Here’s my understanding of what went wrong:

• Google’s API serves down the UTF8 bytes for “Dorfstraße”: 0×44, 0x6f, 0×72, 0×66, 0×73, 0×74, 0×72, 0×61, 0xc3, 0x9f, 0×65
• WebClient Encoding uses the system default. In my case, Windows-1252. If I had set this property to UTF8 I wouldn’t have these Mojibake files.
• In UTF8 the ß character is represented with the bytes 0xc3, 0x9f
• WebClient interprets these bytes with Windows-1252 gets: “Dorfstraße”
• Saving this with File.WriteAllText “uses UTF-8 encoding without a Byte-Order Mark”. This turns “Dorfstraße” into the bytes: 0×44, 0x6f, 0×72, 0×66, 0×73, 0×74, 0×72, 0×61, 0xc3, 0×83, 0xc5, 0xb8, 0×65
• Open the file without a BOM and you see: “Dorfstraße”

I’ve got the files on my disk. How would I fix it? Here’s the plan to put things in reverse..

1. read in the file bytes and interpret as UTF8 so we are back to “Dorfstraße”
2. get the Western European (Windows) bytes for that string
3. interpret those bytes as UTF8

…the code:

• 0:00 Intro
• 0:43 Quick summary of the services
• 4:29 Using the services
• 5:35 Query Request code demo
• 9:10 EC2 intro
• 11:36 EC2 code demo
• 20:29 S3 intro
• 21:19 S3 / CloudFront code demo
• 29:21 Mechanic Turk Intro
• 31:35 Mechanic Turk code demo

It’s best to watch it full screen on the HD resolution.

I was a bit surprised when I stumbled on this article Two Factor SSH with Google Authenticator. Turns out the algorithm used to generate the OTPs is an open standard. When you set-up an account in the smartphone app you are storing a key that’s used to create a HMAC of the current time.

You can read the specifics of the algorithm in the TOTP RFC Draft.  I really like the idea that you can use the smartphone app to generate OTPs for your own application.  I’ve implemented the algorithm in javascript on jsfiddle.   Javascript is nice and readable, but please don’t implement your verification client side!

History

• 20012-Sept-6: jsSHA moved location
• 20012-Sept-12: Something suspect about the way I’m converting BASE32 to bytes. Changed it to grab full bytes from the binary string, and ignore anything left over.

Another quick blog article just to settle/document some concepts in my own head, and hopefully provide a quick intro to someone out there.

The Facebook Developers site documents all the APIs that you can use to integrate Facebook.  Coming into this with no knowledge of the API can be pretty overwhelming.  This article is a quick working example of authentication and the graph API.  You can read the Facebook references here:  Authentication and Graph API.

The authentication here is the OAuth 2.0, this is the protocol you’d use if you want to write server side code to allow people to log into your site via Facebook, or link existing accounts.  The authentication process also gives you an access token needed to read/write Facebook statuses/photos/etc via the Graph API.

• First create an application here: http://developers.facebook.com/setup/
• You will receive an <App ID> and an <App Secret>
• You can use this to build an ‘authorize’ URL. This is the location you’ll send your users to initiate the authentication, e.g. on a ‘log in with Facebook’ button on your website:
  https://graph.facebook.com/oauth/authorize?client_id=<App ID>&redirect_uri=http://localhost/blah.aspx&scope=offline_access,publish_stream,user_photos,read_stream
  https://www.facebook.com/dialog/oauth?client_id=<App ID>&redirect_uri=http://localhost/blah.aspx&scope=offline_access,publish_stream,user_photos,read_stream
  • redirect_uri – the user will be redirected here after they’ve authorized your application
  • scope – the permissions you are requesting to access, more info: Extended Permissions
  • Check out Dialog Form Factors to display a pop-up or mobile styled authentication page
  • Check out OAuth Dialog‘s display property to display a pop-up or mobile styled authentication page
• The user is displayed a page to authorize the access your application is requesting:
  
• The user ‘allows’ your application and gets redirected back to the location you specified in the authorize URL, with a <code> attached:

  http://localhost/blah.aspx?code=<code>

• Your server now takes the code and exchanges it for an access token, performs a GET against the access_token URL:
  https://graph.facebook.com/oauth/access_token?client_id=<App ID>&redirect_uri=http://localhost/blah.aspx&client_secret=<App Secret>&code=<Code>
• Facebook responds with an access token:
  access_token=<access token>

Your application is now authorized to read/write data with the permissions of the user via the Graph API.  Hit some URLs to get a JSON response from facebook:

• The feed: https://graph.facebook.com/me/feed?access_token=<access token>
• Photo albums: https://graph.facebook.com/me/albums?access_token=<access token>

Even better upload a photo on behalf of the user!  Download cURL for your platform: cURL download.  Run cURL from a command-line:

curl.exe
-F access_token=<access token>
-F source=@IMG_2693.jpg
-F "message=Testing yet again!" https://graph.facebook.com/me/photos


The -F arguments build a form POST, the @ attaches the file as a file upload, don’t forget to escape any pipe characters in your access_token with a caret.

So that’s a quick intro to what is possible via the graph and authentication APIs. I see plenty of other APIs on the developer site I haven’t tinkered with yet. If you see anything interesting leave me a comment, and I might pull it apart in a new blog post!

Creating some tasks is very simple, you just create a HTML template with placeholders like this: ${placeholder}, and some HTML elements. Upload a CSV to ‘mail merge’ with the template, and workers get displayed 1 task for each row in the CSV. Any data entered into the form fields are saved and available for download.

I came up with an idea for mturk tasks while I was testing an ipad magazine app at work. The app contains bitmaps exported for the PDF files to create the magazines, so there is no metadata to create hotspots on the index page. The idea: put the index pages up on mturk for workers to draw the hotspots. The workers only interact with the HTML in the template, so it is an option to write some javascript to draw the rectangle and store the co-ordinates in hidden form fields.

I want to take 1 index page, and get a rectangle for each page reference. I’m not really using mturk to its fullest if I give 1 worker a JPG, and have the worker draw multiple rectangles. It would be better to “scale” the task over multiple workers. So instead I type all the page numbers into the CSV (this could also be another task!), and each “task” displays the worker a single page number, and asks them to draw the rectangle. Slight problem, what if a page number is listed more than once? Now I have three columns in my csv:

• pageNumber – the page number I want the workers to find
• pageCount – the number of times the page appears
• pageUrl – the location of the scanned page

If a page appears multiple times I give the worker multiple rectangles to highlight the page.

So what’s it look like? The ‘Design’ tab is used to set up the template

I upload the CSV, and the task appears for workers to complete.

I get a UI of the results in real time, the tasks complete pretty quickly. I can click the results button and see a grid of the results. You can also reject the results you see in the grid, and put the task back to the workers for completion.

See the rectangles drawn on the index page:

There is a bit of an overlap on the rectangles, so I wrote a bit of code to ‘split the difference’ of any overlap. See the results here.

Further reading:

1. Creating a HIT Template – a good reference from Amazon on the workings of a HIT created in the UI.
2. API reference – it is possible to host the IFRAME yourself using an ExternalQuestion

List<TransitionOffset> transitions = new List<TransitionOffset>();
foreach (TimeZoneInfo.AdjustmentRule adj in TimeZoneInfo.FindSystemTimeZoneById("AUS Eastern Standard Time").GetAdjustmentRules())
{
    if (adj.DateEnd.Year >= 2010)
    {
        for (int year = 2010; year <= 2015; year++)
        {
            transitions.Add(new TransitionOffset
            {
                Transition = GetDateTime(year, adj.DaylightTransitionStart),
                Offset = adj.DaylightDelta.TotalHours
            });
            transitions.Add(new TransitionOffset
            {
                Transition = GetDateTime(year, adj.DaylightTransitionEnd),
                Offset = 0
            });

        }
    }
}
transitions = transitions.OrderBy(t => t.Transition).ToList();

..and again in LINQ

var zoneAdjs = from adj in TimeZoneInfo.FindSystemTimeZoneById("AUS Eastern Standard Time").GetAdjustmentRules()
                where adj.DateEnd.Year >= 2010
                from year in Enumerable.Range(2010, 6)
                let trans = new[] { 
                                        new { Transition = GetDateTime(year, adj.DaylightTransitionEnd), 
                                            Offset = 0.0},
                                        new { Transition = GetDateTime(year, adj.DaylightTransitionStart),
                                        Offset = adj.DaylightDelta.TotalHours}
                                    }
                from tran in trans
                orderby tran.Transition
                select tran;

Any more readable / maintainable than the non-LINQ version? It definitely took longer to write while I’m a LINQ “newbie”. The most complicated bit was pulling the DaylightTransitionEnd and DaylightTransitionStart into one collection. This collection is only ever used to create JSON, so I like being able to serialize anonymous types from LINQ, and skip creating classes that are only used to serialize.

The idea was to pull together some visualization of the twitter sample feed.   A work colleague (spud) suggested a heatmap where the points fade out over a few frames.  And here it is: http://twitterheatmap.com/

I’ve got an EXE running on an EC2 micro instance dropping XML files of all the tweets into a folder.  Another EXE fires once a day at midnight UTC, generates the JPGs, creates an AVI (with VirtualDub), and uploads the video to youtube.   The data you see live right now is a bit flakey – displaying a fail whale where I don’t have any data.

Grabbed a lot of handy code off the net to build this:

• Day and Night Map – code to draw the day / night terminator
• Creating Heat Maps with .NET 2.0 (C#)
• jQuery youtube playlist plugin – youtubeplaylist | Gecko New Media Blog

Google released a SDK 1.3.6 for App Engine yesterday: Google App Engine Blog: Multi-tenancy Support, High Performance Image Serving, Increased Datastore Quotas and More Delivered In New App Engine Release. Of particular interest to me was the High Performance Image Serving.

High-Performance Image Serving This release also includes a new, high-performance image serving system for your applications, based on the same infrastructure we use to serve images for Picasa. This feature allows you to generate a stable, dedicated URL for serving web-suitable image thumbnails … This special URL can serve that image resized and/or cropped automatically, and serving from this URL does not incur any CPU or dynamic serving load on your application (though bandwidth is still charged as usual).

So I store some images in an App Engine blob, pass this to get_serving_url and Google give me back a URL for the ‘high performance’ location. I was just interested to see what the URL would look like, and where it would actually be hosted. So I whipped up something real quick to grab the URL ..and here’s what it looks like:

http://lh5.ggpht.com/FIl4UGix2IbadeLOotXq2CgIF4Wi4xUSaSegSDSrfnIy5YyQlZWWaPcDRhyxkJTGepVkxNHUVcv8o1_1zWazbg=s200

Note this URL has the “=s200″ parameter to scale the image down to 200px wide. You can also pass along a crop parameter to crop the image down to a small square, e.g: =s200-c

The code is a hacked up version of the image upload and blob storage examples.

from google.appengine.ext import blobstore
from google.appengine.ext.webapp import blobstore_handlers
from google.appengine.api import images
from google.appengine.api import urlfetch
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app
from google.appengine.ext import db

class ImageModel(db.Model):
    image = db.BlobProperty()
		


class MainPage(webapp.RequestHandler):
	def get(self):
		upload_url = blobstore.create_upload_url('/upload')

		self.response.out.write("""<html><head><title>High Performance Image Tester</title></head>""");
		self.response.out.write("""<body>""");
		self.response.out.write("""
				<form action="%s" enctype="multipart/form-data" method="post">
				<div><label>Image:</label></div>
				<div><input type="file" name="file"/></div>
				<div><input type="submit" value="Upload"></div>
				</form>
			</body>
			</html>""" % upload_url)


class UploadHandler(blobstore_handlers.BlobstoreUploadHandler):
    def post(self):
        upload_files = self.get_uploads('file')  # 'file' is file upload field in the form
        blob_info = upload_files[0]
        self.redirect('/serve/%s' % blob_info.key())

class ServeHandler(webapp.RequestHandler):
	def get(self,resource):
		self.response.out.write("%s %s %s" % (images.get_serving_url(resource, 32),
		images.get_serving_url(resource, 150),
		images.get_serving_url(resource, 80)))

application = webapp.WSGIApplication(
	[('/', MainPage),
	('/upload', UploadHandler),
	('/serve/([^/]+)?', ServeHandler)],

	debug=True)

def main():
	run_wsgi_app(application)

if __name__ == "__main__":
	main()

Same issue as before: host-headers enable you to host several websites on a single IP address. We run into a problem with several SSL certificates on one IP address. The first message a server sends in the SSL handshake contains relevant certificate, the server can only determine which certificate by IP address / port. If there is only one certificate to serve – no need to make a decision!  Here we can serve a UC SSL certificate which is accepted by the web browser for multiple domain names.

I felt like trying this myself, extra curious to know what the certificate will look like in a browser. Managed to get this working in OpenSSL using a combination of two articles:

1. Creating a Self-Signed Certificate using OpenSSL for use with Microsoft Internet Information Services (IIS) 5
2. Creating a Certificate With Multiple Hostnames

Follow the steps for creating a CA in the first article. Grab the CA and import it into your IIS box. Switch over to the second article and add the changes to the openssl.cfg

[ CA_default ]
copy_extensions = copy

[ req ]
req_extensions = v3_req

[ v3_req ]

# Extensions to add to a certificate request
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment

# Some CAs do not yet support subjectAltName in CSRs.
# Instead the additional names are form entries on web
# pages where one requests the certificate...
subjectAltName          = @alt_names

[alt_names]
DNS.1   = stoniessl.info
DNS.2   = freemegahost.info

Create a key for the multi domain certificate. Here we are skipping the “create CSR on IIS” step, and creating the key in OpenSSL.

openssl genrsa -des3 -out keys\multi.key 1024

Create a certificate request

openssl req -new -out requests\multi.txt -key keys\multi.key

Sign the certificate request

openssl.exe ca -policy policy_anything -cert certs\ca.cer -in requests\multi.txt -keyfile keys\ca.key -out multi.cer

Export the certificate AND private key (usually the private key would already be on in IIS box if you created the CSR in IIS)

openssl.exe pkcs12 -export -out multi.pfx -in multi.cer -inkey keys\multi.key

You now have a UC SSL cert you can install on the IIS box. By default IIS won’t allow you to enter a host-header when you are setting the SSL binding. There’s a little trick from this article: Using Host Headers and SLL in IIS 7 – give the certificate a friendly name starting with ‘*’!

And what does the certificate look like? Here it is, in IE and FF:

First thought was wether I should hand over my passwords to the 3rd party, hopefully they are storing them with some serious crypto. Perusing their FAQ I found some info on this:

We only support keeping the encryption done on your computer so LastPass can’t see your sensitive data

…your sensitive data is always encrypted and decrypted locally on your computer before being synchronized. Your master password never leaves your computer and your key never leaves your computer. No one at LastPass (or anywhere else) can decrypt your data without you giving up your password (we will never ask you for it). Your key is created by taking a SHA-256 hash of your password. When you login, we make a hash of your username concatenated with your password, and that hash is what’s sent to verify if you can download your encrypted data.

So they are encrypting everything with a key derived from your password – plus they don’t know your password – so they CANNOT access any of your passwords. Nice. But why stop there? Let’s fire up fiddler to make sure they definitely don’t have my passwords.

I’ve created a junk account on lastpass username: russell.sayers.junk@gmail.com, password: test1234 (the account will be gone by the time you read this!). The first form I see submitted to the server is when I create my account:

I don’t see my password going to their servers in the clear. I can confirm that the hash getting sent is geniune by creating the same hash elsewhere via an online SHA256 generator, or writing some code myself. Try it yourself, the hash created is SHA256(SHA256(username + password) + password) – everything checks out. All the C# code to verify the encryption/hashing is at the end of the article.

The login form:

Again the same hash is being used to verify me when I login. I can see an encrypted username is being sent (although I’m not sure why?), rooting around in the javascript I can see the key being used for encyption is SHA(username+password). Importantly this is different to the hash being used to authenticate me – as we don’t want the server to be able to decypt anything encypted by the client, i.e. they would have to know my password to derive the same key on their side.

The form to add a new website:

We can see here the client is sending encyrypted versions of name, username, password, and the extra info. Again this is encrypted with a key we only have on our client – no one at lastpass could have this key. For some reason the URL is being sent in a HEX representation of the string – not sure why they aren’t just sending the string?

So when I log back into lastpass, I can drill down into this entry and see it again. Let’s make sure everything looks okay here. The HTML that renders this screen is available in fiddler:

<tr>
  <td class='col1'>Name</td>
  <td><input name='name' id='name' type='text' value='pyJlY+AX0Aoczlx50hwlHg==' style='width: 250px'></td>
</tr>
...
<tr>
  <td class='col1'>Username</td>
  <td><input name='username' id='idusername' type='text' value='SgVkuVKH4MjkP+Saz64UhA=='> </td>
</tr>
...
<tr>
  <td class='col1'>Notes</td>
  <td><textarea name='extra' id='extra' rows='6' cols='35'>1rZ3sSuggtdavyCu446GZA==</textarea></td>
</tr> 

The server is sending down our encrypted details, and relying on the client to decrypt everything.

So, yes – you CAN trust lastpass.com with your passwords! Don’t just take my word for it; fire up Fiddler, and compare the hashed/encrypted values with what you expect.

Lastly the c# code to confirm the AES encrypted strings above are geniune.

static void Main(string[] args)
{
    string username = "russell.sayers.junk@gmail.com";
    string password = "test1234";
    string hash_auth = ByteArrayToHexString(SHA256(ByteArrayToHexString(SHA256(username+password)) + password));
    byte[] hash_key = SHA256(username + password);

    Console.WriteLine("hash for authentication => " + hash_auth);
    Console.WriteLine("hash for encryption => " +  ByteArrayToHexString(hash_key));
    Console.WriteLine("'{0}' encrypted => {1}", username, Encrypt(username, hash_key, ""));
    Console.WriteLine("'{0}' encrypted => {1}", "facebook", Encrypt("facebook", hash_key, ""));
    Console.WriteLine("'{0}' encrypted => {1}", "cryptolearner", Encrypt("cryptolearner", hash_key, ""));
    Console.WriteLine("'{0}' encrypted => {1}", "password", Encrypt("password", hash_key, ""));
    Console.WriteLine("'{0}' encrypted => {1}", "no notes", Encrypt("no notes", hash_key, ""));
}

static byte[] SHA256(string data)
{
    byte[] indata = Encoding.UTF8.GetBytes(data);
    SHA256 shaM = new SHA256Managed();
    return shaM.ComputeHash(indata);
}

/// <remarks>From http://social.msdn.microsoft.com/Forums/en-US/csharpgeneral/thread/3928b8cb-3703-4672-8ccd-33718148d1e3</remarks>
static string ByteArrayToHexString(byte[] data)
{
    StringBuilder sb = new StringBuilder(data.Length * 2);
    foreach (byte b in data)
    {
        sb.AppendFormat("{0:x2}", b);
    }
    return sb.ToString();
}

/// <remarks>From http://stackoverflow.com/questions/1079131/c-aes-256-encryption</remarks>
static public string Encrypt(string plaintext, byte[] KeyBytes, string InitialVector)
{
    byte[] PlainTextBytes = Encoding.UTF8.GetBytes(plaintext);
    byte[] InitialVectorBytes = Encoding.ASCII.GetBytes(InitialVector);
    RijndaelManaged SymmetricKey = new RijndaelManaged();
    SymmetricKey.Mode = CipherMode.ECB;
    SymmetricKey.Padding = PaddingMode.PKCS7;
    ICryptoTransform Encryptor = SymmetricKey.CreateEncryptor(KeyBytes, InitialVectorBytes);
    MemoryStream MemStream = new MemoryStream();
    CryptoStream CryptoStream = new CryptoStream(MemStream, Encryptor, CryptoStreamMode.Write);
    CryptoStream.Write(PlainTextBytes, 0, PlainTextBytes.Length);
    CryptoStream.FlushFinalBlock();
    byte[] CipherTextBytes = MemStream.ToArray();
    MemStream.Close();
    CryptoStream.Close();
    return Convert.ToBase64String(CipherTextBytes);
}

]]> http://blog.tinisles.com/2010/01/should-you-trust-lastpass-com/feed/ 8