Gadget Performance

If you have ever published an application on the Android Market Google Play Store, you are familiar with the Android developer console. Besides letting you publish and update your apps, it also shows the number of total and active installs (notoriously broken and not too be taken too seriously, though it's been getting better lately), ratings and comments. Depending on how excited about the whole app publishing business you are, you might want to check it quite often to see how your app is doing, or maybe you just like hitting F5. Most people don't however, so pretty much every developer at some point comes up with the heretic idea that there must be a better way: you should be able to check your app's statistics on your Android device (obviously!), you should get notified about changes automatically and maybe even be able to easily see if today's numbers are better than yesterday's at a glance. Writing such a tool should be fairly easy, so you start looking for an API. If your search ends up empty it's not your search engine's fault: there is none! So before you start scraping those pretty Web pages with your favourite P-language, you check if someone has done this before -- you might get a few hits, and if you are lucky even find the Android app.

Originally developed by Timelappse, and now open source, Andlytics does all the things mentioned above, and more (and if you need yet another feature, consider contributing). So how does it manage to do all of this without an API? Through blood, sweat and a lot of protocol reversing guessing. You see, the current developer console is built on GWT which used to be Google's webstack-du-jour a few years back. GWT essentially consists of RPC endpoints at the server, called by a JavaScript client running in the browser. The serialization protocol in between is a custom one, and the specification is purposefully not publicly available (apparently, to allow for easier changes!?!). It has two main features: you need to know exactly how the transferred objects look like to be able to make any sense of it, and it was obviously designed by someone who used to write compilers for a living before they got into Web development ('string table' ring a bell?). Given the above, Andlytics was quite an accomplishment. Additionally, the developer console changing its protocol every other week and adding new features from time to time didn't really make it any easier to maintain. Eventually, the original developer had a bit too much GWT on his plate, and was kind enough to open source it, so others could share the pain.

But there is a bright side to all this: Developer Console v2. It was announced at this year's Google I/O to much applause, but was only made universally available a couple of weeks ago (sound familiar?). It is a work in progress, but is showing promise. And the best part: it uses perfectly readable (if a bit heavy on null's) JSON to transport data! Naturally, there was much rejoicing at the Andlytics Github project. It was unanimously decided that the sooner we obliterate all traces of GWT, the better, and the next version should use the v2 console 'API'. Deciphering the protocol didn't take long, but it turned out that while to log in to the v1 console all you needed was a ClientLogin (see the next section for an explanation) token straight out of Android's AccountManger, the new one was not so forgiving and the login flow was somewhat more complex. Asking the user for their password and using it to login was obviously doable, but no one would like that, so we needed to figure out how to log in using the Google credentials already cached on the device. Android browser and Chrome are able to automatically log you in to the developer console without requiring your password, so it was clearly possible. The process is not really documented though, and that prompted this (maybe a bit too wide-cast) investigation. Which finally leads us to the topic of this post: to show how to use cached Google account credentials for single sign-on. Let's first see what standard ways are available to authenticate to Google's public services and API's.

Google services authentication and authorization

The official place to start when selecting an auth mechanism is the Google Accounts Authentication and Authorization page. It lists quite a few protocols, some open and some proprietary. If you research further you will find that currently all but OAuth 2.0 and Open ID are considered deprecated, and using the proprietary ones is not recommended. However, a lot of services are still using older, proprietary protocols, so we will look into some of those as well. Most protocols also have two variations: one for Web applications and one for the so called, 'installed applications'. Web applications run in a browser, and are expected to be able to take advantage of all standard browser features: rich UI, free-form user interaction, cookie store and ability to follow redirects. Installed applications, on the other hand, don't have a native way to preserve session information, and may not have the full Web capabilities of a browser. Android native applications (mostly) fall in the 'installed applications' category, so let's see what protocols are available for them.

ClientLogin

The oldest and most widely used till now authorization protocol for installed applications is ClientLogin. It assumes the application has access to the user's account name and password and lets you get an authorization token for a particular service, that can be saved and used for accessing that service on behalf of the user. Services are identified by proprietary service names, for example 'cl' for Google Calendar and 'ah' for Google App engine. A (non-exhaustive) list of supported service names can be found in the Google Data API reference. Here are a few Android-specific ones, not listed in the reference: 'ac2dm', 'android', 'androidsecure', 'androiddeveloper', 'androidmarket' and 'youngandroid' (probably for the discontinued App Inventor). The token can be fairly long-lived (up to two weeks), but cannot be refreshed and the application needs to obtain a new token when it expires. Additionally, there is no way to validate the token short of accessing the associated service: if you get an OK HTTP status (200), it is still valid, if 403 is returned you need to consult the additional error code and retry or get a new token. Another limitation is that ClientLogin tokens don't offer fine grained access to a service's resources: access is all or nothing, you cannot specify read-only access or access to a particular resource only. The biggest drawback for use in mobile apps though is that ClientLogin requires access to the actual user password. Therefore, if you don't want to force users to enter it each time a new token is required, it needs to be saved on the device, which poses various problems. As we saw in the previous post, in Android this is handled by GLS and the associated online service by storing an encrypted password or a master token on the device. Getting a token is as simple as calling the appropriate AccountManger method, which either returns a cached token or issues an API request to fetch a fresh one. Despite it's many limitations, the protocol is easy to understand and straightforward to implement, so it has been widely used. It has been officially deprecated since April 2012 though, and apps using it are encouraged to migrate to OAuth 2.0, but this hasn't quite happened yet. 

No one likes OAuth 1.0 (except Twitter) and AuthSub is not quite suited for native applications, so we will only look at the currently recommended OAuth 2.0 protocol. OAuth 2.0 has been in the works for quite some time, but it only recently became an official Internet standard. It defines different authorization 'flows', aimed at different use cases, but we will not try to present all of them here. If you are unfamiliar with the protocol, refer to one of the multiple posts that aim to explain it at a higher level, or just read the RFC if you need the details.  And, of course, you can watch for this for a slightly different point of view. We will only discuss how OAuth 2.0 relates to native mobile applications.

The OAuth 2.0 specification defines 4 basic flows for getting an authorization token for a resource, and the two ones that don't require the client (in our scenario an Android app) to directly handle user credentials (Google account user name and password), namely the authorization code grant flow and the implicit grant flow, both have a common step that needs user interaction. They both require the authorization server (Google's) to authenticate the resource owner (the user of the our Android app) and establish whether they grant or deny the access request for the specified scope (e.g., read-only access to profile information). In a typical Web application that runs in a browser, this is very straightforward to do: the user is redirected to an authentication page, then to a access grant page that basically says 'Do you allow app X to access data Y and Z?', and if they agree, another redirect, which includes an authorization token, takes them back to the original application. The browser simply needs to pass on the token in the next request to gain access to the target resource. Here's an official Google example that uses the implicit flow: follow this link and grant access as requested to let the demo Web app display your Google profile information. With a native app things are not that simple. It can either

• use the system browser to handle the permission grant step, which would typically involve the following steps:
• launch the system browser and hope that the user will finish the authentication and permission grant process
• detect success or failure and extract the authorization token from the browser on success (from the window title, redirect URL or the cookie store)
• ensure that after granting access, the user ends up back in your app
• finally, save the token locally and use it to issue the intended Web API request
• embed a WebView or a similar control in the apps's UI. Getting a token would generally involve these steps:
• in the app's UI, instruct the user what to do and load the login/authorization page
• register for a 'page loaded' callback, and check for the final success URL each time it's called
• when found, extract the token from the redirect URL or the WebView's cookie jar and save it locally
• finally use the token to send the intended API request

Neither is ideal, both are confusing to the user and to implement the first one on Android you might event have to (temporarily) start a Web server (redirect_uri is set to http://localhost in the API console, so you can't just use a custom scheme). The second one is generally preferable, if not pretty: here's an (somewhat outdated) overview of what needs to be done and a more recent example with full source code. This integration complexity and UI impedance mismatch are the problems that OAuth 2.0 support via the AccountManager initially, and recently Google Play Services aim to solve. When using either of those, user authentication is implemented transparently by passing the saved master token (or encrypted password) to the server side component, and instead of a WebView with a permission grant page, you get the Android native access grant dialog. If you approve, a second request is sent to convey this and the returned access token is directly delivered to the requesting app. This is essentially the same flow as for Web applications, but has the advantages that it doesn't require context switching from native to browser and back, and is much more user friendly. Of course, it only works for Google accounts, so if you wanted to write, say, a Facebook client, you still have to use a WebView to process the access permission grant and get an authorization token.

Now that we have an idea what authentication methods are available, let's see if we can use them to access an online Google service that doesn't have a dedicated API.

Google Web properties single sign-on

Logging in using AccountManager

The only real documentation we could find, besides code comments and READMEs of the unofficial Google client applications mentioned above, is a short Chromium OS design document. It tells us that the standard (at the time) login API for installed applications, ClientLogin, alone is not enough to accomplish Web SSO, and outlines a three step process that lets us exchange ClientLogin tokens for session cookies valid for a particular service:

1. Get a ClientLogin token (this we can do via the AccountManager)
2. Pass it to https://www.google.com/accounts/IssueAuthToken, to get a one-time use, short-lived token that will authenticate the user to any service (the so called, 'ubertoken')
3. Finally, pass the ubertoken to https://www.google.com/accounts/TokenAuth, to exchange it for the full set of browser cookies we need to do SSO

This outlines the process, but is a little light on the details. Fortunately, those can be found in the Chromium OS source code, as well as a few other projects. After a fair bit of digging, here's what we uncovered:

The straightforward way would be to simply implement the flow outlined above using your favourite HTTP client library. We choose to use Apache HttpClient, which supports session cookies and multiple requests using a single instance out of the box. The first step calls for the SID and LSID cookies though, not an authentication token: we need cookies to get a token, in order to get more cookies. Since Android's AccountManager can only give us authentication tokens, and not cookies, this might seem like a hopeless catch-22 situation. However, while browsing the authtokens table of the system's accounts database earlier, we happened to notice that it actually had a bunch of tokens with type SID and LSID. Our next step is, of course, to try to request those tokens via the AccountManager interface, and this happens to work as expected:

String sid = am.getAuthToken(account, "SID", null, activity, null, null)
    .getResult().getString(AccountManager.KEY_AUTHTOKEN);
String lsid = am.getAuthToken(account, "LSID", null, activity, null, null)
    .getResult().getString(AccountManager.KEY_AUTHTOKEN);

Having gotten those, the rest is just a matter of issuing two HTTP requests (error handling omitted for brevity):

String TARGET_URL = "https://play.google.com/apps/publish/v2/";
Uri ISSUE_AUTH_TOKEN_URL = 
 Uri.parse("https://www.google.com/accounts/IssueAuthToken?service=gaia&Session=false");
Uri TOKEN_AUTH_URL = Uri.parse("https://www.google.com/accounts/TokenAuth");

String url = ISSUE_AUTH_TOKEN_URL.buildUpon().appendQueryParameter("SID", sid)
     .appendQueryParameter("LSID", lsid)
     .build().toString();
HttpPost getUberToken = new HttpPost(url);
HttpResponse response = httpClient.execute(getUberToken);
String uberToken = EntityUtils.toString(entity, "UTF-8");
String getCookiesUrl = TOKEN_AUTH_URL.buildUpon()
     .appendQueryParameter("source", "android-browser")
     .appendQueryParameter("auth", authToken)
     .appendQueryParameter("continue", TARGET_URL)
     .build().toString();
HttpGet getCookies = new HttpGet(getCookiesUrl);
response = httpClient.execute(getCookies);

CookieStore cookieStore = httpClient.getCookieStore();
// check for service-specific session cookie
String adCookie = findCookie(cookieStore.getCookies(), "AD");
// fail if not found, otherwise get page content
String responseStr = EntityUtils.toString(entity, "UTF-8");

This lets us authenticate to the Android Developer Console (version 2) site without requiring user credentials and we can easily proceed to parse the result and use it in a native app (warning: work in progress!) from here. The downside is that for this to work, the user has to grant access twice, for two cryptically looking token types (SID and LSID).

Of course, after writing all of this, it turns out that the stock Android browser already has code that does it, which we could have used or at least referenced from the very beginning. Better yet, this find leads us to an yet easier way to accomplish our task. 

The easy way is found right next to the Browser class referenced above, in the DeviceAccountLogin class, so we can't really take any credit for this. It is hardly anything new, but some Googling suggests that it is neither widely known nor used much. You might have noticed that the Android browser is able to silently log you in to Gmail and friends, when you use the mobile site. The way this is implemented is via the 'magic' token type 'weblogin:'. If you use it along with the service name and URL of the site you want to access, it will do all of the steps listed above automatically and instead of a token will give you a full URL you can load to get automatically logged in to your target service. This magic URL is in the format shown below, and includes both the ubertoken and the URL of the target site, as well as the service name (this example is for the Android Developer Console, line is broken for readability):

https://accounts.google.com/MergeSession?args=service%3Dandroiddeveloper%26continue
%3Dhttps://play.google.com/apps/publish/v2/&uberauth=APh...&source=AndroidWebLogin

Here's how to get the MergeSession URL:

String tokenType = "weblogin:service=androiddeveloper&"
+ "continue=https://play.google.com/apps/publish/v2/";
String loginUrl = accountManager.getAuthToken(account,tokenType, false, null, null)
                   .getResult().getString(AccountManager.KEY_AUTHTOKEN);

This is again for the Developer Console, but works for any Google site, including Gmail, Calendar and even the account management page. The only problem you might have is finding the service name, which is hardly obvious in some cases (e.g., 'grandcentral' for Google Voice and 'lh2' for Picasa).

It takes only a single HTTP request form Android to get the final URL, which tells us that the token issuing flow is implemented on the server side. This means that you can also use the Google Play Services client library to issue a weblogin: 'token' (see screenshot below and note that unlike for OAuth 2.0 scopes, it shows the 'raw' token type). Probably goes without saying, but it also means that if you happen to come across someone's accounts.db file, all it takes to log in into their Google account(s) is two HTTPS requests: one to get the MergeSession URL, and one to log in to their accounts page. If you are thinking 'This doesn't affect me, I use Google two-factor authentication (2FA)!', you should know that in this case 2FA doesn't really help. Why? Because since Android doesn't support 2FA, to register an account with the AccountManager you need to use an application specific password (Update: On ICS and later, GLS will actually show a WebView and let you authenticate using your password and OTP. However, the OTP is not required once you get the master token). And once you have entered one, any tokens issued based on it, will just work (until you revoke it), without requiring entering an additional code. So if you value your account, keep your master tokens close and revoke them as soon as you suspect that your phone might be lost or stolen. Better yet, consider a solution that lets you wipe it remotely (which might not work after your revoke the tokens, so be sure to check how it works before you actually need it).


As we mentioned above, this is all ClientLogin based, which is officially deprecated, and might be going away soon (EOL scheduled for April 2013). But some of the Android Google data sync feeds still depend on ClientLogin, so if you use it you would probably OK for a while. Additionally, since the weblogin: implementation is server-based, it might be updated to conform with the latest (OAuth 2.0-based?) infrastructure without changing the client-side interface. In any case, watch the Android Browser and Chormium code to keep up to date.

Summary

Google offers multiple online services, some with both a traditional browser-based interface and a developer-oriented API. Consequently, there are multiple ways to authenticate to those, ranging from form-based username and password login to authentication API's such as ClientLogin and OAuth 2.0. It is relatively straightforward to get an authentication token for services with a public API on Android, either using Android's native AccountManager interface or the newer Google Play Services extension. Getting the required session cookies to login automatically to the Web sites of services that do not offer an API is however neither obvious, nor documented. Fortunately, it is possible and very easy to do if you combine the special 'weblogin:' token type with the service name and the URL of the site you want to use. The best available documentation about this is the Android Browser source code, which uses the same techniques to automatically log you in to Google sites using the account(s) already registered on your device.

Moral of the story: interoperability is so much easier when you control all parties involved.