Single sign-on to Google sites using AccountManager

In the first part of this series, we presented how the standard Android online account management framework works and explored how Google account authentication and authorization modules are implemented on Android. In this article we will see how to use the Google credentials stored on the device to log in to Google Web sites automatically. Note that this is different from using public Google API's, which generally only requires putting an authentication token (and possibly an API key) in a request header, and is quite well supported by the Google APIs Client Library. First, some words on what motivated this whole exercise (may include some ranting, feel free to skip to the next section).

Android developer console API: DIY

If you have ever published an application on the Android Market Google Play Store, you are familiar with the Android developer console. Besides letting you publish and update your apps, it also shows the number of total and active installs (notoriously broken and not too be taken too seriously, though it's been getting better lately), ratings and comments. Depending on how excited about the whole app publishing business you are, you might want to check it quite often to see how your app is doing, or maybe you just like hitting F5. Most people don't however, so pretty much every developer at some point comes up with the heretic idea that there must be a better way: you should be able to check your app's statistics on your Android device (obviously!), you should get notified about changes automatically and maybe even be able to easily see if today's numbers are better than yesterday's at a glance. Writing such a tool should be fairly easy, so you start looking for an API. If your search ends up empty it's not your search engine's fault: there is none! So before you start scraping those pretty Web pages with your favourite P-language, you check if someone has done this before -- you might get a few hits, and if you are lucky even find the Android app.

Originally developed by Timelappse, and now open source, Andlytics does all the things mentioned above, and more (and if you need yet another feature, consider contributing). So how does it manage to do all of this without an API? Through blood, sweat and a lot of protocol reversing guessing. You see, the current developer console is built on GWT which used to be Google's webstack-du-jour a few years back. GWT essentially consists of RPC endpoints at the server, called by a JavaScript client running in the browser. The serialization protocol in between is a custom one, and the specification is purposefully not publicly available (apparently, to allow for easier changes!?!). It has two main features: you need to know exactly how the transferred objects look like to be able to make any sense of it, and it was obviously designed by someone who used to write compilers for a living before they got into Web development ('string table' ring a bell?). Given the above, Andlytics was quite an accomplishment. Additionally, the developer console changing its protocol every other week and adding new features from time to time didn't really make it any easier to maintain. Eventually, the original developer had a bit too much GWT on his plate, and was kind enough to open source it, so others could share the pain.

But there is a bright side to all this: Developer Console v2. It was announced at this year's Google I/O to much applause, but was only made universally available a couple of weeks ago (sound familiar?). It is a work in progress, but is showing promise. And the best part: it uses perfectly readable (if a bit heavy on null's) JSON to transport data! Naturally, there was much rejoicing at the Andlytics Github project. It was unanimously decided that the sooner we obliterate all traces of GWT, the better, and the next version should use the v2 console 'API'. Deciphering the protocol didn't take long, but it turned out that while to log in to the v1 console all you needed was a ClientLogin (see the next section for an explanation) token straight out of Android's AccountManger, the new one was not so forgiving and the login flow was somewhat more complex. Asking the user for their password and using it to login was obviously doable, but no one would like that, so we needed to figure out how to log in using the Google credentials already cached on the device. Android browser and Chrome are able to automatically log you in to the developer console without requiring your password, so it was clearly possible. The process is not really documented though, and that prompted this (maybe a bit too wide-cast) investigation. Which finally leads us to the topic of this post: to show how to use cached Google account credentials for single sign-on. Let's first see what standard ways are available to authenticate to Google's public services and API's.

Google services authentication and authorization

The official place to start when selecting an auth mechanism is the Google Accounts Authentication and Authorization page. It lists quite a few protocols, some open and some proprietary. If you research further you will find that currently all but OAuth 2.0 and Open ID are considered deprecated, and using the proprietary ones is not recommended. However, a lot of services are still using older, proprietary protocols, so we will look into some of those as well. Most protocols also have two variations: one for Web applications and one for the so called, 'installed applications'. Web applications run in a browser, and are expected to be able to take advantage of all standard browser features: rich UI, free-form user interaction, cookie store and ability to follow redirects. Installed applications, on the other hand, don't have a native way to preserve session information, and may not have the full Web capabilities of a browser. Android native applications (mostly) fall in the 'installed applications' category, so let's see what protocols are available for them.

ClientLogin

The oldest and most widely used till now authorization protocol for installed applications is ClientLogin. It assumes the application has access to the user's account name and password and lets you get an authorization token for a particular service, that can be saved and used for accessing that service on behalf of the user. Services are identified by proprietary service names, for example 'cl' for Google Calendar and 'ah' for Google App engine. A (non-exhaustive) list of supported service names can be found in the Google Data API reference. Here are a few Android-specific ones, not listed in the reference: 'ac2dm', 'android', 'androidsecure', 'androiddeveloper', 'androidmarket' and 'youngandroid' (probably for the discontinued App Inventor). The token can be fairly long-lived (up to two weeks), but cannot be refreshed and the application needs to obtain a new token when it expires. Additionally, there is no way to validate the token short of accessing the associated service: if you get an OK HTTP status (200), it is still valid, if 403 is returned you need to consult the additional error code and retry or get a new token. Another limitation is that ClientLogin tokens don't offer fine grained access to a service's resources: access is all or nothing, you cannot specify read-only access or access to a particular resource only. The biggest drawback for use in mobile apps though is that ClientLogin requires access to the actual user password. Therefore, if you don't want to force users to enter it each time a new token is required, it needs to be saved on the device, which poses various problems. As we saw in the previous post, in Android this is handled by GLS and the associated online service by storing an encrypted password or a master token on the device. Getting a token is as simple as calling the appropriate AccountManger method, which either returns a cached token or issues an API request to fetch a fresh one. Despite it's many limitations, the protocol is easy to understand and straightforward to implement, so it has been widely used. It has been officially deprecated since April 2012 though, and apps using it are encouraged to migrate to OAuth 2.0, but this hasn't quite happened yet. 

OAuth 2.0

No one likes OAuth 1.0 (except Twitter) and AuthSub is not quite suited for native applications, so we will only look at the currently recommended OAuth 2.0 protocol. OAuth 2.0 has been in the works for quite some time, but it only recently became an official Internet standard. It defines different authorization 'flows', aimed at different use cases, but we will not try to present all of them here. If you are unfamiliar with the protocol, refer to one of the multiple posts that aim to explain it at a higher level, or just read the RFC if you need the details.  And, of course, you can watch for this for a slightly different point of view. We will only discuss how OAuth 2.0 relates to native mobile applications.

The OAuth 2.0 specification defines 4 basic flows for getting an authorization token for a resource, and the two ones that don't require the client (in our scenario an Android app) to directly handle user credentials (Google account user name and password), namely the authorization code grant flow and the implicit grant flow, both have a common step that needs user interaction. They both require the authorization server (Google's) to authenticate the resource owner (the user of the our Android app) and establish whether they grant or deny the access request for the specified scope (e.g., read-only access to profile information). In a typical Web application that runs in a browser, this is very straightforward to do: the user is redirected to an authentication page, then to a access grant page that basically says 'Do you allow app X to access data Y and Z?', and if they agree, another redirect, which includes an authorization token, takes them back to the original application. The browser simply needs to pass on the token in the next request to gain access to the target resource. Here's an official Google example that uses the implicit flow: follow this link and grant access as requested to let the demo Web app display your Google profile information. With a native app things are not that simple. It can either
  • use the system browser to handle the permission grant step, which would typically involve the following steps:
    • launch the system browser and hope that the user will finish the authentication and permission grant process
    • detect success or failure and extract the authorization token from the browser on success (from the window title, redirect URL or the cookie store)
    • ensure that after granting access, the user ends up back in your app
    • finally, save the token locally and use it to issue the intended Web API request
  • embed a WebView or a similar control in the apps's UI. Getting a token would generally involve these steps:
    • in the app's UI, instruct the user what to do and load the login/authorization page
    • register for a 'page loaded' callback, and check for the final success URL each time it's called
    • when found, extract the token from the redirect URL or the WebView's cookie jar and save it locally
    • finally use the token to send the intended API request
Neither is ideal, both are confusing to the user and to implement the first one on Android you might event have to (temporarily) start a Web server (redirect_uri is set to http://localhost in the API console, so you can't just use a custom scheme). The second one is generally preferable, if not pretty: here's an (somewhat outdated) overview of what needs to be done and a more recent example with full source code. This integration complexity and UI impedance mismatch are the problems that OAuth 2.0 support via the AccountManager initially, and recently Google Play Services aim to solve. When using either of those, user authentication is implemented transparently by passing the saved master token (or encrypted password) to the server side component, and instead of a WebView with a permission grant page, you get the Android native access grant dialog. If you approve, a second request is sent to convey this and the returned access token is directly delivered to the requesting app. This is essentially the same flow as for Web applications, but has the advantages that it doesn't require context switching from native to browser and back, and is much more user friendly. Of course, it only works for Google accounts, so if you wanted to write, say, a Facebook client, you still have to use a WebView to process the access permission grant and get an authorization token.

Now that we have an idea what authentication methods are available, let's see if we can use them to access an online Google service that doesn't have a dedicated API.

Google Web properties single sign-on

Being able to access multiple related, but separate services without needing to authenticate to each one individually is generally referred to as single sign-on (SSO). There are multiple standard ways to accomplish this for different contexts, ranging from Kerberos to SAML-based solutions. We will use the term here in a narrower meaning: being able to use different Google services (Web sites or API's) after having authenticated to only one of them (including the Android login service). If you have a fairly fast Internet connection, you might not even notice it, but after you log in to, say, Gmail, clicking on YouTube links will take you to a completely different domain, and yet you will be able to comment on that neat cat video without having to log in again. If you have a somewhat slower connection and a wide display though, you may notice that there is a lot of redirecting and long parameter passing, with the occasional progress bar going on. What happens behind the scenes is that your current session cookies and authentication tokens are being exchanged for yet other tokens and more cookies, to let you seamlessly log in to that other site. If you are curious, you can observe the flow with Chrome's built-in developer tools (or similar plugins for other browsers), or check out our sample. All of those requests and responses are essentially a proprietary SSO protocol (Google's), which is not really publicly documented anywhere, and, of course, is likely to change fairly often as Google rolls out upgrades to their services. With that said, there is a distinct pattern, and on a higher level you only have two main cases. We are deliberately ignoring the persistent cookie ('Stay signed in')  scenario for simplicity's sake.
  • Case 1: you haven't authenticated to any of the Google properties. If you access, for example, mail.google.com in that state you will get a login screen originating at https://accounts.google.com/ServiceLogin with parameters specifying the service you are trying to access ('mail' for Gmail) and where to send you after you are authenticated. After you enter your credentials, you will generally get redirected a few times around the accounts.google.com, which will set a few session cookies, common (Domain=.google.com) for all services (always SID and LSID, plus a few more). The last redirect will be to the originally requested service and include an authentication token in the redirected location (usually specified with the auth parameter, e.g.: https://mail.google.com/mail/?auth=DQAAA...). The target service will validate the token and set a few more service-specific sessions cookies, restricted by domain and path, and with the Secure and HttpOnly flags set. From there, it might take a couple of more redirects before you finally land at an actual content page.
  • Case 2: you have already authenticated to at least one service (Gmail in our example). In this state, if you open, say, Calendar, you will go through https://accounts.google.com/ServiceLogin again, but this time the login screen won't be shown. The accounts service will modify your SID and LSID cookies, maybe set a few new ones and finally redirect you the original service, adding an authentication token to the redirect location. From there the process is similar: one or more service-specific cookies will be set and you will finally be redirected to the target content.
Those flows obviously work well for browser-based logins, but since we are trying to do this from an Android app, without requiring user credentials or showing WebView's, we have a different scenario. We can easily get a ClientLogin or an OAuth 2.0 token from the AccountManager, but since we are not preforming an actual Web login, we have no cookies to present. The question becomes: is there a way to log in with a standard token alone? Since tokens can be used with the data APIs (where available) of each service, they obviously contain enough information to authenticate us and grant access to the service's resources. What we need is an Web endpoint, that will take our token and give us a set of cookies we could use to access the corresponding Web site in exchange. Clues and traces of such a service are scattered around the Internet, mostly in the code of unofficial Google client libraries and applications. Once we know it is definitely possible, the next problem becomes getting it to work with Android's AccountManger.

Logging in using AccountManager

The only real documentation we could find, besides code comments and READMEs of the unofficial Google client applications mentioned above, is a short Chromium OS design document. It tells us that the standard (at the time) login API for installed applications, ClientLogin, alone is not enough to accomplish Web SSO, and outlines a three step process that lets us exchange ClientLogin tokens for session cookies valid for a particular service:
  1. Get a ClientLogin token (this we can do via the AccountManager)
  2. Pass it to https://www.google.com/accounts/IssueAuthToken, to get a one-time use, short-lived token that will authenticate the user to any service (the so called, 'ubertoken')
  3. Finally, pass the ubertoken to https://www.google.com/accounts/TokenAuth, to exchange it for the full set of browser cookies we need to do SSO
This outlines the process, but is a little light on the details. Fortunately, those can be found in the Chromium OS source code, as well as a few other projects. After a fair bit of digging, here's what we uncovered:
    1. To get the mythical ubertoken, you need to pass the SID and LSID cookies to the IssueAuthToken endpoint like this:
      https://www.google.com/accounts/IssueAuthToken?service=gaia&Session=false&SID=sid&LSID=lsid
    2. The response will give you the ubertoken, which you pass to the TokenAuth endpoint along with the URL of the service you want to use:
      https://www.google.com/accounts/TokenAuth?source=myapp&auth=ubertoken&continue=service-URL
    3. If the token check out OK, the response will give you a URL to load. If your HTTP client is set up to follow redirects automatically, once you load it, needed cookies will be set automatically (just as in a browser), and you will finally land on the target site. As long as you keep the same session (which usually means the same HTTP client instance) you will be able to issue multiple requests, without needing to go through the authentication flow again.
    What remains to be seen is, can we implement this on Android. As usual, it turns out that there is more than one way to do it:

    The hard way

    The straightforward way would be to simply implement the flow outlined above using your favourite HTTP client library. We choose to use Apache HttpClient, which supports session cookies and multiple requests using a single instance out of the box. The first step calls for the SID and LSID cookies though, not an authentication token: we need cookies to get a token, in order to get more cookies. Since Android's AccountManager can only give us authentication tokens, and not cookies, this might seem like a hopeless catch-22 situation. However, while browsing the authtokens table of the system's accounts database earlier, we happened to notice that it actually had a bunch of tokens with type SID and LSID. Our next step is, of course, to try to request those tokens via the AccountManager interface, and this happens to work as expected:

    String sid = am.getAuthToken(account, "SID", null, activity, null, null)
        .getResult().getString(AccountManager.KEY_AUTHTOKEN);
    String lsid = am.getAuthToken(account, "LSID", null, activity, null, null)
        .getResult().getString(AccountManager.KEY_AUTHTOKEN);

    Having gotten those, the rest is just a matter of issuing two HTTP requests (error handling omitted for brevity):

    String TARGET_URL = "https://play.google.com/apps/publish/v2/";
    Uri ISSUE_AUTH_TOKEN_URL = 
     Uri.parse("https://www.google.com/accounts/IssueAuthToken?service=gaia&Session=false");
    Uri TOKEN_AUTH_URL = Uri.parse("https://www.google.com/accounts/TokenAuth");

    String url = ISSUE_AUTH_TOKEN_URL.buildUpon().appendQueryParameter("SID", sid)
         .appendQueryParameter("LSID", lsid)
         .build().toString();
    HttpPost getUberToken = new HttpPost(url);
    HttpResponse response = httpClient.execute(getUberToken);
    String uberToken = EntityUtils.toString(entity, "UTF-8");
    String getCookiesUrl = TOKEN_AUTH_URL.buildUpon()
         .appendQueryParameter("source", "android-browser")
         .appendQueryParameter("auth", authToken)
         .appendQueryParameter("continue", TARGET_URL)
         .build().toString();
    HttpGet getCookies = new HttpGet(getCookiesUrl);
    response = httpClient.execute(getCookies);

    CookieStore cookieStore = httpClient.getCookieStore();
    // check for service-specific session cookie
    String adCookie = findCookie(cookieStore.getCookies(), "AD");
    // fail if not found, otherwise get page content
    String responseStr = EntityUtils.toString(entity, "UTF-8");

    This lets us authenticate to the Android Developer Console (version 2) site without requiring user credentials and we can easily proceed to parse the result and use it in a native app (warning: work in progress!) from here. The downside is that for this to work, the user has to grant access twice, for two cryptically looking token types (SID and LSID).

    Of course, after writing all of this, it turns out that the stock Android browser already has code that does it, which we could have used or at least referenced from the very beginning. Better yet, this find leads us to an yet easier way to accomplish our task. 

    The easy way

    The easy way is found right next to the Browser class referenced above, in the DeviceAccountLogin class, so we can't really take any credit for this. It is hardly anything new, but some Googling suggests that it is neither widely known nor used much. You might have noticed that the Android browser is able to silently log you in to Gmail and friends, when you use the mobile site. The way this is implemented is via the 'magic' token type 'weblogin:'. If you use it along with the service name and URL of the site you want to access, it will do all of the steps listed above automatically and instead of a token will give you a full URL you can load to get automatically logged in to your target service. This magic URL is in the format shown below, and includes both the ubertoken and the URL of the target site, as well as the service name (this example is for the Android Developer Console, line is broken for readability):

    https://accounts.google.com/MergeSession?args=service%3Dandroiddeveloper%26continue
    %3Dhttps://play.google.com/apps/publish/v2/&uberauth=APh...&source=AndroidWebLogin

    Here's how to get the MergeSession URL:

    String tokenType = "weblogin:service=androiddeveloper&"
    + "continue=https://play.google.com/apps/publish/v2/";
    String loginUrl = accountManager.getAuthToken(account,tokenType, false, null, null)
                       .getResult().getString(AccountManager.KEY_AUTHTOKEN);

    This is again for the Developer Console, but works for any Google site, including Gmail, Calendar and even the account management page. The only problem you might have is finding the service name, which is hardly obvious in some cases (e.g., 'grandcentral' for Google Voice and 'lh2' for Picasa).

    It takes only a single HTTP request form Android to get the final URL, which tells us that the token issuing flow is implemented on the server side. This means that you can also use the Google Play Services client library to issue a weblogin: 'token' (see screenshot below and note that unlike for OAuth 2.0 scopes, it shows the 'raw' token type). Probably goes without saying, but it also means that if you happen to come across someone's accounts.db file, all it takes to log in into their Google account(s) is two HTTPS requests: one to get the MergeSession URL, and one to log in to their accounts page. If you are thinking 'This doesn't affect me, I use Google two-factor authentication (2FA)!', you should know that in this case 2FA doesn't really help. Why? Because since Android doesn't support 2FA, to register an account with the AccountManager you need to use an application specific password (Update: On ICS and later, GLS will actually show a WebView and let you authenticate using your password and OTP. However, the OTP is not required once you get the master token). And once you have entered one, any tokens issued based on it, will just work (until you revoke it), without requiring entering an additional code. So if you value your account, keep your master tokens close and revoke them as soon as you suspect that your phone might be lost or stolen. Better yet, consider a solution that lets you wipe it remotely (which might not work after your revoke the tokens, so be sure to check how it works before you actually need it).


    As we mentioned above, this is all ClientLogin based, which is officially deprecated, and might be going away soon (EOL scheduled for April 2013). But some of the Android Google data sync feeds still depend on ClientLogin, so if you use it you would probably OK for a while. Additionally, since the weblogin: implementation is server-based, it might be updated to conform with the latest (OAuth 2.0-based?) infrastructure without changing the client-side interface. In any case, watch the Android Browser and Chormium code to keep up to date.

    Summary

    Google offers multiple online services, some with both a traditional browser-based interface and a developer-oriented API. Consequently, there are multiple ways to authenticate to those, ranging from form-based username and password login to authentication API's such as ClientLogin and OAuth 2.0. It is relatively straightforward to get an authentication token for services with a public API on Android, either using Android's native AccountManager interface or the newer Google Play Services extension. Getting the required session cookies to login automatically to the Web sites of services that do not offer an API is however neither obvious, nor documented. Fortunately, it is possible and very easy to do if you combine the special 'weblogin:' token type with the service name and the URL of the site you want to use. The best available documentation about this is the Android Browser source code, which uses the same techniques to automatically log you in to Google sites using the account(s) already registered on your device.

    Moral of the story: interoperability is so much easier when you control all parties involved.

    Android online account management

    Our recent posts covered NFC and the secure element as supported in recent Android versions, including community ones. In this two-part series we will take a completely different direction: managing online user accounts and accessing Web services. We will briefly discuss how Android manages user credentials and then show how to use cached authentication details to log in to most Google sites without requiring additional user input. Most of the functionality we shall discuss is hardly new -- it has been available at least since Android 2.0. But while there is ample documentation on how to use it, there doesn't see to be a 'bigger picture' overview of how the pieces are tied together. This somewhat detailed investigation was prompted by trying to develop an app for a widely used Google service that unfortunately doesn't have an official API and struggling to find a way to login to it using cached Google credentials. More on this in the second part, let's first see how Android manages accounts for online services.

    Android account management

    Android 2.0 (API Level 5, largely non-existent, because it was quickly succeeded by 2.0.1, Level 6), introduced the concept of centralized account management with a public API. The central piece in the API is the AccountManager class which, quote: 'provides access to a centralized registry of the user's online accounts. The user enters credentials (user name and password) once per account, granting applications access to online resources with "one-click" approval.' You should definitely read the full documentation of the class, which is quite extensive, for more details. Another major feature of the class is that it lets you get an authentication token for supported accounts, allowing third party applications to authenticate to online services without needing to handle the actual user password (more on this later). It also has a whole of 5 methods that allow you to get an authentication token, all but one with at least 4 parameters, so finding the one you need might take some time, with yet some more to get the parameters right. It might be a good idea to start with the synchronous blockingGetAuthToken() and work your way from there once you have a basic working flow. On some older Android versions, the AccountManager would also monitor your SIM card and wipe cached credentials if you swapped cards, but fortunately this 'feature' has been removed in Android 2.3.4.

    The AccountManager, as most Android system API's, is just a facade for the AccountManagerService which does the actual work. The service doesn't provide an implementation for any particular form of authentication though. It only acts as a coordinator for a number of pluggable authenticator modules for different account types (Google, Twitter, Exchange, etc.). The best part is that any application can register an authentication module by implementing an account authenticator and related classes, if needed. Android Training has a tutorial on the subject that covers the implementation details, so we will not discuss them here. Registering a new account type with the system lets you take advantage of a number of Android infrastructure services:
    • centralized credential storage in a system database
    • ability to issue tokens to third party apps
    • ability to take advantage of Android's automatic background synchronization
    One thing to note is that while credentials (usually user names and passwords) are stored in a central database (/data/system/accounts.db or /data/system/user/0/accounts.db on Jelly Bean and later for the first system user), that is only accessible to system applications, credentials are in no way encrypted -- that is left to the authentication module to implement as necessary. If you have a rooted device (or use the emulator) listing the contents of the accounts table might be quite instructive: some of your passwords, especially for the stock Email application, will show up in clear text. While the AccountManger has a getPassword() method, it can only be used by apps with the same UID as the account's authenticator, i.e., only by classes in the same app (unless you are using sharedUserId, which is not recommended for non-system apps). If you want to allow third party applications to authenticate using your custom accounts, you have to issue some sort of authentication token, accessible via one of the many getAuthToken() methods. Once your account is registered with Android, if you implement an additional sync adapter, you can register to have it called at a specified interval and do background syncing for you app (one- or two-way), without needing to manage scheduling yourself. This is a very powerful feature that you get practically for free, and probably merits its own post. As we now have a basic understanding of authentication modules, let's see how they are used by the system.

    As we mentioned above, account management is coordinated by the AccountManagerService. It is a fairly complex piece of code (about 2500 lines in JB), most of the complexity stemming from the fact that it needs to communicate with services and apps that span multiple processes and threads within each process, and needs to take care of synchronization and delivering results to the right thread. If we abstract out the boilerplate code, what it does on a higher level is actually fairly straightforward:
    • on startup it queries the PackageManager to find out all registered authenticators, and stores references to them in a map, keyed by account type
    • when you add an account of a particular type, it saves its type, username and password to the accounts table
    • if you get, set or reset the password for an account, it accesses or updates the accounts table accordingly
    • if you get or set user data for the account, it is fetched from or saves to the extras table
    • when you request a token for a particular account, things become a bit more interesting:
      • if a token with the specified type has never been issued before, it shows a confirmation activity asking (see screenshot below) the user to approve access for the requesting application. If they accept, the UID of the requesting app and the token type are saved to the grants table.
      • if a grant already exits, it checks the authtoken table for tokens matching the request. If a valid one exists, it is returned.
      • if a matching token is not found, it finds the authenticator for the specified account type in the map and calls its getAuthToken() method to request a token. This usually involves the authenticator fetching the username and password from the accounts table (via the getPassword() method) and calling its respective online service to get a fresh token. When one is returned, it gets cached in the authtokens table and then returned to the requesting app (usually asynchronously via a callback).
    • if you invalidate a token, it gets deleted from the authtokens table

    Now that we know how Android's account management system works, let's see how it is implemented for the most widely used account type.

    Google account management

      Usually the first thing you do when you turn on your brand new (or freshly wiped) 'Google Experience' Android device is to add a Google account. Once you authenticate successfully, you are offered to sync data from associated online services (GMail, Calendar, Docs, etc.) to your device. What happens behinds the scenes is that an account of type 'com.google' is added via the AccountManager, and a bunch of Google apps start getting tokens for the services they represent. Of course, all of this works with the help of an authentication provider for Google accounts. Since it plugs in the standard account management framework, it works by registering an authenticator implementation and using it involves the sequence outlined above. However, it is also a little bit special. Three main things make it different:
      • it is not part of any particular app you can install, but is bundled with the system
      • a lot of the actual functionality is implemented on the server side
      • it does not store passwords in plain text on the device
      If you have ever installed a community ROM built off AOSP code, you know that in order to get GMail and other Google apps to work on your device, you need a few bits not found in AOSP. Two of the required pieces are the Google Services Framework (GSF) and the Google Login Service (GLS). The former provides common services to all Google apps such as centralized settings and feature toggle management, while the latter implements the authentication provider for Google accounts and will be the topic of this section.

      Google provides a multitude of online services (not all of which survive for long), and consequently a bunch of different methods to authenticate to those. Android's Google Login Service, however doesn't call those public authentication API's directly, but via a dedicated online service, which lives at android.clients.google.com. It has endpoints both for authentication and authorization token issuing, as well as data feed (mail, calendar, etc.) synchronization, and more. As we shall see, the supported methods of authentication are somewhat different from those available via other public Google authentication API's. Additionally, it supports a few 'special' token types that greatly simplify some complex authentication flows.

      All of the above is hardly surprising: when you are dealing with online services it is only natural to have as much as possible of the authentication logic on the server side, both for ease of maintenance and to keep it secure. Still, to kick start it you need to store some sort of credentials on the device, especially when you support background syncing for practically everything and you cannot expect people to enter them manually. On-device credential management is one of the services GLS provides, so let's see how it is implemented. As mentioned above, GLS plugs into the system account framework, so cached credentials, tokens and associated extra data are stored in the system's accounts.db database, just as for other account types. Inspecting it reveals that Google accounts have a bunch of Base64-encoded strings associated with them. One of the user data entries (in the extras table) is helpfully labeled sha1hash (but does not exist on all Android versions) and the password (in the accounts table) is a long string that takes different formats on different Android versions. Additionally, the GSF database has a google_login_public_key entry, which when decoded suspiciously resembles a 1024-bit RSA public key. Some more experimentation reveals that credential management works differently on pre-ICS and post-ICS devices. On pre-ICS devices, GLS stores an encrypted version of your password and posts it to the server side endpoints both when authenticating for the first time (when you add the account) and when it needs to have a token for a particular service issued. On post-ICS devices, it only posts the encrypted password the first time, and gets a 'master token' in exchange, which is then stored on the device (in the password column of the accounts database). Each subsequent token request uses that master token instead of a password.

      Let's look into the cached credential strings a bit more. The encrypted password is 133 bytes long, and thus it is a fair bet that it is encrypted with the 1024-bit (128 bytes) RSA public key mentioned above, with some extra data appended. Adding multiple accounts that use the same password produces different password strings (which is a good thing), but the first few bytes are always the same, even on different devices. It turns out those identify the encryption key and are derived by hashing its raw value and taking the leading bytes of the resulting hash. At least from our limited sample of Android devices, it would seem that the RSA public key used is constant both across Android versions and accounts. We can safely assume that its private counterpart lives on the server side and is used to decrypt sent passwords before performing the actual authentication. The padding used is OAEP (with SHA1 and MGF1), which produces random-looking messages and is currently considered secure (at least when used in combination with RSA) against most advanced cryptanalysis techniques. It also has quite a bit of overhead, which in practice means that the GLS encryption scheme can encrypt at most 86 bytes of data. The outlined encryption scheme is not exactly military-grade and there is the issue of millions of devices most probably using the same key, but recovering the original password should be sufficiently hard to discourage most attackers. However, let's not forget that we also have a somewhat friendlier SHA1 hash available. It turns out it can be easily reproduced by 'salting' the Google account password with the account name (typically GMail address) and doing a single round of SHA1. This is considerably easier to do and it wouldn't be too hard to precompute a bunch of hashes based on commonly used or potential passwords if you knew the target account name.

      Fortunately, newer version of Android (4.0 and later) no longer store this hash on the device. Instead of the encrypted password+SHA1 hash combination they store an opaque 'master token' (most probably some form of OAuth token) in the password column and exchange it for authentication tokens for different Google services. It is not clear whether this token ever expires or if it is updated automatically. You can, however, revoke it manually by going to the security settings of your Google account and revoking access for the 'Android Login Service' (and a bunch of other stuff you never use while you are at it). This will force the user to re-authenticate on the device next time it tries to get a Google auth token, so it is also somewhat helpful if you ever lose your device and don't want people accessing your email, etc. if they manage to unlock it. The service authorization token issuing protocol uses some device-specific data in addition to the master token, so obtaining only the master token should not be enough to authenticate and impersonate a device (it can however be used to login into your Google account on the Web, see the second part for details).

      Google Play Services

      Google Play Services (we'll abbreviate it to GPS, although the actual package is com.google.android.gms, guess where the 'M' came from) was announced at this year's Google I/O as an easy to use platform that offers integration with Google products for third-party Android apps. It was actually rolled out only a month ago, so it's probably not very widely used yet. Currently it provides support for OAuth 2.0 authorization to Google API's 'with a good user experience and security', as well some Google+ plus integration (sign-in and +1 button). Getting OAuth 2.0 tokens via the standard AccountManager interface has been supported for quite some time (though support was considered 'experimental') by using the special 'oauth2:scope' token type syntax. However, it didn't work reliably across different Android builds, which have different GLS versions bundled and this results in slightly different behaviour. Additionally, the permission grant dialog shown when requesting a token was not particularly user friendly, because it showed the raw OAuth 2.0 scope in some cases, which probably means little to most users (see screenshot in the first section). While some human-readable aliases for certain scopes where introduced (e.g., 'Manage your taks' for 'oauth2:https://www.googleapis.com/auth/tasks'), that solution was neither ideal, nor universally available. GPS solves this by making token issuing a two-step process (newer GLS versions also use this process):
      1. the first request is much like before: it includes the account name, master token (or encrypted password pre-ICS) and requested service, in the 'oauth2:scope' format. GPS adds two new parameters: requesting app package name and app signing certificate SHA1 hash (more on this later). The response includes some human readable details about the requested scope and requesting application, which GPS shows in a permission grant dialog like the one shown below.
      2. if the users grants the permission, this decision is recorded in the extras table in a proprietary format which includes the requesting app's package name, signing certificate hash, OAuth 2.0 scope and grant time (note that it is not using the grants table). GPS then resends the authorization request setting the has_permission parameter to 1. On success this results in an OAuth 2.0 token and its expiry date in the response. Those are cached in the authtokens table in a similar format.

      To be able to actually use a Google API, you need to register your app's package name and signing key in Google's API console. The registration lets services validating the token query Google what app the token was issued for, and thus identify the calling app. This has one subtle, but important side-effect: you don't have to embed an API key in your app and send it with every request. Of course, for a third party published app you can easily find out both the package name and the signing certificate so it is not particularly hard to get a token issued in the name of some other app (not possible via the official API, of course). We can assume that there are some additional checks on the server side that prevent this, but theoretically, if you used such a token you could, for example, exhaust a third-party app's API request quota by issuing a bunch of requests over a short period of time. 

      The actual GPS implementation seems to reuse much of the original Google Login Service authentication logic, including the password encryption method, which is still used on pre-ICS devices (the protocol is, after all, mostly the same and it needs to be able to use pre-existing accounts). On top of that it adds better OAuth 2.0 support, a version-specific account selection dialog and some prettier and more user friendly permission grant UIs. The GPS app has the Google apps shared UID, so it can directly interact with other proprietary Google services, including GLS and GSF. This allows it, among other things, to directly get and write Google account credentials and tokens to the accounts database. As can be expected, GPS runs in a remote service that the client library you link into your app accesses. The major selling point against the legacy AccountManager API is that while its underlying authenticator modules (GLS and GSF) are part of the system, and as such cannot be updated without an OTA, GPS is an user-installable app that can be easily updated via Google Play. Indeed, it is advertised as auto-updating (much like the Google Play Store client), so app developers presumably won't have to rely on users to update it if they want to use newer features (unless GPS is disabled altogether, of course). This update mechanism is to provide 'agility in rolling out new platform capabilities', but considering how much time the initial roll-out took, it is to be seen how agile the whole thing will turn out to be. Another thing to watch out for is feature bloat: besides OAuth 2.0 support, GPS currently includes G+ and AdMob related features, and while both are indeed Google-provided services, they are totally unrelated. Hopefully, GPS won't turn into a 'everything Google plus the kitchen sink' type of library, delaying releases even more. With all that said, if your app uses OAuth 2.0 tokens to authenticate to Google API's, which is currently the preferred method (ClientLogin, OAuth 1.0 and AuthSub have been officially deprecated), definitely consider using GPS over 'raw' AccountManager access.

      Summary

      Android provides a centralized registry of user online accounts via the AccountManager class. It lets you both get tokens for existing accounts without having to handle the actual credentials and register your own account type, if needed. Registering an account type gives you access to powerful system features, such as authentication token caching and automatic background synchronization. 'Google experience' devices come with built-in support for Google accounts, which lets third party apps access Google online services without needing to directly request authentication information from the user. The latest addition to this infrastructure is the recently released Google Play Services app and companion client library, which aim to make it easy to use OAuth 2.0 from third party applications. 

      We've now presented an overview of how the account management system works, and the next step is to show how to actually use it to access a real online service. That will be the topic of the second article in the series.