APIv2: Woulda, coulda, shoulda

As we sunset foursquare APIv1 and announce some amazing new milestones for APIv2, now seemed like as good a time as any to reflect on some of the decisions we made in APIv2 and see how they’re holding up. Fortunately, we were able to draw on both our experience with APIv1 and from tight iteration with our client developers (especially Anoop, who is awesome!). We’re pretty happy with the result, but we still made a few mistakes. Hopefully there are some lessons for anybody else out there who gets stuck designing an API.

JSON

Verdict: no brainer

Flexible. Understood by everybody. (Sorry, Dave Winer!)

OAuth2 and HTTPS-only

Verdict: as good as it can be

OAuth2 is fairly straightforward to implement (although the spec is still churning a bit – the userless flow arrived too late for us to use it!) for both servers and clients, and offloading encryption to HTTPS instead of implementing it ourselves makes it light years easier to get right.

Consumers pushed back a bit on our decision to not let third party applications just use usernames and passwords to authenticate (instead, we force them to use a web-based authorization page). We often tell people, “You shouldn’t be any more comfortable entering your foursquare password into some random app than you are putting your Facebook password into some random app.” Still, the user experience for mobile applications could be better, and on some mobile platforms like Blackberry and Nokia, it’s hard for application developers to interact with the browser in the way that OAuth requires.

The security properties of OAuth2 are far from perfect, but it’s not the protocol’s fault. It would be great if client devices provided a way to keep client secrets secure, even from self-signed SSL certificate man-in-the-middle attacks. And it would be great if the “implicit grant” flow could be more robust against token leaks by the consuming page.

REST lite

Verdict: good!

We decided to use resourceful URLs for certain key objects, like users/USER_ID or venues/VENUE_ID, with actions and connections hanging off of them. But not every resource has a URL (e.g. badge unlocks, todos, comments). This keeps the API minimal and reduces the surface area we need to keep backwards compatible.

On the flip side, we decided not to have deeply nested URLs, e.g. users/USER_ID/tips/TIP_ID, instead opting for flat tips/TIP_ID. This led to much simpler URLs. And having only one path for, say, the details about a tip, let us avoid making developers choose between multiple ways to do the same thing. As a bonus, this explicit, DRY approach to API design makes monitoring and debugging much easier.

Finally, we avoided using the extended set of HTTP verbs beyond the web’s idiomatic POST and GET, in favor of putting the action into the endpoint name, such as tips/add or tips/TIP_ID/markdone. Developers may be using stacks where these verbs are impossible or confusing, and if we supported a workaround for a lack of verb support, we’d be back at having two ways of doing things.

Generic structures and indirection

Verdict: good!

In APIv2, lists use a general structure of { count: X, items: [...] }. This lets us vary the amount of data we want to include in a given context (e.g. the number of tips in a venue response) without breaking clients. And it makes it easy for developers to understand how much data there may be to page in and to share list-handling code across endpoints.

Grouped lists extend this paradigm, where groups are represented via data, not structure. Rather than tips: { friends: [...], others: [...] }, we do tips: { count: X, groups: [ { name: friends, count: Y, items: [...]},.... ] }. This is more wordy, but it allows us to add and remove groups without breaking clients, especially in cases where the data is being pulled into objects where field names map to members. It also allows us to specify an order of groups and to structurally group sub-counts with sample items (e.g. the count of tips from my friends is next to tips from my friends).

In general, we tried to live by not using keys as data. Although it might be intuitive to say something like notifications: { mayor: { .. }, points: { .. } }, this structure suffers from the same problem as the groups above: it’s hard to add and remove groups without breaking consumers, and there’s no order. Instead, we opted for notifications: [ { type: “mayor”, .. }, .. ], which is much more flexible.

Finally, where possible, we use indirection to guard against the introduction of additional data. For example, rather than notification: { type: “special”, item: { ...specialdata... } }, we opted for notification: { type: “special”, item: { special: { ...specialdata... } } }. Although wordier, this format is much more explicit, and it allows us to add additional objects related to a given notification without perverting the form of a “special” object.

Documentation

Verdict: good!

Never underestimate the value of documentation; it’s been one of the largest sources of positive feedback, especially combined with the API explorer, which is built using standard OAuth2. To make it easier to write documentation given our itty bitty team, we hacked together a really simple documentation generation system in python that takes text files and spits out Lift templates, and it’s part of our main code repository. No engineer developing new endpoints has an excuse not to document them, and the documentation is the go-to source of information even for internal developers.

Timestamps as seconds since epoch

Verdict: good!

Not human-readable, but so easy to parse, and nobody has complained.

CamelCase

Verdict: meh

They look okay on field names, but they still look a little funny on parameter names, don’t they? It’s also a little lame that our endpoint names still aren’t camel case, as well as certain, uh, special field names.

Images

Verdict: meh

We had a lot of debates about representing images and surveyed a lot of other APIs. Sadly, we still ended up with slightly different treatments of representing the available sizes and filenames for photos, category icons, badge images, and point icons, depending on the number of sizes we anticipate offering, squareness of images, and our own evolving sense of what is easy to use. In general, the badge response seems the least wordy and most user-friendly way of representing a range of image sizes, and we’re using it where we can throughout the API.

Versioning

Verdict: jury’s still out

The hardest part of designing an API is that you’re stuck with a bunch of decisions you can’t take back easily. But in some cases we needed to take them back anyway, and, for those we came up with the idea of clients sending back a version like 20110708 and promising to be compatible with all changes up to the date. This can lead to some tricky coordination problems between internal test clients and new server releases, and it places a large burden on clients to avoid regressing, but it’s also very simple to explain, and we’re trying to keep the number of breaking changes really, really low. We do wish we’d gone out the gate with this versioning plan, since now we’re forced to break the “default” behavior at some point.

Category representation

Verdict: oops!

Some things require testing with a range of clients to really show their warts, and unfortunately our category representation was exercised too late. It quickly became clear that the way we represent categories in venue JSON is an awkward compromise between brevity and utility by excluding the category ids and icons of parents. For any application where you may want to roll up the category hierarchy, you’re stuck loading the whole hierarchy just to contextualize a small set of results. In the users/USER_ID/badges response, we tried out a sort of “foreign key” approach where part of the response contains IDs pointing back into a dictionary of details that is part of the same response. In hindsight, we should have considered using something like this in more places. Although less straightforward than a totally-inline API like Twitter’s, it still avoids the extra call overhead of a normalized API like Facebook’s.

Object consistency and level of detail

Verdict: meh

This is the more general case of the category problem. It’s always hard to decide how much detail to include in an object, and we largely let the requirements of our own clients dictate what to include where. In APIv1, it felt arbitrary which fields would be present in, say, a user, when seen in a friends list versus in a list of checkins. In APIv2, we tried to be more principled by describing “compact” and “full” JSON representations with a minimal number of optional fields, most of which were optional because of context or absence of data. This sometimes meant sending down extra data in a given context, but we felt it made it easier to consume on the client, especially if the client wanted to cache data.

However, we’re starting to hit the limits of this. Do we add tipCount to compact venues? Does that adversely impact server performance or client speed? Do users always include their twitter contact information, if visible? We’ve discussed adding a detail=high parameter, or something like Facebook’s fields=X,Y,Z parameter, but we’re not excited about the complexity that results.

Envelope

Verdict: good!

Our decision to wrap all responses in { meta: { … }, response: {...} } has been well received, as well as the error details that we pack into the meta block. The decision to only provide error messages intended for developers, plus an error type for developers to switch on, seems to be working relatively well.

Grouping versus ranking

Verdict: meh

Although grouping tips by self, friends, and others seemed reasonable given our UI treatment, this doesn’t really make sense in the long term if we start to more aggressively rank tips from your friends against other popular tips. In fact, we already threw out the grouping of venue search results.

Halllllllp

These are just some of the decisions that we made in building APIv2. Is our assessment correct? Are there other things you’re curious about? Hit us up in the comments.

Enjoy trying to elegantly trade off performance and ease of use in APIs? We’re hiring.

Kushal Dave, foursquare engineer