Developers, grow your numbers! Connected app success stories from our pilot partners
We’ve seen some amazing new apps built since we released the developer preview of our new connected apps framework two months ago. We’re continuing to make improvements, and we can’t wait for the official launch of our App Gallery to our entire community in the coming months.
Connected apps are great for both developers and the foursquare community. For people checking in, connected apps make foursquare more interesting and customized. For developers, connected apps provide an opportunity for more people to discover and engage with your timely and targeted content.
The connected apps effect
Our connected apps launch partners have seen significant jumps in sign-ups and downloads, increased engagement, greater exposure to new audiences, and spikes in web traffic. Here’s what some of them had to say:
Snoball
The Snoball connected app lets people set-up a donation to trigger when they check in to a specific place on foursquare. For example, a person can pledge $1 to charity:water when they check in to their favorite coffee house.
“Since the app’s July launch, the number of check in-based donations within Snoball has increased by 500%, while nonprofits within the platform saw a 40% increase in donations through Snoball with an average donation of $2 per check-in.” – Todd Graves, Software Developer at Snoball
Untappd
Untappd gives beer recommendations to people who check in to a bar and makes it easy to track the beers they drink.
“Through the foursquare connected app, we saw a 10% increase in sign-ups for Untappd and that 10% of all replies sent to the user turned into a beer check-in. Most importantly, our users have told us how much they love getting great recommendations about beer choices while at their favorite establishments, without leaving the foursquare app.” – Greg Avola, Co-Founder of Untappd
Eat This, Not That
Eat This, Not That recommends the healthiest dishes to order when people check in to restaurants.
“Our goal in this partnership was to deliver quality information that is relevant to our viewers when they need it most. We provided the editorial content and foursquare provided the technology… Viewers are literally sitting at these restaurants, checking in, and looking at a list of menu options (usually without calorie counts); serving them helpful tips on what to order (and what to stay away from) is not only timely, but extremely useful.
We plan on launching a second connected app this year with Women’s Health, which will deliver content to its health-conscious readers in the nutrition/fitness category.” – Jackie Lebowitz, Special Projects Editor at Men’s Health
Sonar
Sonar lets users know when their friends and other interesting people who share interests and connections are nearby.
“The impact of being a part of foursquare’s connected apps pilot has been fantastic. We’ve been able to attribute increases across all the standard metrics we track daily to our presence within foursquare, including downloads and engagement. On some days we’ve seen downloads from foursquare as much as doubling our daily download numbers, and we’ve seen a 6000% increase in website traffic sent from foursquare. We’ve also noticed a jump in downloads in new markets where foursquare has a high penetration, including Turkey, Russia and Brazil.” – Katie Smith-Adair, Vice President at Sonar
Want to build a connected app, or have one you’d like us to feature? Learn more or submit your app to be featured in our Connected Apps Gallery.
Checking in at the movies? Here’s how to never miss an after-the-credits teaser again.
The latest addition to the connected apps gallery is a perfect example of how a little bit of information at the right time can make all the difference. Now when you check in to a movie on foursquare, Mantas Vidutis’s After Credits can let you know if there’s something worth sticking around for after the credits roll.
Here’s how to never miss a teaser, blooper reel, or Avengers shawarma-eating-sesh ever again:
1. Click here to connect the After Credits app to your foursquare account. You can also find it in our app gallery online or from the settings page in the foursquare app.
2. The next time you’re at the movies, be sure to check in on foursquare with the flick you’re seeing.
3. After you check in, scroll down to see messages from your connected apps. After Credits will let you know if you should sit tight for something awesome! You can tap on the message for more details.
To connect more apps that add information to your check-ins (like the healthiest dish to order, The Weather Channel forecast, or interesting people you may know nearby), check out our app gallery. We’ll be adding our favorite new apps here as we lead up to our full release in a few months.
Developers, want to build a connected app, or have one you’d like us to feature? Learn more or submit your app.
foursquare now uses Elastic Search (and on a related note: Slashem also works with Elastic Search)!
Historically venue search used to be done in Mongo at foursquare; we later migrated it to Solr and now we are using Elastic Search. While Solr has been great (and we continue to use it for users and to-dos) we want to make sure that we can keep up with the growth of our venues database as our wonderful users keep adding more places.
Elastic Search provides a really nice way to shard search data and manage the related problems around this. By sharding search data we are able to split it up the documents into groups which can easily fit on a single machine and then get the results back from a collection of machines. Elastic Search and Solr have quite a few similarities that come from both being built on top of lucene. The Elastic Search team has been super helpful during the migration process, providing ideas on how to tweak our queries to get better performance out of Elastic Search. We got a non-trivial performance improvement by switching the field types for our geohash to allow us to skip the analyzer. One of the biggest improvements for performance we found was paramaterazing our ranking scripts to avoid the per-search compilation over-head. We also saw an improvement when using a custom scoring plugin (written in scala.)
By far the largest improvement came from switching the type of search we were asking Elastic Search to do. When you are doing a search for “blue bottle coffee shop” you probably care more about the terms “blue bottle” matching then the term “coffee shop” matching, since blue bottle doesn’t show up very often in comparison to “coffee shop”. By switching the search type from DFS_QUERY_THEN_FETCH to QUERY_THEN_FETCH we use the term frequencies in each shard rather than getting a global set of term frequencies. This works really well for us since our venues are fairly uniformly distributed between the shards, but might not work so well if that isn’t the case for your data.
The migration from Solr to Elastic Search has really emphasized the usefulness of having easy to update throttles. The throttles let us switch code paths in our application for different groups of users without having to do deploy. As with deploying any large piece of infrastructure, there were a few hiccups during the roll-out* of Elastic Search and throttles helped us minimize the impact on end users.
As we’ve previously written about, the majority of our search queries are written using Slashem, an in-house DSL for querying search backends. Rather than re-write all of our queries for Elastic Search we instead updated Slashem to generate queries for Elastic Search as well as Solr. The support for Elastic Search in Slashem is not as comprehensive as the Solr support, although its pretty good for basic searching at this point. A lot of the query generation is still fairly naive, but we are working on improving this. As always patches are welcome 
.
Cheers & Hugs,
- @holdenkarau & @adamalix
* Although #sfsearch has normally listened to levels for our deploys.
Say hello to FSNetworking: A Small Networking Library for iOS and Mac
The iOS team here at foursquare is pleased to announce a new open source networking library, FSNetworking. The code is released under the Apache License, version 2.
Back Story
For almost two years the foursquare iOS application depended on ASIHTTPRequest for all of its networking transactions. It served us well, but over time we found that our needs had diverged from what the project offered. After looking at the open alternatives available, we decided to write our own networking class, using NSURLConnection and blocks for callbacks (ASIHTTPRequest was deprecated by its authors shortly thereafter). AFNetworking was released while FSNConnection was in its infancy; we considered switching over but felt that we could come up with an even simpler design (note that AFNetworking has progressed considerably since our initial assessment, and it is fascinating to see just how much our solutions have converged). Since network transactions are the drivetrain of our entire application, it was crucial that we understand the entire mechanism top to bottom.
We are very grateful to the authors of ASIHTTPRequest for all their hard work and generosity. Maintaining an open source project with so many users, who have so many different needs, is a huge undertaking, and projects like ASIHTTPRequest and AFNetworking underpin the success of countless applications. It is in this same spirit that we have chosen to open-source our own solution.
Design and Implementation
The FSNConnection class underwent a number of transformations before reaching its present form. We knew from the start that we wanted result parsing to occur on a background thread; otherwise parsing and caching callbacks would cause congestion in the main run loop and would degrade the user experience. As is so often the case, safe multithreading became the dominant concern throughout the project.
The first version had a request manager singleton, which owned a single background request thread. This thread ran its own private run loop, on which we scheduled every NSURLConnection. The implementation installed a dummy NSPort object to prevent the run loop from exiting, a trick that worked but turned out to be quite brittle. If a connection callback ever crashed on the background thread, then the thread would die but the application would not notice, and subsequent requests would silently pile up.
Rather than delve into the nasty details of detecting crashed threads and restarting them, we converted FSNConnection into an NSOperation subclass, hoping that NSOperationQueue could handle the threading semantics for us. Subclassing came with a certain amount of implementation overhead, and as we got further along the design made less and less sense. The root of the problem was that NSOperationQueue and NSRunLoop provide two distinct paradigms for asynchronous operations, and we were attempting to use both. Scheduling the underlying NSURLConnection on the main runloop seemed to sacrifice some of the efficiency that multithreading promised; creating a new runloop for each operation felt heavy handed and overly complicated. We toyed around with both approaches, but neither was satisfactory.
At some point someone noticed that NSURLConnection had a setDelegateQueue: method in the iOS headers; it was listed in the OS X 10.7 API docs, but at the time was omitted from the iOS 5 docs. This approach held great promise: let Cocoa handle the calling back to the connection delegate inside of a background queue thread. With this method FSNConnection could benefit from an operation queue without having to subclass NSOperation. Unfortunately, the undocumented method proved treacherous; setting the delegate queue on iOS 5 causes the queue to deadlock, and the connection never calls back. This bug has been discussed elsewhere; see the FSNetworking README.md for more details.
Since delegate queues work fine on OS X 10.7, we decided to stick with this approach and work around the problem on iOS 5 by scheduling the NSURLConnection in the main runloop, and then using GCD to dispatch the parse block on the background thread. We found that using the main thread for NSURLConnection callbacks did not slow down the UI too much, and GCD performs wonderfully. We have since verified that iOS 6 fixes the delegate queue bug, so we may deprecate this workaround at some point.
Results
Switching to FSNConnection simplified our networking stack considerably. This was mostly due to the fact that we had previously used a custom request class to wrap ASIHTTPRequest, a design necessitated by the fact that in ASIHTTPRequest, form POSTs are performed by a special subclass. Since FSNConnection does GET, url-encoded POST, and multipart/form-data POST with the same API, the wrapper class became unnecessary. Instead, we simply created a category for all of our foursquare API requests. This category defines constructor class methods that return connection instances configured for a particular need. See the README.md for examples.
The other major benefit of ridding ourselves of the wrapper class was that it enabled a fire-and-forget pattern, in which requests are started, autoreleased, and then never referenced by application code again. The wrapper owned its ASIHTTPRequest object, whereas fire-and-forget required the underlying request to own its wrapper. In contrast, FSNConnection makes this work reliably by relying on the fact that NSURLConnection retains its delegate (which is unusal for Cocoa); once the underlying Cocoa connection completes or fails, the FSN completion block is called, and only after that finishes is the FSNConnection released and deallocated.
Future Work
While FSNConnection serves our current needs, there are several areas that are likely to get attention from the open source community. Additional HTTP methods and MIME types can be added as needed; these should be straightforward additions. The ability to stream upload and download data from and to disk would be very useful. Another interesting feature request is the ability to instrument the authentication callbacks, mostly for the purpose of profiling SSL requests.
Since the code base is so small, we hope that it will appeal to developers who want to add their own features quickly, and we look forward to patch submissions and pull requests on Github. However, please note that we have intentionally limited the scope of this project to Cocoa networking; we would prefer that it become a component of more ambitious frameworks, rather than see it accumulate many optional components itself.
Thanks, and happy hacking!
- George King, foursquare iOS Engineer
Conference fiend? Check out Lanyrd, a connected app for conference-goers!
Every week or so we’ll be featuring a new connected app in the lead up to the full launch in early fall. If you’ve built an app that makes use of the new features, don’t forget to tell us about it – we might feature you in a future post! First up: Lanyrd
Lanyrd is an online social conference directory that helps you keep tabs on the people and events related to conferences you’re attending or should be attending. In addition to a native iOS app, they’ve built a slick HTML5 mobile site that works well across a myriad of devices.
Thanks to their new connected app integration, Lanyrd now gives relevant information to people on foursquare, like details about upcoming events, the moment they check in to a place that’s hosting a conference. If you were already planning on attending the event or are a registered speaker, they also add a post to your check-in telling friends why you’re at the conference. Conference fiends, click below to connect Lanyrd to your foursquare account.
Learn more about how Lanyrd build their connected app on their blog.
Some homemade @foursquareAPI Connected Apps hot out of the @foursquare hq oven
While we were building out the Connected Apps Platform, we couldn’t resist the urge to write a bunch of apps on our own. Some are pretty silly, some are useful, and some are just a first step towards all the things that are possible with connected apps. We’ll start featuring some of our favorite apps from all of you next week (don’t forget to tell us about your connected app!), but in the meantime here are some of our favorite homemade connected apps:
Fourstalgia
Fourstalgia gives you a window into the past at the places you visit. When you check in, you’ll see images of the surrounding area from SepiaTown, a crowd sourced geo database of historic photos.
Choosenear.me
Choosenear.me helps you learn about projects at public schools in your neighborhood that are soliciting donations via DonorsChoose.
Squaredar
Squaredar notifies you when you check in and a foursquare friend has been recently spotted somewhere closer than normal. Find out when a friend from abroad is in the country for a visit or maybe when an old co-worker who lives across town is next door having a burger.
Thanks for Coming in Today
Finally, someone who notices when you clock in at work early! Thanks for Coming in Today keep tabs on the average time you check in at work and pats you on the back when you show up earlier than normal.
TWSS
Squarely in the ‘silly’ category, TWSS will remind you when your check-in shouts deserve some special commentary.
BAM Membership Specials
A lot of places in the neighborhood surrounding The Brooklyn Academy of Music, a local arts center, offer great rewards to donating members, but it can be hard to remember to take advantage of them. This simple BAM Membership Specials app reminds the connected user if they’ve checked in somewhere that offers a reward to BAM members.
These apps are just the beginning of what can be done on the Connected Apps Platform. Ready to start building? Head over to the developer site to get started, and don’t forget to tell us about your app when you’re done! You can check out the apps from our Developer Preview launch in our preliminary app gallery, which we’ll be updating with our wider launch later this year.
Let’s connect? (And you thought we’d never apps.)
Developers, we’ve had a pretty thrilling relationship these last few years and we think it’s finally time to take the next big step. No more users needing to carry state between our applications, no more Push API calls without a reply, no more questions about where content comes from – we want you to be an active part of our users’ lives.
To make it happen, we’ve carved out a special place inside of foursquare for you to make your own. You can now send a customized reply to any user who has connected your application to their foursquare account. The reply will immediately appear on the user’s check-in detail screen, and taps directly into your web or native mobile app.
When a user checks in, foursquare sends a real time push notification to your app’s server. Your app then decides if it wants to reply and what that reply should say. As replies are received by the foursquare servers via the API, they are passed down to the client via APNS (iOS), C2DM (Android), or Push Service (BlackBerry). The foursquare client intercepts these notifications and embeds them in the check-in detail screen – for the user, it feels like the replies are happening nearly instantaneously within foursquare. If foursquare is closed, the notifications appear in the system notification tray, linking the user to their check-in detail page.
These replies can contain up to 200 characters, and also specify a URL and a “contentId.” The reply then taps through to an embedded webview with the URL loaded. If the user has your native app installed, however, we’ll send the user directly to your app, where you can key off of your contentId to show the corresponding content to the user. In both cases, we pass a callback URL so you can send the user back to foursquare when they’re done – by letting them jump in and out of the most relevant parts of your app, we hope to drive more engagement from foursquare users.
In addition to letting apps reply to users when they check in, we’re launching a related set of functionality for users when they post information to their check-ins from another application. For example, if a user shares a song on Soundtracking or spot a dish on Foodspotting and share it with foursquare, foursquare will create a special link back to the original application. The posted content also appears with attribution in the friends tab, helping new users discover and connect apps that their friends are using. We’ll also create a separate story in the friends tab when a user initially connects your app to their foursquare account, which means it’s even easier for your app to spread from friend to friend within our community of over 20 million users.
This new framework was built for everyone from weekend hackers and scrap startups to larger businesses and brands – we think all will benefit from being able to send personalized, contextual, and timely hooks to users when they’re already looking at their phone.
Read our announcement and the full overview here. We’ll continue twiddling and polishing this new framework over the coming months (give us your feedback!) before we fully launch it to users, but it should be stable enough for adventurous developers to start building new apps and features. If you build something, be sure to tell us and we’ll regularly highlight our favorite new apps. We currently have a limited app gallery featuring a few partners who have been helping us test the new endpoints, but when we launch to users, we’ll have a full app gallery and the best apps will get special treatment there. Be sure to subscribe to the foursquare-api mailing list for breaking news about changes.
- Akshay Patil (@ak)
Our Hadoop stack at foursquare
Recently, we hosted the NYC Hadoop Meetup at foursquare HQ. To the over 100 in attendance, we gave an overview of our Hadoop stack. In case you couldn’t make it (or want to take a second look at the slides), here they are:
At foursquare, we’re generating, collecting, and analyzing billions of log events each week. Combined with data exported from mongodb, such as foursquare’s 2 billion check-ins, we use Hadoop to build recommendations, power internal reporting, and drive product features.
We’ve recently introduced a number of new components to our Hadoop stack in order to create a reliable, scalable, and production ready pipeline. These include Oozie for workflow management, Thrift for data serialization, Pig for adhoc analysis, and Scoobi for writing MapReduce jobs in Scala. We’ll share some experiences, lessons learned, and how we’re making it easier for developers and non-developers to use Hadoop at foursquare.
- Joe Crobak (@joecrobak) and Blake Shaw (@metablake)
Stability in the midst of chaos
I gave a talk at MongoNYC (video here!) about how we’ve built resiliency into our Mongo clusters on top of a volatile hardware environment.
We currently run all of our hardware infrastructure on top of Amazon’s EC2 platform. EC2 has allowed us to be really flexible as we’ve grown foursquare in usage and features, but it comes with a trade off of limited hardware options. The most important piece of hardware with regard to databases is the disk, and the disk options on ec2 are limited to local drives on the same machine as the hypervisor, or elastic block storage (EBS), which is network disk. We found that our IO requirements could not be met by the local machine storage, so we run our mongo instances on raided EBS volumes. The steady state performance of EBS is pretty good. However, as a network service, the latency and bandwidth can vary. In some situations, IO operations can be completely blocked for tens of seconds at a time.
Mongo has built-in failover across nodes in a replica set, and it handles outright machine or process failure well, but it does not failover in situations where the disk has degraded performance. For a long time, we reacted manually to those situations, but as the number of servers have grown, the rate of failure has increased and manual response is no longer feasible. This presentation outlines the steps we took to build custom tools to detect degraded disk conditions and the modifications we made to Mongo to automate the failover.
Mongo at foursquare: Stability in the midst of chaos
- Jon Hoffman (@hoffrocket)
Machine Learning with Large Networks of People and Places
I recently gave a talk at foursquare HQ for the New York Machine Learning Meetup. Here’s the abstract, with the slides below. Hope you enjoy!
Foursquare is now aware of over 1.5 billion check-ins from 15 million people at 30 million different places all over the world. Each check-in can be thought of as an edge in a vast network connecting people to each other and to the places that they care about most. Graph-based machine learning algorithms are critical not only for making sense of these networks that emerge out of patterns of human mobility but also for creating useful data-driven products that help people better navigate the real world. In this talk, we will examine two networks that we have observed at foursquare, the Social Graph and the Place Graph, and then discuss various machine learning and big data techniques for better understanding these networks as well as using them to build a novel recommendation engine we call Explore.