Category Archives: Data

Better Round Trip-times (RTT) With CityGrid API Using GZip

One of the biggest questions I get when talking with CityGrid developers are around storing our data. Everyone wants to store the 15 million businesses CityGrid has in their own database. My response is really? You do?

This type of thinking comes from legacy thoughts around database driven applications and concerns about network latency. I counter these with a couple of points:

  1. You don’t want to be in big data business, we have teams of engineers working to ingest, clean, optimize and serve up this data. Do you really want to assume these responsibility.
  2. Our places, images, reviews, offers and other content is updated in real-time. If you store the data, you’re missing out on a lot of content that may change frequently.
  3. Speed is getting to be less of a concern when making calls to our APIs–you are looking at millisecond responses and in some worst case scenarios, 1 or 2 seconds.

Usually I can convince most developers about points 1 and 2. But speed is still an ongoing battle. We are working all the time at CityGrid to improve our API response times. But there are things you the developer can do to improve response times also.

One tactic that isn’t available in our documentation, is the use of GZip. With each API request you can set a HTTP header as Accept-Encoding: gzip, and the CityGrid will compress the responses for you, significantly reducing your round-trip time (RTT). Once you receive the response from CityGrid API you can uncompress using the GZIP decoding functionality available in any programming language.

Reducing the size of the data being transmitted is a great way to increase your RTT, which makes relying on any API for real-time calls, much more realistic. I recommend giving it a try when integrating your web or mobile app with the CityGrid Places, Reviews and Offers APIs.

City of Austin, Texas Has An Open Data Portal

City open data sites are popping up all over the place. This year Austin, TX launched a new open data portal for the city alongside a website refresh for the city, using a Drupal-based website.

As part of a resolution affirming the City council’s commitment to open government, the city worked with open data platform Socrata to build and deploy the portal. The portal provides data downloads and APIs serving up local city data-sets such as water quality sampling data, animal intake reports, lobbying data and affordable housing inventory maps.

The Austin, Texas open data portal one more city to get on the open data bandwagon, following the recent launch of San Francisco, and adding to NYCChicagoBaltimoreSeattleSan Francisco, and New Orleans..

I’m spending time going through all the Socrata powered portals and finding some of the best data-sets CityGrid developers can use to enrich their local web and mobile applications. I am also spending time working with our data ingestion team to make them aware of the most valuable city datasets, as well as reaching out to each of the city data evangelists to make sure we work together around data, hackathons and future city initiatives.

New City of San Francisco Data Portal

The City of San Francisco, one of the early pioneers of open city data, just unveiled a new open data portal at, built using Socrata cloud service platform.

The new city portal has downloadable data-sets and APIs providing access to 175 datasets including the new geocoded case data from San Francisco 311 since 2008, Map-based Crime Data, Active Business Listings as well as several updated administrative and financial datasets.

The open data initiative is a follow-up to, the cities pioneering open data project, and is meant to provide a platform that keeps up with the Mayor’s commitment to providing state of the art cloud infrastructure to drive more innovation in citizen access to information, community engagement and government efficiency.

As Jay Nath, San Francisco’s Chief Innovation Officer says:

“Two years ago, when we launched, Open Data was a visionary experiment in reinventing the government of the future. Today, with increasing worldwide adoption, we view Open Data as part of our new cloud infrastructure to deliver citizen, social, and programmatic interfaces to government services, in a much more cost-effective and agile model.”

The new Socrata-powered San Francisco open data portal offers a wide variety of feature, architecture, and performance enhancements, including:

  • Interfaces that allow non-technical users to interactively explore data, visualize it, and share contextually-relevant information with others, on the site, across the web, and on social networks.
  • Automatic full-text indexing of every data set’s content to facilitate online search, in addition to the ability to download the data in multiple, open, machine-readable formats.
  • Automatic API access to every data set, via the Socrata Open Data API (SODA) and access to technical support and online developer resources, which will lower the access barrier for civic developers.

San Francisco is just the latest, of cities that Socrata provides city data for including NYCChicagoBaltimoreSeattleSan Francisco, and New Orleans.

Making City data more accessible to the public secures San Francisco’s future as the world’s first 2.0 City,” said Mayor Lee. “It’s only natural that we move our Open Data platform to the cloud and adopt modern open interfaces to facilitate the flow of information and develop better tools to enhance City services.”

CityGrid plus municipal data provided by Socrata, provides a perfect source of rich data that developers can use to build mobile and web applications that are not just profitable, but can do some good and potentially make our cities better places to live for everyone.