A System for Distributing Local and Regional News Data

I live in Vancouver, Washington but I commute to Portland for work every week day. For my commute, I usually take I-5 and therefore have to traverse Interstate Bridge to cross over into Oregon. The bridge allows for marine traffic on the Columbia River to pass underneath by lifting the spans and thus closing traffic off until it is clear. Naturally, because of the traffic implications, this is an event that occurs rather infrequently, but it still occurs nonetheless.

There have been times when I’ve been unfortunate enough to be stuck in traffic while a bridge lift was happening. Judging by the amount of other commuters and freight vehicles stuck beside me, I’m guessing that information about spans lifting isn’t being distributed efficiently.

The last time this happened, I tweeted the message: “Does anyone know if ODOT or WADOT make notifications (of any kind – web other otherwise) before they raise the I5 Interstate Bridge?” I was curious about this because I wanted to make a simple web app that tells you if the span is up or down.

A few days later, a response came from the OregonDOT’s twitter account, "Yes, we send traffic alert to Portland area traffic reporters before a bridge lift. @pdxcommute usually tweets about them. They shared a number for me to call for more information and while I have not yet called to get the details my mind started wondering how they would be able to distribute this information in a timely manner to multiple sources. I put the idea out of my head until I read Ilya Grigorik’s post titled HTTP PubSub: Webhooks & PubSubHubbub.

Using that as a springboard, I’ve come up with a conceptual system that addresses these questions.

Let’s say that you operate a web application that distributes traffic information to subscribers via IM, Email, SMS or Twitter. You’ll want to get that information from a reliable source and in Oregon that would be ODOT.

ODOT will surely have more than one service asking for traffic update data, so a polling mechanism on the client end is very inefficient for both the server and the client. Here, PubSubHubbub makes perfect sense. Images below are adapted from this Google Document (PubSubHubbub Subscription Flow Draft 0.1):

In step 1, you can see that my fictional web application (Subscriber) asks the ODOT Publisher to subscribe to traffic data. The Publisher sends information to the subscriber telling it where it can receive that data.

In step 2, our web application subscribes to the data feed and tells the Hub to send updates to a specific URL.

In step 3, there is some security checking from the Hub back to the Subscriber. The Hub sends the data to the URL that the Subscriber specified in step 2.

In step 4, Someone at ODOT has posted a new traffic update and therefore the Publisher updates the specified Hub.

In step 5, the Hub POSTs the update to the endpoint URL that the Subscriber specified in Step 2. Although the diagram below shows the interaction between the Hub and one Subscriber, in theory a Hub can update multiple Subscribers at once.

In the final step, the Subscriber can then re-post the data via IM, Twitter, Email, or SMS or simply display it on their website. Once a Subscriber gets a hold of that data, then it’s up to them what they do with it.

This conceptual method for delivering data is much more efficient. Besides the initial setup, the only data that’s being transferred is actual data and there is no useless polling going on. Although PubSubHubbub is one solution to the polling problem, it’s not the only one. Others are using the IM standard, XMPP, to also distribute information to Subscribers.

If you’re interested in PubSubHubbub, I strongly suggest that you read the documentation at Google’s Code page.