Monday, November 02, 2009

Twitter Search API and Google App Engine

So in my previous post, I talked about using Twitter as a commenting engine in http://zoomaroundtown.appspot.com/, and, towards the end, I briefly mentioned that the Twitter Search API (http://apiwiki.twitter.com/Twitter-API-Documentation) does not really work on Google App Engine (and most other cloud environments).

The reason why making Twitter Search API requests on Google App Engine often fails is simple. Twitter rate-limits requests per IP, and on Google App Engine, you share IPs with loads of other apps, a lot of which are probably trying to do the same thing as you i.e. making Twitter Search requests. Unfortunately Twitter Search API is still not incorporated into the Twitter REST API, which means there is no way of identifying requests via your username, so Twitter can only really rate-limit according to IPs.

To circumvent this, I created a simple PHP proxy on my other website (hosted on a "real" machine") :

$url = $_GET['url'];
$session = curl_init($url);
curl_setopt($session, CURLOPT_HEADER, false);
curl_setopt($session, CURLOPT_RETURNTRANSFER, true);
$json = curl_exec($session);
header("Content-Type: application/json");
echo $json;
curl_close($session);


and in my GAE Python app, instead of sending the requests straight to Twitter using URL Fetch, I send the requests to this proxy, which does a simple straightforward relay job for me.

Of course, this is less than ideal, as this goes in the face of the philosophy of deploying in a scalable cloud environment such as Google App Engine. But until Twitter allows username-based identification in Twitter Search requests, or Google sorts something out with Twitter (and other service API providers, for that matter), this seems to be the best way to get round this problem!

(I have heard that you are get your own IP on Amazon AWS, but I don't know enough about Amazon AWS to comment on it. Anyway no doubt more and more cloud infrastructures relying on shared IPs will be developed, and I can only see it heading that way...)

No comments: