Have been looking at iGoogle gadgets for work (I know they are so 2008, but still a good idea...) Anyway I created a simple Javascript Tester gadget for myself, which some of you may find useful: (See this gadget in the iGoogle directory)
Being an iGoogle gadget, you can add it to you iGoogle homepage, but you can always just use the embedded gadget right here!
Wednesday, November 04, 2009
Monday, November 02, 2009
Twitter Search API and Google App Engine
So in my previous post, I talked about using Twitter as a commenting engine in http://zoomaroundtown.appspot.com/, and, towards the end, I briefly mentioned that the Twitter Search API (http://apiwiki.twitter.com/Twitter-API-Documentation) does not really work on Google App Engine (and most other cloud environments).
The reason why making Twitter Search API requests on Google App Engine often fails is simple. Twitter rate-limits requests per IP, and on Google App Engine, you share IPs with loads of other apps, a lot of which are probably trying to do the same thing as you i.e. making Twitter Search requests. Unfortunately Twitter Search API is still not incorporated into the Twitter REST API, which means there is no way of identifying requests via your username, so Twitter can only really rate-limit according to IPs.
To circumvent this, I created a simple PHP proxy on my other website (hosted on a "real" machine") :
$url = $_GET['url'];
$session = curl_init($url);
curl_setopt($session, CURLOPT_HEADER, false);
curl_setopt($session, CURLOPT_RETURNTRANSFER, true);
$json = curl_exec($session);
header("Content-Type: application/json");
echo $json;
curl_close($session);
and in my GAE Python app, instead of sending the requests straight to Twitter using URL Fetch, I send the requests to this proxy, which does a simple straightforward relay job for me.
Of course, this is less than ideal, as this goes in the face of the philosophy of deploying in a scalable cloud environment such as Google App Engine. But until Twitter allows username-based identification in Twitter Search requests, or Google sorts something out with Twitter (and other service API providers, for that matter), this seems to be the best way to get round this problem!
(I have heard that you are get your own IP on Amazon AWS, but I don't know enough about Amazon AWS to comment on it. Anyway no doubt more and more cloud infrastructures relying on shared IPs will be developed, and I can only see it heading that way...)
The reason why making Twitter Search API requests on Google App Engine often fails is simple. Twitter rate-limits requests per IP, and on Google App Engine, you share IPs with loads of other apps, a lot of which are probably trying to do the same thing as you i.e. making Twitter Search requests. Unfortunately Twitter Search API is still not incorporated into the Twitter REST API, which means there is no way of identifying requests via your username, so Twitter can only really rate-limit according to IPs.
To circumvent this, I created a simple PHP proxy on my other website (hosted on a "real" machine") :
$url = $_GET['url'];
$session = curl_init($url);
curl_setopt($session, CURLOPT_HEADER, false);
curl_setopt($session, CURLOPT_RETURNTRANSFER, true);
$json = curl_exec($session);
header("Content-Type: application/json");
echo $json;
curl_close($session);
and in my GAE Python app, instead of sending the requests straight to Twitter using URL Fetch, I send the requests to this proxy, which does a simple straightforward relay job for me.
Of course, this is less than ideal, as this goes in the face of the philosophy of deploying in a scalable cloud environment such as Google App Engine. But until Twitter allows username-based identification in Twitter Search requests, or Google sorts something out with Twitter (and other service API providers, for that matter), this seems to be the best way to get round this problem!
(I have heard that you are get your own IP on Amazon AWS, but I don't know enough about Amazon AWS to comment on it. Anyway no doubt more and more cloud infrastructures relying on shared IPs will be developed, and I can only see it heading that way...)
Using Twitter as a commenting engine for Zoom around Town
So with the number of routes in http://zoomaroundtown.appspot.com/ increasing daily, the time has come for me to start considering implementing some kind of commenting functionality for the added cycle routes. I have not had the time to implement a user login system (I am still hoping to see more routes in the database first), let alone a commenting system. However one day, a simple idea came to me. What about getting people to comment on routes by creating tweets and then all I have to do is to display these tweets using Twitter Search. I was quite keen on exploring this idea, and I implemented the following in Zoom around Town:
For each route, I programmatically create a bit.ly link for its unique page url e.g. (http://bit.ly/39oF3z for http://zoomaroundtown.appspot.com/findRoutes?id=24001). On the page, I then display a message telling people to comment on the route by sending a tweet starting with @zoomaroundtown http://bit.ly/39oF3z (I'm even including a link that automatically populates the starting text for the user: http://twitter.com/home?status=%40zoomaroundtown%20http%3A%2F%2Fbit.ly%2F39oF3z%20).
To display comments, I use the Search API (http://apiwiki.twitter.com/Twitter-API-Documentation) to search for all tweets sent to zoomaroundtown containing the string http://bit.ly/39oF3z i.e. using the search string to:zoomaroundtown http://bit.ly/39oF3z.
Whether this is a viable way of enabling commenting Zoom around Town, only time can tell. However there are obvious pros and cons:
PROs:
Extremely easy to implement (especially when I don't have a user login system yet)
Increased site exposure on Twitter
CONs:
Twitter Search only seems to return tweets no more than 2 weeks old
I have no control over content (but since these are public tweets, do I really care?)
Last but not least, something unfortunately I only managed to discover after deployment to the live cloud environment - Twitter Search's rate limiter does not seem to like Google App Engine, and it is this point that will lead us to my next post, but for now it suffices to say that Twitter Search rate-limits requests per IP, so it is not surprising that in a cloud environment where one is sharing IPs with loads of other apps, some requests are bound to get rejected!
For each route, I programmatically create a bit.ly link for its unique page url e.g. (http://bit.ly/39oF3z for http://zoomaroundtown.appspot.com/findRoutes?id=24001). On the page, I then display a message telling people to comment on the route by sending a tweet starting with @zoomaroundtown http://bit.ly/39oF3z (I'm even including a link that automatically populates the starting text for the user: http://twitter.com/home?status=%40zoomaroundtown%20http%3A%2F%2Fbit.ly%2F39oF3z%20).
To display comments, I use the Search API (http://apiwiki.twitter.com/Twitter-API-Documentation) to search for all tweets sent to zoomaroundtown containing the string http://bit.ly/39oF3z i.e. using the search string to:zoomaroundtown http://bit.ly/39oF3z.
Whether this is a viable way of enabling commenting Zoom around Town, only time can tell. However there are obvious pros and cons:
PROs:
Extremely easy to implement (especially when I don't have a user login system yet)
Increased site exposure on Twitter
CONs:
Twitter Search only seems to return tweets no more than 2 weeks old
I have no control over content (but since these are public tweets, do I really care?)
Last but not least, something unfortunately I only managed to discover after deployment to the live cloud environment - Twitter Search's rate limiter does not seem to like Google App Engine, and it is this point that will lead us to my next post, but for now it suffices to say that Twitter Search rate-limits requests per IP, so it is not surprising that in a cloud environment where one is sharing IPs with loads of other apps, some requests are bound to get rejected!
Wednesday, October 07, 2009
Circle overlay on Google Map
This is my first "Zoom around Town"-related technical post. This first post looks at the use of GGroundOverlay for rendering circles on Google Maps.
To represent the current search areas in Zoom around Town, I decided to overlay two radar-style circles, on the start point map and on the end point map. These circles have to increase and decrease in diameter depending on the user's range selections. See this page in Zoom around Town for a demonstration. When you change the values of the +/- distance dropdowns, and update the maps, you should see the two circles change in size. The radius of the circle corresponds to the selected +/- value.
While the Google Map API (v2) provides GPolyline and GPolygon for drawing lines and polygons on Google Map, it does not provide an overlay class for drawing circles.
One solution is to create a multi-point GPolygon to approximate a circle. Check out this article for an explanation on how this can be done.
A better (in my opinion) solution is to instead use a GGroundOverlay. GGroundOverlay provides a means to overlay an image on top of the map. The image will be scaled automatically as the user zooms the map, so that the image will always cover the same geography area on the map. So, to create a radar-style circle, one only has to find a transparent circle png file, locate the point on the map where the circle should be centred upon, and specify the southwest corner coordinate and the northeast corner coordinate of the imaginery square that bounds this circle (i.e. the GLatLngBounds). Sounds simple, does'nt it? Well almost, to work out the GLatLngBounds coordinates, we need a bit of good old trigonometry.
Pythagoras told us that, for a 2km-radius circle, the four corners of this bounding square will be sqrt(2*2^2) km from the centre of the circle. So, to draw a 2km-radius circle centred upon a specific point, we need the southwest corner coordinate, which is sqrt(2*2^2) at 225 degrees from the original point and the northeast corner coordinate, which is the same distance at 45 degrees from the original point.
Coordinate of a destination point given distance and bearing from a start point
Unfortunately, as the globe is a sphere, working out a destination point coordinate given the distance and the bearing from a start point is no straightforward maths. Fortunately, the good people at Movable Type Ltd. (not to be confused with http://www.movabletype.org) published this informative page. So based on the formula and the Javascript given on the page, I created the Javascript function:
Now, the rest is easy:
To represent the current search areas in Zoom around Town, I decided to overlay two radar-style circles, on the start point map and on the end point map. These circles have to increase and decrease in diameter depending on the user's range selections. See this page in Zoom around Town for a demonstration. When you change the values of the +/- distance dropdowns, and update the maps, you should see the two circles change in size. The radius of the circle corresponds to the selected +/- value.
While the Google Map API (v2) provides GPolyline and GPolygon for drawing lines and polygons on Google Map, it does not provide an overlay class for drawing circles.
One solution is to create a multi-point GPolygon to approximate a circle. Check out this article for an explanation on how this can be done.
A better (in my opinion) solution is to instead use a GGroundOverlay. GGroundOverlay provides a means to overlay an image on top of the map. The image will be scaled automatically as the user zooms the map, so that the image will always cover the same geography area on the map. So, to create a radar-style circle, one only has to find a transparent circle png file, locate the point on the map where the circle should be centred upon, and specify the southwest corner coordinate and the northeast corner coordinate of the imaginery square that bounds this circle (i.e. the GLatLngBounds). Sounds simple, does'nt it? Well almost, to work out the GLatLngBounds coordinates, we need a bit of good old trigonometry.
Pythagoras told us that, for a 2km-radius circle, the four corners of this bounding square will be sqrt(2*2^2) km from the centre of the circle. So, to draw a 2km-radius circle centred upon a specific point, we need the southwest corner coordinate, which is sqrt(2*2^2) at 225 degrees from the original point and the northeast corner coordinate, which is the same distance at 45 degrees from the original point.
Coordinate of a destination point given distance and bearing from a start point
Unfortunately, as the globe is a sphere, working out a destination point coordinate given the distance and the bearing from a start point is no straightforward maths. Fortunately, the good people at Movable Type Ltd. (not to be confused with http://www.movabletype.org) published this informative page. So based on the formula and the Javascript given on the page, I created the Javascript function:
function getDestLatLng(latLng, bearing, distance) {
var lat1 = latLng.latRadians();
var lng1 = latLng.lngRadians();
var brng = bearing*Math.PI/180;
var dDivR = distance/EARTH_RADIUS;
var lat2 = Math.asin( Math.sin(lat1)*Math.cos(dDivR) + Math.cos(lat1)*Math.sin(dDivR)*Math.cos(brng) );
var lng2 = lng1 + Math.atan2(Math.sin(brng)*Math.sin(dDivR)*Math.cos(lat1), Math.cos(dDivR)-Math.sin(lat1)*Math.sin(lat2));
return new GLatLng(lat2/ Math.PI * 180, lng2/ Math.PI * 180);
}Now, the rest is easy:
var EARTH_RADIUS = 6378.137; //in kilometres
function drawCircle(map, centrePt, rangeValue) {
var boundaries = getBoundaries(centrePt, rangeValue);
var circle = new GGroundOverlay("/images/map_overlays/circle.png", boundaries);
map.addOverlay(circle);
}
function getBoundaries(centrePt, radius) {
var hypotenuse = Math.sqrt(2 * radius * radius);
var sw = getDestLatLng(centrePt, 225, hypotenuse);
var ne = getDestLatLng(centrePt, 45, hypotenuse);
return new GLatLngBounds(sw, ne);
}
Sunday, October 04, 2009
Zoom around Town is live!
Hi, just been spending many late evenings working on my pet project Zoom around Town. It is now live on http://zoomaroundtown.appspot.com/!
Zoom around Town is a global cycle route database, based on user contribution, to help city cyclists get around town.
Please help by adding your commuting routes and by spreading the word!
This project makes heavy use of the Google Map API and has been developed in Python on the excellent and "free" Google App Engine platform. I will no doubt be blogging about some of the interesting technical encounters during the project in the near future.
Zoom around Town is a global cycle route database, based on user contribution, to help city cyclists get around town.
Please help by adding your commuting routes and by spreading the word!
This project makes heavy use of the Google Map API and has been developed in Python on the excellent and "free" Google App Engine platform. I will no doubt be blogging about some of the interesting technical encounters during the project in the near future.
Wednesday, August 05, 2009
Datastore Bulk Upload + ReferenceProperty in Google App Engine
I have been working on my first Google App Engine app (a vintage camera database), and I have just got to the point when I need to upload some real data to the Datastore. Essentially, I have an CSV containing all the camera information, including the manufacturer's information and I want to use google.appengine.tools.bulkloader.Loader to upload all this information to Datastore. I have two separate entities representing "camera" and "manufacturer", and I would like to normalize the data as such.
Did someone shout, "Normalization + Bigtable = Bad"? Well, I have yet to make up my mind on this, and for now, I have decided to go with a more traditional relational database model. This may change in the future, but I decided I should still share my experience with uploading relational data as I feel that many people are probably trying to do the same thing.
Before I carry on, these are the two (simplified) classes:
and the CSV:
Effectively, I want four instances of camera and two instances of manufacturer created, with the cameras correctly referencing the manufacturers.
Google App Engine comes with the google.appengine.tools.bulkloader.Loader class which facilitates the process of bulk uploading data to both the development server (local) datastore and the real (cloud) datastore. It supports basic mapping of CSV fields to object attributes. However, dealing with ReferenceProperty turns out to be not so straightforward. A quick Google search led to a few articles suggesting overridding the HandleEntity (or handle_entity?) function when subclassing google.appengine.tools.bulkloader.Loader. Unfortunately, for some reason, I did not manage to get that to work. After some experimentation, I stumbled across my own solution, which actually is quite simple :-)
DISCLAIMER: I HAVE ONLY TRIED UPLOADING DATA TO MY LOCAL DEVELOPMENT SERVER (GOOGLE APP ENGINE 1.2.3) USING THIS METHOD, AND NOT TO THE REAL CLOUD YET...
Instead of injecting the reference in the HandleEntity function, I create a function called get_manufacturer which looks in the datastore for an existing manufacturer based on the manufacturer's name, and returns its key (or the instance itself) if found, or creates a new instance in the datastore before returning the newly assigned key otherwise. I then make that the transformation function for the manufacturer ReferenceProperty.
The following is my loader class with the magic function:
Please refer to http://code.google.com/appengine/docs/python/tools/uploadingdata.html for details on how to do the actual bulk upload.
Did someone shout, "Normalization + Bigtable = Bad"? Well, I have yet to make up my mind on this, and for now, I have decided to go with a more traditional relational database model. This may change in the future, but I decided I should still share my experience with uploading relational data as I feel that many people are probably trying to do the same thing.
Before I carry on, these are the two (simplified) classes:
class Manufacturer(db.Model):
name = db.StringProperty(required=True)
class Camera(db.Model):
name = db.StringProperty(required=True)
weight = db.IntegerProperty()
introduction_year = db.IntegerProperty()
max_aperture_value = db.FloatProperty()
e_mode_p = db.BooleanProperty(default=False)
e_mode_sp = db.BooleanProperty(default=False)
e_mode_ap = db.BooleanProperty(default=False)
e_mode_m = db.BooleanProperty(default=False)
note = db.TextProperty()
manufacturer = db.ReferenceProperty(Manufacturer)
and the CSV:
"Canonet G-III QL17",Canon,620,1972,1.7,FALSE,TRUE,FALSE,TRUE,
"Canonet G-III QL19",Canon,620,1972,1.9,FALSE,TRUE,FALSE,TRUE,
Demi,Canon,380,1963,2.8,FALSE,TRUE,FALSE,TRUE,"Demi is a half-frame camera"
"TRIP 35",Olympus,390,1967,2.8,TRUE,FALSE,FALSE,FALSE,
Effectively, I want four instances of camera and two instances of manufacturer created, with the cameras correctly referencing the manufacturers.
Google App Engine comes with the google.appengine.tools.bulkloader.Loader class which facilitates the process of bulk uploading data to both the development server (local) datastore and the real (cloud) datastore. It supports basic mapping of CSV fields to object attributes. However, dealing with ReferenceProperty turns out to be not so straightforward. A quick Google search led to a few articles suggesting overridding the HandleEntity (or handle_entity?) function when subclassing google.appengine.tools.bulkloader.Loader. Unfortunately, for some reason, I did not manage to get that to work. After some experimentation, I stumbled across my own solution, which actually is quite simple :-)
DISCLAIMER: I HAVE ONLY TRIED UPLOADING DATA TO MY LOCAL DEVELOPMENT SERVER (GOOGLE APP ENGINE 1.2.3) USING THIS METHOD, AND NOT TO THE REAL CLOUD YET...
Instead of injecting the reference in the HandleEntity function, I create a function called get_manufacturer which looks in the datastore for an existing manufacturer based on the manufacturer's name, and returns its key (or the instance itself) if found, or creates a new instance in the datastore before returning the newly assigned key otherwise. I then make that the transformation function for the manufacturer ReferenceProperty.
The following is my loader class with the magic function:
from google.appengine.ext import db
from google.appengine.tools import bulkloader
import model
def get_manufacturer(name):
manufacturers = db.GqlQuery("select * from Manufacturer where name = :1", name)
if manufacturers.count() == 0:
newManufacturer = model.Manufacturer(name=name)
db.put(newManufacturer)
return newManufacturer
else:
return manufacturers[0]
class CameraLoader(bulkloader.Loader):
def __init__(self):
bulkloader.Loader.__init__(self, "Camera",
[("name", str),
("manufacturer", get_manufacturer),
("weight", int),
("introduction_year", int),
("max_aperture_value", float),
("e_mode_p", bool),
("e_mode_sp", bool),
("e_mode_ap", bool),
("e_mode_m", bool),
("note", str)])
loaders = [CameraLoader]
if __name__ == '__main__':
bulkload.main(CameraLoader)
Please refer to http://code.google.com/appengine/docs/python/tools/uploadingdata.html for details on how to do the actual bulk upload.
Saturday, June 06, 2009
GWT Tips 3 - Popups and IE8
If you are like me and find that your GWT popups start killing your beautiful application in IE8, do not despair. I found this really useful article: http://www.mooreds.com/wordpress/archives/000513
Many thanks to Dan!
Many thanks to Dan!
Friday, June 05, 2009
GWT and Spring Security - sample demo download
As requested, I have put together a sample app to demonstrate the GWT and Spring Security (formerly ACEGI) integration technique discussed in http://seewah.blogspot.com/2009/02/gwt-spring-integration-using.html and http://seewah.blogspot.com/2009/02/gwt-and-spring-security.html.
You can download the demo project here: http://www.fotowuj.com/gwt_spring_security_demo.zip
Essentially this app demonstrates how a GWT client can access a Spring-managed authentication service (via AuthenticationServiceServlet) to authenticate against Spring Security, and subsequently access a secured operation (getNumOfPrivatePublications) in another Spring-managed service (via DocumentServiceServlet). Please refer to the aforementioned blog articles for detailed discussion on how to integrate GWT and Spring Security.
You may also want to download a version of GWT.
NOTE THAT I HAVE ONLY TESTED THE DEMO WITH VERSION 1.5.3 OF GWT.
The zip contains a complete eclipse project consisting of:
1) source (both client and server)
2) all dependent jars
3) WebContent/WEB-INF/web.xml, which gets added to the final WAR
4) tomcat/webapps/ROOT/WEB-INF/web.xml, which is required to run in hosted mode
5) build.xml for running app in hosted mode (gwt-shell) and creating war (war)
6) .launch file for running app in hosted mode inside eclipse
You can run the demo in a number of ways:
1) WITH ECLIPSE - having imported the project into eclipse, right-click the .launch file inside eclipse and select "run as...", which will start the app in hosted mode. Note that you will probably have to change the gwt-dev-windows.jar location specified in the .launch file first.
2) RUNNING "gwt-shell" USING ANT - simply run "ant gwt-shell", which will start the app in hosted mode. Note that you will probably will have change the gwt-dev-windows.jar location specified in the build.xml file first.
3) RUNNING "war" USING ANT AND DEPLOYING IN TOMCAT - simply run "ant war", and deploy in Tomcat. Note that you will probably will have change the gwt-dev-windows.jar location specified in the build.xml file first.
Thanks
PS, a reader recently asked the question, "How come that you declare the services (DocumentService for example) both in the client and in the server package? Why not having the DocumentServiceImpl in the server package implement the DocumentService in the client package?"
And this is my anwser:
1) my design is based on the premises that the Spring layer and the GWT layer are very loosely-coupled e.g. in your company, you may already have all the Spring service beans already developed and you are just bolting a GWT frontend on top of this service layer. There are other reasons why you may want your GWT service interface to be slightly different from your Spring service interface as well.
2) The two interfaces ARE DIFFERENT! The GWT service interface getNumberOfPrivatePublications() method throws a checked ServiceSecurityException while the corresponding method in the Spring service interface does not. There may be alternatives that can potentially circumvent this, but to be honest, I have not really thought hard about this :-)
Anyway as I mentioned before, this is by no means the definitive way to integrate Spring Security in GWT. This method works great for me, but depending on your particular circumstances, this may not work so well. I am just hoping to give you guys some ideas :-)
You can download the demo project here: http://www.fotowuj.com/gwt_spring_security_demo.zip
Essentially this app demonstrates how a GWT client can access a Spring-managed authentication service (via AuthenticationServiceServlet) to authenticate against Spring Security, and subsequently access a secured operation (getNumOfPrivatePublications) in another Spring-managed service (via DocumentServiceServlet). Please refer to the aforementioned blog articles for detailed discussion on how to integrate GWT and Spring Security.
You may also want to download a version of GWT.
NOTE THAT I HAVE ONLY TESTED THE DEMO WITH VERSION 1.5.3 OF GWT.
The zip contains a complete eclipse project consisting of:
1) source (both client and server)
2) all dependent jars
3) WebContent/WEB-INF/web.xml, which gets added to the final WAR
4) tomcat/webapps/ROOT/WEB-INF/web.xml, which is required to run in hosted mode
5) build.xml for running app in hosted mode (gwt-shell) and creating war (war)
6) .launch file for running app in hosted mode inside eclipse
You can run the demo in a number of ways:
1) WITH ECLIPSE - having imported the project into eclipse, right-click the .launch file inside eclipse and select "run as...", which will start the app in hosted mode. Note that you will probably have to change the gwt-dev-windows.jar location specified in the .launch file first.
2) RUNNING "gwt-shell" USING ANT - simply run "ant gwt-shell", which will start the app in hosted mode. Note that you will probably will have change the gwt-dev-windows.jar location specified in the build.xml file first.
3) RUNNING "war" USING ANT AND DEPLOYING IN TOMCAT - simply run "ant war", and deploy in Tomcat. Note that you will probably will have change the gwt-dev-windows.jar location specified in the build.xml file first.
Thanks
PS, a reader recently asked the question, "How come that you declare the services (DocumentService for example) both in the client and in the server package? Why not having the DocumentServiceImpl in the server package implement the DocumentService in the client package?"
And this is my anwser:
1) my design is based on the premises that the Spring layer and the GWT layer are very loosely-coupled e.g. in your company, you may already have all the Spring service beans already developed and you are just bolting a GWT frontend on top of this service layer. There are other reasons why you may want your GWT service interface to be slightly different from your Spring service interface as well.
2) The two interfaces ARE DIFFERENT! The GWT service interface getNumberOfPrivatePublications() method throws a checked ServiceSecurityException while the corresponding method in the Spring service interface does not. There may be alternatives that can potentially circumvent this, but to be honest, I have not really thought hard about this :-)
Anyway as I mentioned before, this is by no means the definitive way to integrate Spring Security in GWT. This method works great for me, but depending on your particular circumstances, this may not work so well. I am just hoping to give you guys some ideas :-)
Subscribe to:
Posts (Atom)
Posts
