Posted by Ajay
Lately I have been instrumental in converting our GIS web based functionality into more modularized service based functionality. A year back, our application had a very tightly coupled custom built WMS/WFS server. Imagine being constrained to only using the client that our web application provided, imagine a rogue application that did not follow the WMS/WFS specification, thats what we had - a tightly coupled rogue. This was fine as long as we serviced just one client. But we could not hope to market our product unless we could shift to a more service oriented framework, or atleast a more modular one.
That is when we envisioned the idea to make our GIS more service oriented and modular. This laid the beginning of our trailblazing path towards geoserverization. Now after a couple of months of effort we have a WMS/WFS server that can be used as a GIS service from any client the user prefers to use. All was nice and dandy with this world. Until the day the user wanted more.
One fine day our users decided they wanted to edit the layers on the fly. They wanted to change layer styles through an admin interface in our web application. They even went all the way to asking us to provide functionality, in the admin GUI, to allow them to modify a layer’s data and update it on the fly, as the application was running. Guess who came to my rescue, none other than the REST functionality packaged within geoserver. The rest of this post gives you an introduction to REST and a quick info about geoserver’s REST interface and how I implemented our functionality using REST.
What are thou REST ?
REST or Representational State Transfer is an architectural style, it is not a design pattern nor a true architecture, but just a style, a set of rules that any architecture should conform to. It defines a common interface set that any REST based design can interpret and use. REST revolves around the use of URIs and the use of the HTTP protocol.
REST provides a lot of inbuilt security safeguards since the method that is used can be an indicator of the action that is to be performed – a GET is safe, POST, PUT and DELETE are relatively unsafe. REST provides authorization and authentication through the web server’s inbuilt functionality.
Key Concepts in REST
- Application state is called as a resource and is identified by a URI. Resources communicate through a standard interface, HTTP. Ex: book is a resource concept. http://myserver.com/book is the URI that returns a file which is a representation of the book resource.
- Information about a resource is called state and is encoded as part of the resource; these encodings along with the resource are called representations of the resource. Ex: http://myserver.com/book/book1 with an Accept header of text/xml is a particular representation of the resource, which returns the data as XML.
- Representations are linked together through hypermedia or links. Ex: I can access readers of a book as follows: http://myserver.com/book/users
- Invoking a REST interface method transfers the state of the resource through representations.
- The key set of standard interactions are
- GET – it says give me the information in my chosen format
- POST – it says add information that I give you
- PUT – it says replace my information with what you have currently
- DELETE - it says delete the information that you have
- REST can evolve with the standard interactions staying the same, new document types can be created and yet the interactions stay the same.
- It extends the statelessness that HTTP provides, every request is completely encapsulated to perform it’s task independent of previous tasks
REST and Geoserver
Geoserver has been providing a very efficient REST based interface to allow access to geoserver’s admin functionality. Geoserver’s REST defines the following key concepts:
- workspace – A group of data stores and feature types. This is equivalent to a namespace.
- datastore – A source of vector based spatial data. It may be a PostGis datastore, shapefile or even a WFS server.
- featuretypes – A vector spatial resource that originates from a data store
- coveragestore – A source of raster based spatial data.
- coverages – A raster based dataset with a coveragestore as the data store
- styles – describe how a resource should be symbolized or rendered
- layers – it is a published resource, equivalent to a feature type or coverage
- layergroups – grouping of layers accessible under a single resource name
One of the key aspects of using REST in my project was to implement two major functionalities:
Provide users a way to dynamically set user styles on the fly.
There are many ways to implement this. The WMS specification allows for SLD and SLD_BODY parameters to be provided as part of the GetMap request. Unfortunately these two methodologies would not work if users store their data in a database. One of the main problems with the SLD parameter is that it requires a SLD file to be created on the server, on the fly which was not an option for us. So. Moving on to SLD_BODY. I was so happy when I implemented this, and it worked, ooooh but I got happy a bit too early. Unfortunately for me I hit upon a restriction in IE, the dreaded GET request length, something I did not even think of since I primarily use Firefox, but unfortunately users use IE. Finally I decide to settle on using the REST interface provided with geoserver.
I decided to use Jersey as my RESTful client API. Jersey builds on a builder pattern to simplify the task of creating a Client, Resources and invoking the RESTful methods, all in a couple of lines of code. At it’s base it uses apache’s HttpClient to perform it’s task with multithreaded efficiency.
A simple PUT as follows:
PUT http://myserver/geoserver/rest/styles/layerName with the SLD string as the request body performed the work for me – modify existing layer’s style with the new style information.
Status codes play a big role in defining how the process was performed. The major ones to be really concerned about are
- 200 – OK - if GET, PUT executed fine
- 201 – Created – if POST executed fine
- 403, 404, 405 – if resource could not be updated for some reason
Allow users to be able to modify a layer
Imagine that you have an application and the user wants to be able to modify a layer on the fly, maybe he has his own data that he wants to upload. Now imagine a restriction that you cannot use the geoserver admin interface. How would you do this? REST baby REST!!
My requirement was to change the originating datastore from a postgis to a shapefile. This was as easy as a PUT to create a new datastore, a DELETE to delete the existing the featuretype and layer, a POST to create a new featuretype, tadaaa…, as easy as pie, all courtesy of the REST API.
How is REST different from RPC
- RPC application is exposed as a network object with one or more exposed public functions, so before communication client must have information about the network object. In REST, this is accomplished through hyperlink relations between objects. RPC uses named operations while REST uses named resources.
- RPC like SOAP can have support for multiple protocols, REST depends on HTTP.
- RPC invariably involves some concept of transaction like WS-AtomicTransaction, REST on the other hand follows HTTP phiolosophy, i.e. deal with failures through retries
- REST provide more diverse formats like JSON,a dn better scalability since it can be cached, SOAP requests cannot be cached
References
- http://rest.blueoxen.net/cgi-bin/wiki.pl
- http://tomayko.com/writings/rest-to-my-wife
- http://en.wikipedia.org/wiki/Representational_State_Transfer
- http://www.infoq.com/articles/rest-introduction
Tags: gis, java, jersey, REST
Posted by Ajay
Ok, so we spent the last two posts talking about geoserver, why we chose it, and some general information about geoserver and it’s architecture. Now let me talk about the geoserverizing of our mapping service.
Mapping our way, in the old times …
I very fondly think back to the old days when we (as in my software development team) used to have our GIS functionality all packed up into a tightly coupled application. We initially began to develop our own WMS/WFS rendering server; since geoserver had not seemed a colorful alternative when my project came into existence oh a couple of years ago (it’s never polite to ask a project his or her age). Our project is a struts based application. So the model view controller that included the rendering code did provide us with some semblance of loose coupling. Here is a look at our application as it was
We use a Mapbuilder based client for our GIS mapping component. Our application handled a lot of the image rendering using geotools. We did all our rendering from using geotiff readers and writers to simple polygon rendering right in our handlers. The handlers would communicate with our layer classes which would use hibernate / plain SQL to talk to the database. It definitely looks modularized doesn’t it? But imagine what happens if you decide to add, a jar of actions doing all other things like managing accounts, managing logs and status boards, and even displaying beautiful charts of our data, none of which is even closely related to our GIS functionality. Imagine the confusion that could happen when the whole world is transitioning towards a service oriented infrastructures. This is exactly what happened.
A New revelation …
We were all fine and happy until one day we realized that our GIS mapping tool could be a potential source of money a.k.a marketable value. That is when we had a rethink about our mapping component and decided to move it out of its monolithic cage. We decided to set our GIS component free.
We had been following the happenings of the geoserver world quite closely for some time and decided it was an appropriate time to try to make the switch to geoserver. Now the only issue was the migration of all our layer and rendering functionality. This task was not really that easy for the following reasons and of course many more too.
- Incompatibility of our jar files
- All our styles were being rendered from the database
- The sheer number of layers we had to transfer to geoserver, somewhere in the range of 200 layers.
- We had some custom layers that could not fit in the geoserver framework
- Pain of having to recreate our base maps as image pyramids.
- Naming conventions used in geoserver.
The Geoserverization has begun…
But in spite of this we managed to go all the way and get geoserver up and running. The diagram below depicts how much more service oriented our application has become, for our mapping functionality.
We now have a clear separation of code between our application and our geographic aspects. We were able to provide our users with a service oriented way to access our GIS mapping data, providing client independence and data independence.
The solution to our problems??
So how many of the problems we talked about before were really solved?
- We definitely had huge problems with jar files especially since we were using older 2.4 geotools library. So we had no choice but to transitioning our application to geotools 2.5. Geotools 2.5 has added new way to do things many things as compared to 2.4. But we were able to get by this issue and were successful in doing the transitioning.
- In order to accommodate the styles coming from the database, we used a feature of Mapbuilder that allows StyleList to be passed through the context document. This works very effectively for us.
- The job of transferring the layers along with their configuration was made really easy by the fact that geoserver uses HTTP as the protocol, providing us with GET requests that were tapped using a very helpful tool called LiveHTTPHeader. We captured the requests being sent and created an ingenious tool that would read the layer list from the database and then send a complete series of GET requests to geoserver for each layer, with each request executing one particular task of the geoserver feature creation work flow. Now with the introduction of REST interface in Geoserver 1.7.3 we feel that we can migrate our tool to using REST to populate geoserver, which would make things much more efficient and simpler.
- Lots of our custom layers are unfortunately still hooked into our main application, so we have been forced to use two server hitting URLs, one for all our generic layers and one for our custom layers. Although I feel that the plug-in mechanism in geoserver could prove to be a potential solution to this problem.
- The basemaps were a pain to convert into image pyramids, but we did it and it works beautifully. Someone really clever once said, “No pain no gain”.
- We have now also shifted to follow the naming convention as in geoserver, with the “namespace” prefix.
Suffice to say our migration has been largely successful, though some aspects of our application like the custom layers do need some thinking to completely separate it from our application, but we are on the road to make our mapping component a full feature web service, courtesy of geoserver.
In future editions of this series I will talk about the geoserver plugin mechanism and possibly WFS-T functionality that geoserver provides. In addition I might also talk about the new REST interface that geoserver 1.7.3 has introduced.
Until then, as my people say, alvida (goodbye)
Tags: geoserver, geotools, wfs, wms
Posted by Ajay
The History behind it
Let me first tell you a story. In a place far far away, couple of million minutes ago, in the year AD 2001, a great entity , well a startup agency actually, called TOPP was looking into designing tools to make democracy and government more transparent, and with a flash came … .Geo server.
Allright enough of all that dramatization, in short the people from TOPP envisioned a geospatial web which led to Geoserver. Geoserver is based on the geotools project. In addition geoserver has a interconnected relationship with OGC, the Open Geospatial Consortium, Refractions Research – creators of PostGIS, OpenLayers and even Google.
Geoserver forms the beginning of the standardization in the GIS arena. It follows the WMS, WFS and WCS specifications to the letter, and forms the platform to develop GIS applications based on these specifications.
It’s design
At a high level view Geoserver essentially consists of many different modules that are actively interacting with each other. All these modules are connected using Spring IOC so that at runtime a module can make use of services provided by other classes.
Geoserver reads data in a wide variety of formats from PostGIS, OracleSpatial, ArcSDE to shapefiles and geotiff. It can also produce KML, GML, Shapefiles, GeoRSS, GeoJSOn and multitudes of other formats.
Geoserver essentially consists of two aspects – the configuration and data store aspect, and the rendering aspect. All configurations in geoserver are done through the admin interfaces and XML configuration files.
- The service.xml defines all service level configuration options. It defines parameters for WFS, WMS and ECS services.
- The catalog.xml on the other hand defines datasources, styles and namespaces. In addition geoserver introduces coverages and featureTypes that provide metadata information about raster layers and vector layers respectively.
Geoserver configurations consists of many different namespaces, to which featuretypes, datastores and styles can be associated. Namespace provides a way to logically group all the data in geoserver. Now digging deeper, we see FeatureTypes.
FeatureType represents each individual layer or feature that needs to be shown on the map. Another entity at the same level of a FeatureType is a Coverage. Coverage is usually created for raster layers and FeatureTypes represent vector layers. All FeatureTypes have a source of data called a DataStore.
DataStore is essentially a source of data for rendering of features. Geoserver supports many different data stores including Web Feature Server, Property files, Shapefiles and database based datasptres. CoverageStore is another entity at the same level as a DataStore but includes raster based data formats like ArcGrid, geotiffs, image mosaics, etc.
Every feature will have certain styles for rendering on the map. All styles for layers are configured using the Style entity. Styles are represented using SLD or Style Layered Descriptor files.
The rendering component follows the WMS, WFS and WCS specification and are uses geotools as the rendering API.
This concludes a very general architectural over view of geoserver. In future posts I will talk about other aspects of geoserver, the OGC specifications and some of the problems encountered on this grand geoserver experiment.
References
Tags: geoserver, gis
Posted by Ajay
The why’s and the wherefores!!
Over the past few weeks I have been working on moving our GIS mapping solution into geoserver, a popular open source WMS/AFS server. We have been using a custom built WMS/WFS server that grossly neglected everything that the WMS specs talks about. I mean, I’m talking willfull disobedience of everything the WMS spec was talking about. It was so tightly coupled with our application that when circumstance arose for us to use the same server for another application, we just fell flat on our face. That prompted us (of course with a little monetary support from our customer) to decide to shift to geoserver, which was a major re-architecture of the way we were doing GIS.
Why did we decide to do this?
Some of the reasons we decided to do this
- Our client wanted us to provide a SOA mechanism to provide GIS rendering independent of the client.
- Freedom from constant tinkering and code maintenance to make our product faster and more effective
- Standardization with OGC specification
Some of the challenges we faced
- Tight coupling of web application functionality with WMS/WFS server
- Custom chemical plume layer
- Incompatible jar files
- Programmatic way to populate geoserver with the 200 different layers that we had to display.
- Providing dynamic styling for many of the layers, where the styles were defined by the user
- User layer admin interface coupled to database
Before I delve more into depth I will spend a couple of posts talking about geoserver and caching mechanisms. So in my next post, I will introduce geoserver. Till then ciao!!
References
http://geoserver.org/display/GEOS/Welcome
Tags: geoserver gis
Posted by Ajay
Eeny meeny miny moose…which mapping API do i choose??
I have been working for quite some time on a Proof of Concept for our GIS web mapping application. Currently our system has a basemap that has been superbly crafted by a good group of GIS engineers, we even have maps that are hill shaded, sounds cool dont it? Trust me it’s cooler than it sounds. Plus we render it using our custom WMS/WFS server with a Mapbuilder / OpenLayers client.
Anyhow, looks like some of our customers were enamored by Google maps (one of those things I call crowd-fluence , getting influenced by the crowd), looks like everyone around is using Google maps so they wanted to use google maps.
So there i was putting on my researcher’s pants, trying to provide a proof of concept for our customers adding a feature in our web application to allow them to switch between Google Maps, Yahoo Maps, Microsoft Virtual Earth and our own custom built basemaps.
In between these lines let me tell you Openlayers is one of those awesomest coolest client APIs that have helped developers like me to add a wooing factor to our open source web applications. Mapbuilder adds onto this to add a lot of useful functionality to OpenLayers in terms of widget. (Why it did not become very
popular i dont know, cause it is so easy to use along with an XML configuration file that allows us to add widgets quite so easily)
Anyhow in the process of my experiments, here are some things I came to realize.
- Yahoo road layer is by far the most coolest looking and of course it is very easy to read. Also it looks like it has built in geocoding which is awesomeness personified. But yahoo maps are not that well developed internationally. Another very cool thing about yahoo map is hillshading on their road basemap that google does not provide, although google’s terrain view is seperate from their map layer and is awesome.
- Google’s terrain view is really cool, but one unfortunate issue with Google map API is that it requires us to have a seperate API key for each machine, whereas Yahoo does not really put that kind of restrictions. But Google’s data is very accurate and up to date.
3. Microsoft’s virtual Earth in my opinion does not match either google or yahoo in terms of road map clarity but it does have a really clear satellite view, with a better color coordination. Also Microsoft’s road map view has lots more details for a given level as compared to google or yahoo which could be a blessing or a curse depending on how you look at it. Of course one thing that Microsoft maps DO NOT have which is really cool is they have NO ADS!! 
4. Of course there is a whole new open source initiative in the form of open street maps (http://www.openstreetmap.org/) that I think has some great potential. Although the level of clarity is not upto either google or yahoo, but they do offer you the benefit of being able to manage a lot of your own GIS data through GPS systems (and of course contribute some of the data collected back to the community). You can even export basemaps so as to work your GIs application without an internet connection, that should prove a boon to many GIS enthusiasts.
References
Tags: google, mapping, openlayers, openstreetmap, virtual earth, yahoo
