REST from a GIS developer’s perspective

Lately I have been instrumental in converting our GIS web based functionality into more modularized service based functionality. A year back, our application had a very tightly coupled custom built WMS/WFS server. Imagine being constrained to only using the client that our web application provided, imagine a rogue application that did not follow the WMS/WFS specification, thats what we had - a tightly coupled rogue. This was fine as long as we serviced just one client. But we could not hope to market our product unless we could shift to a more service oriented framework, or atleast a more modular one.

That is when we envisioned the idea to make our GIS more service oriented and modular. This laid the beginning of our trailblazing path towards geoserverization. Now after a couple of months of effort we have a WMS/WFS server that can be used as a GIS service from any client the user prefers to use. All was nice and dandy with this world. Until the day the user wanted more.

One fine day our users decided they wanted to edit the layers on the fly. They wanted to change layer styles through an admin interface in our web application. They even went all the way to asking us to provide functionality, in the admin GUI, to allow them to modify a layer’s data and update it on the fly, as the application was running. Guess who came to my rescue, none other than the REST functionality packaged within geoserver. The rest of this post gives you an introduction to REST and a quick info about geoserver’s REST interface and how I implemented our functionality using REST.

What are thou REST ?

REST or Representational State Transfer is an architectural style, it is not a design pattern nor a true architecture, but just a style, a set of rules that any architecture should conform to. It defines a common interface set that any REST based design can interpret and use. REST revolves around the use of URIs and the use of the HTTP protocol.

REST provides a lot of inbuilt security safeguards since the method that is used can be an indicator of the action that is to be performed – a GET is safe, POST, PUT and DELETE are relatively unsafe. REST provides authorization and authentication through the web server’s inbuilt functionality.

Key Concepts in REST

1. Application state is called as a resource and is identified by a URI. Resources communicate through a standard interface, HTTP. Ex: book is a resource concept. http://myserver.com/book is the URI that returns a file which is a representation of the book resource.
2. Information about a resource is called state and is encoded as part of the resource; these encodings along with the resource are called representations of the resource. Ex: http://myserver.com/book/book1 with an Accept header of text/xml is a particular representation of the resource, which returns the data as XML.
3. Representations are linked together through hypermedia or links. Ex: I can access readers of a book as follows: http://myserver.com/book/users
4. Invoking a REST interface method transfers the state of the resource through representations.
5. The key set of standard interactions are
   • GET – it says give me the information in my chosen format
   • POST – it says add information that I give you
   • PUT – it says replace my information with what you have currently
   • DELETE - it says delete the information that you have

6. REST can evolve with the standard interactions staying the same, new document types can be created and yet the interactions stay the same.
7. It extends the statelessness that HTTP provides, every request is completely encapsulated to perform it’s task independent of previous tasks

REST and Geoserver

Geoserver has been providing a very efficient REST based interface to allow access to geoserver’s admin functionality. Geoserver’s REST defines the following key concepts:

1. workspace – A group of data stores and feature types. This is equivalent to a namespace.
2. datastore – A source of vector based spatial data. It may be a PostGis datastore, shapefile or even a WFS server.
3. featuretypes – A vector spatial resource that originates from a data store
4. coveragestore – A source of raster based spatial data.
5. coverages – A raster based dataset with a coveragestore as the data store
6. styles – describe how a resource should be symbolized or rendered
7. layers – it is a published resource, equivalent to a feature type or coverage
8. layergroups – grouping of layers accessible under a single resource name

One of the key aspects of using REST in my project was to implement two major functionalities:

Provide users a way to dynamically set user styles on the fly.

There are many ways to implement this. The WMS specification allows for SLD and SLD_BODY parameters to be provided as part of the GetMap request. Unfortunately these two methodologies would not work if users store their data in a database. One of the main problems with the SLD parameter is that it requires a SLD file to be created on the server, on the fly which was not an option for us. So. Moving on to SLD_BODY. I was so happy when I implemented this, and it worked, ooooh but I got happy a bit too early. Unfortunately for me I hit upon a restriction in IE, the dreaded GET request length, something I did not even think of since I primarily use Firefox, but unfortunately users use IE. Finally I decide to settle on using the REST interface provided with geoserver.

I decided to use Jersey as my RESTful client API. Jersey builds on a builder pattern to simplify the task of creating a Client, Resources and invoking the RESTful methods, all in a couple of lines of code. At it’s base it uses apache’s HttpClient to perform it’s task with multithreaded efficiency.

A simple PUT as follows:

PUT http://myserver/geoserver/rest/styles/layerName with the SLD string as the request body performed the work for me – modify existing layer’s style with the new style information.

Status codes play a big role in defining how the process was performed. The major ones to be really concerned about are

• 200 – OK - if GET, PUT executed fine
• 201 – Created – if POST executed fine
• 403, 404, 405 – if resource could not be updated for some reason

Allow users to be able to modify a layer

Imagine that you have an application and the user wants to be able to modify a layer on the fly, maybe he has his own data that he wants to upload. Now imagine a restriction that you cannot use the geoserver admin interface. How would you do this? REST baby REST!!

My requirement was to change the originating datastore from a postgis to a shapefile. This was as easy as a PUT to create a new datastore, a DELETE to delete the existing the featuretype and layer, a POST to create a new featuretype, tadaaa…, as easy as pie, all courtesy of the REST API.

How is REST different from RPC

• RPC application is exposed as a network object with one or more exposed public functions, so before communication client must have information about the network object. In REST, this is accomplished through hyperlink relations between objects. RPC uses named operations while REST uses named resources.
• RPC like SOAP can have support for multiple protocols, REST depends on HTTP.
• RPC invariably involves some concept of transaction like WS-AtomicTransaction, REST on the other hand follows HTTP phiolosophy, i.e. deal with failures through retries
• REST provide more diverse formats like JSON,a dn better scalability since it can be cached, SOAP requests cannot be cached

References

1. http://rest.blueoxen.net/cgi-bin/wiki.pl
2. http://tomayko.com/writings/rest-to-my-wife
3. http://en.wikipedia.org/wiki/Representational_State_Transfer
4. http://www.infoq.com/articles/rest-introduction

Tags: gis, java, jersey, REST

getCurrentSession or openSession - a sessionic dilema

Our hibernate code uses two different types of functions to access the session. openSession() and getCurrentSession(). But then I have always wondered the major difference between them. Now that I realized the difference, it is time to share it with the world.

First let’s talk about sessions in Hibernate.Session is defined as a unit of work. Work begins by starting a session and ends by closing a session.

A Transaction can also be defined similar to a session. But a transaction applies to databases. One kind of transaction works in auto commit mode, where there is one short transaction per SQL statement. Hibernate usually disables auto commit mode. A single Hibernate session has the same scope as a transaction. The most common model used is a session per request model where a single session/database transaction is executed for every request coming in from the client.

session per request

Multi step dialogs like wizards are implemented as session per conversation where a single session includes multiple transactions.

session per conversation

A session is opened whenever getCurrentSession() is called for the first time. This creates a brand new session if one does not exist or an existing one if one alrady exists. In tomcat this associates a session with a thread which is created using the underlying ThreadLocal object. But since Tomcat uses thread pooling it is entirely possible that a request may receive a thread with a session already associated with it, thus introducing the possibility of not even creating a brand new session. The method openSession() on the other hand creates a new session but does not attempt to associate it with a thread. But remember openSession() introduces another hitch in that it expects users to handle the closing and flushing of sessions themselves, instead of letting Hibernate do it automatically for us.

This is one of the advantage of using a JTA enabled container since JTA container will handle the session by binding it to a transaction instead of a thread like it happens in a Tomcat environment without JTA which essentially uses JDBC transactions.

References

http://www.hibernate.org/42.html

Tags: hibernate session jpa

Grizzly Comet - what’s grizzling with comet?? - Part 1

I have been working on implementing an interesting application to manage all the  photographs (I will talk more about this project in a later post). In the process, I have been thinking about a feature I want to add to my application which involves alerting users using my web application to be alerted as and when new additions are made to my photo repository. I came across Grizzly Comet and felt an inane desire to try this out on my application, so here is my technical knowledge sharing about this experience

Let’s talk NIO

Before we talk about Grizzly and Comet, let’s try to understand where this whole concept started. Java 1.4 introduced a collection of Java programming API’s called New I/O with features for intensive I/O operations and was developed as part of JSR 51. The main purpose of NIO is to have an implementation that would allow use of the more efficient underlying platform implementation of I/O. A single thread can operate on a bunch of connections instead of having one thread per connection, which provides high degree of performance. This API provides variety of features that include buffers for primitive data types, character set encoders and decoders, pattern matching features, channels. Details about NIO is probably for another post.

Introducing the “Comet” philosophy

Another interesting concept is what is called Comet application model. The basic idea is that a web server would push HTTP data to a browser without the browser requesting the data. The Comet approach typically uses Ajax with long polling to achieve it’s task.

A simple diagram to illustrate the Comet methodology as opposed to the web application model is as below

Web applications have evolved through various stages, as depicted in the following diagram

In the Page by page model, that most web applications started off from and a model that many still follow, every new browser page would send a request to the web server. So essentially we had to refresh the whole page every time a small bit of data on the page changed, which started to become a real pain.

Thus began the era of Ajax where any new information would take the form of Ajax requests to the server to update the web page asynchronously. Web apps would poll the servers periodically for new information, but still the point of the matter is that users have to periodically hit the server to request information.

Finally Netscape introduced the concept called web server push that has evolved into the comet programming model.  Essentially imagine having the benefit of server to client messaging without issues of fat clients.

Imagine a real time event like a baseball game or the stock market scenario, when users would want to be kept informed of events happening during the game or certain stock market events. Such scenarios cannot be implemented using traditional web application methodologies. This is where Comet or Reverse Ajax methodologies fit in.

Comet is a collection of technologies that provides the functionality of web server push through persistent HTTP connection. Comet can be implanted using Streaming where the browser opens a single persistent connection to the server for all server events or long polling.  Many applications use the comet model, some of them include Meebo, Gmail chat, Jotspot, ICEFaces JSF framework. The Comet approach involves a departure from the usual web based platform approach. It may involve storing some kind of state information about the clients who wish to receive notifications on the server., similar to messaging systems. This is the reason most Comet based approaches rely on custom adapted application servers. In Java Jetty and Grizzly have support for Comet based approaches. This type of design is also being introduced through the continuation concept in other platforms. But how does the browser itself stay in contct with the server applications? Some approaches include long polling, dynamic script tags and Iframes which are all non standards. Many client side Comet designs rely on frameworks to iron away incompatibilities, Dojo for example is an Ajax/Comet implemntation

What is Grizzly?

Writing scalable server application has been a big task always. Threading issues caused issue with scaling. This prompted Grizzly to make it’s appearance. It is a HTTP Connector  based on NIO that ships with Glassfish. It is designed to replace Apache Tomcat’s Coyote connector. All Java based web connectors have scalability limited by the number of available threads.  This is where Grizzly improves by providing plug ability of any kind of thread pool.

Grizzly essentially is based on a task based architecture where each task represents an operation. Every task executes on it’s own thread pool or a shared one. The main entry point is Pipeline which has nothing in common with the Catalina Pipeline. The Grizzly Pipeline is essentially a Thread Pool Wrapper and is responsible to execute a task.  The SelectorThread is another important component where the NIO selector is created.  When processing a request, the SelectorThread will create Task instances and pass it on to the Pipeline. There are three types of Tasks - Accept Task - to handle NIO OP_ACCEPT, ReadTask/AsyncReadTask/ReadBlockingTask - to handle OP_READ, Processor Task - to handle OP_WRITE. This SelectorThread can either create one Pipeline per Task or share a Pipeline among tasks.

Grizzly has introduced Comet support and is implemented on top of the Asynchronous Request Processing extension of Grizzly.

So thats that for my technical sharing session for today, in my next post I will give a brief intro to the Comet API and talk about how I used it in my project. Ciao, for now!

References

http://alex.dojotoolkit.org/2006/03/comet-low-latency-data-for-the-browser/
http://weblogs.java.net/blog/jfarcand/archive/2005/06/grizzly_an_http.html
http://www.pathf.com/blogs/2006/06/infrastructure_/
http://searchsoa.techtarget.com/tip/0,289483,sid26_gci1301487,00.html
http://weblogs.java.net/blog/jfarcand/archive/2006/01/introduction_to.html

Tags: comet, grizzly, java

An Introspection of the JSR 135 Specification

Servlets in Brief

A Servlet is a Java technology that allows a container to generate dynamic content. These containers are called servlet containers. Tomcat is one good example of such a servlet container. The basic Servlet interface defines a service() method to handle client requests. The web container handles concurrent requests to the same servlet by concurrent execution of the service method on different threads. A servlet will also have a variety of methods like doGet, doPost, doPut, doDelete and doHead which is a specialized version of doGet returning only a header. The use of a SingleThreadModel interface will guarantee that only a single thread will execute in a given servlet’s service method.

Most Java web frameworks build on the Servlet API and require configuration of Servlets, Filters and Listeners. The JSR 315 will be a part of Java EE 6

Some of the proposed changes

1. Support for continuations. The general idea of continuations is that you can save the current state of an object and return back to the saved state at a later point of time in your execution lifecycle. It is almost like taking a hiatus from your hectic work schedule, going on a trip around the world and coming back to find that everythingis as you left, no catching up, no making up work, just consitnue on as you were before your hiatus, how cool would that be. This is a concept that was popularized by Scheme and also is a major part of Jetty as Jetty Continuations. Now with JSR 135, the request object will now have a suspend() and resume() to suspend and resume execution from a later thread.
2. Plugability - Today all deployment artifacts are defined in web.xml, which can at times just become large and unmanageable. So multiple frameworks can lead to having us define all the servlet mappings in this web.xml file. But imagine if each framework coems withit’s own web.xml that defines only the components that framework needs, and if we leave the job of assembling the components together to the container, our task would be so much easier, and our project would be so much easier to maintain. This spec introduces web fragements that does exactly this task. In addition to this kind of plug ability the new JSR also provides a feature where servlets can be inserted into the container dynamically at runtime using the servletContext.addServlet() and servletContext.addServletMapping(). One drawback of this feature is that there could be potential security risks since JARS can register their own servlets.
3. Annotations - I am sure many of you are familiar with annotations, a technique to try to eliminate configuration files by introducing the configuration details in the source code file. JSR 135 introduces POJOS for servlets, so no more extending from HttpServlet. Just use the annotation @Servlet(urlmapping = {’mapping‘}, name=“abc” } and then use @GET, @POST, @PUT, @DELETE, @HEAD. Servlet Filters are defined using @ServletFilter and @FilterMapping. Unfortunately this Annotations methodology is not as hunky dory and cool as it looks, there are lots of potential problems that Roy van Rijn has mentioned on his post on theserverside.com which include potential issues with having two methods with the same annotations like @GET, there are no IDEs that are setup to detect Servlet using annotations. Also with annotations what we lose is that previously we could just override the methods, now you cannot do that.
4. Programmatic entry - Potentially there might be support for programmatic login and logout as part of the HttpServletRequest and HttpSession.
5. HttpOnly Cookies - Ability to prevent client side scripting code from getting access to cookies by defining HttpOnly cookies.

References:
http://www.theserverside.com/tt/articles/articles.tss?l=PonderingAboutJSR135
http://weblogs.java.net/blog/mode/archive/2008/04/servlet_30_jsr_1.html

Tags: jsr 135, servlet

transactions in a Spring context - Take 1

Spring as a framework has made a lots of things much simpler for developers. Transactions is one of the area that Spring has simplified a lot. Have you ever wondered how the nitty gritty details of transaction worked in Spring? Well let me share some of what I learnt

What is a transaction?
A Transaction is defined as an indivisible unit of work. It always follows the ACID properties.

• Atomicity- all or none principle - complete the transaction in it’s entirety or do none of it.
• Consistency – do a drew brees (:-P ) - as you go from one state to another, data integrity should be maintained
• Isolation – transactions should not effect each other
• Durability – persistent state should be maintained.

A transaction can either end as a commit or a rollback.

Transaction models

There are many different transaction models available: Local transaction model, programmatic transaction model, declarative transaction model.

• Local transactions are resource specific, for example a transaction associated with JDBC. The local resource manager handles these transactions. With this model the developer will manage the connections to the transaction manager. JDBC transactions are controlled by DBMS’s transaction manager. The Connection object supports transaction demarcation. JDBC has auto commit turned on by default, so commit is done after each SQL statement executes. The problems with this model is that it introduces developer errors when coding the connection logic.
• Programmatic transaction model leverages JTA. The developer writes code to manage the transaction and not the connection as in the local transactions. The UserTransaction interface  comes into play in EJBs and uses the begin(), commit(), rollback() functions. Spring uses TransactionTemplate to perform the same task with the execute() method that takes a TransactionCallback object. The main disadvantage of this method is that lot of responsibility rests on the shoulders of developers to make sure they handle the transaction semantics properly.
• Declarative transaction model is commonly known as Container Managed Transaction in the EJB world. The general idea behind this is that the container will manage the commit or rollback of the transaction, the developers will only to tell the container how to manage the transaction through configuration parameters – ejb-jar.xml in case of EJBs or ApplicationContext.xml in Spring.  In Spring we use the TransactionProxyFactoryBean to implement this transaction model. EJB3.0 introduces the use of annotations for this transaction model. We also need to tell the container how to manage the transaction. This is done using a TransactionAttributeSource bean associated with a transaction attribute which has 6 different settings- Required, Mandatory, RequiresNew, Supports, NotSupported, Never

When do we use programmatic transaction model?

Usually programmatic transactions are useful when user we have client initiated transactions. If a client is making multiple remote service calls for a request, then transaction is maintained on the client side.

Another reason to use this methodology is if using JTA is very expensive in terms of performance. In such cases where we might need to squeeze every ounce of performance, so we might decide to use JTA only when absolutely required. This is called localized JTA transactions.

Another scenario is when there are long running transactions. In EJB’s these are handled using stateless session beans, where you might have multiple stateless session bean methods as a part of a single transaction.

For all other situation is is advisable to use the declarative transaction model.

JTS and JTA

Now we need to understand the difference between JTA and JTS. JTA is the interface developers use to manage transactions. The JTS or Java Transaction Service on the other hand is an underlying service that implements JTA and provides that to application servers. So essentially JTA provides the interface and JTS is the implementation similar to the relation between JDBC and database drivers.

Tags: spring transaction