Friday, October 29, 2010

Standalone XTS and lightweight clients

I've recently been working with a community user to help restore the ability to deploy the Web Services Transactions (XTS) code outside of JBoss AS. XTS is a layer built on top of the JBoss Transactions core which allows you to implement transactional web services and transactional web service clients. Those long enough in the tooth to know about the previous incarnation of the XTS code when it was part of the Arjuna codebase will recall that in the good old, bad old days XTS had to be deployed bundled in with the web service client and/or the web service itself. When we re-implemented XTS to conform to the WSTX 1.1 standard we rebuilt it so that it deployed as a service inside JBoss AS. We also relaid the internal message transport on new foundations. Communications between the client or web service participants and the transaction coordinator were rebuilt over the JaxWS implementation provided by JBossWS-Native. More recently we upgraded XTS so that it also works with JBossWS-CXF.

Although this decision appears to tie XTS tightly into the JBoss AS product it was actually motivated by the desire to render the code independent of a specific transport layer. In theory JaxWS provides an abstraction that is available in many implementation flavours and, in theory, this means that it should be possible to deploy the 1.1 implementation in any container that
  • provides support for deployment of JaxWS endpoints on the server side and
  • allows for creation of JaxWS service proxies on the client side
That's pretty much all that is needed to allow XTS to run in any web service container, except ... it ignores a significant lacuna in the JaxWS specifications.

XTS requires the use of Web Service Addressing (WSA) headers on the protocol messages sent between the client and transaction coordinator or between the participant web service and the coordinator. The original draft JaxWS-WSA specification defined an API which enabled the two sides of a JaxWS exchange to access and manipulate WSA headers. However, this feature got shunted off into the long grass and the final version only provided a minimal WSA capability which made it impossible to implement the WSTX spec using only JaxWS standard functionality.

Luckily for me, JBossWS Native and the CXF project have both always provided an implementation of the original draft WSA feature and, modulo a few annoyingly variations (that's the problem with not standardising), this means that both these WS stacks include all that is needed to deploy the XTS service. JBossWS now includes a small library containing a simple abstraction layer which hides the minor differences in the WSA implementations. That is all that is needed for XTS to deploy inside JBoss AS over either JBossWS-CXF or JBossWS-Native. However, this also re-opens the possibility of bundling the XTS code with the JBossTS code and CXF as a standalone deployment.

Obviously, there is not a great deal of benefit in deploying a transactional web service outside of an application server. It's very likely that you will want such a web service to run inside a container located in a managed environment with 24/7 monitoring, support and so on. A web service is likely to need to employ a database and require configuration of data backups, crash recovery, centralised logging, IT staff on hand to deal with catastrophic failures etc.

Deployment of a lightweight transactional web service client is a much more interesting proposition. If you can bundle the transactional web service client software in a simple desktop Java application then this provides an easy management path for exposing reliable distributed service applications to users. The desktop client negotiate with a networked coordinator service when it creates a transaction. It can use normal web service invocations to request service from transactional web services located in the intranet or even on the internet. The web services negotiate with the same coordinator to ensure that they either all commit their changes or all revert to their original state. The more heavy-weight apparatus needed on the coordinator or web service hosts to ensure reliability, availability, security etc are not actually needed on the client node.

In theory all the transactional client needs to be able to do in order to participate in a Web Service Transaction is talk to the coordinator and the web services using JaxWS client capabilities built over an HTTP client library. There is no reason for the client to require all the capabilities of a full web service container and if definitely does not need to impose the management overheads that running a full container would mandate. The client only requires minimal installation, configuration and maintenance to operate as an HTTP client. It does not need to be involved in automatic or manual crash recovery procedures. It does not need to take part in centralised logging in order to allow monitoring/supervision by IT support staff. Nor does the client need to expose service endpoints on a public network interface and, in consequence, open holes in the client machine's firewall. Bundling the CXF client libraries with the XTS client code in a standalone Java app seems like a very attractive proposition.

In practice things are not quite so simple as that which is why I have been busy for the last month or two. The practical problems were twofold. The first problem was that XTS was built to be deployed as a monolith. When you install the XTS service inside JBoss AS you actually install the code required by all 3 of the agents involved in a transaction: the client, the participant web service and the transaction coordinator. The XTS code actually comprises about half a dozen main software services with their associated web services. These software services are layered so as to support the functions defined by the WSTX specifications. Unfortunately, the needs of the three agents, client, web service, coordinator, cut across these layers making it hard to decouple them.

So, when it comes to deploying a transactional application you might decide that you want to deploy your client application to one JBoss AS instance on Host1 and your web services on two other JBoss AS instances running on Host2 and Host3. For better reliability you might want to the coordinator services to operate in a separate JBoss AS instance on Host4. Unfortunately, up until a few weeks ago you would have to deploy, load and initialise the XTS services for all three types of agent on all of these hosts.

Well, not any more. There is now a simple configuration mechanism which allows you to enable or disable the client, participant web service or coordinator capabilities in a given XTS deployment. There is still a single XTS service archive to drop into the JBoss AS deploy directory. However, you can easily configure the deployment using either dependency injection or file-based property configuration to ensure that only the code and services you want get loaded and initialised. So, you can configure Host1 only to initialise the client capability, and so on.

If you want to deploy to a non-JBoss web container it is now also straightforward to isolate the configuration elements which control deployment of the JaxWS endpoints used internally by XTS. The XTS service archive contains several web archives which do nothing more than map endpoints exposed by the 3 XTS agents. These can easily be unbundled and the mapping can be redefined using an alternative deployment framework like, say, Spring. All you need to do to deploy XTS outside JBoss AS is to bundle CXF, the XTS and JBossTS libraries and the JBossWS WSA compatibility library with your deployment, configure which XTS agents you want to initialise then call the JBossTS and XTS start and stop methods during deployment and undeployment of your application.

This is all you need if you want to deploy to a web container. Unfortunately there is still a second barrier to deploying a simple, lightweight transactional client. This is a harder problem because it lies within the WSTX standard itself. Unfortunately, the WSTX specification, or at least the WSAT component (which specifies the Atomic Transaction protocol), requires a transactional client to expose a service endpoint. WSTX bypasses this issue as far as the WSBA (Business Activity) protocol specification is concerned, but only because it leaves the corresponding part of the WSBA specification undefined! In practice the same requirement applies for WSBA.

This may seem a bit bizarre but it's a rather unfortunate artefact of how WSTX was defined. I guess that at the time WSTX was defined it was felt that anyone implementing a transactional client would only want to deploy it inside an application server. So, this was not considered to be an issue. In order to explain why the spec is like this and what is required to fix it I'll need to go into some of the details of what the spec actually requires a client implementation to do.

The problem does not arise when an application client starts a transaction, either by calling UserTransaction.begin() or UserBusinessActivity.begin(). Under the hood these two calls use JaxWS to talk to an XTS service called the Activation Coordinator. This service employs a request-response message exchange pattern (MEP). The client opens an HTTP connection and sends a create request. The service creates a transaction and uses the HTTP back channel to hand back a transaction context containing a unique identifier for the transaction and the address of another coordinator service called the Registration Coordinator. This is the same information which gets smuggled into client requests to the transactional web services, allowing them to talk to the Registration Coordinator and register as a participant in the WSAT or WSBA transaction (although, confusingly, the standard term used for this operation is enlist).

The application client does not just stash away the context returned by the Activation Coordinator and then return fro the begin() call. It also registers (or enlists) as a participant in the transaction. Although it uses the same operation as the web services it does so for a different purpose. A transactional web service has to enlist so that it can be told when to prepare and commit (or, possibly, abort) any changes made during the transaction. The client has to enlist in order to be able to complete the transaction. In a WSAT transaction the client sends a message to the Registration Coordinator to enlist with the WSAT Completion Coordinator service. This allows it to send a termination request, either Commit or Rollback, later on. Once again the enlist operation uses a request-response MEP but this time the response on the back channel includes the address of the Completion Coordinator service. Likewise, a WSBA client enlists with the Termination Coordinator service and gets back that service's address.

The problem arises when the application client code calls UserTransaction.commit() or UserBusinessActivity.close() to terminate the transaction. The XTS code uses JaxWS to send, respectively, a Commit or Close request to the Completion or Termination Coordinator service. Alternatively, if the application client code calls UserTransaction.rollback() or UserBusinessActivity.cancel() the XTS code uses JaxWS to send, respectively, an Abort or Cancel request. Unfortunately, in all of these cases the WSAT specification mandates that this request empoloys one-way MEP. That means that the client sends the request then closes the connection without waiting for a response. So, how is it supposed to find out when the request has been handled and what happened?

Well, that is why the client needs to expose a JaxWS endpoint. The WSAT client has to implement a Completion Initiator service which will accept incoming messages using a one-way MEP, either a Committed or an Aborted message. After posting , say, a Commit request the XTS code places the calling thread in a wait until one or other messages is received by the Completion Initiator service. This service wakes up the waiting thread and notifies it of the outcome. So, what is essentially a request-response exchange is achieved using two independent messages on two independent HTTP connections one initiated from the application client to the coordinator host and one initiated from the coordinator host back to the client. The WSBA specification does not actually specify a termination protocol but the old Arjuna implementation adopted a similar model based on two one-way messages thus requiring a WSBA client also to expose an end point, in this case for the Termination Participant service.

So, how does the Completion or Termination Coordinator get hold of the address for the client endpoint? Well, the answer to that question really explains how and why this problem comes about. The enlist request supported by the Registration Service expects to be passed three values, a unique identifier for the enlisting party derived from the transaction id, the name of the service the caller wants to enlist for and, finally, a URL providing the missing address. Once the registration request has completed both sides have each others addresses. The protocol expects them to conduct any further communications using a sequence of one way messages labelled with the unique identifier supplied in the enlist call.

One way messages are not a bad idea for communications between the coordinator and the participant web services. Using a request-response MEP requires holding open a connection while waiting for the response. For, say, a Prepare-Prepared exchange this might take a long time, especially if a web service has made a lot of state changes which have to be logged to disk. Holding open lots of connections in a busy web server might cause a resource bottleneck. Also, request-response is only useful if the exchange is always a simple two-way exchange. WSBA sometimes requires longer message sequences not easily mapped onto the request-reponse model. For example, if a WSBA web service is told to complete its changes and finds that it cannot do so the 3-way message exchange, Complete, CannotComplete, NotCompleted is required.

At the client the first of these concerns is less important (there is normally only one connection per client) and the second does not apply. If the completion requests could be implemented using a simple request-response exchange then this would remove the need to expose service endpoints and, hence, allow deployment of a really lightweight client. So, the solution we have adopted is to augment the standard and provide an alternative version of the Completion and Termination Coordinator services which support this type of MEP. Our coordinator service implementations exposes two new endpoints alongside the original completion services, the Completion Coordinator RPC service and the Termination Coordinator RPC service. These services have no corresponding Initiator service. Our client implementation can be configured to use either the normal service or the RPC vwersion. So, if you deploy a client to a web container you can configure it to use the standard one-way MEP requests to the standard services. If you deploy a client as a standalone Java app you can configure it to use request-response MEPs to the extended completion services.

Of course, you can only use this lightweight client if you point it at a JBoss XTS coordinator. If you tried to use another vendor's implementation of WSTX then you would get an error when you begin your transaction because the 3rd party coordinator service would not recognise enlist requests for our extended completion services. This is not quite such a leap into the unknown as it appears. WSBA does not specify completion protocol. So, if you try to use another vendor's coordinator service with our WSBA client library it will fail whichever type of client you deploy. That's a very good reason why the WSTX specification should be extended to include a WSBA completion protocol and to add the request-response MEP variants of the completion protocols so they sit alongside the one-way MEP versions.