Application Context state replication

This article describes what Application context state replication is and how to set it up

Introduction
RMI-cluster enabling a Server
Debugging
Troubleshooting

1 Introduction

As the Servlet specification mandates that the Servlet Context is local to the JVM where it was created and thus not suited for distribution, the Applications Context should be used for Application wide values that should be replicated instead.

The Application Context is the writable (and readable) part of the global JNDI Context that the Application has access to. This context is used to store Application wide state (notice that environment entries and other deployment specific settings are not replicated).

Throughout this article, a cluster will mean a number of RMI-cluster enabled Servers. Each RMI-cluster enabled Server will be refered to as a cluster node.

When a cluster node is started, it is given a specific RMI-cluster service. This service manages replication of the Applications context.

The RMI-cluster services uses multicast packets to send and receive state changes in a cluster. These packets are sent over the network where they are picked up by other cluster nodes using the same multicast address and port.This also means that the data is available to any application, such as a custom cluster monitor, that has registered itself as a listener on the same multicast address and port.

Multicast addresses have IP addresses in the range 224.0.0.0 to 239.255.255.255. Normally, a Multicast packet has a TTL (time to live) value less than 60, which normaly limits the packet to the local network and thereby making sure that they wont flood the network. For more information on multicast and related issues such as TTL, please use a resource as this as multicast in itself is out of scope for this article.

In order for a Application on one cluster node to share the Application context state of the same Application on another cluster node, both Applications need to be deployed with the same Applications deployment name (the name attribute specified in the application tag in server.xml).

Figure 1: The Server-wide RMI-clustering service replicates the Context state for all deployed Applications to all other RMI-cluster enabled Servers listening to the same multicast address/port (in this case 230.0.0.1:9127 which is the default).

The RMI-cluster sevices allows for any remote RMI-objects or Serializable objects to be replicated over a cluster. Other objects will not be replicated and error messages will be put into the Application log file. Serializable objects must be rebound after manipulation in order for the change in state to get replicated.

1.1 Normal behaviour

The normal behaviour of a RMI-cluster enabled server is as follows:

When a Server is started, its RMI-cluster service will send out a context state request. Other nodes on the same cluster will then reply with any replicable context data known to them.
When a value is bound to the Application context, the RMI-cluster service will send out a context state change message. Other nodes on the same cluster will then update their Application context accordingly.
When a value is rebound to the Application context, the RMI-cluster service will send out a context state change message. Other nodes on the same cluster will then update their Application context accordingly.
When a value is unbound from the Application context, the RMI-cluster service will send out a context state change message. Other nodes on the same cluster will then update their Application context accordingly.

1.2 Requirements

In order for RMI-clustering to work, the following requirements must be met:

All cluster nodes must be able to use multicast
All cluster nodes that should share Application context state must use the same deployment name (see identification) for the Applications.
All cluster nodes that should share Application context state must use the same multicast address/port
All cluster nodes that should share Application context state must use the same configured username and password in their RMI-cluster service.
All data put into the Application context must be replicable (Serializable or remote RMI objects) to be shared.

2 RMI-cluster enabling a Server

Go through the following steps to RMI-cluster enable a Server:

Step 1: Configure a RMI-cluster service
Step 2: Verify cluster configuration

2.1 Step 1: Configure a RMI-cluster service

Follow this guide to configure the RMI-cluster service

2.2 Step 2: Verify cluster configuration

Verify the RMI-cluster configuration by starting up the Servers in cluster-debug mode as described here and then access a Application context state altering resource of one of the clustered Servers. Console output on the Servers of the cluster should show that state is being replicated.

If no easily usable state altering resource exists on the Servers, a simle JSP page like the one in listing 1 below can be added to one of the Web-modules on the Servers.

<%@ page import="javax.naming.InitialContext,
java.io.IOException,
javax.naming.NamingException,
javax.naming.NameAlreadyBoundException,
java.util.HashMap"%>
<%
InitialContext context=new InitialContext();
HashMap map=new HashMap();
map.put("a","server a");
map.put("b","server a");
map.put("c","server a");
out.write("Current values: ");
try
{
out.write(context.lookup("jnditest")+ " ");
}
catch (IOException e)
{
out.write(e.getMessage()+" ");
}
catch (NamingException e)
{
out.write(e.getMessage()+" ");
}

try
{
context.bind("jnditest",map);
} catch(NameAlreadyBoundException nabe)
{
context.rebind("jnditest",map);
}

map.put("d","server a"); //not rebound so not replicated
try
{
out.write("After: "+context.lookup("jnditest")+ " ");
} catch (IOException e)
{
out.write(e.getMessage()+" ");
} catch (NamingException e)
{
out.write(e.getMessage()+" ");
}
out.write("<hr>");
%>

Listing 1: A sample Application context state altering JSP page

If the above steps has been completed successfully the Servers should now be able to share their Application contextstate within the cluster.

3 Debugging

This section describes common techniques that can be used to debug RMI-clustering.

Tracing
Application logs

3.1 Tracing

In order to get trace information from the cluster services, start the Orion Server with the cluster.debug and rmi.debug system properties set to true, as exemplified in listing 4 below.

java -Drmi.debug=true -Dcluster.debug=true -jar orion.jar -console

Listing 4: Starting a Orion server with cluster debug system properties turned on

The cluster.debug property will list common debug information to the cosole.

The rmi.debug property will list RMI debug information to the console.

3.2 Application logs

Issues the RMI-clustering service has with either sending or receiving state can be found in the Applications application.log file per default.

4 Troubleshooting

This section describes common problems and possible solutions.

Firewall issues
Proxies/Gateways issues
Non distributable state
Manipulating bound values

4.1 Firewall issues

Make sure that any firewalls allows for the Servers to communicate on the configured multicast address in both directions, both sending and receiving information.

Using a simple multicast sender/receiver test application allows for testing the configured multicast address and make sure that packets can be both sent and received.

4.2 Proxies/Gateways issues

As multicast packets are known to hog down networks if allowed to roam free, most proxies and gateways won't let multicast packets pass. Make sure that the multicast packets can reach all nodes in the cluster and configure any proxies/gateways accordingly.

4.3 Non distributable state

If certain objects are not replicated within the cluster, make sure that these values are either Serializable or remote RMI-objects (this also goes for any objects this object is using/referring to).

4.4 Manipulating bound values

Serializable objects must be rebound after manipulation in order for the change in state to get replicated.