EJB state replication

This article describes what EJB state replication is and how to set it up

Introduction
EJB-cluster enable a Entity bean
EJB-cluster enable a Session bean
Debugging
Troubleshooting

1 Introduction

This article will explain EJB state replication and distribution for the Orion Application Server, as well as give step-by-step instructions for setting up, running and testing EJB clustering.

Throughout this article, a cluster will mean a number of EJB-cluster enabled EJB-modules. Each EJB-cluster enabled EJB-module will be refered to as a cluster node.

The EJB-cluster services uses multicast packets to send and receive state changes in a cluster. These packets are sent over the network where they are picked up by other cluster nodes using the same multicast address and port.This also means that the data is available to any application, such as a custom cluster monitor, that has registered itself as a listener on the same multicast address and port.

Multicast addresses have IP addresses in the range 224.0.0.0 to 239.255.255.255. Normally, a Multicast packet has a TTL (time to live) value less than 60, which normaly limits the packet to the local network and thereby making sure that they wont flood the network. For more information on multicast and related issues such as TTL, please use a resource as this as multicast in itself is out of scope for this article.

In order to identify a cluster node the cluster services uses the Applications deployment name (the name attribute specified in the application tag in server.xml) to identify Applications. To identify EJB-modules, the URI of the EJB-module as it is stated in the Applications application.xml file is used. If the EJB-module is packaged into a JAR, it's the URI of the JAR file without the .jar suffix. Together, these two identifiers are then used together with the beans name to decide if a certain cluster node should be given information about a state change.

As described in Figure 1 below, all cluster-enabled Entities needs to use the same DataSource for persistence in order for state replication to function properly.

Figure 1: The EJB clustering service utilizes the Server-wide RMI-clustering service to replicate the EJB state to all other EJB-cluster nodes listening to the same multicast address/port (in this case 230.0.0.1:9127 which is the default). All clustered EJB Entities are using the same DataSource.

Session beans are not replicated. They are instead distributed from one node to another when explicidly called for, or when the owning node is shut down.

1.1 Normal behaviour

The normal behaviour of a EJB-cluster enabled server is as follows:

When a modified cluster enabled Entity bean is comitted, the EJB-cluster service replicates the data to the other cluster nodes
When a Session bean that is located on another cluster node is explicidly called for, the node will receive the passivated session bean from the node currently holding the Session bean.
When a node is shut down it's Session bean instances are passivated and distributed to another node in the cluster
When a cluster enabled Entity bean without transactional write is manipulated simultaneously on multiple nodes, the last write will rule (see details below)

1.2 Requirements

In order for EJB-clustering to work, the following requirements must be met:

All cluster nodes must be able to use multicast
All cluster nodes that should share EJB state must use the same deployment name (see identification) for the Applications.
All cluster nodes that should share EJBstate must use the same multicast address/port
All cluster nodes that should share EJB state must use the same configured username and password in their RMI-cluster service.
Any Entity bean that should be replicable over the cluster must be cluster enabled
Any Session bean that should be distributed over the cluster must be cluster enabled

1.3 Entity considerations

The default EJB-cluster service behaviour implies that if a Entity is modified simultaneously in a cluster, the last change will rule as Transactional writes are not used by default.

This is normally acceptable for Entities that are mostly read (the majority of Entities). Entities that are often modified can be given transactional writes by setting exlusive-write-access to false in the deployment tag of the Entity as described here. This will have a major impact on performance and should thus only be used when needed.

2 EJB-cluster enable a Entity bean

In order for a Entity bean to be replicated within a cluster the deployment information for the Entity bean needs to be modified.

Go through the following steps to EJB-cluster enable a Entity bean:

Step 1: RMI-cluster enable the Server
Step 2: Locate the orion-ejb-jar.xml file
Step 3: Cluster enable the Entity
Step 4: Enable transactional writes (Optional)

2.1 Step 1: RMI-cluster enable the Server

As the EJB-cluster service uses a underlying RMI-cluster service, the first step is to RMI-cluster enable the Server as described in this guide.

If the Server is already RMI-cluster enabled, continue with the next step.

2.2 Step 2: Locate the orion-ejb-jar.xml file

Locate the orion-ejb-jar.xml file of the EJB-module that the Entity bean belongs to. This file is normaly located in the deployment directory of the Application that the EJB-module belongs to.

In a default installation, the orion-ejb-jar.xml file is located at "orion/application-deployment/[Application]/[EJB-module]/orion-ejb-jar.xml". Substitute [EJB-module] with the EJB-module the Entity is part of and the [Application] with the Application the EJB-module is part of.

2.3 Step 3: Cluster enable the Entity

In the orion-ejb-jar.xml file located in the previous step, locate the deployment tag for the Entity. It should look something like in listing 1 below:

<orion-ejb-jar ... >
<enterprise-beans>
<entity-deployment
name="name"
table="table"
data-source="datasource"
... >
...
</entity-deployment>
...
</enterprise-beans>
...
</orion-ejb-jar>

Listing 1: The deployment tag of an Entity with the implicit clustering-schema attribute (normally not displayed).

Where name is the name of the Entity bean.

Make sure that the data-source and table is the same DataSource and table name for all nodes in the cluster.

Add the attribute clustering-schema with a value of "asynchronous-cache" to cluster-enable the Entity bean (the default value is "default" which means that the Entity is not clustered). By now the deployment tag for the Entity should look similar to the example shown in listing 2 below.

<orion-ejb-jar ... >
<enterprise-beans>
<entity-deployment
name="name"
table="table"
data-source="datasource"
clustering-schema="asynchronous-cache"
... >
...
</entity-deployment>
...
</enterprise-beans>
...
</orion-ejb-jar>

Listing 2: Cluster enabling a Entity bean by setting the clustering-schema attribute.

2.4 Step 4: Enable transactional writes (Optional)

If the entity is written to often consider adding the attribute exclusive-write-access with a value of "false" to the Entity's deployment tag. This will slow down the performance but making sure that all writes are transactional (the default value is "true" which is best suited for Entities that are mostly read.

<orion-ejb-jar ... >
<enterprise-beans>
<entity-deployment
name="name"
table="table"
data-source="datasource"
clustering-schema="asynchronous-cache"
exclusive-write-access="false"
... >
...
</entity-deployment>
...
</enterprise-beans>
...
</orion-ejb-jar>

Listing 3: Enabling transactional writes for a clustered Entity

3 EJB-cluster enable a Session bean

In order for a Session bean to be distributed within a cluster the deployment information for the Session bean needs to be modified.

Go through the following steps to EJB-cluster enable a Entity bean:

Step 1: RMI-cluster enable the Server
Step 2: Locate the orion-ejb-jar.xml file
Step 3: Cluster enable the Session

3.1 Step 1: RMI-cluster enable the Server

As the EJB-cluster service uses a underlying RMI-cluster service, the first step is to RMI-cluster enable the Server as described in this guide.

If the Server is already RMI-cluster enabled, continue with the next step.

3.2 Step 2: Locate the orion-ejb-jar.xml file

Locate the orion-ejb-jar.xml file of the EJB-module that the Session bean belongs to. This file is normaly located in the deployment directory of the Application that the EJB-module belongs to.

In a default installation, the orion-ejb-jar.xml file is located at "orion/application-deployment/[Application]/[EJB-module]/orion-ejb-jar.xml". Substitute [EJB-module] with the EJB-module the Session is part of and the [Application] with the Application the EJB-module is part of.

3.3 Step 3: Cluster enable the Session

In the orion-ejb-jar.xml file located in the previous step, locate the deployment tag for the Session. It should look something like in listing 3 below:

<orion-ejb-jar ...>
<enterprise-beans>
<session-deployment
name="name"
location="location"
... >
</session-deployment>
</orion-ejb-jar>

Listing 3: The deployment tag of the Session bean.

Where name is the name of the Session bean.

Add the attribute replication with a value of "cluster" to cluster-enable the Session bean (the default value is "none" which means that the Session is not clustered). By now the deployment tag for the Session should look similar to the example shown in listing 4 below.

<orion-ejb-jar ...>
<enterprise-beans>
<session-deployment
name="name"
location="location"
...
replication="cluster"
... >
</session-deployment>
</orion-ejb-jar>

Listing 4: The deployment tag of the cluster enabled Session bean.

4 Debugging

This section describes common techniques that can be used to debug RMI-clustering.

Tracing
Application logs

4.1 Tracing

In order to get trace information from the cluster services, start the Orion Server with the cluster.debug and ejb.cluster.debug system properties set to true, as exemplified in listing 5 below.

java -Dejb.cluster.debug=true -Dcluster.debug=true -jar orion.jar -console

Listing 5: Starting a Orion server with cluster.debug system properties turned on

The cluster.debug property will list common cluster debug information to the cosole.

The ejb.cluster.debug property will list EJB cluster debug information to the console.

4.2 Application logs

Issues the EJB-clustering service has with either sending or receiving state can be found in the Applications application.log file per default.

5 Troubleshooting

This section describes common problems and possible solutions.

Firewall issues
Proxies/Gateways issues

5.1 Firewall issues

Make sure that any firewalls allows for the Servers to communicate on the configured multicast address in both directions, both sending and receiving information.

Using a simple multicast sender/receiver test application allows for testing the configured multicast address and make sure that packets can be both sent and received.

5.2 Proxies/Gateways issues

As multicast packets are known to hog down networks if allowed to roam free, most proxies and gateways won't let multicast packets pass. Make sure that the multicast packets can reach all nodes in the cluster and configure any proxies/gateways accordingly.