Tomcat Clustering Series Part 3 : Session Replication

RamkiNovember 27th, 2012Last Updated: November 27th, 2012

3 372 6 minutes read

This is the third part of the Tomcat Clustering Series. In this post we are going to discuss the how to setup session replication in tomcat clustering environment. Session replication makes High availability and full fail-over capability to our clustering environment.

[Check the video below for better understanding]

In my previous post we discussed about setup simple load balancer and how to make session affinity concepts.

How to setup Session Replication in tomcat

Before going to session replication we need to understand 2 important concepts

Multicast
Session Manager in Tomcat

Multicast

Multicast is to transmit a single message to a select group of recipients. here multicast used by tomcat cluster to identify the instances those part of cluster.

There are 2 types of cluster

Static Tomcat Cluster
Dynamic Tomcat Cluster

In static cluster there is no need multicast, because each tomcat we statically defined/configured the other instances. But dynamic Cluster we are not defined anything. so each tomcat in that cluster some how to identify the other tomcat instances.

So here multicast concepts is used. Each and every tomcat first joining to single multicast group. and send the heartbeat signals in periodic interval. so other tomcat instances received these signal and add the member to the cluster.

Session Manager in Tomcat

Session Manager is used to create and manage the session behalf the application. In Servlet Specification request.getSession(); line is mention that container (tomcat) is responsible for create the session. here tomcat use the Session Manager for this purpose.

There are 4 types of Session Manager

Standard Manager
Persistent Manager
Delta Manager
Backup Manager

Standard Manager

It is the default manager used by tomcat. Even though we are not mention in our web application tomcat use this manager for managing our session. If u want to customize the this standard manager then add <Manager> tag in context.xml file.

<Manager className=“org.apache.catalina.session.StandardManager” />

Here org.apache.catalina.session.StandardManager is fully qualified class name of the Standard Manager.

Persistent Manger

This mnager is to store the session information into persistent place after some interval. here two types of store is available.

File Store
JDBC Store

File Store helps to store all session information in separate files in underlying file system (local HDD or shared file-system like NFS,..)

JDBC Store helps to store the session information to relational database.

So using the Persistent Manager we can achieve the tomcat cluster. but its not swapped out in real time. its pushes the information after certain interval. so if anything badly happen(crash) before that interval then in-memory session data is gone.

Delta Manger

In this post we are going to use this manager. Its replicate the session to all other instances. so this manager usually used clustered environment. but not good for large cluster.

Backup Manager

this manager usually used clustered environment. Its like delta manger. but it will replicate to exactly one other instance(backup instance). Its acted like one instance is Primary and another instance as backup.

Steps to make Session Replication in Tomcat Clustering

Here i will continue from exactly where i left in last session affinity post. So check that post and make sure jumRoute all are set properly. So steps are

Enable Multicast routing
Add <Cluster> Entries in conf/server.xml file for all instances.
Enable the Web Application as distributable

1. Enable Multicast routing

In Linux Environment most of the system kernel is capable to process the multicast address. but we need to add route entry in kernel routing table.

sudo route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0

Here eth0 is my Ethernet interface. so change according to your interface

In multicast address is belong to Class D address Range (224.0.0.0 to 239.255.255.255). so we inform to kernel if any one access these address then it goes through eth0 interface.

2. Add <Cluster> Entries in conf/server.xml file for all instances.

This very important part for tomcat clustering. We need to Add <Cluster> tag in conf/server.xml file in all tomcat instances.

<Cluster className='org.apache.catalina.ha.tcp.SimpleTcpCluster'/>

We can add this <Cluster> tag in either inside the<Engine> tag or <Host> tag.

Here SimpleTcpCluster is Tomcat Cluster implementation

This tag is looks like simple but its has many inner tags. if we omitted then its takes the default values. if we want do any cutomization (like change multicat address, receving address port) we need to use complete <Cluster> tag

This is the complete <Cluster>

<Cluster className='org.apache.catalina.ha.tcp.SimpleTcpCluster'

channelSendOptions='8'>

<Manager className='org.apache.catalina.ha.session.DeltaManager'

expireSessionsOnShutdown='false'

notifyListenersOnReplication='true'/>

<Channel className='org.apache.catalina.tribes.group.GroupChannel'>

<Membership className='org.apache.catalina.tribes.membership.McastService'

address='228.0.0.4'
port='45564'

frequency='500'

dropTime='3000'/>
<Sender className='org.apache.catalina.tribes.transport.ReplicationTransmitter'>

<Transport className='org.apache.catalina.tribes.transport.nio.PooledParallelSender'/>

</Sender>

<Receiver className='org.apache.catalina.tribes.transport.nio.NioReceiver'

address='auto'

port='4000'

autoBind='100'

selectorTimeout='5000'

maxThreads='6'/>

<Interceptor

className='org.apache.catalina.tribes.group.interceptors.TcpFailureDetector'/>

<Interceptor

className='org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor'/>

</Channel>
<Valve className='org.apache.catalina.ha.tcp.ReplicationValve' filter=''/>

<Valve className='org.apache.catalina.ha.session.JvmRouteBinderValve'/>

<ClusterListener className='org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener'/>

<ClusterListener className='org.apache.catalina.ha.session.ClusterSessionListener'/>

</Cluster>

Check my sample conf/server.xml file (for reference)

Here most of the code is boiler plate code. just copy and paste. if we need we can customize. for example we can change the multicat address and port number.

<Manager className='org.apache.catalina.ha.session.DeltaManager'/>

here Manager tag define the delta manager. Delta manager means replicate to all instances.

<Channel className='org.apache.catalina.tribes.group.GroupChannel'>

Tomcat Clustering use the Apache Tribes communication framework. This group commnication framework is responsible for dynamic membership (using multicast) , send and receive the session delta information using uni-cast (normal TCP connection).

<Membership className='org.apache.catalina.tribes.membership.McastService'

address='228.0.0.4'

port='45564'

frequency='500'

dropTime='3000'/>

This is Membership definition. here address is multicast address. we can pick any address from Class D address range (224.0.0.0 to 239.255.255.255)and any port number.

Each and every tomcat send the heart beat signal to multicast address in periodic ( frequency) interval. all other tomcat whose joined the multicast address they can receive these signals and add the membership to the cluster. if heat beat signal is not revive some particular interval ( dropTime) from any one of the tomcat, then we need to consider that tomcat is failed.

Note:-

All tomcat instances which are part of the clustering, should have same multicast address and port number.

<Sender className='org.apache.catalina.tribes.transport.ReplicationTransmitter'>

<Transport className='org.apache.catalina.tribes.transport.nio.PooledParallelSender'/>

</Sender>

Here sender use the PooledParallelSender have pooled connections to use the send the session information concurrently. so its speedup the session replication process.

<Receiver className='org.apache.catalina.tribes.transport.nio.NioReceiver'

address='auto'

port='4000'

autoBind='100'

selectorTimeout='5000'

maxThreads='6'/>

Here we define which port Receiver can bind and used for receiving the session replicate information. here two properties are important. address and port. here address is ur system IP address and port is any unused port. Here address=’auto’ its automatically pick the system IP address.

We have some interceptor.

TcpFailureDetector -Its ensure that instance are dead. In some case multicast messages are delayed, all tomcat instances are think about that tomcat is dead. but this interceptor to make tcp unicast to failed tomcat and ensure that instances is actually failed or not

Another important listener is JvmRouteSessionIDBinderListener, we will talk about it later.

3. Enable the Web Application as distributable

We need to make the our web application distribuable. its simple add <distributable/> tag in web.xml file. In according to servlet specification <distributable/> tag in web.xml mention that any container to consider this application can work in distributed environment.

Note:

All session in our web application must me serializable.

Do these steps to all tomcat instances and start the tomcat and httpd server. check my configuration in my github repo or get as ZIP

This is my configuration. all 3 tomcat instances are configured in delta manager and i deployed the distributed web application. all tomcat use the multicast to maintain the membership.

Now client make the request and first tomcat process and create the session, then looks like this

Then tomcat 1 is responsible to replicate the session using Apache tribes group communication framework to replicate the session to all instances.

Now all tomcat instance have exact copy of the session. so if tomcat 1 crashed or shutdown, then any other tomcat still can process the request [see the video below]

We used session affinity like previous post. based on that cookie id contain the tomcat name (worker name). so when first tomcat1 return the session id end with tomcat1. but when tomcat 1 is failed and tomcat 2 take the responsible for all further request. but session id still contain the tomcat1. so its makes the load balancer difficult. because tomcat1 is down. and load balancer pick any other tomcat. but actually tomcat2 takes the responsible. so we need to reflect these changes in session id.

JvmRouteSessionIDBinderListener changes the client session id to tomcat2 when failure is occurred so load balancer redirect to tomcat2 without confusing.

Check the git hub for all configuration files and tomcat clustering setup is available. or u can download as ZIP

Related Links: