{"id":176,"date":"2013-11-06T15:50:46","date_gmt":"2013-11-06T04:50:46","guid":{"rendered":"https:\/\/icicimov.com\/blog\/?p=176"},"modified":"2017-01-02T16:09:43","modified_gmt":"2017-01-02T05:09:43","slug":"tomcat7-clustering-and-session-replication-in-aws","status":"publish","type":"post","link":"https:\/\/icicimov.com\/blog\/?p=176","title":{"rendered":"Tomcat7 clustering and session replication in AWS"},"content":{"rendered":"<p>This was a POC for Tomcat clustering and session replication in AWS. It has been set up and tested on a pair of EC2 instances (ip-172-31-13-11 and ip-172-31-13-12) deployed in one of our test VPC&#8217;s. Since the multicast between the availability zones is still not possible in AWS we need to use the static cluster membership. The traffic for the private subnet is unrestricted for the <code>Security Group<\/code> the instances belong to.<\/p>\n<h2>Configuration<\/h2>\n<h3>Tomcat<\/h3>\n<p>We configure the <code>Cluster<\/code> element in the tomcat&#8217;s <code>server.xml<\/code> file on both nodes.<\/p>\n<pre>\n<cluster className=\"org.apache.catalina.ha.tcp.SimpleTcpCluster\" channelStartOptions=\"3\" channelSendOptions=\"8\">\n      <manager className=\"org.apache.catalina.ha.session.DeltaManager\"\n               expireSessionsOnShutdown=\"false\"\n               notifyListenersOnReplication=\"true\"><\/manager>\n \n      <channel className=\"org.apache.catalina.tribes.group.GroupChannel\">\n        <!--<Membership className=\"org.apache.catalina.tribes.membership.McastService\"\n                    address=\"228.0.0.4\"\n                    port=\"45564\"\n                    frequency=\"500\"\n                    dropTime=\"3000\"\/>-->\n        <receiver className=\"org.apache.catalina.tribes.transport.nio.NioReceiver\"\n                  address=\"auto\"\n                  port=\"4000\"\n                  autoBind=\"100\"\n                  selectorTimeout=\"5000\"\n                  maxThreads=\"6\"><\/receiver>\n \n        <interceptor className=\"org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor\">\n                <!--<Member className=\"org.apache.catalina.tribes.membership.StaticMember\" port=\"4000\" securePort=\"-1\" host=\"ip-172-31-13-11\" domain=\"publish-cluster\" uniqueId=\"{10,0,10,109}\"\/>-->\n                <member className=\"org.apache.catalina.tribes.membership.StaticMember\" port=\"4000\" securePort=\"-1\" host=\"ip-172-31-13-12\" domain=\"publish-cluster\" uniqueId=\"{10,0,10,227}\"><\/member>\n    <\/interceptor>\n \n        <sender className=\"org.apache.catalina.tribes.transport.ReplicationTransmitter\">\n          <transport className=\"org.apache.catalina.tribes.transport.nio.PooledParallelSender\"><\/transport>\n        <\/sender>\n    <interceptor className=\"org.apache.catalina.tribes.group.interceptors.TcpPingInterceptor\"><\/interceptor>\n        <interceptor className=\"org.apache.catalina.tribes.group.interceptors.TcpFailureDetector\"><\/interceptor>\n        <interceptor className=\"org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor\"><\/interceptor>\n      <\/channel>\n \n      <valve className=\"org.apache.catalina.ha.tcp.ReplicationValve\" filter=\"\"><\/valve>\n      <valve className=\"org.apache.catalina.ha.session.JvmRouteBinderValve\"><\/valve>\n \n      <!--<Deployer className=\"org.apache.catalina.ha.deploy.FarmWarDeployer\"\n                tempDir=\"\/tmp\/war-temp\/\"\n                deployDir=\"\/tmp\/war-deploy\/\"\n                watchDir=\"\/tmp\/war-listen\/\"\n                watchEnabled=\"false\"\/>-->\n \n      <clusterlistener className=\"org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener\"><\/clusterlistener>\n      <clusterlistener className=\"org.apache.catalina.ha.session.ClusterSessionListener\"><\/clusterlistener>\n<\/cluster>\n<\/pre>\n<p>Some points to note in the above configuration that are different from the default one:<\/p>\n<ul>\n<li>We have commented out the multicast <code>McastService<\/code> element since we don&#8217;t need it (no multicast possible in AWS)<\/li>\n<li>We add <code>channelStartOptions=\"3\"<\/code> to the Cluster element to disable the multicast (even when not explicitly configured, the multicast service is enabled by default)<\/li>\n<li>We set <code>channelSendOptions=\"8\"<\/code> which means asynchronous session replication, for synchronous mode we need to set this option to 4<\/li>\n<li>We add the <code>StaticMembershipInterceptor<\/code> class and specifying the other static members of the cluster (the local host Member line is commented out, we don&#8217;t want the local host adding it self to the cluster)<\/li>\n<li>We add <code>TcpPingInterceptor<\/code> interceptor that pings other nodes so that all nodes can recognize when other nodes have left the cluster (preventing the session replication to brake down when nodes are removed and re-introduced)<\/li>\n<\/ul>\n<p>Additionally I have set the <code>jvmRoute<\/code> value of the <code>Engine<\/code> element on each node (tomcat1 and tomcat2) so I can easily distinguish the sessions as this value will be appended to each session created (each session will have <code>.tomcat1<\/code> and <code>.tomcat2<\/code> appended). This is also useful in case of Apache load balancer in front with <code>mod_jk<\/code> or <code>mod_proxy<\/code> since the tomcat cookies will also have the above value appended (<code>JSESSIONID=xxxx.tomcat1<\/code> and <code>JSESSIONID=xxxx.tomcat2<\/code>).<\/p>\n<pre>\n<engine name=\"Catalina\" defaultHost=\"localhost\" jvmRoute=\"tomcat1\">\n<\/engine><\/pre>\n<h3>Application<\/h3>\n<p>I have used the Manager application to test the session replication. I deployed the app in <code>$CATALINA_BASE\/webapps<\/code> and set the context file:<\/p>\n<pre>\nroot@ip-172-31-13-11:~# vi \/etc\/tomcat7\/Catalina\/localhost\/manager.xml\n<context privileged=\"true\" antiResourceLocking=\"false\"\n         docBase=\"${catalina.base}\/webapps\/manager\" distributable=\"true\">\n  <valve className=\"org.apache.catalina.valves.RemoteAddrValve\"\n         allow=\"172\\.31\\.13\\..*\"><\/valve>\n<\/context>\n<\/pre>\n<p>and to cluster it we add the <code>&lt;distributable&gt;&lt;\/distributable&gt;<\/code> tag inside it&#8217;s <code>web.xml<\/code> file, <code>\/var\/lib\/tomcat7\/webapps\/manager\/WEB-INF\/web.xml<\/code>, right at the end before the &#8220; tag.<\/p>\n<h3>Runtime<\/h3>\n<p>After that we restart the servers we can see the cluster startup and session replication messages that look as follows on ip-172-31-13-11:<\/p>\n<pre><code>06\/11\/2013 9:55:47 PM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded\nINFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp:\/\/ip-172-31-13-12:4000,ip-172-31-13-12,4000, alive=0, securePort=-1, UDP Port=-1, id={0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 }, payload={}, command={}, domain={112 117 98 108 105 115 104 45 99 ...(15)}, ]\n06\/11\/2013 9:56:14 PM org.apache.catalina.ha.session.DeltaManager startInternal\nINFO: Register manager localhost#\/manager to cluster element Engine with name Catalina\n06\/11\/2013 9:56:14 PM org.apache.catalina.ha.session.DeltaManager startInternal\nINFO: Starting clustering manager at localhost#\/manager\n06\/11\/2013 9:56:14 PM org.apache.catalina.tribes.io.BufferPool getBufferPool\nINFO: Created a buffer pool with max size:104857600 bytes of type:org.apache.catalina.tribes.io.BufferPool15Impl\n06\/11\/2013 9:56:14 PM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions\nINFO: Manager [localhost#\/manager], requesting session state from org.apache.catalina.tribes.membership.MemberImpl[tcp:\/\/ip-172-31-13-12:4000,ip-172-31-13-12,4000, alive=0, securePort=-1, UDP Port=-1, id={0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 }, payload={}, command={}, domain={112 117 98 108 105 115 104 45 99 ...(15)}, ]. This operation will timeout if no session state has been received within 60 seconds.\n06\/11\/2013 9:56:14 PM org.apache.catalina.ha.session.waitForSendAllSessions\nINFO: Manager [localhost#\/manager]; session state send at 11\/6\/13 9:56 PM received in 143 ms.\n<\/code><\/pre>\n<p>We can see ip-172-31-13-11 adding ip-172-31-13-12 to the cluster and sending the session information via <code>DeltaManager<\/code>.<\/p>\n<p>After logging to the manager app the following screen shots of the sessions on both nodes confirm the replication has been successful.<\/p>\n\n\t\t\t<div id='gallery-176-1' class='gallery gallery-176'>\n\t\t\t\t<div class='gallery-row gallery-col-2 gallery-clear'>\n\t\t\t\t\t<figure class='gallery-item col-2'>\n\t\t\t\t\t\t<div class='gallery-icon '><a href='https:\/\/icicimov.com\/blog\/wp-content\/uploads\/2017\/01\/tomcat1-manager-app-sessions.png'><img loading=\"lazy\" decoding=\"async\" width=\"420\" height=\"116\" src=\"https:\/\/icicimov.com\/blog\/wp-content\/uploads\/2017\/01\/tomcat1-manager-app-sessions-420x116.png\" class=\"attachment-thumbnail size-thumbnail\" alt=\"\" srcset=\"https:\/\/icicimov.com\/blog\/wp-content\/uploads\/2017\/01\/tomcat1-manager-app-sessions-420x116.png 420w, https:\/\/icicimov.com\/blog\/wp-content\/uploads\/2017\/01\/tomcat1-manager-app-sessions-744x205.png 744w, https:\/\/icicimov.com\/blog\/wp-content\/uploads\/2017\/01\/tomcat1-manager-app-sessions-768x211.png 768w, https:\/\/icicimov.com\/blog\/wp-content\/uploads\/2017\/01\/tomcat1-manager-app-sessions-1200x330.png 1200w, https:\/\/icicimov.com\/blog\/wp-content\/uploads\/2017\/01\/tomcat1-manager-app-sessions.png 1595w\" sizes=\"auto, (max-width: 420px) 100vw, 420px\" \/><\/a><\/div>\n\t\t\t\t\t\t<figcaption class='gallery-caption'>Sessions tomcat1<\/figcaption>\n\t\t\t\t\t<\/figure>\n\t\t\t\t\t<figure class='gallery-item col-2'>\n\t\t\t\t\t\t<div class='gallery-icon '><a href='https:\/\/icicimov.com\/blog\/wp-content\/uploads\/2017\/01\/tomcat2-manager-app-sessions.png'><img loading=\"lazy\" decoding=\"async\" width=\"420\" height=\"115\" src=\"https:\/\/icicimov.com\/blog\/wp-content\/uploads\/2017\/01\/tomcat2-manager-app-sessions-420x115.png\" class=\"attachment-thumbnail size-thumbnail\" alt=\"\" srcset=\"https:\/\/icicimov.com\/blog\/wp-content\/uploads\/2017\/01\/tomcat2-manager-app-sessions-420x115.png 420w, https:\/\/icicimov.com\/blog\/wp-content\/uploads\/2017\/01\/tomcat2-manager-app-sessions-744x203.png 744w, https:\/\/icicimov.com\/blog\/wp-content\/uploads\/2017\/01\/tomcat2-manager-app-sessions-768x210.png 768w, https:\/\/icicimov.com\/blog\/wp-content\/uploads\/2017\/01\/tomcat2-manager-app-sessions-1200x328.png 1200w, https:\/\/icicimov.com\/blog\/wp-content\/uploads\/2017\/01\/tomcat2-manager-app-sessions.png 1593w\" sizes=\"auto, (max-width: 420px) 100vw, 420px\" \/><\/a><\/div>\n\t\t\t\t\t\t<figcaption class='gallery-caption'>Sessions tomcat2<\/figcaption>\n\t\t\t\t\t<\/figure>\n\t\t\t\t<\/div>\n\t\t\t<\/div><!-- .gallery -->\n\n<p>We can see that each server has created additional backup session from the other cluster member.<\/p>\n<h2>Considerations<\/h2>\n<p>Not being able to use Multicast between the availability zones imposes significant problem to Auto Scaling in case of clustering. There is no way to know before hand how many members will be in the cluster and the IP\/DNS names of the additionally launched instances in event of auto scaling. This makes it very hard to configure the cluster with Auto Scaling feature and static membership. In this case, when Auto Scaling is imperative, the best option we are left with is a shared session persistent storage. I have done some testing with <code>Shared File system<\/code>, <code>MongoDB<\/code>, <code>MySQL<\/code> and <code>Memcached<\/code> (more on this is another post) as session persistent storage and they have all proved to work well. In this case though the session storage HA and the storing speed need to be taken in consideration. The store needs to be clustered or replicated and upon changes the session state needs to be flushed to the storage as soon as possible to avoid loosing it in case of server outage. Also compared to the in-memory replication in case of clustering, there will be some delays between the session changes on the primary node (where the session gets created) and the time the other server nodes get updated so the sticky sessions on the LB&#8217;s will have to stay on.<\/p>\n<h2>References<\/h2>\n<ul>\n<li><a href=\"http:\/\/tomcat.apache.org\/tomcat-7.0-doc\/cluster-howto.html\">Clustering\/Session Replication HOW-TO<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>This was a POC for Tomcat clustering and session replication in AWS. It has been set up and tested on a pair of EC2 instances (ip-172-31-13-11 and ip-172-31-13-12) deployed in one of our test VPC&#8217;s. Since the multicast between the&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-176","post","type-post","status-publish","format-standard","hentry","category-social"],"_links":{"self":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/176","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=176"}],"version-history":[{"count":3,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/176\/revisions"}],"predecessor-version":[{"id":183,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/176\/revisions\/183"}],"wp:attachment":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=176"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=176"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=176"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}