{"id":86,"date":"2016-03-18T13:59:16","date_gmt":"2016-03-18T02:59:16","guid":{"rendered":"https:\/\/icicimov.com\/blog\/?p=86"},"modified":"2017-01-02T15:39:32","modified_gmt":"2017-01-02T04:39:32","slug":"activemq-masterslave-kahadb-on-ocfs2-shared-file-system","status":"publish","type":"post","link":"https:\/\/icicimov.com\/blog\/?p=86","title":{"rendered":"ActiveMQ Master\/Slave KahaDB on OCFS2 shared file system"},"content":{"rendered":"

During my tests of shared storage clusters I wondered if ActiveMQ supports file locking on OCFS2 file system which I used on couple of occasions. While looking into it I came across the following warning on the Apache project site<\/a>:
\n
\nOCFS2 Warning
\nWas testing using OCFS2 and both brokers thought they had the master lock – this is because OCFS2 only supports locking with fcntl and not lockf and flock, therefore mutex file locking from Java isn’t supported.
\n<\/cite>
\nFrom RedHat faq<\/a> we can also see:
\n
\nOCFS2: No cluster-aware flock or POSIX locks
\nGFS: fully supports Cluster-wide flocks and POSIX locks and is supported.
\n<\/cite><\/p>\n

See this JIRA for more discussion: AMQ-4378<\/a><\/p>\n

And true enough, when tested ActiveMQ-5.11.1 on one of the OCFS2 clusters, what happened is that both servers started as master due to lack of lock file on the share. Investigating further it came up that some good man created a patch that Apache never bothered to apply from some unknown reason. The case link AMQ-4694<\/a>. The patch was attached to the ticket so thought I would try it and apply to latest 5.11.4 version.<\/p>\n

Downloaded the source:<\/p>\n

\nroot@drbd01:~# aptitude install zip\nroot@drbd01:~# wget http:\/\/mirror.ventraip.net.au\/apache\/activemq\/5.11.4\/activemq-parent-5.11.4-source-release.zip\nroot@drbd01:~# unzip activemq-parent-5.11.4-source-release.zip\nroot@drbd01:~# cd activemq-parent-5.11.4\/\n<\/pre>\n
and applied the patch:<\/p>\n
\nroot@drbd01:~\/activemq-parent-5.11.4# vi ocfs2-locks.patch\nroot@drbd01:~\/activemq-parent-5.11.4# patch -p0 -i ocfs2-locks.patch\npatching file NOTICE\nHunk #1 succeeded at 45 with fuzz 2 (offset -4 lines).\npatching file activemq-broker\/pom.xml\npatching file activemq-broker\/src\/main\/java\/org\/apache\/activemq\/store\/AbstractResourceLocker.java\npatching file activemq-broker\/src\/main\/java\/org\/apache\/activemq\/store\/SharedFileLocker.java\nHunk #1 FAILED at 16.\n1 out of 1 hunk FAILED -- saving rejects to file activemq-broker\/src\/main\/java\/org\/apache\/activemq\/store\/SharedFileLocker.java.rej\npatching file activemq-broker\/src\/main\/java\/org\/apache\/activemq\/store\/SharedNativeFileLocker.java\npatching file activemq-broker\/src\/main\/java\/org\/apache\/activemq\/util\/LockFile.java\npatching file activemq-broker\/src\/main\/java\/org\/apache\/activemq\/util\/LockResource.java\npatching file activemq-broker\/src\/main\/java\/org\/apache\/activemq\/util\/NativeLockFile.java\npatching file activemq-unit-tests\/src\/test\/java\/org\/apache\/activemq\/util\/NativeLockFileTest.java\npatching file assembly\/pom.xml\nHunk #1 succeeded at 342 (offset 14 lines).\npatching file assembly\/src\/main\/descriptors\/common-bin.xml\nHunk #1 succeeded at 191 (offset -9 lines).\npatching file pom.xml\nHunk #1 succeeded at 881 with fuzz 1 (offset 44 lines).\n<\/pre>\n
Ok, so all good except it failed for the SharedFileLocker.java<\/code> file. Easily fixed by applying that part of the patch manually:<\/p>\n
\nroot@drbd01:~\/activemq-parent-5.11.4# vi activemq-broker\/src\/main\/java\/org\/apache\/activemq\/store\/SharedFileLocker.java\n\/**\n * Licensed to the Apache Software Foundation (ASF) under one or more\n * contributor license agreements.  See the NOTICE file distributed with\n * this work for additional information regarding copyright ownership.\n * The ASF licenses this file to You under the Apache License, Version 2.0\n * (the \"License\"); you may not use this file except in compliance with\n * the License.  You may obtain a copy of the License at\n *\n *      http:\/\/www.apache.org\/licenses\/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n *\/\npackage org.apache.activemq.store;\n \nimport java.io.File;\nimport org.apache.activemq.util.LockFile;\nimport org.apache.activemq.util.LockResource;\n \n\/**\n * An {@link AbstractResourceLocker} that utilizes a {@link LockFile}.\n *\n * @org.apache.xbean.XBean element=\"shared-file-locker\"\n *\n *\/\npublic class SharedFileLocker extends AbstractResourceLocker {\n \n    @Override\n    protected LockResource newLockResource(File lockFileName) {\n        return new LockFile(lockFileName, true);\n    }\n}\n<\/pre>\n
Next installed Maven:<\/p>\n
\nroot@drbd01:~# wget http:\/\/apache.uberglobalmirror.com\/maven\/maven-3\/3.3.9\/binaries\/apache-maven-3.3.9-bin.tar.gz\nroot@drbd01:~# tar -xzvf apache-maven-3.3.9-bin.tar.gz\nroot@drbd01:~# mv apache-maven-3.3.9 \/usr\/local\/\nroot@drbd01:~# export PATH=\/usr\/local\/apache-maven-3.3.9\/bin:$PATH\n<\/pre>\n
Back to AMQ directory and build the project:<\/p>\n
\nroot@drbd01:~\/activemq-parent-5.11.4# mvn -Dtest=false -DfailIfNoTests=false clean install -e -X\n<\/pre>\n
Install the built binaries:<\/p>\n
\nroot@drbd01:~# tar -xzvf \/root\/.m2\/repository\/org\/apache\/activemq\/apache-activemq\/5.11.4\/apache-activemq-5.11.4-bin.tar.gz -C \/opt\/\n<\/pre>\n
and copy the setup from the previous AMQ version into the new one:<\/p>\n
\nroot@drbd01:\/opt# cp -a apache-activemq-5.11.1\/conf\/{log4j.properties,jmx.access,jmx.password,jetty.xml,encompass-activemq.xml,star_encompasshost_com.jks} apache-activemq-5.11.4\/conf\/\nroot@drbd01:\/opt# chown -R activemq\\: apache-activemq-5.11.4\nroot@drbd01:\/opt# rm activemq\nroot@drbd01:\/opt# ln -s apache-activemq-5.11.4 activemq\n<\/pre>\n
The last bit is modifying the KahaDB locker to use the newly introduced shared-native-file-locker one:<\/p>\n
\nroot@drbd01:\/opt# vi activemq\/conf\/activemq.xml\n[...]\n \n    \n           \n              <\/shared>\n           <\/locker>\n     <\/kahadb>\n<\/persistenceadapter>\n[...]\n<\/pre>\n
After copying over the binary tarball to the other server, installing and performing the above steps, I started the server and the lock file was indeed created this time:<\/p>\n
\nroot@drbd01:\/opt# ls -ltr \/share\/activemq-data\/\ntotal 160\ndrwxr-xr-x 3 activemq activemq     3896 Mar 17 15:27 com.broker.name\n-rw-r--r-- 1 root     root            5 Mar 17 15:30 test\n-rw-r--r-- 1 activemq activemq        0 Mar 17 18:16 lock\n-rw-r--r-- 1 activemq activemq 33030144 Mar 17 18:17 db-1.log\n-rw-r--r-- 1 activemq activemq    12304 Mar 17 18:17 db.redo\n-rw-r--r-- 1 activemq activemq    12288 Mar 17 18:17 db.data\n<\/pre>\n
and the other server can see it:<\/p>\n
\nroot@drbd02:\/opt# start activemq && tail -f \/opt\/activemq\/data\/activemq.log | grep -i lock\n2016-03-18 10:00:15,375 | INFO  | Database \/share\/activemq-data\/lock is locked... waiting 5 seconds for the database to be unlocked. Reason: java.io.IOException: failed to acquire lock: [11] ( | org.apache.activemq.store.SharedNativeFileLocker | main\n...\n<\/pre>\n","protected":false},"excerpt":{"rendered":"
During my tests of shared storage clusters I wondered if ActiveMQ supports file locking on OCFS2 file system which I used on couple of occasions. While looking into it I came across the following warning on the Apache project site:…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9,10],"tags":[],"class_list":["post-86","post","type-post","status-publish","format-standard","hentry","category-high-availability","category-server"],"_links":{"self":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/86","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=86"}],"version-history":[{"count":15,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/86\/revisions"}],"predecessor-version":[{"id":171,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/86\/revisions\/171"}],"wp:attachment":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=86"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=86"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=86"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}