What we are going to setup is shown in the ASCII chart below. The storage stack will provide Each client server (drbd01 and drbd02) has three interfaces each connected to three separate networks of which only the external one, Here is the hosts configuration in the The servers will be running CentOS-6.7 since the iSCSI and Pacemaker are generally better supported on RHEL\/CentOS distributions. There are several iSCSI providers ie IET, STGT, LIO and SCST and I’ve chosen SCST for it’s stability, speed and array of features that others don’t provide ie ALUA. Unfortunately it has not been merged in the upstream Linux kernel, LIO got that privilege in the latest kernels, so it needs installation from source and kernel patching for best results. Helper scripts are provided in the source to make this otherwise tedious task very simple.<\/p>\n The We start by installing some needed software (on both nodes, centos01 and centos02:<\/p>\n We need start it up:<\/p>\n Fetch the SCST source from trunk:<\/p>\n Then we run:<\/p>\n This script will build for us a version of the RHEL\/CentOS\/SL kernel we are running with the SCST patches applied on top. Then we install the new kernel and boot into it:<\/p>\n Last step is to compile and install the scst services and modules we need:<\/p>\n Now we check if everything is working properly. We need to set minimum config file to start with:<\/p>\n and start the service:<\/p>\n and check if all has been started and loaded properly:<\/p>\n All looks good so the SCST part is finished. At the end we check if SCST service has been added to auto-start and if yes we remove it since we want to be under Pacemaker control:<\/p>\n We start by installing DRBD-8.4 on both servers:<\/p>\n Then we create our DRBD resource that will provide the backing device for the volume group and logical volume we want to create. Create new file Then we start drbd on both nodes:<\/p>\n create the meta data:<\/p>\n and then on one server only we promote the resource to primary state:<\/p>\n The initial sync of the block device will start and when finished we have:<\/p>\n All is We need to leave this though for after we execute the next step where we create the logical volume.<\/p>\n We use this layer so we can easily extend our storage. We just add new volume to the DRBD resource and extend the PV, VG and LV that we create on top of it. On both nodes:<\/p>\n then we tell LVM to look for VG’s on our system and DRBD device only by setting the filter option. We also make sure the locking is set properly, the file is and remove some possibly stale cache file:<\/p>\n then we can create our VG and LV (on one node only since DRBD will replicate this for us):<\/p>\n We are done with this part.<\/p>\n Corosync is the cluster messaging layer and provides communication for the Pacemaker cluster nodes. On both nodes:<\/p>\n then we add the HA-Cluster repository to install and run:<\/p>\n Now we setup Corosync with dual ring and active mode in the We start the service on both nodes and check:<\/p>\n All looks good. We set Corosync to auto-start and we are done:<\/p>\n Now we start Pacemaker on both nodes and check its status.<\/p>\n All looks good so we enable it to auto-start on both nodes:<\/p>\niSCSI<\/code> service to the clients by exporting LUN's<\/code> via single target. The clients (iSCSI initiators) will access the LUN’s via multipathing for high-availability and keep the files in sync via OCFS2<\/code> clustered file system. On the server side, DRBD is tasked to sync the block storage on both servers and provide fail-over capacity. Both clusters will be managed by Pacemaker<\/code> cluster stack.<\/p>\n\n 192.168.0.0\/24 +----------+\n--------------------------------------------------------------------------------------------------| router |-----------\n drbd01 drbd02 | +----------+\n+----------+ +----------+ +----------+ +----------+ |\n| Service | | Service | | Service | | Service | |\n+----------+ +----------+ +----------+ +----------+ |\n || || || || |\n+------------------------+ +------------------------+ |\n| ocfs2 |< ~~~~~~~~~~>| ocfs2 | |\n+------------------------+ +------------------------+ |\n| multipath | | multipath | |\n+------------------------+ +------------------------+ |\n| sdc,sdd | | sdc,sdd | |\n+------------------------+ +------------------------+ |\n| iscsi | | iscsi | |\n+------------------------+ +------------------------+ |\n | | | | | | 10.10.1.0\/24 |\n----------+---------------------------------------+------------------------------- |\n | | | | 10.20.1.0\/24 |\n------+--------------+------------------------+--------------+-------------------- |\n | | | | |\n--+-------------+-------------------------+-------------+-------------------------------\n | | | |\n+------------------------+ +------------------------+\n| iscsi | | iscsi |\n+------------------------+ +------------------------+\n| lv_vol | | lv_vol |\n+------------------------+ +------------------------+\n| volume group vg1 | | volume group vg1 |\n+------------------------+ +------------------------+\n| physical volume | | physical volume |\n+------------------------+ +------------------------+\n| drbd r0 |< ~~~~~~~~~~>| drbd r0 |\n+------------------------+ +------------------------+\n| sdb | | sdb |\n+------------------------+ +------------------------+\n centos01 centos02\n <\/pre>\n
192.168.0.0\/24<\/code> is routed. The other two, 10.10.1.0\/24<\/code> and 10.20.1.0\/24<\/code>, are private networks dedicated to the DRBD and iSCSI traffic respectively. All 4 servers are connected to the 10.20.1.0\/24<\/code> network ie the storage network.<\/p>\n\/etc\/hosts<\/code> file on all 4 nodes:<\/p>\n...\n10.10.1.17 drbd01.virtual.local drbd01\n10.10.1.19 drbd02.virtual.local drbd02\n10.20.1.17 centos01.virtual.local centos01\n10.20.1.11 centos02.virtual.local centos02\n<\/code><\/pre>\niSCSI Target Servers Setup<\/span><\/h1>\n
sdb<\/code> virtual disk attached to the server VM’s and used as backing device for the iSCSI volumes are created with write-through<\/code> cache. This, or no cache at all, is the best choice when data security is of highest importance on expanse of speed. We’ll be using the vdisk_fileio<\/code> mode in SCST devices since it performs better in virtual environments over vdisk_blockio<\/code> although we are still presenting the LUN’s as block devices to the initiators letting them to format the drive.<\/p>\nSCST<\/span><\/h2>\n
[root@centos01 ~]# yum install svn asciidoc newt-devel xmlto rpm-build redhat-rpm-config gcc make \\\n patchutils elfutils-libelf-devel elfutils-devel zlib-devel binutils-devel \\\n python-devel audit-libs-devel bison hmaccalc perl-ExtUtils-Embed rng-tools \\\n ncurses-devel kernel-devel\n<\/code><\/pre>\nrngd<\/code> since we are running in VM’s and have not enough entropy for generating Corosync authentication key for example. Edit the \/etc\/sysconfig\/rngd<\/code> config file:<\/p>\n...\nEXTRAOPTIONS=\"-r \/dev\/urandom\"\n<\/code><\/pre>\n[root@centos01 ~]# service rngd start\n[root@centos01 ~]# chkconfig rngd on\n<\/code><\/pre>\n[root@centos01 ~]# svn checkout svn:\/\/svn.code.sf.net\/p\/scst\/svn\/trunk scst-trunk\n[root@centos01 ~]# cd scst-trunk\n<\/code><\/pre>\n[root@centos01 scst-trunk]# .\/scripts\/rebuild-rhel-kernel-rpm\n<\/code><\/pre>\n[root@centos01 scst-trunk]# yum -iVh ..\/*.rmp\n[root@centos01 scst-trunk]# shutdown -r now\n<\/code><\/pre>\n[root@centos01 scst-trunk]# make scst scst_install\n[root@centos01 scst-trunk]# make iscsi iscsi_install\n[root@centos01 scst-trunk]# make scstadm scstadm_install\n<\/code><\/pre>\n[root@centos01 ~]# vi \/etc\/scst.conf\nTARGET_DRIVER iscsi {\n enabled 1\n}\n<\/code><\/pre>\n[root@centos01 ~]# service scst start\nLoading and configuring SCST [ OK ]\n<\/code><\/pre>\n[root@centos01 ~]# lsmod | grep scst\nisert_scst 73646 3\niscsi_scst 191131 4 isert_scst\nrdma_cm 36354 1 isert_scst\nib_core 81507 6 isert_scst,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad\nscst 2117799 2 isert_scst,iscsi_scst\ndlm 148135 1 scst\nlibcrc32c 1246 3 iscsi_scst,drbd,sctp\ncrc_t10dif 1209 2 scst,sd_mod\n\n[root@centos01 ~]# ps aux | grep scst\nroot 3008 0.0 0.0 0 0 ? S 16:28 0:00 [scst_release_ac]\nroot 3009 0.0 0.0 0 0 ? S 16:28 0:00 [scst_release_ac]\nroot 3010 0.0 0.0 0 0 ? S 16:28 0:00 [scst_release_ac]\nroot 3011 0.0 0.0 0 0 ? S 16:28 0:00 [scst_release_ac]\nroot 3012 0.0 0.0 0 0 ? S< 16:28 0:00 [scst_uid]\nroot 3013 0.0 0.0 0 0 ? S 16:28 0:00 [scstd0]\nroot 3014 0.0 0.0 0 0 ? S 16:28 0:00 [scstd1]\nroot 3015 0.0 0.0 0 0 ? S 16:28 0:00 [scstd2]\nroot 3016 0.0 0.0 0 0 ? S 16:28 0:00 [scstd3]\nroot 3017 0.0 0.0 0 0 ? S< 16:28 0:00 [scst_initd]\nroot 3019 0.0 0.0 0 0 ? S< 16:28 0:00 [scst_mgmtd]\nroot 3054 0.0 0.0 4152 648 ? Ss 16:28 0:00 \/usr\/local\/sbin\/iscsi-scstd\n<\/code><\/pre>\n[root@centos01 scst-trunk]# chkconfig --list scst\n[root@centos01 scst-trunk]# chkconfig scst off\n[root@centos01 scst-trunk]# chkconfig --list scst\nscst 0:off 1:off 2:off 3:off 4:off 5:off 6:off\n<\/code><\/pre>\nDRBD<\/span><\/h2>\n
# yum install -y drbd84-utils kmod-drbd84\n<\/code><\/pre>\n\/etc\/drbd.d\/vg1.res<\/code>:<\/p>\nresource vg1 {\n startup {\n wfc-timeout 30;\n degr-wfc-timeout 20;\n outdated-wfc-timeout 10;\n }\n syncer {\n rate 40M;\n }\n disk {\n on-io-error detach;\n fencing resource-and-stonith;\n }\n handlers {\n fence-peer \"\/usr\/lib\/drbd\/crm-fence-peer.sh\";\n after-resync-target \"\/usr\/lib\/drbd\/crm-unfence-peer.sh\";\n outdate-peer \"\/usr\/lib\/heartbeat\/drbd-peer-outdater\";\n }\n options {\n on-no-data-accessible io-error;\n #on-no-data-accessible suspend-io;\n }\n net {\n timeout 60;\n ping-timeout 30;\n ping-int 30;\n cram-hmac-alg \"sha1\";\n shared-secret \"secret\";\n max-epoch-size 8192;\n max-buffers 8912;\n after-sb-0pri discard-zero-changes;\n after-sb-1pri discard-secondary;\n after-sb-2pri disconnect;\n }\n volume 0 {\n device \/dev\/drbd0;\n disk \/dev\/sdb;\n meta-disk internal;\n }\n on centos01 {\n address 10.20.1.17:7788;\n }\n on centos02 {\n address 10.20.1.11:7788;\n\n}\n<\/code><\/pre>\n# modprobe drbd\n<\/code><\/pre>\n# drbdadm create-md vg1\n# drbdadm up vg1\n<\/code><\/pre>\n[root@centos01 ~] drbdadm primary --force vg1\n<\/code><\/pre>\n[root@centos01 ~]# cat \/proc\/drbd\nversion: 8.4.7-1 (api:1\/proto:86-101)\nGIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11\n 0: cs:Connected ro:Secondary\/Primary ds:UpToDate\/UpToDate C r-----\n ns:0 nr:25992756 dw:25992756 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0\n<\/code><\/pre>\nUpToDate<\/code> and we are done with DRBD, we can switch it off and disable it on both nodes since it will be managed by Pacemaker:<\/p>\n# service drbd stop\n# chkconfig drbd off\n<\/code><\/pre>\nLVM<\/span><\/h2>\n
# yum install -y lvm2\n<\/code><\/pre>\n\/etc\/lvm\/lvm.conf<\/code>:<\/p>\n...\n filter = [ \"a|\/dev\/sda*|\", \"a|\/dev\/drbd*|\", \"r|.*|\" ]\n write_cache_state = 0\n volume_list = [ \"vg_centos\", \"vg1\", \"vg2\" ]\n...\n<\/code><\/pre>\n# rm -f \/etc\/lvm\/cache\/.cache\n<\/code><\/pre>\n[root@centos01 ~]# vgcreate vg1 \/dev\/drbd0\n[root@centos01 ~]# lvcreate --name lun1 -l 100%vg vg1\n[root@centos01 ~]# vgchange -aey vg1\n<\/code><\/pre>\nCorosync and Pacemaker<\/span><\/h2>\n
# yum install -y pacemaker corosync ipmitool openais cluster-glue fence-agents scsi-target-utils OpenIPMI OpenIPMI-libs freeipmi freeipmi-bmc-watchdog freeipmi-ipmidetectd\n<\/code><\/pre>\ncrmsh<\/code> from, create \/etc\/yum.repos.d\/ha-clustering.repo<\/code> file:<\/p>\n[haclustering]\nname=HA Clustering\nbaseurl=http:\/\/download.opensuse.org\/repositories\/network:\/ha-clustering:\/Stable\/CentOS_CentOS-6\/\nenabled=0\ngpgcheck=0\n<\/code><\/pre>\n# yum --enablerepo haclustering install -y crmsh\n<\/code><\/pre>\n\/etc\/corosync\/corosync.conf<\/code> file:<\/p>\ntotem {\n version: 2\n\n # How long before declaring a token lost (ms)\n token: 3000\n\n # How many token retransmits before forming a new configuration\n token_retransmits_before_loss_const: 10\n\n # How long to wait for join messages in the membership protocol (ms)\n join: 60\n\n # How long to wait for consensus to be achieved before starting a new round of membership configuration (ms)\n consensus: 3600\n\n # Turn off the virtual synchrony filter\n vsftype: none\n\n # Number of messages that may be sent by one processor on receipt of the token\n max_messages: 20\n\n # Stagger sending the node join messages by 1..send_join ms\n send_join: 45\n\n # Limit generated nodeids to 31-bits (positive signed integers)\n clear_node_high_bit: yes\n\n # Disable encryption\n secauth: off\n\n # How many threads to use for encryption\/decryption\n threads: 0\n\n # Optionally assign a fixed node id (integer)\n # nodeid: 1234\n\n # This specifies the mode of redundant ring, which may be none, active, or passive.\n rrp_mode: active\n\n interface {\n member {\n memberaddr: 10.20.1.17\n }\n member {\n memberaddr: 10.20.1.11\n }\n ringnumber: 0\n bindnetaddr: 10.20.1.11\n mcastaddr: 226.94.1.1\n mcastport: 5404\n }\n interface {\n member {\n memberaddr: 192.168.0.178\n }\n member {\n memberaddr: 192.168.0.179\n }\n ringnumber: 1\n bindnetaddr: 192.168.0.179\n mcastaddr: 226.94.41.1\n mcastport: 5405\n }\n transport: udpu\n}\namf {\n mode: disabled\n}\nservice {\n # Load the Pacemaker Cluster Resource Manager\n # if 0: start pacemaker\n # if 1: don't start pacemaker\n ver: 1\n name: pacemaker\n}\naisexec {\n user: root\n group: root\n}\nlogging {\n fileline: off\n to_stderr: yes\n to_logfile: no\n to_syslog: yes\n syslog_facility: daemon\n debug: off\n timestamp: on\n logger_subsys {\n subsys: QUORUM\n debug: off\n tags: enter|leave|trace1|trace2|trace3|trace4|trace6\n }\n}\n<\/code><\/pre>\n# service corosync start\n\n# corosync-cfgtool -s\nPrinting ring status.\nLocal node ID 184620042\nRING ID 0\n id = 10.20.1.11\n status = ring 0 active with no faults\nRING ID 1\n id = 192.168.0.179\n status = ring 1 active with no faults\n<\/code><\/pre>\n# chkconfig corosync on\n# chkconfig --list corosync\ncorosync 0:off 1:off 2:on 3:on 4:on 5:on 6:off\n<\/code><\/pre>\n[root@centos01 ~]# service pacemaker start\n\n[root@centos01 ~]# crm status\nLast updated: Fri Feb 26 12:49:06 2016\nLast change: Fri Feb 26 12:48:47 2016\nStack: classic openais (with plugin)\nCurrent DC: centos01 - partition with quorum\nVersion: 1.1.11-97629de\n2 Nodes configured, 2 expected votes\n12 Resources configured\n\nOnline: [ centos01 centos02 ]\n<\/code><\/pre>\n# chkconfig pacemaker on\n# chkconfig --list pacemaker\npacemaker 0:off 1:off 2:on 3:on 4:on 5:on 6:off\n<\/code><\/pre>\n