{"id":273,"date":"2016-09-18T22:19:06","date_gmt":"2016-09-18T12:19:06","guid":{"rendered":"https:\/\/icicimov.com\/blog\/?p=273"},"modified":"2017-01-12T15:27:21","modified_gmt":"2017-01-12T04:27:21","slug":"adding-adding-drbd-shared-volumes-to-proxmox-to-support-live-migration","status":"publish","type":"post","link":"https:\/\/icicimov.com\/blog\/?p=273","title":{"rendered":"Adding DRBD shared volumes to Proxmox to support Live Migration"},"content":{"rendered":"<p>The plan is to create 2 resources in <code>Primary\/Primary<\/code> mode. The first one <code>r0<\/code> will be used to store disk images for VM&#8217;s running on <code>proxmox01<\/code> and <code>r1<\/code> for the VM&#8217;s running on <code>proxmox02<\/code>. This way we can easily recover from split brain in that way that for r0 we can discard all the changes from proxmox02 and for r1 all the changes from proxmox01.<\/p>\n<p>Then we will create LVM volume from each of the resources and add that as shared LVM storage to the PVE Datacenter. PVE 4.x comes with DRBD 9.0.x installed:<\/p>\n<pre><code>root@proxmox02:~# modinfo drbd\nfilename:       \/lib\/modules\/4.4.6-1-pve\/kernel\/drivers\/block\/drbd\/drbd.ko\nalias:          block-major-147-*\nlicense:        GPL\nversion:        9.0.2-1\ndescription:    drbd - Distributed Replicated Block Device v9.0.2-1\nauthor:         Philipp Reisner &lt;phil @linbit.com&gt;, Lars Ellenberg &lt;lars @linbit.com&gt;\nsrcversion:     17486A03B4EFD83FC6539BB\ndepends:        libcrc32c\nvermagic:       4.4.6-1-pve SMP mod_unload modversions\nparm:           minor_count:Approximate number of drbd devices (1-255) (uint)\nparm:           disable_sendpage:bool\nparm:           allow_oos:DONT USE! (bool)\nparm:           enable_faults:int\nparm:           fault_rate:int\nparm:           fault_count:int\nparm:           fault_devs:int\nparm:           two_phase_commit_fail:int\nparm:           usermode_helper:string\n<\/code><\/pre>\n<p>First we configure the common parameters in <code>\/etc\/drbd.d\/global_common.conf<\/code>:<\/p>\n<pre><code>global {\n    usage-count yes;\n    # minor-count dialog-refresh disable-ip-verification\n    # cmd-timeout-short 5; cmd-timeout-medium 121; cmd-timeout-long 600;\n}\n\ncommon {\n    handlers {\n        # These are EXAMPLE handlers only.\n        # They may have severe implications,\n        # like hard resetting the node under certain circumstances.\n        # Be careful when chosing your poison.\n\n        # pri-on-incon-degr \"\/usr\/lib\/drbd\/notify-pri-on-incon-degr.sh; \/usr\/lib\/drbd\/notify-emergency-reboot.sh; echo b &gt; \/proc\/sysrq-trigger ; reboot -f\";\n        # pri-lost-after-sb \"\/usr\/lib\/drbd\/notify-pri-lost-after-sb.sh; \/usr\/lib\/drbd\/notify-emergency-reboot.sh; echo b &gt; \/proc\/sysrq-trigger ; reboot -f\";\n        # local-io-error \"\/usr\/lib\/drbd\/notify-io-error.sh; \/usr\/lib\/drbd\/notify-emergency-shutdown.sh; echo o &gt; \/proc\/sysrq-trigger ; halt -f\";\n        # fence-peer \"\/usr\/lib\/drbd\/crm-fence-peer.sh\";\n        # split-brain \"\/usr\/lib\/drbd\/notify-split-brain.sh root\";\n        # out-of-sync \"\/usr\/lib\/drbd\/notify-out-of-sync.sh root\";\n        # before-resync-target \"\/usr\/lib\/drbd\/snapshot-resync-target-lvm.sh -p 15 -- -c 16k\";\n        # after-resync-target \/usr\/lib\/drbd\/unsnapshot-resync-target-lvm.sh;\n    }\n\n    startup {\n        # wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb\n        wfc-timeout          300;\n        degr-wfc-timeout     120;\n        outdated-wfc-timeout 120;\n        become-primary-on    both;\n    }\n\n    options {\n        # cpu-mask on-no-data-accessible\n    }\n\n    disk {\n        # size on-io-error fencing disk-barrier disk-flushes\n        # disk-drain md-flushes resync-rate resync-after al-extents\n        # c-plan-ahead c-delay-target c-fill-target c-max-rate\n        # c-min-rate disk-timeout\n        resync-rate  40M;\n        on-io-error  detach;\n        disk-barrier no;\n        disk-flushes no;\n    }\n\n    net {\n        # protocol timeout max-epoch-size max-buffers unplug-watermark\n        # connect-int ping-int sndbuf-size rcvbuf-size ko-count\n        # allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri\n        # after-sb-1pri after-sb-2pri always-asbp rr-conflict\n        # ping-timeout data-integrity-alg tcp-cork on-congestion\n        # congestion-fill congestion-extents csums-alg verify-alg\n        # use-rle\n        protocol      C;\n        fencing       resource-only;\n        after-sb-0pri discard-zero-changes;\n        after-sb-1pri discard-secondary;\n        after-sb-2pri disconnect;\n        allow-two-primaries;\n    }\n}\n<\/code><\/pre>\n<p>and then we create the resources, first for r0 we create <code>\/etc\/drbd.d\/r0.res<\/code> file:<\/p>\n<pre><code>resource r0 {\n    on proxmox01 {\n        device           \/dev\/drbd0 minor 0;\n        disk             \/dev\/vdc1;\n        address          ipv4 10.20.1.185:7788;\n        meta-disk        internal;\n    }\n    on proxmox02 {\n        device           \/dev\/drbd0 minor 0;\n        disk             \/dev\/vdc1;\n        address          ipv4 10.20.1.186:7788;\n        meta-disk        internal;\n    }\n}\n<\/code><\/pre>\n<p>and for r<code>we create<\/code>\/etc\/drbd.d\/r<code>.res<\/code> file:<\/p>\n<pre><code>root@proxmox01:~# vi \/etc\/drbd.d\/r1.res\nresource r1 {\n    on proxmox01 {\n        device           \/dev\/drbd1 minor 1;\n        disk             \/dev\/vdc2;\n        address          ipv4 10.20.1.185:7789;\n        meta-disk        internal;\n    }\n    on proxmox02 {\n        device           \/dev\/drbd1 minor 1;\n        disk             \/dev\/vdc2;\n        address          ipv4 10.20.1.186:7789;\n        meta-disk        internal;\n    }\n}\n<\/code><\/pre>\n<p>and copy over the files to the second node:<\/p>\n<pre><code>root@proxmox01:~# rsync -r \/etc\/drbd.d\/ proxmox02:\/etc\/drbd.d\/\n<\/code><\/pre>\n<p>Then we start the service and create the resources, we do this on both nodes:<\/p>\n<pre><code># service drbd start\n# drbdadm create-md r{0,1}\n# drbdadm up r{0,1}\n<\/code><\/pre>\n<p>Then on one node only we set the resources to Primary state and start the initial sync:<\/p>\n<pre><code>root@hpms01:~# drbdadm primary --force r{0,1}\n\nroot@proxmox01:~# drbdadm status\nr0 role:Primary\n  disk:UpToDate\n  proxmox02 role:Secondary\n    replication:SyncSource peer-disk:Inconsistent done:85.82\n\nr1 role:Primary\n  disk:UpToDate\n  proxmox02 role:Secondary\n    replication:SyncSource peer-disk:Inconsistent done:7.86\n<\/code><\/pre>\n<p>When that finishes we promote the resources to Primary state on the second node too:<\/p>\n<pre><code>root@proxmox02:~# drbdadm primary r{0,1}\n\nroot@proxmox01:~# drbdadm status\nr0 role:Primary\n  disk:UpToDate\n  proxmox02 role:Primary\n    peer-disk:UpToDate\n\nr1 role:Primary\n  disk:UpToDate\n  proxmox02 role:Primary\n    peer-disk:UpToDate\n<\/code><\/pre>\n<p>Then on both nodes we configure LVM in <code>\/etc\/lvm\/lvm.conf<\/code> file to look for volumes on the DRBD devices instead the underlying block devices:<\/p>\n<pre><code>[...]\n    filter = [ \"r|\/dev\/zd*|\", \"r|\/dev\/vdc|\", \"a|\/dev\/drbd.*|\", \"a|\/dev\/vda|\", \"a|\/dev\/vdb|\" ]\n[...]\n<\/code><\/pre>\n<p>Next we create the DRBD physical LVM devices:<\/p>\n<pre><code>root@proxmox01:~# pvcreate \/dev\/drbd{0,1}\n  Physical volume \"\/dev\/drbd0\" successfully created\n  Physical volume \"\/dev\/drbd1\" successfully created\n\nroot@proxmox02:~# pvcreate \/dev\/drbd{0,1}\n  Physical volume \"\/dev\/drbd0\" successfully created\n  Physical volume \"\/dev\/drbd1\" successfully created\n\nand create the volume groups on one of the nodes only:\nroot@proxmox01:~# vgcreate vg_drbd0 \/dev\/drbd0\n  Volume group \"vg_drbd0\" successfully created\n\nroot@proxmox01:~# vgcreate vg_drbd1 \/dev\/drbd1\n  Volume group \"vg_drbd1\" successfully created\n<\/code><\/pre>\n<p>The groups now can be seen on both nodes thanks to the DRBD replication:<\/p>\n<pre><code>root@proxmox01:~# vgs\n  VG         #PV #LV #SN Attr   VSize  VFree\n  pve          1   3   0 wz--n- 31.87g  3.87g\n  vg_drbd0     1   0   0 wz--n-  9.31g  9.31g\n  vg_drbd1     1   0   0 wz--n- 11.68g 11.68g\n  vg_proxmox   1   1   0 wz--n- 20.00g     0\n\nroot@proxmox02:~# vgs\n  VG         #PV #LV #SN Attr   VSize  VFree\n  pve          1   3   0 wz--n- 31.87g  3.87g\n  vg_drbd0     1   0   0 wz--n-  9.31g  9.31g\n  vg_drbd1     1   0   0 wz--n- 11.68g 11.68g\n  vg_proxmox   1   1   0 wz--n- 20.00g     0\n<\/code><\/pre>\n<p>Then we go to the PVE admin web console and add LVM storage under Datacenter, select vg_drbd0 from drop-down list and check the boxes for active and shared. In the Nodes drop-down list we select both nodes proxmox01 and proxmox02 and click Add. Repeat same for <code>vg_drbd1<\/code>.<\/p>\n<p>[serialposts]<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The plan is to create 2 resources in Primary\/Primary mode. The first one r0 will be used to store disk images for VM&#8217;s running on proxmox01 and r1 for the VM&#8217;s running on proxmox02. This way we can easily recover&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[17,9,22,13],"tags":[26,21,25,24,23],"class_list":["post-273","post","type-post","status-publish","format-standard","hentry","category-cluster","category-high-availability","category-kvm","category-virtualization","tag-cluster","tag-drbd","tag-high-availability","tag-kvm","tag-proxmox"],"_links":{"self":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/273","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=273"}],"version-history":[{"count":2,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/273\/revisions"}],"predecessor-version":[{"id":386,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/273\/revisions\/386"}],"wp:attachment":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=273"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=273"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=273"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}