ZFS over iSCSI<\/code> storage in Proxmox PVE<\/code>, see Adding ZFS over iSCSI shared storage to Proxmox<\/a>. It was inspired by the excellent work from Saso Kiselkov<\/a> and his stmf-ha<\/code> project, please see the References section at the bottom of this page for details.<\/p>\nOmniOS<\/a> is an open source continuation of OpenSolaris (discontinued by Oracle when they acquired Sun Microsystems back in 2010) that builds on IllumOS<\/a> project, the OpenSolaris reincarnation. ZFS and iSCSI, or COMSTAR (Common Multiprotocol SCSI Target), have been part of Solaris for very long time bringing performance and stability to the storage solution.<\/p>\nFor the setup I’m using two VM’s, omnios01<\/code> and omnios02<\/code>, connected via two networks, public 192.168.0.0\/24<\/code> and private 10.10.1.0\/24<\/code> one configured on the hypervisor.<\/p>\nOmniOS installation and initial setup<\/span><\/h2>\nDownload the current stable OmniOS iso<\/a>, and launch a VM in Proxmox. Start it up and install accepting the defaults.<\/p>\nChange GRUB default timeout on boot from 30 to 5 seconds:<\/p>\n
root@omnios01:\/root# vi \/rpool\/boot\/grub\/menu.lst\n...\ntimeout 5\n...\n<\/code><\/pre>\nTry telling OmniOS we have 2 virtual cpu’s:<\/p>\n
root@omnios01:\/root# eeprom boot-ncpus=2\nroot@omnios01:\/root# psrinfo -vp\nThe physical processor has 1 virtual processor (0)\n x86 (GenuineIntel F61 family 15 model 6 step 1 clock 1900 MHz)\n Common KVM processor\n<\/code><\/pre>\nwhen we have 1 CPU (socket) with 2 cores.<\/p>\n
Then configure networking:<\/p>\n
root@omnios01:\/root# ipadm create-if e1000g0\nroot@omnios01:\/root# ipadm create-addr -T static -a local=192.168.0.141\/24 e1000g0\/v4\nroot@omnios01:\/root# route -p add default 192.168.0.1\nroot@omnios01:\/root# echo 'nameserver 192.168.0.1' >> \/etc\/resolv.conf\nroot@omnios01:\/root# cp \/etc\/nsswitch.dns \/etc\/nsswitch.conf\nroot@omnios01:\/root# ipadm show-addr\nADDROBJ TYPE STATE ADDR\nlo0\/v4 static ok 127.0.0.1\/8\ne1000g0\/v4 static ok 192.168.0.141\/24\nlo0\/v6 static ok ::1\/128\n<\/code><\/pre>\nSecondary interface:<\/p>\n
root@omnios01:\/root# ipadm create-if e1000g1\nroot@omnios01:\/root# ipadm create-addr -T dhcp e1000g1\/dhcp\nroot@omnios01:\/root# ipadm show-addr\nADDROBJ TYPE STATE ADDR\nlo0\/v4 static ok 127.0.0.1\/8\ne1000g0\/v4 static ok 192.168.0.141\/24\ne1000g1\/dhcp dhcp ok 10.10.1.13\/24\nlo0\/v6 static ok ::1\/128\n<\/code><\/pre>\nIf we want to enable jumbo frames and we have a switch that supports it:<\/p>\n
root@omnios01:\/root# dladm set-linkprop -p mtu=9000 e1000go\n<\/code><\/pre>\nConfigure the hosts file:<\/p>\n
\n- on omnios01\n
127.0.0.1 omnios01\n10.10.1.12 omnios02\n<\/code><\/pre>\n<\/li>\n- on omnios02\n
127.0.0.1 omnios02\n10.10.1.13 omnios01\n<\/code><\/pre>\n<\/li>\n<\/ul>\nConfigure SSH to allow both ssh key and password login for root user:<\/p>\n
root@omnios01:\/root# cat \/etc\/ssh\/sshd_config | grep -v ^# | grep .\nHostKey \/etc\/ssh\/ssh_host_rsa_key\nHostKey \/etc\/ssh\/ssh_host_ecdsa_key\nHostKey \/etc\/ssh\/ssh_host_ed25519_key\nPermitRootLogin yes\nStrictModes yes\nRSAAuthentication yes\nPubkeyAuthentication yes\nAuthorizedKeysFile .ssh\/authorized_keys\nHostbasedAuthentication no\nIgnoreRhosts yes\nPasswordAuthentication yes \nPermitEmptyPasswords no\nChallengeResponseAuthentication no\nGSSAPIAuthentication no \nUsePAM yes \nPrintMotd no\nTCPKeepAlive yes\nUseDNS no\nSubsystem sftp \/usr\/libexec\/sftp-server\nAllowUsers root\n<\/code><\/pre>\nand restart ssh service:<\/p>\n
root@omnios01:\/root# svcadm restart svc:\/network\/ssh:default\n<\/code><\/pre>\nNext check if the STMF service is running:<\/p>\n
root@omnios01:\/root# svcs -l stmf\nfmri svc:\/system\/stmf:default\nname STMF\nenabled true\nstate online\nnext_state none\nstate_time 25 August 2016 05:06:30 AM UTC\nlogfile \/var\/svc\/log\/system-stmf:default.log\nrestarter svc:\/system\/svc\/restarter:default\ndependency require_all\/none svc:\/system\/filesystem\/local:default (online)\n<\/code><\/pre>\nand if not enable it:<\/p>\n
root@omnios01:\/root# svcadm enable stmf\n<\/code><\/pre>\nThen enable COMSTAR iSCSI target service from the GUI or console:<\/p>\n
root@omnios01:\/root# svcadm enable -r svc:\/network\/iscsi\/target:default\nroot@omnios01:\/root# svcs -l iscsi\/target\nfmri svc:\/network\/iscsi\/target:default\nname iscsi target\nenabled true\nstate online\nnext_state none\nstate_time 25 August 2016 05:06:31 AM UTC\nlogfile \/var\/svc\/log\/network-iscsi-target:default.log\nrestarter svc:\/system\/svc\/restarter:default\ndependency require_any\/error svc:\/milestone\/network (online)\ndependency require_all\/none svc:\/system\/stmf:default (online)\n<\/code><\/pre>\nIf the services are missing we need to install the storage-server<\/code> package:<\/p>\n# pkg install group\/feature\/storage-server\n# svcadm enable stmf\n<\/code><\/pre>\nThe following 3 SATA (have to be on SATA bus for VM’s, not sure why) disks, apart from the root one, have been attached to each VM:<\/p>\n
root@omnios01:\/root# format\nSearching for disks...done\n\nAVAILABLE DISK SELECTIONS:\n 0. c2t0d0 <QEMU-HARDDISK-1.4.2 cyl 2085 alt 2 hd 255 sec 63>\n \/pci@0,0\/pci1af4,1100@7\/disk@0,0\n 1. c2t1d0 <QEMU-HARDDISK-1.4.2-10.00GB>\n \/pci@0,0\/pci1af4,1100@7\/disk@1,0\n 2. c2t2d0 <QEMU-HARDDISK-1.4.2-10.00GB>\n \/pci@0,0\/pci1af4,1100@7\/disk@2,0\n 3. c2t3d0 <QEMU-HARDDISK-1.4.2-10.00GB>\n \/pci@0,0\/pci1af4,1100@7\/disk@3,0\nSpecify disk (enter its number): ^C\nroot@omnios01:\/root#\n<\/code><\/pre>\nThey will be used to create a new zfs pool named pool1<\/code> from these 3x10GB disks using RAIDZ1 mirror that I will then use in my ZFS over iSCSI setup in the PVE cluster.<\/p>\niSCSI HA<\/span><\/h2>\nHA packages and stmf-ha setup<\/span><\/h3>\nInstall pre-built HA packages (HeartBeat, Cluster Glue, Pacemaker, OCF Agents) from the bundle created by Saso Kiselkov at (http:\/\/zfs-create.blogspot.com.au):<\/p>\n
root@omnios01:\/root# wget http:\/\/37.153.99.61\/HA.tar.bz2\nroot@omnios01:\/root# tar -xjvf HA.tar.bz2\nroot@omnios01:\/root# cd HA\/prebuilt_packages\nroot@omnios01:\/root# gunzip *.gz\nroot@omnios01:\/root# for PKG in *.pkg ; do pkgadd -d $PKG ; done\nroot@omnios01:\/root# vi ~\/.profile \n[...]\nexport PYTHONPATH=\/opt\/ha\/lib\/python2.6\/site-packages\nexport PATH=\/opt\/ha\/bin:\/opt\/ha\/sbin:$PATH\nexport OCF_ROOT=\/opt\/ha\/lib\/ocf\nexport OCF_AGENTS=\/opt\/ha\/lib\/ocf\/resource.d\/heartbeat\n\nroot@omnios01:\/root# pkg install ipmitool\nroot@omnios01:\/root# pkg install git\nroot@omnios01:\/root# git clone https:\/\/github.com\/skiselkov\/stmf-ha.git\nCloning into 'stmf-ha'...\nremote: Counting objects: 72, done.\nremote: Total 72 (delta 0), reused 0 (delta 0), pack-reused 72\nUnpacking objects: 100% (72\/72), done.\nChecking connectivity... done.\n\nroot@omnios01:\/root# cp stmf-ha\/heartbeat\/ZFS \/opt\/ha\/lib\/ocf\/resource.d\/heartbeat\/\nroot@omnios01:\/root# chmod +x \/opt\/ha\/lib\/ocf\/resource.d\/heartbeat\/ZFS\nroot@omnios01:\/root# perl -pi -e 's\/#DEBUG=0\/DEBUG=1\/' \/opt\/ha\/lib\/ocf\/resource.d\/heartbeat\/ZFS\nroot@omnios01:\/root# mkdir -p \/opt\/ha\/lib\/ocf\/lib\/heartbeat\/helpers\nroot@omnios01:\/root# cp stmf-ha\/heartbeat\/zfs-helper \/opt\/ha\/lib\/ocf\/lib\/heartbeat\/helpers\/\nroot@omnios01:\/root# chmod +x \/opt\/ha\/lib\/ocf\/lib\/heartbeat\/helpers\/zfs-helper\nroot@omnios01:\/root# cp stmf-ha\/stmf-ha \/usr\/sbin\/\nroot@omnios01:\/root# chmod +x \/usr\/sbin\/stmf-ha\nroot@omnios01:\/root# cp stmf-ha\/manpages\/stmf-ha.1m \/usr\/share\/man\/man1m\/\nroot@omnios01:\/root# man stmf-ha\n<\/code><\/pre>\nFix annoying ps<\/code> command error for crm<\/code>:<\/p>\nroot@omnios01:\/root# perl -pi -e 's#ps -e -o pid,command#ps -e -o pid,comm#' \/opt\/ha\/lib\/python2.6\/site-packages\/crm\/utils.py\n<\/code><\/pre>\nFix the IPaddr OCF agent, get patched one from Vincenco’s site see Use pacemaker and corosync on Illumos (OmniOS) to run a HA active\/passive cluster<\/a> for details:<\/p>\nroot@omnios01:\/root# cp \/opt\/ha\/lib\/ocf\/resource.d\/heartbeat\/IPaddr \/opt\/ha\/lib\/ocf\/resource.d\/heartbeat\/IPaddr.default\nroot@omnios01:\/root# wget -O \/opt\/ha\/lib\/ocf\/resource.d\/heartbeat\/IPaddr https:\/\/gist.githubusercontent.com\/vincepii\/6763170efa5050d2d73d\/raw\/bfc0e7df7dda9c673b4e0888240581f7963ff1b6\/IPaddr\n<\/code><\/pre>\nConfigure HeartBeat<\/span><\/h3>\nCreate the config file, we can edit the example on the project site.<\/p>\n
Based on Saso’s config from the git repo with serial link between nodes for heart beat I ended up with the following \/opt\/ha\/etc\/ha.d\/ha.cf<\/code> config file:<\/p>\n# Master Heartbeat configuration file\n# This file must be identical on all cluster nodes\n\n# GLOBAL OPTIONS\nuse_logd yes # Logging done in separate process to\n # prevent blocking on disk I\/O\nbaud 38400 # Run the serial link at 38.4 kbaud\nrealtime on # Enable real-time scheduling and lock\n # heartbeat into memory to prevent its\n # pages from ever being swapped out\n\napiauth cl_status gid=haclient uid=hacluster\n\n# NODE LIST SETUP\n# Node names depend on the machine's host name. To protect against\n# accidental joins from nodes that are part of other zfsstor clusters\n# we do not allow autojoins (plus we use shared-secret authentication).\nnode omnios01\nnode omnios02\nautojoin none\nauto_failback off\n\n# COMMUNICATION CHANNEL SETUP\n#mcast e1000g0 239.51.12.1 694 1 0 # management network\n#mcast e1000g1 239.51.12.1 694 1 0 # dedicated NIC between nodes\nmcast e1000g0 239.0.0.43 694 1 0\nbcast e1000g1 # dedicated NIC between nodes\n\n# STONITH\/FENCING IN CASE OR REAL NODES\n# Use ipmi to check power status and reboot nodes\n#stonith_host omnios01 external\/ipmi omnios02 192.168.0.141 <ipmi_admin_username> <ipmi_admin_password> lan\n#stonith_host omnios02 external\/ipmi omnios01 192.168.0.142 <ipmi_admin_username> <ipmi_admin_password> lan\n\n# NODE FAILURE DETECTION\nkeepalive 2 # Heartbeats every 2 second\nwarntime 5 # Start issuing warnings after 5 seconds\ndeadtime 15 # After 15 seconds, a node is considered dead\ninitdead 60 # Hold off declaring nodes dead for 60 seconds\n # after Heartbeat startup.\n\n# Enable the Pacemaker CRM\ncrm on\n#compression bz2\n#traditional_compression yes\n<\/code><\/pre>\nTo find the list of available STONITH devices run:<\/p>\n
root@omnios02:\/root# stonith -L\napcmaster\napcmastersnmp\napcsmart\nbaytech\ncyclades\ndrac3\nexternal\/drac5\nexternal\/dracmc-telnet\nexternal\/hetzner\nexternal\/hmchttp\nexternal\/ibmrsa\nexternal\/ibmrsa-telnet\nexternal\/ipmi\nexternal\/ippower9258\nexternal\/kdumpcheck\nexternal\/libvirt\nexternal\/nut\nexternal\/rackpdu\nexternal\/riloe\nexternal\/sbd\nexternal\/ssh\nexternal\/vcenter\nexternal\/vmware\nexternal\/xen0\nexternal\/xen0-ha\nibmhmc\nmeatware\nnull\nnw_rpc100s\nrcd_serial\nrps10\nssh\nsuicide\nwti_mpc\nwti_nps\nroot@omnios02:\/root#\n<\/code><\/pre>\nand add it to configuration if you have one.<\/p>\n
Create the authentication file:<\/p>\n
root@omnios01:\/root# (echo -ne \"auth 1\\n1 sha1 \"; openssl rand -rand \/dev\/random -hex 16 2> \/dev\/null) > \/opt\/ha\/etc\/ha.d\/authkeys\n<\/code><\/pre>\nGrant sudo access to the hacluster<\/code> user (on both nodes):<\/p>\nroot@omnios01:\/root\/HA# visudo\n[...]\nhacluster ALL=(ALL) NOPASSWD: ALL\n<\/code><\/pre>\nCreate the logd<\/code> config file:<\/p>\nroot@omnios01:\/root\/HA# cat \/opt\/ha\/etc\/logd.cf\n# File to write debug messages to\n# Default: \/var\/log\/ha-debug\ndebugfile \/var\/log\/ha-debug\n\n#\n#\n# File to write other messages to\n# Default: \/var\/log\/ha-log\nlogfile \/var\/log\/ha-log\n\n#\n#\n# Octal file permission to create the log files with\n# Default: 0644\nlogmode 0640\n\n\n#\n#\n# Facility to use for syslog()\/logger \n# (set to 'none' to disable syslog logging)\n# Default: daemon\nlogfacility daemon\n\n\n# Entity to be shown at beginning of a message\n# generated by the logging daemon itself\n# Default: \"logd\"\n#entity logd\n\n\n# Entity to be shown at beginning of _every_ message\n# passed to syslog (not to log files).\n#\n# Intended for easier filtering, or safe blacklisting.\n# You can filter on logfacility and this prefix.\n#\n# Message format changes like this:\n# -Nov 18 11:30:31 soda logtest: [21366]: info: total message dropped: 0\n# +Nov 18 11:30:31 soda common-prefix: logtest[21366]: info: total message dropped: 0\n#\n# Default: none (disabled)\n#syslogprefix linux-ha\n\n\n# Do we register to apphbd\n# Default: no\n#useapphbd no\n\n# There are two processes running for logging daemon\n# 1. parent process which reads messages from all client channels \n# and writes them to the child process \n# \n# 2. the child process which reads messages from the parent process through IPC\n# and writes them to syslog\/disk\n\n\n# set the send queue length from the parent process to the child process\n#\n#sendqlen 256 \n\n# set the recv queue length in child process\n#\n#recvqlen 256\n<\/code><\/pre>\nand enable the service:<\/p>\n
root@omnios01:\/root\/HA# svcadm enable ha_logd\n<\/code><\/pre>\nFinally start the HA service on both servers omnios01 and omnios02 and check the status:<\/p>\n
root@omnios02:\/root# \/opt\/ha\/lib\/heartbeat\/heartbeat\nheartbeat[3153]: 2016\/08\/26_06:30:47 info: Enabling logging daemon \nheartbeat[3153]: 2016\/08\/26_06:30:47 info: logfile and debug file are those specified in logd config file (default \/etc\/logd.cf)\nheartbeat[3153]: 2016\/08\/26_06:30:47 info: Pacemaker support: on\nheartbeat[3153]: 2016\/08\/26_06:30:47 info: **************************\nheartbeat[3153]: 2016\/08\/26_06:30:47 info: Configuration validated. Starting heartbeat 3.0.5\nroot@omnios02:\/root# \n<\/code><\/pre>\nand verify the cluster state:<\/p>\n
root@omnios02:\/root# crm status\n============\nLast updated: Fri Aug 26 06:30:54 2016\nStack: Heartbeat\nCurrent DC: omnios02 (641f06f8-65a9-44fd-80f4-96b87e9c4062) - partition with quorum\nVersion: 1.0.11-6e010d6b0d49a6b929d17c0114e9d2d934dc8e04\n2 Nodes configured, unknown expected votes\n0 Resources configured.\n============\n\nOnline: [ omnios01 omnios02 ]\n\nroot@omnios02:\/root#\n<\/code><\/pre>\nAfter we confirm it is working fine we can kill the above started process and enable the service:<\/p>\n
root@omnios01:\/root# svcadm enable heartbeat\nroot@omnios01:\/root# svcs -a | grep heart\nonline 7:48:21 svc:\/application\/cluster\/heartbeat:default\n<\/code><\/pre>\nNext we set some parameters for 2 node cluster, ie disable quorum and stonith<\/code> since this is in vm’s:<\/p>\nroot@omnios01:\/root# crm configure property no-quorum-policy=ignore\nroot@omnios01:\/root# crm configure property stonith-enabled=\"false\"\nroot@omnios01:\/root# crm configure property stonith-action=poweroff\n<\/code><\/pre>\nand set some values for resource stickiness (default zero, will move immediately) and migration threshold (default none, will try forever on the same node):<\/p>\n
root@omnios01:\/root# crm configure rsc_defaults resource-stickiness=100\nroot@omnios01:\/root# crm configure rsc_defaults migration-threshold=3\n\nroot@omnios01:\/root# crm configure show\nnode $id=\"11dc182d-5096-cd7c-acc6-eb3b3493f314\" omnios01\nnode $id=\"641f06f8-65a9-44fd-80f4-96b87e9c4062\" omnios02\nproperty $id=\"cib-bootstrap-options\" \\\n dc-version=\"1.0.11-6e010d6b0d49a6b929d17c0114e9d2d934dc8e04\" \\\n cluster-infrastructure=\"Heartbeat\" \\\n no-quorum-policy=\"ignore\" \\\n stonith-enabled=\"false\" \\\n last-lrm-refresh=\"1472435153\" \\\n stonith-action=\"poweroff\"\nrsc_defaults $id=\"rsc-options\" \\\n resource-stickiness=\"100\" \\\n migration-threshold=\"3\"\nroot@omnios01:\/root# \n<\/code><\/pre>\nCreate the first resource, the cluster VIP address:<\/p>\n
root@omnios01:\/root# crm configure\ncrm(live)configure# primitive p_pool1_VIP ocf:heartbeat:IPaddr \\\n params ip=\"10.10.1.205\" cidr_netmask=\"24\" nic=\"e1000g1\" \\\n op monitor interval=\"10s\" \\\n meta target-role=\"Started\"\ncrm(live)configure# verify\ncrm(live)configure# commit\ncrm(live)configure# exit\n<\/code><\/pre>\nand check the status again:<\/p>\n
root@omnios01:\/root# crm status\n============\nLast updated: Fri Aug 26 10:56:23 2016\nStack: Heartbeat\nCurrent DC: omnios02 (641f06f8-65a9-44fd-80f4-96b87e9c4062) - partition with quorum\nVersion: 1.0.11-6e010d6b0d49a6b929d17c0114e9d2d934dc8e04\n2 Nodes configured, unknown expected votes\n1 Resources configured.\n============\n\nOnline: [ omnios01 omnios02 ]\n\n p_pool1_VIP (ocf::heartbeat:IPaddr): Started omnios01\nroot@omnios01:\/root#\n<\/code><\/pre>\nand if we check the links on the server we can see the VIP:<\/p>\n
root@omnios01:\/root# ipadm show-addr\nADDROBJ TYPE STATE ADDR\nlo0\/v4 static ok 127.0.0.1\/8\ne1000g0\/v4 static ok 192.168.0.141\/24\ne1000g1\/cr static ok 10.10.1.205\/24\nlo0\/v6 static ok ::1\/128\n<\/code><\/pre>\nIn case we want to preserve the primary IP of the e1000g1<\/code> interface instead overwriting it with the VIP one we can create a VNIC and use it for the VIP:<\/p>\nroot@omnios01:\/root# dladm create-vnic -l e1000g1 VIP1\nroot@omnios01:\/root# dladm show-link\nLINK CLASS MTU STATE BRIDGE OVER\ne1000g0 phys 1500 up -- --\ne1000g1 phys 1500 up -- --\nVIP1 vnic 1500 up -- e1000g1\n\ncrm(live)configure# primitive p_pool1_VIP ocf:heartbeat:IPaddr \\\n params ip=\"10.10.1.205\" cidr_netmask=\"24\" nic=\"VIP1\" \\\n op monitor interval=\"10s\" \\\n meta target-role=\"Started\"\n\nroot@omnios01:\/root# crm configure show\nnode $id=\"11dc182d-5096-cd7c-acc6-eb3b3493f314\" omnios01 \\\n attributes standby=\"off\" online=\"on\"\nnode $id=\"641f06f8-65a9-44fd-80f4-96b87e9c4062\" omnios02\nprimitive p_pool1_VIP ocf:heartbeat:IPaddr \\\n params ip=\"10.10.1.205\" cidr_netmask=\"24\" nic=\"VIP1\" \\\n op monitor interval=\"10s\" \\\n meta target-role=\"Started\"\nproperty $id=\"cib-bootstrap-options\" \\\n dc-version=\"1.0.11-6e010d6b0d49a6b929d17c0114e9d2d934dc8e04\" \\\n cluster-infrastructure=\"Heartbeat\" \\\n no-quorum-policy=\"ignore\" \\\n stonith-enabled=\"false\" \\\n last-lrm-refresh=\"1472435153\" \\\n stonith-action=\"poweroff\"\nroot@omnios01:\/root# \n\nroot@omnios01:\/root# ipadm show-addr\nADDROBJ TYPE STATE ADDR\nlo0\/v4 static ok 127.0.0.1\/8\ne1000g0\/v4 static ok 192.168.0.141\/24\ne1000g1\/dhcp dhcp ok 10.10.1.13\/24\nVIP1\/cr static ok 10.10.1.205\/24\nlo0\/v6 static ok ::1\/128\n<\/code><\/pre>\nwhich is the way I ended up doing it.<\/p>\n
Now we can create our ZFS pool:<\/p>\n
root@omnios01:\/root# zpool create -m \/pool1 -o autoexpand=on -o autoreplace=on -o cachefile=none pool1 raidz c2t1d0 c2t2d0 c2t3d0\nroot@omnios01:\/root# zpool status pool1\n pool: pool1\n state: ONLINE\n scan: none requested\nconfig:\n\n NAME STATE READ WRITE CKSUM\n pool1 ONLINE 0 0 0\n raidz1-0 ONLINE 0 0 0\n c2t1d0 ONLINE 0 0 0\n c2t2d0 ONLINE 0 0 0\n c2t3d0 ONLINE 0 0 0\n\nerrors: No known data errors\nroot@omnios01:\/root#\n<\/code><\/pre>\nand set some parameters like lz4<\/code> compression:<\/p>\nroot@omnios01:\/root# zpool set feature@lz4_compress=enabled pool1\nroot@omnios01:\/root# zfs set compression=lz4 pool1\nroot@omnios01:\/root# zfs set atime=off pool1\nroot@omnios01:\/root# zfs list pool1\nNAME USED AVAIL REFER MOUNTPOINT\npool1 5.33G 13.9G 28.0K \/pool1\n<\/code><\/pre>\nafter that we have the following state:<\/p>\n
root@omnios01:\/root# zfs get all pool1\nNAME PROPERTY VALUE SOURCE\npool1 type filesystem -\npool1 creation Mon Aug 29 5:56 2016 -\npool1 used 5.33G -\npool1 available 13.9G -\npool1 referenced 28.0K -\npool1 compressratio 1.12x -\npool1 mounted yes -\npool1 quota none default\npool1 reservation none default\npool1 recordsize 128K default\npool1 mountpoint \/pool1 local\npool1 sharenfs off default\npool1 checksum on default\npool1 compression lz4 local\npool1 atime off local\npool1 devices on default\npool1 exec on default\npool1 setuid on default\npool1 readonly off default\npool1 zoned off default\npool1 snapdir hidden default\npool1 aclmode discard default\npool1 aclinherit restricted default\npool1 canmount on default\npool1 xattr on default\npool1 copies 1 default\npool1 version 5 -\npool1 utf8only off -\npool1 normalization none -\npool1 casesensitivity sensitive -\npool1 vscan off default\npool1 nbmand off default\npool1 sharesmb off default\npool1 refquota none default\npool1 refreservation none default\npool1 primarycache all default\npool1 secondarycache all default\npool1 usedbysnapshots 0 -\npool1 usedbydataset 28.0K -\npool1 usedbychildren 5.33G -\npool1 usedbyrefreservation 0 -\npool1 logbias latency default\npool1 dedup off default\npool1 mlslabel none default\npool1 sync standard default\npool1 refcompressratio 1.00x -\npool1 written 28.0K -\npool1 logicalused 6.00G -\npool1 logicalreferenced 13.5K -\npool1 filesystem_limit none default\npool1 snapshot_limit none default\npool1 filesystem_count none default\npool1 snapshot_count none default\npool1 redundant_metadata all default\nroot@omnios01:\/root# \n\nroot@omnios01:\/root# zfs mount\nrpool\/ROOT\/omnios \/\nrpool\/export \/export\nrpool\/export\/home \/export\/home\nrpool \/rpool\npool1 \/pool1\n\nroot@omnios01:\/root# mount | grep pool1\n\/pool1 on pool1 read\/write\/setuid\/devices\/nonbmand\/exec\/xattr\/atime\/dev=42d0012 on Mon Aug 29 05:56:18 2016\n<\/code><\/pre>\nNext step is to copy over the stmf-ha<\/code> config file so pacemaker can take control over COMSTAR resources:<\/p>\nroot@omnios01:\/root# cp stmf-ha\/samples\/stmf-ha-sample.conf \/pool1\/stmf-ha.conf\n<\/code><\/pre>\nNow we can create the resource in pacemaker:<\/p>\n
primitive p_zfs_pool1 ocf:heartbeat:ZFS \\\n params pool=\"pool1\" \\\n op start timeout=\"90\" \\\n op stop timeout=\"90\"\ncolocation col_pool1_with_VIP inf: p_zfs_pool1 p_pool1_VIP\norder o_pool1_before_VIP inf: p_zfs_pool1 p_pool1_VIP\n<\/code><\/pre>\nAfter committing the changes we need to start the resource on the node we created the pool on, in this case omnios01:<\/p>\n
root@omnios01:\/root# crm resource start p_zfs_pool1\n<\/code><\/pre>\nafter which we can see:<\/p>\n
root@omnios01:\/root# crm status\n============\nLast updated: Mon Aug 29 03:39:39 2016\nStack: Heartbeat\nCurrent DC: omnios02 (641f06f8-65a9-44fd-80f4-96b87e9c4062) - partition with quorum\nVersion: 1.0.11-6e010d6b0d49a6b929d17c0114e9d2d934dc8e04\n2 Nodes configured, unknown expected votes\n2 Resources configured.\n============\n\nOnline: [ omnios01 omnios02 ]\n\n p_pool1_VIP (ocf::heartbeat:IPaddr): Started omnios01\n p_zfs_pool1 (ocf::heartbeat:ZFS): Started omnios01\nroot@omnios01:\/root# \n<\/code><\/pre>\nNow we can create ZFS over iSCSI<\/code> resource in Proxmox<\/code> using the VIP address as portal. I created a vm with id of 109 in Proxmox which resulted with the pool1\/vm-109-disk-1<\/code> zvol being created on the OmniOS cluster.<\/p>\nThe last step is enabling the compression on the VM root device after we have created it so we can benefit from this feature:<\/p>\n
root@omnios01:\/root# zfs set compression=lz4 pool1\/vm-109-disk-1\nroot@omnios01:\/root# zfs get all pool1\/vm-109-disk-1\nNAME PROPERTY VALUE SOURCE\npool1\/vm-109-disk-1 type volume -\npool1\/vm-109-disk-1 creation Mon Aug 29 6:13 2016 -\npool1\/vm-109-disk-1 used 5.33G -\npool1\/vm-109-disk-1 available 13.9G -\npool1\/vm-109-disk-1 referenced 5.33G -\npool1\/vm-109-disk-1 compressratio 1.12x -\npool1\/vm-109-disk-1 reservation none default\npool1\/vm-109-disk-1 volsize 6G local\npool1\/vm-109-disk-1 volblocksize 64K -\npool1\/vm-109-disk-1 checksum on default\npool1\/vm-109-disk-1 compression lz4 local\npool1\/vm-109-disk-1 readonly off default\npool1\/vm-109-disk-1 copies 1 default\npool1\/vm-109-disk-1 refreservation none default\npool1\/vm-109-disk-1 primarycache all default\npool1\/vm-109-disk-1 secondarycache all default\npool1\/vm-109-disk-1 usedbysnapshots 0 -\npool1\/vm-109-disk-1 usedbydataset 5.33G -\npool1\/vm-109-disk-1 usedbychildren 0 -\npool1\/vm-109-disk-1 usedbyrefreservation 0 -\npool1\/vm-109-disk-1 logbias latency default\npool1\/vm-109-disk-1 dedup off default\npool1\/vm-109-disk-1 mlslabel none default\npool1\/vm-109-disk-1 sync standard default\npool1\/vm-109-disk-1 refcompressratio 1.12x -\npool1\/vm-109-disk-1 written 5.33G -\npool1\/vm-109-disk-1 logicalused 6.00G -\npool1\/vm-109-disk-1 logicalreferenced 6.00G -\npool1\/vm-109-disk-1 snapshot_limit none default\npool1\/vm-109-disk-1 snapshot_count none default\npool1\/vm-109-disk-1 redundant_metadata all default\npool1\/vm-109-disk-1 org.illumos.stmf-ha:lun 1 local\npool1\/vm-109-disk-1 org.illumos.stmf-ha:guid 600144F721dca2888ba402e411ee3af1 local\nroot@omnios01:\/root# \n<\/code><\/pre>\nGet the I\/O stats for the pool:<\/p>\n
root@omnios02:\/root# zpool iostat -v\n capacity operations bandwidth\npool alloc free read write read write\n---------- ----- ----- ----- ----- ----- -----\npool1 216K 29.7G 0 0 0 1\n raidz1 216K 29.7G 0 0 0 1\n c2t1d0 - - 0 0 6 5\n c2t2d0 - - 0 0 5 5\n c2t3d0 - - 0 0 5 5\n---------- ----- ----- ----- ----- ----- -----\nrpool 5.74G 10.1G 0 3 123 23.9K\n c2t0d0s0 5.74G 10.1G 0 3 123 23.9K\n---------- ----- ----- ----- ----- ----- -----\n<\/code><\/pre>\nWe can also see a COMSTAR target has been created:<\/p>\n
root@omnios01:\/root# itadm list-target -v\nTARGET NAME STATE SESSIONS \niqn.2010-08.org.illumos:stmf-ha:pool1 online 0 \n alias: -\n auth: none (defaults)\n targetchapuser: -\n targetchapsecret: unset\n tpg-tags: default\n<\/code><\/pre>\nand the LUN for the Proxmox VM:<\/p>\n
root@omnios01:\/root# sbdadm list-lu\nFound 1 LU(s)\n GUID DATA SIZE SOURCE\n-------------------------------- ------------------- ----------------\n600144f721dca2888ba402e411ee3af1 6442450944 \/dev\/zvol\/rdsk\/pool1\/vm-109-disk-1\n\nroot@omnios01:\/root# stmfadm list-lu -v\nLU Name: 600144F721DCA2888BA402E411EE3AF1\n Operational Status: Online\n Provider Name : sbd\n Alias : \/dev\/zvol\/rdsk\/pool1\/vm-109-disk-1\n View Entry Count : 1\n Data File : \/dev\/zvol\/rdsk\/pool1\/vm-109-disk-1\n Meta File : not set\n Size : 6442450944\n Block Size : 512\n Management URL : not set\n Vendor ID : SUN \n Product ID : COMSTAR \n Serial Num : not set\n Write Protect : Disabled\n Writeback Cache : Disabled\n Access State : Active\n\nroot@omnios01:\/root# zfs list -rH -t volume pool1 \npool1\/vm-109-disk-1 3.87G 15.3G 3.87G -\n<\/code><\/pre>\nInstall napp-it<\/code> ZFS appliance (optional)<\/span><\/h3>\nIn this case we don’t really need napp-it, we just need to launch 2 x OmniOS instances and install and configure the HA. Napp-it can help though for managing snapshots, clones, backups, rollbacks etc. for which having a web GUI should help a lot.<\/p>\n
root@omnios01:\/root# wget -O - www.napp-it.org\/nappit | perl\n<\/code><\/pre>\nand then connect to the web UI http:\/\/serverip:81<\/code> when finished. Reboot after installation of napp-it<\/code> then update napp-it (Menu About -> Update) or run:<\/p>\nroot@omnios01:\/root# pkg update\n<\/code><\/pre>\nMoving the resources from one node to another manually<\/span><\/h2>\nWe put the node the resource is running on into standby mode:<\/p>\n
root@omnios01:\/root# crm node attribute omnios01 set standby on\n\nroot@omnios01:\/root# crm status\n============\nLast updated: Mon Aug 29 02:10:59 2016\nStack: Heartbeat\nCurrent DC: omnios02 (641f06f8-65a9-44fd-80f4-96b87e9c4062) - partition with quorum\nVersion: 1.0.11-6e010d6b0d49a6b929d17c0114e9d2d934dc8e04\n2 Nodes configured, unknown expected votes\n1 Resources configured.\n============\n\nNode omnios01 (11dc182d-5096-cd7c-acc6-eb3b3493f314): standby\nOnline: [ omnios02 ]\n\nroot@omnios01:\/root# crm status\n============\nLast updated: Mon Aug 29 02:11:03 2016\nStack: Heartbeat\nCurrent DC: omnios02 (641f06f8-65a9-44fd-80f4-96b87e9c4062) - partition with quorum\nVersion: 1.0.11-6e010d6b0d49a6b929d17c0114e9d2d934dc8e04\n2 Nodes configured, unknown expected votes\n1 Resources configured.\n============\n\nNode omnios01 (11dc182d-5096-cd7c-acc6-eb3b3493f314): standby\nOnline: [ omnios02 ]\n\n p_pool1_VIP (ocf::heartbeat:IPaddr): Started omnios02\n<\/code><\/pre>\nand after couple of seconds we can see the VIP has moved to omnios02.<\/p>\n
root@omnios02:\/root# ipadm show-addr\nADDROBJ TYPE STATE ADDR\nlo0\/v4 static ok 127.0.0.1\/8\ne1000g0\/v4 static ok 192.168.0.142\/24\ne1000g1\/dhcp dhcp ok 10.10.1.12\/24\nVIP1\/cr static ok 10.10.1.205\/24\nlo0\/v6 static ok ::1\/128\n<\/code><\/pre>\nAnother test with all resources created:<\/p>\n
root@omnios01:\/root# crm status\n============\nLast updated: Tue Aug 30 07:10:25 2016\nStack: Heartbeat\nCurrent DC: omnios02 (641f06f8-65a9-44fd-80f4-96b87e9c4062) - partition with quorum\nVersion: 1.0.11-6e010d6b0d49a6b929d17c0114e9d2d934dc8e04\n2 Nodes configured, unknown expected votes\n2 Resources configured.\n============\n\nOnline: [ omnios01 omnios02 ]\n\n p_pool1_VIP (ocf::heartbeat:IPaddr): Started omnios01\n p_zfs_pool1 (ocf::heartbeat:ZFS): Started omnios01\nroot@omnios01:\/root#\n\nroot@omnios01:\/root# crm node attribute omnios01 set standby on\n\nroot@omnios01:\/root# crm status\n============\nLast updated: Tue Aug 30 07:14:22 2016\nStack: Heartbeat\nCurrent DC: omnios02 (641f06f8-65a9-44fd-80f4-96b87e9c4062) - partition with quorum\nVersion: 1.0.11-6e010d6b0d49a6b929d17c0114e9d2d934dc8e04\n2 Nodes configured, unknown expected votes\n2 Resources configured.\n============\n\nNode omnios01 (11dc182d-5096-cd7c-acc6-eb3b3493f314): standby\nOnline: [ omnios02 ]\n\n p_pool1_VIP (ocf::heartbeat:IPaddr): Started omnios02\n p_zfs_pool1 (ocf::heartbeat:ZFS): Started omnios02\nroot@omnios01:\/root#\n<\/code><\/pre>\nTo bring the node online again we run:<\/p>\n
root@omnios01:\/root# crm node attribute omnios01 set standby off\n<\/code><\/pre>\nThen we can check the status again:<\/p>\n
root@omnios01:\/root# crm status\n============\nLast updated: Mon Aug 29 02:14:42 2016\nStack: Heartbeat\nCurrent DC: omnios02 (641f06f8-65a9-44fd-80f4-96b87e9c4062) - partition with quorum\nVersion: 1.0.11-6e010d6b0d49a6b929d17c0114e9d2d934dc8e04\n2 Nodes configured, unknown expected votes\n1 Resources configured.\n============\n\nOnline: [ omnios01 omnios02 ]\n\nroot@omnios01:\/root# crm status\n============\nLast updated: Mon Aug 29 02:14:48 2016\nStack: Heartbeat\nCurrent DC: omnios02 (641f06f8-65a9-44fd-80f4-96b87e9c4062) - partition with quorum\nVersion: 1.0.11-6e010d6b0d49a6b929d17c0114e9d2d934dc8e04\n2 Nodes configured, unknown expected votes\n1 Resources configured.\n============\n\nOnline: [ omnios01 omnios02 ]\n\n p_pool1_VIP (ocf::heartbeat:IPaddr): Started omnios01\n<\/code><\/pre>\nand after couple of seconds we can see that omnios01 is back online and the VIP has moved back to omnios01. After setting resource-stickiness=100<\/code> though the resources will stay on omnios02.<\/p>\n\n Please note that I’m NOT<\/strong> using a shared storage for the cluster hence the ZFS resource failover can NOT<\/strong> work.\n<\/p><\/blockquote>\nRepeat the same and create pool2 on omnios02<\/span><\/h2>\nCreate another VIF on both nodes:<\/p>\n
root@omnios02:\/root# dladm create-vnic -l e1000g1 VIP2\nroot@omnios02:\/root# dladm show-link\nLINK CLASS MTU STATE BRIDGE OVER\ne1000g0 phys 1500 up -- --\ne1000g1 phys 1500 up -- --\nVIP1 vnic 1500 up -- e1000g1\nVIP2 vnic 1500 up -- e1000g1\n<\/code><\/pre>\nCreate the pool on omnios02:<\/p>\n
root@omnios02:\/root# zpool create -f -m \/pool2 -o autoexpand=on -o autoreplace=on -o cachefile=none pool2 raidz c2t1d0 c2t2d0 c2t3d0\n<\/code><\/pre>\nConfigure Pacemaker:<\/p>\n
primitive p_pool2_VIP ocf:heartbeat:IPaddr \\\n params ip=\"10.10.1.206\" cidr_netmask=\"24\" nic=\"VIP2\" \\\n op monitor interval=\"10s\" \\\n meta target-role=\"Started\"\nprimitive p_zfs_pool2 ocf:heartbeat:ZFS \\\n params pool=\"pool2\" \\\n op start timeout=\"90\" \\\n op stop timeout=\"90\"\ncolocation col_pool2_with_VIP inf: p_zfs_pool2 p_pool2_VIP\norder o_pool2_before_VIP inf: p_zfs_pool2 p_pool2_VIP\n<\/code><\/pre>\nThe result:<\/p>\n
root@omnios02:\/root# crm status\n============\nLast updated: Wed Feb 22 04:37:07 2017\nStack: Heartbeat\nCurrent DC: omnios02 (641f06f8-65a9-44fd-80f4-96b87e9c4062) - partition with quorum\nVersion: 1.0.11-6e010d6b0d49a6b929d17c0114e9d2d934dc8e04\n2 Nodes configured, unknown expected votes\n4 Resources configured.\n============\n\nOnline: [ omnios01 omnios02 ]\n\n p_pool1_VIP (ocf::heartbeat:IPaddr): Started omnios01\n p_zfs_pool1 (ocf::heartbeat:ZFS): Started omnios01\n p_pool2_VIP (ocf::heartbeat:IPaddr): Started omnios02\n p_zfs_pool2 (ocf::heartbeat:ZFS): Started omnios02\n<\/code><\/pre>\nCheck for the second VIP on omnios02:<\/p>\n
root@omnios02:\/root# ipadm show-addr\nADDROBJ TYPE STATE ADDR\nlo0\/v4 static ok 127.0.0.1\/8\ne1000g0\/v4 static ok 192.168.0.142\/24\ne1000g1\/dhcp dhcp ok 10.10.1.13\/24\nVIP2\/cr static ok 10.10.1.206\/24\nlo0\/v6 static ok ::1\/128\n<\/code><\/pre>\nAfter reboot of omnios02 node the COMSTAR target was also created:<\/p>\n
root@omnios02:\/root# itadm list-target -v\nTARGET NAME STATE SESSIONS \niqn.2010-08.org.illumos:stmf-ha:pool2 online 0 \n alias: -\n auth: none (defaults)\n targetchapuser: -\n targetchapsecret: unset\n tpg-tags: default \n<\/code><\/pre>\nNow we can use this pool for ZFS-over-iSCSI in Proxmox too. In real world scenario, where both head nodes are connected to 2 x JBOD SAS enclousers lets say for full redundancy, when one of the head nodes goes down its hosted pool will be migrated to the other head with no impact on the clients apart from the short pause during failover and VIP migration.<\/p>\n
Just a note rgarding Proxmox, to use these pools we need to generate SSH key for passwordless access from Proxmox nodes to OmniOS nodes, for example:<\/p>\n
root@proxmox01:\/etc\/pve\/priv\/zfs# ssh-keygen -t rsa -b 2048 -f 10.10.1.206_id_rsa -N ''\n<\/code><\/pre>\nand add the \/etc\/pve\/priv\/zfs\/10.10.1.206_id_rsa.pub<\/code> key to the authorized_keys<\/code> file of the root<\/code> user on the OmniOS servers.<\/p>\nReferences<\/span><\/h2>\n\n- Building zfs storage appliance part-1<\/a><\/li>\n
- Building zfs storage appliance part-2<\/a><\/li>\n
- Use pacemaker and corosync on Illumos (OmniOS) to run a HA active\/passive cluster<\/a><\/li>\n
- ZFS iSCSI Configuration<\/a><\/li>\n
- Configuring iSCSI Devices With COMSTAR<\/a><\/li>\n<\/ul>\n
APPENDIX<\/span><\/h2>\nAt the end some commands related to ZFS and COMSTAR that I find useful.<\/p>\n
COMSTAR COMMANDS<\/span><\/h3>\nInstall COMSTAR<\/span><\/h4>\n# pkg install group\/feature\/storage-server\n# svcadm enable stmf\n<\/code><\/pre>\n