{"id":279,"date":"2016-09-21T12:28:34","date_gmt":"2016-09-21T02:28:34","guid":{"rendered":"https:\/\/icicimov.com\/blog\/?p=279"},"modified":"2017-01-02T22:31:09","modified_gmt":"2017-01-02T11:31:09","slug":"adding-zfs-over-iscsi-shared-storage-to-proxmox","status":"publish","type":"post","link":"https:\/\/icicimov.com\/blog\/?p=279","title":{"rendered":"Adding ZFS over iSCSI shared storage to Proxmox"},"content":{"rendered":"

PVE-4.2 has built in support for ZFS over iSCSI for several targets among which is Solaris COMSTAR<\/code>. I built a ZFS VM appliance based on OmniOS<\/code> (Solaris) and napp-it<\/code>, see ZFS storage with OmniOS and iSCSI<\/a>, and managed to create a shared storage ZFS pool over iSCSI and launch vm09<\/code> with root device on zvol<\/code>. This also supports live migrations as well. Being an appliance, napp-it<\/code> provides a web UI at 192.168.0.141:81<\/code> that we can use to create and manage all our resources. It also supports creating a file shares like NFS and Samba over ZFS.<\/p>\n

This is the state on the OmniOS cluster:<\/p>\n

root@omnios01:\/root\/.ssh# zpool list\nNAME    SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT\npool1  29.8G  9.01G  20.7G         -     8%    30%  1.00x  ONLINE  -\nrpool  15.9G  5.03G  10.8G         -    18%    31%  1.00x  ONLINE  -\n\nroot@omnios01:\/root\/.ssh# zfs list\nNAME                                        USED  AVAIL  REFER  MOUNTPOINT\npool1                                      7.92G  13.2G  24.0K  \/pool1\npool1\/vm-109-disk-1                        6.00G  11.3G  6.00G  -\nrpool                                      6.06G  9.32G  22.5K  \/rpool\nrpool\/ROOT                                 3.02G  9.32G    19K  legacy\nrpool\/ROOT\/omnios                          3.02G  9.32G  2.38G  \/\nrpool\/ROOT\/omnios-backup-1                   65K  9.32G  1.67G  \/\nrpool\/ROOT\/omnios-backup-2                    1K  9.32G  2.36G  \/\nrpool\/ROOT\/omniosvar                         19K  9.32G    19K  legacy\nrpool\/ROOT\/pre_activate_16.07f_1472100772     1K  9.32G  2.36G  \/\nrpool\/ROOT\/pre_napp-it-16.07f                34K  9.32G  1.66G  \/\nrpool\/dump                                 2.00G  9.32G  2.00G  -\nrpool\/export                                 38K  9.32G    19K  \/export\nrpool\/export\/home                            19K  9.32G    19K  \/export\/home\nrpool\/proxmox                                 8K  9.32G     8K  -\nrpool\/swap                                 1.03G  10.4G  2.30M  -\n<\/code><\/pre>\n
We can see the pool1\/vm-109-disk-1<\/code> zvol here that Proxmox created upon vm09<\/code> creation. For this to work though we first need to grant root access to the appliance with ssh key from the Proxmox servers. We create the key in Proxmox and add it to the authorized_keys<\/code> file on Omnios. Then we check the connectivity.<\/p>\n
root@proxmox01:\/etc\/pve\/priv\/zfs# ssh-keygen -t rsa -b 2048 -f 192.168.0.141_id_rsa -N ''\nroot@proxmox01:\/etc\/pve\/priv\/zfs# ssh-copy-id -i \/etc\/pve\/priv\/zfs\/192.168.0.141_id_rsa root@192.168.0.141\nroot@proxmox01:\/etc\/pve\/priv\/zfs# \/usr\/bin\/ssh -vvv -o 'BatchMode=yes' -o 'StrictHostKeyChecking=no' -i \/etc\/pve\/priv\/zfs\/192.168.0.141_id_rsa root@192.168.0.141\n<\/code><\/pre>\n
We need to run the last ssh command from both cluster members. If successful then the cluster will gain access to the COMSTAR iSCSI as tested below:<\/p>\n
root@proxmox02:~# \/usr\/bin\/ssh -o 'BatchMode=yes' -i \/etc\/pve\/priv\/zfs\/192.168.0.141_id_rsa root@192.168.0.141 zfs list -o name,volsize,origin,type,refquota -t volume,filesystem -Hr\npool1 - - filesystem  none\npool1\/vm-109-disk-1 6G  - volume  -\nrpool - - filesystem  none\nrpool\/ROOT  - - filesystem  none\nrpool\/ROOT\/omnios - - filesystem  none\nrpool\/ROOT\/omnios-backup-1  - rpool\/ROOT\/omnios@2016-08-25-03:44:18 filesystem  none\nrpool\/ROOT\/omnios-backup-2  - rpool\/ROOT\/omnios@2016-08-25-04:22:55 filesystem  none\nrpool\/ROOT\/omniosvar  - - filesystem  none\nrpool\/ROOT\/pre_activate_16.07f_1472100772 - rpool\/ROOT\/omnios@2016-08-25-04:52:52 filesystem  none\nrpool\/ROOT\/pre_napp-it-16.07f - rpool\/ROOT\/omnios@2016-08-25-03:42:19 filesystem  none\nrpool\/dump  2G  - volume  -\nrpool\/export  - - filesystem  none\nrpool\/export\/home - - filesystem  none\nrpool\/proxmox 8G  - volume  -\nrpool\/swap  1G  - volume  -\n<\/code><\/pre>\n
Of course, we also need to create the COMSTAR iSCSI target in Omnios first manually or via napp-it<\/code> UI before we start anything in Proxmox.<\/p>\n
[serialposts]<\/p>\n","protected":false},"excerpt":{"rendered":"
PVE-4.2 has built in support for ZFS over iSCSI for several targets among which is Solaris COMSTAR. I built a ZFS VM appliance based on OmniOS (Solaris) and napp-it, see ZFS storage with OmniOS and iSCSI, and managed to create…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[17,14,9,16,13],"tags":[26,25,18,24,23,30],"class_list":["post-279","post","type-post","status-publish","format-standard","hentry","category-cluster","category-docker","category-high-availability","category-storage","category-virtualization","tag-cluster","tag-high-availability","tag-iscsi","tag-kvm","tag-proxmox","tag-zfs"],"_links":{"self":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/279","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=279"}],"version-history":[{"count":2,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/279\/revisions"}],"predecessor-version":[{"id":281,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/279\/revisions\/281"}],"wp:attachment":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=279"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=279"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=279"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}