{"id":408,"date":"2014-09-02T09:45:26","date_gmt":"2014-09-01T23:45:26","guid":{"rendered":"https:\/\/icicimov.com\/blog\/?p=408"},"modified":"2017-02-23T23:57:06","modified_gmt":"2017-02-23T12:57:06","slug":"ceph-cluster-on-ubuntu-14-04","status":"publish","type":"post","link":"https:\/\/icicimov.com\/blog\/?p=408","title":{"rendered":"Ceph cluster on Ubuntu-14.04"},"content":{"rendered":"

As pointed on its home page, Ceph<\/a> is a unified, distributed storage system designed for performance, reliability and scalability. It provides seamless access to objects using native language bindings or radosgw (RGW), a REST interface that’s compatible with applications written for S3 and Swift. Ceph’s RADOS Block Device (RBD) provides access to block device images that are striped and replicated across the entire storage cluster. It also provides a POSIX-compliant network file system (CephFS) that aims for high performance, large data storage, and maximum compatibility with legacy.<\/p>\n

I’m setting up a ceph cluster on three VM’s ostack-ceph1, ostack-ceph2 and ostack-ceph3, using the first one as deployment node as well.<\/p>\n

First we make sure the nodes can resolve each other names, we add to \/etc\/hosts<\/code> on each server:<\/p>\n

192.168.122.211 ostack-ceph1.virtual.local  ostack-ceph1\n192.168.122.212 ostack-ceph2.virtual.local  ostack-ceph2\n192.168.122.213 ostack-ceph3.virtual.local  ostack-ceph3\n<\/code><\/pre>\n
Then setup a password-less login for my user from ostack-ceph1 to ostack-ceph2 and ostack-ceph3. Create ssh public-private key pair:<\/p>\n
igorc@ostack-ceph1:~$ ssh-keygen -t rsa -f \/home\/igorc\/.ssh\/id_rsa -N ''\n<\/code><\/pre>\n
and copy the public key over to the other nodes:<\/p>\n
igorc@ostack-ceph1:~$ cat \/home\/igorc\/.ssh\/id_rsa.pub | ssh igorc@ostack-ceph2 \"cat >> ~\/.ssh\/authorized_keys\"\nigorc@ostack-ceph1:~$ cat \/home\/igorc\/.ssh\/id_rsa.pub | ssh igorc@ostack-ceph3 \"cat >> ~\/.ssh\/authorized_keys\"\nigorc@ostack-ceph1:~$ ssh igorc@ostack-ceph2 \"chmod 600 ~\/.ssh\/authorized_keys\"\nigorc@ostack-ceph1:~$ ssh igorc@ostack-ceph3 \"chmod 600 ~\/.ssh\/authorized_keys\"\n<\/code><\/pre>\n
Next set:<\/p>\n
%sudo   ALL=(ALL:ALL) NOPASSWD:ALL\n<\/code><\/pre>\n
in \/etc\/sudoers<\/code> file on each server. Make sure the user is part of the sudo<\/code> group on each node.<\/p>\n
$ sudo usermod -a -G sudo igorc\n<\/code><\/pre>\n
Then we can install ceph-deploy<\/code> on ostak-ceph1:<\/p>\n
igorc@ostack-ceph1:~$ wget -q -O- 'https:\/\/ceph.com\/git\/?p=ceph.git;a=blob_plain;f=keys\/release.asc' | sudo apt-key add -\nigorc@ostack-ceph1:~$ echo deb http:\/\/ceph.com\/debian-dumpling\/ $(lsb_release -sc) main | sudo tee \/etc\/apt\/sources.list.d\/ceph.list\nigorc@ostack-ceph1:~$ sudo aptitude update && sudo aptitude install ceph-deploy\n<\/code><\/pre>\n
Now we can prepare the deployment directory, install ceph on all nodes and initiate the cluster:<\/p>\n
igorc@ostack-ceph1:~$ mkdir ceph-cluster && cd ceph-cluster\nigorc@ostack-ceph1:~\/ceph-cluster$ ceph-deploy install ostack-ceph1 ostack-ceph2 ostack-ceph3\nigorc@ostack-ceph1:~\/ceph-cluster$ ceph-deploy --cluster ceph new ostack-ceph{1,2,3}\n<\/code><\/pre>\n
Then I modify the ceph.conf<\/code> file as shown below:<\/p>\n
igorc@ostack-ceph1:~\/ceph-cluster$ vi ceph.conf \n[global]\nfsid = ed8d8819-e05b-48d4-ba9f-f0bc8493f18f\nmon_initial_members = ostack-ceph1, ostack-ceph2, ostack-ceph3\nmon_host = 192.168.122.211, 192.168.122.212, 192.168.122.213\nauth_cluster_required = cephx\nauth_service_required = cephx\nauth_client_required = cephx\nfilestore_xattr_use_omap = true\npublic_network = 192.168.122.0\/24\n\n[mon.ostack-ceph1]\n     host = ostack-ceph1 \n     mon addr = 192.168.122.211:6789\n\n[mon.ostack-ceph2]\n     host = ostack-ceph2 \n     mon addr = 192.168.122.212:6789\n\n[mon.ostack-ceph3]\n     host = ostack-ceph3 \n     mon addr = 192.168.122.213:6789\n\n# added below config\n[osd]\nosd_journal_size = 512 \nosd_pool_default_size = 3\nosd_pool_default_min_size = 1\nosd_pool_default_pg_num = 64 \nosd_pool_default_pgp_num = 64\n<\/code><\/pre>\n
and continue with Monitors installation:<\/p>\n
igorc@ostack-ceph1:~\/ceph-cluster$ ceph-deploy mon create ostack-ceph1 ostack-ceph2 ostack-ceph3\n<\/code><\/pre>\n
Also collect the admin keyring on the local node and set read permissions:<\/p>\n
igorc@ostack-ceph1:~\/ceph-cluster$ ceph-deploy gatherkeys ostack-ceph1\nigorc@ostack-ceph1:~\/ceph-cluster$ sudo chmod +r \/etc\/ceph\/ceph.client.admin.keyring\n<\/code><\/pre>\n
Now we can check the quorum status of the cluster:<\/p>\n
igorc@ostack-ceph1:~\/ceph-cluster$ ceph quorum_status --format json-pretty\n\n{ \"election_epoch\": 6,\n  \"quorum\": [\n        0,\n        1,\n        2],\n  \"quorum_names\": [\n        \"ostack-ceph1\",\n        \"ostack-ceph2\",\n        \"ostack-ceph3\"],\n  \"quorum_leader_name\": \"ostack-ceph1\",\n  \"monmap\": { \"epoch\": 1,\n      \"fsid\": \"ed8d8819-e05b-48d4-ba9f-f0bc8493f18f\",\n      \"modified\": \"0.000000\",\n      \"created\": \"0.000000\",\n      \"mons\": [\n            { \"rank\": 0,\n              \"name\": \"ostack-ceph1\",\n              \"addr\": \"192.168.122.211:6789\\\/0\"},\n            { \"rank\": 1,\n              \"name\": \"ostack-ceph2\",\n              \"addr\": \"192.168.122.212:6789\\\/0\"},\n            { \"rank\": 2,\n              \"name\": \"ostack-ceph3\",\n              \"addr\": \"192.168.122.213:6789\\\/0\"}]}}\n<\/code><\/pre>\n
We also deploy the MDS component on all 3 nodes for redundancy:<\/p>\n
igorc@ostack-ceph1:~\/ceph-cluster$ ceph-deploy --overwrite-conf mds create ostack-ceph1 ostack-ceph2 ostack-ceph3\n<\/code><\/pre>\n
Next we set-up the OSD’s:<\/p>\n
igorc@ostack-ceph1:~\/ceph-cluster$ ceph-deploy --overwrite-conf osd --zap-disk create ostack-ceph1:\/dev\/sda ostack-ceph2:\/dev\/sda ostack-ceph3:\/dev\/sda\n<\/code><\/pre>\n
after which we can create our first pool:<\/p>\n
igorc@ostack-ceph1:~\/ceph-cluster$ ceph osd pool create datastore 100\npool 'datastore' created\n<\/code><\/pre>\n
The number of placement groups (pgp) is based on 100 x the number of OSD\u2019s \/ the number of replicas we want to maintain<\/code>. I want 3 copies of the data (so if a server fails no data is lost), so 3 x 100 \/ 3 = 100<\/code>.<\/p>\n
Since I want to use this cluster as backend storage for Openstack Cinder and Glance I need to create some users with permissions to access specific pools. First is the client.datastore<\/code> user for Cinder with access to the datastore<\/code> pool we just created. We need to create a keyring, add it to ceph and set the appropriate permissions for the user on the pool:<\/p>\n
igorc@ostack-ceph1:~\/ceph-cluster$ sudo ceph-authtool --create-keyring \/etc\/ceph\/ceph.client.datastore.keyring\nigorc@ostack-ceph1:~\/ceph-cluster$ sudo chmod +r \/etc\/ceph\/ceph.client.datastore.keyring\nigorc@ostack-ceph1:~\/ceph-cluster$ sudo ceph-authtool \/etc\/ceph\/ceph.client.datastore.keyring -n client.datastore --gen-key\nigorc@ostack-ceph1:~\/ceph-cluster$ sudo ceph-authtool -n client.datastore --cap mon 'allow r' --cap osd 'allow class-read object_prefix rbd_children, allow rwx pool=datastore' \/etc\/ceph\/ceph.client.datastore.keyring\nigorc@ostack-ceph1:~\/ceph-cluster$ ceph auth add client.datastore -i \/etc\/ceph\/ceph.client.datastore.keyring\n<\/code><\/pre>\n
Now, we add the client.datastore<\/code> user settings to the ceph.conf<\/code> file:<\/p>\n
...\n[client.datastore]\n     keyring = \/etc\/ceph\/ceph.client.datastore.keyring\n<\/code><\/pre>\n
and push that to all cluster members:<\/p>\n
igorc@ostack-ceph1:~\/ceph-cluster$ ceph-deploy --overwrite-conf config push ostack-ceph1 ostack-ceph2 ostack-ceph3\n<\/code><\/pre>\n
Since we have MON service running on each host we want to be able to mount from each host too so we need to copy the new key we created:<\/p>\n
igorc@ostack-ceph1:~\/ceph-cluster$ scp \/etc\/ceph\/ceph.client.datastore.keyring ostack-ceph2:~ && ssh ostack-ceph2 sudo cp ceph.client.datastore.keyring \/etc\/ceph\/  \nigorc@ostack-ceph1:~\/ceph-cluster$ scp \/etc\/ceph\/ceph.client.datastore.keyring ostack-ceph3:~ && ssh ostack-ceph3 sudo cp ceph.client.datastore.keyring \/etc\/ceph\/\n<\/code><\/pre>\n
We repeat the same procedure for Glance user and pool:<\/p>\n
igorc@ostack-ceph1:~\/ceph-cluster$ ceph osd pool create images 64\nigorc@ostack-ceph1:~\/ceph-cluster$ sudo ceph-authtool --create-keyring \/etc\/ceph\/ceph.client.images.keyring\nigorc@ostack-ceph1:~\/ceph-cluster$ sudo chmod +r \/etc\/ceph\/ceph.client.images.keyring\nigorc@ostack-ceph1:~\/ceph-cluster$ sudo ceph-authtool \/etc\/ceph\/ceph.client.images.keyring -n client.images --gen-key\nigorc@ostack-ceph1:~\/ceph-cluster$ sudo ceph-authtool -n client.images --cap mon 'allow r' --cap osd 'allow class-read object_prefix rbd_children, allow rwx pool=images' \/etc\/ceph\/ceph.client.images.keyring \nigorc@ostack-ceph1:~\/ceph-cluster$ ceph auth add client.images -i \/etc\/ceph\/ceph.client.images.keyring \n<\/code><\/pre>\n
Now, we add the client.images<\/code> user settings to the ceph.conf<\/code> file:<\/p>\n
...\n[client.images]\n     keyring = \/etc\/ceph\/ceph.client.images.keyring\n<\/code><\/pre>\n
and push that to all cluster members:<\/p>\n
igorc@ostack-ceph1:~\/ceph-cluster$ ceph-deploy --overwrite-conf config push ostack-ceph1 ostack-ceph2 ostack-ceph3\n<\/code><\/pre>\n
As previously done we need to copy the new key we created to all nodes:<\/p>\n
igorc@ostack-ceph1:~\/ceph-cluster$ scp \/etc\/ceph\/ceph.client.images.keyring ostack-ceph2:~ && ssh ostack-ceph2 sudo cp ceph.client.images.keyring \/etc\/ceph\/\nigorc@ostack-ceph1:~\/ceph-cluster$ scp \/etc\/ceph\/ceph.client.images.keyring ostack-ceph3:~ && ssh ostack-ceph3 sudo cp ceph.client.images.keyring \/etc\/ceph\/\n<\/code><\/pre>\n
UPDATE: 25\/08\/2015<\/strong><\/p>\n
The ceph fs new<\/code> command was introduced in Ceph 0.84. Prior to this release, no manual steps are required to create a file system, and pools named data<\/code> and metadata<\/code> exist by default. The Ceph command line now includes commands for creating and removing file systems, but at present only one file system may exist at a time.<\/p>\n
igorc@ostack-ceph1:~\/ceph-cluster$ ceph osd pool create cephfs_metadata 64\nigorc@ostack-ceph1:~\/ceph-cluster$ ceph osd pool create cephfs_data 64\nigorc@ostack-ceph1:~\/ceph-cluster$ ceph fs new cephfs cephfs_metadata cephfs_data\nnew fs with metadata pool 2 and data pool 1\n\nigorc@ostack-ceph1:~\/ceph-cluster$ ceph osd lspools\n0 rbd,1 cephfs_data,2 cephfs_metadata,3 datastore,4 images,\n\nigorc@ostack-ceph1:~\/ceph-cluster$ ceph fs ls\nname: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]\n\nigorc@ostack-ceph1:~\/ceph-cluster$ ceph mds stat\ne5: 1\/1\/1 up {0=ostack-ceph1=up:active}\n\nigorc@ostack-ceph1:~\/ceph-cluster$ ceph status\n    cluster 5f1b2264-ab6d-43c3-af6c-3062e707a623\n     health HEALTH_WARN\n            too many PGs per OSD (320 > max 300)\n     monmap e1: 3 mons at {ostack-ceph1=192.168.122.211:6789\/0,ostack-ceph2=192.168.122.212:6789\/0,ostack-ceph3=192.168.122.213:6789\/0}\n            election epoch 4, quorum 0,1,2 ostack-ceph1,ostack-ceph2,ostack-ceph3\n     mdsmap e5: 1\/1\/1 up {0=ostack-ceph1=up:active}\n     osdmap e25: 3 osds: 3 up, 3 in\n      pgmap v114: 320 pgs, 5 pools, 1962 bytes data, 20 objects\n            107 MB used, 22899 MB \/ 23006 MB avail\n                 320 active+clean\n\nigorc@ostack-ceph1:~\/ceph-cluster$ ceph osd tree\nID WEIGHT  TYPE NAME             UP\/DOWN REWEIGHT PRIMARY-AFFINITY \n-1 0.02998 root default                                            \n-2 0.00999     host ostack-ceph1                                   \n 0 0.00999         osd.0              up  1.00000          1.00000 \n-3 0.00999     host ostack-ceph2                                   \n 1 0.00999         osd.1              up  1.00000          1.00000 \n-4 0.00999     host ostack-ceph3                                   \n 2 0.00999         osd.2              up  1.00000          1.00000\n<\/code><\/pre>\n
Remove Ceph<\/h2>\n
To completely remove Ceph follow the procedure below:<\/p>\n
igorc@ostack-ceph1:~\/ceph-cluster$ sudo service ceph-all stop\nigorc@ostack-ceph1:~\/ceph-cluster$ ceph-deploy purge ostack-ceph1 ostack-ceph2 ostack-ceph3\nigorc@ostack-ceph1:~\/ceph-cluster$ ceph-deploy purgedata ostack-ceph1 ostack-ceph2 ostack-ceph3\nigorc@ostack-ceph1:~\/ceph-cluster$ ceph-deploy forgetkeys\n<\/code><\/pre>\n
Calamari Ceph GUI<\/h2>\n
There is a great guide at How to install Calamari for Ceph Cluster on Ubuntu 14.04<\/a>. With some tweaks I was able to setup Calamari on one of my VM’s and integrate with the Ceph cluster created above.<\/p>\n\n\t\t\t