{"id":188,"date":"2015-01-20T16:05:08","date_gmt":"2015-01-20T05:05:08","guid":{"rendered":"https:\/\/icicimov.com\/blog\/?p=188"},"modified":"2017-01-02T16:08:04","modified_gmt":"2017-01-02T05:08:04","slug":"managing-system-resources-with-cgroups-and-ansible","status":"publish","type":"post","link":"https:\/\/icicimov.com\/blog\/?p=188","title":{"rendered":"Managing system resources with Cgroups and Ansible"},"content":{"rendered":"

Sometimes we need to limit particular resource usage for some process, utility or group of processes in order to prioritize or limit their usage. One way to achieve this in the modern Linux kernel is via Cgroups<\/code>. They provide kernel feature that limits, accounts for and isolates the resource usage (CPU, memory, disk I\/O, network, etc.) of a collection of processes.<\/p>\n

Implementation<\/h2>\n

The following example shows using cgroups to limit the CPU usage and number of CPU cores for specific utility, in this case the html-to-pdf conversion tool wkhtmltopdf<\/code>. First we install the needed packages:<\/p>\n

$ sudo aptitude install cgroup-bin cgmanager-utils<\/p>\n

On Ubuntu\/Debian the cgroups file system mounts under \/sys\/fs\/cgroups<\/code> where all available sub systems get created. To find the capabilities and available cgroup subsystems:<\/p>\n

$ cat \/proc\/cgroups\n#subsys_name    hierarchy    num_cgroups    enabled\ncpuset    3    2    1\ncpu    4    2    1\ncpuacct    5    1    1\nmemory    6    1    1\ndevices    7    1    1\nfreezer    8    1    1\nblkio    9    1    1\nperf_event    10    1    1\nhugetlb    11    1    1\n<\/code><\/pre>\n
Now we can set the cgroup and set it’s limits:<\/p>\n
$ sudo cgcreate -g cpu,cpuset:\/group1\n$ sudo cgset -r cpuset.cpus='0,2,4,6' group1\n$ sudo cgset -r cpu.shares='512' group1\n<\/code><\/pre>\n
This will create the group1 under cpu and cpuset subsystems of cgroups. We can check the set values:<\/p>\n
$ cat \/sys\/fs\/cgroup\/cpuset\/group1\/cpuset.cpus\n0,2,4,6\n\n$ cat \/sys\/fs\/cgroup\/cpu\/group1\/cpu.shares\n512\n<\/code><\/pre>\n
The above created cgroup limits its tasks to only run on cpu cores 0,2,4 and 6 and use max of 50% cpu cycles on those cores. The cpu capacity is represented with shares and each core has 1024 shares in total, which means setting the shares to 512 sets around 50% of the cpu cycles for the tasks. The most important part though is that limitations are not going to be applied UNTIL there is other processes running on the same cores competing for 100% of the cpu usage. Meaning the tasks have in total 100% of cpu available on each core and they can use it all if needed. Also the cpu utilization of 50% is a relative value and doesn’t mean that the tasks are given exactly 50% of cpu on each of the 4 cores but rather an overall utilization for the allocated resources. In case of contention they might get 20% on core 2 but 10% on cores 0,4 and 6 for example which will give it a total of 50%.<\/p>\n
Now, if we want to limit a process in terms of cpu utilization we can add it to the group1<\/code> we created. We have several options for this:<\/p>\n
    \n
  • We can add already running process to the group. Example:\n
    # cgclassify -g cpu,cpuset:\/group1 $PID\n<\/code><\/pre>\n<\/li>\n
  • We can start a process or execute command utility bound to the group. Examples:\n
    # cgexec -g cpu,cpuset:\/group1 \/usr\/local\/bin\/wkhtmltopdf file.html\n# cgexec -g cpu,cpuset:\/group1 httpd\n<\/code><\/pre>\n<\/li>\n
  • We can adjust the default startup parameters of a process so it automatically starts in the group. Example in \/etc\/sysconfig\/httpd<\/code> or \/etc\/default\/apache2<\/code>:\n
    ...\nCGROUP_DAEMON=\"cpu,cpuset:\/group1\"\n<\/code><\/pre>\n<\/li>\n
  • We can use cgrep daemon cgrepd<\/code> which assigns tasks to particular groups based on the settings of the \/etc\/cgrules.conf<\/code> file on run-time. Example:\n
    # echo 'tomcat7:wkhtmltopdf    cpu,cpuset    group1' >  \/etc\/cgrules.conf\n\n# cgrulesengd -d -v -f \/var\/log\/cgrulesengd.log &\n\n# cat \/var\/log\/cgrulesengd.log\nCGroup Rules Engine Daemon log started\nCurrent time: Thu Jan 15 16:04:20 2015\n\nOpened log file: \/var\/log\/cgrulesengd.log, log facility: 0, log level: 7\nProceeding with PID 28569\nRule: tomcat7:wkhtmltopdf\nUID: 500\nGID: N\/A\nDEST: group1\nCONTROLLERS:\n  cpu\n  cpuset\n\nStarted the CGroup Rules Engine Daemon.\n<\/code><\/pre>\n<\/li>\n<\/ul>\n
    Now to automate all this on start-up I’ve created systemv init script for the cgred<\/code> daemon for Ubuntu by modifying the one provided in the package for RedHat \/etc\/init.d\/cgred<\/code>:<\/p>\n
    #!\/bin\/bash\n#\n# Start\/Stop the CGroups Rules Engine Daemon\n#\n# Copyright Red Hat Inc. 2008\n#\n# Authors:    Steve Olivieri <sjo @redhat.com>\n# This program is free software; you can redistribute it and\/or modify it\n# under the terms of version 2.1 of the GNU Lesser General Public License\n# as published by the Free Software Foundation.\n#\n# This program is distributed in the hope that it would be useful, but\n# WITHOUT ANY WARRANTY; without even the implied warranty of\n# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.\n#\n# cgred        CGroups Rules Engine Daemon\n# chkconfig:    - 14 86\n# description:    This is a daemon for automatically classifying processes \\\n#        into cgroups based on UID\/GID.\n#\n# processname: cgrulesengd\n# pidfile: \/var\/run\/cgred.pid\n#\n### BEGIN INIT INFO\n# Provides:        cgrulesengd\n# Required-Start:    $local_fs $syslog\n# Required-Stop:    $local_fs $syslog\n# Default-Start:        2 3 4 5\n# Default-Stop:         0 1 6\n# Short-Description:    start and stop the cgroups rules engine daemon\n# Description:        CGroup Rules Engine is a tool for automatically using \\\n#            cgroups to classify processes\n### END INIT INFO\n\nprefix=\/usr;exec_prefix=${prefix};sbindir=${exec_prefix}\/sbin\nCGRED_BIN=$sbindir\/cgrulesengd\nCGRED_CONF=\/etc\/cgrules.conf\n\n# Sanity checks\n[ -x $CGRED_BIN ] || exit 1\n\n# Source function library & LSB routines\n#. \/etc\/init.d\/functions\n. \/lib\/init\/vars.sh\n. \/lib\/lsb\/init-functions\n\n# Read in configuration options.\nif [ -f \"\/etc\/default\/cgred\" ] ; then\n    . \/etc\/default\/cgred\n    OPTIONS=\"$NODAEMON $LOG\"\n    if [ -n \"$LOG_FILE\" ]; then\n        OPTIONS=\"$OPTIONS --logfile=$LOG_FILE\"\n    fi\n    if [ -n \"$SOCKET_USER\" ]; then\n        OPTIONS=\"$OPTIONS -u $SOCKET_USER\"\n    fi\n    if [ -n \"$SOCKET_GROUP\" ]; then\n        OPTIONS=\"$OPTIONS -g $SOCKET_GROUP\"\n    fi\nelse\n    OPTIONS=\"\"\nfi\n\n# For convenience\nprocessname=cgrulesengd\nservicename=cgred\nlockfile=\"\/var\/lock\/$servicename\"\npidfile=\/var\/run\/cgred.pid\n\nstart()\n{\n    echo -n $\"Starting CGroup Rules Engine Daemon: \"\n    if [ -f \"$lockfile\" ]; then\n        log_failure_msg \"$servicename is already running with PID `cat ${pidfile}`\"\n        return 0\n    fi\n    if [ ! -s $CGRED_CONF ]; then\n        log_failure_msg \"not configured\"\n        return 6\n    fi\n    if ! grep \"^cgroup\" \/proc\/mounts &>\/dev\/null; then\n        echo\n        log_failure_msg $\"Cannot find cgroups, is cgconfig service running?\"\n        return 1\n    fi\n    start-stop-daemon --start -b --pidfile \"$pidfile\" -x $CGRED_BIN -- $OPTIONS\n    retval=$?\n    if [ $retval -ne 0 ]; then\n        return 7\n    fi\n    touch \"$lockfile\"\n    if [ $? -ne 0 ]; then\n        return 1\n    fi\n    sleep 2\n    echo \"`pidof $processname`\" > $pidfile\n    return 0\n}\n\nstop()\n{\n    echo -n $\"Stopping CGroup Rules Engine Daemon...\"\n    if [ ! -f $pidfile ]; then\n        log_success_msg\n        return 0\n    fi\n    #killproc -p $pidfile -TERM \"$processname\"\n    start-stop-daemon --stop --pidfile \"$pidfile\" --retry=TERM\/20\/KILL\/5\n    #killall -TERM $processname\n    retval=$?\n    echo\n    if [ $retval -ne 0 ]; then\n        return 1\n    fi\n    rm -f \"$lockfile\" \"$pidfile\"\n    return 0\n}\n\nstatus () {\n    status_of_proc -p $pidfile $servicename \"$processname\"\n}\n\nRETVAL=0\n\n# See how we are called\ncase \"$1\" in\n    start)\n        start\n        RETVAL=$?\n        ;;\n    stop)\n        stop\n        RETVAL=$?\n        ;;\n    status)\n        status && exit 0 || exit $?\n        ;;\n    restart)\n        stop\n        start\n        RETVAL=$?\n        ;;\n    *)\n        echo $\"Usage: $0 {start|stop|restart}\"\n        RETVAL=2\n        ;;\nesac\n\nexit $RETVAL\n<\/code><\/pre>\n
    make it executable and set it for autostart on proper runlevels:<\/p>\n
    $ sudo chmod +x \/etc\/init.d\/cgred\n$ sudo update-rc.d defaults 99 20\n<\/code><\/pre>\n
    Then we can grab thedefault cgred config that comes with the documentation:<\/p>\n
    $ sudo cp \/usr\/share\/doc\/cgroup-bin\/examples\/cgred.conf \/etc\/default\/cgred\n<\/code><\/pre>\n
    and modify it slightly \/etc\/default\/cgred<\/code>:<\/p>\n
    CONFIG_FILE=\"\/etc\/cgrules.conf\"\nLOG_FILE=\"\/var\/log\/cgrulesengd.log\"\nNODAEMON=\"\"\n#NODAEMON=\"--nodaemon\"\nSOCKET_USER=\"\"\n#SOCKET_GROUP=\"cgred\"\nSOCKET_GROUP=\"\"\n#LOG=\"\"\n#LOG=\"--nolog\"\nLOG=\"-v\"\n<\/code><\/pre>\n
    Then we can use the standard service command to start\/stop the daemon:<\/p>\n
    $ sudo service cgred [start|stop|status|restart]\n<\/code><\/pre>\n
    We also need to parse the rules file and dynamically create our cgroup on start-up. We create the following file for this purpose \/etc\/cgconfig.conf<\/code>:<\/p>\n
    group group1 {\n    cpuset {\n        cpuset.memory_spread_slab=\"0\";\n        cpuset.memory_spread_page=\"0\";\n        cpuset.memory_migrate=\"0\";\n        cpuset.sched_relax_domain_level=\"-1\";\n        cpuset.sched_load_balance=\"1\";\n        cpuset.mem_hardwall=\"0\";\n        cpuset.mem_exclusive=\"0\";\n        cpuset.cpu_exclusive=\"0\";\n        cpuset.mems=\"0\";\n        cpuset.cpus=\"0,2,4,6\";\n    }\n}\ngroup group1 {\n    cpu {\n        cpu.cfs_period_us=\"100000\";\n        cpu.cfs_quota_us=\"-1\";\n        cpu.shares=\"512\";\n    }\n}\n<\/code><\/pre>\n
    and run the parsing command on start-up which we add to rc.local file \/etc\/rc.local<\/code>:<\/p>\n
    ...\ncgconfigparser -l \/etc\/cgconfig.conf\n\nexit 0\n<\/code><\/pre>\n
    That’s it, now our cgroup, with cpu and cpuset subsystems attached to it, will get automatically created on start-up and the cgred daemon started to automatically move the wkhtmltopdf processes to the cgroup, but only if there are other processes competing for 100% CPU usage on the same core that the wkhtmltopdf process is also running on.<\/p>\n
    Automating with Ansible<\/h2>\n
    Now we want this done on each server we have the tool running. The following playbook will do the same as above but in dynamic fashion, finding out the number of CPU’s on the server and adjusting the numbers for the cpu cores and cpu shares in the \/etc\/cgconfig.conf<\/code> represented by an Ansible<\/code> template.<\/p>\n
    ---\n#\n# Cgroups setup\n#\n- ec2_facts:\n\n- set_fact:\n   cpu_cores: |\n    {% raw %}{%- for core in range((ansible_processor_vcpus\/2)|round(0,'ceil')|int) -%}\n      {{ core }}{% if not loop.last %},{% endif %}\n    {%- endfor -%}{% endraw %}\n\n- name: Update apt\n  apt: update_cache=yes\n  when: ansible_os_family == \"Debian\"\n  register: result\n  until: result|success\n  retries: 10\n\n- name: install cgroups\n  apt: pkg={{ item }} state=present\n  with_items: [ 'cgroup-lite', 'cgroup-bin', 'numactl' ]\n\n- copy: src=cgsnapshot_blacklist.conf dest=\/etc\/cgsnapshot_blacklist.conf owner=root group=root mode=0644\n\n- name: configure our cpu cgroup\n  template: src=cgconfig.conf.j2 dest=\/etc\/cgconfig.conf owner=root group=root mode=0644\n\n- name: cgrules config file\n  copy: src=cgrules.conf dest=\/etc\/cgrules.conf owner=root group=root mode=0644\n\n- name: create the cgroup\n  command: cgconfigparser -l \/etc\/cgconfig.conf\n\n- name: set cgconfigparser for autostart\n  lineinfile:\n    dest=\/etc\/rc.local\n    line=\"cgconfigparser -l \/etc\/cgconfig.conf\"\n    insertbefore='exit 0'\n\n- name: setup the cgred daemon\n  copy: src=cgred.init dest=\/etc\/init.d\/cgred owner=root group=root mode=0755\n\n- name: set the cgred defaults config\n  copy: src=cgred.conf dest=\/etc\/default\/cgred owner=root group=root mode=0644\n  notify: restart cgred\n\n- name: autostart cgred\n  command: update-rc.d cgred defaults 99 15\n<\/code><\/pre>\n
    the template cgconfig.conf.j2<\/code> looks like:<\/p>\n
    group group1 {\n    cpuset {\n        cpuset.memory_spread_slab=\"0\";\n        cpuset.memory_spread_page=\"0\";\n        cpuset.memory_migrate=\"0\";\n        cpuset.sched_relax_domain_level=\"-1\";\n        cpuset.sched_load_balance=\"1\";\n        cpuset.mem_hardwall=\"0\";\n        cpuset.mem_exclusive=\"0\";\n        cpuset.cpu_exclusive=\"0\";\n        cpuset.mems=\"0\";\n        cpuset.cpus=\"{% raw %}{{ cpu_cores }}{% endraw %}\";\n    }\n}\n\ngroup group1 {\n    cpu {\n        cpu.cfs_period_us=\"100000\";\n        cpu.cfs_quota_us=\"-1\";\n        cpu.shares=\"{% raw %}{{ cpu_shares }}{% endraw %}\";\n    }\n}\n<\/code><\/pre>\n
    The rest of the files are static. We organize all this nicely into a role that we call each time a new server is launched in EC2.<\/p>\n","protected":false},"excerpt":{"rendered":"
    Sometimes we need to limit particular resource usage for some process, utility or group of processes in order to prioritize or limit their usage. One way to achieve this in the modern Linux kernel is via Cgroups. They provide kernel…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11,12,10],"tags":[],"class_list":["post-188","post","type-post","status-publish","format-standard","hentry","category-aws","category-devops","category-server"],"_links":{"self":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/188","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=188"}],"version-history":[{"count":1,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/188\/revisions"}],"predecessor-version":[{"id":189,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/188\/revisions\/189"}],"wp:attachment":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=188"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=188"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=188"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}