{"id":210,"date":"2015-10-13T16:48:14","date_gmt":"2015-10-13T05:48:14","guid":{"rendered":"https:\/\/icicimov.com\/blog\/?p=210"},"modified":"2017-01-02T16:49:27","modified_gmt":"2017-01-02T05:49:27","slug":"haproxy-dynamically-adjust-server-weight-using-external-agent","status":"publish","type":"post","link":"https:\/\/icicimov.com\/blog\/?p=210","title":{"rendered":"HAProxy dynamically adjust server weight using external agent"},"content":{"rendered":"

Trying to utilize HAProxy-1.5\/1.6 agent-check<\/code> feature, see HAProxy documentation<\/a>, I wrote this small script to check Tomcat system load and return back some values that HAP can use to dynamically adjust the server weight in the backend.<\/p>\n

This will run as xinetd service on the Tomcats, for example made available to HAP on port 9707 (some randomly chosen free port).<\/p>\n

Explanation how is this going to work. Each server starts with weight of 100. This health check will run every 5 minutes lets say (the primary runs every 10 seconds) and the agent will return “up” plus weight value or “down” upon heath check. The weight percentage returned is calculated based on the system load in the last 5 minutes (can be 1 or 15 min. too), ie if load is below 90% it will return weight of 100, between 90% and 100% it will reduce the weight to 50% (meaning HAP will send only 25% of the connections to this server in case of 2 servers) and if 100% it will mark this server down.<\/p>\n

This additional, lets call it secondary, heath check will work in conjunction with the already existing one, which only checks if the service is up or down, and provide flexibility in terms of backend load.<\/p>\n

Setup<\/h2>\n

On the backends (Tomcat app servers)<\/h3>\n

Create agent-check script \/usr\/local\/bin\/haproxy-agent-check<\/code>:<\/p>\n

#!\/bin\/bash\nLMAX=90\n\nload=$(uptime | grep -E -o 'load average[s:][: ].*' | sed 's\/,\/\/g' | cut -d' ' -f3-5)\ncpus=$(grep processor \/proc\/cpuinfo | wc -l)\n\nwhile read -r l1 l5 l15; do {\n    l5util=$(echo \"$l5\/$cpus*100\" | bc -l | cut -d\".\" -f1);\n    [[ $l5util -lt $LMAX ]] && echo \"up 100%\" && exit 0;\n    [[ $l5util -gt $LMAX ]] && [[ $l5util -lt 100 ]] && echo \"up 50%\" && exit 0;\n    echo \"down#CPU overload\";\n}; done < <(echo $load)\n\nexit 0\n<\/code><\/pre>\n
Set it to be executable:<\/p>\n
root@app21:~# chmod +x \/usr\/local\/bin\/haproxy-agent-check\n<\/code><\/pre>\n
Next we set it as xinetd service, first install xinetd:<\/p>\n
root@app21:~# aptitude install -y xinetd\n<\/code><\/pre>\n
Then add our service to the system services in \/etc\/services<\/code> file:<\/p>\n
...\nhaproxy-agent-check 9707\/tcp                # haproxy-agent-check\n...\n<\/code><\/pre>\n
Create the xinetd daemon file \/etc\/xinetd.d\/haproxy-agent-check<\/code>:<\/p>\n
# default: on\n# description: haproxy-agent-check\nservice haproxy-agent-check\n{\n        disable         = no\n        flags           = REUSE\n        socket_type     = stream\n        port            = 9707\n        wait            = no\n        user            = nobody\n        server          = \/usr\/local\/bin\/haproxy-agent-check\n        log_on_failure  += USERID\n        only_from       = 172.31.17.11 172.31.11.11 127.0.0.1\n        per_source      = UNLIMITED\n}\n<\/code><\/pre>\n
To allow access from specific subnets instead of hosts we can use the following format:<\/p>\n
only_from       = 10.22.0.0 10.22.1.0 10.22.2.0 127.0.0.1\n<\/code><\/pre>\n
Make it executable, restart xinetd and test:<\/p>\n
root@app21-hap:~# chmod +x \/etc\/xinetd.d\/haproxy-agent-check\nroot@app21-:~# service xinetd restart\nroot@app21:~# telnet 127.0.0.1 9707\nTrying 127.0.0.1...\nConnected to 127.0.0.1.\nEscape character is '^]'.\nup 100%\nConnection closed by foreign host.\n<\/code><\/pre>\n
All looks good so we can setup the HAP’s now.<\/p>\n
On the HAProxy LB’s<\/h1>\n
Open the TCP port 9707 in the APP servers firewall and Test the agent-check connectivity from HAP serers to the app servers:<\/p>\n
root@lb1:~# telnet ip-172-31-17-41 9707\nTrying 172.31.17.41...\nConnected to ip-172-31-17-41.ap-southeast-2.compute.internal.\nEscape character is '^]'.\nup 100%\nConnection closed by foreign host.\n\nroot@lb1:~# telnet ip-172-31-11-41 9707\nTrying 172.31.11.41...\nConnected to ip-172-31-11-41.ap-southeast-2.compute.internal.\nEscape character is '^]'.\nup 100%\nConnection closed by foreign host.\nroot@lb1:~#\n<\/code><\/pre>\n
Add agent-check<\/code>, agent-port<\/code> and agent-inter<\/code> parameters to the backend servers in the \/etc\/haproxy\/haproxy.cfg<\/code> config file:<\/p>\n
...\nlisten https-in\n...\n    default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 200 maxqueue 250 weight 100 error-limit 10 on-error mark-down on-marked-down shutdown-sessions agent-port 9707 agent-inter 30s\n    server ip-172-31-17-41  ip-172-31-17-41:8080  check agent-check observe layer7\n    server ip-172-31-11-41  ip-172-31-11-41:8080  check agent-check observe layer7\n    server localhost 127.0.0.1:8080 maxconn 500 backup weight 1\n<\/code><\/pre>\n
And reload HAP service. Check the health status:<\/p>\n
root@lb1:~# echo \"show stat\" | socat stdio unix-connect:\/run\/haproxy\/admin.sock | cut -d ',' -f1,2,18,19 | grep https\nhttps-in,ip-172-31-17-41,UP,100\nhttps-in,ip-172-31-11-41,UP,100\nhttps-in,localhost,no check,1\nhttps-in,BACKEND,UP,200\n<\/code><\/pre>\n
We can see the weight of both app servers is set to 100 atm but this should change if\/when they come under high load for 5 minutes.<\/p>\n","protected":false},"excerpt":{"rendered":"
Trying to utilize HAProxy-1.5\/1.6 agent-check feature, see HAProxy documentation, I wrote this small script to check Tomcat system load and return back some values that HAP can use to dynamically adjust the server weight in the backend. This will run…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10],"tags":[],"class_list":["post-210","post","type-post","status-publish","format-standard","hentry","category-server"],"_links":{"self":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/210","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=210"}],"version-history":[{"count":1,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/210\/revisions"}],"predecessor-version":[{"id":211,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/210\/revisions\/211"}],"wp:attachment":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=210"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=210"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=210"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}