Table of contents<\/h2>\n\t- \n\t\tInstallation and setup<\/a>\n\t\t
\n\t\t\t- \n\t\t\t\tCentral Server<\/a>\n\t\t\t\t
\n\t\t\t\t\t- \n\t\t\t\t\t\tKernel Setup<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t
- \n\t\t\t\t\t\tInstall Redis Broker Server<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t
- \n\t\t\t\t\t\tInstall and setup Elastic Search<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t
- \n\t\t\t\t\t\tLogstash Install and Config<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t
- \n\t\t\t\t\t\tWeb interface<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t<\/ul>\n\t\t\t
- \n\t\t\t\tClients<\/a>\n\t\t\t<\/li>\n\t\t\t
- \n\t\t\t\tSecurity<\/a>\n\t\t\t<\/li>\n<\/ul>\n<\/ul>\n<\/div>\n
\nLogstash is a tool for managing events and logs. It is very useful for collecting, parsing and storing logs for later use like for example searching. It comes with a web interface for searching through the logs. The picture bellow shows a typical centralized logstash environment. It consists of logstash clients installed on the servers running applications we want to collect logs for and centralized logstash server that does the indexing and storing of the logs. These remote logstash instances are referred as shippers<\/code> and the central one as indexer<\/code> or reader<\/code>.<\/p>\n![\"\"](\"https:\/\/icicimov.com\/blog\/wp-content\/uploads\/2017\/01\/Logstash_central_log_server_architecture.png\")
<\/a>
\nPicture1:<\/strong> Logstash environment<\/em><\/p>\nThe broker of choice is Redis and the storage for indexing is Elastic Search, which comes natural since Logstash is part of Elastic Search project. The web interface will be served via Nginx which is light enough for this purpose.<\/p>\n
Installation and setup<\/span><\/h1>\nCentral Server<\/span><\/h2>\nEverything is installed on a single node. We start with some kernel tuning.<\/p>\n
Kernel Setup<\/span><\/h3>\nThis will provide some system, memory optimization and network stack optimization and tuning for our EC2 instance type. At the end of the \/etc\/sysctl.conf<\/code> we add:<\/p>\n### KERNEL ###\n\n# Core dumps\nkernel.core_uses_pid = 1\nkernel.core_pattern = \/mnt\/core-%e-%s-%u-%g-%p-%t\nfs.suid_dumpable = 2\n\n# Turn on execshild\n#kernel.exec-shield = 1\nkernel.randomize_va_space = 1\n# Reboot after 10sec. on kernel panic\nkernel.panic = 10\n\n### IMPROVE SYSTEM MEMORY MANAGEMENT ###\n\n# Increase size of file handles and inode cache\nfs.file-max = 2097152\n\n# Insure we always have enough memory\nvm.min_free_kbytes = 8192\n\n# Do less swapping\nvm.swappiness = 10\nvm.dirty_ratio = 60\nvm.dirty_background_ratio = 2\n\n### GENERAL NETWORK SECURITY OPTIONS ###\n\n# Avoid a smurf attack\nnet.ipv4.icmp_echo_ignore_broadcasts = 1\n\n# Turn on protection for bad icmp error messages\nnet.ipv4.icmp_ignore_bogus_error_responses = 1\n\n# Turn on syncookies for SYN flood attack protection\nnet.ipv4.tcp_syncookies = 1\nnet.ipv4.tcp_max_syn_backlog = 1024\n\n# Turn on timestamping\nnet.ipv4.tcp_timestamps = 1\n\n# Turn on and log spoofed, source routed, and redirect packets\nnet.ipv4.conf.all.log_martians = 1\nnet.ipv4.conf.default.log_martians = 1\n\n# No source routed packets here\nnet.ipv4.conf.all.accept_source_route = 0\nnet.ipv4.conf.default.accept_source_route = 0\n\n# Turn on reverse path filtering\nnet.ipv4.conf.all.rp_filter = 1\nnet.ipv4.conf.default.rp_filter = 1\n\n# Make sure no one can alter the routing tables\nnet.ipv4.conf.all.accept_redirects = 0\nnet.ipv4.conf.default.accept_redirects = 0\nnet.ipv4.conf.all.secure_redirects = 0\nnet.ipv4.conf.default.secure_redirects = 0\n\n# Don't act as a router\nnet.ipv4.ip_forward = 0\nnet.ipv4.conf.all.send_redirects = 0\nnet.ipv4.conf.default.send_redirects = 0\n\n# Number of times SYNACKs for passive TCP connection.\nnet.ipv4.tcp_synack_retries = 2\n\n# Allowed local port range\nnet.ipv4.ip_local_port_range = 2000 65535\n\n# Protect Against TCP Time-Wait\nnet.ipv4.tcp_rfc1337 = 1\n\n# Decrease the time default value for tcp_fin_timeout connection\nnet.ipv4.tcp_fin_timeout = 15\n\n# Decrease the time default value for connections to keep alive\nnet.ipv4.tcp_keepalive_time = 300\nnet.ipv4.tcp_keepalive_probes = 5\nnet.ipv4.tcp_keepalive_intvl = 15\n# This means that the keepalive process waits 300 seconds for socket \n# activity before sending the first keepalive probe, and then resend\n# it every 15 seconds. If no ACK response is received for 5 consecutive \n# times (75s in this case), the connection is marked as broken.\n\n### TUNING NETWORK PERFORMANCE ###\n\n# Disable IPv6\nnet.ipv6.conf.all.disable_ipv6 = 1\nnet.ipv6.conf.default.disable_ipv6 = 1\nnet.ipv6.conf.lo.disable_ipv6 = 1\n\n# Default Socket Receive Buffer\nnet.core.rmem_default = 31457280\n\n# Maximum Socket Receive Buffer\nnet.core.rmem_max = 12582912\n\n# Default Socket Send Buffer\nnet.core.wmem_default = 31457280\n\n# Maximum Socket Send Buffer\nnet.core.wmem_max = 12582912\n\n# Increase number of incoming connections\nnet.core.somaxconn = 5000\n\n# Increase number of incoming connections backlog\nnet.core.netdev_max_backlog = 65536\n\n# Enable TCP window scaling\nnet.ipv4.tcp_window_scaling = 1\n\n# Increase the maximum amount of option memory buffers\nnet.core.optmem_max = 25165824\n\n# Increase the maximum total buffer-space allocatable\n# This is measured in units of pages (4096 bytes)\nnet.ipv4.tcp_mem = 65536 131072 262144\nnet.ipv4.udp_mem = 65536 131072 262144\n\n# Increase the read-buffer space allocatable\nnet.ipv4.tcp_rmem = 8192 87380 16777216\nnet.ipv4.udp_rmem_min = 16384\n\n# Increase the write-buffer-space allocatable\nnet.ipv4.tcp_wmem = 8192 65536 16777216\nnet.ipv4.udp_wmem_min = 16384\n\n# Increase the tcp-time-wait buckets pool size to prevent simple DOS attacks\nnet.ipv4.tcp_max_tw_buckets = 1440000\n\n# TIME_WAIT socket policy\n# Note: if both enabled then disable\n# net.ipv4.tcp_timestamps for servers \n# behind NAT to prevent dropped incoming connections\nnet.ipv4.tcp_tw_recycle = 0\nnet.ipv4.tcp_tw_reuse = 1\n\n# Enable TCP MTU probing (in case of Jumbo Frames enabled)\n#net.ipv4.tcp_mtu_probing = 1\n\n# Speedup retrans (Google recommended)\nnet.ipv4.tcp_slow_start_after_idle = 0\nnet.ipv4.tcp_early_retrans = 1\n\n# Conntrack\n# 288bytes x 131072 = 37748736 (~38MB) max memory usage\nnet.netfilter.nf_conntrack_max = 131072\nnet.netfilter.nf_conntrack_tcp_loose = 1\n\n# NOTE: Enable this if EC2 instance support it\n# -- 10gbe tuning from Intel ixgb driver README -- #\n# turn off selective ACK and timestamps\n#net.ipv4.tcp_sack = 0\n#net.ipv4.tcp_timestamps = 0\n<\/code><\/pre>\nand run:<\/p>\n
sysctl -p\n<\/code><\/pre>\nto activate the changes.<\/p>\n
Install Redis Broker Server<\/span><\/h3>\nWe need broker services to buffer all the client updates while the central Logstash service is offline, in case of issues with Logstash or update etc.<\/p>\n
root@myserver:~# aptitude install redis-server\n<\/code><\/pre>\nIn case all our remote clients were to connect directly to the broker we need to change the listening address in the config file and replace:<\/p>\n
bind 127.0.0.1\n<\/code><\/pre>\nwith:<\/p>\n
bind 0.0.0.0\n<\/code><\/pre>\nin \/etc\/redis\/redis.conf<\/code>. We are going to do this temporarily for testing only since we are going to use stunnel<\/code> to secure Redis communication.<\/p>\nFor security reasons we want only the authenticated clients to communicate with the server so we set password in the config file:<\/p>\n
requirepass <my-redis-password>\n<\/code><\/pre>\nWe also need to open the tcp port 6379<\/code> in the firewall (EC2 instance SecurityGroup) in case we enabled remote Redis access.<\/p>\nEnable the overcommit kernel memory option so every malloc()<\/code> operation will succeed:<\/p>\nroot@myserver:~# sysctl vm.overcommit_memory=1\n<\/code><\/pre>\nand add it to \/etc\/sysctl.conf<\/code> as well so it persists over bootups. Then start the service:<\/p>\nroot@myserver:~# service redis start\n<\/code><\/pre>\nInstall and setup Elastic Search<\/span><\/h3>\nWe need at least version 0.90.7.<\/p>\n
root@myserver:~# wget https:\/\/download.elasticsearch.org\/elasticsearch\/elasticsearch\/elasticsearch-0.90.9.deb\nroot@myserver:~# dpkg -i elasticsearch-0.90.9.deb\n<\/code><\/pre>\nThen edit the \/etc\/elasticsearch\/elasticsearch.yml<\/code> config file and set some options that suit our needs and VM hardware:<\/p>\ncluster.name: logstash\nnode.name: \"ec2-logger-server\"\npath.data: \/mnt\/elasticsearch,\/mnt2\/elasticsearch\npath.logs: \/mnt2\/log\/elasticsearch\nbootstrap.mlockall: true\nnetwork.host: \"127.0.0.1\"\n\n## Threadpool Settings ##\n# Search pool\nthreadpool.search.type: fixed\nthreadpool.search.size: 20\nthreadpool.search.queue_size: 100\n\n# Bulk pool\nthreadpool.bulk.type: fixed\nthreadpool.bulk.size: 60\nthreadpool.bulk.queue_size: 300\n\n# Index pool\nthreadpool.index.type: fixed\nthreadpool.index.size: 20\nthreadpool.index.queue_size: 100\n\n# Indices settings\nindices.memory.index_buffer_size: 30%\nindices.memory.min_shard_index_buffer_size: 12mb\nindices.memory.min_index_buffer_size: 96mb\n\n# Cache Sizes\nindices.fielddata.cache.size: 15%\nindices.fielddata.cache.expire: 6h\nindices.cache.filter.size: 15%\nindices.cache.filter.expire: 6h\n\n# Indexing Settings for Writes\nindex.refresh_interval: 30s\nindex.translog.flush_threshold_ops: 50000\n<\/code><\/pre>\nWe enable bootstrap.mlockall<\/code> to try to lock the process address space so it won’t be swapped (recommended by ES configuration page). This also requires setting the following options in the \/etc\/default\/elasticsearch<\/code> file:<\/p>\nES_HEAP_SIZE=2g\nMAX_LOCKED_MEMORY=unlimited\n<\/code><\/pre>\nWe have also tuned the search, bulk and index pools giving the bulk pool 3 times more threads since the server will be mostly writing. We also tune the index sizes and cache flushes to reduce the load on the server under heavy writes.<\/p>\n
Later on we had to add an additional drive via path.data:\/mnt\/elasticsearch,\/mnt2\/elasticsearch<\/code> option to expand the index disk storage. This option also adds on speed since the combination of the two drives acts like a RAID0 for files, of course compared to chunks as in real RAID. I have also moved the logs to \/mnt2\/elasticsearch<\/code> drive since it is a faster SSD drive.<\/p>\nRecommended Java for ES is Oracle JDK so we install it and make it default on the server:<\/p>\n
root@myserver:~$ echo oracle-java7-installer shared\/accepted-oracle-license-v1-1 select true | tee \/usr\/bin\/debconf-set-selections && echo \"deb http:\/\/ppa.launchpad.net\/webupd8team\/java\/ubuntu precise main\" | tee -a \/etc\/apt\/sources.list && echo \"deb-src http:\/\/ppa.launchpad.net\/webupd8team\/java\/ubuntu precise main\" | tee -a \/etc\/apt\/sources.list && apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886 && aptitude update && aptitude install -y oracle-java7-installer && aptitude install -y oracle-java7-set-default\n<\/code><\/pre>\nIn case of SSD drives, we are better off with Noop<\/code> scheduler than the default CFQ (Completely Fair Queuing)<\/code> one which is better for spindles disks.<\/p>\nroot@myserver:~# cat \/sys\/block\/xvdg\/queue\/scheduler\nnoop [deadline] cfq\n\nroot@myserver:~# echo noop | tee \/sys\/block\/xvdg\/queue\/scheduler\nnoop\n\nroot@myserver:~# cat \/sys\/block\/xvdg\/queue\/scheduler\n[noop] deadline cfq\n<\/code><\/pre>\nInstall the HEAD and Marvel plugins so we have some overview of ES parameters and indices:<\/p>\n
root@myserver:~# \/usr\/share\/elasticsearch\/bin\/plugin --install mobz\/elasticsearch-head --verbose\nroot@myserver:~# \/usr\/share\/elasticsearch\/bin\/plugin --install elasticsearch\/marvel\/latest --verbose\n<\/code><\/pre>\nThe access to these plugins and ES restful API will be protected via NGINX proxy we set further down on this page. Then we start the service:<\/p>\n
root@myserver:~# service elasticsearch start\n<\/code><\/pre>\nLogstash Install and Config<\/span><\/h3>\nThe installation is a simple download of a jar file that contains all the tools needed:<\/p>\n
root@myserver:~# mkdir \/opt\/logstash\nroot@myserver:~# mkdir \/etc\/logstash\nroot@myserver:~# mkdir \/var\/log\/logstash\nroot@myserver:~# cd \/opt\/logstash\nroot@myserver:\/opt\/logstash# wget https:\/\/download.elasticsearch.org\/logstash\/logstash\/logstash-1.3.3-flatjar.jar\nroot@myserver:\/opt\/logstash# ln -sf logstash-1.3.3-flatjar.jar logstash.jar\n<\/code><\/pre>\nConfiguration, \/etc\/logstash\/reader.conf<\/code> file:<\/p>\ninput {\n redis {\n host => \"127.0.0.1\"\n type => \"redis\"\n data_type => \"list\"\n key => \"logstash\"\n password => \"<my-redis-password>\"\n }\n}\noutput {\n stdout { debug => \"true\" }\n elasticsearch {\n cluster => \"logstash\"\n }\n}\n<\/code><\/pre>\nso pretty simple, we read from the broker and write to the storage which in this case is elastic search node\/cluster we set up before. We start the logstash server from the command line as:<\/p>\n
java -jar \/opt\/logstash\/logstash.jar agent --verbose -f \/etc\/logstash\/reader.conf --log \/var\/log\/logstash\/logstash-reader.log &\n<\/code><\/pre>\nand send it running in the background. This is not very convenient so we set up an initd service script \/etc\/init.d\/logstash-reader<\/code>:<\/p>\n#! \/bin\/sh\n\n### BEGIN INIT INFO\n# Provides: logstash-shipper\n# Required-Start: $remote_fs $syslog\n# Required-Stop: $remote_fs $syslog\n# Default-Start: 2 3 4 5\n# Default-Stop: 0 1 6\n# Short-Description: Start daemon at boot time\n# Description: Enable service provided by daemon.\n### END INIT INFO\n\n. \/lib\/lsb\/init-functions\n\nmode=\"reader\"\nname=\"logstash-$mode\"\nlogstash_bin=\"\/usr\/bin\/java -- -jar \/opt\/logstash\/logstash.jar\"\nlogstash_conf=\"\/etc\/logstash\/$mode.conf\"\nlogstash_log=\"\/var\/log\/logstash\/$name.log\"\npid_file=\"\/var\/run\/$name.pid\"\n\nNICE_LEVEL=\"-n 19\"\n\nstart () {\n command=\"\/usr\/bin\/nice ${NICE_LEVEL} ${logstash_bin} agent --verbose -f $logstash_conf --log ${logstash_log}\"\n\n log_daemon_msg \"Starting $mode\" \"$name\"\n if start-stop-daemon --start --quiet --oknodo --pidfile \"$pid_file\" -b -m --exec $command; then\n log_end_msg 0\n else\n log_end_msg 1\n fi\n}\n\nstop () {\n start-stop-daemon --stop --quiet --oknodo --pidfile \"$pid_file\"\n}\n\nstatus () {\n status_of_proc -p $pid_file \"\" \"$name\"\n}\n\ncase $1 in\n start)\n if status; then exit 0; fi\n start\n ;;\n stop)\n stop\n ;;\n reload)\n stop\n start\n ;;\n restart)\n stop\n start\n ;;\n status)\n status && exit 0 || exit $?\n ;;\n *)\n echo \"Usage: $0 {start|stop|restart|reload|status}\"\n exit 1\n ;;\nesac\n\nexit 0\n<\/code><\/pre>\nand set right permissions and schedule for start up:<\/p>\n
root@myserver:~# chmod +x \/etc\/init.d\/logstash-reader\nroot@myserver:~# update-rc.d logstash-reader defaults\n<\/code><\/pre>\nAt the end we start the service:<\/p>\n
root@myserver:~# service logstash-reader start\n<\/code><\/pre>\nWe also need to manage the log files created by logstash so they don’t grow too big for which I set logrotate job in \/etc\/logrotate.d\/logstash<\/code> file:<\/p>\n\/var\/log\/logstash\/*.log {\n daily\n missingok\n rotate 5\n compress\n delaycompress\n notifempty\n create 644 root root\n sharedscripts\n postrotate\n \/etc\/init.d\/logstash-shipper reload > \/dev\/null\n endscript\n}\n<\/code><\/pre>\nWeb interface<\/span><\/h3>\nFor web interface frontend we use Kibana<\/code>:<\/p>\nroot@myserver:\/opt# wget http:\/\/download.elasticsearch.org\/kibana\/kibana\/kibana-latest.zip\nroot@myserver:\/opt# unzip kibana-latest.zip\nroot@myserver:\/opt# mv kibana-latest \/var\/www\/kibana\n<\/code><\/pre>\nWe will front by NGINX proxy which gives us more stability, security, SSL and also user access control since Kibana doesn’t come with one. See Nginx LDAP module on Debian\/Ubuntu<\/a> for the details about compiling NGINX for LDAP support. See Secure Nginx with Naxsi, SSL, GeoIP and Google Page Speed modules on Debian\/Ubuntu<\/a> for the details about compiling NGINX with Naxsi and SSL support.<\/p>\nWe are going to setup an SSL proxy for Kibana and ElasticSearch in a SSL enabled virtual host. We put the following in the \/etc\/nginx\/sites-available\/kibana<\/code> file we create (just the related part given for brevity):<\/p>\nserver {\n...\n location \/ {\n include \/etc\/nginx\/mysite.rules;\n try_files $uri $uri\/ \/index.html;\n auth_ldap \"Restricted\";\n auth_ldap_servers ldap1;\n auth_ldap_servers ldap2;\n }\n\n ## Kibana proxy\n location ~ ^\/_aliases$ {\n proxy_pass http:\/\/127.0.0.1:9200;\n proxy_read_timeout 90;\n }\n location ~ ^\/.*\/_aliases$ {\n proxy_pass http:\/\/127.0.0.1:9200;\n proxy_read_timeout 90;\n }\n location ~ ^\/_nodes$ {\n proxy_pass http:\/\/127.0.0.1:9200;\n proxy_read_timeout 90;\n }\n location ~ ^\/.*\/_search$ {\n proxy_pass http:\/\/127.0.0.1:9200;\n proxy_read_timeout 90;\n }\n location ~ ^\/.*\/_mapping$ {\n proxy_pass http:\/\/127.0.0.1:9200;\n proxy_read_timeout 90;\n }\n\n ## Password protected end points\n location ~ ^\/kibana-int\/dashboard\/.*$ {\n proxy_pass http:\/\/127.0.0.1:9200;\n proxy_read_timeout 90;\n limit_except GET {\n proxy_pass http:\/\/127.0.0.1:9200;\n auth_ldap \"Restricted\";\n auth_ldap_servers ldap1;\n auth_ldap_servers ldap2;\n }\n }\n location ~ ^\/kibana-int\/temp.*$ {\n proxy_pass http:\/\/127.0.0.1:9200;\n proxy_read_timeout 90;\n limit_except GET {\n proxy_pass http:\/\/127.0.0.1:9200;\n auth_ldap \"Restricted\";\n auth_ldap_servers ldap1;\n auth_ldap_servers ldap2;\n }\n }\n # Protected Proxy access to ES plugins and modules\n location ~ ^\/elastic\/(_.*) {\n proxy_read_timeout 90;\n proxy_pass http:\/\/127.0.0.1:9200\/$1;\n proxy_redirect http:\/\/127.0.0.1:9200 https:\/\/myserver.mydomain.com\/;\n auth_ldap \"Restricted\";\n auth_ldap_servers ldap1;\n auth_ldap_servers ldap2;\n }\n...\n}\n<\/code><\/pre>\nAt the end we enable the Kibana virtual host, disable the default one and restart Nginx:<\/p>\n
root@myserver:\/opt# rm -f \/etc\/nginx\/sites-enabled\/default\nroot@myserver:\/opt# ln -sf \/etc\/nginx\/sites-available\/kibana \/etc\/nginx\/sites-enabled\/kibana\nroot@myserver:\/opt# service nginx configtest\nroot@myserver:\/opt# service nginx restart\n<\/code><\/pre>\nNow we can go and change the ES link for Kibana:<\/p>\n
root@myserver:\/opt# vi \/var\/www\/kibana\/config.js\n...\n \/*elasticsearch: \"http:\/\/\"+window.location.hostname+\":9200\",*\/\n elasticsearch: \"https:\/\/\"+window.location.hostname+\":443\",\n...\n<\/code><\/pre>\nAfter all this in place we should see the following dashbord after log in to https:\/\/myserver.mydomain.com:<\/p>\n
![\"\"](\"https:\/\/icicimov.com\/blog\/wp-content\/uploads\/2017\/01\/kibana.png\")
<\/a>
\nPicture2:<\/strong> Kibana dashboard<\/em><\/p>\nObviously some configuration has been done to the dashboard and some data collected already when the screen shot was made.<\/p>\n
- \n\t\t\t
- \n\t\t\t\tCentral Server<\/a>\n\t\t\t\t
- \n\t\t\t\t\t
- \n\t\t\t\t\t\tKernel Setup<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t
- \n\t\t\t\t\t\tInstall Redis Broker Server<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t
- \n\t\t\t\t\t\tInstall and setup Elastic Search<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t
- \n\t\t\t\t\t\tLogstash Install and Config<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t
- \n\t\t\t\t\t\tWeb interface<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t<\/ul>\n\t\t\t
- \n\t\t\t\tClients<\/a>\n\t\t\t<\/li>\n\t\t\t
- \n\t\t\t\tSecurity<\/a>\n\t\t\t<\/li>\n<\/ul>\n<\/ul>\n<\/div>\n
\nLogstash is a tool for managing events and logs. It is very useful for collecting, parsing and storing logs for later use like for example searching. It comes with a web interface for searching through the logs. The picture bellow shows a typical centralized logstash environment. It consists of logstash clients installed on the servers running applications we want to collect logs for and centralized logstash server that does the indexing and storing of the logs. These remote logstash instances are referred asshippers<\/code> and the central one asindexer<\/code> orreader<\/code>.<\/p>\n<\/a>
\nPicture1:<\/strong> Logstash environment<\/em><\/p>\nThe broker of choice is Redis and the storage for indexing is Elastic Search, which comes natural since Logstash is part of Elastic Search project. The web interface will be served via Nginx which is light enough for this purpose.<\/p>\n
Installation and setup<\/span><\/h1>\n
Central Server<\/span><\/h2>\n
Everything is installed on a single node. We start with some kernel tuning.<\/p>\n
Kernel Setup<\/span><\/h3>\n
This will provide some system, memory optimization and network stack optimization and tuning for our EC2 instance type. At the end of the
\/etc\/sysctl.conf<\/code> we add:<\/p>\n### KERNEL ###\n\n# Core dumps\nkernel.core_uses_pid = 1\nkernel.core_pattern = \/mnt\/core-%e-%s-%u-%g-%p-%t\nfs.suid_dumpable = 2\n\n# Turn on execshild\n#kernel.exec-shield = 1\nkernel.randomize_va_space = 1\n# Reboot after 10sec. on kernel panic\nkernel.panic = 10\n\n### IMPROVE SYSTEM MEMORY MANAGEMENT ###\n\n# Increase size of file handles and inode cache\nfs.file-max = 2097152\n\n# Insure we always have enough memory\nvm.min_free_kbytes = 8192\n\n# Do less swapping\nvm.swappiness = 10\nvm.dirty_ratio = 60\nvm.dirty_background_ratio = 2\n\n### GENERAL NETWORK SECURITY OPTIONS ###\n\n# Avoid a smurf attack\nnet.ipv4.icmp_echo_ignore_broadcasts = 1\n\n# Turn on protection for bad icmp error messages\nnet.ipv4.icmp_ignore_bogus_error_responses = 1\n\n# Turn on syncookies for SYN flood attack protection\nnet.ipv4.tcp_syncookies = 1\nnet.ipv4.tcp_max_syn_backlog = 1024\n\n# Turn on timestamping\nnet.ipv4.tcp_timestamps = 1\n\n# Turn on and log spoofed, source routed, and redirect packets\nnet.ipv4.conf.all.log_martians = 1\nnet.ipv4.conf.default.log_martians = 1\n\n# No source routed packets here\nnet.ipv4.conf.all.accept_source_route = 0\nnet.ipv4.conf.default.accept_source_route = 0\n\n# Turn on reverse path filtering\nnet.ipv4.conf.all.rp_filter = 1\nnet.ipv4.conf.default.rp_filter = 1\n\n# Make sure no one can alter the routing tables\nnet.ipv4.conf.all.accept_redirects = 0\nnet.ipv4.conf.default.accept_redirects = 0\nnet.ipv4.conf.all.secure_redirects = 0\nnet.ipv4.conf.default.secure_redirects = 0\n\n# Don't act as a router\nnet.ipv4.ip_forward = 0\nnet.ipv4.conf.all.send_redirects = 0\nnet.ipv4.conf.default.send_redirects = 0\n\n# Number of times SYNACKs for passive TCP connection.\nnet.ipv4.tcp_synack_retries = 2\n\n# Allowed local port range\nnet.ipv4.ip_local_port_range = 2000 65535\n\n# Protect Against TCP Time-Wait\nnet.ipv4.tcp_rfc1337 = 1\n\n# Decrease the time default value for tcp_fin_timeout connection\nnet.ipv4.tcp_fin_timeout = 15\n\n# Decrease the time default value for connections to keep alive\nnet.ipv4.tcp_keepalive_time = 300\nnet.ipv4.tcp_keepalive_probes = 5\nnet.ipv4.tcp_keepalive_intvl = 15\n# This means that the keepalive process waits 300 seconds for socket \n# activity before sending the first keepalive probe, and then resend\n# it every 15 seconds. If no ACK response is received for 5 consecutive \n# times (75s in this case), the connection is marked as broken.\n\n### TUNING NETWORK PERFORMANCE ###\n\n# Disable IPv6\nnet.ipv6.conf.all.disable_ipv6 = 1\nnet.ipv6.conf.default.disable_ipv6 = 1\nnet.ipv6.conf.lo.disable_ipv6 = 1\n\n# Default Socket Receive Buffer\nnet.core.rmem_default = 31457280\n\n# Maximum Socket Receive Buffer\nnet.core.rmem_max = 12582912\n\n# Default Socket Send Buffer\nnet.core.wmem_default = 31457280\n\n# Maximum Socket Send Buffer\nnet.core.wmem_max = 12582912\n\n# Increase number of incoming connections\nnet.core.somaxconn = 5000\n\n# Increase number of incoming connections backlog\nnet.core.netdev_max_backlog = 65536\n\n# Enable TCP window scaling\nnet.ipv4.tcp_window_scaling = 1\n\n# Increase the maximum amount of option memory buffers\nnet.core.optmem_max = 25165824\n\n# Increase the maximum total buffer-space allocatable\n# This is measured in units of pages (4096 bytes)\nnet.ipv4.tcp_mem = 65536 131072 262144\nnet.ipv4.udp_mem = 65536 131072 262144\n\n# Increase the read-buffer space allocatable\nnet.ipv4.tcp_rmem = 8192 87380 16777216\nnet.ipv4.udp_rmem_min = 16384\n\n# Increase the write-buffer-space allocatable\nnet.ipv4.tcp_wmem = 8192 65536 16777216\nnet.ipv4.udp_wmem_min = 16384\n\n# Increase the tcp-time-wait buckets pool size to prevent simple DOS attacks\nnet.ipv4.tcp_max_tw_buckets = 1440000\n\n# TIME_WAIT socket policy\n# Note: if both enabled then disable\n# net.ipv4.tcp_timestamps for servers \n# behind NAT to prevent dropped incoming connections\nnet.ipv4.tcp_tw_recycle = 0\nnet.ipv4.tcp_tw_reuse = 1\n\n# Enable TCP MTU probing (in case of Jumbo Frames enabled)\n#net.ipv4.tcp_mtu_probing = 1\n\n# Speedup retrans (Google recommended)\nnet.ipv4.tcp_slow_start_after_idle = 0\nnet.ipv4.tcp_early_retrans = 1\n\n# Conntrack\n# 288bytes x 131072 = 37748736 (~38MB) max memory usage\nnet.netfilter.nf_conntrack_max = 131072\nnet.netfilter.nf_conntrack_tcp_loose = 1\n\n# NOTE: Enable this if EC2 instance support it\n# -- 10gbe tuning from Intel ixgb driver README -- #\n# turn off selective ACK and timestamps\n#net.ipv4.tcp_sack = 0\n#net.ipv4.tcp_timestamps = 0\n<\/code><\/pre>\nand run:<\/p>\n
sysctl -p\n<\/code><\/pre>\nto activate the changes.<\/p>\n
Install Redis Broker Server<\/span><\/h3>\n
We need broker services to buffer all the client updates while the central Logstash service is offline, in case of issues with Logstash or update etc.<\/p>\n
root@myserver:~# aptitude install redis-server\n<\/code><\/pre>\nIn case all our remote clients were to connect directly to the broker we need to change the listening address in the config file and replace:<\/p>\n
bind 127.0.0.1\n<\/code><\/pre>\nwith:<\/p>\n
bind 0.0.0.0\n<\/code><\/pre>\nin
\/etc\/redis\/redis.conf<\/code>. We are going to do this temporarily for testing only since we are going to usestunnel<\/code> to secure Redis communication.<\/p>\nFor security reasons we want only the authenticated clients to communicate with the server so we set password in the config file:<\/p>\n
requirepass <my-redis-password>\n<\/code><\/pre>\nWe also need to open the tcp port
6379<\/code> in the firewall (EC2 instance SecurityGroup) in case we enabled remote Redis access.<\/p>\nEnable the overcommit kernel memory option so every
malloc()<\/code> operation will succeed:<\/p>\nroot@myserver:~# sysctl vm.overcommit_memory=1\n<\/code><\/pre>\nand add it to
\/etc\/sysctl.conf<\/code> as well so it persists over bootups. Then start the service:<\/p>\nroot@myserver:~# service redis start\n<\/code><\/pre>\nInstall and setup Elastic Search<\/span><\/h3>\n
We need at least version 0.90.7.<\/p>\n
root@myserver:~# wget https:\/\/download.elasticsearch.org\/elasticsearch\/elasticsearch\/elasticsearch-0.90.9.deb\nroot@myserver:~# dpkg -i elasticsearch-0.90.9.deb\n<\/code><\/pre>\nThen edit the
\/etc\/elasticsearch\/elasticsearch.yml<\/code> config file and set some options that suit our needs and VM hardware:<\/p>\ncluster.name: logstash\nnode.name: \"ec2-logger-server\"\npath.data: \/mnt\/elasticsearch,\/mnt2\/elasticsearch\npath.logs: \/mnt2\/log\/elasticsearch\nbootstrap.mlockall: true\nnetwork.host: \"127.0.0.1\"\n\n## Threadpool Settings ##\n# Search pool\nthreadpool.search.type: fixed\nthreadpool.search.size: 20\nthreadpool.search.queue_size: 100\n\n# Bulk pool\nthreadpool.bulk.type: fixed\nthreadpool.bulk.size: 60\nthreadpool.bulk.queue_size: 300\n\n# Index pool\nthreadpool.index.type: fixed\nthreadpool.index.size: 20\nthreadpool.index.queue_size: 100\n\n# Indices settings\nindices.memory.index_buffer_size: 30%\nindices.memory.min_shard_index_buffer_size: 12mb\nindices.memory.min_index_buffer_size: 96mb\n\n# Cache Sizes\nindices.fielddata.cache.size: 15%\nindices.fielddata.cache.expire: 6h\nindices.cache.filter.size: 15%\nindices.cache.filter.expire: 6h\n\n# Indexing Settings for Writes\nindex.refresh_interval: 30s\nindex.translog.flush_threshold_ops: 50000\n<\/code><\/pre>\nWe enable
bootstrap.mlockall<\/code> to try to lock the process address space so it won’t be swapped (recommended by ES configuration page). This also requires setting the following options in the\/etc\/default\/elasticsearch<\/code> file:<\/p>\nES_HEAP_SIZE=2g\nMAX_LOCKED_MEMORY=unlimited\n<\/code><\/pre>\nWe have also tuned the search, bulk and index pools giving the bulk pool 3 times more threads since the server will be mostly writing. We also tune the index sizes and cache flushes to reduce the load on the server under heavy writes.<\/p>\n
Later on we had to add an additional drive via
path.data:\/mnt\/elasticsearch,\/mnt2\/elasticsearch<\/code> option to expand the index disk storage. This option also adds on speed since the combination of the two drives acts like a RAID0 for files, of course compared to chunks as in real RAID. I have also moved the logs to\/mnt2\/elasticsearch<\/code> drive since it is a faster SSD drive.<\/p>\nRecommended Java for ES is Oracle JDK so we install it and make it default on the server:<\/p>\n
root@myserver:~$ echo oracle-java7-installer shared\/accepted-oracle-license-v1-1 select true | tee \/usr\/bin\/debconf-set-selections && echo \"deb http:\/\/ppa.launchpad.net\/webupd8team\/java\/ubuntu precise main\" | tee -a \/etc\/apt\/sources.list && echo \"deb-src http:\/\/ppa.launchpad.net\/webupd8team\/java\/ubuntu precise main\" | tee -a \/etc\/apt\/sources.list && apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886 && aptitude update && aptitude install -y oracle-java7-installer && aptitude install -y oracle-java7-set-default\n<\/code><\/pre>\nIn case of SSD drives, we are better off with
Noop<\/code> scheduler than the defaultCFQ (Completely Fair Queuing)<\/code> one which is better for spindles disks.<\/p>\nroot@myserver:~# cat \/sys\/block\/xvdg\/queue\/scheduler\nnoop [deadline] cfq\n\nroot@myserver:~# echo noop | tee \/sys\/block\/xvdg\/queue\/scheduler\nnoop\n\nroot@myserver:~# cat \/sys\/block\/xvdg\/queue\/scheduler\n[noop] deadline cfq\n<\/code><\/pre>\nInstall the HEAD and Marvel plugins so we have some overview of ES parameters and indices:<\/p>\n
root@myserver:~# \/usr\/share\/elasticsearch\/bin\/plugin --install mobz\/elasticsearch-head --verbose\nroot@myserver:~# \/usr\/share\/elasticsearch\/bin\/plugin --install elasticsearch\/marvel\/latest --verbose\n<\/code><\/pre>\nThe access to these plugins and ES restful API will be protected via NGINX proxy we set further down on this page. Then we start the service:<\/p>\n
root@myserver:~# service elasticsearch start\n<\/code><\/pre>\nLogstash Install and Config<\/span><\/h3>\n
The installation is a simple download of a jar file that contains all the tools needed:<\/p>\n
root@myserver:~# mkdir \/opt\/logstash\nroot@myserver:~# mkdir \/etc\/logstash\nroot@myserver:~# mkdir \/var\/log\/logstash\nroot@myserver:~# cd \/opt\/logstash\nroot@myserver:\/opt\/logstash# wget https:\/\/download.elasticsearch.org\/logstash\/logstash\/logstash-1.3.3-flatjar.jar\nroot@myserver:\/opt\/logstash# ln -sf logstash-1.3.3-flatjar.jar logstash.jar\n<\/code><\/pre>\nConfiguration,
\/etc\/logstash\/reader.conf<\/code> file:<\/p>\ninput {\n redis {\n host => \"127.0.0.1\"\n type => \"redis\"\n data_type => \"list\"\n key => \"logstash\"\n password => \"<my-redis-password>\"\n }\n}\noutput {\n stdout { debug => \"true\" }\n elasticsearch {\n cluster => \"logstash\"\n }\n}\n<\/code><\/pre>\nso pretty simple, we read from the broker and write to the storage which in this case is elastic search node\/cluster we set up before. We start the logstash server from the command line as:<\/p>\n
java -jar \/opt\/logstash\/logstash.jar agent --verbose -f \/etc\/logstash\/reader.conf --log \/var\/log\/logstash\/logstash-reader.log &\n<\/code><\/pre>\nand send it running in the background. This is not very convenient so we set up an initd service script
\/etc\/init.d\/logstash-reader<\/code>:<\/p>\n#! \/bin\/sh\n\n### BEGIN INIT INFO\n# Provides: logstash-shipper\n# Required-Start: $remote_fs $syslog\n# Required-Stop: $remote_fs $syslog\n# Default-Start: 2 3 4 5\n# Default-Stop: 0 1 6\n# Short-Description: Start daemon at boot time\n# Description: Enable service provided by daemon.\n### END INIT INFO\n\n. \/lib\/lsb\/init-functions\n\nmode=\"reader\"\nname=\"logstash-$mode\"\nlogstash_bin=\"\/usr\/bin\/java -- -jar \/opt\/logstash\/logstash.jar\"\nlogstash_conf=\"\/etc\/logstash\/$mode.conf\"\nlogstash_log=\"\/var\/log\/logstash\/$name.log\"\npid_file=\"\/var\/run\/$name.pid\"\n\nNICE_LEVEL=\"-n 19\"\n\nstart () {\n command=\"\/usr\/bin\/nice ${NICE_LEVEL} ${logstash_bin} agent --verbose -f $logstash_conf --log ${logstash_log}\"\n\n log_daemon_msg \"Starting $mode\" \"$name\"\n if start-stop-daemon --start --quiet --oknodo --pidfile \"$pid_file\" -b -m --exec $command; then\n log_end_msg 0\n else\n log_end_msg 1\n fi\n}\n\nstop () {\n start-stop-daemon --stop --quiet --oknodo --pidfile \"$pid_file\"\n}\n\nstatus () {\n status_of_proc -p $pid_file \"\" \"$name\"\n}\n\ncase $1 in\n start)\n if status; then exit 0; fi\n start\n ;;\n stop)\n stop\n ;;\n reload)\n stop\n start\n ;;\n restart)\n stop\n start\n ;;\n status)\n status && exit 0 || exit $?\n ;;\n *)\n echo \"Usage: $0 {start|stop|restart|reload|status}\"\n exit 1\n ;;\nesac\n\nexit 0\n<\/code><\/pre>\nand set right permissions and schedule for start up:<\/p>\n
root@myserver:~# chmod +x \/etc\/init.d\/logstash-reader\nroot@myserver:~# update-rc.d logstash-reader defaults\n<\/code><\/pre>\nAt the end we start the service:<\/p>\n
root@myserver:~# service logstash-reader start\n<\/code><\/pre>\nWe also need to manage the log files created by logstash so they don’t grow too big for which I set logrotate job in
\/etc\/logrotate.d\/logstash<\/code> file:<\/p>\n\/var\/log\/logstash\/*.log {\n daily\n missingok\n rotate 5\n compress\n delaycompress\n notifempty\n create 644 root root\n sharedscripts\n postrotate\n \/etc\/init.d\/logstash-shipper reload > \/dev\/null\n endscript\n}\n<\/code><\/pre>\nWeb interface<\/span><\/h3>\n
For web interface frontend we use
Kibana<\/code>:<\/p>\nroot@myserver:\/opt# wget http:\/\/download.elasticsearch.org\/kibana\/kibana\/kibana-latest.zip\nroot@myserver:\/opt# unzip kibana-latest.zip\nroot@myserver:\/opt# mv kibana-latest \/var\/www\/kibana\n<\/code><\/pre>\nWe will front by NGINX proxy which gives us more stability, security, SSL and also user access control since Kibana doesn’t come with one. See Nginx LDAP module on Debian\/Ubuntu<\/a> for the details about compiling NGINX for LDAP support. See Secure Nginx with Naxsi, SSL, GeoIP and Google Page Speed modules on Debian\/Ubuntu<\/a> for the details about compiling NGINX with Naxsi and SSL support.<\/p>\n
We are going to setup an SSL proxy for Kibana and ElasticSearch in a SSL enabled virtual host. We put the following in the
\/etc\/nginx\/sites-available\/kibana<\/code> file we create (just the related part given for brevity):<\/p>\nserver {\n...\n location \/ {\n include \/etc\/nginx\/mysite.rules;\n try_files $uri $uri\/ \/index.html;\n auth_ldap \"Restricted\";\n auth_ldap_servers ldap1;\n auth_ldap_servers ldap2;\n }\n\n ## Kibana proxy\n location ~ ^\/_aliases$ {\n proxy_pass http:\/\/127.0.0.1:9200;\n proxy_read_timeout 90;\n }\n location ~ ^\/.*\/_aliases$ {\n proxy_pass http:\/\/127.0.0.1:9200;\n proxy_read_timeout 90;\n }\n location ~ ^\/_nodes$ {\n proxy_pass http:\/\/127.0.0.1:9200;\n proxy_read_timeout 90;\n }\n location ~ ^\/.*\/_search$ {\n proxy_pass http:\/\/127.0.0.1:9200;\n proxy_read_timeout 90;\n }\n location ~ ^\/.*\/_mapping$ {\n proxy_pass http:\/\/127.0.0.1:9200;\n proxy_read_timeout 90;\n }\n\n ## Password protected end points\n location ~ ^\/kibana-int\/dashboard\/.*$ {\n proxy_pass http:\/\/127.0.0.1:9200;\n proxy_read_timeout 90;\n limit_except GET {\n proxy_pass http:\/\/127.0.0.1:9200;\n auth_ldap \"Restricted\";\n auth_ldap_servers ldap1;\n auth_ldap_servers ldap2;\n }\n }\n location ~ ^\/kibana-int\/temp.*$ {\n proxy_pass http:\/\/127.0.0.1:9200;\n proxy_read_timeout 90;\n limit_except GET {\n proxy_pass http:\/\/127.0.0.1:9200;\n auth_ldap \"Restricted\";\n auth_ldap_servers ldap1;\n auth_ldap_servers ldap2;\n }\n }\n # Protected Proxy access to ES plugins and modules\n location ~ ^\/elastic\/(_.*) {\n proxy_read_timeout 90;\n proxy_pass http:\/\/127.0.0.1:9200\/$1;\n proxy_redirect http:\/\/127.0.0.1:9200 https:\/\/myserver.mydomain.com\/;\n auth_ldap \"Restricted\";\n auth_ldap_servers ldap1;\n auth_ldap_servers ldap2;\n }\n...\n}\n<\/code><\/pre>\nAt the end we enable the Kibana virtual host, disable the default one and restart Nginx:<\/p>\n
root@myserver:\/opt# rm -f \/etc\/nginx\/sites-enabled\/default\nroot@myserver:\/opt# ln -sf \/etc\/nginx\/sites-available\/kibana \/etc\/nginx\/sites-enabled\/kibana\nroot@myserver:\/opt# service nginx configtest\nroot@myserver:\/opt# service nginx restart\n<\/code><\/pre>\nNow we can go and change the ES link for Kibana:<\/p>\n
root@myserver:\/opt# vi \/var\/www\/kibana\/config.js\n...\n \/*elasticsearch: \"http:\/\/\"+window.location.hostname+\":9200\",*\/\n elasticsearch: \"https:\/\/\"+window.location.hostname+\":443\",\n...\n<\/code><\/pre>\nAfter all this in place we should see the following dashbord after log in to https:\/\/myserver.mydomain.com:<\/p>\n
<\/a>
\nPicture2:<\/strong> Kibana dashboard<\/em><\/p>\nObviously some configuration has been done to the dashboard and some data collected already when the screen shot was made.<\/p>\n
- \n\t\t\t\t\t\tInstall Redis Broker Server<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t
- \n\t\t\t\t\t\tKernel Setup<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t
![\"\"](\"https:\/\/icicimov.com\/blog\/wp-content\/uploads\/2017\/01\/logstash_marvel_plugin.png\")
<\/a>