Elasticsearch provides a great HTTP API where applications can write to and read from in high performance environments. One of our customers sponsored a feature for Icinga 2 which writes events and performance data metrics to Elasticsearch. This will hit v2.8 later this year.
We’re also concerned about security, and have been looking into security mechanisms such as basic auth or TLS. Unfortunately this isn’t included in the Open Source stack.
Why should you care about securing Elasticsearch and Kibana?
Modern infrastructure deployments commonly require Elasticsearch listening on an external interface and answering HTTP requests. Earlier this year we’ve learned about ransomware attacks on MongoDB and Elasticsearch too.
During development I’ve found this API call – just clear everything inside the database.
[root@icinga2-elastic ~]# curl -X DELETE http://localhost:9200/_all
{"acknowledged":true}
I don’t want any user to run this command from anywhere. You can disable this by setting „action.destructive_requires_name“ to „true“ inside the configuration, but it is not the default.
A similar thing is that you can read and write anything without any access rules in place, no matter if querying Elasticsearch or Kibana directly.
Firewall and Network Segments
A possible solution is to limit the network transport to only allowed hosts via firewall rules and so on. If your Elasticsearch cluster is running on a dedicated VLAN, you would need to allow the Icinga 2 monitoring host to write data into – anyone on that machine could still read/write without any security mechanism in place.
HTTP Proxy with Nginx
Start with the plain proxy pass configuration and forward http://localhosT:9200 to your external interface (192.168.33.7 in this example).
# MANAGED BY PUPPET
server {
listen 192.168.33.7:9200;
server_name elasticsearch.vagrant-demo.icinga.com;
index index.html index.htm index.php;
access_log /var/log/nginx/ssl-elasticsearch.vagrant-demo.icinga.com.access.log combined;
error_log /var/log/nginx/ssl-elasticsearch.vagrant-demo.icinga.com.error.log;
location / {
proxy_pass http://localhost:9200;
proxy_read_timeout 90;
proxy_connect_timeout 90;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Proxy "";
}
}
Restart Nginx and test the connection from the external interface.
# systemctl restart nginx # curl -v http://192.168.33.7:9200
Once this is working, proceed with adding basic auth and TLS.
HTTP Proxy with Basic Auth
This leverages the access level to authenticated users only. Best is to manage the basic auth users file with Puppet, Ansible, etc. – similar to how you manage your Nginx configuration. Our consultants use that method on a regular basis, and provided me with some examples for Nginx. You could do the same for Apache – I would guess that is a matter of taste and performance here.
Generate a username/password combination e.g. using the htpasswd CLI command.
htpasswd -c /etc/nginx/elasticsearch.passwd icinga
Specify the basic auth message and the file which contains the basic auth users.
auth_basic "Elasticsearch auth";
auth_basic_user_file "/etc/nginx/elasticsearch.passwd";
Restart Nginx and connect to the external interface.
# systemctl restart nginx # curl -v -u icinga:icinga http://192.168.33.7:9200
HTTP Proxy with TLS
The Elasticsearch HTTP API does not support TLS out-of-the-box. You need to enforce HTTPS via HTTP Proxy, enable ssl and set up the required certificates.
Enforce the listen address to SSL only. That way http won’t work.
listen 192.168.33.7:9200 ssl;
Enable SSL, specify the certificate paths on disk, use TLSv1 and above and optionally secure the used ciphers.
ssl on; ssl_certificate /etc/nginx/certs/icinga2-elastic.crt; ssl_certificate_key /etc/nginx/certs/icinga2-elastic.key; ssl_session_cache shared:SSL:10m; ssl_session_timeout 5m; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS; ssl_prefer_server_ciphers on; ssl_trusted_certificate /etc/nginx/certs/ca.crt;
Restart Nginx and connect to the external interface on https. Note: Host verification is disabled in this example.
# systemctl restart nginx # curl -v -k -u icinga:icinga https://192.168.33.7:9200
Combine HTTP Proxy, TLS and Basic Auth
A complete configuration example could look like this:
vim /etc/nginx/sites-enabled/elasticsearch.vagrant-demo.icinga.com.conf
# MANAGED BY PUPPET
server {
listen 192.168.33.7:9200 ssl;
server_name elasticsearch.vagrant-demo.icinga.com;
ssl on;
ssl_certificate /etc/nginx/certs/icinga2-elastic.crt;
ssl_certificate_key /etc/nginx/certs/icinga2-elastic.key;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 5m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS;
ssl_prefer_server_ciphers on;
ssl_trusted_certificate /etc/nginx/certs/ca.crt;
uth_basic "Elasticsearch auth";
auth_basic_user_file "/etc/nginx/elasticsearch.passwd";
index index.html index.htm index.php;
access_log /var/log/nginx/ssl-elasticsearch.vagrant-demo.icinga.com.access.log combined;
error_log /var/log/nginx/ssl-elasticsearch.vagrant-demo.icinga.com.error.log;
location / {
proxy_pass http://localhost:9200;
proxy_read_timeout 90;
proxy_connect_timeout 90;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Proxy "";
}
}
The following example query does not verify the offered host certificate. In case you configure the ElasticWriter feature in Icinga 2 v2.8, you’ll find the options to specify certificates for TLS handshake verification.
$ curl -v -k -u icinga:icinga https://192.168.33.7:9200
* Rebuilt URL to: https://192.168.33.7:9200/
* Trying 192.168.33.7...
* TCP_NODELAY set
* Connected to 192.168.33.7 (192.168.33.7) port 9200 (#0)
* WARNING: disabling hostname validation also disables SNI.
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: icinga2-elastic
* Server auth using Basic with user 'icinga'
> GET / HTTP/1.1
> Host: 192.168.33.7:9200
> Authorization: Basic aWNpbmdhOmljaW5nYQ==
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.12.1
< Date: Tue, 12 Sep 2017 13:52:31 GMT
< Content-Type: application/json; charset=UTF-8
< Content-Length: 340
< Connection: keep-alive
<
{
"name" : "icinga2-elastic-elastic-es",
"cluster_name" : "elastic",
"cluster_uuid" : "axUBwxpFSpeFBmVRD6tTiw",
"version" : {
"number" : "5.5.2",
"build_hash" : "b2f0c09",
"build_date" : "2017-08-14T12:33:14.154Z",
"build_snapshot" : false,
"lucene_version" : "6.6.0"
},
"tagline" : "You Know, for Search"
}
* Connection #0 to host 192.168.33.7 left intact
Conclusion
Secure data transfer from your monitoring instances to Elasticsearch is mandatory. Basic access control via basic auth should also be implemented. All of this is possible with the help of a dedicated HTTP Proxy host. Fine granular access control for specific HTTP requests is available in the commercial Shield package or variants. While securing Elasticsearch, also look into Kibana which runs on port 5601.
Since we’ve used the Icinga Vagrant boxes as a development playground, I’ve added Nginx as HTTP Proxy inside the icinga2x-elastic box. This provisions the required basic auth and TLS settings and offers to write data on https://192.168.33.7:9200 (icinga/icinga). The same applies for Kibana. The examples above can be replayed too.
If you want to learn more on this topic, make sure to join our Elastic Stack training sessions or kindly invite one of our consultants for a hands-on workshop. Hint: There is an Elastic Stack workshop at OSMC too 🙂
