Seite wählen

NETWAYS Blog

Let's Encrypt HTTP Proxy: Another approach

Yes, we all know that providing microservices with Docker is a very wicked thing. Easy to use and of course there is a Docker image available for almost every application you need. The next step to master is how we can expose this service to the public. It seems that this is where most people struggle. Browsing the web (generally GitHub) for HTTP proxies you’ll find an incredible number of images, people build to fit into their environments. Especially when it comes to implement a Let’s Encrypt / SNI encryption service. Because it raises the same questions every time: Is this really the definitely right way to do this? Wrap custom API’s around conventional products like web servers and proxies, inject megabytes of JSON (or YAML or TOML) through environment variables and build scrips to convert this into the product specific configuration language? Always my bad conscience tapped on the door while I did every time.
Some weeks ago, I stumble upon Træfik which is obviously not a new Tool album but a HTTP proxy server which has everything a highly dynamic Docker platform needs to expose its services and includes Let’s Encrypt silently – Such a thing doesn’t exist you say?
A brief summary:

Træfik is a single binary daemon, written in Go, lightweight and can be used in virtually any modern environment. Configuration is done by choosing a backend you have. This could be an orchestrator like Swarm or Kubernetes but you can also use a more “open” approach, like etcd, REST API’s or file backend (backends can be mixed of course). For example, if you are using plain Docker, or Docker Compose, Træfik uses Docker object labels to configures services. A simple configuration looks like this:

[docker]
endpoint = "unix:///var/run/docker.sock"
# endpoint = "tcp://127.0.0.1:2375"
domain = "docker.localhost"
watch = true

Træfik constantly watch for changes in your running Docker container and automatically adds backends to its configuration. Docker container itself only needs labels like this (configured as Docker Compose in this example):

whoami:
image: emilevauge/whoami # A container that exposes an API to show its IP address
labels:
- "traefik.frontend.rule=Host:whoami.docker.localhost"

The clue is, that you can configure everything you’ll need that is often pretty complex in conventional products. This is for Example multiple domains, headers for API’s, redirects, permissions, container which exposes multiple ports and interfaces and so on.
How you succeed with the configuration can be validated in a frontend which is included in Træfik.

However, the best thing everybody was waiting is the seamless Let’s Encrypt integration which can be achieved with this snippet:

[acme]
email = "test@traefik.io"
storage = "acme.json"
entryPoint = "https"
[acme.httpChallenge]
entryPoint = "http"
# [[acme.domains]]
# main = "local1.com"
# sans = ["test1.local1.com", "test2.local1.com"]
# [[acme.domains]]
# main = "local2.com"
# [[acme.domains]]
# main = "*.local3.com"
# sans = ["local3.com", "test1.test1.local3.com"]

Træfik will create the certificates automatically. Of course, you have a lot of conveniences here too, like wildcard certificates with DNS verification though different DNS API providers and stuff like that.
To conclude that above statements, it exists a thing which can meet the demands for a simple, apified reverse proxy we need in a dockerized world. Just give it a try and see how easy microservices – especially TLS-encrypted – can be.

Marius Hein
Marius Hein
Head of IT Service Management

Marius Hein ist schon seit 2003 bei NETWAYS. Er hat hier seine Ausbildung zum Fachinformatiker absolviert und viele Jahre in der Softwareentwicklung gearbeitet. Mittlerweile ist er Herr über die interne IT und als Leiter von ITSM zuständig für die technische Schnittmenge der Abteilungen der NETWAYS Gruppe. Wenn er nicht gerade IPv6 IPSec Tunnel bohrt, sitzt er daheim am Schlagzeug und treibt seine Nachbarn in den Wahnsinn.

Secure Elasticsearch and Kibana with an Nginx HTTP Proxy

Elasticsearch provides a great HTTP API where applications can write to and read from in high performance environments. One of our customers sponsored a feature for Icinga 2 which writes events and performance data metrics to Elasticsearch. This will hit v2.8 later this year.
We’re also concerned about security, and have been looking into security mechanisms such as basic auth or TLS. Unfortunately this isn’t included in the Open Source stack.
 

Why should you care about securing Elasticsearch and Kibana?

Modern infrastructure deployments commonly require Elasticsearch listening on an external interface and answering HTTP requests. Earlier this year we’ve learned about ransomware attacks on MongoDB and Elasticsearch too.
During development I’ve found this API call – just clear everything inside the database.

[root@icinga2-elastic ~]# curl -X DELETE http://localhost:9200/_all
{"acknowledged":true}

I don’t want any user to run this command from anywhere. You can disable this by setting „action.destructive_requires_name“ to „true“ inside the configuration, but it is not the default.
A similar thing is that you can read and write anything without any access rules in place, no matter if querying Elasticsearch or Kibana directly.
 

Firewall and Network Segments

A possible solution is to limit the network transport to only allowed hosts via firewall rules and so on. If your Elasticsearch cluster is running on a dedicated VLAN, you would need to allow the Icinga 2 monitoring host to write data into – anyone on that machine could still read/write without any security mechanism in place.
 

HTTP Proxy with Nginx

Start with the plain proxy pass configuration and forward http://localhosT:9200 to your external interface (192.168.33.7 in this example).

# MANAGED BY PUPPET
server {
  listen       192.168.33.7:9200;
  server_name  elasticsearch.vagrant-demo.icinga.com;
  index  index.html index.htm index.php;
  access_log            /var/log/nginx/ssl-elasticsearch.vagrant-demo.icinga.com.access.log combined;
  error_log             /var/log/nginx/ssl-elasticsearch.vagrant-demo.icinga.com.error.log;
  location / {
    proxy_pass            http://localhost:9200;
    proxy_read_timeout    90;
    proxy_connect_timeout 90;
    proxy_set_header      Host $host;
    proxy_set_header      X-Real-IP $remote_addr;
    proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header      Proxy "";
  }
}

Restart Nginx and test the connection from the external interface.

# systemctl restart nginx
# curl -v http://192.168.33.7:9200

Once this is working, proceed with adding basic auth and TLS.
 

HTTP Proxy with Basic Auth

This leverages the access level to authenticated users only. Best is to manage the basic auth users file with Puppet, Ansible, etc. – similar to how you manage your Nginx configuration. Our consultants use that method on a regular basis, and provided me with some examples for Nginx. You could do the same for Apache – I would guess that is a matter of taste and performance here.
Generate a username/password combination e.g. using the htpasswd CLI command.

htpasswd -c /etc/nginx/elasticsearch.passwd icinga

Specify the basic auth message and the file which contains the basic auth users.

    auth_basic                "Elasticsearch auth";
    auth_basic_user_file      "/etc/nginx/elasticsearch.passwd";

Restart Nginx and connect to the external interface.

# systemctl restart nginx
# curl -v -u icinga:icinga http://192.168.33.7:9200

 

HTTP Proxy with TLS

The Elasticsearch HTTP API does not support TLS out-of-the-box. You need to enforce HTTPS via HTTP Proxy, enable ssl and set up the required certificates.
Enforce the listen address to SSL only. That way http won’t work.

  listen       192.168.33.7:9200 ssl;

Enable SSL, specify the certificate paths on disk, use TLSv1 and above and optionally secure the used ciphers.

  ssl on;
  ssl_certificate           /etc/nginx/certs/icinga2-elastic.crt;
  ssl_certificate_key       /etc/nginx/certs/icinga2-elastic.key;
  ssl_session_cache         shared:SSL:10m;
  ssl_session_timeout       5m;
  ssl_protocols             TLSv1 TLSv1.1 TLSv1.2;
  ssl_ciphers               ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS;
  ssl_prefer_server_ciphers on;
  ssl_trusted_certificate   /etc/nginx/certs/ca.crt;

Restart Nginx and connect to the external interface on https. Note: Host verification is disabled in this example.

# systemctl restart nginx
# curl -v -k -u icinga:icinga https://192.168.33.7:9200

 

Combine HTTP Proxy, TLS and Basic Auth

A complete configuration example could look like this:

vim /etc/nginx/sites-enabled/elasticsearch.vagrant-demo.icinga.com.conf
# MANAGED BY PUPPET
server {
  listen       192.168.33.7:9200 ssl;
  server_name  elasticsearch.vagrant-demo.icinga.com;
  ssl on;
  ssl_certificate           /etc/nginx/certs/icinga2-elastic.crt;
  ssl_certificate_key       /etc/nginx/certs/icinga2-elastic.key;
  ssl_session_cache         shared:SSL:10m;
  ssl_session_timeout       5m;
  ssl_protocols             TLSv1 TLSv1.1 TLSv1.2;
  ssl_ciphers               ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS;
  ssl_prefer_server_ciphers on;
  ssl_trusted_certificate   /etc/nginx/certs/ca.crt;
  uth_basic                "Elasticsearch auth";
  auth_basic_user_file      "/etc/nginx/elasticsearch.passwd";
  index  index.html index.htm index.php;
  access_log            /var/log/nginx/ssl-elasticsearch.vagrant-demo.icinga.com.access.log combined;
  error_log             /var/log/nginx/ssl-elasticsearch.vagrant-demo.icinga.com.error.log;
  location / {
    proxy_pass            http://localhost:9200;
    proxy_read_timeout    90;
    proxy_connect_timeout 90;
    proxy_set_header      Host $host;
    proxy_set_header      X-Real-IP $remote_addr;
    proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header      Proxy "";
  }
}

The following example query does not verify the offered host certificate. In case you configure the ElasticWriter feature in Icinga 2 v2.8, you’ll find the options to specify certificates for TLS handshake verification.

$ curl -v -k -u icinga:icinga https://192.168.33.7:9200
* Rebuilt URL to: https://192.168.33.7:9200/
*   Trying 192.168.33.7...
* TCP_NODELAY set
* Connected to 192.168.33.7 (192.168.33.7) port 9200 (#0)
* WARNING: disabling hostname validation also disables SNI.
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: icinga2-elastic
* Server auth using Basic with user 'icinga'
> GET / HTTP/1.1
> Host: 192.168.33.7:9200
> Authorization: Basic aWNpbmdhOmljaW5nYQ==
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.12.1
< Date: Tue, 12 Sep 2017 13:52:31 GMT
< Content-Type: application/json; charset=UTF-8
< Content-Length: 340
< Connection: keep-alive
<
{
  "name" : "icinga2-elastic-elastic-es",
  "cluster_name" : "elastic",
  "cluster_uuid" : "axUBwxpFSpeFBmVRD6tTiw",
  "version" : {
    "number" : "5.5.2",
    "build_hash" : "b2f0c09",
    "build_date" : "2017-08-14T12:33:14.154Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.0"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host 192.168.33.7 left intact

 

Conclusion

Secure data transfer from your monitoring instances to Elasticsearch is mandatory. Basic access control via basic auth should also be implemented. All of this is possible with the help of a dedicated HTTP Proxy host. Fine granular access control for specific HTTP requests is available in the commercial Shield package or variants. While securing Elasticsearch, also look into Kibana which runs on port 5601.
Since we’ve used the Icinga Vagrant boxes as a development playground, I’ve added Nginx as HTTP Proxy inside the icinga2x-elastic box. This provisions the required basic auth and TLS settings and offers to write data on https://192.168.33.7:9200 (icinga/icinga). The same applies for Kibana. The examples above can be replayed too.
If you want to learn more on this topic, make sure to join our Elastic Stack training sessions or kindly invite one of our consultants for a hands-on workshop. Hint: There is an Elastic Stack workshop at OSMC too 🙂