This week we had the pleasure to welcome Jan Doberstein from Graylog. On Monday our consulting team and myself attended a Graylog workshop held by Jan. Since many of us are already familiar with log management (e.g. Elastic Stack), we’ve skipped the basics and got a deep-dive into the Graylog stack.
You’ll need Elasticsearch, MongoDB and Graylog Server running on your instance and then you are good to go. MongoDB is mainly used for caching and sessions but also as user storage e.g. for dashboards and more. Graylog Server provides a REST API and web interface.
Configuration and Inputs
Once you’ve everything up and running, open your browser and log into Graylog. The default entry page greets you with additional tips and tricks. Graylog is all about usability – you are advised to create inputs to send in data from remote. Everything can be configured via the web interface, or the REST API. Jan also told us that some more advanced settings are only available via the REST API.
If you need more input plugins, you can search the marketplace and install the required one. Or you’ll create your own. By default Graylog supports GELF, Beats, Syslog, Kafka, AMQP, HTTP.
One thing I also learned during our workshop: Graylog also supports Elastic Beats as input. This allows even more possibilities to integrate existing setups with Icingabeat, filebeat, winlogbeat and more.
Authentication
Graylog supports „internal auth“ (manual user creation), sessions/tokens and also LDAP/AD. You can configure and test that via the web interface. One thing to note: The LDAP library doesn’t support nested groups for now. You can create and assign specific roles with restrictions. Even multiple providers and their order can be specified.
Streams and Alerts
Incoming messages can be routed into so-called „streams“. You can inspect an existing message and create a rule set based on these details. That way you can for example route your Icinga 2 notification events into Graylog and correlate events in defined streams.
Alerts can be defined based on existing streams. The idea is to check for a specific message count and apply threshold rules. Alerts can also be reset after a defined grace period. If you dig deeper, you’ll also recognise the alert notifications which could be Email or HTTP. We’ve also discussed an alert handling which connects to the Icinga 2 API similar to the Logstash Icinga output. Keep your fingers crossed.
Dashboards
You can add stream message counters, histograms and more to your own dashboards. Refresh settings and fullscreen mode are available too. You can export and share these dashboards. If you are looking for automated deployments, those dashboards can be imported via the REST API too.
Roadmap
Graylog 2.3 is currently in alpha stages and will be released in Summer 2017. We’ve also learned that it will introduce Elasticsearch 5 as backend. This enables Graylog to use the HTTP API instead of „simulating“ a cluster node at the moment. The upcoming release also adds support for lookup tables.
Try it
I’ve been fixing a bug inside the Icinga 2 GelfWriter feature lately and was looking for a quick test environment. Turns out that the Graylog project offers Docker compose scripts to bring up a fully running instance. I’ve slightly modified the docker-compose.yml to export the default GELF TCP input port 12201 on localhost.
vim docker-compose.yml
version: '2'
services:
mongo:
image: "mongo:3"
elasticsearch:
image: "elasticsearch:2"
command: "elasticsearch -Des.cluster.name='graylog'"
graylog:
image: graylog2/server:2.2.1-1
environment:
GRAYLOG_PASSWORD_SECRET: somepasswordpepper
GRAYLOG_ROOT_PASSWORD_SHA2: 8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
GRAYLOG_WEB_ENDPOINT_URI:
depends_on:
- mongo
- elasticsearch
ports:
- "9000:9000"
- "12201:12201"
docker-compose up
Navigate to http://localhost:9000/system/inputs (admin/admin) and add additional inputs, like „Gelf TCP“.
I just enabled the „gelf“ feature in Icinga 2 and pointed it to port 12201. As you can see, there’s some data running into. All screenshots above have been taken from that demo too 😉
More?
Jan continued the week with our official two day Graylog training. From a personal view I am really happy to welcome Graylog to our technology stack. I’ve been talking about Graylog with Bernd Ahlers at OSMC 2014 and now am even more excited about the newest additions in v2.x. Hopefully Jan joins us for OSMC 2017, Call for Papers is already open 🙂
My colleagues are already building Graylog clusters and more integrations. Get in touch if you need help with integrating Graylog into your infrastructure stack 🙂
