OSMC 2021 | Observability will not fix your broken Monitoring , or Culture

OSMC 2021 has been over for almost three months now. It was a pretty interesting conference, and also my first one as a trainee at NETWAYS. The two-day conference including a workshop and hackathon was all about open source monitoring software like Icinga2, CheckMK or Prometheus.
Today I give you some insights about one of the talks:

About the Speaker

Kris Buytaert is a long time Linux and Open Source Consultant. He’s one of the instigators of the devops movement, currently working for Inuits. He is frequently speaking at or organizing different international conferences and has written about the same subjects in different Books, Papers and Articles. He spends most of his time working on bridging the gap between developers and operations with a strong focus on High Availability, Scalability, Virtualisation and Large Infrastructure Management projects hence trying to build infrastructures that can survive the 10th floor test, better known today as the cloud while actively promoting the devops idea!

The difference between Monitoring and Observability

Plenty of companies are jumping on a new hype train, Observability. The hope that is associated with the new technology: a replacement of the “legacy” monitoring stack.
In his talk Kris Buytaert took a look at this development and pointed out why it is a wrong approach for a lot of people. To understand the problems that could come up with the implementation of Observability, he first introduced the differences between Monitoring and Obervability.

What is Monitoring?

So what exactly is monitoring? Summarizied in one question: what is going on in my system?
With this question in mind, experts look at a system to figure out the following points (and some more):

– High level overview of the state of a service or component
– Availability of this services
– Technical components of your setup
– What is the performance of my setup

What is Observability?

Observability on the other hand is trying to answer the question: Why is this going on?
For example, your services behave in a way that they shouldn’t because you or your colleagues didn’t tell the system to do this. With the help of Observability software you could figure out why your tech is behaving the way it is.
In practice Observability consists of three pilars:
– Metrics
– Logs
– Traces

Whats your goal in observability?

After the short introduction of the differences between monitoring and observability, Kris Buytaert showcased some examples of failed implementation of observability. What all of the situations have in common? There was no flawless instance of monitoring on which the observability was placed.

But why are an increasing number of companies who turn to observabilty and what’s the goal of of this development?

In his talk he listed some reasons he experienced himself when talking to companies and employees:

– Expectation of increasing performance problems
– Already existing performance problems
– The monitoring data is chaos, observability will fix it and give a better insight in what is running in the system
– More hipster credit (e.g. ‘Our company is so advanced. We take on every new trend to show how progressive we are.’)

The first steps of implementing observability

To end his talk, the speaker gave a few helpful first steps for everyone who is thining about implementing obervability.

1. Fix your monitoring. It can be an automated single source of truth about what is going on in your system.
2. Keep it GREEN!
3. Fix your metrics. Check if logshipping is partially broken, if theres a regression on shipping your metrics or if you have a broken dashboard.

Apart form the technical standpoint the speaker encourages everyone to ask some important questions before the implementation of a new system:

– Who exactly wants Observability? Is it the devs and ops or is it the upper management who insists on trying out something new.
– What do they really want? Do they really want Observability or are there other reasons they want it in your system.
– Will a new system fit into my existing ecosystem? If a vendor claims it works out of the box for everyone but your developers say it doesn’t…Trust your devs!

As the last words in his talk Kris Buytaert had some advice: ‘You might not need Observability (yet) but you do need to fix your monitoring.’

 

Full talk and more from and about OSMC 2021

Watch the whole talk by Kris Buytaert here:

 

Since OSMC 2021 is unfortunately over we still have something for you: Did you already check out this year’s conference archives? They provide you slides and videos of each talk and also some photographs of the conference itself.

OSMC 2022 will take place from November 14 – 16 and we’re already looking forward  to meeting you all again!

Stay tuned!