Seite wählen

OSMC 2022 | Monitoring multiple Kubernetes Clusters with Thanos

von | Mrz 14, 2023 | OSMC

By now, Prometheus has become the defacto standard for monitoring containerised applications, especially when you are using orchestration tools like Kubernetes or Consul. However, when it comes to monitoring multiple Kubernetes clusters through a single plane of glass, additional tools are required. In his talk at the Open Source Monitoring Conference 2022 (OSMC), Pascal Fries showed how to set up a production monitoring landscape based on Prometheus and Thanos, spanning several Kubernetes clusters. Focussing on examples and best practices. He also elaborates on how to securely communicate between the individual components.

Pascal Fries works as a IT Consultant at the ATIX AG in Garching (near Munich) in Germany. As a specialist for Cloud Native technologies, among his main fields of expertise are configuration and administration of multi cluster Kubernetes environments. Including their monitoring with Prometheus and Thanos.

As an introduction, Paul asked the audience if they were using Kubernetes (in a single or multi cluster setup) or Prometheus. Prometheus works very well at monitoring multiple Kubernetes Clusters. But there are several problem areas that you should be aware of, like long term storage, high availability and redundancy. To illustrate this with an example, he mentioned a customer story about Kubernetes usage where multiple teams use Kubernetes, in sum over 20 clusters. There are shared, managed and owned clusters all together in that environment, and all teams need to have a single endpoint for getting their metrics. Other requirements are long term storage, high availability and push based monitoring. How can we make yure to meet all there requirements with our monitoring setup?

Thanos, a storage layer for Prometheus

To solve these requirements, he introduced us to Thanos. It is basically a storage layer for Prometheus.










To set it up, you can use sample configurations. Its also more secure, as gRPC and TLS auth are being used instead of REST and basic authentication. When Thanos is communicating with Thanos, they use TLS auth – otherwise of course basic auth.

The long term storage is realized by saving the data in an S3, which can be self hosted or on a cloud service like NETWAYS Web Services. For realizing a push-based monitoring, Thanos can act as a receiver and compactor. The receiver is a short term DB and shards/replicates the timeseries by labels. Unfortunately, the results are duplicates in the S3. The compactor on the other side deduplicates in the S3, and downsamples the data for faster queries.

All in all, Thanos is a great tool for a redundant and highly available long term storage for Prometheus Monitoring.

And the OSMC is a great conference with many interesting talks every year! Especially if you want to learn more about everything related to Open Source Monitoring. There were talks about monitoring, automation and open source in general. And many interesting talks with the attendees. Hearing and discussing the different opinions and use cases of and about technology is exciting for me. I have been working at NETWAYS as a Junior Consultant for 2 years now, and it was the second OSMC. I am looking forward to next year already!

The recording and slides of this talk and all other OSMC talks can be found in our Archives. The next OSMC takes place from November 7 – 9, 2023 in Nuremberg. Early Bird tickets are already on sale!

Björn Berg
Björn Berg
Junior Consultant

Björn hat nach seinem Abitur 2019 Datenschutz und IT-Sicherheit in Ansbach studiert. Nach einigen Semestern entschied er sich auf eine Ausbildung zum Fachinformatiker für Systemintegration umzusteigen und fing im September 2021 bei NETWAYS Professional Services an. Auch in seiner Freizeit sitzt er viel vor seinem PC und hat Spaß mit diversen Spielen, experimentiert auch mit verschiedenen Linux-Distributionen herum und geht im Sommer gerne mal campen.

0 Kommentare

Einen Kommentar abschicken

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert

Mehr Beiträge zum Thema OSMC

OSMC 2023 | Save your Workshop Ticket!

Imagine a day filled with expert guidance, hands-on learning, and the opportunity to connect with like-minded individuals who share your passion. Our upcoming monitoring workshops are designed to do just that and more. Join us for deep insights, hands-on learning, and...

OSMC 2023 | Dive into our Conference Timetable

As OSMC 2023 draws nearer, we are thrilled to reveal that the conference program has been finalized! Anticipate two days filled with a grand total of 34 lecture sessions conducted by renowned experts in the open source field. This year’s edition of the event is...

OSMC 2023 | The Line-Up is Set!

Our First Speakers Since our Call for Papers ran out last week, we’re super happy to already announce some of our first confirmed OSMC speakers today! We have an incredible line-up with many international top-level speakers talking about the latest trends in open...