Select Page

OSMC 2022 | What’s new in the Prometheus ecosystem?

by | Dec 13, 2022 | Events, OSMC

About a month ago OSMC took place and I was happy to attend an offline conference again. One of the things I enjoy most at conferences is networking and this is why I know Julien Pivotto already for years. He focused his work a while ago on Prometheus and is a regular speaker at OSMC, typically not only giving a talk but also a workshop. And now he is also a maintainer of Prometheus. So who could be a better speaker to give an update talk summarizing the news from the last year?

Julien started with a short introduction to Prometheus which I want to repeat here as some readers will also likely not be familiar with the tool. Prometheus is an Open Source monitoring solution based on metrics. It provides a Service Discovery to find metric endpoints to pull data from. PromQL – the Prometheus Query Language – is then used to query and visualize data in the frontend. The project while sometimes still called a new competitor in the field of monitoring solutions has already some history and is a graduated CNCF project since 2018, so we do talk about a project that already left its footprints in the monitoring ecosystem.

Helpful when Exploring Metrics

The first topic Julien covered was the new react UI providing a query editor for PromQL including autocompletion and formatting. The metrics explorer now also supports using the local timezone instead of UTC which can be quite helpful when debugging as it removes the need to calculate this. Also very helpful is the option to filter targets and service discovery to reduce the amount of shown data. The option to show and explain CLI flags helps to tailor your installation to your needs. And as every software it now has to provide also a dark mode!

The next feature he introduced was the agent mode allowing to have an instance which only sends metrics and does not provide all the overhead like the UI if not needed. I would say the remote write receiver is a corresponding feature as it allows for federation of writes by rules as another system tailored to your needs.

The list of service discoveries increased by many new cloud and configuration management solutions, but the one to highlight is the Generic HTTP which allows for the most common and easy to implement one, a REST API with JSON body.

Relabeling capabilities are quite nice and simple ones like uppercase or lowercase are a common use case. But setting some specific labels also allows to do some scraping configuration like changing the interval.

Additions to PromQL

The additions to PromQL were a bigger part of Julien’s demo. One of the many new functions to mention is last_over_time which fills the graph with the last value coming in handy if a target does not provide metrics regularly. The option to have a negative offset is also great to mitigate errors in this case if a system was not in sync and did send with an offset of some minutes. But the most important new function is the new modifiers allowing you to get for example the top 5 at a specific time and then get the values for this specific metrics. This allows to visualize how the top 5 developed over time instead of simply showing the top 5 without how they have changed over time. By doing so you get much nicer and informative graphs.

Julien Pivotto doing his talk

Out of time metrics support allows to fill old metrics in the configured interval, so you can allow this temporarily to fill in gaps if needed by adding a timestamp to the metric.

The new LTS version allows the users to have 6+ month stable APIs while the developers can improve and release still every 6 weeks. Julien showed also how prominent they show the LTS version on the download page and documentation. This is something other projects could learn from. Talking about versioning they also synchronized the end user and go module’s minor version to show the compatibility.

Only the Required Features Set

The developers also introduced a plugin system which allows to build Prometheus with only the required feature set like only the needed service discoveries to reduce the size of the binary.

Also the Alertmanager got some new features. There is the option to mute alerts based on time for receivers allowing for example to honor on-duty and holidays. Negative matchers allow for a simpler ruleset and the new receivers provide more ways of notification.

Exporters in general got systemd socket activation and can now bind to multiple addresses. The node exporter providing information about a system got more collectors and the ones for MySQL and PostgreSQL can now handle multiple targets allowing to have only one for multiple databases reducing the amount of installations.

The Prometheus ecosystem also got a new Rust client , 30+ repositories in the prometheus-community organization, a certification program to prove your Prometheus skills and a conformance program for vendors to show compatibility of their software.

Introducing PromLens

And last but not least Julien introduced PromLens, a new separate Querybuilder with a great Explain functionality detailing how your query functioned and how many results are found. While planned as a separate tool a subset of the features should also be integrated in the normal UI to improve the user experience.

PromLens (taken from

As Julien said, it was a great amount of topics to cover and I tried to summarize further. Please watch the recording of his talk in our OSMC Archives for more details. You will get some practical examples when he demos most of the topics mentioned above as he did a great job. If you are new to Prometheus we also have you covered with our NETWAYS training course “Prometheus Fundamentals”. Check it out!

We hope to see you around at OSMC 2023! Stay in touch and subscribe to our Newsletter!

Dirk Götz
Dirk Götz
Principal Consultant

Dirk ist Red Hat Spezialist und arbeitet bei NETWAYS im Bereich Consulting für Icinga, Puppet, Ansible, Foreman und andere Systems-Management-Lösungen. Früher war er bei einem Träger der gesetzlichen Rentenversicherung als Senior Administrator beschäftigt und auch für die Ausbildung der Azubis verantwortlich wie nun bei NETWAYS.


Submit a Comment

Your email address will not be published. Required fields are marked *

More posts on the topic Events | OSMC

stackconf 2024 | Highlights from Day 1

When our Event team asked for some to join stackconf this year, I happily volunteered and so it is me again writing our conference blog. I think the last time when I joined our conference in Berlin it was still its predecessor, so it's my first stackconf. But if you...

stackconf 2024 | Here We Go!

After many weeks of intense planning, meticulous organization, and countless meetings, all preparations are now complete. Tomorrow, we will finally open the doors to stackconf 2024!   Welcome to Berlin As every year, many of you have already arrived today to...

Icinga Summit 2024: So war’s!

Letzte Woche war es endlich so weit - Icinga Summit. Für mich war es dieses Jahr das erste Mal auf einer von Icinga direkt ausgetragenen Veranstaltung und ich muss sagen, dass das Format nicht nur bei mir sehr gut ankam. In der Vergangenheit haben Events in Form der...