Seite wählen

OSMC 2021 | Monitoring Open Infrastructure Logs – With Real Life Examples

von | Feb 15, 2022 | OSMC

This entry is part 6 of 23 in the series OSMC 2021

Last November OSMC 2021 took place. With me being part of NETWAYS since September 2021 it also has been my first OSMC. In the heart of Open Source many speakers talked about their experiences, shared knowledge and showcased what can be done in the field of monitoring using Open Source.
I have already given a little insight on another talk and would like to do so again.

Logs are a necessity

When it comes to IT infrastructure logs are an essential tool for us humans to understand what the machines are doing. They can tell you a lot about workloads, potential scaling needs and current problems.
Without logs we would not be able to „easily“ adjust to changing circumstances and ever-growing demands.

That is where Marcelo Perazolo (IBM) comes in with his talk to tell us about Open Infrastructure Logs specifically.
Let me give you a brief overview about his talk.


Open Infrastructure Foundation / OpenInfra

The Open Infrastructure Foundation (OIF) was initially founded in 2012 to guide and lead the OpenStack project (called the OpenStack Foundation back then). OpenStack itself was started by NASA and Rackspace Technology in 2010 but has been joined by many companies by now.

Today Openstack is maintained by the OIF and the community revolving around OpenStack.

According to the OIF Open Infrastructure means the use of open alternatives for various forms of compute storage and networking. The different open source components should fit together to provide a flexible, scalable and ready to use environment that companies can rely on.


Elastic Stack Components

In his example Perazolo depicts a cluster consisting of three nodes, each of which has a controller that sends data to its own Logstash which in turn sends data to Elasticsearch.
There is also one Kibana instance per node. If an instance goes down however HAProxy will point to another instance ensuring that operations can still continue.


Requirements for log monitoring

Any log monitoring solution should provide the following:

  • must be able to collect millions of logs per day from different sources
  • ability to filter and streamline logs according to specific criteria
  • capability to visualize large quantities of data in a simple way
  • a way to essentially „zoom“ into data changing from a simple overview to a close up view of specific data

The aforementioned Elastic Stack consisting of

  • Filebeat (for collecting logs)
  • Logstash (for filtering and streamlining data)
  • Elasticsearch (for high performance storage/querying)
  • Kibana (for flexible visualizations)

is one possible solution.


After Perazolo talked about the basic requirements and the chosen solution he goes on to explain best practices for collecting and transforming logs and also suggests a way on how to handle constantly growing amounts of logs.

If you would like to know more now, feel free to watch his talk and listen to Marcelo Perazolo for yourself.

Full talk and more from and about OSMC 2021

Watch the whole talk by Marcelo Perazolo here:

https://www.youtube.com/watch?enablejsapi=1&autoplay=0&cc_load_policy=0&cc_lang_pref=&iv_load_policy=1&loop=0&modestbranding=1&rel=1&fs=1&playsinline=0&autohide=2&hl=de_DE&theme=dark&color=red&controls=1&" class="__youtube_prefs__ epyt-facade no-lazyload" data-epautoplay="1" >YouTube playerhttps://www.youtube.com/watch/maxresdefault.jpg" />


Since OSMC 2021 is unfortunately over we still have something for you: Did you already check out this year’s conference archives? They provide you slides and videos of each talk and also some photographs of the conference itself.

OSMC 2022 will take place from November 14 – 16 and we’re already looking forward  to meeting you all again!

Stay tuned!

Matthias Döhler
Matthias Döhler
Junior Consultant

Über ein paar Umwege ist Matthias nun endlich da gelandet, wo er sich wohl fühlt: in der IT! Bei NETWAYS hat er im September 2021 seine Ausbildung zum Fachinformatiker für Systemintegration im Bereich Professional Services begonnen. Wenn er sich zu Hause nicht auch noch mit Themen rund um Linux auseinandersetzt, sieht er sich leidenschaftlich gerne Horrorfilme und solche an, die man als "Trash" bezeichnen könnte. Je seltsamer, desto besser! Den üblichen Beschäftigungen wie Freunde treffen, Bars aufsuchen oder die Sonne im Freien genießen, geht er eben so nach wie pseudophilosophischen Fragen. Daneben spielt er außerdem wahnsinnig gerne Videospiele vergangener Generationen....
Mehr Beiträge zum Thema OSMC

OSMC 2023 | Be a Sponsor – Fuel your Progress!

Meet your Business Partners Businesses have to deal with an increasing complexity of their IT infrastructure. Therefore it’s even more important to ensure availability of all components at any times. The Open Source Monitoring Conference (OSMC) is the leading event...

OSMC 2023 | Get on Stage!

Wohoo! We officially opened our Call for Papers for OSMC 2023! The three-day community event will take place from November 7 – 9 in Nuremberg. Now it’s your chance again to become a hero speaker and inspire our monitoring community with your expertise! This blogpost...


Web Services



Other posts in series: