OSMC 2021 | Monitoring Open Infrastructure Logs – With Real Life Examples

Last November OSMC 2021 took place. With me being part of NETWAYS since September 2021 it also has been my first OSMC. In the heart of Open Source many speakers talked about their experiences, shared knowledge and showcased what can be done in the field of monitoring using Open Source.
I have already given a little insight on another talk and would like to do so again.

Logs are a necessity

When it comes to IT infrastructure logs are an essential tool for us humans to understand what the machines are doing. They can tell you a lot about workloads, potential scaling needs and current problems.
Without logs we would not be able to “easily” adjust to changing circumstances and ever-growing demands.

That is where Marcelo Perazolo (IBM) comes in with his talk to tell us about Open Infrastructure Logs specifically.
Let me give you a brief overview about his talk.

Open Infrastructure Foundation / OpenInfra

The Open Infrastructure Foundation (OIF) was initially founded in 2012 to guide and lead the OpenStack project (called the OpenStack Foundation back then). OpenStack itself was started by NASA and Rackspace Technology in 2010 but has been joined by many companies by now.

Today Openstack is maintained by the OIF and the community revolving around OpenStack.

According to the OIF Open Infrastructure means the use of open alternatives for various forms of compute storage and networking. The different open source components should fit together to provide a flexible, scalable and ready to use environment that companies can rely on.

Elastic Stack Components

In his example Perazolo depicts a cluster consisting of three nodes, each of which has a controller that sends data to its own Logstash which in turn sends data to Elasticsearch.
There is also one Kibana instance per node. If an instance goes down however HAProxy will point to another instance ensuring that operations can still continue.

Requirements for log monitoring

Any log monitoring solution should provide the following:

• must be able to collect millions of logs per day from different sources
• ability to filter and streamline logs according to specific criteria
• capability to visualize large quantities of data in a simple way
• a way to essentially “zoom” into data changing from a simple overview to a close up view of specific data

The aforementioned Elastic Stack consisting of

• Filebeat (for collecting logs)
• Logstash (for filtering and streamlining data)
• Elasticsearch (for high performance storage/querying)
• Kibana (for flexible visualizations)

is one possible solution.

After Perazolo talked about the basic requirements and the chosen solution he goes on to explain best practices for collecting and transforming logs and also suggests a way on how to handle constantly growing amounts of logs.

If you would like to know more now, feel free to watch his talk and listen to Marcelo Perazolo for yourself.

Full talk and more from and about OSMC 2021

Watch the whole talk by Marcelo Perazolo here:

https://www.youtube.com/watch?enablejsapi=1&autoplay=0&cc_load_policy=0&cc_lang_pref=&iv_load_policy=1&loop=0&modestbranding=1&rel=1&fs=1&playsinline=0&autohide=2&hl=en_US&theme=dark&color=red&controls=1&" class="__youtube_prefs__ epyt-facade no-lazyload" data-epautoplay="1" >https://www.youtube.com/watch/maxresdefault.jpg" />

Since OSMC 2021 is unfortunately over we still have something for you: Did you already check out this year’s conference archives? They provide you slides and videos of each talk and also some photographs of the conference itself.

OSMC 2022 will take place from November 14 – 16 and we’re already looking forward  to meeting you all again!

Stay tuned!

Matthias Döhler

Junior Consultant

---

Über ein paar Umwege ist Matthias nun endlich da gelandet, wo er sich wohl fühlt: in der IT! Bei NETWAYS hat er im September 2021 seine Ausbildung zum Fachinformatiker für Systemintegration im Bereich Professional Services begonnen. Wenn er sich zu Hause nicht auch noch mit Themen rund um Linux auseinandersetzt, sieht er sich leidenschaftlich gerne Horrorfilme und solche an, die man als "Trash" bezeichnen könnte. Je seltsamer, desto besser! Den üblichen Beschäftigungen wie Freunde treffen, Bars aufsuchen oder die Sonne im Freien genießen, geht er eben so nach wie pseudophilosophischen Fragen. Daneben spielt er außerdem wahnsinnig gerne Videospiele vergangener Generationen....

Read more from Matthias and meet the Team