pixel
Select Page

OSMC 2021 | Monitoring Open Infrastructure Logs – With Real Life Examples

by | Feb 15, 2022 | OSMC

This entry is part 6 of 23 in the series OSMC 2021

Last November OSMC 2021 took place. With me being part of NETWAYS since September 2021 it also has been my first OSMC. In the heart of Open Source many speakers talked about their experiences, shared knowledge and showcased what can be done in the field of monitoring using Open Source.
I have already given a little insight on another talk and would like to do so again.

Logs are a necessity

When it comes to IT infrastructure logs are an essential tool for us humans to understand what the machines are doing. They can tell you a lot about workloads, potential scaling needs and current problems.
Without logs we would not be able to “easily” adjust to changing circumstances and ever-growing demands.

That is where Marcelo Perazolo (IBM) comes in with his talk to tell us about Open Infrastructure Logs specifically.
Let me give you a brief overview about his talk.

 

Open Infrastructure Foundation / OpenInfra

The Open Infrastructure Foundation (OIF) was initially founded in 2012 to guide and lead the OpenStack project (called the OpenStack Foundation back then). OpenStack itself was started by NASA and Rackspace Technology in 2010 but has been joined by many companies by now.

Today Openstack is maintained by the OIF and the community revolving around OpenStack.

According to the OIF Open Infrastructure means the use of open alternatives for various forms of compute storage and networking. The different open source components should fit together to provide a flexible, scalable and ready to use environment that companies can rely on.

 

Elastic Stack Components

In his example Perazolo depicts a cluster consisting of three nodes, each of which has a controller that sends data to its own Logstash which in turn sends data to Elasticsearch.
There is also one Kibana instance per node. If an instance goes down however HAProxy will point to another instance ensuring that operations can still continue.

 

Requirements for log monitoring

Any log monitoring solution should provide the following:

  • must be able to collect millions of logs per day from different sources
  • ability to filter and streamline logs according to specific criteria
  • capability to visualize large quantities of data in a simple way
  • a way to essentially “zoom” into data changing from a simple overview to a close up view of specific data

The aforementioned Elastic Stack consisting of

  • Filebeat (for collecting logs)
  • Logstash (for filtering and streamlining data)
  • Elasticsearch (for high performance storage/querying)
  • Kibana (for flexible visualizations)

is one possible solution.

 

After Perazolo talked about the basic requirements and the chosen solution he goes on to explain best practices for collecting and transforming logs and also suggests a way on how to handle constantly growing amounts of logs.

If you would like to know more now, feel free to watch his talk and listen to Marcelo Perazolo for yourself.

Full talk and more from and about OSMC 2021

Watch the whole talk by Marcelo Perazolo here:

https://www.youtube.com/watch?enablejsapi=1&autoplay=0&cc_load_policy=0&cc_lang_pref=&iv_load_policy=1&loop=0&modestbranding=1&rel=1&fs=1&playsinline=0&autohide=2&hl=en_US&theme=dark&color=red&controls=1&" class="__youtube_prefs__ epyt-facade no-lazyload" data-epautoplay="1" >YouTube playerhttps://www.youtube.com/watch/maxresdefault.jpg" />

 

Since OSMC 2021 is unfortunately over we still have something for you: Did you already check out this year’s conference archives? They provide you slides and videos of each talk and also some photographs of the conference itself.

OSMC 2022 will take place from November 14 – 16 and we’re already looking forward  to meeting you all again!

Stay tuned!

Matthias Döhler
Matthias Döhler
Junior Consultant

Über ein paar Umwege ist Matthias nun endlich da gelandet, wo er sich wohl fühlt: in der IT! Bei NETWAYS hat er im September 2021 seine Ausbildung zum Fachinformatiker für Systemintegration im Bereich Professional Services begonnen. Wenn er sich zu Hause nicht auch noch mit Themen rund um Linux auseinandersetzt, sieht er sich leidenschaftlich gerne Horrorfilme und solche an, die man als "Trash" bezeichnen könnte. Je seltsamer, desto besser! Den üblichen Beschäftigungen wie Freunde treffen, Bars aufsuchen oder die Sonne im Freien genießen, geht er eben so nach wie pseudophilosophischen Fragen. Daneben spielt er außerdem wahnsinnig gerne Videospiele vergangener Generationen....

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

More posts on the topic OSMC

OSMC 2023 | Save your Workshop Ticket!

Imagine a day filled with expert guidance, hands-on learning, and the opportunity to connect with like-minded individuals who share your passion. Our upcoming monitoring workshops are designed to do just that and more. Join us for deep insights, hands-on learning, and...

OSMC 2023 | Dive into our Conference Timetable

As OSMC 2023 draws nearer, we are thrilled to reveal that the conference program has been finalized! Anticipate two days filled with a grand total of 34 lecture sessions conducted by renowned experts in the open source field. This year’s edition of the event is...

OSMC 2023 | The Line-Up is Set!

Our First Speakers Since our Call for Papers ran out last week, we’re super happy to already announce some of our first confirmed OSMC speakers today! We have an incredible line-up with many international top-level speakers talking about the latest trends in open...

Trainings

Web Services

Events

Series



Other posts in series: