- OSMC 2021 | Introduction into OpenSearch
- OSMC 2021 | pg_stat_monitor: A cool extension for better database (PostgreSQL) monitoring
- OSMC 2021 | On the Bleeding Edge of OpenTelemetry
- OSMC 2021 | Thola – A tool for monitoring and provisioning network devices
- OSMC 2021 | Advanced MySQL optimization and troubleshooting using PMM 2
- OSMC 2021 | Observability will not fix your broken Monitoring , or Culture
- OSMC 2021 | Monitoring Open Infrastructure Logs – With Real Life Examples
- OSMC 2021 | Scaling Naemon deployments to Kubernetes with Merlin
- OSMC 2021 | Open Source API-HUB – Connect Icinga2, Zabbix, CheckMK and more with OpenCelium
- OSMC 2021 | Icinga for Windows – Evolution
- OSMC 2021 | inspectIT Ocelot: Dynamic OpenTelemetry Instrumentation at Runtime
- OSMC 2021 | Still directing the director… and more!
- OSMC 2021 | Current State of Icinga
- OSMC 2021 | Icinga-Installer – Der einfache Weg zum eigenen Icinga
- OSMC 2021 | Open Source Application Performance Monitoring in the Enterprise
- OSMC 2021 | Use OpenSource monitoring for an Enterprise Grade Platform
- OSMC 2021 | Contributing to Open Source with the example of Icinga
- OSMC 2021 | Handling 250K flows per second with OpenNMS: a case study
- OSMC 2021 | Robotmk: You don’t run IT – you deliver services!
- OSMC 2021 | Monitoring Open Source Hardware
- OSMC 2021 | Secure Password Vaults with Naemon
- OSMC 2021 | Gamification of Observability
- OSMC 2021 | Observability is More than Logs, Metrics & Traces
Last November OSMC 2021 took place. With me being part of NETWAYS since September 2021 it also has been my first OSMC. In the heart of Open Source many speakers talked about their experiences, shared knowledge and showcased what can be done in the field of monitoring using Open Source.
I have already given a little insight on another talk and would like to do so again.
Logs are a necessity
When it comes to IT infrastructure logs are an essential tool for us humans to understand what the machines are doing. They can tell you a lot about workloads, potential scaling needs and current problems.
Without logs we would not be able to „easily“ adjust to changing circumstances and ever-growing demands.
That is where Marcelo Perazolo (IBM) comes in with his talk to tell us about Open Infrastructure Logs specifically.
Let me give you a brief overview about his talk.
Open Infrastructure Foundation / OpenInfra
The Open Infrastructure Foundation (OIF) was initially founded in 2012 to guide and lead the OpenStack project (called the OpenStack Foundation back then). OpenStack itself was started by NASA and Rackspace Technology in 2010 but has been joined by many companies by now.
Today Openstack is maintained by the OIF and the community revolving around OpenStack.
According to the OIF Open Infrastructure means the use of open alternatives for various forms of compute storage and networking. The different open source components should fit together to provide a flexible, scalable and ready to use environment that companies can rely on.
Elastic Stack Components
In his example Perazolo depicts a cluster consisting of three nodes, each of which has a controller that sends data to its own Logstash which in turn sends data to Elasticsearch.
There is also one Kibana instance per node. If an instance goes down however HAProxy will point to another instance ensuring that operations can still continue.
Requirements for log monitoring
Any log monitoring solution should provide the following:
- must be able to collect millions of logs per day from different sources
- ability to filter and streamline logs according to specific criteria
- capability to visualize large quantities of data in a simple way
- a way to essentially „zoom“ into data changing from a simple overview to a close up view of specific data
The aforementioned Elastic Stack consisting of
- Filebeat (for collecting logs)
- Logstash (for filtering and streamlining data)
- Elasticsearch (for high performance storage/querying)
- Kibana (for flexible visualizations)
is one possible solution.
After Perazolo talked about the basic requirements and the chosen solution he goes on to explain best practices for collecting and transforming logs and also suggests a way on how to handle constantly growing amounts of logs.
If you would like to know more now, feel free to watch his talk and listen to Marcelo Perazolo for yourself.
Full talk and more from and about OSMC 2021
Watch the whole talk by Marcelo Perazolo here:
Since OSMC 2021 is unfortunately over we still have something for you: Did you already check out this year’s conference archives? They provide you slides and videos of each talk and also some photographs of the conference itself.
OSMC 2022 will take place from November 14 – 16 and we’re already looking forward to meeting you all again!