Einfacher als gedacht: Openstack + Vagrant

Vagrant-LogoOpenstack-LogoVermutlich haben sich bei fast allen durch Covid-19 die Prioritäten verschoben, bei mir kam sehr schnell zu dem Thema “Ausbildung trotz 100% Home-Office aufrecht erhalten” noch ein weiteres dazu. Und zwar das Thema Online-Schulungen bzw. genauer die technische Plattform dafür, denn um andere Aspekte wie die Kommunikations- und Präsentationssoftware, Organisation, Marketing und ähnliches haben sich dankenswerterweise andere Kollegen gekümmert.

Die zu nutzende Infrastruktur sollte logischerweise die Openstack-Plattform von NWS sein, da hier entsprechende Kapazitäten vorhanden sind, somit hatte ich mein Ziel. Die Ausgangsbasis war auch leicht definiert, da fast alle Schulungen zum Erstellen der virtuellen Maschinen für die praktischen Übungen auf Vagrant setzen. Die somit erstellten Systeme werden dann zwar noch exportiert um die Provisionierung der Schulungsnotebooks zu beschleunigen, aber sollten trotzdem einen guten Start darstellen.

Als erstes musste natürlich dann der Provider installiert werden, da ich Fedora als Betriebssystem nutze, ging das mit dnf install vagrant-openstack-provider. Für die anderen Linux-Systeme habe ich keine Pakete gefunden, daher wird der Rest wohl auf den Plugin-Mechanismus mit vagrant plugin install vagrant-openstack-provider zurückgreifen müssen. Wichtig ist dann noch zu wissen, ob es der einzige oder Standard-Provider ist, denn sonst darf man später beim Starten der Systeme mit vagrant up nicht den entsprechender Parameter --provider=openstack vergessen.

Natürlich stand als erstes aber eine Schulung auf dem Plan, die normalerweise ohne virtuelle Maschine auskommt, was aber nicht so schlimm war, denn es gab mir mehr Freiheit zum Experimentieren und Kennenlernen der Unterschiede zwischen Openstack-Provider und den von mir gewohnten. Der vielleicht größte Unterschied ist, dass hier nicht mit bestehenden Boxen von einem zentralen System wie app.vagrantup.com gearbeitet wird, sondern mit bereits in Openstack vorhandenen Images. Außerdem muss eine Anmeldung an einem Openstack-Projekt erfolgen, in dem dann nicht nur die Images vorbereitet sein müssen, sondern auch die Netzwerkkonfiguration inklusive der Sicherheitsgruppen mit entsprechender Portfreigabe und ein zu verwendender SSH-Key.
(mehr …)

Dirk Götz
Dirk Götz
Principal Consultant

Dirk ist Red Hat Spezialist und arbeitet bei NETWAYS im Bereich Consulting für Icinga, Puppet, Ansible, Foreman und andere Systems-Management-Lösungen. Früher war er bei einem Träger der gesetzlichen Rentenversicherung als Senior Administrator beschäftigt und auch für die Ausbildung der Azubis verantwortlich wie nun bei NETWAYS.

Config Management Camp Ghent 2020 – Recap

Cfgmgmtcamp Logo

It seams like Config Management Camp at beginning of February in Ghent gets a fixed date for me. I attended the fifth year in series, gave a talk in the last three and joined the Foreman Construction day on the day after as many times. So why am I still attending while some people are perhaps already telling you that the time of configuration management is over in favor of containers and Kubernetes. While I can not totally agree or disagree with this thesis, my schedule is still full of Foreman, Puppet and Ansible, so it makes sense to keep me updated. Furthermore the event allows to network like not many others with speaker diner, community event (also known as beer event) and Foreman community dinner. And last but not least it is always interesting to hear what the big names think about configuration management in the future and how to adopt to a world of containers and Kubernetes which was a big part of the talks in the main track.

But to get everything in correct order let me start at Sunday morning where Blerim, Aleksander and I started so we can meet Bernd and Markus who attended FOSDEM in advance in our AirBnb before going to speaker diner. I have to admit I really like Ghent’s old City so I was happy the same restaurant right in the middle of Ghent was chosen for diner like last year. And also like last year I joined the Foreman table to meet old and new friends for some hours mixed of small talk and technical discussions.

The first conference day started as always with main tracks only and I can really recommend Ryn Daniels’ talk Untitled Config Game. After lunch I joined the Foreman community room to get latest news from the community and the 2.0 release by Tomer Brisker and Ewoud Kohl van Wijngaarden respectively. The talks about Katello and how to create API and CLI for a Compute Resource where also quite interesting, but my favorite was Marek Hulan who had initially chosen a very similar title for his talk about Foreman’s new Reporting Engine and showed some interesting examples and the future templates documentation which will be automatically rendered in a similar fashion like the API documentation which is always available at /apidoc on a Foreman installation. Last but hopefully not least was my talk about existing solutions which get data from Foreman into central systems like Elasticsearch for logs and Supervisor Authority Plugin which enables Elastic APM to show performance bottlenecks, stacktraces and some metrics and is perhaps the most promising solution for me. As I was the one between the audience and the beer I was quite happy finishing my talk in time and get some more Kriek afterwards.

Day two had also some create talks to start with John Willis telling us he got 99 (or perhaps even far more) problems and a bash DSL ain’t one of them and Bernd how convenience is killing Open Standards. The first was really great in showing how configuration management has evolved compared to the container world which follows the same evolutionary process. The second was not only related to configuration management but IT at all including clouds and many more (and Bernd was fully aware of the discrepancy giving such a talk on a macbook). This day I visited more different tracks to hear about the migration of Pulp 2 to 3 behind Katello, testing of Ansible roles with Molecule (including some chemistry lessons), Ansible modules for pulp, how Foreman handles Secure boot and last but not least to get an update on Mgmt Config. After the talks we joined the Foreman Community diner which was located in a separate room of the same location we visited last year, allowing even more discussions without fearing to disturb others.

The Foreman Construction day like many other community events is a fringe event at the same location allow to hack together on some features and I was happy to make the beginner session I had given already the last years an official workshop. It was based on our official training focusing on installation and provisioning including hints and answering questions. After lunch I joined the hacking session for some time before shopping some Kriek and waffles and then traveling home.

Dirk Götz
Dirk Götz
Principal Consultant

Dirk ist Red Hat Spezialist und arbeitet bei NETWAYS im Bereich Consulting für Icinga, Puppet, Ansible, Foreman und andere Systems-Management-Lösungen. Früher war er bei einem Träger der gesetzlichen Rentenversicherung als Senior Administrator beschäftigt und auch für die Ausbildung der Azubis verantwortlich wie nun bei NETWAYS.

OSMC 2019 – Day 2

OSMC Logo
The social event yesterday evening was a blast and the late lounge afterwards also a must. So while some were still recovering, the room for the first talk was already quite full. This showed the interest in Jochen Kressin‘s talk about “Zero Trusted Networks – why Perimeter Security is dead”. He explained the (old) assumption of perimeter security “I am behind a firewall, so my traffic is secure” and asked the question if this is still true. Showing examples proving it is not true anymore because if it would be, none of these data breaches would have been happen. He explained what has changed in the last years leading to “Zero Trusted Networks” where every system has to be treated as untrusted and how to adopt for it. As one of the developers he used Search Guard as example which adds security to Elasticsearch, one of the great tools that had no security by itself for a long time, being not ready for the zero trusted approach.
Zero Trusted NetworksFluentD
Toshaan Bravani was talking about “Monitoring your Logs with Fluent”. FluentD and the client component FluentBit is an alternative to Logstash I see more and more at customer environments, so I was happy to get a deeper look into it. In addition he showed the complete tool stack to get most out of your data and the automation used to get it up and running.

Open Source landscape for APM
Third one for today was “Improved Observability Using Automated, OpenCensus-based Application Monitoring Solutions” by Tobias Angerstein. He started with a nice overview of the Open Source landscape for Application Performance Management before focusing on inspectIT. Its latest incarnation inspectIT Ocelot focuses on Open Standards like Open Metrics, Open Tracing and Open Census which are forming a new one called Open Telemetry which allows integration with all the well-known tools like Telegraf, Prometheus, Grafana. It also provides End User Monitoring using Boomerang, a javascript agent, and an EUM Server which transforms data to the Open Standards. In his demo he showed the capability of it and my only thought was how helpful something like this would have been to me in my early IT days being a Java developer.

Afterwards we could enjoy another great lunch break and perhaps also an massage, before starting into the afternoon sessions.
Lunch breakLunch BreakLunch Break

Database observability
Charles Judith gave a talk about “How to improve database Observability”. In his job he is responsible for reliability of the company’s databases and told the crowd the problems he started with like having no backup and monitoring at all. So it was his personal goal to have no hidden issues anymore and get transparency into their environment. His way from zero to hero was quite interesting and he compared it with a roller coaster. In the end having metrics to tell users that they are right or wrong with their feeling of the database is slow and having logs and monitoring telling were the real problem lies instead of guessing has improved his daily work already. But he still has some more steps to do like publishing SLA. The WIP version of his toolkit can be found on Github.

Second last one I attended was Jan Doberstein with a non technical talk about behaviour and how it influences your daily life and work, titled “Idiot! – or: Why BOFH is toxic”. He touched the same topic like the open discussion yesterday and I think it is great to get people think about and reflect their behaviour. While most of his examples were matched to the crowd and perhaps people working in IT do communicate much more in electronic fashion than others, it is a topic that everyone should care about.

High available setup
Last but not least Marcel Weinberg showed the high available setup he built for Digital Ocean. He included some very helpful small tips and tricks to increase performance and avoid pitfalls while diving deep into the configuration. Indeed it were too much for me to list them all here.

Pictures are taken again from the OSMC stream at Twitter, thanks to everyone for sharing their impressions. I hope everyone enjoyed the conference like I did. Thanks to everyone who made OSMC such a great experience again this year, starting with my colleagues organizing the event, the sponsors and speakers but this includes every attendee forming this nice community. Save travels for everyone leaving today or see you tomorrow if you join the Hackathon or Open Source Camp on Foreman. I hope I will see everyone next year at the same place on November 16th to 19th for OSMC 2020 or in Amsterdam for IcingaConf on May 12th to 14th.

Dirk Götz
Dirk Götz
Principal Consultant

Dirk ist Red Hat Spezialist und arbeitet bei NETWAYS im Bereich Consulting für Icinga, Puppet, Ansible, Foreman und andere Systems-Management-Lösungen. Früher war er bei einem Träger der gesetzlichen Rentenversicherung als Senior Administrator beschäftigt und auch für die Ausbildung der Azubis verantwortlich wie nun bei NETWAYS.

OSMC 2019 – Day 1

OSMC Logo
As always OSMC started with the workshop day. This time the topics were Prometheus given by Julian Pivotto, Gitlab by Michael, Terraform by Lennart and Foreman by me. After the workshops with coming together for dinner and some drink social networking started, one of things I enjoy most at conferences nowadays.

Day one started also like every time with a warm welcome from Bernd. Quite unexpected for me was the high number of first time attendees who raised hands when Bernd asked. It is always great to see new faces!

Ansible module planned to be released
First talk I attended was “Directing the Director” by Martin Schurz who gave some insights in how the monitoring platform developed at T-Systems Multimedia Solutions GmbH over last years. So they scaled up from single system, solved migration from VMs to Docker of the monitored environment and built knowledge to provide consulting to the teams which run 94 different projects which have to be monitored. With so many different things to monitor the next step of course was automation where all the good from Icinga-Director-API and Ansible came together. But if it is easy for users to build monitoring objects, configuration will grow which comes with the next challenges to make it even more easy and error prove. And from what Martin showed they solved it in a good fashion, but future will tell and I hope he will give another talk providing an update in the future.

Crowded room at OSMC
Second one was Christian with “Windows: One Framework to Monitor them all” who introduced his precious to the crowed. If you could not make it into the crowded room or was not at OSMC at all, have a look at the documentation or the framework itself, the plugins, the kickstart script to get it up and running the background daemon. In a great live demo he showed all the components and explained them in depth. While it is still the first release candidate it looks very promising.

Marcelo Perazolo from IBM Systems was talking about “Monitoring Alerts and Metrics on Large Power Systems Clusters”. He started with an introduction to the Power architecture and the workloads it is specially useful to make everyone familiar with it. The example he used was a big one, the Summit supercomputer. The main topic were two projects CRASSD and Power-Ops which not bring the data from some systems not everyone is familiar with to commonly used tools but also include Ansible playbooks for automation and flexibility “instead of just providing a docker container”. The demo showed some Kibana dashboards which provided in-depth data summarizing the health and performance of Power Systems starting from firmware to service running on it.
Power architecture explainedDesert

Lunch was great as always (and not only the dessert) and to avoid food coma we had the first time ignite talks at OSMC. It started with one from the conference sponsor ilert. Afterwards Blerim talked about “How Observability is not killing Monitoring” where he concluded that Observability should be an addition to Monitoring and not a replacement. Toshaan Bharvani decided for an ignite when sitting in the talk about Power Systems and wanted to add about “Building your own Datacenter” based on OpenPower and software available for it. In his talk “Overengineering your personal website” Bram Vogelaar showed a good (and funny) example where adding things to your infrastructure can escalate.

New entry in the datalist
Marianne Spiller‘s talk “Lorem Icinga puppetdb director amet” was not only a must see because its creative title, but because I like the mix of humor and technical knowledge she always provides. With practical examples she showed the problems of manual work like lazy admins prefer introducing a “Not answered” to the datalist of operating systems instead of maintaining the information on hosts. So instead of a form manually filled by admins she ended using the import from PuppetDB to create monitoring objects based on facts of the system.

Grafana Loki demo
Directly from Grafana Labs represented by Ganesh Vernekar the audience got some news about Loki which is “Like Prometheus, but for Logs”. So Loki avoid huge indexes and allows for better scaling by not indexing log lines but grouping them to streams. Using a similar format to Prometheus it allows to get metrics and logs for a system without a context switch. Running Loki to get this seems quite simple and flexible with only one binary which can scale out easily and also allows for a microservice infrastructure.

First code improvement
“Fast Logs Ingestion” by Nicolas Fränkel showed common coding mistakes and how to avoid them to get better logging in your application. He also covered topics like metadata and searching logs which also should also influence decisions and code. Structured log data even not written to file are also a option to consider like config reload during runtime to enable the user to switch loglevel without downtime. In the end he hopes people take away from his talk that everyone from dev and ops to architects should keep in mind which trade-off between speed and reliability has to be done and why.

Like every year the “Current State of Icinga” by Bernd was held in front of full house. He started with a short introduction to Icinga including the workflow which results in “Icinga makes you happy”. Icinga Workflow Afterwards to start with technical things he looked into the big changes Icinga 2.11 brought with a new network stack, high availability for more features and a new process handling not only helping with containers. Icinga 2.7 brought more translations, markdown support, jQuery 3, modernized styling for forms and lists, color blind theme and improvements for module developers. The vSphere module provides now an Import Source for Director, no code depenency on the Director and some UI improvements. The latest version of Director has also more translation, support for scheduled downtimes and sync previews. The BP Modelling (formerly Business Process Modul) has now drag & drop, export and import and breadcrumbs to make the UI more usable. As the first new feature he introduced the Windows monitoring Christian gave a detailed talk earlier today. Icinga for AWS was an improvement to the one only providing a simple import source for Director which adds support for multiple sources, some property modifiers and sync previews. Icinga Module for Jira includes an Issue overview, Jira notification via Director integration and custom workflows you can create from Icinga Web 2. Icinga DB as replacement for IDO is decoupling status and historic data using Redis and in a demo the new monitoring module based on it was also shown including all the visual improvements. Pull requests are already merged and will be part of the next releases and new things are available separately. The next update you can get on IcingaConf in Amsterdam on May 12 – 14, 2020.

As last topic a open discussion about Code of Conducts suggested and moderated by Stefan Lange took place.

Pictures are taken from several twitter users tagging them with OSMC. Thanks for providing them, expect some better ones from my colleagues from the events and marketing teams. I hope you enjoyed my report for day 1 while I am heading over to the social event at the Loftwerk and try to have at least a short talk to everyone. Day 2 will be covered tomorrow evening.

Dirk Götz
Dirk Götz
Principal Consultant

Dirk ist Red Hat Spezialist und arbeitet bei NETWAYS im Bereich Consulting für Icinga, Puppet, Ansible, Foreman und andere Systems-Management-Lösungen. Früher war er bei einem Träger der gesetzlichen Rentenversicherung als Senior Administrator beschäftigt und auch für die Ausbildung der Azubis verantwortlich wie nun bei NETWAYS.

Einmal lokaler Mirror? Kommt sofort!

RPM Logo

Wer mich kennt, weiß dass ich gerne zu großen umfangreichen Lösungen neige. Daher ist meine bevorzugte Lösung für einen lokalen Mirror Katello, aber es gibt auch Situationen in denen man nur eine Version ohne Staging braucht. Beispiel aus dieser Woche ein “Icinga 2”-Satellite in China, der einfach nicht die Pakete von packages.icinga.com beziehen möchte. Auf seinem übergeordneten Satelliten in Singapur hat noch alles gut funktioniert und auch die Kommunikation zwischen beiden funktioniert auch gut. Also ist nach kurzer Überlegung der Plan gefasst, es soll ein lokaler Mirror her von dem in China installiert werden soll.

Um den Mirror aufzusetzen, setze ich auf die Kommandos reposync und createrepo, welche recht schnell installiert sind und keine Konfiguration benötigen.

yum install -y yum-utils createrepo

Mit reposync kann nur ein bereits konfiguriertes Repository gespiegelt werden. Da in diesem Fall auf beiden Systemen die gleiche Betriebssystemversion installiert ist, für mich kein Problem und es kann gleich weitergehen. Auch ist auf dem Satelliten bereits ein Webserver installiert um Icinga Web 2 als separates Webinterface für die asiatischen Kollegen anzubieten, also auch hier kein Handlungsbedarf. Der Mirror ist also schnell aufgesetzt.

mkdir -p /var/www/html/repo
reposync -r icinga-stable-release -p /var/www/html/repo/ -n
createrepo /var/www/html/repo/icinga-stable-release

Die Optionen bei reposync sind mit -r die Repository-ID aus der Yum-Konfiguration, -p das Zielverzeichnis und -n um nur die jeweils neuste Version herunterzuladen. reposync lädt allerdings nur die Pakete herunter und legt sie in der entsprechenden Struktur ab ohne die benötigten Metadaten. Diese werden dann mit createrepo erzeugt und schon kann mit der neu zur Verfügung gestellten URL das Repository eingebunden werden.

In vielen Fällen ist dies ausreichend, aber hier noch ein paar Tipps wenn es dann doch etwas mehr sein darf.

  • Zum regelmäßigen Updaten einfach die beiden Kommandos reposync und createrepo in einem Cronjob hinterlegen.
  • Ein Repository kann noch weitere Metadaten enthalten, beispielsweise die comps.xml mit Gruppeninformationen. Diese wird durch der Option --downloadcomps von reposync mit heruntergeladen und im aktuellen Arbeitsverzeichnis abgelegt. Bei createrepo wird diese wiederum mit -g comps.xml eingebunden.
  • Die Errata-Informationen können nicht mit reposync heruntergeladen werden, aber beispielsweise yum list-sec lädt diese lokal in den Cache. Kopiert man die updateinfo.xml dann aus dem Repository-Cache in /var/cache/yum/ in das synchronisierte Repository und führt modifyrepo /var/www/html/repo-id/repodata/updateinfo.xml /var/www/html/repo-id/repodata aus, wird diese Teil der Metadaten.
  • Sollen Repositories für ein anderes Betriebssystem zur Verfügung gestellt werden, kann eine Konfiguration erstellt werden, die aber nicht aktiv ist, also enabled=0 enthält. Bei reposync kann dann mit --enablerepo repo-id das Repository nur für die Synchronisation aktiviert werden.

Ich hoffe dieser kleine Artikel hilft dem ein oder anderen. Wem das schnelle einfache Repository nicht genug ist, der kann auch versuchen mit rsync einen vollständigen Mirror aufzusetzen oder mit Katello sogar ein Staging einbauen, damit Updates erst in Entwicklung und Test laden bevor sie in Produktion vielleicht Probleme verursachen. Bei letzterem unterstützen gerne ich oder ein Kollege im Rahmen eines Foreman-Consultings.

Dirk Götz
Dirk Götz
Principal Consultant

Dirk ist Red Hat Spezialist und arbeitet bei NETWAYS im Bereich Consulting für Icinga, Puppet, Ansible, Foreman und andere Systems-Management-Lösungen. Früher war er bei einem Träger der gesetzlichen Rentenversicherung als Senior Administrator beschäftigt und auch für die Ausbildung der Azubis verantwortlich wie nun bei NETWAYS.

Foreman’s 10th birthday – The party was a blast

Birthday Logo

I can still remember when Greg had the idea of celebrating the Foreman’s Birthday four years ago and I volunteered to organize the German one. After two editions and with Foreman being covered on the Open Source Camp last year I asked for others to run the party. And with ATIX doing a great job I asked them to team up on this. So we have grown a great community event with the annual Birthday party.

This year was different to the ones before because we had such a big support by Red Hat. The new Community Managers showed up to introduce them accompanied by Greg who had stepped down earlier this year. A group of Product managers and consultants made the last stop on their European tour. A technical writer came over to discuss the future of documentation. And with Evgeni and Ewoud we had some recurring attendees to give a talk later. ATIX also arrived with a bus full of people. Monika represented iRonin, a company doing custom development on Foreman and I hope to team up in the future, and Timo developing on Foreman for dmTech brought a colleague. So users were slightly under-represented and the prepared demos were mostly used to share knowledge and probably because of the heat instead of hacking many discussions took place. But I think everyone of the about thirty attendees made good use of the first session.

Birthday PartyDemoThe session ended when I brought in the cake. And thanks to our Events team the cake was as tasty as good looking. A nice touch by Ohad was to insist he can not blow off the candles alone as he could not have build Foreman without the community.

Birthday CakeHelmets

After the cake break we started with the talks and the first one was by the Community team giving us a recap of Foreman’s history, data from the community survey and other insights like a first look on the future documentation. This is really the next step to me that Red Hat is also making their Satellite documentation upstream adding a use case driven documentation to the manual which is way more technical. The second talk Quirin showcased the current state of Debian Support which will be fully functional with Errata support being added, but he already promised some usability and documentation improvements afterwards. The third speakers were Dana and Rich who showed Red Hat’s roadmap for features to add to Foreman so they will be pulled into Satellite afterwards. The roadmap will be presented in a community demo and uploaded to the community forum. Having the product managers easily available allowed the audience also to ask any question and I was excited to hear for almost all topics brought up that there is already ongoing work in the background. For example I asked about making subscription management also usable for other vendors and Rich told me he is part of a newly founded team which is evaluating exactly this.

Because of the heat we added a small ice break before starting the next talk and because of Lennart being ill Ohad entered the stage to show his work on containerizing Foreman. He explained that he started it mainly for testing but the interest showed him that expanding it to be fully functional to run Foreman and even Katello on Kubernetes could be a future way. Evgeni gave a shortened version of the talk on writing Ansible modules for Foreman and Katello he created for Froscon. It was a very technical one showing how much work is necessary to build a good base so later work is much easier. From this perspective I can really recommend this talk to all Froscon attendees. Last but not least Ewoud looked into the project’s social aspects which was a nice mixture of official history and personal moments. He also showed off the different swag the project created, ending with a t-shirt signed by as many team and community members as possible while traveling from Czech to US and back as suitable gift to Greg because “Once a foreman, always a foreman”. 😉

For dinner we had Pizza and Beer, but moved to the air-conditioned hotel bar after a short while to finish the evening. I heard people were enjoying conversation until two o’clock in the morning even when the bar closed one hour earlier. 😀

I would say the Party was a blast and I am already looking forward to next year when ATIX will be the host again. But until then there are several other Foreman related events with the Open Source Automation Day on 15. & 16.10.2019 in Munich including Workshops the day before and a Foreman hackday the day after organized by ATIX and the Open Source Camp on 07.11.2019 in Nuremberg right after OSMC by NETWAYS.

Dirk Götz
Dirk Götz
Principal Consultant

Dirk ist Red Hat Spezialist und arbeitet bei NETWAYS im Bereich Consulting für Icinga, Puppet, Ansible, Foreman und andere Systems-Management-Lösungen. Früher war er bei einem Träger der gesetzlichen Rentenversicherung als Senior Administrator beschäftigt und auch für die Ausbildung der Azubis verantwortlich wie nun bei NETWAYS.

Automatisierte Updates mit Foreman Distributed Lock Manager

Foreman Logo

Wer kennt das nicht am besten soll alle nervige, wiederkehrende Arbeit automatisiert werden, damit man mehr Zeit für spaßige, neue Projekte hat? Es gibt nach Backups wohl kein Thema, mit dem man so wenig Ruhm ernten kann, wie Updates, oder? Also ein klarer Fall für Automatisierung! Oder doch nicht weil zu viel schief gehen kann? Nun ja, diese Entscheidung kann ich euch nicht abnehmen. Aber zumindest für eine häufige Fehlerquelle kann ich eine Lösung anbieten und zwar das zeitgleiche Update eines Clusters, was dann doch wieder zum Ausfall des eigentlich hochverfügbaren Service führt.

Bevor ich aber nun zu der von mir vorgeschlagen Lösung komme, will ich kurz erklären wo die Inspiration hierfür herkommt, denn Foreman DLM (Distributed Lock Manager) wurde stark vom Updatemechanismus von CoreOS inspiriert. Hierbei bilden CoreOS-Systeme einen Cluster und über eine Policy wird eingestellt wie viele gleichzeitig ein Update durchführen dürfen. Sobald nun ein neues Update verfügbar ist, beginnt ein System mit dem Download und schreibt in einen zentralen Speicher ein Lock. Dieses Lock wird dann nach erfolgreichem Update wieder freigegeben. Sollte allerdings ein weitere System ein Lock anfordern um sich upzudaten und die maximalen gleichzeitigen Locks werden bereits von anderen Systemen gehalten, wird kein Update zu dem Zeitpunkt durchgeführt sondern später erneut angefragt. So wird sichergestellt, dass die Container-Plattform immer mit genug Ressourcen läuft. CoreOS hat dazu dann noch weitere Mechnismen wie einen einfachen Rollback auf den Stand vor dem Update und verschiedene Channel zum Testen der Software, welche so einfach nicht auf Linux zur Verfügung stehen. Aber einen Locking-Mechanismus zur Verfügung zu stellen sollte machbar sein, dachte sich dmTech. Dass die Wahl auf die Entwicklung als ein Foreman-Plugin fiel lässt sich leicht erklären, denn dieser dient dort als das zentrale Tool für die Administration.

Wie sieht nun die Lösung aus? Mit der Installation des Plugins bekommt Foreman einen neuen API-Endpunkt über den Locks geprüft, bezogen und auch wieder freigegeben werden können. Zur Authentifizierung werden die Puppet-Zertifikate (oder im Fall von Katello die des Subscription-Managers) genutzt, die verschiedenen HTTP-Methoden stehen für eine Abfrage (GET), Beziehen (PUT) oder Freigaben (DELETE) des Lock und die Antwort besteht aus einem HTTP-Status-Code und einem JSON-Body. Der Status-Code 200 OK für erfolgreiche Aktionen und 412 Precondition Failed wenn Beziehen und Freigeben des Locks nicht möglich ist sowie der Body können dann im eigenen Update-Skript ausgewertet werden. Ein einfaches Beispiel findet sich hierbei direkt im Quelltext-Repository. Ein etwas umfangreicheres Skript bzw. quasi ein Framework wurde von einem Nutzer in Python entwickelt und ebenfalls frei zur Verfügung gestellt.
(mehr …)

Dirk Götz
Dirk Götz
Principal Consultant

Dirk ist Red Hat Spezialist und arbeitet bei NETWAYS im Bereich Consulting für Icinga, Puppet, Ansible, Foreman und andere Systems-Management-Lösungen. Früher war er bei einem Träger der gesetzlichen Rentenversicherung als Senior Administrator beschäftigt und auch für die Ausbildung der Azubis verantwortlich wie nun bei NETWAYS.

Verwaltung von SUSE Linux Paketen mit Katello

Katello-Logo

Katello erweitert Foreman um Content-Management oder da es mir primär um Linux-Pakete geht bevorzuge ich den Ausdruck Software-Management. Über Lifecycle-Environments und Content-Views werden hier Snapshots der Repositories erstellt und den verschiedenen Stages nacheinander präsentiert, damit in Produktion auch tatsächlich die Updates landen, die auch vorher getestet wurden. Doch darüber habe ich bereits vor einer Weile geschrieben. Seitdem hat sich zwar einiges weiterentwickelt, insbesondere ist die Unterstützung für Debian dazugekommen. Aber darüber möchte ich berichten wenn auch noch der Support für Errata-Management für Debian soweit ist.

Stattdessen möchte ich auf die Unterstützung für SUSE eingehen. Diese wurde von ATIX entwickelt und als Foreman-Plugin “ForemanSccManager” veröffentlicht. Wer die “Red Hat”-Unterstützung von Katello kennt, wird die Funktionalität recht schnell wieder erkennen. Das Plugin fügt einen neuen Menüpunkt hinzu, der es erlaubt Accounts für den Zugriff auf das SUSE Customer Center anzugeben und die damit verknüpften Softwareprodukte einfach zur Synchronisation auszuwählen. Dies finde ich besonders hilfreich, da SUSE zur Authentifizierung nicht nur mit Benutzer und Passwort sondern auch einem Token in der URL arbeitet, welches das manuelle Handling hier leider erschwert.

Wenn jemand ein paar Screenshots sehen möchte, möchte ich ihn auf die Orcharhino-Dokumentation (einem Produkt auf Basis von Katello) verweisen, denn das Plugin befindet sich schon eine Weile bei ATIX und ihren Orcharhino-Kunden im Praxis-Einsatz. Wer also auf SUSE angewiesen ist und noch eine Lösung für das Softwaremanagement sucht, kann mit Katello und dem ForemanSccManager auf eine modernere Plattform als Spacewalk oder den darauf basierenden SUSE-Manager setzen. Wer bereits auf Katello setzt und SUSE nutzt, dem kann ich nur empfehlen seinen Workflow auf das Plugin umzustellen.

Dirk Götz
Dirk Götz
Principal Consultant

Dirk ist Red Hat Spezialist und arbeitet bei NETWAYS im Bereich Consulting für Icinga, Puppet, Ansible, Foreman und andere Systems-Management-Lösungen. Früher war er bei einem Träger der gesetzlichen Rentenversicherung als Senior Administrator beschäftigt und auch für die Ausbildung der Azubis verantwortlich wie nun bei NETWAYS.

OSMC 2018 – Day 2

The evening event was again great food, drinks and conversation and while it ended in the early morning for some people, rooms were full of attendees again for the first talk. It was a hard choice between probably great talks but in the end I had chosen Rodrigue Chakode with “Make IT monitoring ready for cloud-native systems“. Being a long-term contributor to several Open Source Monitoring he used his experience to develop Realopinsight as a tool bringing existing monitoring tools together and extending them for monitoring cloud-native application platforms. In his live demo he showed the webinterface and Icinga 2, Zabbix and Kubernetes integration including aggregation of the severity for a specific service across the different solutions.
OSMC 2018
Scoring a Technical Cyber Defense Exercise with Nagios and Selenium” by Mauno Pihelgas was a quite uncommon case study. Locked Shields is the biggest Cyber Defense exercise involving 22 teams defending systems provided by vendors against hundreds of attacks. Mauno is responsible for the availability scoring system which gives the defending teams bonus points for availability of the systems, but of course it makes also available for attacks which if successful will cause loss of points. The data collected by Nagios and Selenium are then forwarded to Kafka and Elasticsearch to provide abuse control and overall scoring. To give you some numbers over the 2 days of the exercise about 34 million checks are executed and logged.
Susanne Greiner’s talk “Mit KI zu mehr Automatisierung bei der Fehleranalyse” was on using Artificial Intelligence for automatic failure analyses. Her talk started from anomaly detection and forecasting, went through user experience and ended with machine learning and deep learning. It is always great to see what experts can do with data, so running anomaly detection and forecasting on the data, adding labels for user experience and feeding them to the AI can increase troubleshooting capabilities. And better troubleshooting will result in better availability and user experience of course what perhaps is the main goal of all IT.
At the evening event there was again some gambling and after lunch the guys how managed to win the most chips won some real prices.
OSMC 2018 Gambling Winners
While some still enjoyed the event massage, Carsten Köbke started the afternoon sessions with the best talk title “Katzeninhalt mit ein wenig Einhornmagie” (Cat content with a little bit of unicorn magic). Being the author of the Icinga Web 2 module for Grafana and several themes for Icinga Web 2 he demonstrated and explained his work to the audience. It is very nice to see performance data with annotations extracted from the Icinga database nicely presented in Grafana. The themes part of the talk was based on the idea of every one can do this and monitoring can be fun.
Thomas and Daniel teamed up to focus on log management and help people on choosing their tool wisely in their talk “Fokus Log-Management: Wähle dein Werkzeug weise“. They compared the Elastic stack and Graylog with each other in multiple categories, showing up advantages and disadvantages and which tool fits best for which user group.
Eliminating Alerts or ‘Operation Forest’” by Rihards Olups was a great talk on how he tried reducing alerts to get a better acceptance and handling of the remaining alerts, getting problems solved instead of ignored. The ‘Operation forest’ mentioned in the talk’s title is his synonym for there infrastructure and alerts are trash he does not like in his forest, because trash attracts trash, like alerts attract alerts because if the numbers grows they tend to be ignored and more problems will get critical causing more alerts. It is not a problem of the tool used for monitoring and alerting but he had not only nice hints on changing culture but also technical ones like focusing on one monitoring solution, knowing and using all features or making problems more recognizable like putting them into the message of the day. For those having the same problems in their environment he wrote a shitlist you can check the problems you have and the number of checked items will indicate how shitty your environment is, I recommend having a look at this list.
Last but not least Nicolai Buchwitz talked about the “Visualization of your distributed infrastructure” and with his Map module for Icinga Web 2 he is providing a very powerful tool to visualize it. All the new features you get from the latest 1.1.0 release make it even more useful and the outlook on future extensions looks promising. Nicolai concluded with a nice live demo showing all this functionality.
So it was again a great conference, thanks to all speakers, attendees and sponsors for making this possible. I wish everyone not staying for the hackathon or Open Source camp “Save travels”. Slides, videos and pictures will be online in the near future. I hope to see you on next year’s OSMC on November, 4th – 7th!

Dirk Götz
Dirk Götz
Principal Consultant

Dirk ist Red Hat Spezialist und arbeitet bei NETWAYS im Bereich Consulting für Icinga, Puppet, Ansible, Foreman und andere Systems-Management-Lösungen. Früher war er bei einem Träger der gesetzlichen Rentenversicherung als Senior Administrator beschäftigt und auch für die Ausbildung der Azubis verantwortlich wie nun bei NETWAYS.

OSMC 2018 – Day 1

It is always the same, Winter is coming and it brings people to Nuremberg for OSMC. Our Open Source Monitoring conference still grows every year and after giving three parallel tracks a try last year, we changed format again to include also shorter talks and having always three tracks. It also gets more international and topics get more diverse, covering all different monitoring solutions with speakers (and attendees) from all over the worlds. Like every year also the 13th conference started with a day of workshops enabling the interested ones to get hands on Prometheus, Ansible, Graylog and practical example on using the Puppet modules for Icinga 2. Also this year two days of great talks will be followed by a day of hacking and the second issue of the Open Source Camp takes place, this time focusing on Puppet.
OSMC 2018
And another tradition is Bernd starting the conference with a warm welcome before the first talk. Afterwards Michael Medin talked about his journey in monitoring and being a speaker at OSMC for the eleventh time in “10 years of OSMC: Why does my monitoring still look the same?“. It was a very entertaining talk comparing general innovation with the one happening in monitoring. He was showing up that monitoring solutions changed to reflect the change in culture but still stayed the same mechanism and explained all the problems we probably know like finding the correct metrics and interpreting them resulting from this.
Second talk I attended was “Scaling Icinga2 with many heterogeneous projects – and still preserving configurability” by Max Rosin. He started with the technical debt to solve and requirements to fulfill when migrating from Icinga 1 to Icinga 2 like check latency or 100% automation of the configuration. Their high-available production environment had no outage since going live in January, because the infrastructure design and testing updates and configuration changes in a staging setup, what is pretty awesome. The scripting framework they created for the migration will be released on Github. But this was not all they coded to customize their environment, they added some very helpful extensions for the operations team to Icinga Web 2, which will be available on Github somewhere in the future after separating company specific and upstream ready parts.
For the third session I had chosen Matthias Gallinger with “Netzwerkmonitoring mit Prometheus” (Network monitoring with Prometheus). In his case study he showed the migration from Cacti to Prometheus and Grafana done at a international company based in Switzerland. The most important part is here the SNMP Exporter for Prometheus including a generator for its configuration. All required is part of their labs edition of Open Monitoring Distribution (OMD).
After the lunch Serhat Can started with “Building a healthy on-call culture“. He provided and explained his list of rules which should create such a culture: Be transparent – Share responsibilities – Be prepared – Build resilient and sustainable systems – Create actionable alerts – Learn from your experiences. To sum up he tells everyone to care about the on-call people resulting in a good on-call service and user experience which will prevent a loss of users and money.
The Director of UX at Grafana Labs David Kaltschmidt gave an update on whats new and upcoming in Grafana focusing on the logging feature in “Logging is coming to Grafana“. The new menu entry Explore allows to easily querying Prometheus metrics including functions – just one click away – for rate calculation or average and it works the same for logging entries as a new type of datasource. This feature should be very useful in a Kubernetes environment to do some distributed tracing. If you are interested in this feature it should be available as beta in December.
Distributed Tracing FAQ” was also the title of Gianluca Arbezzano‘s talk. I can really recommend his talk for the good explanation on why and how to trace requests through more and more complex, distributed services of nowadays. If you are more interested in tool links, he recommends Opentracing as library, Zipkin as frontend and of course InfluxDB as backend.
This year Bernd’s talk about the “Current State of Icinga” was crowded and interesting as always. I skip the organizational things like interest in the project is growing according to website views, customers talking about their usage, partners, camps and meetups all over the world. From the technical aspects Icinga 2 had a release bringing more stabilization, improved Syntax Highlighting and as new feature Namespacing. The coming Director release brings support for multiple instances helping with staging, health checks and a configuration basket allowing to easily export and import configuration. A new Icinga Web 2 module X509 helps managing your certificate infrastructure, available next week on github. The one for VMware vSphere (sponsored by dmTECH) is already released and was shown in a demo by Tom who developed it. Icinga DB will replace IDO as a backend moving volatile data to Redis and data to be keeped will be stored to MySQL or PostgreSQL and there will also be a new Monitoring Module for Icinga Web 2 to make use of it, all available hopefully in two weeks.
This year’s OSMC provided something special as the last talk of the first day with an authors’ panel including Marianne Spiller (Smart Home mit openHAB 2), Jan Piet Mens (Alternative DNS Servers – Choice and deployment, and optional SQL/LDAP back-ends), Thomas Widhalm and Lennart Betz (Icinga 2 – Ein praktischer Einstieg ins Monitoring) moderated by Bernd and answering questions from the audience.
If you want to get more details or pictures have a look at Twitter. There will also be a post by Julia giving a more personal view on the conference from interviewing some attendees and one of me covering the talks of the second day, but now I am heading for the evening event.

Dirk Götz
Dirk Götz
Principal Consultant

Dirk ist Red Hat Spezialist und arbeitet bei NETWAYS im Bereich Consulting für Icinga, Puppet, Ansible, Foreman und andere Systems-Management-Lösungen. Früher war er bei einem Träger der gesetzlichen Rentenversicherung als Senior Administrator beschäftigt und auch für die Ausbildung der Azubis verantwortlich wie nun bei NETWAYS.