Open Source Camp on Foreman

Like every year there was an Open Source Camp following the OSMC and as usual we helped organize that. Just in case you aren’t aware of what an Open Source Camp is here is the just of it: It’s meant to be an offer for Open Source projects to present themselves more in depth to the community. This year the Open Source Camp is on that one special yellow helmet we all know and love, Foreman.

Ondřej Ezr started us off with Ansible automation for Foreman (hosts). There are probably more than enough people using puppet only in their Foreman environment. Alternative or complementary to that would be using the plugin foreman_ansible. Ansible and Puppet don’t necessarily need to be better or worse, they are different and both have their advantages and disadvantages. By going through some basic steps, like role assignment, host creation and so on, he showed how one can do all that, but with Ansible. You can easily dynamically allocate roles and installations through Ansible to your Foreman hosts, but to make it even more specific one can set custom variables within the Ansible plugin for it to use, like foreman_repository_version. You could invoke a Job, like an Ansible Playbook, which will overwrite the variables previously set or make your installation more customizable from the get go. Install from git, run a playbook through ssh and more was covered during his talk. The plugin would not be a good alternative or viable if it did not hold up against the standards that puppet sets as a competitor. While Ansible doesn’t offer an inherit solution for reoccurring runs like every hour, the plugin does.

Next up was Bernhard Suttner, who wanted to give us a taste of Salted Foreman. Initially he explained what all that salt was about. The SaltStack a open source project written in python, can be used as a configuration management tool for Foreman. Salt excels at orchestrating cloud environments and network use-cases, but then we got to the Foreman relation. Running a salt and Foreman environment means running a environment of managed hosts, which are salt minions and a foreman_smart_proxy, which will also be the salt master. He showed us what salt in Foreman looks like and gave us some insight on how it works, but even more important from now on there are people dedicated to the project and some day the plugin might be as good as the puppet or ansible plugin. Salt is great and especially effective in terms of scalability. It’s pretty straightforward to use and the initial setup is not so hard. We are excited for what is to come.

Provisioning on Azure Cloud through Foreman by Aditi Puntambekar was going to follow that one. Aditi made sure everyone is familiar with the extend of Foremans capabilities in terms of provisioning. This was especially important because Foremans capabilities differ from its usual when it comes to cloud provisioning. After a quick trip through the configuration of compute resources and imaged-based provisioning templates we went onward to the Azure Resource Manager. She explained how the Azure Resource Manager essentially worked, but what is interesting to us is the foreman_azure_rm. Well and foreman_azure_rm does what you expect it to do. It adds the Microsoft Azure Resource Manager as a compute resource for the foreman. In her demo, she showed us how to use said resource and more.

Martin Bačovský talked about CLI tools with Foreman. He started of with the Foreman API. Of course the Foreman API is fast and has a wide range of tools and libs included within it. Just like Martin said in his talk, if you are interested in the Foreman API check out the documentation, it’s very good. Also interesting in the realm of APIs was his next tool, which is using apipie/apipy, which you are probably aware of if you are more heavy on the python side of things. Up there with the most well-known tools is Martins next, Hammer CLI, a command-line tool for Foreman. After sharing his experience with these rather popular tools with everyone he introduced us to Foreman’s integration of GraphQL. It’s basically a query language, which seems to be promising so far. Martin especially focused on the flexibility of queries and the introspective it has, yet one has to see where the project goes. There were many more tools he told us a lot about. To name just a few more of them, Report Templates, Foreman Ansible Modules and foreman_maintain. If you are interested in one of these tools in particular check out the video of the talk, which will be available soon on our Youtube Channel.


Give your Foreman a greater toolbox with Plugins by our very own Dirk Götz. Like he said himself: I will start of with existing toolbox things and at the end I will show you how to create these things yourself. And that he did. This talk was very demo heavy, thereby everything he explained was plain and simple, because you where able to see it as he did it. At the very top of his agenda was Job Invocation/Remote Execution. Not that exciting you think? Well, more interesting is the best practice advice he threw in on the way, like there is no issue of the configured user because his password is not saved as plain text in the database. Then the development part was up. He showed a couple of jobs that he wrote himself. Easiest, which served as an example is a simple ping check. He pointed out important thoughts to keep in mind, while writing jobs, like default values. Before his talk came to a close he talked a bit about the Web Console which has been introduced and is yet not well known. The web console is pretty much a integration of Cockpit. A well experienced user in the Linux world won’t be that excited about this, but a less experienced user will love this.

The next talk would not have happened, if Dirk didn’t spontaneously offer to step in. So we got another thirty minutes of Dirk Götz and I won’t complain. Katello: Adding content management to Foreman was the title and people where keen to hear about just that. What is Katello? Dirk described it as a defined set of Foreman plugins but not just that. It enriches your content management, as well as subscription management. Wait… content management? Why do I need that? Configuration management should be enough! Not necessarily, depending on your environment. Lets just pick up the points that Dirk made towards content management. For local content it ensures availability. For staging, it allows testing updates and makes builds reproducible. So content management should be seen as an addition to config management. He also talks about content views and how they are used to do the versioning, while they are being held by life cycles. Integration in orchestration was also a rather big point during his talk, which is done via SSH or Ansible. Dirk designs his talk in a way that makes summarizing them impossible, because he covers way to much. Lets just say not announced but very appreciated and most definitely worth checking out at our NETWAYS-Youtube Channel.

It was my second Open Source Camp and if you ask me this kind of exchange is what one wants to see in the open source community. There was variety and judging by the crowd reactions I was not the only one enjoying these talks. Thanks to all the speakers and attendees, safe travels home to everyone. Until the next Open Source Camp, hope to see you there!

Alexander Stoll
Alexander Stoll
Junior Consultant

Alexander ist ein Organisationstalent und außerdem seit Kurzem Azubi im Professional Services. Wenn er nicht bei NETWAYS ist, sieht sein Tagesablauf so aus: Montag, Dienstag, Mittwoch Sport - Donnerstag Pen and Paper und ein Wochenende ohne Pläne. Den Sportteil lässt er gern auch mal ausfallen.

OSMC 2019 – Day 1

As always OSMC started with the workshop day. This time the topics were Prometheus given by Julian Pivotto, Gitlab by Michael, Terraform by Lennart and Foreman by me. After the workshops with coming together for dinner and some drink social networking started, one of things I enjoy most at conferences nowadays.

Day one started also like every time with a warm welcome from Bernd. Quite unexpected for me was the high number of first time attendees who raised hands when Bernd asked. It is always great to see new faces!

Ansible module planned to be released
First talk I attended was “Directing the Director” by Martin Schurz who gave some insights in how the monitoring platform developed at T-Systems Multimedia Solutions GmbH over last years. So they scaled up from single system, solved migration from VMs to Docker of the monitored environment and built knowledge to provide consulting to the teams which run 94 different projects which have to be monitored. With so many different things to monitor the next step of course was automation where all the good from Icinga-Director-API and Ansible came together. But if it is easy for users to build monitoring objects, configuration will grow which comes with the next challenges to make it even more easy and error prove. And from what Martin showed they solved it in a good fashion, but future will tell and I hope he will give another talk providing an update in the future.

Crowded room at OSMC
Second one was Christian with “Windows: One Framework to Monitor them all” who introduced his precious to the crowed. If you could not make it into the crowded room or was not at OSMC at all, have a look at the documentation or the framework itself, the plugins, the kickstart script to get it up and running the background daemon. In a great live demo he showed all the components and explained them in depth. While it is still the first release candidate it looks very promising.

Marcelo Perazolo from IBM Systems was talking about “Monitoring Alerts and Metrics on Large Power Systems Clusters”. He started with an introduction to the Power architecture and the workloads it is specially useful to make everyone familiar with it. The example he used was a big one, the Summit supercomputer. The main topic were two projects CRASSD and Power-Ops which not bring the data from some systems not everyone is familiar with to commonly used tools but also include Ansible playbooks for automation and flexibility “instead of just providing a docker container”. The demo showed some Kibana dashboards which provided in-depth data summarizing the health and performance of Power Systems starting from firmware to service running on it.
Power architecture explainedDesert

Lunch was great as always (and not only the dessert) and to avoid food coma we had the first time ignite talks at OSMC. It started with one from the conference sponsor ilert. Afterwards Blerim talked about “How Observability is not killing Monitoring” where he concluded that Observability should be an addition to Monitoring and not a replacement. Toshaan Bharvani decided for an ignite when sitting in the talk about Power Systems and wanted to add about “Building your own Datacenter” based on OpenPower and software available for it. In his talk “Overengineering your personal website” Bram Vogelaar showed a good (and funny) example where adding things to your infrastructure can escalate.

New entry in the datalist
Marianne Spiller‘s talk “Lorem Icinga puppetdb director amet” was not only a must see because its creative title, but because I like the mix of humor and technical knowledge she always provides. With practical examples she showed the problems of manual work like lazy admins prefer introducing a “Not answered” to the datalist of operating systems instead of maintaining the information on hosts. So instead of a form manually filled by admins she ended using the import from PuppetDB to create monitoring objects based on facts of the system.

Grafana Loki demo
Directly from Grafana Labs represented by Ganesh Vernekar the audience got some news about Loki which is “Like Prometheus, but for Logs”. So Loki avoid huge indexes and allows for better scaling by not indexing log lines but grouping them to streams. Using a similar format to Prometheus it allows to get metrics and logs for a system without a context switch. Running Loki to get this seems quite simple and flexible with only one binary which can scale out easily and also allows for a microservice infrastructure.

First code improvement
“Fast Logs Ingestion” by Nicolas Fränkel showed common coding mistakes and how to avoid them to get better logging in your application. He also covered topics like metadata and searching logs which also should also influence decisions and code. Structured log data even not written to file are also a option to consider like config reload during runtime to enable the user to switch loglevel without downtime. In the end he hopes people take away from his talk that everyone from dev and ops to architects should keep in mind which trade-off between speed and reliability has to be done and why.

Like every year the “Current State of Icinga” by Bernd was held in front of full house. He started with a short introduction to Icinga including the workflow which results in “Icinga makes you happy”. Icinga Workflow Afterwards to start with technical things he looked into the big changes Icinga 2.11 brought with a new network stack, high availability for more features and a new process handling not only helping with containers. Icinga 2.7 brought more translations, markdown support, jQuery 3, modernized styling for forms and lists, color blind theme and improvements for module developers. The vSphere module provides now an Import Source for Director, no code depenency on the Director and some UI improvements. The latest version of Director has also more translation, support for scheduled downtimes and sync previews. The BP Modelling (formerly Business Process Modul) has now drag & drop, export and import and breadcrumbs to make the UI more usable. As the first new feature he introduced the Windows monitoring Christian gave a detailed talk earlier today. Icinga for AWS was an improvement to the one only providing a simple import source for Director which adds support for multiple sources, some property modifiers and sync previews. Icinga Module for Jira includes an Issue overview, Jira notification via Director integration and custom workflows you can create from Icinga Web 2. Icinga DB as replacement for IDO is decoupling status and historic data using Redis and in a demo the new monitoring module based on it was also shown including all the visual improvements. Pull requests are already merged and will be part of the next releases and new things are available separately. The next update you can get on IcingaConf in Amsterdam on May 12 – 14, 2020.

As last topic a open discussion about Code of Conducts suggested and moderated by Stefan Lange took place.

Pictures are taken from several twitter users tagging them with OSMC. Thanks for providing them, expect some better ones from my colleagues from the events and marketing teams. I hope you enjoyed my report for day 1 while I am heading over to the social event at the Loftwerk and try to have at least a short talk to everyone. Day 2 will be covered tomorrow evening.

Dirk Götz
Dirk Götz
Principal Consultant

Dirk ist Red Hat Spezialist und arbeitet bei NETWAYS im Bereich Consulting für Icinga, Puppet, Ansible, Foreman und andere Systems-Management-Lösungen. Früher war er bei einem Träger der gesetzlichen Rentenversicherung als Senior Administrator beschäftigt und auch für die Ausbildung der Azubis verantwortlich wie nun bei NETWAYS.

Milano – PostgreSQL Conference Europe 2019

It has been a while since one of us visited a PostgreSQL Conference. So when the opportunity arose Lennart and I took it. Additionally, it is somewhat of a special pgconf this year, because the conference is held where it originated, Italy. With over 500 attendees the conference set a new all time record. It is going to be a good one. Numerous talks, a lot of variety and gaining more insight into the postgres community. That being said, these talks can by no means be summarized in one blog post, that is why I have decided to bring you my just personal highlights. If you are interested in talks not mentioned here, I highly recommend checking out the official website, where at least most slides of speakers can be found.

Paul Ramseys talk about PostGIS was the first one that spiked my attention. Having used PostGIS before without extensive knowledge, the add-on seemed pretty mundane, but little did I know. And I mean that. Little did I know of the extend of PostGIS or GIS as a whole. Let me try to give you the just of it. There was a phrase that he used a few times that somehow stuck: GIS without the GIS. To be just that PostGIS without the GIS the workload of low-performance scripts had to be moved to more efficient SQL. Additionally, spatial middleware from the database to the web had to go. Lastly, PostGIS had to be build in a way that provides developer with direct access to a GIS analytical engine. After giving everyone a basic introduction with the some example problems and queries to solve them. Well, he told us everything about PostGIS or atleast a lot. SFSQL, OSS, ISO, Indexing, Spatial Joins, the list is long. 189 Slides, If you will.

Next up in my little list of talks I really enjoyed was Wonderful SQL features your ORMs can use (or not) by Louise Grandjonc. She went through various ORMs (Object Relationship Mapper) during her talk like Django, SQLAlchemy, activerecord and Sequel and used them for a little project of hers. With scripts she aggregated Data regarding pop music lyrics and used this data as an example for various queries.

More Than a Query Language: SQL in the 21st Century was the title of the talk I want to tell you about next. Markus Winand talked a lot of history here. He looked at the very beginning and saw how SQL evolved and changed in many ways. A lot of empathisis was put on SQL having been bound to the idea of an relational data model for too long and that not necessarily beening the case today anymore. While jumping through the timeline of SQL development and explaining various features in the process he validated his claim.

I think it was a great event. I really liked the variety of topics and I hope even though I just mentioned a few, you got some insight into the event. PostgreSQL has an awesome community and having that conference as a platform to exchange experience, help and learn from eachother is just great

Alexander Stoll
Alexander Stoll
Junior Consultant

Alexander ist ein Organisationstalent und außerdem seit Kurzem Azubi im Professional Services. Wenn er nicht bei NETWAYS ist, sieht sein Tagesablauf so aus: Montag, Dienstag, Mittwoch Sport - Donnerstag Pen and Paper und ein Wochenende ohne Pläne. Den Sportteil lässt er gern auch mal ausfallen.

Flying blind – accessibility in the world wide web

Do you know how many visually impaired people there are out there?
I can tell you: There are roughly 188.5 million people with mild vision impairment, 1043 million have moderate to severe vision impairment, and 36 million people are blind (According to the world health organisation in 2018). When it comes to colour blindness there are approximately 300 million people affected – that’s almost as many people as the entire population of the USA! (According to the Colour Blind Awareness Community)

That’s a lot, isn’t it? And a lot of those people also use the web. Not everyone in the same way as most do though.

Have you ever used a screen reader? Or heard of the term at all? A screen reader is a tool used to have the content of your screen read out for you. They are usually used by people who have issues reading what’s presented – either because they cannot distinguish between what’s written and the background because of bad contrasts, because fonts that are too small or just because they can not read or see at all.
For anyone who is interested how it works there is a lot of material on youtube, like this video by the SLCC Universal Access channel. If you have the time you should also try it out yourself, there are a lot of free tools out in the www.

In order to make this all possible there are guidelines provided by the w3, which cover:

  • Web Content Accessibility Guidelines (WCAG) – how to make web content accessible
  • Authoring Tool Accessibility Guidelines (ATAG) – make authoring tools accessible
  • User Agent Accessibility Guidelines (UAAG) – make user agents accessible
  • Accessible Rich Internet Applications Suite (ARIA) – make web content and web applications accessible

For all of the (web) developers reading this, those are definitely worth a read! Sticking to those standards makes the web more accessible for everyone, and that’s what everyone should strive for, right?

For people who have difficulties with contrasts, there are standards set by the w3 and also high-contrast ‘themes’ available for some tools.

People affected by some form of colour vision deficiency there are colour blind variations in some tools – often times they are just being overlooked though. I have some pictures for you, that aim to visualise the most common kinds of colour vision:

colours with different simulated vision impairments

I used ‘Color Oracle’ for the simulated colours.

There is also a more detailed blogpost on about our new ‘colour blind’ theme in Icinga Web 2.

Feu Mourek
Feu Mourek

Feu verbrachte seine Kindheit im schönen Steigerwald und kämpfte sich bis zum Abitur durch die Schule. Seit September 2016 unterstützt es die Development Abteilung bei NETWAYS als Developer und UX Designer und widmet sich dort dem Web Development. Seine Freizeit verbringt es hauptsächlich in den virtuellen Welten von 'Dota 2' und diversen anderen Games, an der Kletterwand in der Boulderhalle, oder damit, mit Freunden und Kollegen Brett- und Pen and Paper Rollenspiele zu spielen.

Managing your Ansible Environment with Galaxy

Ansible is known for its simplicity, lightweight footprint and flexibility to configure nearly any device in your infrastructure. Therefore it’s used in large scale environments shared between teams or departments. This leads to even bigger Ansible environments which need to be tracked or managed in version control systems like Git.

Mostly environments grow with their usage over time, in this case it can happen that all roles are managed inside one big repository. Which will eventually lead to quite messy configuration and loss of knowledge if roles are tested or work the way they supposed to work.

Ansible provides a solution which is called Galaxy, it’s basically a command line tool which keeps your environment structured, lightweight and enforces your roles to be available in a specific version.

First of all you can use the tool to download and manage roles from the Ansible Galaxy which hosts many roles written by open-source enthusiasts. 🙂

# ansible-galaxy install geerlingguy.ntp -v
Using /Users/twening/ansible.cfg as config file
 - downloading role 'ntp', owned by geerlingguy
 - downloading role from
 - extracting geerlingguy.ntp to /Users/twening/.ansible/roles/geerlingguy.ntp
 - geerlingguy.ntp (1.6.4) was installed successfully

# ansible-galaxy list
# /Users/twening/.ansible/roles
 - geerlingguy.apache, 3.1.0
 - geerlingguy.ntp, 1.6.4
 - geerlingguy.mysql, 2.9.5

Furthermore it can handle roles from your own Git based repository. Tags, branches and commit hashes can be used to ensure it’s installed in the right version.

ansible-galaxy install git+,v0.2.0
 - extracting ansible-icinga2 to /Users/twening/.ansible/roles/ansible-icinga2
 - ansible-icinga2 (v0.2.0) was installed successfully

It’s pretty neat but how does this help us in large environments with hundreds of roles?

The galaxy command can read requirement files, which are passed with the “-r” flag. This requirements.yml file can be a replacement for roles in the roles path and includes all managed roles of the environment.

# vim requirements.yml
- src:
  version: v0.2.0
  name: icinga2

- src: geerlingguy.mysql
  version: 2.9.5
  name: mysql

Then run ansible-galaxy with the “–role-file” parameter and let galaxy manage all your roles.

# ansible-galaxy install -r requirements.yml
 - icinga2 (v0.2.0) is already installed, skipping.
 - downloading role 'mysql', owned by geerlingguy
 - downloading role from
 - extracting mysql to /Users/twening/.ansible/roles/mysql
 - mysql (2.9.5) was installed successfully

In case you work with Ansible AWX, you can easily replace all your roles with this file in the roles directory and AWX will download and manage your roles directory automatically.

A example project could look like this.

├── example_playbook.yml
├── group_vars
├── host_vars
├── hosts
└── roles
    └── requirements.yml

In summary, in large environments try to keep your code and configuration data separated, try to maintain your roles in their own repository to avoid conflicts at the main project.

Check out our Blog for more awesome posts and if you need help with Ansible send us a message or sign up for one of our trainings!

Thilo Wening
Thilo Wening

Thilo hat bei NETWAYS mit der Ausbildung zum Fachinformatiker, Schwerpunkt Systemadministration begonnen und unterstützt nun nach erfolgreich bestandener Prüfung tatkräftig die Kollegen im Consulting. In seiner Freizeit ist er athletisch in der Senkrechten unterwegs und stählt seine Muskeln beim Bouldern. Als richtiger Profi macht er das natürlich am liebsten in der Natur und geht nur noch in Ausnahmefällen in die Kletterhalle.

Monthly Snap August 2019


August is the typical month for holidays at NETWAYS. The schools are finally out for summer, and you would think everything would slow down a bit. But look at all the blogposts our colleagues wrote! No downtime at the NETWAYS HQ!

What was our Development Team up to?

Johannes started the month with Icinga Web 2 – More Goodies for Developers. He presented some interesting new functions for fellow developers.. Are you also annoyed with the Facebook newsfeed? Marius found a solution, which might actually be an alternative for some of us, in Facebook: endlich Ruhe. Alexander claims that Ansible is not just a hype, but extremely useful in Ansible + Icinga 2 = #monitoringlove.

Cakes and tech.

Foreman’s 10th birthday – The party was a blast. Dirk celebrated the Foreman birthday party with the community.
Lennart had a lot to share in Graphite-API für Grafana und Icinga Web 2, a detailed and thorough how-to. rsync und was dann? Read Markus` road to a database dump with a strict time limit. Max talked about Monitoring your Monitoring in Monitor das Monitoring_by_ssh. Check out how easily it is done with Icinga!

Grab your ticket!

The call for papers for this year`s OSMC is closed, but tickets are still available if you would like to attend! Julia shared a video from last year`s conference in the series OSMC | Take a glance back… Are you already looking forward to the OSDC 2020?  Save the date & win tickets! 

Nicoles` corner

Nicole gave us a little course on sensors in Kleine Sensorkunde. What kinds of sensors are there, and how do you decide which one is right for your needs? Read the blog, and if you are still unsure just contact our shop for consulting. Then she presented News from HW Group in HW group: Neues vom SensDesk Portal.

Project week for the juniors

The annual project week for our trainees took place in the beginning of august. Alexander told us the tale of 9 people from different departments getting together, agreeing upon a project, and actually creating it, in teamwork in just one week. Read about it in Azubiprojektwoche 2019. We have a new post in our blogseries NETWAYS stellt sich vor! Alexander reflected upon his first months at NETWAYS, and why he started here in the first place.

To migrate or not to migrate?

Are you considering a GitLab upgrade to EE? Read Gabriel`s tips in Migration von GitLab mit Upgrade auf EE. Follow his 9 steps and you might avoid unnecessary obstacles.

Hosted in…Nuremberg!

Martin spoke of the advantages of myEngineer by NWS in NETWAYS Cloud: Dein MyEngineer unterstützt Dich. Jederzeit.  Finally, there is a system where you get exactly what you want without any hidden Costs.

Catharina Celikel
Catharina Celikel
Office Manager

Catharina unterstützt seit März 2016 unsere Abteilung Finance & Administration. Die gebürtige Norwegerin ist Fremdsprachenkorrespondentin für Englisch. Als Office Manager kümmert sie sich deshalb nicht nur um das Tagesgeschäft sondern übernimmt nebenbei zusätzlich einen Großteil der Übersetzungen. Privat ist der bekennende Bücherwurm am liebsten mit dem Fahrrad unterwegs.