It all started with a GameBoy

It’s time to reflect and talk about video games. On how I got into programming and what drives me.

MissingNo: The glitch, the legend, my beginnings

Catch them all!

Pokémon, the first generation Game Boy games. They were the first thing that made me wonder just how programs and computers work. Back in the late 20th and early 21th century Pokémon Red and Blue were the talk of the schoolyard, if you are Generation X or a Millennial you probably know what I’m talking about. Fake rumors on how to get the rarest monsters with absurd guides were floating around: Do X while Y and have two of Z but don’t feed them after midnight. What gave these urban legends credibility were bugs in the games which had seemingly unrelated steps lead to weird and for a child even scary results – from characters being cut up and incorrectly reassembled over save games becoming unusable to a constant, never ending screaming sound.

For those who have not played these classics, I’m talking about the now famous MissingNo Glitch. Now we know the bug is caused by incorrect read and writes in the game. The games were written in Assembly and the programmers had very limited memory and space to work with, something was bound to break and such bugs are not uncommon in early console games. Most kids were just happy to have an infinite supply of Master Balls, an item in the game that could only be acquired once in the game, or get their favorite monster to the highest level quickly. For me this was enough at first as well but as time went on I grew more and more intrigued by the bug.

Minus World. A well known bug caused by incorrectly loading a level.

I asked my dad, he had no clue and neither did my mom.The internet was hard to use then still and I did not find my answers then. What I found was more confusing information and ways to manipulate the game, mostly collected by trial and error of other players, but there were also mentions of buffer overruns, memory violation and other terms I could not make sense of and didn’t hear again until I was allowed to watch The Matrix. This knowledge of the games made me the coolest kid on the playground for a while at least.

And after the Elite Four?

Only when I joined the local hackerspace and got involved with the CCC I finally got my entry point to the world of computers and programming. There was referred the book Learn Python The Hard Way and started writing code. Sadly none of my early work exists anymore, of git and GitHub I learned later still. The obvious choice then was to go to Uni and sign up for Computer Science class, three semesters I spent trying to wrap my head around the math needed to pass but ultimately quit because of it.

But my interest in programming was unbroken, I loved classes like Systems Programming which had assignments where you had to implement basic tools yourself, my own shell, my own email server, netcat – everything in C of course. That’s when I found my way to NETWAYS as an apprentice and have stayed here since, they let me write code. The code I write has changed, abstraction and new languages like Go have changed how I program but the lessons I learned from playing Pokémon in my bedroom still hold true: Sometimes it takes time to understand a bug.

If you’d like to join me in hunting bugs or talking retro games over a cup of coffee, head on over to our jobs page!

Jean Flach
Jean Flach

Geboren und aufgewachsen in Bamberg, kam Jean (das "-Marcel" ist still) nach einem Ausflug an die Uni, als Azubi zu NETWAYS. Dort sitzt sie seit 2014 im Icinga 2 Core Entwicklungsteam.

On giving up and trying an IDE

I dislike IDEs, at least I tell myself and others that. A 200 line long .vimrc gives a lot more street cred than clicking on a colored icon and selecting some profile that mostly fits ones workflow. So does typing out breakpoints in GDB compared to just clicking left of a line. Despite those very good reasons I went ahead and gave Goland and CLion, two JetBrains products for Go and C/C++ respectively, a chance. The following details my experiences with a kind of software I never seen much use for, the problems I ran into, and how it changed my workflow.

Installation and Configuration

A picture of my IDE wouldn’t do much good, they all look the same. So here’s a baby seal.
Source: Ville Miettinen from Helsinki, Finland

First step is always the installation. With JetBrains products being mostly proprietary, there are no repositories for easy installation and updating. But for the first time I had something to put in /opt. When going through the initial configuration wizard one plugin caught my eye: “IdeaVim”. Of course I decided to install and activate said plugin but quickly had to realize it does not work the same simply running vim in a window.
This “Vim emulation plug-in for IDEs based on the IntelliJ platform” sadly does for one not offer the full Vim experience and the key bindings often clash with those of the IDE. Further I started getting bothered by having to manage the Vim modes when wanting to write code.
I sometimes miss the ability to easily edit and move text the way Vim allows, the time I spend using the mouse to mark and copy text using the mouse are seconds of my life wasted I’ll never get back!
Except for the underlying compiler and language specific things both IDEs work the same, identical layout, window options, and most plugins are shared, so I won’t talk about the tool chain too much. But it needs to be said that Goland just works, possibly because building a Go project seems simpler than a C++ one. Getting CLion to work with CMake is tiresome, the only way to edit directives is by giving them to the command like you would on the shell. This would not bother me as much if CMake didn’t already have a great GUI.

Coding and Debugging

Yet I wouldn’t be still using those programs if there weren’t upsides. The biggest being the overview over the whole project, easily finding function declarations and splitting windows as needed. These are things Vim can be made to do, but it does not work as seamless as it does in the IntelliJ world. It made me realize how little time is spent the actual typing of code, most of it is reading code, drawing things and staring at a prototype until your eyes bleed confusion (sometimes code is not well commented). The debuggers, again specifically the one of Goland, work great! Sometimes I have to talk to GDB directly since there are many functions but too few buttons, but the typical case of setting a breakpoint and stepping through to find some misplaced condition is simple and easy.

Alright, here it is.

There are a few features I have not found a use for yet e.g. code generators and I still manage my git repositories from the shell. The automatic formatting is cool, again especially in Go where there is one standard and one tool for it. Other than that I run into a few bugs now and then, one that proved to be quite a hassle is the search/search and replace sometimes killing my entire window manager. Were it free software, there’d be a bug report. But for now I work around it. Maybe I’ll drop CLion but I doubt I’ll be writing any Go code in Vim anytime soon.
If you think you have found the perfect IDE or just want to share Vim tips, meet me at the OSMC in November!

Jean Flach
Jean Flach

Geboren und aufgewachsen in Bamberg, kam Jean (das "-Marcel" ist still) nach einem Ausflug an die Uni, als Azubi zu NETWAYS. Dort sitzt sie seit 2014 im Icinga 2 Core Entwicklungsteam.

Flapping in Icinga 2.8.0

The author viewing the code for the first time

Flapping detection is a feature many monitoring suites offer. It is mainly used to detect unfortunately chosen thresholds, but can also help in detecting network issues or similar. In most cases two thresholds are used, high and low. If the flapping value, which is the percentage of state changes over a set time, gets higher than the high threshold, it is considered flapping. It will then stay flapping until the value drops below the low threshold.
Naturally Icinga 2 had such a feature, just that it implemented a different approach and didn’t work. For 2.8.0 we decided it was time to finally fix flapping, so I went to investigate. As I said the flapping was working differently from Icinga 1, Shinken, etc. Instead of two thresholds there was just one, instead of one flapping value there were two and they change based on the time since the last check. Broken down it looks like this:

positive; //value for state changes
negate; //value for stable changes
FLAPPING_INTERVAL; //Compile time constant to smoothen the values
OnCheckResult() {
  if (positive + negative > FLAPPING_INTERVAL) {
    pct = (positive + negative - FLAPPING_INTERVAL) / FLAPPING_INTERVAL;
    positive -= pct * positive;
    negative -= pct * negative;
  weight = now - timeOfLastCheck;
  if (stateChange)
    positive += weight;
    negative += weight;
IsFlapping() {
  return 100 * positive / (negative + positive);

The idea was to have the two flapping values (positive & negative) increase one or the other with every checkresult. Positive for state changes and negative for results which were not state changes, by the time since the last check result. The first problem which arises here, while in most cases the check interval is relatively stable, after a prolonged Icinga outage one of the values could be extremely inflated. Another problem is the degradation of the result, in my tests it took 17 consecutive stable results for a flapping value to calm down.
After some tweaking here and there, I decided it would be wisest to go with the old and proven style Icinga 1 was using. Save the last 20 checkresults, count the state changes and divide them by 20. I took inspiration in the way Shinken handles flapping and added weight to the sate changes, with the most recent one having a value of 1.2 and the 20th (oldest) one of 0.8. The issue of possibly wasting memory on saving the state changes could be resolved by using one integer as a bit array. This way we are actually using slightly less memory now \o/

The above example would then have a value of 39.1%, flapping in the case of default thresholds. More details on the usage and calculation of flapping in Icinga 2 can be found in the documentation once version 2.8.0 is released.

Jean Flach
Jean Flach

Geboren und aufgewachsen in Bamberg, kam Jean (das "-Marcel" ist still) nach einem Ausflug an die Uni, als Azubi zu NETWAYS. Dort sitzt sie seit 2014 im Icinga 2 Core Entwicklungsteam.

From Perl to Python and beyond

I’ve seen and learned plenty of programming languages either during my studies or in work or spare time related projects – be it C/C++/C#, VHDL, Java or PHP/Perl/Python/Ruby even (I’ve removed some in my XING profile to reduce recruiter spam level ;)). Choosing the “right” language is always hard but most of the time the requirements of existing software or newly designed projects allow you to skip that part and already have one in mind.
python-logo-master-v3-TMWhen joining NETWAYS in late 2012, I took over the LConf backend project being entirely written in Perl similar to other plugins and scripts floating around as open source software. Icinga 1.x Core is partially using Perl as well (although embedded Perl in C is a horrible mess). Most recently our development team decided to go for Python as the primary programming language.
So, my Python foo wasn’t that good after some years not really using it. Learning from my colleagues and looking into additional ressources helped a lot. I also found “Head first Python” from O’Reilly in my bookshelf as a gift from the Nagios/Icinga cookbook review which also provides an extensive introduction into Python – when you already know a scripting language like Perl.
We’ve been working on a customer’s project developing a plugin framework for executing checks via a defined transport (e.g. ssh) and parsing cli output. Find some collected hints and tricks I’ve learned below.

Remove colors from shell output

The EMC “isi” cli command puts colors into the shell output (e.g. for a critical state turning the background red). While one could disable color support on the shell, this was not possible in that environment. The fix by Gunnar was easy: Parsing text output into a list called “lines” and clean the line string from shell color codes:

lines = text.split('\n')
lines = [re.sub('\x1b\[[0-9;]*m', '', line).strip() for line in lines]

Convert datetime to unix timestamp

Consider having a datetime string whose output format is not fixed, which means some strftime print magic does not work for proper parsing. Luckily there’s a Python module called dateutil providing the required functionality.

import time
import datetime
import dateutil.parser
def datetime2unixts(time_str):
    dt = dateutil.parser.parse(time_str)
    return time.mktime(dt.timetuple())
$ python
Python 2.7.8 (default, Apr 15 2015, 09:26:43)
[GCC 4.9.2 20150212 (Red Hat 4.9.2-6)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import time, datetime, dateutil.parser
>>> def datetime2unixts(time_str):
...     dt = dateutil.parser.parse(time_str)
...     return time.mktime(dt.timetuple())
>>> time_str = "Wed May 13 16:15:34 CEST 2015"
>>> datetime2unixts(time_str)

Initialize nested dictionary

Similar to what I am used to do with Perl and hashes I was stumbling over this with Python. In Perl one does not care about initializers, but just keeps populating all items like so:

my $hash = {}
$hash{'key1'}{'key2'} = "val1";

In Python this does not work out-of-the-box. There are some modules available such as “collectors”, but I prefer a somewhat native implementation making it work on older distributions. I came across this solution which I consider pure awesome 🙂

class AutoVivification(dict):
    """Implementation of perl's autovivification feature."""
    def __getitem__(self, item):
            return dict.__getitem__(self, item)
        except KeyError:
            value = self[item] = type(self)()
            return value
hash = AutoVivification()
hash["key1"]["key2"] = "val1"

Note: It certainly does create non-existing keys if you are checking for their existance. My implementation does not care about key checks, but requires known-to-exist values from a flat dictionary with “id_label” as key stashed into a nested dictionary where id and label are separated/nested for further operations.

        jobs = AutoVivification()  # use a pre-initialized nested dict
        for k, v in sorted(perfdata.iteritems()):
            (job_id, label) = k.split("_", 1)
            if label == "ended_ts":
                jobs[job_id][label] = float(v)
            if label == "started_ts":
                jobs[job_id][label] = float(v)

Fix the indent

If your editor doesn’t do that automatically (e.g. converting tabs into 4 spaces) it might still work, but could also cause weird behaviour in Python depending on the indent only (no brackets as I am used to with Perl). Fixing this is fairly easy by using autopep8 – I’m using Fedora 22:

dnf install python-autopep8
autopep8 check_plugin -i

Note: ‘-i’ replaces the given file with all the changes. Make sure to commit your other changes to git before.


I never thought that I would dig deep in Python again that easy. I used to hack that in the past with Win2k usb driver test frameworks, and also checkmk plugins, but gaining back experience worked out pretty well.
Icinga 2’s code, value types and configuration look a lot like Python as well – take dictionaries for custom attributes using apply for loops, eh? (Hint: Testdrive this config snippet inside the Icinga 2 Docker container).

object Host "dns" {
  import "generic-host"
  address = ""
  address6 = "::1"
  vars.dns_checks["dns"] = {
    dns_lookup = ""
    dns_server = ""
    dns_expected_answers = ""
  vars.dns_checks["dns"] = {
    dns_lookup = ""
    dns_server = ""
    dns_expected_answers = ""
apply Service for (dns_check => config in host.vars.dns_checks) {
  check_interval = 1m
  retry_interval = 30s
  check_command = "dns"
  vars += config

Icinga 2, plugins, your project – here I come 🙂
PS: See you at the OSMC hackathon.

Michael Friedrich
Michael Friedrich
Senior Developer

Michael ist seit vielen Jahren Icinga-Entwickler und hat sich Ende 2012 in das Abenteuer NETWAYS gewagt. Ein Umzug von Wien nach Nürnberg mit der Vorliebe, österreichische Köstlichkeiten zu importieren - so mancher Kollege verzweifelt an den süchtig machenden Dragee-Keksi und der Linzer Torte. Oder schlicht am österreichischen Dialekt der gerne mit Thomas im Büro intensiviert wird ("Jo eh."). Wenn sich Michael mal nicht in der Community helfend meldet, arbeitet er am nächsten LEGO-Projekt oder geniesst...