DEV Retreat 2019 Recap

In this Episode: Everybody gets blown to smithereens (digitally).

Those team events are often paired with team-building exercises, causing possibly uncomfortable situations with your co-workers or let you find out things you never wanted to know about them. That’s at least what every comedy involving such an event leads us to believe. So for the breaks between remembering what we did well and not so well since our last DEV Retreat we had a game of “Keep Talking and Nobody Explodes“: The only game released since 2000 to require a manual, it’s also great fun.

The method for retrospection did not change much from last time but we had better sticky notes this time, which helped! We started off with a general assessment of our well being and mood and I am happy to report nobody was too grumpy. This was followed by thinking back what had happened, which projects did we do, what has changed and our unfinished business. Surprisingly a lot of us remembered not only big releases of Icinga 2, Icinga Web 2 or other larger projects but also small things, like our Trainees first merge, taking a break from PHP in favor of JS or making a brochure to advertise apprenticeship at NEWTAYS.

After making concrete plans on how to improve with out team dinner was served. The Hotel had a habit of making the portions a bit smaller than some would have liked but after the dessert nobody was hungry anymore and ready for a mellow evening with Gin and Tonic.

Of course not all is happy, rainbow and unicorns. But there weren’t any problems we aren’t confident we can solve. Except what to do for our next DEV Retreat. But our colleagues went for a round of arrow tag, maybe we’ll steal that idea for next time.

Diana Flach
Diana Flach

Geboren und aufgewachsen in Bamberg kam Diana, nach einem Ausflug an die Uni, als Azubi zu NETWAYS. Dort sitzt sie seit 2014 im Icinga 2 Core Entwicklungsteam.

Lessons learned – how not to make beginner’s mistakes: Migrating server

At Netways we try to learn all the time. Often you can simply read man pages, change logs, or even tabtab through your shell command at hand.

Trainings and conferences are a bit more time consuming, but can offer you one priceless advantage: the direct communication with someone, who has seen several sides of the topic at hand. I think, you learn best from other peoples experiences – even more from the situations where failure ensued. Here it is critical to not only look at the failure itself, but why it occured and how it was finally resolved.

Using other persons failure as a mean to learn  might seem a bit cynically at first. In an ideal world, however, the same failure would only ocurr once.

So let’s begin with my contribution to a better world, maybe even as a start of a series. This entry is not meant as full post-morten per se, it only describes mistakes you can make and should avoid.

As many might know, our office moved. Having been 10ish years in our “old” office, you might figure that quite a lot “historical infrastructure” can grow in this time. Especially in an IT environment where everybody wants to try something new, better, and undocumented.

Being in the distinguished situation of having direct access to our DC via 1GBit-Fiber, it was possible to use some of our external IPs in our office. VLAN-Tagging, firewall policies, iptables rules – all well known and understood best practices. The services got installed, used and worked flawlessly for all the time and have lived happily ever after.

With the announcement of moving to the new premises, the blue sky became a bit hazy. We had to move all needed hardware devices (NASA could tell you how small and well hidden some of them can be) and also separate them from the unneeded devices. Todays story is about two specific systems, each consisting of two 1U server. Those were some of the systems which used external IPs to provide services to the public. They have been running flawlessly for quite some time and given their age, had started to develop some adorable quirks.

It was my task to move these “dear old ladies” out of the cozy office into the cold, professional DC downstairs.

What harm can 4 old and lovable server possibly do, you might ask? The answer is: None, if you treat them the way they were used to.

First things first: How can you gain access to the DC? Is there a registration process, which has to be followed? How long does this take? Can you access the DC after business hours easily? (Hints: yes, long, no)

Also don’t try to rush things when it comes to shutdown the machines for moving  them. The machines owners like to what is going to happen.

Grab all the tools you will need to remove the server from your previous rack and install them in their new home (cordless screwdriver, all bits you can find)

Are all installation material available? This is not only referring to rack rails, but also cage nuts, screws (size matters) and front covers for feng-shui and air flow. (depends, mostly: no)

Cables! Just collect all the cables you need and then some. Usually, they will be too short. Too long is not an issue you can’t fix with zip ties (you will forget these)

Do you have network access in the DC for debugging, communication etc.? (Hint: depends)

Do you want to move more than one server? Be cautios and do them step by step. You’re absolutely allowed to install them all at the same time. Be aware you might experience crooked rails, incorrect cabling and other time consuming things.

When you have installed the machines, definitely take your time to check these with a KVM device. Whichever is in reach. (you guessed it: there won’t be any when you need them the most). Don’t rely on the machine and its fancy blinkenlights: Some may flash when everything is ok, some flash to indicate errors, some don’t flash at all.

Check all your cabling at least twice, give them a gentle pull – if they come lose, you have to start over again.

Take breaks between different machines. Either try to find a cool spot in the DC (haha) or get outside, have something to drink and return refreshed. The noise (ear plugs, ANC Headphones), temperature, confinement while working in the rack will wear you out eventually.

If you route the machines traffic through several VLANs, make sure all needed switch ports are tagged (or untagged? You decide!) and firewall policies applied for the new location.

Always have a piece of paper and a (working!) pen with you – it’s faster to scribble something on paper than to crawl through yor rats nest of cabling, climb over all the machines you’re up to install and then find your trusty notebook with dead battery.

Before you finally leave, make sure to give everything one last check and, if possible, communicate with the owners of the respective machines. Collect all the tools you brought with you. If you didn’t bring them but “found” them somewhere and used them: make sure to return them.

If you run into any issues, make sure all colleagues you could ask for assistance are currently at the party you’re rushing to attend.

Also make sure to communicate only via phone, so you don’t leave any paper trail when it comes to DC access, network config or time accounting.

When you don’t experience some of these mistakes because of this post, this post was a success. Of course some points are missing (feel free to comment), but I hope the overall pattern is visible:

be prepared, double check and take your time


Oh, and don’t forget the key to your racks. There is just one key, right?

Tim Albert
Tim Albert
System Engineer

Tim kommt aus einem kleinen Ort zwischen Nürnberg und Ansbach, an der malerischen B14 gelegen. Er hat in Erlangen Lehramt und in Koblenz Informationsmanagement studiert, wobei seine Tätigkeit als Werkstudent bei IDS Scheer seinen Schwenk von Lehramt zur IT erheblich beeinflusst hat. Neben dem Studium hat Tim sich außerdem noch bei einer Werkskundendienstfirma im User-Support verdingt. Blerim und Sebastian haben ihn Anfang 2016 zu uns ins Managed Services Team geholt, wo er sich nun insbesondere...

Digging up hidden information for a scavenge hunt?

Since today is Ascension Day – which is a public holiday here in Germany – I figured, why not write a post that is also relevant to our readers who have the day off?

So what could be relevant and fun?
Most of the people here who read this blog probably work in tech. And a trait, that I found that a lot of people working in tech have, is enjoying solving puzzles – be it on the computer or in the outside world. I got introduced into something that appeals to me in exactly that way: Geocaching – Mystery caches (yes, yes, I know, geocaching is sooooo 2015 – but solving puzzles is always going to be cool!)

Map of Nuremberg with about 200 caches marked on it

Map of Nuremberg with caches marked on it – each icon is a hidden treasure

A little explanation what that even is (stolen from Wikipedia):
Geocaching is an outdoor recreational activity, in which participants use a GPS receiver or mobile device and other navigational techniques to hide and seek containers, called “geocaches” or “caches”, at specific locations marked by coordinates all over the world.
Mystery/puzzle caches require one to discover information or solve a puzzle to find the cache. Some mystery caches provide a false set of coordinates with a puzzle that must be solved to determine the final cache location. In other cases, the given location is accurate, but the name of the location or other features are themselves a puzzle leading to the final cache. Alternatively, additional information is necessary to complete the find, such as a padlock combination to access the cache.


text: Ugh? Ugh. Ugh? Ugh. Ugh? Ugh. Ugh? Ugh. Ugh? Ugh. Ugh! Ugh! Ugh? Ugh! Ugh. Ugh? Ugh! Ugh! Ugh! Ugh! Ugh! Ugh. Ugh. Ugh? Ugh! Ugh! Ugh! Ugh! Ugh! Ugh! Ugh! Ugh. Ugh. Ugh? Ugh. Ugh. Ugh. Ugh. Ugh! Ugh. Ugh. Ugh? Ugh. Ugh. Ugh. Ugh. Ugh. Ugh. Ugh! Ugh. Ugh! Ugh! Ugh! Ugh! Ugh!

Or about 100 lines that look something like this

So what does that have to do with the tech world?
Well, it’s the way people hide the coordinates.


The difficulty of the riddle and the reachability of the box is measured with a scale from 1 – 5 stars. If you’re looking for a real challenge in solving riddles, you want to go for caches with the rating D3 – D5 (D stands for difficulty here).

Sometimes you just get a cryptic cache description with pretty much only “[Coordinates]: 0e9bfe8f349ed4b75d480743d1ab55e6e83c8176” as a hint

Others just consist of an image. Images can hold a lot of information – if you know where to look.
Steganography – the practice of concealing a file, message, image, or video within another file, message, image, or video – is also often used to hide coordinates from plain view!

I really enjoy geocaching because it’s a nicely balanced pastime. I get to use both my brain and my legs! (A lot of the caches are also wheelchair accessible too though)
Sometimes you even get to see some really cool new places you would have never gone to!

If this sounds like something you could enjoy – just check your area at or the open source alternative where you can select your preferred language and country in the header. Both are using OpenStreetMap for their maps.

My preferred app for caching is c:geo which is an open source app for Android – check out their project on github.

Look for the little  '?' icon  icons for mystery caches


Caches I hinted in my post:
1: Password Swordfish
2: Ugh! Ugh?
3: Nano
4: The series: How Do I Solve These #@&%$ Puzzles?!!

Feu Mourek
Feu Mourek

Feu verbrachte ihre Kindheit im schönen Steigerwald und kämpfte sich bis zum Abitur durch die Schule. Seit September 2016 unterstützt sie nun im Rahmen ihrer Ausbildung zum Fachinformatiker die Development Abteilung bei Netways und widmet sich dort dem Web Development. Ihre Freizeit verbringt sie hauptsächlich in den virtuellen Welten von 'Dota 2' und diversen anderen Games, an der Kletterwand in der Boulderhalle oder damit ihren Freunden und Kollegen auf die Nerven zu gehen.
How to: Merge multiple Git repositories into one

How to: Merge multiple Git repositories into one

Some time ago, I worked on a project that was split into multiple Git repositories. After a few weeks we decided, that it wasn’t longer necessary to have multiple repositories for this project, so we decided to merge them. The question was, if it is possible to merge multiple code bases without losing their history. The answer is yes. We have two ways on how to tackle this. The first way uses one of the repositories as the new main repository. The second option is to create a new repository for that purpose. We chose option one, because we already had a repository that acted as the main repository. If you choose to create a new repository, you can still follow the steps below. The only difference is, that you need at least one commit on that new repository, to be able to merge into it.

Before merging our repositories, we might first have to do some preparation on them. To prevent merge conflicts, you could move all files into a new directory, before merging them.

# Create a new directory
mkdir repo1
# Move all files into repo1
mv * repo1
# Commit those changes
git commit -am "Prepare for repository merge"
# Push changes to origin
git push

When thats done, add the repository as a remote on the new main repository.

git remote add repo1

The merge:
After that the merge is quite simple. We just have to append --allow-unrelated-histories to allow the merge of unrelated code bases.

git merge repo1/master --allow-unrelated-histories

If thats successful, the merge is done.

Redo the steps above for every repository you want to merge.

Noah Hilverling
Noah Hilverling

Nachdem Noah bei einer vierjährigen Exkursion nach Belgien seine Liebe zum Programmieren entdeckte, holte der gebürtige Euskirchener innerhalb kürzester Zeit gleich zwei Schulabschlüsse nach. Danach verließ Noah sogar den schönen Chiemsee, um sich ab September 2016 im Rahmen der Ausbildung zum Fachinformatiker für Anwendungsentwicklung bei NETWAYS voll und ganz dem Programmieren hinzugeben und viele unterschiedliche Erfahrungen zu sammeln. Wenn er mal nicht am Programmieren und Zocken ist, brettert er mit seinem Snowboard die Pisten runter,...

It all started with a GameBoy

It’s time to reflect and talk about video games. On how I got into programming and what drives me.

MissingNo: The glitch, the legend, my beginnings

Catch them all!

Pokémon, the first generation Game Boy games. They were the first thing that made me wonder just how programs and computers work. Back in the late 20th and early 21th century Pokémon Red and Blue were the talk of the schoolyard, if you are Generation X or a Millennial you probably know what I’m talking about. Fake rumors on how to get the rarest monsters with absurd guides were floating around: Do X while Y and have two of Z but don’t feed them after midnight. What gave these urban legends credibility were bugs in the games which had seemingly unrelated steps lead to weird and for a child even scary results – from characters being cut up and incorrectly reassembled over save games becoming unusable to a constant, never ending screaming sound.

For those who have not played these classics, I’m talking about the now famous MissingNo Glitch. Now we know the bug is caused by incorrect read and writes in the game. The games were written in Assembly and the programmers had very limited memory and space to work with, something was bound to break and such bugs are not uncommon in early console games. Most kids were just happy to have an infinite supply of Master Balls, an item in the game that could only be acquired once in the game, or get their favorite monster to the highest level quickly. For me this was enough at first as well but as time went on I grew more and more intrigued by the bug.

Minus World. A well known bug caused by incorrectly loading a level.

I asked my dad, he had no clue and neither did my mom.The internet was hard to use then still and I did not find my answers then. What I found was more confusing information and ways to manipulate the game, mostly collected by trial and error of other players, but there were also mentions of buffer overruns, memory violation and other terms I could not make sense of and didn’t hear again until I was allowed to watch The Matrix. This knowledge of the games made me the coolest kid on the playground for a while at least.

And after the Elite Four?

Only when I joined the local hackerspace and got involved with the CCC I finally got my entry point to the world of computers and programming. There was referred the book Learn Python The Hard Way and started writing code. Sadly none of my early work exists anymore, of git and GitHub I learned later still. The obvious choice then was to go to Uni and sign up for Computer Science class, three semesters I spent trying to wrap my head around the math needed to pass but ultimately quit because of it.

But my interest in programming was unbroken, I loved classes like Systems Programming which had assignments where you had to implement basic tools yourself, my own shell, my own email server, netcat – everything in C of course. That’s when I found my way to NETWAYS as an apprentice and have stayed here since, they let me write code. The code I write has changed, abstraction and new languages like Go have changed how I program but the lessons I learned from playing Pokémon in my bedroom still hold true: Sometimes it takes time to understand a bug.

If you’d like to join me in hunting bugs or talking retro games over a cup of coffee, head on over to our jobs page!

Diana Flach
Diana Flach

Geboren und aufgewachsen in Bamberg kam Diana, nach einem Ausflug an die Uni, als Azubi zu NETWAYS. Dort sitzt sie seit 2014 im Icinga 2 Core Entwicklungsteam.
Rootconf 2019 in Bangalore

Rootconf 2019 in Bangalore

I am very happy and thankful to be part of Rootconf in Bangalore again. My friends in Bangalore organise an amazing event and I can definitely recommend it.

Here is the official press release …

Setting up and managing infrastructure in technology companies is a complex, expensive and challenging proposition. HasGeek’s Rootconf community brought together by the annual conference will discuss and deliberate on this topic including:

  1. Running secure systems and using security tools such as OSINT to track vulnerabilities.
  2. Automating infrastructure security, and gearing teams towards a culture of development, security and operations.
  3. How companies such as Uber, Amazon, GO-JEK, Alibaba Cloud and others run their infrastructure at scale and in a distributed manner, across geographies.
  4. Importance of open standards and how engineers can choose their tools wisely to ensure that they don’t get locked-in with vendors.

Dates:​ 21-22 June 2019
Time:​ 9:30 AM – 6:00 PM
Venue:​ NIMHANS Convention Centre, Bangalore
Event information:​ ​

Systems engineers, DevOps programmers, SRE, team leads, VPs of engineering, security professionals and communities will participate in the 2019 edition of Rootconf. Speakers include Bernd Erk from NETWAYS, Brian McKenna of Atlassian, Kushal Das from Freedom of Press Foundation, Lavakumar Kuppan of Ironwasp security, Shrey Agarwal from Paytm, and Chandrashekhar Bhonsale of CRED.

Apart from talks, Birds of a Feather (BoF) sessions on the following topics will allow participants to discuss:

  • Security paranoid OS​ – led by Sayan Chowdhury, Red Hat
  • Art and science of choices in engineering​ – led by Srujan Akumarthi, Webengage
  • DevSecOps​ ​and the problems with over-engineering security ​- led by Neelu Tripathy, Thoughtworks
  • Building a 100% remote workplace​ – led by Ratnadeep Debnath, Zapier

Other participating organizations in Rootconf include Razorpay, WalmartLabs, Zapier, Red Hat, Atlassian, CRED and Paytm. The 2019 edition of Rootconf is expected to have a footfall of over 600+ participants.

Bernd Erk
Bernd Erk

Bernd ist Geschäftsführer der NETWAYS Gruppe und verantwortet die Strategie und das Tagesgeschäft. Bei NETWAYS kümmert er sich eigentlich um alles, was andere nicht machen wollen oder können (meistens eher wollen). Darüber hinaus startet er das wöchentliche Lexware-Backup und investiert seine ganze Energie in den Rest der Truppe und versucht für kollektives Glück zu sorgen. In seiner Freizeit macht er mit sinnlosen Ideen seine Frau verrückt und verbündet sich dafür mit seinem Sohn.