Select Page


GitLab-CI / YAML – Write less with Anchors, Extends and Hidden Keys

Have you ever wanted to execute a GitLab-CI job for multiple operating systems and just copied every line of YAML multiple times?
Anchors, extends and hidden keys are coming to rescue!

Let’s say you have two jobs and the only difference between them being a single environment variable:

  - echo

  stage: echo
    - echo $ECHO_TEXT
    ECHO_TEXT: "Hello world!"

  stage: echo
    - echo $ECHO_TEXT
    ECHO_TEXT: "Bye bye!"

Anchors and extends

Writing the same job two times can already get quite messy and hard to maintain. The more jobs you add, the worse it gets.
But don’t worry, YAML has got you covered. Anchors and extends let you reuse parts of your config and extend on them.

In this example, we create the echo-hello job and extend on it in the echo-bye task:

  - echo

echo-hello: &echo #create an anchor named "echo"
  stage: echo
    - echo $ECHO_TEXT
    ECHO_TEXT: "Hello world!"

  <<: *echo #use the anchor created above and extend it by using "<<"
    ECHO_TEXT: "Bye bye!"

Templating with hidden keys

One thing you can do to further improve on that is, by using a separate task just for templating using hidden keys.
Hidden keys can be defined in YAML using a . in front of a keys name. This prevents GitLab-CI from executing a job and allows us to use it as a template.

In our last example, we create an echo template job containing our stage and script. The echo job is then extended on in echo-hello and echo-bye:

 - echo

.echo: &echo #keys (jobs in this case) with a dot in front are hidden keys and won't be executed by GitLab
  stage: echo 
    - echo $ECHO_TEXT  

  <<: *echo 
    ECHO_TEXT: "Hello world!"  

  <<: *echo  
    ECHO_TEXT: "Bye bye!"

Some real world examples can be found in our public Icinga 2 packaging repositories:

GitLab CI Runners with Auto-scaling on OpenStack


With migrating our CI/CD pipelines from Jenkins to GitLab CI in the past months, we’ve also looked into possible performance enhancements for binary package builds. GitLab and its CI functionality is really really great in this regard, and many things hide under the hood. Did you know that “Auto DevOps” is just an example template for your CI/CD pipeline running in the cloud or your own Kubernetes cluster? But there’s more, the GitLab CI runners can run jobs in different environments with using different hypervisors and the power of docker-machine.

One of them is OpenStack available at NWS and ready to use. The following examples are from the Icinga production environment and help us on a daily basis to build, test and release Icinga products.



Install the GitLab Runner on the GitLab instance or in a dedicated VM. Follow along in the docs where this is explained in detail. Install the docker-machine binary and inspect its option for creating a new machine.

curl -L | sudo bash
apt-get install -y gitlab-runner
curl -L `uname -s`-`uname -m` -o /usr/local/bin/docker-machine
chmod +x /usr/local/bin/docker-machine
docker-machine create --driver openstack --help

Next, register the GitLab CI initially. Note: This is just to ensure that the runner is up and running in the GitLab admin interface. You’ll need to modify the configuration in a bit.

gitlab-runner register \
  --non-interactive \
  --url \
  --tag-list docker \
  --registration-token SUPERSECRETKEKSI \
  --name "docker-machine on OpenStack" \
  --executor docker+machine \
  --docker-image alpine


Docker Machine with OpenStack Deployment

Edit “/etc/gitlab-runner/config.toml” and add/modify the “[[runners]]” section entry for OpenStack and Docker Machine. Ensure that the MachineDriver, MachineName and MachineOptions match the requirements. Within “MachineOptions”, add the credentials, flavors, network settings just as with other deployment providers. All available options are explained in the documentation.

vim /etc/gitlab-runner/config.toml

    IdleCount = 4
    IdleTime = 3600
    MaxBuilds = 100
    MachineDriver = "openstack"
    MachineName = "customer-%s"
    MachineOptions = [
      "openstack-image-name=Debian 10.1",
      "openstack-keypair-name=GitLab Runner"

The runners cache can be put onto S3 granted that you have this service available. NWS luckily provides S3 compatible object storage.

    Type = "s3"
    Shared = true
      ServerAddress = "s3provider.domain.localdomain"
      AccessKey = "supersecretaccesskey"
      SecretKey = "supersecretsecretkey"
      BucketName = "openstack-gitlab-runner"

Bootstrap Docker in the OpenStack VM

Last but not least, these VMs need to be bootstrapped with Docker inside a small script. Check the “–engine-install-url” parameter in the help output:

root@icinga-gitlab:/etc/gitlab-runner# docker-machine create --help
  --engine-install-url ""							Custom URL to use for engine installation 

You can use the official way of doing this, but putting this into a small script also allows customizations like QEMU used for Raspbian builds. Ensure that the script is available via HTTP e.g. from a dedicated GitLab repository ūüėČ

# This script helps us to prepare a Docker host for the build system
# It is used with Docker Machine to install Docker, plus addons
# See --engine-install-url at docker-machine create --help

set -e

run() {
  (set -x; "$@")

echo "Installing Docker via"
run curl -LsS -o /tmp/
run sh /tmp/

echo "Installing QEMU and helpers"
run sudo apt-get update
run sudo apt-get install -y qemu-user-static binfmt-support

Once everything is up and running, the GitLab runners are ready to fire the jobs.



Jobs and builds are not run all the time, and especially with cloud resources, this should be a cost-efficient thing. When building Icinga 2 for example, the 20+ different distribution jobs generate a usage peak. With the same resources assigned all the time, this would tremendously slow down the build and release times. In that case, it is desirable to automatically spin up more VMs with Docker and let the GitLab runner take care of distributing the jobs. On the other hand, auto-scaling should also shut down resources in idle times.

By default, one has 4 VMs assigned to the GitLab runner. These builds run non-privileged in Docker, the example below also shows another runner which can run privileged builds. This is needed for Docker-in-Docker to create Docker images and push them to GitLab’s container registry.

root@icinga-gitlab:~# docker-machine ls
NAME                                               ACTIVE   DRIVER      STATE     URL                      SWARM   DOCKER     ERRORS
runner-privileged-icinga-1571900582-bed0b282       -        openstack   Running   tcp://           v19.03.4
runner-privileged-icinga-1571903235-379e0601       -        openstack   Running   tcp://           v19.03.4
runner-non-privileged-icinga-1571904408-5bb761b5   -        openstack   Running   tcp://           v19.03.4
runner-non-privileged-icinga-1571904408-52b9bcc4   -        openstack   Running   tcp://           v19.03.4
runner-non-privileged-icinga-1571904408-97bf8992   -        openstack   Running   tcp://           v19.03.4
runner-non-privileged-icinga-1571904408-97bf8992   -        openstack   Running   tcp://           v19.03.4

Once it detects a peak in the pending job pipeline, the runner is allowed to start additional VMs in OpenStack.

root@icinga-gitlab:~# docker-machine ls
NAME                                               ACTIVE   DRIVER      STATE     URL                      SWARM   DOCKER     ERRORS
runner-privileged-icinga-1571900582-bed0b282       -        openstack   Running   tcp://           v19.03.4
runner-privileged-icinga-1571903235-379e0601       -        openstack   Running   tcp://           v19.03.4
runner-non-privileged-icinga-1571904408-5bb761b5   -        openstack   Running   tcp://           v19.03.4
runner-non-privileged-icinga-1571904408-52b9bcc4   -        openstack   Running   tcp://           v19.03.4
runner-non-privileged-icinga-1571904408-97bf8992   -        openstack   Running   tcp://           v19.03.4
runner-non-privileged-icinga-1571904408-97bf8992   -        openstack   Running   tcp://           v19.03.4


runner-non-privileged-icinga-1571904534-0661c396   -        openstack   Running   tcp://           v19.03.4
runner-non-privileged-icinga-1571904543-6e9622fd   -        openstack   Running   tcp://           v19.03.4
runner-non-privileged-icinga-1571904549-c456e119   -        openstack   Running   tcp://           v19.03.4
runner-non-privileged-icinga-1571904750-8f6b08c8   -        openstack   Running   tcp://           v19.03.4


In order to achieve this setting, modify the runner configuration and increase the limit.

vim /etc/gitlab-runner/config.toml

  name = "docker-machine on OpenStack"
  limit = 24
  output_limit = 20480
  url = ""
  token = "supersecrettoken"
  executor = "docker+machine"

This would result in 24 OpenStack VMs after a while, and all are idle 24/7. In order to automatically decrease the deployed VMs, use the OffPeak settings. This also ensures that resources are available during workhours while spare time and weekend are considered “off peak” with shutting down unneeded resources automatically.

    OffPeakTimezone = "Europe/Berlin"
    OffPeakIdleCount = 2
    OffPeakIdleTime = 1800
    OffPeakPeriods = [
      "* * 0-8,22-23 * * mon-fri *",
      "* * * * * sat,sun *"

Pretty neat functionality ūüôā


Troubleshooting & Monitoring

“docker-machine ls” provides the full overview and tells whenever e.g. a connection to OpenStack did not work, or if the VM is currently unavailable.

root@icinga-gitlab:~# docker-machine ls
NAME                                               ACTIVE   DRIVER      STATE     URL                      SWARM   DOCKER     ERRORS
runner-privileged-icinga-1571900582-bed0b282       -        openstack   Error                                      Unknown    Expected HTTP response code [200 203] when accessing [GET], but got 404 instead

In case you have deleted the running VMs to start fresh, provisioning might take a while and the above can be a false positive. Check the OpenStack management interface to see whether the VMs booted correctly. You can also remove a VM with “docker-machine rm <id>” and run “gitlab-runner restart” to automatically provision it again.

Whenever the VM provisioning fails, a gentle look into the syslog (or runner log) unveils what’s the problem. Lately we had used a wrong OpenStack flavor configuration which was fixed after investigating in the logs.

Oct 18 07:08:48 3 icinga-gitlab gitlab-runner[30988]:  #033[31;1mERROR: Error creating machine: Error in driver during machine creation: Unable to find flavor named 1234-customer-id-4-8#033[0;m  #033[31;1mdriver#033[0;m=openstack #033[31;1mname#033[0;m=runner-non-privilegued-icinga-1571375325-3f8176c3 #033[31;1moperation#033[0;m=create

Monitoring your GitLab CI runners is key, and with the help of the REST API, this becomes a breeze with Icinga checks. You can inspect the runner state and notify everyone on-call whenever CI pipelines are stuck.



Developers depend on fast CI feedback these days, speeding up their workflow – make them move fast again. Admins need to understand their requirements, and everyone needs a deep-dive into GitLab and its possibilities. Join our training sessions for more practical exercises or immediately start playing in NWS!

GitLab Commit London Recap

A while ago, when GitHub announced CI/CD for the upcoming actions feature, I’ve been sharing with Priyanka on Twitter how we use GitLab. The GitLab stack with the runners, Docker registry and all-in-one interface not only speeds up our development process and packaging pipelines, it also scales our infrastructure deployments even better. With the love for GitLab, we’ve also created our GitLab training sharing all the knowledge about this great tool stack.


GitLab, GitLab, GitLab

Priyanka was so kind to invite us over to GitLab Commit in London, GitLab’s first European user conference. At first glance, Bernd and I didn’t know what would happen – turns out, this wasn’t a product conference. Instead, meeting new people and learning how they use GitLab in their environments was put into focus. Sid Sijbrandij, GitLab’s CEO, kicked off the event in the morning and after sharing the roots of GitLab being in Europe, he asked the audience to “meet your neighbour and connect”. Really an icebreaker for the coming talks and sessions.

The next keynote was presented by engineers from Porsche sharing their move to GitLab. With Java Boot Spring and iOS application development, and the requirement of deeper collaboration between teams, they took the challenge and are using GitLab since ~1 year. Interesting to learn was that nearly everyone at GitLab Commit uses Terraform for deployments. Matt did so too in his live coding session with a full blown web application Kubernetes container setup all managed and deployed with GitLab and Terraform. After 20 minutes of talking way too fast, it worked. What a great way of showing what’s possible with today’s tool stack!


DevOps for everyone

One thing I’ve also recognized – everyone seems to be moving to Kubernetes and Terraform. Rancher, Jenkins and other tools in the same ecosystem seem to be falling short in modern DevOps environments. I really liked the security panel where ideas like automated dependency scanning in merge requests have been shared. Modern days with easy to use libraries typically pull in lots of unforeseen dependencies and who really knows about all the vulnerabilities? Blocking the merge request in case of emergency is a killer feature for current development workflows.

In terms of the product roadmap, GitLab has a huge vision which is not easy to summarize. On the other hand, having a maybe-not-reachable vision empowers a great team to work even harder. The short term improvements for CI are for example Directed Acyclic Graphs allowing parallel pipelines to continue faster. This will greatly enhance our package pipelines in the future. While tweeting about this, Jason was so kind to share the build matrix feature known from Travis coming soon with GitLab 12.6. Spot on, testing e.g. different PHP versions for the same job is greatly missed being as easy as Travis. GitLab Runners will receive support for ARM soon, and also Vault integration is coming. GitLab also announced their startup Meltano, an open source data to dashboard workflow platform Рlooks really promising.

The afternoon sessions were split into 3 tracks each, with even more user stories. Moving along from Delta with their many of thousands of repositories, we’ve also learned more about VMware’s cloud architects and how they incorporate GitLab & Terraform for deployments. Last but not least we’ve joined Philipp sharing his story on migrating from Jenkins to GitLab CI. Since we struggled from the same problems (XML config, plugins breaking upgrades, etc.) we were enlightened to see that he even developed a GitHub to GitLab issue migrator, fully open source. Moving to a central platform and away from 5+ browser tabs really is a key argument in stressful (development) times. Avoiding context switches for developers improves quality and ensures better releases from my experience.


Get the party started

The evening event took place at Swingers, a bar which had a Mini Golf playground built-in. We were going there with the iconic London Bus, and the nice people from GitLab even ensured that Gin&Tonic made this a great starter. Right after arriving, the fire alarm rang off and we had to move outside. Party like NETWAYS ūüėȬ†And finally we met Priyanka to say hi, lovely memory. We also met Brian, proud Irish, challenging us with funny stories and finding out that Germans do not know everything. Really charming and much to laugh.

Thanks GitLab for this top notch event and see you next year!

Migration von GitLab mit Upgrade auf EE

Wer GitLab CE produktiv im Einsatz hat und mit den zus√§tzlichen Features der EE Version lieb√§ugelt, der wird sich beim Umstieg zwangsl√§ufig mit den¬†Migrationsschritten auseinandersetzen, sofern der GitLab Server nicht von einem Hoster betreut wird. In diesem Post zeige¬†ich, wie der Wechsel inklusive Migration auf einen anderen Server gelingen kann und beziehe mich dabei auf die Omnibus Version basierend auf Ubuntu 18.04.¬†Der Ablauf ist gar nicht so kompliziert. B√ľgelt man die EE-Version einfach √ľber die aktuelle CE Version, dann hat man nur zwei Schritte zu beachten.¬†Wenn man allerdings einen EE Server parallel hochzieht, um dann auf diesen zu migrieren, so kommen ein paar mehr Schritte hinzu.
Deshalb zeige ich im Folgenden wie man per Backup und Restore auch zum Ziel kommt. Die ersten Schritte sind ziemlich identisch mit dem, was GitLab vorgibt.

  1. Zuerst erstellt man sich ein Backup:
    gitlab-rake gitlab:backup:create STRATEGY=copy

    Einfach um sich den aktuellen Stand vor der Migration zu sichern.
    Hier kann nicht viel schief gehen. Man sollte aber darauf achten, dass gen√ľgend Speicherplatz unter /var/opt/gitlab/backups zur Verf√ľgung steht.¬†Es sollten mindestens noch zwei Drittel des Speicherplatzes frei sein.¬†Das resultierende Tar-Archiv sollte man sich anschlie√üend weg kopieren, da im weiteren Verlauf ein weitere Backup erstellt wird.

  2. ¬†Nun f√ľhrt man das Script von GitLab aus, das die apt-Sourcen hinzuf√ľgt und ein paar ben√∂tigte Pakete vorinstalliert:
    curl | sudo bash
  3. Jetzt checkt man noch kurz welche Versionsnummer von GitLab CE momentan installiert ist:
    dpkg -l |grep gitlab-ce
  4. Danach installiert man die GitLab EE Version. Dabei wird die CE Version gel√∂scht und eine Migration durchgef√ľhrt.¬†Um den Versionsstand kompatibel zu halten, verwendet man die gleiche Versionsnummer wie aus dem vorherigen Schritt.¬†Lediglich der teil ‘ce’ wird zu ‘ee’ abge√§ndert:
    apt-get update && sudo apt-get install gitlab-ee=12.1.6-ee.0
  5. Nun erstellt man erneut ein Backup:
    gitlab-rake gitlab:backup:create STRATEGY=copy
  6. Das neu erstellte Backup transferiert man einschließlich der Dateien /etc/gitlab/gitlab-secrets.json und /etc/gitlab/gitlab.rb auf den EE Server. Das lässt sich z.B. per scp bewerkstelligen:
    scp /var/opt/gitlab/backups/*ee_gitlab_backup.tar ziel-server:./
    scp /etc/gitlab/gitlab-secrets.json ziel-server:./
    scp /etc/gitlab/gitlab.rb ziel-server:./
  7. Auf dem Zielserver sollte man nat√ľrlich GitLab EE in der entsprechenden Version installieren.¬†Hier h√§lt man sich am besten an die offizielle Anleitung.¬†Nicht vergessen bei der Installation die Versionsnummer anzugeben.
  8. Jetzt verschiebt man die Dateien die man per scp transferiert hat und setzt die Dateiberechtigungen:
    mv ~/*ee_gitlab_backup.tar /var/opt/gitlab/backups
    mv ~/gitlab-secrets.json ~/gitlab.rb /etc/gitlab/
    chown root:root /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab.rb
    chown git:git /var/opt/gitlab/backups/*ee_gitlab_backup.tar
  9. Dann startet man den Restore:
    gitlab-rake gitlab:backup:restore

    Man quittiert die einzelnen Abfragen mit ‘yes’.¬†Hat sich die URL f√ľr den neuen EE Server ge√§ndert, dann sollte man das in der /etc/gitlab.rb anpassen.¬†In diesem Fall sind auch √Ąnderungen an den GitLab Runnern vorzunehmen. Es reicht dann allerdings wenn man auf dem jeweiligen Runner in der Datei config.toml die URL in der [[runners]] Sektion anpasst, da der Token gleich bleibt.


Es ist allerdings auch möglich, dass es zu Problemen mit den Runnern kommt. Dies zeigt sich z.B. dadurch, dass der Runner in seinen Logs 500er-Fehler beim Verbindungsversuch zum GitLab meldet. In diesem Fall sollte man zuerst versuchen den Runner neu zu registrieren. Falls die Fehler bestehen bleiben, ist es möglich, dass diese durch einen CI-Job verursacht werden, der während der Migration noch lief. So war es zumindest bei mir der Fall. Mit Hilfe der Anleitung zum Troubleshooting und den Infos aus diesem Issue kam ich dann zum Ziel:

gitlab-rails dbconsole
UPDATE ci_builds SET token_encrypted = NULL WHERE status in ('created', 'pending');

Wenn man sich das alles sparen möchte, dann lohnt es sich einen Blick auf unsere GitLab EE Angebote der NETWAYS Web Services zu werfen.

Gabriel Hartmann
Gabriel Hartmann
Senior Systems Engineer

Gabriel hat 2016 als Auszubildender Fachinformatiker f√ľr Systemintegrator bei NETWAYS angefangen und 2019 die Ausbildung abgeschlossen. Als Mitglied des Web Services Teams k√ľmmert er sich seither um viele technische Themen, die mit den NETWAYS Web Services und der Weiterentwicklung der Plattform zu tun haben. Aber auch im Support engagiert er sich, um den Kunden von NWS bei Fragen und Problemen¬†zur Seite zu stehen.

Monthly Snap May

May at NETWAYS was above all the month of the OSDC, but our team still found the time to share some news and thoughts with us all!

It all started with development…

Alexander started the month with his Blogpost Generics waren gestern. Lang lebe Golangs Reflection! And he raises the question if we really need Generics, or if Golangs would do too.
Jean wrote that it all started with a Gameboy and let us take a peek at her personal journey into programming.
Digging up hidden information for a scavenger hunt? Feu reflected on the thrill in geocaching, solving puzzles and hidden riddles.
Noah got technical! How to: Merge multiple Git repositories into one

Live from the OSDC!

Michael shared the latest news live from Berlin: OSDC 2019: Buzzwo…erm…DevOps, Agile & YAML programmers
And so did Tim: OSDC 2019 Part 2 ‚Äď Automating patching, VMs in containers & much more

News from the NETWAYS shop!

Immer im Takt bleiben ‚Äď mit Gude. As Nicole informed us, it is possible to synchronize the inbuilt clock in your hardware per GPS!
TinkerForge is still new in the NETWAYS shop, and in May Nicole presented a new Check-Plugin for TinkerForge Hardware.
Nicole also proudly announced that SMS Eagle sponsored the OSDC for the first time! Welcome SMS Eagle!

Spectacular lottery!

Did you know that our NWS team decided to hold a little competition with several great prices? The first price was winning our very own CEO Bernd Erk for a whole day! Read all about it in Marius` blogpost NETWAYS Web Services Raffle

Poetic and philosophical consultants!

Thomas once again showed his poetic skills in a fun rhyme on Icinga. This is definitely worth the read! Irgendwer muss es halt machen!
Gedanken tanken is Tobias idea of feeding the brain useful but also entertaining Input.
Our apprentice Tobias went on his first business trip! Meine erste Reise bei NETWAYS! He held an Icinga advanced training with Lennart in Munich. Meanwhile Afeef, another junior consultant, dove deep into comparing Podman vs Docker.
Viel hilft viel- nicht immer, Thomas claims! He used Elasticsearch to monitor thresholds in Java and shared useful tips with the community.

Just fit- just awesome!

Healthy living (or at least a bit healthier) is still an important subject at the NETWAYS HQ. Nadja gave us tips on how to treat your back well Just fit ‚Äď just awesome: R√ľckenschule How should we sit? What exercises can easily be carried out while seated in front of the PC? Many helpful tips in this one!
Then Catharina gave a summary on the first ever NETWAYS Wellbeing day! A day packed with healthy recipes, business yoga and other interesting components that helped us heighten our awareness and to lock out the world for a moment.

What`s next?

Julia invited us to the Foreman birthday party taking place here in Nuremberg on July 25! Read all about it and register for a great program. It is all for free! New and Next in Automation: Notes on OSCamp Berlin! Julia reviewed the OSCamp and reminded us of the next one on Foreman. It will take place in Nuremberg on November 7, right after the OSMC! The highlight in May was definitely the OSDC in Berlin. Julia reported about a terrific conference with a great atmosphere. OSDC 2019: Unique views and phenomenal people. And soon afterwards the OSDC Archive was online!

And the others?

Achim contemplated the advantages of mtr compared to traceroute for tracing your routing paths in your network. Finde den Weg mit mtr und traceroute
Martin presented our Icinga starter pack, perfect for the first steps with Icinga, and Blerim from Team Icinga announced future Icinga Camps in Stockholm, Mailand und Z√ľrich!
And last, but not least Bernd is looking forward to Rootconf 2019 in Bangalore taking place June 21 to 22.

Catharina Celikel
Catharina Celikel
Office Manager

Catharina unterst√ľtzt seit M√§rz 2016 unsere Abteilung Finance & Administration. Die geb√ľrtige Norwegerin ist Fremdsprachenkorrespondentin f√ľr Englisch. Als Office Manager k√ľmmert sie sich deshalb nicht nur um das Tagesgesch√§ft sondern √ľbernimmt nebenbei zus√§tzlich einen Gro√üteil der √úbersetzungen. Privat ist der bekennende B√ľcherwurm am liebsten mit dem Fahrrad unterwegs.