pixel
Seite wählen

NETWAYS Blog

HUGO: GitLab-CI/CD-Pipeline für eine statische Website

Vor etwa 4 Monaten habe ich hier einen Blogpost geschreiben, in dem ich Hugo vorgestellt habe – eine Software zum Generieren statischer Webseiten aus Markdown-Dateien.

Im Lauf meiner Ausbildung zum Fachinformatiker für Systemintegration bei NETWAYS habe ich vor kurzem an einer GitLab Fundamentals Schulung teilgenommen, um mehr über Git im allgemeinen und die Besonderheiten von GitLab im speziellen zu lernen. Auf Basis dieser Schulung und dem Projekt hinter oben genannten Blogpost habe ich nun eine CI/CD-Pipeline – CI/CD steht für Continous Integration and Continous Deployment – zum automatisierten Testen, Bauen und Ausrollen einer mit Hugo erzeugten Website gebaut.

Für dieses Projekt habe ich NETWAYS Web Services (NWS) eine GitLab CE App gestartet und außerdem in der Cloud von NWS zwei Webserver – einen als Testumgebung, einen als Produktivumgebung. Mithilfe meines Laptops als Client habe ich an der Website gearbeitet und die anfallenden Daten regelmäßig in ein eigenes GitLab Repository gepusht. Zum Testen, Bauen und Ausrollen auf die beiden Webserver laufen in der GitLab App zwei GitLab Runner. Das sind im Prinzip Agenten die für die GitLab App auf einem anderen System Befehle ausführen können.

Die CI/CD Pipeline

Die CI/CD Pipeline wird über die .gltiab-ci.yml gesteuert. Anfangs werden in der Pipeline die Quelldateien mithilfe zweier Markdown-Linter (vale.sh und markdownlint) getestet – in der .gitlab-ci.yml schaut das so aus:

lint:
   stage: lint
   tags: 
    - hugo
   allow_failure: true
   script:
    - cd tutorial
    - mdl ./content
    - vale ./content

Diese überprüfen die Inhalte der Website auf die Einhaltung eines Styleguides und auch auf die Sprachliche Korrektheit. Anschließend wird die Webseite mit Hugo gerendert, das heißt aus den Markdown-Dateien für den Websiteinhalt entsteht nun die wirkliche Website:

testBuild:
   stage: build
   tags:
    - test
   script:
    - cd tutorial
    - mkdir test
    - hugo -DEF --debug -d test
   artifacts:
   paths:
    - tutorial/test

Falls diese Operation auf der Testumgebung erfolgreich ist, wird sie auch auf der Produktivumgebung durchgeführt. Als Abschluss wird die gerenderte Webseite noch für den genutzten Webserver (z.B. Apache HTTPD oder nginx) bereitgestellt):

testDeploy:
   stage: deploy
   needs: [testBuild]
   tags:
    - hugotest
   script:
    - cp -r tutorial/test/* /var/www/html/
   only: 
    - main

Grafisch sieht diese Pipeline so aus:

Wenn auch Du solche interessanten Projekte in Deiner Ausbildung zum Fachinformatiker machen möchtest, kannst du Dich gerne für eine Ausbildung ab Herbst 2022 bewerben!

Björn Berg
Björn Berg
Junior Consultant

Björn hat nach seinem Abitur 2019 Datenschutz und IT-Sicherheit in Ansbach studiert. Nach einigen Semestern entschied er sich auf eine Ausbildung zum Fachinformatiker für Systemintegration umzusteigen und fing im September 2021 bei NETWAYS Professional Services an. Auch in seiner Freizeit sitzt er viel vor seinem PC und hat Spaß mit diversen Spielen, experimentiert auch mit verschiedenen Linux-Distributionen herum und geht im Sommer gerne mal campen.

GitLab – Merge Requests

Merges werden verwendet, um den Code zwischen anderen Personen, die Sie an einem Projekt vorgenommen haben, auszutauschen und die Änderungen einfach miteinander zu konsolidieren.

Schritt 1: Vor dem Erstellen einer neuen Merge sollte im GitLab ein Branch erstellt werden.
Schritt 2: Melden Sie sich bei Ihrem GitLab-Konto an und gehen Sie zu Ihrem Projekt im Abschnitt Projekte.

Schritt 3: Klicken Sie auf die Registerkarte Merge und dann auf die Schaltfläche Neue Merge.

Schritt 4: Um den Request zu mergen, wählen Sie den Quell-Branch und den Ziel-Branch aus der Dropdown-Liste aus und klicken Sie dann auf die Schaltfläche Zweige vergleichen und fortfahren, wie unten gezeigt.

Merge kann verwendet werden, um den Code zwischen anderen Personen, die Sie an einem Projekt vorgenommen haben, auszutauschen und die Änderungen einfach mit ihnen zu besprechen.

Schritt 5: – Sie sehen den Titel, die Beschreibung und andere Felder wie Zuweisen des Benutzers, Festlegen des Meilensteins, Beschriftungen, Name des Quell-Branches und Name des Ziel-Branches und klicken auf die Schaltfläche “Merge senden”

Schritt 6: – Nach dem Absenden der Merge wird ein neuer Bildschirm für Merge angezeigt (siehe unten).

Noch kein Gitlab? Jetzt bei uns im NWS anmelden und Gitlab sorgenfrei 30 Tage lang testen.

Moumen Amneh
Moumen Amneh
Junior Systems Engineer

Mou startete seine Ausbildung zum Fachinformatiker Systemintegration im September 2019 bei uns. Mit vollem Ehrgeiz widmet er sich jetzt Puppet, Icinga und Co. und lernt jeden Tag neues dazu. Neben der Arbeit brilliert er auf dem Fussballplatz und das zum verwechseln ähnlich mit seinem Namensvetter bei Liverpool.

From Monitoring to Observability: Distributed Tracing with Jaeger

Modern cloud environments and microservice architectures need a changed mindset when it comes to monitoring. Classic host/service object relations are not always applicable, containers run in Kubernetes are short lived, and application performance within distributed networks is hard to monitor. Especially with applications running millions of operations, where to start the root cause analysis for slow client responses in your web shop? Is it just slow connections to MySQL, or does the application’s debug log slow down the entire fleet?

This is where observability with tracing comes into play. In the cloud native space, OpenTracing evolved as vendor neutral standardized API including client instrumentation. Famous tools are Zipkin and Jaeger which was contributed from Uber to the CNCF.

Let’s have a look into Jaeger today.

 

Getting Started

The easiest way to try Jaeger is with using the Docker container explained in the documentation.

docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14268:14268 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.16

Navigate to http://localhost:16686 to get greeted by Jaeger.

 

Try it

A sample application is available as container. I’m using a modified port mapping with 8081-8084 here since port 8080 is already assigned.

docker run --rm -it \
  --link jaeger \
  -p8081-8084:8080-8083 \
  -e JAEGER_AGENT_HOST="jaeger" \
  jaegertracing/example-hotrod:1.16 \
  all

Navigate to http://localhost:8081 and click the buttons to order some cars.

Within Jaeger itself, start analyzing the traces and for example learn that Redis runs into timeouts quite often delaying the HTTP responses to the clients.

 

Traces for applications

Jaeger provides officially supported client libraries for Go, Java, Python, NodeJS, C/C++, C#/.NET, others are in the making. Let’s see if we can add it into Icinga 2 and do some tracing here.

First off, clone the repository, build and install the client libraries. You’ll need CMake, a C++ compiler, etc. – basically everything which is required for Icinga 2 too and documented here. In this example, I’m compiling on my Macbook. There are additional libraries and headers required. Hint: Boost 1.72 has a bug which needs a patch.

brew install yaml-cpp thrift 
git clone https://github.com/opentracing/opentracing-cpp && cd opentracing-cpp
# 1.6.0 doesn't work atm
git checkout v1.5.1
mkdir -p build && cd build
cmake ..
make && make install
cd ..

Then clone, cmake, make, install.

git clone https://github.com/jaegertracing/jaeger-client-cpp && cd jaeger-client-cpp
git checkout v0.5.0
# regenerate thrift headers for 0.11.0
find idl/thrift/ -type f -name \*.thrift -exec thrift -gen cpp -out src/jaegertracing/thrift-gen {} \;
mkdir -p build && cd build
cmake ..
make && make install
cd ..

In order to add Jaeger into Icinga 2, there are the following steps necessary:

  • Add CMake functions to lookup yaml-cpp, opentracing, jaeger headers and libraries
  • Optionally enable Jaeger tracing code, link the icinga2 binary against it
  • Add a new tracer into the Config Compiler CLI command to measure the timing points

The majority of development time today was to resolve compile and linking issues, adding spans and traces is really simple. You can follow my progress in this Icinga PR – developers, get started wtih the client libraries and help your colleagues with enhanced observability!

 

Conclusion

Tracing application performance, cluster messages, end2end tests and any sort of event span provides valuable insights for both, devs and ops. With the new cloud native landscape evolving fast, we have additional possibilities to analyze our environments. Next to the now standardized tools for parsing and ingesting logs with Elastic Stack or Graylog, tracing has found its place in the observability stack.

Jaeger Tracing also is part of the GitLab observability suite which will be moved to the free core edition in 2020. Metrics, logging, alerts and tracing are key elements in modern cloud environments. Prometheus monitors everything from classic load checks to Kubernetes containers, Jaeger provides application performance insights and on top of that, Grafana combines the view on problems and trends. You can learn more about GitLab’s vision here.

Exploring these cool new features in GitLab are our mission in future trainings and workshops, watch this space in 2020!

GitLab-CI / YAML – Write less with Anchors, Extends and Hidden Keys

Have you ever wanted to execute a GitLab-CI job for multiple operating systems and just copied every line of YAML multiple times?
Anchors, extends and hidden keys are coming to rescue!

Let’s say you have two jobs and the only difference between them being a single environment variable:

stages:
  - echo

echo-hello:
  stage: echo
  script:
    - echo $ECHO_TEXT
  variables: 
    ECHO_TEXT: "Hello world!"

echo-bye:
  stage: echo
  script:
    - echo $ECHO_TEXT
  variables: 
    ECHO_TEXT: "Bye bye!"

Anchors and extends

Writing the same job two times can already get quite messy and hard to maintain. The more jobs you add, the worse it gets.
But don’t worry, YAML has got you covered. Anchors and extends let you reuse parts of your config and extend on them.

In this example, we create the echo-hello job and extend on it in the echo-bye task:

stages:
  - echo

echo-hello: &echo #create an anchor named "echo"
  stage: echo
  script:
    - echo $ECHO_TEXT
  variables: 
    ECHO_TEXT: "Hello world!"

echo-bye:
  <<: *echo #use the anchor created above and extend it by using "<<"
  variables: 
    ECHO_TEXT: "Bye bye!"

Templating with hidden keys

One thing you can do to further improve on that is, by using a separate task just for templating using hidden keys.
Hidden keys can be defined in YAML using a . in front of a keys name. This prevents GitLab-CI from executing a job and allows us to use it as a template.

In our last example, we create an echo template job containing our stage and script. The echo job is then extended on in echo-hello and echo-bye:

stages:
 - echo

.echo: &echo #keys (jobs in this case) with a dot in front are hidden keys and won't be executed by GitLab
  stage: echo 
  script: 
    - echo $ECHO_TEXT  

echo-hello: 
  <<: *echo 
  variables:  
    ECHO_TEXT: "Hello world!"  

echo-bye: 
  <<: *echo  
  variables: 
    ECHO_TEXT: "Bye bye!"

Some real world examples can be found in our public Icinga 2 packaging repositories: https://git.icinga.com/packaging/rpm-icinga2

GitLab CI Runners with Auto-scaling on OpenStack

 

With migrating our CI/CD pipelines from Jenkins to GitLab CI in the past months, we’ve also looked into possible performance enhancements for binary package builds. GitLab and its CI functionality is really really great in this regard, and many things hide under the hood. Did you know that “Auto DevOps” is just an example template for your CI/CD pipeline running in the cloud or your own Kubernetes cluster? But there’s more, the GitLab CI runners can run jobs in different environments with using different hypervisors and the power of docker-machine.

One of them is OpenStack available at NWS and ready to use. The following examples are from the Icinga production environment and help us on a daily basis to build, test and release Icinga products.

 

Preparations

Install the GitLab Runner on the GitLab instance or in a dedicated VM. Follow along in the docs where this is explained in detail. Install the docker-machine binary and inspect its option for creating a new machine.

curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
apt-get install -y gitlab-runner
  
curl -L https://github.com/docker/machine/releases/download/v0.16.2/docker-machine-`uname -s`-`uname -m` -o /usr/local/bin/docker-machine
chmod +x /usr/local/bin/docker-machine
  
docker-machine create --driver openstack --help

Next, register the GitLab CI initially. Note: This is just to ensure that the runner is up and running in the GitLab admin interface. You’ll need to modify the configuration in a bit.

gitlab-runner register \
  --non-interactive \
  --url https://git.icinga.com/ \
  --tag-list docker \
  --registration-token SUPERSECRETKEKSI \
  --name "docker-machine on OpenStack" \
  --executor docker+machine \
  --docker-image alpine

 

Docker Machine with OpenStack Deployment

Edit “/etc/gitlab-runner/config.toml” and add/modify the “[[runners]]” section entry for OpenStack and Docker Machine. Ensure that the MachineDriver, MachineName and MachineOptions match the requirements. Within “MachineOptions”, add the credentials, flavors, network settings just as with other deployment providers. All available options are explained in the documentation.

vim /etc/gitlab-runner/config.toml

  [runners.machine]
    IdleCount = 4
    IdleTime = 3600
    MaxBuilds = 100
    MachineDriver = "openstack"
    MachineName = "customer-%s"
    MachineOptions = [
      "openstack-auth-url=https://cloud.netways.de:5000/v3/",
      "openstack-tenant-name=1234-openstack-customer",
      "openstack-username=customer-login",
      "openstack-password=sup3rS3cr3t4ndsup3rl0ng",
      "openstack-flavor-name=s1.large",
      "openstack-image-name=Debian 10.1",
      "openstack-domain-name=default",
      "openstack-net-name=customer-network",
      "openstack-sec-groups="mine",
      "openstack-ssh-user=debian",
      "openstack-user-data-file=/etc/gitlab-runner/user-data",
      "openstack-private-key-file=/etc/gitlab-runner/id_rsa",
      "openstack-keypair-name=GitLab Runner"
    ]

The runners cache can be put onto S3 granted that you have this service available. NWS luckily provides S3 compatible object storage.

  [runners.cache]
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "s3provider.domain.localdomain"
      AccessKey = "supersecretaccesskey"
      SecretKey = "supersecretsecretkey"
      BucketName = "openstack-gitlab-runner"

Bootstrap Docker in the OpenStack VM

Last but not least, these VMs need to be bootstrapped with Docker inside a small script. Check the “–engine-install-url” parameter in the help output:

root@icinga-gitlab:/etc/gitlab-runner# docker-machine create --help
  ...
  --engine-install-url "https://get.docker.com"							Custom URL to use for engine installation 

You can use the official way of doing this, but putting this into a small script also allows customizations like QEMU used for Raspbian builds. Ensure that the script is available via HTTP e.g. from a dedicated GitLab repository 😉

#!/bin/sh
#
# This script helps us to prepare a Docker host for the build system
#
# It is used with Docker Machine to install Docker, plus addons
#
# See --engine-install-url at docker-machine create --help

set -e

run() {
  (set -x; "$@")
}

echo "Installing Docker via get.docker.com"
run curl -LsS https://get.docker.com -o /tmp/get-docker.sh
run sh /tmp/get-docker.sh

echo "Installing QEMU and helpers"
run sudo apt-get update
run sudo apt-get install -y qemu-user-static binfmt-support

Once everything is up and running, the GitLab runners are ready to fire the jobs.

 

Auto-Scaling

Jobs and builds are not run all the time, and especially with cloud resources, this should be a cost-efficient thing. When building Icinga 2 for example, the 20+ different distribution jobs generate a usage peak. With the same resources assigned all the time, this would tremendously slow down the build and release times. In that case, it is desirable to automatically spin up more VMs with Docker and let the GitLab runner take care of distributing the jobs. On the other hand, auto-scaling should also shut down resources in idle times.

By default, one has 4 VMs assigned to the GitLab runner. These builds run non-privileged in Docker, the example below also shows another runner which can run privileged builds. This is needed for Docker-in-Docker to create Docker images and push them to GitLab’s container registry.

root@icinga-gitlab:~# docker-machine ls
NAME                                               ACTIVE   DRIVER      STATE     URL                      SWARM   DOCKER     ERRORS
runner-privileged-icinga-1571900582-bed0b282       -        openstack   Running   tcp://10.10.27.10:2376           v19.03.4
runner-privileged-icinga-1571903235-379e0601       -        openstack   Running   tcp://10.10.27.11:2376           v19.03.4
runner-non-privileged-icinga-1571904408-5bb761b5   -        openstack   Running   tcp://10.10.27.20:2376           v19.03.4
runner-non-privileged-icinga-1571904408-52b9bcc4   -        openstack   Running   tcp://10.10.27.21:2376           v19.03.4
runner-non-privileged-icinga-1571904408-97bf8992   -        openstack   Running   tcp://10.10.27.22:2376           v19.03.4
runner-non-privileged-icinga-1571904408-97bf8992   -        openstack   Running   tcp://10.10.27.22:2376           v19.03.4

Once it detects a peak in the pending job pipeline, the runner is allowed to start additional VMs in OpenStack.

root@icinga-gitlab:~# docker-machine ls
NAME                                               ACTIVE   DRIVER      STATE     URL                      SWARM   DOCKER     ERRORS
runner-privileged-icinga-1571900582-bed0b282       -        openstack   Running   tcp://10.10.27.10:2376           v19.03.4
runner-privileged-icinga-1571903235-379e0601       -        openstack   Running   tcp://10.10.27.11:2376           v19.03.4
runner-non-privileged-icinga-1571904408-5bb761b5   -        openstack   Running   tcp://10.10.27.20:2376           v19.03.4
runner-non-privileged-icinga-1571904408-52b9bcc4   -        openstack   Running   tcp://10.10.27.21:2376           v19.03.4
runner-non-privileged-icinga-1571904408-97bf8992   -        openstack   Running   tcp://10.10.27.22:2376           v19.03.4
runner-non-privileged-icinga-1571904408-97bf8992   -        openstack   Running   tcp://10.10.27.23:2376           v19.03.4

...

runner-non-privileged-icinga-1571904534-0661c396   -        openstack   Running   tcp://10.10.27.24:2376           v19.03.4
runner-non-privileged-icinga-1571904543-6e9622fd   -        openstack   Running   tcp://10.10.27.25:2376           v19.03.4
runner-non-privileged-icinga-1571904549-c456e119   -        openstack   Running   tcp://10.10.27.27:2376           v19.03.4
runner-non-privileged-icinga-1571904750-8f6b08c8   -        openstack   Running   tcp://10.10.27.29:2376           v19.03.4

 

In order to achieve this setting, modify the runner configuration and increase the limit.

vim /etc/gitlab-runner/config.toml

[[runners]]
  name = "docker-machine on OpenStack"
  limit = 24
  output_limit = 20480
  url = "https://git.icinga.com/"
  token = "supersecrettoken"
  executor = "docker+machine"

This would result in 24 OpenStack VMs after a while, and all are idle 24/7. In order to automatically decrease the deployed VMs, use the OffPeak settings. This also ensures that resources are available during workhours while spare time and weekend are considered “off peak” with shutting down unneeded resources automatically.

    OffPeakTimezone = "Europe/Berlin"
    OffPeakIdleCount = 2
    OffPeakIdleTime = 1800
    OffPeakPeriods = [
      "* * 0-8,22-23 * * mon-fri *",
      "* * * * * sat,sun *"
    ]

Pretty neat functionality 🙂

 

Troubleshooting & Monitoring

“docker-machine ls” provides the full overview and tells whenever e.g. a connection to OpenStack did not work, or if the VM is currently unavailable.

root@icinga-gitlab:~# docker-machine ls
NAME                                               ACTIVE   DRIVER      STATE     URL                      SWARM   DOCKER     ERRORS
runner-privileged-icinga-1571900582-bed0b282       -        openstack   Error                                      Unknown    Expected HTTP response code [200 203] when accessing [GET https://cloud.netways.de:8774/v2.1/servers/], but got 404 instead

In case you have deleted the running VMs to start fresh, provisioning might take a while and the above can be a false positive. Check the OpenStack management interface to see whether the VMs booted correctly. You can also remove a VM with “docker-machine rm <id>” and run “gitlab-runner restart” to automatically provision it again.

Whenever the VM provisioning fails, a gentle look into the syslog (or runner log) unveils what’s the problem. Lately we had used a wrong OpenStack flavor configuration which was fixed after investigating in the logs.

Oct 18 07:08:48 3 icinga-gitlab gitlab-runner[30988]:  #033[31;1mERROR: Error creating machine: Error in driver during machine creation: Unable to find flavor named 1234-customer-id-4-8#033[0;m  #033[31;1mdriver#033[0;m=openstack #033[31;1mname#033[0;m=runner-non-privilegued-icinga-1571375325-3f8176c3 #033[31;1moperation#033[0;m=create

Monitoring your GitLab CI runners is key, and with the help of the REST API, this becomes a breeze with Icinga checks. You can inspect the runner state and notify everyone on-call whenever CI pipelines are stuck.

 

Conclusion

Developers depend on fast CI feedback these days, speeding up their workflow – make them move fast again. Admins need to understand their requirements, and everyone needs a deep-dive into GitLab and its possibilities. Join our training sessions for more practical exercises or immediately start playing in NWS!