Seite wählen

NETWAYS Blog

Einfacher als gedacht: Openstack + Vagrant

Vagrant-LogoOpenstack-LogoVermutlich haben sich bei fast allen durch Covid-19 die Prioritäten verschoben, bei mir kam sehr schnell zu dem Thema “Ausbildung trotz 100% Home-Office aufrecht erhalten” noch ein weiteres dazu. Und zwar das Thema Online-Schulungen bzw. genauer die technische Plattform dafür, denn um andere Aspekte wie die Kommunikations- und Präsentationssoftware, Organisation, Marketing und ähnliches haben sich dankenswerterweise andere Kollegen gekümmert.

Die zu nutzende Infrastruktur sollte logischerweise die Openstack-Plattform von NWS sein, da hier entsprechende Kapazitäten vorhanden sind, somit hatte ich mein Ziel. Die Ausgangsbasis war auch leicht definiert, da fast alle Schulungen zum Erstellen der virtuellen Maschinen für die praktischen Übungen auf Vagrant setzen. Die somit erstellten Systeme werden dann zwar noch exportiert um die Provisionierung der Schulungsnotebooks zu beschleunigen, aber sollten trotzdem einen guten Start darstellen.

Als erstes musste natürlich dann der Provider installiert werden, da ich Fedora als Betriebssystem nutze, ging das mit dnf install vagrant-openstack-provider. Für die anderen Linux-Systeme habe ich keine Pakete gefunden, daher wird der Rest wohl auf den Plugin-Mechanismus mit vagrant plugin install vagrant-openstack-provider zurückgreifen müssen. Wichtig ist dann noch zu wissen, ob es der einzige oder Standard-Provider ist, denn sonst darf man später beim Starten der Systeme mit vagrant up nicht den entsprechender Parameter --provider=openstack vergessen.

Natürlich stand als erstes aber eine Schulung auf dem Plan, die normalerweise ohne virtuelle Maschine auskommt, was aber nicht so schlimm war, denn es gab mir mehr Freiheit zum Experimentieren und Kennenlernen der Unterschiede zwischen Openstack-Provider und den von mir gewohnten. Der vielleicht größte Unterschied ist, dass hier nicht mit bestehenden Boxen von einem zentralen System wie app.vagrantup.com gearbeitet wird, sondern mit bereits in Openstack vorhandenen Images. Außerdem muss eine Anmeldung an einem Openstack-Projekt erfolgen, in dem dann nicht nur die Images vorbereitet sein müssen, sondern auch die Netzwerkkonfiguration inklusive der Sicherheitsgruppen mit entsprechender Portfreigabe und ein zu verwendender SSH-Key.
mehr lesen…

Dirk Götz
Dirk Götz
Principal Consultant

Dirk ist Red Hat Spezialist und arbeitet bei NETWAYS im Bereich Consulting für Icinga, Puppet, Ansible, Foreman und andere Systems-Management-Lösungen. Früher war er bei einem Träger der gesetzlichen Rentenversicherung als Senior Administrator beschäftigt und auch für die Ausbildung der Azubis verantwortlich wie nun bei NETWAYS.

NETWAYS Web Services auf der TECHWEEK

Der deutsche Cloud-Markt wächst stetig und die digitale Industrie in Deutschland boomt. Die Veranstaltungsreihe Techweek ist das Event zur digitalen Transformation. Natürlich dürfen wir mit den NETWAYS Web Services bei der Techweek in Frankfurt nicht fehlen! Wir freuen uns darauf, mit den wichtigsten Entscheidungsträgern der deutschen Unternehmenslandschaft ins Gespräch zu kommen und sie von unserer NETWAYS Cloud zu überzeugen!

Ihr seid auch auf der Techweek in Frankfurt? Kommt vorbei, holt euch die neueste NETWAYS Cloud Broschüre und sprecht mit uns! Ihr findet uns an Stand 975.

techweekfrankfurt.de / nws.netways.de/cloud

Julia Hornung
Julia Hornung
Senior Marketing Manager

Julia ist seit Juni 2018 bei NETWAYS. Mit ihren Spezialgebieten Texte/Konzepte, Branding und PR ist sie für Tone of Voice und Wording von NETWAYS und Icinga verantwortlich. Davor war sie als Journalistin und in der freien Theaterszene spannenden Geschichten auf der Spur. Ihre Leidenschaft gilt gutem Storytelling, klarer Sprache und ausgefeilten Texten. Ihre innere Mitte findet sie beim Klettern und Yoga.

GitLab CI Runners with Auto-scaling on OpenStack

 

With migrating our CI/CD pipelines from Jenkins to GitLab CI in the past months, we’ve also looked into possible performance enhancements for binary package builds. GitLab and its CI functionality is really really great in this regard, and many things hide under the hood. Did you know that “Auto DevOps” is just an example template for your CI/CD pipeline running in the cloud or your own Kubernetes cluster? But there’s more, the GitLab CI runners can run jobs in different environments with using different hypervisors and the power of docker-machine.

One of them is OpenStack available at NWS and ready to use. The following examples are from the Icinga production environment and help us on a daily basis to build, test and release Icinga products.

 

Preparations

Install the GitLab Runner on the GitLab instance or in a dedicated VM. Follow along in the docs where this is explained in detail. Install the docker-machine binary and inspect its option for creating a new machine.

curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
apt-get install -y gitlab-runner
  
curl -L https://github.com/docker/machine/releases/download/v0.16.2/docker-machine-`uname -s`-`uname -m` -o /usr/local/bin/docker-machine
chmod +x /usr/local/bin/docker-machine
  
docker-machine create --driver openstack --help

Next, register the GitLab CI initially. Note: This is just to ensure that the runner is up and running in the GitLab admin interface. You’ll need to modify the configuration in a bit.

gitlab-runner register \
  --non-interactive \
  --url https://git.icinga.com/ \
  --tag-list docker \
  --registration-token SUPERSECRETKEKSI \
  --name "docker-machine on OpenStack" \
  --executor docker+machine \
  --docker-image alpine

 

Docker Machine with OpenStack Deployment

Edit “/etc/gitlab-runner/config.toml” and add/modify the “[[runners]]” section entry for OpenStack and Docker Machine. Ensure that the MachineDriver, MachineName and MachineOptions match the requirements. Within “MachineOptions”, add the credentials, flavors, network settings just as with other deployment providers. All available options are explained in the documentation.

vim /etc/gitlab-runner/config.toml

  [runners.machine]
    IdleCount = 4
    IdleTime = 3600
    MaxBuilds = 100
    MachineDriver = "openstack"
    MachineName = "customer-%s"
    MachineOptions = [
      "openstack-auth-url=https://cloud.netways.de:5000/v3/",
      "openstack-tenant-name=1234-openstack-customer",
      "openstack-username=customer-login",
      "openstack-password=sup3rS3cr3t4ndsup3rl0ng",
      "openstack-flavor-name=s1.large",
      "openstack-image-name=Debian 10.1",
      "openstack-domain-name=default",
      "openstack-net-name=customer-network",
      "openstack-sec-groups="mine",
      "openstack-ssh-user=debian",
      "openstack-user-data-file=/etc/gitlab-runner/user-data",
      "openstack-private-key-file=/etc/gitlab-runner/id_rsa",
      "openstack-keypair-name=GitLab Runner"
    ]

The runners cache can be put onto S3 granted that you have this service available. NWS luckily provides S3 compatible object storage.

  [runners.cache]
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "s3provider.domain.localdomain"
      AccessKey = "supersecretaccesskey"
      SecretKey = "supersecretsecretkey"
      BucketName = "openstack-gitlab-runner"

Bootstrap Docker in the OpenStack VM

Last but not least, these VMs need to be bootstrapped with Docker inside a small script. Check the “–engine-install-url” parameter in the help output:

root@icinga-gitlab:/etc/gitlab-runner# docker-machine create --help
  ...
  --engine-install-url "https://get.docker.com"							Custom URL to use for engine installation 

You can use the official way of doing this, but putting this into a small script also allows customizations like QEMU used for Raspbian builds. Ensure that the script is available via HTTP e.g. from a dedicated GitLab repository 😉

#!/bin/sh
#
# This script helps us to prepare a Docker host for the build system
#
# It is used with Docker Machine to install Docker, plus addons
#
# See --engine-install-url at docker-machine create --help

set -e

run() {
  (set -x; "$@")
}

echo "Installing Docker via get.docker.com"
run curl -LsS https://get.docker.com -o /tmp/get-docker.sh
run sh /tmp/get-docker.sh

echo "Installing QEMU and helpers"
run sudo apt-get update
run sudo apt-get install -y qemu-user-static binfmt-support

Once everything is up and running, the GitLab runners are ready to fire the jobs.

 

Auto-Scaling

Jobs and builds are not run all the time, and especially with cloud resources, this should be a cost-efficient thing. When building Icinga 2 for example, the 20+ different distribution jobs generate a usage peak. With the same resources assigned all the time, this would tremendously slow down the build and release times. In that case, it is desirable to automatically spin up more VMs with Docker and let the GitLab runner take care of distributing the jobs. On the other hand, auto-scaling should also shut down resources in idle times.

By default, one has 4 VMs assigned to the GitLab runner. These builds run non-privileged in Docker, the example below also shows another runner which can run privileged builds. This is needed for Docker-in-Docker to create Docker images and push them to GitLab’s container registry.

root@icinga-gitlab:~# docker-machine ls
NAME                                               ACTIVE   DRIVER      STATE     URL                      SWARM   DOCKER     ERRORS
runner-privileged-icinga-1571900582-bed0b282       -        openstack   Running   tcp://10.10.27.10:2376           v19.03.4
runner-privileged-icinga-1571903235-379e0601       -        openstack   Running   tcp://10.10.27.11:2376           v19.03.4
runner-non-privileged-icinga-1571904408-5bb761b5   -        openstack   Running   tcp://10.10.27.20:2376           v19.03.4
runner-non-privileged-icinga-1571904408-52b9bcc4   -        openstack   Running   tcp://10.10.27.21:2376           v19.03.4
runner-non-privileged-icinga-1571904408-97bf8992   -        openstack   Running   tcp://10.10.27.22:2376           v19.03.4
runner-non-privileged-icinga-1571904408-97bf8992   -        openstack   Running   tcp://10.10.27.22:2376           v19.03.4

Once it detects a peak in the pending job pipeline, the runner is allowed to start additional VMs in OpenStack.

root@icinga-gitlab:~# docker-machine ls
NAME                                               ACTIVE   DRIVER      STATE     URL                      SWARM   DOCKER     ERRORS
runner-privileged-icinga-1571900582-bed0b282       -        openstack   Running   tcp://10.10.27.10:2376           v19.03.4
runner-privileged-icinga-1571903235-379e0601       -        openstack   Running   tcp://10.10.27.11:2376           v19.03.4
runner-non-privileged-icinga-1571904408-5bb761b5   -        openstack   Running   tcp://10.10.27.20:2376           v19.03.4
runner-non-privileged-icinga-1571904408-52b9bcc4   -        openstack   Running   tcp://10.10.27.21:2376           v19.03.4
runner-non-privileged-icinga-1571904408-97bf8992   -        openstack   Running   tcp://10.10.27.22:2376           v19.03.4
runner-non-privileged-icinga-1571904408-97bf8992   -        openstack   Running   tcp://10.10.27.23:2376           v19.03.4

...

runner-non-privileged-icinga-1571904534-0661c396   -        openstack   Running   tcp://10.10.27.24:2376           v19.03.4
runner-non-privileged-icinga-1571904543-6e9622fd   -        openstack   Running   tcp://10.10.27.25:2376           v19.03.4
runner-non-privileged-icinga-1571904549-c456e119   -        openstack   Running   tcp://10.10.27.27:2376           v19.03.4
runner-non-privileged-icinga-1571904750-8f6b08c8   -        openstack   Running   tcp://10.10.27.29:2376           v19.03.4

 

In order to achieve this setting, modify the runner configuration and increase the limit.

vim /etc/gitlab-runner/config.toml

[[runners]]
  name = "docker-machine on OpenStack"
  limit = 24
  output_limit = 20480
  url = "https://git.icinga.com/"
  token = "supersecrettoken"
  executor = "docker+machine"

This would result in 24 OpenStack VMs after a while, and all are idle 24/7. In order to automatically decrease the deployed VMs, use the OffPeak settings. This also ensures that resources are available during workhours while spare time and weekend are considered “off peak” with shutting down unneeded resources automatically.

    OffPeakTimezone = "Europe/Berlin"
    OffPeakIdleCount = 2
    OffPeakIdleTime = 1800
    OffPeakPeriods = [
      "* * 0-8,22-23 * * mon-fri *",
      "* * * * * sat,sun *"
    ]

Pretty neat functionality 🙂

 

Troubleshooting & Monitoring

“docker-machine ls” provides the full overview and tells whenever e.g. a connection to OpenStack did not work, or if the VM is currently unavailable.

root@icinga-gitlab:~# docker-machine ls
NAME                                               ACTIVE   DRIVER      STATE     URL                      SWARM   DOCKER     ERRORS
runner-privileged-icinga-1571900582-bed0b282       -        openstack   Error                                      Unknown    Expected HTTP response code [200 203] when accessing [GET https://cloud.netways.de:8774/v2.1/servers/], but got 404 instead

In case you have deleted the running VMs to start fresh, provisioning might take a while and the above can be a false positive. Check the OpenStack management interface to see whether the VMs booted correctly. You can also remove a VM with “docker-machine rm <id>” and run “gitlab-runner restart” to automatically provision it again.

Whenever the VM provisioning fails, a gentle look into the syslog (or runner log) unveils what’s the problem. Lately we had used a wrong OpenStack flavor configuration which was fixed after investigating in the logs.

Oct 18 07:08:48 3 icinga-gitlab gitlab-runner[30988]:  #033[31;1mERROR: Error creating machine: Error in driver during machine creation: Unable to find flavor named 1234-customer-id-4-8#033[0;m  #033[31;1mdriver#033[0;m=openstack #033[31;1mname#033[0;m=runner-non-privilegued-icinga-1571375325-3f8176c3 #033[31;1moperation#033[0;m=create

Monitoring your GitLab CI runners is key, and with the help of the REST API, this becomes a breeze with Icinga checks. You can inspect the runner state and notify everyone on-call whenever CI pipelines are stuck.

 

Conclusion

Developers depend on fast CI feedback these days, speeding up their workflow – make them move fast again. Admins need to understand their requirements, and everyone needs a deep-dive into GitLab and its possibilities. Join our training sessions for more practical exercises or immediately start playing in NWS!

Trefft unser NETWAYS Web Services Team auf der DOST!

Die DOST hat sich nachhaltig als anerkannte Fachkonferenz rund um den Betrieb modularer Cloud-Plattformen etabliert. Da dürfen unsere Kollegen von NETWAYS Web Services natürlich nicht fehlen! Wir werden als Konferenzteilnehmer vor Ort vertreten sein, aber auch als Sponsor mit einem Stand in der Sponsorenausstellung. Egal ob nach oder vor einem Vortrag oder an unserem Stand – sprecht uns an!

Wir freuen uns auf interessante Gespräche zu Cloud und Co.!

Erzählt uns von euren Systemen und Lösungen und erfahrt von uns alles, was es zur NETWAYS Cloud auf OpenStack-Basis zu wissen gibt!

Das Programm, Tickets und alles weitere zur DOST: https://openstack-tage.de/

Alles zur NETWAYS Cloud: https://nws.netways.de/de/cloud/

Julia Hornung
Julia Hornung
Senior Marketing Manager

Julia ist seit Juni 2018 bei NETWAYS. Mit ihren Spezialgebieten Texte/Konzepte, Branding und PR ist sie für Tone of Voice und Wording von NETWAYS und Icinga verantwortlich. Davor war sie als Journalistin und in der freien Theaterszene spannenden Geschichten auf der Spur. Ihre Leidenschaft gilt gutem Storytelling, klarer Sprache und ausgefeilten Texten. Ihre innere Mitte findet sie beim Klettern und Yoga.

NWS – because web services don’t have to be complicated

Web services platforms are a common way nowadays, to run your services without the need of your own infrastructure. No matter if you need a safe place for your data, an app to communicate with people or just a place to run your own services. But what do you know about the platform of your choice?

  • Do you know how your data is stored?
  • Do you know where the servers are located in detail?
  • Do you know the people which are giving you support?
  • Do you know what happens in case of a critical issue?

I bet, most of you, or at least some of you, don’t. But that’s the point where we start. We want to be different. We want to be transparent and uncomplicated.

Our servers are located in Nuremberg – Germany. We are running them in two different datacenters to ensure high security.

Working with our apps is pretty simple. Just start them! There is no complicated registration process, just fill in the form and you are good to go. The best thing is: you don’t even need a domain or some know how of SSL. It’s already configured for you! Of course you can use your own domain and certificates if you want, but you don’t have to.
If you are curious about how our apps are working in detail or what features are available in our apps, just have a look at our FAQs
Still can’t find what you are looking for? Contact us via live chat! If there is no live chat agent available, an automated ticket will be created, so we can get in touch with you.

But whats so special about our support? Well, it’s quick, it’s professional, it’s individual and most important of all: Our developers and administrators are your support team.

We also want to create our bills as transparent as possible. In case of our OpenStack cloud, you won’t just get a bill with an amount. You will get a detailed report of all resources you used in the past month. Which IP was used how long? How many CPUs were used how long and which VM did use them? But thats still not all. We want to give you the possibility to check at a various day of the month, how much money you spent on our cloud so far and how much you will spent till the end of the month.

Keep track of whats going on! And even if you have your applications running on a different platform – we will help you migrating your data with almost no downtime. Because we take care.

Most features we released for our apps and the platform itself, were not features “we wanted” to release. Don’t get me wrong here, we also wanted to release them, but most of them were features our customers requested and needed.

And we got one more thing. If you don’t know if you can trust us, you can just give it a try. Ask us anything or just try our services – 30 days for free (OpenStack is excluded here – but just get in touch with us. We will find a solution that fit your needs.).

Marius Gebert
Marius Gebert
Systems Engineer

Marius ist seit 2013 bei NETWAYS. Er hat 2016 seine Ausbildung zum Fachinformatiker für Systemintegration absolviert und ist nun im Web Services Team tätig. Hier kümmert er sich mit seinen Kollegen um die NWS Plattform und alles was hiermit zusammen hängt. 2017 hat Marius die Prüfung zum Ausbilder abgelegt und kümmert sich in seiner Abteilung um die Ausbildung unserer jungen Kollegen. Seine Freizeit verbringt Marius gerne an der frischen Luft und ist für jeden Spaß zu...

Veranstaltungen

Dez 01

Icinga 2 Fundamentals Training | Online

Dezember 1 @ 09:00 - Dezember 4 @ 17:00
Dez 03

DevOps Meetup

Dezember 3 @ 17:30 - 20:30
Dez 08

Terraform mit OpenStack Training | Online

Dezember 8 @ 09:00 - Dezember 9 @ 17:00
Dez 08

Icinga 2 Advanced Training | Online

Dezember 8 @ 09:00 - Dezember 10 @ 17:00
Dez 15

GitLab Training | Online

Dezember 15 @ 09:00 - Dezember 16 @ 17:00