Managing your Ansible Environment with Galaxy

Ansible is known for its simplicity, lightweight footprint and flexibility to configure nearly any device in your infrastructure. Therefore it’s used in large scale environments shared between teams or departments. This leads to even bigger Ansible environments which need to be tracked or managed in version control systems like Git.

Mostly environments grow with their usage over time, in this case it can happen that all roles are managed inside one big repository. Which will eventually lead to quite messy configuration and loss of knowledge if roles are tested or work the way they supposed to work.

Ansible provides a solution which is called Galaxy, it’s basically a command line tool which keeps your environment structured, lightweight and enforces your roles to be available in a specific version.

First of all you can use the tool to download and manage roles from the Ansible Galaxy which hosts many roles written by open-source enthusiasts. 🙂


# ansible-galaxy install geerlingguy.ntp -v
Using /Users/twening/ansible.cfg as config file
 - downloading role 'ntp', owned by geerlingguy
 - downloading role from https://github.com/geerlingguy/ansible-role-ntp/archive/1.6.4.tar.gz
 - extracting geerlingguy.ntp to /Users/twening/.ansible/roles/geerlingguy.ntp
 - geerlingguy.ntp (1.6.4) was installed successfully

# ansible-galaxy list
# /Users/twening/.ansible/roles
 - geerlingguy.apache, 3.1.0
 - geerlingguy.ntp, 1.6.4
 - geerlingguy.mysql, 2.9.5

Furthermore it can handle roles from your own Git based repository. Tags, branches and commit hashes can be used to ensure it’s installed in the right version.


ansible-galaxy install git+https://github.com/Icinga/ansible-icinga2.git,v0.2.0
 - extracting ansible-icinga2 to /Users/twening/.ansible/roles/ansible-icinga2
 - ansible-icinga2 (v0.2.0) was installed successfully

It’s pretty neat but how does this help us in large environments with hundreds of roles?

The galaxy command can read requirement files, which are passed with the “-r” flag. This requirements.yml file can be a replacement for roles in the roles path and includes all managed roles of the environment.


# vim requirements.yml
- src: https://github.com/Icinga/ansible-icinga2.git
  version: v0.2.0
  name: icinga2

- src: geerlingguy.mysql
  version: 2.9.5
  name: mysql

Then run ansible-galaxy with the “–role-file” parameter and let galaxy manage all your roles.


# ansible-galaxy install -r requirements.yml
 - icinga2 (v0.2.0) is already installed, skipping.
 - downloading role 'mysql', owned by geerlingguy
 - downloading role from https://github.com/geerlingguy/ansible-role-mysql/archive/2.9.5.tar.gz
 - extracting mysql to /Users/twening/.ansible/roles/mysql
 - mysql (2.9.5) was installed successfully

In case you work with Ansible AWX, you can easily replace all your roles with this file in the roles directory and AWX will download and manage your roles directory automatically.

A example project could look like this.


awx_project/
├── example_playbook.yml
├── group_vars
├── host_vars
├── hosts
└── roles
    └── requirements.yml

In summary, in large environments try to keep your code and configuration data separated, try to maintain your roles in their own repository to avoid conflicts at the main project.

Check out our Blog for more awesome posts and if you need help with Ansible send us a message or sign up for one of our trainings!

Thilo Wening
Thilo Wening
Consultant

Thilo hat bei NETWAYS mit der Ausbildung zum Fachinformatiker, Schwerpunkt Systemadministration begonnen und unterstützt nun nach erfolgreich bestandener Prüfung tatkräftig die Kollegen im Consulting. In seiner Freizeit ist er athletisch in der Senkrechten unterwegs und stählt seine Muskeln beim Bouldern. Als richtiger Profi macht er das natürlich am liebsten in der Natur und geht nur noch in Ausnahmefällen in die Kletterhalle.

Monthly Snap August 2019

 

August is the typical month for holidays at NETWAYS. The schools are finally out for summer, and you would think everything would slow down a bit. But look at all the blogposts our colleagues wrote! No downtime at the NETWAYS HQ!

What was our Development Team up to?

Johannes started the month with Icinga Web 2 – More Goodies for Developers. He presented some interesting new functions for fellow developers.. Are you also annoyed with the Facebook newsfeed? Marius found a solution, which might actually be an alternative for some of us, in Facebook: endlich Ruhe. Alexander claims that Ansible is not just a hype, but extremely useful in Ansible + Icinga 2 = #monitoringlove.

Cakes and tech.

Foreman’s 10th birthday – The party was a blast. Dirk celebrated the Foreman birthday party with the community.
Lennart had a lot to share in Graphite-API für Grafana und Icinga Web 2, a detailed and thorough how-to. rsync und was dann? Read Markus` road to a database dump with a strict time limit. Max talked about Monitoring your Monitoring in Monitor das Monitoring_by_ssh. Check out how easily it is done with Icinga!

Grab your ticket!

The call for papers for this year`s OSMC is closed, but tickets are still available if you would like to attend! Julia shared a video from last year`s conference in the series OSMC | Take a glance back… Are you already looking forward to the OSDC 2020?  Save the date & win tickets! 

Nicoles` corner

Nicole gave us a little course on sensors in Kleine Sensorkunde. What kinds of sensors are there, and how do you decide which one is right for your needs? Read the blog, and if you are still unsure just contact our shop for consulting. Then she presented News from HW Group in HW group: Neues vom SensDesk Portal.

Project week for the juniors

The annual project week for our trainees took place in the beginning of august. Alexander told us the tale of 9 people from different departments getting together, agreeing upon a project, and actually creating it, in teamwork in just one week. Read about it in Azubiprojektwoche 2019. We have a new post in our blogseries NETWAYS stellt sich vor! Alexander reflected upon his first months at NETWAYS, and why he started here in the first place.

To migrate or not to migrate?

Are you considering a GitLab upgrade to EE? Read Gabriel`s tips in Migration von GitLab mit Upgrade auf EE. Follow his 9 steps and you might avoid unnecessary obstacles.

Hosted in…Nuremberg!

Martin spoke of the advantages of myEngineer by NWS in NETWAYS Cloud: Dein MyEngineer unterstützt Dich. Jederzeit.  Finally, there is a system where you get exactly what you want without any hidden Costs.

Catharina Celikel
Catharina Celikel
Office Manager

Catharina unterstützt seit März 2016 unsere Abteilung Finance & Administration. Die gebürtige Norwegerin ist Fremdsprachenkorrespondentin für Englisch. Als Office Manager kümmert sie sich deshalb nicht nur um das Tagesgeschäft sondern übernimmt nebenbei zusätzlich einen Großteil der Übersetzungen. Privat ist der bekennende Bücherwurm am liebsten mit dem Fahrrad unterwegs.

Foreman’s 10th birthday – The party was a blast

Birthday Logo

I can still remember when Greg had the idea of celebrating the Foreman’s Birthday four years ago and I volunteered to organize the German one. After two editions and with Foreman being covered on the Open Source Camp last year I asked for others to run the party. And with ATIX doing a great job I asked them to team up on this. So we have grown a great community event with the annual Birthday party.

This year was different to the ones before because we had such a big support by Red Hat. The new Community Managers showed up to introduce them accompanied by Greg who had stepped down earlier this year. A group of Product managers and consultants made the last stop on their European tour. A technical writer came over to discuss the future of documentation. And with Evgeni and Ewoud we had some recurring attendees to give a talk later. ATIX also arrived with a bus full of people. Monika represented iRonin, a company doing custom development on Foreman and I hope to team up in the future, and Timo developing on Foreman for dmTech brought a colleague. So users were slightly under-represented and the prepared demos were mostly used to share knowledge and probably because of the heat instead of hacking many discussions took place. But I think everyone of the about thirty attendees made good use of the first session.

Birthday PartyDemoThe session ended when I brought in the cake. And thanks to our Events team the cake was as tasty as good looking. A nice touch by Ohad was to insist he can not blow off the candles alone as he could not have build Foreman without the community.

Birthday CakeHelmets

After the cake break we started with the talks and the first one was by the Community team giving us a recap of Foreman’s history, data from the community survey and other insights like a first look on the future documentation. This is really the next step to me that Red Hat is also making their Satellite documentation upstream adding a use case driven documentation to the manual which is way more technical. The second talk Quirin showcased the current state of Debian Support which will be fully functional with Errata support being added, but he already promised some usability and documentation improvements afterwards. The third speakers were Dana and Rich who showed Red Hat’s roadmap for features to add to Foreman so they will be pulled into Satellite afterwards. The roadmap will be presented in a community demo and uploaded to the community forum. Having the product managers easily available allowed the audience also to ask any question and I was excited to hear for almost all topics brought up that there is already ongoing work in the background. For example I asked about making subscription management also usable for other vendors and Rich told me he is part of a newly founded team which is evaluating exactly this.

Because of the heat we added a small ice break before starting the next talk and because of Lennart being ill Ohad entered the stage to show his work on containerizing Foreman. He explained that he started it mainly for testing but the interest showed him that expanding it to be fully functional to run Foreman and even Katello on Kubernetes could be a future way. Evgeni gave a shortened version of the talk on writing Ansible modules for Foreman and Katello he created for Froscon. It was a very technical one showing how much work is necessary to build a good base so later work is much easier. From this perspective I can really recommend this talk to all Froscon attendees. Last but not least Ewoud looked into the project’s social aspects which was a nice mixture of official history and personal moments. He also showed off the different swag the project created, ending with a t-shirt signed by as many team and community members as possible while traveling from Czech to US and back as suitable gift to Greg because “Once a foreman, always a foreman”. 😉

For dinner we had Pizza and Beer, but moved to the air-conditioned hotel bar after a short while to finish the evening. I heard people were enjoying conversation until two o’clock in the morning even when the bar closed one hour earlier. 😀

I would say the Party was a blast and I am already looking forward to next year when ATIX will be the host again. But until then there are several other Foreman related events with the Open Source Automation Day on 15. & 16.10.2019 in Munich including Workshops the day before and a Foreman hackday the day after organized by ATIX and the Open Source Camp on 07.11.2019 in Nuremberg right after OSMC by NETWAYS.

Dirk Götz
Dirk Götz
Principal Consultant

Dirk ist Red Hat Spezialist und arbeitet bei NETWAYS im Bereich Consulting für Icinga, Puppet, Ansible, Foreman und andere Systems-Management-Lösungen. Früher war er bei einem Träger der gesetzlichen Rentenversicherung als Senior Administrator beschäftigt und auch für die Ausbildung der Azubis verantwortlich wie nun bei NETWAYS.

Icinga Web 2 – More Goodies for Developers

Tuesday version 2.7 of Icinga Web 2 has been released and introduced some interesting new functionality for module developers. Now I’d like to tell you, my fellow colleague, some more details about this.

 

jQuery v3 – Migration Required

First a friendly warning. We’ve upgraded the jQuery version we ship to v3.4.1. This has previously been v2.1.0 so now with this major upgrade some deprecated methods and interfaces are gone.

Though, don’t worry, you don’t need to hurry to avoid everyone complaining your module is incompatible with v2.7. We also ship jQuery migrate now which ensures that the usage of removed methods/interfaces still works. It also emits console warnings if it detects such a usage. The warnings are not active by default. They only appear when using the non-minimized javascript code. Put _dev in your address bar to instruct Icinga Web 2 to serve the non-minimized version. (e.g. /icingaweb2/dashboard?_dev)

Then start using the front-end as usual. Interact with all widgets you’ve written your own Javascript for and look for console entries starting with JQMIGRATE. Any of these messages will only appear once, repeated usages are not reported. If you’ve got a warning then, consult jQuery migrate’s warnings.md in order to get hints how to solve it.

jQuery migrate will be removed with Icinga Web 2 version 2.8.0. While this is still some time ahead, this (and the note in the upgrade documentation) is probably the only warning.

 

Persistence and Collapsible Containers

While we’re at it, let’s stay with the topic of Javascript. If you don’t already know about the localStorage and sessionStorage, it’s now time to inform yourself. (That’s an entire blogpost if described thoroughly)

There’s now an abstraction layer for this shipped with Icinga Web 2. It hides all the handling of complex datatypes and conflicts with other apps using the storage from you. That’s the object Icinga.Storage which utilizes the localStorage by default but also supports the sessionStorage. Take a look here to see how this is used for Icinga Web 2’s sidebar.

Though, this is only the basic stuff. If you need to store more complex data and want to benefit from a storage’s event processing, take a look at the object Icinga.Storage.StorageAwareMap. This is a proxy for Map and allows to subscribe to change events of particular keys in the map. It also keeps track of a key’s age and removes it automatically if it hasn’t been accessed for 90 days.

Another new addition are collapsible HTML containers. This is provided by a behavior which makes use of the StorageAwareMap, a perfect example use-case.

Making a container collapsible is as easy as possible. Just apply the CSS class collapsible and you’re done. If you’re not satisfied with the default height, apply the data attribute data-visible-height and give it the desired height in pixels. (For table‘s and ul‘s or ol‘s there is also data-visible-rows.) Then, if you fancy a custom control by which users expand or collapse the container you can pass a CSS selector to data-toggle-element which (if a direct descendant of the container) then acts as the toggle.

 

Custom XHR Without Dirty Hacks

Have you ever wanted/tried to process link clicks or form submissions by yourself? Well, I have and it was a nightmare every single time. Most of Icinga Web 2’s processing is fine. But of course there ever is this single behavior or side-effect which keeps getting in the way. This has now come to an end.

Meet data attribute data-no-icinga-ajax which does exactly what it’s name suggests. Applied to an element it causes Icinga Web 2 to ignore click and submit events triggered by the element itself and all descendants.

Couldn’t be more simple, can it?

 

Hooks For Everyone

Previously hooks were only processed for logged-in users with the permission to access the module providing the hook. This for example prevented the audit module to register logins from users without the permission module/audit and also didn’t allow to log failed logins.

When providing a hook it is now possible to have it run always. ($this->provideHook($name, $implementation = null, $alwaysRun = false);)

Another case of hooks not being processed was the issue that, unlike in the web front-end, enabled modules were not loaded automatically on the CLI. Thus also their hooks were not registered. Now this has changed and also on the CLI all enabled modules are automatically loaded. If you’ve previously loaded the modules explicitly this is not required anymore. If you don’t need any other modules and want to avoid the overhead of loading them, you can disable this of course.

If you still don’t have enough of this, there’s also an entirely new hook available: ConfigFormEventsHook This hook enables you to influence every configuration form of Icinga Web 2. Extending a form’s validation or doing additional work once submission succeeded is now on the table.

 

That’s it. I hope these things are as useful to you as they were to us. And remember, we don’t mind any suggestions to further improve the integration of modules. You are the developer, you’ll know best what’s… best.

Johannes Meyer
Johannes Meyer
Developer

Johannes ist seit 2011 bei uns und hilft bei der Entwicklung zukünftiger Knüller (Icinga2, Icinga Web 2, ...) aus dem Hause NETWAYS.

Cleanup your Docker Environment

Using Docker is pretty common meanwhile and a very good idea for development. Using many versions of your favourite language without messing up your host system, different types of deployments (e.g. web servers) or just testing production environment without operational support. The only drawback is that normally you don’t have a clue what’s going on behind the scenes. If you run out of disk space for the first time, you’re exactly at that point.

To apply first aid, you will be advised to use some curious cli hacks to clean up your system. Breaking fingers between grep, sed and awk works out well but is not very helpful – Especially if you want to remember what you did 3 months before 😉

Since Docker Api version 1.25 you have a couple of high level cli commands available doing exactly this job:

Minimal Cheat Sheet:

$ # Claimed disk space
$ docker system df
TYPE            TOTAL       ACTIVE      SIZE          RECLAIMABLE
Images          59          5           10.74GB       9.038GB (84%)
Containers      6           1           991kB         991kB (99%)
Local Volumes   216         1           5.876GB       5.876GB (100%)
Build Cache     0           0           0B            0B

$ # Cleanup disk space
$ docker system prune
WARNING! This will remove:
        - all stopped containers
        - all networks not used by at least one container
        - all dangling images
        - all dangling build cache
Are you sure you want to continue? [y/N] Y
Deleted Containers:
02401e1555e8e752d36198d982b5e4114d0999c7cca34a2353e8dc332faa4db5
997eac76d4a46515797027103967c61b46219ff8c70f6e0bb39bc2b975297fa5
23983ed8abaa60198b497e4b3788bb6de7d39d03f171f43e4ee865c0df318ab8
65bb90b9e7edcd2d13da3129664f8b74a72b011d56136cb28c687f1f8dd8e473
5218788bff77cc0c0cc03f79888ea61c3e27bf3ef0003e41fc231b8b6ecdcdc2

Deleted Images:
deleted: sha256:dccdc3cf7d581b80665bad309b66ba36d88219829e1ade951912dc122b657bfc
[...]

There is also an equivalent for images only:

$ docker image prune

You should definitely take a deeper look into the CLI commands. There are a lot of things that helps you to solve your every day problems!

Marius Hein
Marius Hein
Head of Development

Marius Hein ist schon seit 2003 bei NETWAYS. Er hat hier seine Ausbildung zum Fachinformatiker absolviert, dann als Application Developer gearbeitet und ist nun Leiter der Softwareentwicklung. Ausserdem ist er Mitglied im Icinga Team und verantwortet dort das Icinga Web.

DEV Retreat 2019 Recap

In this Episode: Everybody gets blown to smithereens (digitally).

Those team events are often paired with team-building exercises, causing possibly uncomfortable situations with your co-workers or let you find out things you never wanted to know about them. That’s at least what every comedy involving such an event leads us to believe. So for the breaks between remembering what we did well and not so well since our last DEV Retreat we had a game of “Keep Talking and Nobody Explodes“: The only game released since 2000 to require a manual, it’s also great fun.

The method for retrospection did not change much from last time but we had better sticky notes this time, which helped! We started off with a general assessment of our well being and mood and I am happy to report nobody was too grumpy. This was followed by thinking back what had happened, which projects did we do, what has changed and our unfinished business. Surprisingly a lot of us remembered not only big releases of Icinga 2, Icinga Web 2 or other larger projects but also small things, like our Trainees first merge, taking a break from PHP in favor of JS or making a brochure to advertise apprenticeship at NEWTAYS.

After making concrete plans on how to improve with out team dinner was served. The Hotel had a habit of making the portions a bit smaller than some would have liked but after the dessert nobody was hungry anymore and ready for a mellow evening with Gin and Tonic.

Of course not all is happy, rainbow and unicorns. But there weren’t any problems we aren’t confident we can solve. Except what to do for our next DEV Retreat. But our colleagues went for a round of arrow tag, maybe we’ll steal that idea for next time.