Monthly Snap November


 
Remember one month ago? The first talks were about to start soon, you had your first coffee, enjoyed the rich Holiday Inn breakfast, took your journal, in which you had your plan for the day scetched, you were happy to meet an old friend in the hallway on your way to Saal Jacobi, where Bernd Erk was preparing to kick-off OSMC. …
OSMC – good times! Take some of its spirit with you into the cold and dark december days and let it warm your heart like a cup of hot Christkindlesmarkt mullet wine.

Form OSMC to Open Infrastructure

How? Make yourself a cup of hot wine and read through OSMC 2018 – Day 1 and OSMC 2018 – Day 2 by Dirk, or OSMC 2018 – or: The Thirteen-Star Conference, in which our attendees share their experiences. Read Michi’s Hackathon stories and Dirk’s report about OSCAMP on Puppet. You’re more of a visual type?  Browse through our OSMC Archive and find all the videos, pics and slides.
Right after OSMC our NWS-Team hit the road to Berlin for OpenStack Summit. Check out their managed Open Infrastructure service at nws.netways.de – now with MyEngineer support!

Icinga Seminar & Berlin Camp

Lennart Betz continues his blogpost-seminar Icinga 2 – Monitoring automatisiert mit Puppet Teil 9: Profile and Teil 10: Profile Part II. Join the Icinga community in 2019: Icinga Camp Berlin: Call for Papers is running until Decemeber 31! Find all November Icinga updates here.

For the tech hearts

Achim tell you to Get your RadosGW metrics into Ceilometer. Loei explains the use and functionality of RFID & NFC. Niko shares Neuheiten beim IntelliJ-Update. Tim knows Ceph Mimic | Using loop devices as OSD
In git gud! Henrik tells the story of: How I Learned to Stop Worrying and Love my Code. Noah introduces you with GitKraken to a nearly perfect Git-Client, while Johannes shares another useful tool: PDF manipulieren mit pdftk.

Presents

In November Silke announced Black Friday mal ganz in Orange – Es darf gespart werden und Die neue Generation der SMSEagle NXS-97xx Linie stellt sich vor. It‘s always a good idea to check our Shop for special offers. Especially if you are looking for Christmas presents. In that sense: Entschleunigung vom Weihnachtsstress bei Netways!
Have a wonderful December and Christmas season! Stay tuned!

Julia Hornung
Julia Hornung
Marketing Manager

Julia ist seit Juni 2018 Mitglied der NETWAYS Family. Vor ihrer Zeit in unserem Marketing Team hat sie als Journalistin und in der freien Theaterszene gearbeitet. Ihre Leidenschaft gilt gutem Storytelling, klarer Sprache und ausgefeilten Texten. Privat widmet sie sich dem Klettern und ihrer Ausbildung zur Yogalehrerin.

Get your RadosGW metrics into Ceilometer

With OpenStack it is an ease to provide an object store backed up from your Ceph RadosGW. But getting the usage metrics from Ceph into Ceilometer and Gnocchi is a little bit tricky. At least if you use OpenStack Pike. Following the official Pike documentation we need a RadosGW user with the correct credentials, usage logging enabled and a little bit of Ceilometer configuration:
 

$ radosgw-admin user create --uid ceilometer --display-name "ceilometer" --caps "usage=read;metadata=read;users=read;buckets=read"
# ceilometer.conf
[service_types]
radosgw = object-store
 
[rgw_admin_credentials]
access_key = access
secret_key = secret

 
Additionally we need to add the radosgw.* metrics to your ceilometer polling.yaml. At this point the official documentation lets you down with inconsistent metric names. In Pike, there are no entry points for radosgw.* metrics, only for the old rgw.* metrics. This is described in the bug #1726458. But following the defined entry points into the code, we see that these creates the new radosgw.* metrics. Well, adding the old metric names to your polling.yaml delivers you the new metric names… ¯\_(ツ)_/¯
 

# polling.yaml
- name: radosgw_pollsters
    interval: 1200
    meters:
    -  rgw.containers.objects
    -  rgw.containers.objects.size
    -  rgw.objects
    -  rgw.objects.size
    -  rgw.objects.containers
    -  rgw.usage

To bring it to an end, restarting “ceilometer central” will deliver your RadwosGW metrics to Gnocchi.

Achim Ledermüller
Achim Ledermüller
Lead Senior Systems Engineer

Der Exil Regensburger kam 2012 zu NETWAYS, nachdem er dort sein Wirtschaftsinformatik Studium beendet hatte. In der Managed Services Abteilung ist unter anderem für die Automatisierung des RZ-Betriebs und der Evaluierung und Einführung neuer Technologien zuständig.

Ceph Mimic | Using loop devices as OSD

For quite some time we have been using ceph-deploy to deploy OSD in folders during the Ceph trainings held by Netways. This worked perfectly with jewel, but newer versions don’t allow this behaviour anymore.
There are several reasons for this, however, as we have a quite regulated setup for our training notebooks, we had to come up with some kind of workaround. This approach is, while working fine in our training setup, not recommended for production!
The following steps apply to a current CentOS 7 system.
As stated before, we will deploy an OSD on a block device. Though you could use a separate partition for this, we will use a loop device. For this, the first step is to create a file:
For this, create an OSD directory
 

$ mkdir -p /home/training/my-cluster/osd-$HOSTNAME
$ cd /home/training/my-cluster/osd-$HOSTNAME/

in this folder, create a file for later use

$ fallocate -l 30G 30GB.img

test it

# losetup -l -P /dev/loop1 "/home/training/my-cluster/osd-$HOSTNAME/30GB.img"
# wipefs -a /dev/loop1
# lsblk

This should then display your new loopdevice.
As loop devices are not reboot safe, you need to go some steps further. If you like to use rc.local for this, you’re free to do so.
We’re going to create a service, which will essentially execute the prior mentioned losetup command. For this, we need a script with the command and a .service file, which will execute the script:

rebloop.sh
#!/bin/bash
sudo losetup -l -P /dev/loop1 "/home/training/my-cluster/osd-$HOSTNAME/30GB.img"

and the service file:

rebloop.service
[Unit]
Description=Reattach loop device after reboot
[Service]
Type=simple
ExecStart=/bin/bash /usr/bin/rebloop.sh
[Install]
WantedBy=multi-user.target

These files have to be executable and be copied to the correct folders. Afterwards, the service must be enabled and can be started.

# chmod +x rebloop.*
# cp rebloop.sh /usr/bin/rebloop.sh
# cp rebloop.service /etc/systemd/system
# systemctl enable rebloop.service
# systemctl start rebloop.service

Ceph, however, will still not want to create an OSD on this device, instead give you following error message:

-->  RuntimeError: Cannot use device (/dev/mapper/<name>). A vg/lv path or an existing device is needed 

You have to make changes to /usr/lib/python2.7/site-packages/ceph_volume/util/disk.py on the OSD host:
in line 201, add “or TYPE==’loop'”:

# use lsblk first, fall back to using stat
TYPE = lsblk(dev).get('TYPE')
if TYPE:
return TYPE == 'disk' or TYPE == 'loop'

and in line 286, change the “skip_loop” switch from “True” to “False”:

 def get_block_devs(sys_block_path="/sys/block", skip_loop=False): 

For testing purposes, simply reboot your system and verify if the loop device gets reattached correctly.
If yes, you can deploy an OSD. We’re using ceph-deploy here:

$ ceph-deploy osd create --data /dev/loop1 $HOSTNAME

When the command was successfully executed on your hosts, you can create your first pool

# ceph osd pool create rbd 100 100 replicated

examine ceph status

# ceph status

tag the pool with an application

# ceph osd pool application enable rbd rbd

As you can see, there are quite some changes to be made, and each of it is failure-prone.
Best practice is to simply use a real block device and for production you really should stick to this.
If, however, there are certain needs to be fulfilled, ceph can be convinced to comply.
 
Sources:
http://tracker.ceph.com/issues/36603
https://tracker.ceph.com/issues/23337
https://github.com/NETWAYS/ceph-training
 
 
 

Tim Albert
Tim Albert
System Engineer

Tim kommt aus einem kleinen Ort zwischen Nürnberg und Ansbach, an der malerischen B14 gelegen. Er hat in Erlangen Lehramt und in Koblenz Informationsmanagement studiert, wobei seine Tätigkeit als Werkstudent bei IDS Scheer seinen Schwenk von Lehramt zur IT erheblich beeinflusst hat. Neben dem Studium hat Tim sich außerdem noch bei einer Werkskundendienstfirma im User-Support verdingt. Blerim und Sebastian haben ihn Anfang 2016 zu uns ins Managed Services Team geholt, wo er sich nun insbesondere...

Monthly Snap June


June kept everyone busy with and excited about the Open Source Data Conference in Berlin. Eleven days before OSDC Keya started the „OSDC 2018 Countdown“. Second week of June the NETWAYS headquarter in Nuremberg was quite quiet. Everyone flew off to Berlin. Everyone? Well, not entirely… One small group of NETWAY-ers kept the NETWAYS flag flying in Nuremberg. Thankfully they had sent a great conference reporter out: Every evening Dirk summed up what had happened in „The Future of Open Source Data Center Solutions – OSDC 2018 – Day 1“ and „… 2“. He also wrote about the „Open Source Camp Issue #1“ . OSCamp will give Open Source projects a platform to present themselves to the Community. This year it started with Foreman and Graylog.
Berlin Events are over for this year, but other great events cast their shadows ahead: „Now is the time to register“ for the upcoming Open Source Monitoring Conference. OSMC takes place in Nuremberg, November 5 to 8.
There is this German saying: „Alles neu macht der Mai“ – for NETWAYS it was June: For OSMC we have created new presentations formats, learn more in „OSMC 2018: Choose what suits you!” And: Julia is new. She just started this month as Marketing Manager and introduced herself in our blogseries „NETWAYS stellt sich vor“. Also new: We have published a „Ceph Training “, as Tim is happy to announce.
At times of DSVGO for Christoph it’s time to reconsider data protection of monitoring servers. In „Einfaches verschlüsseltes Backup“ he explains how one can use GPG to encrypt an icinga2 backup. Nicole shared her thoughts on the „Microsoft and GitHub – merge conflict?“ and recommends to get your own GitLab instance, whereas Michael explains „Continuous Integration with Golang and GitLab“. „Wie überwache ich eine Cluster-Applikation in Icinga 2?“, asked Daniel being at a customer – solving the problem with a little help from his friends. Eric explains „Filter for Multiple Group Memberships in SQL“, that will become even more powerful with the upcoming Icinga Web 2 release. In „Fresh from the shelfDavid reports about command-lines with Ranger, Progress and fzf, and Dirk inspired the Open Source Community about „Contributing as a Non-Developer“. One month, so much going on… Stay tuned!

Julia Hornung
Julia Hornung
Marketing Manager

Julia ist seit Juni 2018 Mitglied der NETWAYS Family. Vor ihrer Zeit in unserem Marketing Team hat sie als Journalistin und in der freien Theaterszene gearbeitet. Ihre Leidenschaft gilt gutem Storytelling, klarer Sprache und ausgefeilten Texten. Privat widmet sie sich dem Klettern und ihrer Ausbildung zur Yogalehrerin.

NETWAYS' Upcoming Training #Summer2018

Our Open Source trainings are designed to support your modern business processes

New Ways with NETWAYS – Our goal is to design your learning process as comfortable as possible! 


Foreman – Life Cycle Management for Servers
2 days training sessions | 03.07.2018 to 04.07.2018
CEPH – Scale out Storage
2 days training sessions | 03.07.2018 to 04.07.2018
GitLab – Modern Software Development
2 days training sessions | 10.07.2018 to 11.07.2018
Icinga 2 Fundamentals – Monitoring Redesigned
4 days training sessions | 10.07.2018 to 13.07.2018
Bareos – Level 1 – Introduction to Bareos
3 days training sessions | 16.07.2018 to 18.07.2018
Graylog – Centralized Logfile Management
2 days training sessions | 17.07.2018 to 18.07.2018
Bareos – Level 2  – Backup with LTO
2 days training sessions | 19.07.2018 to 20.07.2018
Ansible  Configuration Management
2 days training sessions | 24.07.2018 to 25.07.2018
Ansible AWX (Tower) 
1 day training session | 26.07.2018


For the whole NETWAYS training schedule visit: netways.de/training. For assistance please contact us.
 

Keya Kher
Keya Kher
Marketing Manager

Keya ist seit Oktober 2017 in unserem Marketing Team. Sie kennt sich mit Social Media Marketing aus und ist auf dem Weg, ein Grafikdesign-Profi zu werden. Wenn sie sich nicht kreativ auslebt, entdeckt sie andere Städte oder schmökert in einem Buch. Ihr Favorit ist “The Shiva Trilogy”.  

Ceph-Training Insights

2018’s first ceph-training started today with a hot cup of coffee and a little breakfast snack at NETWAYS’ Kesselhaus. The two days training mainly focuses on distributed storage offering performance, resilience and scalability.

Our trainer Tim gives an in-depth insight on structure and basic of ceph, working in detail with practical examples. The training course covers the following contents:

  • Ceph Architecture and Components
  • Installation, Structure and Configuration of a CEPH Cluster
  • Cluster Extensions
  • Crushmap Ruleset
  • Dealing with Rados Block Devices (Snapshot, Import/Export, libvirt, Kernel Module)
  • Installation and Configuration of a Rados Gateway
  • Using Cache Tiers
  • Monitoring and Analysis
  • Best Practices

Feel sorry you missed the March Ceph Training session? Here are the next dates for upcoming ceph-Trainings:
03.07.2018 – 04.07.2018 | Nuremberg
13.11.2018 – 14.11.2018 | Nuremberg
For more details about ceph and all our other training, take a look at our website.

Keya Kher
Keya Kher
Marketing Manager

Keya ist seit Oktober 2017 in unserem Marketing Team. Sie kennt sich mit Social Media Marketing aus und ist auf dem Weg, ein Grafikdesign-Profi zu werden. Wenn sie sich nicht kreativ auslebt, entdeckt sie andere Städte oder schmökert in einem Buch. Ihr Favorit ist “The Shiva Trilogy”.  

February Snap 2018 > Icinga camp, Open Source Datacenter Conference 2018, Ceph, Braintower , NETWAYS Monitor


Freezing February !! And we froze our selection for Open Source Datacenter Conference 2018’s speaker listAlexander said Apple Pi is simple and secure, have you tried it yet? Philipp was happy to celebrate his six -month anniversary at NETWAYS with his fellow trainees Afeef and KillianJulia announced a 25% discount for Ceph training in March.
Marius showed us how to upgrade MariaDB from 10.0.x to 10.2.x. in 10 easy steps. Jean-Marcel talked about the quest for su on Windows. Lennart educated us all with Icinga 2 “Eat or Die”, the variable scope. Nicole announced detailes on the Nextcloud 13 update. Dirk recapped the Configuration Management Camp Ghent 2018.
Isabel invited you to NETWAYS – Monitor your server rooms with us! And told us about MultiTech MTD- H5-2.0 USB powered- easy SMS sending, and, also about the Braintower SMS Gateway with 2 variants. Blerim had the honor of presenting the program for the Icinga Camp Berlin 2018. And, last, but not least,Christoph wanted to Samba Samba all night!!

Keya Kher
Keya Kher
Marketing Manager

Keya ist seit Oktober 2017 in unserem Marketing Team. Sie kennt sich mit Social Media Marketing aus und ist auf dem Weg, ein Grafikdesign-Profi zu werden. Wenn sie sich nicht kreativ auslebt, entdeckt sie andere Städte oder schmökert in einem Buch. Ihr Favorit ist “The Shiva Trilogy”.  

Satte Rabatte – 25% Rabatt auf Foreman & Ceph Trainings im März!


Das Lifecycle Management System Foreman unterstützt bei der Provisionierung virtueller und physikalischer Systeme und beim Konfigurationsmanagement. Die Inhalte der Schulung decken die Bereiche Architektur, Installation, Provisionierung, Konfiguration, sowie die Grundlagen der Automatisierung ab. Seit Oktober 2017 ist außerdem auch Ansible und die Monitoring Konfiguration Teil der zweitägigen Schulung.
Ceph hingegen ist ein distributed Storage und bietet Performance, Ausfallsicherheit und Skalierbarkeit. Da es in modernen Infrastrukturen immer wichtiger wird, auf dynamische und schnelle Anforderungen schnell zu reagieren und um dem immer größer werdendem Datenwachstum gerecht zu werden, bietet Ceph als idealer Cloudstorage für alle Arten von Objekten, Medien und Archiven die optimale Lösung.
Die Qualität der NETWAYS-Schulungen zeichnet sich dadurch aus, dass durch eine geringe Teilnehmerzahl der zielorientierte und effiziente Ablauf jeder Schulung gewährleistet ist. Dadurch ermöglicht sich die genaue Auseinandersetzung mit dem Schulungsstoff und es bleibt für die Trainer genug Zeit, sich mit den individuellen Fragen der Teilnehmer auseinander zu setzen. Auch beim gemeinsamen Abendessen bleibt viel Raum für Diskussionen und Networking unter den Trainingsteilnehmern.
Für die Foreman und Ceph Trainings im März 2018 gewähren wir jeweils 25% Rabatt. Dazu einfach den oben genannten Discountcode im Anmeldeformular für Ceph oder Foreman eingeben und sparen!

Ceph: Increasing placement groups in production

Ceph LogoIf you are running a Ceph cluster for several years you will come to the point where the number of placement groups (PGs) no longer fits to the size of your cluster. Maybe you increased the number of OSDs several times, maybe you misjudged the growth of a pool years ago.
Too few PGs per OSD can lead to drawbacks regarding performance, recovery time and uneven data distribution. The latter can be examined with the command ceph osd df. It shows the usage, weight, variance and number of PGs for each OSD as well the min/max variance and the standard deviation of your OSD usage.
Increasing the number of the PGs can lead to a more even data distribution,
but there are several warnings in blogs or mailings lists, especially for production environments. Doubling the PGs (e.g. from 1024 to 2048) may bring down your cluster for some minutes, because creating, activating and peering the new PGs may strongly influence your client’s traffic.
Regarding this warnings, we decided to increase the PGs in slices of 128. Between each slice we waited until all PGs peered successfully, which needed only a few seconds and did not influence the client traffic at all.
Increasing the number of PGs is done with two simple commands:

$ ceph osd <pool> pg_num <int>
$ ceph osd <pool> pgp_num <int>

Increasing pg_num creates new PGs, but the data rebalancing and backfilling will only start after increasing pgp_num (the number of placement groups for placement), too. pg_num and pgp_num should always have the same value. Increasing your PGs usually comes with a huge amount of backfilling which should not be a problem for a well configured cluster.

Achim Ledermüller
Achim Ledermüller
Lead Senior Systems Engineer

Der Exil Regensburger kam 2012 zu NETWAYS, nachdem er dort sein Wirtschaftsinformatik Studium beendet hatte. In der Managed Services Abteilung ist unter anderem für die Automatisierung des RZ-Betriebs und der Evaluierung und Einführung neuer Technologien zuständig.

Deutsche Open Stack Tage 2017 – Programm

In 6 Wochen ist es soweit: Die deutschen OpenStack Tage (DOST) in München gehen in die 3. Runde!
Die Feinschliffarbeiten sind bereits in vollem Gange und das Programm steht selbstverständlich auch schon fest! Was wird euch alles erwarten?
Die Konferenz lockt jedes Jahr ca. 300 begeisterte OpenStack Sympathisanten an, die sich in zwei Tagen über die neuesten Neuigkeiten rund um den Enterprise-Einsatz von OpenStack informieren und die Gelegenheit wahrnehmen, sich wertvolle Tipps von Profis zu holen.
Das fertige Konferenzprogramm deckt ein breites Themenspektrum ab. Zum einen wird es Vorträge von Vertretern führender, internationaler Unternehmen geben, zum anderen gibt es etliche Fachvorträge von OpenStack Experten zu Case Studies und Best Practices. Am ersten Konferenztag wird es hierzu außerdem Workshops zu den Themen „Konfigurationsmanagement mit Puppet“, „Ceph Storage Cluster“, „Administration von OpenStack“ und „Docker – Die andere Art der Virtualisierung“ geben. Informationen zu den Vorträgen und Referenten findet ihr auf der Konferenzwebseite.
Ergänzend zu den Vorträgen besteht für alle die Möglichkeit, im Foyer des Veranstaltungshotels die Sponsorenausstellung zu besuchen. Hier können die Besucher direkt mit den Referenten und Sponsoren in Kontakt treten. Euch erwarten interessante Diskussionen, aktuellstes Know-How und tolle Networkingmöglichkeiten. Unterstützt werden die 3. Deutschen OpenStack Tage von Noris Network, Fujitsu, Rackspace,  Mirantis, SUSE, Nokia, Canonical, Cumulus, Netzlink, Juniper, Telekom, Vmware, Mellanox, Cisco und NetApp.
An dieser Stelle möchten wir neben den Sponsoren außerdem unseren tollen Medienpartner danken, die auch einen großen Anteil am Erfolg der OpenStack Tage haben. Unsere hochlobenden Dankeshymnen gehen dieses Jahr raus an das deutsche Linux Magazin und den IT-Administrator. Wir haben die Zusammenarbeit mit euch jederzeit genossen und würden uns freuen, euch auch nächstes Jahr wieder mit an Bord zu haben.