Lessons learned – how not to make beginner’s mistakes: Migrating server

At Netways we try to learn all the time. Often you can simply read man pages, change logs, or even tabtab through your shell command at hand.

Trainings and conferences are a bit more time consuming, but can offer you one priceless advantage: the direct communication with someone, who has seen several sides of the topic at hand. I think, you learn best from other peoples experiences – even more from the situations where failure ensued. Here it is critical to not only look at the failure itself, but why it occured and how it was finally resolved.

Using other persons failure as a mean to learn  might seem a bit cynically at first. In an ideal world, however, the same failure would only ocurr once.

So let’s begin with my contribution to a better world, maybe even as a start of a series. This entry is not meant as full post-morten per se, it only describes mistakes you can make and should avoid.

As many might know, our office moved. Having been 10ish years in our “old” office, you might figure that quite a lot “historical infrastructure” can grow in this time. Especially in an IT environment where everybody wants to try something new, better, and undocumented.

Being in the distinguished situation of having direct access to our DC via 1GBit-Fiber, it was possible to use some of our external IPs in our office. VLAN-Tagging, firewall policies, iptables rules – all well known and understood best practices. The services got installed, used and worked flawlessly for all the time and have lived happily ever after.

With the announcement of moving to the new premises, the blue sky became a bit hazy. We had to move all needed hardware devices (NASA could tell you how small and well hidden some of them can be) and also separate them from the unneeded devices. Todays story is about two specific systems, each consisting of two 1U server. Those were some of the systems which used external IPs to provide services to the public. They have been running flawlessly for quite some time and given their age, had started to develop some adorable quirks.

It was my task to move these “dear old ladies” out of the cozy office into the cold, professional DC downstairs.

What harm can 4 old and lovable server possibly do, you might ask? The answer is: None, if you treat them the way they were used to.

First things first: How can you gain access to the DC? Is there a registration process, which has to be followed? How long does this take? Can you access the DC after business hours easily? (Hints: yes, long, no)

Also don’t try to rush things when it comes to shutdown the machines for moving  them. The machines owners like to what is going to happen.

Grab all the tools you will need to remove the server from your previous rack and install them in their new home (cordless screwdriver, all bits you can find)

Are all installation material available? This is not only referring to rack rails, but also cage nuts, screws (size matters) and front covers for feng-shui and air flow. (depends, mostly: no)

Cables! Just collect all the cables you need and then some. Usually, they will be too short. Too long is not an issue you can’t fix with zip ties (you will forget these)

Do you have network access in the DC for debugging, communication etc.? (Hint: depends)

Do you want to move more than one server? Be cautios and do them step by step. You’re absolutely allowed to install them all at the same time. Be aware you might experience crooked rails, incorrect cabling and other time consuming things.

When you have installed the machines, definitely take your time to check these with a KVM device. Whichever is in reach. (you guessed it: there won’t be any when you need them the most). Don’t rely on the machine and its fancy blinkenlights: Some may flash when everything is ok, some flash to indicate errors, some don’t flash at all.

Check all your cabling at least twice, give them a gentle pull – if they come lose, you have to start over again.

Take breaks between different machines. Either try to find a cool spot in the DC (haha) or get outside, have something to drink and return refreshed. The noise (ear plugs, ANC Headphones), temperature, confinement while working in the rack will wear you out eventually.

If you route the machines traffic through several VLANs, make sure all needed switch ports are tagged (or untagged? You decide!) and firewall policies applied for the new location.

Always have a piece of paper and a (working!) pen with you – it’s faster to scribble something on paper than to crawl through yor rats nest of cabling, climb over all the machines you’re up to install and then find your trusty notebook with dead battery.

Before you finally leave, make sure to give everything one last check and, if possible, communicate with the owners of the respective machines. Collect all the tools you brought with you. If you didn’t bring them but “found” them somewhere and used them: make sure to return them.

If you run into any issues, make sure all colleagues you could ask for assistance are currently at the party you’re rushing to attend.

Also make sure to communicate only via phone, so you don’t leave any paper trail when it comes to DC access, network config or time accounting.

When you don’t experience some of these mistakes because of this post, this post was a success. Of course some points are missing (feel free to comment), but I hope the overall pattern is visible:

be prepared, double check and take your time

 

Oh, and don’t forget the key to your racks. There is just one key, right?

Tim Albert
Tim Albert
System Engineer

Tim kommt aus einem kleinen Ort zwischen Nürnberg und Ansbach, an der malerischen B14 gelegen. Er hat in Erlangen Lehramt und in Koblenz Informationsmanagement studiert, wobei seine Tätigkeit als Werkstudent bei IDS Scheer seinen Schwenk von Lehramt zur IT erheblich beeinflusst hat. Neben dem Studium hat Tim sich außerdem noch bei einer Werkskundendienstfirma im User-Support verdingt. Blerim und Sebastian haben ihn Anfang 2016 zu uns ins Managed Services Team geholt, wo er sich nun insbesondere...
OSDC 2019 Part 2 – Automating patching, VMs in containers & much more

OSDC 2019 Part 2 – Automating patching, VMs in containers & much more

After having a really successful and sublime evening event on Tuesday, today was the last day of the Open Source Data Center Conference.

Andreas Lehr and Rico Spiesberger showed off with their automated patch management using Ansible and Rundeck at Lidl and Kaufland.
Things to keep in mind:

  • rebooting bare metal takes time
  • firmware updates might change things
  • have enough space in /var/yum and /tmp -> might result in kernel panics (:

And why not doing a Live Demo at a production system and simply patch the spanish webshop?

On the next slot, it was our pleasure to welcome Kosisochukwu Anyanwu with kinvolk.io. She showed us how to use KVM as a Hypervisor, running a VM in a Docker Container! Get in touch with her at @kosyfrances and feel free to ask her for their use case.

Nikhil Kathole presented how to simplify your IT Workflow with Katello and Foreman. You can use Foreman for provisioning, configuration and monitoring (to some extend) of your hosts. It also provides you with many plugins for flexibility for provisioning tools and infrastructure. If you want to manage your .rpms/.debs, katello should be your choice

 

Troy Harvey started with tossing shirts and gave a quite interesting introduction into the concept of privacy. Privacy demands ethical behaviour of professionals and they should always try to automate things

 

Furthermore, if you want to have to use slack, try to get a job at Carta.

Colin Charles closed the conference with mysql & mariadb security. Obviously breaches are bad and there are the issues you can easily avoid. Use TLS for replication – also update your instances!

I don’t know how you feel after these two days. But I feel like I want more of that! Save the date for next OSDC!

 

 

Tim Albert
Tim Albert
System Engineer

Tim kommt aus einem kleinen Ort zwischen Nürnberg und Ansbach, an der malerischen B14 gelegen. Er hat in Erlangen Lehramt und in Koblenz Informationsmanagement studiert, wobei seine Tätigkeit als Werkstudent bei IDS Scheer seinen Schwenk von Lehramt zur IT erheblich beeinflusst hat. Neben dem Studium hat Tim sich außerdem noch bei einer Werkskundendienstfirma im User-Support verdingt. Blerim und Sebastian haben ihn Anfang 2016 zu uns ins Managed Services Team geholt, wo er sich nun insbesondere...

Aufmerksamkeit – ein super Mittel gegen schädliche Emails

Vor etwa einem halben Jahr stand der Autor im Kontakt mit einer lokalen Metzgerei im Vorwahlbereich 0911. Es ging um leckere Weißwürste mit ebenso leckeren Brezen.

Diese wurden geliefert und NETWAYS-typisch mit vollem Einsatz verzehrt.

Wir nehmen an, es wäre Mittwoch, 20.02.19, 09:19. Emaileingang am persönlichen Postfach:

Erstmal verblüffend:

a) Ich hatte keine Rechnungskopie angefordert.

b) Der Absender hat nichts mit der Metzgerei meines Vertrauens zu tun – die zensierte Emailadresse in der Signatur wäre allerdings korrekt.

c) Die Vorwahl in der Signatur zeigt ins Nirvana

Nun gut, aber man will ja ein verlässlicher Geschäftspartner sein, eventuell liegt es ja nur am “von meinem Samsung gesendet”?

Also klicken wir fröhlich den Link und landen auf einer Seite aus Beheshti Avenue,Tehra,1577837414,Iran??

Außerdem wird eine .doc-Datei angeboten. Auch die kleinste Metzgerei hat es mittlerweile hinbekommen, Rechnungen per pdf zu verschicken.

Aber gut, schauen wir doch mal, was LibreOffice damit so anfängt:

Hier habe ich dann das Experiment abgebrochen.

Es waren zuvor schon viele “Red Flags”, die einen aufmerksamen Emailbenutzer stutzig machen sollten.

Und im Zweifelsfall kann man seinen Geschäftskontakt auch schlicht anrufen, wenn man denn auf Nummer Sicher gehen will.

Wer sich weitere Beispiele anschauen möchte, kann bspw. hier klicken.

edit (TA): typos

Tim Albert
Tim Albert
System Engineer

Tim kommt aus einem kleinen Ort zwischen Nürnberg und Ansbach, an der malerischen B14 gelegen. Er hat in Erlangen Lehramt und in Koblenz Informationsmanagement studiert, wobei seine Tätigkeit als Werkstudent bei IDS Scheer seinen Schwenk von Lehramt zur IT erheblich beeinflusst hat. Neben dem Studium hat Tim sich außerdem noch bei einer Werkskundendienstfirma im User-Support verdingt. Blerim und Sebastian haben ihn Anfang 2016 zu uns ins Managed Services Team geholt, wo er sich nun insbesondere...

Startup Days bei Netways Vol. II

Dass Bernd einen gewissen Hang zum TrashTV hat, dürfte allgemein bekannt sein. Was würde also näher liegen, als sich von einem ehemaligen (stv.) Mitglied im Kunstbeirat des Deutschen Bundestages aus Nürnberg inspirieren zu lassen und seine Netways-Familie in eine selbstkonzipierte Löwenhöhle zu schicken.

Letztes Jahr wurde das Projekt initial gepitcht und dabei fiel unser neues Konferenzbuchungssystem raus.

Bis repetita non placent, könnte man meinen, aber dieses Jahr treten wir mit noch mehr Ideen an als schon 2017.
Zwischen Oktober 2018 und heute kamen etwa 100 Commits auf unsere Wiki-Page und brachten somit 12 Projekte zu Stande.
Zwar wird gibt es hier nochmal einen weed out, aber hier ein kurzer Abriss über die Projekte mit der höchsten Resonanz bisher:
Christian möchte weiter an seinem Windows Monitoring Konzept mit Icinga schrauben und sammelt hierfür Wünsche und Vorschläge.
Marius und Eric planen die Weltherrschaft bis Merz per automatisertem Aktientrading und hoffen auf mehr als 39,24% bzw. 48,52 % in der Endabstimmung.
Max verfolgt einen technischeren Ansatz und will mit seinem “SkyNET(ways)” heute das Office und morgen die Welt automatisiert kontrollieren.

(what could possibly go wrong)

Vanessa dagegen möchte etwas für unsere Gesundheit zu tun, wobei sie fachkundig von Julia unterstützt wird.
Nicole geht mit ihrem Projekt in eine Produktevaluation von Tinkerforge um unser Shop-Portfolio eventuell zu erweitern.
Wie die/der eine oder andere mitbekommen hat, sind wir dabei, in neue Buroräume zu ziehen. Die dort neue Dachterasse versuche ich, unter anderem mit Daniel, in eine Spielwiese für Urban Gardening und Gartenautomation umzuwandeln.
Wie man sehen kann, sind die Interessen bei Netways nicht ausschließlich technisch ausgerichtet.
Unseren Projektverläufen folgen kann man auf twitter: #lifeatnetways und #startupdays
Wer nächstes Jahr mitmachen möchte, darf gerne auf jobs.netways.de vorbeikommen!

Tim Albert
Tim Albert
System Engineer

Tim kommt aus einem kleinen Ort zwischen Nürnberg und Ansbach, an der malerischen B14 gelegen. Er hat in Erlangen Lehramt und in Koblenz Informationsmanagement studiert, wobei seine Tätigkeit als Werkstudent bei IDS Scheer seinen Schwenk von Lehramt zur IT erheblich beeinflusst hat. Neben dem Studium hat Tim sich außerdem noch bei einer Werkskundendienstfirma im User-Support verdingt. Blerim und Sebastian haben ihn Anfang 2016 zu uns ins Managed Services Team geholt, wo er sich nun insbesondere...

Ceph Mimic | Using loop devices as OSD

For quite some time we have been using ceph-deploy to deploy OSD in folders during the Ceph trainings held by Netways. This worked perfectly with jewel, but newer versions don’t allow this behaviour anymore.
There are several reasons for this, however, as we have a quite regulated setup for our training notebooks, we had to come up with some kind of workaround. This approach is, while working fine in our training setup, not recommended for production!
The following steps apply to a current CentOS 7 system.
As stated before, we will deploy an OSD on a block device. Though you could use a separate partition for this, we will use a loop device. For this, the first step is to create a file:
For this, create an OSD directory
 

$ mkdir -p /home/training/my-cluster/osd-$HOSTNAME
$ cd /home/training/my-cluster/osd-$HOSTNAME/

in this folder, create a file for later use

$ fallocate -l 30G 30GB.img

test it

# losetup -l -P /dev/loop1 "/home/training/my-cluster/osd-$HOSTNAME/30GB.img"
# wipefs -a /dev/loop1
# lsblk

This should then display your new loopdevice.
As loop devices are not reboot safe, you need to go some steps further. If you like to use rc.local for this, you’re free to do so.
We’re going to create a service, which will essentially execute the prior mentioned losetup command. For this, we need a script with the command and a .service file, which will execute the script:

rebloop.sh
#!/bin/bash
sudo losetup -l -P /dev/loop1 "/home/training/my-cluster/osd-$HOSTNAME/30GB.img"

and the service file:

rebloop.service
[Unit]
Description=Reattach loop device after reboot
[Service]
Type=simple
ExecStart=/bin/bash /usr/bin/rebloop.sh
[Install]
WantedBy=multi-user.target

These files have to be executable and be copied to the correct folders. Afterwards, the service must be enabled and can be started.

# chmod +x rebloop.*
# cp rebloop.sh /usr/bin/rebloop.sh
# cp rebloop.service /etc/systemd/system
# systemctl enable rebloop.service
# systemctl start rebloop.service

Ceph, however, will still not want to create an OSD on this device, instead give you following error message:

-->  RuntimeError: Cannot use device (/dev/mapper/<name>). A vg/lv path or an existing device is needed 

You have to make changes to /usr/lib/python2.7/site-packages/ceph_volume/util/disk.py on the OSD host:
in line 201, add “or TYPE==’loop'”:

# use lsblk first, fall back to using stat
TYPE = lsblk(dev).get('TYPE')
if TYPE:
return TYPE == 'disk' or TYPE == 'loop'

and in line 286, change the “skip_loop” switch from “True” to “False”:

 def get_block_devs(sys_block_path="/sys/block", skip_loop=False): 

For testing purposes, simply reboot your system and verify if the loop device gets reattached correctly.
If yes, you can deploy an OSD. We’re using ceph-deploy here:

$ ceph-deploy osd create --data /dev/loop1 $HOSTNAME

When the command was successfully executed on your hosts, you can create your first pool

# ceph osd pool create rbd 100 100 replicated

examine ceph status

# ceph status

tag the pool with an application

# ceph osd pool application enable rbd rbd

As you can see, there are quite some changes to be made, and each of it is failure-prone.
Best practice is to simply use a real block device and for production you really should stick to this.
If, however, there are certain needs to be fulfilled, ceph can be convinced to comply.
 
Sources:
http://tracker.ceph.com/issues/36603
https://tracker.ceph.com/issues/23337
https://github.com/NETWAYS/ceph-training
 
 
 

Tim Albert
Tim Albert
System Engineer

Tim kommt aus einem kleinen Ort zwischen Nürnberg und Ansbach, an der malerischen B14 gelegen. Er hat in Erlangen Lehramt und in Koblenz Informationsmanagement studiert, wobei seine Tätigkeit als Werkstudent bei IDS Scheer seinen Schwenk von Lehramt zur IT erheblich beeinflusst hat. Neben dem Studium hat Tim sich außerdem noch bei einer Werkskundendienstfirma im User-Support verdingt. Blerim und Sebastian haben ihn Anfang 2016 zu uns ins Managed Services Team geholt, wo er sich nun insbesondere...

Multiplatform multimeaning multiprotocol desktop messenger – Franz

Gunnar hatte es hier schon einmal kurz angerissen, ich möchte heute nochmal verstärkt darauf zeigen:
Die Application “Franz” auf dem Desktop.
Mit Franz ist hier nicht nur ein ehemaliger Kaiser Österreichs gemeint, sondern eine Multimessengerapp. Diese unterstützt nicht nur unterschiedliche Messenger wie WhatsApp, Slack usw. sondern läuft auch auf Windows, Mac und Linux.
In den Funktionalitäten konnte ich keine Unterschiede zwischen den “nativen” Clients und Franz feststellen.

mmehrere Mmessenger, eine Oberfläche


Das Programm kommt aus Wien und der Datenschutz wird somit auch europäisch reguliert – also gerne ausprobieren!

Tim Albert
Tim Albert
System Engineer

Tim kommt aus einem kleinen Ort zwischen Nürnberg und Ansbach, an der malerischen B14 gelegen. Er hat in Erlangen Lehramt und in Koblenz Informationsmanagement studiert, wobei seine Tätigkeit als Werkstudent bei IDS Scheer seinen Schwenk von Lehramt zur IT erheblich beeinflusst hat. Neben dem Studium hat Tim sich außerdem noch bei einer Werkskundendienstfirma im User-Support verdingt. Blerim und Sebastian haben ihn Anfang 2016 zu uns ins Managed Services Team geholt, wo er sich nun insbesondere...

Adfree Internet with pi-hole

There are two ways to motivate people: intrinsic, motivation stems from the person her/himself or extrinsic, where an external force pushes a person to do something.
I myself experienced this external motivation when I bought a new TV and wanted to test the YouTube-Application.
Using ublock origin on both private and work-related devices, I was surprised by the amount of advertisements being displayed – before and during the video.

mildly annoying, no offense Ben Schmid!


I don’t want to dive to deep into the discussion whether ads and tracking are good or bad, but show you a way how to get rid of most of the ads in your network.
One of the most basic protocol of the internet is DNS. It resolves Domain Names (like nws.netways.de) that you type into your browser to IP Adresses (185.11.252.146) which the browser then invokes. A common analogy would be “phone directory of the Internet”. This is the part we’re going to have closer look on.
Most ad blocking techniques use lists of hosts which deliver the ads (like noisy.popups.com) to your device and direct your device to 127.0.0.1 (localhost) instead. You’re essentially calling yourself.
For web browsers, there are very popular plugins available, even for (rooted) mobile phones similar solutions are known, but have you ever tried to root your TV or Apple device and side load an application?
This is exactly the point where pi-hole is getting handy.
It uses aforementioned lists to block ads but not only for the device you installed an adblocker on but for all devices in your network.
You just have to use pihole as DNS-Server in your network – if you can’t configure a specific DNS fuss-free on your device or router you might want to use pi-hole as your DHCP-Server.
This way it pushes itself as DNS to your devices.
If you want to have a look at it, there are several very good installation guides in multiple languages available.
Feel free to invest one hour of setting it up and enjoy a new ad free network and don’t forget to support the project 🙂

Tim Albert
Tim Albert
System Engineer

Tim kommt aus einem kleinen Ort zwischen Nürnberg und Ansbach, an der malerischen B14 gelegen. Er hat in Erlangen Lehramt und in Koblenz Informationsmanagement studiert, wobei seine Tätigkeit als Werkstudent bei IDS Scheer seinen Schwenk von Lehramt zur IT erheblich beeinflusst hat. Neben dem Studium hat Tim sich außerdem noch bei einer Werkskundendienstfirma im User-Support verdingt. Blerim und Sebastian haben ihn Anfang 2016 zu uns ins Managed Services Team geholt, wo er sich nun insbesondere...

Ceph Training opensourced

 
Besides several other trainings, such as gitlab, foreman or graphite/grafana we’re happy to announce our newest member in the NETWAYS OpenSource training community.
You might have guessed it after reading the headline, we’ve published a ceph training.
 

This training is designed as a two days hands-on training introducing Ceph, its basics, cluster setup and many best practices.
The training participants will get an in-depth insight into the Ceph basics and configuration. They also learn about Ceph Cache Tier, Rados Gateway, RBD, CephFS, Monitoring and Sizing.

– README.md

As ceph is a project with huge momentum, not only regarding spreading but also development, changes are certain to happen.
The training right now is not very specific regarding OpenStack-Integration, so this will be in the focus in the next iteration.
Commits to the trainings are always welcome!
When you’re interested in a training with more interactivity and personal discussion, feel free to get an overview here.

Tim Albert
Tim Albert
System Engineer

Tim kommt aus einem kleinen Ort zwischen Nürnberg und Ansbach, an der malerischen B14 gelegen. Er hat in Erlangen Lehramt und in Koblenz Informationsmanagement studiert, wobei seine Tätigkeit als Werkstudent bei IDS Scheer seinen Schwenk von Lehramt zur IT erheblich beeinflusst hat. Neben dem Studium hat Tim sich außerdem noch bei einer Werkskundendienstfirma im User-Support verdingt. Blerim und Sebastian haben ihn Anfang 2016 zu uns ins Managed Services Team geholt, wo er sich nun insbesondere...

Change Your AD Password easily via OWA

In many environments, Microsoft Active Directory is used to manage users, their roles and permissions and of course their passwords.
When you have set up an  Exchange Server as well, you may want to provide your users with the Outlook Web App. Here also users without Microsoft Windows are able to tune their mail settings and their Active Directory passwords, too.
Sometimes the method may not be completely clear, so please feel free to use this following guide whenever you need to.
Disclaimer: The screenshots display a German OWA and my GIMP skills are, well, improvable.
First you need to login. Please don’t forget to add your domain name.

Then find the small gear and click it.

Use the drop down and navigate to “change password”

You’ll now be prompted to enter your old (1) and your new password twice (2,3). Hit “Save”(4) for using your new password and you’re done!

Tim Albert
Tim Albert
System Engineer

Tim kommt aus einem kleinen Ort zwischen Nürnberg und Ansbach, an der malerischen B14 gelegen. Er hat in Erlangen Lehramt und in Koblenz Informationsmanagement studiert, wobei seine Tätigkeit als Werkstudent bei IDS Scheer seinen Schwenk von Lehramt zur IT erheblich beeinflusst hat. Neben dem Studium hat Tim sich außerdem noch bei einer Werkskundendienstfirma im User-Support verdingt. Blerim und Sebastian haben ihn Anfang 2016 zu uns ins Managed Services Team geholt, wo er sich nun insbesondere...

Philips Brilliance 258B6 and macOS

With Apples policy to get rid of established connection methods and switching entirely to USB-C, we faced the need for keeping hardware connected to the new Apple devices.
As there are many offers for adapting USB-C to any other connectors, we looked thoroughly through these possibilities.
In the end we found a charming solution, Philips’ Brilliance 258B6 (Review)
It provides USB-A, HDMI, DSUB, DVI, Audio connectivity, Ethernet, Display Connection and Power Supply via one single USB-C Slot so other devices could still be connected directly at the MacBooks USB-C ports.
While this setup has been runnig smoothly with mostly Dell Notebooks with Windows and Fedora, macOS was sometimes having issues.
The most annoying one has been intermittend Network connection loss. We have discussed this topic at length and could determine, that there is no fault at our network setup.
After a lot of testing, Philips came up with a solution: Installing the correct driver! You may find version 1.0.17 here.
Having installed this specific version, the monitors started to act as expected.
Another issue some users had to face, was a random vertical shift by one pixel.
Using the OSD you might want to switch off “Pixel Orbiting” and this strange behaviour stopped.
So if you’re thinking about buying a new monitor for your new devices, this monitor might be a very good choice.

Tim Albert
Tim Albert
System Engineer

Tim kommt aus einem kleinen Ort zwischen Nürnberg und Ansbach, an der malerischen B14 gelegen. Er hat in Erlangen Lehramt und in Koblenz Informationsmanagement studiert, wobei seine Tätigkeit als Werkstudent bei IDS Scheer seinen Schwenk von Lehramt zur IT erheblich beeinflusst hat. Neben dem Studium hat Tim sich außerdem noch bei einer Werkskundendienstfirma im User-Support verdingt. Blerim und Sebastian haben ihn Anfang 2016 zu uns ins Managed Services Team geholt, wo er sich nun insbesondere...