Select Page

Lessons learned – how not to make beginner’s mistakes: Migrating server

by | Jul 3, 2019 | NETWAYS, Hardware, Web Services, Network

At Netways we try to learn all the time. Often you can simply read man pages, change logs, or even tabtab through your shell command at hand.

Trainings and conferences are a bit more time consuming, but can offer you one priceless advantage: the direct communication with someone, who has seen several sides of the topic at hand. I think, you learn best from other peoples experiences – even more from the situations where failure ensued. Here it is critical to not only look at the failure itself, but why it occured and how it was finally resolved.

Using other persons failure as a mean to learn  might seem a bit cynically at first. In an ideal world, however, the same failure would only ocurr once.

So let’s begin with my contribution to a better world, maybe even as a start of a series. This entry is not meant as full post-morten per se, it only describes mistakes you can make and should avoid.

As many might know, our office moved. Having been 10ish years in our “old” office, you might figure that quite a lot “historical infrastructure” can grow in this time. Especially in an IT environment where everybody wants to try something new, better, and undocumented.

Being in the distinguished situation of having direct access to our DC via 1GBit-Fiber, it was possible to use some of our external IPs in our office. VLAN-Tagging, firewall policies, iptables rules – all well known and understood best practices. The services got installed, used and worked flawlessly for all the time and have lived happily ever after.

With the announcement of moving to the new premises, the blue sky became a bit hazy. We had to move all needed hardware devices (NASA could tell you how small and well hidden some of them can be) and also separate them from the unneeded devices. Todays story is about two specific systems, each consisting of two 1U server. Those were some of the systems which used external IPs to provide services to the public. They have been running flawlessly for quite some time and given their age, had started to develop some adorable quirks.

It was my task to move these “dear old ladies” out of the cozy office into the cold, professional DC downstairs.

What harm can 4 old and lovable server possibly do, you might ask? The answer is: None, if you treat them the way they were used to.

First things first: How can you gain access to the DC? Is there a registration process, which has to be followed? How long does this take? Can you access the DC after business hours easily? (Hints: yes, long, no)

Also don’t try to rush things when it comes to shutdown the machines for moving  them. The machines owners like to what is going to happen.

Grab all the tools you will need to remove the server from your previous rack and install them in their new home (cordless screwdriver, all bits you can find)

Are all installation material available? This is not only referring to rack rails, but also cage nuts, screws (size matters) and front covers for feng-shui and air flow. (depends, mostly: no)

Cables! Just collect all the cables you need and then some. Usually, they will be too short. Too long is not an issue you can’t fix with zip ties (you will forget these)

Do you have network access in the DC for debugging, communication etc.? (Hint: depends)

Do you want to move more than one server? Be cautios and do them step by step. You’re absolutely allowed to install them all at the same time. Be aware you might experience crooked rails, incorrect cabling and other time consuming things.

When you have installed the machines, definitely take your time to check these with a KVM device. Whichever is in reach. (you guessed it: there won’t be any when you need them the most). Don’t rely on the machine and its fancy blinkenlights: Some may flash when everything is ok, some flash to indicate errors, some don’t flash at all.

Check all your cabling at least twice, give them a gentle pull – if they come lose, you have to start over again.

Take breaks between different machines. Either try to find a cool spot in the DC (haha) or get outside, have something to drink and return refreshed. The noise (ear plugs, ANC Headphones), temperature, confinement while working in the rack will wear you out eventually.

If you route the machines traffic through several VLANs, make sure all needed switch ports are tagged (or untagged? You decide!) and firewall policies applied for the new location.

Always have a piece of paper and a (working!) pen with you – it’s faster to scribble something on paper than to crawl through yor rats nest of cabling, climb over all the machines you’re up to install and then find your trusty notebook with dead battery.

Before you finally leave, make sure to give everything one last check and, if possible, communicate with the owners of the respective machines. Collect all the tools you brought with you. If you didn’t bring them but “found” them somewhere and used them: make sure to return them.

If you run into any issues, make sure all colleagues you could ask for assistance are currently at the party you’re rushing to attend.

Also make sure to communicate only via phone, so you don’t leave any paper trail when it comes to DC access, network config or time accounting.

When you don’t experience some of these mistakes because of this post, this post was a success. Of course some points are missing (feel free to comment), but I hope the overall pattern is visible:

be prepared, double check and take your time

 

Oh, and don’t forget the key to your racks. There is just one key, right?

Tim Albert
Tim Albert
Senior Systems Engineer

Tim kommt aus einem kleinen Ort zwischen Nürnberg und Ansbach, an der malerischen B14 gelegen. Er hat in Erlangen Lehramt und in Koblenz Informationsmanagement studiert. Seit Anfang 2016 ist er bei uns tätig. Zuerst im Managed Services Team, dort kümmerte Tim sich um Infrastrukturthemen und den internen Support, um dann 2019 - zusammen mit Marius - Gründungsmitglied der ITSM Abteilung zu werden. In seiner Freizeit engagiert sich Tim in der Freiwilligen Feuerwehr – als Maschinist und Atemschutzgeräteträger -, spielt im Laientheater Bauernschwänke und ist auch handwerklich ein absolutes Allroundtalent. Angefangen von Mauern hochziehen bis hin zur KNX-Verkabelung ist er jederzeit...

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

More posts on the topic NETWAYS | Hardware | Web Services | Network

Monthly Snap Mai 2024

Hallo zusammen! Der Mai ist unglaublich schnell verflogen mit Star Wars Day, DevOpsDays und den vielen Feiertagen. Aber rückblickend war es ein schöner Frühlingsmonat. Unser Blog war im Mai stark auf die Konferenzen fokussiert, denn da warten ja noch einige Highlights...

Grab One of the Last Tickets to stackconf 2024!

The clock is ticking, and if you haven't secured your spot at stackconf 2024 yet, now is the time! This premier conference, held in Berlin on June 18-19, is a must-attend event for anyone involved in IT infrastructure, cloud native solutions, and DevOps. With a...

Praktikumswoche 2024 | Das Ticket zu Deiner IT-Zukunft

Wir von NETWAYS sind voller Vorfreude darauf, junge Talente während der Praktikumswoche Mittelfranken zu begrüßen und ihnen die faszinierende Welt der Informationstechnologie näherzubringen.   Das ist die Praktikumswoche Die Praktikumswoche ist eine fantastische...

Azubi-Projektwoche 2024

Hallo zusammen, wie jedes Jahr fand auch dieses Jahr unsere spannende Azubi-Projektwoche bei NETWAYS statt. Für alle, die noch nicht wissen, worum es geht: In dieser Woche kommen alle NETWAYS Azubis zusammen, um gemeinsam ein Projekt zu realisieren. Besonders...