Seite wählen

NETWAYS Blog

5 Steps to a DevOps Transformation by Dan Barker | OSDC 2019

This entry is part 3 of 6 in the series OSDC 2019 | Recap

YouTube player

 

„It’s not what we believe, it’s what we do that defines our culture“, was on his first slide. At the Open Source Data Center Conference (OSDC) 2019 Dan Barker presented „5 Steps to a DevOps Transformation“. Those who missed the talk back then now get the chance to see the video of Dan’s presentation and read a summary (below).

The former OSDC will be held for the first time in 2020 under the new name stackconf. With the changes in modern IT in recent years, the focus of the conference has increasingly shifted from a mainly static infrastructure approach to a broader spectrum that includes agile methods, continuous integration, container, hybrid and cloud solutions. This development is taken into account by changing the name of the conference and opening the topic area for further innovations. Transformation rules!

Due to concerns around the coronavirus (COVID-19), the decision was made to hold stackconf 2020 as an online conference. The online event will now take place from June 16 – 18, 2020. Join us, live online! Save your ticket now at stackconf.eu/ticket/


5 Steps to a DevOps Transformation

In order to be successful in the new digital economy, it is essential to continuously improve the quality, speed and efficiency of your own organization.

„In this session, we’ll walk through the five steps to transformational change that I’ve found to be important. These are really applicable to any continuously improving organization or any large amount of change in a system. Establish the vision. Create shared experiences. Educate, educate, educate. Find evangelists; Get feedback. I’ll elaborate on each item with methods I’ve used in real transformations at multiple companies. I’ll also describe how these all tie into the DevOps culture, which is really the transformation that’s occurring within the company.“

DevOps professionals primarily work in the tech and software world, creating new technology products, software, and other user services. You will play a key role in the development of new ideas for products and services and manage the process of turning these ideas into realities.

Establish the vision

„A strong team can take any crazy vision and turn it into reality“ – John Carmack

The vision creates empowerment

  • But I‘m not a leader!!!
  • Bold
  • Inspiring
  • Actionable

Pathological – Power oriented

Bureaucratic – Rule oriented

Generative – Performance oriented

If your company values increased productivity, profitability, and market share then DevOps is essential. Even if your goals are non-financial, DevOps will enhance your ability to achieve those goals. The State of DevOps report soundly backs up these claims. More importantly, if your competition has already implemented DevOps and you haven’t, you are already behind. That’s how Walmart feels now that Amazon has built the world’s most efficient shopping platform.

Bad vision → bad outcomes

  • Biased for failure
  • No vision
  • IT-focused
  • Lack of clarity – JFK Moonrace
  • Not actionable

Find evangelists

„It is not about whether you call yourself a leader or not. It is about what you have to show to people as a leader. Leadership is contagious, you carry it and share it“ – Israelmore Ayivor

The control mechanisms that are currently in place to manage your people and projects may not be suited for the DevOps world. You have to be willing to look at items that prevent agility, scalability, and responsiveness and change them. DevOps will provide agility, scalability, and responsiveness, so anything that hinders that process needs to be aligned with the new model.

You can‘t do it alone

  • Use anyone willing to help
  • Nurture this team
  • This team is a bellwether
  • Publicly praise team members

When your organization moves towards developing a DevOps culture, it’s signaling to everyone that participates in the production and release of software they have an equal stake in the success of the company. It’s an all for one, one for all mentality that will break down the communication barriers between teams and make everyone accountable. Once DevOps roles and responsibilities are implemented positive changes will occur, and everyone wins.

Create shared experiences

„Words are symbols for shared memories. If I use a word, then you should have some experience of what the word stands for. If not, the word means nothing to you.“ – Jorge Luis BorgesIm

Bringing people together by sharing

  • Two levels
    • Leadership
    • Organization
  • Equally important

Leadership teams need landmarks

  • Shared information model
  • Reference point
  • Provides inspiration
  • Repeat

To start down your path to DevOps success you need to build a proper DevOps organization which includes all the proper team members. However, the size of your organization plays a big role on how granular you can be with your team. But size doesn’t really matter if you properly define the roles and responsibilities across the organization. The important thing is to make a commitment to the process and get started

The core responsibility that needs to exist is the person who owns the entire DevOps process. This person would usually be someone in a senior position. They are the keeper of the process and procedures and guarantor of the delivery of DevOps value. I like to think of this person as the DevOps evangelist. Aside from the leader, you would need to establish, at a minimum, the following roles: Code Release Manager, Automation Expert, Quality Assurance, Software Developer/Tester, and Security Engineer. The DevOps duties for each of these resources are described below.

Don‘t leave everyone else behind

  • Shared information model
  • Provides motivation
  • Leaders should be leading
  • How?

Educate,…

„An investment in knowledge pays the best interest“ – Benjamin Franklin

Learn something new to build something new

  • Knowledge changes outcomes
  • Make it priority
  • Make it available
  • Monitor it

Measure what matters

  • Accelerate by Dr. Forsgren
  • Westrum Culture Survey
  • User Surveys
  • 1:1 Feedback
  • CultureAmp

Everyone in the company is sailing on the same ship. If the tide goes up so does the ship and everyone on it. But if the tide goes down so does the ship, but no one on the ship is to blame.

Everyone learns differently

  • Online training
  • In-person classes
  • Newsletters
  • Conferences
  • Hackathons

Get feedback

„True intuitive expertise is learned from prolonged experience with good feedback on mistakes“ – Daniel Kahneman

Quellen und Nachschlagewerke

DevOpsDays Berlin: Call for Papers open. Get involved!

The organizers of DevOpsDays Berlin once again invite IT professionals with DevOps experience to propose a talk and show their interest in contributing. DevOpsDays is a technical conference, aimed at developers, sysadmins and anyone else involved in technology, whether expert or beginner. The idea behind it: Understanding each others way of working, improving code and cooperation. It was the series of DevOpsDays conferences that made the term DevOps as popular as it is today.

The Call for Papers for DevOpsDays Berlin runs until July 05, 2020. Papers can be submitted at cfp.devopsdays.berlin/2020/.

If you have a hiring success story, can talk about diversity in teams or leadership, have a great tale to tell about spreading DevOps to the rest of your company or organization: Don’t hesitate to propose.

There are three different formats

  • 30-minute talk: Presented during the conference, usually in the mornings. Anyone can submit a presentation. The DevopsDays Berlin program council will choose the best presentations among these submissions, which will appear in the agenda of the conference.
  • Ignite talk: Presented during the Ignite sessions (scheduling varies). The ignite talks are a valuable portion of knowledge, lasting for 5 minutes. 20 slides change automatically every 15 seconds giving a condensed overview on a topic.
  • Hands-On session: Presentations of technical nature lasting for 45 minutes. Speakers should demonstrate a part of their work and give people the chance to ask questions. No product demos allowed.

There will also be Open Space sessions during the conference. The Open Space format is a very interesting and simple one. Short time before the Open Space starts, all attendees can suggest topics and vote on what topic is the most wanted. After this, participants create discussion groups. It is not necessary to propose ahead of time. The topics are suggested in person at the conference.

Get involved!

DevOpsDays events are unique in that they are entirely volunteer-supported, and not hosted by an industry vendor or for profit. If you are in any way interested in supporting that, be it by speaking or by volunteering at the event: Reach out to the organizers. Get involved!

For more information visit https://devopsdays.org/events/2020-berlin/welcome/

tmux – terminal multiplexer

In diesem Artikel möchte ich euch kurz das Tool tmux vorstellen, dass zu meinen Lieblingstools unter Linux gehört. In einem vorherigen Artikel hat Johannes schon einmal die Basisconfig von tmux gezeigt.

Da tmux eine Server Client Struktur hat, bevorzuge ich es in der Regel die Finger von der tmux config zu lassen. Da man die sonst auf jedem Server pflegen muss, mit dem man arbeiten möchte. Leicht hat es an der Stelle derjenige, der Puppet oder Ansible nutzt und seine eigenen Server pflegt. Ich als Consultant habe meistens Kundenserver, auf denen ich nichts verändern möchte was nicht nötig ist.

tmux hat für mich 2 Einsatzzwecke:

  • Bei Präsentationen Bildschirm- und Beamerterminal synchronisieren.
  • Bei Servern meine Sessions schützen, so dass ich nach Verbindungsabbrüchen weiterarbeiten kann.

Besonders der zweite Zweck ist sehr praktisch, da ich auch Bahnfahrer bin.

CLI

Jetzt zur eigentlichen Benutzung: Gestartet wird tmux über den jeweiligen User. Wenn man ein Terminal vor sich hat sind folgende Basisbefehle praktisch:

  • # tmux -> startet eine neue unbenannte Session
  • # tmux new -s „MeineSession“ -> startet eine benannte Session
  • # tmux list-sessions -> listet Sessions
  • # tmux -a -> verbindet zu einer unbenanntenten Session
  • # tmux -a -t „MeineSession“

Inside tmux

Wenn wir jetzt in einer tmux Session sind kann das folgendermaßen aussehen:

tmux ScreenshotTmux unterscheidet zwischen „window“ und „pane“. Wir sehen in diesem Beispiel 4 panes in einem window.
Um innerhalb von tmux zu navigieren braucht gibt es diverse Keybindings die immer mit Strg+b eingeleitet werden. Hinweis: Bei screen, einem ähnlichen aber älteren tool als tmux, war es Strg+a. Wem das lieber ist, der kann bei Johannes nachschlagen, wie man das umstellt.

Im Window hat man folgende Befehle:

  • Strg-b, % Horizontaler Split
  • Strg-b, “ Vertikaler Split
  • Strg-b, x Schließt den aktuellen Pane
  • Strg-b, <Pfeil-Tasten> Zwischen Panes hin und her springen
  • Strg-b, d detach, von tmux abmelden ohne es zu beenden

Ein Window ist vergleichbar mit einem Tab in deinem Browser. Zwischen einzelnen Windows navigiert man mit:

  • Strg-b, c Neues Window
  • Strg-b, w Windows auflisten
  • Strg-b, [0…9] zum Window Nr. 0-9 wechseln

Kopieren

Ein praktisches Feature ist das kopieren. Das funktioniert ähnlich wie in Vim aber, da es Terminals sind, auch zwischen verschiedenen Servern.

  1. Als Erstes braucht man mindestens 2 Panes.
  2. Strg-b, [ Öffnet den Scroll-Modus
  3. <Space> startet den Markier-Modus unterm Cursor
  4. <CR/Enter> kopiert
  5. Nach dem Wechsel ins nächste Pane . . .
  6. Strg-b, ] fügt ein

Am besten probiert man das alles selber mal aus. Es gibt auch eine Menge Cheat-Sheets zu dem Thema, die vor allem am Anfang hilfreich sein können.

Christoph Niemann
Christoph Niemann
Senior Consultant

Christoph hat bei uns im Bereich Managed Service begonnen und sich dort intensiv mit dem internen Monitoring auseinandergesetzt. Seit 2011 ist er nun im Consulting aktiv und unterstützt unsere Kunden vor Ort bei größeren Monitoring-Projekten und PERL-Developer-Hells.

Ansible – should I use omit filter?

When we talk about Ansible, we more and more talk about AWX or Tower. This Tool comes in handy when you work with Ansible in a environment shared with colleagues or multiple teams.
In AWX we can reuse the playbooks we developed and share them with our colleagues on a GUI Platform.

Often we need a bit of understanding how a playbook is designed or if a variable need to be defined for the particular play. This can be much more tricky when sharing templates to people unaware of your work.

This is where the omit filter can be used. The easiest way to explain this, if the variable has no content or isn’t defined, omit the parameter.

The following example is an extract from the documentation:


- name: touch files with an optional mode
  file:
    dest: "{{ item.path }}"
    state: touch
    mode: "{{ item.mode | default(omit) }}"
  loop:
    - path: /tmp/foo
    - path: /tmp/bar
    - path: /tmp/baz
      mode: "0444"

In AWX we can create surveys, those are great to ask a few questions and provide a guide on how to use the underlying play. But often we need to choose between two variables whether one or another action should happen. Defined by the variable in use. If we leave one of both empty, Ansible will see those empty as defined but „None“ (Python null) as content.

With the omit filter we can remove the parameter from the play, so if the parameter is empty it won’t be used.

The following code is the usage of icinga2_downtimes module which can create downtimes for hosts or hostgroups but the parameters cannot be used at the same time. In this case I can show the variable for hostnames and hostgroups in the webinterface. The user will use one variable and the other variable will be removed and therefore no errors occur.


- name: schedule downtimes
  icinga2_downtimes:
    host: 
    username: icinga_downtime
    password: "{{ icinga_downtime_password }}"
    hostnames: "{{ icinga2_downtimes_hostnames | default(omit) }}"
    hostgroups: "{{ icinga2_downtimes_hostgroups | default(omit) }}"
    all_services: "{{ icinga2_downtimes_allservices | default(False) }}"

The variables shown in the AWX GUI on the template.

This filter can be used in various other locations to provide optional parameters to your users.

If you want to learn more about Ansible, checkout our Ansible Trainings or read more on our blogpost.

Thilo Wening
Thilo Wening
Manager Consulting

Thilo hat bei NETWAYS mit der Ausbildung zum Fachinformatiker, Schwerpunkt Systemadministration begonnen und unterstützt nun nach erfolgreich bestandener Prüfung tatkräftig die Kollegen im Consulting. In seiner Freizeit ist er athletisch in der Senkrechten unterwegs und stählt seine Muskeln beim Bouldern. Als richtiger Profi macht er das natürlich am liebsten in der Natur und geht nur noch in Ausnahmefällen in die Kletterhalle.

From Monitoring to Observability: Distributed Tracing with Jaeger

Modern cloud environments and microservice architectures need a changed mindset when it comes to monitoring. Classic host/service object relations are not always applicable, containers run in Kubernetes are short lived, and application performance within distributed networks is hard to monitor. Especially with applications running millions of operations, where to start the root cause analysis for slow client responses in your web shop? Is it just slow connections to MySQL, or does the application’s debug log slow down the entire fleet?

This is where observability with tracing comes into play. In the cloud native space, OpenTracing evolved as vendor neutral standardized API including client instrumentation. Famous tools are Zipkin and Jaeger which was contributed from Uber to the CNCF.

Let’s have a look into Jaeger today.

 

Getting Started

The easiest way to try Jaeger is with using the Docker container explained in the documentation.

docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14268:14268 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.16

Navigate to http://localhost:16686 to get greeted by Jaeger.

 

Try it

A sample application is available as container. I’m using a modified port mapping with 8081-8084 here since port 8080 is already assigned.

docker run --rm -it \
  --link jaeger \
  -p8081-8084:8080-8083 \
  -e JAEGER_AGENT_HOST="jaeger" \
  jaegertracing/example-hotrod:1.16 \
  all

Navigate to http://localhost:8081 and click the buttons to order some cars.

Within Jaeger itself, start analyzing the traces and for example learn that Redis runs into timeouts quite often delaying the HTTP responses to the clients.

 

Traces for applications

Jaeger provides officially supported client libraries for Go, Java, Python, NodeJS, C/C++, C#/.NET, others are in the making. Let’s see if we can add it into Icinga 2 and do some tracing here.

First off, clone the repository, build and install the client libraries. You’ll need CMake, a C++ compiler, etc. – basically everything which is required for Icinga 2 too and documented here. In this example, I’m compiling on my Macbook. There are additional libraries and headers required. Hint: Boost 1.72 has a bug which needs a patch.

brew install yaml-cpp thrift 
git clone https://github.com/opentracing/opentracing-cpp && cd opentracing-cpp
# 1.6.0 doesn't work atm
git checkout v1.5.1
mkdir -p build && cd build
cmake ..
make && make install
cd ..

Then clone, cmake, make, install.

git clone https://github.com/jaegertracing/jaeger-client-cpp && cd jaeger-client-cpp
git checkout v0.5.0
# regenerate thrift headers for 0.11.0
find idl/thrift/ -type f -name \*.thrift -exec thrift -gen cpp -out src/jaegertracing/thrift-gen {} \;
mkdir -p build && cd build
cmake ..
make && make install
cd ..

In order to add Jaeger into Icinga 2, there are the following steps necessary:

  • Add CMake functions to lookup yaml-cpp, opentracing, jaeger headers and libraries
  • Optionally enable Jaeger tracing code, link the icinga2 binary against it
  • Add a new tracer into the Config Compiler CLI command to measure the timing points

The majority of development time today was to resolve compile and linking issues, adding spans and traces is really simple. You can follow my progress in this Icinga PR – developers, get started wtih the client libraries and help your colleagues with enhanced observability!

 

Conclusion

Tracing application performance, cluster messages, end2end tests and any sort of event span provides valuable insights for both, devs and ops. With the new cloud native landscape evolving fast, we have additional possibilities to analyze our environments. Next to the now standardized tools for parsing and ingesting logs with Elastic Stack or Graylog, tracing has found its place in the observability stack.

Jaeger Tracing also is part of the GitLab observability suite which will be moved to the free core edition in 2020. Metrics, logging, alerts and tracing are key elements in modern cloud environments. Prometheus monitors everything from classic load checks to Kubernetes containers, Jaeger provides application performance insights and on top of that, Grafana combines the view on problems and trends. You can learn more about GitLab’s vision here.

Exploring these cool new features in GitLab are our mission in future trainings and workshops, watch this space in 2020!