Seite wählen

NETWAYS Blog

5 Steps to a DevOps Transformation by Dan Barker | OSDC 2019

von | Mai 12, 2020 | , , ,

YouTube player

 

„It’s not what we believe, it’s what we do that defines our culture“, was on his first slide. At the Open Source Data Center Conference (OSDC) 2019 Dan Barker presented „5 Steps to a DevOps Transformation“. Those who missed the talk back then now get the chance to see the video of Dan’s presentation and read a summary (below).

The former OSDC will be held for the first time in 2020 under the new name stackconf. With the changes in modern IT in recent years, the focus of the conference has increasingly shifted from a mainly static infrastructure approach to a broader spectrum that includes agile methods, continuous integration, container, hybrid and cloud solutions. This development is taken into account by changing the name of the conference and opening the topic area for further innovations. Transformation rules!

Due to concerns around the coronavirus (COVID-19), the decision was made to hold stackconf 2020 as an online conference. The online event will now take place from June 16 – 18, 2020. Join us, live online! Save your ticket now at stackconf.eu/ticket/

5 Steps to a DevOps Transformation

In order to be successful in the new digital economy, it is essential to continuously improve the quality, speed and efficiency of your own organization.

„In this session, we’ll walk through the five steps to transformational change that I’ve found to be important. These are really applicable to any continuously improving organization or any large amount of change in a system. Establish the vision. Create shared experiences. Educate, educate, educate. Find evangelists; Get feedback. I’ll elaborate on each item with methods I’ve used in real transformations at multiple companies. I’ll also describe how these all tie into the DevOps culture, which is really the transformation that’s occurring within the company.“

DevOps professionals primarily work in the tech and software world, creating new technology products, software, and other user services. You will play a key role in the development of new ideas for products and services and manage the process of turning these ideas into realities.

Establish the vision

„A strong team can take any crazy vision and turn it into reality“ – John Carmack

The vision creates empowerment

  • But I‘m not a leader!!!
  • Bold
  • Inspiring
  • Actionable

Pathological – Power oriented

Bureaucratic – Rule oriented

Generative – Performance oriented

If your company values increased productivity, profitability, and market share then DevOps is essential. Even if your goals are non-financial, DevOps will enhance your ability to achieve those goals. The State of DevOps report soundly backs up these claims. More importantly, if your competition has already implemented DevOps and you haven’t, you are already behind. That’s how Walmart feels now that Amazon has built the world’s most efficient shopping platform.

Bad vision → bad outcomes

  • Biased for failure
  • No vision
  • IT-focused
  • Lack of clarity – JFK Moonrace
  • Not actionable

Find evangelists

„It is not about whether you call yourself a leader or not. It is about what you have to show to people as a leader. Leadership is contagious, you carry it and share it“ – Israelmore Ayivor

The control mechanisms that are currently in place to manage your people and projects may not be suited for the DevOps world. You have to be willing to look at items that prevent agility, scalability, and responsiveness and change them. DevOps will provide agility, scalability, and responsiveness, so anything that hinders that process needs to be aligned with the new model.

You can‘t do it alone

  • Use anyone willing to help
  • Nurture this team
  • This team is a bellwether
  • Publicly praise team members

When your organization moves towards developing a DevOps culture, it’s signaling to everyone that participates in the production and release of software they have an equal stake in the success of the company. It’s an all for one, one for all mentality that will break down the communication barriers between teams and make everyone accountable. Once DevOps roles and responsibilities are implemented positive changes will occur, and everyone wins.

Create shared experiences

„Words are symbols for shared memories. If I use a word, then you should have some experience of what the word stands for. If not, the word means nothing to you.“ – Jorge Luis BorgesIm

Bringing people together by sharing

  • Two levels
    • Leadership
    • Organization
  • Equally important

Leadership teams need landmarks

  • Shared information model
  • Reference point
  • Provides inspiration
  • Repeat

To start down your path to DevOps success you need to build a proper DevOps organization which includes all the proper team members. However, the size of your organization plays a big role on how granular you can be with your team. But size doesn’t really matter if you properly define the roles and responsibilities across the organization. The important thing is to make a commitment to the process and get started

The core responsibility that needs to exist is the person who owns the entire DevOps process. This person would usually be someone in a senior position. They are the keeper of the process and procedures and guarantor of the delivery of DevOps value. I like to think of this person as the DevOps evangelist. Aside from the leader, you would need to establish, at a minimum, the following roles: Code Release Manager, Automation Expert, Quality Assurance, Software Developer/Tester, and Security Engineer. The DevOps duties for each of these resources are described below.

Don‘t leave everyone else behind

  • Shared information model
  • Provides motivation
  • Leaders should be leading
  • How?

Educate,…

„An investment in knowledge pays the best interest“ – Benjamin Franklin

Learn something new to build something new

  • Knowledge changes outcomes
  • Make it priority
  • Make it available
  • Monitor it

Measure what matters

  • Accelerate by Dr. Forsgren
  • Westrum Culture Survey
  • User Surveys
  • 1:1 Feedback
  • CultureAmp

Everyone in the company is sailing on the same ship. If the tide goes up so does the ship and everyone on it. But if the tide goes down so does the ship, but no one on the ship is to blame.

Everyone learns differently

  • Online training
  • In-person classes
  • Newsletters
  • Conferences
  • Hackathons

Get feedback

„True intuitive expertise is learned from prolonged experience with good feedback on mistakes“ – Daniel Kahneman

Quellen und Nachschlagewerke

DevOpsDays Berlin: Call for Papers open. Get involved!

von | Mrz 24, 2020 | ,

The organizers of DevOpsDays Berlin once again invite IT professionals with DevOps experience to propose a talk and show their interest in contributing. DevOpsDays is a technical conference, aimed at developers, sysadmins and anyone else involved in technology, whether expert or beginner. The idea behind it: Understanding each others way of working, improving code and cooperation. It was the series of DevOpsDays conferences that made the term DevOps as popular as it is today.

The Call for Papers for DevOpsDays Berlin runs until July 05, 2020. Papers can be submitted at cfp.devopsdays.berlin/2020/.

If you have a hiring success story, can talk about diversity in teams or leadership, have a great tale to tell about spreading DevOps to the rest of your company or organization: Don’t hesitate to propose.

There are three different formats

  • 30-minute talk: Presented during the conference, usually in the mornings. Anyone can submit a presentation. The DevopsDays Berlin program council will choose the best presentations among these submissions, which will appear in the agenda of the conference.
  • Ignite talk: Presented during the Ignite sessions (scheduling varies). The ignite talks are a valuable portion of knowledge, lasting for 5 minutes. 20 slides change automatically every 15 seconds giving a condensed overview on a topic.
  • Hands-On session: Presentations of technical nature lasting for 45 minutes. Speakers should demonstrate a part of their work and give people the chance to ask questions. No product demos allowed.

There will also be Open Space sessions during the conference. The Open Space format is a very interesting and simple one. Short time before the Open Space starts, all attendees can suggest topics and vote on what topic is the most wanted. After this, participants create discussion groups. It is not necessary to propose ahead of time. The topics are suggested in person at the conference.

Get involved!

DevOpsDays events are unique in that they are entirely volunteer-supported, and not hosted by an industry vendor or for profit. If you are in any way interested in supporting that, be it by speaking or by volunteering at the event: Reach out to the organizers. Get involved!

For more information visit https://devopsdays.org/events/2020-berlin/welcome/

DevOpsDays Berlin: Last Tickets

von | Nov 12, 2019 | ,

On Nov 27 – 28 2019 the DevOpsDays Berlin are taking place!

Success factor open space

One of the success factors of previous DevOpsDays has been the open space format. It’s quick, it’s easy, it’s communicative. It fosters the sharing of ideas and insights in the good old fashion, in a non-technical way by just talking to each other. This doesn’t require any effort besides registering and showing up at the event.

Great international line-up

We have introduced the wonderful DevOpsDays line-up before, but let me drop some of these names again, because of the pleasant expectations they arise: Monica Sarbu (Elastic), Beth North, Daniel Löffelholz (ThoughtWorks), Leonardo di Donato (Sysdig), Jessica Anderson (Meltwater)… Or, put differently: Wrapped around the open spaces will be a great line-up of international speakers! Learn more about it here.

Register now, listen carefully and throw yourself into discussion!

DevOpsDays Ghent: Celebrating 10 years DevOps culture

von | Okt 31, 2019 | , ,

Ten years ago the DevOps movement was started by Patrick Debois and Kris Buytaert in Ghent. Who would have guessed that it could become such an innovative movement and community. Today we’ve seen more than 80 DevOpsDays all over the world on each continent. All with the love and help from the core organizers team.

The NETWAYS family made their way to the 10th DevOpsdays anniversary in Ghent to participate and celebrate with the community. This time it also was really special with attending many DevOpsDays organizers from all over the world.

From a historical view, DevOpsDays started out small. In 2015 they’ve started to document how to organize and create communities and it all spread around the globe. More than 50 different countries and organizers met on the first day sharing their experiences and have a good time together. As with anything else in the Open Source world, good documentation is key for the first success to move on and spread the love.

 

DevOps is a culture, not a job title

Patrick shared his journey with making new friends in a new company after leaving the DevOpsDays organizer team five years ago. He’s identified the bottlenecks and silos in every department, and they all solved it with learning from each other. DevOps also is about valueing others work and understand their feelings and emotions.

After 10 years with many DevOpsDays held all over the world, DevOps as a term still is used wrong and needs improvements. It isn’t just creating a new team called „DevOps Engineers“ now replacing the ops team. Neither is it about putting devs on call letting them eat their own dog food. Anyone trying to sell you the perfect DevOps world from a marketing slide with certifications and job titles is just plain wrong. There are many great tools in the wild which can help with bringing the DevOps mindset into the enterprise environment. Not every company is able to immediately dive into the culture, sometimes it takes months or even years to encourage for a change.

DevOpsDays is about sharing these thoughts and emotions, care about diversity and tell you not the ordinary tech story but something to think about. Raise awareness that DevOps is about culture and finding the harmony in your daily workflows. Achieve goals and visions in a shorter amount of time, combine tools and be a role model with sharing your expertise. The DevOps movement is not only a place to sharing experiences with tools and best practices but also talking about work ethics, communication styles and soft skills. With overemphasizing on the technical toolchain, sociotechnity seems to balance this as well in many environments. Emotions are a thing, software is not only code.

From the full-stack engineers not writing device drivers to the most important message: Our definition of the „full stack“ only covers what we understand, and not what’s actually required to run the application. Stop pretending that things are easy, being on call 24/7 doesn’t burn out and new products will solve old problems. Care about high performing teams with the need of psychological safety. Now wait for the recordings, these talks are really interesting to learn from.

 

Wait for it

Slides changing every 15 seconds, 5 minutes time to pitch your story. I like them a lot being entertaining and sometimes you just feel the speaker’s pain with „waiting for slide“. The tradition says that this actually was a malfunction of Kris‘ Linux notebook 😉

That way we’ve heard stories about hot takes, myths and false hoods about DevOps and the danger of DevOps certifications. Before Jason could start his ignite, Kris jumped in announcing that ConfigMgmtCamp registrations are now open. In return, he announced DeliveryConf and jumped right into the meta story being ignites. Watch the recordings when available, it literally made my day. Since announcing confs in ignites was now a thing, Blerim did so too with IcingaConf next year before diving into „Why monitoring is NOT killing observability“. Monversability combing both is a good idea with moving from traditional blackbox monitoring to applications providing metrics and insights. On the other hand, watching graphs all days still requires business process dashboards to immediately visualize failure with alerts. Last but not least, learning about Kubernetes in 5 minutes really nailed it.

 

Move on

We really enjoyed meeting friends old and new and aside from the talks, exploring beautiful Ghent with Belgium waffles and beer. One thing to note – the half marathon from the ground up to the ball room with many stairs was hard in the beginning. On the other hand, our fitness trackers were very happy 😉

Thanks to the organizers and sponsors for the great event – onwards to the next decade!

PS: DevOpsDays Berlin is happening soon. And many more great events near your city. If you don’t have one, kindly contact the core organizers, they are all in with helping to kick-off an amazing event and culture.

 

GitLab CI Runners with Auto-scaling on OpenStack

von | Okt 24, 2019 | , , , ,

 

With migrating our CI/CD pipelines from Jenkins to GitLab CI in the past months, we’ve also looked into possible performance enhancements for binary package builds. GitLab and its CI functionality is really really great in this regard, and many things hide under the hood. Did you know that „Auto DevOps“ is just an example template for your CI/CD pipeline running in the cloud or your own Kubernetes cluster? But there’s more, the GitLab CI runners can run jobs in different environments with using different hypervisors and the power of docker-machine.

One of them is OpenStack available at NWS and ready to use. The following examples are from the Icinga production environment and help us on a daily basis to build, test and release Icinga products.

 

Preparations

Install the GitLab Runner on the GitLab instance or in a dedicated VM. Follow along in the docs where this is explained in detail. Install the docker-machine binary and inspect its option for creating a new machine.

curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
apt-get install -y gitlab-runner
  
curl -L `uname -s`-`uname -m` -o /usr/local/bin/docker-machine
chmod +x /usr/local/bin/docker-machine
  
docker-machine create --driver openstack --help

Next, register the GitLab CI initially. Note: This is just to ensure that the runner is up and running in the GitLab admin interface. You’ll need to modify the configuration in a bit.

gitlab-runner register \
  --non-interactive \
  --url https://git.icinga.com/ \
  --tag-list docker \
  --registration-token SUPERSECRETKEKSI \
  --name "docker-machine on OpenStack" \
  --executor docker+machine \
  --docker-image alpine

 

Docker Machine with OpenStack Deployment

Edit „/etc/gitlab-runner/config.toml“ and add/modify the „[[runners]]“ section entry for OpenStack and Docker Machine. Ensure that the MachineDriver, MachineName and MachineOptions match the requirements. Within „MachineOptions“, add the credentials, flavors, network settings just as with other deployment providers. All available options are explained in the documentation.

vim /etc/gitlab-runner/config.toml

  [runners.machine]
    IdleCount = 4
    IdleTime = 3600
    MaxBuilds = 100
    MachineDriver = "openstack"
    MachineName = "customer-%s"
    MachineOptions = [
      "openstack-auth-url=https://cloud.netways.de:5000/v3/",
      "openstack-tenant-name=1234-openstack-customer",
      "openstack-username=customer-login",
      "openstack-password=sup3rS3cr3t4ndsup3rl0ng",
      "openstack-flavor-name=s1.large",
      "openstack-image-name=Debian 10.1",
      "openstack-domain-name=default",
      "openstack-net-name=customer-network",
      "openstack-sec-groups="mine",
      "openstack-ssh-user=debian",
      "openstack-user-data-file=/etc/gitlab-runner/user-data",
      "openstack-private-key-file=/etc/gitlab-runner/id_rsa",
      "openstack-keypair-name=GitLab Runner"
    ]

The runners cache can be put onto S3 granted that you have this service available. NWS luckily provides S3 compatible object storage.

  [runners.cache]
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "s3provider.domain.localdomain"
      AccessKey = "supersecretaccesskey"
      SecretKey = "supersecretsecretkey"
      BucketName = "openstack-gitlab-runner"





Bootstrap Docker in the OpenStack VM


Last but not least, these VMs need to be bootstrapped with Docker inside a small script. Check the „–engine-install-url“ parameter in the help output:


root@icinga-gitlab:/etc/gitlab-runner# docker-machine create --help
  ...
  --engine-install-url "https://get.docker.com"							Custom URL to use for engine installation 



You can use the official way of doing this, but putting this into a small script also allows customizations like QEMU used for Raspbian builds. Ensure that the script is available via HTTP e.g. from a dedicated GitLab repository 😉


#!/bin/sh
#
# This script helps us to prepare a Docker host for the build system
#
# It is used with Docker Machine to install Docker, plus addons
#
# See --engine-install-url at docker-machine create --help

set -e

run() {
  (set -x; "$@")
}

echo "Installing Docker via get.docker.com"
run curl -LsS https://get.docker.com -o /tmp/get-docker.sh
run sh /tmp/get-docker.sh

echo "Installing QEMU and helpers"
run sudo apt-get update
run sudo apt-get install -y qemu-user-static binfmt-support



Once everything is up and running, the GitLab runners are ready to fire the jobs.


 


Auto-Scaling


Jobs and builds are not run all the time, and especially with cloud resources, this should be a cost-efficient thing. When building Icinga 2 for example, the 20+ different distribution jobs generate a usage peak. With the same resources assigned all the time, this would tremendously slow down the build and release times. In that case, it is desirable to automatically spin up more VMs with Docker and let the GitLab runner take care of distributing the jobs. On the other hand, auto-scaling should also shut down resources in idle times.


By default, one has 4 VMs assigned to the GitLab runner. These builds run non-privileged in Docker, the example below also shows another runner which can run privileged builds. This is needed for Docker-in-Docker to create Docker images and push them to GitLab’s container registry.


root@icinga-gitlab:~# docker-machine ls
NAME                                               ACTIVE   DRIVER      STATE     URL                      SWARM   DOCKER     ERRORS
runner-privileged-icinga-1571900582-bed0b282       -        openstack   Running   tcp://10.10.27.10:2376           v19.03.4
runner-privileged-icinga-1571903235-379e0601       -        openstack   Running   tcp://10.10.27.11:2376           v19.03.4
runner-non-privileged-icinga-1571904408-5bb761b5   -        openstack   Running   tcp://10.10.27.20:2376           v19.03.4
runner-non-privileged-icinga-1571904408-52b9bcc4   -        openstack   Running   tcp://10.10.27.21:2376           v19.03.4
runner-non-privileged-icinga-1571904408-97bf8992   -        openstack   Running   tcp://10.10.27.22:2376           v19.03.4
runner-non-privileged-icinga-1571904408-97bf8992   -        openstack   Running   tcp://10.10.27.22:2376           v19.03.4



Once it detects a peak in the pending job pipeline, the runner is allowed to start additional VMs in OpenStack.


root@icinga-gitlab:~# docker-machine ls
NAME                                               ACTIVE   DRIVER      STATE     URL                      SWARM   DOCKER     ERRORS
runner-privileged-icinga-1571900582-bed0b282       -        openstack   Running   tcp://10.10.27.10:2376           v19.03.4
runner-privileged-icinga-1571903235-379e0601       -        openstack   Running   tcp://10.10.27.11:2376           v19.03.4
runner-non-privileged-icinga-1571904408-5bb761b5   -        openstack   Running   tcp://10.10.27.20:2376           v19.03.4
runner-non-privileged-icinga-1571904408-52b9bcc4   -        openstack   Running   tcp://10.10.27.21:2376           v19.03.4
runner-non-privileged-icinga-1571904408-97bf8992   -        openstack   Running   tcp://10.10.27.22:2376           v19.03.4
runner-non-privileged-icinga-1571904408-97bf8992   -        openstack   Running   tcp://10.10.27.23:2376           v19.03.4

...

runner-non-privileged-icinga-1571904534-0661c396   -        openstack   Running   tcp://10.10.27.24:2376           v19.03.4
runner-non-privileged-icinga-1571904543-6e9622fd   -        openstack   Running   tcp://10.10.27.25:2376           v19.03.4
runner-non-privileged-icinga-1571904549-c456e119   -        openstack   Running   tcp://10.10.27.27:2376           v19.03.4
runner-non-privileged-icinga-1571904750-8f6b08c8   -        openstack   Running   tcp://10.10.27.29:2376           v19.03.4



 


In order to achieve this setting, modify the runner configuration and increase the limit.


vim /etc/gitlab-runner/config.toml

[[runners]]
  name = "docker-machine on OpenStack"
  limit = 24
  output_limit = 20480
  url = "https://git.icinga.com/"
  token = "supersecrettoken"
  executor = "docker+machine"



This would result in 24 OpenStack VMs after a while, and all are idle 24/7. In order to automatically decrease the deployed VMs, use the OffPeak settings. This also ensures that resources are available during workhours while spare time and weekend are considered „off peak“ with shutting down unneeded resources automatically.


    OffPeakTimezone = "Europe/Berlin"
    OffPeakIdleCount = 2
    OffPeakIdleTime = 1800
    OffPeakPeriods = [
      "* * 0-8,22-23 * * mon-fri *",
      "* * * * * sat,sun *"
    ]



Pretty neat functionality 🙂


 


Troubleshooting & Monitoring


„docker-machine ls“ provides the full overview and tells whenever e.g. a connection to OpenStack did not work, or if the VM is currently unavailable.


root@icinga-gitlab:~# docker-machine ls
NAME                                               ACTIVE   DRIVER      STATE     URL                      SWARM   DOCKER     ERRORS
runner-privileged-icinga-1571900582-bed0b282       -        openstack   Error                                      Unknown    Expected HTTP response code [200 203] when accessing [GET https://cloud.netways.de:8774/v2.1/servers/], but got 404 instead



In case you have deleted the running VMs to start fresh, provisioning might take a while and the above can be a false positive. Check the OpenStack management interface to see whether the VMs booted correctly. You can also remove a VM with „docker-machine rm <id>“ and run „gitlab-runner restart“ to automatically provision it again.


Whenever the VM provisioning fails, a gentle look into the syslog (or runner log) unveils what’s the problem. Lately we had used a wrong OpenStack flavor configuration which was fixed after investigating in the logs.


Oct 18 07:08:48 3 icinga-gitlab gitlab-runner[30988]:  #033[31;1mERROR: Error creating machine: Error in driver during machine creation: Unable to find flavor named 1234-customer-id-4-8#033[0;m  #033[31;1mdriver#033[0;m=openstack #033[31;1mname#033[0;m=runner-non-privilegued-icinga-1571375325-3f8176c3 #033[31;1moperation#033[0;m=create



Monitoring your GitLab CI runners is key, and with the help of the REST API, this becomes a breeze with Icinga checks. You can inspect the runner state and notify everyone on-call whenever CI pipelines are stuck.


 


Conclusion


Developers depend on fast CI feedback these days, speeding up their workflow – make them move fast again. Admins need to understand their requirements, and everyone needs a deep-dive into GitLab and its possibilities. Join our training sessions for more practical exercises or immediately start playing in NWS!


			
			

				

				

				 
			

				
				
				
				
			

			

				
				
				
				
				

				
				
				
				
				


Trainings








Web Services








Events