Select Page

NETWAYS Blog

Kubernetes Custom Resources with Kubeless and Metacontroller by Michael Grüner | OSDC 2019

This entry is part 1 of 6 in the series OSDC 2019 | Recap

YouTube player

 

This talk is about our journey from Nginx & Docker Swarm to Traefik & Nomad. At the Open Source Data Center Conference (OSDC) 2019 in Berlin, Jan Martens invited to audience to travel with him in his talk „Evolution of a Microservice-Infrastructure”. You have missed him speaking? We got something for you: See the video of Michael’s presentation and read a summary (below).

The former OSDC will be held for the first time in 2020 under the new name stackconf. With the changes in modern IT in recent years, the focus of the conference has increasingly shifted from a mainly static infrastructure approach to a broader spectrum that includes agile methods, continuous integration, container, hybrid and cloud solutions. This development is taken into account by changing the name of the conference and opening the topic area for further innovations.

Due to concerns around the coronavirus (COVID-19), the decision was made to hold stackconf 2020 as an online conference. The online event will now take place from 16 to 18 June 2020. Join us, live online! Save your ticket now at: stackconf.eu/ticket/


 

Kubernetes Custom Resources with Kubeless and Metacontroller

Mostly Michael is working with all the Kubernetes stuff in his company (also with Foreman). Developing additional tools e.g. scripts and tools for deploy applications. Custom Resources from Kubernetes and how to use them.

The goal of Michael‘s talk: Create your own Kubernetes resources that look like, behave like and can be used like every other Kubernetes resource. For example, if you want to let your users manage helm for Kubernetes, you have to give them access to your instance, what means, they can control the hole cluster. That’s not what you want.

The recipe:

  • A resource definition
  • Some logic to make it do something
  • Some API magic to make it all stick together

 

1) Resource definition

CustomResourceDefinition = It is used to tell Kubernetes about your new resource.

It consists of three parts:

  • API group
  • some metadata
  • the kind (the name of your resource)

 

2) Some logic to make it to something

For this, Michael uses a tool called Kubeless. It is a so called function as a service framework. It is basically a framework where you can say ‘Here is my code, make it work as a webservice’. (That’s a very oversimplified description of what it is, but for the use case in this presentation it is enough.)

Use Kubeless to create our actual resource logic in form of a web service. How do we do that? As everytime in Kubernetes we use a Kubernetes resource (in this case it is a function).

Example:

  • API
  • kind (function)
  • metadata (name, namespace, etc.)
  • spec (runtime, handler, checksum, function-content-type, the actual function)

The function (logic) can now be found at .

 

3) Some API magic to make it all stick together

How to connect your custom resource with your logic? As always, we use a Kubernetes resource. The special feature here is to give the resource the parentResource, that means, the apiVersion of the custom resource. We also have to tell, what childResource it has to create. In our case just “pods”. 

Furthermore, we have to define what to do if the resource changes. For example, what should happen if we change the parent resource or add some configuration? At the end of the configuration we have to tell where to find the logic we built. In our example it is .

And this ties our two pieces together. It connects our custom resource definition with our logic. Now we have on the right our logic, on the left our resource definition and meta controller for the communication in the middle.

 

If the user now creates a new instance of our custom resource, the logic gets notified by the API, that ensures the meta controller gets notified. After this, the meta controller calls our function and returns a JSON structure containing the status and the list of children. Meta Controller then uses these information to do whatever API requests are necessary to create these children.

If you are interested now, watch the full video to have a look at a live demo how this works!

Tobias Bauriedel
Tobias Bauriedel
Assistant Manager Operations

Tobias ist ein offener und gelassener Mensch, dem vor allem der Spaß an der Arbeit wichtig ist. Bei uns hat er seine Ausbildung zum Fachinformatiker für Systemintegration abgeschlossen und arbeitet nun im NETWAYS Professional Services - Team Operations und entwickelt nebenbei Projekte für die NPS. In seiner Freizeit engagiert er sich ehrenamtlich aktiv bei der Freiwilligen Feuerwehr als Atemschutzgerätetrager und Maschinist, bereist die Welt und unternimmt gerne etwas mit Freunden.

Evolution of a Microservice-Infrastructure by Jan Martens | OSDC 2019

This entry is part 2 of 6 in the series OSDC 2019 | Recap

YouTube player

 

At the Open Source Data Center Conference (OSDC) 2019 in Berlin, Jan Martens invited to audience to travel with him in his talk „Evolution of a Microservice-Infrastructure”. You have missed him speaking? We got something for you: See the video of Jan‘s presentation and read a summary (below).

The former OSDC will be held for the first time in 2020 under the new name stackconf. With the changes in modern IT in recent years, the focus of the conference has increasingly shifted from a mainly static infrastructure approach to a broader spectrum that includes agile methods, continuous integration, container, hybrid and cloud solutions. This development is taken into account by changing the name of the conference and opening the topic area for further innovations.

Due to concerns around the coronavirus (COVID-19), the decision was made to hold stackconf 2020 as an online conference. The online event will now take place from June 16 to 18, 2020. Join us, live online! Save your ticket now at: stackconf.eu/ticket/


 

Evolution of a Microservice-Infrastructure

Jan Martens signed up with a talk titled “Evolution of a Microservice Infrastructure” and why should I summarize his talk if he had done that himself perfectly: “This talk is about our journey from Ngnix & Docker Swarm to Traefik & Nomad.”

But before we start getting more in depth with this talk, there is one more thing to know about it. This is more or less a sequel to “From Monolith to Microservices” by Paul Puschmann a colleague of Jan Martens, but it’s not absolutely necessary to watch them in order or both.

YouTube player

 

So there will be a bunch of questions answered by Jan during the talk, regarding their environment, like: “How do we do deployments? How do we do request routing? What problems did we encounter, during our infrastructural growth and how did we address them?”

After giving some quick insight in the scale he has to deal with, that being 345.000 employees and 15.000 shops, he goes on with the history of their infrastructure.

Jan works at REWE Digital, which is responsible for the infrastructure around services, like delivery of groceries. They started off with the takeover of an existing monolithic infrastructure, not very attractive huh? They confronted themselves with the question: “How can we scale this delivery service?” and the solution they came up with was a micro service environment. Important to point out here, would be the use of Docker/Swarm for the deployment of micro services.

Let’s skip ahead a bit and take a look at the state of 2018 REWE Digital. Well there operating custom Docker-Environment consists of: Docker, Consul, Elastic Stack, ngnix, dnsmasq and debian

Jan goes into explaining his infrastructure more and more and how the different applications work with each other, but let’s just say: Everything was fine and peaceful until the size of the environment grew to a certain point. And at that point problems with nginx were starting to surface, like requests which never reached their destination or keepalive connections, which dropped after a short time. The reason? Consul-template would reload all ngnix instances at the same time. The solution? Well they looked for a different reverse proxy, which is able to reload configuration dynamically and best case that new reverse proxy is even able to be configured dynamically.

The three being deemed fitting for that job were envoy, Fabio and traefik, but I have already spoiled their decision, its treafik. The points Jan mentioned, which had them decide on traefik were that it is dynamically configurable and is able to reload configuration live. That’s obviously not all, lots of metrics, a web ui, which was deemed nice by Jan and a single go binary, might have made the difference.

Jan drops a few words on how migration is done and then invests some time in talking about the benefits of traefik, well the most important benefit for us to know is, that the issues that existed with ngnix are gone now.

Well now that the environment was changed, there were also changes coming for swarm, acting on its own. The problems Jan addresses are a poor container spread, no self-healing, and more. You should be able to see where this is going. Well the candidates besides Docker Swarm are Rancher, Kubernetes and Nomad. Well, this one was spoiled by me as well.

The reasons to use nomad in this infrastructure might be pretty obvious, but I will list them anyway. Firstly, seamless consul integration, well both are by HashiCorp, who would have guessed. Nomad is able to selfheal and comes in a single go binary, just like traefik. Jan also claims it has a nice web UI, we have to take his word on that one.

Jan goes into the benefits of using Nomad, just like he went into the benefits of ngnix and shows how their work processes have changed with the change of their environment.

This post doesn’t give enough credit to how much information Jan has shared during his talk. Maybe roughly twenty percent of his talk are covered here. You should definitely check it out the full video to catch all the deeper more insightful topics about the infrastructure and how the applications work with each other.

Alexander Stoll
Alexander Stoll
Consultant

Alex hat seine Ausbildung zum Fachinformatiker für Systemintegration bei NETWAYS Professional Services abgeschlossen und ist nun im Consulting tätig. Vereinzelt kommt es auch vor das er an Programmierprojekten mitarbeitet. Auch privat setzt er sich sehr viel mit Informationstechnologie auseinander, aber jenseits davon ist auch viel Zeit für Fußballabende, Handwerkerprojekte und das ein oder andere Buch.

5 Steps to a DevOps Transformation by Dan Barker | OSDC 2019

This entry is part 3 of 6 in the series OSDC 2019 | Recap

YouTube player

 

“It’s not what we believe, it’s what we do that defines our culture”, was on his first slide. At the Open Source Data Center Conference (OSDC) 2019 Dan Barker presented “5 Steps to a DevOps Transformation”. Those who missed the talk back then now get the chance to see the video of Dan’s presentation and read a summary (below).

The former OSDC will be held for the first time in 2020 under the new name stackconf. With the changes in modern IT in recent years, the focus of the conference has increasingly shifted from a mainly static infrastructure approach to a broader spectrum that includes agile methods, continuous integration, container, hybrid and cloud solutions. This development is taken into account by changing the name of the conference and opening the topic area for further innovations. Transformation rules!

Due to concerns around the coronavirus (COVID-19), the decision was made to hold stackconf 2020 as an online conference. The online event will now take place from June 16 – 18, 2020. Join us, live online! Save your ticket now at stackconf.eu/ticket/


5 Steps to a DevOps Transformation

In order to be successful in the new digital economy, it is essential to continuously improve the quality, speed and efficiency of your own organization.

“In this session, we’ll walk through the five steps to transformational change that I’ve found to be important. These are really applicable to any continuously improving organization or any large amount of change in a system. Establish the vision. Create shared experiences. Educate, educate, educate. Find evangelists; Get feedback. I’ll elaborate on each item with methods I’ve used in real transformations at multiple companies. I’ll also describe how these all tie into the DevOps culture, which is really the transformation that’s occurring within the company.”

DevOps professionals primarily work in the tech and software world, creating new technology products, software, and other user services. You will play a key role in the development of new ideas for products and services and manage the process of turning these ideas into realities.

Establish the vision

“A strong team can take any crazy vision and turn it into reality” – John Carmack

The vision creates empowerment

  • But I‘m not a leader!!!
  • Bold
  • Inspiring
  • Actionable

Pathological – Power oriented

Bureaucratic – Rule oriented

Generative – Performance oriented

If your company values increased productivity, profitability, and market share then DevOps is essential. Even if your goals are non-financial, DevOps will enhance your ability to achieve those goals. The State of DevOps report soundly backs up these claims. More importantly, if your competition has already implemented DevOps and you haven’t, you are already behind. That’s how Walmart feels now that Amazon has built the world’s most efficient shopping platform.

Bad vision → bad outcomes

  • Biased for failure
  • No vision
  • IT-focused
  • Lack of clarity – JFK Moonrace
  • Not actionable

Find evangelists

“It is not about whether you call yourself a leader or not. It is about what you have to show to people as a leader. Leadership is contagious, you carry it and share it” – Israelmore Ayivor

The control mechanisms that are currently in place to manage your people and projects may not be suited for the DevOps world. You have to be willing to look at items that prevent agility, scalability, and responsiveness and change them. DevOps will provide agility, scalability, and responsiveness, so anything that hinders that process needs to be aligned with the new model.

You can‘t do it alone

  • Use anyone willing to help
  • Nurture this team
  • This team is a bellwether
  • Publicly praise team members

When your organization moves towards developing a DevOps culture, it’s signaling to everyone that participates in the production and release of software they have an equal stake in the success of the company. It’s an all for one, one for all mentality that will break down the communication barriers between teams and make everyone accountable. Once DevOps roles and responsibilities are implemented positive changes will occur, and everyone wins.

Create shared experiences

“Words are symbols for shared memories. If I use a word, then you should have some experience of what the word stands for. If not, the word means nothing to you.” – Jorge Luis BorgesIm

Bringing people together by sharing

  • Two levels
    • Leadership
    • Organization
  • Equally important

Leadership teams need landmarks

  • Shared information model
  • Reference point
  • Provides inspiration
  • Repeat

To start down your path to DevOps success you need to build a proper DevOps organization which includes all the proper team members. However, the size of your organization plays a big role on how granular you can be with your team. But size doesn’t really matter if you properly define the roles and responsibilities across the organization. The important thing is to make a commitment to the process and get started

The core responsibility that needs to exist is the person who owns the entire DevOps process. This person would usually be someone in a senior position. They are the keeper of the process and procedures and guarantor of the delivery of DevOps value. I like to think of this person as the DevOps evangelist. Aside from the leader, you would need to establish, at a minimum, the following roles: Code Release Manager, Automation Expert, Quality Assurance, Software Developer/Tester, and Security Engineer. The DevOps duties for each of these resources are described below.

Don‘t leave everyone else behind

  • Shared information model
  • Provides motivation
  • Leaders should be leading
  • How?

Educate,…

“An investment in knowledge pays the best interest” – Benjamin Franklin

Learn something new to build something new

  • Knowledge changes outcomes
  • Make it priority
  • Make it available
  • Monitor it

Measure what matters

  • Accelerate by Dr. Forsgren
  • Westrum Culture Survey
  • User Surveys
  • 1:1 Feedback
  • CultureAmp

Everyone in the company is sailing on the same ship. If the tide goes up so does the ship and everyone on it. But if the tide goes down so does the ship, but no one on the ship is to blame.

Everyone learns differently

  • Online training
  • In-person classes
  • Newsletters
  • Conferences
  • Hackathons

Get feedback

“True intuitive expertise is learned from prolonged experience with good feedback on mistakes” – Daniel Kahneman

Quellen und Nachschlagewerke

Tick Tock: What the heck is time-series data? by Tanay Pant | OSDC 2019

This entry is part 6 of 6 in the series OSDC 2019 | Recap

YouTube player

 

The rise of IoT and smart infrastructure has led to the generation of massive amounts of complex data. In his talk at the Open Source Data Center Conference (OSDC) 2019 Tanay Pant brought up a question to gather insights: Tick Tock: What the heck is time-series data? See the video of Tanay‘s presentation and read a summary (below).

The former OSDC will be held for the first time in 2020 under the new name stackconf. With the changes in modern IT in recent years, the focus of the conference has increasingly shifted from a mainly static infrastructure approach to a broader spectrum that includes agile methods, continuous integration, container, hybrid and cloud solutions. This development is taken into account by changing the name of the conference and opening the topic area for further innovations.

Due to concerns around the coronavirus (COVID-19), the decision was made to hold stackconf 2020 as an online conference. The online event will now take place from June 16 – 18, 2020. Join us, live online! Save your ticket now at: stackconf.eu/ticket/


Tick Tock: What the heck is time-series data?

Today we are going to talk about topics like what is time-series and how the load of different file forms are distributed, different use cases where time-series are used frequently. Then we’ll talk about how Create-DB helps to communicate with machine files.

What are time series?

To answer this question we present a sensor that sends the files in a period of time. When we want to read in or display this file, the time would be an axis. Compared to other workloads this file is not added to the database as an update, the time-series is added as an input and this is the primary way for this process. Time-series in database is basically introducing efficiencies through temporal treatment and this allows us to intuitively have this set of files like monitoring in different times in all aspects of our operation.

Now we have a view on time-series. If you create an abstract, look at different use cases of time-series and the way the data was generated. You can categorize them in two different ways. The first one is IT and monitoring, what can be described as a traditional use of time-series databases. When we have a look at the properties in this, one can say there are tens or hundreds of metrics or sensors as well as a lot of complex data and queries that are often larger than several gigabytes. Flux DB is a good example in this category.

We have industrial sensor data and this is an emerging sector that has not been much talked about. There are also hundreds or thousands of sensors or metrics, too. So the real-time queries are under pressure, which must be able to access all the gigabytes of data. Create-DB is a good example in this case.

We start with core technology and see what exactly Create-DB is and how it differs from other databases in this segment. Create-DB is a new type of distribution continuation database that is best suited for handling industrial sensor data, due to its ease of use and ability to handle a lot of different data, as well as a thousand different sensor data. Create-DB supports distributed SQL with full-text search and data queries, and also coordinates different nodes in a DB Cluster seamlessly with one another. In addition, the execution of write and query operations across nodes in clusters are automatically distributed. Create-DB has columnar caches for time-series in memory SQL performance so time-series normally require all data in main memory to fit, which limits the amount of data that can be managed within a specific time.

One solution for time-series performance without data volume restrictions is to implement the residence of memory in filled caches at each node, so that the caches tell the query engine whether there are any records on this node and where those records are. Distributed query processing also contributes to fast performance and a query planner that makes wise decisions about which nodes are best suited for execution. And it has machine data functions with a cloud native that makes it seamless in the cloud. Finally, we look at a few advantages of Create-DB. The Create-DB installation is simple. You can create an instance of Create-DB with a single line on the terminal or docker. It has a distributed query engine that supports full-text queries. It can handle economic hardware and instances well, and it is easy to scale the architecture.

Saeid Hassan-Abadi
Saeid Hassan-Abadi
Systems Engineer

Saeid hat im Juli 2022 seine Ausbildung als Fachinformatiker für Systemintegration bei uns abgeschloßen, und arbeitet nun in Operation-Team. Der gebürtige Perser hat in seinem Heimatland Iran Wirtschaftsindustrie-Ingenieurwesen studiert. Er arbeitet leidenschaftlich gerne am Computer und eignet sich gerne neues Wissen an. Seine Hobbys sind Musik hören, Sport treiben und mit seinen Freunden Zeit verbringen.

Fast log management for your infrastructure by Nicolas Frankel | OSDC 2019

This entry is part 4 of 6 in the series OSDC 2019 | Recap

YouTube player

 

Nicolas Frankel is a Developer Advocate with 15+ years experience consulting for many different customers, in a wide range of contexts. “Fast log management for your infrastructure” was his topic at the Open Source Data Center Conference (OSDC) 2019 in Berlin. Those who missed the talk back then now have the opportunity to see the video of Nicolas’ presentation and read a summary (below).

The former OSDC will be held for the first time in 2020 under the new name stackconf. With the changes in modern IT in recent years, the focus of the conference has increasingly shifted from a mainly static infrastructure approach to a broader spectrum that includes agile methods, continuous integration, container, hybrid and cloud solutions. This development is taken into account by changing the name of the conference and opening the topic area for further innovations.

We are proud to announce that Nicolas Frankel is in our speaker lineup this year, too. We are looking forward to his talk: “Real Continuous Deployment of JVM applications”.

Due to concerns around the coronavirus (COVID-19), the decision was made to hold stackconf 2020 as an online conference. The online event will now take place June 16 – 18, 2020. Be there, live online! Save your ticket now at: stackconf.eu/ticket/


Fast log management for your infrastructure

Fast log management for your infrastructure”, well that is one way to get OSDC visitors excited. Nicolas Frankel signed up with that one and he did not disappoint. The issues, he was tackling, were issues produced by optimization, that being said do you think about the logs when it comes to migrating your application to reactive micro services?

Before we get to all that, Nicolas had to take a little detour through programming logic and how logging works, and he also points out some misconceptions of how things are done and how they work. Like for example, his so called “[…] root of all evil”.

[bash]
LOGGER.debug(
"Cart price is now {}", cart.getPrice())
[/bash]

He states the question, who believes that in case of the log level being above debug the statements will be ignored? That’s what is to be expected, however it is not the case. In a small demo section he gives further insight on the topic from the perspective of a software developer.

From the developer point of view one should only do physical logging is the statement he ends his demo explanation run on. Directly afterwards he states that developers do not like to think that they are dealing with the physical world, then he goes further on about the respective storage possibilities like the write time regarding SSDs, HDDs or on an NFS, which should be taken into account.

Tackled some issues already, Nicolas keeps switching back and forth between the perspective of a software developer and an operator. He puts a lot of empathizes on these perspective changes to make sure that everyone involved starts to understand where the issue lies and if there is an issue at all.

For example the writing process and the opening and closing of streams for single log statements. It would be great if the stream could be continuously open and log statements can be written until the stream can be closed. But arguably and in most cases by default, logging is blocking. While most frameworks allow asynchronous logging, there is no right or wrong. And it also doesn’t have to be a software development mistake nor a bad infrastructure.

He dives deeper into asynchronous logging, because if you want to use it, you have to understand it: from queue size to discarding thresholds, the difference between blocking and dropping messages, everything. Nicolas also covers some logging basics, like metadata and what is especially important. Most essential metadata named timestamp, log level, line number and more. You may ask, why? Because some metadata is more expensive to get than others.

After some more detours through log aggregations and common pitfalls, with searching in logs or mandatory metadata, we get to a well-known application stack in the world of logging, the Elastic Stack.

He explains the basic architecture of the Elastic Stack and how the applications work with each other. Especially Filebeat and Logstash take the spotlight during this part. Step by step he works his way through an abstraction of the path a log takes from Filebeat to Logstash until you get a JSON you are familiar with. Then common misunderstandings like “Why do I need Logstash at all?” are being tackled by him, before he goes onto how he is doing logging at Exoscale.

They are using syslog-ng instead of Filebeat, basically just because when they started Filebeat was not ready for production. Then a regular Logstash and before we come to Elasticsearch there is a Kafka running. The reason why they are using Kafka is that Kafka being a decentralized data store, and using Logstash to get data out of it there is lower risk of dropping data instead of buffering towards elasticsearch, because there are not multiple nodes writing at once.

Nicolas summarizes his talk at the end with six short statements or maybe even lessons for log management. If you want, head over to the video above to learn about them from Nicolas himself or experience him live to learn from him.

Alexander Stoll
Alexander Stoll
Consultant

Alex hat seine Ausbildung zum Fachinformatiker für Systemintegration bei NETWAYS Professional Services abgeschlossen und ist nun im Consulting tätig. Vereinzelt kommt es auch vor das er an Programmierprojekten mitarbeitet. Auch privat setzt er sich sehr viel mit Informationstechnologie auseinander, aber jenseits davon ist auch viel Zeit für Fußballabende, Handwerkerprojekte und das ein oder andere Buch.