pixel
Select Page

NETWAYS Blog

stackconf 2022 | Vitess: Running MySQL with confidence on Kubernetes

This entry is part 1 of 5 in the series stackconf 2022

How about taking a walk down memory lane and seeing which insights stackconf 2022 has brought? Let’s start this blog series with Sebstian’s talk “Vitess: Running MySQL with confidence on Kubernetes”.

 

What to Expect

Kubernetes and cloud native computing changed the way how applications and workloads are deployed and operated. Besides stateless applications there is of course still the need for more static components like MySQL databases.

In his talk Sebastian discusses the question if it’s a good idea to run MySQL databases on Kubernetes.

 

Enjoy his valuable Expert Know-How and watch the Video!

YouTube player

 

You’re convinced by our top-level speaker and want to gain even more knowledge from our open source experts? Then you should definitely join this year’s edition of the event. stackconf 2023 takes place from September 13 – 14, 2023 in Berlin. Save your Early Bird ticket until April 30 and you’re in!

In case you also want to share some know-how about several topics around the whole DevOps lifecycle, feel free to submit your proposal until May 31.

We’re looking forward to hearing from you!

Katja Kotschenreuther
Katja Kotschenreuther
Marketing Specialist

Katja ist seit Oktober 2020 Teil des Marketing Teams. Als Online Marketing Specialist kümmert sie sich neben der Optimierung unserer Websites und Social Media Kampagnen hauptsächlich um die Bewerbung unserer Konferenzen und Trainings. In ihrer Freizeit ist sie immer auf der Suche nach neuen Geocaches, bereist gern die Welt, knuddelt alle Tierkinder, die ihr über den Weg laufen und stattet ihrer niederbayrischen Heimat Passau regelmäßig Besuche ab.

stackconf 2023 | Submit your Talk now!

Submissions for stackconf 2023 are now open! The event will take place on September 13 & 14 in Berlin and for it to take place at all, we are currently looking for speakers who will shape this year’s conference program.

 

Why Speaking at stackconf

The actual question is: why not speaking at stackconf? So let’s take a look at the benefits you will enjoy as a speaker.

Meet your target audience, capture them with your expertise and get infected by their prevailing enthusiasm. Join the two-day conference inclusive its awesome evening event for free. And last but not least take the advantage of the various networking opportunities at the conference, exchange ideas with other attendees and make new friends.

 

What we are Looking for

Lectures focusing on a comprehensive technical insight are very appreciated. Let your audience benefit by learning about new topics or open source projects, technical backgrounds, how-tos, etc. Proof your expert know-how in Q&A session following after your talk.

Chose the lecture format that suites you best and customize your speaking session.

  • IGNITE TALK | Five minutes to talk accompanied by 20 slides, 15 seconds each.
  • MEDIUM TALK | The short lecture format limited to 30 minutes, including a Q&A session.
  • FULL TALK | The long lecture format of 45 minutes, including a Q&A session.

 

Call for Sponsors is Open

As a sponsor at stackconf you have the chance to present your company in a focused environment to international IT professionals & decision makers. Realize your goals and network with potential partners, grow your business with lead generation or promote your brand. Customize your presence at stackconf with our versatile and custom-fit sponsorship program.

 

Early Bird Ticket Alert

In case you want to join stackconf 2023 as an attendee you now have the opportunity to grab one of our coveted Early Bird tickets. They are on sale until April 30. So save your seat now and you will be part of the coolest DevOps event this year!

Katja Kotschenreuther
Katja Kotschenreuther
Marketing Specialist

Katja ist seit Oktober 2020 Teil des Marketing Teams. Als Online Marketing Specialist kümmert sie sich neben der Optimierung unserer Websites und Social Media Kampagnen hauptsächlich um die Bewerbung unserer Konferenzen und Trainings. In ihrer Freizeit ist sie immer auf der Suche nach neuen Geocaches, bereist gern die Welt, knuddelt alle Tierkinder, die ihr über den Weg laufen und stattet ihrer niederbayrischen Heimat Passau regelmäßig Besuche ab.

stackconf 2022 | How to Be a Good Corporate Citizen in Open Source

Dr. Dawn Foster is a unix sysadmin for VMware, did her doctorate on linux kernel development and has been following her tech career for over twenty years! Her main focus is community and open source work. In her talk she enlightened us about how to be a Good Corporate Citizen. If you would prefer to watch a recording head on over to YouTube to listen to her talk – or if you prefer to read all about it, go ahead and read on!

This is what I’ve learned from Dawn in her talk “How to Be a Good Corporate Citizen in Open Source“:

Collaboration in OSS Projects: Individuals, Companies, Communities

Intro Slide with Pictures of Dr Foster

Open source communities have a variety of different people involved.

A project has developers, a release team, localisation and translation teams, marketing, community managers, tech writers, users and lots of other people involved. All of these people are working together as one community towards the goal of a good project.

 

Balance

This community that works on the project is what makes the decisions on where the project goes, an outside corporate entity can not force them to adapt changes they don’t want – or that go against the direction of the project. As a company, you need to align your needs with the needs of the project. This is important to understand when making contributions, so you don’t put your employees in a position where they have to either do harm to the project or their employment.

Contribution Strategy and Plans

Aligning Goals

The first and most important step is to make sure that your companies and the project’s goal are in alignment. If this is the case, it will be so much easier to justify putting resources and effort into the project. It also makes it easier to make the team that works on the project understand the importance of their work.

Finding and focusing on projects is an important point. Look at your operations team and what tools they use – those might be a great fit to support. Are there development or deployment tools that are open source and you could support? These questions can help you figure out what to support, in order to make a better point to your superiors to help support those projects.

Communication

Make sure that all of your teams that work on open source projects communicate with each other, to avoid having conflicts in public open source projects. If your vision is aligned, you will have a lot fewer issues. You can even help organise meetups, provide discussion channels and events to further help foster productive discourse.

Which Projects?

Slide "Which Projects?"

After you know which projects to support, you need people to contribute. Maybe you already have people that have contributed in the past. Keep in mind that contributing to open source projects needs a different skill set than working on internal projects – they need to for example be comfortable receiving and reacting to feedback in public.

Staffing

You can also hire people that are already contributing to those projects – but you might need to be careful with that, because you do not want to get the reputation to aggressively poach contributors from projects. It requires a bit of nuance to make it known that you are hiring for a project, without coming across too strong.

Guidelines

Having guidelines and best practices ready for people to engage in open source projects. Try to find a good balance between providing help and guidance and not being too overbearing or scaring your employees away from making contributions. Help engineers understand what they want to do and why.

Measure Success

You also want to make sure you can measure outcomes and results. How do you pull that kind of data? It really depends on what you want to achieve – examples would be: for the goal “improving performance” check the softwares performance data. For the goal “gain influence” check your employees in meaningful positions in the project. You might also want to overmeasure a little bit, to have some extra data at hand, in case your focus shifts in the future.

Making Contributions as a Good Corporate Citizen in OSS

Slide "Getting Started"Before hopping into a new project, you might want to look around a little and understand how the community works and feel around a little. Look at the documentation, especially at the contribution docs and the code of conduct. All projects work differently and understanding how things work to not violate any community norms.

Start with small contributions and work your way up, instead of just working on a big addition to the project and just dumping them unannounced.

 Learn from Feedback

When you start participating in a project you need to expect feedback. Sometimes feedback will be kind, sometimes it will be worded a bit more harshly. What you need to do is stay focused on what changes you need to make on your contribution, stay kind and maybe have someone proofread what you write to catch any unwanted harshness in how you write your answers. Try not getting defensive and iterate on what you mean.

Work with the Community

You might want to connect with people that worked on similar areas that you are touching on, and collaborate. Get in touch with the people who run a project and discuss strategies with them, to offer better help and be more productive in the process!

Break up your work into smaller contributions to make it easier for the maintainers to work with you and to iterate through the process.

Remember that you have a lot less control over other people working on the project, unlike in a company where you are able to escalate issues to managers. Meet people where they are and be kind!

Relationships

Having good relationships with people you collaborate with makes it a lot more easy and fun to work together. Conferences and meetups are very important to solve issues when you can talk about something in person. Knowing the human being behind the other side of your screen can make a big difference! When you need to do something new, or have questions – having someone you know that can put you in the right direction is an incredibly valuable thing to have.

Upstream your Patches

When you maintain your patches internally, every time the project has an update there is a risk that someone will forget to apply them, or has to fix places that were touched by the upstream and the patch. If you get your patches in the upstream repository you will not run into those issues and you might help other people with them as well!

Maintenance Expectations

If you are adding larger features to a project’s codebase, make sure that you can help with its maintenance and have someone constantly assigned to that task. If you make additions to a project and then bail on it, you create a big workload for the maintainers, which will make you and your organisation look bad and future contributions to this or other projects will be a lot less well received.

Open Source Your Software

If you are open sourcing your projects, don’t just dump dead projects onto the internet and hope someone is going to take over. This is at best naive, and will also make your company look bad. Take care of your software, just the same way you would under a proprietary licence!
Maintaining a project with the community involved is a lot of work, but it pays off in the long run. Tend to your pull requests and issues and you will reap the hard work others have put into it.

If you have read through this all, you’ll be happy to hear that there is more content like this on this blog – or if you also enjoy a video about it, check out our YouTube channel with lots of recordings from our conferences!

Take a look at our conference website to learn more about stackconf, check out the archives and register for our newsletter to stay tuned!

Feu Mourek
Feu Mourek
Developer Advocate

Feu verbrachte seine Kindheit im schönen Steigerwald, bevor es sich aufmachte die Welt zu Erkunden. Seit September 2016 unterstützt es Icinga zunächst als Developer und seit 2020 als Developer Advocate, und NETWAYS als Git und GitLab Trainer. Seine Freizeit verbringt es hauptsächlich damit Video-, und Pen and Paper Rollenspiele zu spielen, sich Häuser zu designen (die es sich nie leisten können wird) oder ganz lässig mit seinem Cabrio durch die Gegend zu düsen.

stackconf 2022 | Spotify’s outage of 8.3.2022, explained

We’re still excited about stackconf 2022! Our Open Source Infrastructure Conference, which for the very first time took place in person in Berlin this year. We’ve had many awesome speakers on stage and one of their outstanding lectures I will present to you in the following.

Spotify had one of its most disruptive outages in recent history in the evening of 8.3.2022 Tue 19:00 CET, which resulted in over an hour of downtime and users getting logged out. Kat Liu, Senior Software Engineer at Spotify Berlin, explained the storm of this incident.

Kat was enjoying her day off because of International Women’s Day when she received the first alert. In a short time, she received many such alerts, and it became clear that there was a serious issue. Hundreds of people have posted online that they have been logged out and can no longer log in.

The Outage

As you can see in the screenshot above, there is a warning with the message: Failed to resolve name. The reason for this warning was that the internal system could not resolve the name of service2 because service2 was down, which caused the outage.

The Fix

The solution for this problem was very simple, just revert all services back to using the Nameless system. The outage was mostly restored by 19:40 CET.

But why were users logged out?

The screenshot above shows how service1 calls service2. Since Service2 was not available, an incorrect NOT_FOUND error was returned, causing the user to be logged out and unable to log back in.

This error was later changed to UNAVAILABLE.

The Aftermath

An outage lasting about 40 minutes resulted in about 50 million login sessions were disrupted.

Over the next few days/weeks, 3 million new duplicate accounts were created as many users were not regularly logging into Spotify and had forgotten their credentials.

That was just a short summary of Kat Liu’s talk at stackconf 2022. You can watch her full talk on our YouTube channel. Enjoy!
And don’t forget to register for the stackconf newsletter to stay tuned about the upcoming plans for next year’s stackconf! See you there!

Sukhwinder Dhillon
Sukhwinder Dhillon
Developer

Sukhwinder hat 2021 seine Ausbildung als Fachinformatiker für Anwendungsentwicklung bei NETWAYS erfolgreich abgeschlossen. In seiner Freizeit fährt er gerne Fahrrad, trifft sich mit Freunden, geht Joggen oder sitzt vorm Computer und lernt etwas Neues.

stackconf 2022 | DevOps or DevX – Lessons We Learned Shifting Left the Wrong Way

stackconf 2022 was a full success! On July 19 and 20, our conference took place in Berlin and we very much enjoyed the event, which has been on site for the very first time! stackconf was all about open source infrastructure solutions in the spectrum of continuous integration, container, hybrid and cloud technologies. We’re still excited about our expert speaker sessions. In the following you get a deeper insight into one of our talks.

We kicked off the lecture program with a talk by Hannah Foxwell on “DevOps or DevX – Lessons We Learned Shifting Left The Wrong Way”. Here is what I’ve learned from Hannah:

Once Upon A Time

DevOps is such a common term now that it has almost lost its accurate meaning. Once upon a time there were two teams, Devs and Ops, with different missions and goals – rapid development vs. stable user experience. Changes were handed over just like that and great effort was put into getting even the smallest features into production to the customer in a stable way. For sure: This needed to change!

While some people felt the problem was the Ops team. Here, NoOps was a thing. This misconception came from thinking that the Ops team didn’t care about users because the Ops team didn’t want to release the new features fast enough. As a result, more and more typical Ops tasks like backup, monitoring or cost management were outsourced to developers. At a certain point, these additional tasks became too much for the dev team, which some developers were also unhappy with.

Focus on Team Health

According to a report by Haystack Analytics, 83% of all developers suffer from burnout, mostly triggered by the demands of having to learn and consider more and more technologies and areas.
Here you have to pay more attention to HumanOps again to focus on the health of the team.

Just like the old ways of splitting everything into silos, the NoOps approach was the wrong way to go. Here, it’s important to use mixed teams with a product-owner mentality for the different layers. Each team is responsible for delivering the best possible experience for their users.

Hannah also touched on how important the proper site reliabitily is and how it can impact the team. With a 99% reliability over 28 days, you have 400 minutes, enough time for manual intervention. The larger the reliability, the less time and more stress the team has until only automatic interventions are possible to stay within the time. Here, no human can react fast enough.

On Site Realiability

But you also have to see if this is needed by the user. Many users don’t even notice a short disruption, and if they do, some aren’t even bothered by it – contrast this with the cost and effort of taking measures. Depending on the level of site reliability needed, monitoring measures range from user input to active monitoring to automatic rollbacks.

You also have to decide how to allocate this downtime at each level – the closer you are to the physical hardware, the lower the downtime needs to be.
Whereas site reliability should not be a single responsibility, this is where all teams need to work together.

Finally, Hannah explained the security aspects that need to be considered with software. Bugs like Log4Shell can be avoided with the right security mindset. An open culture is important here, where you can also discuss and criticize your own concept.

When creating the security concept, you should also consider the people who implement the measures as well as how to automate it. Some security aspects should also not be carried out by individual teams alone, but across entire teams. You can avoid a strong leftward slide towards the dev team with this and still not work in isolated silos if you have a user-centric focus with it and pay attention to the people in the process.

That was just a short summary of Hannah Foxwell’s talk at stackconf 2022. You can watch her full talk on our YouTube Channel.
I’m already looking forward to the talks at the next stackconf and the opportunity to share thoughts and experiences with a wide variety of cool people there.

Take a look at our conference website to learn more about stackconf, check out the archives and register for our newsletter to stay tuned!

Michael Kübler
Michael Kübler
Systems Engineer

Michael war jahrelang in der Gastronomie tätig, bevor er 2022 seine Umschulung als Fachinformatiker bei Netways abschloss. Seitdem unterstützt er unsere Kunden bei ihren Projekten als MyEngineer und sucht auch nebenbei kleinere Projekte, die er realisieren kann. Privat geht er gerne Campen und fährt Rad. Er genießt auch einen entspannten Abend daheim mit einem Buch und Whisky.