We’d finally like to announce the release of Kubernetes v1.24 and v1.25 on our Kubernetes Platform. Since 1.24 brought many under the hood changes, our deployment process had to be refactored as well. While Version 1.24 and 1.25 were available on our platform for some time now, we can now safely say that both versions are completely stable and safe to use. For the various changes coming with the new version we recommend creating a test cluster to test your applications. But what are these big changes? Let’s go through some of the highlights.
Deprecation of the docker shim
Easily the biggest change is the deprecation of the docker shim that broke our current deployment method where every component and workload runs in Docker containers. But with this release the docker integration is no more. Kubernetes now officially only supports Container Runtimes implementing the CRI specification. Container Runtimes that implement said spec would be Containerd, CRI-O, Kata and gVisor for example.
Since Containerd is the most common runtime, we settled for it. Even though docker internally uses Containerd, the migration was pretty tricky, as it involved installing and configuring Containerd, as well as making sure that Kubernetes uses it. Finally when kubelet restarts, all containers will be recreated.
Containerd
We initially thought Containerd would behave exactly like docker since Containerd runs the docker containers after all. But for one reason or another that is not the case. Through the CRI interface the containers get created based on a CRI-spec. Unfortunately, the default configuration of Containerd does not set the ulimit properly, which results in some application working and others will be killed by the OOMKiller. It turns out that some application try to check the ulimit by „trial and error“ and since the limit is set too high, the kernel will eventually kill the respective process. Weirdly enough, almost exclusively older applications were effected. For example mysql:8 would work fine, but mysql:5.7 will crash almost instantly. The same problem can be observed with the nfs-server-provisioner and rabbitmq for example.
API Deprecations
Like with any other Kubernetes release, there were a lot of API deprecations that needed us to move the Flannel CNI and other deployments to a new release gracefully. The official Kubernetes Blog has a great write-up on this topic.
Configuration
Another problem we faced is the change in configuration. The core Kubernetes component kubelet now only supports being configured with a special configuration file, which in turn meant that we had to rebuild the configuration from scratch, as all of our configuration involved command line flags that now no longer work. CoreDNS as well had some new configuration options helping to conquer overloading the pod with many concurrent DNS queries. We even support adding new static host entries in CoreDNS. The ConfigMap coredns-extra-hosts sets the entries. This entry hosts.list is empty by default, but can the modified like any other hosts file ( 1.2.3.4 example.com ). After restarting the coredns deployment the host can be queried.
Deprecation of PodSecurityPolicy
PodSecurityPolicies have been deprecated since 1.21. But since 1.24 is the last release with it still active, it’s the last chance to get started with PodSecurityAdmission. However, they don’t provide the same feature set, as it enforces the policy based on 3 Pod Security Standards namespace wide. This means, in order to get the same and even more features solutions like OPA Gatekeeper or Kyverno have to be implemented.
ServiceAccount Tokens
Another noteworthy change is the changed behaviour in token creation. Until now, every ServiceAccount automatically gets a new secret including a token to access the Kubernetes api. This is no longer the case. If you need to create a token, make sure to use the kubectl create token command instead.
Happy upgrading!
