Cgroup "Misc controller" introduce instability in Kubelet

This document (000021270) is provided subject to the disclaimer at the end of this document.

Environment

Kubernetes version build with runc library < 1.1.6
A node running Cgroup V1

You can check the Runc library version of your Kubernetes version in the go.mod file in the Kubernetes repo.
Eg: See the below example for Kubernetes 1.24.6 using runc 1.1.1
https://github.com/kubernetes/kubernetes/blob/v1.24.6/go.mod#L66

Situation

The Kubelet on the node report the following error message

kubelet_getters.go:300] "Path does not exist" path="/var/lib/kubelet/pods/<ID_NUMBER>/volumes"

These lines fill up the kubelet log ending up in gigabyte of logs.
This can also cause high CPU usage.

Resolution

To fix the issue you need to either :

Update the node to use Cgroup V2

Upgrade to a Kubernetes version using runc libraries >= 1.1.6

It is recommended to upgrade your Kubernetes version.
Doing a change at the Kernel level may introduce unkowns to the stability of your system.

Cause

The issue comes from the integration of the so called Misc controller in Kernel 5.13.
Problem comes from a discrepancy between the code creating the "Misc" Cgroup and the code cleaning the Cgroup that doesn't handle the "Misc" Cgroup, leaving it behind.

runc integrated a fix in 1.1.6, HOWEVER
kubelet depends on runc's cgroup libraries
In order to clean up pods using the new "Misc" controller, runc cgroup library need to be updated to be aware of it.
So even if your system run 1.1.6 but kubelet is not build with these library, the problem occurs.

Ex:
Kubernetes 1.24.8 uses runc library 1.1.1
https://github.com/kubernetes/kubernetes/blob/v1.24.8/go.mod#L66

The first kubernetes release with runc library 1.1.6 in k8s 1.24.x is K8s 1.24.14
https://github.com/kubernetes/kubernetes/blob/v1.24.14/go.mod#L59
with the help of this PR
https://github.com/kubernetes/kubernetes/pull/117892

The following comment list all PR for all impacted Kubernetes minor releases
https://github.com/opencontainers/runc/issues/3849#issuecomment-1544519250

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.