How to enable container log rotation with k3s or containerd

Task

In a Kubernetes cluster running an alternative container runtime, such as containerd, instead of Docker, the kubelet manages container logs. The kubelet default values in relation to log rotation can be found in the upstream kubelet | Kubernetes documentation. These values can be adjusted by adding flags to the kubelet process.

Pre-requisites

These steps have been validated for a k3s cluster using the default containerd runtime, in theory these same flags should work for any Kubernetes cluster which does not use Docker as the container runtime.

Resolution

Two kubelet flags need to be added to configure log rotation, the flags will take effect only at start time.

In the case of k3s, passing the needed flags can be done a number of ways, the most common perhaps is with the INSTALL_K3S_EXEC environment variable when installing k3s as a service. These same flags can be added to a previous install command to update the service configuration of an existing install of k3s.

Note When updating an existing k3s install, the following command will restart k3s.

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--kubelet-arg "container-log-max-files=4" --kubelet-arg "container-log-max-size=50Mi"" sh -

However, flags can also be supplied to the k3s binary directly if a service is not being used. A restart of k3s is required, using the updated flags.

k3s server --kubelet-arg container-log-max-files=4 --kubelet-arg container-log-max-size=50Mi

Note please adjust the values to suit your needs, for demonstration purposes the above commands used 4 log files of 50MB, allowing for 200MB of total space to be retained per container.

Further reading

Please reference the k3s and kubelet documentation pages to find more information on these flags.