Filesystem fills up due to podman logging console output to ctrl.log without limits

This document (000019954) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15
SUSE Enterprise Storage 7

Situation

The /var filesystem for containers run by podman as root, or the home directory of the user of a podman run root-less container, fills up.

Resolution

Create or edit /etc/containers/containers.conf for podman running as root, or $HOME/.config/container /containers.conf for root-less containers, to contain "log_size_max=SIZE" in the [containers] section, with SIZE being the maximum size in bytes for the log files. For example:

[containers]
log_size_max=10485760

to set a maximum log size of 10 MiB. Afterwards, restart all containers to activate the change.

Cause

The log grows indefinitely due to the console log size is unlimited by default. The impact depends on how talkative the programs running in the container are. Some even have their syslog daemon configured to log to console, in that case the filesystem of the directory where ctr.log is located might fill up very quickly.

Status

Reported to Engineering

Additional Information

The file ctrl.log is podman's log of the container's console output (stdout and stderr) and is used to replay the console log with "podman logs CONTAINER_ID". The path of the file is /var/lib/containers/storage/btrfs-containers/CONTAINER_ID/userdata/ctr.log for podman started as root, or $HOME/.local/share/containers/storage/overlay-containers/CONTAINER_ID/userdata/log/ctr.log for root-less containers.

Removing ctr.log manually or by logrotate doesn't free any space, as podman keeps ctr.log open the entire time and thus only the directory entry gets removed with podman continues to log to the open file descriptor.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.