Configuring and tuning process and threads limits

This document (000020429) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15 SP3

Situation

Process or thread creation fails, in some conditions this may affect also the creation of kernel threads which manifests as a workqueue lockup (dmesg message `BUG: workqueeue lockup` (beware not all workqueue lockups are caused by hitting the task limits)).

Resolution

Check what limits are set in the configuration files and adjust them accordingly.

Mechanism	Scope	Kernel default	Userspace default	Where configured
pid_max (/proc/sys/kernel/pid_max)	pid namespace (effectively global)	max (nr_cpus * 1024, 32768)	none (note: systemd > v237 propagates pids.max of the root cgroup here)	/etc/sysctl.conf /etc/sysctl.d/.conf/usr/lib/sysctl.d/.conf
threads-max (/proc/sys/kernel/threads-max)	global	memory / 128 KiB	none	/etc/sysctl.conf /etc/sysctl.d/.conf /usr/lib/sysctl.d/.conf
pids.max (cgroup attribute of PID controller)	cgroup (and hierarchical ancestors)	infinitely	infinitely (SLE-10123)	/etc/systemd/system.conf /etc/systemd/logind.conf /usr/lib/systemd/system/user-.slice.d/10-defaults.conf or particular systemd units (TasksMax= directive)
RLIMIT_NPROC (setrlimit(2) inherited by forks)	tasks of given uid (limit can be modified for chilren in forking tree)	threads-max / 2	none	/etc/security/limits.conf DefaultLimitNPROC= (systemd/system.conf) or LimitNPROC= in systemd units

The HW derived defaults take into account the system at boot, nr_cpus refers to "possible cpus" (/proc/cpuinfo lists only "online" cpus, the possible CPUs are reported on x86_64 in kernel log message `[ 0.020731] smpboot: Allowing 8 CPUs, 0 hotplug CPUs`).

The formula factors apply for SLE15 SP3.

Whatever the names of the mechanism suggests, the accounted entity is a kernel task.

It's suggested to override config files rather than to overwrite them. (https://documentation.suse.com/sles/15-SP3/html/SLES-all/cha-systemd.html#sec-boot-systemd-custom-drop-in)

Cause

The Linux kernel provides several mechanisms that limit the number of tasks that exist at given time in a particular scope. The default limit values are derived from available HW resources based on simple heuristics and these should work just fine for most workloads. (Persistent) user configuration may override these defaults, primarily dedicated to fine tuning but that may also backfire when HW capacity changes without appropriate configuration update.

The most stringent limit for the given scope applies.

Note: The accounted kernel resource is a task. Each thread of a process is a task.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.