cgroup: fork rejected by pids controller in..." found in system log

This document (7022001) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 12 SP5
SUSE Linux Enterprise Server 12 SP4
SUSE Linux Enterprise Server 12 SP3
SUSE Linux Enterprise Server 12 SP2
SUSE Linux Enterprise Server 12 SP1
SUSE Linux Enterprise Server for SAP Applications 12 SP5
SUSE Linux Enterprise Server for SAP Applications 12 SP4
SUSE Linux Enterprise Server for SAP Applications 12 SP3
SUSE Linux Enterprise Server for SAP Applications 12 SP2
SUSE Linux Enterprise Server for SAP Applications 12 SP1

Situation

Applications cannot fork processes and start to behave unpredictable. The system log contains error messages like:

... kernel: [...] cgroup: fork rejected by pids controller in...

Below are two examples:

2017-10-04T15:55:12.049219+02:00 lv8038 kernel: [  669.909917] cgroup: fork rejected by pids controller in /system.slice/sshd.service 

2017-10-04T17:10:29.352438+02:00 lv8038 kernel: [  117.147910] cgroup: fork rejected by pids controller in /user.slice/user-0.slice/session-1.scope

To resolve the issue it is important to identify the correct slice. The first example shows a problem in the system slice and the second in the user slice.

On SLES 15/SLES for SAP Applications 15 this issue should not occur. TasksMax and UserTasksMax are set to infinity.

Resolution

To choose the correct solution, it is important to verify that processes are in the intended slice.
A misconfigured PAM prohibits the transition of newly created tasks from the system slice into the user slice.

Please verify your PAM configuration first. The file /etc/pam.d/common-session or the relevant PAM configuration should have pam_systemd.so listed:

...
session optional        pam_systemd.so
...

Also an empty user slice indicates a PAM configuration issue:

 # systemd-cgls -a
 ...
 └─user.slice

If this is the case, correct the PAM configuration and reboot the system!
Except for the situation described in TID 000019129 SAP Instances failed stop on shutdown (PACEMAKER, SYSTEMD, SAP) skipping or removing pam_systemd.so is not supported!

Changing the task limit of a system slice

The log message tells, which service was effected. This example uses sshd.service (/system.slice/sshd.service).

To check the current limit, run:

# systemctl status sshd
...
    Tasks: 509 (limit: 512)
...

The current limit is 512. If there is no limit information at all, the default is used.

Please check Additional Information how to obtain default values.

To change the limit, create an override file /etc/systemd/system/sshd.service.d/override.conf that contains:

[Service]
TasksMax=MAX_TASKS|infinity

MAX_TASKS is the numeric value of allowed tasks. For no limit, use the key word infinity.
After a reload or restart of the service, the new limit should take effect.

As an alternative the value DefaultTasksMax=512 in /etc/systemd/system.conf can be altered, but this would effect every service.

Changing the task limit of the user slice

To check the current limit, run:

# systemctl status user.slice
...
    Tasks: 803 (limit: 950)
...

This is the limit for processes in all users slices. That means, all users can have not more than 950 processes together, regardless of there own limit.

To see the individual limit each user can have (here for user 0) run:

# systemctl status user-0.slice
...
    Tasks: 803 (limit: 200)
...

The log message contains information, which user slice was affected.
If there is no limit information, than the default is used.

To change the overall limit, create the file /etc/systemd/system/user.slice.d/override.conf with the following content:

[Slice] 
TasksMax=MAX_TASKS|infinity

MAX_TASKS is the numeric value of allowed tasks. For no limit, use the key word infinity. After a systemctl daemon-reload, the new limit takes effect. You can check with systemctl status user.slice.

To change the individual limit for all users, create the file /etc/systemd/logind.conf.d/override.conf with the following content:

[Login]
UserTasksMax=MAX_TASKS|infinity

MAX_TASKS is the numeric value of allowed tasks. For no limit, use the key word infinity.
After a reboot, the new limit takes effect. You can check with systemctl status user-<uid>.slice.

Cause

A process limit set by systemd has been hit and new processes cannot be forked.

Additional Information

Verify current/default settings :

Looking at the system level DefaultTasksMax settings:

systemctl show --property DefaultTasksMax

returns eg.:

DefaultTasksMax=512

Looking for a service (e.g.: sshd.service) TaskMax value:

systemctl status sshd.service |grep -e Tasks

return eg.:

Tasks: 1 (limit: 18100)

Looking for what is the current TasksMax set for a specific user:

As example:
Checking TasksMax value for user "example" with uid 1001
(this translates to user-slice : user-1001.slice)

Knowing the user-slice check the TasksMax value for the "user-1001.slice":

systemd-analyze dump | sed -n "/-> Unit user-1001.slice:/,/-> Unit /p"| grep -e "TasksMax="

returns for example:

TasksMax=18000

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

Document ID:7022001
Creation Date: 04-Oct-2017
Modified Date:30-Apr-2021
- SUSE Linux Enterprise Server
- SUSE Linux Enterprise Server for SAP Applications

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com