error: fork: Resource temporarily unavailable

SUSE Linux Enterprise Server for SAP Applications 15
SUSE Linux Enterprise Server for SAP Applications 12
SUSE Linux Enterprise High Availability Extension 15
SUSE Linux Enterprise High Availability Extension 12

System hangs because it cannot spawn new child processes.
Errors found in /var/log/messages.

fork failed: Resource temporarily unavailable (errno = 11)
fatal: fork of unprivileged child failed
error: fork: Resource temporarily unavailable

If crash dump was captured, the crash logs show the following where the overall number of processes is close to pid_max configured.

crash> ps -S
  RU: 14
  IN: 65269
  WA: 1
  ZO: 3
crash>

The sar log files show that the system is close to maximum of allowed PIDs, 65536.

00:40:01      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
...
02:50:01            0     65279      0.57      1.05      1.40         0
03:10:01            0     65279      0.89      1.48      1.51         0
...

pid_max configured as below.

# cat /proc/sys/kernel/pid_max
65536

Increase the kernel variable pid_max to 131072, and monitor the system behavior.

For a temporary change, do the following:

sysctl -w kernel.pid_max=131072

For a permanent change, kernel.pid_max=131072 entry must be inserted in /etc/sysctl.conf file.

* The pid_max value depends much on the system load, thus the value must be set accordingly.

System hangs because the maximum number of processes has been reached and is not able spawn new child processes.

The maximum value allowed on a 64 bit system is approximately 4 million.
More info regarding pid_max could be found on 'man 5 proc'
/proc/sys/kernel/pid_max (since Linux 2.5.34)

This file specifies the value at which PIDs wrap around (i.e., the value in this file is one greater than the maximum PID). PIDs greater than this value are not allocated; thus, the value in this file also acts  as  a system-wide  limit  on  the  total number of processes and threads.  The default value for this file, 32768, results in the same range of PIDs as on earlier kernels.  On 32-bit platforms, 32768 is the maximum value for pid_max.  On 64-bit systems, pid_max can be set to any value up to 2^22 (PID_MAX_LIMIT, approximately 4 million).

Possible reasons for the errors include:
1. There is a misbehaving service or process running, consuming more resources than expected.
2. The system was not able to create new processes, because of the limits set for proc in /etc/security/limits.conf.
3. The system ran out of memory and new processes were unable to start because they could not allocate memory.
4. There is not an available ID to assign to the new process. A unique value less than kernel.pid_max must be available.