SAP applications are using ephemeral port 40403 or 40404 excessively

SUSE Linux Enterprise Server 15
SUSE Linux Enterprise Server 12

On a Linux system with SAP applications installed, when applications ask for temporary port usage, also known as ephemeral ports, certain ports (often ports 40403 and/or 40404) are being assigned far more often than any other port.

SAP installations often set a very large system wide "ip_local_port_range" for use, but then set a tremendously detailed list for "ip_local_reserved_ports" which is a list of ports to be avoided.

The practice of overlapping these two ranges is allowed, but it impacts the algorithm which selects ephemeral ports when applications request one.  The ports directly after the largest reserved range will be thousands of times more likely to be selected than other ports.

To avoid this behavior, instead of defining a very wide range for "ip_local_port_range" and defining exceptions within that with "ip_local_reserved_ports", define a smaller range for "ip_local_port_range" and then do not define any reserved ports within that range.

For example, after most SAP installations, the following ranges are the largest available without interruption:

10516 - 19199
or
21214 - 29999

Select ONE of those ranges, i.e. within /etc/sysctl.conf:

net.ipv4.ip_local_port_range = 21214 29999

And then make sure nothing within that range is specified within
net.ipv4.ip_local_reserved_ports

Then the algorithm will have nearly 9000 ports to chose from and should distribute the choices fairly evenly.  This range of ports is smaller than the default range, but in some cases a mid-sized range that is evenly used may be better than a larger range where only a few ports are likely to get selected.

NOTE:  If SAP is enforcing its port ranges somewhere other than /etc/sysctl.conf, their location may need to be tracked down and changed.

Regardless of what is present in /etc/sysctl.conf, you can check the settings actually in effect with:

sysctl -a | grep ip_local

The port selection algorithm is strongly impacted by having ip_local_port_range and ip_local_reserved_ports settings which overlap each other.  The larger a range that is included in ip_local_reserved_ports, the higher he probability that the ports immediately after that range will be selected for use.  For example, if a range of 10000 ports is reserved, the ports just after that range will be 5000 times more likely to be selected than other ports.

SUSE has also released a couple of code changes which can slightly mitigate the port selection algorithm and this behavior.  However, these changes have a very small impact.  The port range settings in the "Resolution" section above are expected to be the main, necessary solution.  The code changes (with minor impact) are found beginning in:

SLES 12 SP4 kernel-default 4.12.14-95.96
SLES 12 SP5 kernel-default 4.12.14-122.116
SLES 15 SP1 kernel-default 4.12.14-150100.197.123
SLES 15 SP2 kernel-default 5.3.18-150200.24.129
SLES 15 SP3 kernel-default  5.3.18-150300.59.63
SLES 15 SP4 present in all kernels.

Another way to identify if your SUSE kernel contains the changes is the following command, substituting your kernel package name, for example, if you have kernel-azure:

rpm -q --changelog kernel-azure | grep -A1 -B1 1180153

Which (if the changes are present) should return entries such as:

- tcp: add some entropy in __inet_hash_connect() (bsc#1180153).
- tcp: change source port randomizarion at connect() time
  (bsc#1180153).


The Linux community discusses adding a caution to the doc which discusses these settings:
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=a7a80b17c750