vsftpd and other processes terminated due to OOM scenario

This document (000020252) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15 SP3
SUSE Linux Enterprise Server 15 SP2

Situation

When multiple short lived FTP connections are established and terminated in quick succession, the vsftpd process and other processes may be terminated due to an out of memory scenario.

Following messages may be observed in dmesg and /var/log/messages:

kernel: vsftpd invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[…]
kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0-1,global_oom,task_memcg=/,task=sshd,pid=48038,uid=0
kernel: Out of memory: Killed process 48038 (sshd) total-vm:127000kB, anon-rss:0kB, file-rss:5200kB, shmem-rss:0kB

Resolution

This "regression" is the result of the 5.3 kernel being more efficient and not throttling the creation of new network namespaces. This is not a bad thing, and reintroducing throttling at kernel level would negate these improvements. SUSE's suggestion is to throttle the rate at the userspace (vsftpd) level or use netfilter rules.

Workaround:
The issue is worked around by setting 'isolate_network=NO' in /etc/vsftpd.conf

Cause

By default, the 'isolate_network' configuration directive is set to YES and results in the creation of multiple net namespaces, one per client connection.
Each namespace may create many raw sockets which are used as control sockets.
The number of raw sockets created depends on the number of possible cpus.

The out of memory scenario is triggered because the kmem cache memory for RAW and RAWv6 sockets are not getting freed as soon as the FTP connection is closed.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.