Listing directory contents on NFS is sometimes slow

SUSE Linux Enterprise Server 15 SP3 (LTSS)
SUSE Linux Enterprise Server 15 SP4
SUSE Linux Enterprise Server 15 SP4 (LTSS)
SUSE Linux Enterprise Server 15 SP5
SUSE Linux Enterprise Server 15 SP5 (LTSS)
SUSE Linux Enterprise Server for SAP Applications 15 SP3
SUSE Linux Enterprise Server for SAP Applications 15 SP4
SUSE Linux Enterprise Server for SAP Applications 15 SP5

When the contents of a directory residing on an NFS volume is listed, for example with "ls -l", it may happen that a repeated invocation of "ls -l" will experience increased latency. The first invocation is always fast. Any repeated invocations are fast as long as the file attributes cached by the client have not expired. Invocations of "ls -l" that occur in the time frame between the file attributes expiring and the directory contents still being cached are slow. The increase in latency becomes noticeable when the number of the files in the directory being listed exceeds thousands.

This NFS performance bug also affects the performance of any software that iterates over files in a directory and reads the attributes of those files.  
In particular, application software that carries out log rotation as part of processing requests experiences an increase in request processing latency caused by this NFS performance bug. Application software that has areas of code where application threads wait for file I/O request completion while looking up directory contents can in extreme cases suffer thread exhaustion if the aforementioned problem lasts for a longer time and/or there is higher workload.

Please update the kernel to the following versions or higher to address this issue: 
SUSE Linux Enterprise Server 15 SP3 (LTSS): 5.3.18-150300.59.195.1
SUSE Linux Enterprise Server 15 SP4 (LTSS): 5.14.21-150400.24.150.1 
SUSE Linux Enterprise Server 15 SP5 (LTSS): 5.14.21-150500.55.94.1

Workaround

Reducing the number of the files in the directory being listed mitigates this NFS performance bug.

Changes to improve throughput for reading large directories by processing more entries at a time had an inadvertent side effect of hurting performance for "ls -l" and similar work loads. The heuristic for determining whether file attribute information should be collected together with the directory listing was triggered too late, after time had already been wasted. This was fixed by triggering the heuristic earlier.

Specifically, the issue was introduced by backporting 1a34c8c9a49e ("NFS: Support larger readdir buffers") into SUSE Linux Enterprise Server 15 SP3 (kernel 5.3.18-150300.59.170.1). The issue was always present in SUSE Linux Enterprise Server 15 SP4/SP5.