Access to CIFS mounts fail after working for a period of time

SUSE Linux Enterprise Server 15 SP3
SUSE Linux Enterprise Server 15 SP2
SUSE Linux Enterprise Server 12 SP5

SLES 12 SP5, 15 SP3, and 15 SP2 systems with CIFS mounts can use them successfully for a while, but after some period of time, access to the CIFS-mounted file system fails.   For example, in one case, various attempts to use the mount returned the error:

Bad file descriptor

In another case, access attempts appeared to hang or stall.  Tcpdump reveals that various operations are still being attempted, but looping on errors such as STATUS_FILE_CLOSED or STATUS_TOO_MANY_OPENED_FILES.

The problem can be temporarily solved by umounting and mounting again, but the behavior will eventually come back.

This problem affects SLES 12 SP5, SLES 15 SP2, and SLES 15 SP3.  It does not affect previous or later SPx versions.

A problem was identified in the caching of the root file handle of the CIFS share.  A code fix is available.  Update to the following kernel versions (or higher) depending on your SLES version:

SLES 12 SP5:  Kernel 4.12.14-122.159.1  (early May 2023).
SLES 15 SP3 LTSS:  Kernel 5.3.18-150300.59.127.1  (mid July 2023).
SLES 15 SP2 LTSS:  Kernel 5.3.18-150200.24.160.2  (late August 2023).

Where the fix is not available or cannot be installed, a workaround is to add "nohandlecache" to the CIFS mount options.

For other flavors of the kernel (such as Real Time, Azure) you can check if the kernel has the needed fix with the following command (and output shown):
 

# rpm -q --changelog <kernel-flavor-and-version> | grep 1209342
- cifs: fix open leaks in open_cached_dir() (bsc#1209342).

This issue is not expected to effect SLES 15 SP4 / SP5, which use a 5.14.21 kernel.  There, the relevant caching features have been expanded and enhanced, and the precise cause should no longer exist.  However, due to ongoing changes to cifs, if a similar issue is encountered on 15 SP4/SP5, it is suggested to update to the latest available kernel.