Excessive slowness in sapxpg tool, after SUSE updates.

SUSE Linux Enterprise Server 15
SUSE Linux Enterprise Server 12

Some mutual SAP and SUSE customers have come to SUSE Support saying that after recent updates, the "sapxpg" tool takes a long time to finish simple operations such as batches of renaming, deleting, or moving files.

SUSE Support has investigated these complaints and they appear to be caused by the way sapxpg (or a parent program which calls it) is coded.  After every individual operation (such as deleting one file) the code calls "close" on nearly the entire range of possible file descriptors, one by one.  On a typical Linux system, this might not cause a noticeable impact, because often the limit on the number of file descriptors (the "nofile" value) is low, such as 1024.  However, SAP systems typically have higher "nofile" settings, at SAP's recommendation.

SAP updates it recommendations from time to time, and in SAP Note 1771258 v6, they recommended raising various nofile setting for their components to 1048576.  SUSE adopted that recommendation into our "sapconf" package in mid 2022.  The previous limit in sapconf was 65536.  After a system received the mid-2022 update to the sapconf package, the sapxpg tool now must cycle through "close" calls on 1 million+ invalid file descriptors after every operation.  This is causing delays, especially during large batches of work.

To SUSE's understanding, the root cause needs to be addressed within sapxpg code, which needs to adopt a more efficient practice.

But to offer a possible workaround:

If the launching of sapxpg (or its parent program) can be controlled, it may be possible to force it to launch with a lower limit for "nofile".  SUSE has had trouble getting timely feedback from SAP on this point, so the exact best method is unknown.  Hypothetically, if the command to be launched is "sapxpg" then possibly it can be launched as:

prlimit --nofile=65536 sapxpg

Or if a SAP utility is already up and you can identify it's PID, then you could change its nofile value on the fly with (assuming PID is 1234):

prlimit --nofile=65536 --pid 1234

You can learn the current limit of a process with:
prlimit --nofile --pid 1234

However, those examples are purely hypothetical.  These utilities come from SAP, so their precise usage and best practices may require input form SAP Support.

This TID is not necessarily expected to offer a final solution to customers.  It is mainly an initial discussion of the issue being encountered, and encouragement to pursue a final best solution with SAP.  If the ideal solution is identified, it can be included in this TID if reported to:    tidfeedback[at]suse.com