Addressing file system performance issues on NUMA machines

SUSE Linux Enterprise Server 11 Service Pack 1

On some NUMA systems performance issues can occur when large amounts of data are copied. This can be mainly observed with NFS. There is a low probability that it can happen with local file systems.

This can for example be addressed by tuning zone_reclaim in the kernel. zone_reclaim is a tunable that is set automatically on NUMA machines when the penalty for accessing remote memory is high. This is an optimal setting where all of the machine's  memory bank sets have the same size that are assigned to the CPUs.

A problem occurs if the machine is not tuned to fit within a NUMA partition. The server can stall as the system performs the work necessary to keep the process local to a node. These stalls can be significant if the process is also writing data. There are five recommendations as to how the problem may be mitigated.

1. Disable zone_reclaim if the workload has not been partitioned to fit within NUMA nodes with:

echo 0 >/proc/sys/vm/zone_reclaim_mode

2. Disable NUMA support in the BIOS, effectively using the interleave memory policy for all processes.
3. Use numactl to use the interleave memory policy on the processes writing data by preceding the process start command with

numactl --interleave=all <command>

4. Tune vm.dirty_background_bytes to a value based on the backing storage such that stalls due to IO are within an acceptable margin:

echo $((128*1024*1024)) >/proc/sys/vm/dirty_background_bytes
echo $((1024*1024*1024)) >/proc/sys/vm/dirty_bytes

To set one or several of these parameters permanently edit the file /etc/sysctl.conf