Improving network performance using Receive Packet Steering (RPS)

SUSE Linux Enterprise Server 11 Service Pack 3 (SLES 11 SP3)

Modern network interface devices can move enough packets that the host
can be a limiting factor to achieving maximum performance.

In recent years, CPU speed increases have slowed, but the number of
cores have been increasing. In order to keep up, the system must be
able to distribute the work across  multiple CPU cores.

Some modern network interfaces can help distribute the work to multiple
CPU cores through the implementation of multiple transmission and multiple
receive queues in hardware. However, others are only equipped with a single
queue and the driver must deal with all incoming packets in a single, serialized
stream, in which case, in order to distribute the work across multiple CPUS,
the operating system must "parallelize" the stream.

Receive Packet Steering (RPS) creates a hash from the IP addresses and port
numbers, which it then uses to determine to which CPU to enqueue the packet.

The use of the hash ensures that packets for the same stream of data are sent
to the same CPU, which helps to increase performance.

RPS is configured per device receive queue, and therefore the path
of the file configure it depends both on the interface name and receive
queue to be configured.

The configuration file can be found at:

/sys/class/net/<device>/queues/<rx-queue>/rps_cpus

where <device> is the network device, such as eth0, eth1, etc
and <rx-queue>  is the receive queue, such as rx-0, rx-1, etc

If the network interface hardware only supports a single receive queue,
only rx-0 will exist. If it supports multiple receive queues, there will be an
rx-<queue> directory for each receive queue, where <queue> is the number
of the queue.

This configuration file contains a comma-delimited list of CPU bitmaps.

By default, all bits will be 0 disabling RPS, and therefore the CPU which
handles the interrupt will also process the packet .

To enable RPS and enable specific CPUs to process packets for the
receive queue of the interface, you must set the value of their positions
in the bitmap to 1.

For example, to enable CPUs 0-3 to process packets for the first receive
queue for eth0, you would need to set bit positions 0-3 to 1 in binary, this
value is 00001111. In hex, this value is 'F'.

# cat /sys/class/net/eth0/queues/rx-0/rps_cpus
00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000

# echo f > /sys/class/net/eth0/queues/rx-0/rps_cpus

# cat /sys/class/net/eth0/queues/rx-0/rps_cpus
00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,0000000f

If you wanted to enable CPUs 8-15

1111 1111 0000 0000 (binary)
15     15     0       0       (decimal)
F        F      0       0       (hex)

The value that you would echo would be ff00.
 
On NUMA machines, best performance can be achieved by
configuring RPS to use the CPUs on the same NUMA node as the
interrupt for the interface's receive queue.

On non-NUMA machines, all CPUs can be used, and excluding the
CPU handling the network interface can boost performance if the
interrupt rate is very high.

The CPU being used for the network interface can be determined from
/proc/interrupts:

For example:

# cat /proc/interrupts
            CPU0       CPU1       CPU2       CPU3
...
  51:  113915241          0          0          0      Phys-fasteoi   eth0
...

In this case, CPU 0, is the only CPU processing interrupts for eth0, since
only CPU0 contains a non-zero value.

On x86 and AMD64/Intel64 platforms, irqbalance can be used to distribute hardware interrupts
across CPUs. Instructions on it's use is outside of the scope of this document, but can be found
in the irqbalance man page.

Receive Packet Steering can be used in virtual environments.