Persistent Shielding of Hardware Interrupts to Certain Processors

SUSE Linux Enterprise Server 11 Service Pack 1
SUSE Linux Enterprise Real Time 11

How can shielding hardware interrupts to certain processors be configured as persistent across reboots?

1. Make sure the irqbalance daemon is disabled.
# /etc/init.d/irq_balancer stop
# chkconfig irq_balancer off

2. Add the commands to modify the smp_affinity in the /etc/init.d/cset service script. NOTE: Read TID7007601 - "Read Making CPUSets Persistent Across Reboots" to learn about creating /etc/init.d/cset. For example, I would modify my start() section from this:

  start)
    echo -n "Starting cpuset "
    $CSET_BIN shield --userset=rtcpus --cpu="$CSET_USER" --kthread=on
    rc_status -v
    ;;
  stop)

to this:

  start)
    echo -n "Starting cpuset "

    IRQ_CPUS=1
    echo "$IRQ_CPUS"> /proc/irq/default_smp_affinity
    for irqlist in $(ls /proc/irq/*/smp_affinity)
    do
      echo "$IRQ_CPUS"> $irqlist
    done

    $CSET_BIN shield --userset=rtcpus --cpu="$CSET_USER" --kthread=on
    rc_status -v
    ;;
  stop)

3. Reboot the server
# reboot
 

Real Time Wiki - https://rt.wiki.kernel.org/index.php/CPU_shielding_using_/proc_and_/dev/cpuset
SLERTE 12 Documentation - https://documentation.suse.com/sle-rt/12-SP5/

The Real Time Wiki provides additional background about cpusets and shielding interrupts. You can follow this doc to manually to define CPUSETS but why when there is the cset command!

Using cset shield is like using a swiss army knife, it takes care of the kernel thread interrupt handlers whereas the link above shows you the manual method for creating cpusets. The cset shield command with the --kthread=on isolates the unbound interrupt threads to the system cpuset. The only kernel threads that cannot be moved are the per-CPU bound interrupt kernel threads.

The real hardware interrupts, which support IRQ affinity, can be restricted to certain CPUs by using smp_affinity which is explained in the Real Time Wiki link. The link explains how modifying smp_affinity can be done manually.

Changing the smp_affinity programmtically can be accomplished by adding a little bit of code to the /etc/init.d/cset script so the configuration can survive a reboot which is described above.

The lines that perform the interrupt shielding magic are “echo 1 > /proc/irq/default_smp_affinity” and “ echo 1 > $irqlist”.  The “1” in the line is a bitmask of allowed CPUs that can service the interrupt.  The Linux kernel documentation, /usr/src/linux-rt/Documentation/IRQ-affinity.txt, states the default bitmask is 0xffffffff.  Use the following command to verify the default bitmask of your system.

# cat /proc/irq/*/smp_affinity

So how do you calculate the bitmask?  Let's say you want to shield to first CPU on the system as the first example.

Below is an 8 CPU system where each bit represents a CPU.

 7    6    5    4    3    2    1    0    = CPU number
2^7  2^6  2^5  2^4  2^3  2^2  2^1  2^0   = binary representation of the CPUs
128  64   32   16    8    4    2    1    = decimal representation of binary digits
 0    0    0    0    0    0    0    1    = the first CPU is the allowed CPU

The interrupt shielding is configured by echoing the hex number “1” to all the real hardware interrupts in /proc/irq/*.  Not all real hardware interrupts can be shielded so an input/output error will be displayed for certain interrupts.

Let shield to processors 4 – 7 for the second example.

 7    6    5    4    3    2    1    0    = CPU number
2^7  2^6  2^5  2^4  2^3  2^2  2^1  2^0   = binary representation of the CPUs
128  64   32   16    8    4    2    1    = decimal representation of binary digits
 1    1    1    1    0    0    0    0    = CPUs 4 – 7 are the allowed CPUs

128 + 64 + 32 + 16 = 240 (decimal) or f0 (hexidecimal)

So echo “f0” to all the real hardware interrupts in /proc/irq/*.