SUSE Support

Here When You Need Us

Persistent Shielding of Hardware Interrupts to Certain Processors

This document (7007602) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 11 Service Pack 1
SUSE Linux Enterprise Real Time 11

Situation

How can shielding hardware interrupts to certain processors be configured as persistent across reboots?

Resolution

1. Make sure the irqbalance daemon is disabled.
# /etc/init.d/irq_balancer stop
# chkconfig irq_balancer off

2. Add the commands to modify the smp_affinity in the /etc/init.d/cset service script. NOTE: Read TID7007601 - "Read Making CPUSets Persistent Across Reboots" to learn about creating /etc/init.d/cset. For example, I would modify my start() section from this:

  start)
    echo -n "Starting cpuset "
    $CSET_BIN shield --userset=rtcpus --cpu="$CSET_USER" --kthread=on
    rc_status -v
    ;;
  stop)

to this:

  start)
    echo -n "Starting cpuset "

    IRQ_CPUS=1
    echo "$IRQ_CPUS"> /proc/irq/default_smp_affinity
    for irqlist in $(ls /proc/irq/*/smp_affinity)
    do
      echo "$IRQ_CPUS"> $irqlist
    done

    $CSET_BIN shield --userset=rtcpus --cpu="$CSET_USER" --kthread=on
    rc_status -v
    ;;
  stop)

3. Reboot the server
# reboot
 

Additional Information

Real Time Wiki - https://rt.wiki.kernel.org/index.php/CPU_shielding_using_/proc_and_/dev/cpuset
SLERTE 12 Documentation - https://documentation.suse.com/sle-rt/12-SP5/

The Real Time Wiki provides additional background about cpusets and shielding interrupts. You can follow this doc to manually to define CPUSETS but why when there is the cset command!

Using cset shield is like using a swiss army knife, it takes care of the kernel thread interrupt handlers whereas the link above shows you the manual method for creating cpusets. The cset shield command with the --kthread=on isolates the unbound interrupt threads to the system cpuset. The only kernel threads that cannot be moved are the per-CPU bound interrupt kernel threads.

The real hardware interrupts, which support IRQ affinity, can be restricted to certain CPUs by using smp_affinity which is explained in the Real Time Wiki link. The link explains how modifying smp_affinity can be done manually.

Changing the smp_affinity programmtically can be accomplished by adding a little bit of code to the /etc/init.d/cset script so the configuration can survive a reboot which is described above.

The lines that perform the interrupt shielding magic are “echo 1 > /proc/irq/default_smp_affinity” and “ echo 1 > $irqlist”.  The “1” in the line is a bitmask of allowed CPUs that can service the interrupt.  The Linux kernel documentation, /usr/src/linux-rt/Documentation/IRQ-affinity.txt, states the default bitmask is 0xffffffff.  Use the following command to verify the default bitmask of your system.

# cat /proc/irq/*/smp_affinity

So how do you calculate the bitmask?  Let's say you want to shield to first CPU on the system as the first example.

Below is an 8 CPU system where each bit represents a CPU.

 7    6    5    4    3    2    1    0    = CPU number
2^7  2^6  2^5  2^4  2^3  2^2  2^1  2^0   = binary representation of the CPUs
128  64   32   16    8    4    2    1    = decimal representation of binary digits
 0    0    0    0    0    0    0    1    = the first CPU is the allowed CPU


The interrupt shielding is configured by echoing the hex number “1” to all the real hardware interrupts in /proc/irq/*.  Not all real hardware interrupts can be shielded so an input/output error will be displayed for certain interrupts.

Let shield to processors 4 – 7 for the second example.

 7    6    5    4    3    2    1    0    = CPU number
2^7  2^6  2^5  2^4  2^3  2^2  2^1  2^0   = binary representation of the CPUs
128  64   32   16    8    4    2    1    = decimal representation of binary digits
 1    1    1    1    0    0    0    0    = CPUs 4 – 7 are the allowed CPUs


128 + 64 + 32 + 16 = 240 (decimal) or f0 (hexidecimal)

So echo “f0” to all the real hardware interrupts in /proc/irq/*.

 

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:7007602
  • Creation Date: 19-Jan-2011
  • Modified Date:23-Mar-2021
    • SUSE Linux Enterprise Real Time
    • SUSE Linux Enterprise Server

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

tick icon

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

tick icon

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.

tick icon

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.