kABI breakage causing kernel panic with third-party drivers

This document (000021148) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15 Service Pack 4

Situation

Server suffers kernel panic similar to:

crash64> bt
PID: 11187    TASK: ffffa2675cf48000  CPU: 334  COMMAND: "openibd"
 #0 [ffffb2b841e27a20] machine_kexec at ffffffffa047fca3
 #1 [ffffb2b841e27a78] __crash_kexec at ffffffffa05820ca
 #2 [ffffb2b841e27b38] crash_kexec at ffffffffa0582fc4
 #3 [ffffb2b841e27b48] oops_end at ffffffffa043eb18
 #4 [ffffb2b841e27b68] do_trap at ffffffffa043b0fa
 #5 [ffffb2b841e27bb0] do_error_trap at ffffffffa043b1f4
 #6 [ffffb2b841e27bf0] exc_invalid_op at ffffffffa0e417f3
 #7 [ffffb2b841e27c10] asm_exc_invalid_op at ffffffffa1000dcd
    [exception RIP: __static_call_validate+41]
    RIP: ffffffffa04475c9  RSP: ffffb2b841e27cc8  RFLAGS: 00010282
    RAX: 0000000000000019  RBX: 0000000000000000  RCX: 0000000000000000
    RDX: 0000000000000000  RSI: ffffa3e1048a2940  RDI: ffffa3e1048a2940
    RBP: 0000000000000001   R8: 0000000000000000   R9: 0000000000000001
    R10: 000000000000029f  R11: ffffb2b841e27af0  R12: ffffffffc0e6bdc0
    R13: 0000000000000000  R14: ffffffffc0f2b5b8  R15: ffffffffc0e6bdc0
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #8 [ffffb2b841e27cc0] __static_call_validate at ffffffffa04475c9
 #9 [ffffb2b841e27cc8] __arch_static_call_transform at ffffffffa0447682
#10 [ffffb2b841e27cf8] ____static_call_update at ffffffffa06578e1
#11 [ffffb2b841e27d78] tracepoint_add_func at ffffffffa05c27c3
#12 [ffffb2b841e27db8] __tracepoint_probe_register_prio at ffffffffa05c295e
#13 [ffffb2b841e27e08] __ftrace_event_enable_disable at ffffffffa05eb98a
#14 [ffffb2b841e27e40] event_enable_write at ffffffffa05ec5a2
#15 [ffffb2b841e27e70] vfs_write at ffffffffa074ae59
#16 [ffffb2b841e27ea8] ksys_write at ffffffffa074b1e5
#17 [ffffb2b841e27ee8] do_syscall_64 at ffffffffa0e411d8
#18 [ffffb2b841e27f50] entry_SYSCALL_64_after_hwframe at ffffffffa1000099
    RIP: 00007f84a8ebd9d3  RSP: 00007ffe10d40f28  RFLAGS: 00000246
    RAX: ffffffffffffffda  RBX: 0000000000000002  RCX: 00007f84a8ebd9d3
    RDX: 0000000000000002  RSI: 0000557b5c5bc940  RDI: 0000000000000001
    RBP: 0000557b5c5bc940   R8: 000000000000000a   R9: 0000000000000000
    R10: 0000557b5c6a0c80  R11: 0000000000000246  R12: 00007f84a8fa1500
    R13: 0000000000000002  R14: 00007f84a8fa6c00  R15: 0000000000000002
    ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b

Resolution

The fix for this issue is available in kernel 5.14.21-150400.24.74 (Aug 03, 2023) and later:

* Wed Mar 15 2023 jslaby@suse.cz
- kABI: do not check external trampolines for signature (kabi
  bsc#1207894 bsc#1211243).
- commit b4624ad

Above text is seen in the changelog for the kernel binary.

Cause

The cause is a kABI change introduced in kernel 5.14.21-150400.24.11.1 by the following patch:

       * Fri Jul 01 2022 bp@suse.de
       - static_call,x86: Robustify trampoline patching (bsc#1199657
         CVE-2022-29900 CVE-2022-29901).
       - commit 6790036

Above text is seen in the changelog for the kernel binary.

Additional Information

Using a driver built with with the OFED version of openibd is likely to cause a kernel panic if running on any kernel from 5.14.21-150400.24.11 to 5.14.21-150400.24.69 (inclusive).

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.