SUSE Linux Enterprise Server 15
SUSE Linux Enterprise Desktop 15
SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Desktop 12
A system or an application is hung and there is no obvious way how to debug the problem. Perhaps, kdump is not enabled or the system got stuck before the kdump service had been started. This document lists simple debugging steps that can be taken for most systems and that provide valuable debugging information with little effort.
Prerequisite
It is often necessary to have a serial console operational and to be able to store its output into a file. Text format is preferred over images. In case image is the only way, please use an OCR software to convert its content to text if possible. A serial console is only required if the kernel log messages do not got stored to disk, for example due to rsyslogd not being operational, or journald just logging to memory but not to disk.
System Request Keys
There is a simple way to gather some data from the system without resetting or crashing it for kdump: system request (SysRq) keys. SysRq keys are predefined (hard-coded in the kernel) key combinations that trigger various actions.
Note: The kernel configs "CONFIG_MAGIC_SYSRQ=y", "CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1" are set in kernels built by SUSE under General Support[1]. In case you build your own, please refer to the official[2] documentation.
How to use the magic SysRq key
The action triggered depends on the command key used in the SysRq key combination. The command keys most useful for debugging are.
- "t" prints a stack trace for every process on the system into the kernel log. The output allows one to see what all processes were doing at that moment. On a busy, fully booted system, the output may be tens of thousands of lines long.
- "l" prints the stack traces of all processes that are currently running on active CPUs into the kernel log.
- "w" ("z" on AZERTY keyboards) prints the stack traces of all processes that are blocked in uninterruptible sleep into the kernel log. This command key is used for debugging I/O issues. The output should be much shorter than that of the "t" command key because not all processes are printed out.
- "m" ("," on AZERTY) prints current memory information into the kernel log. Useful if a memory-related issue is suspected.
- "c" - will crash the system. A kernel core dump will be stored if kdump is enabled, see [3].
When getting debugging data, it is a good idea to send the command keys repeatedly (with the exception of "c") with at least several seconds in between the commands keys. In this way, the state of the system at different points in time is captured.
There are also command keys for rebooting a machine with as minimal an impact as possible:
- "r" - turns off keyboard raw mode and sets it to XLATE.
- "s" - will attempt to sync all mounted filesystems. This lessens the chance of data loss.
- "e" - sends a SIGTERM to all processes, except for init.
- "i" - sends a SIGKILL to all processes, except for init.
- "u" - will attempt to remount all mounted filesystems read-only.
- "b" - will reboot your system immediately (without syncing or unmounting your disks).
There is a mnemonic for remembering the order of the above command keys:
- Raising Skinny Elephants Is Utterly Boring.
How to trigger SysRq on different systems
Desktop machine (x86 architecture):
If a PS2 or a USB keyboard is connected to the machine, a SysRq key combination is sent to the kernel by pressing the Alt key together with the Print Screen/SysRq key together with a command key, e.g. Alt-SysRq-m for memory information.
Server with SSH:
Log into the machine with ssh. To send a SysRq key to the kernel, it is enough to write the command key it into /proc/sysrq-trigger as root. E.g.
# echo m > /proc/sysrq-trigger
# echo w > /proc/sysrq-trigger
# echo l > /proc/sysrq-trigger
Server with serial console only:
It is often necessary to resort to using the serial console when a system starts misbehaving. The /proc/sysrq-trigger file could be used for sending SysRq keys in case it is still possible to log into the system. Otherwise, Sysrq keys can also be sent over a serial line by sending a break followed by a command key within 5 seconds.
Note: The type of terminal you have will define how to 'send break', e.g.: in ipmitool the break character is "~B" (tilde followed by a capital B).
Azure:
On Azure, SysRq keys can be sent from the GUI interface of the serial console of a virtual machine. To get to the console, select "Support + troubleshooting/Serial console" in the menu of the machine on the Azure portal. The top bar of the console holds a tool for sending SysRq keys.
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.
