SUSE Support

Here When You Need Us

Boot drops to emergency shell due udev running out of memory

This document (000019906) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 12 SP5 on s390x or x86_64
SUSE Linux Enterprise Server 12 SP4 on s390x or x86_64
 

Situation

When system has lots of LUNs and each LUN has multiple paths, it can lead to the event that the boot process does not complete successfully. Instead, the boot process stops and asks to drop to emergency shell. Accessing emergency shell and just exiting it will cause the boot process to continue.

Resolution

Blacklist the device driver for HBA that has the huge amount of LUNs allocated. If for example on s390x the LUNs are behind an zFCP HBA, omit the loading of zfcp driver as outlined:

Method 1

Add modprobe.blacklist=zfcp in /etc/default/grub to the GRUB_CMDLINE_LINUX_DEFAULT arguments .

Method 2

Blacklist the zFCP driver from being included in the initrd

echo 'omit_drivers+=" zfcp "' > /etc/dracut.conf.d/50-zfcp.conf
Next, add an action to load the zfcp module after system has done initial booting. For example, add the following line in /etc/initd.d/boot.local file
modprobe zfcp
Finally, recreate the intird image with mkinitrd or dracut -f command, then refresh bootloader
update-bootloader --refresh

Cause

As part for disk detection during boot process for each path udev worker is spawned. The underlying problem is that the individual udev workers will receive an OOM during booting, as the calculation on how many udev workers should be spawned is very fragile.

Background: udev does not do any memory allocation, but rather allocates everything on the stack. So an OOM can not be captured as one would normally when 'malloc' returns NULL; rather the OOM will directly kill the process with no way out. This leads to events not being processed, and a hanging system waiting for these events.
 

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000019906
  • Creation Date: 11-Mar-2021
  • Modified Date:22-Apr-2024
    • SUSE Linux Enterprise Server

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

tick icon

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

tick icon

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.

tick icon

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.