System exit to emergency shell at boot with multipath enabled

This document (000019607) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15
SUSE Linux Enterprise Server 12

Situation

The following is observed :

[  OK  ] Listening on multipathd control socket.
         Starting Device-Mapper Multipath Device Controller...
[  OK  ] Listening on Device-mapper event daemon FIFOs.
         Starting Device-mapper event daemon...
         Expecting device dev-disk-by\x2duuid-34be48b2\x2dc21...32dd9.device...
         Expecting device dev-sda2.device...
[  OK  ] Listening on udev Kernel Socket.
[  OK  ] Listening on udev Control Socket.
         Starting udev Coldplug all Devices...
         Expecting device dev-disk-by\x2duuid-1172afe0\x2d63c...5d0a7.device...
         Expecting device dev-disk-by\x2duuid-c4a3d1de\x2d4dc...ef77d.device...
[  OK  ] Started Create list of required static device nodes ...current kernel.
         Starting Create static device nodes in /dev...
[  OK  ] Started Collect Read-Ahead Data.
[  OK  ] Started Device-mapper event daemon.
[  OK  ] Started udev Coldplug all Devices.
         Starting udev Wait for Complete Device Initialization...
[  OK  ] Started Replay Read-Ahead Data.
         Starting Load Kernel Modules...
         Starting Remount Root and Kernel File Systems...
[  OK  ] Started Create static devices
[   13.682489] floppy0: no floppy controllers found
[*     ] (1 of 4) A start job is running for dev-disk-by\x2du...(7s / 1min 30s)
[*     ] (1 of 4) A start job is running for dev-disk-by\x2du...(7s / 1min 30s)

...

Timed out waiting for device dev-disk-by\x2duuid-c4a...cfef77d.device.
[DEPEND] Dependency failed for /opt.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for Postfix Mail Transport Agent.
Welcome to emergency shell
Give root password for maintenance
(or press Control-D to continue):

This is the exit to the emergency shell

This issue is logical consequence of the multipath integration in systemd and occurs if the root filesystem is not on multipath but multipath is enabled.

In this setup multipath tries to setup its paths for all devices that are not blacklisted, as the device of the root filesystem is mounted already it is inaccessible causing multipath to fail.

Resolution

A proper filter needs to be created in

/etc/multipath.conf

Issue for example :

multipath -v2

and identify what is the device for the root filesystem.

This will result in an output similar to:

Jul 29 10:10:03 | 3600508b1001030343841423043300400: ignoring map

The string between '| ' and ':' is the wwid, which should be added to /etc/multipath.conf as:

blacklist {
  wwid "3600508b1001030343841423043300400"
}

Once this is added multipath will not try to access the wrong (the non-multipathed) device for the root file system during boot.

Another way of dealing with this would be to enable multipath in the initrd; re-run 'mkinitrd' and include the 'multipath' dracut module. Then the system will always come up multipathed, which is clearly not the intention if the root device is not multipathed.

As an immediate short term workaround one could also disable multipath in the root filesystem as this :

systemctl stop multipathd
systemctl stop multipathd.socket
systemctl disable multipathd

CAVEAT: Please keep in mind that this wwid is most likely local to the specific machine.
As such :

/etc/multipath.conf

should then be removed from syncing of configuration files between different machines. Which applies for example to csync2.

Cause

Multipath is enabled, however the root filesystem is not located on a MPIO device

Additional Information

This is former TID 7015462

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.