Troubleshooting boot issues (multipath with lvm)

SUSE Linux Enterprise Server 15
SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Server 11

After update or upgrade of a system using multipath with logical volumes, the system doesn't boot normally anymore. Instead, it goes on emergency mode.

In the logs, messages similar to below can be observed :

# journalctl -b 1 -xl

Jan 08 13:55:41 sles12sp2-MPIO systemd[1]: Dependency failed for Local File Systems.

-- Subject: Unit local-fs.target has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit local-fs.target has failed.
--
-- The result is dependency.

1) Insert a SLES install medium which corresponds to the current running version on the system. Then, set bios to boot primarily from the CD/iso inserted. On the menu, select “Rescue system” or “more” and “Rescue system”.

2) By default, the rescue system will activate the LVM volume group right from the boot, which is not optimal when rescuing a system with multipath configured.

To deactivate the volume group, use the following command :

# vgchange -an

Now, start the multipath service with :

# modprobe dm_multipath

For SLES 12 :
# systemctl restart multipathd

For SLES 11 :
# service multipathd restart

Confirm that the paths are visible with the command :

# multipath -ll

3) Make sure that the filter on /etc/lvm/lvm.conf is set to reach the devices over multipath by changing the "filter =" line as described below:

filter = [ "a|/dev/disk/by-id/dm-uuid-.*mpath.*|", "r/.*/" ]

An alternative filter would be:

filter = [ "a|/dev/disk/by-id/dm-name.*|", "r/.*/" ]

Important: check which device links exist on the system!

After that, on SLES12 restart lvmetad with command:

# systemctl restart lvm2-lvmetad (SLES 12 only)
# pvscan ; vgscan

If set correctly, the “pvscan” command shouldn't output "Found duplicate PV" messages anymore.

4) Mount the root LV using the /dev/mapper device and bind mount /proc, /dev, /sys and /run to :

# mount /dev/mapper/<rootvg>-<rootlv> /mnt

For SLES 12 :
# for i in dev proc sys run ; do mount -o bind /$i /mnt/$i ; done

For SLES 11 :
# for i in dev proc sys ; do mount -o bind /$i /mnt/$i ; done

Then change root to /mnt :

# chroot /mnt

5) Once in  chroot environment, make sure that the multipath is enabled again and that the paths are visible too:

For SLES12 and SLES15:
# systemctl enable --now multipathd

For SLES 11 :
# chkmod multipath on
# service multipathd start ;

# multipath -ll

6) Make sure local and multipath devices are correctly listed in /etc/fstab.
For multipath devices, use "UUID=" or "/dev/disk/by-id/dm-name-*" instead of "/dev/sd*" or "/dev/disk/by-*", which should be used only for local disks.

7) As we are still on the system activated by the chroot, change the /etc/lvm/lvm.conf and reload the lvmetad if necessary as described on step 3. Also make sure that there are  no “Found duplicate PV” entries after running the “pvscan” command. Make sure that all volumes were correctly mounted and listed with “mount” or "findmnt" command.

It is also a good idea to verify if the output of “cat /proc/mounts” match with the “cat /etc/mtab”. If there are any differences between the two, please do as below :

# cat /proc/mounts > /etc/mtab 

This can be added using the following command :

# echo 'force_drivers+=" dm_multipath dm_service_time "' >> /etc/dracut.conf.d/10-mp.conf

NOTE: which modules to force depends on the path selector usage. While dm_multipath is a must have, dm_service_time can be replaced by dm_rou# systemctl start multipathd
nd_robin or dm_queue_length, depending on the system requirements. NOTE: dracut expects a whitespace between " and the module name and will complain if it's missing!

NOTE: above will add multipath related kernel modules. However, it is better to add the dracut multipath module. This will also include the needed kernel modules.

echo 'add_dracutmodules+=" multipath "' > /etc/dracut.conf.d/10-mpio.conf

10) Next, backup the old initrd and create a new one using the following commands:

# cd /boot
# mkdir brokeninitrd
# cp initrd-<version> brokeninitrd
# mkinitrd 

 Now force creating initramfs with multipath

For SLES 12 :
# dracut --kver <kernelversion>-default -f -a multipath

For SLES 11 :
# mkinitrd -f multipath -k vmlinuz-<kernelversion>-default -i initrd-<kernelversion>-default

And also create a new grub config  file using YaST2:

# yast bootloader

In the YaST interface, at the “Bootloader Options” tab, change the value (by increasing or decreasing) on “Timeout in Seconds”.
This action will force the system to recognize the changes made and to  recreate the grub configuration. It is also a good idea to mark the “Boot from Master Boot Record” on “Boot Code Options” tab. 

• Bad entries in /etc/fstab.
• For some reason, the multipath or lvm modules were not included on the initrd image following a kernel update.
• Bad entry in /boot/grub2/device.map
• Bad entry in /etc/default/grub_installdevice
• Dependency failures on the modules included on the initrd image.

Additionally, it is also possible to manually check the init files in /boot in order to verify if there are missing dependencies or wrong entries. To do that, extract the initrd and modify it as described below :

# cd /boot
# mkdir brokeninitrd
# cp initrd-<version> brokeninitrd
# cd brokeninitrd
# xzcat /initrd-<version> |cpio -id

This will extract the initrd file, then it's structure can be verified. On SLES 12 versions, every initrd file should have a link called "init" pointing  systemd at /usr/lib/systemd/systemd

Once the necessary changes are done, re-recreate the initrd file once again with :

# find . | cpio --create -c |xz -z --format=lzma >> ../initrd-<version>