System does not boot after extending btrfs root filesystem

SUSE Linux Enterprise Server 12 Service Pack 5 - System Z (s390x)
SUSE Linux Enterprise Server 12 Service Pack 4 - System Z (s390x)

After adding a disk volume to the volume group of the root filesystem, or in case of btrfs, to the root filesystem itself, the root filesystem cannot be mounted on next IPL.
Systemd endlessly prints messages similar to:
"A start job is running for dev-disk...ab12cdef.device (60s / no limit)."

Example:
btrfs device add /dev/dasdb1 /

If you do this and have not rebooted, see TID 000019088 - System does not boot anymore after extending root filesystem for list of commands to run BEFORE reboot.

If you have rebooted and the system will not boot the following are steps to recover the SLES VMs running on s390, IBM Z.

In short, the recovery process, is to run "grub2-install" but from a rescue boot, not from mounting on another VM.
Then you have to copy the new initrd from /boot/zipl/ to /boot/

There are variables that will need to be adjusted for each environment.
It is assumed you know how to use the IBM tools to adjust parmfiles, minidisk and IPL the system.

We assume you know how to boot the VM so it can start an FTP or HTTP install,
we'll add the "rescue" flag to the boot parameters instead of completing a normal install.

Here is an example parmfile for that:

InstNetDev=osa OsaInterface=qdio Layer2=1 PortNo=0 OSAHWAddr=
HostIP=10.1.1.10/20 Hostname=sles12sp5.domain.com
Gateway=10.1.1.1 Nameserver=10.1.1.1 Domain=domain.com
ReadChannel=0.0.0600 WriteChannel=0.0.0601 DataChannel=0.0.0602
install=ftp://ftp-install.server.com/install/SLP/SLE-12-SP5-Server-GM/s390x/DVD1/
ssh=1 sshpassword=sshpasswd rescue=1

You will need the SLES OS s390 ISO.
The SLES12-SP5 s390 ISO is:
SLE-12-SP5-Server-DVD-s390x-GM-DVD1.iso

You will need to put these 4 files from the SLES12-SP5 s390 ISO on the "a" disk.
The files on the ISO are:
boot/s390x/linux
boot/s390x/initrd
boot/s390x/sles12.exec
boot/s390x/parmfile

(With the parmfile edited to what is posted above).
The parmfile will need to be changed and customized for your environment.

The files are punched to the virtual card reader and then IPL'd
(booted) from the reader.

to start the VM run:
sles12 exec

Once you're booted into rescue mode we want to name
the devices what they are actually named in the running OS.
Typically that would be 0100 and 0101.

chzdev -e 0.0.0100
chzdev -e 0.0.0101

lsdasd (to confirm)
btrfs filesystem show / (to confirm both disks)
mount /dev/dasda2 /mnt/
mount /dev/dasda1 /mnt/boot/zipl
mount --rbind /dev /mnt/dev
mount --rbind /proc /mnt/proc
mount --rbind /sys /mnt/sys
chroot /mnt/

Do not run "mkinitrd" or any of the other commands.

Only run:

grub2-install

Then we need to copy the NEW initrd that was created in /boot/zipl/ to /boot/

Using "ls", you can check the dates, the initrd in /boot/zipl should be the current date:
(Depending on kernel version)

ls -la /boot/zipl/initrd-4.12.14-122.12-default

This one should be old:

ls -la /boot/initrd-4.12.14-122.12-default

Then:

cp /boot/zipl/initrd-4.12.14-122.12-default /boot/initrd-4.12.14-122.12-default

(Change the kernel numbers to match the version you are running.)

Exit chroot and reboot:

exit
umount /mnt/sys
umount /mnt/proc
umount /mnt/dev
umount /mnt/boot/zip/
umount /mnt/
reboot

Then the VM should boot normally.

Grub2 and the initrd will only set disk devices online that are needed to mount the root filesystem to avoid long waiting times when a large amount of disks are visible to the system. Thus, both need to be updated to contain the information that an additional disk is required. This affects only filesystems mandatory to start the installed system, for filesystems that get mounted afterwards this is not necessary.
 

Related to TID 000019088 .