grub2 on Power system can't find lvmid device when a 2nd disk is added to the root volume group
This document (7024181) is provided subject to the disclaimer at the end of this document.
Environment
SUSE Linux Enterprise Server 12 for POWER including all Service Packs
SUSE Linux Enterprise Server 15 for POWER including all Service Packs
Situation
After rebooting the system, grub2 can't find the root volume anymore and is stuck with an error message similar to:
error: disk 'lvmid/eBY8sJ-l5YB-nVGq-dhld-lbDa-hQxT-P8eXl9/zbpPOS-yUU1-pgTr-QLJC-5jeK-d3ts-gNwtzw' not found.
The root volume group (LVM) was consisting of one physical volume (PV) but was running out of space, thus the VG was extended with new PV - the PV is located on a disk without partitions, ie. raw disk.
Resolution
On Power systems every boot disk must have partitions and its first partition must be PReP type.
An example of two root VG PVs which has PReP partition:
An example of two root VG PVs which has PReP partition:
If a disk where root VG PV is located is a raw disk or does not have PReP partition as the first partition, new disk has to be used, partitioned, and existing LVM physical extends (PEs) have to be moved from to the correctly partitioned disk, that is, to the PV located on such disk.
GRUB2 uses PReP partitions to save boot data, thus raw disks, formatted as physical volumes (LVM), do not have a reserved space for the bootloader and boot may fail.
GRUB2 uses PReP partitions to save boot data, thus raw disks, formatted as physical volumes (LVM), do not have a reserved space for the bootloader and boot may fail.
To correct the boot issue, the correct workflow would be (for demonstration purposes let's assume the raw disk PV is /dev/mapper/360000000000000000000000000000000):
- assign another LUN to the system (e.g. /dev/mapper/360000000000000000000000000000010)
- boot from ISO
- setup multipath if required (see link in 'Additional information')
- load dm-mirror module with modprobe dm-mirror command as pvmove command needs that module and that module might not be available for the rescue boot kernel on installed system.
- chroot to the installed system (see link in 'Additional information')
- update grub2 package to latest version! (old versions of GRUB2 had issues if root VG was extended over multiple PVs); eg: zypper up grub2\*
- create a GPT partition table on the newly added disk; eg. via fdisk /dev/mapper/360000000000000000000000000000010
- create a PReP partition of about 8MB size on the newly added disk; eg. /dev/mapper/360000000000000000000000000000010-part1
- create a second partition consuming all free space on the LUN, eg: /dev/mapper/360000000000000000000000000000010-part2
- use `pvcreate' to create PV on the second partition on the newly added disk; eg: pvcreate /dev/mapper/360000000000000000000000000000010-part2
- extend root VG with newly created PV; eg: vgextend system /dev/mapper/360000000000000000000000000000010-part2
- use `pvmove' to move all physical extends (PEs) from the raw disk PV to the newly added PV; eg: pvmove /dev/mapper/360000000000000000000000000000000 /dev/mapper/360000000000000000000000000000010
- modify /etc/default/grub_installdevice to refer to first partition on all disks where root VG is located
- reinstall bootloader to first partition on all disks where root VG is located; eg: grub2-install /dev/mapper/360000000000000000000000000000000-part1 and continue with first partitions on the remaining root volume group disks
- regenerate bootloader configuration: grub2-mkconfig -o /boot/grub2/grub.cfg
Cause
NOTE: This issue is only seen on POWER, other architectures are not affected as it's caused by an issue in OpenFirmware.
Whenever there is more than one disk/LUN in the LVM volume group Openfirmware cannot find the bootloader so there must a PReP partition on each LUN and the bootloader needs to be installed (or refreshed in case of any changes) on all disks/LUNS.
Additional Information
- TID 7022520 - Troubleshooting boot issues (multipath with lvm)
- TID 7018126 - How to Chroot in Rescue Mode
Disclaimer
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.
- Document ID:7024181
- Creation Date: 15-Oct-2019
- Modified Date:29-Mar-2024
-
- SUSE Linux Enterprise Server
- SUSE Linux Enterprise Server for SAP Applications
For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com
