GRUB Bootloader Enter Rescue Shell on VMware guest

This document (000019654) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Server 15

Situation

VMware has a hard limit where only the first 8 disks are presented to the VM at boot. If the root or boot disk is outside the first 8 disks, grub will drop into the grub rescue prompt and stop booting. One common example is when a new LVM Physical volume is added to the root Volume Group:

The system use BIOS (not UEFI) and has more than 8 disks. LVM root Logical Volume was recently extended (adding /dev/sdi - 9 MB disk) and now system Volume Group is allocated on multiple disks:

linux-623v:~ # lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0   22G  0 disk 
├─sda1            8:1    0    8M  0 part 
└─sda2            8:2    0   22G  0 part 
  ├─system-swap 254:0    0  3.9G  0 lvm  [SWAP]
  ├─system-root 254:1    0 12.7G  0 lvm  /
  └─system-home 254:2    0  5.5G  0 lvm  /home
sdb               8:16   0    1M  0 disk 
sdc               8:32   0    2M  0 disk 
sdd               8:48   0    3M  0 disk 
sde               8:64   0    4M  0 disk 
sdf               8:80   0    5M  0 disk 
sdg               8:96   0    6M  0 disk 
sdh               8:112  0    8M  0 disk 
sdi               8:128  0    9M  0 disk 
sr0              11:0    1  663M  0 rom  
linux-623v:~ # lsscsi 
[0:0:0:0]    disk    VMware   Virtual disk     2.0   /dev/sda 
[0:0:1:0]    disk    VMware   Virtual disk     2.0   /dev/sdb 
[0:0:2:0]    disk    VMware   Virtual disk     2.0   /dev/sdc 
[0:0:3:0]    disk    VMware   Virtual disk     2.0   /dev/sdd 
[0:0:4:0]    disk    VMware   Virtual disk     2.0   /dev/sde 
[0:0:5:0]    disk    VMware   Virtual disk     2.0   /dev/sdf 
[0:0:6:0]    disk    VMware   Virtual disk     2.0   /dev/sdg 
[0:0:8:0]    disk    VMware   Virtual disk     2.0   /dev/sdh 
[0:0:9:0]    disk    VMware   Virtual disk     2.0   /dev/sdi 
[1:0:0:0]    cd/dvd  NECVMWar VMware SATA CD00 1.00  /dev/sr0 

linux-623v:~ # pvs
  PV         VG     Fmt  Attr PSize  PFree
  /dev/sda2  system lvm2 a--  21.99g    0 
  /dev/sdi1  system lvm2 a--   4.00m    0

After reboot, GRUB2 Bootloader is not able able to access LVM root disk. It print the following error and enter grub rescue mode:

Resolution

To fix this issue, all disks belongs to system Volume Group should be placed in the first 8 disks. E.g we moved the new disk ( 9 MB ) to top.

linux-623v:~ # lsblk 
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0   22G  0 disk 
├─sda1            8:1    0    8M  0 part 
└─sda2            8:2    0   22G  0 part 
  ├─system-swap 254:0    0  3.9G  0 lvm  [SWAP]
  ├─system-root 254:1    0 12.7G  0 lvm  /
  └─system-home 254:2    0  5.5G  0 lvm  /home
sdb               8:16   0    9M  0 disk 
└─sdb1            8:17   0    8M  0 part 
  └─system-root 254:1    0 12.7G  0 lvm  /
sdc               8:32   0    1M  0 disk 
sdd               8:48   0    2M  0 disk 
sde               8:64   0    3M  0 disk 
sdf               8:80   0    4M  0 disk 
sdg               8:96   0    5M  0 disk 
sdh               8:112  0    6M  0 disk 
sdi               8:128  0    8M  0 disk 
sr0              11:0    1  663M  0 rom  
linux-623v:~ # lsscsi 
[0:0:0:0]    disk    VMware   Virtual disk     2.0   /dev/sda 
[0:0:1:0]    disk    VMware   Virtual disk     2.0   /dev/sdb 
[0:0:2:0]    disk    VMware   Virtual disk     2.0   /dev/sdc 
[0:0:3:0]    disk    VMware   Virtual disk     2.0   /dev/sdd 
[0:0:4:0]    disk    VMware   Virtual disk     2.0   /dev/sde 
[0:0:5:0]    disk    VMware   Virtual disk     2.0   /dev/sdf 
[0:0:6:0]    disk    VMware   Virtual disk     2.0   /dev/sdg 
[0:0:8:0]    disk    VMware   Virtual disk     2.0   /dev/sdh 
[0:0:9:0]    disk    VMware   Virtual disk     2.0   /dev/sdi 
[1:0:0:0]    cd/dvd  NECVMWar VMware SATA CD00 1.00  /dev/sr0

Cause

GRUB2 Bootloader was not able to access all disks hence it cannot assemble the VG and not able to access it's configuration files e.g. grub.cfg

Additional Information

The system has the following setup:

linux-623v:~ # lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0   22G  0 disk 
├─sda1            8:1    0    8M  0 part 
└─sda2            8:2    0   22G  0 part 
  ├─system-swap 254:0    0  3.9G  0 lvm  [SWAP]
  ├─system-root 254:1    0 12.7G  0 lvm  /
  └─system-home 254:2    0  5.5G  0 lvm  /home
sdb               8:16   0    1M  0 disk 
sdc               8:32   0    2M  0 disk 
sdd               8:48   0    3M  0 disk 
sde               8:64   0    4M  0 disk 
sdf               8:80   0    5M  0 disk 
sdg               8:96   0    6M  0 disk 
sdh               8:112  0    8M  0 disk 
sdi               8:128  0    9M  0 disk 
sr0              11:0    1  663M  0 rom  

linux-623v:~ # lsscsi 
[0:0:0:0]    disk    VMware   Virtual disk     2.0   /dev/sda 
[0:0:1:0]    disk    VMware   Virtual disk     2.0   /dev/sdb 
[0:0:2:0]    disk    VMware   Virtual disk     2.0   /dev/sdc 
[0:0:3:0]    disk    VMware   Virtual disk     2.0   /dev/sdd 
[0:0:4:0]    disk    VMware   Virtual disk     2.0   /dev/sde 
[0:0:5:0]    disk    VMware   Virtual disk     2.0   /dev/sdf 
[0:0:6:0]    disk    VMware   Virtual disk     2.0   /dev/sdg 
[0:0:8:0]    disk    VMware   Virtual disk     2.0   /dev/sdh 
[0:0:9:0]    disk    VMware   Virtual disk     2.0   /dev/sdi 
[1:0:0:0]    cd/dvd  NECVMWar VMware SATA CD00 1.00  /dev/sr0

Let us assume all disks are in-use except /dev/sdi and we need to use it to extend /dev/system/root filesystem.

linux-623v:~ # echo -e "n\n\n\n\n\nw\n" | fdisk /dev/sdi

Welcome to fdisk (util-linux 2.33.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x68a93c8f.

Command (m for help): Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): 
Using default response p.
Partition number (1-4, default 1): First sector (2048-18431, default 2048): Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-18431, default 18431): 
Created a new partition 1 of type 'Linux' and of size 8 MiB.

Command (m for help): The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

linux-623v:~ # pvcreate /dev/sdi1
  Physical volume "/dev/sdi1" successfully created.
linux-623v:~ # vgextend system /dev/sdi1
  Volume group "system" successfully extended
linux-623v:~ # lvresize -r -L +4M /dev/system/root
  Size of logical volume system/root changed from 12.69 GiB (3248 extents) to 12.69 GiB (3249 extents).
  Logical volume system/root successfully resized.
Resize '/' of '1:13627293696'
linux-623v:~ # grub2-install /dev/sdi
Installing for i386-pc platform.
Installation finished. No error reported.
linux-623v:~ #

Rebooting the system now will cause above error. GRUB2 will enter in rescue shell.

This is a VMware limitation and not a limitation of SLES.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.