Server hangs after line "Started Plymouth switch root service" during boot process

This document (000020901) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15
SUSE Linux Enterprise Server 12

Situation

Server repeatedly hangs during the boot process. The last line printed on the boot screen is "Started Plymouth switch root service".

When accessing the root filesystem via rescue mode, some of the binaries are zero byte and trying to run some of the binaries fails.

Resolution

As there was no available backup of the server's root file system, the server was booted from the SUSE installation media and the option to 'upgrade' was selected. This allowed all programs and files related to the operating system to be re-installed. After this was done, the server was able to boot without issue. The server was then patched to the desired state and normal operation resumed.

Cause

It was determined that the issue was related to operating system binary/file corruption, as opposed to filesystem metadata corruption or format corruption.

Additional Information

Booting the server under rescue mode and repairing the root filesystem did not help with this issue. The server was also not using btrfs as the root filesystem so no roll-back option was available.

In most cases, using a recovery procedure that re-writes the operating system files, in order to overcome binary file corruption, will allow a server to resume normal function. However, there is always a risk of existing configuration files being changed, over-written or renamed, which may need manual intervention to return affected files to the desired configuration state.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.