Unmounting Runtime Directory Failed During Shutdown

This document (7023942) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 12

Situation

A system was upgraded from SLES 11 SP4 to SLES 12 during shutdown the following error can be seen:

dracut: Disassembling device-mapper devices , Command failed

this error can be seen on s390x but also on x86 when enabling debugging during shutdown. Detailed output:

Apr 25 16:32:38 linux-aoie systemd[1]: Unmounting Runtime Directory...
Apr 25 16:32:38 linux-aoie systemd[1]: Unmounted Lock Directory.
Apr 25 16:32:38 linux-aoie umount[2531]: umount: /var/run: target is busy
Apr 25 16:32:38 linux-aoie umount[2531]: (In some cases useful info abou
Apr 25 16:32:38 linux-aoie umount[2531]: use the device is found by lso
Apr 25 16:32:38 linux-aoie systemd[1]: var-run.mount: Mount process exited, code
Apr 25 16:32:38 linux-aoie systemd[1]: Failed unmounting Runtime Directory.

Resolution

To fix such problem we need to make sure /var/run is pointing to /run.

This can be done via two method:

1) Interactive by booting the system from SLES DVD and select "Rescue Mode"

then mounting the root disk to /mnt
then remove /mnt/var/run
next, create a link /mnt/var/run -> /mnt/run

Note in the case /var is separate volume then mount var instead of / and adjust above paths

2) Automated by using convertfs dracut module:

run the following command :

dracut --add convertfs -f

add the following two kernel command line parameters to the kernel Command Line (use 'yast bootloader'):

rd.convertfs rootflags=rw systemctl reboot

After doing above, the result should look something similar to below :

linux-aoie# ll -ld /var/run*
lrwxrwxrwx 1 root root 6 May 13 09:22 /var/run -> ../run
drwxr-xr-x 22 root root 4096 Apr 30 11:05 /var/run.runmove~

And the var-run.mount unit file should not be running:

linux-aoie# systemctl status var-run.mount
● var-run.mount - Runtime Directory
Loaded: loaded (/usr/lib/systemd/system/var-run.mount; static; vendor preset: disabled)
Active: inactive (dead

Cause

The cause of this issue is duplicated tmpfs Filesystem. The upgraded system end up with two run-time directories:

linux-aoie# mount | grep run
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /var/run type tmpfs (rw,nosuid,nodev,mode=755)

In SLES 11 SP4 /var/run is used as tmpfs ; In SLES 12 /run is used as tmpfs. /var/run is a link to /run:

linux-aoie# ll /var/run
lrwxrwxrwx 1 root root 4 May 2 2017 /var/run -> /run

But when /var/run is a directory (like with SLES 11), var-run.mount systemd unit gets activated and mounts /var/run as tmpfs

linux-aoie# systemctl status var-run.mount
● var-run.mount - Runtime Directory
Loaded: loaded (/usr/lib/systemd/system/var-run.mount; static; vendor preset: disabled)
Active: active (mounted) since Thu 2019-04-25 16:39:46 CEST; 13min ago
Where: /var/run
What: tmpfs
Process: 678 ExecMount=/usr/bin/mount /run /var/run -t bind -o bind (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 512)

This could be seen also in journalctl output:

Apr 25 16:39:46 linux-aoie systemd[1]: var-run.mount: Directory /var/run to mount over is not empty, mounting anyway.
Apr 25 16:39:46 linux-aoie systemd[1]: Mounting Runtime Directory...
Apr 25 16:39:46 linux-aoie systemd[1]: Mounted Runtime Directory.

The following lines prevent systemd from mounting /var/run as tmpfs but systemd still mounting /var/run as tmpfs because its a directory.

linux-aoie# grep Condition /usr/lib/systemd/system/var-run.mount
ConditionPathIsDirectory=/var/run
ConditionPathIsSymbolicLink=!/var/run

Additional Information

Above is a workaround for this problem. And should fix the issue. However SUSE Engineering is working on a fix that might be included in SLES12 SP5 DVD.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.