Systemd ordering dependency issues with local fs mountpoints on top of subdirectories of network mount points

This document (000021285) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15 SP4

Situation

The system has no network connection after a patch activity. The wicked.service fails to start after a system reboot. The service is found in the following state:

sles15-sp4:~ # systemctl status wicked.service 
○ wicked.service - wicked managed network interfaces
     Loaded: loaded (/usr/lib/systemd/system/wicked.service; enabled; vendor preset: disabled)
     Active: inactive (dead)

Nov 30 15:02:23 sles15-sp4 systemd[1]: mnt-fonts\x2dstaging-shared-dir-media.mount: Found dependency on mnt-fonts\x2dstaging-shared-dir-media-inquiry.mount/start
Nov 30 15:02:23 sles15-sp4 systemd[1]: mnt-fonts\x2dstaging-shared-dir-media.mount: Found dependency on mnt-fonts\x2dstaging-shared-dir-media.mount/start
Nov 30 15:02:23 sles15-sp4 systemd[1]: mnt-fonts\x2dstaging-shared-dir-media.mount: Job network-online.target/start deleted to break ordering cycle starting with mnt-fonts\x2dstaging-shared-dir-media.mount/start
Nov 30 15:02:24 sles15-sp4 systemd[1]: mnt-fonts\x2dstaging-shared-dir-media.mount: Found ordering cycle on network-online.target/start
Nov 30 15:02:24 sles15-sp4 systemd[1]: mnt-fonts\x2dstaging-shared-dir-media.mount: Found dependency on wicked.service/start
Nov 30 15:02:24 sles15-sp4 systemd[1]: mnt-fonts\x2dstaging-shared-dir-media.mount: Found dependency on wickedd-dhcp4.service/start
Nov 30 15:02:24 sles15-sp4 systemd[1]: mnt-fonts\x2dstaging-shared-dir-media.mount: Found dependency on local-fs.target/start
Nov 30 15:02:24 sles15-sp4 systemd[1]: mnt-fonts\x2dstaging-shared-dir-media.mount: Found dependency on mnt-fonts\x2dstaging-shared-dir-media-inquiry.mount/start
Nov 30 15:02:24 sles15-sp4 systemd[1]: mnt-fonts\x2dstaging-shared-dir-media.mount: Found dependency on mnt-fonts\x2dstaging-shared-dir-media.mount/start
Nov 30 15:02:24 sles15-sp4 systemd[1]: mnt-fonts\x2dstaging-shared-dir-media.mount: Job network-online.target/start deleted to break ordering cycle starting with mnt-fonts\x2dstaging-shared-dir-media.mount/start

Similarly, the below output could also be found:

sles15-sp4:~ # systemctl status network.service
— wicked.service - wicked managed network interfaces
     Loaded: loaded (/usr/lib/systemd/system/wicked.service; enabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Wed 2023-11-08 10:12:39 CET; 5min ago
    Process: 1198 ExecStart=/usr/sbin/wicked --systemd ifup all (code=exited, status=1/FAILURE)
   Main PID: 1198 (code=exited, status=1/FAILURE)

Nov 08 10:12:39 sles15-sp4 systemd[1]: Starting wicked managed network interfaces...
Nov 08 10:12:39 sles15-sp4 wicked[1198]: /org/opensuse/Network/Interface.getManagedObjects failed. Server responds:
Nov 08 10:12:39 sles15-sp4 wicked[1198]: org.freedesktop.DBus.Error.ServiceUnknown: The name org.opensuse.Network was not provided by any .service files
Nov 08 10:12:39 sles15-sp4 wicked[1198]: Couldn't refresh list of active network interfaces
Nov 08 10:12:39 sles15-sp4 systemd[1]: wicked.service: Main process exited, code=exited, status=1/FAILURE
Nov 08 10:12:39 sles15-sp4 systemd[1]: wicked.service: Failed with result 'exit-code'.
Nov 08 10:12:39 sles15-sp4 systemd[1]: Failed to start wicked managed network interfaces.

Resolution

Check the /etc/fstab for local file systems mount on top of a network mounts. For example:

nfs-server:/opt/data/media/dir1 /mnt/fonts-staging/shared/dir/media     nfs     ro,auto,soft    0 0
/dev/vdb1 /mnt/fonts-staging/shared/dir/media/inquiry                   ext3    ro,auto,user    0 2

Add the _netdev option to the local mount, as follows:

nfs-server:/opt/data/media/dir1 /mnt/fonts-staging/shared/dir/media     nfs     ro,auto,soft       0 0
/dev/vdb1 /mnt/fonts-staging/shared/dir/media/inquiry                   ext3    ro,auto,user,_netdev    0 2

The fixed systemctl command results after applying the fix is similar to the following:

sles15-sp4:~ # systemctl status wicked.service 
● wicked.service - wicked managed network interfaces
     Loaded: loaded (/usr/lib/systemd/system/wicked.service; enabled; vendor preset: disabled)
     Active: active (exited) since Thu 2023-11-30 15:08:33 CET; 3h 24min ago
   Main PID: 758 (code=exited, status=0/SUCCESS)

Nov 30 15:08:16 sles15-sp4 systemd[1]: Starting wicked managed network interfaces...
Nov 30 15:08:33 sles15-sp4 wicked[758]: lo              up
Nov 30 15:08:33 sles15-sp4 wicked[758]: eth0            up
Nov 30 15:08:33 sles15-sp4 systemd[1]: Finished wicked managed network interfaces.

Cause

As soon as a "local" mount depends on a "network" mount, the local mount becomes a "network" one and, as such, should be marked with the _netdev option. This is mostly caused by dependency cycles between network and non-network mounts, which are difficult for systemd to determine automatically.

Status

Reported to Engineering

Additional Information

The system logs would report the "ordering cycle" messages within the system logs (/var/log/messages, for example), similar to the following:

Nov 28 15:49:10 sles15-sp4 systemd[1]: wickedd.service: Found ordering cycle on wickedd-dhcp6.service/start
Nov 28 15:49:10 sles15-sp4 systemd[1]: wickedd.service: Job wickedd-dhcp6.service/start deleted to break ordering cycle starting with wickedd.service/start
Nov 28 15:49:10 sles15-sp4 systemd[1]: sysinit.target: Found ordering cycle on grub2-once.service/start
Nov 28 15:49:10 sles15-sp4 systemd[1]: sysinit.target: Job grub2-once.service/start deleted to break ordering cycle starting with sysinit.target/start
Nov 28 15:49:10 sles15-sp4 systemd[1]: firewalld.service: Found ordering cycle on basic.target/start
Nov 28 15:49:10 sles15-sp4 systemd[1]: firewalld.service: Job paths.target/start deleted to break ordering cycle starting with firewalld.service/start
Nov 28 15:49:10 sles15-sp4 systemd[1]: sysinit.target: Found ordering cycle on auditd.service/start
Nov 28 15:49:10 sles15-sp4 systemd[1]: sysinit.target: Job auditd.service/start deleted to break ordering cycle starting with sysinit.target/start
Nov 28 15:49:10 sles15-sp4 systemd[1]: wickedd.service: Found ordering cycle on wickedd-dhcp4.service/start
Nov 28 15:49:10 sles15-sp4 systemd[1]: wickedd.service: Job wickedd-dhcp4.service/start deleted to break ordering cycle starting with wickedd.service/start
Nov 28 15:49:11 sles15-sp4 systemd[1]: wickedd-dhcp6.service: Found ordering cycle on local-fs.target/start
Nov 28 15:49:11 sles15-sp4 systemd[1]: wickedd-dhcp6.service: Job local-fs.target/start deleted to break ordering cycle starting with wickedd-dhcp6.service/start
Nov 28 15:49:11 sles15-sp4 systemd[1]: wickedd.service: Found ordering cycle on wickedd-dhcp6.service/start
Nov 28 15:49:11 sles15-sp4 systemd[1]: wickedd.service: Job wickedd-dhcp6.service/start deleted to break ordering cycle starting with wickedd.service/start
Nov 28 15:49:11 sles15-sp4 systemd[1]: grub2-once.service: Found ordering cycle on local-fs.target/start
Nov 28 15:49:11 sles15-sp4 systemd[1]: grub2-once.service: Job local-fs.target/start deleted to break ordering cycle starting with grub2-once.service/start

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

Document ID:000021285
Creation Date: 28-Nov-2023
Modified Date:10-Jan-2024
- SUSE Linux Enterprise Server
- SUSE Linux Enterprise Server for SAP Applications

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com