Troubleshooting systemd jobs that are hung or stuck

SUSE Linux Enterprise Server 15
SUSE Linux Enterprise Server 12
SUSE Linux Enterprise High Availability Extension 15
SUSE Linux Enterprise High Availability Extension 12
SUSE Linux Enterprise Server for SAP Applications 15
SUSE Linux Enterprise Server for SAP Applications 12

The purpose of this TID is to give a troubleshooting example of a systemd service that appears stuck or hung.

The server is taking longer than usual to boot. Some system services are not starting as expected. The systemctl list-jobs command shows running and waiting jobs. For example:

> sudo /bin/systemctl --no-pager list-jobs
JOB UNIT                                 TYPE  STATE  
208 getty@tty1.service                   start waiting
216 systemd-user-sessions.service        start waiting
213 hostinfo.service                     start running
207 getty.target                         start waiting
117 multi-user.target                    start waiting
219 systemd-update-utmp-runlevel.service start waiting
255 after-local.service                  start waiting
235 plymouth-quit-wait.service           start waiting
253 issue-generator.service              start waiting
240 plymouth-quit.service                start waiting

10 jobs listed.

 

Usually the running job is holding up others. In this example the hostinfo.service is running and should be the focus of troubleshooting efforts. Show the services that depend on the running job. They cannot start until the running job completes.

> sudo /bin/systemctl --no-pager --after list-jobs
JOB UNIT                                 TYPE  STATE  
208 getty@tty1.service                   start waiting
    waiting for job 207 (getty.target/start)
216 systemd-user-sessions.service        start waiting
    waiting for job 117 (multi-user.target/start)
    waiting for job 208 (getty@tty1.service/start)
    waiting for job 235 (plymouth-quit-wait.service/start)
    waiting for job 240 (plymouth-quit.service/start)
213 hostinfo.service                     start running
    waiting for job 117 (multi-user.target/start)
    waiting for job 253 (issue-generator.service/start)
207 getty.target                         start waiting
    waiting for job 117 (multi-user.target/start)
    waiting for job 255 (after-local.service/start)
117 multi-user.target                    start waiting
    waiting for job 219 (systemd-update-utmp-runlevel.service/start)
219 systemd-update-utmp-runlevel.service start waiting
255 after-local.service                  start waiting
    waiting for job 117 (multi-user.target/start)
235 plymouth-quit-wait.service           start waiting
    waiting for job 117 (multi-user.target/start)
    waiting for job 208 (getty@tty1.service/start)
253 issue-generator.service              start waiting
    waiting for job 117 (multi-user.target/start)
    waiting for job 216 (systemd-user-sessions.service/start)
240 plymouth-quit.service                start waiting
    waiting for job 117 (multi-user.target/start)

10 jobs listed.

The multi-user.target and issue-generator.service cannot start until the hostinfo.service completes. Find out the service application hostinfo.service starts. If you have locate installed, you can use it to search for the hostinfo.service unit file. Many systemd units are kept in /usr/lib/systemd/system or /etc/systemd/system.

# locate hostinfo.service
/etc/systemd/system/default.target.wants/hostinfo.service
/usr/lib/systemd/system/hostinfo.service

# rpm -qf /usr/lib/systemd/system/hostinfo.service
hostinfo-1.0-18.15.noarch

# grep ExecStart /usr/lib/systemd/system/hostinfo.service
ExecStart=/usr/sbin/hostinfo

# rpm -qf /usr/sbin/hostinfo
hostinfo-1.0-18.15.noarch

# chkbin /usr/sbin/hostinfo
# cat /var/log/scc_chkbin_hostinfo_1937.txt 
####################################################################
Binary Check Tool, v1.02.1
Date:   05/27/21, 12:24:39
Kernel: 5.3.18-24.64-default, Hardware: x86_64
####################################################################

Checking Binary Ownership                                   ... Done
Checking for Shared Libraries                               ... Done
Validating Unique RPMs                                      ... Warning

#--[ Checking File Ownership ]-------------------------------------#
/usr/sbin/hostinfo                  - from RPM: hostinfo-1.0-18.15.noarch
 :Shell script

#--[ Validating Unique RPMs ]--------------------------------------#
Validating RPM: hostinfo-1.0-18.15.noarch            [   Warning   ]
.......T.  c /etc/hostinfo.conf
S.5....T.    /usr/sbin/hostinfo

In this case the size, md5sum and timestamp have all changed since /usr/sbin/hostinfo was installed. Reinstall the hostinfo RPM package and test again. If you have a server with the same service working correctly, compare the chkbin output from the working and affected servers.

If no issues are reported with the application, you might consider opening a support case with SUSE and include a supportconfig after running chkbin on the running services.