Processes in an Uninterruptible Sleep (D) State

This document (7002725) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15
SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server 10

Situation

Applications seem slow to respond or appear hung. Slow server performance. Processes stuck in a "D" state.

#==[ Checking Health of Processes ]=================#
# egrep " D| Z" /root/scc_server_210831_0955/psout.yKkLeVtM
vv
root 3744 3654 0.0 0.0 16420 768 D+ 00:00:00 ls -l /data
root 3964 3962 0.0 0.0 4480 668 D+ 00:00:00 /bin/df -h
^^

Resolution

Processes in a "D" or uninterruptible sleep state are usually waiting on I/O. The ps command shows a "D" on processes in an uninterruptible sleep state. The vmstat command also shows the current processes that are "blocked" or waiting on I/O. The vmstat and ps will not agree on the number of processes in a "D" state, so don't be too concerned. You cannot kill "D" state processes, even with SIGKILL or kill -9. As the name implies, they are uninterruptible. You can only clear them by rebooting the server or waiting for the I/O to respond. It is normal to see processes in a "D" state when the server performs I/O intensive operations.

If performance becomes an issue, you may need to check the health of your disks. Make sure your firmware and kernel disk drivers are updated.

In the example above, the NFS server had failed while the client still had an active mount to it. Restarting the NFS server would resolve this particular issue.

There are two ways to find more about the processes in D state.

1. ps -eo ppid,pid,user,stat,pcpu,comm,wchan:32
This prints a list of all processes where in the last column either a '-' is displayed when the process is running or the name of the kernel function in which the process is sleeping if the process is currently sleeping. This includes also processes which are interruptible. Processes that are in uninterruptible sleep can be determined via the fourth column which would then show a D.

# ps -eo ppid,pid,user,stat,pcpu,comm,wchan:32 | grep " D"
3654 3744 root D+ 0.0 ls -
1 3964 root D 0.0 df -

2. echo w > /proc/sysrq-trigger
This command produces a report and a list of all processes in D state and a full kernel stack trace to /var/log/messages. This shows much more information than the first option described above.

# echo w > /proc/sysrq-trigger
# dmesg -T
[Tue Aug 31 10:44:05 2021] task:ls state:D stack: 0 pid: 3744 ppid: 3654 flags:0x00004006
[Tue Aug 31 10:44:05 2021] Call Trace:
[Tue Aug 31 10:44:05 2021] __schedule+0x2ff/0x760
[Tue Aug 31 10:44:05 2021] ? __rpc_wait_for_completion_task+0x30/0x30 [sunrpc]
[Tue Aug 31 10:44:05 2021] ? rpc_exit+0x30/0x30 [sunrpc]
[Tue Aug 31 10:44:05 2021] schedule+0x2f/0xa0
[Tue Aug 31 10:44:05 2021] rpc_wait_bit_killable+0x1e/0xa0 [sunrpc]
[Tue Aug 31 10:44:05 2021] __wait_on_bit+0x2c/0x90
[Tue Aug 31 10:44:05 2021] out_of_line_wait_on_bit+0x91/0xb0
[Tue Aug 31 10:44:05 2021] ? init_wait_var_entry+0x50/0x50
[Tue Aug 31 10:44:05 2021] __rpc_execute+0x10c/0x420 [sunrpc]
[Tue Aug 31 10:44:05 2021] ? nfs_weak_revalidate+0x67/0xd0 [nfs]
[Tue Aug 31 10:44:05 2021] rpc_execute+0x71/0xd0 [sunrpc]
[Tue Aug 31 10:44:05 2021] rpc_run_task+0x10d/0x140 [sunrpc]
[Tue Aug 31 10:44:05 2021] rpc_call_sync+0x50/0x90 [sunrpc]
[Tue Aug 31 10:44:05 2021] nfs3_rpc_wrapper.constprop.13+0x2f/0xa0 [nfsv3]
[Tue Aug 31 10:44:05 2021] nfs3_proc_getattr+0x58/0xb0 [nfsv3]
[Tue Aug 31 10:44:05 2021] __nfs_revalidate_inode+0xff/0x330 [nfs]
[Tue Aug 31 10:44:05 2021] nfs_getattr+0x140/0x2d0 [nfs]
[Tue Aug 31 10:44:05 2021] vfs_statx+0x89/0xe0
[Tue Aug 31 10:44:05 2021] __do_sys_statx+0x3b/0x80
[Tue Aug 31 10:44:05 2021] do_syscall_64+0x5b/0x1e0
[Tue Aug 31 10:44:05 2021] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[Tue Aug 31 10:44:05 2021] RIP: 0033:0x7f0c9822943b
[Tue Aug 31 10:44:05 2021] Code: Bad RIP value.
[Tue Aug 31 10:44:05 2021] RSP: 002b:00007fff051af2c0 EFLAGS: 00000206 ORIG_RAX: 000000000000014c
[Tue Aug 31 10:44:05 2021] RAX: ffffffffffffffda RBX: 00007fff051b0796 RCX: 00007f0c9822943b
[Tue Aug 31 10:44:05 2021] RDX: 0000000000000100 RSI: 00007fff051b0796 RDI: 00000000ffffff9c
[Tue Aug 31 10:44:05 2021] RBP: 0000000000000100 R08: 00007fff051af2e0 R09: 00000000ffffff9c
[Tue Aug 31 10:44:05 2021] R10: 000000000000025e R11: 0000000000000206 R12: 00007fff051af2e0
[Tue Aug 31 10:44:05 2021] R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000002
[Tue Aug 31 10:44:05 2021] task:df state:D stack: 0 pid: 3964 ppid: 1 flags:0x00004002
[Tue Aug 31 10:44:05 2021] Call Trace:
[Tue Aug 31 10:44:05 2021] __schedule+0x2ff/0x760
[Tue Aug 31 10:44:05 2021] ? __rpc_wait_for_completion_task+0x30/0x30 [sunrpc]
[Tue Aug 31 10:44:05 2021] ? rpc_exit+0x30/0x30 [sunrpc]
[Tue Aug 31 10:44:05 2021] schedule+0x2f/0xa0
[Tue Aug 31 10:44:05 2021] rpc_wait_bit_killable+0x1e/0xa0 [sunrpc]
[Tue Aug 31 10:44:05 2021] __wait_on_bit+0x2c/0x90
[Tue Aug 31 10:44:05 2021] out_of_line_wait_on_bit+0x91/0xb0
[Tue Aug 31 10:44:05 2021] ? init_wait_var_entry+0x50/0x50
[Tue Aug 31 10:44:05 2021] __rpc_execute+0x10c/0x420 [sunrpc]
[Tue Aug 31 10:44:05 2021] ? nfs_weak_revalidate+0x67/0xd0 [nfs]
[Tue Aug 31 10:44:05 2021] rpc_execute+0x71/0xd0 [sunrpc]
[Tue Aug 31 10:44:05 2021] rpc_run_task+0x10d/0x140 [sunrpc]
[Tue Aug 31 10:44:05 2021] rpc_call_sync+0x50/0x90 [sunrpc]
[Tue Aug 31 10:44:05 2021] nfs3_rpc_wrapper.constprop.13+0x2f/0xa0 [nfsv3]
[Tue Aug 31 10:44:05 2021] nfs3_proc_getattr+0x58/0xb0 [nfsv3]
[Tue Aug 31 10:44:05 2021] __nfs_revalidate_inode+0xff/0x330 [nfs]
[Tue Aug 31 10:44:05 2021] nfs_getattr+0x140/0x2d0 [nfs]
[Tue Aug 31 10:44:05 2021] vfs_statx+0x89/0xe0
[Tue Aug 31 10:44:05 2021] __do_sys_newstat+0x39/0x70
[Tue Aug 31 10:44:05 2021] do_syscall_64+0x5b/0x1e0
[Tue Aug 31 10:44:05 2021] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[Tue Aug 31 10:44:05 2021] RIP: 0033:0x7f0505d71095
[Tue Aug 31 10:44:05 2021] Code: Bad RIP value.
[Tue Aug 31 10:44:05 2021] RSP: 002b:00007ffd3c786828 EFLAGS: 00000246 ORIG_RAX: 0000000000000004
[Tue Aug 31 10:44:05 2021] RAX: ffffffffffffffda RBX: 00005636b25c87a0 RCX: 00007f0505d71095
[Tue Aug 31 10:44:05 2021] RDX: 00007ffd3c7868d0 RSI: 00007ffd3c7868d0 RDI: 00005636b25c7be0
[Tue Aug 31 10:44:05 2021] RBP: 0000000000000000 R08: 00005636b25c87a0 R09: 0000000000000003
[Tue Aug 31 10:44:05 2021] R10: 00005636b25c5010 R11: 0000000000000246 R12: 00005636b25c7c20
[Tue Aug 31 10:44:05 2021] R13: 00005636b25c7b80 R14: 0000000000000000 R15: 00007ffd3c7868b0

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.