autofs service is failing due to exhausted inodes in "/tmp" filesystem

This document (000020955) is provided subject to the disclaimer at the end of this document.

Environment

All versions of SUSE Linux Enterprise

Situation

Autofs cannot create a test mount into /tmp filesystem because filesystem limit hit, thus automount operation ends with failure message "test mount forbidden or incorrect kernel protocol version".

automount[2189]: /usr/sbin/automount: test mount forbidden or incorrect kernel protocol version, kernel protocol version 5.00 or above required.

Failed autofs.service in systemd:

# systemctl status autofs.service -l --no-pager
● autofs.service - Automounts filesystems on demand
     Loaded: loaded (/usr/lib/systemd/system/autofs.service; enabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Wed 2023-01-25 10:29:57 CET; 12s ago
       Docs: man:automount(8)
             man:autofs(5)
    Process: 2198 ExecStart=/usr/sbin/automount $AUTOFS_OPTIONS -p /run/automount.pid (code=exited, status=1/FAILURE)

Jan 25 10:29:57 sle15-n02 systemd[1]: Starting Automounts filesystems on demand...
Jan 25 10:29:57 sle15-n02 automount[2198]: /usr/sbin/automount: test mount forbidden or incorrect kernel protocol version, kernel protocol version 5.00 or above required.
Jan 25 10:29:57 sle15-n02 systemd[1]: autofs.service: Control process exited, code=exited, status=1/FAILURE
Jan 25 10:29:57 sle15-n02 systemd[1]: autofs.service: Failed with result 'exit-code'.
Jan 25 10:29:57 sle15-n02 systemd[1]: Failed to start Automounts filesystems on demand.

Resolution

1) Check inode availability on the file system with df -iT. The inodes used shows 100%:

# df -iT /tmp/
Filesystem     Type Inodes IUsed IFree IUse% Mounted on
/dev/sda       ext4 65536 65536     0  100% /tmp

NOTE: The value returned in supportconfig might not reflect the current status

2) Check the number of allocated inodes in the /tmp filesystem or directory:

# du -s --inode /tmp
65527	/tmp

3) Free filesystem inodes. This can be done by deleting unwanted and useless files, links, and directories from /tmp filesystem.

Since the limits are high and probably not exhausted manually, it may be wise to find what caused excessive usage of inodes. There is high possibility of an application or script dynamically creating files for temporary usage, without ever cleaning them up.

Cause

Excessive inode usage related to filesystem.

Status

Top Issue

Additional Information

These limits change with type of filesystem. Table below shows filesystem inode limits in most common filesystem in SLES environment.

The limitations of traditional SLES filesystems:

FILESYSTEM	MAX INODES
ext3	32k
ext4	64k
xfs	dynamic; related to size of FS/allocation groups
btrfs	2^64

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.