No space left on device - how to determine which directory occupies the most inodes

This document (7014803) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server 11 Service Pack 1 (SLES 11 SP1)
SUSE Linux Enterprise Server 11 Service Pack 2 (SLES 11 SP2)
SUSE Linux Enterprise Server 11 Service Pack 3 (SLES 11 SP3)
SUSE Linux Enterprise Desktop 11
SUSE Linux Enterprise Desktop 11 Service Pack 1 (SLED 11 SP1)
SUSE Linux Enterprise Desktop 11 Service Pack 2 (SLED 11 SP2)
SUSE Linux Enterprise Desktop 11 Service Pack 3 (SLED 11 SP3)

Situation

An ext2/3 system reports no space left on device but checking df -h output shows plenty disk space free.

Resolution

Run the command df -i to check for the state of available inodes, if an entry shows 100% the next step is to validate which directory stores the most files. In order to scan directories on a single device, the following command may be useful:

find / -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n

This will show the directory using the most inodes as:

susemanager:~ # cd /
susemanager:/ # find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n
     15 tmp
     39 root
    100 bin
    163 lib64
    190 sbin
   2158 lib
   2901 etc
   7029 srv
41669 usr
59915 var

To narrow down the directory structure:

cd /var
find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n

which will show

susemanager:/var # find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n
      1 opt
      8 spool
      9 lock
     24 run
    201 log
    223 adm
   1459 cache
   5612 lib
52378 spacewalk

Repeat this until the problematic directory has been identified and check if files can be deleted to free up inodes.

Cause

A likely cause for this is a process created lots of small files which occupy all inodes of a given filesystems.

Additional Information

Note: The SUSE Manager in this case was only used to demonstrate the procedure.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.