When Disk Space Is Not Disk Space: The Inode Exhaustion Problem

You run df -h, everything looks fine. Plenty of space left. Then your application throws "No space left on device" and you spend an hour debugging the wrong thing.

That's inode exhaustion. Here's how it works and how to deal with it.

What Even Is an Inode

Every file on a filesystem has an inode. It's a metadata structure that holds:

• File permissions and ownership
• Timestamps
• Pointers to the actual data blocks
• A unique inode number

The inode table has a fixed size when you format the filesystem. You can have millions of bytes of free space but if you run out of inodes, you can't create new files.

# Check inode usage
df -i

# Example output:
# Filesystem      Inodes  IUsed  IFree IUse% Mounted on
# /dev/sda1      3276800 3276800      0  100% /data

Notice the IFree column is 0. That's your problem right there.

Why This Sneaks Up on You

Small files are the usual culprit. Each file consumes at least one inode regardless of size. A directory with millions of tiny files (cache files, session data, log shards) can burn through your inode table fast.

Common situations:

• /tmp filling with unnamed files from crashed processes
• Mail spools with tiny deferred messages
• Object storage caches creating countless small blobs
• Build artifacts that leave behind many tiny intermediate files

How to Find the Culprit

# Find directories with high inode counts
sudo find / -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head -20

# Specifically look at /tmp
sudo find /tmp -type f | wc -l

# Check a specific directory tree
sudo find /var -type f | wc -l

Once you find the directory, you can drill down:

sudo find /tmp -type f -ls 2>/dev/null | sort -k7 -n | tail -20

The tmpfs Gotcha

A less obvious trap: tmpfs filesystems (RAM-backed storage) mounted at /tmp or /run.

tmpfs has configurable size limits:

# Check tmpfs sizes
mount | grep tmpfs

# Example output:
# tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=512M)
# tmpfs on /run type tmpfs (rw,nosuid,nodev,size=100M)

The size= parameter limits both the byte size AND the inode count. If you set size=512M but don't set nr_inodes, the kernel auto-calculates inodes based on the size. Low memory systems often end up with surprisingly few inodes.

When tmpfs fills up, you get OOM behavior even though it's supposed to be backed by RAM. The kernel starts killing processes to free memory, which is rarely what you want.

Prevention

1. Monitor inodes, not just space. Add df -i to your alerting.

2. Set explicit inode limits on tmpfs if you're using one
   

   tmpfs /tmp tmpfs rw,nosuid,nodev,size=1G,nr_inodes=1M 0 0

1. Clean up strategies for temp directories. Logrotate isn't enough for /tmp - processes should clean their own mess, but they don't always.

2. Choose filesystems wisely for small-file workloads. Some filesystems (ext4 with bytes-per-inode ratio) let you tune inode density at mkfs time.

The Quick Fix When You're Stuck

# If /tmp is the problem and you need breathing room NOW
# (this clears all files in /tmp - be careful)
sudo find /tmp -type f -delete

# If you need to identify files by inode number for targeted deletion
sudo find /path/to/problem -inum 12345 -delete

The real fix is figuring out why you're creating files faster than you're cleaning them up.

Bottom Line

df -h is incomplete information. When something reports "no space" but df -h looks fine, check df -i before anything else. Inode exhaustion is rare but extremely annoying when it hits, and it always hits at the worst time.