DEV Community

jun liu
jun liu

Posted on

Inode Exhaustion: The Disk Full Error That Makes No Sense

#devops #infrastructure #linux

You run df -h and the root partition shows 85% used. Plenty of space left, right? Then you try to create a new file and get: No space left on device.

You check again. You reboot. You even resize the partition. Same error.

The problem isn't your disk space. It's your inodes.

What Are Inodes?

An inode is a data structure that stores metadata about a file: permissions, owner, timestamps, and pointers to the actual data blocks on disk. Every file, every directory, every symlink gets one inode.

When you create a file, the system needs two things:

  • Data blocks to store the actual content (governed by disk space)
  • An inode to describe the file (governed by inode count)

These are separate limits. df -h only shows data block usage. It says nothing about inodes.

The Classic Scenario

Here's a real example I saw last month on a production server hosting a Node.js application:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       100G   85G   15G  85% /

$ df -i
Filesystem     Inodes IUsed   IFree IUse% Mounted on
/dev/sda1     6553600 6553600       0  100% /
Enter fullscreen mode Exit fullscreen mode

The disk is 85% full but has zero inodes remaining. That's the contradiction that confused the on-call engineer for an hour.

What Causes Inode Exhaustion?

Tiny Files Accumulate

Every npm install, every webpack cache entry, every log rotation creates thousands of small files. A single node_modules directory can easily contain 50,000+ files.

# Count files in node_modules
find . -type f | wc -l
# Often: 80000-150000 files
Enter fullscreen mode Exit fullscreen mode

Log Rotation Gone Wrong

Misconfigured logrotate creates compressed copies without deleting originals. Or a backup script writes hourly snapshots to /tmp without cleanup:

# Culprit pattern: backup creates a file per minute
ls /backup/ | wc -l
# Result: 43,200 small files
Enter fullscreen mode Exit fullscreen mode

Ephemeral Session Files

PHP sessions, temporary upload files, or cache directories that grow without bounds. Each request creates a new file. Nothing cleans them up.

# Check common inode traps
ls /var/lib/php/sessions/ | wc -l
ls /tmp | wc -l
Enter fullscreen mode Exit fullscreen mode

Diagnosing the Culprit

Step 1: Confirm Inode Exhaustion 

df -i
Enter fullscreen mode Exit fullscreen mode

If any partition shows 100% in Use%, you've found the problem.

Step 2: Find the Largest Directories 

# Top-level breakdown (slow on large filesystems)
sudo du -x --max-depth=1 / 2>/dev/null | sort -rh | head -20
Enter fullscreen mode Exit fullscreen mode

Focus on the directory with the most files, not necessarily the most bytes.

Step 3: Identify the Pattern

Once you find the suspect directory, check the file characteristics:

# Check if they're tiny files (the inode killer)
find /path/to/suspect -type f -size -1k | wc -l
# If most files are under 1KB, that's your inode bomb
Enter fullscreen mode Exit fullscreen mode

The Fix: Clean Up and Prevent

Immediate Relief

Delete the smallest, most numerous files first. Each deletion frees one inode. Prioritize:

# Clear npm cache (frees thousands of tiny files)
npm cache clean --force

# Rotate and compress old logs
sudo journalctl --vacuum-time=3d

# Remove temporary files older than 7 days
find /tmp -type f -mtime +7 -delete
Enter fullscreen mode Exit fullscreen mode

Long-Term Prevention

Configure logrotate properly so old logs compress and rotate instead of piling up:

/var/log/myapp/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 0640 www-data adm
    postrotate
        systemctl reload myapp
    endscript
}
Enter fullscreen mode Exit fullscreen mode

Then add a cron job that warns you before inodes hit critical:

0 6 * * * df -i | awk 'NR==1{next} $5 ~ /[9][0-9]%/{print "INODE ALERT: " $0}' | mail -s "Inode Usage Critical" admin@example.com
Enter fullscreen mode Exit fullscreen mode

When Cleanup Isn't Enough

Sometimes the filesystem was created with a small inode count. Check your filesystem creation parameters:

tune2fs -l /dev/sda1 | grep -i inode
# Inode count:           6553600
Enter fullscreen mode Exit fullscreen mode

If your inode count is too low for your workload, you have three options:

  1. Resize the filesystem (risky, requires unmount or online resize depending on filesystem type)
  2. Move data to a larger partition with more inodes
  3. Recreate the filesystem with higher inode density (last resort)

For ext4, the inode ratio is controlled at format time:

# Default: 1 inode per 16KB of space
mke2fs -T default /dev/sda1

# For file-heavy workloads, increase inode density
mke2fs -i 8192 /dev/sda1  # 1 inode per 8KB
Enter fullscreen mode Exit fullscreen mode

The Key Takeaway

df -h tells you about disk space. df -i tells you about inodes. Run both.

When you see "No space left on device" on a partition that clearly has free space, check inodes before you panic about disk capacity. The fix is rarely more storage — it's usually deleting thousands of tiny files you didn't know were piling up.

More ops notes and server reliability guides: jiunotes · ops notes

Top comments (0)