DEV Community

Cover image for No Space Left on Device, df Shows Space?
devTalk
devTalk

Posted on Originally published at dev-talk.com

No Space Left on Device, df Shows Space?

#linux #troubleshooting #inodes #diskspace

"No space left on device" is a confusing error when df -h says the disk is only 4% full. The usual cause is that you have not run out of space, you have run out of inodes. This short guide explains what that means, how to confirm it in one command, and how to find and clear whatever is using them up.

What an inode is

Every file and every directory on a Linux filesystem needs one inode, a small record that stores its details. The number of inodes is limited, and on filesystems like ext4 it is fixed when the filesystem is created. Millions of tiny files, such as session files, cache files or mail queue entries, can use up every inode while barely touching the actual storage.

Step 1: confirm it with df -i

The normal df -h shows space. Add -i to see inodes instead. Here is a real example from a test filesystem with a tiny inode limit. Writing a file fails:

echo test > /data/new.txt
# -bash: /data/new.txt: No space left on device
Enter fullscreen mode Exit fullscreen mode

Yet the disk looks almost empty:

df -h /data
Enter fullscreen mode Exit fullscreen mode 
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            20M  788K   20M   4% /data
Enter fullscreen mode Exit fullscreen mode

Now check the inodes:

df -i /data
Enter fullscreen mode Exit fullscreen mode 
Filesystem     Inodes IUsed IFree IUse% Mounted on
tmpfs             200   200     0  100% /data
Enter fullscreen mode Exit fullscreen mode

IUse% is 100% and IFree is 0. That is the whole answer. I also reproduced it on a real ext4 filesystem created with only 64 inodes: df -h showed 3% used while df -i showed 100%, and the 54th file failed with the same error.

Step 2: find the directory that holds all the files

Count inodes per directory with du --inodes, then sort so the biggest comes last:

du --inodes -d1 /data | sort -n | tail
Enter fullscreen mode Exit fullscreen mode 
1   /data/cache
198 /data/sessions
200 /data
Enter fullscreen mode Exit fullscreen mode

Here /data/sessions holds 198 of the 200 inodes. On a real server, start at / and drill down one level at a time (du --inodes -d1 /var, then the next folder). Add -x to stay on one filesystem.

If du --inodes is not available on an old system, this counts files per top-level folder instead:

for d in /var/*; do echo "$(find "$d" -xdev 2>/dev/null | wc -l) $d"; done | sort -n | tail
Enter fullscreen mode Exit fullscreen mode

Common culprits

  • Session files. PHP's file sessions are stored as sess_* files, and Laravel's file session driver writes to storage/framework/sessions.
  • File cache. A file-based cache driver creates one file per key.
  • Mail queues and temp folders that nothing cleans up.
  • Old log files and uploads split into thousands of small pieces.

Step 3: delete the files safely

Once you know the folder, remove old files rather than everything. This deletes files older than 7 days:

find /data/sessions -type f -mtime +7 -delete
Enter fullscreen mode Exit fullscreen mode

In a test with 150 session files, 5 of them older than a week, this reduced inode use from 152 to 147, exactly the 5 old files and nothing else. Preview first by swapping -delete for -print, so you can see what will go.

Avoid rm /data/sessions/* on a folder with a huge number of files. The shell expands the * into a giant list and can fail with "Argument list too long". find ... -delete does not have that problem.

Step 4: stop it happening again

  • Clean up automatically. Add a scheduled job that deletes old session and cache files, or move sessions and cache to Redis or the database so they do not create files at all.
  • Monitordf -i. Alert at 80% inode use, the same way you alert on disk space.
  • Plan inode count for new volumes. On ext4 the count is set when the filesystem is made, and you cannot add more later without recreating it. mkfs.ext4 -N sets the number of inodes directly. XFS and Btrfs allocate inodes dynamically, so this problem is much rarer on them.

If df -i is fine too

If both df -h and df -i look healthy, the problem is something else. Two other possibilities are files that were deleted but are still held open by a running process, which you can list with lsof +L1, and a disk quota for your user. Those are separate issues from inode exhaustion.

Quick reference

Goal Command
Check disk space df -h
Check inodes df -i
Find the folder with the most files `du --inodes -d1 /path \
Preview old files {% raw %}find /path -type f -mtime +7 -print
Delete old files find /path -type f -mtime +7 -delete

Originally published on DEV Talk.

Top comments (0)