I've been working on some Linux troubleshooting labs recently, and one of my favorite scenarios involves an error that almost every sysadmin has seen:
No space left on device
Normally, the first thing you'd check is disk usage.
df -h
And in most cases, that tells you exactly what you need to know.
But sometimes the output looks completely normal:
Filesystem Size Used Avail Use%
/dev/vda1 40G 17G 21G 45%
There's 21 GB of free space. Nothing particularly suspicious.
Yet the application still can't create files. Even trying to create an empty file in the affected directory fails:
touch /var/cache/myapp/test
touch: cannot touch '/var/cache/myapp/test':
No space left on device
So what's going on?
The problem isn't always disk space
The important thing to remember is that filesystems have more limits than just available storage.
On ext4, for example, files need inodes. An inode stores metadata about a file, and the filesystem has a finite number of them.
You can have plenty of free disk space and still run out of inodes.
That's why I usually check this next:
df -i /var/cache/myapp
Here's an example of what you might find:
Filesystem Inodes IUsed IFree IUse%
/dev/vda1 2621440 2621440 0 100%
There it is. No free inodes.
The filesystem has room for more data, but it can't allocate another inode for a new file.
How do you end up here?
Usually, something has been creating a ridiculous number of small files.
A cache directory is a good example.
Imagine an application writing thousands of tiny cache files and never cleaning them up. Each file barely uses any disk space, but each one consumes an inode.
Keep doing that long enough, and eventually you hit the limit.
One command I find useful when investigating this is:
sudo du --inodes -x -d 2 /var/cache
Instead of showing disk usage in bytes, it counts inode usage.
The -x option keeps the scan on the same filesystem, which is helpful when you're trying to find where the inodes went.
I wouldn't immediately run it across the entire server, though. Scanning millions of files can take a while. It's better to start with directories you already suspect.
Once you've found the problem, the fix depends on what's creating those files.
Maybe an application isn't rotating its cache. Maybe a cleanup job stopped working. Maybe somebody left a script running that generates temporary files indefinitely.
Removing unnecessary files gets the service working again, but unless you fix the underlying cause, you'll probably be dealing with the same incident next week.
One thing worth mentioning
This particular problem is common on filesystems such as ext4, where the inode count is generally determined when the filesystem is created.
Other filesystems handle inode allocation differently, so inode exhaustion isn't always the explanation.
And No space left on device can have other causes too.
That's why I wouldn't jump straight to deleting files just because df -i is the next command in this example.
First, make sure you're checking the filesystem where the failure actually happens. /tmp, for example, might be mounted separately.
Why I like this scenario
What makes this interesting to me isn't the solution itself. Once you know about inode exhaustion, it's a fairly straightforward problem to recognize.
It's the assumption people make at the beginning.
The error says there's no space, so you check free space. The numbers look fine, and suddenly the obvious explanation doesn't work anymore.
That's where troubleshooting actually starts.
I've been trying to build more exercises around situations like this. Not necessarily difficult commands or obscure Linux features, but problems where you need to question your first assumption.
Because knowing which command to run is useful.
Knowing why you're running it is much more useful.
Top comments (0)