"No space left on device" is one of those errors that shows up at the worst possible time — during a deploy, in the middle of a database write, at 11 PM when you just wanted to finish one thing. The good news: finding the culprit on Linux takes about sixty seconds once you know three commands. The better news: most of the time it is logs, caches, or Docker, and cleaning them is safe.
Here is the exact sequence I follow, from "which disk?" to "what do I delete?".
Step 1: Which filesystem is actually full?
df -h
Look for the filesystem at 100% (or close). Usually it is /, but on servers with separate mounts it might be /var or a data disk. The Use% column is your friend, and -h gives you human-readable sizes.
Do not skip this step. I have watched people clean their home directory for ten minutes while the full filesystem was /var/lib/docker. Clean the right disk.
Step 2: Find the big directories
My favorite one-liner, run from the root of the full filesystem:
sudo du -xhd1 / | sort -rh | head -15
What the flags do: -x stays on one filesystem (so you do not wander into mounted disks), -h human sizes, -d1 just one level deep. Then sort -rh puts the biggest on top.
Now you just repeat the same command, drilling into whatever is biggest:
sudo du -xhd1 /var | sort -rh | head -10
sudo du -xhd1 /var/log | sort -rh | head -10
Two or three rounds of this and you are looking at the culprit. It is almost always one of: /var/log, /var/lib/docker, a cache directory, or your own forgotten backups.
If you prefer a visual explorer, sudo ncdu / gives you an interactive version — arrow keys to drill in, d to delete. Install it with sudo apt install ncdu. It is one of those tiny tools that feels like a superpower the first time you use it.
Step 3: The usual suspects (and how to clean them safely)
Old logs. Check first, then clean:
sudo journalctl --disk-usage
sudo journalctl --vacuum-size=500M # keep only the last 500 MB
For old rotated log files, sudo find /var/log -name '*.gz' -delete clears the compressed archives. Resist the urge to rm a log file a process is actively writing to — the space will not be freed until the process lets go. Truncate instead: sudo truncate -s 0 /var/log/big.log.
Docker. If you use containers, this is the big one:
docker system df
docker system prune -a
prune -a removes stopped containers, unused networks, and all images not currently in use. Read that last part again: if you have an image you meant to run later, tag it or accept the re-download. For a gentler clean, docker system prune (without -a) removes only dangling images.
Package caches. sudo apt clean frees the apt archive, usually hundreds of MB. Language caches are similar: npm cache clean --force, pip cache purge, go clean -modcache — all safe, all rebuilt on demand.
Your own trash and forgotten files. ~/.local/share/Trash can hold gigabytes for years. And everyone has a backup-final-v2-REAL.tar.gz somewhere. du finds them; you decide their fate.
Step 4: When the disk is full but nothing looks big
Two classic gotchas:
Deleted files still held open. A process deleted a log but kept writing to the handle, so the space is not freed. df says full, du says nothing is there. Find them with:
sudo lsof +L1
Restarting the offending process (or the service) releases the space immediately.
Reserved blocks. ext4 reserves ~5% for root by default. On a big data disk that is a lot of invisible space; tune2fs -m 1 /dev/sdX1 lowers it to 1% on non-root filesystems.
Make it not happen again
A full disk is usually a slow problem pretending to be a sudden one. Two small habits prevent most repeats:
-
Log rotation and journal limits — cap systemd's journal in
/etc/systemd/journald.confwithSystemMaxUse=500M -
A simple disk alert — even a cron job that mails you above 85% usage is enough.
df -h / | awk 'NR==2 {print $5}'in a weekly reminder works too
And keep the one-liner handy: sudo du -xhd1 / | sort -rh | head -15. Next time the error appears at 11 PM, you will be back to your evening in five minutes instead of an hour.
Top comments (0)