Disk full alerts at 2am? Learn the exact commands to find what's eating your Linux server's disk space and fix it fast.
You get the alert: disk usage at 94%. Your app starts throwing errors, logs stop writing, and databases refuse to accept new rows. Finding the culprit fast matters — but on a server with millions of files, knowing where to look is half the battle. Here's a systematic approach to track down disk hogs in minutes, not hours.
Start With the Big Picture: df
Before you dig into directories, confirm which filesystem is actually full. Run:
- df -h — shows all mounted filesystems with human-readable sizes
- df -h / — focus on the root filesystem
- df -i — check inode usage (a filesystem can be 'full' even with free space if inodes are exhausted) Pay attention to the 'Use%' column. If you see 100% on /var or /home but not /, that tells you exactly which mount point to investigate. Inode exhaustion — df -i showing 100% — is easy to miss and causes the same symptoms as a full disk, so always check both.
Drill Down With du
Once you know which mount point is full, use du to find the largest directories. Start from the top of that mount point and work down:
- du -sh /* 2>/dev/null — sizes of every top-level directory, errors suppressed
- du -sh /var/* 2>/dev/null — drill into /var if that's the culprit
- du -ah /var | sort -rh | head -20 — list the 20 largest files and folders inside /var The pattern is always the same: run du -sh on the suspicious directory, find the largest subdirectory, repeat one level deeper. You'll usually hit the real culprit within three or four iterations. Common offenders are /var/log (runaway logs), /var/lib/docker (unused images and volumes), and /tmp (applications that don't clean up after themselves).
Find Large Files Directly With find
Sometimes a single enormous file is the problem — a core dump, a forgotten database export, or a log that rotated incorrectly. Use find to surface files above a size threshold:
- find / -xdev -size +500M -ls 2>/dev/null — files over 500 MB on the current filesystem only (-xdev stops it crossing into other mount points)
- find /var/log -name '*.log' -size +100M — large log files specifically
- find / -xdev -name 'core' -o -name '*.dump' 2>/dev/null — core dumps that can appear silently after crashes Always use -xdev with find when searching from / — without it, find will cross into other mount points and give you confusing results about space that belongs elsewhere.
Track Down Deleted Files Still Holding Space
One of the most frustrating disk space mysteries on Linux: a file gets deleted but disk usage doesn't drop. This happens when a process still has the file open — the space isn't reclaimed until that process closes or releases the file. To find these ghost files:
- lsof +L1 — lists all open files where the link count has dropped to zero (i.e., deleted but still held open)
- lsof +L1 | awk 'NR>1 {print $7, $1, $2}' | sort -rn | head -10 — sort by file size descending
- sudo lsof +L1 | grep deleted — simpler alternative if you just want to see what's deleted but open The fix is usually to restart the process holding the file. If it's a log file held open by a service like nginx or a Java app, restarting the service releases it. Alternatively, if restarting isn't an option immediately, you can truncate the file: > /proc//fd/ — but a restart is cleaner.
Clean Up Common Space Wasters
Once you've identified the culprit, here are targeted cleanup commands for the most frequent offenders:
- Docker: docker system prune -a --volumes — removes stopped containers, unused images, and orphaned volumes. Be sure you actually want this in production.
- Old journals: journalctl --vacuum-size=200M or journalctl --vacuum-time=7d — caps systemd journal size
- Package manager leftovers: apt autoremove && apt clean on Debian/Ubuntu; dnf autoremove on RHEL-based systems
- Rotated logs not cleaning up: check /etc/logrotate.conf and run logrotate -f /etc/logrotate.conf to force a rotation cycle
- Core dumps: check /proc/sys/kernel/core_pattern to see where they go, then remove them — consider setting kernel.core_pattern=/dev/null in /etc/sysctl.conf if you don't need them For Docker specifically, it's worth making prune a scheduled task rather than a reactive one. A weekly cron job running docker system prune -f prevents volumes from silently accumulating over months.
Make Disk Pressure Visible Before It Becomes a Crisis
Tracking down disk usage reactively works, but the real fix is knowing about gradual disk growth before it hits 100%. Set up alerts at 75% and 85% so you have time to investigate calmly rather than under pressure. Trend data matters too — a disk filling at 1 GB per day needs a different response than one that jumped 20 GB overnight.
This is where Opservo helps: it monitors disk usage across your servers, surfaces which directories are growing fastest, and flags anomalies like a log file that suddenly triples in size — before you get the 2 am alert. If you're managing production servers without a dedicated SRE, having that context surfaced automatically means you spend time fixing problems, not hunting for them.
The commands above will get you out of trouble today. The longer-term win is building visibility so slow-growing disk issues never become emergencies in the first place.
Originally published on the Opservo blog — Opservo is the AI ops engineer for teams without an SRE. Free for 2 servers → https://getopservo.com/welcome
Top comments (0)