DEV Community

Cover image for Disk Full on a Linux VPS? How to Find What Is Using the Space
Pro Gineous
Pro Gineous

Posted on Originally published at blog.progineous.com

Disk Full on a Linux VPS? How to Find What Is Using the Space

#linux #devops #docker #sysadmin

A full disk rarely announces itself. MySQL stops accepting writes, WordPress can't upload an image, apt fails halfway through an upgrade, or you get "No space left on device" from a command that worked yesterday. On a 20 or 30 GB VPS, one chatty log or a forgotten backup folder is enough.

The fix is almost never "delete random big things until it works". It's a short triage: confirm what kind of full you are, find where the space went, clear the parts that are safe to clear, and leave the rest alone. Below is the order we work through, with the commands and real output from a test server in the same state.

Step 0: buy a little room first

At 100%, some tools won't even start because they can't write a temp file. Two commands are safe on almost any Debian or Ubuntu server and usually free a few hundred megabytes:

sudo apt-get clean
sudo journalctl --vacuum-size=200M
Enter fullscreen mode Exit fullscreen mode

The first removes downloaded .deb files, which apt can fetch again. The second trims old journal files down to 200 MB. Neither touches your site, database or config.

Step 1: is it space or inodes? 

df -h /
df -i /
Enter fullscreen mode Exit fullscreen mode

df -h shows blocks, which is what people usually mean by disk space. df -i shows inodes, one per file. A disk can run out of inodes while plenty of gigabytes are free, usually because something writes millions of tiny files: PHP sessions, a cache plugin, a stuck mail queue. If IUse% is the one near 100%, look for directories with huge file counts (du --inodes -x / | sort -n | tail) instead of big files.

Also notice that Size minus Used doesn't equal Avail. On ext4, 5% of the filesystem is reserved for root by default, so the system can still log in and write a few things after normal users hit zero. On the test server below that's about 1.5 GB of a 30 GB disk. You can lower it with tune2fs -m, but on a root filesystem that margin is what keeps a full server reachable.

Step 2: walk down from / with du 

sudo du -xh --max-depth=1 / | sort -h
Enter fullscreen mode Exit fullscreen mode

The -x flag keeps du on the root filesystem, so it skips /proc and other mounts. sort -h understands the K, M and G suffixes, so the biggest directory lands at the bottom. Then repeat one level deeper on whatever is biggest, usually /var or /home.

Terminal screenshot of df -h showing the root disk 99% full with 345M available, df -i at 9% inode use, and du drilling down from / to /var and /var/lib, where Docker takes 9.1G

If ncdu is installed (apt install ncdu), sudo ncdu -x / does the same walk interactively with the arrow keys. Plain du works everywhere, though.

Step 3: look for single big files

Directories tell you where. A file search tells you what:

sudo find / -xdev -type f -size +500M -exec ls -lh {} +
Enter fullscreen mode Exit fullscreen mode

On our test server that turned up the usual crowd: a 1.5 GB database dump in /root from before an upgrade, an old site zip, nine weekly backup archives at 1.2 GB each, a 940 MB nginx access log, and a 3.6 GB file deep inside /var/lib/docker/containers. That last one gets its own section below.

Step 4: check the usual suspects

The systemd journal. By default journald may use up to 10% of the filesystem, capped at 4 GB. Check it, trim it, then set a permanent limit with SystemMaxUse=500M in /etc/systemd/journald.conf and systemctl restart systemd-journald.

$ journalctl --disk-usage
Archived and active journals take up 1.3G in the file system.
Enter fullscreen mode Exit fullscreen mode

Package cache and old kernels. du -sh /var/cache/apt/archives showed 205M here. apt-get clean handles that. On Ubuntu, old kernels add a few hundred megabytes each in /boot and /usr/lib/modules. sudo apt autoremove --purge removes the ones no longer needed and keeps the running one. Read the list before you type yes.

Docker. Start with docker system df:

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          10        7         5.285GB   1.922GB (36%)
Containers      7         5         1.099kB   0B (0%)
Local Volumes   4         4         243.5MB   0B (0%)
Build Cache     0         0         0B        0B
Enter fullscreen mode Exit fullscreen mode

Two things this table doesn't tell you. Old stopped containers keep their images "active", so check docker ps -a and remove the ones you really don't need before docker image prune -a. And container logs aren't counted at all. That 3.6 GB file was the JSON log of one busy worker container. Cap logs for new containers in /etc/docker/daemon.json:

{ "log-driver": "json-file", "log-opts": { "max-size": "50m", "max-file": "3" } }
Enter fullscreen mode Exit fullscreen mode

Existing containers keep their old settings until you recreate them. And be careful with docker volume prune. Volumes are where your database lives.

Logs that never rotate. An access.log.1 the size of the current log means rotation runs but compression doesn't, or the app writes somewhere logrotate doesn't know about. sudo logrotate -d /etc/logrotate.conf does a dry run and shows what it would do for each file without changing anything.

Step 5: when df and du disagree

Look back at the first screenshot. du counts 27G, while df says 28G is used. When the gap is bigger than rounding, look for a file that was deleted while a process still had it open. The name is gone, but the blocks stay allocated until the process closes the file. The classic way to get there is rm on a big log an app is still writing to.

sudo lsof -nP +L1
Enter fullscreen mode Exit fullscreen mode

+L1 lists open files with a link count below one, meaning deleted. Without lsof, ls -l /proc/*/fd 2>/dev/null | grep deleted finds the same thing.

Terminal screenshot where df reports 28G used but du counts 27G, lsof +L1 finds a 900 MB deleted worker.log still held open by a python3 process, and emptying it through /proc brings the disk back to 96%

Restarting the service that holds the file releases the space. If you can't restart it right now, empty the file through its descriptor, as in the screenshot. Next time, empty a live log with truncate -s 0 file instead of deleting it.

What not to delete

Checklist for cleaning a full disk in three columns: safe to clear (apt cache, old journal files, old kernels, unused Docker images), check first (backups, live logs, stopped containers, deleted open files) and leave alone (database folders, /var/lib/docker, Docker volumes, /boot)

A few things look like junk and aren't:

  • Anything under /var/lib/mysql or /var/lib/postgresql. Binary logs and WAL files get big, but you clear them from inside the database (PURGE BINARY LOGS in MySQL, fixing a stuck replication slot in PostgreSQL), never with rm.
  • /var/lib/docker by hand. Use the docker commands. Deleting overlay2 folders directly breaks images and containers in ways that are painful to fix.
  • Files in /boot. Remove kernels with apt so the bootloader config is updated too.
  • Your only copy of a backup. Those weekly archives belong on another machine anyway. Copy them off the server first, check that the copy opens, then delete the old ones. Our guide on backing up a VPS and testing the restore covers where to send them.

Keep it from happening again

Once there's room, put the limits in place: SystemMaxUse for the journal, log-opts for Docker, and a logrotate entry for every app that writes its own log. Add a disk check to whatever monitoring you already run, alerting at 80% rather than 95%. If you set up the server with our 15-minute VPS security checklist, unattended-upgrades is already installing new kernels, so run apt autoremove --purge now and then.

We put the read-only checks from this article into one script, disk-usage-report.sh, in our server toolkit on GitLab. It prints suggested cleanup commands and deletes nothing.

And if the disk keeps filling because the site has simply grown, a bigger disk is cheaper than cleaning up every week. Pro Gineous VPS plans start at $6.96/mo.

Related reading

Originally published on the Pro Gineous blog.

Top comments (0)