Disk full is still the #1 outage nobody watches: a 15-minute fix

Ask a room of ops people which failure has bitten them most, and the answer is embarrassingly boring: the disk filled up. Not the fancy Kubernetes incident. The boring one — logs growing, a tmp directory nobody owned, and at 3am the database stops accepting writes because it literally cannot write.

I write operational playbooks for small teams (250+ articles on this profile), and in scheduled-job audits, disk exhaustion is the failure I find most often that nobody had an alarm for. Here's the fix — it costs fifteen minutes.

Why "it's fine" is a lie

Disk fills are silent because they're slow. You go from 40% to 78% over weeks. Every daily check sees "still under 80, looks fine." Then a single verbose job or a stuck log rotation pushes you from 78% to 100% in one night, and everything that writes — databases, queues, uploads — fails at once. You don't have one incident. You have every incident at the same time.

The 15-minute fix

1. Alarm on trend, not threshold. A 90% alert fires once — usually the night you're asleep and it passes 89→100 in an hour. Alert on growth rate instead: "if the disk grew more than 2% in 24 hours, tell me." Slow leaks get caught days early.
2. Find the eaters. One command: du -x / | sort -h | tail -20. Ninety percent of the time it's logs (fix rotation), package caches (fix retention), or temp files from one job that never cleans up after itself.
3. Give every writer a budget. Log rotation with hard caps (maxsize, retain 7), uploads on their own volume, and a weekly cron that reports the top 20 disk consumers to you. When the report arrives every week, drift becomes visible while it's still cheap.
4. Leave a kill floor. Keep the disk under 75% by policy. That headroom is what turns "database refused writes" into "nothing happened."

The drill

Right now, before you close this tab: run the du command on your most important box. If the answer surprises you, you just found next weekend's incident. If you don't know where the output would even go — that's the real gap, and it's the one alarms can't fix.

The full kit

The monitoring patterns in this article — trend alarms, weekly disk reports, the quiet-failure checklist — are packaged with the rest of the operational basics in the downloadable kit:

→ Automation Starter Pack

Want the outage-side drills too? First 30 Minutes: Emergency Response Kit covers what to do when the disk (or anything else) already won, and Ops Starter Kit has the daily checklists that catch drift before it catches you.

What's eating your disk right now — and did you actually know before running the command? Tell me in the comments; I'll suggest the retention fix for the worst one.