The disk filled at 2:47am. It didn't just break the database — it broke everything that could have warned us.

At 2:47am, the database stopped writing. No error in the app logs — because the app couldn't write logs anymore. The disk had filled, and it didn't just break the database: log rotation died, the backup cron wrote zero bytes, and the monitoring agent that should have been screaming couldn't record its own scream.

Three hours later the disk had space again. The backup that should have run that night didn't. That was the part that cost us.

Disk-full is never one incident — it's a family. The disk fills, and then the failure spreads: the database stalls, the log shipper buffers into RAM, the backup silently produces nothing (see our backup post), and the next restart may fail outright. The runbook below keeps it a one-page incident instead of a week of archaeology.

First: buy breathing room without deleting the evidence.

• Truncate active logs; don't delete them. Deleting a file a running service holds open frees zero space and destroys history. : > big.log or journalctl --vacuum-size=200M keeps handles valid and frees space immediately.
• Clear only caches you can explain: package-manager caches, old build artifacts, stale /tmp. Not the database directory. Never the backup directory.
• If the database stalled because the disk filled, freeing space is the incident response — recovery follows space.

Then find the top offenders — two minutes, not two hours.
du -x --max-depth=2 (or ncdu) and the tree usually explains itself. In our case: a debug log left at DEBUG from a February incident, a journal that had never been vacuumed, and six months of Docker layers. Boring. Predictable. Huge.

The safe cleanup order, as a habit:

1. Top offenders first (du / ncdu) — the tree explains itself.
2. Truncate active logs, vacuum journals — usually buys the whole day.
3. Prune containers with the tool's own commands (docker system prune), never by hand in /var/lib.
4. Move ambiguous things to a quarantine directory instead of deleting — decide their fate in the review, when there's no pressure.
5. Verify recovery today: the database writes, the cron runs, and the backup completes tonight, not "next week."

The guardrails that stop the sequel:

• The 80% alert is non-negotiable. One threshold, one channel, outside view. Every disk that filled to 100% was going to page somebody — the only question is whether it pages you at 80% or the database at 100%.
• Rotation with size caps on every log. Rotate, compress, delete after N days. Unbounded logging is a delayed incident.
• Backups land off-box. A backup that fills the production disk isn't a backup; it's a second incident scheduled for the worst week.
• A weekly five-minute disk review — scripted, top-10 largest directories — catches the slow leaks that never seem urgent until they are.

The one rule that survives every disk-full: never rm -rf a directory you can't explain. Disk pressure plus improvisation is how teams delete their own backups at 3am and only find out a year later.

The full runbook — safe cleanup order, the usual suspects, guardrail setup — is free:

→ https://hive80-lab.github.io/ops-notes/disk-full-incident-runbook.html

The 80% disk line is one of the five signals in our monitoring checklist (also free), and the Automation Starter Pack ($19) turns the disk review and rotation sweep into workflows that run themselves. When checks do fire, the Ops Starter Kit ($14) covers the incident side. Launch-week pricing: 30% off with code HIVE-LAUNCH30 at https://hive80lab.gumroad.com

Free starting point: The First 30 Minutes — the one-page quick-start for any outage: https://hive80lab.gumroad.com/l/first-30-minutes