DEV Community

Hive80-lab
Hive80-lab

Posted on

The disk-full alert fired 6 days late — here's the 20-line watchdog I run on every box now

#devops #sre #linux #monitoring

The worst outage I ever caused didn't start with a crash. It started with a warning email nobody read.

Our disk alerting had a threshold: 90%. The log volume on a scratch box crossed 90% on a Tuesday, stayed there, and kept growing in slow motion — 0.5 GB a day. By Sunday the database couldn't write its WAL and the whole thing fell over. Total time from "avoidable" to "page the whole team": six days.

The lesson: trend beats threshold. A threshold tells you the disk is full. A trend tells you the disk will be full — while you still have days.

The 20-line watchdog

This runs from cron every 15 minutes on every box I touch now. No agent, no SaaS, no dependencies:

#!/usr/bin/env bash
# disk-trend.sh — alert when a filesystem will be full in N days, not when it is.
FS="/"            # filesystem to watch
HORIZON=3         # days-ahead threshold
STATE=/var/tmp/$(basename $0).state

cur=$(df -k "$FS" | awk 'NR==2 {print $3}')
if [ -f "$STATE" ]; then
  read -r prev ts < "$STATE"
  hours=$(( ($(date +%s) - ts) / 3600 ))
  [ "$hours" -lt 1 ] && exit 0
  rate=$(( (cur - prev) / hours ))          # KB per hour
  free=$(( $(df -k "$FS" | awk 'NR==2 {print $4}') ))
  if [ "$rate" -gt 0 ]; then
    days=$(( free / (rate * 24 / 1) ))      # rough: KB free / KB-per-day
    if [ "$days" -le "$HORIZON" ]; then
      echo "ALERT: $FS full in ~${days}d at current growth (${rate} KB/h, ${free} KB free)"         | mail -s "[$(hostname)] disk horizon ${days}d" ops@example.com
    fi
  fi
fi
echo "$cur $(date +%s)" > "$STATE"
Enter fullscreen mode Exit fullscreen mode

Drop it in cron: */15 * * * * /usr/local/bin/disk-trend.sh. Two runs and it has a growth rate. Three and it has a forecast. It has caught every slow-motion disk since — including a runaway debug log that a 90% threshold would have missed for another week.

The other two patterns that page nobody

That watchdog is 1 of 3 in my "quiet killers" set. The other two, same philosophy:

  • Log-rotation amnesia — the rotate config exists but the service was never restarted, so the open file handle keeps growing. 5-line check: find /var/log -size +1G -mtime +1.
  • The cron that quietly dies — cron jobs whose last success date is older than their interval. The 2 a.m. silence pattern: an alert on absence, not on failure.

Most incident cost hides in the first 10 minutes after something was already knowable for days. Kill the knowable-days-early class of incident and on-call stops being a horror story.

Where the full set lives

I packaged the watchdogs, the runbook templates, and the escalation tables from my first 90 days of ops into four pay-once, download-tonight packs — no webinar, no upsell:

👉 Ops Starter Kit — runbooks, templates, checklists from the first 90 days
👉 Agent Ops 24/7 — the always-on agent monitoring loop as shipped configs
👉 Automation Starter Pack — the scripts that deleted the top-five recurring pages
👉 Ops Mega Bundle — everything, bundled

What's the oldest knowable-in-advance incident your team still hits? If it's a common one, tell me in the comments — I'll write that watchdog next and publish it free.

Top comments (0)