DEV Community

Hive80-lab
Hive80-lab

Posted on

Your disk will fill on a Friday night. Here's the 20-line script that catches it early.

#monitoring #sre #devops #linux

Every SRE has one scar like this: pager silent all week, then Friday 11 PM — write failed: No space left on device. Not because disk fills are unpredictable, but because nobody watches growth rate, only thresholds.

A disk at 80% with 1 GB/day growth is a Friday problem. A disk at 95% with 10 MB/day growth is a Tuesday problem. Static thresholds can't tell the difference. Rate-based checks can.

The 20-line watcher 

#!/usr/bin/env bash
# disk_guard.sh — alert on GROWTH RATE, not static %
PART=/var
STATE=/var/tmp/.disk_guard_state
NOW_USED=$(df -B1 --output=used "$PART" | tail -1)
echo "$NOW_USED $(date +%s)" >> "$STATE"
PREV=$(tail -2 "$STATE" | head -1)
read -r P_BYTES P_TS <<< "$PREV"
# window: last entry is >= 60s old
[ $(( $(date +%s) - P_TS )) -lt 60 ] && exit 0
RATE=$(( (NOW_USED - P_BYTES) / ( $(date +%s) - P_TS ) ))   # bytes/sec
DAYS_LEFT=$([ "$RATE" -gt 0 ] && echo $(( (df -B1 --output=avail "$PART" | tail -1) / (RATE * 86400) )) || echo 999)
if [ "$DAYS_LEFT" -lt 7 ]; then
  echo "ALERT $PART: fills in ${DAYS_LEFT}d at $(numfmt --to=iec $RATE)B/s" \
    | tee /dev/stderr | mail -s "DISK: $DAYS_LEFT days left" oncall@yourco.example
fi
Enter fullscreen mode Exit fullscreen mode

Run it from cron every 10 minutes. Tune the 7-day window. That's it: you just converted every surprise disk-full into a scheduled ticket on Wednesday afternoon.

Why "days-until-full" beats "percent used"

  • Percent tells you where you are. Days-left tells you when you die.
  • Logs rotate in waves — your rate check naturally smooths them out.
  • One number, one action: under 7 days → investigate or grow the volume.

The same rate-window pattern works for memory leaks, log dirs, and queue depths. It's the single highest-value 20 lines most ops teams never write.

Steal the hardened version

We packaged this plus 40+ other copy-paste ops scripts (paging trees, alert budgets, Friday-rotation handoffs, silent-failure catchers) in:

🔥 Agent-Ops 24/7 Kit — monitoring + escalation + disk/log guards, deploy in an afternoon ($29).

📘 White-Label Runbook Kit — agencies rebrand the whole runbook set into their managed contracts.

📦 Ops Mega Bundle — every kit in one pack ($79, save 60%).

Put the cron line in tonight. Your Friday self sends thanks; your Monday self sends a raise.

Top comments (0)