Nobody pages you for the alert you silenced. They page you when the disk fills, the cert expires, or the backup you never tested turns out to be an empty directory. Here are the three tiny scripts I keep in every server's crontab — and the drill that proves they work.
1. Disk canary — 60 minutes of warning, not 60 seconds
df -H | awk -v t=85 'NR>1 && $5+0 >= t {print $6" at "$5}' \
| while read p; do echo "DISK NEARLY FULL: $p" | mail -s "disk alert $HOSTNAME" ops@example.com; done
The point isn't the mail. It's that you learn about the log-spewing process while you can still delete one file, not when the database corrupts.
2. Cert expiry canary
echo | openssl s_client -servername example.com -connect example.com:443 2>/dev/null \
| openssl x509 -noout -enddate | awk '{print $4}' | xargs -I{} sh -c \
'd=$(( ($(date -d {} +%s) - $(date +%s)) / 86400 )); [ $d -lt 14 ] && echo "CERT EXPIRES IN $d DAYS"'
Let's Encrypt auto-renews — until it doesn't. This one-liner has caught three silent renewal failures for me.
3. Backup truth test
latest=$(ls -t /backups/*.tar.gz 2>/dev/null | head -1); \
[ -z "$latest" ] && echo "NO BACKUPS AT ALL" && exit 1; \
age=$(( ($(date +%s) - $(stat -c %Y "$latest")) / 86400 )); \
[ $age -gt 1 ] && echo "BACKUP STALE: $latest is ${age}d old"
A backup that's three days old is a rumor. This turns the rumor into a measurable number.
The drill that makes all three real
Scripts fail. Restores get skipped. Once a month, in a throwaway VM or container:
mkdir /tmp/restore-drill && tar -xzf "$latest" -C /tmp/restore-drill && ls -lhR /tmp/restore-drill | head -20
If you can't extract it and see your real files, you don't have a backup — you have a habit.
I packaged the full version of this drill — cron-ready canaries, a restore-drill checklist, and a runbook template — as App Monetization & Ops Playbooks on Gumroad. If you run servers for a living, the drill alone is worth the price of a coffee.
Test your restore this week. Future-you at 3AM is depending on it.
Running a stack like this? Our Ops Starter Kit ships the runbooks, checklists, and monitoring patterns behind these posts — grab it and cut your next incident short.
Top comments (0)