DEV Community

Hive80-lab
Hive80-lab

Posted on

Your backups have been failing for weeks and every checkmark says green

#backup #devops #sre #productivity

The lie of the successful backup

Ask a small team when they last restored something. The silence is the answer. Most "working" backup setups haven't restored a single file since they were written — and the scariest part isn't that restores fail. It's that the backups quietly stopped happening weeks ago, and every dashboard still shows green.

Backup jobs fail silently for boring reasons: an SSH key rotated and nobody told cron, a mount point moved, disk filled up, a password expired. The job crashes at 2am, nothing pages anyone, and the "last successful backup" date drifts into legend.

The 12-line freshness guard

Don't monitor backups. Monitor freshness — the age of the newest file the backup job actually produced:

#!/bin/bash
# cron: every hour
DIR=/backups/app          # where the job writes
MAX_HOURS=26              # daily backup = 26h grace
NEWEST=$(find "$DIR" -type f -mmin -1440 | head -1)
if [ -z "$NEWEST" ]; then
  curl -s "https://your-alert-hook/?backup=STALE&dir=$DIR" > /dev/null
  logger "BACKUPGUARD: no file newer than 24h in $DIR"
fi
Enter fullscreen mode Exit fullscreen mode

Why this beats checking the job:

  1. It checks the artifact, not the process. "The job ran" and "a backup exists" are different sentences. A job can run perfectly and produce zero bytes — the artifact check can't be fooled.
  2. One stale alert = investigate today. You catch it 24 hours after the last good backup, not 40 days later during the first restore attempt.
  3. It survives infrastructure drift. Keys rotate, paths change, servers move — as long as something lands in the directory on schedule, it's green. When nothing does, you know within a day.

The uncomfortable audit

Run this on every backup directory you own right now:

for d in /backups/*; do
  printf "%-30s %s\n" "$d" "$(find $d -type f -mtime -1 | wc -l | tr -d ' ') files in last 24h"
done
Enter fullscreen mode Exit fullscreen mode

Any zero is a fire already in progress. Most teams find at least one.

Then make it boring

One guard per directory is a good afternoon. The next level is a small library of these sentinels — backup freshness, disk headroom, cert expiry, hung jobs — deployed identically on every box, each with a one-page runbook so whoever gets paged fixes it in minutes instead of archaeology. Boring, repeatable, and it turns your 3am from an investigation into a copy-paste.

If you'd rather not assemble that library yourself, the full set is packaged at https://hive80lab.gumroad.com — guards, runbooks, and the wiring that makes them escalate instead of whisper.

Top comments (0)