A green backup tick means nothing: the 5-minute script that proves your backups actually restore

I once inherited a client whose backup dashboard was green for 11 months. Beautiful green squares, daily emails: "Backup completed successfully."

Then their file server died. We opened the backups. Every single archive since March was a 4KB stub. The backup job had been "successfully" backing up an empty folder for almost a year.

That day I adopted one rule, and I've never broken it since:

A backup that has never been restored is a rumor, not a backup.

If you run ops for a small team — or you're the one person everyone calls when "the server is slow" — this post gives you the exact script I now run on every client's estate. It takes 5 minutes and it has caught three more silent failures since.

The 3 ways backups lie to you

1. The empty-source backup. The job points at a folder someone moved/deleted. It "succeeds" — 4KB of nothing. (My client above.)
2. The never-expires archive. Backups pile up until the disk fills, then new backups silently fail while old ones look healthy.
3. The untestable tarball. 40GB of .tar.gz nobody has ever opened. Could contain data. Could contain a corrupted stream from a failing disk. Nobody knows.

All three produce green ticks. All three fail on restore day.

The restore-first check (the script)

The only test that matters: pick the newest backup, restore it to a temp dir, prove real files came out, clean up. Run it daily from cron. Here's the whole thing:

#!/usr/bin/env python3
"""restore_drill.py — prove the newest backup actually restores. Run daily."""
import tarfile, tempfile, shutil, sys, time
from pathlib import Path
from datetime import datetime

BACKUP_DIR = Path("/backups")          # where your archives land
MIN_FILES   = 10                        # a real backup has content
MIN_BYTES   = 1024 * 1024               # and at least 1MB restored
REPORT      = Path("/var/log/restore_drill.log")

def newest_archive(d: Path):
    arcs = sorted(d.glob("*.tar.gz"), key=lambda p: p.stat().st_mtime)
    return arcs[-1] if arcs else None

def drill() -> bool:
    arc = newest_archive(BACKUP_DIR)
    if arc is None:
        return fail("no archives found — backup job is dead or pointed nowhere")
    age_h = (time.time() - arc.stat().st_mtime) / 3600
    if age_h > 26:
        return fail(f"newest backup is {age_h:.0f}h old — last night's run FAILED silently")
    tmp = Path(tempfile.mkdtemp(prefix="restore_drill_"))
    try:
        with tarfile.open(arc, "r:gz") as t:
            members = [m for m in t.getmembers() if m.isfile()]
            if len(members) < MIN_FILES:
                return fail(f"only {len(members)} files inside — empty-source backup?")
            # extract up to 200 files as a sample restore
            for m in members[:200]:
                m2 = t.extractfile(m)
                if m2 is None:
                    continue
                out = tmp / m.name
                out.parent.mkdir(parents=True, exist_ok=True)
                out.write_bytes(m2.read())
        bytes_out = sum(p.stat().st_size for p in tmp.rglob("*") if p.is_file())
        if bytes_out < MIN_BYTES:
            return fail(f"restored only {bytes_out} bytes — contents are stubs")
        # spot-check: newest file inside must be recent, else you're backing up a ghost tree
        newest = max((p.stat().st_mtime for p in tmp.rglob("*") if p.is_file()), default=0)
        if newest < time.time() - 7*86400:
            return fail("nothing inside was modified in 7+ days — you may be backing up a stale tree")
        ok(f"RESTORE DRILL PASSED: {len(members)} files, {bytes_out//1024}KB restored from {arc.name}")
        return True
    except (tarfile.TarError, EOFError) as e:
        return fail(f"archive corrupt: {e}")
    finally:
        shutil.rmtree(tmp, ignore_errors=True)

def ok(msg):
    log(f"OK   {datetime.utcnow():%Y-%m-%d %H:%M} {msg}")

def fail(msg):
    log(f"FAIL {datetime.utcnow():%Y-%m-%d %H:%M} {msg}")
    return False

def log(line):
    print(line)
    REPORT.parent.mkdir(parents=True, exist_ok=True)
    with REPORT.open("a") as f:
        f.write(line + "\n")

if __name__ == "__main__":
    sys.exit(0 if drill() else 1)

Note what it checks beyond "can I open it": archive freshness (the 26-hour tripwire), file count (empty-source trap), restored byte volume (stub trap), and content recency (the ghost-tree trap). Exit code 1 → your monitoring pings you. Green tick with no drill is worth exactly nothing.

The cron line

30 6 * * * /usr/local/bin/restore_drill.py || mail -s "RESTORE DRILL FAILED" [e-mail] < /var/log/restore_drill.log

One line, every morning, before standup. The whole drill runs in seconds because it samples 200 files instead of restoring 40GB.

The 5-point backup honesty checklist

Print this. Tape it to the backup server.

• [ ] Newest archive < 26 hours old
• [ ] Archive contains ≥ 10 real files
• [ ] Sample restore produced ≥ 1MB of real bytes
• [ ] Newest file inside was modified this week
• [ ] Someone received a restore drill email this month — not just a backup success email

If you can tick all five, you're ahead of most small IT shops I've audited.

What I ship to clients

This drill is now a permanent cron entry on every client box I manage, and it's part of the ops kit I built after one too many 2am "the backup was green!" conversations. If you want the full set — the drill, the on-call pager test, the overnight no-surprise check, and the templates I hand to clients — I packaged them:

• Ops Starter Kit — $14 — 20+ battle-tested scripts + runbook templates (code LAUNCH50 for 50% off this week)
• Automation Starter Pack — $19 — the cron/monitoring/reporting automation layer

Backup day is a terrible day to learn your backups don't work. Run the drill tomorrow at 6:30. Future-you, standing in front of a dead file server, will send thanks back in time.