The restore drill: why your backups are worthless until you run this 20-minute test

Every team I've met has backups. Almost none of them have ever restored one under time pressure. A backup you've never restored is not a backup — it's a hope. Here's the drill that turned our hope into a fact, and it takes 20 minutes a month.

The drill

1. Pick the smallest real thing — a single production database, not a full region. Real credentials, real data, but contained blast radius.
2. Restore it to a fresh environment — new hostname, new port. Restoring in place proves nothing; you'll learn only that the files still exist where they already were.
3. Start a timer and an honest log — every step where you had to guess, look something up, or ask someone is a finding. "Where's the encryption key?" is a finding.
4. Point a real read at it — run the app's health check or one real query against the restored copy. A restore that returns rows nobody can read has failed.
5. Write down the RTO you just achieved — not the RTO in the policy doc, the one on the stopwatch. The gap between those two numbers is your incident plan's biggest lie.

What the first drill always reveals

The first one is always humbling. Ours found: the backup job silently skipped the largest table for six months (a schema change broke the include-list); the restore docs referenced a bastion host that had been decommissioned; and nobody on the team knew the S3 bucket key policy. Total real restore time: 3 hours 40 minutes against a policy RTO of 30 minutes. The second drill, one month later: 41 minutes. The third: 28.

Make it a rhythm, not a project

• Monthly, calendar-blocked, 20 minutes. Rotating owner so knowledge isn't one person's head.
• Keep the drill log next to the runbook. When the real outage hits, you run the procedure you rehearsed, not the one you wrote.
• When a drill finds a gap, the fix ships that week. A gap found in a drill and not fixed is worse than not drilling — it's documented false confidence.

Insurance you've never claimed on is still insurance. A restore drill is the claim form, filled out in advance, in pencil. Twenty minutes a month is the cheapest uptime you will ever buy.

---

Every drill in this post ships in the Ops Starter Kit ($29). All five kits in one download: Ops Mega Bundle ($49). Or let the always-on ops desk run them for you.