Every IT team says "we have backups." Almost none have proved it in the last 90 days. Then the restore fails, and the meeting gets very quiet.
We started doing a formal restore drill every quarter after watching a client lose 3 days of work to a "successful" backup job that had been silently restoring garbage for months. Here's the drill that fits in one afternoon.
The 5-step drill
1. Pick your worst-case file. Not a text file. The largest, most complex thing you own — a production database dump, a 40GB mailbox, a Terraform state file. If the drill only tests the easy 1KB files, it's theater.
2. Restore to a clean environment. Never restore over the live copy. Spin up a scratch VM or container, restore there. This catches the "backup is fine but my restore target is gone" class of failure nobody talks about.
3. Time it against your RTO. You wrote an RTO in your DR plan. If the restore takes 14 hours and your RTO says 4, you don't have a DR plan — you have a wish. Record the real number.
4. Verify content, not existence. Open the file. Run the app against the restored DB. Checksum before/after. A file that exists but is corrupt is worse than no file, because it makes everyone confident.
5. Document what broke. The drill's output is a list of missing pieces: runbook steps, credentials, network paths, dependencies nobody wrote down. Fix one per drill. In a year you'll have a restore process that actually works under stress.
What our drills caught
- A backup job running as a user whose password expired (backups failing for 6 weeks, alerts going to a dead mailbox).
- A 2TB dataset that "backed up" nightly but had never once been restored — restore tooling choked on the format.
- A DR runbook that referenced a server decommissioned a year earlier.
None of these show up in backup dashboard green checks. All of them show up in a drill.
Make it repeatable, not heroic
The drill only works if it's scheduled and scripted. We keep a drill checklist, verification scripts, and a results template in one place so any team member can run it — not just the person who set up the backups. If you want the same kit, the Ops Starter Kit ($14) has the restore-drill checklist and verification scripts, and the Agent-Ops 24/7 kit adds the automated monitoring layer that tells you when a backup silently stops running. Everything ships as plain scripts you run on your own machines.
Question for the comments: when did you last restore, not back up — actually restore? Be honest, it's anonymous.
Top comments (0)