A backup you have never restored is not a backup — it's a wish with a cron job. Backup software reports the only metric it can see: "job completed." It cannot see whether the catalog matches the objects, whether the credentials still decrypt the vault, or whether restoring a Friday's worth of work takes forty minutes or four days.
The fix is small and boring: once a quarter, actually restore something, time it, and write down what was missing.
Why green backups still fail
The nightly job only tests the first link of a chain:
- The job runs, the vault doesn't open. The vault key was rotated and nobody updated the restore tooling. Jobs keep "succeeding"; the data is unreadable.
- The catalog is stale. The listing says the file exists; the object behind it was pruned by a retention policy running on different terms. Old-date restores fail selectively — on exactly the dates you'll ask for.
- Nobody knows where it lives. The one person who knew the passphrase and the portal left in March.
- SaaS data was never covered. "We have backups" meant a CRM export downloaded in 2023 and never opened.
- The restore target doesn't exist. The plan says "restore to clean hardware" but no machine was ever named — so the real incident starts with three hours of improvising.
Every one of these is invisible until you rehearse.
The drill: six steps
- Start the clock, open the runbook — not the chat history. Steps that exist only in someone's head are finding #1.
- List the catalog for a random date 30+ days back. Confirm the file set you expect exists. Catalog-vs-reality gaps are finding #2.
- Restore real content — a document tree, a database dump, a mailbox folder. Open it. Verify the data is current to that date. Existence isn't restoration.
- Prove the credentials work from the documented location.
- Stop the clock. Minutes from "go" to "verified content" — that number is your measured RTO, not the one on the slide.
- Write the log line. One dated row with a fix owner. It's the evidence your insurer and auditors actually want.
Ground rules: restore into an isolated target (sandbox folder, spare VM — never over the live copy), and use a sixty-minute timebox. If you're not done in an hour, that is the finding.
The five-column scorecard
One sheet, one row per drill:
| Column | Question it answers |
|---|---|
| What was restored | file server, mailbox, DB dump, SaaS export |
| Date restored to | the random past date |
| Minutes taken | your measured RTO |
| What was missing | credentials, docs, tooling — blank is a win |
| Fix + owner | name and date; row is done when the gap closes |
Four rows a year is a restorability record you can hand to anyone who asks "can you actually recover?"
The Tuesday scare (worked example)
09:40: the shared projects folder is missing half its client work — deleted since Friday, cause unknown. Because this team drilled in July, the response is boring: restore the folder tree from Friday's backup into an isolated directory in 22 minutes, discover the real gap is four hours (two folders were created after the last good job), write the scorecard row, assign the fix, done by coffee.
The un-drilled version of the same Tuesday ends with: "our backup vendor says the job has been failing since June." And if the deletion turns out to be malicious, the drill is also your rehearsal for ransomware recovery — attackers delete or encrypt backups first.
Paste-in drill log (CSV)
date,restored_what,restore_date,minutes,missing,fix_owner,fix_date
2026-03-12,fileserver/projects,2026-02-02,38,none,n/a,n/a
2026-06-11,mailbox@owner,2026-04-15,52,vendor-portal MFA reset,alex,2026-06-14
2026-09-10,crm pg-dump,2026-07-01,71,restore VM image stale,sam,2026-09-12
Where this fits
The drill is the proof step of your disaster recovery plan; the measured numbers belong in your downtime budget, and cron job monitoring keeps the job that feeds this drill watched — an unmonitored backup job fails silently for months.
The full one-page version — six failure modes, the scorecard, the drill log, and the isolated-target rules — is on the site: Backup Restore Drill Checklist for Small Teams.
If you want this as printable cards: The First 30 Minutes (free incident quick-start), Ops Field Cards — $4, Ops Starter Kit — $14, or the Ops Mega Bundle — $29.
Templates, not legal or insurance advice.
Top comments (0)