We tested our backups for the first time in 2 years. The restore failed in 4 minutes.

We had 3 years of green backup checks and a restore that failed in 4 minutes. Here's the drill that fixed it.

The dashboard said everything was fine. Nightly job, green badge, 640 days of consecutive success. Then we ran our first real restore drill — sandbox, stopwatch, documented procedure — and the archive wouldn't even open. Wrong credentials. The ones in the runbook had been rotated in July, the vault path changed twice, and the only person who knew the manual sequence had left in March.

Nothing was lost that day, because nothing had happened yet. That's the trap: the failure only shows up during the disaster, which is the one moment you can't afford a failed restore. A backup you've never restored is a rumor, not a capability.

The 20-minute drill (quarterly, per critical dataset)

1. Pick the target before the clock starts. One service, one dataset — the one whose loss would hurt most. Restore into a sandbox, never production.
2. Pull from the source of truth. Download the archive from where the job actually writes it, not from a cached copy on someone's laptop.
3. Start the stopwatch at "disaster declared." The drill's output is a number: minutes to "data usable." No number, no drill.
4. Restore using the documented procedure only. If the runbook fails, that's a finding. Don't improvise — the 2am version of you only gets the doc.
5. Verify data, not process. Row counts, checksums, newest-record spot-check, one end-to-end action. Green job + corrupt data is the most common failure mode.
6. Log the result on one page. Date, operator, backup tested, declare→usable time, defects found, fix owner, due date.

What our first drill found

• Restore credentials in the runbook: dead since July (credential decay is finding #1 in nearly every first drill)
• Retention policy: 14 copies of archives nobody had ever opened — we paid storage for a rumor
• Time to usable data: 4 minutes to fail, 11 minutes after the doc fix — nowhere near the 60-minute promise we'd made to customers

Pass/fail criteria that keep it honest

• The time budget is a product promise. "Under 60 minutes" is measurable; "it eventually worked" is theater.
• Checksums get verified during the drill, not trusted from the backup job's own logs.
• One restore drill beats one retention audit. Prune what the drill proves is unreadable.
• The drill log feeds the next incident plan — the defects it lists are exactly the gaps that turn an incident into an outage.

---

If you want the full pack — restore drill log, backup policy one-pager, and the incident plan the drill plugs into — the Ops Starter Kit ($14) has it, and launch week takes 30% off any paid kit with code HIVE-LAUNCH30 at checkout.

The free checklist version of this drill is also on our ops notes site, alongside the incident-response and on-call templates.

And if the routine restore-adjacent chores (log rotation, backup verification pings, archive pruning) still need a human at 2am, that's exactly what the Automation Starter Pack ($19) picks-first workflows solve — same launch code covers it.