Every company has a backup. Almost none have a restore. The difference matters the day you need the second one, which is roughly 11 minutes before you lose the customer.
Our first restore test was a two-week project: dig out old scripts, find where the database actually lived, discover the backup bucket policy had rotated keys out from under everything. The backup was a theory. The restore drill made it a fact.
The index-card version
- One card, one drill. "Postgres nightly: restore last night's dump to a scratch box, run row-count check, record timestamp lag." If the drill doesn't fit on an index card, nobody runs it.
- A spreadsheet is the tool. Columns: date, what was restored, minutes taken, data age (how old was the newest record), pass/fail, who ran it. No ticket system, no dashboard. A spreadsheet gets filled in; a dashboard gets stared at.
- One cron job, zero meetings. The drill runs weekly on a scratch instance. The cron job emails the filled row to the team. Attendance at the monthly "backup review meeting" was zero; spreadsheet completion rate is 100%.
- The number that matters is data age, not backup age. A backup taken at 2am restored at 5pm is 15 hours of lost orders. That number on a visible spreadsheet is what finally got the replication gap fixed — not any amount of policy language.
What the first month found
Three of our five systems restored fine. One restore silently failed (the bucket had been renamed in a migration). One restored successfully but to a database nobody's app pointed at — useful for audits, useless for recovery. Two real defects, found by an 11-minute drill, in 30 days.
Your backup is a theory until you have restored from it under a clock, on a schedule, with the results visible to people who don't read policies.
Every drill in this post ships ready-made in the Ops Starter Kit ($29). All five kits in one download: Ops Mega Bundle ($49). Or let the always-on ops desk run them for you.
Top comments (0)