Our DR plan was a 40-page PDF. The outage never opened it.

Our DR plan was a 40-page PDF. The outage never opened it.

The region went down at 6:40pm on a Thursday, and I can tell you exactly what the plan did: nothing. It was a beautiful document — 41 pages, a RACI matrix, color-coded failure scenarios — produced over three weeks for an insurance questionnaire. During the actual outage, nobody opened it. We opened a chat window and started improvising. The same question — "wait, where do the backups actually restore to?" — got answered three times, three different ways, and the two wrong answers cost us about two hours.

That's the dirty secret of small-team disaster recovery: the plan isn't what's in the folder. The plan is what the person who happens to be awake can reconstruct from memory, under pressure, with Slack and your monitoring provider possibly also on fire.

What we replaced it with

One page. Five fill-in blanks. It fits on a phone screen, which matters, because your laptop might be part of the disaster.

1. The trigger. Not vibes — a test. Ours: "primary region unreachable for 15+ minutes from two networks, OR database restore ETA exceeds 2 hours." If your trigger needs a human to debate whether it's bad enough, you wrote a debate, not a trigger.

2. The three numbers. RTO — how long until you're back, and it has to be a number you've actually achieved once, not a wish. RPO — how much data you can lose, which is purely a statement about backup frequency. And who's called, in order — with phone numbers, not Slack handles. Your chat platform is not a reliable notification system during the exact disasters it was meant for.

3. The failover path, as exact commands. "Restore from backups" is not a path. aws s3 cp s3://dr-bucket/latest.sql.gz . && psql < it && ./smoke-tests.sh is a path. If the person on call at 2am can't copy-paste it, it's a description of a path someone else used to have.

4. The degraded mode. This was our biggest miss. Should we pause orders? Reply to customers with a template? Nobody decided that in advance, so the most awake person decided it at 2am. A DR plan without a degraded mode makes full recovery the only acceptable outcome — and full recovery is the slowest one.

5. The drill. Quarterly, one person owns it, the result gets one log line. If the drill fails, it becomes a ticket with a date. That's the whole program.

The drill found what the document never would

First drill: restore to a scratch instance from the previous night's backup, run smoke tests, time it. Our RTO on paper was 4 hours. The actual restore? 3 hours 41 minutes of mostly waiting — plus we discovered the DR bucket was in the same region as production, which would have made the whole plan a joke if the outage had been a region outage. The document had survived three compliance reviews. The drill killed it in twenty minutes, on a quiet Tuesday, when killing it was free.

That's the deal with DR: the drill is the plan. The document is just where you write down what the drill taught you.

The rules that survived our incident

• One page or it's dead. Nobody reads 41 pages at 2am. The operational plan is one page; appendices can be long.
• It lives where the outage can't reach it. Printed, in the password manager, on two phones. Assume every tool you normally use is part of the disaster.
• Degraded mode is a business decision written in advance. "Pause orders" at 2am from the most awake person is not a decision, it's a guess with confidence.
• Every drill failure becomes a ticket with a date. A failed drill with no follow-up is a rehearsal of failure.
• DR changes go through the same change record as everything else. A stale runbook means DR was exempted from change control. It isn't exempt.

The free version

The full template — the five blanks, the filled-in example table, the drill procedure, the numbers that tell the truth — is on our ops notes site, next to the backup restore test and the incident response guides it plugs into.

If you'd rather not assemble it yourself: the Ops Starter Kit Vol. 2 ($27) includes the full continuity pack — the one-page DR plan, the restore drill card, the call list — plus the advanced incident response and comms templates. Launch week: 30% off any paid kit with code HIVE-LAUNCH30 at checkout.

And for the part of recovery that shouldn't need a human at all — nightly backups, WAL shipping, health checks, the drill's smoke tests running on schedule — the Automation Starter Pack ($19) covers the pick-first workflows that make recovery boring.

What did your last drill break? The one that made you rewrite the plan the same afternoon — I'm collecting those stories.