Yesterday's piece ended with a test: four items, PASS / FAIL / UNKNOWN, UNKNOWN counts as FAIL. It was a good test as far as it went. It had a hole, and the hole is the whole subject of today.
The test asks whether the box contains what a successor needs. It does not ask whether anyone has ever opened it. Those are different questions, and only one of them is answerable from the outside.
Two states that look identical
Consider two companies on the day before they need continuity.
Company A has an escrow: source, build environment, credential rotation runbook, a documented core. Everything passes. The box has never been opened since it was sealed two years ago.
Company B has a folder named escrow-2024 containing a tarball of a repository that has since been rewritten twice, and three PDFs about a vendor they no longer use.
From the outside — from the perspective of the buyer doing due diligence, the auditor filling in a questionnaire, the founder answering "yes, we have that" — these two companies are indistinguishable. Both say the right sentence. Both have a file. Neither has evidence.
They become distinguishable on exactly one day: the day it matters. And on that day, the difference is not a matter of degree. Company A discovers, over the following six hours, which of its four items were true. Company B already knows.
The purpose of a rehearsal is to move that discovery to a Tuesday afternoon when nobody is on fire, and to spend it in an environment where the wrong answer costs four hours instead of a customer.
Why nobody rehearses
The economics are genuinely bad, which is why this is rare and why "we should do a drill" survives in the same drawer as "we should write tests for the legacy service."
- The cost is paid now and is certain. Two engineers, half a day each, a scratch environment, someone to babysit the credentials.
- The benefit is probabilistic and lands in the future. You are buying the difference between a four-hour recovery and a four-day one, on a day that may not come this year.
- The benefit is unattributable. When the drill goes well, nothing breaks. Nobody gets credit for the incident that didn't happen, or for the two days of downtime that were actually three hours. There is no artifact that says "this was the rehearsal paying off."
That third point is the real blocker, and it's solvable. A rehearsal can be made to produce a thing that a person can hold up: a diff between what your documentation says and what your infrastructure does. That diff is the deliverable. It exists whether or not an incident ever comes.
Four drills, in ascending order of discomfort
Each one is useful on its own. Do not skip to four.
Drill 0 — Open the box and read it. No execution. Someone who has never touched the escrow retrieves it, lists what's inside, and writes down what's missing using the same four-item test. One hour. The usual result: the source is there, the build environment is a sentence, the rotation runbook is a screenshot from 2023, and the "core 20%" document describes a service that was deprecated.
Drill 1 — Restore into a scratch environment. Not production. A disposable box, a fresh account, no access to the original infrastructure. Restore, build, start. The rule that makes this a drill rather than a demo: any step that requires a human to remember something is a failure. Not a friction point — a failure. If the restore only works because the person who wrote it was in the room, you have measured the person, not the box.
Drill 2 — Rotate one credential through the documented path. This is the one that separates documentation from authority. Your runbook says: "issue new key, update the three consumers, revoke old, verify." Doing it for real answers a different question: can the person on call actually do this at 3am without waking anyone? Often the answer is no — the vault policy requires an approver who is on holiday, the API token needed to mint tokens is itself expiring, the consumer list in the runbook is missing the fourth consumer that was added in March. Drill 2 is where organizations discover that their recovery plan has an approval chain, and that the approval chain is a single point of failure with a nicer name.
Drill 3 — Hand it to someone who didn't build it. The successor test. A different engineer, given only the box and a one-line goal, attempts the restore. No hints, no "oh just ask Priya." This is the only drill that tests the artifact instead of the author, and it is the one that predicts what actually happens when a team loses the person who understood the system. Expect to fail. The failures are the highest-value output in the whole exercise, because they are exactly the failures your successor will hit, minus the incident.
What counts as passing
A drill passes when it produces evidence, not when it produces no errors.
Three artifacts, or it didn't happen:
- A timestamped log of what was done, in order, including the parts that were improvised.
- A named human who did it — not a team, not a role. Names make the absence of drills visible next quarter.
- A doc-reality diff. Every place where the runbook and reality disagreed, written down, with the wrong version marked. This is the piece people skip, and it is the piece that has value on a normal day: your runbook is not documentation of your system, it is a record of what someone believed about your system at some point.
The corollary: an exercise that cannot fail is not a test. If the drill is run in the one environment where all the credentials still work and the person who built everything is watching, it will pass, and it will tell you nothing. The value is concentrated in the awkward parts — the missing approver, the expired token, the consumer nobody remembered. Design for those to surface.
The cheap version
Two drills a year is a realistic target, and the reason most teams do zero is that they imagine the expensive version.
The cheap version costs four hours, twice a year, and looks like this:
- Q1: Drill 0 + Drill 1. Read the box. Restore it somewhere disposable. Write the diff. Half a day, two people.
- Q3: Drill 2 + Drill 3. Rotate one credential for real, out of hours, with the on-call person doing it from the runbook. Then have someone new restore from the box alone. Half a day, three people including the observer.
- Every drill: the diff goes into the repo next to the runbook, dated. Next drill starts by reading the previous diff, so failures either get fixed or get re-confirmed as accepted risk. An unfixed, re-confirmed failure is a decision. An unfixed, forgotten one is a surprise waiting.
Four hours twice a year is roughly the cost of one bad afternoon of incident response — except you get to choose which afternoon.
The uncomfortable part
Rehearsal tests the box, and the box is a snapshot. The moment the drill ends, reality starts diverging again: a new consumer, a new credential, a dependency that moved. This is why one drill is not a program, and why the diff matters more than the pass. The diff is the only thing that tells you the rate at which your continuity decays.
Every organization I've seen pass this test had the same property, and it wasn't better documentation. It was that someone had, on a calm day, deliberately tried to break their own recovery path and written down what broke. The ones that failed had a file, a sentence, and a confident answer.
An escrow you have never opened is a hypothesis. A rehearsal is the only thing that turns it into a plan — and the only thing that tells you how much of the plan has already expired.
Next: the drift layer — what happens between two rehearsals, and why the box you validated last quarter is a description of a system that no longer exists.
Top comments (0)