Every team has an escalation policy. Almost nobody has a tested one. The difference shows up at 3am, when the page goes out, the primary doesn't answer in five minutes, and the secondary discovers they're actually the tertiary — and the tertiary's phone number was rotated in March.
Here's the cheap version of the test that catches this before the outage does.
The 20-minute escalation drill (run it quarterly, on a Tuesday)
Minute 0 — pick the scenario. Not a fires drill. Pick your most boring real failure: disk full on the metrics host, cert expiry on the edge, the queue that backs up every Tuesday. Boring failures are what actually page you; test those.
Minute 5 — page the primary the real way. Not a Slack DM. The actual pager, the actual rotation, the actual first message. Watch the clock: how long from page to human ack? If your SLA says "ack in 5 minutes" and it takes 11, you just found your first gap.
Minute 10 — the primary hands off. They get one minute of context and must hand to the secondary using the actual handoff doc. If the handoff doc is "call Dave," and Dave changed teams in June, the drill ends here — with a finding, not an outage.
Minute 15 — the secondary executes the runbook's first step only. They don't fix it. They read step one and say whether they could do it at 3am. Ambiguity in step one of a runbook is a finding; nobody reads further in real incidents anyway.
Minute 20 — write the 5-field review. What page, ack time, handoff time, first-step verdict, one fix with an owner. That's the whole document. If your postmortem template takes an hour, people stop doing it, and the escalation chain silently rots.
What the drill almost always finds
- Ack times are fantasy. Teams assume minutes; reality is often "until I noticed the notification was muted."
- Contact data rots silently. Phone numbers, on-call apps, personal emails — every quarter, someone in the chain changed something. The drill is the only thing that checks the whole chain.
- Step one is written for the person who wrote it. "Restart the service" is not a runbook step unless it says where it runs and what counts as healthy after.
- The secondary doesn't have access. The worst finding: the escalation chain leads to someone who can't touch the system. Test access, not intent.
The honest cost
Twenty minutes per quarter, per rotation. Compare that to the 45-minute 3am scramble when the chain fails for real — and the customer-visible minutes that pile up while your pager routes to a dead end.
If you want this drill pre-written — escalation drill script, 5-field review template, handoff doc format, and the rest of the small-team ops kit — we packaged the field-tested versions: Ops Starter Kit. One download, instant, written by people who have run the 3am shift.
More field notes on real-ops for small teams: dev.to/hive80lab
Top comments (0)