We test our technical failover regularly and we are good at it. What we had never tested was the part that happens before anybody touches a keyboard, which is the sequence of phone calls that turns an alert at an antisocial hour into a person who is awake, informed and allowed to act.
So we ran the call, unannounced, at ten past two on a Wednesday. It took fifty-one minutes to assemble the four people the runbook said we needed.
Every delay was ordinary. The second-line number in the escalation document belonged to an engineer who had moved to another team in 2023 and had politely stopped being on call. The storage vendor's contract says twenty-four hours a day, which is true, and the severity that unlocks a response out of hours is assessed by their service desk against their own definitions, so our first call produced a ticket rather than an engineer. The bridge we use for major incidents required a licence that two of the people on the list did not have. The duty manager's phone was on silent, correctly, because his escalation depended on a paging system that had been decommissioned with the tool it came with. And nobody on our side was confident they were permitted to wake a director, which meant everyone waited for somebody else to decide.
None of this appears in a technical test, because the estate behaved perfectly throughout.
The contact register now has a single owner and is verified quarterly by calling every number on it rather than reading it. Supplier escalation is rehearsed once a year, in their words, so we know which phrase produces an engineer and which produces a ticket. Every role has two named deputies. The authority to wake senior people is written down, by name, so that nobody spends twenty minutes being considerate. There is a printed card, because the directory is exactly what we may not have.
We measure one number now: from alert to a person who can make a decision. It was fifty-one minutes. It is eleven.
An estate that recovers in minutes is still down for an hour if nobody has picked up the phone.
– Serguey Shinder
Top comments (1)
Impressive, 51 minutes down to 11. And "alert to a person who can make a decision" is exactly the right metric.
On our side it's the opposite. I'm on call, and when an alert fires at night I'm the one who gets up, alone. If I can't fix it myself, my job is to limit the impact until the team arrives around 9am. No bridge to assemble: the time to a decision is the time it takes me to wake up.
Out of curiosity: what are the four roles in your runbook, and can the first person who picks up act on their own?