Every ops team I have worked on has the same folklore: the one senior who "just knows" how to restart the payment gateway, drain the queue, or fail over the database. That knowledge feels like an asset. It is actually a liability, and it becomes obvious the moment that person is on a plane, asleep, or gone to another company.
I learned this the hard way during a Saturday outage where the only person who understood our certificate rotation was unreachable for six hours. We had monitoring, we had access, we had permissions. What we did not have was a written sequence of steps, so three of us guessed, and two of our guesses made things worse. The postmortem action item wrote itself.
Since then I hold a simple rule on my teams. If a task is important enough to page someone at 3am, it is important enough to have a runbook. Not a wiki page written from memory a year ago, but a tested document that someone who is not the expert can follow under pressure. The real test is not whether it reads well. It is whether a tired colleague who has never done the procedure can execute it correctly on their first attempt.
The trick is writing runbooks at the right moment. Right after an incident, while the pain is fresh and the exact commands are still in your shell history, is gold. Six weeks later you will have forgotten the flag that mattered and the order that could not be reversed. So I now treat "write the runbook" as part of closing the incident, not as a follow-up ticket that ages into irrelevance.
A good runbook also states what success looks like and what to do when a step fails. "Restart the service" is not a runbook. "Restart the service, confirm the health check returns 200 within 90 seconds, and if it does not, roll back using this exact command" is. The difference is whether the next person acts with confidence or with fear.
The goal is not documentation for its own sake. The goal is that no single human is a single point of failure. That is what resilient operations actually mean.
– Serguey Shinder
Top comments (0)