Most of the time an outage runs long, the bug was not the hard part.
The hard part is that four competent people saw the alert, and all four started working on it inside the same two minutes, in four different terminals, without telling each other. That is what turns twenty minutes into two hours. It happens on teams with good engineers, good dashboards and an expensive pager tool.
None of what follows is about tooling. It works the same whether you run Kubernetes or one VM, whether your alerts come from a serious observability contract or a cron job that curls a health endpoint. This is the process layer, and it is the layer most teams have never written down.
What goes wrong when everyone helps
Three failure modes, and they compound.
Collisions. Two people apply two mitigations inside the same minute. The graph recovers. Now nobody knows which change did it, so nobody knows what to put in the postmortem, and nobody knows what to do when it comes back on Thursday. The worse version: one person rolls back while another scales up, and the system is now in a state that has never existed in staging.
Unobserved changes. Someone restarts a node, flushes a cache, or bumps a connection pool, and doesn't say so. Ten minutes later a second engineer is staring at a metric that moved for a reason that appears in no dashboard. You are now debugging your own team.
The status tax. The person deepest into the problem is also the person everyone asks for updates. Every question costs them the thread they were pulling. On a bad incident, the most informed engineer in the company spends the outage writing chat messages.
Notice what those have in common. In all three, nobody decided who was doing what.
One person stops touching the keyboard
The fix is a role. Someone declares themselves incident commander, and then does not debug.
That second clause is the whole thing, and it is the part that gets dropped. The instinct of anyone senior enough to end up commanding is to open a terminal. If the commander is also investigating, there is no commander. There is a busy person with a title.
The commander's job for the first ten minutes:
- Say the incident exists, out loud, in one channel. "Declaring an incident on checkout failures. I'm commander." Ambiguity about whether this is even an incident is itself a cost.
- State the symptom in customer terms. Not "p99 is up on the orders service." Instead: "roughly a third of checkouts have been failing since 14:06." The customer-terms version is what later tells you whether you're done.
- Take a roll call. Who is here, and what is each person's one job. Said explicitly: "you own the theory that it's the deploy, you own the database theory, nobody else runs commands."
- Choose between mitigating and diagnosing. They compete for the same people. Pick one and say which you picked.
- Set a timebox before the first theory gets tested. "If we don't have a cause by 14:30, we roll back anyway."
- Log it as it happens, with timestamps.
Roles past commander are optional and depend on how many people you have. Above about four, split off a comms role whose entire job is telling everyone outside the incident what is happening, so the commander doesn't have to. If it's two of you at 3am, you're both wearing everything and the roll call takes four seconds. What matters is that the assignment happened, and that somebody said it somewhere the others could read.
Commander is not a rank. The most junior person on the call can command an incident that a principal engineer is investigating, and that works, because the commander isn't there to be the smartest person present. They're there to track who is doing what.
Mitigate before you understand
The rule engineers argue with most: restore service first, find the cause later.
It feels wrong. Rolling back without knowing what broke is an admission that you don't know what broke. But the last deploy is guilty often enough that "roll it back and watch" is a real diagnostic rather than a surrender.
During an outage the number that matters is how long customers are affected. Afterwards, the number that matters is why. Producing both at once mostly produces neither.
Two honest exceptions. If the mitigation could destroy data, say a rollback across a schema migration or a restore that overwrites newer writes, then understanding comes first, because that class of action doesn't reverse. And if you truly can't mitigate without knowing which of two subsystems is at fault, diagnosis is the mitigation path. Either way, say out loud which situation you think you're in, because the rest of the room is assuming the default.
One more thing about mitigation. The incident ends when someone has written down what temporary state the system is now in and who is unwinding it. A recovered graph is not that. Feature flags left off for four months are sediment from incidents that got declared over too early.
Severity levels that survive 3am
Most severity tables are arguments waiting to happen, because they are written in terms of cause. "Sev1: database failure." At 3am nobody yet knows whether it's the database.
Write them in terms of what a customer can observe, and put the response obligation in the same row so the level means something operational.
| Level | What's true | What it obliges |
|---|---|---|
| Sev1 | Core function broken for most users, or data at risk | Page immediately, commander named, comms out within 30 min |
| Sev2 | Core function degraded, or broken for a subset | Page in hours, commander named |
| Sev3 | Non-core broken, workaround exists | Ticket, next working day |
Three levels is usually enough. Five levels means the first ten minutes of every incident contain a debate about whether this is a 3 or a 4.
Two rules make the table work: anyone can declare, and severity can be revised down without embarrassment. On a team where declaring an incident reads as an accusation, people don't declare, and you find out about the outage from a customer.
The channel is the artifact
Write in one place, as it happens. Do it because the postmortem needs a timeline, and a timeline reconstructed from four people's memories the next afternoon is partly fiction.
The log line worth writing looks like this:
14:12 ROLLED BACK deploy 4471 on orders-api (all 3 regions).
Reason: error onset 14:06 matches deploy at 14:05.
Expect: checkout errors drop within 2 min if this is it.
Owner: Marco
Time, action, reason, expectation, owner. The expectation line is the one people skip and the one that pays for itself. It turns every action into a test with a stated prediction, so when the errors don't drop you have learned something, instead of merely having done something.
When this is too much process
For two people on a small product, most of the above is ceremony. The rough threshold: more than three people involved, or fifteen minutes elapsed, or any risk to customer data. Under that, one person saying "I've got this, shout before you touch anything" captures most of the value.
The more common problem is the opposite of ceremony anyway. It's a process that lives only in a wiki page nobody has opened since it was written. A one-page card people actually use beats a twelve-page runbook they don't.
The card
All of it, in the form worth printing and sticking to something:
FIRST TEN MINUTES
1. Declare it. One channel. Say the words.
2. Name a commander. The commander does not debug.
3. Symptom in customer terms, plus since when.
4. Roll call: who's here, one job each.
5. Mitigate or diagnose. Pick one. Say which.
6. Timebox it: "if not X by <time>, we do Y."
7. Log every action: time / action / reason / expected / owner.
8. Severity is revisable. Declaring costs nothing.
I'd rather hear where this breaks than defend it. The two parts I'm least sure travel: whether "the commander does not debug" survives a team of three, and whether anyone has ever written a severity table that people genuinely don't argue about at 3am.
So what do your first ten minutes look like? And if you don't have a first ten minutes, what happens instead?
Disclosure, since it should be out in the open: I write and sell guides for engineers, though nothing on incidents yet, so there's nothing to sell you at the bottom of this one. If the engineering-management end of my stuff is useful to you, the interview question bank is free. Otherwise, ignore this paragraph. Everything useful in this post is in the post.
Top comments (0)