If you collect enough post-mortems — internal ones, or the public ones big shops publish — a pattern shows up fast: the 2 AM failure is rarely one broken thing. It is a handoff that lost three facts on the way from the person who knew to the person who acted.
We keep coming back to the same four fields. A handoff — between engineers, between agents, between shifts — survives if it carries all four, and dies if it drops any of them:
1. What broke (symptom, not theory). "Checkout 500s in EU, since 01:42" beats "probably the new deploy?" The person receiving the handoff needs the observable fact first. Theories second. Both labeled.
2. Blast radius. Which customers, which regions, which flows. Every escalation argument at 2 AM is really a blast-radius argument: is this worth waking the third person? You can't make that call if the handoff says "some errors."
3. First move that is safe. One command, one rollback, one documented step that makes things no worse. If your runbook doesn't name a first move for the alert that fired, the alert will page someone into improvising. Improvised 2 AM actions are how outages get interesting.
4. Who to call, and what to say. The escalation contact with a one-line summary they can read in ten seconds. Not "see the ticket." Nobody sees the ticket at 2 AM.
The uncomfortable part: these four fields are boring. They don't demo well. They are also the difference between an incident that ends in an hour and one that eats a weekend — because the second one usually starts as the first one, mishandled.
A quick test for your own rotation: take the last three pages that fired at night and ask, out loud, where each of the four fields lived. If any of them lived in "someone's head" or "chat scrollback," that is your gap. Fix it in the runbook next to the alert, not in a policy doc nobody opens.
We package this as ready-to-print incident cards, paging rules, and handoff templates: Agent Ops 24/7 on our storefront. Longer field notes land weekly on our Dev.to.
If you run a lean on-call rotation, steal the four fields wholesale — and reply with the field you think we missed. We read everything.
Top comments (0)