The 3AM Incident Playbook: What Senior SREs Do Before the Pager Fires

Most on-call engineers react to incidents. Senior SREs pre-act — they have a playbook that runs before the pager fires. Here's the playbook structure we run in production, distilled from hundreds of post-mortems.

The 3AM Rule: Every Alert Must Answer 4 Questions

When an alert fires at 3AM, the responder should never have to think. The alert itself must carry:

1. What user-facing surface is degraded (not "CPU is high" — "checkout is failing")
2. The last known-good state (when did it last work?)
3. The one command to run for a safe first response
4. The escalation path if the first response fails

If your alerts don't answer these, you don't have monitoring — you have noise.

The Pre-Incident Checklist (run weekly, not during incidents)

1. Alert Fatigue Audit

Pull your alert counts from the last 30 days. Any alert that fired >5 times with zero action taken is noise — delete it or make it actionable.

# alerts that fired but nobody acked = noise
noisy = [a for a in alerts if a.fired_count > 5 and a.actioned_count == 0]

2. Runbook Gap Check

For every service, ask: "If this dies at 3AM, what command do I run?" If the answer involves reading source code, you have a runbook gap.

3. Restore Path Drill

Once a week, actually run your restore path in staging. A backup you've never restored is Schrödinger's backup.

The 4-Phase Incident Response (what actually works)

Phase	Action	Time budget
Detect	Alert fires with the 4 answers	0 min
Triage	Classify: Sev1 user-facing / Sev2 internal	5 min
Mitigate	Safe first response: restart, rollback, failover	15 min
Verify	Synthetic check confirms recovery	10 min

The killer mistake: jumping to root-cause during Phase 3. Mitigation first, diagnosis later. Users don't care why checkout is down; they care that it's up.

The Post-Incident Loop That Actually Improves Reliability

Most post-mortems die in a doc nobody reads. The fix: every post-mortem must produce exactly one automated check. Not action items — code. A check that would have caught this incident, running forever.

Ops Automation Starter Kit

If you're building out your monitoring/ops playbook, we packaged the foundation — runbook templates, alert hygiene scripts, and the incident response checklists above — as the Ops Starter Kit.

It's the toolkit version of this playbook: copy the templates, plug in your services, stop improvising at 3AM.

What does your incident response look like? Drop your worst 3AM story in the comments.