The kill-switch page that stops 60% of our 3am pages without waking anyone

The first hour of every on-call rotation at a small company is wasted on the same ten incidents. Disk full on the log host. Stuck queue consumer. Cache memory spike. The response is always the same four steps, and the engineer performing them at 3am is the same person who performed them yesterday.

So we wrote a kill-switch page: one runbook page per recurring incident, with the exact commands, in the exact order, written so that a cron job — not a human — can execute them.

The page has five rules

1. One incident class per page. Not "database issues" — "consumer lag > 60s on payments-queue." Specificity is what makes it automatable.
2. Preconditions first. What must be true before it's safe to run: replicas healthy, no deploy in progress, error rate below X. If a precondition fails, the script stops and pages a human.
3. The fix, as a script, not prose. kill -9 and restart, trim the log with the exact path, scale the pool. If you can't write it as a script, it isn't a kill-switch.
4. An undo. Every remediation page ships with the reverse command. This is the line that makes on-call trust it.
5. A success check. The script's last line verifies the customer-facing metric recovered, not the internal one. Lag returning to zero means nothing if checkout is still failing.

The safety valve

Every kill-switch script checks a global "is a human looking at this right now" file before it fires. On-call flips it when they want to investigate manually. It costs nothing and it prevents the one failure mode that kills trust: the machine "fixing" something while a human was mid-diagnosis.

We logged two quarters of pages and measured: 61% of after-hours pages now resolve via kill-switch before anyone's phone lights up. The remaining 39% are genuinely novel, which is exactly what a human should be woken for.

The hard part isn't the scripting. It's noticing that the same incident has happened thirty times and refusing to wake anyone a thirty-first time.

---

Every drill in this post ships ready-made in the Ops Starter Kit ($29). All five kits in one download: Ops Mega Bundle ($49). Or let the always-on ops desk run them for you.