The escalation tree that ends 3 AM chaos (with copy-paste paging script)

At 3:11 AM the on-call phone lights up. No context, no notes, one line: "prod is down, someone look?"

What happens in the next twenty minutes decides whether the client renews or churns. Most teams lose those minutes to one missing artifact: an escalation tree everyone pretends exists.

Why escalation fails at night

• The tree lives in someone's head, not on a wall.
• Nobody knows who owns the incident at minute zero — so everyone waits.
• "Escalate" means wait longer, not page the next tier.
• No written handoff, so the morning shift re-diagnoses from zero.

The tree that actually works

Tier 0 — the responder. Isolate the blast radius first: one command, not a debate. Preserve evidence before restarting anything.

Tier 1 — the service owner. Paged automatically at minute 5 if the responder hasn't posted a status. No reply = auto-escalate. Waiting politely is how 20-minute outages become 4-hour outages.

Tier 2 — vendor/upstream. Paged at minute 15 with the evidence bundle already attached. You should never be hunting a vendor's support line mid-incident.

Every tier has one rule: acknowledge in one line, or the next tier fires. The tree ends when a human writes "I own this."

The timer script (copy-paste)

This is the spine of the tree — a cron-able escalation timer that pages tier 1 and tier 2 on dead air:

#!/usr/bin/env python3
"""escalate.py — dead-air escalation timer.
Usage: escalate.py --incident INC-1042 --tier1 [phone] --tier2 [phone] --timeout 300
Fires tier2 page if tier1 doesn't write an ack file within timeout seconds.
"""
import argparse, json, time
from pathlib import Path

ACK_DIR = Path("/var/lib/ops/acks")          # ack files land here from any channel
def acked(incident):
    return (ACK_DIR / f"{incident}.ack").exists()

def page(tier, incident, note):
    # wire this to your pager (Twilio, Slack webhook, PagerDuty Events API)
    print(f"[PAGE] {tier} for {incident}: {note}")

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--incident", required=True)
    ap.add_argument("--tier1", required=True)
    ap.add_argument("--tier2", required=True)
    ap.add_argument("--timeout", type=int, default=300)
    args = ap.parse_args()

    page(args.tier1, args.incident, "You own INC. Ack in writing within "
         f"{args.timeout}s or tier 2 fires.")
    t0 = time.time()
    while time.time() - t0 < args.timeout:
        if acked(args.incident):
            print(f"[OK] {args.incident} acknowledged; tree stops here.")
            return
        time.sleep(10)
    page(args.tier2, args.incident,
         f"Tier 1 dead air on {args.incident}. Evidence bundle: /var/lib/ops/incidents/{args.incident}/")

if __name__ == "__main__":
    main()

Pair it with an ack-writer (a two-line HTTP endpoint or even touch /var/lib/ops/acks/INC-1042.ack) and the "who owns this?" question disappears from every incident.

Make it someone else's problem (in a good way)

The teams that sleep well print the tree, laminate it, and hand it to whoever answers the phone. Agencies go further: they white-label the whole runbook kit and ship it inside their managed contracts — paging rules, shift handoffs, escalation trees, no per-seat fees.

🔥 Agent-Ops 24/7 Kit — the automated monitoring layer: monitors, escalation timers, silent-failure alerts. Deploy in an afternoon ($29).

📕 White-Label Runbook Kit — agencies rebrand it into their managed contracts.

📦 Ops Mega Bundle — every kit, checklist and script in one pack ($79, save 60%).

Print the tree tonight. The next 3 AM call is not a question — it's a schedule.