A page is the most expensive message in your company: it costs sleep, and sleep is what keeps the next incident fixable. Every alerting setup drifts the same way — someone adds an alert because a thing once broke, nobody removes one, and within a year the on-call phone is a noise machine. When the real page finally fires, it arrives as the eleventh buzz of the night and gets swiped away like the rest.
The fix is a written paging bar. Full version with the worked example is on our ops notes. Core below.
1. Three response classes, one per alert
- Page — a human wakes now. Reserved for symptoms burning customer trust or money while everyone sleeps.
- Ticket — a human reads it in business hours. Where most "urgent" alerts belong: capacity trends, certs with 30 days left, replica lag without customer impact.
- Log — nobody reads it until asked. Fine, as long as you don't call it monitoring.
An alert with no assigned class is a ticket by default. The burden of proof is on the page.
2. The four-question page test
All four must be yes; three yeses = ticket with a fast SLA:
- Does it need a human decision now?
- Can a human actually fix it now? (Unfixable at night = ticket with a timer.)
- Will ignoring it make it worse? Self-healing = log line.
- Are customers feeling it, or about to?
3. What a page must carry
- One page, one symptom. Four thresholds in one page is four pages in a trench coat.
- Symptom, not cause. "Checkout error rate 4%" wakes the right person; "iostat latency high" wakes whoever knows iostat.
- Runbook link, mandatory. A page without a runbook is a riddle at 2 a.m. Link it or demote it.
- Anti-flap: 3 pages from one alert within an hour → the alert is auto-demoted to ticket pending tuning, and the demotion is announced. The alert is broken, not the on-call.
4. The sleep budget
Target: median ≤ 2 pages per person per shift. Two enforcement rules:
- Repeat-offender: an alert firing at the same hour three weeks running gets tuned or deleted. It's a subscription, not a signal.
- Night-page review: every overnight page gets 60 seconds at morning handoff — real or noise? Three noisy nights in a month means the bar moved.
Worked example
8-person SaaS, 41 alerts, 19 paging, 9 pages/week median — and the on-call had started muting the phone after midnight, which is how a real cache-layer outage waited 40 minutes for a customer email. One afternoon re-barring: 19 → 6 paging alerts (runbook links fresh, two demoted as unfixable-at-night), 11 to ticket with SLAs, rest deleted. Six weeks later: 1.4 pages/shift median, 100% of pages carried runbook links, and the "CPU > 80%" alert that never correlated with anything was gone. The next real page — a certificate chain break — was answered in 3 minutes, because the phone still meant something.
Metrics: median ≤ 2 pages/shift · ≥ 95% actionable · 100% runbook-linked · < 5% repeat pages within 24h.
Start here: The First 30 Minutes — a free incident quick-start checklist. When you need the full system, the Ops Starter Kit ($14) covers incident response, severity, and post-mortems for a 1–50 person team. Every template referenced in this post is free on HIVE80lab Ops Notes.
Top comments (0)