Your monitoring is talking. Nobody is listening. That is not a discipline problem — it's a design problem. Alert fatigue is what happens when a small team's pager sends more noise than signal, and the team does the rational thing: it starts ignoring everything, including the one alert that actually mattered.
The 3am rationality trap
Every ignored alert was, at some point, obeyed. The team responded, found nothing actionable, responded again, found nothing again — and then updated its priors: "these alerts don't mean anything." That's not laziness; it's Bayesian. If 30 alerts a week are noise, "ignore it" is the statistically correct response. The fix isn't telling people to care more. It's making alerts true.
Why fatigue happens (in order)
- Thresholds copy-pasted from tutorials. "CPU > 80%" means nothing on a box that idles at 75% and nothing on one that spikes to 95% every morning at backup time.
- Alerts without owners. If nobody is assigned, nobody responds, and the alert trains the whole channel to ignore.
- Alerts without actions. If the response to an alert is "look at the dashboard," it's not an alert — it's a nudged dashboard.
- Noise stacking. One flapping dependency can generate 400 alerts a week, each technically "real," each collectively meaningless.
The 10-line alert contract
Before any alert enters the pager, it must answer ten things — in the alert itself:
- SYMPTOM: what is observably wrong ("checkout API returning 5xx at 12%")
- IMPACT: what the user cannot do right now ("customers cannot pay")
- SCOPE: how many users/requests ("~120 checkout attempts in 10 min")
- STARTED: timestamp it began
- FIRST CHECK: the one thing to look at first ("GET /health on the payments pod")
- LIKELY CAUSE: ranked 2 guesses ("payments deploys; upstream Stripe webhook")
- DO: the first mitigation command, pasteable
- DO NOT: the known-wrong reflex ("don't restart the DB")
- ESCALATE WHEN: hard threshold ("no progress in 15 min → page [name]")
- CLOSE: what "fixed" looks like from the user side, not the dashboard side
If an alert can't fill these lines, it doesn't get to page a human. It goes to a dashboard, a weekly digest, or the trash. This contract does more for on-call sanity than any rotation scheme, because it kills noise at the source.
The noise audit (30 minutes, once a quarter)
Pull the last 90 days of alerts. For each distinct alert: did a human act within 30 minutes? Three buckets:
- ACTED and it mattered → keep.
- ACTED and it didn't matter → downgrade to digest or delete.
- IGNORED → delete on the spot. An alert everyone ignores is worse than no alert, because it spends the team's attention — your scarcest ops resource — at negative interest.
Then add one rule going forward: every new alert must ship with its 10-line contract filled in, and every alert that pages twice falsely gets auto-downgraded. Treat false pages like production bugs, because that's what they are — bugs in your monitoring code.
Where agents fit
If part of your stack runs unattended (scripts, cron jobs, AI agents), the contract gets stricter, not looser: an agent that "pages for everything" is just automated alert fatigue. The design goal is asymmetric — agents self-handle the routine, and the human pager fires only for the genuinely human decisions: customer-visible impact, money movement, data deletion, anything irreversible.
The litmus test
If you woke up to 6 alerts this morning and 5 were ignorable, your pager cost you 83% of your trust budget last night. Ten-line contract. Quarterly noise audit. Auto-downgrade on false pages. That's the whole fix — and it takes an afternoon.
This is one chapter of Agent Ops 24/7 — the solo operator's playbook for running agents unattended ($19): watchdog patterns, escalation ladders, and the monitoring design that pages a human only when a human is actually needed. Launch week: 30% off any paid kit with code HIVE-LAUNCH30 at checkout.
Free start: The First 30 Minutes — a one-page incident quick-start checklist, no email required.
Top comments (0)