Alert fatigue isn't a tooling problem. It's a math problem: every noisy alert you keep taxes the attention you need for the one that matters.
Last quarter I pulled 100 consecutive pages from a mid-size on-call rotation and scored each one on two axes: was it actionable? and did it require a human right now?
The numbers
| Verdict | Count |
|---|---|
| Actionable + urgent | 11 |
| Actionable but deferrable | 7 |
| Noise (self-healed / duplicate / dashboard check) | 82 |
82% of the pager's inventory was spent attention on nothing. Here is the exact checklist we used to claw it back.
The 4-question alert audit (run on every alert ≥3 fires/week)
- "If I ignore this for 30 minutes, does anything bad happen?" If no → demote to dashboard or daily digest.
- "Does this alert tell me what to DO?" An alert without a runbook link is a riddle, not a page. Riddles go to Slack, not SMS.
- "Would a machine have closed this ticket?" Retries, disk cleanup, rolling restarts — automate the remediation, keep the record, delete the page.
- "Did the last 5 instances resolve on their own?" Then it is a trend, not an incident. Trend-lines belong in weekly reviews.
What to do with the noise
- Auto-remediate the top 10 self-healing alerts. Most are cron-jobs and restarts you can script in an afternoon.
- Batch the deferrables into one daily "drift report" email.
- Budget pages like money: if a service pages more than twice a week, it is spending more than it earns. Cap it.
The result after one audit pass: pages dropped 70%+ and mean time to acknowledge improved, because real pages stopped hiding in the noise.
The boring truth
Nobody ships alert hygiene because it isn't shiny. But it is the highest-leverage afternoon you can spend on your rotation. We packaged the scripts, runbook templates, and the full audit checklist we use into a starter kit — grab it here if you want to skip the blank-page problem:
You'll get: alert-audit checklist, 10 bash auto-remediation scripts, incident runbook templates, and the paging budget spreadsheet.
If this saved you a page tonight, follow me — I write one ops-automation teardown every week.
Top comments (0)