Our pager used to go off 40+ times a week. After applying four rules, it went off twice last month — and both were real.
If you run on-call for a small team (or are the whole team), these are the rules that actually moved the number.
Rule 1: Every alert must be actionable, or it dies
"CPU over 80%" is not an alert. It's a number. If an alert fires and your first thought is "huh, that again," it's noise — and noise trains you to ignore the fire when it's real. Delete it or make it actionable.
Rule 2: Severity tiers, or everything is screaming
One severity level means every page feels like 3am. Ours:
- SEV1: customers can't do the thing → page now, all hands.
- SEV2: degraded, workaround exists → slack message, fix within hours.
- SEV3: cosmetic / internal-only → ticket, fix this week. Most of what pages you today is honestly a SEV3 wearing a SEV1 costume.
Rule 3: Runbooks next to the alert
An alert without a runbook is a riddle sent to a tired person at 2am. Every page links to: what this means, first 3 checks, escalation path, and the fix that worked last time. Thirty minutes of writing saves dozens of 2am spirals.
Rule 4: Alert budget — 5 per week, hard cap
If we exceed 5 pages in a week, the NEXT retro is mandatory and something gets deleted. Sounds extreme. It's the only thing that made us prune for real.
The uncomfortable math
Alert fatigue isn't a tooling problem, it's a standards problem. Better alerting tools won't save a team that pages itself for CPU spikes.
If your on-call rotation needs the same rebuild, we packaged the system: severity matrix templates, runbook formats, the 2am checklist, and the alert-budget retro script — Agent Ops 24/7 kit. Buy once, deploy this week, no subscription.
Full catalog of ops/automation kits: Hive80Lab store.
What's the dumbest recurring alert you've killed? Comments open.
P.S. The best on-call engineer is one who sleeps through the night because the alerts are actually good.
Running a stack like this? Our Ops Starter Kit ships the runbooks, checklists, and monitoring patterns behind these posts — grab it and cut your next incident short.
Top comments (0)