Your on-call engineer got 47 pages last night. Fourteen of them meant nothing. Two meant everything. Nobody slept, and the two that mattered were almost buried in the noise.
That's not a monitoring problem. It's an alert fatigue problem — and it's the most expensive silent tax on a small ops team. Here's how to fix it in one afternoon.
Step 1: The Silence Audit (30 minutes)
Export your last 30 days of alerts into a spreadsheet. For each alert, write three columns:
- Action taken: what did the human actually do? ("restarted it", "nothing, it self-healed", "ignored it")
- Would missing it have hurt? yes / no / probably
- Seconds to triage: how long to decide it was noise?
Every alert where "action taken" is "nothing" or "ignored" and missing it wouldn't have hurt is a false contract: your monitoring promised danger and delivered nothing. Kill it or demote it to a dashboard metric.
Step 2: The 3-Severity Rule
Everything you page on must pass one test: "If this fires at 3 AM, will you get out of bed?" If no, it doesn't page. Small teams need exactly three severities:
- PAGE (phone rings): customer-facing money path down or about to be. Human needed NOW.
- TICKET (next business day): something is degrading but users are fine.
- DASHBOARD (informational): someone will look at this during business hours anyway.
Most "monitoring setups" have ten severities in practice and one real one. Collapsing to three usually cuts page volume by 60-80% the first week — and the pages that remain finally get respect.
Step 3: The Re-Page Guard
Two rules that end 90% of night noise:
- No page repeats within 15 minutes for the same root cause. If disk-fill fires 6 times as the disk fills, that's 1 page + 5 insults. Group by cause, not by symptom.
- Every page must link to a runbook. A page without a copy-pasteable first action is a riddle, not an alert. If the runbook doesn't exist, the alert isn't ready to page.
Step 4: The Weekly Flap Review (10 minutes, Friday)
Every Friday: top 5 noisiest alerts of the week. For each — fix the threshold, fix the flaky check, or delete it. Noise you allow this week becomes the noise you sleep through next week, when it's real.
I packaged this whole loop — the silence-audit spreadsheet, severity decision tree, and re-page guard runbook templates — into the Ops Starter Kit. If you want the monitoring half automated (alerts wired to runbooks automatically), the Automation Starter Pack covers that side. And HIVE-LAUNCH30 takes 30% off the bundle this week.
How many pages did your team get last night — and how many were real? Tell me in the comments.
Top comments (0)