There's a specific kind of failure that keeps me up at night, and it isn't the outage. It's the outage that sat unseen because the alert announcing it was buried under two hundred others that didn't matter.
Most teams treat alert fatigue as an annoyance: a noisy channel, a phone that buzzes too much. It isn't an annoyance. It's a reliability risk. Every alert that fires without needing a human response is a small withdrawal from the same account: the team's trust that a page means something. Drain that account far enough and people start muting, skimming, and reflexively acknowledging without looking. At that point your monitoring is still running, still green on the dashboard, and functionally blind. The danger was never the noise itself. It's what the noise trains people to ignore.
After years of keeping production systems observable, I've come to think the goal of good alerting isn't fewer alerts for their own sake. It's a team that trusts every page, because every page is real. Here's how I get there.
Why noise accumulates
Noise is rarely one bad decision. It's the compound interest of small, reasonable-at-the-time choices that nobody revisited. A few patterns show up almost everywhere:
Static thresholds that outlived their context. Someone set "alert if latency > 800ms" during a launch two years ago. Traffic patterns changed, the baseline moved, and the threshold quietly became either a constant false alarm or a rubber stamp that never fires. Thresholds decay. Most teams never schedule the review.
Alerting on causes instead of symptoms. CPU at 90% pages someone. But 90% CPU is not a problem. It's a machine doing work. The problem is a user getting an error or waiting too long. When you alert on every internal cause, you generate a page for every twitch in the system, most of which resolve themselves before anyone reads the notification.
No severity tiering, so everything pages. When the informational, the interesting, and the genuinely urgent all arrive through the same channel with the same urgency, the person on call has to sort them out in their own head, doing the triage the alerting system should have done for them. That cognitive tax is where fatigue actually lives.
One incident wearing three costumes. A single upstream failure lights up the load balancer, the application layer, and the database monitor as three independent alerts. The responder now spends their first five minutes proving to themselves that it's one incident, not three, during the exact window when minutes matter most.
None of these come from carelessness. They come from adding monitoring faster than anyone prunes it.
The reframe: alert on symptoms, not on every metric
The shift that changes everything is deceptively small. Stop asking "is this metric abnormal?" and start asking "is a user being hurt right now, and can a human do something about it?"
That second question is a much higher bar, and it's the right one. It moves your alerting off the internals of the system and onto the experience of the people using it. An abnormal metric is a curiosity. A hurt user with an actionable cause is a page.
This is where Service Level Objectives earn their keep. When you define what "working" means from the user's side (a success rate, a latency ceiling, an availability target), you get a principled way to decide what deserves human attention. You're no longer alerting on raw metrics; you're alerting on whether you're spending your error budget faster than you can afford to. A slow burn against your budget is a ticket to look at tomorrow. A fast burn that will exhaust the month's budget in an hour is a page tonight. Same underlying data, completely different urgency, and the difference is decided by user impact, not by a number crossing a line.
A practical example
Consider a checkout service. A traditional alert might look like this:
Alert: CheckoutHighErrorRate
Condition:
http_requests_total{status=~"5.."} > 100
Duration: 5m
Severity: Critical
The problem is that raw error counts say nothing about customer impact. One customer stuck in a retry loop can generate hundreds of errors. Fifty different customers each failing once is a much bigger business problem, and this alert can't tell the two apart.
A better alert ties together user impact and SLO burn:
Alert: CheckoutSLOBurn
Condition:
burn_rate > 2
AND affected_customers > 50
AND duration > 10m
Severity: Critical
Same system. Very different signal. The first alert asks, "did errors increase?" The second asks, "are enough real users being hurt that someone should wake up?"
From metrics to action
Raw signals aren't the thing you page on. They're the thing you refine, through correlation and user-impact and burn rate, until what's left is worth a human's sleep.
A framework you can apply Monday
Principles are easy to nod at. Here's the operational version: four moves you can start on this week.
1. The actionability test. For every alert that pages a human, answer one question: what specific action does the person take when they receive it? If you can name the action (restart this, fail over to that, roll back the deploy), it's a legitimate page. If the honest answer is "check whether it's still happening," it isn't a page. It's a dashboard panel or, at most, a ticket. This single test retires more noise than any other. Run it against your current paging alerts and you'll be surprised how many fail.
2. Tie severity to impact, not to the metric. Build three lanes and route deliberately. Page for user-facing harm with an actionable cause and meaningful error-budget burn. Ticket for things that need attention but not tonight: degradations, slow trends, capacity you'll hit next quarter. Silent dashboard for the informational signal you want visible but never want to be woken by. The discipline is refusing to let a lane creep upward. Most alert bloat is ticket-worthy signal that somehow got wired to the pager.
3. Deduplicate and group. Collapse correlated alerts before they reach a person. If a dependency is down, suppress the downstream alerts that are merely reporting the same root failure. Dependency-aware suppression turns three costumes back into one incident. Respect maintenance windows so planned work doesn't page anyone. The responder should receive the incident, not the incident's echoes across every layer it touched.
4. Enrich with context. An alert that says "error rate high" makes the responder start from zero. An alert that arrives with the runbook link, the last few deploys, and a clear statement of what is failing and why we think so lets them start from the middle. Attach the filtering and correlation logic that already knows which paths are affected and which are noise. The responder's first job should be deciding what to do, not reconstructing what happened.
A quick decision tree
If every alert had to walk this path before reaching a phone, most of them wouldn't make it, and that's the point.
Knowing it's working
You can't manage what you don't measure, and "the channel feels quieter" is not a metric. Track the ones that are:
| Metric | Why it matters |
|---|---|
| Alert volume | Measures overall noise |
| Actionable pages | Indicates trust in the system |
| Auto-resolved alerts | Shows suppression opportunities |
| False positives | Direct measure of alert quality |
| MTTR | Measures operational improvement |
If I had to pick one, it's this: measure the reduction in pages that led to no action. That number is the purest expression of the whole effort. Driving it down means you're returning human attention to the system, one retired alert at a time.
One caution before you automate
Everything above can be automated: generated thresholds, auto-suppression rules, ML-driven alerting. You should automate it. But keep one thing in mind, because it's the trap I see teams fall into most. Automation amplifies whatever judgment you feed it. An auto-suppression rule will silence a noisy alert, and it will just as faithfully silence a real failure that happens to match the same pattern. A generated threshold propagates your best thinking and your worst assumptions with equal enthusiasm, and the bad ones scale faster because nobody's watching. The answer isn't to stop automating. It's to automate the execution and keep a human review gate on the judgment.
The takeaway
If you do nothing else, do these three things this week:
- Run the actionability test against your current paging alerts. Demote every one that has no concrete human action attached.
- Pick your single noisiest alert and either retire it, re-tier it, or enrich it so it's worth reading.
- Start tracking pages that led to no action. That one number tells you whether any of this is working.
Final thoughts
Every mature monitoring platform eventually collects thousands of alerts. The best teams aren't the ones that monitor the most. They're the ones that continually remove everything that no longer deserves a human's attention.
That's the real job: not building alerts, but building trust. Because at three in the morning, nobody is reading every notification. They're betting that the one which woke them up is worth believing.
Top comments (2)
"Every alert that fires without needing a human response is a withdrawal from trust" is exactly the right framing. The practical win with SLO-driven paging is that it forces teams to separate operator symptoms from interesting-but-nonactionable telemetry. I've found the hard part isn't defining the first good alerts, it's scheduling the periodic review so old thresholds don't quietly become permanent noise. The decision tree angle is especially useful because it gives teams a way to prune alerts without pretending they're reducing observability.
Thanks Raju. The review cadence is the part I'd underline too. When I audited a couple of hundred Tier-1 alerts, the blocker wasn't spotting the noise, it was that half of them had no clear owner, so nobody felt they could retire anything. A one-off audit clears the backlog. Only the recurring review stops the decay.