Your on-call engineer gets 47 alerts before lunch. Twelve are the same flapping health check. Eight are downstream effects of a single database hiccup. Six are informational warnings that never require action. By 2 PM, a new alert arrives — checkout is returning 500s from three regions — and it sits in the channel for nine minutes because the person on call has learned to stop reading.
That is alert fatigue. It is one of the most common ways a well-monitored system ends up with a longer incident response time than a system with no monitoring at all.
What alert fatigue looks like in practice
Alert fatigue is not laziness. It is a rational response to a noisy system. When most alerts are false positives, low-priority warnings, or duplicates of the same underlying problem, engineers train themselves to ignore the pager — because ignoring it is usually the right call.
PagerDuty's State of Digital Operations research found that the median on-call team receives over 300 alerts per week. Of those, roughly 30–40% are noise: alerts that fire, get acknowledged, and close without any action taken. The remaining alerts often cluster around a handful of real incidents, buried under duplicates and false alarms.
The damage compounds over time. Engineers rotate off on-call shifts feeling burned out. Response times creep upward. Critical pages blend in with the noise. The team builds informal workarounds — muting channels, filtering notifications, checking Slack "when they get around to it" — that undermine the entire alerting pipeline. For more on how this affects on-call teams specifically, see on-call rotation best practices.
Why your alerting is too noisy
Alert fatigue rarely has one cause. It accumulates from a series of reasonable decisions that compound into an unreasonable system.
Too many monitors, not enough intent
Teams add monitors reactively. An outage happens, someone creates an alert so it "never happens again," and the alert stays forever — even after the underlying architecture changes. The monitor count grows while the number of meaningful signals stays flat.
The question is not "does this metric matter?" It usually does. The question is "does this metric need to wake someone up?" Most metrics belong on a dashboard, not in a pager.
No severity differentiation
When every alert pages the on-call engineer with the same urgency, the engineer cannot tell what is actually urgent. A certificate expiring in 28 days should not arrive in the same channel, with the same sound, as a checkout outage affecting live transactions.
Severity levels exist to solve this. Critical means customers are affected right now. Warning means something needs attention during business hours. Info means the team should know, but nobody needs to act. Without that separation, everything feels equally important — which means nothing does. See incident severity levels for a practical framework.
Duplicate alerts from correlated failures
A single database going unhealthy can trigger alerts from every service that depends on it. The API returns 500s — alert. The background job queue stalls — alert. The health check for the admin panel fails — alert. The status page component goes red — alert. One root cause, four pages.
The on-call engineer spends the first ten minutes triaging instead of fixing the database. Worse, the volume itself signals "something big is happening" without clarifying what.
Alerts without actions
"CPU at 78%." So what? Should someone scale up? Is it a runaway query? Is it normal traffic? If the alert does not point to a specific action or a runbook, it creates noise without value. Every alert that fires without a next step trains the team to treat alerts as background chatter.
How to fix it
The fix is not adding more rules. It is removing the ones that do not earn their place and restructuring the rest so each alert is worth the interruption.
Audit the alert-to-action ratio
Pull the last 30 days of alert history. For each alert rule, count how many times it fired and how many of those resulted in a human taking action. If an alert fires 50 times a month and gets acted on twice, it needs to be downgraded, tuned, or removed.
A healthy target: at least 70% of pages should lead to a meaningful response. If your ratio is below 50%, the pager has become a notification feed, not an incident signal.
Set severity thresholds that mean something
Map each alert to a severity level and route accordingly:
| Severity | Meaning | Channel |
|---|---|---|
| Critical | Customers affected now, revenue at risk | Phone call or push notification to on-call |
| Warning | Needs attention within hours, not minutes | Slack or email to the owning team |
| Info | Useful context, no action required | Dashboard or daily digest |
Then enforce the mapping in your notification policies. Critical pages the on-call rotation. Warning posts to a team channel. Info goes nowhere near a phone. This separation alone can cut page volume by 40–60% without losing coverage.
Group correlated failures into one alert
When five services fail because one dependency went down, the on-call engineer needs one alert that says "the database is unhealthy and these services are affected" — not five independent pages.
Resource groups help here. Group monitors by shared dependency — your payment provider, your primary database, your CDN — so that correlated failures produce a single notification instead of a cascade. The engineer sees one alert with context, not a wall of symptoms.
Tune flapping and transient detection
A health check that fails once because of a network blip and recovers 30 seconds later is not an incident. But if your alerting fires on the first failure, the on-call engineer gets paged for something that resolved before they could open a terminal.
Require consecutive failures before alerting — two or three failed checks in a row from the same region, or failures from multiple regions simultaneously. This filters transient noise without delaying real outage detection.
Require every alert to have a runbook link
This is the simplest and most effective rule to adopt. If the alert does not link to a runbook or documented response procedure, it should not page anyone. The runbook does not need to be perfect — a three-line document that says "check the database dashboard, look for long-running queries, escalate to the database team if replication lag exceeds 30 seconds" is enough.
This forces the team to think through the response before wiring the alert, which naturally filters out alerts nobody knows how to act on.
Review and prune quarterly
Alert rules are not permanent. Services change, architectures evolve, and the alert that mattered six months ago may now fire on a deprecated endpoint. Schedule a quarterly review: sort alerts by frequency, check the action ratio, and delete or downgrade anything that has become noise.
The alert your team cannot afford to miss
Alert fatigue is not a tooling problem at its root. It is a prioritization problem. The goal is not zero alerts — it is a system where every page is urgent, actionable, and trusted. When the on-call engineer's phone buzzes, they should think "something real is happening" instead of "probably another flapping check."
Start with the audit. Pull your alert history. Find the rules that fire without action. Set severity levels that route alerts to the right channel. Group correlated failures so one root cause produces one page. Attach a runbook to every alert rule. And prune what no longer matters.
The on-call shift should be boring most of the time. When it is not boring, every alert should matter. If you're ready to fix the noise, DevHelm's notification policies let you map severity levels to channels, group correlated monitors into resource groups, and require multi-region confirmation before any alert pages your team. For a deeper dive into the configuration, see the full monitoring alerts guide. To reduce mean time to resolution once a real alert fires, pair it with standardized runbooks and clear escalation paths.
Originally published on DevHelm.
Top comments (0)