A client pinged me at 7 AM: "Did you see anything weird last night? PagerDuty says we had 47 alerts."
I pulled the night log. 47 alerts fired between midnight and 6 AM. Three of them mattered. The other 44 were the same two problems firing on loop: a flaky disk-latency check and an app that logs a warning every time a crawler hits an endpoint that doesn't exist.
Here's the part that should scare you: the three real alerts were buried in the noise, and one of them was the first sign of the disk that died two weeks later.
Why alert fatigue isn't a monitoring problem
It's an accountability problem. Every alert that fires and gets ignored trains your team to ignore the next one. That's how a $400 disk failure becomes a $40,000 outage with a postmortem.
The math nobody does:
- If your on-call sees 40 alerts a night, by week two they trust none of them.
- The signal-to-noise ratio isn't a "nice to have" metric. It's the difference between an alert system and an alert costume.
The $0 fix, in the order that matters
- Kill every alert that fired more than 3 times last week and resulted in zero action. Not "lowered priority." Deleted. If it fires and nobody moves, it's noise. If it's really important, it'll show up in a different way that DOES get action.
- Deduplicate at the source. One flaky check firing every 5 minutes is 72 alerts. It should be one alert that says "this check failed 72 times." Most monitors have a "re-alert after X minutes" setting — set it to hours, not minutes.
- Tier by consequence. Tier 1 wakes a human. Tier 2 waits for business hours. Tier 3 goes in a weekly digest. If everything is Tier 1, nothing is.
- Add a dead-man's switch to the monitors themselves. The scariest failure isn't noise — it's the monitor that silently stops checking. Your watchdog should scream if IT doesn't hear from the watchdog.
That fourth one is the one almost everyone skips, and it's the one that bites hardest. A green dashboard that stopped updating at 2 AM looks exactly like a healthy system to anyone glancing at it.
What the noise-free night looks like
After the cleanup at that client: 4 alerts fired the next week. All 4 got acted on within 10 minutes. One of them was a real problem (the disk, caught early this time). Cost of the fix: one afternoon. Cost of the outage they eventually avoided: conservatively five figures.
I run 24/7 ops for client systems using exactly this playbook — the dead-man's-switch scripts, the watchdog-that-watches-the-watchdog, and the weekly digest that replaces 90% of noise. I packaged the core of it as a self-serve starter pack (canaries, watchdog, evidence reports, restore drill):
https://hive80lab.gumroad.com/l/automation-starter-pack
And if you want the full operations layer — the system that keeps itself honest 24/7, not just the scripts:
https://hive80lab.gumroad.com/l/agent-ops-24-7
If your alert queue has 40 items in it right now, do step 1 before you do anything else today. It costs nothing and it buys back your team's attention.
I write about the boring infrastructure failures that cost real money. Follow if you'd rather catch them at $400 than $40,000.
Running a stack like this? Our Ops Starter Kit ships the runbooks, checklists, and monitoring patterns behind these posts — grab it and cut your next incident short.
Top comments (0)