When I joined the team, our on-call channel got roughly 400 alerts a week. The on-call engineer's job had quietly become "acknowledge things and go back to sleep." That's the worst state a system can be in, because a real alert now looks exactly like the 399 fake ones.
I did an audit. I pulled a month of alerts and sorted them by how often each one fired and how often anyone actually did something in response. The pattern was brutal and familiar: a handful of alerts fired constantly and never required action. One alert had fired 61 times and the response every single time was "wait, it recovers on its own." That's not an alert. That's a status update with a pager attached.
We deleted it. We deleted a lot of them. If an alert fires and the correct human response is to do nothing, it should not page a human. It can go to a dashboard, a log, a weekly review — anywhere but the thing that wakes someone at 3am.
Then we rebuilt what was left around symptoms, not causes. The old alerts screamed about CPU, memory, queue depth, individual host health — internal plumbing. But a customer doesn't care that a queue is deep. They care that their request is slow or failing. So we alerted on the things a user feels: error rate, latency at the 99th percentile, and whether requests were completing at all. Everything else became context you look at after the symptom alert fires, not a page in its own right.
Alert volume dropped from ~400 a week to under 20. And here's the part that matters: when one of those 20 fires now, people move. Trust in the pager came back.
Good observability isn't more dashboards or more alerts. It's a ruthless answer to one question: what does the user experience, and how would I know if it broke? Alert on that. Everything else is data you consult, not noise you inflict on a tired human at night.
Every alert that doesn't demand action is training your team to ignore the ones that do.
– Sergey Shinder
Top comments (0)