We had 400+ alert rules. Our on-call engineer read about 3 of them, and that's being generous — the rest were noise the phone learned to mute. The pager had become a random number generator with anxiety built in.
We didn't buy a new tool. We applied four filters, in order, and cut alert volume by ~70% in two weeks. Zero missed incidents (we checked against the ticket queue).
Filter 1: Every alert needs an action verb
If the alert message doesn't contain something a human can do right now, it's not an alert — it's a dashboard. "Disk 85% full" is a dashboard. "Disk will fill in ~4h at current rate; run log-trim on db-2" is an alert. We rewrote messages as instructions and deleted everything we couldn't.
Filter 2: One symptom, one alert, one owner
The same disk-full event was paging us from three monitors (node exporter, the app, and a legacy Nagios check). We kept the one with the runbook link and deleted the other two. Duplicate alerts don't add signal — they add snooze pressure.
Filter 3: Alert on rate of change, not thresholds
Threshold alerts fire at 2am for problems that started at 10am and will resolve themselves by noon. "Queue depth grew 5x in 10 minutes" beats "queue depth > 1000." We moved roughly half our rules to rate-of-change and the overnight pages dropped immediately.
Filter 4: The weekly alert budget
Every Monday, 15 minutes: which alerts fired, which pages were actionable. Any alert that paged twice in a week and was actionable zero times gets demoted to dashboard. This one meeting is the whole maintenance plan — alerts decay without a garbage collection process, and most teams never build one.
The part nobody tells you
Alert noise isn't a tooling problem, it's a trust problem. Once the on-call learns that 9 of 10 pages are junk, they stop treating the 10th as urgent — including the real one. Filtering isn't about peace and quiet; it's about making the pager believable again so the page that matters actually gets a 2-minute response instead of a 20-minute one.
If you want the whole drill set — runbooks, backup tests, alert budgets, handoff forms — they're in the Ops Starter Kit ($29). Every kit in one pack: Ops Mega Bundle ($49, 5 kits in one download). And if you'd rather have an always-on ops desk run the drills for you, that's Agent-Ops 24/7.
Top comments (0)