Disclosure: I'm the founder of SentinelReady, the tool I mention in the later sections. I built it with AI-assisted development using Claude. That's separate from the runtime, where triage runs on a local model on your own hardware. The first half of this article doesn't depend on the product at all.
Most alert-fatigue advice comes down to "alert on less." I've spent most of my career in operations: as an SRE supporting national telecom carrier video streaming on AWS EKS, as a Cisco High Touch Engineer, and running a 24×7 network support center. My experience is that the alert you tune away is the one you need at 3 a.m. Even with a team triaging around the clock and escalating to me, I still got paged at 2 a.m. for noise and still missed what mattered.
This article covers a different approach. Keep alerting on everything, and put a layer next to your monitoring that remembers patterns, groups related alerts into incidents, and writes sitreps. That layer runs on a local model, so alerts never leave your network, and if the model fails, the raw alert still gets through.
1. Why more human triage didn't fix it
On paper, our setup was solid: a team watched alerts 24/7 and escalated to me when something needed an engineer. In practice, three things kept happening.
- Every alert was worked from scratch. The same alert fired on Tuesday as on Thursday, and each time someone re-read it, re-checked it and re-decided. Nobody remembered it in a structured way.
- Escalation became the safe default. When you're on the triage desk and unsure, you escalate. That's rational for the person on the desk, and it's exactly how 2 a.m. pages for noise happen.
- Signal got buried. The alert that mattered arrived alongside a dozen that didn't, and it looked just like them.
More people watching the firehose doesn't change the shape of the firehose. The problem wasn't effort. It was memory and context.
2. Why "alert on less" is the wrong fix
Tuning thresholds, adding suppression windows and deleting noisy rules all have their place. But each one trades noise for a blind spot, and you don't find out which blind spot until it costs you.
Some practical alert hygiene before you reach for any tool:
- Separate "page a human now" from "someone should know." Most noise is legitimate information delivered on the wrong channel at the wrong time. Moving it to a digest is better than deleting it.
- Write down your sacred alerts. Agree on a short list of conditions that always page, no matter what, and never let automation or tuning touch them.
- Make alerts carry their own context. Include the service, the environment and a link to the runbook in the payload. An alert that needs ten minutes of digging before anyone knows what it is will always get escalated.
- Track repeat offenders. If the same alert fires often and is resolved the same way every time, that's a pattern you're paying a human to rediscover. Either automate the response, fix the root cause, or at least record the known answer.
- Review pages after on-call. For each page, ask whether it needed a human right then. The answers make a better tuning backlog than any dashboard.
None of this requires deleting alerts. The aim is to organize them, not silence them.
3. Pattern memory: stop working the same alert from scratch
The biggest waste in alert triage is re-deciding things that have already been decided. If an alert has fired before, been looked at and turned out to be routine, the next occurrence should arrive with that history attached.
That's what I mean by pattern memory: recognizing prior occurrences of an alert and carrying forward what was learned. You can approximate it yourself with a shared notes doc keyed by alert name, runbook links in payloads, and tags in your incident tool. The discipline is the same either way. Capture the answer once and reuse it.
One warning about measurement. How much quieter this makes things depends entirely on how repetitive your alerts are, and a busy environment benefits far more than a quiet one. Be skeptical of any single noise-reduction percentage, mine included. Measure it on your own alerts. In SentinelReady, that's the "Answered from memory" count on the dashboard, which reflects your environment and not a marketing slide.
4. From alerts to incidents to sitreps
Individual alerts are the wrong unit for humans to reason about. When one upstream issue triggers alerts in five downstream services, that's one incident, not five pages.
A workable flow looks like this:
- Group related alerts into incidents, so the on-call engineer sees one thing with context instead of a burst of fragments.
- Urgent alerts come straight through. Your sacred list always escalates.
- Everything else goes on a sitrep, a situation report the team reads in the morning instead of being woken for.
Sitreps also solve a problem engineers often overlook: management visibility. Leaders want to know what's going on, and handing them the raw firehose helps nobody. A clean sitrep gives them the picture without pulling engineers into status updates.
5. Why a local model
Using a language model for triage is useful because alerts are messy, semi-structured text, and models are good at reading them in context. But alert payloads contain hostnames, internal topology, customer identifiers and sometimes worse. Sending them to a cloud AI service is a real data-handling decision, not a detail.
Running the model on your own hardware avoids that decision. Alerts never leave your network, and air-gapped setups become possible. That's the design SentinelReady uses: triage runs on a local model on the customer's own hardware, not in a cloud AI service.
Even if you build your own pipeline instead, I'd apply the same rule. Keep alert content inside the boundary you already trust.
6. Fail-open design: never let the smart layer eat an alert
This is the part I'd insist on with any triage layer, whether you build it or buy it. The triage layer should sit alongside the alert path, not in it.
If the model is unreachable, slow or returns nonsense, the alert should still be delivered, unenriched but delivered. A system that silently drops alerts when its AI fails is worse than having no system.
Test it yourself, and don't rely on anyone's word, including mine. Stop the model container, fire a test alert, and confirm it still arrives. It takes a minute, and it's the first test I'd run on any tool in this category.
7. Fitting it into an existing stack
Nobody wants to rip out monitoring to fix alert noise. The organizing layer should take alerts from what you already run and leave that setup alone.
SentinelReady takes alerts by webhook from Grafana, CloudWatch, Datadog, UniFi Protect or any JSON payload, and it deploys with Docker on your own box. Your monitoring stays exactly as it is. It's not an alert-management or paging tool, and it doesn't try to replace one. It organizes the alerts you already get.
If you're evaluating anything in this space, ask three questions:
- Does it require changing my existing alert rules?
- What happens to an alert when it fails?
- Where does my alert data go?
8. Trying it, and what I'd like to hear
Everything in sections 1 to 6 works without any product: sacred alerts, digests instead of deletions, recorded answers for repeat alerts, incident grouping, local processing and a fail-open path.
If you want to see it packaged, SentinelReady has a free Community tier at sentinelready.io, with paid tiers above it. The container image is proprietary, so it isn't open source. I'd rather you hear that from me up front.
I'd especially like to hear from people who carry a pager:
- Does "organize, don't silence" match your pain, or do you just want fewer alerts?
- What's the first thing that would make you distrust a triage layer?
- What belongs in a sitrep that you actually want to read?
Blunt feedback is welcome in the comments.
Top comments (0)