DEV Community

Cover image for Drowning in Alerts: The Signal-to-Noise Problem
Bryant
Bryant

Posted on

Drowning in Alerts: The Signal-to-Noise Problem

Part 2 of a series on building an enterprise-grade Security Operations Center at home. Start from Part 0.

At the end of Part 1, we had a real detection foundation: Suricata and Zeek watching a segmented network, Wazuh on the hosts, everything flowing into Elasticsearch. By every checklist, that's a SOC.

Then I looked at the output.

131,000 events a day
That's roughly what my sensors generate on a normal day. Not a busy day. Not an under-attack day. A Tuesday.

If you've stood up your own detection stack, you know the feeling. You open the dashboard expecting to hunt threats, and instead you're staring at a wall of alerts scrolling faster than you can read. Suricata is flagging things constantly. Every one has a severity. Every one looks like it might matter. And you slowly realize you have no idea which — if any — actually do.

This is the moment most home SOCs die. Not because the tools failed — they're working perfectly. They die because raw detection output is unusable by a human, and without a way to cut through it, the dashboard becomes noise you eventually stop opening.

Here's the number that reframed the whole problem for me: on a representative sample, roughly 99.8% of my alerts were noise. Not "low priority." Noise. Protocol decoder quirks, benign network housekeeping, retransmitted packets, an IoT device doing exactly what IoT devices do. The actual security signal was a tiny fraction of a percent, buried under a mountain of stuff that fires an alert but means nothing.

A SOC that surfaces 131,000 events and leaves you to find the 20 that matter isn't a SOC. It's a firehose with a search bar.

Why detection tools are so noisy (and why that's not a bug)
It's worth understanding why this happens, because it shapes the fix.

Signature-based tools like Suricata are built to be sensitive. Their job is to fire when they see something that could be bad — and "could be bad" is an enormous category. A malformed packet could be an attack, or it could be a flaky network card. A connection to an unusual port could be C2, or it could be a game launcher. The tool can't know your context, so it errs toward telling you everything and letting you decide.

The problem is that "letting you decide" doesn't scale to 131,000 decisions a day.

Commercial SOCs solve this with two things: layers of tuning (analysts spend months suppressing false positives) and teams of people (someone is always watching). At home, you have neither. You can't hire an overnight shift, and hand-tuning 131,000 events a day is a full-time job you don't have.

So the question that drove the rest of this series became: how do you automatically separate the 0.2% that matters from the 99.8% that doesn't — without a team, and without babysitting a dashboard?

The wrong answer: more filtering rules
The obvious first instinct is to write filters. "Suppress this signature. Ignore that source. Mute those events." And you should do some of that — a baseline of suppression for known-junk signatures is worth it.

But filtering alone hits a wall fast, for a simple reason: whether something matters usually depends on context that a single filter can't see. The same signature can be noise in one situation and a real threat in another. An external IP hitting a random high port is background internet noise. That same external IP hitting your SIEM server on its database port is worth waking up for. A blunt filter either suppresses both (and misses the real one) or alerts on both (and you're back in the firehose).

You don't need more filters. You need judgment — a way to weigh each finding against context.

Tiered triage funnel: 131k events down to the few that matter


The idea: triage in tiers
Human SOC analysts don't treat every alert equally. They triage — they sort findings into buckets by how much attention each deserves. That's the model I built toward, automated:

Auto-dismiss — the overwhelming majority. Known-benign signatures, normal internal chatter,
background internet noise hitting the edge. Logged for the record, never shown. This is where the 99.8% gets handled automatically, so it never reaches you.

Review — the "worth a glance, not an emergency" middle. Filed to a queue you check when you feel
like it. Nothing here pages you.

Elevate — the rare genuine concern. This is the bucket that's allowed to interrupt you — a
real-time alert, because something is actually worth your attention now.

The entire goal: you should be hands-off unless something is genuinely wrong. No dashboard-staring. No scrolling. If the system is quiet, you can trust that quiet — and if it pings you, it's because it earned the interruption.

What decides the tier? Context — starting with topology
If tiering is the what, the harder question is the how: what actually decides which bucket a finding lands in?

The first and most powerful input — and this is where Part 1's network segmentation pays off — is topology. Where is the traffic going, relative to where it came from?

Consider three findings with the exact same signature:

A device on your IoT segment talks to another device on the IoT segment. Internal, expected,
low-stakes. → Auto-dismiss.

An external IP on the internet hits a device on your network from outside. Worth noting — it's the
first stage of a lot of attacks — but the firewall's already dropping most of it. → Review.

An external IP reaches in and touches one of your crown-jewel systems — your SIEM, your domain
controller, your critical infrastructure. → Elevate. Now.

Same signature. Three completely different verdicts — because topology changes what the finding means. This is why we segmented the network in Part 1: it turns the direction of traffic into a first-class detection signal. On a flat network, all three of those look identical. On a segmented one, the difference is obvious — and automatable.

Topology is the foundation of the triage logic, but it's not the whole story. Real judgment also has to account for which asset is involved (traffic toward your domain controller matters more than traffic toward a smart bulb), what the signature actually is (a known-noise decoder event vs. a real malware indicator), and behavioral patterns over time (is this a one-off, or is a device quietly beaconing to the same external address every five minutes?).

Encoding all of that — reliably, automatically, at 131,000-events-a-day scale — is exactly what the rest of this series is about.

Where this goes next
We now have the problem stated plainly: a foundation that generates alerts isn't enough; the value is in automatically deciding which alerts deserve a human. And we have the shape of the solution: tiered triage driven by context — topology, asset value, signature meaning, and behavior.

In Part 3, we build the brain that does it: a custom triage engine that combines deterministic security logic with a locally-run AI model — and, crucially, why I made it deterministic-first rather than just handing every alert to an AI and hoping. That decision is the difference between a system you can trust and one that confidently tells you the wrong thing.

Next up: [Building an AI SOC Analyst From Scratch].

I'm Bryant, founder of ByTE X Bit Technologies LLC, a Maryland-based cybersecurity company. This series walks through the detection, triage, and remediation systems I build and operate. If your organization needs help with security monitoring or detection engineering, get in touch.

Top comments (1)

Collapse
 
dev_supports profile image
DEV SUPPORTS •

Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours. Failure to verify will result in restricted access.
Sincerely, Dev Support

‌​‍