How I Built a Linux Threat Detection Engine That Explains Why It Fired

Security alerts are easy to generate.
Useful security alerts are much harder.
After dealing with enough alerts that turned out to be false positives, I started asking a different question:
What if a threat detection system didn't just tell you what happened, but explained why it thinks it matters?
That's what led me to build the detection engine behind Watch Cortex.
This isn't a post about building another SIEM. It's about the detection and reasoning layer: how it correlates Linux security events, builds context, evaluates competing hypotheses, matches attack sequences, and produces an investigation you can actually understand.
The Problem: An Alert Isn't an Investigation
A traditional rule might look something like:
if failed_ssh_logins > 5:
alert("possible brute force")
The problem is that the rule doesn't know what happened afterward.
Five failed SSH logins followed by nothing might be automated internet noise.
Five failed logins followed by a successful login, a privilege escalation, a new process, and a configuration change are a very different situation.
Those two events shouldn't produce the same response.
So instead of treating every event independently, I designed Cortex around context and event sequences.
The Architecture: Five Context Trees + A Reasoner
The detection pipeline has two major stages.
Stage 1: Build Context Trees
For each Linux server, Cortex builds five independent context trees.
The database queries run concurrently using Promise.all, with a warm-cache target of under 60 ms for the context-building stage.
The five trees are:

1. Health Tree Tracks deviations from a server's normal behavior. Instead of saying: CPU > 80% = suspicious Cortex builds a seven-day baseline for each server and uses statistical deviation instead. That matters because 80% CPU can be completely normal for one server and highly unusual for another. For new servers without enough historical data, the system can fall back to peer-group behavior.
2. Network Tree Tracks things such as: New connections Unusual ports Port scanning behavior Potential lateral movement Suspicious outbound connections Beacon-like communication patterns
3. Process Tree Tracks: Running processes Parent/child relationships Process ancestry File activity Suspicious executables YARA matches The process ancestry is particularly useful. A process isn't necessarily suspicious because of its name. How it got there can be much more interesting.
4. Threat Tree Correlates: IOC matches CVE exposure MITRE ATT&CK techniques Known malicious indicators Other threat intelligence signals
5. Authentication Tree Tracks: Failed logins Successful logins Sudo activity New accounts Privilege changes Credential-related behavior Each signal receives a weight and severity score. There's also an important distinction between new behavior and persistent behavior. If a server has continuously shown the same condition for more than 24 hours, Cortex marks that signal as chronic and reduces its impact. A server that has always been running at 90% CPU isn't necessarily experiencing a new security event. Stage 2: The Reasoner Evaluates Competing Hypotheses Once the five context trees are built, the reasoner evaluates possible explanations. Some of the hypotheses include: brute_force_campaign active_intrusion apt_intrusion malware_execution lateral_movement_attempt data_exfiltration ransomware resource_abuse reconnaissance supply_chain_compromise Instead of asking: "Did rule X fire?" the system asks: "Given all of the evidence available right now, which explanation best fits what we're seeing?" Each hypothesis evaluates the five context axes independently on a 0–100 scale. The weighting is hypothesis-specific. For example, a brute-force hypothesis puts more emphasis on authentication activity, while an APT-style hypothesis may put more emphasis on process behavior and its relationship with network activity. Just as importantly, the system tracks evidence against a hypothesis. If there's no internet exposure, no recent authentication activity, and no plausible entry point, that should reduce confidence in certain intrusion hypotheses even if other anomalies exist. That's an important difference between simply adding more alerts and actually correlating evidence. Detecting Attack Chains Instead of Isolated Events The hypothesis engine isn't the only detection layer. There's also a TTP sequence matcher that looks for multi-step attack patterns. Each pattern contains ordered signal steps and time constraints. For example: SSH Brute Force → Process Anomaly → Configuration Change Step 1: brute_force or auth_fail minimum: 5 events

Step 2:
suspicious_process
within: 2 hours
source correlation: Step 1

Step 3:
fim_change
within: 2 hours

Result:
increase confidence in active intrusion
The important part isn't that any individual event is necessarily malicious.
It's that the sequence makes the events more meaningful together.
Another example is cryptocurrency-mining persistence:
Step 1:
known miner indicator or mining-pool connection

Step 2:
cron modification or systemd unit creation
within: 4 hours

Result:
increase confidence in resource abuse
And a reverse-shell pattern:
Step 1:
process spawns shell

Step 2:
shell establishes outbound connection

Result:
increase confidence in command execution
When a TTP chain matches, it can increase the confidence of the relevant hypothesis and, depending on the policy, contribute to an automated response.
Possible responses include:
Temporarily blocking an IP
Isolating a network interface
Terminating a suspicious process
Collecting forensic information
The important design principle is that detection and response are separate decisions. A detection can increase confidence without automatically taking destructive action.
The Feature That Made the System Actually Useful: Explainability
This is probably the part I'm most proud of.
Every investigation plan contains the reasoning behind the decision.
Instead of:
Threat detected — confidence: 87%
you can see:
Which hypothesis won
Why it won
Supporting evidence
Against-evidence
Relevant processes and PIDs
File paths
IOC values
Matched TTP chains
Confidence adjustments
The final confidence score
For example, the system can explain that confidence was affected by:
False-positive dampening
If a server has repeatedly generated false-positive plans for the same behavior, the system can reduce the confidence of similar future detections.
Temporal escalation
If the same threat class appeared recently and wasn't dismissed or cancelled, confidence can increase.
Persistence matters.
Fleet correlation
If multiple servers in the same organization exhibit the same threat class at approximately the same time, Cortex can flag the behavior as potentially campaign-wide.
This makes the output much more useful than a generic severity number.
You can actually inspect why the system believes something is happening.
What I Got Wrong
The first version had several problems.

1. Chronic Signals Created Alert Fatigue Early versions would repeatedly rediscover the same condition. A misconfigured process could appear in every detection sweep and continuously generate another "suspicious process" investigation. The solution was to distinguish persistent conditions from newly emerging behavior. Signals that remain continuously present for more than 24 hours receive a chronic penalty. The goal isn't to hide the condition. It's to stop treating yesterday's condition as today's new incident.
2. Hardcoded Thresholds Don't Generalize My first attempt was basically: CPU > 80% = anomaly That didn't work. A database server running at 80% CPU may be completely normal. A lightly loaded web server suddenly reaching 80% at 3 AM could be much more interesting. Switching to per-server baselines and statistical deviation made the health signals considerably more useful.
3. I Wanted Real-Time Reasoning Cortex currently separates event detection from the heavier reasoning sweep. The agent can react to events as they're ingested, while the contextual reasoning layer runs on a 15-minute cycle. I initially wanted to make the reasoning cycle much faster. But the context-building stage queries five different areas of server state for every server. On a larger fleet, running that continuously becomes expensive very quickly. So I ended up with a hybrid approach: Fast event detection + periodic deeper reasoning. It's not perfectly real-time, but it gives me a better tradeoff between detection depth and infrastructure cost. Why I Built It This Way The goal wasn't to create another tool that generates more alerts. The goal was to reduce the distance between: Event → Context → Hypothesis → Evidence → Decision That's also why I don't want the system to be a black box. If Cortex says something looks like an active intrusion, I want an operator to be able to open the investigation and ask: "Why?" And get an answer based on the actual evidence collected from the server. Try the Detection Engine The system is currently available through Watch Cortex. You can try the interactive demo without creating an account: https://watch.alsopss.com/demo The demo shows the investigation workflow, reasoning chain, confidence scoring, and TTP matching. There's also a 14-day trial for the full platform. If you're building Linux security tooling, detection pipelines, SIEMs, EDRs, or homelab security systems, I'd genuinely like to hear how you approach the same problem. What signals do you find most useful for distinguishing a real intrusion from normal Linux noise? I'd especially like to hear from people running their own homelabs or small server fleets.