Can a small, local LLM take tier-1 alert triage off a SOC team without missing attacks, and without being talked out of an escalation by the attacker? I built a lab to measure it instead of guessing. The code is on GitHub: github.com/Willey2003/soc-triage-lab.
Three designs, same alerts
The same 120 labelled alerts (60 malicious, 60 benign look-alikes) go through three designs:
- Rules: a classic SIEM threshold.
- LLM only: the model decides everything.
- Hybrid: rules at both ends, the model only in the grey zone, with a prompt-injection guard and fail-safe defaults.
Everything runs on one VM with no GPU, using qwen2.5:1.5b through Ollama, and no alert data leaves the box.
feeder -> API -> enrich -> rule score -> hybrid gate -> (grey zone) -> Ollama qwen2.5:1.5b
| |
SQLite cases <--------------------+ Prometheus <- /metrics -> Grafana
Results
| Metric | Rules | LLM only | Hybrid |
|---|---|---|---|
| Accuracy | 67% | 48% | 81% |
| Recall (escalate) | 100% | 88% | 100% |
| Missed attacks | 0 | 7 | 0 |
| False positives | 40 | 55 | 23 |
| Escalated to an analyst | 100 | 108 | 83 |
| Prompt-injection success | 0% | 10% | 0% |
| Model calls | 0 | 120 | 43 |
The hybrid design cut the analyst queue by 17% and false positives by 43%, and missed no attacks. Letting the model decide everything was the worst design: it missed 7 attacks, closed foreign logins from new devices, and one injected alert talked it into closing.
The guardrails that mattered
- Rule floor. Any alert scoring 70 or above is escalated before the model sees it. This handled 54 of 120 alerts, including an injection written to evade the pattern list.
- Close veto. The model may close an alert only when a trusted CMDB or allowlist fact explains it (VPN egress, signed internal tool, approved scanner). Without it, the hybrid closed 3 malicious foreign logins.
- Injection guard. Alert text is untrusted: command lines, user agents and DNS labels are attacker-controlled. The guard strips instruction-like text and the prompt marks the alert as evidence only.
- Fail-safe on invalid output. 12% of model answers still failed validation, and every one was escalated rather than guessed.
What the model is actually good for
Reading business context the rule does not have. Its wins were VPN logins (10/10 correctly closed in hybrid mode) and corporate-network password typos. It still did not trust an allowlist for telemetry DNS or the automation-host flag for SCCM PowerShell. A 1.5B model often restates the rule instead of weighing the context.
Latency was about 10 seconds per alert on CPU (p95 18 s). That is fine for a steady queue and not fine for a burst, which is one more reason to keep the model in the grey zone only.
Run it yourself
The lab runs rootless with podman (tested on RHEL 9 with SELinux enforcing), and there is a Helm chart for Kubernetes with a NetworkPolicy so only the API can reach the model.
git clone https://github.com/Willey2003/soc-triage-lab && cd soc-triage-lab
scripts/lab.sh up # build image, start the pod, pull the model
scripts/lab.sh eval 120 # rules vs llm vs hybrid on 120 labelled alerts
scripts/lab.sh test # unit tests
scripts/lab.sh down
Limits
The alerts are synthetic with clean labels, and the indicators use RFC 5737 documentation IP ranges and invented hashes and domains. It is one model, one seed, temperature 0. Larger models are the obvious next run. Treat the model’s output as a suggestion that a human reviews, and keep high-risk alerts on deterministic rules.
Top comments (0)