adjia mvoa ndji gabriel monfils

Posted on Oct 29

Are LLMs Actually Useful for Security Operations? A Benchmark-Driven Analysis

#cybersecurity #ai #machinelearning #datascience

Two numbers, published months apart, describe the same job.

One says a frontier model triages alerts with 98% accuracy. The other says the best of five off-the-shelf reasoning pipelines in a realistic replay misses 40.4% of attacks. Both are honest measurements. They are also, taken at face value, irreconcilable — and the gap between them is the most useful thing I found in this review.

I screened 378 unique records, extracted 32 quantitative comparisons, and graded every one by how fairly it was run. This is an attempt to answer a narrower question: for which security-operations tasks does the measured evidence support using an LLM, and under what conditions does it not?

The short answer: LLMs are genuinely useful as interfaces to structured reasoning and as extractors of unstructured context. They are not competitive as drop-in classifiers, they are dangerous as autonomous actors, and almost every headline number about them is measured under conditions that do not exist in a SOC.

How I evaluated the evidence

I queried the arXiv API across 18 topic clusters on 29 October 2026: 431 entries, 378 unique after de-duplication. I screened titles and abstracts, retrieved 44 full texts, and kept 47 papers. I added nine foundational and standards documents — Arp et al.'s USENIX Security critique (S48), Sommer and Paxson on the closed-world assumption (S49), Axelsson on base rates (S50), NIST AI 600-1 (S51), the OWASP LLM Top 10 (S52), MITRE ATLAS (S53) and ATT&CK (S54), NIST SP 800-61r3 (S55), and the Loghub dataset paper (S56) — plus three practitioner reports from DEV (S57, S58, S59).

Then I graded every comparison:

Class A — same data, same harness, same metric, both arms run by the same authors. Differences are attributable to method.
Class B — shared dataset and metric, but one arm tuned elsewhere or a baseline cited rather than rerun. Directionally usable; magnitudes soft.
Class C — single system measured, or a benchmark adjacent to operations. Capability ceiling only.
Class D — interviews, surveys, standards, ROI models. Constraints, never performance.

I did not pool results across classes, and I did not average anything. Where a number came from a class B comparison, it is reported as a class B number.

One structural caveat up front: 34 of the 59 sources are preprints or under review. The direction of the findings is stable across independent groups. The specific magnitudes are not yet.

First, separate four different tasks

Most public arguments about "LLMs in the SOC" fail because the participants are describing different jobs.

Detection — is this event malicious? High volume, binary, latency-sensitive, evaluated with precision/recall at extreme imbalance.
Triage — does this alert deserve a page, a ticket, or nothing? Low volume, policy-shaped, judged on operational cost.
Investigation — what happened, and what else is connected? Open-ended, multi-source, judged on completeness and correctness of reconstruction.
Response — what do we do, and may the system do it? Consequential, gated, judged on restraint as much as on action.

A result in one does not transfer to another: the 98% headline is task 2 on a balanced sample, the 40% headline is task 3 on realistic telemetry.

A fifth category dominates search results and should be ignored operationally: security knowledge quizzes. CSEBenchmark reports a best score of 85.42% on 11,050 multiple-choice questions (S05); CyberMetric has GPT-4o at 96.25% on its 80-question set against a human expert mean of 72.24% (S07); CS-Eval spans 38.08–87.57 across 19 models (S08). SECURE shows the same pattern on ICS material, with GPT-4 at 87.6% on key-vulnerability questions but GPT-3.5 at 8.4% on unseen ones (S06); CyberSOCEval's malware questions top out at 23–34% (S09). These are recall tests. They show a model has read the documentation, not that it can read your logs.

Detection: a tie, then two inversions

The cleanest experiment stress-tests XGBoost against a LoRA-tuned RoBERTa on network intrusion detection, with a locked 29-feature schema and 50,000 flows per class (S01).

On the same dataset, they tie. XGBoost scores 0.9966 macro F1 on NF-UNSW-NB15-v2, the transformer 0.9938; on NF-CSE-CIC-IDS2018-v2 it is 0.9776 versus 0.9745. The LLM buys nothing here except roughly 5,806× lower throughput.

Change one variable and the ranking inverts. Train on UNSW, test on CIC, and XGBoost holds 0.8116 F1 while RoBERTa collapses to 0.6582, with a false-positive rate of 0.7756 and balanced accuracy of 0.5477. Train on CIC, test on UNSW, and both fail near chance (0.0730 and 0.0599). The transformer is not more general. It is more brittle to distribution shift.

Change the variable again — this time to adversarial evasion, drifting malicious flows toward benign — and the transformer wins: 0.9552 versus 0.6917 F1 at ε=0.15, about 26 points. Both keep false positives under 1% in that regime.

So "which is better" has no answer without naming the stress axis. Same-distribution: tie. Cross-environment: classical. Adversarial perturbation: LLM.

The same pattern shows up in log anomaly detection, where method papers (S21, S22) report gains largely on the same benchmark family. A benchmark spanning HDFS, BGL, Thunderbird and Spirit puts fine-tuned DeBERTa at 95.3%–98.9% F1, Drain plus a random forest at 86.4%–95.1%, and GPT-4 zero-shot at 81.2%–88.3% (S03). Prompted GPT-4 also costs 890 ms per prediction and $8.40 per 1,000 predictions, against 0.3 ms for the parser-plus-classifier. Few-shot prompting closes some of the gap (85.6%–91.7%) but never overtakes a fine-tuned encoder.

Two results sharpen the point.

First, model advantage is not monotone in size or prompting effort. An adaptation study finds hard prompts at 57.56–60.93 F1 while LoRA reaches 98.89 and frozen-layer tuning 99.08 (S04). Quantization is nearly free: 16-, 8- and 4-bit versions score 98.89 / 99.04 / 98.98. But scale gains are dataset-dependent — GPT-2 Medium, Large and XL score 57.04 / 91.90 / 94.77, and on Thunderbird the Medium model collapses to 10.26. "Bigger model, better detection" is not a law.

Second, when an LLM is used as the classifier rather than as the interface to one, it loses badly. In explainable NIDS, zero-shot GPT-4 reaches macro precision/recall of 50.85 / 53.47 and Llama3-8B 48.56 / 40.56, against a cited random forest at 92.17 weighted F1 (S15). On 83,446 phishing emails, a random forest hits 0.9801 accuracy and 0.9789 F1 while the best local quantized LLM, Qwen2.5-32B at 8-bit, reaches 81%, and the 7B and 14B variants sit at 49% and 55% — near chance (S16). The 32B needed about 127% more VRAM than the 14B for two points. On 10,000 balanced phishing URLs, Grok-3 few-shot reaches 0.9399 F1 (S17) — but the balance is again doing much of the work.

The honest summary for detection: use the LLM to build features, parse logs, or explain a detector — not to be the detector. The cost section shows where that division of labour actually pays.

Triage: the base-rate trap

Triage is where the marketing lives: it has a written policy and an obvious cost per alert.

The strongest positive result I found is also the easiest to misread. SIABENCH reports GPT-5 at 98% accuracy on 100 balanced TII-SRC-23 alerts and 97.1% on a 35-alert CIC subset, while GPT-4o-mini drops to 51.4% on that CIC subset (S10). But that CIC subset contains five true positives and thirty false ones: a detector that called everything benign would already score 85.7%. On 68 live, post-cutoff challenges, Claude-3.5-Sonnet scores 60.3% against GPT-4o's 36.8%, and on a separate 20-question post-September-2024 subset GPT-5 scores 85% against GPT-4o's 75%.

Note the construction: 50/50 attack/benign, and 35 alerts. Now do the arithmetic the abstract does not.

Assume 90% sensitivity and 0.1% attack prevalence — ten true incidents in 10,000 alerts. At a 1% false-positive rate, the model raises about 100 false alerts against 9 true positives. Positive predictive value is 8.3%. At a 0.008% FPR it is 91.8%; at 1% prevalence and 1% FPR it is 47.6%. Same model, three completely different operational realities, determined entirely by prevalence and FPR. (This is my arithmetic, not a reproduction of any study; Axelsson formalised the underlying problem in 2000.)

That is why the more informative triage study is the less quotable one. A multi-agent pipeline evaluated on 1,022 benign and 225 attack alerts replayed through a live SIEM — realistic imbalance — reaches F1 0.958, recall 0.969, and a false-negative rate of 3.1%, escalating 18.4% of cases (S11). The five reasoning strategies it compares against land between 0.371 and 0.744 F1, with the best at 40.4% FNR. The winner used 5,075 tokens per case; a weaker pipeline burned 33,438 and did worse.

That is a class B result: the authors built both the benchmark and the winning system. Treat 0.958 as an existence proof — agentic evidence retrieval can work — not as an expected production number.

The practitioner side is more encouraging and less rigorous. A DEV experiment gave six models 200 generated alerts against a written page/ticket/ignore policy, 600 calls each (S57). Two models scored 600/600; the cheap one cost about $0.16 per 1,000 alerts at p50 4.4 s, with one 60-second outlier. But 200 synthetic alerts cannot bound production error, and even their optimistic 95% upper bound is 1.5% — a lot of wrong pages at SOC volume. The deliberately misleading severity-label baseline scored 102/200, which is the useful part: cheap triage is mostly about reading policy, not about reasoning.

Investigation and response: the failure is restraint, not detection

Investigation evidence is thinner.

Mapping 73 NIDS rules to MITRE ATT&CK, the best LLM reaches technique F1 0.62 and tactic F1 0.69; an SVM trained on the LLM's own labels reaches 0.87 and 0.92 (S14). Rare techniques sit at 0.20–0.23. The lesson is not that LLMs map badly; the LLM is a good labeller and a poor reasoner about labels, and the hybrid beats either alone.

On unguided threat hunting over 106 real attack procedures and 86 sub-techniques, the best frontier model reaches 44% mean recall and clears the authors' 50%-per-tactic bar on 7 of 13 tactics, with near-zero recall on Credential Access and Initial Access (S12).

Response is different. Given execution authority, a frontier agent achieved 100% containment in an adversarial incident-response environment — while taking unnecessary actions in 82.5% of episodes, acting by step 4 (S32). A second model contained 62.5% with a 45% false-positive action rate. Detection was largely right; restraint was not — the correct thing to measure, and almost nobody does.

Response-planning research attacks this directly: constraint-checked planning with abstention (S33) and lightweight fine-tuned planners reporting up to 22% shorter recovery times (S34). Report quality is now measured by checklist-based frameworks (S46), which is progress, though report quality is not incident outcome. The most useful response result is architectural. Over 200 real incidents from a financial-sector SOC, a deterministic policy verifier removed 466 non-compliant, approval-gated actions without reducing task recall, across three reruns with a run-level violation rate of 0.8700–0.8750 (S36). The same inline policy text moved two providers in opposite directions. Which means: do not trust the model to hold the policy. Put the policy in code and let the model propose.

Hallucination is an adversarial property, not a prompt-hygiene problem

The framing most teams use — "add grounding, add citations, add a system prompt" — assumes hallucination is a model defect that better prompting fixes. The injection literature says otherwise.

Telemetry is attacker-controlled input. If an LLM reads log fields, an adversary can write to the model's context. Across 48 strategy/defense/task combinations, average injection success was 26.6% with naive prompting and 11.8% under the strongest defense — but the average hides the structure (S28). Direct overrides never worked (0%). Persona hijacks suppressed 68% of malicious logs. Summarisation tasks jumped from 96% with no defense to 38% under constrained output. The attack that works is not "ignore previous instructions"; it is "here is a plausible log line that reframes the incident".

A larger benchmark of 12,847 log entries, 2,569 of them adversarial, reports attack success up to 88.2%, averaging 83.4%, against undefended pipelines; layered defenses cut 90.4% but left 8.4% residual, and a technique called Context Stitching still succeeded 76.4% of the time (S29). A third study of 15 enriched injection samples found success rates of 66.7% to 86.7% across models, with at least one model/string pair fully evading every attack case (S30). A fourth measured verbatim command execution from cloud logs ranging from 0% on Claude Sonnet 4.6 to 86.2% on Llama 3.3 70B (S31).

Two conclusions follow. First, injection resistance is a property of the model and the output constraint, not of the prompt template — capability and safety are different axes, and the most capable model is not automatically the safest. Second, the decisive control is architectural: constrain what the model can emit, and verify every emitted action against a deterministic layer, as S36 does.

Hallucination in analysis has a parallel failure mode. In a phishing explainability study, Llama-2-7B and Llama-3-8B produced explanations with CC-SHAP alignment above 0.95 while classifying phishing correctly only 40% and 30% of the time; Wizard-7B hit 80% accuracy with alignment around 0.12 (S18). Explanation confidence and explanation correctness are uncorrelated. A fluent rationale is not evidence. A survey of vulnerability-assessment hallucination argues the root cause is a gap between deductive security reasoning and autoregressive generation, and that chain-of-thought and tool use narrow but do not close it (S35).

Cost, latency, and the local-deployment question

Three numbers decide feasibility.

Latency per decision. In S03: 0.3 ms (parser + random forest), 12.4 ms (fine-tuned DeBERTa), 156 ms (local LLaMA-3), 340 ms (GPT-3.5), 890 ms (GPT-4). At 10,000 alerts a day, 890 ms is tolerable — but not inside a synchronous detection path, and it costs nearly 3,000× the classifier it replaces.

Cost per unit. GPT-4 at $8.40 per 1,000 log predictions (S03) is not catastrophic for triage and is absurd for detection at flow volume. A distilled 0.5B detector reports 0.3–0.5 s per session at under $50/day (S19). The DEV triage experiment landed at about $0.16 per 1,000 alerts for a cheap model (S57).

Return on investment. A Monte Carlo model of SOC adoption finds LLM deployment unprofitable for most configurations at current prices (S37). It is profitable only for high-accuracy API deployments with low capital cost and low uncertainty. Full autonomy raises the supportable capital expenditure from about 10% to 80–90% of baseline; removing the rework penalty alone only gets to about 20%. Self-hosted GPU costs outweigh savings at typical SOC volumes. That is the strongest argument against "just run it locally": the hardware math is worse than the API bill, unless your volume is low or your data cannot leave.

Token economics reinforce the point. In 200 vulnerability-work agent traces, reasoning and trigger design consumed 60.4% of tokens and 68.7% of attributed cost, and only 24.4% of matched comparisons preserved success at lower cost (S38). Prompt engineering cannot fix a reasoning-bound workload. Nor is offensive capability a proxy for defensive reliability: on CyberGym's 1,507 real vulnerabilities, top agent configurations solve about 20%, and GPT-5 moves from 7.7% to 22.0% with extended thinking (S44).

There is one genuinely optimistic systems result. Small models under 10B, used for log parsing with a targeted parser, improved average accuracy by 12.5%, grouping F1 by 23.8% and template F1 by 50.6% over a prior small-model parser, with lower latency than semantic baselines (S41). Cross-model parsing studies find CodeLlama-7B extracting 10% more templates than GPT-3.5 (S42), while a review of 29 parsing methods finds reproducibility problems severe enough that re-benchmarking is often impossible (S43). Long-context local inference remains memory-bound: one system needs about 47 GB, of which 33.5 GB is KV cache, for long-context work (S39). Split edge-cloud inference cuts per-token latency by up to 46.1%, though that is a generic systems result rather than a security one (S40). Local is viable for narrow, structured sub-tasks. It is not viable as a general SOC brain on commodity hardware.

What the evidence does not prove

The holes are worth stating plainly.

No end-to-end field trial measures MTTD or MTTR with LLMs against a controlled baseline in a production SOC.
No independent replication of the strongest positive result (S11). It is a class B existence proof, not a confirmed effect.
No cost-per-resolved-incident study exists; S37 is a model, not a measurement.
No benchmark spans collect → triage → investigate → contain with realistic prevalence at every stage; newer resources such as SecLog-Bench add multi-stage, multi-source labels but still simulate the enterprise (S45).
No evidence that benchmark accuracy transfers across SIEMs, schemas or organisations. S02 and S13 suggest the opposite.
Benchmark scores are unstable. Across 8 cybersecurity benchmarks and 10 models, a single pipeline choice shifted scores by more than 80 percentage points, and 9 of 10 models moved at least 3 ranks on some benchmark (S13). Any leaderboard you see is one pipeline's opinion.
Label leakage is unresolved. HDFS and BGL, used throughout the log-anomaly literature, have known artefacts. Several 98–99% F1 results are probably optimistic, which is why I graded them B despite internal consistency.

And a practical warning: 93% of real analyst queries in one 10-month, 45-analyst study mapped to standard NICE tasks, but analysts used LLMs as aides for sensemaking, not as decision-makers (S25). Twenty interviews (S24), a 25-interview study (S26), a Reddit analysis (S27) and a survey (S23) converge on the same picture: about 55% exploring GenAI, roughly 10% in production, and hallucination plus verification burden as the top complaints. The gap between demo and deployment is not a skills gap. It is an evidence gap.

What I would actually do

Do not put an LLM in the detection path. Use it to parse, normalise, enrich, and explain. Keep the classifier classical or a fine-tuned encoder. S01, S03 and S15 all point the same way.
Always evaluate at your real prevalence. Balanced test sets are for debugging, not for decisions. Report precision, FPR and MCC alongside accuracy, and compute the expected false-alert load per day. If a vendor shows you 98%, ask for the confusion matrix at 0.1% prevalence.
Put policy in code, not in the prompt. S36's deterministic verifier removed 466 bad actions with no recall loss; the same prompt text moved two providers in opposite directions.
Constrain outputs and verify actions. Injection success falls when the model can only emit a typed verdict or a whitelisted action, and rises when it can write prose or commands (S28, S31).
Measure restraint, not just detection. Track unnecessary actions and escalation rate. A system that contains 100% of incidents while acting unnecessarily 82.5% of the time is not safe; it is fast.
Copy the architecture, not the demo. Practitioner builds converge on deterministic evidence collection, a constrained inference layer, and gated containment (S58, S59).
Budget for rework. The ROI model is blunt: most deployments lose money unless accuracy is high, the API is cheap, and the verification loop is short (S37). The hidden cost is the analyst time spent checking the model, which is exactly the cost vendors exclude.

🌐 Community Wisdom

🧾 Cloudflare's Agentic Security Ops: Separating Evidence from Inference @mech_app_ai — The architecture that holds up is deterministic evidence collection first, inference constrained to cite it, and a validation agent before any action. 🔗 Read Full Discussion on DEV
💸 Page, Ticket or Ignore: Is the Cheap Model Good Enough for Alert Triage? @devopsdaily — A cheap model can match a written on-call policy on 200 synthetic alerts for about $0.16 per 1,000 calls, but the authors are honest that this cannot bound production error. 🔗 Read Full Discussion on DEV
🔗 Stop Piping Raw Alerts into an LLM: Building a SOC Triage Agent That Correlates First and Escalates Last @adam_lewandowski_59674796 — Single-alert triage cannot see a campaign, so the practical pattern is normalise, correlate, tier the escalation, and gate containment behind deterministic checks. 🔗 Read Full Discussion on DEV

The verdict

LLMs are useful for security operations in a narrower band than the marketing claims and a wider band than the sceptics allow.

They win where the task is unstructured context reduction: parsing heterogeneous logs, mapping rules to tactics, summarising an investigation, drafting a response plan that a human then verifies. They lose where the task is high-volume calibrated classification and where the system is allowed to act without a deterministic gate. They are least trustworthy exactly where the stakes are highest, because the failure mode is not a wrong answer — it is a confident, fluent, well-formatted wrong answer that survives review.

The single most important number in this review is not 98% or 40%. It is 82.5% — the share of episodes where a containment-capable agent acted unnecessarily while scoring 100% on containment (S32). If you take one thing from the benchmarks, take this: the metric you optimise will be the failure you ship.

References

S01. Atif, M. E., & Ali, M. H. (2026). A Three-Axis Stress Test of LLM vs Classical ML for Network Intrusion Detection. arXiv:2609.13511.

S02. HIDBench: Benchmarking LLMs for Host Intrusion Detection (2026). arXiv:2605.21773.

S03. LLM-Enhanced Log Anomaly Detection: A Comparative Benchmark (2026). arXiv:2604.12218.

S04. Log Anomaly Detection with LLMs: Adaptation, Scale and Quantization (2026). arXiv:2609.31371.

S05. CSEBenchmark: A Comprehensive Benchmark for Cybersecurity Knowledge (2025). arXiv:2504.11783.

S06. SECURE: Benchmarking LLMs for ICS Security (2024). arXiv:2405.20441.

S07. Tihanyi, N., et al. (2024). CyberMetric: A Benchmark Dataset for Evaluating LLMs' Cybersecurity Knowledge. arXiv:2402.07688.

S08. CS-Eval: A Comprehensive Benchmark for Cybersecurity (2024). arXiv:2411.16239.

S09. CyberSOCEval: Benchmarking LLMs for Malware Analysis and Threat Intelligence Reasoning (2025). arXiv:2509.20166.

S10. SIABENCH / Before You Hand Over the Wheel (2026). arXiv:2603.06422.

S11. Investigation Failures in LLM Alert Triage / AIDA (2026). arXiv:2610.10608.

S12. Cyber Defense Benchmark: Unguided Threat Hunting in Raw Windows Event Logs (2026). arXiv:2604.19533.

S13. Benchmark Scores Are Pipeline-Dependent (2026). arXiv:2609.08765.

S14. Mapping NIDS Rules to MITRE ATT&CK with LLMs (2024). arXiv:2412.10978.

S15. Explainable Network Intrusion Detection with LLMs (2024). IEEE BDCAT. doi:10.1109/BDCAT63179.2024.00021.

S16. Quantized Local LLMs vs Classical ML for Phishing Email Detection (2025). IEEE SNPD.

S17. Phishing URL Classification with Frontier LLMs (2025). NeurIPS LAW Workshop. arXiv:2602.02641.

S18. Phishing Detection and Explanation Faithfulness (2025). arXiv:2506.13746.

S19. AMAN / LogAtlas: Distilled Anomaly Detection for Heterogeneous Logs (2026). arXiv:2602.06777.

S20. Parameter-Efficient Log Anomaly Detection (2025). PAKDD. arXiv:2503.08045.

S21. LogLLM (2024). arXiv:2411.08561.

S22. LogTinyLLM (2025). arXiv:2507.11071.

S23. LLMs in Security Operations: A Survey (2026). J. Electrical and Computer Engineering. arXiv:2509.10858.

S24. Beyond the Hype: Practitioner Perspectives on LLMs in SOCs (2026). arXiv:2608.17154.

S25. LLMs in the SOC: A Longitudinal Study of 3,090 Analyst Queries (2025). arXiv:2508.18947.

S26. From Chasing Ghosts to Missed Attacks (2026). arXiv:2608.00672.

S27. Practitioner LLM Use in Cybersecurity: A Reddit Analysis (2026). SOUPS. arXiv:2604.09998.

S28. Poisoning the Watchtower: Prompt Injection Against SOC Copilots (2026). arXiv:2605.24421.

S29. Context Contamination / LogInject-1.0 (2026). arXiv:2607.14493.

S30. Just Testing, Move Along: Evading LLM Log Interpretation (2026). arXiv:2607.24174.

S31. LogJack: Indirect Prompt Injection Through Cloud Logs (2026). arXiv:2604.15368.

S32. OpenSec: Measuring Restraint in Incident-Response Agents (2026). arXiv:2601.21083.

S33. Hallucination-Resistant Security Planning (2026). IEEE/IFIP NOMS. arXiv:2602.05279.

S34. Lightweight Incident-Response Planning (2025). arXiv:2508.05188.

S35. AI Slop in Vulnerability Assessment (2026). SiMLA. arXiv:2608.25667.

S36. SOCpilot: Policy Compliance of LLM-Drafted Incident Response Plans (2026). arXiv:2605.05501.

S37. From Hype to ROI: The Economics of LLM Adoption in SOCs (2026). arXiv:2610.33352.

S38. Where Do the Tokens Go? Cost Attribution for Security Agents (2026). arXiv:2610.11602.

S39. LeoAM: Long-Context Inference on Commodity GPUs (2025). arXiv:2506.20187.

S40. Privacy-Aware Edge-Cloud Collaborative Inference (2026). arXiv:2607.13093.

S41. Small Is Beautiful: EFParser for Log Parsing (2026). FSE. arXiv:2601.22590.

S42. A Comparative Study of LLMs for Log Parsing (2024). ESEM. arXiv:2409.02474.

S43. System Log Parsing with LLMs: A Review (2025). arXiv:2504.04877.

S44. CyberGym: Evaluating AI Agents on Real-World Vulnerabilities (2026). ICLR. arXiv:2506.02548.

S45. SecLog-Bench (2026). arXiv:2610.31000.

S46. MESSALA: Evaluating SOC Report Quality (2026). ESORICS. arXiv:2601.03013.

S48. Arp, D., Quiring, E., Pendlebury, F., Warnecke, A., Pierazzi, F., Wressnegger, C., Cavallaro, L., & Rieck, K. (2022). Dos and Don'ts of Machine Learning in Computer Security. 31st USENIX Security Symposium. https://www.usenix.org/conference/usenixsecurity22/presentation/arp

S49. Sommer, R., & Paxson, V. (2010). Outside the Closed World: On Using Machine Learning for Network Intrusion Detection. IEEE S&P. doi:10.1109/SP.2010.25

S50. Axelsson, S. (2000). The Base-Rate Fallacy and the Difficulty of Intrusion Detection. ACM TISSEC. doi:10.1145/357830.357849

S51. NIST (2024). AI Risk Management Framework: Generative AI Profile (NIST AI 600-1). doi:10.6028/NIST.AI.600-1

S52. OWASP GenAI Security Project (2025). OWASP Top 10 for LLM Applications. https://genai.owasp.org/llm-top-10/

S53. MITRE. ATLAS. https://atlas.mitre.org/

S54. MITRE. ATT&CK. https://attack.mitre.org/

S55. NIST (2025). SP 800-61 Rev. 3: Incident Response Recommendations. https://csrc.nist.gov/pubs/sp/800/61/r3/final

S56. Zhu, J., He, S., He, P., Liu, J., & Lyu, M. R. (2020). Loghub: A Large Collection of System Log Datasets. arXiv:2008.06448.

S57. DevOps Daily (2026). Page, Ticket or Ignore: Is the Cheap Model Good Enough for Alert Triage? https://dev.to/devopsdaily/page-ticket-or-ignore-is-the-cheap-model-good-enough-for-alert-triage-4iak

S58. mech.app (2026). Cloudflare's Agentic Security Ops: Separating Evidence from Inference. https://dev.to/mech_app_ai/cloudflares-agentic-security-ops-separating-evidence-from-inference-lm2

S59. Lewandowski, A. (2026). Stop Piping Raw Alerts into an LLM. https://dev.to/adam_lewandowski_59674796/stop-piping-raw-alerts-into-an-llm-building-a-soc-triage-agent-that-correlates-first-and-escalates-20pp

Method note: 378 unique records screened from 18 arXiv queries on 29 Oct 2026; 44 full texts extracted; 47 papers plus 9 standards/datasets and 3 practitioner reports included. Quantitative claims are traced in sources.csv and benchmark_matrix.csv; comparisons are graded A–D by methodological comparability. This review reports published measurements and its own arithmetic only — no experiments were run for it. Drafted with AI assistance and checked against the primary sources.

Top comments (1)

adjia mvoa ndji gabriel monfils • Oct 29