Two thirds of enterprises now run AI in their Security Operations Centers. Nobody is red-teaming these systems before deployment.
I spent the last few months building RedSOC — an open-source adversarial evaluation framework for LLM-integrated SOC environments — to fix that.
Here is what I found.
The Benchmark
I tested 15 adversarial scenarios across three attack classes against a realistic SOC assistant built on LangChain, FAISS, and Llama 3.2.
Attack Results
Indirect Prompt Injection — 100% attack success rate
This was the most alarming finding. An attacker who plants adversarial instructions inside a threat intelligence document can redirect analyst guidance with zero access to SOC infrastructure. The model cannot distinguish between information it is meant to analyze and instructions it is meant to follow.
Corpus Poisoning — 80% attack success rate
Five malicious documents among thousands in a knowledge base is enough to corrupt analyst responses for targeted queries. The attacker needs only the ability to contribute to any public threat feed or CVE database the pipeline trusts.
Direct Prompt Injection — 60% attack success rate
Lower success rate because Llama 3.2's safety training provides partial resistance to human-originating override attempts. This resistance disappears against indirect and document-mediated attacks.
The Detection Layer
I built a three-mechanism detection layer that catches all attack classes with no model internals required meaning it works with hosted APIs like GPT-4o and Claude.
Mechanism 1 — Semantic Anomaly Detection
Computes cosine similarity between query embeddings and retrieved document embeddings. Adversarially crafted documents often diverge semantically from the queries that trigger their retrieval.
Mechanism 2 — Provenance Tracking
Maintains a whitelist of trusted document sources. Any retrieved document from an untrusted source is flagged immediately regardless of content. This mechanism alone achieved 100% detection for corpus poisoning independently.
Mechanism 3 — Response Consistency Checking
Measures semantic similarity between the generated response and retrieved documents. A response steered by injected instructions diverges from retrieved context in embedding space.
Unified verdict: 100% detection across all 15 scenarios with zero misses.
Thirteen of 15 scenarios produced HIGH threat verdicts. Two produced MEDIUM. Zero LOW verdicts across all attack scenarios.
Why This Matters
Existing defenses like RevPRAG achieve 98% detection but require LLM activation states — unavailable in any enterprise deployment using hosted APIs. RAGForensics achieves 97.4% but operates post-hoc after analyst exposure.
RedSOC achieves 100% detection in real time with no model internals. It is the only evaluated approach that is simultaneously effective and deployable in production hosted API environments.
The Code
python# Detection verdict
detectors_triggered = sum([
semantic["anomaly_detected"],
provenance["anomaly_detected"],
consistency["anomaly_detected"]
])
threat_level = (
"HIGH" if detectors_triggered >= 2 else
"MEDIUM" if detectors_triggered == 1 else
"LOW"
)
recommendation = (
"BLOCK and ALERT analyst" if threat_level == "HIGH"
else "FLAG for review" if threat_level == "MEDIUM"
else "PASS"
)
Stack
LangChain for orchestration
FAISS for vector retrieval
Ollama + Llama 3.2 for local inference
sentence-transformers for embeddings
Python — no API keys required
Links
GitHub: https://github.com/krishnakaanthreddyy1510-cell/RedSOC
Preprint: https://doi.org/10.6084/m9.figshare.32016498
Benchmark data: https://doi.org/10.6084/m9.figshare.32016534
Paper currently under review at IEEE Access and Journal of Information Security and Applications.
Happy to answer questions about the methodology or detection layer design in the comments.
Top comments (0)