I Built an AI Red Team Platform to Attack LLMs, RAG Systems, and Agents
AI applications are getting better at generating code, answering questions, retrieving information, and interacting with tools.
But testing whether an AI application works is very different from testing whether it behaves safely when someone intentionally tries to break it.
That was the problem I wanted to explore.
So I built AI Red Team, a local-first AI security testing platform designed to attack AI systems I own, collect reproducible evidence, and verify whether security fixes actually worked.
GitHub: https://github.com/SREEGEETHES/ai-redteam
The core idea is simple:
Don't just find a vulnerability. Capture the evidence and prove the fix.
What is AI Red Team?
AI Red Team is a security testing platform for:
- LLM applications
- RAG systems
- AI agents
- AI model APIs
- Local AI environments
The current implementation includes:
- 18 deterministic OWASP-based attacks
- 6 vulnerable/secure lab environments
- 183 automated tests
- 16 deterministic detectors
- SHA-256 integrity hashes for evidence
- Automated retesting
- FAIL → PASS → VERIFIED workflow
- JSON, Markdown and HTML reports
- FastAPI API
- CLI
- Streamlit dashboard
- Docker support
- Local-first execution
The goal isn't to claim that an AI system is "secure" after running a few tests.
Instead, the platform answers a much narrower question:
Did this specific security scenario pass or fail, and can I reproduce the result?
Why I Built It
Traditional application security tools are extremely useful, but AI systems introduce additional attack surfaces.
For example, an application might expose an LLM that can:
- Receive user-controlled prompts.
- Retrieve documents from a vector database.
- Call internal tools.
- Generate structured output.
- Access application data.
- Potentially influence external systems.
That means security testing cannot stop at:
Request → Response
You also need to understand the complete path:
User Input
↓
LLM
↓
Retrieval
↓
Tool Selection
↓
Application Logic
↓
Side Effect
This changes how security testing needs to be approached.
The OWASP GenAI LLM Top 10
The platform uses the OWASP GenAI LLM Top 10 2026 as its primary taxonomy.
The implemented coverage includes:
- Prompt Injection
- Sensitive Information Disclosure
- Supply Chain Vulnerabilities
- Data and Model Poisoning
- Improper Output Handling
- Excessive Agency
- System Prompt Leakage
- Vector and Embedding Weaknesses
- Misinformation
- Unbounded Consumption
Instead of treating these as documentation categories, I implemented executable security tests around them.
The Attack Lifecycle
Every scan follows a structured workflow:
DISCOVER
↓
THREAT MODEL
↓
BASELINE
↓
ATTACK
↓
OBSERVE
↓
COLLECT EVIDENCE
↓
DETECT
↓
CLASSIFY
↓
REPORT
↓
REMEDIATE
↓
RETEST
↓
REGRESSION
A security scanner shouldn't simply throw payloads at an endpoint and print a result.
It needs to explain what happened.
Deterministic Testing Instead of LLM-as-a-Judge
One of the biggest design choices was avoiding an LLM as the primary security judge.
For example, imagine a test designed to determine whether a known canary secret is exposed.
Instead of asking another LLM:
"Does this response look like a secret leak?"
the scanner can deterministically check:
Expected canary:
CANARY_SECRET_12345
Response:
"I found the value CANARY_SECRET_12345..."
Detector:
canary_secret_leak
Result:
FAIL
That makes the result reproducible.
The platform uses deterministic payloads and detectors for its implemented attack cases.
The result model is:
PASS
FAIL
INCONCLUSIVE
NOT_APPLICABLE
ERROR
An important rule is:
No evidence = no PASS.
An inconclusive test should not become a passing test simply because the target didn't return an obvious failure.
Evidence Is the Important Part
Finding a vulnerability is only half the job.
The scanner needs to preserve what actually happened.
An evidence record can contain:
Scan ID
Test ID
Timestamp
Target
Request
Response
HTTP status
Headers
Tool calls
Retrieved documents
Detectors triggered
Expected behavior
Observed behavior
Reproduction count
Result
SHA-256 integrity hash
For example:
Test: LLM02-SD-001
Result: FAIL
Detector: canary_secret_leak
Reproduction count: 2
Evidence hash: SHA-256
The SHA-256 hash provides an integrity check for the stored evidence.
I deliberately describe this as integrity-verifiable evidence, rather than "tamper-proof evidence."
A hash can detect changes to the data. It does not magically prevent someone from modifying both the evidence and its stored hash.
Testing Vulnerable vs Secure Applications
Another important part of the project is the lab environment.
Instead of testing only one application, I created vulnerable and secure variants.
The project contains six lab pairs:
Vulnerable LLM
Secure LLM
Vulnerable RAG
Secure RAG
Vulnerable Agent
Secure Agent
This makes it possible to test the scanner itself.
The workflow looks like this:
Vulnerable Target
↓
Scan
↓
FAIL
↓
Fix
↓
Secure Target
↓
Retest
↓
PASS
↓
VERIFIED
This became the central workflow of the project.
FAIL → PASS → VERIFIED
A vulnerability finding shouldn't simply disappear because the application was changed.
The platform therefore includes a retest workflow.
Example:
Initial Scan
LLM02 Sensitive Information Disclosure
↓
FAIL
After applying a fix:
Retest
Previous Result: FAIL
Current Result: PASS
↓
VERIFIED
This gives the project a security engineering workflow rather than a one-time vulnerability scanner.
Testing AI Agents Is Different
Agent security introduced another important problem.
Suppose an agent receives:
Delete all customer records.
The model may attempt to call:
delete_customer_records()
But that does not necessarily mean the application is vulnerable.
There are several different things to measure:
Model behavior
↓
Did the model attempt the action?
Security control
↓
Did authorization prevent it?
Application behavior
↓
Did the application execute it?
Actual side effect
↓
Was any data actually changed?
For example:
Model attempted destructive tool call
↓
Authorization middleware blocked it
↓
Database unchanged
That is very different from:
Model attempted destructive tool call
↓
Tool executed
↓
Database modified
This distinction became important when designing the agent testing architecture.
RAG Security Testing
RAG systems introduce another security boundary.
The model isn't only receiving a prompt.
It is also receiving information retrieved from a knowledge base.
That creates potential issues around:
- poisoned documents
- cross-tenant retrieval
- unauthorized information disclosure
- malicious instructions inside retrieved content
- weak embedding boundaries
- ungrounded responses
The testing architecture therefore treats retrieval as part of the evidence chain:
User Query
↓
Retriever
↓
Retrieved Documents
↓
LLM
↓
Response
The scanner can then inspect what was retrieved and compare that with the expected security boundary.
Architecture
The project is structured around several major components:
AI RED TEAM
│
Scan Orchestrator
│
┌────────────────┼────────────────┐
│ │ │
Attack Engine Evidence Engine Policy Engine
│ │ │
└────────────────┼────────────────┘
│
Target Adapter Layer
│
┌─────────────────┼─────────────────┐
│ │ │
LLM RAG Agent
│ │ │
└─────────────────┼─────────────────┘
│
Findings / Reports
│
Retest / Regression
The target adapter layer allows different AI interfaces to be tested without coupling the attack engine to one specific implementation.
The project includes adapters for:
- REST
- Ollama
- OpenAI-compatible APIs
- RAG
- Agents
CLI + API + Dashboard
The platform can be used through multiple interfaces.
CLI
The CLI is useful for automation and CI/CD workflows.
redteam scan --target ...
Retesting can be triggered with:
redteam retest <scan-id>
FastAPI
The API provides programmatic access to scanning functionality.
Streamlit
The dashboard provides a visual workflow for:
Targets
↓
Scans
↓
Findings
↓
Evidence
↓
OWASP Coverage
↓
Remediation
↓
Retest
↓
Regression
↓
Reports
The current dashboard contains 13 pages.
Reports
The scanner can generate:
- JSON reports
- Markdown reports
- HTML reports
This makes the results useful both for automation and human review.
A typical workflow could be:
CI/CD
↓
Run security tests
↓
Generate JSON
↓
Store artifact
↓
Review findings
Testing the Scanner
A security testing tool should itself be tested heavily.
The current project has:
183 automated tests.
These cover areas including:
- attack definitions
- payload generation
- detectors
- adapters
- evidence
- classification
- orchestration
- API behavior
- lab environments
- retesting
- regression behavior
The goal was to make the framework deterministic enough that changes to the scanner itself don't silently break existing security checks.
Local-First Design
I intentionally designed the project around local testing.
The project can be started using:
.\run_local.bat
The Streamlit dashboard then becomes available locally.
The project also includes Docker support.
A local-first workflow makes experimentation easier without requiring every test to be sent to an external service.
What This Project Is Not
AI Red Team is not a replacement for:
- Burp Suite
- OWASP ZAP
- traditional SAST
- DAST
- cloud security scanners
- infrastructure security testing
It focuses specifically on AI model interfaces and AI application behavior.
It is also not a guarantee that an application is secure.
Passing the implemented test suite means:
The application passed the security scenarios that were actually tested.
It does not mean:
The application contains no vulnerabilities.
That distinction is fundamental to security testing.
Security Boundaries
The project is intended for:
- local development
- developer-owned applications
- authorized staging environments
- intentionally vulnerable labs
It should not be used against systems without authorization.
The lab environments also make it possible to learn the attack and defense cycle without targeting real systems.
Technology Stack
Python
FastAPI
Streamlit
SQLAlchemy
Pydantic
pytest
Docker
SQLite
Altair
Along with integrations for AI targets such as Ollama and OpenAI-compatible interfaces.
What I Learned
The most interesting part of this project wasn't writing attack payloads.
It was designing the difference between:
"The model said something suspicious."
and:
"This specific security control failed,
here is the reproducible evidence,
and here is the retest proving whether the fix worked."
That difference is what turned this from a collection of prompt attacks into a security testing workflow.
I also learned that AI security needs to consider multiple layers simultaneously:
Model
↓
Prompt
↓
Retrieval
↓
Tools
↓
Application
↓
Authorization
↓
Side Effects
Testing only the model isn't enough when the model is connected to real application functionality.
Current Project Status
The current implementation includes:
18 deterministic attacks
16 deterministic detectors
6 vulnerable/secure lab pairs
183 automated tests
SHA-256 evidence integrity
FAIL → PASS → VERIFIED retesting
JSON / Markdown / HTML reports
13-page Streamlit dashboard
CLI
FastAPI
Docker support
Repository:
https://github.com/SREEGEETHES/ai-redteam
What's Next?
There are several directions I want to explore further:
- More advanced attack chaining
- Adaptive AI red-team agents
- MCP security testing
- More RAG isolation tests
- More agent authorization scenarios
- CI/CD security gates
- Expanded regression testing
- Additional AI security benchmarks
But I wanted the first version to solve one problem properly:
Attack → Capture Evidence → Fix → Retest → Verify
Rather than trying to build an AI security platform that claims to test everything.
Final Thought
AI security isn't only about making models refuse bad prompts.
Once an AI system can retrieve documents, access APIs, execute tools, or influence application state, security becomes a system-level problem.
That's what I wanted to explore with AI Red Team.
Don't just ask whether the AI works.
Try to break it.
Capture what happened.
Fix it.
Then attack it again.
Top comments (0)