Every testing tool for AI agents tests individual agents. But production failures don't happen inside agents — they happen between them.
I learned this the hard way.
The Problem Nobody Is Solving
I built a 14-agent document processing system using CrewAI. Each agent worked perfectly in isolation. In production, the system failed constantly — and I couldn't figure out why.
The problem wasn't any single agent. It was the interactions:
- One agent failing silently took down 12 others
- Agents were sharing sensitive data across boundaries they shouldn't cross
- Three agents formed a communication clique that bypassed the orchestrator
- Every agent depended on one central orchestrator with zero fallback
No existing tool could find these issues. Arize, Langfuse, Braintrust — they all monitor individual agents. None of them test the graph of agent interactions.
So I built one.
What I Built: swarm-test
swarm-test builds a NetworkX interaction graph of your multi-agent system and runs 6 chaos engineering tests against it:
- Cascade Failure — which agents bring down the whole system if they fail
- Context Leakage — sensitive data (API keys, PII, credentials) crossing agent boundaries
- Intent Drift — agents acting outside their role or being manipulated
- Collusion Detection — agents communicating outside the orchestrator's oversight
- Blast Radius — single points of failure and critical dependency paths
- Timeout Resilience — agents with no fallback if upstream is slow
3-line API:
from swarm_test import SwarmProbe
probe = SwarmProbe(crew)
report = probe.run_all()
report.print_summary()
What It Found On My Real System
I ran swarm-test on my 14-agent system. The results were brutal:
54 total findings:
- 15 CRITICAL (14 cascade failures + 1 SPOF)
- 13 HIGH (9 timeout vulnerabilities + 4 collusion cliques)
- 26 MEDIUM (13 intent drift + 13 missing timeout handling)
The worst agent: OrchestratorAgent scored 4 out of 100. It's a single point of failure with 92% blast radius — if it fails, 12 of 14 agents go down. And it had zero timeout handling.
The scariest finding: EvolutionAgent has 100% blast radius. If it fails, every other agent in the system is affected.
Three agents (OrchestratorAgent, FileOptimizerAgent, PrintOptimizerAgent) formed a collusion clique — communicating directly with each other and bypassing orchestrator oversight.
None of this was visible from testing individual agents. It only appeared when I tested the interaction graph.
I Shipped 7 Features in 7 Days
After launching, I shipped one feature every day:
| Day | Feature | Impact |
|---|---|---|
| 0 | Launch — 5 chaos tests, GitHub + PyPI | First multi-agent testing tool on PyPI |
| 1 | Timeout resilience test | Found 22 new issues in my system |
| 2 | JSON export | Another developer integrated it into his runtime gate within hours |
| 3 | LangGraph adapter | Now supports CrewAI + LangGraph |
| 4 | Sensitive data detection (23 patterns) | Catches AWS keys, JWT tokens, credit cards crossing agent boundaries |
| 5 | Per-agent health scores (0-100) | Know exactly which agent to fix first |
| 6 | Before/after comparison | Measure if your refactor actually improved reliability |
| 7 | ASCII agent graph | See your agent topology right in the terminal |
94 tests passing. Two frameworks supported. And growing.
The First Integration
Within 48 hours of launch, another developer built an integration. He has a runtime action-gate that blocks dangerous agent actions before execution. He connected swarm-test's findings as "priors" — so when swarm-test flags an edge as high-risk, his gate becomes more cautious on that edge.
The result: the same run_sql action went from "CONFIRM" (risk 62) to "HUMAN_REQUIRED" (risk 78) when swarm-test's cascade finding was attached.
Structural testing (swarm-test) + runtime enforcement (his gate) = the full reliability stack for multi-agent systems.
Why This Matters Now
According to recent industry research:
- 88% of organizations report AI agent security incidents
- Only 14.4% of agents go live with full security approval
- OWASP classified cascade failures as ASI08 — a top AI security risk
Multi-agent systems are going to production faster than anyone can secure them. The tools exist for single-agent monitoring. Nothing existed for multi-agent interaction testing — until now.
Try It
pip install swarm-test
from swarm_test import SwarmProbe
# Works with CrewAI
probe = SwarmProbe(your_crew)
report = probe.run_all()
report.print_summary()
report.to_html("report.html") # Interactive D3 graph
report.to_json("report.json") # Machine-readable for CI/CD
GitHub: github.com/surajkumar811/swarm-test
Open source. MIT licensed. Solo founder building in public.
What reliability tests would YOU want for your multi-agent systems? Drop a comment — I'm shipping features based on real feedback.
Top comments (1)
If anyone wants to see what swarm-test finds on their own agent system, I'm happy to run it and share the results. Just describe your agent setup (framework, number of agents, how they connect) and I'll generate a report.