There's a blind spot in how we test multi-agent AI systems, and almost every team has it.
We test agents individually — prompt evals, output scoring, regression sets. The individual agent has never been better tested. And yet multi-agent systems keep failing in production, in ways no eval predicted.
The reason is simple and uncomfortable: the failures don't live in the agents. They live in the wiring between them.
The math nobody runs
14 agents, each 95% reliable. End-to-end: 0.95^14 ≈ 49%.
Your system can be a coin flip while every individual agent passes every test you throw at it. Reliability doesn't average across a pipeline — it multiplies. Every handoff is a multiplication.
The three failures per-agent testing cannot see
1. The loop with no exit. Agent A calls B, B's output routes back to A. Under the right input they ping-pong indefinitely — burning tokens with every lap, throwing no error, dashboards green. You find it on the invoice.
2. The SPOF nobody designed. Teams build an orchestrator (sensible), and over time everything accretes onto it — every handoff, every retry. Nobody decided it should be a single point of failure; it just became one. One slow call in that agent stalls the entire pipeline, silently.
3. The unguarded handoff. Agent A passes structurally-valid but semantically-broken data to B. B processes it happily. The error surfaces three agents downstream, where no one can trace it back.
Common thread: each agent involved passes its own tests. The failure only exists at the system level — in the edges, not the nodes.
Why this stays invisible
Our whole testing culture is node-centric. Unit tests test functions. Agent evals test agents. But a multi-agent system is a graph, and graphs fail in graph ways: cycles, articulation points, cascade paths, unvalidated edges.
Here's the part that should change how you work: these are structural properties. A loop with no exit edge is detectable from the topology alone. So is a hidden articulation point. So is a handoff with no validator between two agents. You don't need to run the system to find them — the same way an architect doesn't need the building to collapse to spot the missing load-bearing wall in the blueprint.
Test the edges
This is the gap I've been building for. swarm-test maps your agent topology (CrewAI, LangGraph, AutoGen, or a plain agents/edges description) as a directed graph and statically flags the structural failures — no-exit loops, hidden SPOFs, cascade blast radius, unguarded handoffs, orchestrator-bypass cycles. Deterministic, milliseconds, no LLM calls, runs in CI (--ci fails your build on structural regressions).
Structural analysis is the first layer — the honest scope of what static analysis can see. The direction from here is using that structure to make deeper testing targeted: the graph tells you exactly where a system is fragile, which is exactly where runtime testing should aim.
pip install swarm-test
Open source, MIT. If you're running agents in production, run it on your topology or if you want I run and give you result and help you — worst case, you confirm your wiring is clean. And if you've hit a silent wiring failure of your own, tell me about it in the comments. Those stories are how we all learn where the edges break.
If this was useful, a star on GitHub is how other devs find it.
Top comments (0)