DEV Community: suraj kumar

swarm-test v0.3.3 — I Visualized My 14-Agent System and the Bottleneck Was Obvious

suraj kumar — Thu, 18 Jun 2026 18:27:13 +0000

Documentation rots. You draw your multi-agent architecture once, then six months later it's wrong because the system changed and nobody updated the picture.

swarm-test v0.3.3 generates the diagram from your actual agent topology, so it stays accurate. And when I ran it on my own 14-agent system, the problem jumped out immediately.

[ATTACH THE AREENGINE DIAGRAM HERE]

That red node is OrchestratorAgent — a single point of failure with 92% blast radius. Look at the shape: nearly every one of the 14 agents funnels into two hubs, OrchestratorAgent and TrainerAgent. If either goes down, the system collapses. You can't see that in a console table. You see it in one glance at the diagram.

Generating it is one command:

swarm-test graph my_crew.py --format mermaid

You get Mermaid syntax to paste straight into a GitHub README:

graph TD
OrchestratorAgent[OrchestratorAgent ⚠️ SPOF]:::spof
TrainerAgent[TrainerAgent]:::healthy
ImageValidatorAgent --> OrchestratorAgent
classDef spof fill:#ff4444,stroke:#cc0000,color:#fff
classDef healthy fill:#44cc44,stroke:#22aa22,color:#fff

GitHub, GitLab, and Notion render this natively. Single points of failure show red, healthy agents green, moderate-risk yellow — the same risk classification from the reliability analysis.

Three formats depending on where the diagram is going:

Mermaid — for READMEs and wikis. Renders inline, version-controllable as text, diffs cleanly in PRs.

DOT — for Graphviz pipelines and custom tooling.

PNG — for slide decks and external docs:

swarm-test graph my_crew.py --format png --output topology.png

(PNG needs matplotlib: pip install swarm-test[png])

The value scales with system size. On a 3-agent crew you can already see the structure in the console. But on a 14-agent system with 40+ edges, the diagram reveals clusters, bottlenecks, and isolated agents that a table can't show. The shape of the problem becomes visible — and the fix becomes obvious. In my case: break the Orchestrator bottleneck by distributing routing across multiple agents.

Because the diagram comes from the same graph analysis that runs the reliability tests, your documentation and your testing never disagree.

Works across CrewAI, LangGraph, AutoGen, and custom orchestrators.

pip install swarm-test --upgrade
GitHub: github.com/surajkumar811/swarm-test

swarm-test v0.3.2 — Write Your Own Multi-Agent Reliability Tests

suraj kumar — Wed, 17 Jun 2026 05:29:08 +0000

swarm-test v0.3.2 adds a plugin system. You can now write custom reliability tests for your specific multi-agent architecture.

The built-in tests cover universal failure modes — cascade failures, context leakage, intent drift, collusion, blast radius, timeout resilience, sensitive data detection, and contract violations. But every team has domain-specific risks that a generic tool can't anticipate. Maybe you need to check that your billing agent never communicates directly with your data deletion agent. Maybe you need to verify that no agent chain exceeds 5 hops. Maybe you have compliance requirements unique to your industry.

Now you can build those checks yourself and they run alongside everything else.

Writing a plugin takes about 10 lines:

from swarm_test.plugins import BasePlugin, PluginResult
from swarm_test.core.models import Finding

class MaxHopsPlugin(BasePlugin):
name = "max_hops_check"
version = "0.1.0"
description = "Warns if any agent chain exceeds N hops"

  def run(self, graph, agents, edges, config):
      findings = []
      # your test logic using the NetworkX graph
      return PluginResult(
          test_name=self.name,
          status="passed" if not findings else "failed",
          score=100,
          findings=findings,
          duration_ms=0.0
      )

[project.entry-points."swarm_test.plugins"]
max_hops_check = "my_package:MaxHopsPlugin"

Install your package and swarm-test discovers it automatically:

swarm-test plugins list

Plugin findings appear everywhere — console output, JSON export, HTML reports, GitHub Action annotations, CI/CD gates. They respect the same YAML config filtering (enabled_tests/disabled_tests) as built-in tests. One failing plugin doesn't crash the rest of the run.

The graph object your plugin receives is a full NetworkX DiGraph with all agent nodes, edges, and metadata. You have access to every graph algorithm NetworkX provides — centrality, shortest paths, connected components, community detection. The agents and edges lists give you the full swarm-test model with roles, tools, health scores, and redundancy data.

What I'd love to see the community build: rate limit validation (does any agent path exceed API rate limits), cost estimation plugins (token counting per path), compliance-specific checks (HIPAA, SOC 2, GDPR agent isolation), framework-specific tests that go deeper than the generic adapters.

If you build a plugin, open an issue on the repo and I'll add it to a community plugins directory.

pip install swarm-test --upgrade
GitHub: github.com/surajkumar811/swarm-test

swarm-test v0.3.1 — Interactive HTML Reports and Developer Experience Overhaul

suraj kumar — Sun, 14 Jun 2026 19:15:30 +0000

Major update to swarm-test — the open-source multi-agent reliability testing tool.

The problem with CLI output: tools dump everything. You run the test, get 200 lines, and scroll back trying to find what matters. For CI scripts, you need one line. For debugging, you need everything. For daily use, you need something in between. Most tools pick one mode. That's wrong.

swarm-test v0.3.1 adds three output modes:

Default — first line is the verdict: "Swarm Score: 0/100 — CRITICAL (5 critical, 1 high findings)" followed by only CRITICAL and HIGH findings with actionable fixes. Lower-severity findings hidden with a note.

Quiet (--quiet) — one line only. "Swarm Score: 10/100 — CRITICAL (2 critical findings)". Exit code does the rest. 0 = pass. 1 = threshold exceeded. Perfect for CI scripts.

Verbose (--verbose) — everything. All findings including LOW and INFO. Full graph metrics. All agent health details. Complete redundancy table.

Every finding now ends with a specific fix, not just a problem statement:

CRITICAL | cascade_failure
Catastrophic cascade potential: Hub failure cascades to 5 agents
→ Add a fallback agent for 'Hub' or distribute its responsibilities across multiple agents.

The big addition is the interactive HTML report. Run: swarm-test run crew.py --output-format html --output-path report.html --open

Your browser opens with a full dashboard:

Swarm Score Gauge — large circular gauge showing 0-100 with certification level (EXCELLENT, GOOD, NEEDS IMPROVEMENT, AT RISK, CRITICAL). One look tells you the state of your system.

Agent Interaction Graph — D3 force-directed graph. Nodes are agents, sized by connections, colored by health (green/yellow/red). SPOF agents get a pulsing red border. Drag to reposition, scroll to zoom, click to highlight edges.

Interaction Heatmap — NxN grid showing which agent pairs communicate most. Darker = more interactions. Red overlay = findings on that edge. Instantly see where the risky connections are.

Health Scores Table — sortable with colored progress bars. Each agent shows its score, status, and specific risk details like "100% blast radius, SPOF, high cascade depth."

Redundancy Table — replaceability scores from IRREPLACEABLE (0-20) to FULLY REDUNDANT (81-100). SPOFs highlighted in red with green progress bars for safe agents.

Findings Section — filter buttons (ALL / CRITICAL / HIGH / MEDIUM / LOW). Each finding is collapsible — click to expand for full description, affected agents, and remediation steps.

Everything else still works. Same 8 reliability tests (cascade failure, context leakage, intent drift, collusion detection, blast radius, timeout resilience, sensitive data detection, contract violation). Same 3 framework adapters (CrewAI, LangGraph, AutoGen). Same YAML config with auto-discovery. Same GitHub Action for CI/CD gating. Same JSON and Markdown exports. Nothing removed, everything improved.

Install: pip install swarm-test --upgrade

What's next: plugin system — write your own custom reliability tests with a simple BasePlugin interface.

GitHub: github.com/surajkumar811/swarm-test

swarm-test is now a GitHub Action — multi-agent reliability testing on every PR

suraj kumar — Sat, 13 Jun 2026 16:32:11 +0000

swarm-test v0.3.0 turns multi-agent reliability testing into a CI/CD gate.

The Setup

Add this to .github/workflows/reliability.yml:

name: Agent Reliability
on: [pull_request]
jobs:
swarm-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: surajkumar811/swarm-test@v0.3.0
with:
script: my_crew.py
fail-on-severity: high

That's it. Every PR now gets tested for cascade failures, blast radius, context leakage, intent drift, collusion, timeout resilience, contract violations, and single points of failure.

What You See on the PR

Findings show up as inline annotations:

Critical findings → errors (block the merge)
High findings → warnings
Medium findings → notices

Plus a job summary with your Swarm Score and the top findings with remediation steps.

Why This Matters

Most teams test individual agents and call it done. But the failures that take down production live in the interactions between agents — and those only surface when you test the whole graph.

Running this manually means you test when you remember. Running it in CI means you test every single change, automatically, before it merges.

Works Across Frameworks

CrewAI, LangGraph, AutoGen — same action, same config. The graph topology is what gets tested, not the framework.

pip install swarm-test --upgrade
GitHub: github.com/surajkumar811/swarm-test

swarm-test v0.2.8 — This Reliability Report Found a Catastrophic Failure Nobody Caught

suraj kumar — Wed, 10 Jun 2026 10:49:06 +0000

I ran swarm-test v0.2.8 on a 4-agent system. The result: Risk Score 90/100.
One agent is holding the entire system together — and nobody knew.
What the report found:
The "Hub" agent has a health score of 15/100. It's classified as IRREPLACEABLE with a blast radius of 100%. If Hub fails, every downstream agent — Worker1, Worker2, Worker3 — receives corrupted input. The system throws no errors. Logs look clean. Output is silently garbage.
Meanwhile all three workers scored 80-100/100 and are FULLY REDUNDANT. The individual agents are fine. The architecture is the vulnerability.
No code review caught this. No unit test caught this. Only graph-based chaos testing exposed the structural weakness.
What's new in v0.2.8:
Per-agent redundancy scoring with SPOF detection. Every agent in your system gets classified: IRREPLACEABLE, PARTIALLY REDUNDANT, or FULLY REDUNDANT. You see exactly which agents are safe to lose and which ones would take down your entire pipeline.
This sits on top of the 6 chaos tests swarm-test has run since v0.1.0: cascade failure analysis, blast radius mapping, intent drift measurement, context leakage detection, collusion detection, and timeout resilience testing.
Why this matters:
I found 54 of these failures in my own 14-agent production system. 15 were CRITICAL. The system had been running for weeks looking perfectly healthy. It wasn't.
Every team building multi-agent AI systems with CrewAI, LangGraph, or AutoGen has these hidden vulnerabilities. The only question is whether you find them before your users do.Try it:
Search "swarm-test" on PyPI. MIT licensed. 78 tests passing. Works with CrewAI and LangGraph.
What does YOUR agent system's risk score look like?

swarm-test now supports AutoGen — 3 frameworks, 1 reliability testing tool

suraj kumar — Mon, 08 Jun 2026 07:51:00 +0000

Quick update: swarm-test v0.2.7 adds AutoGen support.

The Problem

The multi-agent ecosystem is fragmenting. Teams build with CrewAI, LangGraph, AutoGen, or a mix. But the failure modes are identical across all of them:

Cascade failures where one agent takes down the chain
Context leaking between agents that shouldn't share data
Intent drift where instructions get distorted through handoffs
Contract violations where Agent A outputs something Agent B doesn't expect

Testing shouldn't fragment just because your framework choice did.

What's New

swarm-test v0.2.7 adds full AutoGen support:

GroupChat and GroupChatManager detection
ConversableAgent, AssistantAgent, UserProxyAgent extraction
Speaker transition mapping (allowed_transitions, speaker_selection_method)
Tool/function extraction from agent function maps

Same 7 reliability tests run identically across all three frameworks:

Cascade failure
Context leakage
Intent drift
Collusion detection
Blast radius mapping
Timeout resilience
Output contract validation

Usage

pip install swarm-test --upgrade

# Test a CrewAI crew
swarm-test run my_crew.py

# Test a LangGraph graph
swarm-test run my_graph.py

# Test an AutoGen GroupChat
swarm-test run my_groupchat.py

Framework is auto-detected. No flags needed.

With YAML Config

# .swarmtest.yml
fail_on_severity: high
max_blast_radius: 0.75
enabled_tests:
  - cascade
  - blast_radius
  - contract_violation

Same config works across all frameworks. Drop it in your project root, swarm-test picks it up automatically.

Why This Matters

Most teams pick a framework and build testing around its specific API. Then they add a second framework for a different use case and their testing breaks. Or they migrate from CrewAI to LangGraph and lose all their reliability coverage.

swarm-test tests the interaction graph, not the framework. The graph topology, blast radius, and failure modes are the same whether you built with CrewAI, LangGraph, or AutoGen.

What's Next

Redundancy scoring — how replaceable is each agent?
GitHub Action — swarm-test as a CI/CD gate on every PR
Interaction heatmap — visual map of agent communication patterns

GitHub: github.com/surajkumar811/swarm-test

🐝 swarm-test v0.2.4 — Quick Scan CLI

suraj kumar — Wed, 03 Jun 2026 07:50:16 +0000

Test any agent system in 30 seconds. No Python needed:

swarm-test scan \
--agents "Researcher,Analyst,Writer,Reviewer" \
--edges "Researcher>Analyst,Analyst>Writer,Writer>Reviewer"

One command. 13 findings. 2 SPOFs. Health scores for every agent.

pip install swarm-test --upgrade

I Found 54 Reliability Issues in My 14-Agent AI System — Here's What Broke

suraj kumar — Sun, 31 May 2026 00:46:10 +0000

Every testing tool for AI agents tests individual agents. But production failures don't happen inside agents — they happen between them.

I learned this the hard way.

The Problem Nobody Is Solving

I built a 14-agent document processing system using CrewAI. Each agent worked perfectly in isolation. In production, the system failed constantly — and I couldn't figure out why.

The problem wasn't any single agent. It was the interactions:

One agent failing silently took down 12 others
Agents were sharing sensitive data across boundaries they shouldn't cross
Three agents formed a communication clique that bypassed the orchestrator
Every agent depended on one central orchestrator with zero fallback

No existing tool could find these issues. Arize, Langfuse, Braintrust — they all monitor individual agents. None of them test the graph of agent interactions.

So I built one.

What I Built: swarm-test

swarm-test builds a NetworkX interaction graph of your multi-agent system and runs 6 chaos engineering tests against it:

Cascade Failure — which agents bring down the whole system if they fail
Context Leakage — sensitive data (API keys, PII, credentials) crossing agent boundaries
Intent Drift — agents acting outside their role or being manipulated
Collusion Detection — agents communicating outside the orchestrator's oversight
Blast Radius — single points of failure and critical dependency paths
Timeout Resilience — agents with no fallback if upstream is slow

3-line API:

from swarm_test import SwarmProbe

probe = SwarmProbe(crew)
report = probe.run_all()
report.print_summary()

What It Found On My Real System

I ran swarm-test on my 14-agent system. The results were brutal:

54 total findings:

15 CRITICAL (14 cascade failures + 1 SPOF)
13 HIGH (9 timeout vulnerabilities + 4 collusion cliques)
26 MEDIUM (13 intent drift + 13 missing timeout handling)

The worst agent: OrchestratorAgent scored 4 out of 100. It's a single point of failure with 92% blast radius — if it fails, 12 of 14 agents go down. And it had zero timeout handling.

The scariest finding: EvolutionAgent has 100% blast radius. If it fails, every other agent in the system is affected.

Three agents (OrchestratorAgent, FileOptimizerAgent, PrintOptimizerAgent) formed a collusion clique — communicating directly with each other and bypassing orchestrator oversight.

None of this was visible from testing individual agents. It only appeared when I tested the interaction graph.

I Shipped 7 Features in 7 Days

After launching, I shipped one feature every day:

Day	Feature	Impact
0	Launch — 5 chaos tests, GitHub + PyPI	First multi-agent testing tool on PyPI
1	Timeout resilience test	Found 22 new issues in my system
2	JSON export	Another developer integrated it into his runtime gate within hours
3	LangGraph adapter	Now supports CrewAI + LangGraph
4	Sensitive data detection (23 patterns)	Catches AWS keys, JWT tokens, credit cards crossing agent boundaries
5	Per-agent health scores (0-100)	Know exactly which agent to fix first
6	Before/after comparison	Measure if your refactor actually improved reliability
7	ASCII agent graph	See your agent topology right in the terminal

94 tests passing. Two frameworks supported. And growing.

The First Integration

Within 48 hours of launch, another developer built an integration. He has a runtime action-gate that blocks dangerous agent actions before execution. He connected swarm-test's findings as "priors" — so when swarm-test flags an edge as high-risk, his gate becomes more cautious on that edge.

The result: the same run_sql action went from "CONFIRM" (risk 62) to "HUMAN_REQUIRED" (risk 78) when swarm-test's cascade finding was attached.

Structural testing (swarm-test) + runtime enforcement (his gate) = the full reliability stack for multi-agent systems.

Why This Matters Now

According to recent industry research:

88% of organizations report AI agent security incidents
Only 14.4% of agents go live with full security approval
OWASP classified cascade failures as ASI08 — a top AI security risk

Multi-agent systems are going to production faster than anyone can secure them. The tools exist for single-agent monitoring. Nothing existed for multi-agent interaction testing — until now.

Try It

pip install swarm-test

from swarm_test import SwarmProbe

# Works with CrewAI
probe = SwarmProbe(your_crew)
report = probe.run_all()
report.print_summary()
report.to_html("report.html")  # Interactive D3 graph
report.to_json("report.json")  # Machine-readable for CI/CD

GitHub: github.com/surajkumar811/swarm-test

Open source. MIT licensed. Solo founder building in public.

What reliability tests would YOU want for your multi-agent systems? Drop a comment — I'm shipping features based on real feedback.

I Ship One AI Testing Feature Every Day — Here's What 6 Days Looks Like

suraj kumar — Thu, 28 May 2026 17:13:57 +0000

I launched swarm-test 6 days ago. It's the first reliability testing tool
for multi-agent AI systems. Here's what I've shipped every single day:

Day 0: Launch — 5 chaos tests, GitHub + PyPI
Day 1: Timeout resilience — found 22 new issues in my 14-agent system
Day 2: JSON export — another developer integrated it into his runtime gate within hours
Day 3: LangGraph adapter — doubled the addressable user base
Day 4: Sensitive data detection — 23 pattern types (AWS keys, JWT, credit cards)
Day 5: Per-agent health scores — every agent gets a 0-100 rating
Day 6: Before/after comparison — measure if changes actually helped

[Include screenshots from each day]

What I learned: shipping daily does three things:

Forces you to keep features small and shippable
Gives you something to post about every day
Shows users the project is alive and actively maintained

What's next: AutoGen adapter, GitHub Action, YAML config,
plugin system for community tests.

Try it: pip install swarm-test
GitHub: github.com/surajkumar811/swarm-test .

🐝 swarm-test v0.1.5— Per-Agent Health Scores

suraj kumar — Wed, 27 May 2026 19:04:05 +0000

My 14-agent system's results:

OrchestratorAgent scored 4/100 — single point of failure with 3 collusion cliques
FaceDetectorAgent scored 44/100 — high cascade depth
EvolutionAgent scored 50/100 — 100% blast radius

Stop guessing which agent to fix. Let the data tell you.

5 features in 5 days. 84 tests. CrewAI + LangGraph.

github.com/surajkumar811/swarm-test

AI #MultiAgent #Reliability #OpenSource