The agent generates code. The orchestrator tries to find reasons not to trust it.
That sentence is the entire pivot. Earlier versions of swarm-orchestrator coordinated multiple agents working on the same task. v7 wraps a single agent CLI (Copilot, Claude Code, or Codex) and runs five independent checks on the patch before allowing a merge.
The five checks
| # | Layer | What it catches | Gate type |
|---|---|---|---|
| 1 | Intent verification | Patch doesn't actually fix the stated problem | Hard |
| 2 | Regression verification | Patch breaks existing behavior, or test coverage is too weak to know | Hard |
| 3 | Solution quality | Agent gamed the test (hardcoded values, swallowed exceptions, modified tests) | Composite |
| 4 | Behavioral verification | Patch works on the happy path, crashes on edge cases | Composite |
| 5 | Provenance | No signed attestation produced for the merge | Composite |
1. Intent verification
The patch must make a previously-failing test pass. For SWE-bench instances, that's the FAIL_TO_PASS test from the instance JSON. For user-facing goals, a reviewer synthesizes a regression test before the worker runs and confirms it fails against the base commit first.
2. Regression verification
Existing tests must pass. Then mutation testing runs on the modified files to check whether coverage around the change is actually strong enough to catch regressions. A patch that works but lives in weakly-tested code gets flagged.
Mutation score thresholds are configurable in Mutation testing tooling per language
.swarm/gates.yaml. Defaults: below 0.6 fails, 0.6 to 0.8 warns, above 0.8 passes.
3. Solution quality
A Semgrep rule pack scans for the specific shortcuts agents take when they're being graded:
- Hardcoded values matching test expectations
-
try/catchblocks swallowing the exact exception a failing test was asserting on - Modifications to test files outside the stated scope
- Mock mutations that make tests pass without changing the implementation
4. Behavioral verification
Property-based testing runs against modified functions for 60 seconds each, using Hypothesis (Python) or fast-check (TypeScript). Counterexamples that crash the patched code or violate type contracts get reported.
5. Provenance
A signed SLSA v1.0 attestation is generated for each verified merge and attached to the commit as a git note. Signed via cosign keyless OIDC. The attestation contains agent identity, model version, per-layer results, and the composite score.
swarm attest verify <commit>
That command pulls the note and verifies the signature. Useful when something breaks in production three months later and someone asks which agent wrote the offending code and what was checked at merge time.
Status
Where this goes next
v8 brings parallel execution back, applied to verification instead of generation.
The orchestrator will compute a risk score for each patch, then spawn a population of independent falsifiers sized to that risk. Falsifiers share findings through a coordination channel so a discovery from one steers the targeting of others. A bandit selects which falsifier types to spawn based on past outcomes.
The v7 five-layer battery becomes the seed pool that v8 grows from. The project name finally fits.
moonrunnerkc
/
swarm-orchestrator
Independent verification battery for patches written by AI coding agents. Wraps Copilot, Claude Code, and Codex; applies a five-layer falsification battery (intent, mutation, cheat detection, property tests, signed attestation) to gate merges.
Swarm Orchestrator
Independent verification battery for patches written by AI coding agents.
Wraps third-party coding-agent CLIs (Copilot, Claude Code, Codex), runs worker and reviewer steps on isolated git branches, and applies a five-layer falsification battery to each agent-authored patch. Hard gates block patches that fail intent or regression checks; advisory layers feed a composite score.
You run this around an agent CLI, not instead of one. The agent produces the patch; the orchestrator tries to break it. Patches that survive merge to main; patches that don't are rolled back with a verification report.
Features
-
Five-layer falsification battery. Intent verification, regression and mutation testing, cheat detection, property-based testing, and signed attestation. Layers 1 and 2 are hard gates; layers 3 to 5 feed an advisory composite score. Implementations live under
src/verification/. - Isolated worker and reviewer steps. Each step runs…
Top comments (0)