DEV Community: Brad Kinnard

How Swarm Orchestrator v8 Tries to Break Its Own AI Patches

Brad Kinnard — Sun, 10 May 2026 02:10:05 +0000

Most AI coding tools commit when their own checks pass. Swarm Orchestrator v8 adds a second adversarial layer: independent falsifier adapters that try to break each patch before it merges. v8.0.1 is on main with that subsystem on by default.

This post walks through the v8 architecture, the four verification points, the producer/falsifier adapter split, and the limitations that haven't been solved in v8.0 yet.

What is Swarm Orchestrator? A contract-first AI coding swarm with hash-chained evidence and verifier-gated commits. It compiles a natural-language goal into a typed contract, dispatches it to a population of personas inside one cached Anthropic session, races candidate diffs per obligation, and commits only what passes verification. It wraps an LLM; it doesn't replace one.

The shape of a run

You hand it a goal in plain English. The contract compiler turns that into contract.jsonl plus a manifest.json carrying the goal, repo context, extractor provenance, and a SHA-256 of the canonical contract bytes. Identical inputs produce identical contract hashes.

goal (text)
   |
   v
contract compiler  ->  contract.jsonl + manifest.json
   |
   v
+-------------------------------------------------+
|        population manager (single session)      |
|                                                 |
|  ledger (jsonl, hash-chain) <- personas (8)     |
|       ^                          |              |
|       | tournament + verifier scoring           |
|       |                                         |
|  WASM deterministic floor (zero-LLM obligs)     |
+-------------------------------------------------+
   |                              |
   v                              v
streaming verifier      post-merge integration
   |                              |
   +--------------+---------------+
                  v
       falsifier adapters (Codex, Copilot)
                  |
                  v
            committed diffs

The population manager opens one cached Anthropic session and walks each obligation. It picks the persona whose trigger predicate matches the obligation type. In tournament mode, N candidates run in parallel; the verifier scores them, the top scorer is a commit candidate, and losers get logged but never merge.

Two adapter subsystems

The most common confusion in v6 was treating the coding CLIs and the falsifiers as one thing. v8 splits them cleanly.

Producer adapters (src/adapters/) wrap third-party coding CLIs as the worker in the v6 verified-branch pipeline. Backends: Copilot, Claude Code, Codex, Claude Code Teams. All four are opt-in via swarm run --v6.

Falsifier adapters (src/falsification/adapters/) take a patch the producer's verifier already accepted and try to falsify the obligation by surfacing a counter-example, regression fixture, or property-violation trace. A confirmed counter-example flips the obligation back to failed.

Falsifier	Default	Obligation types
`CodexFalsifier`	on	`property-must-hold`
`CopilotFalsifier`	on	`import-graph-must-satisfy`, `function-must-have-signature`
`ClaudeCodeFalsifier`	off (per-adapter opt-in)	all three

The CLI surface is one flag: --falsifiers <on|off> (default on). Per-adapter selection happens at the API layer via defaultAdapterRegistry({ includeCopilot, includeClaudeCode }).

Four verification points

A patch has to survive these before it merges:

Pre-generation memoization. Skip generation if the obligation result is already cached.
Mid-stream abort. During generation, the streaming verifier can abort the call. Works in --mode single only; tournament mode skips it.
Post-generation per-obligation verifier. Scores the candidate diff. In tournament mode, top scorer wins; in single mode it's pass/fail.
Post-merge integration check. After the diff lands, the integration check confirms the broader system still holds.

The architectural rule from the README: nothing commits without passing the obligation's verifier. Then the falsifiers get a shot.

The hash-chained ledger

Every action lands in .swarm/ledger/<run-id>.jsonl with the SHA of the prior entry. Tampering is detectable; runs resume from any prior state. If a process is killed mid-run, swarm v8 resume <run-id> walks the ledger and picks up where it left off.

The ledger format is shared with v6, but v8 writes more granular events (per-persona dispatch, per-candidate score, falsifier verdict) so a run can be replayed or audited end-to-end.

Quick start

git clone https://github.com/moonrunnerkc/swarm-orchestrator.git
cd swarm-orchestrator && npm install && npm run build && npm link

# Compile a goal, then run it
swarm v8 compile "add a /health endpoint that returns 200 OK" --yes
swarm v8 run .swarm/contracts/<contract-id>

# Or both in one step (defaults to v8)
swarm run --goal "add a /health endpoint that returns 200 OK"

# Resume a killed run
swarm v8 resume <run-id>

Requires Node >= 20, git >= 2.40, and ANTHROPIC_API_KEY. Pass --extractor stub --session stub to run offline.

There's also a GitHub Action:

- uses: moonrunnerkc/swarm-orchestrator@v8
  with:
    goal: "add a /health endpoint"
    contract-only: false
    cost-cap: "5.00"
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

What v8.0 doesn't do

Limitations worth reading before adopting

Tournament mode doesn't stream. --mode tournament plus --forbid-import skips the streaming abort; streaming verification is --mode single only.
Post-merge failure doesn't auto-rollback. The run is marked failed; per-obligation worktree snapshots are post-v8.0.
--cost-cap is enforced post-obligation, not mid-call. Cumulative spend is checked at the end of each obligation against estimated Sonnet 4 pricing. Exit code 6 if exceeded.
Bandit dispatch is not built (Phase 5). Codex and Copilot have disjoint obligation types, so there's nothing to arbitrate between yet.
Cross-vendor producer race is deferred (Phase 6).

Full list with rationale lives in docs/v8-architecture-deviations.md.

Repo

moonrunnerkc / swarm-orchestrator

Contract-first AI coding swarm with hash-chained evidence. Compiles a goal into typed obligations, races persona candidates per obligation in a single cached inference session, verifies before commit, and logs every action in an append-only ledger.

Swarm Orchestrator

Contract-first AI coding swarm with hash-chained evidence and verifier-gated commits.

swarm compiles a natural-language goal into a typed contract, dispatches it to a population of personas inside one cached Anthropic session, races candidate diffs per obligation, and commits only the diffs that pass verification. After the producer's verifier accepts a patch, registered falsifier adapters get a chance to break it before it merges. Every action lands in an append-only hash-chained ledger you can audit, resume, or replay.

It wraps an LLM; it does not replace one. The model writes the code, the orchestrator decides what reaches your repo.

Status

Version 8.0.1 on main. Node >= 20 (CI matrix: 20, 22). License ISC. The v8 architecture is the default for swarm run; the v6 verified-branch pipeline is preserved under swarm run --v6 and the swarm swarm / swarm execute commands Falsifier subsystem: Codex on, Copilot on, ClaudeCode…

View on GitHub

How to Write a CLAUDE.md Rule That Actually Gets Enforced

Brad Kinnard — Sun, 10 May 2026 02:06:45 +0000

Open a CLAUDE.md file at random and you'll find build commands, architecture notes, and rules. The rules tend to be the unenforceable kind. "Write clean code." "Be careful with types." "Follow our conventions." The author meant every word. The agent reads them. And nothing checks whether the agent followed them, because nothing can.

In a corpus of 580 CLAUDE.md, AGENTS.md, and .cursorrules files from public GitHub repos with 10+ stars, 74% contained zero machine-extractable rules. Not because the authors didn't care about rules. Because most rules were written in a form no parser could pull out as a deterministic check.

This post is about the difference. Specifically: how to phrase a rule so a parser can extract it and a verifier can check it, without sacrificing what you actually meant.

The principle

Enforceability comes from a verifiable surface. A rule is verifiable when there's a concrete pattern in code that either matches it or doesn't. "Use camelCase for function names" is verifiable: read the AST, list the function names, check the casing. "Name things consistently" is not: there's no concrete pattern to check, only a judgment to make.

The gap between the two is the gap between intent and enforcement. You meant the same thing in both cases. But only one of them survives translation into a check.

Here's the heuristic I use: could a junior engineer with no context mechanically check whether code follows this rule, just by reading the rule and looking at the code? If yes, the rule is enforceable. If they'd have to ask "what does 'consistent' mean here?", it isn't.

Three pairs, walked through

Take a few common intents and look at how they fail or succeed.

Type safety. You want strong typing.

Bad:  Be careful with types.
Good: No `any` types in `src/`. Async functions require explicit return types.

The bad version has no surface. "Careful" isn't a check. The good version has two: a forbidden token (any) and a structural property (return type annotation on async function declarations). Both check directly against the AST.

Module structure. You want predictable imports and exports.

Bad:  Prefer clean module structure.
Good: Named exports only. No default exports. Filenames in kebab-case.

"Clean" is meaningless to a parser. Named-only and kebab-case are both binary properties of code that exist or don't. The first version sounds like more guidance because it's broader, but breadth is the problem: it covers everything and enforces nothing.

Preferences over alternatives. You want React functional components, not class components.

Bad:  Write modern React.
Good: Prefer functional components over class components.

"Modern" is a moving target with no fixed surface. The "prefer X over Y" pattern, on the other hand, has a clean check: count instances of each, compute a ratio, score against a threshold. This is one of the most useful patterns in instruction files because it captures real-world preference (not absolute prohibition) in a measurable way.

The reference table

Twelve common intents, paired:

Intent	Unenforceable	Enforceable
Naming functions	Name things consistently	Use `camelCase` for function names
Filenames	Pick reasonable filenames	All filenames in kebab-case
Type safety	Be careful with types	No `any` types
Async return types	Make types clear	Async functions require explicit return types
Module exports	Prefer clean module structure	Named exports only, no default exports
File size	Keep files manageable	Maximum 300 lines per file
Logging	Be mindful of logging	Never use `console.log`; use `src/logger.ts`
Component style	Write modern React	Prefer functional components over class components
Package manager	Use the right package manager	Use `pnpm`, not `npm`
Test files	Keep tests organized	All test files end with `.test.ts`
Error handling	Handle errors properly	Async functions must use `try/catch` or return a `Result` type
Commit format	Write clear commit messages	Use conventional commits (`feat:`, `fix:`, `chore:`)

Every right-hand cell points at a concrete check: a token, a casing rule, a count, a file pattern, a configured tool. Every left-hand cell points at a judgment.

Real-world rules usually carry scope: "no any in src/," "named exports outside index.ts files," "no console.log in production code paths." Scope makes a rule narrower and more accurate without making it less enforceable. The interop layer that genuinely needs any keeps it; the rest of the codebase doesn't.

What kinds of checks exist

Worth knowing what's available, because it shapes what's writable. Static analysis tools targeting AI instruction files generally support a few classes of check:

AST-level: function names, type annotations, import patterns, forbidden tokens, structural properties
Filesystem: file existence, naming conventions, directory layout, file size limits
Regex: literal strings, content patterns, conventional formats
Tooling: presence and configuration of linters, formatters, package managers, test runners
Config-file: contents of .eslintrc, tsconfig.json, .prettierrc, etc.
Git-history: commit message formats, branch naming conventions
Preference ratios: "prefer X over Y" with a compliance percentage instead of a binary verdict

If your rule maps to one of these classes, it's enforceable. If it doesn't, it isn't. The trick when writing instruction files is to keep that map in mind: when you're about to write "be careful with X", ask which of these classes "carefulness with X" lives in. Usually the answer points at a concrete reformulation.

The unenforceable rules aren't worthless

Here's a real tension: most of what makes a CLAUDE.md useful isn't enforceable at all. Project context (what the repo does, where the architecture lives), agent behavior directives (be succinct, ask before deleting, don't touch /legacy), and onboarding instructions are all valuable. None of them extract as rules.

Don't try to make them enforceable. They're a different kind of content with a different purpose. Project context grounds the agent. Behavior directives shape its style. Neither is supposed to be checked against output; they're checked against the agent's process, which is a different problem.

The mistake worth avoiding is letting unenforceable prose crowd out enforceable rules. Anthropic's Claude Code best practices doc recommends deleting any instruction the model already follows correctly without it. Most "write clean code" style rules fail that test: the model already does its version of clean code, so the line is taking up attention budget your agent could be spending on the specific, verifiable rules that actually distinguish your codebase from a generic project. Cut what the model already does. Keep the checks.

A test for your own files

Pull up your CLAUDE.md or AGENTS.md right now. For each line that looks like a rule, ask:

Could a junior engineer check this without asking clarifying questions?
Does it name a specific pattern, file, token, casing, or value?
Would 5 different reviewers all agree on whether a piece of code passes this rule?

If a rule fails 1 or 2, it's not a rule, it's a wish. If it fails 3, it's ambiguous. Rewrite or delete.

If you want a mechanical version of this test, RuleProbe parses CLAUDE.md, AGENTS.md, .cursorrules, .windsurfrules, GEMINI.md, and copilot-instructions.md against 102 matchers and tells you which lines extracted as rules and which didn't:

npm install -g ruleprobe
ruleprobe parse ./CLAUDE.md --show-unparseable

The --show-unparseable flag is the interesting one. It surfaces every line that looked rule-shaped but didn't map to a check. That list is your rewrite queue.

RuleProbe on GitHub

What this leaves out

The hardest case is rules like "follow the existing error handling pattern in this codebase." That's enforceable in principle (compare new code's structural shape against the codebase's dominant pattern), but not by simple AST or regex matching. It needs codebase-aware analysis. Some tools handle that; most don't. If you find yourself writing those kinds of rules, know that they'll either need a tool that does pattern profiling or they'll stay aspirational.

The other thing enforceability doesn't catch: an agent that follows every rule and still writes broken code. Static rules reduce variance, they don't eliminate it. A function with any removed and an explicit return type can still have wrong logic. Treat passing rule checks as a floor, not a ceiling.

I Dropped Multi-Agent Coordination for a 5-Layer Falsification Battery

Brad Kinnard — Sat, 02 May 2026 15:00:11 +0000

Swarm Orchestrator just lost its swarm. Dropped the multi-agent parallel coordination layer. Running one agent now and putting all the weight on a five-layer post-merge falsification battery instead.

This is an experiment, not an endpoint. v8 will bring proper multi-agent swarming back. The reason for cutting it temporarily: I want to know whether the value I was getting from coordinated parallel agents was the coordination itself, or the verification pressure that coordination produced. Easier to measure with one variable. Intended side effect: cost reduction, since the previous architecture spun up multiple CLI agent instances per run. Real benchmarks pending.

TL;DR: every patch survives a five-layer post-merge battery before the orchestrator declares success. Layers 1 and 2 are hard gates. Layers 3, 4, 5 are advisory and feed a composite score. Hard-gate failure throws before attestation, before final gates, before any external success signal.

Pipeline Order

The battery runs once per orchestrator execution against the merged working tree, not per-step branches. The per-step verifier is a separate component. Layers fire in fixed order:

Differential gate (hard)
Mutation gate (hard)
Cheat detector (advisory)
Property gate (advisory)
Attestation (advisory on first run, signed after)

If the hard gate fails, the composite is forced to 0 and the orchestrator throws falsification battery blocked the patch before any external success signal can fire.

Layer 1: Differential Gate (Hard)

Before any agent touches the repo, a synthesizer generates a regression test against the goal. Layer 1 then runs that test in two detached worktrees: one at the base commit, one at the patch commit.

The contract: the test must fail at base and pass at patch.

If the test passes at base, the layer returns INVALID_TEST. This catches the specific failure mode where an agent writes a tautological test that passes against any code. Without this gate, that pattern slips through every other check downstream.

If no command can be synthesized and the caller doesn't pass --differentialTestCommand, the layer fails closed. Deliberate policy.

Layer 2: Mutation Gate (Hard)

Runs Stryker for JS/TS, mutmut for Python, PITest (Gradle/Maven) for Java, against changed files only. First runs the regression command; if that fails, the layer fails immediately. On pass, dispatches the mutation tool and parses the reporter output, including Stryker's clear-text table layout.

Score	Status
< 0.6	FAIL
< 0.8	WARNING
≥ 0.8	PASS

Patches with no meaningful test coverage of the changed lines fail here even when npm test is green.

Layer 3: Cheat Detector (Advisory)

Diff-static scan for known agent-shortcut patterns. Each rule maps to a real category of behaviour I've watched agents fall into under pressure.

Cheat detector rule IDs

test-modification: agent edited the test instead of the implementation
complexity-mismatch: trivial implementation paired with a non-trivial test
mock-mutation: real call replaced with a mock that returns the expected value
exception-swallowing: caught and ignored exceptions to make tests pass
hardcoded-answer: literal return of the expected output

Optional Semgrep integration loads any registered cheat-rule packs into a transient config. The scan reads added and removed lines and patterns; it does not execute the patch. An agent that hides a hardcoded answer behind a small computation can still pass this layer. Honest about that.

Layer 4: Property Gate (Advisory)

Discovers modified TS/JS/Python functions, parses their parameter types, maps each to a fast-check arbitrary or Hypothesis strategy, generates a harness, runs it. Counterexamples surface as findings. Untyped or unsupported types degrade to a low-severity advisory finding rather than blocking.

Layer 5: Attestation (Advisory on First Run)

Reads the refs/notes/swarm-attestation git note for the patch commit, validates the in-toto SLSA v1.0 envelope's subject SHA against the patch commit, then verifies the cosign signature. On the first run for a commit there's no note yet, so this layer reports advisory-warn and the post-battery attestation step writes the note.

The note is verifiable later via swarm attest verify <commit>. A downstream consumer can verify the patch survived the battery without trusting the running orchestrator process.

Composite Scoring

When the hard gate passes, a weighted composite is computed across the three advisory layers and any optional advisory quality-gate results. Failed advisory gates each subtract a fixed penalty.

Default scoring (overridable via .swarm/gates.yaml)

composite threshold: 0.7
weights: cheat detector 0.4, property gate 0.4, attestation 0.2
advisory gate penalty: 0.02 per failure

humanReviewRequired is true when the composite score is below threshold or any advisory layer is in advisory-warn status.

Where It Actually Runs

Three real call sites, not just unit tests:

Production orchestrator on every swarm run
Synthetic calibration corpus (36 paired test specs across 6 broken-category families) executing in CI on every push
SWE-bench harness using Layer 1 and Layer 4 as standalone spot-check eval drivers

Honest Caveats

These are in docs/known-gaps.md and I won't hide them:

Differential gate is host-Python-sensitive on legacy codebases. The synth-eval can reflect import-chain errors rather than assertion outcomes. The authoritative resolution gate in the per-instance Docker image is unaffected.
Mutation gate skips quietly when no changed files match supported languages. YAML, Markdown, Rust, Go diffs don't get mutation-tested.
Cheat detector is diff-static, not behavioural. The hidden-computation-around-hardcoded-answer pattern can pass it.
Attestation signing is best-effort. Cosign-not-installed errors get logged and the run proceeds without a note. The note's absence is reflected in Layer 5's advisory-warn on subsequent runs but does not block.

Why Run This Experiment

If the falsification battery alone produces patches that survive scrutiny at acceptable quality, then a lot of the apparent value of multi-agent coordination was actually the verification pressure it created, not the agent diversity itself. If the battery alone isn't enough, then v8 multi-agent gets a clearer mandate: the swarm is the value, not the side effect.

Either result is useful. The point of the rewrite is to make the answer measurable.

View on GitHub

swarm-orchestrator v7.0.0-alpha.0

Brad Kinnard — Thu, 30 Apr 2026 05:28:06 +0000

The agent generates code. The orchestrator tries to find reasons not to trust it.

That sentence is the entire pivot. Earlier versions of swarm-orchestrator coordinated multiple agents working on the same task. v7 wraps a single agent CLI (Copilot, Claude Code, or Codex) and runs five independent checks on the patch before allowing a merge.

TL;DR

Five-layer verification battery sits between any agent CLI and your main branch. Two layers are hard gates. Three feed a composite score. Every verified merge gets a signed SLSA attestation as a git note.

The five checks

#	Layer	What it catches	Gate type
1	Intent verification	Patch doesn't actually fix the stated problem	Hard
2	Regression verification	Patch breaks existing behavior, or test coverage is too weak to know	Hard
3	Solution quality	Agent gamed the test (hardcoded values, swallowed exceptions, modified tests)	Composite
4	Behavioral verification	Patch works on the happy path, crashes on edge cases	Composite
5	Provenance	No signed attestation produced for the merge	Composite

1. Intent verification

The patch must make a previously-failing test pass. For SWE-bench instances, that's the FAIL_TO_PASS test from the instance JSON. For user-facing goals, a reviewer synthesizes a regression test before the worker runs and confirms it fails against the base commit first.

2. Regression verification

Existing tests must pass. Then mutation testing runs on the modified files to check whether coverage around the change is actually strong enough to catch regressions. A patch that works but lives in weakly-tested code gets flagged.

Mutation testing tooling per language

JS / TS: Stryker
Python: mutmut
Java: PITest

Mutation score thresholds are configurable in .swarm/gates.yaml. Defaults: below 0.6 fails, 0.6 to 0.8 warns, above 0.8 passes.

3. Solution quality

A Semgrep rule pack scans for the specific shortcuts agents take when they're being graded:

Hardcoded values matching test expectations
try/catch blocks swallowing the exact exception a failing test was asserting on
Modifications to test files outside the stated scope
Mock mutations that make tests pass without changing the implementation

4. Behavioral verification

Property-based testing runs against modified functions for 60 seconds each, using Hypothesis (Python) or fast-check (TypeScript). Counterexamples that crash the patched code or violate type contracts get reported.

5. Provenance

A signed SLSA v1.0 attestation is generated for each verified merge and attached to the commit as a git note. Signed via cosign keyless OIDC. The attestation contains agent identity, model version, per-layer results, and the composite score.

swarm attest verify <commit>

That command pulls the note and verifies the signature. Useful when something breaks in production three months later and someone asks which agent wrote the offending code and what was checked at merge time.

Status

Alpha. SWE-bench Verified 50-instance sweeps across Copilot CLI, Claude Code, and Codex are running now. Headline metric is the falsification catch rate: of the patches each agent claimed succeeded, what percentage failed at least one layer. Numbers drop in a follow-up post when the sweeps complete.

Where this goes next

v8 brings parallel execution back, applied to verification instead of generation.

The orchestrator will compute a risk score for each patch, then spawn a population of independent falsifiers sized to that risk. Falsifiers share findings through a coordination channel so a discovery from one steers the targeting of others. A bandit selects which falsifier types to spawn based on past outcomes.

The v7 five-layer battery becomes the seed pool that v8 grows from. The project name finally fits.

moonrunnerkc / swarm-orchestrator

Independent verification battery for patches written by AI coding agents. Wraps Copilot, Claude Code, and Codex; applies a five-layer falsification battery (intent, mutation, cheat detection, property tests, signed attestation) to gate merges.

Swarm Orchestrator

Independent verification battery for patches written by AI coding agents.

Quick Start · How It Works · Documentation · Contributing

Wraps third-party coding-agent CLIs (Copilot, Claude Code, Codex), runs worker and reviewer steps on isolated git branches, and applies a five-layer falsification battery to each agent-authored patch. Hard gates block patches that fail intent or regression checks; advisory layers feed a composite score.

You run this around an agent CLI, not instead of one. The agent produces the patch; the orchestrator tries to break it. Patches that survive merge to main; patches that don't are rolled back with a verification report.

Features

Five-layer falsification battery. Intent verification, regression and mutation testing, cheat detection, property-based testing, and signed attestation. Layers 1 and 2 are hard gates; layers 3 to 5 feed an advisory composite score. Implementations live under src/verification/.
Isolated worker and reviewer steps. Each step runs…

View on GitHub

94% of Published SKILL.md Files Skip the Spec's Two Most Basic Patterns

Brad Kinnard — Wed, 29 Apr 2026 02:30:37 +0000

The agentskills.io spec recommends two things in every description: start with an action verb, and include a trigger phrase like "use when..." that tells the routing layer when to fire the skill. They take five seconds to add and they're the difference between a skill an agent picks up and a skill that sits unused in the catalog.

I sampled 500 skills at random from a 1,436-skill public corpus and measured both. 5.8% follow both recommendations. 61.8% follow neither.

The full breakdown of what the SKILL.md ecosystem actually looks like in production, as of late April 2026.

Methodology

Corpus: sickn33/antigravity-awesome-skills at HEAD on April 29, 2026. This is the largest publicly bundled SKILL.md collection in a single repo (1,436 indexed skills with metadata for category, source, and risk classification).

Sample: 500 skills, random with seed 42 for reproducibility.

Tool: skillcheck v1.2.0 from PyPI.

Per-skill features captured: every skillcheck diagnostic (rule, severity, message), description quality score, body line count, body and metadata token estimates, activation entropy and top-hypothesis score from --activation-hypotheses, structural features computed locally (description length in chars and words, action verb in first position, trigger-phrase presence, presence of resources//scripts//references/ subdirectories, frontmatter field count and which fields), plus the antigravity-supplied category, source, and risk metadata.

Caveat one: skillcheck's description quality score is a heuristic that includes action-verb and trigger-phrase detection as positive signals. So the correlation between these two features and the score is partly mechanical. The headline finding is not "we discovered these patterns predict quality." It's "the spec recommends these patterns, the linter that encodes the spec rewards them, and almost nobody is using them."

Caveat two: antigravity's bundler injects risk, source, date_added, and category fields into the SKILL.md frontmatter when packaging skills. The author-original frontmatter analysis below excludes these injected fields.

Reproduce in five commands:

pip install skillcheck==1.2.0
git clone --depth 1 https://github.com/sickn33/antigravity-awesome-skills.git
cd antigravity-awesome-skills
# Then sample from skills_index.json with seed 42 and run skillcheck against each
# Full analysis script: see the dataset link at the bottom

The two-pattern adoption gap

Every skill description was classified on two binary features: does it start with an action verb (Generates, Validates, Creates, Builds, Analyzes, etc., from a 90-verb allowlist), and does it contain a trigger phrase (use when, use this skill when, when the user, when working with, whenever, etc.)?

Pattern	Count	%
Has both action verb and trigger phrase	29	5.8%
Action verb only	108	21.6%
Trigger phrase only	54	10.8%
Neither	309	61.8%

The same four groups, scored against skillcheck's description quality metric:

Group	n	Median score	% scoring 70+
Has both	29	90.0	100.0%
Action verb only	108	70.0	72.2%
Trigger phrase only	54	70.0	94.3%
Neither	309	50.0	8.4%

The 100% rate in the both-features group isn't magic. It reflects that skillcheck's heuristic was designed around the spec's recommendations and rewards skills that follow them. What's actually striking is the bottom line: 309 of 500 published skills skip both recommendations. That's the working majority of the ecosystem leaving easy quality on the floor.

What authors actually fill in

Outside name and description, frontmatter is mostly empty. The median author-original frontmatter (excluding the bundler's injected fields) has just two fields. Two.

Field	Adoption
name	99.6%
description	99.6%
author	10.8%
tags	10.6%
tools	8.8%
license	3.8%
allowed-tools	2.8%
version	2.2%
triggers	0.6%
user-invokable	0.6%
capabilities	0.2%

The spec offers version, author, tags, allowed-tools, model, agent, hooks, user-invocable, disable-model-invocation, skills, mode. Almost none of them are being used. 80% of authors stop after name and description. There's an entire optional metadata layer the spec defines and the ecosystem ignores.

Progressive disclosure adoption is 16%

The spec's load-bearing concept is progressive disclosure: keep metadata tiny so the routing layer scans it cheaply, keep the body lean so it fits the agent's context window, push heavy material into resources/, scripts/, or references/ subdirectories that load only when needed.

Subdirectory	Adoption
`resources/`	6.4%
`scripts/`	4.4%
`references/`	8.2%
Any of the three	16.0%

84% of skills inline everything in SKILL.md. The whole architectural promise of progressive disclosure (multiple skills can sit in the agent's catalog without overwhelming context) requires authors to actually use the pattern. Most don't.

Body bloat is real

23% of skills triggered disclosure.body-bloat warnings, meaning they contain code blocks over 50 lines or tables over 20 rows in the SKILL.md body itself. These are exactly the things the progressive disclosure pattern was designed to push out into references/.

13.6% exceeded the spec's 500-line soft cap on body length. 8.4% exceeded the 5,000-token body budget when skillcheck's tokenizer flagged them (the rest weren't measured because they didn't trip the warning threshold).

Description length sweet spot

Quality scores rise with description length up to about 175-225 characters, then plateau:

Length range (chars)	n	Median quality
25-49	16	50.0
50-99	90	50.0
100-149	158	60.0
150-199	131	70.0
200-249	62	67.5
250-299	38	60.0

The spec's character cap is 1,024. Almost nobody's pushing it. The ecosystem clusters between 100 and 200 chars (median 145), which is roughly the bottom edge of the quality plateau. Authors writing 150+ char descriptions get noticeably better routing signal density.

Cross-source patterns

Antigravity's index classifies each skill's source. Quality patterns by source class:

Source class	n	Median quality	% action verb	% trigger	% progressive disclosure
community	394	60.0	26.6%	17.5%	16.2%
external_repo	38	65.0	34.2%	31.6%	18.4%
official_org	9	60.0	77.8%	0.0%	33.3%
personal	14	50.0	0.0%	0.0%	0.0%

Three observations. Skills from official org repos (Anthropic, Hugging Face, etc.) hit 77.8% action-verb adoption, miles above the community baseline, but zero trigger-phrase use; their descriptions are direct and verb-led without the "use when" preamble. Skills from individual external repos (someone's personal GitHub project) actually hit the highest trigger-phrase rate (31.6%), suggesting individual maintainers writing for their own activation problem think harder about it than community contributors writing for a shared list. Skills tagged "personal" (someone's curated set of their own work) hit 0% on both patterns, which is the cleanest signal that "I made this for me" doesn't translate to "an agent will pick this up."

Skillcheck v1.2.0 against the corpus

The new version was released April 28, 2026. The skillcheck rule set found:

1 of 500 skills produced an actual ERROR (0.2%): android_ui_verification, which has invalid characters in its name.
499 of 500 produced WARNINGs (99.8%).
0 skills passed completely clean.

Most-fired rules:

Rule	Count
`frontmatter.field.unknown`	500
`description.quality-score`	499
`disclosure.body-bloat`	115
`compat.unverified`	81
`disclosure.metadata-budget`	70
`sizing.body.line-count`	68
`disclosure.body-budget`	42
`frontmatter.description.person-voice`	27
`frontmatter.field.ecosystem`	19
`sizing.body.token-estimate`	14
`frontmatter.name.reserved-word`	11

The frontmatter.field.unknown warning fires on every file because antigravity injects bundler-only fields into the frontmatter (risk, source, date_added); strip those and the genuine unknown-field rate drops dramatically. Worth knowing if you're running skillcheck against bundled corpora versus author-original repos.

What this means if you publish skills

Four things, all reversible in a single commit per skill:

Start the description with an action verb (Generates, Validates, Creates, Analyzes, Refactors, Audits, etc.). Not Expert in, not Comprehensive, not One-stop. The verb tells the routing layer what the skill does in two syllables.
Include a trigger phrase (Use when ..., Trigger when ..., Use this skill when the user ...). The agent's routing decision is "should I activate this." A trigger phrase answers it directly.
Aim for 175-225 characters in the description. Short descriptions don't carry enough routing signal; long ones bury it.
Push large code blocks (>50 lines), large tables (>20 rows), and detailed reference material out of SKILL.md and into resources/, scripts/, or references/. The body should describe the work; the reference files should hold the work.

That's it. Four changes that move a skill from the 61.8% of the ecosystem ignoring spec recommendations to the 5.8% following them.

Methodology, for anyone who wants to push back

Tool: skillcheck v1.2.0 from PyPI (released April 28, 2026)
Corpus: sickn33/antigravity-awesome-skills at HEAD on April 29, 2026 (1,436 indexed skills)
Sample: 500 skills, drawn with random.seed(42) then random.sample
Per-skill processing: skillcheck path --format json --skip-ref-check plus skillcheck path --activation-hypotheses --format json
Feature extraction: action-verb match against a 90-verb allowlist (gerund and base forms); trigger-phrase match against 9 regex patterns; structural facts computed from filesystem and parsed frontmatter
Quality score: pulled from skillcheck's description.quality-score info diagnostic (a published heuristic whose source is at src/skillcheck/rules/description.py in the skillcheck repo)
Frontmatter analysis: bundler-injected fields (risk, source, date_added, category, id) excluded from the author-original counts above

The full dataset (500 skills, all features, all diagnostics) and the analysis output are in the skillcheck repo under docs/. Anyone who wants to verify a finding, slice it differently, or run the same pipeline against a different corpus has everything they need.

What's next

This study used skillcheck's symbolic mode and the activation-hypotheses generator. The agent-native critique mode (--ingest-critique) and capability graph extraction (--ingest-graph) weren't run here because they require a real agent in the loop and would have made the corpus run significantly longer. A follow-up study using those modes on a smaller subset (50-100 skills) would tell us what an agent actually sees in a skill versus what a static linter can measure. That's the next post.

moonrunnerkc / skillcheck

Cross-agent skill quality gate for SKILL.md files. Validates frontmatter, scores description discoverability, checks file references, enforces three-tier token budgets, and flags compatibility issues across Claude Code, VS Code/Copilot, Codex, and Cursor.

Cross-agent skill quality gate for SKILL.md files.

What This Does

skillcheck validates SKILL.md files against the agentskills.io specification: frontmatter structure, description quality, body size, file references, and cross-agent compatibility. New in v1.0: agent-native semantic self-critique, heuristic capability graph extraction with five structural analyzers, and a per-skill validation history ledger. It does not call any LLM API, execute skill instructions, or modify files.

Why This Exists

Analysis of 580 AI instruction files found that 96% of their content cannot be verified by any static tool. A separate survey found that 22% of SKILL.md files fail basic structural validation. Skills get written, committed, and published to catalogs; nobody proves they work.

skillcheck addresses both gaps with a two-mode design. When a calling agent is present, it uses that agent for semantic self-critique and capability graph extraction: the agent reads the skill's instructions and reports whether they are clear, complete, and internally…

View on GitHub

The Jupyter notebook bug that only crashes for other people

Brad Kinnard — Tue, 28 Apr 2026 04:53:59 +0000

Cell 0 uses df. Cell 1 defines df.

Notebook works for you because your kernel ran the cells in some other order and the variable's still in memory. You commit. Someone clones the repo, hits Restart and Run All, dies on cell 0.

Standard Python linters can't catch this. ruff, flake8, mypy operate on one source file at a time. A notebook is N cells whose execution order in your kernel may have nothing to do with their order on disk. The bug isn't inside any single cell. It's in the relationship between cells.

nborder is a static linter for that relationship.

Rules

Code	Flags
NB101	`execution_count` decreases in source order
NB201	Name used in cell N, only defined in cell M where M > N
NB102	Name used somewhere, never defined anywhere
NB103	Stochastic call (numpy, torch, tensorflow, stdlib random) before any seed

How the cross-cell analysis works

Each cell gets parsed with libCST. A visitor extracts symbol definitions (assignments, function defs, class defs, imports) and symbol uses (name references, attribute roots) per cell. Connect them across cells in source order, you get a dataflow graph at notebook scope.

NB201 findings are uses whose nearest matching definition lives in a later cell. NB102 findings are uses with no matching definition anywhere.

The graph also makes the auto-fix safe. When NB201 fires, the fixer runs a topological sort over cell dependency edges. Sort succeeds, cells get reordered to respect dataflow and execution counts get cleared. Cycle detected, fixer bails with an explicit message naming the cycle.

NB201 fix example

Input:

# cell 0
result = df.head()

# cell 1
import pandas as pd
df = pd.DataFrame({'a': [1, 2, 3]})

Run nborder check --fix notebook.ipynb:

notebook.ipynb:cell_0:1:10: NB201 Variable `df` used in cell 0 is only defined in cell 1. The notebook will fail on Restart-and-Run-All. [*]
Fix outcomes:
  reorder: applied (reordered 2 cells and cleared execution counts)

Output:

# cell 0
import pandas as pd
df = pd.DataFrame({'a': [1, 2, 3]})

# cell 1
result = df.head()

Cell IDs preserved. Execution counts cleared. Second nborder check exits 0.

NB103 and seed injection

NB103 walks the same graph for stochastic calls (np.random.rand, torch.rand, tf.random.normal, random.random) firing before any matching seed. The fix injects a single seed cell at the right position. Multi-library notebooks get one cell:

import numpy as np
np.random.seed(42)
rng = np.random.default_rng(42)
import torch
torch.manual_seed(42)

Alias-aware. import numpy as numpy_lib produces a seed line using numpy_lib, not a redundant fresh import. After fixing a NumPy notebook, computed cell outputs are byte-identical across consecutive jupyter nbconvert --execute runs.

JAX and scikit-learn get diagnostic-only handling. JAX needs PRNGKey threading through call signatures. sklearn random_state=None needs a value chosen against your testing strategy. Neither is a single line you can inject.

Byte-stable writer

Parse a notebook, modify nothing, write it back, bytes match exactly. Verified against nbformat v4.0, v4.4, v4.5 fixtures plus a real-world notebook corpus. When the writer does mutate during a fix, only the cells that actually changed get rewritten. Cell IDs, metadata, and unrelated cells stay verbatim.

Outputs

Four reporters:

text: ruff-style path:cell:line:col: NB### message
json: machine-readable
github: ::error file=...,line=...,title=NB201:: annotations for PR inline comments
sarif: SARIF 2.1.0, schema-validated

Pre-commit hook and a composite GitHub Action included:

- uses: moonrunnerkc/nborder@v0.1.4
  with:
    path: notebooks/
    select: NB201,NB103

What it doesn't do

Doesn't execute notebooks. Pair with nbval or papermill for kernel-level validation.
Doesn't lint cell-internal style. That's nbqa.
Dynamic name resolution (exec, getattr, **kwargs, monkey-patching) is invisible. Same limitation as any static analyzer.
Cell magics are stripped before analysis. Names introduced by %%capture get tracked. Anything magic-internal does not.

Install

pip install nborder
nborder check path/to/notebooks/

Python 3.10+.

moonrunnerkc / nborder

A fast, opinionated linter and auto-fixer for Jupyter notebook hidden-state and execution-order bugs.

nborder

A fast, opinionated linter and auto-fixer for Jupyter notebook hidden-state and execution-order bugs.

What this catches

Code	Name	One-line example
NB101	Non-monotonic execution counts	Cell 1 ran with `In [3]:` after cell 0 ran with `In [5]:`.
NB102	Won't survive Restart-and-Run-All	`print(df)` references a name no cell in the notebook defines.
NB201	Use-before-assign across cells	Cell 0 uses `df`; `df = ...` only appears in cell 1.
NB103	Stochastic library used without seed	`np.random.rand(3)` runs with no seed call before it.

Each rule has a docs page under docs/rules/ explaining the bug class, a bad and good example, and the auto-fix behaviour. The four sections below walk through each rule with the diagnostic nborder actually emits.

NB101: out-of-order execution

The execution_count field on each cell records the order Jupyter actually ran cells in, not the order they appear in the file. When those orders disagree, the recorded…

View on GitHub

Four Security Bugs That Shipped in AI-Generated Code (and How They Got Caught)

Brad Kinnard — Wed, 15 Apr 2026 18:36:15 +0000

A single Copilot CLI run against a FastAPI application produced four distinct security issues. The code worked. Tests passed. The endpoint did what was asked. None of the issues would surface during a demo or a code review focused on functionality.

User input rendered as raw HTML

The application tracks satellite data. Satellite names come from user input. The agent rendered them directly into HTML templates in four separate locations:

html += f"<strong>{t.risk}</strong>: {t.sat1} vs {t.sat2}"

No escaping. Four blocks, same pattern. A single-purpose security scanning agent found all four and applied markupsafe.escape(). A general-purpose agent reviewing the same code caught three of four, missing one buried in a conditional branch.

The difference isn't model quality. The security-focused agent had a narrower scope and explicit instructions to scan for unescaped user input in template rendering. Scope and prompt specificity determined the outcome.

Health endpoint that lies to the load balancer

The agent built a /health endpoint. It returned HTTP 200 unconditionally, including when the database was unreachable.

Kubernetes liveness and readiness probes interpret 200 as "this instance is healthy, keep routing traffic." An instance that returns 200 with a dead database stays in the rotation. Users hit it. Requests fail. The cluster thinks everything is fine.

The correct response is 503 (Service Unavailable). The orchestrator's verification caught this because runtime behavior checks are part of the quality gate surface, not just static analysis.

This one's subtle. The endpoint "works" in every test environment where the database is actually running. It only fails in the exact production scenario it was designed to protect against.

Exception details returned to clients

Error handlers used str(e) as the response body:

except Exception as e:
    return {"error": str(e)}

Database connection strings, file paths, internal state. All returned directly to whoever triggered the error. In a security audit this is an information disclosure finding. In a FastAPI app behind an API gateway, it's a path to mapping internal infrastructure.

Deprecated datetime API

datetime.utcnow() has been deprecated since Python 3.12. The replacement is datetime.now(timezone.utc). The agent also used time.time() for uptime tracking, which is affected by NTP clock adjustments and can report negative uptime if the system clock steps backward. time.monotonic() exists specifically for this case.

Neither of these will cause a production outage today. Both are the kind of technical debt that accumulates when generated code isn't checked against current language standards.

Why this matters

None of these bugs required a sophisticated analysis to find. They're patterns: unescaped user input in templates, unconditional success responses in health checks, raw exception strings in error responses, deprecated stdlib usage. Each one is a known category with a known fix.

The problem is attention. A general-purpose agent optimizing for "make this feature work" doesn't allocate attention to these categories unless explicitly prompted. The feature works. The tests pass. The agent moves on.

This is where orchestration changes the economics. Instead of one agent covering everything, specialized agents with narrow scopes check specific categories. A security auditor scans for injection and information disclosure. A runtime checker validates health endpoint semantics. Each agent's prompt is focused enough that known bug patterns get caught.

The alternative is what most developers do today: manually reprompt. "Now check for XSS." "Now add proper error handling." "Now fix the health check to actually check health." We measured this on the same codebase. 14 follow-up prompts to bring the standalone output to the same level. Each prompt required reading the previous output, identifying what was wrong, and writing a specific correction. About 45 minutes of continuous supervision.

The orchestrated run took 22 minutes, unattended. 7 premium requests vs 15. Zero human review cycles.

Swarm Orchestrator v5.0.0

The tool that caught these is open source. It wraps existing agent CLIs (Copilot, Claude Code, Codex) and adds verification, quality gates, and parallel execution. It doesn't generate code. It delegates code generation and verifies the output against outcome-based checks: git diff, build success, test pass, runtime behavior.

v5.0.0 adds three features relevant to this problem:

Spec-aware planning reads the quality gate configuration before generating agent prompts. Security requirements, test coverage thresholds, and configuration standards get injected before agents write code, not discovered through iteration afterward.

SARIF output exports quality gate violations as SARIF 2.1.0 JSON compatible with GitHub code scanning. Same PR annotation workflow teams already use for CodeQL.

Per-project gate configuration via .swarm/gates.yaml lets teams override thresholds and disable gates that don't apply to their project type.

1,386 passing tests, 84 source files, 7 documented benchmarks. The release notes include commit hashes for every bug fix.

Swarm Orchestrator on GitHub

What categories of bugs do you consistently find in AI-generated code that could be caught by a specialized check rather than manual review?

We Parsed 580 AI Instruction Files. 96% of the Content Can't Be Verified.

Brad Kinnard — Wed, 15 Apr 2026 04:31:26 +0000

Every AI coding agent reads an instruction file. CLAUDE.md, AGENTS.md, .cursorrules, whatever your agent uses. You write rules in it. The agent says "Done." And you have no idea whether it followed any of them.

We wanted to know what's actually inside these files. Not what people think they contain, but what a machine can extract and verify through static analysis. So we scraped instruction files from 568 public GitHub repos with 10+ stars, ran them through a parser backed by 102 matchers across 8 verifier engines (AST, filesystem, regex, tree-sitter, preference, tooling, config-file, git-history), and counted what came out.

The short version: across the entire corpus, 3.8% of lines were extracted as verifiable coding rules. The other 96% is markdown headers, code examples, project descriptions, build commands, agent behavior directives, and contextual prose.

The dataset

580 instruction files from 568 repos, including Sentry (43k stars), PingCAP/TiDB (40k), Lerna (36k), Dragonfly (30k), Kubernetes/kops (17k), javascript-obfuscator (16k), RabbitMQ (14k), Google APIs (14k), Redpanda (12k), and hundreds of others. Six file formats represented: AGENTS.md (149 files), CLAUDE.md (111), .cursorrules (102), .windsurfrules (95), GEMINI.md (89), and copilot-instructions.md (34). This sample skews toward larger public repos. Enterprise internal repos with stricter governance, or solo projects with tightly scoped instruction files, may look different. We'd like to see that data.

The parser reads each file and classifies every line: is this a rule that can be checked against code, or is it something else? "Something else" includes headers, blank lines, code blocks, explanatory prose, build instructions, and agent personality configuration.

Corpus stats: 8,222 total instruction lines parsed. 309 rules extracted. 7,913 lines classified as non-rule content.

What instruction files actually contain

The 96% that isn't rules breaks down into several categories. Some of it is necessary context (project structure explanations, build command documentation). Some of it is agent behavior configuration ("be succinct," "avoid providing explanations"). Some of it is just markdown formatting overhead.

Here's what stood out: 430 of the 580 files (74%) had zero extractable rules. Of those, 67 were completely empty to the parser: zero extracted, zero unparseable. Many were single-line redirects. Dragonfly's .cursorrules (30k stars) says "READ AGENTS.md." Umi's .cursorrules (16k stars) contains the single word "RULE.md." Mautic's GEMINI.md says "Read and follow all instructions in ./AGENTS.md."

At the other end, a few files were dense with rules. Apache Skywalking-java's CLAUDE.md extracted 6 rules from 26 lines (23%). Cloudflare chanfana's AGENTS.md: 5 rules from 21 lines (24%). But those files tend to be short, focused lists of concrete instructions.

The heavy files tell a different story. javascript-obfuscator's CLAUDE.md (16k stars): 197 lines, zero rules extracted. These files are documentation with no machine-verifiable instructions embedded.

Parse rate distribution across all 580 files

Parse Rate	Files	Percentage
0% (no rules)	430	74.1%
1-9%	70	12.1%
10-19%	54	9.3%
20-29%	13	2.2%
30-49%	11	1.9%
>= 80%	2	0.3%

Only 2 files (0.3%) had parse rates at or above 80%. Nearly three quarters had zero.

Types of content the parser correctly skips

"3.8% extraction rate" sounds like the parser is broken. It isn't. These are lines that genuinely aren't rules:

Markdown structure (headers, horizontal rules, blank lines). Code examples showing how to use a function or run a command. Project descriptions explaining what the repo does. Build and deployment instructions. Links to external documentation. Agent behavior directives that have no code-level representation ("be concise," "ask before making changes"). Workflow instructions ("use this branch strategy," "run tests before pushing").

The parser isn't failing on these. It's correctly identifying them as not-rules. The denominator is every line in the file, not every line that looks like it could be a rule.

A second metric tells the complementary story. 150 of 580 files (25.9%) contained at least one extractable rule. Across those 150 files, 309 rules is an average of 2.1 rules per file. So only a quarter of instruction files contain anything enforceable at all, and when they do, they typically contain two rules. The 3.8% describes the corpus-wide line ratio. The 25.9% and 2.1-per-file numbers describe what rule-writers are actually producing.

What a "verifiable rule" looks like

The 309 rules that did get extracted map to concrete checks. Things like:

"Use camelCase for function names" (AST naming check)
"No any types" (TypeScript type safety check)
"Use named exports, not default exports" (import pattern check)
"Prefer const over let" (preference ratio check)
"Test files must exist for every source file" (filesystem check)
"Use Yarn, not npm" (tooling check)

Each rule gets a category, a verifier type (AST, filesystem, regex, tree-sitter, preference, tooling, config-file, or git-history), and a qualifier (always, prefer, when-possible, avoid-unless, try-to, never).

Rule extraction by category

Category	Rules Extracted
naming	169
structure	44
code-style	29
forbidden-pattern	24
type-safety	20
dependency	12
error-handling	5
import-pattern	4
test-requirement	2

Naming rules dominate: 55% of all extracted rules. That's likely a combination of two factors. Naming conventions ("use camelCase," "kebab-case filenames") are the most concrete, unambiguous instructions people write, so they appear frequently. They're also the rule class that static analysis matchers handle most cleanly, so the parser has high affinity for them. We can't fully separate how much of the 55% is user behavior vs. parser strength, but both contribute.

Rule extraction by instruction file type

Type	Files	Files with Rules	Rules	Total Lines	Rate
copilot-instructions.md	34	13	33	556	5.9%
.cursorrules	102	37	79	1,508	5.2%
AGENTS.md	149	49	97	1,961	4.9%
.windsurfrules	95	22	50	1,866	2.7%
CLAUDE.md	111	20	38	1,501	2.5%
GEMINI.md	89	9	12	830	1.4%

copilot-instructions.md had the highest extraction rate (5.9%), likely because those files tend to be shorter and more prescriptive. GEMINI.md files had the lowest (1.4%).

E2E verification: does excalidraw follow its own instruction files?

This is a pipeline demonstration on one repo, not broad validation across ecosystems. We ran the full pipeline on excalidraw (~95k stars) because it's large, well-maintained, and has instruction files with extractable rules: both a CLAUDE.md and a copilot-instructions.md.

The parser found 9 verifiable rules across both files. Deterministic analysis scored 66.1% compliance. Semantic analysis (structural fingerprinting of 626 source files) produced 9 verdicts, all resolved via fast-path vector similarity. Zero LLM calls, zero cost:

Rule	Compliance	Method
Prefer functional components	0.976	structural-fast-path
PascalCase type naming	0.976	structural-fast-path
Async try/catch usage	0.983	structural-fast-path
Contextual error logging	0.979	structural-fast-path
Yarn as package manager	0.50	no matching topic
TypeScript required	0.50	no matching topic
Optional chaining preference	0.50	no matching topic
camelCase variables	0.50	no matching topic
UPPER_CASE constants	0.50	no matching topic

Rules that match established code pattern topics (component-structure, error-handling) score 0.97+, meaning the codebase's structural fingerprint strongly matches the instruction. The remaining five rules scored a neutral 0.50 because they describe tooling choices and naming conventions that don't have structural AST representations. That's itself a finding: even among the 4% of lines that get extracted as verifiable rules, some fall into categories that resist automated verification beyond simple presence checks. The verifier is real, but not comprehensive. No static analysis tool covers every rule class, and pretending otherwise would be dishonest.

Privacy note: 626 files scanned, all file IDs are opaque sequential integers. No source code strings, file paths, variable names, or comments appear in any payload. In this case, no LLM was even called.

What this means for anyone writing instruction files

Two clarifications before the takeaways. First, "96% can't be verified" means can't be verified through static analysis, not "is useless." Agent behavior configuration, project context, and workflow documentation all have value. They guide the agent even if no tool can confirm compliance after the fact. Second, the 4% that is verifiable still matters. Excalidraw's 9 extractable rules produced a 66.1% deterministic compliance score with specific failures at specific line numbers. Nine rules doesn't sound like much until three of them fail and you find the agent ignored your naming conventions across 626 files.

The real problem isn't that instruction files contain documentation. It's that most people don't know which of their lines are enforceable and which are suggestions the agent can silently drop. That ratio isn't fixed, either. People write unverifiable instructions because nobody's told them which phrasings produce checkable rules.

To write rules that can actually be checked:

Use imperative verbs with specific targets. "Use camelCase for all function names" is verifiable. "Follow good naming conventions" isn't.

Specify the tool or pattern, not the principle. "Prefer const over let" is a ratio check. "Write immutable code" is philosophy.

Include the file patterns your rules apply to. "All .ts files must use named exports" scopes the check. "Use named exports" is vague.

Keep rules and documentation separate. Rules are instructions. Documentation explains why. Mixing them dilutes both.

RuleProbe on GitHub: parse your own instruction files and see what's actually verifiable

The tool

RuleProbe is the parser and verifier behind this analysis. It reads 7 instruction file formats, extracts machine-verifiable rules using 102 built-in matchers across 14 categories, and checks agent output against each one. Deterministic by default, no API keys needed for the core pipeline. Optional semantic analysis for pattern-matching and consistency rules.

npx ruleprobe parse CLAUDE.md --show-unparseable
npx ruleprobe verify CLAUDE.md ./src --format summary

The --show-unparseable flag shows you exactly which lines were skipped and why. That's often the most useful output: it tells you which of your "rules" aren't rules at all.

moonrunnerkc / ruleprobe

Verify whether AI coding agents follow the instruction files they're given

RuleProbe

Verify whether AI coding agents actually follow the instruction files they're given

Why

Every AI coding agent reads an instruction file. None of them prove they followed it.

You write CLAUDE.md or AGENTS.md with specific rules: camelCase variables, no any types, named exports only, test files for every source file. The agent says "Done." But did it actually follow them? Your code review catches some violations, misses others, and doesn't scale.

RuleProbe reads the same instruction file, extracts the machine-verifiable rules, and checks agent output against each one. Compliance scores with file paths and line numbers as evidence. Deterministic and reproducible by default. Optional semantic analysis for pattern-matching and consistency rules that require codebase-aware judgment.

Quick Start

npm install -g ruleprobe

Or run it directly:

npx ruleprobe --help

Parse an instruction file to see what rules RuleProbe can extract:

ruleprobe parse CLAUDE.md
ruleprobe parse AGENTS.md --show-unparseable

Verify agent output…

View on GitHub

Independent convergence on specification-first AI code verification

Brad Kinnard — Sat, 11 Apr 2026 14:42:02 +0000

On March 26, 2026, Christo Zietsman published "The Specification as Quality Gate: Three Hypotheses on AI-Assisted Code Review" on arXiv.

Paper: arXiv:2603.25773

The paper's core argument (direct quote from abstract):

The combined argument implies an architecture: specifications first, deterministic verification pipeline second, AI review only for the structural and architectural residual.

I noticed this because my own open-source project, Swarm Orchestrator, implements a very similar layered approach. I built it from real usage patterns with AI coding agents, not from the paper (neither of us referenced the other's work).

moonrunnerkc / swarm-orchestrator

CI/CD for AI-generated code. Run Copilot, Claude Code, or Codex in parallel; verify every claim against evidence; gate merges on 8 automated quality checks.

Swarm Orchestrator

CI/CD for AI-generated code. Run Copilot, Claude Code, or Codex in parallel; verify every claim against evidence; gate merges on 8 automated quality checks.

Not an autonomous system builder: an accountability layer around agents you already trust enough to run, but not enough to merge blind. Each step runs on its own isolated branch. Each claim (tests pass, build clean, commit made) is cross-referenced against the transcript and the actual filesystem. Failures are auto-classified, repaired with targeted strategies, and re-verified. Nothing reaches main without passing both the verification engine and the quality gate pipeline. The metric that matters is cost per rubric point, not wall-clock time.

Quick Start · What Is This · Benchmarking · Usage · GitHub Action · Recipes · Architecture · Contributing

Quick Start

See it run end-to-end

npm install -g swarm-orchestrator
# then set up any one of the agent CLIs below, and:

…

View on GitHub

How the tool works (current state as of April 2026)

Agents run as untrusted subprocesses on isolated git branches. Acceptance criteria are injected into each agent's prompt before generation.

After execution, a deterministic verification pipeline checks claims against concrete evidence (commit SHAs, test output, build results, file diffs). No LLM is used as the primary gate.

Eight configurable quality gates then run: scaffold leftovers, duplicate blocks, hardcoded config, README accuracy, test isolation, test coverage, accessibility, runtime correctness. All are regex/AST/diff/threshold checks.

An optional --governance Critic wave runs after the deterministic layers. It scores steps on weighted axes and pauses for human review on flags. Scores are advisory only.

Full details and flow: github.com/moonrunnerkc/swarm-orchestrator (80 stars, 50 passing tests across 95 files, latest release v4.2.0 on April 9.)

The original Copilot-focused version went public on dev.to January 25, 2026 with the core isolation + evidence-based verification already present.

Why this alignment matters

Zietsman cites the DORA 2026 report showing that higher AI code generation correlates with higher throughput and higher instability. Time saved writing code gets re-spent on auditing. His paper argues that simply adding more AI review does not fix the structural issue when there is no external specification layer.

Swarm Orchestrator was built to address exactly that pattern. The deterministic gates catch the repeatable failure modes (security headers, test depth, config externalization) that standalone agents consistently miss in head-to-head runs. The Critic layer is available only for the residual judgment calls where human or AI insight can still add value.

I am not claiming this proves or validates the paper. It is simply an independent practical example that landed on closely aligned principles at roughly the same time. If you are working with AI coding agents and wrestling with verification, the repo is open for review, issues, or contributions.

swarm-orchestrator on GitHub

AI Coding Agents Can Verify Some of Their Work Now. Here's What They Still Miss.

Brad Kinnard — Thu, 09 Apr 2026 05:40:48 +0000

Copilot and Claude Code both ship with verification features now. Copilot's Agent mode runs terminal commands, detects build failures, and iterates fixes. Claude Code plans changes across files and can run your test suite after modifications. Both have improved significantly since 2025.

They're still not catching everything.

Developers consistently report agents declaring tasks complete while skipping accessibility attributes, test isolation, config externalization, dark mode, responsive layout, and meta tags. The agent runs the build, sees green, and moves on. But "build passes" and "the output is production-ready" are different bars. The reprompt cycle for quality attributes the agent never attempted in the first place is still a significant time sink on any non-trivial project.

That gap is where Swarm Orchestrator sits. Not replacing the agent's self-verification, but adding the checks it doesn't run.

What It Does

You give it a goal. It builds a dependency-aware plan, assigns steps to specialized agents, and launches them in parallel on isolated git branches. Each step runs through outcome-based verification (build, test, diff, expected files) and eight quality gates covering scaffold leftovers, duplicate code, hardcoded config, README accuracy, test isolation, test coverage, accessibility, and runtime correctness.

Before the agent runs, the orchestrator injects acceptance criteria based on the project type. For web apps, that's 16 requirements: semantic HTML, responsive layout, dark mode via CSS custom properties, prefers-reduced-motion, image alt attributes, heading hierarchy, ARIA labels, focus-visible styles, and more. For everything else, 6 baseline criteria covering error handling, documentation, input validation, logging, and test coverage.

The agent sees these as hard requirements. After execution, the quality gates check whether they were met. The agent's own verification handles "does it compile and do tests pass." The orchestrator handles "did it actually do what was asked, completely."

Benchmark context

Head-to-head runs against standalone Copilot CLI, Claude Code, and Codex on the same goals showed a consistent pattern: quality attributes the agent never attempted were absent from unassisted output. These aren't build failures the agent would catch on its own. They're requirements like skip-to-content links, prefers-reduced-motion media queries, CSS custom properties on :root, dual theme-color meta tags, module separation between logic and presentation, zero-dependency test runners. Each is at least one follow-up prompt. Several take 2-3 rounds.

The orchestrator caught and enforced all of them in a single pass.

Steps that fail don't get blindly retried. The orchestrator classifies the failure (build, test, missing artifact, dependency, timeout) and sends the agent back with the actual error output and context. This works alongside the agent's own retry capabilities, not instead of them.

What's New in v4.2.0

Three additions.

Multi-Tool Adapters

The --tool flag existed in previous versions. It parsed from the CLI, reached the options object, and then did nothing. The orchestrator always spawned Copilot CLI internally regardless of what you passed.

That's fixed. resolveAdapter() now routes through real adapter implementations with a shared process supervisor.

swarm run --goal "Add auth" --tool copilot              # default, unchanged behavior
swarm run --goal "Add auth" --tool claude-code
swarm run --goal "Add auth" --tool claude-code-teams --team-size 3

Agent Teams mode spawns a team lead per wave for native multi-agent coordination. If the team lead fails, it falls back to per-step sequential execution automatically.

Every adapter shares the same process supervisor: 5-minute stall timeout, 10-second heartbeat checking stdout activity, SIGTERM on stall, SIGKILL after 5-second grace. Previously only the Copilot path had stall detection. A hung claude process would block your entire run indefinitely. That's gone.

OWASP ASI Compliance Mapping

The orchestrator already enforced branch isolation (ASI-03: Excessive Agency), outcome-based verification (ASI-05: Improper Output Handling), and failure-classified repair. Those behaviors map directly to risks in the OWASP Top 10 for Agentic Applications.

--owasp-report formalizes that mapping. After every run, it generates a per-risk assessment with evidence pulled from actual execution metadata.

swarm run --goal "Build REST API" --governance --owasp-report

6 of 10 ASI risks are assessed. 4 are marked not-applicable with explicit rationale (the orchestrator doesn't store user data, doesn't communicate across networks, doesn't train models). If it doesn't apply, it says so and explains why.

Which ASI risks are assessed?

ASI Risk	Assessed	Rationale
ASI-01: Prompt Injection	Yes	Agent prompts controlled by orchestrator, user goals parameterized into plan steps
ASI-02: Insecure Tool Use	Yes	Tool invocations verified against transcript evidence
ASI-03: Excessive Agency	Yes	Scope enforcement via isolated worktrees and boundary declarations
ASI-04: Unreliable Execution	Yes	Failure classification, targeted repair, retry with error context
ASI-05: Improper Output Handling	Yes	Build/test/diff verification independent of agent self-reporting
ASI-10: Uncontrolled Autonomy	Yes	Governance mode with Critic scoring, human-in-the-loop approval
ASI-06, 07, 08, 09	N/A	No model training, no data storage, no cross-network communication, no supply chain

Structured Run Reports

Every run already produced artifacts: session state, metrics, cost attribution, per-step verification reports, and now OWASP compliance. Pulling a coherent picture from those files meant opening each one individually.

swarm report runs/my-run-id                             # generate from any completed run
swarm report --latest --stdout                          # most recent run, print to terminal
swarm report runs/my-run-id --format json               # JSON only

One command. Markdown and JSON. Missing sections (cost data, OWASP) are handled gracefully and just don't appear in the output.

Where This Sits

The agents have gotten better at self-verification. That's a good thing. The orchestrator isn't competing with that. It's adding a layer the agents don't cover: acceptance criteria enforcement, quality gates for attributes agents don't check on their own, independent verification that doesn't rely on the agent's self-reporting, and an auditable trail of everything that happened.

	Standalone Agent (2026)	With Orchestrator
Build/test verification	Built-in (Copilot Agent, Claude Code)	Independent check on isolated branch
Quality attributes	Whatever you prompt for	16 web-app / 6 baseline criteria injected and verified
Failure handling	Agent retries with some context	Classified failure, targeted repair prompt with error output
Audit trail	Chat history, some checkpoints	Transcripts, verification reports, cost attribution, OWASP compliance
Merge safety	Agent says it's done	Proof required across verification + 8 quality gates

GitHub: moonrunnerkc/swarm-orchestrator

TypeScript. ISC license. Requires Node 20+ and Git.

Same Instruction File, Same Score, Completely Different Failures

Brad Kinnard — Tue, 07 Apr 2026 00:24:41 +0000

Two AI coding agents were given the same task with the same 10-rule instruction file. Both scored 70% adherence. Here's the breakdown:

Rule	Agent A	Agent B
camelCase variables	PASS	FAIL
No `any` type	FAIL	PASS
No console.log	FAIL	PASS
Named exports only	PASS	FAIL
Max 300 lines	PASS	FAIL
Test files exist	FAIL	PASS

Agent A had a type safety gap. It used any for request parameters even though it defined the correct types in its own types.ts file. Agent B had a structural discipline gap. It used snake_case for a variable, added a default export following Express conventions over the project rules, and generated a 338-line file by adding features beyond the task scope.

Same score. Completely different engineering weaknesses. That table came from RuleProbe.

About this case study
The comparison uses simulated agent outputs with deliberate violations, not live agent runs. Raw JSON reports are in the repo under docs/case-study-data/. This is documented in the case study.

What RuleProbe is

RuleProbe is an open source CLI that reads AI coding agent instruction files and verifies whether the agent's output followed the rules. It covers six formats: CLAUDE.md, AGENTS.md, .cursorrules, copilot-instructions.md, GEMINI.md, and .windsurfrules.

Verification is deterministic. No LLM in the pipeline. The same input produces the same report every time.

GitHub Repo

How it checks

Three methods, depending on the rule:

AST analysis via ts-morph handles code structure. Variable and function naming (camelCase), type and interface naming (PascalCase), type annotations (any detection), export style (named vs default), JSDoc presence on public functions, and import patterns (path aliases, deep relative imports).

Filesystem inspection handles file-level rules. File naming conventions (kebab-case) and whether test files exist for source files.

Regex handles content patterns like max line length.

v0.1.0 has 15 matchers across those three methods, covering TypeScript and JavaScript. ts-morph is the AST engine, so other languages aren't supported.

Output looks like this:

RuleProbe Adherence Report
Rules: 14 total | 11 passed | 3 failed | Score: 79%

PASS  naming/naming-camelcase-variables-5
PASS  naming/naming-pascalcase-types-7
FAIL  forbidden-pattern/forbidden-no-any-type-1
      src/handler.ts:12 - found: req: any
      src/handler.ts:24 - found: data: any
FAIL  forbidden-pattern/forbidden-no-console-log-10
      src/handler.ts:18 - found: console.log("handling request")
FAIL  test-requirement/test-files-exist-11
      src/handler.ts - found: no test file found

File, line, violation. No ambiguity.

The conservative parser

This is a design choice worth explaining. When RuleProbe reads an instruction file, it only extracts rules it can map to a deterministic mechanical check. Everything else gets reported as unparseable.

ruleprobe parse CLAUDE.md --show-unparseable

"Write clean code" is unparseable. "Use the repository pattern" is unparseable. "Handle errors gracefully" is unparseable. These can't be verified without judgment, and judgment means variance between runs. RuleProbe doesn't do that.

The tradeoff: a 30-rule instruction file might produce 12 verified rules and 18 unparseable ones. You see both counts so you know exactly what's being checked and what isn't.

Running it

npx ruleprobe --help

Parse an instruction file

ruleprobe parse CLAUDE.md

Extracted 14 rules:

  forbidden-no-any-type-1
    Category: forbidden-pattern
    Verifier: ast
    Pattern:  no-any (*.ts)
    Source:    "- TypeScript strict mode, no any types"

  naming-kebab-case-files-4
    Category: naming
    Verifier: filesystem
    Pattern:  kebab-case (*.ts)
    Source:    "- File names: kebab-case"

Verify agent output

ruleprobe verify CLAUDE.md ./agent-output --format text

Supports --format json, --format markdown, and --format rdjson (reviewdog-compatible). Exit code 0 means all rules passed, 1 means violations found.

Compare two agents

ruleprobe compare AGENTS.md ./claude-output ./copilot-output \
  --agents claude,copilot --format markdown

CI with the GitHub Action

name: RuleProbe
on: [pull_request]
jobs:
  check-rules:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
      - uses: moonrunnerkc/ruleprobe@v0.1.0
        with:
          instruction-file: AGENTS.md
          output-dir: src
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

No external API keys. Posts results as a PR comment. Supports reviewdog rdjson for inline annotations if you use reviewdog in your pipeline. Exposes score, passed, failed, and total as step outputs, so you can gate merges on adherence thresholds in downstream steps.

All action inputs

Input	Default	What it does
`instruction-file`	(required)	Path to your instruction file
`output-dir`	`src`	Directory of code to verify
`agent`	`ci`	Agent label for report metadata
`model`	`unknown`	Model label for report metadata
`format`	`text`	text, json, or markdown
`severity`	`all`	error, warning, or all
`fail-on-violation`	`true`	Fail the check if any rule is violated
`post-comment`	`true`	Post results as a PR comment
`reviewdog-format`	`false`	Also output rdjson

Programmatic API

Five functions if you want to integrate verification into your own tooling:

import {
  parseInstructionFile,
  verifyOutput,
  generateReport,
  formatReport,
  extractRules
} from 'ruleprobe';

parseInstructionFile reads the instruction file. verifyOutput runs the rules. generateReport builds the adherence report with summary stats. formatReport renders it as text, JSON, markdown, or rdjson. extractRules works on raw markdown content if you don't have a file path.

What it doesn't cover

15 matchers is a starting point, not full coverage. Real instruction files have rules RuleProbe can't verify yet: architectural patterns, error handling conventions, dependency constraints, API design rules. The parser will tell you what it skipped.

TypeScript and JavaScript only. ts-morph is the AST engine. Other languages would need a different parser.

No automated agent invocation. You run the agent separately and point RuleProbe at the output directory.

Security and dependencies

RuleProbe never executes scanned code, never makes network calls, never writes to the scanned directory. Paths are resolved and bounded to process.cwd(). Symlinks outside the project are skipped by default.

Four runtime dependencies: chalk 5.6.2, commander 12.1.0, glob 11.1.0, ts-morph 24.0.0. All pinned to exact versions. No semver ranges.

npm: ruleprobe | MIT license

moonrunnerkc / ruleprobe

Verify whether AI coding agents follow the instruction files they're given

RuleProbe

Verify whether AI coding agents actually follow the instruction files they're given

Why

Every AI coding agent reads an instruction file. None of them prove they followed it.

Quick Start

npm install -g ruleprobe

Or run it directly:

npx ruleprobe --help

Parse an instruction file to see what rules RuleProbe can extract:

ruleprobe parse CLAUDE.md
ruleprobe parse AGENTS.md --show-unparseable

Verify agent output…

View on GitHub

AI coding agents lie about their work. Outcome-based verification catches it.

Brad Kinnard — Sun, 29 Mar 2026 21:59:27 +0000

AI coding agents have a consistency problem. Ask one to add authentication to your project and it'll tell you it's done. Commits made, tests passing, middleware wired up. Check the branch and you'll find a half-written JWT helper, no tests, and a build that doesn't compile.

This isn't a hallucination problem. The agent did produce code. It just didn't verify that any of it worked before declaring victory. And neither did the tools sitting between the agent and your main branch.

The transcript trust problem

Most orchestration tools that coordinate AI agents verify work by reading transcripts. The agent says "committed 3 files" or "all tests passing" and the verifier pattern-matches those strings as evidence of completion.

That's trusting the agent's self-report.

The issue isn't that agents are deliberately deceptive. It's that they generate completion language as part of their output pattern regardless of the actual state of the codebase. An agent will write "tests passing" into its response while the test suite has syntax errors. It'll claim files were created that only exist in the prompt's hypothetical, not on disk.

Transcript parsing catches the obvious failures: agent errored out, produced no output, didn't mention anything about the task. It misses the subtle ones: agent produced code that looks right, described it correctly, but the code doesn't compile, doesn't pass tests, or doesn't do what was asked.

Outcome-based verification

The alternative is checking what actually happened instead of what the agent said happened.

This is what Swarm Orchestrator 4.0 implements. After each agent step runs on its isolated git branch, the verifier executes a series of checks against the branch itself:

Check	What it does	Fails when
`git_diff`	Diffs the branch against the recorded base SHA	No file changes detected
`build_exec`	Runs the detected build command in the worktree	Non-zero exit code
`test_exec`	Runs the detected test command in the worktree	Non-zero exit code
`file_existence`	Checks that expected output files exist	Expected files missing
`transcript`	Parses agent output for completion evidence	(supplementary only)

Transcript analysis still runs. But when outcome checks are present, transcript-based checks get demoted to required: false. The build and test execution results gate the merge decision.

Stack detection is automatic. The verifier reads package.json, Makefile, pyproject.toml, Cargo.toml, or whatever project configuration exists and runs the appropriate commands. No per-repo configuration.

What happens when verification fails

Blind retry is the default across most agent tooling. Step fails, same prompt runs again, up to some retry limit. The agent has no idea what went wrong.

Swarm Orchestrator's RepairAgent takes the structured output from the verification checks and feeds it back into the retry prompt. Which check failed, the last 20 lines of build or test output, which files were expected but aren't there. The failure gets classified (build failure, test failure, missing files, no changes) and the repair strategy adapts to the type.

On the final attempt the prompt includes an explicit priority shift: get something working over getting something complete.

The difference between "retry with context" and "blind retry" is measurable. An agent that knows the build failed on a missing import has a realistic path to fixing it. An agent re-running the same prompt that produced a broken build has roughly the same odds of producing another broken build.

Agent-agnostic by design

4.0 drops the Copilot CLI dependency. The adapter layer supports Copilot CLI, Claude Code, and Codex out of the box. The interface is minimal:

export interface AgentAdapter {
  name: string;
  spawn(opts: {
    prompt: string;
    workdir: string;
    model?: string;
    timeout?: number;
  }): Promise<AgentResult>;
}

One flag at the CLI level (--tool claude-code) or per-step in a plan file. The orchestrator treats the agent as an interchangeable subprocess. Verification doesn't change based on which agent ran. The branch either builds or it doesn't.

This also means you can mix agents within a single plan. Use Claude Code for the architecture step, Codex for the boilerplate, Copilot for the tests. Each step gets verified the same way regardless of which agent produced it.

CI integration

The tool ships as a GitHub Action:

- uses: moonrunnerkc/swarm-orchestrator@swarm-orchestrator
  with:
    goal: "Add unit tests for all untested modules"
    tool: claude-code
    pr: review

The Action outputs a JSON result with per-step verification status. You can gate downstream jobs on the verification outcome the same way you'd gate on any other CI check.

Most orchestrators in this space are desktop-first or local-development tools. Even those that support remote execution do not run natively in CI with outcome-verified results. That's the gap Swarm fills.

Recipes for repeatable tasks

Generating a plan from scratch for "add tests to this project" is wasteful when the plan structure is the same every time. 4.0 ships with seven parameterized recipes:

swarm use add-tests --tool codex --param framework=vitest
swarm use add-auth --param strategy=jwt
swarm use security-audit

Each recipe is a JSON file with {{parameter}} placeholders. Custom recipes are one file in templates/recipes/. The knowledge base tracks recipe outcomes across runs so success rates and failure patterns accumulate over time.

Current state

1,112 tests passing, 1 pending. TypeScript strict mode. ISC license. Five phases of upgrades shipped in this release across the adapter layer, verification engine, repair pipeline, CI integration, and recipe system.

moonrunnerkc / swarm-orchestrator

CI/CD for AI-generated code. Run Copilot, Claude Code, or Codex in parallel; verify every claim against evidence; gate merges on 8 automated quality checks.

Swarm Orchestrator

CI/CD for AI-generated code. Run Copilot, Claude Code, or Codex in parallel; verify every claim against evidence; gate merges on 8 automated quality checks.

Quick Start · What Is This · Benchmarking · Usage · GitHub Action · Recipes · Architecture · Contributing

Quick Start

See it run end-to-end

npm install -g swarm-orchestrator
# then set up any one of the agent CLIs below, and:

…

View on GitHub