DEV Community: Brad Kinnard

It Fits and It Benchmarks Well. Will It Do Your Job?

Brad Kinnard — Mon, 20 Jul 2026 03:05:45 +0000

A fit calculator answers one question: will these weights load in your memory. A leaderboard answers a different one: is this model good at tests in general. The question that decides what you actually run is neither. It's "will Q4_K_M hurt my invoice extraction accuracy on my machine," and the only reliable way to answer it is to run your extraction on Q4_K_M and measure.

QuantProof automates that. Point it at a folder of real examples from your task. It sweeps every model you already have in Ollama (or whatever a Rapid-MLX server is serving), scores every output with deterministic scorers, measures time to first token, tokens per second, and peak memory, then recommends the smallest quant whose quality stays within 2% of the best measured result.

npm install -g quantproof
quantproof ingest my-tasks.md   # a local model drafts the task pack from your notes
# or: quantproof init my-task   # scaffold it yourself
quantproof run --pack my-tasks
quantproof report --markdown

Why measure instead of estimate

Three task packs ran on an M5 Max: ticket classification, invoice extraction, config generation. Three models, 540 scored generations. No model won all three.

An 8B at Q8_0 took classification at 0.950, beating a 30B at 45% of the memory. The 30B was the only model to score 1.000 on extraction. They tied at 1.000 on config generation, where the 8B needs about 17 GiB less.

The 8B's zero on extraction was a token-budget truncation issue, not a quality failure. On 57 of 60 generations it spent its entire 512-token budget on reasoning and never emitted content. Because every raw output is journaled, the report flags the zero trunc! and names the fix (raise max_tokens) instead of letting it read as "this model can't extract."

Which model is "best" changed with every task. That is the leaderboard problem in one paragraph: the ranking is a property of the model and the workload together, and a leaderboard only knows the model.

Determinism is a backend version property, not a model property

Every example runs 3 times at temperature 0 with a fixed seed, and repetitions are compared byte for byte. On Apple Metal, every candidate repeated exactly, across 240+ scored units. Then the same check ran on an RTX 5070. Same pack, same seed, same driver, two Ollama versions:

candidate	ollama 0.15.2	ollama 0.32.1
llama3.1:latest (Q4_K_M)	nondet	deterministic
llama3.1:8b-instruct-q4_0	nondet	deterministic
gemma3:4b	nondet	nondet
gemma3:1b	deterministic	nondet

Both 8B quants became deterministic after the upgrade. The 1B flipped the other way.

Not even the direction of the effect survived a backend upgrade, which is why the check runs on every sweep instead of being trusted once. One example answered "As needed" on repetition 1 and "Weekly" on repetitions 2 and 3, at temperature 0 with a seed: identical conditions, different outputs.

Predictions print next to measurements

The fit prediction is deliberately simple: weights on disk, plus f16 KV cache for the context, plus a fixed 1 GiB compute allowance. On the 5070, three of four candidates measured within 15% of prediction; the 1B ran over 20% under its intentionally conservative estimate. Every report shows both. Use predictions for planning. Measured peak is your actual constraint.

Peak memory comes from the most honest source each machine has:

nvidia-smi on NVIDIA GPUs
Backend process RSS on Apple Silicon (validated against Ollama's own accounting)
Metal accounting on Rapid-MLX
/proc/meminfo-based RSS on everything else

A machine with no working source reports "not measured." Nothing is ever estimated.

Anyone can recheck the scores

quantproof report --bundle exports a zip with every raw output, all scores, model digests, sampler params, and the backend version. Re-scoring the 5070 sweep from bundle contents alone reproduced 264 of 264 scores exactly, on both Ollama versions. A number you can't recheck is marketing.

The fine print (read before citing any of these numbers)

Memory is sampled at roughly 200 ms. The recorded peak is the highest sample, a floor on the true peak, not an allocator trace.
TTFT is measured from HTTP request to first streamed chunk, including connection overhead and prompt evaluation. The real latency a user experiences, not a pure decode metric.
Scorers are deterministic only: schema conformance, field comparison, label match, pattern presence, numeric tolerance. Tasks that need judgment (summarization, open QA) are out of scope, not approximated with a judge model.
Partial GPU offload is flagged with its reasoning, not measured. Ollama decides offload, not the harness.
A pack drafted by quantproof ingest measures agreement with its drafting model until human review. One run: drafter got 1 of 22 expected values wrong. The provenance label exists because of that.

Run it on your task

Node 22+, Ollama or Rapid-MLX. Out-of-memory failures are recorded as results, not crashes, and the same packs can also sweep Claude models over the Anthropic API when no GPU is around.

moonrunnerkc / quantproof

Run your real task on quantized models and get measured quality, latency, and peak VRAM, plus the smallest quant that holds quality.

Why measure · How it works · Case study · Run it on your task · API runs · Documentation

QuantProof

Point QuantProof at a folder of real examples from your task, and it tells you which quantized model performs best on your hardware.

It measures, not estimates:

Quality
Latency
Peak VRAM usage

Then it recommends the smallest quantized model whose quality is within 2% of the best-performing model.

Why measure

Fit calculators only tell you whether a model's weights will fit in memory Benchmark leaderboards rank models on general tests, not your specific workload.

If your real question is:

"Will Q4_K_M hurt my invoice extraction accuracy?"

there's only one reliable way to answer it: run your invoice extraction on Q4_K_M and measure the results.

How it works

QuantProof automates that process. It:

Tests candidate models one at a time
Scores every output deterministically
Measures latency and peak VRAM
Reports…

View on GitHub

Run your own 20 examples

AI Agents Cheat on Pull Requests. I Mined 327 of Them to Prove It.

Brad Kinnard — Thu, 09 Jul 2026 22:45:30 +0000

AI coding agents cheat. Not maliciously, and usually not on purpose. They optimize for "looks done," because shipping code that appears complete is easier than shipping code that is complete. Every reward signal an agent sees pushes it toward the green checkmark, and the green checkmark is cheaper to fake than to earn.

That was a curiosity when one engineer babysat one agent. It stops being a curiosity when a fleet of agents opens PRs faster than any human ever did, and a reviewer is asked to catch subtle shortcuts at a volume review was never built for.

So I went and measured it. I mined 327 agent-attributed pull requests from public GitHub, looked for the ones maintainers publicly called out as cheating, and then tried to catch the same cheats with software. This post is what I found, including the parts that did not work.

What "cheating" actually means

The word "cheating" invites eye-rolling AI-doom takes, so let me make it concrete. Here are real, recognizable patterns. Every one of these is something you have already seen a human do on a bad day.

Swallowed errors. The failure path is caught and dropped on the floor.

try {
  await syncRemoteState(payload);
} catch (err) {
  // handled
}

Nothing is handled. The test that expected no throw now passes.

Relaxed assertions. A strict matcher becomes a loose one.

- expect(result).toEqual({ id: 7, status: 'settled', total: 4200 });
+ expect(result).toBeTruthy();

{} is truthy. The test is now green for almost any output.

Assertion stripping. The checks that actually pin behavior quietly disappear.

  const rows = await repo.findByOrg(orgId);
- expect(rows).toHaveLength(3);
- expect(rows[0].email).toBe('a@b.co');
+ expect(rows).toBeDefined();

No-op fix. The PR claims to fix a bug. The source is untouched; only the test changed to stop failing.

Fake refactor. A symbol is renamed at its definition, callers still reference the old name, and it only compiles because a type got loosened somewhere.

Type/lint suppression. A @ts-ignore or eslint-disable lands directly over the line that stopped type-checking after the change.

+ // @ts-ignore
  return handler(req as AuthedRequest);

None of these are exotic. That is the point. The cheat hides inside patterns your codebase already contains legitimately.

Why this matters now

Agents do not cheat more per PR than a rushed human does. They cheat at the same modest rate, across far more PRs, with none of the social friction that makes a human think twice before deleting an assertion. A cheat rate that was tolerable at ten PRs a week becomes a review backlog at two hundred. It is a signal-to-noise problem, and the noise floor is rising.

The data, honestly

Of the 327 agent-attributed PRs I mined, 27 (about 8%) carried a maintainer complaint that named a cheat. Agents cheat in the wild, and maintainers do catch them: 20 of the 27 were rejected at review. The other 7 merged anyway, including on microsoft/testfx and outline/outline.

Now the caveat that most AI posts skip. That 8% is a loose bar: any maintainer comment naming a cheat counts, including terse ones and self-flags. When I re-audited the same 27 against a strict independent-human bar (a second person, reading the actual diff, agreeing it is a cheat), only 7 survived. So the honest range is "8% by a generous reading, closer to 2% by a strict one." Both numbers are in the repo, and I would rather you see the gap than trust a single tidy figure.

Why your existing tools miss it

Linters and SAST (Semgrep, ESLint security rules, and friends) do not catch these, because a cheat is usually structurally valid code. An empty catch block is legal. A renamed function is legal. A loosened matcher is legal.

Two of the merged examples above make the point:

microsoft/testfx#8513 deleted a test in C#, a language the structural detectors do not even parse.
outline/outline#12197 removed a single jest.mock line. There is nothing malformed to match. The behavior changed; the syntax did not.

A pattern matcher keyed on "bad syntax" has nothing to grab onto. You need something that reasons about whether the diff delivers what the PR claims, and something that can run the code.

What I built

Swarm Orchestrator is an open-source auditor for exactly this. It walks a PR diff through eleven cheat detectors (test relaxation, mock-of-hallucination, assertion strip, no-op fix, swallowed error, dead-branch insertion, fake refactor, type suppression, and more), fingerprints which agent wrote the PR, and posts a finding back.

The design rule is one sentence: flags are tips, blocks are proof.

Detectors are advisory by default. They flag cheat-shaped patterns and never block a merge on suspicion alone. No detector is allowed to block on its own opinion.
A finding only escalates to a block when the tool can reproduce a runtime proof in a fresh checkout: revert the suspicious hunk, rerun the suite, and confirm the code only "passed" because of the doctored change. The block ships with the exact command to reproduce it.

It runs offline against a committed diff, so nothing here is a number you have to take on faith:

git clone https://github.com/moonrunnerkc/swarm-orchestrator
cd swarm-orchestrator && npm install && npm run build
git diff origin/main | node dist/src/cli.js audit --diff-stdin

The honest limitations

This is the section that should make you trust the rest.

No detector clears the bar to auto-block. Zero of eleven are gate-eligible. Every one is advisory.
The proof gate proved zero of the 27 wild cheats on its own. The kind of cheat a machine can prove without a human (a merged test-tamper whose restoration provably fails) simply did not occur in the sample it could execute. Cheats that need judgment to see stayed invisible to the gate, by design, because the gate refuses to fire on an accusation.
There is a known false-positive class. Refactors that relocate test coverage (moving a check into a golden-file test the engine cannot see) once tripped a false proof. It is pinned by a regression test now, but it is a real limit, not a solved problem.
The advisory tier is where the daily value lives: across the hunt it flagged 148 candidates and corroborated all 27 human-caught cheats. On a defect-injection corpus it recovers 301 of 325 planted cheats (92.6%). Corroboration with a known ceiling, not a replacement for review.

The uncomfortable finding under all of this: agent cheating is common and mostly the kind only a human currently catches. Automation can raise the signal and prove the rare merged case. It cannot yet stand in for the reviewer.

If you want to poke holes in it

Run it against your own diffs. Try to make a detector fire on legitimate code, or slip a real cheat past it. Every claim in this post regenerates from a committed script, so the fastest way to disagree with me is with a reproduction.

Repo: https://github.com/moonrunnerkc/swarm-orchestrator

It is an open problem, not a finished product.

Catching the shortcuts AI coding agents take to look done

Brad Kinnard — Fri, 05 Jun 2026 21:40:35 +0000

A green test suite is supposed to mean the change works. It doesn't. A test can be weakened just enough to pass. An error can be caught and thrown away. A rename can stop halfway and still compile. None of that turns red, and none of it shows up in the linters most teams already run.

Swarm Orchestrator is built to catch exactly that class of problem in AI-written pull requests.

Two parts. One audits AI-written PRs for the shortcuts that fake "done" (11 checks). The other gates a patch against a contract you define: it builds, passes tests, satisfies your requirement, and survives a falsifier that tries to break it.

TypeScript, Node 20, ISC license. The audit side runs with no model credentials.

The gap linters leave

Semgrep and ESLint are built around risky APIs and known-bad code patterns. Whether a diff is honest is a different question. They won't tell you a test was edited until it passed, or that a catch block quietly eats the error it caught. That's the gap.

Two examples from merged Cloudflare pull requests:

PR	Finding	Semgrep + ESLint
`workers-sdk#14063`	Function renamed, some callers still using the old name	No finding
`workers-sdk#14132`	Empty catch block hiding errors	No finding

Across 72 known-bad pull requests from 12 repositories, that pair of analyzers produced one finding. The auditor flagged 67.

What the auditor checks

Eleven checks total. Eight run by default. The other three exist but stay off, because they haven't shown useful signal on real pull requests yet, and a noisy check is worse than no check.

The default set looks for things like:

Errors caught and ignored
Renames left unfinished
Test coverage reduced
Tests weakened
Assertions removed
New @ts-ignore or eslint-disable comments
Test-only fixes with no code change behind them
Mocks pointing at modules that don't exist

Measured, not assumed

The detection rate isn't a guess. Known defects get injected into real pull requests, then the auditor runs against them. It caught 253 of 300, or 84 percent.

Reproduce it:

npm run benchmarks:full

Runtime mode (optional)

The checks can also execute code instead of only reading a diff: mutation testing, coverage, and reproducing reported issues.

On trpc#6098 it found mutations surviving on lines a later hotfix changed. The tests passed. They weren't actually exercising that code.

Why this mode stays optional
Running code is louder than reading a diff: it averages about 3.4 findings on a clean pull request. That noise is fine when you're deliberately hunting, but it's too much to leave on by default, so it's opt-in.

Defining "done" with a contract

The second command is swarm run. You write down what done means:

obligations:
  - type: build-must-pass
    command: npm run build
  - type: test-must-pass
    command: npm test

A patch is accepted only if every obligation passes and the falsifier can't break it. The default provider is deterministic, so identical inputs give identical results, and every input and hash gets written to a hash-chained ledger.

Blocking merges

Findings are advisory out of the box. Gate mode can block a merge, but only on reproducible evidence. The structural checks throw too many false positives to trust as automatic blockers on their own.

Right now no runtime signal has enough real-world evidence to justify auto-rejection, so the gate stays open and reports that fact directly instead of pretending otherwise.

Who it's for

If you review a lot of AI-written pull requests and want signals the usual linters skip, that's the case this is built for. It also emits CycloneDX-ML and SPDX AI BOM documents with --emit-aibom, supports TypeScript and JavaScript, and runs offline.

It points reviewers at the code worth inspecting. It doesn't claim to prove anything bug-free.

View the repo on GitHub

moonrunnerkc / swarm-orchestrator

Reviews pull requests for the shortcuts AI coding agents take to look done without being done: relaxed tests, swallowed errors, fake renames, 11 checks in all. Flags them for a human by default, or blocks the merge if you turn that on. Can also turn a goal into a checklist and only accept a patch once every check passes.

Swarm Orchestrator

A CLI for auditing AI-generated PRs and grading patches against typed contracts.

= 20">

Install · Quick start · What it does · Results · Detectors · AI-BOM · Reference

What This Does

Swarm Orchestrator reads a pull-request diff and flags the shortcuts an AI coding agent takes to look done without being done: relaxed tests, stripped assertions, swallowed errors, fake renames, eleven checks in all. On a benchmark of planted cheats it recovers 253 of 300 (84%, up 20.5% from the prior version), and on real merged Cloudflare PRs it caught two cheats that Semgrep and the ESLint security rules missed, both reproducible offline. Findings are advisory by default, so it never blocks a merge unless you turn that on.

Who it's for

You review AI-written PRs at volume and want a "this change may be gaming the tests" signal that ordinary linters do not give you.
You have…

View on GitHub

AI wrote the PR. How do you know it actually works?

Brad Kinnard — Wed, 03 Jun 2026 01:26:44 +0000

AI agents open a lot of pull requests now. Most are fine. Some quietly cheat to make the checks go green: they delete the failing test, weaken an assertion, wrap the broken call in an empty catch so the error disappears. The diff looks done. A reviewer skimming forty agent PRs a day will not catch that by eye.

swarm audit is a command-line tool that does. I maintain it. It runs three jobs on AI-written code, all offline, no API key.

1. Catch the cheat

Eleven checks read a pull-request diff and flag the shortcut patterns: a deleted test with no matching code change, a function renamed while its callers still use the old name, an error swallowed by an empty catch, a mock of a package that exists in no manifest, a type-checker suppression dropped over a changed line, and more.

The detection is measured, not asserted. Hide one known cheat in each of 300 real merged PRs, run the auditor, count the catches: 254, about 85%, reproducible with one command.

The catch that matters most is on real code. On two merged Cloudflare PRs it flagged a rename that left two callers pointing at a dead function, and an empty catch {} that throws every error away. Semgrep (210 rules) and ESLint's security rules flagged neither, because they hunt for dangerous code like an injection or a leaked secret, and a deleted test is not dangerous code, it is missing code. The auditor also names the agent that wrote the PR: on a live fetch it tagged the author as Devin. Findings ship advisory, so it reports and never blocks your merge unless you ask it to.

2. Prove the change meets its spec

The second mode runs before a change is accepted. You hand it a plain goal. It compiles that goal into a contract of machine-checkable obligations: build passes, tests pass, coverage holds, a named function has the right signature, a property holds, performance does not regress. Candidate patches get generated, and one is admitted only if it satisfies every obligation. Adversarial falsifiers actively try to break a patch before it counts.

In a fresh project it compiled a goal into two obligations, verified both, confirmed nothing regressed after the merge, and spent zero tokens doing it. Turn on gate mode and it becomes a hard merge rule: a diff that strips a test exits non-zero and never lands.

3. Document it for compliance

If you ship or buy AI-written code under the EU AI Act or CISA's SBOM-for-AI guidance, someone will ask for a record of the AI involvement. The tool emits one: a CycloneDX 1.6 ML bill-of-materials and an SPDX 3.0 AI-Profile, both valid against their specs, plus a hash-chained evidence ledger where altering any entry breaks the chain. It ships with the mappings to EU AI Act Annex IV and the CISA minimum elements.

Try it

git clone https://github.com/moonrunnerkc/swarm-orchestrator
cd swarm-orchestrator && npm install && npm run build

npm run benchmarks:oracle   # the ~85% number
node dist/src/cli.js audit --diff-file benchmarks/real-prs/diffs/cloudflare-workers-sdk/14132.diff --detectors all
swarm init && swarm run --goal "verify this project builds and tests pass"

Open Source Repo: https://github.com/moonrunnerkc/swarm-orchestrator

Your AI's tests pass. That doesn't mean the code works.

Brad Kinnard — Sun, 31 May 2026 22:21:05 +0000

You ask a coding agent to fix a bug. It writes the code, writes the tests, CI goes green, you merge. The bug's still there.

The agent's job was to turn the check green. The honest way to do that is to fix the code. The lazy way is to write a test that passes no matter what the code does. CI can't tell those two apart. A green check means the tests passed, not that the code is right.

It's easy to miss in review, because the test sits right there looking like proof:

test("parses the config", () => {
  const result = parseConfig(rawInput);
  expect(result).toBeDefined();
});

That passes whether parseConfig works perfectly or returns nothing useful on every input. It checks nothing. Adding more tests like it just raises your coverage number, not your odds of catching a bad change.

So I built ClaimCheck (https://github.com/moonrunnerkc/claimcheck). Instead of trusting the agent's tests, it tries to break them. If a test still passes after the supposedly fixed code is broken on purpose, the test was never really checking the fix, and it gets blocked. Same answer every time, no AI making the call. So far it's caught every cheat in a set of twelve hand-built cases. Twelve is small, and there's no public release yet, so treat that as a direction, not a finished result.

Some cheats slip through anyway. If the agent writes a real, solid test that locks in the wrong answer, every check passes. The only way to know the answer's wrong is to already know the right one, and nothing in the pull request can tell you that except the agent you're trying to catch. The one thing that helps is a clue from outside it, like a human-written bug report you can run the fix against.

There's a second, wider tool, Swarm Orchestrator (https://github.com/moonrunnerkc/swarm-orchestrator). It flags suspicious changes and keeps a tamper-evident record for audits. The record-keeping is the solid part. The catching is not: on real pull requests its accuracy is still low, and that's the half I'm hardening now.

The next step is comparing the old code's behavior to the new directly. The catch is that a wrong change and a harmless cleanup can look the same from the outside, and a tool that blocks good code is worse than one that lets a bad change through. That's the part I'm still working out.

Audit AI-Generated PRs Before You Merge Them (Swarm Orchestrator 10.3.0)

Brad Kinnard — Sun, 24 May 2026 20:54:59 +0000

If you let Claude Code, Cursor, Devin, Aider, Copilot, or any other coding agent open PRs against your repo, you already know the problem. The diff looks fine on a fast read. CI is green. You merge it. A week later you find the test that "passed" got deleted, or the error handling is a silent catch {}, or the "fix" was a comment swap that never touched the bug.

Swarm Orchestrator looks at those PRs and flags the suspicious bits before you click merge.

What it is

A CLI and a GitHub Action. Open source. Node 20 or later. You point it at a PR (or a local diff) and it scores the patch against a set of cheat-pattern detectors. It posts a comment back to the PR with what it found and why.

swarm audit moonrunnerkc/swarm-orchestrator#42

That's the whole interface for most people.

What it does

The default detector set has four checks, all aimed at patterns AI agents actually produce on real PRs:

error-swallow: a new empty or comment-only catch block in non-test code.
mock-of-hallucination: a jest.mock or vi.mock against a module that doesn't exist anywhere in the repo.
no-op-fix: tests changed without source, or source changed without tests, when the diff claims to fix something.
fake-refactor: an exported symbol renamed in source, with no caller in the diff updated.

Six more detectors live behind --detectors experimental for shadow runs. They're not scored well enough on real PRs to be on by default, and the README says so.

Every finding renders with its measured precision number inline, so a reviewer sees the false-positive rate every time the bot speaks.

If you need compliance artifacts, --emit-aibom cyclonedx-ml writes a CycloneDX 1.6 ML-BOM and an SPDX 3.0 AI-Profile per audit. That covers the EU AI Act Annex IV and CISA SBOM-for-AI minimums without bolting on a separate vendor.

Who it's for

Teams that let AI agents open PRs and want a second pair of eyes that runs in CI, costs nothing per call, and produces a deterministic comment instead of vibes. Also useful for procurement and security folks who need an AI-BOM next to their SBOM and don't want another tool in the chain.

If you have one developer eyeballing every line of every AI PR by hand, you probably don't need this yet. If you have ten agents pushing diffs to a queue at 2am, you do.

What's new in 10.3.0

Four things:

no-op-fix got a v2.0 with a gated LLM judge. The judge is off by default and only fires when you set --enable-llm-judge (or SWARM_AUDIT_LLM_JUDGE=1) and have an Anthropic key. Verdicts are content-addressed and cached, so the same diff and title always gets the same answer. The model id is pinned in the ledger so replay stays deterministic.
--shadow-output <path>. One JSON file per audit with detector verdicts, judge call count, and the rendered comment. Drops into a directory you can jq later. The existing --shadow <repo> per-repo rollup still works.
Public leaderboard on GitHub Pages. Fetches the real-corpus score snapshot and renders precision, recall, F1, and a sortable per-detector table. No build step, no CDN, just an HTML page and one JS file: moonrunnerkc.github.io/swarm-orchestrator/docs/leaderboard/.
Real-corpus headline rescored against the v2.0 detectors. F1 moved from 0.109 (P 0.067, R 0.300) to 0.167 (P 0.100, R 0.500). mock-of-hallucination picked up two true positives the v1 shape missed.

The honest part

The real-corpus F1 is 0.167 across 205 AI-labeled PRs (10 broken, 195 clean, eight agent vendors). Precision is 0.100. Recall is 0.500.

That precision number is exactly why the default mode is advise and not gate. Most flags will be false positives. The tool is calibrated to be useful as a reviewer-assist signal, not a merge blocker. If you want it to block, opt in: --mode gate.

The 205-PR corpus is currently labeled by an AI judge with "pending human review" stamped on every entry. That's the largest credibility hole in the project and the next milestone closes it. The labeling rubric, the kappa script, and the labels-v2 scaffold already live in the repo.

Don't read this as "ship this into your release gate today." Read it as "here's a tool you can run in shadow mode, look at what it flags, and decide for yourself if those flags are useful."

Try it

git clone https://github.com/moonrunnerkc/swarm-orchestrator.git
cd swarm-orchestrator
npm install
npm run build
npm link

# audit a PR (advisory, never blocks)
GITHUB_TOKEN=... swarm audit owner/repo#PR

Or wire it into a workflow with uses: moonrunnerkc/swarm-orchestrator@main and audit-mode: true.

Sources:

Repo: https://github.com/moonrunnerkc/swarm-orchestrator
Leaderboard: https://moonrunnerkc.github.io/swarm-orchestrator/docs/leaderboard/
Real-corpus score snapshot: https://github.com/moonrunnerkc/swarm-orchestrator/blob/main/benchmarks/real-corpus/scores/latest.json
CycloneDX 1.6 ML-BOM spec: https://cyclonedx.org/specification/overview/
SPDX 3.0 AI Profile: https://spdx.dev/use/specifications/
EU AI Act Annex IV: https://artificialintelligenceact.eu/annex/4/

Cryptographic Forensics for AI Coding Agent Sessions

Brad Kinnard — Wed, 20 May 2026 14:58:25 +0000

A Claude Code or Codex CLI session writes a JSONL file to disk. If the agent runs rm -rf on a training-data directory or terraform destroy -auto-approve on production, that file is where an incident review starts.

A JSONL file is not evidence. Anyone with shell access can rewrite it. To a third party who doesn't trust the machine it came from, it proves nothing.

That gap matters once agents have credentials to real infrastructure. Most agent observability tooling is built for debugging and quality, not for the moment after damage is done. This post is about the three cryptographic properties that turn a transcript into something an auditor or regulator can verify, and how the DEPOSE project wires them together.

Three properties

Assume the machine that produced the bundle can't be trusted. Three things need to hold at once:

Tamper-evident. Any byte change has to be detectable. Hash chain over events: change a byte, replay fails.
Authenticated. The record has to be bound to a key the producer controls and publishes a fingerprint for. Ed25519 signatures over a manifest.
Anti-backdated. A party other than the producer has to anchor the record in time. RFC 3161 tokens from a public TSA.

The primitives are old and well understood. The hard part is wiring them through a normalized event schema and shipping a verifier that doesn't depend on the producer's runtime.

No LLM in the signed path

Every event is captured at execution time or normalized from the session JSONL, then committed to the hash chain. The human-readable narrative is generated separately, from deterministic Handlebars templates over the signed events. It's excluded from the root hash.

If generated prose became part of the signed record, verification would depend on model behavior staying stable and reproducible. DEPOSE avoids that dependency. The signed record is event data and hashes. The prose is templated commentary with [#evt-<ulid>] citations back to the signed events. You can rewrite the narrative without affecting verification. Change an event and verification fails.

What's in a bundle

A DEPOSE bundle is a directory, not an opaque archive:

incident-01JABC.../
├── manifest.json            bundleId, rootHash, eventsJsonlSha256, sigs, timestamps
├── events.jsonl             every event in canonical JSON, byte-pinned by manifest
├── rules/destructive.yaml   ruleset used at reconstruction time
├── narrative.md / .html     templated prose with per-event citations
├── verify.txt               human-readable verification summary
├── artifacts/               captured file diffs, payloads
├── attestations/            Ed25519 signatures, RFC 3161 timestamp tokens
└── raw/                     source JSONL, shell history, capture records

Change a byte of events.jsonl, manifest.json, or rules/destructive.yaml and verification fails. Canonical JSON follows RFC 8785 (JCS), which is what lets a Go verifier check a TypeScript-produced bundle without either side trusting the other's serializer.

Two binaries

The producer is TypeScript. The verifier is a separately-built static Go binary, depose-verify. The separation is deliberate: you hand the binary to whoever needs to check the bundle (auditor, opposing counsel, regulator, a customer's security team) and they run it on their own machine. No producer stack required.

A passing run prints:

parse        OK
signature    OK
chain-replay OK
artifacts    OK
timestamp    OK
PASS  bundleId=...  rootHash=...

The cryptography here is mostly off-the-shelf. The actual engineering work is in normalization: getting Go and Node to serialize identically, getting timing and ordering right across capture sources, deciding what counts as one event versus two. Canonical JSON is the unsexy part. Float formatting, key ordering, unicode escapes: Go and Node have to agree byte-for-byte or the verifier rejects a bundle the producer thinks is fine. That's what the cross-language conformance vectors in tests/conformance/ are for.

Verifiers can pin a producer's expected key fingerprint and consult a revocation list, both at the command line. The RFC 3161 timestamp does double duty here: a bundle stamped before a key is revoked stays time-anchored, so "when was this signed" remains answerable even if the key is later compromised.

Capture modes

Two modes, different coverage.

Reconstruction reads the Claude Code session JSONL after the fact, compares it against shell history (bash, zsh, fish) and git reflog where available, and builds a bundle. Lower-bound mode. It can verify integrity after packaging. It can't prove the original session file was complete before capture.

Active capture installs a Claude Code PreToolUse hook and POSIX shell shims for the binaries that tend to do destructive things: terraform, aws, gh, kubectl, psql, gcloud, railway, rm. Records land under ~/.depose/captures/ at execution time. A later depose package merges them with the session JSONL so every covered event has a verified pre-execution intent on record.

DEPOSE can prove integrity of captured events. It can't prove an uninstrumented system captured everything. An agent that shells out to a binary not in the shim list, or hits an API directly, still shows up in the JSONL but won't have an active-capture record. The coverage matrix is in the repo.

macOS and Linux only. Windows isn't supported (POSIX 0600 on the key store, POSIX shell scripts for the shims). WSL2 works.

Release pipeline

Releases ship with SBOMs, provenance attestations, and signed checksums. The specifics: CycloneDX for both halves, SLSA L3 provenance, and SHA256SUMS signed via cosign keyless. CI rebuilds the two checked-in example bundles (an rm -rf on training data, a terraform destroy on infrastructure) on every push and runs three semantic tamper rejections to confirm the verifier fails closed.

Right now most coding-agent session logs are treated like disposable debug output. That assumption gets weaker the moment an agent can modify infrastructure.

https://github.com/Aftermath-Technologies-Ltd/depose

Gemma.Witness - Offline Multimodal Evidence Capture with Gemma 4

Brad Kinnard — Sun, 17 May 2026 03:43:46 +0000

This is a submission for the Gemma 4 Challenge: Build with Gemma 4

What I Built

Gemma.Witness is an offline-first multimodal evidence capture system built for environments where cloud access, trust, or chain-of-custody assumptions fail.

The system records audio alongside supporting images, runs local multimodal analysis through Gemma 4, and produces a signed evidence bundle containing:

Structured incident reports
Timestamped evidence metadata
Local reasoning traces
Hash-linked verification artifacts
Exportable forensic bundles

The focus was reliability and local verification instead of "AI assistant" behavior.

Most evidence tooling today assumes internet access, centralized APIs, or mutable storage. Gemma.Witness was designed around the opposite assumption: the network may be unavailable, the machine may be isolated, and every generated output may eventually need independent verification.

The application runs fully local through a desktop interface using Rust, Tauri, and local inference orchestration.

Demo

github: https://github.com/moonrunnerkc/gemma-witness

Code

Source code is available at the repository above.

How I Used Gemma 4

Gemma 4 is the reasoning layer behind the entire evidence pipeline.

I used Gemma 4's multimodal capabilities to process:

Audio-derived transcripts
Scene images
Cross-evidence consistency analysis
Structured incident extraction
Reasoning trace generation

The model is used in a multi-pass workflow instead of a single prompt-response cycle. Each pass validates or expands on the previous stage before the final signed bundle is emitted.

This matters because evidence systems fail quietly when models hallucinate details, merge assumptions into facts, or overstate certainty. The pipeline was intentionally designed to separate:

Raw observations
Inferred conclusions
Confidence scoring
Verifiable artifacts

Gemma 4 was a strong fit because it could operate locally while still handling multimodal reasoning tasks without requiring cloud APIs or external orchestration services.

The project prioritizes:

Offline operation
Verifiable outputs
Local ownership of evidence
Minimal trust assumptions
Reproducible forensic artifacts

A surprising challenge was not getting the model to generate reports. That part was easy.

The difficult part was building guardrails around evidence integrity so the system does not quietly become a very confident fiction generator wearing a necktie.

Tech Stack

Gemma 4
Rust
Tauri
Node.js
Local multimodal inference
Cryptographic hashing and bundle verification

Repository

moonrunnerkc / gemma-witness

Offline multimodal evidence capture that emits a signed, locally verifiable .witness bundle. Tauri + Rust + Gemma 4 + Ed25519. Static HTML verifier runs with no server.

Gemma.Witness

Offline, tamper-evident evidence capture for field journalism. Signed in your hand, verified in a browser, with no server in the loop

demo.mov

Why · Status · Install · Usage · Configuration · Threat model · Limitations · Verify yourself · Contributing

Why this matters

A reporter is working in a country where journalists are detained for their reporting. She records a witness account. She attaches the photos she just took. She seals the file before she leaves the room.

A week later, an editor on another continent opens a single static HTML page in any browser and drags the file in. Three checks turn green:

the signature comes from the reporter's device
the audio and the photos have not been altered by a single byte
the AI model in the chain is bit-for-bit the published Gemma model her manifest names, by model_id, revision, and model.safetensors SHA-256

In…

View on GitHub

Swarm Orchestrator v8.0.2

Brad Kinnard — Tue, 12 May 2026 02:31:18 +0000

v8.0.2 is out now and it cleans up several rough edges that kept showing up under heavy tournament and falsification workloads.

The biggest operational change is that all four previously documented architectural limitations are now closed in the same release (7b68867).

Notable Changes

Tournament mode now streams through the same pipeline as single mode. If one candidate fails streaming verification, it gets aborted independently instead of poisoning the whole run.
Live cost-cap enforcement is now real-time. Concurrent streams project cumulative USD usage continuously and abort the moment projected spend crosses the configured cap.
Snapshot cleanup is automatic now and supports retention policies like:

  retain-last:N
  max-age:<dur>
  max-disk:<sz>

Adaptive falsifier dispatch using UCB1 is available behind:

  --falsifier-scheduler ucb1

ARIES-style rollback support landed for falsified obligations. If a counter-example appears after apply, the workspace restores from the pre-apply snapshot and verifies the rollback by hashing the restored bytes against the original SHA.

New Command

There is also a new command:

swarm v8 stats <run-id>

That surfaces persisted falsifier metrics directly from:

.swarm/falsifier-stats.json

including regression discoveries, false positives, success counts, and latency.

Replay Determinism

One important detail: replay determinism remains intact across all of this. Every scheduler decision and abort event still lands in the ledger so replay reproduces the same winner consistently. That part was non-negotiable.

Release

https://github.com/moonrunnerkc/swarm-orchestrator/

How Swarm Orchestrator v8 Tries to Break Its Own AI Patches

Brad Kinnard — Sun, 10 May 2026 02:10:05 +0000

Most AI coding tools commit when their own checks pass. Swarm Orchestrator v8 adds a second adversarial layer: independent falsifier adapters that try to break each patch before it merges. v8.0.1 is on main with that subsystem on by default.

This post walks through the v8 architecture, the four verification points, the producer/falsifier adapter split, and the limitations that haven't been solved in v8.0 yet.

What is Swarm Orchestrator? A contract-first AI coding swarm with hash-chained evidence and verifier-gated commits. It compiles a natural-language goal into a typed contract, dispatches it to a population of personas inside one cached Anthropic session, races candidate diffs per obligation, and commits only what passes verification. It wraps an LLM; it doesn't replace one.

The shape of a run

You hand it a goal in plain English. The contract compiler turns that into contract.jsonl plus a manifest.json carrying the goal, repo context, extractor provenance, and a SHA-256 of the canonical contract bytes. Identical inputs produce identical contract hashes.

goal (text)
   |
   v
contract compiler  ->  contract.jsonl + manifest.json
   |
   v
+-------------------------------------------------+
|        population manager (single session)      |
|                                                 |
|  ledger (jsonl, hash-chain) <- personas (8)     |
|       ^                          |              |
|       | tournament + verifier scoring           |
|       |                                         |
|  WASM deterministic floor (zero-LLM obligs)     |
+-------------------------------------------------+
   |                              |
   v                              v
streaming verifier      post-merge integration
   |                              |
   +--------------+---------------+
                  v
       falsifier adapters (Codex, Copilot)
                  |
                  v
            committed diffs

The population manager opens one cached Anthropic session and walks each obligation. It picks the persona whose trigger predicate matches the obligation type. In tournament mode, N candidates run in parallel; the verifier scores them, the top scorer is a commit candidate, and losers get logged but never merge.

Two adapter subsystems

The most common confusion in v6 was treating the coding CLIs and the falsifiers as one thing. v8 splits them cleanly.

Producer adapters (src/adapters/) wrap third-party coding CLIs as the worker in the v6 verified-branch pipeline. Backends: Copilot, Claude Code, Codex, Claude Code Teams. All four are opt-in via swarm run --v6.

Falsifier adapters (src/falsification/adapters/) take a patch the producer's verifier already accepted and try to falsify the obligation by surfacing a counter-example, regression fixture, or property-violation trace. A confirmed counter-example flips the obligation back to failed.

Falsifier	Default	Obligation types
`CodexFalsifier`	on	`property-must-hold`
`CopilotFalsifier`	on	`import-graph-must-satisfy`, `function-must-have-signature`
`ClaudeCodeFalsifier`	off (per-adapter opt-in)	all three

The CLI surface is one flag: --falsifiers <on|off> (default on). Per-adapter selection happens at the API layer via defaultAdapterRegistry({ includeCopilot, includeClaudeCode }).

Four verification points

A patch has to survive these before it merges:

Pre-generation memoization. Skip generation if the obligation result is already cached.
Mid-stream abort. During generation, the streaming verifier can abort the call. Works in --mode single only; tournament mode skips it.
Post-generation per-obligation verifier. Scores the candidate diff. In tournament mode, top scorer wins; in single mode it's pass/fail.
Post-merge integration check. After the diff lands, the integration check confirms the broader system still holds.

The architectural rule from the README: nothing commits without passing the obligation's verifier. Then the falsifiers get a shot.

The hash-chained ledger

Every action lands in .swarm/ledger/<run-id>.jsonl with the SHA of the prior entry. Tampering is detectable; runs resume from any prior state. If a process is killed mid-run, swarm v8 resume <run-id> walks the ledger and picks up where it left off.

The ledger format is shared with v6, but v8 writes more granular events (per-persona dispatch, per-candidate score, falsifier verdict) so a run can be replayed or audited end-to-end.

Quick start

git clone https://github.com/moonrunnerkc/swarm-orchestrator.git
cd swarm-orchestrator && npm install && npm run build && npm link

# Compile a goal, then run it
swarm v8 compile "add a /health endpoint that returns 200 OK" --yes
swarm v8 run .swarm/contracts/<contract-id>

# Or both in one step (defaults to v8)
swarm run --goal "add a /health endpoint that returns 200 OK"

# Resume a killed run
swarm v8 resume <run-id>

Requires Node >= 20, git >= 2.40, and ANTHROPIC_API_KEY. Pass --extractor stub --session stub to run offline.

There's also a GitHub Action:

- uses: moonrunnerkc/swarm-orchestrator@v8
  with:
    goal: "add a /health endpoint"
    contract-only: false
    cost-cap: "5.00"
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

What v8.0 doesn't do

Limitations worth reading before adopting

Tournament mode doesn't stream. --mode tournament plus --forbid-import skips the streaming abort; streaming verification is --mode single only.
Post-merge failure doesn't auto-rollback. The run is marked failed; per-obligation worktree snapshots are post-v8.0.
--cost-cap is enforced post-obligation, not mid-call. Cumulative spend is checked at the end of each obligation against estimated Sonnet 4 pricing. Exit code 6 if exceeded.
Bandit dispatch is not built (Phase 5). Codex and Copilot have disjoint obligation types, so there's nothing to arbitrate between yet.
Cross-vendor producer race is deferred (Phase 6).

Full list with rationale lives in docs/v8-architecture-deviations.md.

Repo

moonrunnerkc / swarm-orchestrator

Contract-first AI coding swarm with hash-chained evidence. Compiles a goal into typed obligations, races persona candidates per obligation in a single cached inference session, verifies before commit, and logs every action in an append-only ledger.

Swarm Orchestrator

Contract-first AI coding swarm with hash-chained evidence and verifier-gated commits.

swarm compiles a natural-language goal into a typed contract, dispatches it to a population of personas inside one cached Anthropic session, races candidate diffs per obligation, and commits only the diffs that pass verification. After the producer's verifier accepts a patch, registered falsifier adapters get a chance to break it before it merges. Every action lands in an append-only hash-chained ledger you can audit, resume, or replay.

It wraps an LLM; it does not replace one. The model writes the code, the orchestrator decides what reaches your repo.

Status

Version 8.0.1 on main. Node >= 20 (CI matrix: 20, 22). License ISC. The v8 architecture is the default for swarm run; the v6 verified-branch pipeline is preserved under swarm run --v6 and the swarm swarm / swarm execute commands Falsifier subsystem: Codex on, Copilot on, ClaudeCode…

View on GitHub

How to Write a CLAUDE.md Rule That Actually Gets Enforced

Brad Kinnard — Sun, 10 May 2026 02:06:45 +0000

Open a CLAUDE.md file at random and you'll find build commands, architecture notes, and rules. The rules tend to be the unenforceable kind. "Write clean code." "Be careful with types." "Follow our conventions." The author meant every word. The agent reads them. And nothing checks whether the agent followed them, because nothing can.

In a corpus of 580 CLAUDE.md, AGENTS.md, and .cursorrules files from public GitHub repos with 10+ stars, 74% contained zero machine-extractable rules. Not because the authors didn't care about rules. Because most rules were written in a form no parser could pull out as a deterministic check.

This post is about the difference. Specifically: how to phrase a rule so a parser can extract it and a verifier can check it, without sacrificing what you actually meant.

The principle

Enforceability comes from a verifiable surface. A rule is verifiable when there's a concrete pattern in code that either matches it or doesn't. "Use camelCase for function names" is verifiable: read the AST, list the function names, check the casing. "Name things consistently" is not: there's no concrete pattern to check, only a judgment to make.

The gap between the two is the gap between intent and enforcement. You meant the same thing in both cases. But only one of them survives translation into a check.

Here's the heuristic I use: could a junior engineer with no context mechanically check whether code follows this rule, just by reading the rule and looking at the code? If yes, the rule is enforceable. If they'd have to ask "what does 'consistent' mean here?", it isn't.

Three pairs, walked through

Take a few common intents and look at how they fail or succeed.

Type safety. You want strong typing.

Bad:  Be careful with types.
Good: No `any` types in `src/`. Async functions require explicit return types.

The bad version has no surface. "Careful" isn't a check. The good version has two: a forbidden token (any) and a structural property (return type annotation on async function declarations). Both check directly against the AST.

Module structure. You want predictable imports and exports.

Bad:  Prefer clean module structure.
Good: Named exports only. No default exports. Filenames in kebab-case.

"Clean" is meaningless to a parser. Named-only and kebab-case are both binary properties of code that exist or don't. The first version sounds like more guidance because it's broader, but breadth is the problem: it covers everything and enforces nothing.

Preferences over alternatives. You want React functional components, not class components.

Bad:  Write modern React.
Good: Prefer functional components over class components.

"Modern" is a moving target with no fixed surface. The "prefer X over Y" pattern, on the other hand, has a clean check: count instances of each, compute a ratio, score against a threshold. This is one of the most useful patterns in instruction files because it captures real-world preference (not absolute prohibition) in a measurable way.

The reference table

Twelve common intents, paired:

Intent	Unenforceable	Enforceable
Naming functions	Name things consistently	Use `camelCase` for function names
Filenames	Pick reasonable filenames	All filenames in kebab-case
Type safety	Be careful with types	No `any` types
Async return types	Make types clear	Async functions require explicit return types
Module exports	Prefer clean module structure	Named exports only, no default exports
File size	Keep files manageable	Maximum 300 lines per file
Logging	Be mindful of logging	Never use `console.log`; use `src/logger.ts`
Component style	Write modern React	Prefer functional components over class components
Package manager	Use the right package manager	Use `pnpm`, not `npm`
Test files	Keep tests organized	All test files end with `.test.ts`
Error handling	Handle errors properly	Async functions must use `try/catch` or return a `Result` type
Commit format	Write clear commit messages	Use conventional commits (`feat:`, `fix:`, `chore:`)

Every right-hand cell points at a concrete check: a token, a casing rule, a count, a file pattern, a configured tool. Every left-hand cell points at a judgment.

Real-world rules usually carry scope: "no any in src/," "named exports outside index.ts files," "no console.log in production code paths." Scope makes a rule narrower and more accurate without making it less enforceable. The interop layer that genuinely needs any keeps it; the rest of the codebase doesn't.

What kinds of checks exist

Worth knowing what's available, because it shapes what's writable. Static analysis tools targeting AI instruction files generally support a few classes of check:

AST-level: function names, type annotations, import patterns, forbidden tokens, structural properties
Filesystem: file existence, naming conventions, directory layout, file size limits
Regex: literal strings, content patterns, conventional formats
Tooling: presence and configuration of linters, formatters, package managers, test runners
Config-file: contents of .eslintrc, tsconfig.json, .prettierrc, etc.
Git-history: commit message formats, branch naming conventions
Preference ratios: "prefer X over Y" with a compliance percentage instead of a binary verdict

If your rule maps to one of these classes, it's enforceable. If it doesn't, it isn't. The trick when writing instruction files is to keep that map in mind: when you're about to write "be careful with X", ask which of these classes "carefulness with X" lives in. Usually the answer points at a concrete reformulation.

The unenforceable rules aren't worthless

Here's a real tension: most of what makes a CLAUDE.md useful isn't enforceable at all. Project context (what the repo does, where the architecture lives), agent behavior directives (be succinct, ask before deleting, don't touch /legacy), and onboarding instructions are all valuable. None of them extract as rules.

Don't try to make them enforceable. They're a different kind of content with a different purpose. Project context grounds the agent. Behavior directives shape its style. Neither is supposed to be checked against output; they're checked against the agent's process, which is a different problem.

The mistake worth avoiding is letting unenforceable prose crowd out enforceable rules. Anthropic's Claude Code best practices doc recommends deleting any instruction the model already follows correctly without it. Most "write clean code" style rules fail that test: the model already does its version of clean code, so the line is taking up attention budget your agent could be spending on the specific, verifiable rules that actually distinguish your codebase from a generic project. Cut what the model already does. Keep the checks.

A test for your own files

Pull up your CLAUDE.md or AGENTS.md right now. For each line that looks like a rule, ask:

Could a junior engineer check this without asking clarifying questions?
Does it name a specific pattern, file, token, casing, or value?
Would 5 different reviewers all agree on whether a piece of code passes this rule?

If a rule fails 1 or 2, it's not a rule, it's a wish. If it fails 3, it's ambiguous. Rewrite or delete.

If you want a mechanical version of this test, RuleProbe parses CLAUDE.md, AGENTS.md, .cursorrules, .windsurfrules, GEMINI.md, and copilot-instructions.md against 102 matchers and tells you which lines extracted as rules and which didn't:

npm install -g ruleprobe
ruleprobe parse ./CLAUDE.md --show-unparseable

The --show-unparseable flag is the interesting one. It surfaces every line that looked rule-shaped but didn't map to a check. That list is your rewrite queue.

RuleProbe on GitHub

What this leaves out

The hardest case is rules like "follow the existing error handling pattern in this codebase." That's enforceable in principle (compare new code's structural shape against the codebase's dominant pattern), but not by simple AST or regex matching. It needs codebase-aware analysis. Some tools handle that; most don't. If you find yourself writing those kinds of rules, know that they'll either need a tool that does pattern profiling or they'll stay aspirational.

The other thing enforceability doesn't catch: an agent that follows every rule and still writes broken code. Static rules reduce variance, they don't eliminate it. A function with any removed and an explicit return type can still have wrong logic. Treat passing rule checks as a floor, not a ceiling.

I Dropped Multi-Agent Coordination for a 5-Layer Falsification Battery

Brad Kinnard — Sat, 02 May 2026 15:00:11 +0000

Swarm Orchestrator just lost its swarm. Dropped the multi-agent parallel coordination layer. Running one agent now and putting all the weight on a five-layer post-merge falsification battery instead.

This is an experiment, not an endpoint. v8 will bring proper multi-agent swarming back. The reason for cutting it temporarily: I want to know whether the value I was getting from coordinated parallel agents was the coordination itself, or the verification pressure that coordination produced. Easier to measure with one variable. Intended side effect: cost reduction, since the previous architecture spun up multiple CLI agent instances per run. Real benchmarks pending.

TL;DR: every patch survives a five-layer post-merge battery before the orchestrator declares success. Layers 1 and 2 are hard gates. Layers 3, 4, 5 are advisory and feed a composite score. Hard-gate failure throws before attestation, before final gates, before any external success signal.

Pipeline Order

The battery runs once per orchestrator execution against the merged working tree, not per-step branches. The per-step verifier is a separate component. Layers fire in fixed order:

Differential gate (hard)
Mutation gate (hard)
Cheat detector (advisory)
Property gate (advisory)
Attestation (advisory on first run, signed after)

If the hard gate fails, the composite is forced to 0 and the orchestrator throws falsification battery blocked the patch before any external success signal can fire.

Layer 1: Differential Gate (Hard)

Before any agent touches the repo, a synthesizer generates a regression test against the goal. Layer 1 then runs that test in two detached worktrees: one at the base commit, one at the patch commit.

The contract: the test must fail at base and pass at patch.

If the test passes at base, the layer returns INVALID_TEST. This catches the specific failure mode where an agent writes a tautological test that passes against any code. Without this gate, that pattern slips through every other check downstream.

If no command can be synthesized and the caller doesn't pass --differentialTestCommand, the layer fails closed. Deliberate policy.

Layer 2: Mutation Gate (Hard)

Runs Stryker for JS/TS, mutmut for Python, PITest (Gradle/Maven) for Java, against changed files only. First runs the regression command; if that fails, the layer fails immediately. On pass, dispatches the mutation tool and parses the reporter output, including Stryker's clear-text table layout.

Score	Status
< 0.6	FAIL
< 0.8	WARNING
≥ 0.8	PASS

Patches with no meaningful test coverage of the changed lines fail here even when npm test is green.

Layer 3: Cheat Detector (Advisory)

Diff-static scan for known agent-shortcut patterns. Each rule maps to a real category of behaviour I've watched agents fall into under pressure.

Cheat detector rule IDs

test-modification: agent edited the test instead of the implementation
complexity-mismatch: trivial implementation paired with a non-trivial test
mock-mutation: real call replaced with a mock that returns the expected value
exception-swallowing: caught and ignored exceptions to make tests pass
hardcoded-answer: literal return of the expected output

Optional Semgrep integration loads any registered cheat-rule packs into a transient config. The scan reads added and removed lines and patterns; it does not execute the patch. An agent that hides a hardcoded answer behind a small computation can still pass this layer. Honest about that.

Layer 4: Property Gate (Advisory)

Discovers modified TS/JS/Python functions, parses their parameter types, maps each to a fast-check arbitrary or Hypothesis strategy, generates a harness, runs it. Counterexamples surface as findings. Untyped or unsupported types degrade to a low-severity advisory finding rather than blocking.

Layer 5: Attestation (Advisory on First Run)

Reads the refs/notes/swarm-attestation git note for the patch commit, validates the in-toto SLSA v1.0 envelope's subject SHA against the patch commit, then verifies the cosign signature. On the first run for a commit there's no note yet, so this layer reports advisory-warn and the post-battery attestation step writes the note.

The note is verifiable later via swarm attest verify <commit>. A downstream consumer can verify the patch survived the battery without trusting the running orchestrator process.

Composite Scoring

When the hard gate passes, a weighted composite is computed across the three advisory layers and any optional advisory quality-gate results. Failed advisory gates each subtract a fixed penalty.

Default scoring (overridable via .swarm/gates.yaml)

composite threshold: 0.7
weights: cheat detector 0.4, property gate 0.4, attestation 0.2
advisory gate penalty: 0.02 per failure

humanReviewRequired is true when the composite score is below threshold or any advisory layer is in advisory-warn status.

Where It Actually Runs

Three real call sites, not just unit tests:

Production orchestrator on every swarm run
Synthetic calibration corpus (36 paired test specs across 6 broken-category families) executing in CI on every push
SWE-bench harness using Layer 1 and Layer 4 as standalone spot-check eval drivers

Honest Caveats

These are in docs/known-gaps.md and I won't hide them:

Differential gate is host-Python-sensitive on legacy codebases. The synth-eval can reflect import-chain errors rather than assertion outcomes. The authoritative resolution gate in the per-instance Docker image is unaffected.
Mutation gate skips quietly when no changed files match supported languages. YAML, Markdown, Rust, Go diffs don't get mutation-tested.
Cheat detector is diff-static, not behavioural. The hidden-computation-around-hardcoded-answer pattern can pass it.
Attestation signing is best-effort. Cosign-not-installed errors get logged and the run proceeds without a note. The note's absence is reflected in Layer 5's advisory-warn on subsequent runs but does not block.

Why Run This Experiment

If the falsification battery alone produces patches that survive scrutiny at acceptable quality, then a lot of the apparent value of multi-agent coordination was actually the verification pressure it created, not the agent diversity itself. If the battery alone isn't enough, then v8 multi-agent gets a clearer mandate: the swarm is the value, not the side effect.

Either result is useful. The point of the rewrite is to make the answer measurable.

View on GitHub