DEV Community: Brad Kinnard

Reproducible vulnerability scans with dumpscan

Brad Kinnard — Fri, 04 Sep 2026 01:54:07 +0000

Vulnerability scan results can change even when the code does not. Advisory data, comparison rules, or exclusions may change between scans, making earlier results difficult to explain.

I built dumpscan to make vulnerability scans reproducible.

Each scan records four inputs by digest:

Dependency lockfile
OSV vulnerability snapshot
Version comparison rules
Applied exclusions

Using the same inputs produces the same findings byte for byte. When results differ, dumpscan identifies which input changed.

Create a reproducible scan

Instead of relying on the latest advisory data, dumpscan requires a content-addressed OSV snapshot:

dumpscan scan package-lock.json \
  --snapshot ./snapshot \
  --out scan.bundle.json

The resulting bundle contains the input digests, findings, and findings root needed to verify the result later.

This is useful during audits, incident reviews, and investigations where teams need to understand what the scanner knew at a specific time.

Sign and verify results

A signature proves that a file was signed, but not that its findings can be reproduced.

dumpscan records the input digests and findings root in an in-toto statement. It also supports keyless signing with Sigstore:

dumpscan scan package-lock.json \
  --snapshot ./snapshot \
  --out scan.bundle.json \
  --sign

Verify the bundle without running another scan:

dumpscan verify scan.bundle.json

Verification checks the signature, payload binding, findings root, findings count, input manifest, and lockfile digest. It can also verify the certificate issuer and workflow identity.

Replay a scan

Replay derives the findings again from the recorded inputs:

dumpscan replay scan.bundle.json \
  --lockfile package-lock.json \
  --snapshot ./snapshot

The replay must produce the same findings root. If it does not, dumpscan reports the expected and observed digests.

Explain changed results

When two scans differ, dumpscan diff determines whether the change came from the project or the scanning inputs:

dumpscan diff scan-day-1.bundle.json scan-day-2.bundle.json \
  --snapshot-a ./snapshot-day-1 \
  --snapshot-b ./snapshot-day-2

It compares the lockfile, vulnerability feed, version rules, and exclusions. When snapshots are available, it can also identify changed advisory fields.

This helps distinguish a dependency update from an advisory being rescored, withdrawn, or assigned a new fixed version.

Keep uncertainty visible

Excluded, withdrawn, and unevaluated findings remain in the findings set with their status and reason.

Unsupported version ranges are not treated as unaffected, and exclusions are recorded as scan inputs. This keeps the result complete and makes policy decisions visible during review.

Get started

dumpscan requires Node 22.22.2 or later:

npm install --global dumpscan

You can also run it without a global installation:

npx dumpscan --help

Create a snapshot from local OSV records:

dumpscan snapshot \
  --from fixtures/osv/synthetic/records \
  --out ./snapshot

Generate SARIF 2.1.0 output for GitHub code scanning:

dumpscan scan package-lock.json \
  --snapshot ./snapshot \
  --out scan.bundle.json \
  --sarif scan.sarif

The SARIF file is derived output. The bundle remains the reproducible source of truth.

Scope

The first release focuses on lockfile scanning. It does not provide reachability analysis, exploitability analysis, risk scoring, container scanning, or SBOM generation.

These limits are intentional. dumpscan prioritizes reproducible, verifiable results over broad vulnerability-management coverage.

dumpscan is available under the Apache 2.0 license:

github.com/moonrunnerkc/dumpscan

skillcheck Update: Scorer Fixes, Cleaner Failures, Honest Token Numbers

Brad Kinnard — Sun, 23 Aug 2026 03:04:23 +0000

skillcheck is a static analyzer for SKILL.md files, the format agents like Claude Code, Copilot, Codex, and Cursor use to load reusable skills. It validates frontmatter, scores description discoverability, checks file references, enforces token budgets, and flags cross-agent compatibility issues. No network calls, no LLM calls, no file mutations. Runs as a CLI, a GitHub Action, or a pre-commit hook.

pip install skillcheck
skillcheck skills/

Latest pass was hardening and accuracy, not features. Here's what changed and why.

Description scores went up. Skills that were scoring low because the scorer was broken will now see a jump in scoring. Median across the reference corpus went from 75 to 90. --explain-score also now tells you which pattern hits or misses instead of just a number. The score exists to predict whether an agent will actually find and trigger your skill, so a scorer that under-credits good descriptions defeats the point. The fix was validated against real-world skills, and the separation held: filler still scores 28-65, well-written descriptions 85-100.

Corrupt files now fail cleanly instead of crashing. Before, a bad history ledger or non-UTF-8 skillcheck.toml above the skill dumped a Python traceback. It's now a clear error naming the file and byte offset (exit code 2). Config discovery walks up the directory tree, so one bad file could break every scan under it. Now every untrusted read (ingest, history, config) goes through the same guard before parsing, so they all reject the same way.

README has been corrected in regards to token estimates. Without tiktoken, expect roughly 20-30% over-estimation, so install the extra if you're near a budget limit. The offline heuristic feeds the budget checks and its accuracy had never actually been measured, just assumed. It's benchmarked against tiktoken across the full corpus now, and the documented numbers are the measured ones.

pip install "skillcheck[tiktoken]"

The rest of the pass is invisible on purpose: flag-conflict logic consolidated to one source of truth, golden-file tests pinning exact diagnostic output, coverage floor raised from 75% to 80% (actual sits at 90%). Diagnostic output across the corpus verified byte-for-byte identical before and after. Nothing changed except what's above.

moonrunnerkc / skillcheck

Cross-agent skill quality gate for SKILL.md files. Validates frontmatter, scores description discoverability, checks file references, enforces three-tier token budgets, and flags compatibility issues across Claude Code, VS Code/Copilot, Codex, and Cursor.

Static analyzer for SKILL.md files. Validates frontmatter, body sizing, file references, and cross-agent compatibility against the agentskills.io specification. No network calls. No LLM API calls. No file mutations.

1058 tests cover all rule modules.

Install

pip install skillcheck

Requires Python 3.10 or later. For more accurate token estimates, install the optional extra:

pip install "skillcheck[tiktoken]"

Token estimation accuracy

Token counts are estimates, and the sizing rules report them as such. The bands below are measured, not estimated: scripts/measure_token_error.py compares the offline heuristic against tiktoken cl100k_base across a corpus of 61 real SKILL.md files, for the three spans the rules actually size.

Span	Rule	Median error	p95 error	Direction
Whole file	`sizing.total-tokens`	23.0%	30.7%	over-estimates 61/61
Frontmatter	`disclosure.metadata-budget`	25.9%	35.3%	over-estimates 61/61
Body	`disclosure.body-budget`	22.7%	30.7%	over-estimates 61/61

The bias is one-directional: the offline heuristic read high on every file in the corpus. For a budget check that…

View on GitHub

If skillcheck flags something in your skills that looks wrong, open an issue. The reference corpus grows from real-world cases and the scorer improves with them.

It Fits and It Benchmarks Well. Will It Do Your Job?

Brad Kinnard — Mon, 20 Jul 2026 03:05:45 +0000

A fit calculator answers one question: will these weights load in your memory. A leaderboard answers a different one: is this model good at tests in general. The question that decides what you actually run is neither. It's "will Q4_K_M hurt my invoice extraction accuracy on my machine," and the only reliable way to answer it is to run your extraction on Q4_K_M and measure.

QuantProof automates that. Point it at a folder of real examples from your task. It sweeps every model you already have in Ollama (or whatever a Rapid-MLX server is serving), scores every output with deterministic scorers, measures time to first token, tokens per second, and peak memory, then recommends the smallest quant whose quality stays within 2% of the best measured result.

npm install -g quantproof
quantproof ingest my-tasks.md   # a local model drafts the task pack from your notes
# or: quantproof init my-task   # scaffold it yourself
quantproof run --pack my-tasks
quantproof report --markdown

Why measure instead of estimate

Three task packs ran on an M5 Max: ticket classification, invoice extraction, config generation. Three models, 540 scored generations. No model won all three.

An 8B at Q8_0 took classification at 0.950, beating a 30B at 45% of the memory. The 30B was the only model to score 1.000 on extraction. They tied at 1.000 on config generation, where the 8B needs about 17 GiB less.

The 8B's zero on extraction was a token-budget truncation issue, not a quality failure. On 57 of 60 generations it spent its entire 512-token budget on reasoning and never emitted content. Because every raw output is journaled, the report flags the zero trunc! and names the fix (raise max_tokens) instead of letting it read as "this model can't extract."

Which model is "best" changed with every task. That is the leaderboard problem in one paragraph: the ranking is a property of the model and the workload together, and a leaderboard only knows the model.

Determinism is a backend version property, not a model property

Every example runs 3 times at temperature 0 with a fixed seed, and repetitions are compared byte for byte. On Apple Metal, every candidate repeated exactly, across 240+ scored units. Then the same check ran on an RTX 5070. Same pack, same seed, same driver, two Ollama versions:

candidate	ollama 0.15.2	ollama 0.32.1
llama3.1:latest (Q4_K_M)	nondet	deterministic
llama3.1:8b-instruct-q4_0	nondet	deterministic
gemma3:4b	nondet	nondet
gemma3:1b	deterministic	nondet

Both 8B quants became deterministic after the upgrade. The 1B flipped the other way.

Not even the direction of the effect survived a backend upgrade, which is why the check runs on every sweep instead of being trusted once. One example answered "As needed" on repetition 1 and "Weekly" on repetitions 2 and 3, at temperature 0 with a seed: identical conditions, different outputs.

Predictions print next to measurements

The fit prediction is deliberately simple: weights on disk, plus f16 KV cache for the context, plus a fixed 1 GiB compute allowance. On the 5070, three of four candidates measured within 15% of prediction; the 1B ran over 20% under its intentionally conservative estimate. Every report shows both. Use predictions for planning. Measured peak is your actual constraint.

Peak memory comes from the most honest source each machine has:

nvidia-smi on NVIDIA GPUs
Backend process RSS on Apple Silicon (validated against Ollama's own accounting)
Metal accounting on Rapid-MLX
/proc/meminfo-based RSS on everything else

A machine with no working source reports "not measured." Nothing is ever estimated.

Anyone can recheck the scores

quantproof report --bundle exports a zip with every raw output, all scores, model digests, sampler params, and the backend version. Re-scoring the 5070 sweep from bundle contents alone reproduced 264 of 264 scores exactly, on both Ollama versions. A number you can't recheck is marketing.

The fine print (read before citing any of these numbers)

Memory is sampled at roughly 200 ms. The recorded peak is the highest sample, a floor on the true peak, not an allocator trace.
TTFT is measured from HTTP request to first streamed chunk, including connection overhead and prompt evaluation. The real latency a user experiences, not a pure decode metric.
Scorers are deterministic only: schema conformance, field comparison, label match, pattern presence, numeric tolerance. Tasks that need judgment (summarization, open QA) are out of scope, not approximated with a judge model.
Partial GPU offload is flagged with its reasoning, not measured. Ollama decides offload, not the harness.
A pack drafted by quantproof ingest measures agreement with its drafting model until human review. One run: drafter got 1 of 22 expected values wrong. The provenance label exists because of that.

Run it on your task

Node 22+, Ollama or Rapid-MLX. Out-of-memory failures are recorded as results, not crashes, and the same packs can also sweep Claude models over the Anthropic API when no GPU is around.

moonrunnerkc / quantproof

Run your real task on quantized models and get measured quality, latency, and peak VRAM, plus the smallest quant that holds quality.

Why measure · How it works · Case study · Run it on your task · API runs · Documentation

QuantProof

Point QuantProof at a folder of real examples from your task, and it tells you which quantized model performs best on your hardware.

It measures, not estimates:

Quality
Latency
Peak VRAM usage

Then it recommends the smallest quantized model whose quality is within 2% of the best-performing model.

Why measure

Fit calculators only tell you whether a model's weights will fit in memory Benchmark leaderboards rank models on general tests, not your specific workload.

If your real question is:

"Will Q4_K_M hurt my invoice extraction accuracy?"

there's only one reliable way to answer it: run your invoice extraction on Q4_K_M and measure the results.

How it works

QuantProof automates that process. It:

Tests candidate models one at a time
Scores every output deterministically
Measures latency and peak VRAM
Reports…

View on GitHub

Run your own 20 examples

AI Agents Cheat on Pull Requests. I Mined 327 of Them to Prove It.

Brad Kinnard — Thu, 09 Jul 2026 22:45:30 +0000

AI coding agents cheat. Not maliciously, and usually not on purpose. They optimize for "looks done," because shipping code that appears complete is easier than shipping code that is complete. Every reward signal an agent sees pushes it toward the green checkmark, and the green checkmark is cheaper to fake than to earn.

That was a curiosity when one engineer babysat one agent. It stops being a curiosity when a fleet of agents opens PRs faster than any human ever did, and a reviewer is asked to catch subtle shortcuts at a volume review was never built for.

So I went and measured it. I mined 327 agent-attributed pull requests from public GitHub, looked for the ones maintainers publicly called out as cheating, and then tried to catch the same cheats with software. This post is what I found, including the parts that did not work.

What "cheating" actually means

The word "cheating" invites eye-rolling AI-doom takes, so let me make it concrete. Here are real, recognizable patterns. Every one of these is something you have already seen a human do on a bad day.

Swallowed errors. The failure path is caught and dropped on the floor.

try {
  await syncRemoteState(payload);
} catch (err) {
  // handled
}

Nothing is handled. The test that expected no throw now passes.

Relaxed assertions. A strict matcher becomes a loose one.

- expect(result).toEqual({ id: 7, status: 'settled', total: 4200 });
+ expect(result).toBeTruthy();

{} is truthy. The test is now green for almost any output.

Assertion stripping. The checks that actually pin behavior quietly disappear.

  const rows = await repo.findByOrg(orgId);
- expect(rows).toHaveLength(3);
- expect(rows[0].email).toBe('a@b.co');
+ expect(rows).toBeDefined();

No-op fix. The PR claims to fix a bug. The source is untouched; only the test changed to stop failing.

Fake refactor. A symbol is renamed at its definition, callers still reference the old name, and it only compiles because a type got loosened somewhere.

Type/lint suppression. A @ts-ignore or eslint-disable lands directly over the line that stopped type-checking after the change.

+ // @ts-ignore
  return handler(req as AuthedRequest);

None of these are exotic. That is the point. The cheat hides inside patterns your codebase already contains legitimately.

Why this matters now

Agents do not cheat more per PR than a rushed human does. They cheat at the same modest rate, across far more PRs, with none of the social friction that makes a human think twice before deleting an assertion. A cheat rate that was tolerable at ten PRs a week becomes a review backlog at two hundred. It is a signal-to-noise problem, and the noise floor is rising.

The data, honestly

Of the 327 agent-attributed PRs I mined, 27 (about 8%) carried a maintainer complaint that named a cheat. Agents cheat in the wild, and maintainers do catch them: 20 of the 27 were rejected at review. The other 7 merged anyway, including on microsoft/testfx and outline/outline.

Now the caveat that most AI posts skip. That 8% is a loose bar: any maintainer comment naming a cheat counts, including terse ones and self-flags. When I re-audited the same 27 against a strict independent-human bar (a second person, reading the actual diff, agreeing it is a cheat), only 7 survived. So the honest range is "8% by a generous reading, closer to 2% by a strict one." Both numbers are in the repo, and I would rather you see the gap than trust a single tidy figure.

Why your existing tools miss it

Linters and SAST (Semgrep, ESLint security rules, and friends) do not catch these, because a cheat is usually structurally valid code. An empty catch block is legal. A renamed function is legal. A loosened matcher is legal.

Two of the merged examples above make the point:

microsoft/testfx#8513 deleted a test in C#, a language the structural detectors do not even parse.
outline/outline#12197 removed a single jest.mock line. There is nothing malformed to match. The behavior changed; the syntax did not.

A pattern matcher keyed on "bad syntax" has nothing to grab onto. You need something that reasons about whether the diff delivers what the PR claims, and something that can run the code.

What I built

Swarm Orchestrator is an open-source auditor for exactly this. It walks a PR diff through eleven cheat detectors (test relaxation, mock-of-hallucination, assertion strip, no-op fix, swallowed error, dead-branch insertion, fake refactor, type suppression, and more), fingerprints which agent wrote the PR, and posts a finding back.

The design rule is one sentence: flags are tips, blocks are proof.

Detectors are advisory by default. They flag cheat-shaped patterns and never block a merge on suspicion alone. No detector is allowed to block on its own opinion.
A finding only escalates to a block when the tool can reproduce a runtime proof in a fresh checkout: revert the suspicious hunk, rerun the suite, and confirm the code only "passed" because of the doctored change. The block ships with the exact command to reproduce it.

It runs offline against a committed diff, so nothing here is a number you have to take on faith:

git clone https://github.com/moonrunnerkc/swarm-orchestrator
cd swarm-orchestrator && npm install && npm run build
git diff origin/main | node dist/src/cli.js audit --diff-stdin

The honest limitations

This is the section that should make you trust the rest.

No detector clears the bar to auto-block. Zero of eleven are gate-eligible. Every one is advisory.
The proof gate proved zero of the 27 wild cheats on its own. The kind of cheat a machine can prove without a human (a merged test-tamper whose restoration provably fails) simply did not occur in the sample it could execute. Cheats that need judgment to see stayed invisible to the gate, by design, because the gate refuses to fire on an accusation.
There is a known false-positive class. Refactors that relocate test coverage (moving a check into a golden-file test the engine cannot see) once tripped a false proof. It is pinned by a regression test now, but it is a real limit, not a solved problem.
The advisory tier is where the daily value lives: across the hunt it flagged 148 candidates and corroborated all 27 human-caught cheats. On a defect-injection corpus it recovers 301 of 325 planted cheats (92.6%). Corroboration with a known ceiling, not a replacement for review.

The uncomfortable finding under all of this: agent cheating is common and mostly the kind only a human currently catches. Automation can raise the signal and prove the rare merged case. It cannot yet stand in for the reviewer.

If you want to poke holes in it

Run it against your own diffs. Try to make a detector fire on legitimate code, or slip a real cheat past it. Every claim in this post regenerates from a committed script, so the fastest way to disagree with me is with a reproduction.

Repo: https://github.com/moonrunnerkc/swarm-orchestrator

It is an open problem, not a finished product.

Catching the shortcuts AI coding agents take to look done

Brad Kinnard — Fri, 05 Jun 2026 21:40:35 +0000

A green test suite is supposed to mean the change works. It doesn't. A test can be weakened just enough to pass. An error can be caught and thrown away. A rename can stop halfway and still compile. None of that turns red, and none of it shows up in the linters most teams already run.

Swarm Orchestrator is built to catch exactly that class of problem in AI-written pull requests.

Two parts. One audits AI-written PRs for the shortcuts that fake "done" (11 checks). The other gates a patch against a contract you define: it builds, passes tests, satisfies your requirement, and survives a falsifier that tries to break it.

TypeScript, Node 20, ISC license. The audit side runs with no model credentials.

The gap linters leave

Semgrep and ESLint are built around risky APIs and known-bad code patterns. Whether a diff is honest is a different question. They won't tell you a test was edited until it passed, or that a catch block quietly eats the error it caught. That's the gap.

Two examples from merged Cloudflare pull requests:

PR	Finding	Semgrep + ESLint
`workers-sdk#14063`	Function renamed, some callers still using the old name	No finding
`workers-sdk#14132`	Empty catch block hiding errors	No finding

Across 72 known-bad pull requests from 12 repositories, that pair of analyzers produced one finding. The auditor flagged 67.

What the auditor checks

Eleven checks total. Eight run by default. The other three exist but stay off, because they haven't shown useful signal on real pull requests yet, and a noisy check is worse than no check.

The default set looks for things like:

Errors caught and ignored
Renames left unfinished
Test coverage reduced
Tests weakened
Assertions removed
New @ts-ignore or eslint-disable comments
Test-only fixes with no code change behind them
Mocks pointing at modules that don't exist

Measured, not assumed

The detection rate isn't a guess. Known defects get injected into real pull requests, then the auditor runs against them. It caught 253 of 300, or 84 percent.

Reproduce it:

npm run benchmarks:full

Runtime mode (optional)

The checks can also execute code instead of only reading a diff: mutation testing, coverage, and reproducing reported issues.

On trpc#6098 it found mutations surviving on lines a later hotfix changed. The tests passed. They weren't actually exercising that code.

Why this mode stays optional
Running code is louder than reading a diff: it averages about 3.4 findings on a clean pull request. That noise is fine when you're deliberately hunting, but it's too much to leave on by default, so it's opt-in.

Defining "done" with a contract

The second command is swarm run. You write down what done means:

obligations:
  - type: build-must-pass
    command: npm run build
  - type: test-must-pass
    command: npm test

A patch is accepted only if every obligation passes and the falsifier can't break it. The default provider is deterministic, so identical inputs give identical results, and every input and hash gets written to a hash-chained ledger.

Blocking merges

Findings are advisory out of the box. Gate mode can block a merge, but only on reproducible evidence. The structural checks throw too many false positives to trust as automatic blockers on their own.

Right now no runtime signal has enough real-world evidence to justify auto-rejection, so the gate stays open and reports that fact directly instead of pretending otherwise.

Who it's for

If you review a lot of AI-written pull requests and want signals the usual linters skip, that's the case this is built for. It also emits CycloneDX-ML and SPDX AI BOM documents with --emit-aibom, supports TypeScript and JavaScript, and runs offline.

It points reviewers at the code worth inspecting. It doesn't claim to prove anything bug-free.

View the repo on GitHub

moonrunnerkc / swarm-orchestrator

Reviews pull requests for the shortcuts AI coding agents take to look done without being done: relaxed tests, swallowed errors, fake renames, 11 checks in all. Flags them for a human by default, or blocks the merge if you turn that on. Can also turn a goal into a checklist and only accept a patch once every check passes.

Swarm Orchestrator

A CLI for auditing AI-generated PRs and grading patches against typed contracts.

= 20">

Install · Quick start · What it does · Results · Detectors · AI-BOM · Reference

What This Does

Swarm Orchestrator reads a pull-request diff and flags the shortcuts an AI coding agent takes to look done without being done: relaxed tests, stripped assertions, swallowed errors, fake renames, eleven checks in all. On a benchmark of planted cheats it recovers 253 of 300 (84%, up 20.5% from the prior version), and on real merged Cloudflare PRs it caught two cheats that Semgrep and the ESLint security rules missed, both reproducible offline. Findings are advisory by default, so it never blocks a merge unless you turn that on.

Who it's for

You review AI-written PRs at volume and want a "this change may be gaming the tests" signal that ordinary linters do not give you.
You have…

View on GitHub

AI wrote the PR. How do you know it actually works?

Brad Kinnard — Wed, 03 Jun 2026 01:26:44 +0000

AI agents open a lot of pull requests now. Most are fine. Some quietly cheat to make the checks go green: they delete the failing test, weaken an assertion, wrap the broken call in an empty catch so the error disappears. The diff looks done. A reviewer skimming forty agent PRs a day will not catch that by eye.

swarm audit is a command-line tool that does. I maintain it. It runs three jobs on AI-written code, all offline, no API key.

1. Catch the cheat

Eleven checks read a pull-request diff and flag the shortcut patterns: a deleted test with no matching code change, a function renamed while its callers still use the old name, an error swallowed by an empty catch, a mock of a package that exists in no manifest, a type-checker suppression dropped over a changed line, and more.

The detection is measured, not asserted. Hide one known cheat in each of 300 real merged PRs, run the auditor, count the catches: 254, about 85%, reproducible with one command.

The catch that matters most is on real code. On two merged Cloudflare PRs it flagged a rename that left two callers pointing at a dead function, and an empty catch {} that throws every error away. Semgrep (210 rules) and ESLint's security rules flagged neither, because they hunt for dangerous code like an injection or a leaked secret, and a deleted test is not dangerous code, it is missing code. The auditor also names the agent that wrote the PR: on a live fetch it tagged the author as Devin. Findings ship advisory, so it reports and never blocks your merge unless you ask it to.

2. Prove the change meets its spec

The second mode runs before a change is accepted. You hand it a plain goal. It compiles that goal into a contract of machine-checkable obligations: build passes, tests pass, coverage holds, a named function has the right signature, a property holds, performance does not regress. Candidate patches get generated, and one is admitted only if it satisfies every obligation. Adversarial falsifiers actively try to break a patch before it counts.

In a fresh project it compiled a goal into two obligations, verified both, confirmed nothing regressed after the merge, and spent zero tokens doing it. Turn on gate mode and it becomes a hard merge rule: a diff that strips a test exits non-zero and never lands.

3. Document it for compliance

If you ship or buy AI-written code under the EU AI Act or CISA's SBOM-for-AI guidance, someone will ask for a record of the AI involvement. The tool emits one: a CycloneDX 1.6 ML bill-of-materials and an SPDX 3.0 AI-Profile, both valid against their specs, plus a hash-chained evidence ledger where altering any entry breaks the chain. It ships with the mappings to EU AI Act Annex IV and the CISA minimum elements.

Try it

git clone https://github.com/moonrunnerkc/swarm-orchestrator
cd swarm-orchestrator && npm install && npm run build

npm run benchmarks:oracle   # the ~85% number
node dist/src/cli.js audit --diff-file benchmarks/real-prs/diffs/cloudflare-workers-sdk/14132.diff --detectors all
swarm init && swarm run --goal "verify this project builds and tests pass"

Open Source Repo: https://github.com/moonrunnerkc/swarm-orchestrator

Your AI's tests pass. That doesn't mean the code works.

Brad Kinnard — Sun, 31 May 2026 22:21:05 +0000

You ask a coding agent to fix a bug. It writes the code, writes the tests, CI goes green, you merge. The bug's still there.

The agent's job was to turn the check green. The honest way to do that is to fix the code. The lazy way is to write a test that passes no matter what the code does. CI can't tell those two apart. A green check means the tests passed, not that the code is right.

It's easy to miss in review, because the test sits right there looking like proof:

test("parses the config", () => {
  const result = parseConfig(rawInput);
  expect(result).toBeDefined();
});

That passes whether parseConfig works perfectly or returns nothing useful on every input. It checks nothing. Adding more tests like it just raises your coverage number, not your odds of catching a bad change.

So I built ClaimCheck (https://github.com/moonrunnerkc/claimcheck). Instead of trusting the agent's tests, it tries to break them. If a test still passes after the supposedly fixed code is broken on purpose, the test was never really checking the fix, and it gets blocked. Same answer every time, no AI making the call. So far it's caught every cheat in a set of twelve hand-built cases. Twelve is small, and there's no public release yet, so treat that as a direction, not a finished result.

Some cheats slip through anyway. If the agent writes a real, solid test that locks in the wrong answer, every check passes. The only way to know the answer's wrong is to already know the right one, and nothing in the pull request can tell you that except the agent you're trying to catch. The one thing that helps is a clue from outside it, like a human-written bug report you can run the fix against.

There's a second, wider tool, Swarm Orchestrator (https://github.com/moonrunnerkc/swarm-orchestrator). It flags suspicious changes and keeps a tamper-evident record for audits. The record-keeping is the solid part. The catching is not: on real pull requests its accuracy is still low, and that's the half I'm hardening now.

The next step is comparing the old code's behavior to the new directly. The catch is that a wrong change and a harmless cleanup can look the same from the outside, and a tool that blocks good code is worse than one that lets a bad change through. That's the part I'm still working out.

Audit AI-Generated PRs Before You Merge Them (Swarm Orchestrator 10.3.0)

Brad Kinnard — Sun, 24 May 2026 20:54:59 +0000

If you let Claude Code, Cursor, Devin, Aider, Copilot, or any other coding agent open PRs against your repo, you already know the problem. The diff looks fine on a fast read. CI is green. You merge it. A week later you find the test that "passed" got deleted, or the error handling is a silent catch {}, or the "fix" was a comment swap that never touched the bug.

Swarm Orchestrator looks at those PRs and flags the suspicious bits before you click merge.

What it is

A CLI and a GitHub Action. Open source. Node 20 or later. You point it at a PR (or a local diff) and it scores the patch against a set of cheat-pattern detectors. It posts a comment back to the PR with what it found and why.

swarm audit moonrunnerkc/swarm-orchestrator#42

That's the whole interface for most people.

What it does

The default detector set has four checks, all aimed at patterns AI agents actually produce on real PRs:

error-swallow: a new empty or comment-only catch block in non-test code.
mock-of-hallucination: a jest.mock or vi.mock against a module that doesn't exist anywhere in the repo.
no-op-fix: tests changed without source, or source changed without tests, when the diff claims to fix something.
fake-refactor: an exported symbol renamed in source, with no caller in the diff updated.

Six more detectors live behind --detectors experimental for shadow runs. They're not scored well enough on real PRs to be on by default, and the README says so.

Every finding renders with its measured precision number inline, so a reviewer sees the false-positive rate every time the bot speaks.

If you need compliance artifacts, --emit-aibom cyclonedx-ml writes a CycloneDX 1.6 ML-BOM and an SPDX 3.0 AI-Profile per audit. That covers the EU AI Act Annex IV and CISA SBOM-for-AI minimums without bolting on a separate vendor.

Who it's for

Teams that let AI agents open PRs and want a second pair of eyes that runs in CI, costs nothing per call, and produces a deterministic comment instead of vibes. Also useful for procurement and security folks who need an AI-BOM next to their SBOM and don't want another tool in the chain.

If you have one developer eyeballing every line of every AI PR by hand, you probably don't need this yet. If you have ten agents pushing diffs to a queue at 2am, you do.

What's new in 10.3.0

Four things:

no-op-fix got a v2.0 with a gated LLM judge. The judge is off by default and only fires when you set --enable-llm-judge (or SWARM_AUDIT_LLM_JUDGE=1) and have an Anthropic key. Verdicts are content-addressed and cached, so the same diff and title always gets the same answer. The model id is pinned in the ledger so replay stays deterministic.
--shadow-output <path>. One JSON file per audit with detector verdicts, judge call count, and the rendered comment. Drops into a directory you can jq later. The existing --shadow <repo> per-repo rollup still works.
Public leaderboard on GitHub Pages. Fetches the real-corpus score snapshot and renders precision, recall, F1, and a sortable per-detector table. No build step, no CDN, just an HTML page and one JS file: moonrunnerkc.github.io/swarm-orchestrator/docs/leaderboard/.
Real-corpus headline rescored against the v2.0 detectors. F1 moved from 0.109 (P 0.067, R 0.300) to 0.167 (P 0.100, R 0.500). mock-of-hallucination picked up two true positives the v1 shape missed.

The honest part

The real-corpus F1 is 0.167 across 205 AI-labeled PRs (10 broken, 195 clean, eight agent vendors). Precision is 0.100. Recall is 0.500.

That precision number is exactly why the default mode is advise and not gate. Most flags will be false positives. The tool is calibrated to be useful as a reviewer-assist signal, not a merge blocker. If you want it to block, opt in: --mode gate.

The 205-PR corpus is currently labeled by an AI judge with "pending human review" stamped on every entry. That's the largest credibility hole in the project and the next milestone closes it. The labeling rubric, the kappa script, and the labels-v2 scaffold already live in the repo.

Don't read this as "ship this into your release gate today." Read it as "here's a tool you can run in shadow mode, look at what it flags, and decide for yourself if those flags are useful."

Try it

git clone https://github.com/moonrunnerkc/swarm-orchestrator.git
cd swarm-orchestrator
npm install
npm run build
npm link

# audit a PR (advisory, never blocks)
GITHUB_TOKEN=... swarm audit owner/repo#PR

Or wire it into a workflow with uses: moonrunnerkc/swarm-orchestrator@main and audit-mode: true.

Sources:

Repo: https://github.com/moonrunnerkc/swarm-orchestrator
Leaderboard: https://moonrunnerkc.github.io/swarm-orchestrator/docs/leaderboard/
Real-corpus score snapshot: https://github.com/moonrunnerkc/swarm-orchestrator/blob/main/benchmarks/real-corpus/scores/latest.json
CycloneDX 1.6 ML-BOM spec: https://cyclonedx.org/specification/overview/
SPDX 3.0 AI Profile: https://spdx.dev/use/specifications/
EU AI Act Annex IV: https://artificialintelligenceact.eu/annex/4/

Cryptographic Forensics for AI Coding Agent Sessions

Brad Kinnard — Wed, 20 May 2026 14:58:25 +0000

A Claude Code or Codex CLI session writes a JSONL file to disk. If the agent runs rm -rf on a training-data directory or terraform destroy -auto-approve on production, that file is where an incident review starts.

A JSONL file is not evidence. Anyone with shell access can rewrite it. To a third party who doesn't trust the machine it came from, it proves nothing.

That gap matters once agents have credentials to real infrastructure. Most agent observability tooling is built for debugging and quality, not for the moment after damage is done. This post is about the three cryptographic properties that turn a transcript into something an auditor or regulator can verify, and how the DEPOSE project wires them together.

Three properties

Assume the machine that produced the bundle can't be trusted. Three things need to hold at once:

Tamper-evident. Any byte change has to be detectable. Hash chain over events: change a byte, replay fails.
Authenticated. The record has to be bound to a key the producer controls and publishes a fingerprint for. Ed25519 signatures over a manifest.
Anti-backdated. A party other than the producer has to anchor the record in time. RFC 3161 tokens from a public TSA.

The primitives are old and well understood. The hard part is wiring them through a normalized event schema and shipping a verifier that doesn't depend on the producer's runtime.

No LLM in the signed path

Every event is captured at execution time or normalized from the session JSONL, then committed to the hash chain. The human-readable narrative is generated separately, from deterministic Handlebars templates over the signed events. It's excluded from the root hash.

If generated prose became part of the signed record, verification would depend on model behavior staying stable and reproducible. DEPOSE avoids that dependency. The signed record is event data and hashes. The prose is templated commentary with [#evt-<ulid>] citations back to the signed events. You can rewrite the narrative without affecting verification. Change an event and verification fails.

What's in a bundle

A DEPOSE bundle is a directory, not an opaque archive:

incident-01JABC.../
├── manifest.json            bundleId, rootHash, eventsJsonlSha256, sigs, timestamps
├── events.jsonl             every event in canonical JSON, byte-pinned by manifest
├── rules/destructive.yaml   ruleset used at reconstruction time
├── narrative.md / .html     templated prose with per-event citations
├── verify.txt               human-readable verification summary
├── artifacts/               captured file diffs, payloads
├── attestations/            Ed25519 signatures, RFC 3161 timestamp tokens
└── raw/                     source JSONL, shell history, capture records

Change a byte of events.jsonl, manifest.json, or rules/destructive.yaml and verification fails. Canonical JSON follows RFC 8785 (JCS), which is what lets a Go verifier check a TypeScript-produced bundle without either side trusting the other's serializer.

Two binaries

The producer is TypeScript. The verifier is a separately-built static Go binary, depose-verify. The separation is deliberate: you hand the binary to whoever needs to check the bundle (auditor, opposing counsel, regulator, a customer's security team) and they run it on their own machine. No producer stack required.

A passing run prints:

parse        OK
signature    OK
chain-replay OK
artifacts    OK
timestamp    OK
PASS  bundleId=...  rootHash=...

The cryptography here is mostly off-the-shelf. The actual engineering work is in normalization: getting Go and Node to serialize identically, getting timing and ordering right across capture sources, deciding what counts as one event versus two. Canonical JSON is the unsexy part. Float formatting, key ordering, unicode escapes: Go and Node have to agree byte-for-byte or the verifier rejects a bundle the producer thinks is fine. That's what the cross-language conformance vectors in tests/conformance/ are for.

Verifiers can pin a producer's expected key fingerprint and consult a revocation list, both at the command line. The RFC 3161 timestamp does double duty here: a bundle stamped before a key is revoked stays time-anchored, so "when was this signed" remains answerable even if the key is later compromised.

Capture modes

Two modes, different coverage.

Reconstruction reads the Claude Code session JSONL after the fact, compares it against shell history (bash, zsh, fish) and git reflog where available, and builds a bundle. Lower-bound mode. It can verify integrity after packaging. It can't prove the original session file was complete before capture.

Active capture installs a Claude Code PreToolUse hook and POSIX shell shims for the binaries that tend to do destructive things: terraform, aws, gh, kubectl, psql, gcloud, railway, rm. Records land under ~/.depose/captures/ at execution time. A later depose package merges them with the session JSONL so every covered event has a verified pre-execution intent on record.

DEPOSE can prove integrity of captured events. It can't prove an uninstrumented system captured everything. An agent that shells out to a binary not in the shim list, or hits an API directly, still shows up in the JSONL but won't have an active-capture record. The coverage matrix is in the repo.

macOS and Linux only. Windows isn't supported (POSIX 0600 on the key store, POSIX shell scripts for the shims). WSL2 works.

Release pipeline

Releases ship with SBOMs, provenance attestations, and signed checksums. The specifics: CycloneDX for both halves, SLSA L3 provenance, and SHA256SUMS signed via cosign keyless. CI rebuilds the two checked-in example bundles (an rm -rf on training data, a terraform destroy on infrastructure) on every push and runs three semantic tamper rejections to confirm the verifier fails closed.

Right now most coding-agent session logs are treated like disposable debug output. That assumption gets weaker the moment an agent can modify infrastructure.

https://github.com/Aftermath-Technologies-Ltd/depose

Gemma.Witness - Offline Multimodal Evidence Capture with Gemma 4

Brad Kinnard — Sun, 17 May 2026 03:43:46 +0000

This is a submission for the Gemma 4 Challenge: Build with Gemma 4

What I Built

Gemma.Witness is an offline-first multimodal evidence capture system built for environments where cloud access, trust, or chain-of-custody assumptions fail.

The system records audio alongside supporting images, runs local multimodal analysis through Gemma 4, and produces a signed evidence bundle containing:

Structured incident reports
Timestamped evidence metadata
Local reasoning traces
Hash-linked verification artifacts
Exportable forensic bundles

The focus was reliability and local verification instead of "AI assistant" behavior.

Most evidence tooling today assumes internet access, centralized APIs, or mutable storage. Gemma.Witness was designed around the opposite assumption: the network may be unavailable, the machine may be isolated, and every generated output may eventually need independent verification.

The application runs fully local through a desktop interface using Rust, Tauri, and local inference orchestration.

Demo

github: https://github.com/moonrunnerkc/gemma-witness

Code

Source code is available at the repository above.

How I Used Gemma 4

Gemma 4 is the reasoning layer behind the entire evidence pipeline.

I used Gemma 4's multimodal capabilities to process:

Audio-derived transcripts
Scene images
Cross-evidence consistency analysis
Structured incident extraction
Reasoning trace generation

The model is used in a multi-pass workflow instead of a single prompt-response cycle. Each pass validates or expands on the previous stage before the final signed bundle is emitted.

This matters because evidence systems fail quietly when models hallucinate details, merge assumptions into facts, or overstate certainty. The pipeline was intentionally designed to separate:

Raw observations
Inferred conclusions
Confidence scoring
Verifiable artifacts

Gemma 4 was a strong fit because it could operate locally while still handling multimodal reasoning tasks without requiring cloud APIs or external orchestration services.

The project prioritizes:

Offline operation
Verifiable outputs
Local ownership of evidence
Minimal trust assumptions
Reproducible forensic artifacts

A surprising challenge was not getting the model to generate reports. That part was easy.

The difficult part was building guardrails around evidence integrity so the system does not quietly become a very confident fiction generator wearing a necktie.

Tech Stack

Gemma 4
Rust
Tauri
Node.js
Local multimodal inference
Cryptographic hashing and bundle verification

Repository

moonrunnerkc / gemma-witness

Offline multimodal evidence capture that emits a signed, locally verifiable .witness bundle. Tauri + Rust + Gemma 4 + Ed25519. Static HTML verifier runs with no server.

Gemma.Witness

Offline, tamper-evident evidence capture for field journalism. Signed in your hand, verified in a browser, with no server in the loop

demo.mov

Why · Status · Install · Usage · Configuration · Threat model · Limitations · Verify yourself · Contributing

Why this matters

A reporter is working in a country where journalists are detained for their reporting. She records a witness account. She attaches the photos she just took. She seals the file before she leaves the room.

A week later, an editor on another continent opens a single static HTML page in any browser and drags the file in. Three checks turn green:

the signature comes from the reporter's device
the audio and the photos have not been altered by a single byte
the AI model in the chain is bit-for-bit the published Gemma model her manifest names, by model_id, revision, and model.safetensors SHA-256

In…

View on GitHub

Swarm Orchestrator v8.0.2

Brad Kinnard — Tue, 12 May 2026 02:31:18 +0000

v8.0.2 is out now and it cleans up several rough edges that kept showing up under heavy tournament and falsification workloads.

The biggest operational change is that all four previously documented architectural limitations are now closed in the same release (7b68867).

Notable Changes

Tournament mode now streams through the same pipeline as single mode. If one candidate fails streaming verification, it gets aborted independently instead of poisoning the whole run.
Live cost-cap enforcement is now real-time. Concurrent streams project cumulative USD usage continuously and abort the moment projected spend crosses the configured cap.
Snapshot cleanup is automatic now and supports retention policies like:

  retain-last:N
  max-age:<dur>
  max-disk:<sz>

Adaptive falsifier dispatch using UCB1 is available behind:

  --falsifier-scheduler ucb1

ARIES-style rollback support landed for falsified obligations. If a counter-example appears after apply, the workspace restores from the pre-apply snapshot and verifies the rollback by hashing the restored bytes against the original SHA.

New Command

There is also a new command:

swarm v8 stats <run-id>

That surfaces persisted falsifier metrics directly from:

.swarm/falsifier-stats.json

including regression discoveries, false positives, success counts, and latency.

Replay Determinism

One important detail: replay determinism remains intact across all of this. Every scheduler decision and abort event still lands in the ledger so replay reproduces the same winner consistently. That part was non-negotiable.

Release

https://github.com/moonrunnerkc/swarm-orchestrator/

How Swarm Orchestrator v8 Tries to Break Its Own AI Patches

Brad Kinnard — Sun, 10 May 2026 02:10:05 +0000

Most AI coding tools commit when their own checks pass. Swarm Orchestrator v8 adds a second adversarial layer: independent falsifier adapters that try to break each patch before it merges. v8.0.1 is on main with that subsystem on by default.

This post walks through the v8 architecture, the four verification points, the producer/falsifier adapter split, and the limitations that haven't been solved in v8.0 yet.

What is Swarm Orchestrator? A contract-first AI coding swarm with hash-chained evidence and verifier-gated commits. It compiles a natural-language goal into a typed contract, dispatches it to a population of personas inside one cached Anthropic session, races candidate diffs per obligation, and commits only what passes verification. It wraps an LLM; it doesn't replace one.

The shape of a run

You hand it a goal in plain English. The contract compiler turns that into contract.jsonl plus a manifest.json carrying the goal, repo context, extractor provenance, and a SHA-256 of the canonical contract bytes. Identical inputs produce identical contract hashes.

goal (text)
   |
   v
contract compiler  ->  contract.jsonl + manifest.json
   |
   v
+-------------------------------------------------+
|        population manager (single session)      |
|                                                 |
|  ledger (jsonl, hash-chain) <- personas (8)     |
|       ^                          |              |
|       | tournament + verifier scoring           |
|       |                                         |
|  WASM deterministic floor (zero-LLM obligs)     |
+-------------------------------------------------+
   |                              |
   v                              v
streaming verifier      post-merge integration
   |                              |
   +--------------+---------------+
                  v
       falsifier adapters (Codex, Copilot)
                  |
                  v
            committed diffs

The population manager opens one cached Anthropic session and walks each obligation. It picks the persona whose trigger predicate matches the obligation type. In tournament mode, N candidates run in parallel; the verifier scores them, the top scorer is a commit candidate, and losers get logged but never merge.

Two adapter subsystems

The most common confusion in v6 was treating the coding CLIs and the falsifiers as one thing. v8 splits them cleanly.

Producer adapters (src/adapters/) wrap third-party coding CLIs as the worker in the v6 verified-branch pipeline. Backends: Copilot, Claude Code, Codex, Claude Code Teams. All four are opt-in via swarm run --v6.

Falsifier adapters (src/falsification/adapters/) take a patch the producer's verifier already accepted and try to falsify the obligation by surfacing a counter-example, regression fixture, or property-violation trace. A confirmed counter-example flips the obligation back to failed.

Falsifier	Default	Obligation types
`CodexFalsifier`	on	`property-must-hold`
`CopilotFalsifier`	on	`import-graph-must-satisfy`, `function-must-have-signature`
`ClaudeCodeFalsifier`	off (per-adapter opt-in)	all three

The CLI surface is one flag: --falsifiers <on|off> (default on). Per-adapter selection happens at the API layer via defaultAdapterRegistry({ includeCopilot, includeClaudeCode }).

Four verification points

A patch has to survive these before it merges:

Pre-generation memoization. Skip generation if the obligation result is already cached.
Mid-stream abort. During generation, the streaming verifier can abort the call. Works in --mode single only; tournament mode skips it.
Post-generation per-obligation verifier. Scores the candidate diff. In tournament mode, top scorer wins; in single mode it's pass/fail.
Post-merge integration check. After the diff lands, the integration check confirms the broader system still holds.

The architectural rule from the README: nothing commits without passing the obligation's verifier. Then the falsifiers get a shot.

The hash-chained ledger

Every action lands in .swarm/ledger/<run-id>.jsonl with the SHA of the prior entry. Tampering is detectable; runs resume from any prior state. If a process is killed mid-run, swarm v8 resume <run-id> walks the ledger and picks up where it left off.

The ledger format is shared with v6, but v8 writes more granular events (per-persona dispatch, per-candidate score, falsifier verdict) so a run can be replayed or audited end-to-end.

Quick start

git clone https://github.com/moonrunnerkc/swarm-orchestrator.git
cd swarm-orchestrator && npm install && npm run build && npm link

# Compile a goal, then run it
swarm v8 compile "add a /health endpoint that returns 200 OK" --yes
swarm v8 run .swarm/contracts/<contract-id>

# Or both in one step (defaults to v8)
swarm run --goal "add a /health endpoint that returns 200 OK"

# Resume a killed run
swarm v8 resume <run-id>

Requires Node >= 20, git >= 2.40, and ANTHROPIC_API_KEY. Pass --extractor stub --session stub to run offline.

There's also a GitHub Action:

- uses: moonrunnerkc/swarm-orchestrator@v8
  with:
    goal: "add a /health endpoint"
    contract-only: false
    cost-cap: "5.00"
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

What v8.0 doesn't do

Limitations worth reading before adopting

Tournament mode doesn't stream. --mode tournament plus --forbid-import skips the streaming abort; streaming verification is --mode single only.
Post-merge failure doesn't auto-rollback. The run is marked failed; per-obligation worktree snapshots are post-v8.0.
--cost-cap is enforced post-obligation, not mid-call. Cumulative spend is checked at the end of each obligation against estimated Sonnet 4 pricing. Exit code 6 if exceeded.
Bandit dispatch is not built (Phase 5). Codex and Copilot have disjoint obligation types, so there's nothing to arbitrate between yet.
Cross-vendor producer race is deferred (Phase 6).

Full list with rationale lives in docs/v8-architecture-deviations.md.

Repo

moonrunnerkc / swarm-orchestrator

Contract-first AI coding swarm with hash-chained evidence. Compiles a goal into typed obligations, races persona candidates per obligation in a single cached inference session, verifies before commit, and logs every action in an append-only ledger.

Swarm Orchestrator

Contract-first AI coding swarm with hash-chained evidence and verifier-gated commits.

swarm compiles a natural-language goal into a typed contract, dispatches it to a population of personas inside one cached Anthropic session, races candidate diffs per obligation, and commits only the diffs that pass verification. After the producer's verifier accepts a patch, registered falsifier adapters get a chance to break it before it merges. Every action lands in an append-only hash-chained ledger you can audit, resume, or replay.

It wraps an LLM; it does not replace one. The model writes the code, the orchestrator decides what reaches your repo.

Status

Version 8.0.1 on main. Node >= 20 (CI matrix: 20, 22). License ISC. The v8 architecture is the default for swarm run; the v6 verified-branch pipeline is preserved under swarm run --v6 and the swarm swarm / swarm execute commands Falsifier subsystem: Codex on, Copilot on, ClaudeCode…

View on GitHub