Our AI verification job passed for months. It was checking the wrong branch.

Paul Redman — Fri, 23 Oct 2026 07:35:19 +0000

I run a nightly pipeline of more than 20 AI agents: research, architecture, security audit, build, QA. Every morning there's a report of what passed.

For months, everything passed.

Then I found out why. The final verification job was checking out the wrong branch. It re-tested a tree that had already passed, so it could never fail. Nothing crashed. Nothing errored. It just quietly approved the wrong thing, every night.

The failure nobody alerts on

Monitoring is built for things that break. A job times out, a test goes red, a deploy falls over, and someone gets paged.

This was different. The job ran, finished and reported green. Its only problem was that the green tick was about different code from the code we were shipping.

Agents make this kind of failure much more common:

An agent reviews a file and says "pass". Ten minutes later the file changes. The pass is still sitting there.
An agent says it checked the code against the spec. Which version of the spec?
An agent never produces a verdict at all, and nothing notices the gap.

In every case the verdict looks fine. It's just about the wrong bytes.

Bind the verdict to what was read

The fix I ended up with is boring on purpose. When an agent gives a verdict, it records the SHA-256 of every file it based that verdict on:

{
  "schema": "slipmark/attestation@1",
  "by": "security-audit",
  "verdict": "cleared",
  "at": "2026-10-17T09:00:00.000Z",
  "subjects": [
    { "path": "src/auth/session.ts", "sha256": "9f3c…" }
  ]
}

Before anything merges, a gate re-hashes those files and compares:

Slipmark verify: 1 verified · 1 refused

  ✓ verified        review-agent verified examples/attestation/brief.md
  ✗ hash_mismatch   security-audit-agent gave "cleared" for a different version of
                    examples/attestation/spec.md. The file changed after the verdict.

If a file changed after the verdict, the verdict stops counting. If a file you require was never attested, the gate refuses. If the agent's own verdict wasn't a pass, it refuses. An empty gate is a refusal, never a silent pass.

No model calls. Nothing to tune. Same inputs, same answer, every time.

Check content, not names

This is the part that would have caught the original bug. The gate checks content, not names. Branch names, job names and timestamps can all be wrong without anyone noticing. A hash can't. Run the check on the exact tree you're about to merge, and a verdict produced anywhere else simply won't match.

When I pointed it at our 23 open agent pull requests, it refused all 23.

That was a bad morning. It was also the most useful thing the pipeline had told me in months.

What it doesn't do

It doesn't judge whether the agent was right. A bad review of the right file still passes. All it proves is that the verdict is about the code you're actually shipping. You still need good reviewers, human or otherwise.

It's also strict. Run a formatter after the review and the verdict stops counting, the same as any other change. For now, run formatters before the agent attests (first in your pre-commit hooks) and most of that goes away. Formatting-tolerant hashing is next, built on one rule: a false mismatch is annoying, but a false match would make the whole thing worthless.

Try it

I've open-sourced it as Slipmark, under Apache-2.0.

npx slipmark init

That adds an attestation folder, a GitHub workflow that starts in report-only mode, and a short block in AGENTS.md (and CLAUDE.md, Cursor rules or Copilot instructions if you have them) telling your coding agents how to attest. There's also a Claude Code plugin that asks Claude to re-check a file it signed off when that file changes.

As a GitHub Action:

- uses: actions/checkout@v4
- uses: founder-ship-it/Slipmark@v0.4.0
  with:
    require: |
      specs/your-spec.md

The gate is free and needs no account. There's an optional hosted ledger if you need the record as well as the build result.

Code and docs: github.com/founder-ship-it/Slipmark