DEV Community

Cover image for Our AI verification job passed for months. It was checking the wrong branch.
Paul Redman
Paul Redman

Posted on AI-assisted

Our AI verification job passed for months. It was checking the wrong branch.

I run a nightly pipeline of more than 20 AI agents: research, architecture, security audit, build, QA. Every morning there's a report of what passed.

For months, everything passed.

Then I found out why. The final verification job was checking out the wrong branch. It re-tested a tree that had already passed, so it could never fail. Nothing crashed. Nothing errored. It just quietly approved the wrong thing, every night.

The failure nobody alerts on

Monitoring is built for things that break. A job times out, a test goes red, a deploy falls over, and someone gets paged.

This was different. The job ran, finished and reported green. Its only problem was that the green tick was about different code from the code we were shipping.

Agents make this kind of failure much more common:

  • An agent reviews a file and says "pass". Ten minutes later the file changes. The pass is still sitting there.
  • An agent says it checked the code against the spec. Which version of the spec?
  • An agent never produces a verdict at all, and nothing notices the gap.

In every case the verdict looks fine. It's just about the wrong bytes.

Bind the verdict to what was read

The fix I ended up with is boring on purpose. When an agent gives a verdict, it records the SHA-256 of every file it based that verdict on:

{
  "schema": "slipmark/attestation@1",
  "by": "security-audit",
  "verdict": "cleared",
  "at": "2026-10-17T09:00:00.000Z",
  "subjects": [
    { "path": "src/auth/session.ts", "sha256": "9f3c…" }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Before anything merges, a gate re-hashes those files and compares:

Slipmark verify: 1 verified · 1 refused

  ✓ verified        review-agent verified examples/attestation/brief.md
  ✗ hash_mismatch   security-audit-agent gave "cleared" for a different version of
                    examples/attestation/spec.md. The file changed after the verdict.
Enter fullscreen mode Exit fullscreen mode

If a file changed after the verdict, the verdict stops counting. If a file you require was never attested, the gate refuses. If the agent's own verdict wasn't a pass, it refuses. An empty gate is a refusal, never a silent pass.

No model calls. Nothing to tune. Same inputs, same answer, every time.

Check content, not names

This is the part that would have caught the original bug. The gate checks content, not names. Branch names, job names and timestamps can all be wrong without anyone noticing. A hash can't. Run the check on the exact tree you're about to merge, and a verdict produced anywhere else simply won't match.

When I pointed it at our 23 open agent pull requests, it refused all 23.

That was a bad morning. It was also the most useful thing the pipeline had told me in months.

What it doesn't do

It doesn't judge whether the agent was right. A bad review of the right file still passes. All it proves is that the verdict is about the code you're actually shipping. You still need good reviewers, human or otherwise.

It's also strict. Run a formatter after the review and the verdict stops counting, the same as any other change. For now, run formatters before the agent attests (first in your pre-commit hooks) and most of that goes away. Formatting-tolerant hashing is next, built on one rule: a false mismatch is annoying, but a false match would make the whole thing worthless.

Try it

I've open-sourced it as Slipmark, under Apache-2.0.

npx slipmark init
Enter fullscreen mode Exit fullscreen mode

That adds an attestation folder, a GitHub workflow that starts in report-only mode, and a short block in AGENTS.md (and CLAUDE.md, Cursor rules or Copilot instructions if you have them) telling your coding agents how to attest. There's also a Claude Code plugin that asks Claude to re-check a file it signed off when that file changes.

As a GitHub Action:

- uses: actions/checkout@v4
- uses: founder-ship-it/Slipmark@v0.4.0
  with:
    require: |
      specs/your-spec.md
Enter fullscreen mode Exit fullscreen mode

The gate is free and needs no account. There's an optional hosted ledger if you need the record as well as the build result.

Code and docs: github.com/founder-ship-it/Slipmark

If you're letting agents sign off work in CI, I'd like to know how you check their verdicts are about the right code, and where this breaks for you.

Top comments (14)

Collapse
 
vladzoff profile image
Vlad Zoff •

I'd bind the green result to the exact tree, not just a branch name. Branches move. A verification record should carry the head SHA, base SHA, and tool/config version, and the merge gate should compare that SHA with the candidate. Otherwise, a valid pass can quietly refer to different code. This is one of the problems I'm addressing with verification receipts in Codapult Guard.

Collapse
 
paulredman profile image
Paul Redman •

Agreed, and that's where it ended up. Slipmark binds each verdict to the SHA-256 of every file it covered, not the branch or even the commit. So a pass can't refer to different code, and an unrelated commit elsewhere doesn't invalidate it.

The config part landed in v0.5.0. The policy is read from the base branch and its hash goes in every report. The agent's own instructions (AGENTS.md, skills, MCP config) are hashed too, so a verdict lapses if what the agent was told changes.

Interested to see how you're doing receipts in Codapult Guard. Are you signing them?

Collapse
 
vladzoff profile image
Vlad Zoff •

Not currently. Guard's verification receipt is machine-readable evidence, but it's not cryptographically signed.

It records the commit, policy base and fingerprint, approval decision IDs, check statuses, finding counts, contract issues, waiver state, and Guard version. The effective policy revision and fingerprint are also recorded in the run manifest, so the receipt can be traced back to the policy used for that run.

But I don't currently claim that this proves who ran the check or that the artifact hasn't been replaced. That's a separate integrity and provenance problem.

Your per-file hashing approach is interesting, especially because it binds the verdict to exactly what was checked rather than invalidating it on every unrelated commit. Signing receipts and tying them to CI identity is something I'd consider as a separate extension, rather than pretending the current receipt already provides that guarantee.

Thread Thread
 
paulredman profile image
Paul Redman •

That's an honest line to draw. Slipmark had the same gap until it added signing. An unsigned attestation is only a claim about who produced it. Now attest --sign uses the CI job's own GitHub identity through Sigstore, so the gate can require verdicts signed by a specific workflow on main. Your receipt fields would map well onto an in-toto statement if you go that way. Happy to compare notes.

Thread Thread
 
vladzoff profile image
Vlad Zoff •

That makes sense. I'd keep signing outside Guard's core runtime: Guard should produce a deterministic, content-bound receipt, and CI should wrap it in an in-toto attestation using its OIDC identity.
The important part is the verification policy, not just a valid signature. The gate needs to require the expected issuer and trusted workflow identity, then match the attested scope, PR head/base, and policy fingerprint against the change being evaluated. A pull_request workflow can come from the PR's merge commit, so I wouldn't let a workflow modified by that PR become the authority signing its own verdict.
I'd be interested in comparing how Slipmark binds and verifies those claims.

Thread Thread
 
paulredman profile image
Paul Redman •

Agreed on both points, and that's how the verify side works. A signature counts for nothing until the policy names who it trusts. signatures.trusted lists exact workflow identities and signatures.issuer pins the OIDC issuer (GitHub Actions by default). A verdict signed by anything else is refused as signer_not_trusted.

Your pull_request point is the one that catches people out. On a PR the identity ends in @refs/pull/<n>/merge, which the PR author controls. So the docs say to trust only the workflow at refs/heads/main, run by push, workflow_dispatch or pull_request_target, or a reusable workflow pinned to a protected ref. Since v0.5 the trust list comes from the base branch's policy file, so a PR can't add itself to it.

On scope, Slipmark doesn't match on PR head and base SHAs. The signature covers the attestation, which names the hash of every file it covered, and the gate re-hashes those files in the tree being merged. That's why an unrelated commit leaves a verdict standing.

Details are under Signed verdicts in github.com/founder-ship-it/Slipmar.... If Guard ends up emitting in-toto statements, I'd be keen to see whether a Guard receipt and a Slipmark verdict could sit in the same bundle.

Thread Thread
 
vladzoff profile image
Vlad Zoff •

I checked the in-toto Bundle spec, and it looks like a natural fit for this. It supports multiple independently authenticated envelopes in one JSONL file, with different subjects, predicate types and signing identities.
Guard already supports in-toto statements, so its architecture verification could be one entry, while Slipmark's verdict remains a separate attestation. Neither tool would need to own the other's predicate or trust policy.
One implementation detail matters: does Slipmark's --sign produce a DSSE envelope, or a Sigstore bundle using messageSignature? The former could be included directly as an in-toto Bundle entry. The latter would need a small envelope adapter or a separate outer container. Also, the gate would need to require both entries explicitly, since the in-toto Bundle itself isn't authenticated as a whole.

Thread Thread
 
paulredman profile image
Paul Redman •

DSSE. --sign calls Sigstore's attest, so the bundle carries a dsseEnvelope, not a messageSignature. You could lift the envelope out and add it as a line in an in-toto Bundle as it is.

The catch is the payload. It's Slipmark's own attestation as canonical JSON, with payload type application/vnd.slipmark.attestation+json, not an in-toto Statement. So it would go into the Bundle, but tooling that expects subject and predicateType couldn't read it. What would close that is an option to sign an in-toto Statement instead: the attested files and their SHA-256 hashes as subject, and the Slipmark verdict as the predicate. It's on my list, since it's the same direction as the context field I proposed on in-toto #603.

Agreed on the last point. The Bundle isn't authenticated as a whole, so the gate has to require each entry by predicate type and signer, and check each one's subjects against the change. A missing entry has to be a refusal, never a pass.

When the Statement option lands, I'll put up an example on the Slipmark side with a Guard-style receipt next to a Slipmark verdict in one Bundle, and the gate requiring both. One question back: do teams using Guard ask for evidence of what the agent checked, or is it mostly about stopping bad changes?

Thread Thread
 
paulredman profile image
Paul Redman •

To be clear on the last part: I'd like to do this with you rather than alongside you. If Guard's Statement entry and Slipmark's verdict can share one Bundle, that's a better example for both projects than either of us mocking up the other's half. Happy to pick it up on GitHub, or by email at hello@replyintelligencegroup.co.uk.

Thread Thread
 
vladzoff profile image
Vlad Zoff •

Both, but there's an important distinction: Guard is primarily designed to prevent architectural regressions. The evidence is about what Guard itself verified, not an attempt to record everything the agent read or checked.

I'll work on publishing a versioned Guard predicate v1 specification covering subject semantics, receipt structure, verification scope, fingerprint and validation requirements. I also plan to support both an aggregate verification-context subject and file-level subjects, so different consumers can use the representation that best fits their needs.
The CI workflow will provide exports for both DSSE Envelopes (.intoto.jsonl) and full Sigstore Bundles (.sigstore.jsonl), along with a consumer that validates required stages and rejects incomplete or unstable receipts.

Could you open an integration issue in the Slipmark repo and tag me (@vladzdev)? I'll open a corresponding issue in Guard, and we can cross-link them.

Thread Thread
 
paulredman profile image
Paul Redman •

Sorry for the slow reply, Vlad. Done: github.com/founder-ship-it/Slipmar...

I've split it into Guard side, Slipmark side and shared. The Statement option on my side is the first thing to build, since nothing can read a Slipmark entry in a Bundle until it exists. Link the Guard issue there when it's up and we'll take it from there.

Collapse
 
innokentyb profile image
Kent Bodrov •

I'd add an adversarial case for the list of required subjects itself. Keep all recorded file hashes correct, but remove an important path from require, or add a new implementation file that no attestation names. Does the gate still refuse? The file receipt can be sound while the coverage policy is incomplete; I'd keep that policy outside the change being reviewed, then bind its version into the decision too.

Collapse
 
paulredman profile image
Paul Redman •

Thats a good catch, and you're right !

The new file case is covered if you turn on diff coverage (coverage-base: origin/main). Any changed file withut a current verdict gets refused. It's off by default though, which probably needs rethinking.

The policy one is a real gap. Right now verify reads policy.json from the PR's own checkout, so a PR can loosen the rules it's being checked against. Same for the require list in the workflow.

The fix I'm going with.... read the policy from the base branch, not the PR, and put the policy hash in the report so you can see which rules actually made the decision.

Going into the next release. Thanks for this, exactly the kind of thing I was hoping someone would poke at.

Collapse
 
paulredman profile image
Paul Redman •

Shipped in v0.5.0: github.com/founder-ship-it/Slipmar...

The policy is now read from the branch you're merging into, not from the PR. Drop a path from require, loosen the policy, delete it, and the gate still judges you by the rules on main. The change only applies after merge.

Every report now records which policy made the decision, with its hash. That's your "bind its version into the decision" point. Required paths can live in the policy file now too, so they come from base as well.

The one gap left is the workflow file itself, since GitHub runs a PR's workflow from the PR. The docs say to put CODEOWNERS on .github/workflows/ and .slipmark/ for that.

Getting it reviewed turned up three more holes in the fix itself (subfolder runs, a git tag posing as main, a symlinked .slipmark folder), all closed now. So thanks twice over. You're credited in the changelog and the release notes.