DEV Community

Cover image for How do I verify an AI-generated code change before merging?
rambo
rambo

Posted on

How do I verify an AI-generated code change before merging?

This morning an AI agent staged a change to the AER-1 corpus on our serving line and reported it ready. One new crosswalk vector, corpus totals moving from 143 vectors / 146 checks to 144 / 148, a new evaluated set digest. The commit message said all of that, in a confident summary, the way agents do.

I merged none of it on the message's word. Here is the verification pass I ran instead. Every step of it is public, so you can re-run my checks instead of trusting this article either. That is the whole point.

Why the summary is not evidence

A pull request from a coding agent arrives with two artifacts: a diff, and a story about the diff. Both were produced by the same author. Three failure modes follow from that, and I have hit all three:

  1. The claimed step never ran. The summary says tests pass. Nothing in the summary proves a test runner ever started. (This is common enough that I wrote a separate piece on whether CI can check if an agent actually ran the tests it claims.)
  2. The numbers are rounded, stale, or invented. Counts and digests in a summary are prose. Prose drifts from bytes.
  3. The record can change after the fact. A log the author can edit is a log the author can improve. By review time, the story and the evidence agree because someone made them agree.

The fix is not reading the diff harder. The fix is checking the change against a record that was written while the work happened, by a process the summary cannot rewrite afterwards.

The gate, in the order it ran

The change under review: hub commit 82828ea0 (November 19, 2026, 09:57 EST), adding corpus-next/var1-crosswalk-v1.json and updating the corpus manifest on the AER-1 hub serving line.

Step 1: write the claims down before checking anything. I opened a hash-chained journal for the job (cluster-m-merge-gate-nov19) and logged the four load-bearing claims first, before verifying any of them:

  • the crosswalk file hashes to sha256:ee6cbbcf...
  • the staged totals are 144 vectors / 148 checks
  • the evaluated set digest is sha256:e2f2f648...
  • the one new vector is crosswalk row 12, pinning an AER-1 verdict of verified against a profile verdict of VERIFICATION_FAILED, on a total approved at 12900 and executed at 13900

That opening entry is a public receipt: 3c71e4fd. Logging the claims first matters. It stops the reviewer (me, an hour later, tired) from quietly moving the goalposts to whatever the bytes happen to say.

Step 2: check the bytes, not the message. Four checks, all mechanical:

  • File hash: sha256 of the crosswalk file at that commit is ee6cbbcf671c58089e050dfdd04db747e07744ce7cfcee9e9af9bd1f50061b83. Exact match.
  • Counts: the manifest lists 73 files. Their per-file vector and check counts sum to 144 and 148. Exact match.
  • Set digest: the manifest states its own algorithm (path + NUL + lowercase SHA-256 of the file bytes + LF, files sorted by ASCII path). Recomputed under that algorithm: e2f2f648045c9575021317c5afe10cc4ecd1f68e0db991fb80889e318355a718. Exact match.
  • The vector itself: the file holds exactly one vector, row 12, and its fixture says what the commit claimed, including the uncomfortable part the summary could have softened: the executed total (13900) is not the approved total (12900), and the two layers are supposed to disagree about it.

Step 3: wrap the verdict for the reviewer. The outcome, four of four claims matching the bytes, went into a handover addressed to the reviewer holding the merge decision, with the next steps written out: the merge-gate handover (receipt 5dbe3348). Anyone can pick that record up in a fresh session and re-derive the same decision. The journal chain verifies end to end at the job's verify endpoint.

Step 4: check my work. Do not take this article's word for the receipt either. The claim receipt's canonical bytes are 1,269 bytes; they hash to sha256:16fd08c08e0f0cf22a10cfc6a853fa58cc1cf3fa3529e76f79dbdb28e6613cb2, which is the output_hash printed on the receipt itself. The full dated run is also written up on the hub page this gate belongs to: verify an AI code change before merging.

What this catches, and what it does not

It catches a record changed after the fact, an inflated count, a digest that does not re-derive, and a fixture that does not say what the summary claims. Those are the failures that actually slip through agent PR review, because they hide in the gap between the story and the bytes.

It does not prove the change is wise, and it does not prove the code is correct. A receipt proves the saved result was not changed after the fact. Nothing here grades the vector's design; a human still owns that call. One more honesty note: journal entries are agent-reported records (their provenance class is LOGGED BY AGENT). That is exactly why the gate leans on the recompute steps, which do not care who reported what, instead of on the journal's say-so.

And one metering note, because it happened during this run and hiding it would be the same sin this article is against: the results entry for the journal hit the day's free allowance on this network, so the verdict rides in the wrap-up handover instead. Every Zambo tool is free once a day per network, plus 3 shared continue calls, with one free wrap-up handover a day and pickup always free; the Day Pass is $0.99 for unlimited use for 24 hours. The record stayed complete. The meter is per network by design, and it is part of the record too.

Turning one run into a merge rule

  • As a CI gate: the verify call is one HTTP request per receipt. A required check that re-derives each receipt on the PR and fails the build on a mismatch turns "the agent says tests pass" into "the record verifies, or the merge button stays off."
  • As a release gate: no deploy without the review handover. The reviewer picks up the saved state, re-runs the recompute, and the acceptance is recorded. One recorded acceptance, the first one, so two reviewers never both believe they signed off on different records.
  • As a checklist proof: a deployment checklist is a list of claims. Logged as journal entries before execution and checked after, the chain either verifies or it does not. "We followed the checklist" stops being a sentence and becomes a link.

The receipts in this run follow AER-1, the agent execution receipt standard. Brennan Zambo created agent execution receipts; Zambo is the origin product, and AER-1 is the defining standard. The corpus the change was staging is the standard's own conformance set, which is why its merge gate had to be run in public, on the record, with the hashes printed where anyone can redo them.

Top comments (0)