DEV Community

Cover image for How do I verify an AI-generated code change before merging?
rambo
rambo

Posted on

How do I verify an AI-generated code change before merging?

The short answer

Run a gate with three moves, in this order. First, write down the change's claims before you check anything: which files it says it touched, which behavior it says it implements, which tests it says it ran. Second, check each claim against bytes and records, not against the author's summary: hash the submitted artifacts yourself, recompute any receipt the author presents, and run your own analysis steps through tools that mint receipts for their results. Third, merge only when every load-bearing claim has a check behind it, and record the verdict where the next reviewer can find it. The agent's summary is an input to the gate. It is never the gate's output.

Disclosure, once: I run ops for Zambo, the endpoint used for the receipted analysis step below.

Brennan Zambo created agent execution receipts; Zambo is the origin product, and AER-1 is the defining standard.

Why summaries fail as merge evidence

An AI-generated pull request arrives with a fluent description of itself, and fluency is exactly the problem. The summary was written by the same process that wrote the code, from the same context, with the same blind spots. If the agent misunderstood the ticket, the summary misunderstands it confidently and in complete sentences. Reviewers then verify the prose against their expectations instead of verifying the change against reality, and the failure modes that reach production are the ones the summary never mentioned: the file edited outside the stated scope, the test suite described as run when only a linter was, the document whose numbers were carried over from a draft the author meant to replace.

None of this requires a dishonest agent. It requires only an ordinary one and a reviewer in a hurry. The gate below assumes the summary is a set of claims, possibly sincere, possibly wrong, and treats sincerity as irrelevant. Claims get checked or they get marked unchecked. There is no third state called "sounds right."

The gate, step by step

1. Claims first, in writing. Before opening the diff, extract the PR's claims as a numbered list: files, behaviors, tests, documents, numbers. This ordering matters. A claim list written after you have seen the checks will quietly shrink to whatever passed.

2. Hash what was actually submitted. For each artifact the change delivers, compute its SHA-256 over the exact bytes in the PR. If the author or their tooling presents a receipt, recompute it: take the record's canonical bytes, hash them, compare against the published output_hash. A mismatch ends the review until explained. A match proves the saved result was not changed after it was recorded, which is the foundation everything else stands on.

3. Run your own checks through receipted tools. The reviewer's steps deserve the same discipline as the author's. When you run an analysis over the change, run it through a tool that mints a receipt for the result, so your verdict rests on records a second reviewer can recompute rather than on your say-so in a comment thread.

4. Decide per claim, then merge or do not. Each claim ends in one state: checked and holding, checked and failed, or not checked. AER-1 names the anchor outcomes for records (verified, verification failed, rejected: unresolvable record, not checked, absent), and the same discipline applies to claims. A PR with a not checked load-bearing claim is not approved. It is postponed, which is a different and much more honest word.

A real gate run, this morning, on a real submission

On November 22, 2026, an interop lab's pull request carried a submitted document, a mapping of a live AER-1 receipt into a second framework, and the mapping's correctness was the claim under review. The gate ran on it before any agreement was recorded:

  • The artifact, hashed by the reviewer: MAPPING.md as committed, 9,237 bytes, SHA-256 sha256:b4ce7b7958bbfa667ffa48a3189ae118ffa91a6e67d0554e5ef55d7131f031cc, recomputed locally from the committed bytes rather than taken from the PR description.
  • The analysis step, receipted: the document was run through doc_intel at https://zambo.dev/mcp , which returned its analysis (1,360 words; the summary correctly identifying the asserted-only grounding and the absent versus not checked anchor distinction) and minted receipt 34d47e29-1b53-40a9-84df-c9cac9841245 at 2026-11-22T09:05:56Z: https://zambo.dev/run/34d47e29-1b53-40a9-84df-c9cac9841245 . The receipt carries 3,580 canonical bytes; recomputing SHA-256 over them gives sha256:0a8ceb6330503a1f181f4fc3b0089a663e7af5d74eb6edf805435abbaf5f2e37, matching its output_hash exactly. MATCH, outcome verified.
  • The verdict, recorded after the checks: the run's expectations had been frozen before the run, both appraisers matched them in all five cases, and only then was agreement posted on the result row, in the open, where the next reviewer can re-derive every step.

Translate that shape to code and nothing structural changes: the diff replaces the document, the test run replaces the analysis step, and the merge decision waits for the recompute either way. The tooling that receipts the steps is one endpoint, 153 tools in total with 37 on the default surface, and the checks themselves are free for whoever runs them.

What the gate does not prove

A receipt proves the saved result was not changed after it was saved. It never proves the result was right. A test run can be receipted faithfully and still be the wrong test suite, run against a stale branch, in a dirty environment. The gate catches substitution, drift, and invention: the summary claiming a step that never ran, the artifact swapped after review, the numbers copied from an earlier draft. It does not catch bad judgment, and it is not a substitute for reading the diff. Provenance has the same boundary: journal and receipt entries record who the system says ran a step, which is a producer assertion. The gate leans on recompute precisely because assertions are cheap.

Price the discipline honestly, too. Hashing and recomputing are minutes. The expensive outcome is the merge you cannot reconstruct in February, when someone asks what exactly shipped and the only witness is a chat thread.

What it costs

Every tool free once a day, plus 3 continue calls. One free wrap-up handover per day, and pickup is always free. A daily merge gate over real changes fits inside the free tier for most reviewers, and verifying anyone's receipts never costs anything. Paid tiers, stated once: Day Pass $0.99 unlimited for 24 hours, Zambo Pass $29/mo, $ZAMBO holder Pass $19/mo, Lifetime Pass $250. There is no annual tier.

Try it: Day Pass, $0.99 unlimited for 24 hours: https://zambo.dev/day-pass/ and add Zambo to your AI: https://zambo.dev/install/ (redirects to the add page, verified live today).

Sources

Top comments (0)