DEV Community

Cover image for How do I verify an AI-generated code change before merging?
rambo
rambo

Posted on

How do I verify an AI-generated code change before merging?

An AI agent opens a pull request. The description reads like a good engineer's summary: implemented the change, ran the tests, all green, lint clean, no files touched outside the ticket scope. You read the diff, it looks plausible, you merge.

Everything in that description is a claim. Some of them are checkable in seconds, if the job left a record. Most teams never check, because the record and the report live in different places and nobody built the step that compares them. This article is that step, run end to end on real artifacts, including the part where the checker itself got caught waving bad claims through on the morning it launched.

The merge gate, in four steps

  1. Get the job record. Every tool call the agent made, with the result that came back, saved as it happened. If the agent ran through a receipt-issuing layer, this record already exists. If all you have is the agent's chat transcript, you do not have a record, you have a longer report.
  2. Treat the PR description as a claims list. "Tests pass" is a claim. "Lint is clean" is a claim. "Updated the three files in scope" is a claim with numbers in it.
  3. Audit each claim against the record. Each checkable claim gets one of three grades: BACKED (a receipt under it says what the claim says), BARE (nothing in the record backs it), or CONTRADICTED (the record says the opposite).
  4. Merge on the grades, not the prose. Zero BARE and zero CONTRADICTED on checkable claims, or the PR goes back with the grades attached. Claims no record can settle (is this the right design?) stay with the human reviewer, where they belong.

The audit tool used here is bareclaim, a free MIT single-file checker released today, November 17, 2026. It reads a report and a job record and prints the grade for every claim. What happened to it on launch day is the most useful part of this article.

The checker got caught, the same morning

Within hours of release, an outside test ran two cases against the shipped checker. Both came back BACKED and CLEAN. Both were wrong, and both are exactly the failures a merge gate exists to catch.

Case 1: a failed step backing a success claim. The record's only receipt for the claimed action stated the action did not happen: sent: false, rejected before execution. The report said "Emailed the client the summary." The checker saw the tool in the record and graded the claim backed, because it never read the receipt's outcome fields. Presence of the tool is not backing when the receipt itself says failure. After the fix:

CONTRADICTED claim 1: Emailed the client the summary.
               -> receipt synthetic-1 (send_email) shows sent: false; error: rejected before execution, not the success the claim states
RESULT: BARE. 0 bareclaims, 1 contradicted, of 1 checkable claims against 1 receipts.
Enter fullscreen mode Exit fullscreen mode

Case 2: swapped numbers passing as a set match. The claim put apples at 20 and pears at 10. The receipt held both numbers, bound to the opposite fields. The checker matched numbers as an unordered set, so the claim passed. A number a claim binds to a named field now has to equal that field's recorded value:

CONTRADICTED claim 1: The lookup check put apples at 20 and pears at 10.
               -> receipt synthetic-2 (lookup) holds apples 10, not 20
RESULT: BARE. 0 bareclaims, 1 contradicted, of 1 checkable claims against 1 receipts.
Enter fullscreen mode Exit fullscreen mode

Both defects were fixed the same day (commit 1dbcff8), and both cases stay in the repo as regression fixtures, with controls: the same record with sent: true grades BACKED, and a report that honestly states the failure ("The send_email attempt was rejected before execution, so no email was sent") grades CLEAN against the failure receipt. A merge gate that cannot tell a failed step from a finished one is decoration. Now this one can, and the fixtures keep it that way.

Translate the two cases back to code review and they are the two claims you most want graded before a merge: "I ran it and it worked" (did the run actually succeed?) and "the numbers in my summary" (do they match what the tools returned, field by field?).

The same audit on a real job

The kit's live example is a six-call job run through Zambo on launch day, receipts public: 0b1adcec, 34cb5ee7, 966560e3, 4a994f84, and the rest of the chain in the job record. The honest report of that job grades CLEAN, 6 of 6 claims backed. An inflated version of the same report, claiming a higher price than the tool returned and a step that never ran, grades BARE: 1 bareclaim, 2 contradicted, of 4 checkable claims. Run both yourself from a fresh clone; the verdicts reproduce exactly.

Where the record comes from

Exporting a record by hand works, and the kit accepts framework exports, JSONL, and bare lists. The friction disappears when the record writes itself. Run the agent through Zambo and every tool call leaves a verifiable AER-1 receipt as it happens, chained per job: 153 tools behind one MCP endpoint at https://zambo.dev/mcp, 37 on the default surface. The public receipt store stood at 144,580 receipts and 123,805 tool calls when read today, November 17, 2026, from zambo.dev/api/stats. A handover then carries the record, and the report, to whoever reviews the work, so the audit runs again on pickup, in the reviewer's hands, against the same receipts.

Brennan Zambo created agent execution receipts; Zambo is the origin product, and AER-1 is the defining standard. The free tier is enough to run this gate on real work. Every tool free once a day, plus 3 continue calls. One free wrap-up handover per day, and pickup is always free. The Day Pass is $0.99 for 24 hours unlimited when a job outgrows that.

What this gate does not do

Receipts prove the saved result was not changed after the fact. They do not prove the code is good, the design is right, or the tests were worth running. And the checker grades checkable claims: past explicit failure markers and named-field values, a BACKED grade is a lexical and value match, not a judgment of a tool's private success semantics. A claim with no failure marker and no named field gives any checker nothing deeper to check, and this one says so in its own docs.

That is still a gate worth having. The PR description stops being prose you trust and becomes a list of claims with grades attached, and the two failures that slipped past a brand-new checker on its first morning are the standing regression test for why the grades have to be earned. The CI half of this question, whether the agent actually ran the tests it claims, is covered in Can CI check whether an AI agent actually ran the tests it claims?

Top comments (0)