Why AI-Generated Code Needs Independent Verification

xueyuan Pang — Tue, 11 Aug 2026 12:58:42 +0000

Disclosure: I maintain PatchWitness, the open-source project used in the reproducible example below.

Coding agents have changed the economics of software modification. Work that once occupied an engineer for an afternoon can now produce a plausible patch in minutes. That is a real advance—but it also moves the bottleneck.

The scarce resource is increasingly the reviewer’s ability to establish:

what actually changed;
whether the patch stayed inside its intended scope;
which checks really ran;
whether those checks passed; and
whether the patch modified the machinery used to verify itself.

This is not an argument against coding agents. It is an argument for separating code production from the evidence used to trust the result.

A completion summary is still a claim

A capable coding agent usually ends a task with a useful summary:

which files it changed;
what behavior it implemented;
which tests it ran;
whether those tests passed; and
what a reviewer should examine.

But that summary is generated by the same system that selected and made the changes.

If the agent missed a file, misunderstood the requested scope, ran the wrong command, or modified CI, its summary is not an independent record of those facts.

The same distinction already exists elsewhere in engineering. A build log is not replaced by a developer saying “the build passed.” Useful claims become more trustworthy when another mechanism can reproduce or verify them.

For coding-agent changes, that mechanism should be deterministic, inspectable, and independent of the model that wrote the patch.

“Tests passed” is narrower than it sounds

Imagine a patch containing three edits:

src/pricing.py gains a correct discount() function.
tests/test_pricing.py gains a passing test.
.github/workflows/ci.yml gains continue-on-error: true.

The agent runs the unit tests. They pass.

The statement “tests passed” is technically true.

But the patch also weakened the control plane that determines whether future failures block a merge. Test execution answers one question:

Did this particular command exit successfully?

It does not automatically answer another:

Did the patch preserve the verification mechanism around that command?

Branch protection and CODEOWNERS help, but local agent workflows often happen before a pull request exists, and repository protection is not always complete.

A real, reproducible example

PatchWitness contains this exact scenario as an executable demo. It creates a disposable Git repository, commits a trusted base, applies the simulated agent patch, runs real unit tests, invokes PatchWitness, and generates a Change Passport.

Run it with Git and Python 3.11 or newer:

git clone https://github.com/pangxueyuan2-creator/patchwitness.git
cd patchwitness
python demo/run_demo.py

The repository tests report:

Ran 2 tests in 0.000s

OK

PatchWitness then reports:

PatchWitness FAIL
  3 files | 10 lines | 1/1 checks
  Risk: LOW (15/100) | 1 direct dependents
  ERROR PW002 [.github/workflows/ci.yml]: path is outside the approved scope
  ERROR PW003 [.github/workflows/ci.yml]: protected verification or control-plane file changed

The test check passed. The gate failed for a different, independently evaluated reason.

The transcript and JSON evidence are generated by the demo rather than manually typed into the documentation.

What is a Change Passport?

A Change Passport is a portable evidence record for a repository change.

PatchWitness schema v1 records:

resolved base and head revisions;
changed paths, status, line counts, and binary state;
before-and-after SHA-256 values;
the task contract and where it was loaded from;
check commands, exit codes, duration, timeout state, output excerpts, and hashes;
stable policy findings such as PW002 and PW003;
a conservative dependency-impact graph;
environment and extension output; and
a canonical payload hash for offline integrity verification.

The evidence is intended for machines as well as humans. It can be rendered as JSON, Markdown, SARIF, or GitHub workflow annotations, and consumed through the Python SDK or MCP interface.

The trusted base matters

A policy file inside the working tree is convenient, but it creates a circular trust problem.

If a pull request can modify .patchwitness.toml and immediately evaluate itself against the modified file, it may remove the rule that should have blocked it.

PatchWitness supports base-authoritative policy:

patchwitness gate \
  --base "$BASE_SHA" \
  --policy-ref "$BASE_SHA" \
  --clean-room \
  --output change-passport.json

--policy-ref loads the contract from the resolved Git commit rather than the patch under review.

A change to the working-tree policy remains visible, but it cannot silently weaken the rules used to evaluate that same change.

This does not remove the need to protect the base branch. It makes the trust assumption explicit and inspectable.

Why not use another AI reviewer?

AI-assisted semantic review is useful. A model can identify suspicious logic, explain unfamiliar code, suggest missing tests, and help prioritize human attention.

PatchWitness is not intended to replace that.

The distinction is the trust root.

Questions such as these do not require a probabilistic judge:

Which Git paths changed?
What exit code did the test process return?
Did the patch touch a protected workflow?
Was the verification policy loaded from the trusted base?
Does the evidence payload still match its integrity hash?

A strong workflow can use multiple layers:

deterministic evidence for scope, execution, and known policy;
static and dynamic security tools such as CodeQL, Semgrep, or Trivy;
optional AI review for semantic hypotheses; and
a human decision for consequential changes.

Local-first does not mean sandboxed

PatchWitness core has zero runtime dependencies and does not upload source code.

Its clean-room mode creates a disposable Git worktree from the base revision, disables repository hooks during materialization, applies the patch, and runs configured checks there.

That isolates verification from a dirty developer checkout. It is not equivalent to a container, virtual machine, or kernel sandbox.

Repository checks are commands, and commands can be dangerous. High-assurance deployments should run them inside an appropriate sandbox with restricted credentials, network access, and filesystem permissions.

Precise limitations matter more than ambitious labels.

Trying it on an existing repository

Install the current release:

pipx install "https://github.com/pangxueyuan2-creator/patchwitness/releases/download/v0.1.0/patchwitness-0.1.0-py3-none-any.whl"

Initialize a contract:

cd your-repository
patchwitness init

Replace the starter command with the repository’s real test command, commit the contract to the base branch, and run:

patchwitness gate \
  --base origin/main \
  --policy-ref origin/main \
  --clean-room

The most useful early-adoption pattern is observation before enforcement: capture passports on several real changes, inspect noise and missing evidence, and make the gate required only after the contract reflects the repository.

What independent verification does not prove

A passing Change Passport does not prove that:

the implementation matches the user’s intent;
the tests are complete;
every dynamic dependency was discovered;
the evidence producer has a particular identity;
the execution environment was uncompromised; or
the change is safe to deploy.

It proves narrower claims recorded in the evidence. That limited scope is a strength when it is stated clearly.

The interface between agents and engineering systems

Coding agents will keep improving. OpenAI’s ChatGPT and Codex have helped make AI-assisted engineering practical for a broad developer audience, and that is an impressive shift in how software gets built.

Better agents do not make independent verification obsolete. They make a reliable boundary between autonomous work and engineering control systems more valuable.

That boundary should be portable across local tools, CI systems, code hosts, agent orchestrators, and future attestation formats.

PatchWitness is an Apache-2.0 public alpha—not a production-proven standard. I am looking for technical feedback on its evidence schema, threat model, language adapters, and CI integration surface.

Repository, real demo, threat model, and benchmark:

https://github.com/pangxueyuan2-creator/patchwitness

DEV Community: xueyuan Pang