A coding agent changed the code. What proves the task is done?

Sarthak Agrawal — Fri, 14 Aug 2026 22:10:55 +0000

A coding agent can produce a clean diff, explain every choice, and still leave the task broken.

I kept running into the same problem while reviewing agent-written changes. Code review could tell me that something looked suspicious. A test command could tell me that a test suite was green. Neither answer proved that the requested behavior worked.

That gap is what led me to build CodeVetter.

Review findings are leads, not verdicts

Static review is useful. It can spot missing error handling, authorization mistakes, suspicious state changes, and code that does not match the surrounding repository. But a plausible finding is not a reproduced failure.

The reverse is also true. A green command is not automatically proof. The test might miss the requested behavior. The environment might differ from production. An existing failure might be reported as a new regression. A command without its task and repository revision is just output without context.

Verification needs to connect the task, the exact change, the environment, the command, its output, and the resulting verdict.

The record I want after an agent finishes

For each task, I want to be able to answer five questions:

What outcome was requested, including the acceptance boundary?
Which exact repository revision and patch did the agent produce?
Which checks ran against the changed behavior?
Which commands, outputs, artifacts, and failure classifications were retained?
What passed, failed, or remained unverified?

The last question matters more than it sounds. Missing evidence must stay missing. A verifier should not turn an unavailable check into a pass because the rest of the change looks convincing.

CodeVetter is built around this task-to-evidence loop. The CLI and MCP boundary produce a machine-readable verification bundle, while the desktop app provides a local way to inspect it.

Where execution catches what review misses

Runtime checks matter most when an agent changes authorization rules, browser state, API contracts, persistence, concurrency, or regression-sensitive behavior. These changes can look reasonable line by line while failing in the real interaction.

For example, a browser test that clicks a button is not enough if the task also requires a particular account, URL, and durable state change. The evidence should retain those conditions so another reviewer can tell what the test established.

The same standard applies to API work. A 200 response does not prove that the response used the right schema, wrote the intended state, or rejected an unauthorized request. Each acceptance criterion needs evidence that addresses it directly.

A benchmark should state what it cannot prove

CodeVetter has a public synthetic recognition benchmark with 27 cases and 29 labeled findings. The cases, reviewer outputs, scorer, and limitations are public.

That benchmark can show how the current reviewer handles those cases and false positives. It cannot establish performance across large production repositories. I would rather publish a narrow result with inspectable inputs than stretch it into a general claim.

You can inspect the benchmark at https://codevetter.com/benchmark.

The practical standard

Before I call an agent task complete, I now look for a chain that another person can inspect:

task -> revision -> check -> output -> evidence -> verdict

If part of that chain is absent, the task may still be useful, but it is not fully verified. Keeping that uncertainty visible is better than manufacturing confidence.

The full verification model is at https://codevetter.com/coding-agent-verification. CodeVetter is available for macOS, Windows, and Linux at https://codevetter.com/download, and the source is at https://github.com/Codevetter/codevetter.