DEV Community

Cover image for How to Verify AI-Generated Code With Evidence
jamilxt
jamilxt

Posted on Edited on Originally published at jamilxt.com

How to Verify AI-Generated Code With Evidence

An AI coding agent says the change is complete. The tests pass. Is that enough to merge?

Not always. The code and its tests may share the same mistaken assumption. A unit test can pass while the feature fails when a real user follows the full workflow. A code review can miss a bug if the reviewer sees the same incomplete context that shaped the implementation.

On November 16, 2026, Addy Osmani described how the Claude Code team approaches this problem. He says about 90% of the team's code is written and reviewed by Claude. That figure comes from Anthropic, not an independent audit. The post's practical value is its verification design. Run the change. Use different checks. Keep each check separate from the change it judges. Measure whether the checks catch real defects.[1]

The same ideas apply to any coding agent. This article turns that account into a workflow you can adapt to an existing project. It does not claim that tests can prove software has no bugs. The goal is narrower: collect evidence that matches the change, then make the remaining uncertainty visible.

Start with the behavior, not the test command

A green test suite answers one question: the checks that ran passed. It does not answer whether the right checks ran or whether they cover the user's task.

Before asking an agent to change code, write down the behavior that must remain true. Keep it concrete. For a paginated endpoint, require no more than the requested number of parent records. Each record should include the expected child data. Also check that the database does not load every match into application memory. These are requirements to verify, not claims about a particular framework or database.

This short list gives the agent a target and gives the reviewer a basis for checking the result. It also makes a useful distinction: implementation evidence shows that the code changed as expected; behavior evidence shows that the running system still meets the requirement.

The Claude Code team's post makes running the application part of verification, not an optional final check. It describes project-specific instructions for starting the app and exercising a change before saving a commit (a snapshot of the code in version control). It also says failures that remain after three repair rounds are handed to a person.[1] The lesson is not that every team needs the same number of attempts. It is that an agent should not keep retrying indefinitely or label a change complete while a required check still fails.

Use tests that look at different failure modes

No single test type catches every class of defect. A reliable workflow uses checks that fail for different reasons.

End-to-end tests check a real path

An end-to-end (E2E) test drives the application through a user-visible flow. A test might create a record. Then it opens the detail page and changes a field. After a reload, it checks the new value.

This can catch failures isolated unit tests miss. The route may not be wired. A response shape can be wrong. A browser control may stay disabled. Data can disappear between layers. The Claude Code team's article recommends investing in E2E tests because they give an agent a running system to inspect and a clear result to act on.[1]

E2E tests cost more time and can fail for reasons outside the code under test. Keep a small, stable path for the behavior being changed. Make the failure useful: preserve the step that failed, the relevant output, and a screenshot or trace when the test environment supports them. A bare timeout gives less help than a report that says which action failed and what the page showed.

Property-based tests check rules across many inputs

A normal example-based test checks a few inputs chosen by a person. A property-based test states a rule, then uses a library to generate many inputs and look for a counterexample (an input that breaks the rule). The rule might say that encoding and then decoding a value returns the original value, or that an account balance never becomes negative.

The Claude Code post discusses round-trip encoding and invariants, which are conditions that must always hold. It also covers idempotency, where repeating an operation has no extra effect, and checking optimized code against a simpler reference implementation. Generated inputs can reveal edge cases that hand-picked tests miss.[1]

Property-based tests are useful when a rule has a broad input space. They are less useful when the requirement is vague or when a generated case cannot be interpreted. Start with a property that a product owner or maintainer can explain in one sentence. Keep the failing input when the library finds a counterexample; that input can become a regression test (a test that catches the same bug if it returns).

Differential tests compare old and new behavior

A differential test runs two implementations with the same inputs and compares their outputs. This helps during a migration or rewrite when the expected behavior should stay the same, even if the internals change.

For instance, a team could run a new parser and the existing parser against the same sample documents. If their outputs differ, the difference needs investigation. Some differences may be intended, so normalize irrelevant details and record approved changes rather than treating every mismatch as a bug. The Claude Code article recommends random inputs or a sample of production traffic for this kind of comparison.[1]

Differential testing has a boundary: the old system is a reference, not an infallible specification. If the old behavior contains a defect that the migration should fix, matching it would preserve the defect. Record which differences are expected and why.

Keep the verifier independent from the patch

A test can be green and still be weak if the code under review can rewrite the test or the rule that runs it. This is a problem of independence as well as test coverage.

The Claude Code team's article describes keeping review rules on the base branch (the target branch for a proposed change) rather than reading a modified copy from the proposal. It also says review results are tied to one commit. That stops a passing result from silently applying to newer code.[1] These controls make the evidence refer to the exact change that was checked.

A project can apply the same principle in simpler ways:

  • Keep required checks in continuous integration (CI) configuration or another trusted location.
  • Have the verifier report the commit identifier it checked.
  • Re-run checks after any code change.
  • Make changes to test rules visible to a human reviewer.
  • Do not let a code change weaken a failing assertion (the condition a test checks) just to make the suite pass.

For a new gate, start in observe mode if the system supports it. It logs which changes would have been blocked without stopping work. Review false alarms, adjust the rule, and enforce it after the check behaves as expected. The Claude Code team describes using this gradual rollout for its commit gate.[1]

This does not mean tests should never change alongside product code. New behavior often needs new tests. It means changes to the judging rules deserve separate scrutiny. When a passing code change removes a check or changes expected output, the reviewer should understand why the old expectation was wrong.

Turn incidents into checks that match the diff

A static checklist can become long enough that nobody uses it. The Claude Code team says it keeps a reliability list of 39 items. Each item has a trigger that can be recognized in a diff (the set of code changes), a check, and a link to the incident or pull request (a proposed code change for review) that motivated it. The agent checks the items whose triggers match the change, usually three to ten items, rather than applying the full list every time.[1]

A small project can use the same structure without copying that list. When a production bug or a review miss happens, record the code change that would have exposed it. Then write a check that runs only when a later diff touches the same behavior. For example, when session storage or message types change, check that older transcripts still work when a new field is missing. That example follows the Claude Code team's post.[1]

This keeps lessons close to the code that needs them. It also gives the reviewer a reason for each extra check. The list needs regular cleanup: remove outdated rules, merge duplicates, and confirm that each check still catches the failure it was added to detect.

For Claude Code users, skills can store reusable instructions for a procedure. Hooks can run defined actions at events such as a tool call or session end. The official skills documentation describes how Claude loads skills for relevant tasks. The hooks reference documents commands and HTTP calls. It also covers other hook types.[2][3] These features can help make checks repeatable, but they do not make a weak check reliable. The project still owns the test commands and their results.

Make failure behavior explicit

An agent needs a stopping rule. Without one, it may repeat a flaky test (one that passes or fails unpredictably). It may alter a timeout or keep editing until the failure disappears. It might do this without finding the cause.

The Claude Code team says it does not let Claude skip flaky tests or add sleeps. It also does not let it retry until a test passes. If it cannot identify the cause, the team removes the test and records why.[1] That is the team's process, not a universal rule that deleting a test is always correct. Another project may set a flaky test aside temporarily. It can record a bug report and block the affected release until the test is repaired.

A useful failure policy answers four questions:

  1. Which failures may the agent repair without approval?
  2. How many repair attempts are allowed before it stops?
  3. What evidence must it include when asking a person to decide?
  4. What happens to a flaky or unavailable check before merge?

If the agent changes a test, it should explain the old assertion. It should say why that assertion was wrong or incomplete. It should also show how the replacement catches the intended regression. A passing result alone cannot answer those questions.

Check whether the checks catch important bugs

A test suite can grow while its value stays unclear. Reviewers can also become noisy: they may report issues that are already present or miss defects that later force the team to undo a change.

The post points to a Code Review team's evaluation based on past incidents and changes later undone. It separates a tuning set (examples used to adjust the check) from a fixed test set (kept aside for evaluation). It compares findings with merged changes that have stayed stable to reduce false alarms. It also measures recall (the share of known defects found) at three stages: raw findings, findings after verification, and final review.[1]

A small team does not need to copy that evaluation system in full. Start by saving examples of defects that escaped review and changes that were incorrectly flagged. For each new check, ask:

  • Does it catch the intended failure in a known example?
  • Does it stay quiet on a similar change that is correct?
  • Is the result actionable, or does it only say that something looks suspicious?
  • Does the check still work after the surrounding code changes?

This shifts review from saying the agent found a bug to asking whether the finding matches a known failure mode and points to evidence. It also helps you decide whether to refine a check or remove one that creates noise.

A practical verification loop

For a small change, the workflow can be direct:

  1. State the expected behavior. Write a short acceptance condition before implementation.
  2. Ask the agent to inspect the relevant code and tests. Require it to identify the commands it plans to run.
  3. Make the smallest useful change. Keep unrelated formatting and cleanup out of the proposed change.
  4. Run focused checks first. Run the narrow test or property check tied to the change.
  5. Run the application path. Exercise the real user flow or the part of the service affected by the code change.
  6. Run broader checks before review. Build the app, run automated style checks, and run the relevant test suite.
  7. Inspect the diff and evidence. Confirm that tests were not weakened and that the reported results belong to the final commit.
  8. Stop on unresolved failures. Let a person decide when the failure is unexplained, flaky, or outside the agent's permission.

The sequence is the useful part. Define what done means and test at more than one level. Keep evidence tied to the final change. Make unresolved failures visible.

The point is evidence, not autonomy

More agents do not remove the need for engineering judgment. They increase the amount of work that can be proposed, which makes a clear verification process more important.

The Claude Code team's account offers a practical pattern. Run the change end to end. Add tests for broad input spaces or comparisons with old behavior. Keep review rules independent. Then evaluate whether checks catch the defects that matter. The account also shows a limit: the team escalates unresolved failures to a human.[1]

For any coding agent, the useful completion message is not simply "done." It should say what changed and which checks ran. It should report the results, note what was not checked, and say whether any failure remains. That gives a reviewer evidence to assess instead of asking the reviewer to trust the agent's summary.

Sources

[1] Addy Osmani, "How the Claude Code team thinks about verification" (November 16, 2026). The reported team practices and 90% figure are Anthropic's own account.

[2] Claude Code documentation: Skills.

[3] Claude Code documentation: Hooks.

Top comments (0)