An AI coding agent returns a clean test log and says the task is complete.
The log may be genuine. The command may have passed. The result may still be insufficient for the claim you are being asked to accept.
The problem is not that logs are useless. The problem is that a log is usually supplied by the same workflow making the completion claim. Independent review begins when a second person can connect that output to the current artifact and reproduce the important part.
A log answers one narrow question
A test log can show that a command produced particular output. By itself, it may not show:
- which revision was tested;
- whether files changed after the run;
- which environment or configuration was active;
- whether the command exercised the requested behavior;
- whether the output was truncated;
- whether the reviewer can run the same check.
None of those gaps proves the implementation is wrong. They limit what the log can support.
Reconstruct the evidence chain
Use five fields:
- Claim — the exact behavior you are being asked to accept.
- Artifact — the changed files, patch, build, or other delivered item.
- Execution — the command, output, exit status, and timestamp.
- Provenance — the revision, workspace state, environment, and configuration.
- Coverage — what the check exercised and what remained outside its scope.
If one field is missing, keep the verdict narrow. “A test log was supplied” is different from “the requested behavior is independently reproducible on the current revision.”
Example: the real log from the wrong moment
Suppose a focused test passes at 10:14. At 10:19, the agent updates a lockfile and configuration. At 10:22, it returns the earlier log and says the delivery is ready.
The 10:14 result may be real. It is stale for the final workspace until the affected check is rerun.
The smallest useful request is not “prove everything.” It is:
Rerun this command against the final revision and return the exit status plus the current revision identifier.
That request binds the result to the artifact without pretending one rerun certifies security, deployment, or production readiness.
Reproduction is a review action
For a low-risk change, reading the diff and rerunning one focused command may be enough. For data, security, billing, or release behavior, the reviewer may need additional checks and a human owner.
The important distinction is between supplied evidence and reproduced evidence:
- Supplied evidence helps you decide what to inspect.
- Reproduced evidence gives you a result from a check you controlled.
Both can be useful. They support different levels of confidence.
A narrow verdict is still a useful verdict
Use one of four outcomes:
- Supported — the current artifact and reproduced check support the exact claim.
- Unverified — the evidence is missing, stale, too narrow, or not reproduced.
- Blocked — access, environment, or permissions prevent the next check.
- Conflicted — the artifacts disagree.
This avoids turning “I cannot verify this yet” into “the code is bad,” while preventing a green log from silently becoming a broad release decision.
A reusable review prompt
Before accepting an AI coding handoff, ask:
- What exact claim does this log support?
- Which revision produced it?
- What changed after the run?
- Can I reproduce the important check?
- What remains outside the evidence boundary?
I made the free AI Completion Evidence Auditor Lite as a manual worksheet for structuring that review. It does not run tests or authenticate supplied logs:
Top comments (0)