DEV Community: Reliable AI Delivery

A Test Log Can Support a Claim Without Being Independent Proof

Reliable AI Delivery — Thu, 23 Jul 2026 20:47:41 +0000

An AI coding agent returns a clean test log and says the task is complete.

The log may be genuine. The command may have passed. The result may still be insufficient for the claim you are being asked to accept.

The problem is not that logs are useless. The problem is that a log is usually supplied by the same workflow making the completion claim. Independent review begins when a second person can connect that output to the current artifact and reproduce the important part.

A log answers one narrow question

A test log can show that a command produced particular output. By itself, it may not show:

which revision was tested;
whether files changed after the run;
which environment or configuration was active;
whether the command exercised the requested behavior;
whether the output was truncated;
whether the reviewer can run the same check.

None of those gaps proves the implementation is wrong. They limit what the log can support.

Reconstruct the evidence chain

Use five fields:

Claim — the exact behavior you are being asked to accept.
Artifact — the changed files, patch, build, or other delivered item.
Execution — the command, output, exit status, and timestamp.
Provenance — the revision, workspace state, environment, and configuration.
Coverage — what the check exercised and what remained outside its scope.

If one field is missing, keep the verdict narrow. “A test log was supplied” is different from “the requested behavior is independently reproducible on the current revision.”

Example: the real log from the wrong moment

Suppose a focused test passes at 10:14. At 10:19, the agent updates a lockfile and configuration. At 10:22, it returns the earlier log and says the delivery is ready.

The 10:14 result may be real. It is stale for the final workspace until the affected check is rerun.

The smallest useful request is not “prove everything.” It is:

Rerun this command against the final revision and return the exit status plus the current revision identifier.

That request binds the result to the artifact without pretending one rerun certifies security, deployment, or production readiness.

Reproduction is a review action

For a low-risk change, reading the diff and rerunning one focused command may be enough. For data, security, billing, or release behavior, the reviewer may need additional checks and a human owner.

The important distinction is between supplied evidence and reproduced evidence:

Supplied evidence helps you decide what to inspect.
Reproduced evidence gives you a result from a check you controlled.

Both can be useful. They support different levels of confidence.

A narrow verdict is still a useful verdict

Use one of four outcomes:

Supported — the current artifact and reproduced check support the exact claim.
Unverified — the evidence is missing, stale, too narrow, or not reproduced.
Blocked — access, environment, or permissions prevent the next check.
Conflicted — the artifacts disagree.

This avoids turning “I cannot verify this yet” into “the code is bad,” while preventing a green log from silently becoming a broad release decision.

A reusable review prompt

Before accepting an AI coding handoff, ask:

What exact claim does this log support?
Which revision produced it?
What changed after the run?
Can I reproduce the important check?
What remains outside the evidence boundary?

I made the free AI Completion Evidence Auditor Lite as a manual worksheet for structuring that review. It does not run tests or authenticate supplied logs:

https://frankster8205.gumroad.com/l/ai-completion-evidence-auditor-lite?src=devto_us_article3_log_provenance_v1&utm_source=devto&utm_medium=organic_content&utm_campaign=reliable_ai_delivery_wave1&utm_content=devto_us_article3_log_provenance_v1

When a Retry Returns “Done” but Loses the Evidence

Reliable AI Delivery — Thu, 23 Jul 2026 19:21:44 +0000

A retry can recover a response without recovering the context needed to trust it.

Consider this fictional sequence:

A coding agent receives a task with three acceptance conditions.
It edits several files and starts a test.
The response stream fails before the handoff reaches the reviewer.
The system retries.
The reviewer receives a confident completion message, but not the original changed-file list or the complete test output.

The retry may have returned a fluent answer. That does not tell us whether the answer belongs to the same task state, the same workspace revision, or the same verification run.

This is not proof that the implementation is wrong. It is a provenance gap.

What was lost?

The useful question is not only “Did the request retry?” It is “Which parts of the delivery record survived the retry?”

For a reviewable handoff, preserve at least:

the original task and acceptance conditions;
the assistant’s exact completion claims;
the changed files or diff;
the exact command that ran;
the complete output and exit state;
the revision, branch, or working-tree state;
anything unverified, inaccessible, or conflicting.

If the retry returns only the final claim, the reviewer cannot safely infer the missing artifacts.

Do not turn missing context into a green result

Suppose the retry says:

“Implemented the change and all tests pass.”

But the current record contains no command, no output, and no link between the reported run and the current revision.

Three different statements are possible:

PROVEN: the supplied material directly supports the specific claim.
UNPROVEN: the claim may be true, but the required support is absent.
BLOCKED: the reviewer cannot obtain the required artifact or access the environment.

In this example, “all tests pass” should not become PROVEN merely because the sentence is confident. If the evidence is missing, keep it UNPROVEN or BLOCKED.

Ask for a correction packet, not a longer summary

A useful follow-up request is narrow:

Restate the exact claim being corrected.
List the current changed files or provide diff evidence.
Provide the exact verification command and complete result, or state that it was not run.
Identify the revision or workspace state that produced the output.
List known limitations and unverified areas.

The purpose is not to force the agent to repeat the whole conversation. It is to reconstruct the smallest evidence packet another reviewer can inspect.

When should a human rerun the check?

A supplied log can support a consistency review, but it is not automatically independent proof. The reviewer still has to decide whether the risk justifies a fresh run.

For a copy edit, the supplied record may be enough. For a data migration, authentication rule, billing change, security boundary, or production behavior, a human-owned rerun may be necessary.

The risk controls the depth of verification.

A practical manual workflow

The free AI Completion Evidence Auditor Lite is a worksheet for this kind of second pass. It helps compare one completion message with the task, changed-file information, and test or build output you actually have.

It does not run code, authenticate logs, or certify production readiness.

Free download:

https://frankster8205.gumroad.com/l/ai-completion-evidence-auditor-lite?src=devto_us_article2_response_loss_retry_v1&utm_source=devto&utm_medium=organic_content&utm_campaign=reliable_ai_delivery_wave1&utm_content=devto_us_article2_response_loss_retry_v1

How to Catch False “Done” Claims in AI-Generated Code

Reliable AI Delivery — Sat, 18 Jul 2026 07:36:52 +0000

AI-generated code can sound finished long before it is ready to accept. A message such as “tests passed” is a claim, not a verdict. Before accepting delivery, ask a more useful question: what evidence actually supports the claim that the requested work is complete?

Why “tests passed” is not the finish line

A passing test can be real and still be incomplete. It may cover only a narrow helper, skip the reported failure, use an old output, or test behavior that is adjacent to rather than identical to the requested change. A green result does not tell you which files changed, whether the relevant command ran, or whether the output belongs to the current work.

The goal is not to distrust every report. It is to separate a plausible statement from a supported one.

Use a claim → evidence → verdict check

A simple review loop makes that separation practical:

Claim: Write the completion statement precisely. For example: “The password-reset redirect now returns people to the sign-in page, and the regression test passes.”
Evidence: Gather the changed-file list, a diff summary, the exact test command, the test output, and any acceptance checks that matter for the goal.
Check: Ask whether the evidence is current, relevant, complete, and consistent with the claim.
Verdict: Mark the claim PROVEN, UNPROVEN, or BLOCKED. “Unproven” does not mean false; it means the available record is not enough to accept it yet.

This keeps a review focused. Instead of arguing about confidence, you can identify the next missing piece of evidence.

What to inspect before accepting a completion claim

Changed files

A changed-file list tells you whether the reported work touched the expected area. It can also reveal scope drift: a tiny fix that unexpectedly changes deployment settings, unrelated dependencies, or authentication code deserves a closer look.

Test output

Useful test evidence includes the command that ran, the result, and enough context to connect it to the current change. “All tests passed” is less useful than a specific command and its output. The strongest evidence also makes clear which scenario was exercised.

Missing evidence

Missing evidence is often the whole story. If a report says a browser flow is fixed but supplies only a unit test, the relevant end-to-end evidence is still missing. If a report claims no regressions but only one focused test ran, broader coverage may remain unproven.

Contradictory evidence

Look for conflicts instead of averaging them away. A success message beside a failing log, a claim that nothing changed beside a diff, or a test output from before the latest edit should move the verdict away from PROVEN until the contradiction is resolved.

A small example

Imagine an assistant reports: “The password-reset redirect is fixed and tests passed.” The supplied record contains one unit test for URL construction, but no changed-file list, no browser or route test, and no output for the reset flow.

The correct verdict is UNPROVEN. That does not say the fix is wrong. It says the available evidence does not yet justify accepting the full completion claim. The next action is clear: provide the changed files and run a test that exercises the reset flow itself.

Evidence helps humans decide what to verify next

An evidence check is not a substitute for engineering judgment. It helps a reviewer see what is supported, what is missing, and what conflicts with the report. That makes it easier to ask for a narrow follow-up instead of reopening the entire task.

There is one important limitation: a tool that reviews an evidence packet does not re-run commands or independently verify test output supplied by a user. A log can be organized and challenged, but it does not become trustworthy merely because it was pasted into a report. When the risk is high, the right next step may still be to reproduce the result in a controlled environment.

Start with a lightweight evidence check

If you want a structured way to label completion claims before you accept delivery, try the free Auditor. It is designed to surface unsupported “done” statements, missing proof, and contradictions without pretending that an unverified record is a completed result.

Get the free AI Completion Evidence Auditor Lite:
https://frankster8205.gumroad.com/l/ai-completion-evidence-auditor-lite?src=devto_us_article1_v1