DEV Community

hefty
hefty

Posted on

Your Coding Agent Needs an `unverified` State

Imagine a coding agent patches a checkout bug. The focused unit test passes. The process exits cleanly. Its final message says the task is done.

One problem: the request was about a browser checkout flow, and nobody ran that flow.

The patch may be correct. It may also fail when the payment form, navigation state, and browser runtime meet. The agent has evidence for one claim and no evidence for another. Calling the whole task successful turns a gap in verification into a green badge.

Coding-agent systems need an explicit unverified outcome. A run can finish without proving every requested behavior, and the interface should say so.

"Done" is carrying too many meanings

Most agent workflows collapse several events into one status:

  1. The process stopped.
  2. The tool changed something.
  3. The requested behavior was accepted.

Those events are related, but they are not interchangeable.

A zero exit code tells you that a process completed according to its own exit convention. Reading the file back tells you that bytes were written. A passing unit test tells you that its named assertion held for the candidate and environment it checked. None of those facts automatically proves the whole task.

The gap gets easier to miss as agents gain more tools. An agent can edit files, launch commands, inspect output, and produce a polished summary. More activity creates more observations. It does not turn those observations into acceptance evidence by itself.

The workflow needs two axes:

  • execution_status: Did the attempt finish, fail, or remain blocked?
  • claim-level verification: Which requested behaviors were verified, contradicted, or left unverified?

An attempt can be finished while one of its claims remains unverified. That is not a system error. It is an honest description of the available evidence.

Track claims, not confidence

A single task badge hides too much. Break the request into claims that can be checked independently.

For the hypothetical checkout patch, the claims might be:

  • the price calculation returns the expected total for a named case
  • the payment button becomes enabled after valid input
  • the browser completes the requested checkout path
  • the patch does not change an unrelated account flow

These claims may not all need the same evidence. A focused unit test could cover the calculation. A browser check could cover the interaction. A diff review might show that the patch never touches the account flow, although that still would not guarantee the absence of indirect effects.

Each claim gets one of three outcomes:

  • verified: a named check produced evidence for the claim
  • contradicted: a named check produced evidence against the claim
  • unverified: the workflow does not have enough relevant evidence either way

Unverified is not a softer word for failure. No check was run, the environment was unavailable, or the available check did not cover the claim. Contradicted means relevant evidence exists and points the other way. Mixing those outcomes makes repair routing worse: one needs a check, while the other needs a code change, a changed requirement, or rejection.

Claim-level outcomes prevent the opposite mistake too. One failed check should not stain every claim in the task. If the calculation test passes but the browser flow fails, preserve both facts. The system now knows what not to redo and where the repair belongs.

Use a small verification record

The record does not need to become a new observability platform. It needs enough structure to stop prose from laundering uncertainty.

Here is an illustrative format, not an existing protocol or a claim about any project's API:

execution_status: finished
candidate_ref: "<exact patch or revision under review>"
claims:
  - id: checkout-total
    expected: "<named calculation behavior>"
    outcome: verified
    evidence: "<focused test artifact>"
  - id: browser-checkout
    expected: "<requested browser behavior>"
    outcome: unverified
    evidence: null
    gap: "<browser environment was not available>"
next_action: "<run the named browser scenario against this candidate>"
Enter fullscreen mode Exit fullscreen mode

The angle brackets matter. They keep the example from looking like a real execution log. In production, those references must point to actual artifacts created by the workflow.

A useful record answers four practical questions:

  • What exact candidate was evaluated?
  • What did each check claim to establish?
  • Where is the evidence?
  • What bounded action would resolve the remaining gap?

It should not say all tests passed when only one test command ran. It should name the check and the requirement it covers. It should not use a file hash as proof of behavior. A hash can identify the candidate; it cannot tell you whether checkout works.

Evidence expires when the relevant candidate changes

Verification belongs to a candidate, not to a conversation.

Suppose the browser flow passes, then the agent edits the checkout component again. The old browser evidence may no longer apply. Keeping the green result attached to the task rather than the checked revision creates a stale proof problem.

That does not mean every edit invalidates every check. The workflow should connect claims to the files, runtime surfaces, and assumptions they depend on. A documentation correction probably does not invalidate a calculation test. A change to shared form state may invalidate both unit and browser evidence.

The useful rule is narrower: when the candidate changes, reassess the evidence for affected claims. Preserve unaffected evidence and rerun or downgrade the rest.

Aiden's published design separates attempts, effects, evidence, verification, and verdicts. The separation lets an effect record survive without being mistaken for a verdict. A parent process can inspect what a worker changed, then decide whether the supplied evidence covers the assigned claim.

Parent agents must verify the handoff

Multi-agent systems make false completion easier.

A helper returns a confident paragraph: "Implemented the fix and verified the result." The parent sees clean prose, not the boundary of the check. If it accepts the sentence as evidence, the hierarchy only amplifies the helper's confidence.

The parent should evaluate the handoff against the assignment:

  • Is there a candidate artifact to inspect?
  • Does the evidence belong to that candidate?
  • Does the check address the assigned claim?
  • Are unresolved gaps still visible?

This is not a demand for the parent to rerun every command. It is a demand that the parent judge evidence, not tone. A worker's successful exit proves that the worker stopped successfully. Its summary may help a human read the record, but the summary is not the record.

The same rule applies to human review. Architecture, security, and product behavior often involve judgments that a focused test cannot settle. The system should route those claims to review instead of silently treating machine-checkable evidence as universal approval.

Route the gap instead of hiding it

An unverified state can look like unfinished work. The work was already unfinished; the state only stops the interface from hiding that fact.

A good unresolved handoff should include:

  • the last candidate that was checked
  • the claims that were verified
  • the exact claim that remains open
  • the evidence collected so far
  • the reason verification stopped
  • one safe next action

The reason matters. Missing browser infrastructure calls for a different response than missing authority to use a payment sandbox. A time budget expiring is different from a check returning a contradiction. If the workflow preserves those distinctions, the next operator can continue without reconstructing the task from a transcript.

An unresolved handoff also needs a stop condition. If the required environment does not exist, the agent should not invent validation or keep retrying unrelated commands. It should preserve the candidate, name the blocked claim, and stop at the authority or infrastructure boundary.

Preserving the gap gives the next step a typed input instead of a vague success message.

Bring the missing check back into view

Return to the checkout patch. The unit test passed, so the calculation claim can be verified for the checked candidate. The browser flow was never run, so that claim is unverified. The task is not globally green, and it is not globally failed.

The next action is small: run the named browser scenario against the same candidate. If it passes, attach the evidence and update that claim. If it fails, mark the claim contradicted and route the patch back for repair. If the environment remains unavailable, keep the gap visible for a person who can resolve it.

A coding agent should be allowed to finish its run without pretending it proved the result. An honest unverified state gives the next developer something better than confidence: it gives them the exact boundary of what is known.


Source notes

Top comments (0)