How AIClaw's Harness Runtime Stops Premature "Done" Answers

AI agents are good at sounding finished before they have actually finished. They can say a file was generated when no file exists, claim a task is blocked without showing the failed tool call, or return a progress update as if it were a final answer.

AIClaw's recent harness runtime work addresses that gap by moving completion from "the model said it is done" to "the execution layer has enough contract and evidence to allow completion."

This article is based on the current AIClaw repository, especially the harness runtime design and the July 6 and July 9 agent changes that added the verifier layer, execution evidence ledger, explicit finish tool, plan bootstrap, and final-answer gates.

The Problem: LLM Confidence Is Not Execution Truth

In a tool-using agent, the model can produce a polished answer even when:

• a required tool was never called
• a tool failed and the final answer ignores the failure
• a promised attachment was never created
• the task plan is still incomplete
• the assistant is only narrating progress instead of delivering a result

AIClaw already had runtime plan state, tool execution, generated files, and execution logs. The new harness runtime ties those pieces together with a validation layer that decides whether a run can proceed, needs correction, or is allowed to finish.

The Runtime Model

The core pipeline in pkg/harness is:

Contract -> Evidence -> Validate -> Correct

In practice that means:

• TaskContract derives what the run is expected to do.
• EvidenceLedger records what the run actually did.
• validators check whether the current state satisfies the contract.
• correction prompts push the model back into execution when the answer is incomplete.

This is not a separate executor. The main loop still lives in the agent runtime. The harness adds staged validation and correction around that loop.

What The Contract Captures

The contract is derived from the user objective and current execution context. According to the current design and code, it can incorporate:

• the user goal
• runtime plan requirements
• file-delivery intent
• tool evidence expectations
• attachment context
• sub-agent context
• correction budget

That matters because "done" means different things for different tasks. A question-answering turn might only need a grounded text answer. A file-generation turn needs actual artifact evidence. A multi-step task may require an active plan and a terminal plan outcome before the final response is allowed.

What Counts As Evidence

AIClaw's verifier collects structured execution evidence rather than relying on the final assistant message alone. The evidence ledger includes:

• business tool calls, excluding internal control tools like plan, tool_search, and finish
• tool events with tool name, argument summary, output summary, status, duration, files, and failure classification
• generated file artifacts persisted as AIClaw files
• plan snapshots from the active plan manager
• validation and correction events
• blocker evidence such as permission, auth, policy, timeout, rate limit, or not-found failures

This gives the final gate something concrete to inspect. If a tool failed, the harness can tell whether it is a recoverable timeout or a terminal auth problem. If a file was promised, the harness can check that file evidence actually exists.

The Four Validation Gates

AIClaw now validates at four different points in the run:

Stage	What it checks
pre_tool	Whether a tool call is allowed before execution
post_tool	Whether the current tool round produced the evidence the run now has
pre_final	Whether the candidate final answer is truly ready to finish
pre_save	Whether the final content is still valid after attachment links are rendered

This staged approach is the important design choice.

Many agent systems only validate at the end. AIClaw also validates before tool execution and between tool rounds, so it can keep the model inside the task until it either gathers enough evidence or reaches a real blocker.

What Gets Rejected Before Final Answer

The current runtime can stop finalization when the candidate answer is:

• empty
• only a progress update
• missing required successful evidence
• ignoring a terminal blocker
• missing promised artifacts
• trying to finish while the plan is still open

The logic is visible in the harness design and outcome classifier. Candidate answers are assessed as success, blocked, partial, progress_only, or unknown, then compared with the evidence ledger.

That makes the final gate more than a string check. It is deciding whether the answer matches what the run actually accomplished.

An Explicit finish Tool Instead Of Implicit Guessing

One practical addition is the built-in finish tool.

Instead of relying only on "no more tool calls means we must be done," the model can explicitly submit the final user-facing answer through finish. AIClaw captures that answer, runs the same final gate, and only closes the turn if the harness allows it.

This is a clean separation:

• the model proposes the final answer
• the harness decides whether the answer is allowed to end the run

The finish tool is also excluded from normal business-tool evidence, which prevents completion signaling from being counted as actual execution work.

Plan State And Harness Runtime Work Together

AIClaw already uses runtime Plan State instead of chat-visible todo lists. The recent harness work tightens that integration.

If a contract requires a plan and there is no active plan, the runtime can bootstrap one from an initial template. During execution:

• only one plan item is allowed to be running
• successful rounds can advance the active item
• tool or LLM failures can mark the running item as failed
• the final answer is linked to the final plan snapshot

This is useful because the harness is not only validating text quality. It is validating whether the run reached a terminal execution state.

Better Failure Semantics

The runtime now classifies blocker types such as:

• permission denied
• auth failed
• policy blocked
• not found
• rate limited
• timeout
• generic tool error

It also distinguishes recoverable from non-recoverable failures. A timeout can lead to another attempt or a correction loop. Missing permissions or invalid authentication should be surfaced clearly as blockers instead of being buried under a generic apology.

That makes the execution log more operationally useful, especially when you are debugging real tool-using agents.

Observable By Design

When validation or correction fails, AIClaw records harness-specific execution steps such as:

• validate_pre_tool
• validate_post_tool
• validate_pre_final
• validate_pre_save
• correct_pre_final
• correct_post_tool
• continue_execution
• recover_llm_round

The metadata includes the stage, violation codes, required actions, evidence summary, and correction outcome. Successful validation without violations stays quiet to avoid polluting the trace.

That is a good tradeoff: rich debugging data when something goes wrong, low noise when the run is healthy.

Why This Matters In Daily Use

For users, this changes the feel of an agent in a practical way:

• file-delivery tasks are less likely to end without actual files
• long tasks are less likely to stop at a narrative progress update
• tool failures are more likely to be explained with actionable blocker context
• execution logs become a better source of truth than the assistant's confidence

For developers building on AIClaw, the bigger value is architectural. The stable pkg/harness package gives you a place to evolve contracts, evidence, validation, and correction rules without rewriting the whole agent loop.

A Small But Important Shift

The key idea behind this feature is simple:

an agent should not be considered done because it sounds done. It should be considered done because the runtime can prove it did the work, produced the promised artifacts, or reached a well-explained blocker.

That is the shift AIClaw's harness runtime is making.

If you are building self-hosted agents that need to do more than chat, this is one of the most important layers to get right.