CodeSmith Meta-Ability: All-Green Tests Only Count After a Replay

Meta-Ability: "Tests All Green" Only Counts After You Replay It

Source version of CodeSmith: v0.5.0 (commit 3a74c82f). All paths are relative to the repo root; line numbers refer to this version.
Intended audience: anyone who has been handed an agent's report that "all the tests passed," then watched everything turn red the moment they ran the tests themselves.

When the model says "tests all green," what are those words worth? v0.5.0's answer: at the instant a turn ends and the model has just finished saying it, the engine replays, exactly as-is, the verification commands the model itself ran that turn, and reconciles the books against the exit codes. The verdict is written back into the conversation as a four-tuple, on by default, in a module of 914 lines. Commits ab45cdee and 39e77390 together closed the gap where "evolution had records but no closed loop" — how that loop actually turns is gathered up in Article 23.

The Going Rate of "Tests All Green"

Anyone who has put an agent to work knows the scene: the model finishes changing the code, runs the tests once, then writes in its summary "all tests passed." You trust it, you commit, CI goes red. Only when you dig through the logs does it come out — it ran the tests in a stale directory, skipped the failing cases, or never ran anything at all; it just rounded the sentence off the way it always does.

This is not a moral failing; it is the factory setting of a probabilistic brain. Article 12 made the case: deterministic problems belong to the interpreter; the LLM is fit only to be the translator. If parameters can be handled that way, why not conclusions?

The ROADMAP keeps the ledger entry (ROADMAP.md:16-17): an audit against field notes found that "continuing evolution has records but no closed loop," and the improvement item was numbered P3-8. The slop ledger booked the debt, MEMORY.md stored the lessons, but nothing ever laid the model's "claims" and the world's "facts" on the same table to reconcile them.

With v0.5.0, the reconciliation begins. The first paragraph of the module docs stakes out the position (crates/agent-runtime/src/engine/result_verifier.rs:1-9): this is the first domino of the continuing-evolution closed loop, the one verifier in the whole set "most deserving of being built first."

The Verdict Four-Tuple

Start with the shape of the output. The design notes require every verdict to carry four elements: conclusion, dimension, evidence location, failure type. Landed in code, that is four verdict kinds (result_verifier.rs:366-389):

pub(crate) enum VerdictKind {
    VerifiedPass,
    VerifiedFail,
    /// The re-run itself did not complete (timeout / sandbox denial / tool
    /// unavailable). Never reported as pass or fail — an unverifiable check
    /// must not masquerade as either.
    VerifyError,
    /// The model claimed success but executed no verification-class command
    /// during the turn — the highest-value signal this checker produces
    /// (a claim with no execution evidence behind it).
    Unsubstantiated,
}

Plain-English translation: verified-pass and verified-fail need no explaining; verify-error is "I could not verify" — the replay timed out, the sandbox refused, the tool was unavailable, and none of these may masquerade as pass or fail; unsubstantiated is "you claimed success, but this turn never ran a single verification-class command."

The fourth kind is the most valuable signal in the room. Its detection cost is zero — nothing needs replaying; the check only has to notice that no execution evidence stands behind the claim. The comment in the config file puts it more bluntly (config.example.toml:720-721): this is the strongest hallucination signal this checker produces, because it does not even need a re-run.

The rendering of the verdict text obeys the four-tuple as well (result_verifier.rs:417-455): a verdict line, a dimension: result line, an evidence: line (the replayed command, the original tool_use_id, the exit code, the head of the output), plus failure_type: test-failure on failure.

Replay Only, No New Authorization

The most important thing in this module is not "what gets verified" but "by what right it gets verified at all." Replaying a command the model ran means the engine executes a shell behind the user's back one more time — why doesn't that ever end badly?

The safety contract is written into the module docs (result_verifier.rs:17-34), and clause one is named approved-replay only: the replay travels through the conversation's own tool dispatcher, reusing character-for-character the input the model already sent this turn — same command, same cwd, same timeout. Verification grants no permission the turn did not already have: this input cleared the approval gate once; now it is simply being played again.

Clause two: the eligibility gate. Not just any exec_shell history can be replayed; only commands that begin with a known test/build prefix qualify. The test-class whitelist holds 20 prefixes, the build class 14 (result_verifier.rs:87-128), from cargo test to dotnet test. Compound commands are split on && and ; and judged segment by segment, so cd crate && cargo test counts too; a VAR=value environment assignment at the head of a segment is stripped off first.

The whitelist's inclusions and omissions show real craft (result_verifier.rs:111-112): the build class deliberately excludes bare make (it would match make install, make clean) and cargo clippy (that is a lint, not a build). Background invocations are never replayed — replaying task_shell_start would mean launching a detached process all over again.

The claim detection itself has its refinements (result_verifier.rs:47-53): it uses contiguous-phrase matching rather than regexes, because "测试未通过" does not contain the substring "测试通过" and "tests failed" does not contain "tests pass" — a negation naturally breaks the phrase apart. One residual false positive is known, "not all tests pass"; the cost is one extra replay and one more honest exit code on the books.

How the Verdict Returns to the Conversation

Computing the verdict is only half the job; how it gets delivered back into the context is where the design shows. The module docs prescribe the route: flow like LSP diagnostics (result_verifier.rs:11-15) — at end of turn the verdict hangs on a pending buffer, and before the next API request it is flushed in as a synthetic <codesmith:runtime_event> user message, appended at the tail only, never rewriting history.

Why so roundabout? Because the rule from Article 9 still stands: an append-only conversation history is the KV cache's lifeline. Insert the verdict into the middle of history and the prefix cache is scrapped wholesale; hang it on the tail, and the model still has to face the bill head-on — the moment it opens its mouth next turn, it must first explain why it said all green last turn while the replay came back red.

On the engine side, the wiring is exactly two hook points. The end-of-turn trigger condition is one line (crates/agent-runtime/src/engine/mod.rs:1695): the final assistant message is scanned only when result_claim_verifier is on and the turn state is Completed.

The flush happens before the request (crates/agent-runtime/src/engine/turn/postprocess.rs:209): the buffered verdict is rendered into one runtime_event and advanced into history.

Ledger and Readings

A single verdict is for the model to see; the long-term ledger is for humans. 39e77390 added the observation layer: every verdict is appended to ~/.codesmith/evolution/verdicts.jsonl, one JSON per line — timestamp, verdict, the claim verbatim, the command, the exit code, and which model said it (crates/agent-runtime/src/evolution_log.rs:33-45).

The logging is best-effort: if path resolution fails or the write fails, it is WARN-logged and swallowed, because "the log must never break the engine that produces the events" (evolution_log.rs:62-63). Malformed lines are skipped on read — a half-line truncated by a crash must not be allowed to bury all the rest.

The readings come from /verify stats (crates/tui/src/commands/verify.rs). The headline metric is the claim-match rate, and its accounting basis is written into a comment (evolution_log.rs:150-157): the denominator counts only verifiable claims that truly had a command behind them; the numerator counts the ones that still stand after the replay:

pub fn claim_match_rate(&self) -> Option<f64> {
    let checked = self.verified_pass + self.verified_fail;
    if checked == 0 {
        return None;
    }
    Some(self.verified_pass as f64 / checked as f64)
}

Plain-English translation: this accounting basis is honest. unsubstantiated never enters the denominator — mix "never verified at all" and "verified but crashed" into one ratio, and the reading loses its ability to tell hallucination types apart. Where a metric draws its denominator exposes what the system wants you to see.

Closing: The First Domino of the Closed Loop

Chain the mechanism into a single thread:

1. The turn ends normally; the engine scans the final assistant message, and phrase matching catches claims of the "tests passed / build succeeded" class;
2. It searches this turn's history in reverse for the most recent verification-class command (20+14 prefixes, segment splitting, stripping environment assignments, excluding background calls);
3. The command is replayed character-for-character through the conversation's own dispatcher — same command, same cwd, same timeout, no fresh authority granted;
4. The exit code settles into one of four verdicts; the four-tuple is rendered as a runtime_event hung on the conversation's tail, and the model must reconcile next turn;
5. The same verdict lands in the jsonl ledger, where /verify stats serves up the per-class counts and the claim-match rate.

Three properties, and their prices:

1. On by default: trust is no longer free — the price is one extra replay for every model fond of declaring victory, and repos with slow tests will want to weigh the [verification] result_claims switch;
2. Replay only, no new authorization: the security boundary stays conservative — the price is that when the model never ran a verification command, the engine cannot run one on its behalf; all it can report is unsubstantiated;
3. The verdict is evidence, not an instruction: the engine rewrites no prompts and touches no approval gate — the price is that course correction depends on the model settling its own account, and with a model that refuses to blink, the bill takes a few more turns to collect.

The four meta-ability installments close here. Look back, and you will notice they kept landing on the same foundation: verdicts may only hang on the conversation's tail, snapshots only append, the ledger is append-only — any byte written into context or history may never be quietly changed. Why that discipline is worth the 100× price tag on a single byte, the four context installments before this (8-11) have already reckoned in detail. The next set of questions returns to the pair of hands that pour content into the context — the tools themselves, and how they are built.