The Agent That Writes the Code Cannot Be the One That Certifies It

#ai #llm #testing #python

Every agent demo ends the same way

The model writes the file, runs the thing, and prints "done". Nobody checks.
So the deliverable that looked finished at 2am turns out to be a file that does
not exist, a test that was never run, and a summary that was confident.

We hit this enough times to stop trusting the word of the agent that did the
work. Not because the model is lazy — because an LLM cannot certify its own
artifact. It declares success from the inside, where it cannot see the failure.

Three failures, one root cause

Building a validation loop for our agents surfaced three distinct ways it breaks:

Forged self-evaluation. The model that generated the artifact is the model evaluating it. Same context, same blind spots. It says "looks correct" about a diff it never executed.
FoFo — Found on False. The AI must not correct its own proof. Whoever writes the code cannot write the test that validates it, and whoever judges cannot share context with whoever built.
Cheatable verification. If the verifier lives where the agent can reach, the agent forges the verifier. A fake pytest.py that returns 0 is not a hypothetical — it is the obvious move.

The first two are about authority: only external evidence can declare DONE.
The third is about reach: the evidence has to be produced somewhere the
Builder cannot touch.

The held-out gate

So the architecture splits the loop and puts the proof outside:

Task ──▶ [SOUL / Builder] ──▶ [Held-out Gate] ──▶ [Judge / Reviewer] ──▶ Envelope
              ▲                    │ FAIL              │ REJECT
              └──── required_fix ──┘                   │ (max_attempts → ESCALATED)
                     (re-gate, strict)                 ▼ APPROVE → SUCCESS

Builder / SOUL. Drafts from persona + procedural memory + task. It may reference a secret by name ($VAULT_X), never by value.
Held-out Gate. The unreachable environment. Runs the real tool — HTTP over curl, nc, ffprobe, an EXPLAIN dry-run, HMAC-SHA256, a declared regex — and returns (passed: bool, log: str). Evidence is re-run on every attempt. A green gate from round 1 is never reused in round 2.
Judge / Reviewer. A session without the Builder's history, confronting draft + gate log against the injected rules. It emits a routing verdict, not the final text. Rejection injects required_fix into the next Builder prompt, invisible to the human channel.

The contract is typed (Pydantic v2 — no LangChain, no LangGraph, decision
documented): the delivery envelope carries soul_id, a status restricted to
SUCCESS | ESCALATED_TO_DM | TOOL_FAILURE, and an audit trail with
self_correction_count, rules_applied, latency_ms, hedge_warning and the
raw gate_log.

The golden rule of the loop

The LLM may suggest it finished. Only external evidence may declare DONE.

SUCCESS exists only when gate_pass == true and judge_pass == true.
Everything else terminates as ESCALATED_TO_DM (attempt ceiling) or
TOOL_FAILURE. The agent stops where the authorization ends — which we proved
on a live pilot that hit a real 404 and returned no invented payload instead of
hallucinating one.

What it takes to trust the tester

Gates come in two kinds, both declared in one registry:

subprocess: argv rendered with the artifact, executed as a host subprocess.
static: an in-process deterministic validator (a MEDIA: prompt regex), still outside the LLM interpreter and still declared in the registry.

Parameters are injected through string.Template, so shell/HTTP/JSON literals
(%{http_code}, {"exact":false}) never collide with interpolation.

The harness measures itself with a pure function, no LLM in the middle, across
six dimensions — deterministic contracts, orchestrator self-test, end-to-end
orchestration, cost policy, live operation, documentation. Each dimension
returns one of four states (PASS | FAIL | NOT_VERIFIABLE | ERROR), and the
verdict is derived, not judged:

APTO          all dimensions PASS and total >= 90% of the (fixed) ceiling
COM RESSALVAS no FAIL/ERROR, but something is NOT_VERIFIABLE (>= 70%)
REPROVADO     any FAIL/ERROR, or total < 70%

Unexecuted dimensions stay in the denominator, and exit codes are 0 / 1 / 2 —
so CI can call it. Inconclusive never passes the gate: that rule came from a
broken machine that would otherwise have shipped green.

Where it stands

Honest numbers, all from local measurement:

44 pytest tests green
2 end-to-end pilots (two unrelated projects, same engine)
1 real delivery to a human channel (message_id=43843)
cost policy enforced: the gate never calls a paid model

Not a framework. No dependency mountain, no agent-of-agents marketing. The
whole claim is narrow on purpose: it does not make your agent smarter, it
makes "done" mean something.

The open-source version of the loop engine, including the gate registry and the
quality harness, is here:

→ Download the free AI Test Loop skill

DEV Community