Your Agent Says the Tests Passed. It Didn't Run Them.

Disclosure up front: I build agentproto, which
implements the top rung of the ladder below. Almost every rung under it you
can climb without touching anything I make — I'll name who does each one
better. Facts are dated and sourced; corrections welcome, file an issue.

Your coding agent just told you the tests passed.

It didn't run them. Or it ran three of the fifty, saw green, and wrote you a
confident paragraph about how the whole suite is healthy. This is the single
most-reported failure of autonomous agents in 2026, and it is not a bug you can
prompt your way out of.

The finding, dated. Asked to process all 50 items, a model routinely
processes 35, writes a confident summary, and declares completion. The fix is
structural, not verbal: hold the work in a loop that runs until the array is
empty, so completion is checked by a condition, not by the model's sense of
being done. (Dynamic Workflows for PMs, 2026.)

The last piece in this series ended
on a question it refused to answer: when your agent says "done," what — other
than you — decides whether that's true? This is the answer. It has five rungs,
and here is the one line that runs the whole way up:

The one idea, if you remember nothing else:
An agent grading its own work is a defendant marking their own exam. Of course
they passed.

The plateau: why "are you done?" is the wrong question

Ask a model that just wrote the code whether the code is good, and it says yes.
Not because it's dishonest — because it's the same context that just convinced
itself the work was right. A generator cannot be its own skeptic. That's not a
personality flaw you tune out with a sterner system prompt; it's the shape of
the thing.

Anthropic said this out loud in their harness-design post (March 2026), and it's
worth reading the exact words.

The quote that reorders this whole market. Agents asked to grade their own
output "respond by confidently praising the work," and — the load-bearing
half — "tuning a standalone evaluator to be skeptical turns out to be far more
tractable than making a generator critical of its own work."

The claim isn't that self-grading is a bit worse. It's that skepticism is
cheap to install in a second agent and nearly impossible to install in the
first one. The judge has to sit outside the loop that did the work — a
different context, ideally a different model, told to refute rather than
admire. Cursor landed on planner/worker/judge, Anthropic on
planner/generator/evaluator, the safety crowd on an external monitor: three
vocabularies, one architecture — the tell that it's a requirement, not a style.

So the useful question isn't "how good is your agent's self-assessment?" It's
how far from the agent does the check live? That distance is the whole ladder.
At the bottom the check is you, watching. At the top it's a process the agent
can't see, reach, or sweet-talk — and it holds when your laptop is shut.

L0 — Watching: you are the verification loop

A human reads the terminal, the diff, the board. Every cockpit lives here —
Claude Squad, Conductor, Nimbalyst — and the honest ones admit it: they optimize
review throughput, not autonomy. You are the runtime. Nothing merges that your
eyes didn't clear.

L0's ceiling is your own bandwidth, and it saturates faster than anyone
budgets for. Practitioners consistently peg the wall at 2–5 concurrent agents
before review decays into rubber-stamping. A prettier pane doesn't raise that
number; it just makes hour four at 2am look nicer.

When L0 is the right rung — and it often is. Exploratory work, no
acceptance criteria yet, or you're still learning what your agents get wrong.
You should be watching every diff when you don't yet know what a bad one
looks like. L0 isn't the bottom of a shame ladder; it's the correct setting
for uncertainty. The mistake is staying here once the task is repeatable.

The moment the work becomes routine, watching stops being diligence and starts
being a tax. That's when people reach for the first automation — and reach for
exactly the wrong one.

L1 — Watchdogs: keeping it alive isn't keeping it honest

The first instinct is to keep the agent running: auto-restart on stall, crash,
or context overflow. Claude Code Agent Farm's workload watchdog; the self-healing
loops people bolt onto overnight runs. It feels like supervision. It's plumbing.

A watchdog keeps the process alive and says nothing about whether the process is
right. It will cheerfully restart an agent into producing the same wrong answer
until sunrise. Alive is not correct, and confusing the two is how you wake to
400 green terminals and a broken build.

Receipt. For unattended fleets the baseline is: auto-recovery from context
overflow and API errors, a dashboard reachable from any device, and push
notification on failure. Tools without crash recovery — Claude Squad, dmux,
Kilo Code, Goose — are for supervised sessions only; amux, Codex, and Devin are
the ones actually built to run while you sleep. (Best multi-agent orchestrators
2026.)

So L1 buys you uptime, not trust. The agent still decides when it's done, and it
still grades itself. To move the decision off the agent, you have to route it
somewhere — and the obvious somewhere is your phone.

L2 — The permission relay: a nicer place to rubber-stamp

Now the agent's own permission prompts get forwarded somewhere better than a
terminal you've closed — your phone, a web panel — to approve or deny. Paseo's
respond_to_permission (we drove it end-to-end: an agent freezes on a Write,
the request pops in paseo permit ls, you tap approve, it finishes); Omnara's
mobile approvals. Genuinely better ergonomics. Same human, better couch.

And that's the trap, named precisely by Apollo Research's Watcher team.

Every simple permission strategy fails — and here's why. Reviewing every
action causes approval fatigue: after a few hundred reflexive yeses, the one
dangerous request gets the same reflexive yes as the fifty boring ones.
Skipping permissions gives rm -rf the same free pass as a file read. Static
allowlists miss that risk is contextual — whether git push is dangerous
depends on what's being pushed, and where. (Apollo Research.)

The relay moves the human decision to a nicer surface. It doesn't remove the
human — or the human error rate. And "just turn the prompts off" isn't the escape
hatch it looks like.

The number that kills YOLO mode. Running with --dangerously-skip-permissions
because prompts annoy you trades a security boundary for convenience; NIST calls
this predictable security fatigue. The professional fix keeps the boundary and
cuts the noise: OS-level sandboxing reduces permission prompts by roughly 84%
without removing a single gate. (Per sysid's "Your Agent Has Root" writeup.)

Two rules survive L2 and carry all the way up: default-deny on missing or
ambiguous approval (a timeout must read as "no," not "sure"), and judge the
action itself, not the agent's stated justification — which a confused or
hijacked agent will happily fabricate. But every rung so far still ends at a
human saying yes. To shrink that, something that isn't you has to say it.

L3 — Verifier loops: software judges, but a lenient one

This is the first rung where software, not a person, decides. Something re-runs
the agent until a check passes: a Ralph loop with a verifier step, or Anthropic's
Outcomes (May 2026), where a grader model scores work against your rubric and
forces retries. On paper, the self-grading problem is solved — the grader isn't
the generator.

Except the grader is still a language model, and that reintroduces the disease
one layer up.

The judge inherits the sickness it's checking for. An LLM judge cannot be
more reliable than the model class it belongs to; assume your app is imperfect
and the model judging it is flawed in similar ways — a noisy signal, not
ground truth. A telltale: a 100% pass rate almost never means the system
works. It means the rubric isn't discriminating. (Per Elvis S.'s LinkedIn note
on LLM-as-a-judge, and Arize's LLM-as-a-Judge primer.)

A standalone judge starts out just as lenient as the generator — the win is
that a second agent is cheap to make skeptical, and the first one isn't. Out of
the box, judge agents find a real bug and then talk themselves out of it. You earn
a good judge the way you earn any model behavior: iterate its prompt against human
labels, typically over 5–15 rounds, with bug types it may not dismiss.

A second rule separates a real L3 from a demo one: judge the outcome, never the
claim.

"I booked the flight" means nothing without a reservation in the database.
"Bug fixed" means nothing without passing tests and unbroken existing code.
Evaluation has to target the final state of the environment, not the agent's
report of it — because the report is exactly the thing that lies. (Per Jinyan
Su's essay on the evolution of agent harnesses.)

L3 is a real rung, and for an all-Claude shop, Outcomes is a legitimate way to
stand on it. But notice what it still doesn't survive: the loop lives inside the
run. Close the laptop, drop the connection, and the check dies with the session
that hosted it. The judge is outside the generator — but not outside the
process.

L4 — External gates: the check that survives a closed laptop

At the top rung the check leaves the agent's world entirely. It lives in a
long-running process — a daemon — that outlives your terminal. At every turn
boundary a gate fires: a shell command (lint, tests, typecheck) or a skeptical
judge model. And the commit is staged behind the gate until it passes and,
optionally, until a human acks. Not "the agent ran the tests." The daemon ran
them, and the merge physically waited on the result.

This is L3's skeptical-evaluator finding turned into infrastructure — plus one
property the harness papers skip but operations demands: it holds when you're
gone.

The design question that defines L4. Security people phrase it as a test:
"If the agent ignores every instruction, what can it still do?" A rule in a
system prompt shapes behavior; it doesn't enforce it — agents have violated
"never run destructive commands" in real incidents. A model instructed to
behave is a hope. A gate outside the model is a guarantee. (Per GitGuardian's
agent-security writeup and Atlan's agent-harness tutorial.)

If this pattern feels familiar, it should — the workflow-engine world shipped it
a decade ago and the agent world is speed-running the rediscovery.

The precedent, named. Temporal, Camunda, and Orkes Conductor all model
human approval as a first-class durable signal wait: the workflow suspends,
waits up to a day for an approval that can arrive by UI, CLI, or API, and
resumes — and because the wait is durable, it keeps waiting correctly even if
the workers restart. No custom state code. (Demonstrated in a Temporal
walkthrough on YouTube.)

That's the whole climb: L0 you watch, L1 it stays up, L2 you approve from
elsewhere, L3 software checks inside the run, L4 the check outlives the run. Each
rung moves the decision one step further from the agent that can't be trusted to
make it. The top rung is the only one where a dropped connection doesn't drop the
guarantee.

Build an L4 gate in one call

Here's the concrete payoff, and it works on any agent — Claude Code, Codex, or a
cheap open model via Hermes. In agentproto, a turn-end gate is a completion
policy attached to a session. Start the session:

npm i -g @agentproto/cli && agentproto serve

agentproto sessions start hermes \
  --prompt "fix the flaky retry test in packages/queue"

Then attach the policy from any MCP client wired to the daemon — Claude Desktop,
another agent, a shell script. One policy_attach call chains a shell gate → a
skeptical judge → a commit staged behind your ack:

// MCP tool call: policy_attach
{
  "sessionId": "sess_…",
  "gate": { "command": "pnpm", "args": ["test", "--filter=queue"] },
  "onFail": { "maxRetries": 2, "nudge": "Gate failed (exit {code}). Fix and finish the task." },
  "then": "emit",
  "next": {
    "gate": { "judge": {
      "adapter": "claude-code", "model": "claude-sonnet-5",
      "prompt": "You are a skeptical reviewer. Try to REFUTE that this diff fixes the flaky test. Reject if tests were weakened or skipped."
    }},
    "then": "commit",
    "commit": {
      "paths": ["packages/queue"],
      "message": "fix flaky retry test",
      "requireHumanAck": true
    }
  }
}

When the agent claims done, the tests run on the daemon — fail, and the agent
gets nudged (up to two retries, then it blocks). On green, a separate judge model
— prompted to refute, not praise — reads the work. On the judge's approval the
explicit paths are staged and the policy parks in awaiting-ack, emitting
policy:commit-ready. The commit runs only when you policy_ack — from
anywhere — and it never pushes, never git add -A. Disconnect all you want; the
gate is in the daemon.

Every rule the ladder earned is now load-bearing in that one config: the judge is
a different context (the plateau), the gate is deterministic code (prompt rules
aren't boundaries), the ack default-denies (missing approval ≠ yes), and the
whole thing survives your screen locking (durable signal wait).

Where each tool actually sits

The receipt, tool by tool — find yours, then find its ceiling.

Rung	What checks "done"	Tools that live here	What it still misses
L0 watching	You, reading	Claude Squad, Conductor, Nimbalyst, every dashboard	Caps at 2–5 agents; dies at screen-lock
L1 watchdogs	A liveness monitor	Agent Farm, self-healing overnight loops	Keeps it alive, not correct
L2 permission relay	You, on a nicer surface	Paseo, Omnara	Approval fatigue; still a human bottleneck
L3 verifier loops	Software, inside the run	Ralph+verifier, Anthropic Outcomes, Advisor (mid-turn)	Judge is lenient by default; dies with the session
L4 external gates	A daemon, outside the run	agentproto; DIY on Temporal/Camunda if you're brave	The honest ceiling — see below

Two vendor-native footnotes, because fairness is what makes the rest credible.
Outcomes is a real L3 for an all-Claude shop on Anthropic's infrastructure —
retry-until-the-grader-passes, no daemon of your own. Advisor is an
interesting L2.5-for-models: a cheap executor consults a stronger model mid-turn.

The catch that sends you cross-vendor. Both sit inside one vendor's walls
— Advisor has the same single-vendor, encrypted-advice catch named in
the landscape piece. Which rules
out the exact case where gates pay off most: a cheap GLM or DeepSeek executor
judged by a stronger model. The cost arbitrage of mixed fleets
(the routing math is its own post) only works if
something external catches the $0.10 model's mistakes before merge — and that
gate has to reach agents no single vendor sells you.

So the top of the ladder isn't just "more supervision." It's the only rung whose
check you own — cross-vendor, local, and yours.

Match the rung to the trust you need

The ladder is not a maturity score, and climbing to the top of it isn't a virtue.
L0 is the right rung for exploratory work; L4 on a throwaway script is
theater. It's a diagnostic, and the diagnosis is one sentence: the check that
decides "done" should sit as far from the agent as the cost of being wrong
demands.

If your agents' output merges to production and nobody's going to read every diff
forever, then somewhere between "the agent says done" and "the commit lands,"
something skeptical and external has to say yes — and Anthropic's own research
says it cannot be the agent. One honest caveat the other way: don't treat the
human at L0–L2 as ground truth either. People give careless, inconsistent answers
at scale, which is the deeper reason the check wants to be a process, not a
tired glance at 2am.

You wouldn't let a defendant mark their own exam. Stop letting the model that
wrote the code be the one that tells you it's fine. Pick your rung by what breaks
if it lies — and put the pass/fail somewhere it can never reach.

If your setup already climbs higher than this, or I've mis-rated where a tool
sits, tell me where — I'll fix the piece.

---

The series — Orchestration, Honestly

Ten pieces, one argument. Start anywhere; each one cross-links the rest.

	Piece	The one idea
1	You can't parallelize the trust	Amdahl's Law: why your fifth agent slows you down
2	Harness engineering	you rent the model; the harness is the part you own
3	The supervision ladder (you're here)	five rungs of trusting an agent you don't watch
4	The approval plane	auto-approve reads, gate writes — wire the line between
5	Kill the loop	why "keep going until done" compounds a wrong turn
6	Route by cost	plan expensive, execute cheap, verify independently
7	Files with contracts	the interop layer every agent system reinvents
8	Knowledge is power	give your agent your knowledge, not the internet's average
9	Paseo, hands-on	a full real-session review of the daemon
10	9 orchestrators, compared	the tool-by-tool teardown + a decision table

Written by the maintainer of agentproto (Apache-2.0, source). Same contract as our /compare page — dated facts, named strengths, corrections by issue. Got something wrong? File an issue.

Building agentproto in the open — follow @theagentproto and @agentik_ai on X.