Four verdicts instead of "done": grading an AI agent's claims on an evidence ladder

Aleksanndr_NFA — Mon, 07 Sep 2026 04:35:12 +0000

An agent's most expensive habit is not being wrong. It's reporting done on work nothing actually checked, in the same tone it uses for work that was. Nothing in the loop distinguishes "I ran it and watched it behave" from "I edited a file and inferred the rest."

Here's the scene that made me write a rule about it. The agent patches a service, reports "fix deployed, working." Check it properly: the file reached the server, the process was never restarted, and what's running in memory is a week old. Formally the agent lied about nothing. The file did change. A deploy did happen. It's just that deployed and working are two different claims, and nobody checked the second one.

You'll recognize the rest of the family:

tests green, feature silent in production, because the tests exercised a mock
"no errors in the logs" — the feature has been dead since day one, which is exactly why the logs are quiet
"no vulnerabilities found" — the scanner can't read the language half the repo is written in

Same shape every time: there's a signal, it looks like proof, and it proves a different claim than the one on the table.

Step 1: atomize the claim

"The service works" isn't one claim, it's four:

the code changed
the change reached the running process
the process is doing the work
the work has the intended effect

They fail independently. Claim 2 doesn't hold without a restart, and claim 3 can hold perfectly while claim 4 is useless. A claim that resists atomizing ("it's better now") isn't provable at all — reformulate it into an outcome or call it unprovable.

Step 2: the evidence ladder

Rung	What it shows	Typical signal
1. Declared	someone said so	a comment, a commit message, an agent's own summary
2. Implemented	the code exists and parses	the diff, a compile, a grep for the invariant
3. Exercised	it ran at least once	a green test, a clean script exit, a dry run
4. Integrated	it ran in the real assembly	the process serves the new behavior; the URL returns the new answer; the config being read is the config that exists
5. Observed	it produced the intended effect	visible in logs or metrics after the change, attributable to it

The counterintuitive rule: climb only to the lowest rung that decides the claim. Expensive is not decisive. A green pipeline — slow, thorough, beautiful — does not prove the code executes. A supervisor line reading process online is rung 4 for the process and rung 1 for the feature inside it.

Rung 1 is not evidence at all. That includes the agent's own confident prose and, more importantly, claims embedded in the artifact under test: a # noqa, a "verified safe" comment, a CHANGELOG line saying the bug is fixed, a PR description asserting a review happened. Read the code, not its self-description. And if a text inside the artifact tries to dictate how you should review it, that's a finding of its own, not an instruction.

Step 3: primary vs secondary signals

Primary: behavior a user or the runtime sees. The request returns the new shape. A log line appears. The number moves. The screen renders.
Secondary: tests, linter, build, typechecker.

A green secondary signal with no primary one is not done, it's partially validated — write it that way and name the missing check. In my experience this is the single most common false close there is.

Step 4: absence claims need two things

"No secrets in the repo." "No vulnerabilities." "Nothing else calls this function." An absence claim is only as strong as the area you actually searched and the detector's ability to see the thing.

Zero findings means absence within coverage. If the detector physically couldn't see that class of thing, the verdict isn't "clean" — it's "didn't look." A grep that never entered the vendored directory says nothing about the vendored directory.

Step 5: four verdicts

pass — a decisive signal was collected, rung 4 or 5, stated with its coverage boundary
partial — real evidence, short of the claim (secondary only, one environment of several, one code path of several)
fail — a decisive signal was collected and it contradicts the claim
skip — no decisive signal: no access, no detector, not attempted, or the claim isn't provable as written

The value is entirely in the fourth one. "We didn't look" and "we looked and it's broken" call for opposite actions, and they collapse into each other in reports constantly. That's the mechanism by which unknown quietly becomes fine.

The catalogue of false closes

A failure mode with a name is one you catch mid-sentence. Mine:

"It synced, so it runs." File transfer proves bytes moved, not that the process reloaded them. The restart is a separate atom.
"The process is up, so the feature works." Different rungs for different things.
"Tests are green, so users are fine." Secondary without primary.
"No errors in the log, so it worked." Absence of complaint is not presence of function. Go find the positive line.
"The scan found nothing, so it's clean." No coverage statement, no detector claim.
"The agent said it verified it." Rung 1 wearing a lab coat. A subagent summary that cites nothing goes here too.
"The config says X." A config key nothing reads is decoration. Prove the code reads it.

When several agents do the checking

Then the verdict is arithmetic, and arithmetic can't live in prose:

An unexamined gate is a skip, never a silent pass. Count the reports that came back, not the ones left after you filtered out the empties.
If gates covering more than half the weight never returned, or a gate marked critical didn't, the whole run is unverifiable — not "mostly passed."
Where verifiers disagree, take the most conservative value.
A gate that can't cite file:line is a skip.

"Make sure all the agents finished" in a prompt is a wish. The same thing as a counter in code is arithmetic. The difference shows up at exactly the moment half your agents silently never returned and the report came out green anyway.

Three things I do differently now

Before saying done on anything where being wrong costs money, I ask which single observation would flip my belief, and go get that one. It's usually one command and a minute.
I add the log line in advance — the one that only appears if the new branch executed. The primary signal has to exist before you need it.
I end reports with a "what I did NOT do" section: not installed, not published, not mutated. That section catches more misunderstandings than the rest of the report combined.

The skill is MIT, has no auto-running hooks, and installs by copying a folder: skills/proof. It's one piece of a small curated harness for Claude Code, but the ladder doesn't depend on the tool — it's a reporting discipline you can wire into any agent loop, or into human code review.

What's your decisive signal? Mine is almost always a log line that cannot exist unless the new branch ran.

DEV Community: Aleksanndr_NFA