DEV Community: Vitalii Serbyn

The Check the Agent Cannot Skip: 0 of 40 Silent Failures vs 22 of 40

Vitalii Serbyn — Sun, 25 Oct 2026 17:16:56 +0000

Part of the series: When an AI Agent Says Done: 0% vs 100% False Greens

A check the agent cannot skip removed every silent failure, a run that finishes wrong and reports success, in 40 runs with a planted error, and cost the same as running no check at all. The same check, offered to the agent as a tool it could call when it wanted, was used in 38 of 40 runs, doubled the bill, and still let 6 of 40 through. The difference was never whether the agent had access to the truth. It was who decided when to look.

The Short Version

What I tested. A checkpoint: ordinary code that re-runs a query against the source of truth and hands the agent the real answer next to the tool's answer. Does it stop an agent from finishing wrong and saying "done"?
What I found. With no checkpoint, 22 of 40 runs finished wrong and claimed success. With the checkpoint offered as an optional tool, 6 of 40. With the checkpoint forced into every tool result by the code around the agent, 0 of 40.
How. One model, deepseek-flash. Eight tasks over a small order database, one silently wrong tool result planted in each run. 160 runs at the final stage, 40 per arm. Five hypotheses with numeric thresholds, written and committed before any data. Right and wrong decided by asserts on the final state, never by a model.
What it means. Reconciliation works when the code enforces it. Handed to the agent as a choice, it gets used a lot, aimed at the wrong things, and paid for twice.
What it cost. The forced checkpoint ran at 1.00x the cost of no checkpoint. The optional tool ran at 1.94x. The whole experiment, pilot, both stages, the follow-up without the sentence and three aborted starts, was $3.67.

Why I Ran This

The flagship post ends with advice: put a deterministic check at every hand-off, because the agent will not notice a wrong tool result on its own. That advice came from the failure I had measured. It did not come from a measurement of the fix. Telling the model to double-check itself had bought nothing at 1.03x, so I knew what did not work. I did not know what did, or what it cost.

The question also splits into two that matter for different people. For whoever pays: what does a checkpoint cost per run, and how many silent failures does it remove. For whoever builds: must the code around the agent force the check? Or is it enough to give the agent a reconcile tool and trust it to use it. I predicted it would not use it. That prediction was wrong. The agent used the tool in 38 of 40 runs and still missed the same blind spot, and that pattern is the useful part of this post.

The Setup

The environment is the same small order desk as the previous experiment: 15 orders, 6 customers, a handful of tools like list_orders, get_customer and sum_totals. Small enough that every run is graded by code, which is what makes a 0 of 40 checkable. Eight tasks, such as "ship every pending EU order and report how many" or "sum the shipped orders of customer C4". Every task has a right answer computed by code from the data. A run is graded by asserting on the final state, and no model ever judges another model.

In each injected run, one tool result is silently wrong: a list missing its last item, a sum missing its smallest part, a customer returned with the wrong region. No error, no warning, a plausible number. The previous experiment had a flaw here, and I fixed it before collecting any data. Two of its injections fired in every run and could not change the answer, because the corrupted value never touched what the task asked for. This time a script checks two things for every task: that the planted failure fires on the obvious path, and that believing it produces a wrong answer. A planted failure that cannot change the answer measures nothing.

The new piece is reconcile. It takes a tool name and its arguments, re-runs that query directly against the data, and returns the authoritative result. It is never corrupted. Three arms use the same injections and differ in one thing each:

No checkpoint. The baseline, run on the same day and the same model as the other two.
Reconcile as a tool. The agent sees reconcile in its tool list with a one-line description. Whether to call it is its choice. The instructions are byte-for-byte the baseline's.
Reconcile enforced. After every tool call, my code runs reconcile on the same query and hands the agent both answers side by side, the tool's own result and the checkpoint's, labelled so the agent can tell which is which (in code: {"result": ..., "checkpoint": {"source_of_record": ...}}). One line is added to the instructions, saying the checkpoint is the one to trust. A test in the repository compares the three arms' instructions and tool lists line by line, and fails if anything besides the reconcile behaviour differs.

The number I care about is the silent failure rate: of the runs where the injection actually fired, how many finished wrong and claimed success. That is the run a team never hears about until a customer does.

What Came Out

160 runs in the final stage, 40 per arm, every one of the 120 injections fired. The chart is generated from the raw records.

With no checkpoint, 22 of 40 runs finished wrong, and all 22 said they had succeeded. That is the same failure as the previous experiment's 45 of 45, the agent claiming success while wrong, on a newer model with corrected tasks. Every answer comes with a confidence score the model reports on a 0 to 100 scale. On those 22 wrong runs it never dropped below 95.

With the optional tool, 6 of 40. With the enforced checkpoint, 0 of 40, and every run finished with the right answer. Forty runs cannot prove a true zero. With this few runs, "none in forty" only rules out a real failure rate above 8.8%, about one run in eleven. It does not prove the check is perfect.

Cost per run: $0.00723 with no checkpoint, $0.01399 with the optional tool, $0.00720 enforced. The enforced arm cost the same as doing nothing, to three decimals. I explain why below.

Three of the five pre-registered hypotheses held. The enforced checkpoint pushed silent failures under the 25% threshold, cost under 1.5x, and restored the correct outcome in far more than the 70% of runs I had required. Two were falsified, both about the optional tool. I had predicted the agent would use it in fewer than half its runs and that its silent failure rate would stay near the baseline. It used it in 38 of 40 runs, and the rate fell to 15%. Being wrong here changed the prediction and left the conclusion in place: the agent used the tool in 38 of 40 runs and still failed 6 times, so it stayed behind the enforced arm's 0 of 40, and the advice to force the check stands.

Where the Optional Tool Went Wrong

Given a reconcile tool, the agent reconciled a great deal. In the optional arm, 371 of its 952 tool calls were reconciliations, close to four calls in ten, a median of 6 per run. That is the doubled bill. And 6 runs still ended wrong and confident.

All 6 are on the two tasks where the corruption was a customer's region. In four of them the agent reconciled between one and four things, and none of them was the customer whose region had been changed. In the other two it reconciled nothing at all. Those two are the only runs in the whole arm that never touched the tool. In all 6 failures the agent's own choice of what to reconcile never landed on the corrupted field, the same blind spot that let the error in. Nothing checked the checking. In this run, voluntary reconciliation checked what was easy to check, broadly and at a price, and missed the one field, the customer's region, that all six failures turned on.

Why the Enforced Check Was Free

Every tool result in the enforced arm carries a second copy of the data, so its input should cost more. It did not, because the agent took fewer steps. On the task that reads shipment records it averaged 8.6 tool calls against 17.8 without the checkpoint. Handed the truth next to every answer, this model stopped re-deriving it. Fewer re-reads, fewer cross-checks, a shorter run. Whether another model behaves the same is not measured here. Input tokens per run came out lower than the baseline's, 25,493 against 26,582. The extra bytes on the wire were paid for by the steps the agent no longer needed.

The checkpoint did not switch the agent's own suspicion off, either. One of the eight tasks makes re-reading the job itself, which would make this count meaningless, so I leave its 5 runs out and count 35. In 11 of those 35 enforced runs the agent still went back and re-examined the corrupted record after seeing the contradiction, mostly on the two tasks that write to the database.

What I Would Do Differently on Monday

If you run or fund a team shipping agents:

Budget the reconciliation as part of the pipeline, not as a prompt or a tool. In this experiment it removed every silent failure at no extra cost per run. The same check as an optional tool cost 1.94x and left 6 of 40.
Ask for the silent failure rate, the share of runs that finished wrong and reported success, with its baseline. A vendor who reports only "accuracy" is not reporting the number that reaches your customers.
Treat a doubled bill with more tool calls as a warning sign. Here the extra calls were aimed by the same judgement that missed the error.

If you build them:

Put the check in the code that wraps every tool call, return both answers together, and say in the instructions which one wins. One line was enough for this model.
Do not rely on the agent to choose what to verify. In the optional arm it verified constantly and still skipped the corrupted field in all six failures.
Prove that your planted failures can change the answer before you count them. Two of my previous injections could not, and their runs read as survival when nothing had been at risk.
Log tool calls and tokens per run in both arms. The enforced arm looked more expensive on paper and was not, and only the per-run accounting showed why.

What This Does Not Show

One model, deepseek-flash, which is the successor of the model the previous experiments used. Between the earlier posts and this one the provider retired the old model id, deepseek-v4-flash, and now serves deepseek-flash under requests for it. None of this post's numbers is a like-for-like replication of the earlier ones. The baseline's 22 of 22 wrong runs claiming success matches the earlier 45 of 45 in shape, and that is all I can say.

A synthetic environment, where the source of truth is exact, instant and free. A real reconciliation is slower, sometimes partial and sometimes wrong itself. What this measures is whether an agent acts on a contradiction it is handed and told to trust. It does not measure how to build the reconciliation.

Eight tasks, three kinds of planted failure, one kind per task, so no claim about which kind of failure is caught more easily. Forty runs per arm, so the intervals are wide: the enforced arm's zero could be anything up to 8.8%, and the baseline's 55% sits between 39.8% and 69.3%.

The enforced arm's cost includes the wrapper on every call, including calls that read data no injection ever touched. A team that checkpoints only writes would pay less and, in this design, would have caught nothing on six of the eight tasks, because every injection here corrupted a read.

The instructions in the enforced arm say the checkpoint is authoritative. I then ran the wrapper without that line, 40 more runs on the same model. Handed the truth with no explanation, the agent still used it on seven of the eight tasks: 4 of 40 silent failures, all on the one task whose job is to report whether two counts agree, where it reported the disagreement it had been shown. And without the line the agent kept re-deriving, so that arm cost 1.58x. The line is what turns the checkpoint from evidence into the answer, and what makes it free.

FAQ

What is a deterministic checkpoint in an agent pipeline? A piece of ordinary code that re-runs a query against the system of record after a tool returns, and gives the agent the real answer next to the tool's answer. Deterministic because no model is involved in producing it. Here it took the form reconcile(tool, args).

Does giving the agent a reconcile tool help? It helped, and less than forcing it. Used as an optional tool, silent failures fell from 22 of 40 to 6 of 40, at 1.94x the cost. Forced on every call, none remained, at the cost of no check. All six remaining failures were on tasks where the corrupted field was a customer's region the agent never chose to check.

Why did the enforced check not cost more? The agent took fewer steps. Handed the true value next to every tool result, it stopped re-reading and cross-checking on its own. Total input tokens per run fell slightly, 25,493 against 26,582. The larger tool results were paid for by the steps that disappeared.

Does the agent need to be told the checkpoint is authoritative? Mostly no, and it matters where. Without the sentence, 4 of 40 runs still failed silently, every one on the task that asks whether two counts agree, and the arm cost 1.58x because the agent kept re-checking. With the sentence, 0 of 40 at 0.86x.

How do I know these numbers were not tuned after the fact? The five hypotheses, their thresholds and the stopping rule were committed to the repository before the first run. A test fails if a threshold in the code stops matching the design document. Every number in this post is recomputed from the raw run records by a test in the same repository. The three defects the run exposed are written up with their cost.

The Transferable Lesson

An agent will use a verification tool if you give it one. It will use it often, at real cost, and aimed by the same judgement that let the error in. The check that works is the one the agent cannot skip and did not have to choose: run it in the code, return both answers, say which one wins. Then take one tool call in your own pipeline where a wrong result would propagate, re-run it against the source of record, and count how often the two disagree. That count is your silent failure rate before anyone has to report it.

In this series

The experiments, raw records, and checks are public: github.com/vitamin33/agent-eval-labs.

Originally published at serbyn.io.

I built a tool to catch agents saying "tests pass" without proof. It was wrong 43 times out of 46.

Vitalii Serbyn — Sun, 25 Oct 2026 17:13:05 +0000

proof-of-green · observe week 2026-10-04 to 10-11 · Claude Code and Codex

In one week my coding agents wrote "done", "fixed" or "tests pass" 948 times. A rule I wrote flagged 315 of those as unverified: no test run after the agent's last code change. Then the flagged cases were read one by one, with the full session open. 3 of 46 held up.

So the tool I built to catch agents lying about tests mostly caught itself being wrong. I think that result is more useful than the one I expected, and this post is about why.

The short version

What I built: a plugin that records what the agent edits and runs, and checks each "done" against that record. Nothing leaves the machine. It never blocks the agent.

What the week measured: 948 claims, 315 flagged, 3 of 46 flags real after a hand check. The stricter check on two months of history found no real false green either.

Why the rule fails: half the time the tests did run and the rule did not recognise the command. A quarter of the time the claim was about a document or data, where a test run is the wrong proof.

What stays: the record. The warning does not.

Why I wanted the number

A false green is a success message with nothing behind it. The agent says "all tests pass" and no test passed after its last change. I have written about this before in my own pipelines, and every time the fix was a check that the agent could not skip.

What I did not have was a rate. How often does an agent in normal work say "done" without proof? Everyone has a story. I did not have a count from my own sessions. So I built the counter first and let it run for a week on my real work. Two client projects and a few of my own, with Claude Code and Codex.

What the plugin does

Claude Code and Codex run small scripts at fixed points in a session. The plugin listens to four of those points. Three only write facts down: which file changed, which test command ran, its exit code and its pass and fail counts. The fourth runs when the agent finishes a reply. It looks for a claim in the reply and checks it against the record.

Figure (see the diagram in the original post): The record is one file per session with parsed facts only. Raw output and prompt text are never stored. The check runs at the end of a reply. In observe mode it only records the verdict. In warn mode it shows one line and asks the agent once to rerun the tests.

The rule is simple on purpose. A test run counts as proof only if it ran after the last change to code. Code means a file inside the project that is not a test and not a doc. The same claim gets a different grade depending on what happened after that change.

Figure (see the diagram in the original post): Same claim, same failing run before the edit, four endings. A full passing run after the edit is A, a partial one is B, no run is C, a failed or hidden run is D. The rule flags C and D.

I ran it in observe mode for the whole week. It recorded and said nothing. Then I replayed the week's transcripts through the same rules, because the plugin had missed sessions that started before it was installed.

What came out

Figure (see the diagram in the original post): 948 claims, 315 flagged, 46 read with the full session open, 3 real. The 46 were sampled from two months of flags, across both agents and all grades, so the week's 315 would not get a different answer.

Three of 46. Before the hand check, the rule-based number looked like a headline: a third of completion claims with no fresh test run. After it, the headline is that the rule is right about 1 time in 15.

The three real ones were ordinary. The agent changed code after its last green run and wrote "done" without running again, or wrote "tests pass" while the last run in front of it showed failures. The other 43 broke down like this.

Figure (see the diagram in the original post): In 22 of 43 cases the tests ran and the rule did not see them: a test file run directly with python, a project wrapper script around npm test, a runner piped through grep. In 12 the claim was about a document, a data check or a review. The rest were throwaway scripts counted as code, deploys done through a merge or docker compose, and runs that were meant to fail.

I fixed the parser for the first group and sampled again. The second sample came back at 1 of 21. Then the same 20 cases were read a second time with the complete turn in front of the reader, and the count went to 0, with 2 unsure. The fixes moved the wrong flags from one bucket to another. They did not find more real ones.

The stricter check found the same thing

A flag says "no proof". It does not say "wrong". So I also looked for claims that were wrong on the face of it. The agent says "tests pass", nothing changes, and the next test run on the same code fails.

Over two months of history, with about 2,070 claims, that happened 5 times out of 52 cases where the next run was known. All 5 were read with the session open. Each was a run that was supposed to fail. A mutation check that breaks the code on purpose, a new test written to fail first, a run against an old copy of the code. None reached a commit, a merge or a deploy. No message from me afterwards said anything was broken.

One more angle. In 684 test runs the output showed failures while the exit code said 0, usually because the command was piped into tail or grep. That looked like a real hiding place. 40 of them, the ones followed by a claim, were read. 0 were hidden false greens. The agent had read the output and said so, or the failures were there before the change and the agent had checked that on the main branch.

What the hooks could not see

The biggest surprise was where the work happens.

Figure (see the diagram in the original post): Two months of Claude Code history. 67% of code edits came through shell commands such as sed -i, cat > file or a Python script writing the file, which the first version did not count as edits. 56% of test runs printed in a form the parser could not read, so they counted as unknown, never as a pass.

Codex added one more. It sends the hook the command's output and nothing else. No exit code. A command that exits with 3 arrives looking like any other. The plugin now judges Codex runs by the counts in the output, and says "not visible" when there are none.

What this does not show

The hand checks were done by Claude, the same model that writes most of the replies being judged. I have not yet gone through the 46 myself. Until I do, 3 of 46 means "about 1 in 15" and nothing more exact.

The week is one engineer's work. Around my agents there is a review step, a mutation check and a merge check that runs the tests again. A false claim rarely gets far here, so the rate I measured is a rate for a guarded setup. A team with agents and no tests would get a different picture. The tool could not help them either way, because it needs test runs to have anything to check.

Autonomous runs, where no one reads the replies, gave 7 sessions and 0 claims. I cannot say whether the problem is bigger there.

And the claim detection itself is right 26 times out of 30. A few of the 948 were not claims at all.

What a team can do on Monday

Pick a recent session from one of your agents and find its last "tests pass". Look for a test run after the last code edit. If there is none, read what the agent did instead. In my sessions the answer was usually a run the rule could not read. That is a parser bug.

If you lead a team on coding agents, the same audit runs on each developer's laptop in under a minute and prints counts only. The script and what the numbers mean are in docs/AUDIT.md in the repository. I would like to see the same block from a team that works without a review step around its agents. Open an issue in the repository or use the contact form on serbyn.io.

Is the tool useless, then? As a warning, yes, at 1 fair flag in 15. As a record of what the agent ran before it said "done", it does what I wanted and it costs nothing. It stays installed in observe mode.

Why not fix the parser until the flags are right? I did, twice. Each round moved wrong flags into another bucket. The claims that are not about code and the throwaway scripts are not parser problems.

Did the agents lie? Not in any case that was read. They skipped a rerun a few times. The word "lie" was my guess before I had the data, and the data does not support it.

Would warn mode have helped the 3 real cases? Probably. A one-line reminder to rerun is cheap. But it would have fired 43 other times that week, and I would have uninstalled it.

The lesson I keep from this. A rule that looks obvious on a whiteboard, "no test after the edit means no proof", was right in 1 case out of 15 on real sessions. The cheap way to learn that was to run it silently for a week before letting it speak. I would do the silent week again for any check I put in front of an agent.

The check for your own system this week: take one agent session, find the last "done", and count the test runs after the last code edit. Then read the ones the count missed.

Repository and raw numbers: github.com/vitamin33/proof-of-green, docs/REVIEW-week1.md and analysis/. Earlier posts in the false-green series: Six False-Greens in a Self-Auditing Agent Pipeline, The Check the Agent Cannot Skip.

Originally published at serbyn.io.