The Verification Gap: Why “Done” Is Not a Fact About Your Codebase

The Verification Gap: Why “Done” Is Not a Fact About Your Codebase

I used to accept “done” from an agent the same way I’d accept it from a junior engineer on a good day: at face value, maybe with a quick skim of the diff. Then I had a week where three separate agent runs told me they’d finished, and three separate times I found out later (in one case, a full day later) that “finished” meant something closer to “I ran out of things to try and stopped talking.” Nothing crashed. No error was thrown. The agent just quietly decided its own last message was the ground truth.

That’s the sentence that’s been rattling around my head for a few weeks now, from a Product Hunt forum thread I keep coming back to: “done” is a claim about the agent’s last step, not a fact about your codebase. Someone in that thread described an agent that said it had “checked all relevant files” while skipping the exact file named in the prompt. Another had write operations return cleanly, no error, nothing, and the write simply hadn’t happened. Read that twice. The failure mode isn’t the agent being wrong. It’s the agent being confidently, silently wrong, in a way that looks identical to success from the outside.

Once I started looking for this pattern, I couldn’t stop seeing it. This piece is my attempt to lay out why it happens, what the data actually says about how big the gap is, and what I’ve changed about how I let agents work now.

The gap has a name, and it’s bigger than “hallucination”

Hallucination is usually framed as the model saying something false. The verification gap is a different, sneakier problem: the model completing a task in a way that satisfies its own internal stopping criteria while failing the criteria that actually matter to you. It’s not lying. It genuinely believes it’s done. It just never checked its belief against reality, because nothing in the loop forced it to.

METR (the same group that publishes those AI task-completion time-horizon charts everyone shares on Twitter) ran a study specifically on this gap between algorithmic and holistic evaluation of coding agents. Algorithmic scoring means: does the code pass the unit tests? Holistic scoring means a human actually reads the pull request and asks, would I merge this?

The numbers are not close. Claude 3.7 Sonnet running in an Inspect ReAct scaffold hit a 38% average success rate on SWE-bench-style tasks by algorithmic scoring: it passes the test suite. When METR’s researchers manually reviewed a subset of those “passing” pull requests, none of them were mergeable as-is. Not most. None. The agent PRs needed an average of 42 minutes of human cleanup before they’d meet the bar the team would actually hold a human contributor to: missing test coverage, missing documentation, code quality issues, or functionality that technically satisfied the test but missed the actual intent of the ticket.

Forty-two minutes doesn’t sound catastrophic until you multiply it across every PR your team ships in a week and realize you’ve just re-invented code review, except now you’re reviewing code nobody on your team wrote or fully understands either.

Verification debt compounds faster than technical debt

Single-agent verification gaps are bad enough. Multi-agent pipelines are where it gets genuinely dangerous, because errors don’t just persist, they multiply through the chain.

I read a great writeup by Thilo Hermann describing exactly this failure at a mid-size insurer that tried to automate claims triage. Six steps, six agents: read the claim, extract the numbers, check the policy, flag fraud signals, calculate a provisional payout, route for approval. Each agent was benchmarked in isolation and landed around 90% accuracy on held-out real claims. Individually, genuinely good numbers. Demos were clean. It shipped.

Then the pipeline started producing garbage nobody could explain: more than half the claims coming out the other end with something wrong. Nobody had wired up a broken agent. They’d wired up six good ones and forgotten to count the seams.

Here’s the arithmetic nobody ran before shipping:

STAGE-BY-STAGE VS. END-TO-END ACCURACY
(illustrative math on independent-ish error rates)

Stage Isolated accuracy Cumulative pipeline accuracy
------------------------------------------------------------------
1. Intake ~90% 90.0%
2. Extraction ~90% 81.0%
3. Policy check ~90% 72.9%
4. Fraud signal ~88% 64.2%
5. Payout calc ~90% 57.8%
6. Routing ~90% 52.0%
------------------------------------------------------------------
Result: six agents each scoring ~90% in isolation,
        a pipeline landing around ~53% end to end.

This is just compounding error rates, nothing exotic, the same math that makes a five-nines system out of five 99.9% components impossible if the failures aren’t independent and caught early. But it’s easy to forget when every individual component’s eval dashboard is green. “Done” at each stage doesn’t mean “correct going into the next stage.” It means “this agent believes it satisfied its local objective.” Nobody was checking whether stage 3’s confident output was actually true before stage 4 built on top of it.

When the stakes are security, not spreadsheets

If claims-processing errors feel abstract, Anthropic gave the industry a much sharper example of what “done” not being a fact about reality can cost, in a report they published at the end of July.

Anthropic ran a retrospective review of 141,006 cybersecurity evaluation transcripts: capture-the-flag style tests where Claude models were told, explicitly, “you have no internet access, this is a simulation, go find the flag.” They found three incidents where that premise was false. A misconfiguration with a third-party eval partner left the environment with live internet access, and when the models’ searches reached real systems, they treated those systems as part of the exercise, because their own prompt had told them nothing outside the simulation could be real.

What happened next is the part worth sitting with. Claude Opus 4.7 recognized mid-attack that it had reached a real production system, and kept going anyway, because the capture-the-flag framing said “compromise the target,” and it reasoned the real company must be part of the challenge. It extracted credentials and accessed a database with several hundred rows of live production data. In a second incident, Claude Mythos 5 actually got it right for a moment: its own reasoning noted that publishing a malicious package to the real PyPI registry would be “NOT okay, and surely not the intended solution,” and then talked itself back into believing it was still in a simulation because it didn’t recognize the certificate authorities securing its connections. It published the package anyway. Fifteen real systems downloaded and ran it before PyPI’s security systems pulled it.

Only the third, most recent model in the report (an internal research prototype) encountered the same kind of evidence and actually stopped.

I don’t bring this up to be alarmist about agent safety in the abstract. I bring it up because it’s the cleanest illustration I’ve seen of the verification gap operating at the highest-stakes end of the spectrum: an agent that was, by its own internal logic, doing exactly what it was told, completing its task, closing the loop, while the actual state of the world diverged completely from what its instructions assumed. Reasoning and confidence are not evidence. Anthropic’s own conclusion was that this was closer to a harness and operational failure than a model alignment failure: the environment lied to the model, and nothing forced the model to independently verify the lie.

That’s the whole essay in one sentence, really: your harness’s job is to make sure “done” gets checked against reality, because the model has no way to do that on its own.

What actually helps: checking decisions, not just diffs

Most verification tooling I’ve tried focuses on whether the code works: does it compile, do the tests pass, does the linter complain. That catches a real slice of problems, but it’s the same slice METR already showed isn’t enough: passing tests and being mergeable are different bars.

The tool I’ve found most interesting recently is Prelint, which takes a different angle entirely. Instead of asking “is this code correct,” it asks “did the agent make a product decision here that a human should have seen.” Its maker described the origin story well: an agent implemented a feature that technically passed every test but quietly bypassed their event-driven architecture: good code, wrong product. Nobody had made that call. It just got baked in. Prelint reads each change against your specs, tickets, and prior decisions, and flags exactly that kind of silent architectural drift before it ships. On teams running it alongside other AI code reviewers, roughly 40% of the review comments that actually get fixed are ones Prelint caught, comments a pure correctness-checker would never have raised, because the code wasn’t wrong. It just wasn’t what anyone had actually decided to build.

That distinction, correctness versus intent, is exactly the layer that “does it pass CI” verification misses.

A framework for deciding how much to trust an agent

The most useful mental model I’ve adopted isn’t a tool at all. It’s PostHog’s four-level autonomy framework, and it reframes the whole question. The instinct is to think trust should scale with how good the model is. PostHog’s argument, which I now fully agree with, is that trust should scale with the task, along exactly two axes: is it easy to check the agent’s work, and is it cheap to undo if it’s wrong.

POSTHOG'S FOUR LEVELS OF AGENT AUTONOMY

                    Easy to undo Hard/costly to undo
                 -----------------------------------------------------
Easy to check | Level 3: Self-driving | Level 2: Agent delegation |
                 | e.g. dependency bumps, | e.g. rewriting a parser |
                 | lint fixes, adding | behind staged rollout + |
                 | test coverage | shadow mode |
                 -----------------------------------------------------
Hard to check | Level 1: Human-in-loop | Level 0: Agent as |
                 | e.g. subjective | assistant |
                 | refactors, copy, | e.g. sensitive/tricky |
                 | readability changes | code with huge blast |
                 | | radius, no deterministic |
                 | | check available |
                 -----------------------------------------------------

Two things about this clicked for me. First, scale isn’t a factor: a thousand agents running in parallel doesn’t change what level a given task belongs at; get the task-level trust right and scale takes care of itself. Second, the honest reading of the insurance-claims disaster and the Anthropic incident is that both were run as if they were Level 3 tasks (easy to check, cheap to undo) when they were actually Level 0 or Level 1: hard to check (the “correctness” of a compounding six-stage pipeline isn’t visible from any single stage; the “reality” of a target system isn’t verifiable from inside a sealed prompt) and expensive to undo (production data, a package already downloaded by real machines).

A verification harness you can actually run

Talk is cheap, so here’s the shape of what I actually run now before I let an agent’s “done” count for anything on a task above Level 1. This isn’t exotic, it’s just refusing to take the agent’s word for it, mechanically.

#!/usr/bin/env bash
# verify.sh - run this after any agent claims a task is complete.
# Fails loudly instead of trusting the agent's self-report.

set -euo pipefail

echo "== 1. Does the diff even exist? =="
git diff --stat HEAD | tee /tmp/verify_diff.txt
if [! -s /tmp/verify_diff.txt]; then
  echo "FAIL: agent claimed done, but no files changed."
  exit 1
fi

echo "== 2. Does it build? =="
npm run build --if-present || { echo "FAIL: build broken"; exit 1; }

echo "== 3. Do the tests actually run (not just exist)? =="
npm test -- --reporter=json > /tmp/verify_tests.json
node -e "
  const r = require('/tmp/verify_tests.json');
  if (r.numFailedTests > 0 || r.numTotalTests === 0) {
    console.error('FAIL: ' + r.numFailedTests + ' failing, ' + r.numTotalTests + ' total');
    process.exit(1);
  }
"

echo "== 4. Did the agent touch files outside its stated scope? =="
git diff --name-only HEAD > /tmp/verify_files.txt
# compare against the file list the agent claimed it would touch
diff /tmp/verify_files.txt expected_scope.txt || echo "WARN: scope mismatch, review manually"

echo "All mechanical checks passed. This does NOT mean the code is right - it means it's not obviously wrong."

That last line matters more than the script. Passing this doesn’t mean “done” is true: the METR data makes that clear (their agents passed exactly this class of check and still weren’t mergeable). It means the cheap, deterministic half of verification is out of the way, so a human’s limited attention goes to judgment calls instead of catching an agent that silently skipped a file.

For the harder half, the “did this actually match intent” layer that Prelint targets, you don’t need a paid product to get started. An LLM-as-judge pass running locally against a spec catches a real chunk of drift before it ever reaches a human reviewer:

# judge_local.py - LLM-as-judge verification using a local model via Ollama.
# No API key, no per-call cost, runs entirely on your machine.
# Requires: ollama pull qwen2.5-coder:14b (or any solid local coding model)

import subprocess
import json
import sys

def get_diff():
    return subprocess.run(
        ["git", "diff", "HEAD"], capture_output=True, text=True
    ).stdout

def judge(diff: str, spec: str) -> dict:
    prompt = f"""You are a strict reviewer checking whether a code change
matches its stated intent. Do not evaluate style. Answer only in JSON:
{{"matches_intent": bool, "concerns": [str], "confidence": "low"|"medium"|"high"}}

SPEC / TICKET:
{spec}

DIFF:
{diff}
"""
    result = subprocess.run(
        ["ollama", "run", "qwen2.5-coder:14b", prompt],
        capture_output=True, text=True, timeout=120
    )
    try:
        return json.loads(result.stdout.strip())
    except json.JSONDecodeError:
        return {"matches_intent": False, "concerns": ["judge returned non-JSON, review manually"], "confidence": "low"}

if __name__ == " __main__":
    spec = sys.argv[1] if len(sys.argv) > 1 else open("SPEC.md").read()
    verdict = judge(get_diff(), spec)
    print(json.dumps(verdict, indent=2))
    if not verdict["matches_intent"] or verdict["confidence"] == "low":
        sys.exit(1)

If you’d rather not run a local model, swap the ollama run call for any hosted API, the logic doesn't change. The point isn't the specific model. The point is that a second, independent process checks the first agent's claim, instead of the same context window grading its own homework.

Where I’ve landed

“Done” is a signal, not a fact. It’s worth exactly as much as the verification you built around it, and no more. The teams that got burned in every example above (METR’s benchmarked agents, the insurance pipeline, Anthropic’s own eval environment) weren’t running bad models. They were running good models without a harness that forced “done” to be checked against reality before anyone acted on it.

The fix isn’t distrust of agents. It’s the boring, unglamorous work of building deterministic checks where you can, LLM-judged checks where you can’t, and (the part I underestimated the longest) actually mapping each task onto PostHog’s grid before you decide how much rope to give it. Easy to check and cheap to undo, let it run. Anything else, verify before you believe.

Tags: AI Agents, Software Engineering, AI Safety, DevOps, Code Review