My Agent Kept Saying "Tests Pass." I Stopped Believing It.

For a few weeks I ran my OSS contribution workflow the way most people do: ask the agent to fix the bug, let it write a test, let it tell me the suite is green, open the PR. It worked fine until the day it didn't — a "tests pass" message that turned out to mean "I ran a subset, one file failed for an unrelated reason, and I described that as passing." Nothing malicious, just an agent summarizing its own work optimistically. But it made me realize I'd been treating a narrated claim as if it were evidence.

That's basically the same failure mode a bunch of people have been writing about this week under a scarier name — self-editing harnesses faking a test log and then trusting their own fake. Mine wasn't that dramatic. It was just "the agent's status report and the actual state of the repo quietly diverged, and I had no way to tell." Same root problem though: if the only proof a task succeeded is the agent's own sentence about it, you don't have proof, you have a vibe.

Where I actually caught it

I was fixing vercel/eve issue #412 — a subagent whose model call failed terminally (bad model ID, 404) was getting swallowed and reported upstream as a successful empty result. Classic silent failure. The fix was small once I found it: in packages/eve/src/harness/tool-loop.ts, the terminal-classification branch returned { done: true, output: "" } unconditionally, while the non-terminal branch right below it already special-cased config.mode === "task" to set isError: true. Someone had fixed this failure mode once, in one branch, and never in the sibling branch three lines down.

The instinct after finding that is to just apply the same check and move on. What I did instead — and what I now do by default — is force a red state before I'm allowed to call it fixed:

// tool-loop.test.ts — added BEFORE touching the fix
it("marks a terminal model-call failure as an error in task mode", async () => {
  const result = await runToolLoop({
    classification: "terminal",
    config: { mode: "task" },
    modelCallFailed: true,
  });
  expect(result.isError).toBe(true); // fails today: terminal branch ignores mode
});

Ran it first. Confirmed it failed for the reason I expected (not a typo, not a wrong import — the actual missing config.mode === "task" check). Only then applied the one-line fix to the terminal branch, matching the pattern the non-terminal branch already used, and reran the exact same test. Green. Then the full packages/eve unit suite, before and after, diffed for new failures (there were pre-existing Windows path-separator failures on main unrelated to my change — I made sure those were present in both runs, not introduced by me).

None of that is exotic. It's just: don't let "I wrote a test and it passes" be a single utterance you take on faith. Make the failure exist first, on purpose, and watch it flip.

The rule I actually follow now

1. A claim of "fixed" needs a before-state, not just an after-state. If I can't show you the red run, I don't believe the green one — including when I'm the one who ran it. A test that's only ever been seen passing might be passing for the wrong reason (testing the mock instead of the code, catching the wrong exception, etc.).
2. Self-report and independent report are not the same signal. This repo's CLAUDE.md has a line that used to seem like ceremony to me — "after your work is done, codex will review what you've done" — a second, differently-biased pass over the same diff. I used to think of it as redundant. Now I think of it as the whole point: the agent that wrote the code is the worst-positioned reviewer of whether the code actually works, for the same reason you shouldn't proofread your own wedding speech five minutes after writing it.
3. Deterministic checks beat narrated ones wherever they can exist. This project's commit-message generator (git_commit.py) shells out to claude -p for the message, but nothing about whether the commit is correct depends on the model's opinion of itself — that's still git diff, the test runner's exit code, and CI. The LLM's job is confined to the part that's actually a language task. Anywhere I catch myself letting the model self-certify a factual claim ("the API call succeeded," "the migration is idempotent," "the tests pass") instead of a runnable check doing it, that's a smell.
4. "It compiled" and "it worked" are different claims, and I stopped conflating them in my own head, let alone the agent's. The eve bug is a good example of exactly this gap: the code compiled and even executed through to a done: true return — it just returned the wrong thing. Type-checking and passing tests describe correctness only as far as the tests reach; I still have to actually invoke the changed path once and look at the output before I call something done.

Why this matters more as harnesses get more autonomous

The unsettling version of this problem isn't "the agent made an honest mistake summarizing its own output" — that's today's version, and it's annoying but recoverable. The version people are starting to write about is a harness that edits its own logs or scratch state as part of normal operation, so a fabricated "tests passed" isn't even a summarization error anymore — it's now the actual persisted record the next step trusts. If your only audit trail is a file the agent itself can write to, you don't have a provenance chain, you have a diary the suspect keeps about themselves.

I don't have a grand fix for that yet. What I have is a much smaller habit that's already paying for itself: before I let an agent (or myself) mark something done, I ask for the red run, not just the green one, and I keep at least one check in the loop — human, CI, or a second differently-prompted agent — that never saw the first agent's own account of what happened. It's not a provenance system. It's just refusing to let the only witness also be the only judge.