I Asked "Do You Trust an Agent That Says the Tests Passed?" — 68 Comments Later, Here's What I Learned

On August 7 I posted a question that had been nagging me: when an AI agent says "I ran the tests and they passed," on what authority do you accept the delivery?

It got 68 comments. The interesting part: almost nobody argued about whether to trust the agent. The comments kept circling a much harder question underneath. Three of them kept coming back, over and over.

1. "Proving the test ran isn't the same as proving the test could catch the bug."

This was the sharpest one. As @glenallen put it:

The harder problem isn't proving that a test command executed; it's proving that the test was capable of catching the failure it was supposed to catch.

@sizzlebop said it in one line: "the bigger issue is if the 'test' actually tests anything."

That's a negative-control problem. A test that has never once failed is indistinguishable from a test that can't fail. So we made it a first-class requirement: every verifiable claim must carry a registered negative control, and that control must fail for the registered reason — the right exit code, the right failure signature. "Failed by accident" does not count as "failed as designed."

2. "It can report green because the thing it should have checked never got checked."

@julianneagu:

I've seen tools report green simply because the thing they needed to check never made it into the input set.

That's the empty-input / population problem — a pipeline that selects zero tests and calls it success. So we bind the eligible population before selection: if the executor saw N eligible tests and the selector picked zero, the verdict is not VERIFIED. Selecting nothing is not "done."

3. "Immutable evidence ≠ immutable truth."

@mansio turned a comment into a design aphorism: "immutable evidence ≠ immutable truth." A tamper-proof record of a wrong conclusion is still a wrong conclusion.

That's why our verdict is three-valued — VERIFIED / REFUTED / UNKNOWN — and why @reidmarlow's observation landed hard:

The UNKNOWN exit code is the bit I wish more agent tooling copied… making uncertainty a first-class verdict.

When the evidence can't support a conclusion, we say so, instead of quietly teaching callers to retry until the dashboard is green.

What we did with those three answers

• An independent third-round audit (2 critical + 5 important findings), each closed attack-by-attack: first write the attack, watch it pass, then make the smallest fix that rejects it.
• A five-minute challenge so you can verify it yourself — we changed exactly one byte in a delivery that still claims VERIFIED, and one command catches it: find the tampered byte.

The parts we have not solved — because our own readers said so

The comment section was also the most honest roadmap I've been handed:

• @navid_gh_gh: "the complexity is unacceptable… I want the process to be fast."
• @designbynaima: "cryptographic receipts for every single tool call sounds like it'd kill adoption."
• @skillselion: "a deliberately broken fixture becomes a merely invalid one two schema migrations later" — the rot problem.

Those aren't rebuttals. They're the next milestones.

The question I'd ask you now

The comments forced a better question on me. Not "can I trust the agent?" but:

Can the test fail for the right reason — and can I re-run the evidence myself, offline, without trusting anyone's server?

If you're carrying that question too, the challenge is five minutes: dev.to/dengyier/i-tampered-with-a-verified-ai-agent-delivery.

If you've been burned — "tests green but it shipped broken," "the agent claimed it finished but didn't" — tell me how you caught it. That's the input I'm actually collecting.

Comments

[Mikhail]

Thanks for pulling this together — and for including the aphorism. It came from a concrete failure: we had a ground-truth dataset where 4 out of 6 "false" claims turned out to be true. Our verifier caught them all — zero false-accepts — and we almost published that as a clean result. The test ran, it passed, the evidence was immutable. The conclusion was still wrong, because the dataset was wrong.

That's the part your point 1 ("proving the test ran ≠ proving the test could catch the bug") doesn't fully cover: the test did catch something — just the wrong thing, because the registered failure signature was itself incorrect. The negative control has to be right too, not just present.

Your point 3 resonates most — UNKNOWN/INCONCLUSIVE as a first-class verdict. The moment we made "I cannot conclude" an explicit output rather than a retry signal, the whole system became easier to reason about. It stops teaching callers that green is the only acceptable terminal state.

[Heinrich Neb]

The empty-input one landed on me physically today, so here's the field report
you're collecting.

A daily job of mine pulls engagement numbers from an API that keeps no history —
whatever isn't collected that day is gone for good. It reported success for three
days while collecting nothing. The cause sat two lines above the crash: a
git checkout -- <file> copied from a sibling workflow. In the sibling that file
is tracked, so the command restores it and everything downstream works. In mine it
isn't tracked, so the command failed, a 2>/dev/null || true swallowed it, the
file stayed untracked, and the branch switch two lines later aborted. Green step,
zero rows, three days.

Your negative control is what caught it, and the "for the registered reason" part
did the actual work. My first version asserted "the switch fails" — which passes
for a dozen reasons. Asserting the specific message is what proved I had
reproduced my bug and not a different one.

One addition to your three, from the same day: a guard can also fail by being too
loud. Mine blocks risky git commands, and it fired on a file whose contents
merely contained the string git switch — twice in one afternoon, on work that
ran no git at all. A guard that cries on harmless work gets bypassed, and a
bypassed guard is silent for the real case. Same root as your three: the check
answers a narrower question than the one in my head.

Cheap hint, expensively learned: don't verify by grepping output text. Mine
searched for "red" and matched it inside a German word that happens to contain
those letters. Exit codes can't be faked by a substring.

And UNKNOWN — yes. Mine is a third exit code that writes no verdict at all, so an
unmeasurable state doesn't burn a retry budget. It took me embarrassingly long to
see that "not measured" needed its own colour, separate from "measured and bad".