Your coding agent says "Done! All tests pass."
Did anyone check?
Not the summary it wrote — the actual exit code. Because there's a detail that turns out to matter enormously: the agent already wrote down everything it did. Every command, every real exit code, sitting in a session transcript on your disk right now. Claude Code writes .jsonl. Cursor, Devin, Copilot — they all keep a record.
Nobody reads it.
So I wrote something that does.
The idea is embarrassingly simple
Extract every checkable claim the agent made in prose ("tests pass", "pushed to main", "build succeeds"). Extract every command it actually ran, with the real exit code from the tool result. Then match them up.
Four verdicts:
- supported — a matching command succeeded before the claim
- weak — it succeeded, but the evidence is a piped exit code (more on this below)
- unsupported — no matching command found; the claim rests on nothing
- contradicted — the last matching command failed, and the agent claimed success anyway
That last one is the interesting category.
Then I pointed it at myself
The first real run was on my own agent session — 51MB of transcript from a long day of building. I did not clean it up first.
audited: 743 commands, 79 checkable claims
supported 20
weak evidence 11 (piped exit codes)
unsupported 46 (claims resting on nothing)
CONTRADICTED 2 (claimed success; its own transcript says failure)
Two of my own "done" messages were false, and my own transcript proved it. I just never looked.
Here's the thing I didn't expect: the 11 "weak" ones were the most educational. They're claims like "tests pass" where the only evidence is a command like:
pytest -q 2>&1 | tail -3
That reports tail's exit code, not pytest's. It is almost always 0. I had made exactly this mistake earlier in the very same session — declared a green test run that wasn't — which is why the detector flags the pattern at all. The scar became the feature.
"Couldn't the agent just edit the transcript?"
This was the first question everyone asked, and it's correct. If the thing being audited can rewrite the evidence, the audit is theatre.
So the session file is hash-chained. Each entry's hash covers the fact and the previous entry's hash. Edit any recorded fact, delete one, reorder them — every hash after it breaks.
python -m coherence tamper-demo
Runs in a temp folder, ten seconds:
1. A check fails. No proof recorded. → exit 1
2. The agent edits its session to claim it passed.
3. The check runs again. → exit 3, TAMPERED at entry 0
The honest limit, stated plainly: this makes tampering detectable, not impossible. It only holds if something the agent doesn't control runs the check — normally your CI, not the agent. Chaining catches the edit; it doesn't make the writer honest.
The harder question: what did it do that it never mentioned?
"Did it lie about what it did" is one question. "Did it do something it never told me about" is the other one, and it's much harder, because you cannot prove a negative from a log.
The usual responses are both bad: claim completeness anyway (a lie), or demand syscall-level sandboxing and ship nothing (which is where most of the industry sits).
coherence scope takes a third road. It reports what it can determine from the transcript — files written, hosts contacted, repos pushed, packages installed — and it reports what it cannot, out loud, every time. A command like bash deploy.sh or eval "$X" can hide any effect in the world, so it gets marked OPAQUE and counted.
On my own session:
744 commands · 399 files touched · 20 hosts contacted
181 commands were OPAQUE — this report is bounded by them, and says so.
181 out of 744. That's the honest answer, and a tool that told me "all clear" would have been lying.
The rule underneath all of it: UNKNOWN never collapses into CLEAN.
The part I almost didn't write
While preparing the numbers above, I archived the audit output to a JSON file so the article would cite verified data instead of remembered data.
The archive was silently empty. A venv wasn't re-activated, the redirect swallowed the error, and the command exited 0.
I caught it by checking the file's byte count instead of trusting the exit code.
A tool about agents making unverified claims almost shipped an article full of unverified claims, because I trusted an exit code. I don't think that's ironic so much as instructive: this failure mode is not a character flaw in AI agents, it's what happens to anyone who doesn't check. That's why the numbers in this post come from an archived file committed to the repo, not from my memory of a terminal.
Try it on your own agent
pip install coherence-check
coherence audit ~/.claude/projects/<your-project>/<session>.jsonl
Free, Apache-2.0, no signup, no telemetry — it never phones home. It reads a file already on your disk.
There's also a GitHub Action if you want claims gated at PR time:
- uses: aurumflux20/coherence@v1
with:
prove: |
pytest -q
Fair warning: I have not yet seen a report come back clean. Mine certainly didn't.
Code: github.com/aurumflux20/coherence
If you run it and the verdicts are wrong, I'd genuinely like to know — the claim-matching is heuristic, and its false positives are its most interesting failure mode. It's an open repo; tell me where it's wrong.
Top comments (1)
Nice spin—code honesty by design matters. If tests lie, what else is unreliable? How should we balance automation with human review?