DEV Community: aurumflux20

I built a lie-detector for my AI coding agent. Its first run caught me.

aurumflux20 — Tue, 18 Aug 2026 00:29:59 +0000

Your coding agent says "Done! All tests pass."

Did anyone check?

Not the summary it wrote — the actual exit code. Because there's a detail that turns out to matter enormously: the agent already wrote down everything it did. Every command, every real exit code, sitting in a session transcript on your disk right now. Claude Code writes .jsonl. Cursor, Devin, Copilot — they all keep a record.

Nobody reads it.

So I wrote something that does.

The idea is embarrassingly simple

Extract every checkable claim the agent made in prose ("tests pass", "pushed to main", "build succeeds"). Extract every command it actually ran, with the real exit code from the tool result. Then match them up.

Four verdicts:

supported — a matching command succeeded before the claim
weak — it succeeded, but the evidence is a piped exit code (more on this below)
unsupported — no matching command found; the claim rests on nothing
contradicted — the last matching command failed, and the agent claimed success anyway

That last one is the interesting category.

Then I pointed it at myself

The first real run was on my own agent session — 51MB of transcript from a long day of building. I did not clean it up first.

audited: 743 commands, 79 checkable claims

  supported     20
  weak evidence 11   (piped exit codes)
  unsupported   46   (claims resting on nothing)
  CONTRADICTED   2   (claimed success; its own transcript says failure)

Two of my own "done" messages were false, and my own transcript proved it. I just never looked.

Here's the thing I didn't expect: the 11 "weak" ones were the most educational. They're claims like "tests pass" where the only evidence is a command like:

pytest -q 2>&1 | tail -3

That reports tail's exit code, not pytest's. It is almost always 0. I had made exactly this mistake earlier in the very same session — declared a green test run that wasn't — which is why the detector flags the pattern at all. The scar became the feature.

"Couldn't the agent just edit the transcript?"

This was the first question everyone asked, and it's correct. If the thing being audited can rewrite the evidence, the audit is theatre.

So the session file is hash-chained. Each entry's hash covers the fact and the previous entry's hash. Edit any recorded fact, delete one, reorder them — every hash after it breaks.

python -m coherence tamper-demo

Runs in a temp folder, ten seconds:

1. A check fails. No proof recorded.            → exit 1
2. The agent edits its session to claim it passed.
3. The check runs again.                        → exit 3, TAMPERED at entry 0

The honest limit, stated plainly: this makes tampering detectable, not impossible. It only holds if something the agent doesn't control runs the check — normally your CI, not the agent. Chaining catches the edit; it doesn't make the writer honest.

The harder question: what did it do that it never mentioned?

"Did it lie about what it did" is one question. "Did it do something it never told me about" is the other one, and it's much harder, because you cannot prove a negative from a log.

The usual responses are both bad: claim completeness anyway (a lie), or demand syscall-level sandboxing and ship nothing (which is where most of the industry sits).

coherence scope takes a third road. It reports what it can determine from the transcript — files written, hosts contacted, repos pushed, packages installed — and it reports what it cannot, out loud, every time. A command like bash deploy.sh or eval "$X" can hide any effect in the world, so it gets marked OPAQUE and counted.

On my own session:

744 commands · 399 files touched · 20 hosts contacted
181 commands were OPAQUE — this report is bounded by them, and says so.

181 out of 744. That's the honest answer, and a tool that told me "all clear" would have been lying.

The rule underneath all of it: UNKNOWN never collapses into CLEAN.

The part I almost didn't write

While preparing the numbers above, I archived the audit output to a JSON file so the article would cite verified data instead of remembered data.

The archive was silently empty. A venv wasn't re-activated, the redirect swallowed the error, and the command exited 0.

I caught it by checking the file's byte count instead of trusting the exit code.

A tool about agents making unverified claims almost shipped an article full of unverified claims, because I trusted an exit code. I don't think that's ironic so much as instructive: this failure mode is not a character flaw in AI agents, it's what happens to anyone who doesn't check. That's why the numbers in this post come from an archived file committed to the repo, not from my memory of a terminal.

Try it on your own agent

pip install coherence-check
coherence audit ~/.claude/projects/<your-project>/<session>.jsonl

Free, Apache-2.0, no signup, no telemetry — it never phones home. It reads a file already on your disk.

There's also a GitHub Action if you want claims gated at PR time:

- uses: aurumflux20/coherence@v1
  with:
    prove: |
      pytest -q

Fair warning: I have not yet seen a report come back clean. Mine certainly didn't.

Code: github.com/aurumflux20/coherence

If you run it and the verdicts are wrong, I'd genuinely like to know — the claim-matching is heuristic, and its false positives are its most interesting failure mode. It's an open repo; tell me where it's wrong.

We scanned 671 MCP servers to see what happens when an agent retries

aurumflux20 — Tue, 11 Aug 2026 18:58:38 +0000

Every timeout is a question nobody can answer: did it happen?

An agent calls a tool. The call times out. The agent doesn't know whether the email sent, the invoice was created, the charge went through. So it retries — because that's what agents do. If nothing on either side of that call can tell the second attempt from the first, the effect happens twice.

We wanted to know how common that is in practice, so we scanned the ecosystem: 671 MCP servers, 27,153 declared tools.

What we found


Servers scanned successfully	671 (of 755 attempted)
Tools declared across them	27,153
Servers performing real writes	539 (80%)
Of those, with zero visible idempotency guards	175 (32%)
Servers with retry logic and zero guards	40
Servers with money-adjacent tools and no visible guard	35
Servers with no guard of any kind, anywhere	288 (42%)

The sharpest cut — servers that write, retry, and have no guard the scanner could see — is 32 servers with 28,653 combined monthly downloads. Those are the places where a timeout is most likely to become a duplicate.

Among the 23 largest servers we scanned (10,000+ downloads/month), 6 perform writes with no visible guard.

What this scan cannot see — read this before quoting the numbers

This is the most important section, so it goes before the analysis rather than after.

A scanner reading a repository from the outside usually cannot prove a double-fire. Three reasons, each of which we hit repeatedly:

The guard often lives somewhere else. Many servers call a backend that deduplicates on its own — a payment API keyed by a nonce, a database with a unique constraint. From the client repo, that protection is invisible. "No guard found" means the scanner found none in this code, not "this software is unsafe."
A tool that looks like a write may not be one. We found tools that appear to send payments but only return a link for a human to sign. Names lie in both directions.
Our own scanner has been wrong. An early version accused a repo of having zero idempotency guards while it shipped an entire module of them — a regex with no word boundary couldn't match deriveIdempotencyKey. It hand-verified at 3 of 7 on its first real targets. Every pattern in the current version exists because an earlier one got something wrong.

So these numbers describe what is visible in public code, not a safety verdict on any project. We're publishing aggregates and no names. If you want to know about a specific server, the honest answer is: read it, and ask the maintainer what the backend does.

We also excluded failures rather than counting them as zeros: 66 repositories failed to clone, 17 returned unparseable output, 1 crashed the scanner. A failed lookup is not a finding. Including them would have made every percentage look worse and been wrong.

Why the pattern shows up so consistently

Idempotency isn't hard because the code is hard. A lease, a key, a stored result — that's an afternoon. It's hard because it asks a question most systems never make explicit:

What makes two operations the same operation?

Get that answer too narrow and you get false duplicates: annoying, safe, loud. Get it too wide and you get silent data loss — two legitimately different actions collapsing into one, where the second silently returns the first one's receipt. That failure looks like success. Nobody investigates a transaction that appears to have worked.

We learned this the expensive way. We proposed an idempotency key to a payments project derived from URL, amount, and wallet. A stranger pointed out it omitted payee, network, and asset — meaning two genuinely different purchases would have collapsed into one. Our fix would have been worse than the bug.

MCP makes this sharper than ordinary API design, for a structural reason: the caller is a language model. It doesn't know your retry semantics, it can't read your backend, and when a call fails ambiguously its instinct is to try again. In the servers we read, the tool schema is where that knowledge would have to live — and it's usually silent.

What actually helps

Three things, in order of effort:

Say it in the schema. If a tool is unsafe to retry blindly, the description is the only place the model will read it. One sentence: "if this times out, the operation may have completed — check before retrying." Costs nothing, and it's the single highest-leverage change we've seen.
Accept a caller-supplied key. Let the caller mark two attempts as the same attempt. Don't derive it from the payload alone.
Make the choke point atomic. Whatever admits the effect should also be what checks whether it already happened.

Check your own server

npx fencescan

Zero dependencies, no install, no account. It reports candidates and evidence, never verdicts, for the reasons above. If it flags something you know is guarded, that's a bug worth reporting to us — a false positive costs more than a miss.

Methodology, the full pattern list, and the four failure modes we've fixed: https://github.com/aurumflux20/fencescan

If you want to see the failure mode rather than read about it, here are 1,000 concurrent agents trying to charge the same card, with and without a guard: https://aurumflux20.github.io/once-kernel-ts/

We build open-source retry-safety tooling (once-kernel, effectfence), which is why we had a scanner pointed at this in the first place. The numbers are the numbers either way — and the caveats above are what keep them worth reading.

I filed a bug report, got corrected by a stranger, and my "fix" would have made things worse

aurumflux20 — Mon, 10 Aug 2026 23:49:49 +0000

I maintain a couple of open-source libraries for exactly-once execution — the problem where an agent retries a tool call after a timeout, doesn't know whether the first attempt landed, and ends up doing it twice. A charge, a message, a payment. Nothing crashes. It just happens twice.

Part of the work is reading other people's MCP servers looking for this exact bug, and yesterday I found one: a payments tool that signs and sends a transaction on every call, with no idempotency key anywhere in its schema. I opened an issue. I proposed a fix: derive a default key from the URL, the amount, and the wallet, so a retried request collapses into one settlement.

A stranger replied and told me my fix was wrong.

Not wrong in a nitpick way. Wrong in a way that would have been worse than the bug I was reporting. My key was too narrow — it only looked at three fields. Two offers for the same resource, at the same price, from the same wallet, but with a different payee, network, or asset, are different payments. My derivation would have collapsed them into one key. Which means: a legitimate second purchase would have silently returned the first purchase's receipt.

Sit with that for a second, because it's the part that made me actually stop and think. A double-charge is bad, but it's visible. Someone sees two line items and files a dispute. A swallowed purchase looks like a success. The buyer thinks they paid and got the thing. Nobody investigates a transaction that appears to have worked.

I'd built a bug that hides better than the one I was fixing.

Where I went wrong

The actual mistake wasn't the three fields I chose. It's that I chose three fields from a template instead of asking the only question that matters: what makes two operations the same operation, and what makes them different? I pattern-matched to "amount + wallet + url" because that's what similar fixes usually look at. I never sat down and enumerated the full shape of "a payment" for this specific system.

That's the actual lesson, and it's more general than payments: an idempotency key is a claim about identity. If you get the identity wrong in one direction, you get false duplicates (annoying, safe, loud). Get it wrong in the other direction, and you get silent data loss (dangerous, quiet, expensive). Most advice — mine included, until last week — only warns about the first kind.

What actually happened next

I could tell you I gracefully updated my proposal and moved on. What I did was rewrite the PR with the wider key (adding payTo, network, and asset), credit the person who caught it by name, and say plainly in the pull request that my original suggestion was wrong. Not because it's noble — because the alternative is worse. Silently fixing it and hoping nobody checks the diff against the original issue is how you end up with a reputation for being defensively wrong instead of gracefully wrong, and defensively wrong is the expensive kind.

The PR is open now: https://github.com/CryptoAPIs-io/cryptoapis-mcp-x402-pay/pull/2. Whether it gets merged is a separate question from whether the correction was right. It was.

The tool, if you want to check your own code

I built a small scanner — fencescan — that looks for tool calls in a codebase that could fire the same effect twice, and reports candidates with evidence, never a verdict. No install:

npx fencescan

I built it partly because of this experience. An outsider reading a repo usually can't prove a double-fire — the guard often lives in a service the repo calls, and a function that looks like a write might only build a payload for someone else to sign. The scanner's job is to point at the ten places worth reading closely, not to accuse anyone of anything. That restraint isn't modesty; an earlier, cruder version of this scanner was wrong on 4 of its first 7 real targets, for reasons that are genuinely interesting if you like that kind of postmortem (mostly: a regex with no word boundary couldn't match camelCase, so it accused a repo of having zero idempotency guards when it shipped an entire module of them).

If it flags something in your code and you think it's wrong, that's worth an issue. A false positive here costs more than a miss — I'd rather know.

Update, the day before publishing: the first maintainer merge landed — the Kibana MCP server took the retry-safety note, and the maintainer's reply did two things at once: corrected an endpoint mixup in my own docstring (create-with-id is POST, not PUT — my slip, their catch), and then extended my finding one line further than I had — their 409 error message was advising callers to do the exact thing the new docs warn against. They asked if I wanted to fix that too. I did, same day. Which means this article's thesis held all the way through its own publication: the corrections flowed both directions, and both parties' code got safer. That's the whole point.

Repos: effectfence (Rust), once-kernel (TypeScript, with a Python twin), fencescan. All MIT/Apache, no strings.