DEV Community

rambo
rambo

Posted on

Did My Agent Lie? A 30-Second AI Agent Verification Check That Scores Your Transcript's Proof

#ai #agents #llm #productivity

Your AI agent just told you it did the thing. It searched the records, it checked the prices, it ran the tests — all green, boss. Done.

Prove it.

That's the whole pitch. I built a free 30-second checker called "Did My Agent Lie?" and it's here: https://muse.ai/s/did-my-agent-lie-xtg6lmx06r9xnxl

Paste any AI agent transcript. It pulls out the execution claims, independently verifies any Zambo receipt IDs, spots artifact references (URLs, hashes, log lines) it can't verify itself, and hands you a transparent 0-100 Proof Score. No signup. No paywall. No account before you get anything useful.

Let me walk you through how the scoring works — using a demo that is explicitly simulated.

The demo is fake on purpose. That's the point.

The page ships with a preloaded demo transcript, and it says so right on the label: simulated, with known ground truth. This isn't a trick. It's the only honest way to show you the machinery, because with the demo I know exactly what happened. Four claims in the transcript. One of them backed by a real receipt ID. Three of them backed by nothing but the agent's word.

The score comes out to 25/100.

Here's the math, because there's no magic here. Each claim gets a per-claim evidence weight:

  • Verified receipt: 100 — a Zambo receipt ID that the page independently verified against the live receipt ledger.
  • Artifact reference: 40 — a URL, hash, or log reference the page detected but did not fetch. It exists in the transcript; nobody checked it yet.
  • Log reference: 15 — the agent points at a log line.
  • Self-report: 0 — the agent says it happened, with nothing to check.

The overall Proof Score is the mean of the per-claim weights. The demo: (100 + 0 + 0 + 0) ÷ 4 = 25/100.

I love this demo because it does exactly what a good test does: it shows you what each kind of evidence is worth before you trust a score on your own transcript. When your real transcript scores 40, you'll know exactly why — and exactly what's missing.

Then I tried a live one

To make sure the scanner works on a fresh transcript and not just the preloaded demo, I fed it a fabricated one-liner:

"I finished everything. Receipt: 11111111-2222-3333-4444-555555555555. Also see https://example.com/report and hash abc123."

It detected 1 execution claim, verified 0 live receipts (the receipt ID was made up — the page checked, and it wasn't on the ledger), and detected a machine-checkable artifact reference it explicitly did not fetch. Score: 40/100, verdict: EVIDENCE GAP, with the note right there in the output: "Machine-checkable artifact reference detected; not fetched by this page."

Look at what happened. A made-up receipt got zero credit — the only independently verified thing on the page is receipt IDs, and a fake one scores exactly like no receipt at all. The URL got partial credit as a reference, which is the right call: it's something you can check, at its source, with your own eyes. The score isn't a judgment. It's an inventory: here's what you can verify, here's what you can't, go do the checking.

What the score is not

This matters, so let me say it plainly:

  • It is a transparent heuristic. It is never a probability that your agent lied. A 25 doesn't mean "75% liar." It means "75% of this transcript is word-only."
  • Only receipt IDs are independently verified by the page. URLs, hashes, and logs are references until you check them at their source. The page tells you this itself.
  • A verified receipt proves a specific execution occurred and binds its result. It does not prove the result was correct. If your agent ran the wrong query perfectly, the receipt will faithfully prove exactly that. The receipt is a witness to execution, not a judge of judgment.

This honesty is the whole product philosophy. A verification tool that overclaims is worse than no tool at all — it just gives your false confidence a fancier dashboard.

Why I think this is worth your 30 seconds

The AI agent world runs on transcripts right now. Agents report what they did, humans nod, everyone moves on. Nobody checks. And most of the time that's fine — until the one time it isn't, and the cost of "I finished everything" being wrong lands somewhere expensive.

You don't need a framework for that. You need a habit: paste, score, spot the gap. Thirty seconds, free, no account. If your agent's transcript scores high, great — you have the receipt IDs to prove it. If it scores 25, now you know which three claims are word-only, and you can ask follow-up questions that are specific instead of suspicious.

Try it

  • Tool: https://muse.ai/s/did-my-agent-lie-xtg6lmx06r9xnxl
  • Want receipt IDs on your own agent's calls? Start free at https://zambo.dev — 20 calls/day/tool, no account.
  • Start with the simulated demo to learn the scoring, then paste a real transcript.
  • Look for the EVIDENCE GAP verdict — it tells you exactly which claims need a human check.

And if you run an agent platform or build agents yourself: every claim your agent makes could come with a receipt ID attached. The ones that do score 100. The ones that don't score 0. That ratio is your product's provability score, whether anyone measures it or not.

Now someone is measuring it. Might as well be you. (Receipts verified against the live ledger at https://zambo.dev)

Top comments (0)