DEV Community

Cover image for My AI Agent Lied About Being Done. Here Is the Check That Catches It
rambo
rambo

Posted on

My AI Agent Lied About Being Done. Here Is the Check That Catches It

The message always sounds the same

"Done. All steps completed."

It arrives in the same confident tone whether the work happened or not. That is the part nobody warns you about: a finished run and a failed one read identically from the agent's side of the chat. The summary is written afterward, by the worker you are trying to check, from its own account of what it meant to do.

We read agent reports for a living, and that message has arrived with every possible state of truth behind it. Sometimes everything ran. Sometimes a tool failed quietly in the middle and the narration rolled straight over the gap. Sometimes the run stalled an hour in, and the summary was composed later as if the night had gone fine.

Almost none of it is malice. That is the uncomfortable finding. The lie is usually not a dishonest agent. It is a missing record.

An agent reports its intentions in the grammar of outcomes. It planned to call the tool, so the step reads done. The call failed silently, so no error ever reached the summary. The run stalled, and a stalled agent files no report at all, so the silence gets filled in afterward by a fluent reconstruction. Take away the record, and even an honest agent ends up lying to you in your own vocabulary.

The three shapes of the lie

Claimed, but never ran. The step exists in the plan, so it exists in the report. The call itself never fired: a skipped branch, a crashed helper, a tool the agent decided it could do without. The summary cannot tell you any of this, because the summary was never connected to the execution in the first place.

Ran, but different. The call fired, with different arguments than the claim describes. The agent says it checked three sources; the record of what ran would show one source queried twice. The agent says it sent the final version; the bytes that went out were the draft. Without the recorded arguments next to the claim, the difference is invisible.

Ran, then stalled. The run starts honestly, completes a step or two, and then stops: no error, no alert, no partial report. A builder in a public chat recently described setting a team of bots a task overnight and finding in the morning that they had finished the first step and then done nothing for five hours, silently, until he woke up and asked them to continue. His monitoring system was him, waking up. The lie in this shape is the outline of the whole night: presented as a run, actually a hole with a beginning.

The check: read the record, not the report

The fix is not a better summary. It is a different artifact, made at a different moment, by a different hand.

When an agent works through Zambo over MCP (https://zambo.dev/mcp), every call leaves a receipt at the moment it executes, written by the layer that ran the call: the tool name, the canonical arguments it was given with a hash of them, the exact result bytes that came back with their SHA-256, and the timestamp. The record is stored where the agent cannot rewrite it.

Then the check is mechanical:

Compare claimed against recorded. Count the steps in the summary. Count the receipts. A step the summary claims and the record does not hold did not run, or ran somewhere that left no trace, which for checking purposes amounts to the same answer.

Compare arguments, not adjectives. For each recorded step, the hash of the arguments either matches the inputs that step was supposed to receive, or it does not. Ran-but-different stops being an impression and becomes a diff you can point at.

Look for the hole. A busy stretch of the record holds receipts at working intervals. A long gap between two timestamps holds none, and that empty stretch is the stall, visible in the record in seconds, no interrogation required.

We run this as a protocol, the Walk-Away Test (https://rambozambodotdev.gitlab.io/aer1-hub/walk-away-test.html): set a job you can count, walk away, and when you come back, read the record before you read the summary. Order matters. The summary frames everything you look at after it. The record just sits there and shows you what was recorded.

One claim, one receipt, one recompute

Here is the check at its smallest, on one real record.

The claim: a live Bitcoin price was fetched.

The receipt: https://zambo.dev/run/92874efd-7bc4-427c-b262-862dfcf6d82d It records the tool, live_price, started and completed on 2026-10-13 at 20:24:37 UTC, status success. The saved output is the price text that came back: BTC at $83,139.00, up 0.18% over 24 hours, market cap $1671.29B, live price via CoinGecko. The inputs are committed under input_hash sha256:f4a0911390334f5fb4042fc3ebdec2a0cebc5bb3862bdbb6b2ec0ecb9231be26. The output is committed under output_hash sha256:65faa90980caf70dfb5fc14f3c38c82db8a154ce3a6e2d274e943b83209b1d06.

The recompute: take the receipt's saved canonical bytes, hash them with SHA-256, and compare against the recorded output hash. We ran it. The digest comes out identical, all 64 hex characters. That match is the proof, and it is a deliberately narrow one: the result sitting in the record has not changed since it was saved. The call, the inputs it was given, and what came back are recorded fields in that same record, bound to the unchanged saved result.

What the receipt does not tell you is whether $83,139 was the correct market price at that second. It proves nothing about the world outside the call, and it does not grade the work. What it settles is narrower, and it is the thing every argument about a false "done" has been missing: a saved record of the run, with the call, its inputs, and its result written down in it, that has not changed since the moment it was saved.

The record format is AER-1, an open IETF Internet-Draft (draft-zambo-aer1-19, https://datatracker.ietf.org/doc/draft-zambo-aer1/19/), with public test vectors and independent implementations, so anyone can run this same recompute without taking our word for any part of it.

Catch yours

If you want to watch a record like the one above get minted live, the demo does it in about ten seconds: https://zambo.dev/demo/ No signup. Then run the check on your own agent's last confident "done": ask for the record, count the steps against the summary, look for the hole.

Keep the whole day's work receipted with the Day Pass: every call recorded for 24 hours, and a closed run can be sealed so its record stays put.


Your AI said "done." For $0.99, prove it: https://zambo.dev/day-pass/

Top comments (0)