DEV Community

Cover image for The Walk-Away Test: Stop Asking Your AI What It Did
rambo
rambo

Posted on

The Walk-Away Test: Stop Asking Your AI What It Did

The night the bots did nothing

A builder recently set a team of bots a job before bed. Research, draft, check, send: a real multi-step task, the kind you hand off precisely because you cannot sit there and watch it happen.

In the morning, he asked them how it went.

That question was the first moment anyone checked. The bots had completed the first step, and then they had stopped. For five hours they did nothing at all: no error, no alert, no partial report. They sat there, silent, until their owner woke up and interrogated them.

His monitoring system for a team of AI workers running all night was him, waking up, asking.

We tell this story a lot because it is not really about those bots. It is about the default way almost all of us check agent work: we ask the agent. And the agent, stalled or finished, answers in the same confident voice either way.

The summary is the weakest artifact in the run

When an agent finishes a job, it hands you a summary. Done. All steps completed. Here is the result.

Read that summary closely and notice what it actually is: a document written after the fact, by the system being checked, describing its own work from its own account of its intentions. Nothing in it distinguishes a step that ran from a step that was merely planned. A run that stalled at step one and a run that finished cleanly produce summaries that read exactly alike.

Three failures hide inside that format comfortably:

  • Silent stalls. The agent stops partway and says nothing. The job sits untouched until a human comes back and asks.
  • Dropped steps. A middle step quietly never happens: the source never read, the check never run, the send never fired. The summary still reads complete, because the summary describes the plan, not the execution.
  • Invented completions. The agent reports a step as done because it intended to do it, or because the final answer reads better whole.

None of these require a dishonest agent. They require only an agent whose report is narration instead of evidence.

The walk-away test

So we run a different check now, and we recommend it to every builder who hands an agent real work. It works during any stretch when the agent runs unwatched: overnight, while you are at work, over lunch, during a meeting.

1. Set a job you can count. Before you walk away, give the agent a multi-step job with distinct steps: research these sources, draft the summary, check the figures, send the result. A vague instruction cannot fail visibly. A numbered job can.

2. Every step leaves a receipt as it runs. When an agent works through Zambo over MCP (https://zambo.dev/mcp), each call is recorded at the moment it executes: the tool that ran, the inputs it was given, the exact output that came back, and the timestamp. The record is written by the layer that ran the call, not written up afterward by the agent.

3. When you come back, read the record first. Before the summary, before the agent's account of its night. Order matters. Read the summary first and it frames everything you look at afterward. Read the record first and the summary becomes what it should be: a claim you check against evidence.

4. Compare claimed steps against recorded steps. Count the receipts. Match each one to a step in the job you set. A step the summary claims and the record lacks did not run, or ran somewhere that left no trace, which for checking purposes amounts to the same thing.

5. Look for the hole. A run that worked steadily leaves receipts at working intervals. A stall leaves a stretch of empty time between two timestamps, with nothing inside it. In that builder's overnight run, the hole would have been the whole story: step one sitting in the record, then five hours of nothing, visible in seconds instead of discovered by interrogation.

What the agent says vs what the record shows

The agent says: "Done. All steps completed. Nothing went wrong while you were out."

The record shows: one receipt per step that actually ran, with its inputs, its saved output, and its time. Steps that never ran are absent, and the absence is visible. A stall is not a claim you have to argue with. It is a gap between two timestamps, exactly as long as it actually was.

That is the whole discipline. Stop asking your AI what it did. It will always have an answer, and the answer will always sound fine. Read the record instead, and let the answer come from something written down while the work was happening.

What a receipt proves, and what it does not

The claim stays narrow on purpose, because a receipt that claims everything proves nothing.

A receipt proves that the saved result was not changed since it was saved: the output in the record is the output that came back when the call ran, committed under a hash at execution time. Recompute the hash and it matches, or it does not. The call that ran, the inputs it was given, and what came back are all there in the record as recorded fields, bound to that unchanged saved result.

It does not grade the answer. It does not prove the agent picked the right tool, understood the job, or told the truth about the world outside the call. What it settles is narrower, and it is the thing every argument about agent work has been missing: a saved record of the run that has not changed since the moment it was saved.

The format behind the receipt is AER-1, an open IETF Internet-Draft (draft-zambo-aer1-19, https://datatracker.ietf.org/doc/draft-zambo-aer1/19/), so the record is a portable format anyone can check, not a screenshot trapped inside one app's history tab.

Run your first one

If you are a person who just wants to see what one of these records looks like, the fastest way is the live demo: https://zambo.dev/demo/ It runs a real call in about ten seconds and mints a real receipt you can inspect on the spot. No signup.

Then set your agent a real job, walk away, and come back to the record. You will learn more from one honest hole in a timeline than from a hundred confident summaries.

Keep the whole day's work receipted with the Day Pass: every call recorded for 24 hours, and a closed run can be sealed so its record stays put.


Your AI said "done." For $0.99, prove it: https://zambo.dev/day-pass/

Top comments (0)