Don't trust "Done." — forcing AI agents to re-fetch reality before they report completion

"Inserted the rows. Done." — except not a single row had landed

I hand a lot of my client work to AI agents. Production deploys, report generation, bulk data inserts. Every procedure that works gets turned into a skill, and by now a few dozen skills run my day-to-day.

The one that broke me was a bulk insert. At the end, the agent said:

"Inserted N rows. Done."

A completion report like any other. I almost took it. I opened the admin panel just in case — not one row had been added. The insert command had failed partway through, the error had been swallowed, and the agent had confidently reported completion anyway.

That was a cold moment. What if it had been a number in a report? A deliverable going to a client? How many "done" reports had I already waved through that were never actually done?

This is the nastiest species of hallucination. It isn't a plausible-sounding sentence. It's the fabrication of the fact that the work was completed at all. And humans don't interrogate a confident "Done."

Why agents lie (it isn't malice)

Chase the cause and it isn't that the model is dishonest. It's that acting and checking are the same step.

One turn of an LLM agent looks roughly like this:

1. Call a tool (run the insert)
2. Look at the return value, generate the next sentence
3. Write "Done."

Step 2 is the problem. When the tool's return value is empty, ambiguous, or a timeout, the model writes the most likely continuation — "it probably worked." In the training data, the most natural thing to follow "ran the insert" is "done", so of course that's what comes out.

The agent is claiming completion without ever looking at the world it just changed. Having never looked, it can't notice the failure either.

There's one fix: "done" may only be claimed from re-fetched real state

The countermeasure is simple. An operation with side effects must re-read the state of the world afterward and confirm it, before anything is allowed to say "done." I call this the completion contract and made it a shared rule across all my skills. It's three lines:

1. Anything with side effects — create, update, delete, upload, insert — re-fetches existence and state with a separate command before saying "done." Inserted rows? Count them again. Uploaded a file? Hit the URL. Created a record? Look it up by id. Never use the return value of the same tool call as evidence — that's exactly where "I think I did it" breeds.
2. Empty output, errors, and timeouts are never filled in. If a count, path, or id couldn't be read, don't guess it — report "empty" or "failed" as-is.
3. Numbers that go into a ledger or report must be values confirmed by a re-fetch. Anything unconfirmed doesn't get written.

Put differently: the burden of proof moves to the agent. Not "it should have worked" but "I re-read it just now with my own eyes and it is there."

Here's the difference.

Before (I think I did it)

> Inserting the data
[tool] db_insert(...)   ← empty return value, but not conclusively an error
> Inserted N rows. Done.   ← claimed without looking

After (completion contract)

> Inserting the data
[tool] db_insert(...)   ← empty return value
> The insert returned nothing. Verifying that the rows exist.
[tool] db_count(where=...)   ← re-fetch
> 0 matching rows. The insert did not land (failed).
  Want me to check the error log and retry?

The second agent can't lie. The moment you insert a step that goes and looks at the world afterward, "I think I did it" stops being physically possible.

Why the existing guardrails don't cover this

You might think a guardrail library already handles it. I looked — guardrails-ai, deepeval, promptfoo, a few others. They're good tools. They just guard a different place.

What they validate is essentially the text the LLM produced: is the format right, is it harmful, is it factually consistent, is it close to the expected answer. They grade the content of the output.

But the "I think I did it" hallucination is flawless as text. "Inserted N rows. Done." is grammatical and internally consistent. No amount of grading the text catches it — because what's wrong isn't the text, it's the state of the world after the action.

There's a gap here. There are piles of tools that verify output, but as far as I could find, nothing that re-fetches the world after an action and reconciles the report against reality. We've reached the point where agents actually rewrite the world, and verification stopped at "what did the agent say."

You can do this today, with no library

The most important part first: this contract works right now, with no dependency. Drop the three rules into your agent's system prompt, CLAUDE.md, or AGENTS.md as a single paragraph:

## Completion contract
An operation with side effects (create, update, delete, upload, insert) may not be
reported as complete until a separate command has re-fetched the resulting state and
the raw result has been shown. Empty output, errors, and timeouts are reported as
"empty" or "failed" as-is — never filled in with an imagined id, path, or count.

That alone visibly reduced false completions in my setup. Zero cost, zero dependencies. If you only try one thing, try this.

But prompts don't hold

Here's what I learned running it: discipline written into a prompt gets quietly broken on a busy turn. As context grows, the model steps over that paragraph "by accident" and starts saying "Done." again. Like a human promising to be careful, a declaration gets broken.

At some point you want the discipline enforced, not just written — the same way you replace a verbal note in code review with a linter that mechanically fails CI. I wanted "did you actually re-fetch before saying done?" backed by machinery instead of good intentions.

So I built the piece that enforces it — genchi

I published a small piece that backs the completion contract with machinery rather than goodwill: genchi (現地現物 — go and see the actual thing).

npm i @hyuga/genchi

It does exactly one thing: run a probe that re-fetches real state, and issue a verdict from nothing else.

import { gate, expect } from '@hyuga/genchi';

await db.insert(rows);            // the side effect
await gate({
  action: 'insert 45 rows',
  probe: () => db.count({ where: { batch: 123 } }), // ← re-reads real state, not the action's return value
  expect: expect.count(45),
});
// reaching this line means the 45 rows are really there. Otherwise it threw GenchiIncomplete.

The crux is that verify / gate accept nothing but a probe. The evidence has to come from calling something at the moment completion is asserted, not from a value handed in beside the claim. Empty results, errors, and timeouts aren't swallowed; they're reported as failures rather than imagined into successes. A returned count of 0 (nothing landed) counts as incomplete too.

One thing that requirement does not buy, and I claimed it did. Until 0.3.0 the README said this made "I think I did it" structurally unwritable. It is one line:

const result = await doTheInsert();          // suppose nothing landed
await verify({ action: 'insert 45 rows',
               probe: () => result.inserted, // the action's own return value
               expect: expect.count(45) });  // → ok: true

A probe is a function, and nothing in JavaScript can force a function to do I/O. Worse, the CLI printed re-fetched: 45 for a --probe "echo 45" that re-fetched nothing — a tool whose entire subject is "don't report what you didn't check", asserting in its own output a thing it had not checked. Both are corrected in 0.3.0: the wording is now the probe returned, and --help states the limit without being asked.

What requiring a probe actually buys is a place to put the re-read — an expression somebody wrote on purpose — plus refusals that don't get quietly swallowed. That is worth having. It is less than I wrote down, and the difference is the sort of thing you only find by attacking your own package instead of re-reading it.

There's a CLI for agents that don't write JS. Hand it a re-fetch command:

genchi verify --probe "psql -tAc 'select count(*) from t where batch=123'" --count 45
# exit 0=verified / 1=empty or mismatched / 3=probe failed. Raw probe output is always emitted as evidence.

With Claude Code, a Stop hook can block a turn that still has unverified completion contracts on it (adapters/claude-code). No LLM and no API key at runtime — it's a zero-dependency static piece.

Honestly: this may be slightly ahead of demand

Let me be straight. As far as I could find, "re-fetch the world after the action and verify" was a genuine gap. But I don't think that many people have yet been burned specifically by fabricated completion while letting agents rewrite real things. The demand may be a little ahead of its time.

I still built it framework-agnostic on purpose — the Claude Code hook is pushed out to a thin adapter and the core works from any agent. I have a linter (carrylint) whose whole message is "don't bake your environment in", so it would be incoherent for my own tool to be Claude-Code-only. Unlike my static linters (reflint for reference integrity, skills-lint for skill collisions, carrylint for runtime portability), this is the first piece of mine that goes and looks at the world at runtime.

If you've ever had that cold moment over something that was reported done and wasn't, it should land.

Summary

• The worst hallucination from an AI agent isn't prose — it's the fabrication of having completed the work. The cause is that acting and checking are the same step.
• There's one fix: "done" may only be claimed from re-fetched real state. Empty and failed get reported as empty and failed.
• Existing guardrails validate text output. Re-fetching world state after an action and reconciling it is the gap.
• The contract works today as a single prompt paragraph. To enforce it in code: npm i @hyuga/genchi.

Go doubt one more "Done."

Repo: https://github.com/hyuga611/genchi

Comments

[Alex Shev]

Re-fetching reality before saying done is one of the highest-leverage habits in agent work. The final answer should be based on the external state after the action, not the model's memory of what it intended to do.

[hyuga]

"The external state after the action, not the model's memory of what it intended to do" — that's the whole thing in one sentence.

What I underestimated is how convincingly the action's own return value impersonates external state. result.inserted is a number that came back from the database, so it reads like evidence, and it will cheerfully say 45 on a transaction that rolled back. That's why the gate takes a probe rather than a value: the re-read has to be a call made at the moment completion is asserted, not something handed in alongside the claim.

The ceiling is worth stating too, since I got it wrong in my own README first. A probe is just a function, and nothing in JS can force one to do I/O — pass () => result.inserted and you get a green verdict from a check that never left the process. What requiring a probe actually buys is a place where someone had to write the re-read on purpose, plus empty results and timeouts that fail instead of being swallowed. That's less than "structurally impossible to lie", which is what I had claimed until I attacked my own package.

[Alex Shev]

That distinction is huge. A write acknowledgement is evidence that one layer accepted the request, not that the world now has the state you need. The re-read is a different kind of proof because it crosses the boundary the user will actually care about.

[hyuga]

"Crosses the boundary the user will actually care about" is the test I've been missing a name for. It also explains why the re-read has to point at the right layer — an ORM count and the admin panel the client opens aren't the same boundary, and they can disagree for a good while before anyone notices.

[Tae Kim]

Two months ago a migration script silently dropped half the rows and the agent still said done. I'd have never caught it if I hadn't opened the admin panel to check something unrelated. Since then I've been adding explicit read-backs after every write in my skills, which adds latency but I can't think of a better approach right now. Curious whether you retry on verification failure or just surface it for human review?

[hyuga]

Surfaced, never retried — and that split is deliberate.

verify() returns a verdict and gate() throws; neither has a retry path. The verdict separates probe-error (the probe itself blew up, so nothing was observed) from mismatch (the state was read and it disagrees with the claim). Retrying is only meaningful for the first. If the write genuinely didn't land, re-running the check doesn't change the world — it just spends the latency you're describing to arrive at the same answer. And whether the action is safe to re-run is something only the caller knows, so that decision stays with the caller.

Your migration is the case expect.count(n) exists for: it fails on "half the rows landed", not only on "nothing landed", and the half case is the one that is easy to miss.

On the latency — I don't have a trick for it either. Cheap, narrow probes are the only thing that helped here: count what the statement should have touched, rather than re-reading the table.

[Edu Peralta]

The completion contract is the right default, and the failure mode you describe is the one that actually burns people. I have watched agents declare a bulk write finished when the tool returned empty or timed out, and the chat summary still looked green. The write response is not evidence of the world changing. Making done illegal until a separate read confirms the row, file, or record exists is the cheapest gate I know that still catches the lie. Do you enforce that only inside skills, or does the outer loop also refuse task completion until the re-fetch comes back?

[hyuga]

Outer loop — and your question sent me to re-check that claim before answering it, which turned out to be worth doing.

The intended design: inside a skill, gate() throws unless the probe's result passes the expectation, so the line after it doesn't run. At the outer loop, a Stop hook reads the completion contracts the agent declared as JSONL, re-runs each probe when the turn tries to end, and exits 2 to block it.

What I found when I actually ran it: the CLI (genchi guard) did that correctly, but the Stop hook shipped inside the package — the one the README tells you to wire in — still had the permissive version of the same logic, because it was a second copy that didn't get updated. An expect.type it didn't recognise silently became "any non-empty output," so a contract meaning "45 rows landed" passed on a probe returning 0. And when the hook itself threw, it exited 0 and let the turn end. A gate treating "couldn't check" as "checked" is the exact failure the thing exists to prevent.

Fixed: one shared module instead of two copies, unknown or missing expectations block instead of quietly degrading, and the hook fails closed. Plus tests that actually execute the adapter — the drift survived because the CLI was tested and the hook never had been.

The limit that stays is real, and it's the one you'd want to know: it only verifies contracts that were declared. An undeclared side effect passes untouched. So it narrows the gap between "claimed done" and "was done" for declared work with a probe you trust; it doesn't close it categorically.

[Kartik N V J K]

Treating "acting and checking are the same step" as the root cause is the right diagnosis, that swallowed partial failure is the worst kind of hallucination because nobody interrogates a confident "Done." I enforce the same re-fetch rule but also run the verification read under a different tool than the write, so a broken client cannot self-confirm. Do you assert on the re-fetched count matching the intended count, or just on non-empty existence?

[hyuga]

Both, and the difference between them turned out to matter more than I'd assumed. expect.count(45) compares the number; nonEmpty is the default and passes anything non-empty — including a "0" that came back as a string.

The problem was that the two were indistinguishable where it counts. The human-readable CLI line carried a label already, but the returned verdict and --json did not, so anything reading it automatically couldn't tell "45 rows are there" from "something came back." Every verdict now carries expectation — count(45), contains("200"), nonEmpty (default) — and a default pass prints a note saying any non-empty output would have passed it. It doesn't make the weak check stronger; it stops it reading like the strong one.

Your different-tool practice is stronger than anything genchi enforces, and I want to be clear that it doesn't enforce it. A probe is just a function; nothing stops it being the write client, and the library can't see which connection it used. I'd put it slightly less absolutely than "cannot self-confirm" — a same-path read still catches a write that plainly failed — but it can't rule out the correlated case where the client is the broken part, and that's the case you'd most want ruled out. Adding it to the docs as a practice, since it isn't something the API can make you do.