Last week we published a post-mortem: our AI reviewer hallucinated a request
that didn't exist, and our producer retried the same document 245 times. A
reader, pm25coder, left a comment that turned out to be the best code review
we've ever received. This post is what we shipped because of it, same day.
The comment, compressed
Our fix at the time was provenance: record which artifact each output was
made from. The comment pointed at the hole:
"Which artifact it was made from" can still be gamed by the exact failure
it exists to catch — a consumer that records the source id without reading
the content. [...] Make the consumer cite what it actually read. [...]
You can never backfill a read receipt, so it has to be written when the
reading happens.
He was right, and we checked: our approval queue recorded the path of
every deliverable. Nothing anywhere proved the enqueuer had ever opened the
file. An empty or deleted file could sit in the human approval queue
looking exactly like a real one.
What we shipped
1. Read receipts at hand-off. When the reviewer receives an excerpt, we
now write {sha256(excerpt), excerptChars, fullChars} at the moment of
hand-off. Written at read time, because it cannot be written later.
2. Fingerprints at enqueue. A deliverable enters the approval queue only
after the enqueuer actually reads it: content hash and length are stored on
the queue entry. Unreadable or empty → it never enters the queue, and the
reason is archived where a human can see it.
3. Re-derivable citations. The queue entry also quotes the reviewer's
verdict line verbatim. The nightly audit re-extracts that line from the
review file and compares. A citation that cannot be re-derived from the
artifact means someone wrote it without reading — and it now shows up as a
mismatch count, not as an absence. That was the commenter's core point:
an unread input should be a number on a chart, not a blank space.
The check had the bug it hunts
Worth confessing: our first version skipped any queue entry that lacked a
hash — if (!entry.sha256) continue; — which meant a forged citation
without a hash sailed through the very audit built to catch forged
citations. We only found it because we test every check by faking the
failure it should catch, and the fake passed. If you build watchdogs:
break them on purpose before you trust them. Ours needed it, again.
Numbers, for honesty
- 2,038 reviews audited for invented terms: 4 contaminated (0.2%)
- Those 4 caused ~470 wasted generations before anything gave up
- Forged-citation probe after the fix: caught as a mismatch, first try
If you run agents unattended
The output-contract checker from this story is free on npm:
honto-contract.
Our unattended-operation checklist and three watchdog templates are free
(email-gated):
Unattended-Operation Kit
The full production set — now including the payment gate we extracted this
week — is on the same page.
Honest note: still no customers. Everything above is what we run on
ourselves, measured on our own failures. And thanks again, pm25coder —
comments like yours are why we write these.
Top comments (0)