A hallucination checker returned PASS and cited its source. A consistency checker returned PASS across all eight surfaces. Both were correct. The sentence they cleared was false.
In June a video of mine went into the publish queue teaching people the wrong arithmetic about their own birth year. The Japanese narration said, in effect, 1990 plus 18 makes 2009. It does not. The same wrong rule sat in the cover copy, the formula slide, the intro, the outro, and the metadata — eight places in one piece, two days from a scheduled public release.
Five checks had already run on that file. None of them was asleep. I have gone back through what each one actually did, and the uncomfortable finding is that the two I can reconstruct in detail both returned PASS correctly.
Everything I write about this setup circles the same question: when nobody else is checking, how do I know the work is real? The last piece answered it with an org chart — the agent that does the work never grades it. Here the org chart was already correct. An independent checker, with its own job and no stake in finishing the task, looked straight at the bad sentence and cleared it. So this piece is about the part the reporting line doesn’t determine: what the checker compares the claim to.
The first gate said PASS, and it was right
The pipeline runs a grounding check before anything renders: every fact in the narration has to appear in the source article the video is built from. Its report on this file was clean — no new figures, no new proper nouns, verdict PASS — and it named the line in the source article where the rule lived.
It was telling the truth. The source article said birth year + 18. The narration said birth year + 18. Provenance confirmed.
The source was wrong. It had been wrong, publicly, on the site, for about six weeks.
A provenance check can only find claims that disagree with the source. Feed it a claim that agrees with a wrong source and it will hand you a citation. That is not a bug in the checker; it is the entire definition of what it does. And it gets worse for anything derived: the video’s memorable-number framing didn’t exist in the article at all, so there was no line to compare it against, and a claim with no comparison basis passes by default.
The second gate said PASS, and it was also right
A different check looks across every surface of the piece and demands that a number land the same way everywhere. Its output on the broken file, verbatim:
✅ add-N 단일(+18), 전 면 일관 (8곳)
One value. Eight surfaces. Perfect agreement — on the wrong number.
Consistency is not correctness. That check exists for a failure I hit constantly, which is the half-finished fix: I correct a number in the narration and forget the metadata, and the piece ships arguing with itself. Against that, it is excellent. Against an error that was uniform from the start, it is a green tick on a lie.
The check that was missing
Here is what makes this incident worth writing down. Nobody had to know anything about Japanese school entry, or read the source, or hold context, to know that sentence was false.
1990 plus 18 makes 2009 refutes itself. The operands and the asserted result are both sitting inside the claim. The comparison isn’t against a source or a sibling surface — it’s against the claim’s own arithmetic.
So the third check compares a claim to itself. The whole mechanism is one line:
if (a + n === b) return true;
That sits inside a 199-line script with three regexes for written arithmetic and one concession to the real world — a modulo, so that Korean and Japanese two-digit school-year notation resolves. It reads the script and metadata files, extracts every written triple, redoes the addition in integers, and exits 0 for pass, 1 for fail, 2 if you pointed it at nothing. In the pipeline that ships the video it runs per language, and the orchestrator’s contract is four words long: a nonzero exit throws.
I keep the broken file. Running the gate against it today still prints:
06-formula-18.text: 「1990足す18で2009」 → 기대 2008, 표기 2009
→ FAIL: 산술 모순은 절대 통과 불가
— exit 1.
The part that would be easy to lie about
I did not catch this with the script. The script did not exist.
The error was found by a person redoing the arithmetic by hand during an audit two days before release, and the gate was written roughly fourteen minutes after the corrected file landed. If I tell this story as the regex caught what the models missed, I get a much better story and a false one. What the regex bought was not that instance. It was the class: that shape of error now cannot reach the queue.
That distinction matters more than it sounds. Every gate I own was written the same way — after something got through. A gate is not foresight. It is a scar with an exit code.
| Comparison target | What a PASS establishes | What it does not establish |
|---|---|---|
| Source article | The claim agrees with its source | The source itself is correct |
| Other output surfaces | The same value appears consistently | The shared value is correct |
| Recalculation | The stated arithmetic holds | The real-world rule behind it is valid |
A verdict is a claim about a comparison. The three referents have different blind spots, and only one of them was available here.
Three things a check can compare a claim to
Working backwards from this, the useful question about any check is not how smart it is. It is what it holds the claim up against — and there are only three answers.
Against a source. Does this appear in the thing we’re supposed to be faithful to? Cheap, exact, and the right tool for invented facts. Blind to anything the source itself got wrong, and blind by construction to anything derived, because derived values have no line to match.
Against its siblings. Does this value agree with every other place it appears? Also exact — it reduces to counting distinct values across a known set of fields. Catches drift and partial fixes. Says nothing whatsoever about whether the agreed-upon value is true.
Against itself. Does the claim survive being recomputed? Only available for claims that carry their own refutation — arithmetic, dates, units, totals — but where it’s available it is unarguable. No world knowledge, no context, nothing to flatter.
The design rule I now use is narrow on purpose: if a claim contains everything needed to refute it, no model gets a vote. Not because models are bad at arithmetic — most are fine at arithmetic. Because a model asked does this look right? is answering a different question than does this add up, and it will answer the first one fluently while the second goes unasked.
The corollary is the part people skip. That rule covers a much thinner slice of “verification” than the word implies. Tone, market fit, whether a draft reads as translated, whether a piece overclaims — none of those has a comparison target you can write down, and I’ve watched the attempt fail: a translation-quality check specified in prose as a tidy similarity threshold turned out, when I went looking for the implementation, not to exist anywhere. It read like an algorithm and was a wish. The checks in my pipeline that do that kind of work are model critics with rubrics, and they stay.
There is one more category, though, and it’s the one the incident actually belongs to.
What no gate covered
Every automated check in that pipeline behaved correctly and faithfully reproduced an error, because the error entered upstream of all of them, in a source that was itself wrong. There is no check in my setup that asks is this rule true about the world. That question has never been delegated, and after this I don’t think it can be.
Which is also why the fix isn’t as clean as the story wants. +18 is not simply wrong — it's correct for people born in the early months of the year, under a rule the video was too compressed to explain. So the gate ships an explicit flag to allow more than one add-value, and the corrected video still teaches an 18 somewhere. The gate can be turned off at the orchestrator too, and nothing in CI enforces it, because there is no CI. The honest claim is not this cannot be bypassed. It's that the one path that actually ships a video runs it, and a nonzero exit stops the run.
One artifact from that night I’ve left alone deliberately. The corrected slide now reads birth year plus 19. Its id is still 06-formula-18 — the wrong number preserved in the name of the slide that got it wrong. I keep meaning to rename it and I keep not doing it, because a fossil that specific is worth more than a tidy filename.
Compared to what?
The lesson I took isn’t add more checks. It’s that a green tick is not a statement about the truth of a claim. It is a statement about one comparison, and the comparison is usually invisible in the output.
So now, when something reports PASS, the question I ask is: compared to what? If the answer is “the source,” I’ve verified faithfulness, not correctness. If it’s “the other surfaces,” I’ve verified agreement, not correctness. If it’s “itself,” I’ve verified the only thing I can trust while nobody is watching — and that answer is available far less often than I’d like.
The next piece is about the case where none of the three referents is available at all. My pipeline ships narrated audio in three languages and not one agent in it can hear a sound; the next essay, My AI can’t hear the audio it ships, is how it QAs that anyway — and why the workaround is a proxy I only half trust.
Originally published on Medium in August 2026. This DEV edition was prepared by an AI agent from that existing essay, with an image placeholder replaced by a text comparison table. The incident and implementation details describe the original retrospective, not a fresh audit of the current system.
Top comments (0)