Originally published on hexisteme notes.
I run a small pipeline that turns a shared brief into published pages, and it won't let a draft go live until it clears five separate automated checks. I recently used it to build three pages at once — a PayPal Goods-and-Services fee explainer, a general merchant-fee anatomy page, and a Stripe-vs-Square comparison — written in parallel by three separate workers from the same brief, each working against the same set of published rate pages from the providers themselves.
Every one of the three pages passed every one of the five gates on the first try — all five check one document at a time, and all five ran. warrant_gate confirmed that every number cited in the prose actually appears there. numeral_gate confirmed that every digit on the page traces back to a declared piece of evidence. expression_gate checked each page's prose against the sources it cited, for near-duplicate sentences. disclosure_gate and revenue_axis_gate passed too. On top of the five machine gates, each worker ran its own 7-item reader checklist before calling itself finished, and each reported 7 out of 7. The step that decides whether a draft is clean enough to put in front of a human exited clean — three times over.
Then one read-through of the actual rendered text — one script pulls all three pages down to roughly 5,500 words of running prose for a person to read in one sitting — rejected all three pages.
This isn't a gate that got lied to
It's worth being precise about what this wasn't. All five publishing gates had actually run, against real rendered text, checking real evidence entries, on each page on its own. Nothing here was a checklist item nobody executed, and no page asserted a number without a source for it. Every one of those five gates did exactly the job it was built to do, correctly. What got through lived in three narrower places instead: in the word standing next to a correctly sourced number, in what one page denied about a source its own sibling page was busy quoting, and — this is the one that stings — in a check that lives one layer downstream of those five, in the site builder rather than the publishing pipeline, whose entire job is comparing sibling pages against each other. For three pages each built on its own, that check never ran at all.
Three ways to be wrong while citing your sources correctly
1. The right number, the wrong owner
PayPal charges an additional 1.50% on what it calls an "international commercial transaction" — its term for a cross-border sale. Two of the three pages described that same 1.50% figure as triggering "when the buyer's card was issued outside the United States." That's a real trigger condition. It just belongs to Stripe and Square, not PayPal. The number was correct, it was sourced, and it was attached to the right provider's fee line — it was just carrying someone else's rule for when that fee applies.
A gate that checks "does this number have a citation" has no opinion on that sentence. The citation is real. The digit is real. The predicate wrapped around the digit was copied by pattern from a different company's rate page rather than re-read from the specific source sitting in front of the worker that used it.
2. A denial one page made that its own sibling disproved
The merchant-fee anatomy page stated that "neither PayPal nor Square publishes a keyed-in rate on the pages cited here." The Stripe-vs-Square page, built in the same run, quoted "3.5% + 15¢" for keyed-in cards — from the very same Square URL the first page had just declared silent on the subject.
Both pages were internally correct about their own citations. Both passed every gate that checks a page against its own sources. Set side by side, they flatly contradict each other, and nothing in the pipeline ever put them side by side while it still mattered.
3. Wording with nothing to check it against
The rest were smaller, but the same shape: "buyer protection," asserted with no evidence entry behind it, on a source page that never uses those words. "Each row below," describing a table that actually rendered above the sentence. "A flat 1.5% surcharge" — a percentage described with a word that means it doesn't scale. "Four tenths of a percentage point," a number spelled out in words in exactly the place no gate is looking for digits.
None of these needed a wrong citation. They needed no citation at all to be wrong — a non-numeric claim, a spatial reference, an adjective, a number written as words instead of digits. A pipeline built to check digits against sources has nothing to say about any of them, because none of them is a digit.
Why five green gates didn't add up to one correct page
Two structural gaps and one process failure, and the process failure is the one worth remembering.
warrant_gate and numeral_gate both check that a number is sourced. Neither reads the sentence wrapped around the number closely enough to know whose rule it's describing. "Sourced" and "attributed to the right actor" are different properties, and only the first one had a check.
A negative claim like "does not publish" has no number and no evidence cell to anchor to. A per-document gate can verify what a page asserts with a value. It has no mechanism for verifying what a page denies about a source it didn't fully quote — and even less way to know that a sibling page, built the same hour, was about to quote exactly that.
Catching the second gap at all would have needed something outside those five gates entirely. One layer downstream, in the site builder rather than the publishing pipeline, a separate check named NEAR_DUPLICATE_PAGE measures sentence overlap between sibling pages in the same build. It never ran for these three workers: each built its page alone, in its own single-page run, and a single-page run has no siblings to measure against, so the check is skipped — what the build reports for that situation is, in substance, "no sibling pages to compare". It first actually ran later, when the coordinator built the full site across nine pages, and passed. Every worker had still reported "gate PASS" for a check that, for them, had never once run.
The self-check missed it for a more human reason. The checklist item that should have caught the third defect class was worded as "no claim beyond the evidence," illustrated with a single example. None of the three workers generalized from that one example out to buyer protection, table position, or a spelled-out fraction. A checklist item is only ever as good as the example riding along with it.
What changed, and why it isn't a sixth gate
The fix went into the brief, not into the gate pipeline. Three new checklist items: each provider keeps its own trigger term, so a fee's condition gets re-read from that provider's own page instead of pattern-matched off a sibling's; never write that a source "does not publish" something — write that "this page does not compute" the number, which is a claim about what got fetched rather than a claim about what exists; and non-numeric claims need an evidence entry exactly the way numeric ones do, which also covers direction words, "flat," and spelled-out fractions. Separately, it's now written down that a single-page build skips the site builder's sibling-comparison check, and a report of "gate PASS" has to name which gates actually ran rather than asserting a clean run in general.
A sixth gate was the other option on the table, and it's worth explaining why it lost. Catching an unfalsifiable "does not publish" mechanically would mean fetching and fully indexing the entire text of every source cited anywhere in a batch, not just confirming that a cited number appears somewhere in it — a much bigger, slower, and more fragile system to maintain than the five gates already running. The brief rule removes the same class of defect for the cost of one sentence in a document a worker was already required to read.
What generalizes
The generic caution here is a familiar one: a passed check only tells you about whatever it actually looked at. What's worth keeping isn't that line by itself, but the specific shape it took across three independently gated documents. A fact can ride on a citation that is completely correct and still describe the wrong actor's rule, because "this number has a source" and "this number's condition belongs to the company it's attributed to" are different properties, and only the first one anywhere had a check. A claim can be false in a way nothing catches simply because nothing else in the batch was compared against it — until a sibling document happens to quote the very source it denied. And a check built to compare documents against each other is worthless the moment it runs on a batch of one, because from the outside, "ran and found nothing wrong" and "never ran" report themselves the same way.
None of that called for a sixth gate. It called for three sentences added to a brief three workers were already reading, and a rule that a report of "gate PASS" has to name what actually ran.
How this would be shown wrong
If the next batch, briefed with these three new items, still produces a mechanism-drift or an unfalsifiable-negative-claim defect, the brief wasn't the fix, and something more structural is needed — a per-vendor trigger-term table the export step enforces, say, rather than a paragraph a worker is trusted to have absorbed. And if someone builds a per-document gate that catches the negative-claim class at a reasonable cost, without fetching and indexing every cited source in full, then "this isn't a gate problem" was wrong for that class specifically, and it should become one.
What it cost
One reply per worker fixed everything in the next round. The real registration then passed all five gates three times, picked up the human signature, and the batch went out clean. Start to fixed, the whole detour cost one extra round-trip per page. That's cheap, as long as somebody actually reads the rendered result before the first clean run gets treated as done.
Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.
More notes at hexisteme.github.io/notes.
Top comments (0)