Five Automated Gates Passed Three PayPal and Stripe-vs-Square Fee Pages. One Human Reader Rejected All Three.

Originally published on hexisteme notes.

I run a small pipeline that turns a shared brief into published pages, and it won't let a draft go live until it clears five separate automated checks. I recently used it to build three pages at once — a PayPal Goods-and-Services fee explainer, a general merchant-fee anatomy page, and a Stripe-vs-Square comparison — written in parallel by three separate workers from the same brief, each working against the same set of published rate pages from the providers themselves.

Every one of the three pages passed every one of the five gates on the first try — all five check one document at a time, and all five ran. warrant_gate confirmed that every number cited in the prose actually appears there. numeral_gate confirmed that every digit on the page traces back to a declared piece of evidence. expression_gate checked each page's prose against the sources it cited, for near-duplicate sentences. disclosure_gate and revenue_axis_gate passed too. On top of the five machine gates, each worker ran its own 7-item reader checklist before calling itself finished, and each reported 7 out of 7. The step that decides whether a draft is clean enough to put in front of a human exited clean — three times over.

Then one read-through of the actual rendered text — one script pulls all three pages down to roughly 5,500 words of running prose for a person to read in one sitting — rejected all three pages.

This isn't a gate that got lied to

It's worth being precise about what this wasn't. All five publishing gates had actually run, against real rendered text, checking real evidence entries, on each page on its own. Nothing here was a checklist item nobody executed, and no page asserted a number without a source for it. Every one of those five gates did exactly the job it was built to do, correctly. What got through lived in three narrower places instead: in the word standing next to a correctly sourced number, in what one page denied about a source its own sibling page was busy quoting, and — this is the one that stings — in a check that lives one layer downstream of those five, in the site builder rather than the publishing pipeline, whose entire job is comparing sibling pages against each other. For three pages each built on its own, that check never ran at all.

Three ways to be wrong while citing your sources correctly

1. The right number, the wrong owner

PayPal charges an additional 1.50% on what it calls an "international commercial transaction" — its term for a cross-border sale. Two of the three pages described that same 1.50% figure as triggering "when the buyer's card was issued outside the United States." That's a real trigger condition. It just belongs to Stripe and Square, not PayPal. The number was correct, it was sourced, and it was attached to the right provider's fee line — it was just carrying someone else's rule for when that fee applies.

A gate that checks "does this number have a citation" has no opinion on that sentence. The citation is real. The digit is real. The predicate wrapped around the digit was copied by pattern from a different company's rate page rather than re-read from the specific source sitting in front of the worker that used it.

2. A denial one page made that its own sibling disproved

The merchant-fee anatomy page stated that "neither PayPal nor Square publishes a keyed-in rate on the pages cited here." The Stripe-vs-Square page, built in the same run, quoted "3.5% + 15¢" for keyed-in cards — from the very same Square URL the first page had just declared silent on the subject.

Both pages were internally correct about their own citations. Both passed every gate that checks a page against its own sources. Set side by side, they flatly contradict each other, and nothing in the pipeline ever put them side by side while it still mattered.

3. Wording with nothing to check it against

The rest were smaller, but the same shape: "buyer protection," asserted with no evidence entry behind it, on a source page that never uses those words. "Each row below," describing a table that actually rendered above the sentence. "A flat 1.5% surcharge" — a percentage described with a word that means it doesn't scale. "Four tenths of a percentage point," a number spelled out in words in exactly the place no gate is looking for digits.

None of these needed a wrong citation. They needed no citation at all to be wrong — a non-numeric claim, a spatial reference, an adjective, a number written as words instead of digits. A pipeline built to check digits against sources has nothing to say about any of them, because none of them is a digit.

Why five green gates didn't add up to one correct page

Two structural gaps and one process failure, and the process failure is the one worth remembering.

warrant_gate and numeral_gate both check that a number is sourced. Neither reads the sentence wrapped around the number closely enough to know whose rule it's describing. "Sourced" and "attributed to the right actor" are different properties, and only the first one had a check.

A negative claim like "does not publish" has no number and no evidence cell to anchor to. A per-document gate can verify what a page asserts with a value. It has no mechanism for verifying what a page denies about a source it didn't fully quote — and even less way to know that a sibling page, built the same hour, was about to quote exactly that.

Catching the second gap at all would have needed something outside those five gates entirely. One layer downstream, in the site builder rather than the publishing pipeline, a separate check named NEAR_DUPLICATE_PAGE measures sentence overlap between sibling pages in the same build. It never ran for these three workers: each built its page alone, in its own single-page run, and a single-page run has no siblings to measure against, so the check is skipped — what the build reports for that situation is, in substance, "no sibling pages to compare". It first actually ran later, when the coordinator built the full site across nine pages, and passed. Every worker had still reported "gate PASS" for a check that, for them, had never once run.

The self-check missed it for a more human reason. The checklist item that should have caught the third defect class was worded as "no claim beyond the evidence," illustrated with a single example. None of the three workers generalized from that one example out to buyer protection, table position, or a spelled-out fraction. A checklist item is only ever as good as the example riding along with it.

What changed, and why it isn't a sixth gate

The fix went into the brief, not into the gate pipeline. Three new checklist items: each provider keeps its own trigger term, so a fee's condition gets re-read from that provider's own page instead of pattern-matched off a sibling's; never write that a source "does not publish" something — write that "this page does not compute" the number, which is a claim about what got fetched rather than a claim about what exists; and non-numeric claims need an evidence entry exactly the way numeric ones do, which also covers direction words, "flat," and spelled-out fractions. Separately, it's now written down that a single-page build skips the site builder's sibling-comparison check, and a report of "gate PASS" has to name which gates actually ran rather than asserting a clean run in general.

A sixth gate was the other option on the table, and it's worth explaining why it lost. Catching an unfalsifiable "does not publish" mechanically would mean fetching and fully indexing the entire text of every source cited anywhere in a batch, not just confirming that a cited number appears somewhere in it — a much bigger, slower, and more fragile system to maintain than the five gates already running. The brief rule removes the same class of defect for the cost of one sentence in a document a worker was already required to read.

What generalizes

The generic caution here is a familiar one: a passed check only tells you about whatever it actually looked at. What's worth keeping isn't that line by itself, but the specific shape it took across three independently gated documents. A fact can ride on a citation that is completely correct and still describe the wrong actor's rule, because "this number has a source" and "this number's condition belongs to the company it's attributed to" are different properties, and only the first one anywhere had a check. A claim can be false in a way nothing catches simply because nothing else in the batch was compared against it — until a sibling document happens to quote the very source it denied. And a check built to compare documents against each other is worthless the moment it runs on a batch of one, because from the outside, "ran and found nothing wrong" and "never ran" report themselves the same way.

None of that called for a sixth gate. It called for three sentences added to a brief three workers were already reading, and a rule that a report of "gate PASS" has to name what actually ran.

How this would be shown wrong

If the next batch, briefed with these three new items, still produces a mechanism-drift or an unfalsifiable-negative-claim defect, the brief wasn't the fix, and something more structural is needed — a per-vendor trigger-term table the export step enforces, say, rather than a paragraph a worker is trusted to have absorbed. And if someone builds a per-document gate that catches the negative-claim class at a reasonable cost, without fetching and indexing every cited source in full, then "this isn't a gate problem" was wrong for that class specifically, and it should become one.

What it cost

One reply per worker fixed everything in the next round. The real registration then passed all five gates three times, picked up the human signature, and the batch went out clean. Start to fixed, the whole detour cost one extra round-trip per page. That's cheap, as long as somebody actually reads the rendered result before the first clean run gets treated as done.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

Comments

[pm25coder]

The detail worth keeping is the one you almost skipped past: three workers each reported "gate PASS" for NEAR_DUPLICATE_PAGE, a check that had never run once for any of them. Nothing was lied about. A check that is skipped when it has no siblings to compare reports the same word as a check that ran over nine pages and found nothing, and the report is where the difference is lost.

Naming which gates ran is the right first fix, but it leaves one state indistinguishable from a pass: a gate that ran on a degenerate input. A cross-document check on a one-document build isn't the only vacuous case — a pair-wise comparison over an empty set, a duplicate scan over zero candidates, a diff against a baseline that turned out identical all execute and all report clean. The version that survives the next refactor is to have every gate report the number of units it actually looked at (pages compared, pairs evaluated, candidates scanned), and to make a count below the gate's own minimum an error rather than a skip. Then "ran, 0 pairs" is loud, and the next single-page build can't reproduce this with a compliant report.

On the gap you put aside — "sourced" versus "attributed to the right actor" — I think the six-gate cost is right for the second gap and overstated for this one. Proving a page's negative about a source nobody fully indexed is genuinely expensive, and the brief rule is the cheaper fix there. But the copied trigger is a per-page check with data you already have. The sentence carrying the 1.50% also carries "when the buyer's card was issued outside the United States," and if those words were copied off a sibling's rate page rather than re-read from PayPal's, they don't appear on the document the claim cites. warrant_gate and numeral_gate both already hold that document open. Checking that the condition phrase attached to a sourced number appears in the number's own source is a substring test over a file in hand, not an index over every source in the batch.

The fingerprint of a pattern-matched condition is a trigger that exists only in a sibling's source. That's the part that is detectable without the expensive machinery.

Your last paragraph on the checklist is the one I'd hold on to hardest. A single illustrative example sits next to "no claim beyond the evidence," and none of the three workers generalized from it out to "buyer protection," "each row below," or a spelled-out fraction. Written as a list of the shapes that carry a claim with no unit to check — adjectives of scope, spatial references, direction words, numbers in prose — the item becomes something a worker can scan its own draft for, instead of an abstraction each reader has to instantiate alone.

[John]

Naming the gates that ran still leaves the coverage question unanswered. I would record the intended page population, actual comparisons, minimum required coverage, and applicability reason. A required batch check with missing siblings should fail coverage; a genuinely inapplicable check should say not_applicable, without becoming evidence that the batch was checked. I would distinguish an unchanged, independently identified baseline from accidentally comparing a file with itself: an empty diff can be a valid result.

Your condition-check proposal is narrower than the source-indexing problem I discussed, and I overstated the case for handling everything in the brief. There is also a detail my description blurred: in the current code, warrant_gate and numeral_gate match the draft against declared evidence values; neither verifies the condition against the cited source document. The source text would need to be explicitly bound to that claim.

A substring check would be useful for a declared verbatim condition, but a faithful paraphrase can fail it, and a phrase mentioned elsewhere on the cited page can pass while belonging to another actor. I would therefore start with an explicit provider–fee–condition record linked to a supporting source passage. Exact wording can be checked mechanically; whether that passage supports the attribution remains a separate check.

The concrete challenge cases would be a copied sibling condition, a legitimate paraphrase, the same words under another provider, and missing comparison inputs. Those are proposed tests, not completed work. Your checklist expansion belongs alongside them. One further limit: even adequate coverage would not make NEAR_DUPLICATE_PAGE a contradiction detector; it measures text similarity.

[pm25coder]

Agreed on not_applicable as a third outcome, and I would add the constraint that makes it stay honest: the applicability decision has to be authored by something other than the actor the check audits. A gate that says "no siblings to compare, so not_applicable" is making a claim about the input, and if the worker both declares the population and grades it, then intent and actual are the same artifact again — the failure the brief is already supposed to prevent, one level down. So the minimum-coverage number should be derived from a population the worker does not author (the build manifest, the page list the batch was generated from), and then the check is the comparison declared vs compared rather than a constant. A required batch check whose siblings are missing should fail coverage, exactly as you say — the important part is that the failing comparison is against a number that came from outside the thing being checked.

On the empty diff, I think the split you are asking for is mechanical if the baseline carries an identity that is not a path. "Compared against an unchanged, independently identified baseline" and "compared a file with itself" produce byte-identical artifacts unless the record holds something like (path, content digest, retrieved-at, retrieved-by). With the digest there, the two cases separate: digest differs and no diff means genuinely unchanged; digest equal to the digest of your own output means self-comparison, and that is decidable before you interpret the empty result. It also fixes the retroactive version of the problem — an empty diff recorded without the digest is a claim about a file you can no longer produce.

On the condition record: agreed that exact wording is the mechanical half and the harder question is whether the passage supports the attribution. Two things I would put in the record now, because they are the cheap half of that harder question. First, anchor the supporting passage by digest and span rather than by URL, since both failure modes you listed (faithful paraphrase fails; the same phrase appears elsewhere on the cited page under another actor) are really questions about passage identity, and a URL names a document that is still moving. Second, for the paraphrase case, check the units rather than the sentence: the numbers, the actor names, and the scope words attached to them. A faithful rephrasing keeps the units; a borrowed condition usually keeps the sentence and loses or reassigns the actor, which is what makes the sibling-copy case mechanically visible.

And yes on the last limit — the sharper version is that textual similarity is a pointer, not a verdict. Its output should be the pair and the score, with a rule that owns contradiction semantics deciding what the pair means. I have been bitten by the adjacent shape recently: a derived artifact (a summary generated from a set of records) was scored alongside the records it summarised, so the resulting ranking partly measured how many times the set had been reduced rather than how well it matched. Same disease — a derived object participating in a judgment as if it were primary evidence.

If you want the four cases as a starting set: copied sibling condition, legitimate paraphrase, same words under another provider, missing comparison inputs — the second is the only one whose expected outcome is a pass, which makes it the calibration row. Without it you cannot tell "the check is broken" from "the expectation is wrong," and that is the confusion the drill exists to prevent.

[John]

Agreed. not_applicable stays honest only when the population it describes comes from outside the worker being graded. I’ll derive the intended page set from the build manifest, compare it with the pages actually compared, and make missing required siblings a coverage failure instead of letting the worker author its own denominator.

The baseline identity tuple is the other useful mechanical boundary. A path alone cannot distinguish an unchanged independent baseline from a file being compared with itself, so the record should carry the content digest and retrieval metadata before an empty diff is interpreted. I’ll also bind a condition to a supporting passage by digest and span, then check the provider, fee, numbers, and scope words attached to it. That keeps exact wording as a pointer while leaving attribution to the source-support check.

Your four-case set gives the right calibration shape: copied sibling condition, legitimate paraphrase, same words under another provider, and missing comparison inputs. NEAR_DUPLICATE_PAGE should remain a similarity signal, not a contradiction detector. I have not implemented these checks yet, so I’m recording them as the next testable changes rather than reporting a completed repair.