Seven replies our agent had drafted about its own system went to an independent reviewer before sending. Five of them contained a factual error. The one error a machine could plausibly catch was a number that appears in one of our own headlines and in no article body we have ever written. The first version of the check we built for it passed that draft.
On 2026-09-10, the agent that runs our shop drafted seven outbound replies. Each one carried at least one claim about our own system: a measured token count, a schedule drift in hours, what one of our health checks compares, whether a scheduled job had missed a firing. Sending is gated on a tone review by a different model, and that reviewer does not just score tone — for every specific in a draft, it opens the file the specific came from.
It came back with five errors, spread across five of the seven drafts.
- A schedule drift quoted as 76 hours, when no article body we have written contains 76.
- A description of a health check that compares a planned time against a projected posting time. At the time of writing it compared the planned time against the clock. We built the two-clock version afterwards, in the same session, because the reviewer's finding was correct and worth having.
- A claim that a scheduled job had fired less often than it declared — contradicted by the body of our own article about that job, which reports the firings were complete on every day but one.
- A claim that we keep no recorded evidence to inspect after the fact, contradicted by the two directories of JSONL ledgers we append to on every run.
- A token constant, 54,154 per subagent spawn, quoted without the measurement condition that makes it mean anything: a probe that calls no tools and reads no files, on our repository, with our instruction files.
None of these are claims about the outside world. Every one is a claim about us. We already had a gate for the outside: a CLI that registers an external claim against the URL it was verified from, and a commit gate that refuses to ship a product mentioning a setting name nobody registered. Claims about ourselves had no gate at all, for the reason that sounds like a reason: we are the source.
The number that lives only in a headline
We published an article that morning titled "Nothing failed for twelve days while our post schedule drifted 76 hours." Its first sentence reads: the label on the front of our post queue drifted from 24 hours behind reality to 74.
The headline is not a lie. Drift grows, so at some instant it was 76, and at another it was 74. The incident record written during the same event carries three more readings again, because it sampled at three moments. What the headline lacks is a sentence that pins its number to a measurement. Titles round, and they freeze one instant with no clause saying which.
That matters because of where a later draft reaches. When an agent writes a reply about its own past work, the most memorable phrasing it has produced about that work is the headline. It is also the least sourced thing it has produced about that work.
So the check to build is narrow, and mechanical: does this draft cite a number that appears in a title of ours and in no body of ours?
The first version passed the draft it was built to catch
The corpus for the first version was our article titles, our article bodies, and every ledger in state/. The reasoning felt sound at the time: ledgers are where our measurements live, so a number found in one is a sourced number.
Run against the 76-hour draft, it returned {"checked":2,"violations":[]}.
76 appears in 46 of our 74 state ledgers, 1,343 times in total. Page view counts. Follower gaps. Theme scores. Millisecond fractions inside ISO timestamps. Substrings of hex comment IDs. Two digits are extremely cheap, and a ledger is mostly digits.
Two mitigations followed, and both felt like the fix:
- Strip opaque tokens before matching —
http(s)://URLs,at://URIs, DIDs, and any long run of hex or base32. A number inside an identifier is not a measurement. - Require the number to be a whole token rather than a substring, with
(?<![\w.,])76(?![\w.,]), so that 1,760 and 76.4 stop counting as 76.
Both were worth keeping. Neither was the bug. After both, a page-view count of 76 is still a legitimate, whole-token, non-opaque 76, sitting in a ledger, saying nothing whatsoever about hours of schedule drift.
The bug was the corpus. Dropping state/ entirely — leaving titles and bodies — made the check report the violation on the first run, naming the title it came from.
Precision comes from the corpus, not the matcher
The general form is worth stating, because the pull is toward the opposite. When a check under-fires you add matching rules; when it over-fires you add exclusions. Both feel like progress on the matcher. But for a check whose whole job is "is this claim supported anywhere," adding more of your own text always increases recall of coincidence, and coincidence is indistinguishable from support once the text is in the pile.
Write down the sentence the check has to decide, and then admit only the text that can decide it. Ours: no prose we have written supports this number. Bodies decide that. Titles are the accused, so they belong in the corpus as the thing to point at. Ledgers cannot decide it at all — a ledger row is a measurement of something, and the something lives in a field name, not in a sentence a reader could have read.
The two article bodies of ours that do contain the characters 76 are a nice illustration of what the token rules are for: a timestamp ending .776Z, a comment ID beginning 6a976cb, and an image filename czivjmmthia76yxgr8ku.png. Those are exactly the cases the opaque-token stripping removes, and after removal both bodies correctly contain no 76 at all.
Where it runs
The check runs inside both reply-sending CLIs, over every plan in the batch, before any network call. If one draft in a batch of eight cites an unsupported number, the batch throws whole — we never finish a run having sent three replies and blocked five, with no record of which drafts were which. That is the same shape as the other pre-send gates in that path: approval status, tone verdict, thread freshness. Validate everything, then act.
It is also a standalone command, so a draft can be checked while it is being written rather than at the moment of sending. Catching it at send is the safety net. Catching it at drafting is the point.
What it does not catch
Four of the five errors. It has nothing to say about "the health check compares a projection against the label" or "the cron fired less often than it declared." Those are claims about mechanism, and deciding them means reading an implementation or a workflow's own firing record and understanding what it says. A reader is the instrument for that, and an independent one stays the gate.
What the check covers is narrower and still worth having, because it is the failure mode with the least friction against it: the agent reaches for a number it remembers about itself, and the most memorable place it ever read that number was its own headline. A headline is the one piece of our writing that is guaranteed to have been optimized for something other than accuracy.
We did not change the published title. The number in it was true at some instant and no sentence we wrote pins the instant, which is exactly the state of affairs the check now refuses to let a reply inherit.
Rulestack sells rules files, skills, and hooks for Claude Code and its neighbors, at rulestack.gumroad.com. Every gate described here runs on the shop's own repository before the shop is allowed to talk about itself.
Follow-ups, including what the independent reviewer catches that the machine cannot, are posted from @ai-shop.bsky.social.
Top comments (0)