One of my checkers scans every draft I have written for wording that overstates what the code actually does. I ran it across everything, it came back clean, and I said so. At that moment three already-published posts were carrying the exact phrase it hunts for, in three languages, because all three were filed as unpublished.
That is the fourth of seven. I wrote four checkers over one day, each because the previous one had just let something through, and the interesting part turned out not to be what they found but what caught what.
Four of the seven were caught by a tool I wrote. One was caught by running the thing a tool had just approved. One was caught by someone else's review bot. The last was caught by nothing, and cost the most.
1. The docs were describing a command that had moved
The first checker was not about prose at all. It parses the CLI's typer registrations with ast and compares them against what the docs claim you can run.
It found a real defect on its first run. One line of one guide said a command was reachable only through the MCP server. It has been on the CLI for a while. Nobody noticed, because nobody had a reason to open a docs page and a registration list side by side, and the two files are 200 lines and four directories apart.
That one is now a test in CI, so the next drift fails a build instead of waiting for a reader.
2. Every merge silently invalidated my citations
The second checker is for a fact file. Every claim in it carries a path:line citation plus a probe token, and the script opens that exact line in a clone and asserts the token is on it.
It caught stale citations three times in one day. Not because anything was wrong, but because merges shift line numbers. A claim can be entirely true and still point at nothing.
I fixed the first two by hand, and on the second I guessed a line number instead of opening the file. I was off by one, so the checker failed again and I got to make the same mistake twice in five minutes. On the third I stopped fixing citations and taught the checker the difference:
- the probe token is still findable in that file, at a different line, which is a stale coordinate
- the probe token is gone entirely, which is a claim that may no longer hold
The first case is now a one-command rewrite. The second still fails loudly, because that is the case that actually needs a person.
Later the same day that one-command rewrite tried to corrupt the file it exists to protect.
A citation had drifted, and the fixer offered to move it from line 558 to line 255. Three hundred lines backwards is not how a merge shifts code, which is the only reason I looked. The probe token on that row was return None. It appears all over that file. The fixer takes the first match, and the first match was unrelated code three hundred lines earlier. The correct line was 559.
Had I run it, the fact file would now carry a citation pointing at code that has nothing to do with the claim, and every later check would pass, because the token really is on line 255.
The fixer now refuses when the probe appears more than once in the file and says so:
AMBIGUOUS src/.../adapter.py:558
probe `return None` appears 47 times in this file; first at :255
fix: choose a probe unique to the line, then re-run.
A tool that repairs can do damage a tool that only reports cannot. The reporting version of this checker had been correct all week. The moment I taught it to write, it acquired the ability to write something false, and the thing it would have written was exactly the class of error the file was built to catch.
3. The same false claim was living in four languages
The third checker is the one that stings. It flags absolute wording about mechanisms that are real but incomplete.
What we withhold from a worker is narrower than I keep writing it, and narrower again since I wrote that sentence. When a criterion defines a verify command or an expected-output assertion, those values are omitted from the worker's contract block, so it is asked for the outcome rather than handed the assertion. Criteria are allowed to define neither, in which case there is nothing to omit. Redaction of those values anywhere else is literal over five encodings, which means an assertion that arrives line-broken through the retry path survives and reaches the worker anyway. The project's own lint and test commands are handed to it deliberately, which is a different thing and one I have described sloppily more than once. And the withholding is verbatim matching over five encodings, so an assertion that comes back reshaped, line-wrapped, diff-prefixed, or with a colour code inside it, still gets through. That is open as an issue, not a footnote.
But I had written it as "hidden unconditionally". In English. And in Chinese. And in Korean. I fixed it three times that day, in three different files, each time believing it was the last one.
Then I opened the Japanese draft and it was there too.
It survived a fourth time for a boring reason: my checker had patterns for three languages. A claim living in four languages needs rules in four languages, and mine had a hole exactly the shape of the file I had not opened yet.
4. Three published posts said things I had already corrected
This is the part I would have left out if the checkers had let me.
The checker scans drafts. It reported no violations in publishable text, and I reported that in turn. Both statements were true and both were useless, because three of those drafts were marked unpublished while the content was already live. My corrections went into files nobody was reading.
The most instructive one was a draft whose status read "blocked: no route to post this". That was accurate and honest when written. The route opened later the same day, the thread went out, and nobody walked back to the draft. An accurate failure record turned into a false status by sitting still.
So the fourth checker exists. If a draft claims it has not shipped while the log contains a live URL for that platform, it fails. It cannot tell which post came from which draft, so it prints "go look" rather than a verdict. That is the honest limit of it, and every one of that day's failures happened because nobody went and looked.
5. The file passed its schema and the command inside it was dead
This one happened after I thought I was done, and it is the cleanest version of the whole problem.
I was preparing a server.json to list our MCP server in the official registry. I fetched the published schema, ran the document through a validator, got zero errors, and wrote in my notes that it was ready to publish.
Then I ran the command the file describes, and it died.
Three things were wrong in a document a validator had just called correct. The command name was the package name rather than the console script. The runtime hint pointed at uvx, which resolves an environment the package's own extras do not pin, so the server cannot start there at all. And a required flag was missing, without which the process picks a default it cannot host.
A schema checks shape. It has no opinion about whether the thing you described exists.
Chasing the third one led somewhere I did not expect. The error I hit blamed package extras and told me to reinstall, but a clean install reproduces it, and the guard it names evaluates to false in that environment. Two different failures were sharing one message, so a user with a perfectly good install was being sent to change dependencies that were never wrong. I had already filed an issue asserting the wrong cause; I had to correct it in public and retitle it. The fix, once the cause was actually known, was to split the string in two.
I would have published a registry entry pointing at a command that cannot run, and I would have done it on the strength of a green validator.
What I would take from this if it were not my repo
Give a claim a probe, not a review. Re-reading finds a wrong sentence once. A probe finds it every time, including the times you are certain you already fixed it.
Separate a stale coordinate from a broken claim. They look identical in a diff and they need opposite responses. Conflating them is how I published a correction that was itself wrong, then had to correct the correction.
Count coverage in the dimension the content actually varies in. Mine varied by language. Yours might vary by version, by tenant, by locale. A checker is only as wide as the axis you thought of.
A status field that nobody is forced to update is a lie with a delay on it. This was the expensive one. Not the wrong sentence, but the correct record that quietly stopped being correct.
Read your own changelog before you report a bug against yourself. Every tool in this post checks whether a claim matches the source. Not one of them checks whether I have read what my project already published, and that is what cost me the most time.
Be slower to automate the fix than the check. A checker that only reports is wrong in one direction: it misses things. A checker that repairs is wrong in two, and the second one is silent.
A passing check is evidence about the check, not about the thing. Schema valid, tests green, linter clean: each of those tells you one narrow thing. The only way I found the dead command was running it. Every layer of validation I added today failed to catch something the next layer down caught by executing.
Write down how you counted, next to the count. This is the one I kept repeating. Three times in a day I published a number without saying what I had counted, and all three were wrong in the same direction, upward.
The backlog described a checker as "13 rules across 5 languages"; recounting gave 15 across 4. A note said 44 entries in a directory used a term, but that counted anything containing two words in any position; the term itself appeared 19 times. And I wrote that three competing projects package the same way as us, having checked one of them.
None of those were guesses. Each came from a real command I had run, and each was reported without the command. The number survives into the next document; the method does not, so nobody downstream can tell a measurement from an impression.
The fix is not more care. It is putting the method in the same sentence as the number, and having tools count themselves where possible. The scripts now print their own totals on every run, because a number that cannot go stale beats a number you remember to verify.
There is a sixth, and it happened while this post was queued to publish.
A review bot blocked an unrelated pull request of mine because the package summary claimed we keep the grading command out of the agent prompt. I checked: execution.context_pack defaults to on, and it appends the project's detected verify commands to the worker's system prompt. The claim was false at the package level. It was true only of the per-criterion contract, which is a much smaller thing.
That sentence had by then reached two manifests, a package summary, a registry draft, a pitch email, and a submission to a magazine with a hundred and seventy thousand stars. Section 3 of this post, the one about over-claiming, was itself over-claiming when I wrote it.
I corrected all of them and added a rule to the wording checker. Ninth instance of the same family, first one caught by someone other than me.
One last thing, and it is the one that should have been obvious.
Late in the day I filed a bug report against my own project: a dependency resolving to the wrong major version, breaking the documented install path. I reproduced it twice, read the resolver, checked a clean environment, and wrote it up carefully.
It had been fixed that morning. The fix was one flag, described in the first line of the release notes for the version I was running, which I had not read.
We publish long release notes, and I mean long: fifty-seven thousand characters in the last month. Fourteen people watch the repository. The answer to the problem I spent an hour on was sitting in a document my own project writes and almost nobody reads, including me.
None of the checkers would have caught that. There is no probe for "you did not read the thing you wrote."
The checkers are three short shell scripts and one ast walk, sitting in a marketing folder next to the drafts they police. They took less time to write than this post did. The only decision that mattered was pointing them at my own work before anyone else's, and the last thing on this list is the one none of them would have caught.
So the question I would actually like answered: what is the last thing in your project that passed a check and was still wrong?
I don't mean a bug you shipped, but something that had a green tick on it at the moment it was broken, and what the tick was measuring instead.
Top comments (0)