I asked one question. All four of my linters failed it.

Three separate times, in one of my linters, the same bug came back wearing a different hat: a text rule was looking at mentions and treating them as execution.

That matters more than it sounds, because of who writes mentions. If you never think about scripts/deprecated.sh, you never type it. The person who writes Never run scripts/deprecated.sh — it was removed in the migration is the person being careful. My rules were reading that line as a step and reporting it. The tool punished exactly the people who did it right.

After the third occurrence I stopped fixing instances and asked whether the shape was in my other tools. It was. In all of them.

The question

I did not ask for a code review. Reviews come back with style notes. I asked a single adversarial question and gave it the source:

Construct an input that punishes the person who wrote it carefully.

Four tools, four hits.

Tool	Version	What the careful writer got
skills-lint	0.9.0	A prohibition silenced a real broken reference
tracklint	0.7.0	Comments counted as code — in both directions
tokenlint	0.5.0	React inline styles missed entirely, inflating the headline number
reflint	0.12.0	Commands and paths inside prohibition sentences counted as steps

The worst one was not a false positive

skills-lint checks whether a SKILL.md references files that do not exist. It has a sensible exemption: if the document says the skill produces a file, that file is not expected to be there yet. PRODUCES matched on stems like generat.

So this happened:

Never generate `scripts/missing.py`.
When asked to deploy, execute `scripts/missing.py`.

Line 1 registered scripts/missing.py as an artifact this skill produces. Once a name is registered, references to it are exempt anywhere in the document — including line 2, which is a genuinely broken reference to a file that does not exist.

The tool printed ✓ all clean.

This is the part I keep thinking about. A false positive is loud: someone opens an issue, you fix it. A warning that never fires is invisible. If a user had hit this, they would have concluded their skill was fine. There is no complaint to receive. The only reason I found it is that I went looking for the input that would cause it.

Broken in both directions at once

tracklint checks conversion tracking on web forms, and it was matching regexes against raw text. Comments are raw text.

The direction that punishes the honest:

<!-- This site intentionally does not call gtag() or load Google Analytics. -->

That one line convinced the tool the project has analytics, which switched on every wiring rule. An identical form went from clean to 1 error because someone documented that they don't track.

The direction that hides the real problem:

// gtag must never be called here; conversion tracking is not implemented yet.

Counted as a real call, so an AJAX form with genuinely no success-time tracking stopped reporting ajax-no-conversion.

The fix was a stripComments() that blanks comments while preserving length and newlines, so line numbers don't shift. I deliberately made it conservative — https:// and src="//cdn..." are not comments, and over-stripping would make the tool miss the analytics install and go quiet. Between a fix that over-reports and a fix that under-reports, I took over-reporting, for the same reason as above: noise is visible, silence isn't.

When a miss flatters your own headline number

tokenlint reports a design-token coverage percentage: tokenized / (tokenized + hardcoded).

Its markup scanner only understood quoted HTML attributes, style="...". It had never once matched:

<div style={{ color: "#ff0000" }}>

That is the ordinary way to hardcode a color in React. Files consisting entirely of hardcoded colors reported zero hardcoded values — and because the miss lands in the denominator, every miss pushes the coverage number up. My tool's own advertised metric was wrong in the flattering direction.

Worth separating two things: this was a miss, not a judgment call. A // line comment containing a Tailwind class is still not scanned, and that one stays — // cannot be told apart from https:// without hiding whole URL lines, which trades a false positive for a miss. That trade I'm making on purpose. The React one I just never checked.

One caveat about how this was found

The model I asked could not run node in its sandbox, so it validated by calling internal functions and reasoning about them. That is not evidence. Every claim it made, I re-ran through the real CLI myself before writing a fix. After publishing, I installed each package fresh from npm and re-ran the failing path against the released build.

An adversarial reviewer is very good at generating the input you would not have thought of. It is not a substitute for executing that input.

Two questions worth asking your own tool

1. "Construct an input that punishes the person who wrote it carefully." Better than "review this." It forces a concrete artifact you can run instead of a list of concerns.
2. "What does this change stop reporting?" Every one of these bugs was easy to see once framed as a miss, and invisible while framed as a fix. If your tool has a coverage percentage or a clean/dirty verdict, ask which direction a bug in it would move that number.

Sources, if you want the actual diffs. The changelogs are written in Japanese, but each entry contains the literal failing input, and that part reads fine in any language:

• skills-lint
• tracklint
• tokenlint
• reflint

All four were green on their own test suites the entire time. That's the other thing worth saying out loud: a test suite you wrote is a picture of what you already thought of.

Comments

[Alex Shev]

This is a good reminder that linters encode assumptions, not truth. When all four fail the same question, the useful artifact is the missing invariant: what intent did none of the tools know how to represent?

[hyuga]

The invariant none of them had is modality — the author's stance toward a name they typed.

All four could represent "scripts/deprecated.sh occurs at line 12." None could represent "line 12 forbids it," "line 12 is an example," or "line 12 says this file gets created later." Naming and doing were the same event, so a sentence that exists because the author was careful read identically to a step.

Being honest, though: that's three of the four. tokenlint's was a plain miss — style={{ }} was a syntax its scanner had simply never matched, no intent involved. It's in the same table because the miss flattered the coverage number, not because it shared the shape.

And the fixes don't encode the invariant either, they approximate it lexically. reflint now skips any line matching a prohibition pattern. So this, with docs/handbook.md genuinely absent:

Never run `npm run release`; use `docs/handbook.md` instead.

prints reflint: all references resolve and exits 0. That's the skills-lint bug from the post, scoped to a line instead of the whole document. Move the reference to the next line and it's caught, so the blast radius is bounded — but the shape survived its own fix.

Which is your point, I think: I moved the assumption rather than removing it. Representing modality properly means parsing the sentence, and none of these are parsers.