My Harness Reported 0 Findings on 100 Files. All 100 Were Ignored.

I ran fourteen lint rules over 100 files and got back zero findings. Every
rule silent, clean sweep, nothing to fix.

The correct answer was 308 findings in 54 of those files. The harness had read
none of them, exited 0, and printed nothing to stderr.

Here are the three ways that happens, all of them measured this week, all of
them silent.

1. The files were outside the base path

ESLint resolves what it is allowed to lint relative to cwd. Point it at a
directory that is not under the cwd of the process, and every file comes back
ignored:

File ignored because outside of base path.

That string arrives as a message with a null ruleId — the same shape as a
parse error. If your loop counts messages.filter(m => m.ruleId), or skips
files whose messages look unfamiliar, you have just discarded the only
explanation you were given.

On the CLI it is quieter still. Exit code 0. Empty stderr.

Fix: set cwd to the tree you are measuring.

2. The .ts files were never eligible

Flat config (ESLint 9.39.2 here) matches files by pattern. Omit files, and
the default covers .js and friends — not .ts, in an ecosystem that is
mostly TypeScript.

// eslint.config.mjs
export default [{ rules: { "no-eval": "error" } }];

src/a.ts   ← contains eval()
src/b.js   ← clean

$ eslint .
$ echo $?
0

No output. Exit 0. The .ts file contains eval() and was never opened.

This one is worse than the first, because a partial corpus still produces
plausible numbers. A repository that is 80% TypeScript reports on the other
20% and calls it a result.

Fix: files: ['**/*.ts', '**/*.tsx'], explicitly, and check the count of files
actually linted against a count you obtained some other way.

3. The build under test was not the build you ship

In a monorepo, node_modules/<your-package> is usually a symlink into the
workspace. Anything you measure through it is unreleased source.

I asked a plugin which of its rules carry a fixer. From the workspace:

prefer-event-target: code

From the published package, same command, clean directory:

prefer-event-target: no fixer

Both true. Different questions. I nearly published a correction to a correct
article on the strength of the first one.

Fix: measure in a scratch directory with an npm i <pkg>@<exact-version>, and
say which tree the number came from.

Why all three survive review

Nothing throws. Nothing warns. The exit code is 0 in every case — and the one
time I saw a 1, it came from a real finding in a file that was linted, not
from the file that was skipped. "It exited non-zero, so it noticed" is the
wrong conclusion and I drew it once.

They also fail at three different layers — which files are eligible, which
eligible files are covered, and which build is under test — so fixing one
leaves the others exactly as they were.

And they all fail toward good news. There is no version of these bugs that
invents findings. That asymmetry is what makes them survive: a number that is
too high gets investigated, and a number that is too low gets celebrated.

The check

One rule, before you believe any of it: write the code the tool must flag,
and make sure it flags it.

export function C({ items }: { items: string[] }) {
  return <ul style={{ color: "red" }}>
    {items.map((i) => <li key={i} onClick={() => log(i)}>{i}</li>)}
  </ul>;
}

Four lines, tripping at least two of the rules under test. If the canary is
silent, the corpus number is not a number.

That is what caught mine. The hundred-file zero was completely believable —
it was the four-line file reporting zero that was not.

A count from an instrument you have never seen fail is not a measurement, it is
a hope.

This is the cheap half of what ground truth catches that unit tests
miss:
the tests all passed, because the harness they exercised was working — it was
the corpus it pointed at that was empty. It is also why a result you cannot
re-run is not yet a result, which is the whole distinction between
reproducibility and replicability.
And a zero that means "read nothing" corrupts the one cell every other metric
is derived from — the true-negative box of the confusion
matrix.

Part of the Interlace ESLint ecosystem. Source on GitHub · Follow: Dev.to/ofri-peretz

What's the last clean report you got — and would you know if the tool had read nothing?

Comments

[GnomeMan4201]

The distinction between finding nothing and examining nothing is especially significant in security research, where a clean result can become someone's evidence that a system is safe.

Questions like this have started consuming almost everything I work on. How do we know a tool did what it claims, instead of just trusting its output?

My working answer is that we may not be able to prove it completely, but we can leave a record others can check: what was in scope, what was skipped and why, and which version ran. Your canary shows the instrument can fail, but that record would show what it actually covered.

I've been testing that idea on research review with a draft protocol and offline verifier, and I still don't know if that level of verification is fully achievable but i don't know a lot of things. do you think its achievable?