My scanner was wrong 22 times and its own proof certified every one

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

I built a scanner that audits the verification layer of a repository rather than the code. One of its checks looks for a file the build reads that git never committed: the file is on your disk untracked, so every local run is green while a clean clone or a CI runner cannot read it at all. The diff looks innocent because the defect is an absence.

Because a scanner that only asserts is not much better than the tests it audits, it also has a proof mode. Every finding carries a machine-checkable command. --prove runs that command, reads the output for the markers the finding predicted, then certifies the finding as REPRODUCED. Anything it cannot reproduce is retracted and dropped, so a CI gate never fails a build on a claim the tool could not show on your own tree.

Then I pointed it at five real repositories I had lying around. It produced 22 findings on four of them. Every single one was wrong. Here is the part that still bothers me: the proof certified all 22.

What the false positive looked like

Reduced to the shape it kept taking, on a repository anybody can build in ten seconds:

mkdir -p web/public/tour web/src && git init
echo webm-bytes > web/public/tour/clip.webm
echo 'export const clip = "/tour/clip.webm";' > web/src/app.js
printf 'node_modules\nweb/dist\n' > .gitignore
git add -A && git commit -m "app plus the clip it plays, dist ignored"
npm run build     # or just: mkdir -p web/dist/tour && cp web/public/tour/clip.webm web/dist/tour/

clip.webm is committed. It sits in web/public, which is exactly where a Vite or Next project keeps assets it wants copied verbatim. The build then writes a second copy into web/dist, which is gitignored because a clean clone rebuilds it.

My check walked the source, found "/tour/clip.webm", resolved it against the files on disk, hit web/dist/tour/clip.webm first, saw that git ignores it and reported HIGH: the build reads a file that is not in the commit.

1. web/dist/tour/clip.webm is read by web/src/app.js but git ignores it, so it is not in the commit
   HIGH  ignored-source  web/dist/tour/clip.webm
   ignore rule: .gitignore:2:web/dist
   why: A clean clone or a CI runner cannot read this file.

Read on its own, that finding is defensible. The file it names really is untracked. Every fact in it is true. It is still wrong, because the thing the reader asked for is in the commit. A clean clone answers the request perfectly.

On the real repositories it was the same fault at scale: a Vite app reporting its whole public directory, a Foundry project reporting contracts/out (declared out = "out" in its own foundry.toml), a CLI reporting dist/cli.mjs (declared outdir: "dist" in its own build script). 22 findings, 4 repositories, zero real defects.

The part that actually scared me

Here is the same fixture through proof mode, on the unfixed scanner:

1. web/dist/tour/clip.webm is read by web/src/app.js but git ignores it, so it is not in the commit
   HIGH  ignored-source  web/dist/tour/clip.webm  REPRODUCED
   MARGYN_ABSENT_FROM_HEAD
   MARGYN_PRESENT_ON_DISK

2 findings: 2 reproduced.

REPRODUCED. The proof ran, both markers matched and the tool certified its own mistake with a green badge.

The proof was this:

git archive HEAD | tar -t | grep -qx 'web/dist/tour/clip.webm' || echo 'ABSENT from HEAD: web/dist/tour/clip.webm'
test -f 'web/dist/tour/clip.webm' && echo 'PRESENT on disk: web/dist/tour/clip.webm'

Both lines are correct. web/dist/tour/clip.webm genuinely is absent from HEAD and genuinely is present on disk. The proof passes because it asks the same wrong question the check asked: is this path committed, when the question that decides the bug is does anything in the commit answer the path the reader asked for.

That is the lesson I paid for. A check and its proof can agree, both be internally valid and both be wrong, because they inherited the same premise. Independent verification means asking a different question, not running the assertion twice.

The two fixes

A reader asks for a path, not for a file on your disk. The check now collects every needle a reader names, then reports only when no committed file answers it. web/public/tour/clip.webm in the commit satisfies /tour/clip.webm, whatever the build left in dist. The proof asks that question too, so a finding that slips through retracts itself instead of failing somebody's build over a copy of a committed file.

Regenerated output is not missing source. The check skips output a tool in the repository declares it writes: Foundry's out from foundry.toml, a Vite or Next or Cargo target from the script that runs it, an outdir in a build script or a tsconfig. Read from declarations and resolved to real paths, never from the directory happening to be called dist.

That second rule matters more than it looks. One of the five repositories had vendored real source into vendor/dist, a path no tool there declares. That is exactly the defect this check exists for. A crude "ignore anything named dist" would have deleted the only true positive in the set.

So the fixture keeps both cases. Same tree, one import added, scanner at HEAD:

1. vendor/dist/real.mjs is read by web/src/vendorised.js but git ignores it, so it is not in the commit
   HIGH  ignored-source  vendor/dist/real.mjs
   reproduce:
     git archive HEAD | tar -t | grep -qE '(^|/)dist/real\.mjs$' || echo 'NOTHING in HEAD answers dist/real.mjs'

The build copy is gone. The vendored module is still reported. And look at the new reproduce line: it asks whether anything in HEAD answers the reference, which is the question the fix is built on.

Verification

• 4 regression tests in test/checks.test.mjs, one per direction: quiet when the commit answers the path, still loud when nothing does, quiet on declared build output, still loud on vendored source under a path that only looks built.
• The fixture above, run against the pre-fix commit and against HEAD: 2 findings before, 1 after, then the one that survives is the real one.
• Full suite 108 tests passing. The tool's own mutation proof mutates all 27 mutable files of 40 tracked and reports no survivors.

While I was in there, one more thing turned up that nothing was watching: the SARIF helpUri for the mutation rule pointed at /docs#mutation while the docs heading carried id="mutation-check", so every uploaded finding linked to nothing. There is a test now that resolves the helpUri of every check the scanner can emit.

What I would tell my past self

Write the check and the proof from different premises. Otherwise the proof is a spell-check on the check's own reasoning. Mine was, for 22 findings, with a green badge on each.

Code: the fix is one commit, efc5657, in github.com/zkasuran/margyn. The scanner runs with no account and no config: npx margyn-scan /path/to/repo. --prove is the mode that got humbled here.

AI assistance (Claude, Anthropic) was used while writing this post and while implementing the fix. The bug, the reduced fixture, the two rules and every number above were verified by me by running them: the outputs quoted are captured stdout from the pre-fix commit and from HEAD, not retyped.