I built a scanner that audits your test suite instead of your code

A passing test suite is supposed to mean something. Here are three ways it lies. All three stay green forever.

A test that asserts nothing. It calls the function, nothing throws, the runner prints a tick. It counts toward coverage and guards nothing. fastify at commit 6e95cb9 has seven of them.

An assertion that cannot fail. A literal answered inside a catch, a swallowed assertion, a status list that accepts both the success and the failure. The test reads as rigorous and holds whatever the code does. fastify has two.

A gate nobody invokes. A test-cov or verify script sits in package.json, reads as coverage to every reviewer, then no workflow ever calls it. express declares three.

None of that is visible to a code reviewer, to a coverage percentage or to an AI reading the diff, because the absence of a check does not appear in a diff.

So the tool reads the checks rather than the code

npx margyn-scan /path/to/repo

No account, no config file, zero dependencies, nothing uploaded. It answers in about a second. Every finding prints one command that reproduces it on your machine.

Six checks: no-assertion, cannot-fail, ignored-source (a file the build reads that git never committed, so a clean clone cannot build), unrun-check, lint-blindspot (linter exclusions that come from the ignore file rather than the linter's own config) and a mutation proof.

Measured at pinned commits, nothing tuned for the demo and nothing left out because the number was inconvenient:

Repository	Commit	Findings
chalk/chalk	661317e	0
sindresorhus/execa	8017b27	0
sindresorhus/got	e3924aa	1 unrun gate
expressjs/express	a371447	3 unrun gates
fastify/fastify	6e95cb9	13 (7 assert nothing, 2 cannot fail, 4 unrun)

Proof mode, because a scanner should not be believed either

npx margyn-scan . --prove

It runs each finding's own command, reads the output for the markers that finding predicted, then certifies it REPRODUCED. Anything it cannot reproduce is RETRACTED and dropped, so a CI gate never fails a build on a claim the tool could not show on your own tree. On fastify today: 13 findings, 13 reproduced.

That mode exists because the tool was badly wrong once. An earlier version of ignored-source produced 22 false positives across four real repositories. Proof mode certified every one of them, because the check and its proof had inherited the same wrong premise. I wrote that up separately, fixture included, since it is the more interesting half of building this.

It runs in CI and posts on the pull request

- uses: zkasuran/margyn@v0
  with:
    path: .
    comment: true
    sarif: true

It comments on the request, edits that same comment in place on every push, writes the report to the job summary and uploads the findings to code scanning, all with your own GITHUB_TOKEN and no service of mine in the path.

There is a live run to click rather than a screenshot to trust: pull request 1 plants two faults on purpose, a test that calls the scanner and asserts nothing plus a script no workflow invokes. The suite still reads 109 tests passing. The comment lists both findings with a reproduction each, both appear in the Security tab, then the job exits 1, which is the gate working.

What it does to itself

The mutation proof inverts a line, runs your suite, then reports the suite that stayed green anyway. Over this repository: 40 files tracked as source, 27 carry a mutation it knows how to make, 27 mutated, 27 caught, no survivors, across 108 tests.

Two honest numbers to go with that. It takes 208 seconds, because every mutation runs the whole suite once, which is why the default cap is four rather than every file. And zero survivors is a claim about this suite against seven operators, not a claim that the code is correct.

The first time it ran on itself it found eight survivors, one of them inside the mutation checker. The one that mattered most was in the worker: inverted, the entitlement branch handed a licence to every customer who had not paid. Nothing was watching that line.

What it will not do

It does not test your product, it audits the suite you already have. It does not upload your code, because there is no endpoint that accepts it. It will not tell you your code is correct. It tells you which of your existing checks cannot fail, which is a smaller claim and a checkable one.

The five static checks are free forever. The mutation proof is the paid one, because it runs your suite N times and that is the part with a real cost.

Try it on something you own

npx margyn-scan .

MIT, no telemetry, no account: margyn.xyz and github.com/zkasuran/margyn

The question I would genuinely like answered, because the default is still open: when a tool finds a hollow test in your suite, should it fail the build or only comment on the pull request? Margyn defaults to failing. I am not certain that is right.

Built during the tiun x Microlaunch challenge on hackwithus.dev.

AI assistance (Claude, Anthropic) was used while building this and while writing this post. The design, the review and every number above are mine. Each number ships with the command that re-derives it.