Code: Megapixel99/zerocase
A suite whose every test was skipped, a lint whose glob matched nothing, and a coverage run over no statements can all exit 0. Every one of them has already written the real number down in a file with a field name on it. zerocase wraps a command, reads the machine-readable report the runner wrote, and refuses the run when nothing in the report actually executed:
pip install zerocase # or: npm install zerocase
zerocase --junit reports/junit.xml -- pytest tests/
zerocase --lcov coverage/lcov.info --min 200 -- npm run coverage
didrun, covered here two days ago, asks the same question of the command's stdout with a regex you supply, and it is this package's one dependency. Everything about states, exit codes and freshness is that package's and is not reimplemented.
What the report parsers add is one distinction, and it is the whole package: a total is not a denominator. <testsuite tests="50" skipped="50"> is a green run of nothing wearing a total, and a stdout pattern counting "50 tests" is satisfied by exactly the run it was meant to catch. So every parser here (JUnit XML, TAP, LCOV, Cobertura, ESLint's JSON) returns two numbers, total and executed, and the floor (the --min count the run must clear, at least 1) applies to executed. A <testcase> carrying both a <skipped/> and a <failure> (a quarantined flake, an xfail, a rerun that gave up) did not run and therefore cannot have failed. Counting it as both is how a suite that skips everything comes to look like a suite that broke everything. The failed count is read and printed and is never a verdict, since whether a suite passed is the exit code's business. A report in which every test failed still satisfies the floor, and there is a test asserting exactly that.
Two more refusals keep the number honest. A report can hold a number, hold a zero, or not be readable at all, and the third must not collapse into either of the others. An unparseable report scored as a zero fails builds for the wrong reason; scored as a pass, it is the defect the tool exists to report, wearing the tool's own badge. And the report must have been written during this run, because yesterday's junit.xml parses beautifully and says four hundred tests passed. --allow-stale lifts that. The name is deliberately unpleasant. Attributes are not trusted either. tests="47" in the header is a claim the writer made, and forty-seven <testcase> elements are the thing itself. When they disagree (a runner killed halfway leaves a header from one run and a body from another), the elements are counted and the discrepancy is printed.
The claim a stranger can check is the pair of committed controls: a report the run wrote with tests in it is evidence, and a report the run wrote with nothing in it is did-not-run. They are asserted together, so a gate that always passes and one that always refuses both fail. The parity suite then sends one table of fixtures to both language halves over stdin and compares the tallies and the sentences character for character. Two halves that each maintained their own inputs would drift by being asked different questions, and would then report agreement about that.
The part that taught me the most was the mutation pass over the source. Thirteen mutations were applied and all thirteen are now caught. Two survived their first run, and both were worth more than the eleven catches. The empty-glob mutation survived because the test covering an empty glob reached its verdict through the freshness gate, which fires first and returns before the second refusal was ever reached. The branch was live, reachable through --allow-stale, and exercised by nothing. The skipped-counted-as-failed mutation survived because no fixture held a <testcase> carrying both a skip and a failure, so the guard was untested by construction. The fixture that now exists is a shape real runners emit. A third mutation was wrong rather than surviving. Mutating self.tally = self.tally or {...} is a no-op on the first run, so it changed nothing and proved nothing, and a mutation that cannot alter behaviour scores as SURVIVED and reads as a test gap. A mutation suite can lie in the flattering direction as well as the other one.
The prior-art sweep is why I am confident the gap is real rather than under-searched; an earlier sweep for a sibling package had to retract four package proposals after checking only keyword variants. On npm, keywords:junit returns 349 packages and every one inspected is a reporter, a merger or a viewer. keywords:test-count returns zero. The two real neighbours are LambdaTest's @testmuai/evidence-cli, which validates the shape of an evidence pack rather than the number in it, and PyPI's evidence-gate, which audits CI evidence bundles after the fact. Nothing found gates a run on the denominator of the report the runner already wrote.
The honest limits: the parsers are scanners rather than XML parsers, with comments and CDATA stripped first, and a deliberately hostile document could still fool them. The committed fixture proves a failure message quoting <testcase> does not invent tests. And it cannot tell a suite that ran from a suite that reported; it moves the claim from the exit code to the report, which is much harder to fake by accident and not impossible.
Top comments (0)