My mutation score was 1.000 in two places and the denominators disagreed

I published a number, went to confirm it, and the confirmation disagreed with it. The number was fine. The confirmation was broken, and it was broken in a way I had already written a warning about.

The number

My evidence-bundle verifier has one metric I care about more than coverage: for every place in the code where the tool can refuse, is there a test that notices if I delete that refusal? So the sweep replaces each refusal site with a pass, one at a time, and reruns the suite. A site nobody catches is a gate that is not a gate.

CI, on the pushed commit:

baseline clean; 168 refusal sites (4 excluded, see EXCLUDE)
MUTATION SCORE: 168/168 = 1.000

The build fails under 0.99, so this is a gate rather than a dashboard.

The confirmation

I ran the same script locally, expecting 168:

MUTATION SCORE: 170/170 = 1.000

Same tool, same command, same score, two different denominators. 168 and 170 are both "everything", which is exactly why it is easy to miss. If the local run had said 0.94 I would have investigated in seconds. It said 1.000, so it looked like agreement.

Why

My working tree had 28 files that were not committed: 15 modified, 13 untracked. The verifier source itself was clean, so no new refusal sites had appeared in the thing being measured. What had changed was the suite and the fixtures around it, which decides which mutants get caught and which sites get excluded.

So the population moved. The sweep did not measure a different quality of the same system; it measured a different system.

The part I would rather not write

Here is a comment sitting in my own CI config, committed months ago, above the sibling checkouts:

# The score is only comparable if the suite is. Without these checkouts
# the real-bundle tests SKIP, four mutants stop being caught, and the
# denominator changes -- a mutation score that depends on what happens
# to be on the machine is the same host-dependence class this verifier
# exists to refuse. Check them out so CI measures what a developer does.

I wrote that. It names the failure precisely. And I still ran a local sweep to check a published figure, because checking felt like the careful thing to do.

Knowing a rule is not the same as being governed by it. The gate that actually held here was not my knowledge, it was CI, because CI had no choice about which tree it ran on.

What I changed

The number I put in front of other people is the one CI computed on the commit they can check out. Not the one my laptop computed on a tree only I have.

That sounds obvious written down. It was not obvious while I was doing it, because the local run felt like more diligence rather than less.

Two questions I would genuinely like answers to:

1. If you publish a metric out of your own repository, is its denominator stable across a clean clone and your working tree? Have you checked, or does it just feel stable?
2. Does your number get quoted from a CI log, or from whatever you last ran locally?

If you check and the two disagree, I would like to hear what moved. I suspect this is common and mostly invisible, because agreement on the headline figure hides disagreement underneath it.