Run the same LLM-as-judge eval twice and you can get pass, then fail, on identical input. Now try to build anything on top of that verdict. This is the problem mutation testing runs straight into when the eval is a judge, and here's how muteval deals with it.
Why one run isn't a measurement
Mutation testing needs a verdict per mutant: did the suite catch it (killed) or not (survived)? If the eval is a deterministic check, that verdict is stable. If it's an LLM judge, it's a coin with an unknown bias — the same mutant can flip between runs. A single run gives you one flip of that coin, not a measurement of the mutant.
Four things that make it robust
- Repeat and take a majority. Evaluate each mutant N times and only count it killed on a strict majority of runs (ties survive). One flaky verdict can't flip the outcome. If you'd rather set an explicit fail-rate threshold, you can.
- Flag what flipped. Any mutant whose verdict actually changed between runs (caught sometimes, missed other times) gets surfaced as flaky. That list is signal in itself — it's telling you where your judge is unstable enough to matter.
- Report an interval, not a point. With a noisy judge and a finite number of mutants, a single percentage is falsely precise. muteval reports a Wilson 95% interval, so you read the uncertainty instead of a confident-looking number that isn't.
- Run the cheap checks first. Rule-based checks run before the LLM judge and short-circuit — a mutant a deterministic check already kills never pays for a judge call. Less cost, and less exposure to judge noise, for free.
What this does not fix
None of this makes a bad judge good. It makes the measurement robust to a judge's noise — the run-to-run randomness. If your judge is systematically wrong (not just noisy), majority-voting it just gives you a stable wrong answer, confidently. Separating a noisy judge from a biased one is a different job, and it's the subject of the next part.
Also honest: more runs means more cost and time, and it scales with mutants × cases, so it adds up fast. And the confidence interval widens quickly when you have few mutants — which is a feature (it's telling you not to over-read a small run), but it's worth expecting.
The question
If your evals use an LLM judge: do you run them once? How do you know a green result in CI isn't a lucky flip that would have been red on the next run? I'd like to hear whether people repeat-and-vote, or just accept the noise.
Top comments (6)
Two things to add, because your closing question has a cheaper answer than "repeat more".
How many repeats you need is only a number once you have the flip rate. Under the strict-majority rule as written, the chance the majority is wrong is a binomial tail: at a 10% per-run flip rate, 3 runs give 2.8% and 5 runs 0.86%; at 20% you need 7 runs to get under 5% and 13 for 1%; at 30% it is 17 and 31. So repeat-and-vote is nearly free while the judge is stable, and goes vertical exactly in the regime where you can least afford it.
Which suggests the right move at high flip rates is not more N but a third outcome. If a mutant's per-run verdicts straddle 50%, you have not earned a binary killed/survived - you have an unresolved mutant. Carrying
unresolvedas first-class (excluded from the score numerator and denominator, reported as its own rate, CI on the reduced set) costs nothing extra and is honest about what was actually measured. Worth noting the tie rule pushes the other way:killed = fails * 2 > len(runs)(src/muteval/runner.py:382) means ties survive, so an even N and a genuinely undecided judge both default to "survived". For a gate whose job is to block regressions that asymmetry points the wrong way - an uninformative judge at N=2 produces a kill 25% of the time. Escalating N on a tie, and blocking when a high-severity mutant is still tied after escalation, gets you the severity stratification and the adaptive N in one move.Last thing: the judge is not the only thing that flips between runs. We re-ran a 60-cell benchmark from a fresh clone on a second machine - same code, same seeds, same pinned versions - and 42/60 cells matched exactly while the heavy cells drifted by up to about 10. That drift is a systematic offset for the whole session, so unlike judge noise it does not average out with N; every repeat inherits it. The practical split that survived contact with the second machine was to run the control arm inside the same invocation so the comparison is paired, then assert invariants exactly (this region spikes, that one does not) and magnitudes inside a band with the tolerance written next to the number.
Checked the line before answering since you cited it: runner.py:382 is exactly killed = fails * 2 > len(runs), ties survive. You're right on all three counts.
The binomial tail reframes "repeat more" as only honest in the stable regime — nearly free at a 10% flip rate, vertical exactly where you can least afford it. Which means a fixed N is the wrong knob; it should be a function of the measured flip rate, not a constant.
The third outcome is the real fix, and it's the strongest point. A mutant whose per-run verdicts straddle 50% hasn't earned a binary verdict — forcing it into killed/survived is manufacturing a result out of noise. Carrying unresolved as first-class — out of both numerator and denominator, reported as its own rate, CI on the reduced set — is honest about what was actually measured and costs nothing. muteval doesn't have it; it should. And you've caught that the tie rule makes it worse than neutral: with ties surviving, an undecided judge at N=2 still resolves to a definite verdict a quarter of the time, and for a gate you never want an undecided high-severity mutant silently defaulting either way. The move you describe — escalate N on a straddle, block a high-severity mutant that's still unresolved after escalation — quietly delivers two things other readers asked for in one mechanism: adaptive N (spend passes only where the verdict is close) and severity stratification (the block decision is per-tier). unresolved + escalate-on-tie is going on the list as a single change.
Session drift is the part I hadn't priced in. You're right that judge flip isn't the only source, and a per-session systematic offset is worse because it doesn't average out with N — every repeat inherits it. The one thing in muteval's favor: it scores a differential, not an absolute. The verdict is whether the eval flips baseline→mutant, and both are graded in the same invocation, so a session offset that hits both sides largely cancels in the contrast that gets scored — in a way it wouldn't if the baseline came from another machine. That's not a free pass, though: the individual eval verdicts are still absolute, so a judge sitting near its threshold can flip both, and a user eval that asserts an exact magnitude stays brittle no matter how muteval votes. Your discipline — assert invariants exactly, magnitudes in a band with the tolerance written next to the number — is the fix on the eval-authoring side. muteval can keep the comparison paired; it can't make a brittle assertion robust. That belongs in the docs.
Thanks — the middle point in particular ties together a couple of the other threads on this post.
Reporting a Wilson interval instead of a point percentage is the right discipline. If a judge has even a 10% flip rate, a single green check in CI is just an unpriced lottery ticket on prompt variance.
The bottleneck usually isn't statistical theory; it's the invoice. Running an N-pass majority vote across hundreds of mutants turns a linear eval budget into a painful fixed cost on every pull request. A squad shipping forty PRs a day with five model passes per mutant burns through their token allocation before lunch.
That cost convexity explains why so many teams retreat back to single-pass judges. Engineers understand the coin has an unknown bias, but the budget forces them to treat a lucky flip as deterministic verification. Short-circuiting with deterministic rule checks upstream is the only way to keep the expected cost from pricing the entire test suite out of CI.
Exactly — the invoice is the real constraint, and it's worse than linear because it's the product of mutants × cases × runs × judges. Five passes across a few hundred mutants per PR is how a linear eval budget becomes a fixed tax on every merge.
muteval leans on the levers you'd expect, and I'd call them load-bearing, not nice-to-haves: cheap rule-based checks run before the judge and short-circuit (a mutant a deterministic guardrail already kills never reaches a paid pass); an inert mutant whose output is byte-identical to the baseline reuses the baseline's outcomes for zero judge calls; an identical re-run is fully cached; and --max-calls fails closed before you overspend, not after. On a suite with real deterministic guardrails, most mutants die cheap and the judge only ever sees the survivors — which is exactly the upstream short-circuit you're describing.
But here's the honest gap, and I think it's where the cost curve actually bends: muteval runs a fixed N passes per mutant. You don't need five passes on a mutant whose first two agree 2–0 — you need them on the mutants sitting near the decision boundary. Adaptive N — stop early when the passes agree, escalate only when they don't (sequential testing) — is the right answer to the invoice, and muteval doesn't do it yet. Fixed-N is the naive version. That's going on the list; framing it as cost convexity is the clarifying lens.
I would also segment stability by consequence. A judge can look reliable across the full mutation set while remaining inconsistent on the small class of payment, permission, or compliance failures that matter most.
Do you calculate mutation survival and confidence intervals per failure class or risk tier? That would separate harmless variance from instability that should block acceptance.
This is the sharper version of the stability question, and honestly — no, muteval computes the score and the Wilson interval globally, not per risk tier. It has the ingredients but not this: every mutant carries a severity, critical-pattern text (refund / PII / permission / compliance) escalates it, and there's a --fail-on-severity gate that blocks on any surviving high-severity mutant. So risk is tagged per mutant and gated — but the aggregate stability signal (the judge flip-rate, the CI) is pooled across everything.
And you've named exactly why that isn't enough: a global 8% flip-rate could be the judge being rock-solid on the easy 90% and genuinely unsure precisely on the payment/permission class — the one place you'd want it to block. Pooling hides the instability that matters most. Stratifying survival + CI + flip-rate by severity is the honest fix, and it's a clean extension of machinery that's already there. Going on the list.
What's nice is it converges with the cost point in the other thread: risk-tiering is also a budget lever — spend your (adaptive) N passes on the high-consequence mutants where you need the stability, single-pass the rest. "Judge harder where it's expensive to be wrong" turns out to be both the cheaper policy and the more trustworthy one.