Why I made my eval tool refuse to give a score

Most eval tools always hand you a number. Mine sometimes refuses — and I think that refusal is the more useful behavior. Here's why.

The vacuous green

muteval reports a mutation score: of the regressions I injected into your system, what percent did your eval suite catch? Naively that's killed / total. But there's a trap hiding in it.

Suppose your eval suite doesn't even pass on your original, un-mutated system — a broken assertion, a flaky judge, a bad threshold. Now every mutant also fails the suite… so every mutant looks "killed"… so the score is a triumphant 100%. It's a perfect number that means the exact opposite of perfect. Same story if no mutants could be generated (a one-line prompt with nothing to degrade): 0/0 is undefined, and a naive tool rounds that to a happy 1.0.

A tool whose entire job is "tell me whether my tests are any good" cannot afford to lie by omission in precisely the place you were trying to remove doubt.

Fail closed

So muteval earns a score only when two things are true: the baseline passed, and at least one mutant produced a clean verdict. Otherwise it doesn't guess — it returns a status instead of a number:

• baseline_failed / baseline_errored — your suite doesn't pass (or errored) on the original system. Fix that first; every downstream number is meaningless until you do.
• no_mutants / no_evaluated_mutants — nothing to test, or everything errored. No evidence is not a perfect score.
• partial_errors — some mutants errored (timeouts, API blips) above a budget, so the score would be computed over a shrunken denominator. Untrusted.

In all of these, score is None, CI exits non-zero, and no badge is written. The number only appears when it's earned.

And a number with an interval

When there is a score, it's a proportion over a finite number of mutants, so a point estimate is falsely precise. muteval reports a Wilson 95% interval alongside it: 2 of 3 mutants killed is 67% [21–94%], not 67%. On small runs, trust the interval, not the point.

What this costs

Fail-closed is occasionally annoying: a genuinely flaky judge can trip partial_errors mid-run, and there are flags to accept an error budget when you want to push through. That's a deliberate trade — the honesty is the default, the ergonomics are opt-in. I'd rather the tool irritate you by admitting it doesn't know than reassure you when it doesn't.

The question

When your eval tooling can't actually tell whether things are fine — a broken baseline, too little data, too many errors — what does it do? Does it green-light you, or does it admit it doesn't know? I've come to think that's one of the more important properties a measurement tool has, and one of the least advertised.

Repo, if you want to see the status codes in practice: https://github.com/AshwinUgale/muteval