A tool tells you your tests miss something. Before you rush off to write the test — is the gap actually real? Mutation testing has a decades-old problem here, and it matters just as much for evals.
The equivalent mutant
Mutation testing degrades the system and checks whether your suite notices. A mutant the suite doesn't catch looks like a coverage gap. But sometimes the mutation didn't change behavior at all — and then the suite passing is correct, not a gap.
Two concrete examples from the eval world:
Weakening must to should on a rule the model follows either way. The output is the same; nothing to catch.
Dropping a retrieved document the current question never needed. The answer is unchanged; retrieval quality wasn't tested by that case.
If you count those as gaps, two bad things happen: your score is deflated, and your survivor list fills with noise. And noise is what kills a tool like this — if it hands you 100 "gaps" and 95 are nonsense, you stop reading it.
What muteval does about it (and what it doesn't claim)
For every survivor, muteval diffs the mutant's output against the baseline's:
Output changed, evals still passed → a real coverage gap. Your suite watched the behavior shift and didn't flinch.
Output identical on the samples we ran → observationally unchanged. No output-based eval could have caught it here, so it's dropped from the "effective" score.
So there are two numbers: the raw score (killed / evaluated) and the effective score (which excludes the observationally-unchanged mutants). Report the effective one — but know exactly what it assumes.
The caveat I won't bury
"Identical output on N samples" is not proof of equivalence for a stochastic system. At temperature > 0, or with a flaky judge, a mutant that matched on the samples you saw might differ on one you didn't. That's why muteval labels these mutants observationally unchanged, never equivalent. Raising the number of runs per mutant hardens the claim — more samples, more confidence — but it never reaches certainty, and pretending otherwise would be the exact dishonesty this whole approach is supposed to remove.
The classic static "is this mutant equivalent?" analysis is undecidable in general anyway. This is a practical heuristic that catches the common cases and is honest about the ones it can't.
The question
In your own eval work — automated or by hand — how do you separate a real coverage gap from a mutation that never actually changed anything? Because getting that wrong in either direction (crying wolf, or missing a real hole) is where these tools lose trust.
Top comments (2)
Thank you for your thoughtful comment, Luis! Determinism is one of the core principles of Vireo. The strict state machine ensures that all agents follow the same lifecycle, regardless of the implementation language.
Regarding scalability:
The architecture is built on asynchronous messaging (Message Bus)
Each agent runs in its own container/process
Adding new roles does not require changes to the core
Testing is conducted with 59+ conformance tests
We are actively looking for partners for integrations and testing. I would be happy to discuss collaboration details — feel free to reach out to me in private messages.
Who's Luis? And what are you talking about buddy? Control your agents