Give Your Eval Harness Cases That Must Fail

An eval suite whose score never moves is not evidence of stability; more often it is evidence that the graders stopped looking. Pass rates drift upward on their own as prompts get longer, schemas get looser, and substring checks keep matching the wrong text. The cheapest way to detect that decay is to add inputs your harness is required to reject: negative controls, borrowed from lab practice, where an assay that cannot register a known positive is declared broken before any result is trusted.

This article builds that audit for a prompt-evaluation harness. You will mutate a set of human-approved reference answers, run your existing checks against the mutants, and publish a per-check kill rate that fails CI when sensitivity drops. Everything below is runnable Python and a shell command, with no dependency on a specific model vendor.

The failure mode is a grader with no obligation to fail

Consider the check most teams write first, something like "refund" in output.lower(). It passes on a correct policy answer, and it also passes on an answer that cites the wrong clause, drops the citation, or states a 30-day window when the policy says 14. The check is not wrong so much as unpriced: nobody measured what it can distinguish, so its green tick is treated as information when it is closer to decoration. Silent regressions live exactly in that gap, because the prompt change, the truncation, and the schema relaxation all leave a cluster of harmless tokens intact.

The same blindness appears one layer up when the grader is a model. A grader model with a vague rubric, or one that has quietly been pointed at a different family than the generator, will hand out pass with high confidence and pleasing consistency. Averages hide this well, since a uniformly generous grader produces a flat, tidy scoreline that reviews nicely.

Negative controls turn the question around

Instead of asking how many cases pass, ask how many deliberately broken outputs your checks refuse. Each mutant is a small, mechanical corruption of a known-good answer that preserves plausibility while destroying a property you care about. A check that cannot kill any mutant is not a weak check; it is a check that should be deleted or replaced, because it is consuming maintenance attention while contributing no discrimination.

The implementation is short. Keep reference answers as data, keep checks as pure functions of the output string, and keep mutators in a registry so the audit can grow without touching the runner.

# harness/mutants.py
import re
from dataclasses import dataclass
from typing import Callable, Iterable

@dataclass(frozen=True)
class Case:
    id: str
    output: str                                   # human-approved reference answer
    checks: tuple[Callable[[str], bool], ...]     # True means accepted

Mutator = Callable[[str], str]

def drop_last_sentence(text: str) -> str:
    parts = [p for p in re.split(r"(?<=[.!?])\s+", text.strip()) if p]
    return " ".join(parts[:-1]) if len(parts) > 1 else text

def strip_citations(text: str) -> str:
    return re.sub(r"\[\d+\]", "", text)

def corrupt_first_number(text: str) -> str:
    return re.sub(r"\d+", lambda m: str(int(m.group()) + 1), text, count=1)

def truncate_at_half(text: str) -> str:
    return text[: max(1, len(text) // 2)]

def identity(text: str) -> str:                   # control: must survive every check
    return text

MUTATORS: dict[str, Mutator] = {
    "drop_last_sentence": drop_last_sentence,
    "strip_citations": strip_citations,
    "corrupt_first_number": corrupt_first_number,
    "truncate_at_half": truncate_at_half,
    "identity_control": identity,
}

The audit walks the cross product of cases and mutators, records which checks rejected each mutant, and reports a kill rate per mutator. Mutators that happen to be no-ops on a given case are skipped, because a mutant identical to the original measures nothing.

# harness/audit.py
from collections import defaultdict
from harness.mutants import Case, MUTATORS

def audit(cases: list[Case]) -> dict[str, dict]:
    stats = defaultdict(lambda: {"killed": 0, "total": 0, "survivors": []})
    for case in cases:
        for name, mutate in MUTATORS.items():
            mutant = mutate(case.output)
            if mutant == case.output and name != "identity_control":
                continue
            failed = [check for check in case.checks if not check(mutant)]
            bucket = stats[name]
            bucket["total"] += 1
            if failed:
                bucket["killed"] += 1
            else:
                bucket["survivors"].append(case.id)
    return {
        name: {"kill_rate": v["killed"] / v["total"], "survivors": v["survivors"]}
        for name, v in stats.items()
        if v["total"]
    }

Reading the report, and gating CI on it

The identity control inverts the expectation: its kill rate must be zero, and any nonzero value means your checks reject good answers and the suite is noisy rather than sensitive. The other mutators get floors, set per property rather than globally, since truncation should be caught essentially always while a citation check may legitimately miss a few cases that carry no numeric markers. The trace below is an illustrative shape rather than a benchmark; replace every number with your own run before publishing anything about it.

$ python -m harness.audit --cases cases/refunds.jsonl
strip_citations       0.94   survivors: c-118, c-204
truncate_at_half      1.00
corrupt_first_number  0.61   survivors: c-007, c-031, c-052, c-077
drop_last_sentence    0.88   survivors: c-016, c-093
identity_control      0.00   false alarms: none

The survivors list is the actual deliverable, because each surviving pair names a case and a property your checks cannot see. c-007 surviving corrupt_first_number usually means the numeric claim is graded by an LLM rubric with no extraction step, and the fix is a small deterministic assertion that pulls the figure out and compares it to the source field. Adding that assertion is a one-line change; finding it without the audit typically costs a week of confused incident review.

The gate should be boring and fail loudly. Floors live in a config file, the exit code is nonzero when a floor is breached, and the report is attached to the build so the diff between runs is visible.

FLOOR = {"strip_citations": 0.90, "corrupt_first_number": 0.80, "truncate_at_half": 0.95}

def gate(report: dict[str, dict]) -> None:
    breached = {
        name: round(row["kill_rate"], 2)
        for name, row in report.items()
        if name in FLOOR and row["kill_rate"] < FLOOR[name]
    }
    if breached:
        raise SystemExit(f"harness lost sensitivity: {breached}")

Where a second model family and a spare box fit

If your checks include a model grader, the audit acquires a second dimension, because a grader from the same family as the generator tends to share its blind spots and will accept mutants that an unrelated model rejects. Running the same 40 mutants through a grader from a different family is a cheap monoculture test, and the disagreement set is a short list of cases worth human review. Stamp the grader identifier and a hash of the rubric next to the model identifier in the report, so a sensitivity drop can be attributed to a prompt edit rather than guessed at later.

For the nightly sweep, the constraint is usually not compute but containment: the audit reads internal reference answers, so it should run somewhere you control and can wipe. MonkeyCode provides free model access and a free server option, which covers both the secondary grader slot and a scheduled run that does not compete with anyone's laptop; the project page documents current limits, and you should read them before designing around the free tier. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you already have a CI runner with egress rules you trust, use that instead and skip the extra moving part.

Limits, and who should not adopt this

Mutation testing only covers failure modes you can enumerate, so a semantically wrong answer that preserves citations, numbers, and sentence count will survive every mutant in the registry above. That is why the audit complements human spot checks rather than replacing them, and why the mutator list should grow every time a real incident reveals a property nobody was testing. Teams without stable, human-approved reference answers should not start here, since the mutants are derived from those answers and a shaky baseline produces noise instead of signal.

If your product decisions do not depend on a score, do not build any of this. The whole apparatus exists to protect a number that someone will cite in a review, and if no one cites your eval score, the maintenance cost of a sensitivity gate buys nothing. For teams that do cite it, the kill rate is the number that tells you whether the other number means anything.