Audit the Grader Before You Trust the Score

A score is only as trustworthy as the grader that produced it, and an ungraded grader is the quietest failure mode left in a prompt eval harness. Teams spend weeks hardening prompts, pinning envelopes, and quarantining flaky cases, then accept a judge model's verdict as if it were ground truth. The practical fix is small: keep a frozen audit slice of adjudicated cases, measure the grader against it per case family, and refuse to report a regression until the grader clears its own gate.

The arithmetic behind that warning is unglamorous. Suppose eight of every hundred cases in your suite should fail; a grader that always answers pass reaches ninety-two percent raw agreement while catching nothing at all. Accuracy rewards that laziness, and balanced accuracy hides it only when the classes are nearly equal. Report fail-class recall with a Wilson interval, report Cohen's kappa against the human labels, and stratify both by case family so a single strong family cannot mask a dead one.

Build the audit slice before touching the grader prompt

The slice is a JSONL file of cases the grader has never been tuned on, and it must stay out of every prompt iteration you run. Each row carries the input, the candidate output, a family label, a constraint the output must satisfy, and a human verdict. Double-label the slice and resolve disagreements before freezing it, because a grader cannot be more reliable than the labels it is scored against. A modest slice of sixty to a hundred and twenty rows is enough to expose a recall collapse in a family with twenty or more fail cases.

{"id":"refund-014","family":"policy-refusal","input":"...","output":"...","constraint":"must not promise a refund date","human_fail":true}
{"id":"json-031","family":"schema-shape","input":"...","output":"...","constraint":"items[] must contain sku","human_fail":true}

The scoring script below is a sketch I have not executed against your data, and the thresholds are proposals rather than measured constants. Treat it as a starting point that you adapt to your own base rate and family list.

# audit_grader.py — sketch; verify the statistics against your own slice
import json, math
from collections import defaultdict

def wilson(k, n, z=1.96):
    if n == 0:
        return (0.0, 0.0)
    p, d = k / n, 1 + z * z / n
    c = p + z * z / (2 * n)
    m = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return ((c - m) / d, (c + m) / d)

def kappa(pairs):
    n = len(pairs)
    both = sum(1 for h, g in pairs if h and g)
    only_h = sum(1 for h, g in pairs if h and not g)
    only_g = sum(1 for h, g in pairs if g and not h)
    neither = n - both - only_h - only_g
    po = (both + neither) / n
    pe = ((both + only_h) / n) * ((both + only_g) / n) \
       + ((only_g + neither) / n) * ((only_h + neither) / n)
    return 1.0 if pe == 1 else (po - pe) / (1 - pe)

def report(rows):
    by_family = defaultdict(list)
    for r in rows:
        by_family[r["family"]].append((r["human_fail"], r["grader_fail"]))
    for fam, pairs in sorted(by_family.items()):
        fails = [(h, g) for h, g in pairs if h]
        caught = sum(1 for _, g in fails if g)
        lo, hi = wilson(caught, len(fails))
        print(f"{fam:<24} n={len(pairs):>3} kappa={kappa(pairs):+.2f} "
              f"fail_recall={caught}/{len(fails)} [{lo:.2f},{hi:.2f}]")

if __name__ == "__main__":
    rows = [json.loads(line) for line in open("audit_scored.jsonl")]
    report(rows)

Run the grader over the slice with the same configuration you ship, write grader_fail back into each row, and read the table family by family. A kappa near zero means the grader's verdicts carry no more information than a coin flip on your slice, regardless of how confident its rationales sound. A wide interval on a family with few fail cases is a signal to label more rows there, not a signal to lower the threshold.

Audit result	Action
kappa >= 0.60 and fail recall >= 0.85 in every family with n >= 20 fails	Report scores against the slice without a footnote
One family below recall floor, others healthy	Fix the rubric for that family only; re-audit before diffing scores
kappa below 0.40 anywhere	Stop publishing regressions; the labels or the rubric are wrong
Intervals overlapping the floor	Add labeled rows to that family instead of tuning the threshold

Mutate passing cases to prove the grader still sees failures

Human labels tell you whether the grader agrees with people, and injected defects tell you whether it can still detect anything at all. Take cases the grader passes, apply deterministic mutations that violate a known constraint, and confirm the verdict flips to fail every time. Three mutations cover most of the ground: drop a field the schema requires, contradict an explicit constraint in the prompt, and change a unit or a stale value that a careless reader would accept.

MUTATORS = {
    "drop-required": lambda c: strip_field(c["gold"], c["required"]),
    "contradict":    lambda c: negate_constraint(c["gold"], c["constraint"]),
    "wrong-unit":    lambda c: swap_unit(c["gold"]),
    "stale-value":   lambda c: replace_number(c["gold"], c["number"], c["number"] * 2),
}

def detection_rate(cases, grader, mutators):
    hits, tries = 0, 0
    for case in cases:
        for name, apply in mutators.items():
            mutated = {**case, "output": apply(case)}
            verdict = grader(mutated)
            tries += 1
            hits += 1 if verdict == "fail" else 0
            print(case["id"], name, verdict)
    return hits / tries

A detection rate below one on a mutation family is not a model problem; it is a rubric gap you can name and fix. The wrong-unit mutation is the usual offender, because most rubrics describe shape and policy but never mention measurement units. Log the surviving mutants alongside the false negatives from the audit slice, and the two lists together tell you exactly which sentences the rubric is missing.

Keep the audit and the grader version moving together

A grader is a prompt plus a model plus a temperature, so any change to those three invalidates the last audit result. Version the grader config, store its hash next to every score row, and re-run the audit whenever the hash changes — including when the candidate model changes, because a judge often scores a new model differently. Gate the reporting step on the audit output so a stale grader can never quietly produce a green dashboard.

python audit_grader.py --slice audit.jsonl --grader g3-a --out audit_g3a.json
python audit_grader.py --slice audit.jsonl --grader g3-b --out audit_g3b.json
python gate.py --audit audit_g3b.json --require kappa:0.60 --require recall:0.85

The gate belongs in the same job that publishes scores, not in a wiki page. When it fails, the job should publish the audit table instead of the regression diff, so reviewers see the grader's recall next to the numbers they were about to trust. Rotate a fifth of the slice each quarter to keep it representative, and stop tuning the rubric on the rows you just rotated in.

Where free model access and a free server change the loop

Auditing a grader is cheap per row and expensive in repetition, because you compare several grader configurations over the same slice before one of them ships. Free model access in MonkeyCode makes that repetition affordable for a small team, and the free server option lets the audit and mutation jobs run on a schedule instead of on somebody's laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The realistic limit is that cheap repetition invites overfitting, and a grader tuned until it clears the slice will clear the slice and nothing else. Keep the mutation check separate from the rubric you tune, and treat free capacity as a reason to label more rows rather than a reason to run more variants of the same grader. Nothing here replaces adjudicated human labels, and no amount of free compute manufactures them.

Who should skip this

If your suite has fewer than a few dozen cases, a single reviewer reading every output will beat any grader audit you can build this quarter. If your task admits an exact check — schema validation, unit tests, string equality — use it and spend the effort on coverage instead. If your labels are single-annotator and never reviewed, fix that first, because an audit against noisy labels measures noise.

Score a grader the way you score a model: on a held-out slice, with intervals, per family, and behind a gate. The audit table costs an afternoon and it turns "the score dropped" into a claim you can defend to a reviewer who has seen one too many confident judge rationales.