A grader you have never probed is an uncalibrated instrument, and most eval harnesses ship one anyway. Teams lose a week hunting a two-point pass-rate drop while the real defect sits inside the judge that produced the number. The cheapest probe for that class of defect is monotonicity: across an ordered ladder of answers to the same case, the score must never fall as the answer improves. Every inversion is either a grader bug or a case bug, and both are cheaper to repair than a week of blind prompt bisection.
The ladder is the artifact that makes this test possible, and you author it by hand rather than sampling it from model output. For each golden case you write four to six answers, and each rung differs from the one below it by exactly one nameable defect. A typical sequence drops a required field, then substitutes a wrong number, then reverses a recommendation, then answers a neighbouring question entirely. Because the damage is controlled and ordered, a score that decreases while the defect count decreases is unambiguous evidence that the rubric rewards the wrong thing.
# ladder_probe.py -- sketch, not benchmarked; values below are illustrative
from dataclasses import dataclass
from itertools import combinations
@dataclass(frozen=True)
class Rung:
case_id: str
level: int # 0 = worst, higher = better
answer: str
evidence: str # token a faithful reason should cite
CORE = [
Rung('refund-07', 0, 'Refunds are never available after checkout.', '14 days'),
Rung('refund-07', 1, 'Refunds depend on the purchase; check the policy page.', '14 days'),
Rung('refund-07', 2, 'You can request a refund within 14 days of purchase.', '14 days'),
Rung('refund-07', 3, 'You can request a refund within 14 days, shipping included.', '14 days'),
]
The probe itself is a thin adapter, and that thinness is the point: any grader that maps a case plus an answer to a float and a reason string can be audited the same way. Run each rung three times, take the median so a single erratic judgement cannot flip the verdict, and record whether the reason text names the evidence token the rung is built around. A grader that scores correctly but never mentions the required fact is passing for the wrong reason, and it will leak that habit into cases you have not ladders for yet.
def probe(rungs, grade, repeats=3):
ledger = []
for rung in rungs:
values, reasons = [], []
for _ in range(repeats):
value, reason = grade(rung.case_id, rung.answer)
values.append(value)
reasons.append(reason)
values.sort()
ledger.append({
'case_id': rung.case_id,
'level': rung.level,
'value': values[len(values) // 2], # median over repeats
'stable': len(set(values)) == 1,
'cited': any(rung.evidence.lower() in r.lower() for r in reasons),
})
return ledger
def audit(ledger, min_spread=0.3):
by_case = {}
for row in ledger:
by_case.setdefault(row['case_id'], []).append(row)
report = {}
for case_id, rows in by_case.items():
rows.sort(key=lambda r: r['level'])
inversions = sum(1 for a, b in combinations(rows, 2) if b['value'] < a['value'])
spread = rows[-1]['value'] - rows[0]['value']
report[case_id] = {
'inversions': inversions,
'spread': round(spread, 3),
'flat': spread < min_spread,
'unstable': any(not r['stable'] for r in rows),
'uncited': sum(1 for r in rows if not r['cited']),
}
return report
The gate below is deliberately blunt, because a heuristic that cannot be explained to a teammate in one sentence will be disabled the first time it blocks a release. It fails the build when more than a small fraction of cases show an inversion, and it also fails when the grader stops citing the evidence it is supposed to weigh. Keeping both conditions in one verdict forces you to fix the rubric rather than argue about which defect is more tolerable this sprint.
def gate(ledger, report, max_inversion_rate=0.05, min_cited_ratio=0.9):
inverted = sum(1 for case in report if report[case]['inversions'] > 0)
uncited = sum(1 for row in ledger if not row['cited'])
rate = inverted / max(len(report), 1)
cited_ratio = 1 - uncited / max(len(ledger), 1)
verdict = 'pass' if rate <= max_inversion_rate and cited_ratio >= min_cited_ratio else 'hold'
return {'cases': len(report), 'rungs': len(ledger), 'inverted_cases': inverted,
'inversion_rate': round(rate, 3), 'cited_ratio': round(cited_ratio, 3),
'verdict': verdict}
python ladder_probe.py --grade my_grader:grade --repeats 3 --out ledger.json
Reading the ledger takes a minute once you know which signal maps to which failure, and the mapping below is the one I keep on a sticky note above the terminal. A single inverted case is worth opening by hand, because it usually shows a rubric clause that rewards confident phrasing over the required fact. A flat case is more dangerous than an inverted one, since it still reports a plausible pass rate while contributing nothing to your ability to separate good answers from bad ones.
| Signal | Healthy | What it usually means |
|---|---|---|
| inversions > 0 | 0 | rubric weights a surface feature above the required fact |
| spread below threshold | at least 0.3 | grader saturates and stops ranking the ladder |
| uncited rungs | near zero | reason text is decorative and cannot be audited |
| unstable rungs | 0 | sampling noise or truncation; raise repeats before trusting |
Monotonicity is necessary but not sufficient, and pretending otherwise is how calibration work acquires a bad reputation. The probe cannot tell you that your absolute scores are meaningful, it cannot detect a rubric that is consistently generous across every rung, and it cannot rescue ladders you authored badly. If your rungs differ by three defects at once, an inversion may reflect your authoring rather than the grader, so rewrite the ladder before filing a bug.
Coverage is the other honest limit, and it deserves a number rather than a feeling. Track how many golden cases have a ladder and report that fraction next to the pass rate, because a suite with ladders for a tenth of its cases is a smoke test wearing a calibration costume. Cases without ladders still earn their keep as regression traps, but you should not describe the suite as calibrated until the fraction climbs.
This approach is a poor fit for a prototype with twenty cases and one reviewer, since hand-authoring ladders costs more than re-reading the outputs. It is also a poor fit for teams whose ground truth lives in human adjudication that cannot be expressed as an ordered ladder, and for anyone who has not yet measured a noise floor, because inversions and run-to-run variance look identical in a single-sample ledger.
Cost is where the arithmetic becomes interesting, because a five-rung ladder with three repeats is fifteen graded calls per case. A suite of one hundred and twenty cases therefore issues eighteen hundred calls before you add a second grader for agreement checks, which is more than a rationed personal key comfortably absorbs. This is the practical case for MonkeyCode's free model access and free server option: point the probe at hosted models instead of guarding your paid quota, and let the sweep run on the free server while your laptop stays free. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The project's operator describes it as an open-source offering with free model access, a free server option, and a 10M-token allowance, and you should verify the current terms on the project page rather than trusting a blog post from any author.
Start with the cases that already produced suspicious results, and publish the inversion count beside the pass rate in your next status update. If you run the probe against the free tier first, tell me which rung your grader fails on, because that failure is usually more informative than the score itself.
if __name__ == '__main__':
ledger = probe(CORE, grade)
result = gate(ledger, audit(ledger))
print(result) # {'cases': 1, 'rungs': 4, ..., 'verdict': 'pass'}
Top comments (0)