model-drift grades models weekly on a frozen suite with an exact-match grader — no LLM judge, so a score change is real. That design has a sharp edge: a call that returns no valid answer scores identically to a wrong one. A refusal, a max_tokens truncation, a timeout, a parser failure on a quietly-changed schema — all land as zero, indistinguishable from the capability collapse the board exists to catch.
I hit the catchable version: a model appeared to drop from 69% to 3% overnight. The run log showed a 429 on 34 of 35 calls — a rate limit scored as a regression. Rate limits are catchable only because they leave a status code; refusals and truncations don't.
The board excludes by aggregate reliability: it drops a run's accuracy point only when that run's reliability falls below 50% — a catastrophic-infra floor, never a single failed call. That restraint matters because failures aren't missing-at-random: the long, hard prompts are the ones that hit token caps and timeouts, so a blanket drop-every-failure rule would exclude failures that correlate with difficulty and inflate accuracy on the discriminating tasks. An eval board has to tell a wrong answer from an absent one; miss that and you're scoring the provider's uptime, not the model.
The design was already exclude-by-class, not blanket. Code: github.com/egnaro9/model-drift
Writing the methodology in public is how it gets read this closely.
Top comments (1)
The 50% threshold is arbitrary for a specific reason: it's trying to infer infrastructure-vs-capability from the same zero-score signal that already collapsed the distinction, one level up. Reliability computed from exact-match scores is circular exactly where you need it not to be, because a hard task and a rate-limited task both produce the same zero, and the threshold is asking that zero to somehow carry information it discarded on the way in.
The 429 case works because status code isn't that signal. It comes from the transport layer, not the model, so it's the one piece of evidence in this design that's actually independent of the thing being graded. That's a different kind of signal than a statistical threshold, not a better-tuned version of it, and it's worth treating as its own tier rather than folding into reliability math: classify by status code when it exists, full stop, no threshold needed, because it's not a rate you're estimating, it's a fact you're reading.
Refusals and truncations don't have that, but they're not fully signal-free either, exact-match just discards what they do have. A refusal has a different response shape than a wrong answer: finish_reason=length instead of stop, content matching refusal boilerplate instead of an attempted answer. That's weaker than a status code, it's inferred from the model's own output rather than an external fact, but it's still more than nothing. So the real hierarchy is three tiers, not two: transport-signal (status code, external, trust it directly), response-shape-signal (refusal/truncation patterns, weaker, model-authored but still informative), and true zero-signal (a hard task that just failed, where reliability math is the only tool left and the 50% cutoff is a real judgment call, not a bug). Collapsing all three into one threshold is what makes 50% feel arbitrary, because it's doing three different jobs with one number.