model-drift grades models weekly on a frozen suite with an exact-match grader — no LLM judge, so a score change is real. That design has a sharp edge: a call that returns no valid answer scores identically to a wrong one. A refusal, a max_tokens truncation, a timeout, a parser failure on a quietly-changed schema — all land as zero, indistinguishable from the capability collapse the board exists to catch.
I hit the catchable version: a model appeared to drop from 69% to 3% overnight. The run log showed a 429 on 34 of 35 calls — a rate limit scored as a regression. Rate limits are catchable only because they leave a status code; refusals and truncations don't.
The board excludes by aggregate reliability: it drops a run's accuracy point only when that run's reliability falls below 50% — a catastrophic-infra floor, never a single failed call. That restraint matters because failures aren't missing-at-random: the long, hard prompts are the ones that hit token caps and timeouts, so a blanket drop-every-failure rule would exclude failures that correlate with difficulty and inflate accuracy on the discriminating tasks. An eval board has to tell a wrong answer from an absent one; miss that and you're scoring the provider's uptime, not the model.
The design was already exclude-by-class, not blanket. Code: github.com/egnaro9/model-drift
Writing the methodology in public is how it gets read this closely.
Top comments (0)