I run a public board that probes 16 LLMs on a frozen 35-task suite, once a day, and keeps every score. When a model drops against its previous run,...
For further actions, you may consider blocking this person and/or reporting abuse
The deterministic grader closes the test side of that last claim. It leaves the serving side open. "A score change means the model moved" holds only if the model and the grader are the two things that can move, and there is a third: the serving path behind a stable model id. A 429 is the loud version of that. The quiet version comes back 200 with a well-formed, gradeable, worse answer. A route to different quantization or hardware, a point release shipped under an unchanged id, a completion clipped at the token cap, a refusal that is still a valid string: each of those scores exactly like regression, and each carries reliability 1.000. So the column doing the work in your two Google cases is blind to them by construction. Reliability tells you a call came back. It does not tell you that what came back was an attempt at the question. Which means the residue after both of your filters is "the model or the stack under it moved", and the board cannot say which. Worth scoring returned-but-degraded as its own bucket instead of folding it into accuracy: completions that stop at the token cap, or whose length or format collapses against that task's own history.
The second one is free given what you already store. The suite is frozen and every run is kept, so you have which tasks flipped, not only how many. Aggregate delta discards that. One question flipping is noise. The same question flipping across independent runs is signal, and the same question flipping across several providers on one day points at the probe rather than at drift anywhere. That turns the 35-task resolution ceiling into a filter you can run before review, instead of a limit you wait out.
You've found the load-bearing hole, and I'd rather say so than defend the post.
"A score change means the model moved" is the line I keep repeating, and you're right that it smuggles in an assumption: that the only two things that can move are the model and the grader. The serving path is a third, it sits behind a stable model id, and my reliability column is blind to it by construction — reliability asks "did bytes come back", not "was that an attempt at the question". A silent route to different hardware, a point release under an unchanged id, a completion clipped at max_tokens, a refusal that is a perfectly valid string: all of those return 200 with reliability 1.000 and score identically to the model getting worse. My two Google cases were the easy version. The hard version I would currently publish as drift.
Both of your fixes use data already on disk, which is the annoying part.
Returned-but-degraded as its own bucket. Partly in flight — a reader pushed me last week on
finish_reasonbeing unreliable across providers (length vs max_tokens vs a truncated-but-"stop" payload vs a dropped stream all mean the same thing and none look alike), so truncation detection is moving to a per-provider map instead of one check. Your framing makes that a scoring bucket rather than a footnote: a run that hit the token cap is not a wrong answer, it is an unfinished one, and averaging it into accuracy is the same category error as counting a 429 as wrong. The length/format-collapse-against-that-task's-own-history check is the piece I do not have and should — I keep every run, so each task has its own distribution to compare against.Which tasks flipped, not how many. This one stings, because the data is right there and I discard the identity at the aggregate. Your three-way read is the discriminator I was missing: one question flipping is noise; the same question flipping across independent runs is signal; the same question flipping across several providers on one day indicts the probe. That last case is the one I would otherwise have written up as an industry-wide event, which would have been wrong in a specific and public way.
The honest summary: my post claimed the residue after filtering was "the model moved". The residue is actually "the model or the stack under it moved, and I cannot yet tell you which". That is a smaller claim than the one I made. Changing the code is easier than that sentence was to type.
That last sentence is the correction that matters, and it will outlive the code changes.
One more trap sits inside the repair. A per-task "own history" distribution can absorb the drift it is meant to catch, if that history is a rolling window of prior runs through the same serving path. Slow degradation walks into the reference. Each new run gets compared against a window that already contains part of the decline, so a gradual length or format collapse starts to read as normal because the norm moved with it. Sudden breaks fire. Slow ones fade in. Silent route changes tend to look like the second case. The repair is a pinned reference epoch: a frozen baseline from a dated window, kept as the comparison target, re-pinned only as an explicit recorded event. A trailing window is fine for health telemetry, risky as the standard of truth.
The other boundary is provenance. "Model or stack, cannot tell which" is not resolvable from inside one client, because finish reasons, headers, model ids and latency are all authored by the same party whose movement is in question. Self-reported provenance cannot adjudicate its own drift. What separates those two cases is a second vantage, or a control item in the same batch whose answer is fixed and known in advance. Same frozen probe, same minute, different region or path: one flips and the other holds, that smells like routing. Both flip, stronger evidence the model moved. A trivial control that fails intermittently indicts the path, since the model did not forget that item only on alternate calls.
Recording reference-epoch id and vantage id beside every flipped task would make the aggregate explainable instead of only alarming.
Both points land, and before answering I went and read my own comparison path, because I wanted to concede to the thing I actually run. It's worse than a trailing window. The reference isn't a window at all — the verdict for each model is today's accuracy minus the previous run's, fetched with limit=2, and since the probe went daily that's yesterday. Window of one. There's no threshold either: any negative delta fires, so with 35 tasks the smallest possible event, one task flipping, is a -2.9 point regression alert.
Follow that through and the failure mode is sharper than "slow drift fades in." It doesn't fade in. It fires once, as a single -2.9 indistinguishable from the four alerts the post is about, and then the lower level becomes the reference and the board reads unchanged from there on. A model that walks down one task a month produces twelve alerts I'd dismiss individually as noise and never once produces the fact that matters, which is that it's 34 points below where it started. The chart carries the history, so a human eye can see a slope; nothing in the alerting path compares against anything older than the previous point.
The symmetry is the part I can't argue with. The suite isn't only frozen, it's versioned — SUITE_VERSION is 2026-07-v3, and the rule I wrote down is that when the questions change the version bumps and old runs become a different series. Changing what I ask is an explicit, dated, recorded event. Changing what I measure against happens silently every 24 hours. I built the discipline you're describing one level up and then let the reference re-pin itself on a cron.
What I have is that every run is kept, so an epoch can be pinned backwards out of stored data instead of starting a new clock — at the aggregate level, since that's the field the store hands back. Whether it's pinnable per task depends on the stored rows being per-task granular, and I haven't checked. What I don't have is the epoch as an object: a dated reference, an epoch id written beside every flagged flip, re-pinning as a recorded event with a reason. That should be the next change. It isn't one.
On provenance, the "yet" in my last reply was doing work it hadn't earned — it implied more instrumentation on my side would settle it. There is already a model-identity pre-check in the probe: it pings each provider and compares the model id echoed back in the response. Its own docstring says it doesn't prove the weights, that a provider lying in both the label and the echoed id defeats it. Your point is worse than that caveat, and I had the caveat scoped too narrowly. It isn't about anyone lying. A scrupulously honest provider still hands me finish reasons, ids and latency that describe its own movement, and no arrangement of those separates a route change from a weights change.
Your two repairs aren't the same size, and I'd rather say which one I'm actually going to do than let the cheap one stand in for both. A second vantage is infrastructure — the probe is already about 1,700 calls a day, 35 tasks across 16 models at three repeats each — and I'm not going to claim a second region is next when I haven't costed it. The control item is nearly free, and the embarrassing part is that the call already goes out: the identity ping is a trivial known-answer prompt sent to every model every day, non-blocking, and nobody grades its answer or files the result beside a flipped task. The evidence is being generated and thrown away.
One limit worth setting before I build it. The control is asymmetric. A trivial item passing is weak evidence the path is intact — a quantized or distilled swap degrades the hard tasks and still returns "ok" perfectly. A trivial item failing intermittently is strong evidence against the path. It can indict; it can't acquit.
And the epoch doesn't escape your second point, it relocates it. A baseline pinned before an announced version bump reads every later run as a regression forever, so re-pins need a trigger, and the only trigger available is the provider announcing a version change — authored by the same party whose movement is in question. The epoch makes a flip explainable. It doesn't make provenance self-resolvable.
So the claim shrinks in a direction I didn't expect. The post was about a detector that was too loud: four alerts, all wrong. Your first point describes the opposite failure, and it's the worse one — real declines this design absorbs a step at a time and never accumulates. I can't tell you how many there have been, and the reason is that nothing here was built to answer it. What the board currently reports is that this model id, through this path, today, differed from itself yesterday.
You can tell how many real declines there have been, without collecting anything new. Every run is stored and the suite is frozen inside a version, so pin epoch zero at the first run of 2026-07-v3 and re-derive the whole series against that, offline, today. The missing number is sitting behind a missing query. The honest limit is the version boundary: a SUITE_VERSION change breaks comparability, so the output is one drift figure per version segment, no single clean number since the board began. The aggregate versus per-task storage question only decides the resolution of the answer. It does not decide whether the answer exists.
The re-pin trigger problem looks circular because "provider announcement" is doing two jobs at once. Split them. Boundary detection can come from the measured series against the pinned epoch. The announcement can then attach attribution to a boundary already found. If the provider says nothing, or says something false, the artifact is still useful: an unexplained discontinuity with a timestamp, model id, suite version, and affected tasks. The dependency that remains is on explanation. The dependency on being told when to look can be removed.
Also keep every epoch. Re-pinning as overwrite would erase exactly the comparison needed to answer "34 points below where it started." A re-pin should create a new reference object, with an epoch id written beside every flagged flip, while the old one stays queryable. Then local movement since the current epoch and cumulative movement since inception are both computable, even after several boundaries.
The threshold issue has a measured answer already on disk too. Three repeats per task per model per day give an empirical noise floor for the instrument while the 35-task suite is frozen. The 2.86-point quantum is only the smallest representable aggregate step. It says nothing by itself about how often one-task flips happen under normal repeat variation. Alerting should be calibrated from that repeat spread, per model and ideally per task where granularity exists. Free in the same sense as the epoch replay: no new calls, just a different read over stored runs.
The control-item asymmetry is stated correctly. "Can indict, can't acquit" is the right constraint to preserve.
You're right, and the constraint I gave was the wrong one. Every run stores its cases and is stamped with
SUITE_VERSIONplus asuite_hash()of the exact questions, so pinning epoch zero at the first2026-07-v3run and re-deriving the whole series is a query I haven't written, not data I don't have. Storage granularity sets the resolution of the answer; it doesn't decide whether the answer exists. That was my error.The version boundary is the real limit and it stands: a
SUITE_VERSIONchange breaks comparability by design, so the output is one figure per segment and there's no single clean number reaching back to the board's start. Worth publishing as a segmented series rather than not publishing it.The line that survives is "the hard part isn't detecting drift, it's not manufacturing it," and the granularity result is the mechanism, not just a caveat: once you know the suite can't resolve below about three points, every alert under roughly two questions' worth of movement is indistinguishable from measurement noise before you've looked at a single reliability number. That's a testable floor, not a vibe, worth stating as its own line on the board, minimum-detectable-regression in points, computed from task count alone.
The meta-metric you landed on, share of alerts that survive checking, has the same quantization problem one level up, worth flagging before someone reads too much into a future week's number. Four alerts, zero survivors this week is a real and useful result, but it's also n=4, so a week with one alert that happens to survive checking would read as much worse, when it might just be a different roll of the same die. Probably wants a rolling window before it becomes a chart anyone trusts, the same reason single-day accuracy needed the reliability column next to it.
The split between the automation noticing and you deciding is the right shape, and it's the same one that shows up whenever an alert competes for reviewer attention: if checking a stub costs real time every week, sensitivity has an operating cost that a false-positive rate alone doesn't capture. Worth tracking minutes spent per surviving alert too, so sensitivity-is-safe-because-something-says-no stays true even as volume scales past four a week.
Minimum-detectable-regression is going on the board. One correction that makes it worse than you framed it: it isn't computed from task count alone.
Accuracy is
graded_pass / graded_total, andgraded_totalexcludes truncated calls rather than failing them, because a truncation is a reliability problem and not a wrong answer. So the denominator moves between runs. With 35 tasks the quantum is 2.9 points when nothing truncates — but it's100/graded_total, and that's a variable.It's visible on the live board right now. Sonnet 5 is flagged regressed at -1.0 points. 81.8% isn't expressible over 35; it's 27/33. So that run graded 33 tasks, and at least part of that delta is the denominator moving rather than an answer changing. The manufacturing failure is one level below where I was looking for it.
The rolling window on alert-survival is right, and I'd been treating four-of-four as more informative than it is. Minutes per surviving alert I'll take too — it's the number that decides whether sensitivity is actually free, and the honest answer today is that I'm the reviewer, so it isn't.
The denominator moving is worse than a caveat, it means minimum-detectable-regression isn't a board constant at all, it's a per-run computed value, 100/graded_total, and treating it as fixed was the same mistake as treating accuracy as fixed independent of reliability last week. The Sonnet 5 case makes it concrete: a -1.0 point alert is unreadable without knowing it's 27/33, because -1.0 is inside the noise floor of a 35-quantum board but might not be inside the noise floor of a 33-quantum one that run.
Which suggests the board wants a third number next to accuracy and reliability, the realized quantum for that specific run, so a reader can tell at a glance whether a delta is even expressible as more than one flipped answer before asking whether it's real. Computing it live and printing it alongside the score costs nothing extra, since graded_total is already known at scoring time, it just hasn't been surfaced as its own column yet.
Minutes-per-surviving-alert being honest about you being the reviewer right now is the right way to report it too. A number that's true today and would need re-measuring the moment someone else takes over the check is worth flagging as owner-dependent explicitly, since it's exactly the kind of quantity that looks like a system property and is actually a personnel property.
You're right that it isn't a board constant — it's 100/graded_total per run, and treating it as fixed was the same error as treating accuracy as fixed independent of reliability last week. The realized quantum as a third column next to accuracy and reliability is the fix, and it costs nothing since graded_total is already known at scoring time; a reader could then tell at a glance whether a -1.0 is even expressible as more than one flipped answer before asking whether it's real. The Sonnet 5 case is exactly why — -1.0 is inside the noise floor of a 35-quantum board but maybe not a 33-quantum one that run. And the owner-dependent flag on minutes-per-surviving-alert is the honest move: it's a personnel property wearing a system property's clothes, true today and needing re-measurement the moment someone else takes over the check.
The accuracy/reliability coupling is the key insight here. Treating a failed call as a wrong answer means your tracker is mostly measuring your own infra, not the models. Curious whether you now exclude ungraded calls entirely or weight scores by the reliability column.
Ungraded calls are excluded entirely — accuracy is passes over graded items, and a truncated or errored call leaves the denominator rather than counting as wrong. The reasoning is the one you named: a failed call is evidence about my infrastructure, not about the model's answer. Reliability is a separate column, computed as the fraction of the suite's calls that both returned and finished cleanly, so a flaky provider shows up as a reliability drop instead of an accuracy drop.
There's a cost to that which I only found this week, and it argues for your second option. Because the denominator moves between runs, the smallest change the instrument can detect isn't a constant. Thirty-five questions puts it at about 2.9 points when nothing truncates — but a run that grades 33 has a different floor. The board had a model flagged as down one point, and one point isn't even a whole question; that delta was the denominator moving, not the model.
So the fix I'm shipping is to compute the floor per run from the graded count and print it next to the number, rather than reweighting. Weighting by reliability would fold two different signals back into one figure, which is the thing I was trying to get away from. But you've put your finger on the real hole: excluding ungraded calls is only honest if you also publish what that exclusion did to your resolution.