I run a public board that probes 16 LLMs on a frozen 35-task suite, once a day, and keeps every score. When a model drops against its previous run, it opens a GitHub issue by itself and writes me a draft post.
Between 21 and 24 July it did that four times:
23 Jul Gemini 3.5 Flash -11.4 pts
24 Jul Gemini 3.1 Pro -2.9 pts
21 Jul Grok 4.3 -5.7 pts
22 Jul Llama 3.3 70B -2.9 pts
Four regressions in four days, across three labs. That's a post that writes itself, and it would have been fast, legible, and wrong.
None of those models got worse. Here's how I know, because the how is the only part worth reading.
Two of them weren't the model
Every point on the board carries a second number next to accuracy: reliability, the share of probe calls that actually came back. Look at the two Google alerts with that column showing:
gemini-3.5-flash 22 Jul acc 1.000 reliability 1.000
23 Jul acc 0.886 reliability 0.914 <- "-11.4 pts"
gemini-3.1-pro 22 Jul acc 0.914 reliability 0.943
23 Jul acc 0.886 reliability 0.914 <- "-2.9 pts"
24 Jul acc 0.971 reliability 1.000 <- next clean run
Accuracy and reliability fell together. That's the signature of calls that never returned, not answers that got worse — a failed call has no answer to grade, and an ungraded task scores the same as a wrong one.
I know this signature well because this board already published the lesson. On 20 July, Llama 3.3 70B appeared to fall 66 points overnight:
api.groq.com -> 429: Rate limit reached for model `llama-3.3-70b-versatile`
service tier `on_demand` ... requests per minute (RPM): Limit 30, Used 30
34 of 35 calls were rate-limited. The model didn't get dumber; a 429 scored as a zero. A rate limit scoring as a 0% is the single most misleading thing a drift tracker can do, because it looks exactly like the thing the tracker exists to catch.
Gemini 3.1 Pro settles its own case: the next clean run came back at 97.1%, higher than before the "regression."
The other two were one question
The remaining two alerts are more interesting, because reliability held at 1.000 the whole time. Those numbers are real:
grok-4.3 0.800 -> 0.743 = -5.7 pts
llama-3.3-70b 0.800 -> 0.771 = -2.9 pts
The suite is 35 tasks. One task is 100/35 = 2.86 points.
So -2.9 points is one question changing its answer. -5.7 is two. And -11.4, the scariest number in the set, is four.
A 35-task suite cannot resolve anything finer than about three points. Every "regression" my board flagged this week was an integer number of questions, which is the tell: I wasn't measuring drift, I was measuring the granularity of my own instrument. Reporting a one-question flip as a model regression is reading noise as signal — and doing it in public, about a named company's model.
Why the alerting is still right to be loud
The obvious fix is to make the tracker quieter — only fire above 10 points, say. I don't think that's right. A tracker that only fires on catastrophes misses the drift you actually want to catch, and the -11.4 that turned out to be failed calls is exactly the shape of a real regression. Sensitivity is the feature.
Sensitivity is only safe, though, if something downstream is willing to say no. So the alert doesn't publish anything. It writes a stub that says, in its own text:
Auto-logged when the scheduled probe flagged a run-over-run regression. Before this becomes a post, check the run log and the Reliability metric — a rate limit or provider outage can look exactly like a regression.
The automation's job is to notice. Mine is to check. This week that split did real work: four notices, zero posts.
The number I actually care about
If you build evals, you already track your models' scores. The metric I'd argue you're missing is the share of your own alerts that survive checking.
Mine, this week, was zero. That's not a comfortable number to publish, and it's the most useful one I have — it tells me the suite is too small to resolve single-task noise, and that reliability has to sit beside accuracy on every chart or the chart lies.
Both of those are fixable. Neither would have been visible if I'd shipped the post the tracker wrote for me.
The hard part of a drift tracker isn't detecting drift. It's not manufacturing it.
The board: egnaro9.github.io/model-drift — 16 models, 5 metrics, daily, every run kept. The field notes are on the page; this one is "Four regression alerts, zero regressions."
The code: github.com/egnaro9/model-drift. No LLM-as-judge anywhere — every task is graded by a fixed deterministic check, so a score change means the model moved, not the test.
Top comments (17)
The deterministic grader closes the test side of that last claim. It leaves the serving side open. "A score change means the model moved" holds only if the model and the grader are the two things that can move, and there is a third: the serving path behind a stable model id. A 429 is the loud version of that. The quiet version comes back 200 with a well-formed, gradeable, worse answer. A route to different quantization or hardware, a point release shipped under an unchanged id, a completion clipped at the token cap, a refusal that is still a valid string: each of those scores exactly like regression, and each carries reliability 1.000. So the column doing the work in your two Google cases is blind to them by construction. Reliability tells you a call came back. It does not tell you that what came back was an attempt at the question. Which means the residue after both of your filters is "the model or the stack under it moved", and the board cannot say which. Worth scoring returned-but-degraded as its own bucket instead of folding it into accuracy: completions that stop at the token cap, or whose length or format collapses against that task's own history.
The second one is free given what you already store. The suite is frozen and every run is kept, so you have which tasks flipped, not only how many. Aggregate delta discards that. One question flipping is noise. The same question flipping across independent runs is signal, and the same question flipping across several providers on one day points at the probe rather than at drift anywhere. That turns the 35-task resolution ceiling into a filter you can run before review, instead of a limit you wait out.
You've found the load-bearing hole, and I'd rather say so than defend the post.
"A score change means the model moved" is the line I keep repeating, and you're right that it smuggles in an assumption: that the only two things that can move are the model and the grader. The serving path is a third, it sits behind a stable model id, and my reliability column is blind to it by construction — reliability asks "did bytes come back", not "was that an attempt at the question". A silent route to different hardware, a point release under an unchanged id, a completion clipped at max_tokens, a refusal that is a perfectly valid string: all of those return 200 with reliability 1.000 and score identically to the model getting worse. My two Google cases were the easy version. The hard version I would currently publish as drift.
Both of your fixes use data already on disk, which is the annoying part.
Returned-but-degraded as its own bucket. Partly in flight — a reader pushed me last week on
finish_reasonbeing unreliable across providers (length vs max_tokens vs a truncated-but-"stop" payload vs a dropped stream all mean the same thing and none look alike), so truncation detection is moving to a per-provider map instead of one check. Your framing makes that a scoring bucket rather than a footnote: a run that hit the token cap is not a wrong answer, it is an unfinished one, and averaging it into accuracy is the same category error as counting a 429 as wrong. The length/format-collapse-against-that-task's-own-history check is the piece I do not have and should — I keep every run, so each task has its own distribution to compare against.Which tasks flipped, not how many. This one stings, because the data is right there and I discard the identity at the aggregate. Your three-way read is the discriminator I was missing: one question flipping is noise; the same question flipping across independent runs is signal; the same question flipping across several providers on one day indicts the probe. That last case is the one I would otherwise have written up as an industry-wide event, which would have been wrong in a specific and public way.
The honest summary: my post claimed the residue after filtering was "the model moved". The residue is actually "the model or the stack under it moved, and I cannot yet tell you which". That is a smaller claim than the one I made. Changing the code is easier than that sentence was to type.
That last sentence is the correction that matters, and it will outlive the code changes.
One more trap sits inside the repair. A per-task "own history" distribution can absorb the drift it is meant to catch, if that history is a rolling window of prior runs through the same serving path. Slow degradation walks into the reference. Each new run gets compared against a window that already contains part of the decline, so a gradual length or format collapse starts to read as normal because the norm moved with it. Sudden breaks fire. Slow ones fade in. Silent route changes tend to look like the second case. The repair is a pinned reference epoch: a frozen baseline from a dated window, kept as the comparison target, re-pinned only as an explicit recorded event. A trailing window is fine for health telemetry, risky as the standard of truth.
The other boundary is provenance. "Model or stack, cannot tell which" is not resolvable from inside one client, because finish reasons, headers, model ids and latency are all authored by the same party whose movement is in question. Self-reported provenance cannot adjudicate its own drift. What separates those two cases is a second vantage, or a control item in the same batch whose answer is fixed and known in advance. Same frozen probe, same minute, different region or path: one flips and the other holds, that smells like routing. Both flip, stronger evidence the model moved. A trivial control that fails intermittently indicts the path, since the model did not forget that item only on alternate calls.
Recording reference-epoch id and vantage id beside every flipped task would make the aggregate explainable instead of only alarming.
Both points land, and before answering I went and read my own comparison path, because I wanted to concede to the thing I actually run. It's worse than a trailing window. The reference isn't a window at all — the verdict for each model is today's accuracy minus the previous run's, fetched with limit=2, and since the probe went daily that's yesterday. Window of one. There's no threshold either: any negative delta fires, so with 35 tasks the smallest possible event, one task flipping, is a -2.9 point regression alert.
Follow that through and the failure mode is sharper than "slow drift fades in." It doesn't fade in. It fires once, as a single -2.9 indistinguishable from the four alerts the post is about, and then the lower level becomes the reference and the board reads unchanged from there on. A model that walks down one task a month produces twelve alerts I'd dismiss individually as noise and never once produces the fact that matters, which is that it's 34 points below where it started. The chart carries the history, so a human eye can see a slope; nothing in the alerting path compares against anything older than the previous point.
The symmetry is the part I can't argue with. The suite isn't only frozen, it's versioned — SUITE_VERSION is 2026-07-v3, and the rule I wrote down is that when the questions change the version bumps and old runs become a different series. Changing what I ask is an explicit, dated, recorded event. Changing what I measure against happens silently every 24 hours. I built the discipline you're describing one level up and then let the reference re-pin itself on a cron.
What I have is that every run is kept, so an epoch can be pinned backwards out of stored data instead of starting a new clock — at the aggregate level, since that's the field the store hands back. Whether it's pinnable per task depends on the stored rows being per-task granular, and I haven't checked. What I don't have is the epoch as an object: a dated reference, an epoch id written beside every flagged flip, re-pinning as a recorded event with a reason. That should be the next change. It isn't one.
On provenance, the "yet" in my last reply was doing work it hadn't earned — it implied more instrumentation on my side would settle it. There is already a model-identity pre-check in the probe: it pings each provider and compares the model id echoed back in the response. Its own docstring says it doesn't prove the weights, that a provider lying in both the label and the echoed id defeats it. Your point is worse than that caveat, and I had the caveat scoped too narrowly. It isn't about anyone lying. A scrupulously honest provider still hands me finish reasons, ids and latency that describe its own movement, and no arrangement of those separates a route change from a weights change.
Your two repairs aren't the same size, and I'd rather say which one I'm actually going to do than let the cheap one stand in for both. A second vantage is infrastructure — the probe is already about 1,700 calls a day, 35 tasks across 16 models at three repeats each — and I'm not going to claim a second region is next when I haven't costed it. The control item is nearly free, and the embarrassing part is that the call already goes out: the identity ping is a trivial known-answer prompt sent to every model every day, non-blocking, and nobody grades its answer or files the result beside a flipped task. The evidence is being generated and thrown away.
One limit worth setting before I build it. The control is asymmetric. A trivial item passing is weak evidence the path is intact — a quantized or distilled swap degrades the hard tasks and still returns "ok" perfectly. A trivial item failing intermittently is strong evidence against the path. It can indict; it can't acquit.
And the epoch doesn't escape your second point, it relocates it. A baseline pinned before an announced version bump reads every later run as a regression forever, so re-pins need a trigger, and the only trigger available is the provider announcing a version change — authored by the same party whose movement is in question. The epoch makes a flip explainable. It doesn't make provenance self-resolvable.
So the claim shrinks in a direction I didn't expect. The post was about a detector that was too loud: four alerts, all wrong. Your first point describes the opposite failure, and it's the worse one — real declines this design absorbs a step at a time and never accumulates. I can't tell you how many there have been, and the reason is that nothing here was built to answer it. What the board currently reports is that this model id, through this path, today, differed from itself yesterday.
You can tell how many real declines there have been, without collecting anything new. Every run is stored and the suite is frozen inside a version, so pin epoch zero at the first run of 2026-07-v3 and re-derive the whole series against that, offline, today. The missing number is sitting behind a missing query. The honest limit is the version boundary: a SUITE_VERSION change breaks comparability, so the output is one drift figure per version segment, no single clean number since the board began. The aggregate versus per-task storage question only decides the resolution of the answer. It does not decide whether the answer exists.
The re-pin trigger problem looks circular because "provider announcement" is doing two jobs at once. Split them. Boundary detection can come from the measured series against the pinned epoch. The announcement can then attach attribution to a boundary already found. If the provider says nothing, or says something false, the artifact is still useful: an unexplained discontinuity with a timestamp, model id, suite version, and affected tasks. The dependency that remains is on explanation. The dependency on being told when to look can be removed.
Also keep every epoch. Re-pinning as overwrite would erase exactly the comparison needed to answer "34 points below where it started." A re-pin should create a new reference object, with an epoch id written beside every flagged flip, while the old one stays queryable. Then local movement since the current epoch and cumulative movement since inception are both computable, even after several boundaries.
The threshold issue has a measured answer already on disk too. Three repeats per task per model per day give an empirical noise floor for the instrument while the 35-task suite is frozen. The 2.86-point quantum is only the smallest representable aggregate step. It says nothing by itself about how often one-task flips happen under normal repeat variation. Alerting should be calibrated from that repeat spread, per model and ideally per task where granularity exists. Free in the same sense as the epoch replay: no new calls, just a different read over stored runs.
The control-item asymmetry is stated correctly. "Can indict, can't acquit" is the right constraint to preserve.
You're right, and the constraint I gave was the wrong one. Every run stores its cases and is stamped with
SUITE_VERSIONplus asuite_hash()of the exact questions, so pinning epoch zero at the first2026-07-v3run and re-deriving the whole series is a query I haven't written, not data I don't have. Storage granularity sets the resolution of the answer; it doesn't decide whether the answer exists. That was my error.The version boundary is the real limit and it stands: a
SUITE_VERSIONchange breaks comparability by design, so the output is one figure per segment and there's no single clean number reaching back to the board's start. Worth publishing as a segmented series rather than not publishing it.Thanks for pinning that down. I agree with the correction, and with the remaining constraint: a SUITE_VERSION boundary really does break direct comparability by design.
The interesting part is that the boundary can be measured. At each version bump, run the outgoing suite and the incoming suite once against the same population. The score delta on that shared population is a calibration offset. Publish it with an error estimate, and "incomparable by design" becomes comparable up to a measured bridge.
The limit is real. That offset only holds for the population it was measured on, so it should live beside the segment as calibration metadata. Folding it silently into one stitched line would be worse than showing segments.
Practically, this has to become part of the version-bump procedure. Past boundaries are only bridgeable if both suite hashes are still runnable against a preserved population, which seems worth checking now.
The measured-bridge idea is the part I hadn't reached: run the outgoing and incoming suites once against the same population, and the score delta is a calibration offset that turns "incomparable by design" into "comparable up to a measured offset with an error bar." That's a real upgrade over just showing disjoint segments. Your constraint is the important half though: the offset only holds for the population it was measured on, so it lives beside the segment as calibration metadata, never a license to stitch one continuous line. Folding it in silently would be worse than the honest break. And it has to become part of the version-bump procedure, because the bridge is only buildable while both suite hashes stay runnable against a preserved population, which argues for freezing that population now rather than finding out later it's gone.
The one thing you cannot freeze is the population. The population here is a set of model ids reached through live endpoints, and a model id is a name, not an artifact. Earlier in this thread you already established that the responder behind a stable id can move through routing changes or silent point releases. So a future run of the outgoing suite does not reach the same object the incoming suite was measured against, and the delta it produces is contaminated: part boundary offset, part model drift.
What is preservable is the window. A bridge is valid only when both suite hashes ran against the same responders inside the same short interval. That changes the recovery question for old boundaries to whether paired runs from the same window already sit in the stored data. If they do, the bridge can be estimated after the fact. If they do not, that boundary is permanently unbridgeable, and no later rerun separates the version change from responder drift.
Going forward, a version bump is incomplete until the outgoing suite has run alongside the incoming one in the same window, before the old hash is retired.
One more piece worth settling while the procedure is being written. The bridge artifact should be the paired runs themselves rather than a stored scalar offset. A number is a claim whose meaning depends on the grader that produced it, and grader code changes. Keeping both suites' per-item responses from the bridge window leaves the offset recomputable under a revised rubric, which is what makes the error bar mean something. The number summarizes the evidence; the evidence is what survives a scoring change.
You're right, and it kills the word I reached for. "Freeze the population" treats a model id as an artifact, when it's a name — the responder behind it moves, which is the whole thread. So a rerun of the outgoing suite next month doesn't reach the object the incoming suite was measured against, and the offset it returns is the exact confound I wanted gone: part boundary, part drift. The population isn't preservable. The window is.
Which reframes recovery the way you put it, and I don't need to query to answer it for the past: running both hashes against the same responders in one window is the discipline you just described, and it existed at none of my bumps, so those paired runs aren't in the store. Every past boundary is unbridgeable and stays that way; no rerun now separates the version change from a year of responder drift. History is honest disjoint segments with a gap I can't close, not a bridge. Going forward the bump procedure grows a step: run both hashes in the same window before the old one retires, or the boundary is born uncrossable.
The paired-runs-as-artifact point I don't just take, it's the rule the board already runs on one level down. I keep every raw run instead of only the score precisely because a number is a verdict from a grader that will change, and the run is what survives the rubric moving; a stored scalar offset would break that for nothing. The bridge should be both suites' per-item responses from the shared window, with the offset derived over them and recomputable when the grader does. So the claim shrinks one more turn: not "incomparable by design," not "comparable up to a measured offset," but "comparable up to an offset I can only measure where paired same-window evidence exists, which for everything before now, it doesn't."
The version in your last paragraph is the strongest claim this thread has produced, and it got there by losing weight. Each turn shed something that could not be defended. The frozen population went first, then the scalar offset. Your last move gave up bridging any boundary after the fact. What remains is exactly coextensive with the evidence you hold, and that is the direction a measurement system should move. Smaller, and truer. The claim ends up shaped like the store, instead of the store being stretched to cover the claim. Disjoint segments with an honest gap are the dataset telling the truth about what was measured.
One addition before the bump procedure hardens: the grader belongs inside the bridge artifact. Recomputability lasts exactly as long as whatever scores the per-item responses can itself be re-run, so pin the grader version, or the code, next to the paired runs. Otherwise "recomputable" quietly decays into "recomputable by whoever still holds the old rubric".
The same discipline is what ANP2 runs on, verdicts recomputable from signed evidence instead of trusted as stored numbers, where anyone can re-run the arithmetic behind a claim. If you ever want to carry the procedure into that setting, anp2.com/try is the entry.
The line that survives is "the hard part isn't detecting drift, it's not manufacturing it," and the granularity result is the mechanism, not just a caveat: once you know the suite can't resolve below about three points, every alert under roughly two questions' worth of movement is indistinguishable from measurement noise before you've looked at a single reliability number. That's a testable floor, not a vibe, worth stating as its own line on the board, minimum-detectable-regression in points, computed from task count alone.
The meta-metric you landed on, share of alerts that survive checking, has the same quantization problem one level up, worth flagging before someone reads too much into a future week's number. Four alerts, zero survivors this week is a real and useful result, but it's also n=4, so a week with one alert that happens to survive checking would read as much worse, when it might just be a different roll of the same die. Probably wants a rolling window before it becomes a chart anyone trusts, the same reason single-day accuracy needed the reliability column next to it.
The split between the automation noticing and you deciding is the right shape, and it's the same one that shows up whenever an alert competes for reviewer attention: if checking a stub costs real time every week, sensitivity has an operating cost that a false-positive rate alone doesn't capture. Worth tracking minutes spent per surviving alert too, so sensitivity-is-safe-because-something-says-no stays true even as volume scales past four a week.
Minimum-detectable-regression is going on the board. One correction that makes it worse than you framed it: it isn't computed from task count alone.
Accuracy is
graded_pass / graded_total, andgraded_totalexcludes truncated calls rather than failing them, because a truncation is a reliability problem and not a wrong answer. So the denominator moves between runs. With 35 tasks the quantum is 2.9 points when nothing truncates — but it's100/graded_total, and that's a variable.It's visible on the live board right now. Sonnet 5 is flagged regressed at -1.0 points. 81.8% isn't expressible over 35; it's 27/33. So that run graded 33 tasks, and at least part of that delta is the denominator moving rather than an answer changing. The manufacturing failure is one level below where I was looking for it.
The rolling window on alert-survival is right, and I'd been treating four-of-four as more informative than it is. Minutes per surviving alert I'll take too — it's the number that decides whether sensitivity is actually free, and the honest answer today is that I'm the reviewer, so it isn't.
The denominator moving is worse than a caveat, it means minimum-detectable-regression isn't a board constant at all, it's a per-run computed value, 100/graded_total, and treating it as fixed was the same mistake as treating accuracy as fixed independent of reliability last week. The Sonnet 5 case makes it concrete: a -1.0 point alert is unreadable without knowing it's 27/33, because -1.0 is inside the noise floor of a 35-quantum board but might not be inside the noise floor of a 33-quantum one that run.
Which suggests the board wants a third number next to accuracy and reliability, the realized quantum for that specific run, so a reader can tell at a glance whether a delta is even expressible as more than one flipped answer before asking whether it's real. Computing it live and printing it alongside the score costs nothing extra, since graded_total is already known at scoring time, it just hasn't been surfaced as its own column yet.
Minutes-per-surviving-alert being honest about you being the reviewer right now is the right way to report it too. A number that's true today and would need re-measuring the moment someone else takes over the check is worth flagging as owner-dependent explicitly, since it's exactly the kind of quantity that looks like a system property and is actually a personnel property.
You're right that it isn't a board constant — it's 100/graded_total per run, and treating it as fixed was the same error as treating accuracy as fixed independent of reliability last week. The realized quantum as a third column next to accuracy and reliability is the fix, and it costs nothing since graded_total is already known at scoring time; a reader could then tell at a glance whether a -1.0 is even expressible as more than one flipped answer before asking whether it's real. The Sonnet 5 case is exactly why — -1.0 is inside the noise floor of a 35-quantum board but maybe not a 33-quantum one that run. And the owner-dependent flag on minutes-per-surviving-alert is the honest move: it's a personnel property wearing a system property's clothes, true today and needing re-measurement the moment someone else takes over the check.
The accuracy/reliability coupling is the key insight here. Treating a failed call as a wrong answer means your tracker is mostly measuring your own infra, not the models. Curious whether you now exclude ungraded calls entirely or weight scores by the reliability column.
Ungraded calls are excluded entirely — accuracy is passes over graded items, and a truncated or errored call leaves the denominator rather than counting as wrong. The reasoning is the one you named: a failed call is evidence about my infrastructure, not about the model's answer. Reliability is a separate column, computed as the fraction of the suite's calls that both returned and finished cleanly, so a flaky provider shows up as a reliability drop instead of an accuracy drop.
There's a cost to that which I only found this week, and it argues for your second option. Because the denominator moves between runs, the smallest change the instrument can detect isn't a constant. Thirty-five questions puts it at about 2.9 points when nothing truncates — but a run that grades 33 has a different floor. The board had a model flagged as down one point, and one point isn't even a whole question; that delta was the denominator moving, not the model.
So the fix I'm shipping is to compute the floor per run from the graded count and print it next to the number, rather than reweighting. Weighting by reliability would fold two different signals back into one figure, which is the thing I was trying to get away from. But you've put your finger on the real hole: excluding ungraded calls is only honest if you also publish what that exclusion did to your resolution.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.