There is a mature literature on what happens when you swap one measuring instrument for another, and it is not in machine learning.
The standard treatment is Bland and Altman, "Statistical methods for assessing agreement between two methods of clinical measurement", Lancet 1986; i: 307-310. Their core move is the one worth borrowing: do not correlate the two instruments. Plot their difference against their average, and look at the shape.
Correlation is the wrong tool because it measures association rather than agreement, and it inflates with the spread of whatever sample you happened to use. Two instruments can correlate at 1.00 while one of them reads a fifth of the scale high everywhere it has room to. A constant offset is a perfect linear relationship, so correlation reports agreement that is not there.
An LLM-as-judge is an instrument. When we replaced ours with its successor and the weekly quality score moved from 0.78 to 0.83 in the same week, we had changed the system being measured and the thing measuring it at once, and the honest position was that we did not know which had moved. Our scores run 0 to 1.
The usual response is to estimate an offset and subtract it. That is a reasonable instinct and it is also a model, with assumptions, and the assumptions are checkable. Three checks, and the third one I got wrong the first time I ran it.
The apparatus
An anchor set: a fixed collection of (input, output) pairs, with the outputs frozen, scored under both judges. Paired, same items, same text, two raters.
Ours had 60 items. On it, sd(d) was 0.12, where d is the per-item difference, new judge minus old. Both judges had a score sd of 0.22 on the anchor set, so the ratio sd(new)/sd(old) came out at 1.0 to the precision I can report from two figures rounded to two decimals. That ratio matters more than it looks, and check two is where I explain why. The scores themselves are continuous, a weighted rubric rolled up to 0 to 1, not a grid of discrete levels; that turns out to matter for check three.
Check one: the offset, and how badly you know it
The offset is the mean of d, and the useful number is not the offset but its confidence interval:
half-width = 1.96 x sd(d) / sqrt(n)
Rearranged, that gives the anchor set size you need for a given precision:
±0.05 on the offset: 10 items at sd(d) = 0.08, 23 at 0.12, 50 at 0.18
±0.03 on the offset: 28 items at sd(d) = 0.08, 62 at 0.12, 139 at 0.18
±0.02 on the offset: 62 items at sd(d) = 0.08, 139 at 0.12, 312 at 0.18
±0.01 on the offset: 246 items at sd(d) = 0.08, 554 at 0.12, 1,245 at 0.18
That table uses z = 1.96, the large-sample form. Under the t distribution, which is what Bland and Altman specify because sd(d) is estimated rather than known, every cell rises by two or three items. Proportionally it only matters at the top: the 10 becomes 13 and the 23 becomes 25, while the 554 becomes 556.
At 60 items and sd(d) = 0.12, our offset is pinned to ±0.030.
Now put the two numbers next to each other. We were trying to detect a product improvement of about 0.05, using a correction whose own uncertainty was ±0.030. The error bar on the correction was about 60% of the thing being measured. Subtracting that offset relabels the confounding rather than removing it, and it does so in a way that looks precise because a single number has been subtracted.
Note which interval that is, because I conflated the two for longer than I would like. The ±0.030 is the precision of the mean offset, which is the right number for correcting an aggregate. Bland and Altman's headline statistic is the limits of agreement, mean ± 1.96 x sd(d), which for us is ±0.235, and at 60 items each of those limits is itself uncertain by roughly ±0.05. Any individual rescaled score carries the wide interval, not the narrow one. If you plan to correct a per-item score and act on it, you are working with something close to eight times the width you computed.
A word on why the paired design is worth the trouble, since the reason usually given is wrong. Scoring the same frozen outputs twice does not merely make sd(d) small. Against two disjoint samples of frozen outputs, one per judge, the paired design needs 2/(1 - rho) times fewer items and 1/(1 - rho) times fewer judge calls, so it wins on items even when the two judges are uncorrelated. What rho buys is the judge calls. Our sd(d) of 0.12 against a score sd of 0.22 implies rho of about 0.85, and at ±0.01 that is 3,719 items per arm unpaired, 7,438 in total, against 554 scored by both judges: thirteen times the items, or 6.7 times the judge calls.
(The identity sd(d)² = 2 x sd(score)² x (1 - rho) assumes both judges have the same score sd, which ours did. If yours do not, use rho = (sd_new² + sd_old² - sd_d²) / (2 x sd_new x sd_old), because the shortcut will quietly mis-state rho and everything downstream of it.)
Check two: did the spread change, or only the location?
Subtracting an offset assumes the new judge is the old judge shifted. In what I have seen, it is at least as common for a new judge to be compressed: more agreeable, more clustered in the middle, fewer confident lows. The mean moves and the standard deviation shrinks.
The diagnostic is the ratio sd(new) / sd(old) on the anchor set, plus the difference-against-average plot. If the difference d trends with the average score, or fans out at one end, the relationship is not a shift and no single constant will fix it.
Ours came back at 1.00, which is the boring answer and the one that let me use the simple rho identity above. When it does not come back near 1, the consequence is specific: a compressed judge attenuates every future comparison toward zero, your real improvements shrink, your real regressions shrink, and the series goes quiet. This is the same shape as the attenuation I wrote about last week, arriving by a different route. There the effect shrank because a binary judge's flips are bounded. Here it would shrink because the judge's scale is narrower.
I owe this check the same precision arithmetic I gave check one, and I have not done it. There is a proper paired-variance test (Pitman and Morgan's, which works by testing the correlation between the sum and the difference), and I have not worked out what interval 60 items buys on the ratio. So the point estimate is 1.00 and I am not going to pretend I know how tightly. I would rather say that than quote a number I have not derived.
Check three: did the ranking change, and what would "unchanged" have looked like?
The offset and the spread are both about the scale. The third question is about order: does the new judge rank items the way the old one did? Kendall's tau or Spearman's rho on the pairs.
Here is the mistake I made, and it is the reason this section exists.
I computed tau, got 0.65, and read that as substantial rank movement. It is not. Before reading tau you have to work out what tau you would get if nothing but noise had changed. For jointly normal scores, tau = (2/pi) x arcsin(rho), and our implied rho of 0.85 gives tau = 0.6483 on its own. So the gap I thought I was reading was 0.002, which at 60 items is not a gap at all. The arcsin value is the floor and I had mistaken it for a finding.
And then the uncomfortable corollary, which is the reason I now treat this check as the weakest of the three. Under joint normality, rho and tau carry the same information, so comparing observed tau against the rho-implied floor is not really a rank-change detector at all. It is a test of the normality and linearity assumption. It fires when the relationship between the judges is monotone but curved, or when a handful of extreme pairs are propping up the Pearson correlation.
One caveat on the floor itself, which I got wrong on the first attempt. The arcsin identity is a continuous-normal result. Our anchor scores are continuous, so it applies. If yours sit on a discrete rubric grid, ties break it in both directions at once: tau-a is dragged down by tied pairs, while tau-b's tie correction over-restores and lands above the continuous value. On a grid the arcsin number is simply the wrong null, and the floor has to be recomputed on your actual grid before any of this means anything. Say which variant you computed, too.
So the check is worth running, and what it earns you is a reason to look harder at the difference-against-average plot. It does not on its own earn you the conclusion that you have two series.
When you do conclude the ranks have moved, it decides whether splicing is possible. If ranks are preserved, some monotone transformation maps one scale to the other, and with enough anchor items you can estimate it. If they have moved, there is no item-level rescaling that makes a score from March comparable to a score from August. You can still equate the two score distributions and keep the aggregate series roughly comparable, at the cost of no individual rescaled score being trustworthy.
Anchor-set construction: freeze the outputs
The outputs must be frozen artefacts, not re-generated.
It is tempting to take last quarter's 200 eval inputs, run today's system on them, and score that with both judges. It feels like the same thing. It is not: the model, the prompt and the retrieval index have all moved, so the paired differences now contain both judge drift and system drift, and the whole point of the exercise was to separate those. Store the text, score the text, and accept that the anchor set stops being a test suite the moment you freeze it.
When the ranks have moved
I have found only one honest option, and it is expensive: run both judges in parallel on a sample of traffic for a transition window, publish both series side by side, and let the old series end rather than pretending it continues. Annotate the chart at the switch date. Anyone who reads across the boundary is then doing so knowingly.
The alternative, which I have watched happen, is that the offset gets subtracted, the chart stays smooth, and six weeks later a genuine regression is invisible because it is the same size as the correction nobody re-examined.
FAQ
How often does the judge actually change? More often than the calendar suggests. A version pin protects you from silent provider updates but not from your own team changing the rubric text, the temperature, the few-shot examples, or the parser. Every one of those is a new instrument by this argument, and the rubric edit is the most frequent and least announced.
Can I skip this if I only look at relative comparisons within a week? Partly. The judge's offset cancels out of a comparison where both arms were scored by the same judge in the same window. Its scale does not: a compressed judge attenuates the contrast by the same factor in both weeks, so the comparison stays valid in sign and direction but not in magnitude. That is still a good reason to structure reporting around within-week contrasts rather than a long-running level, because the level is where the trouble is worst, and levels are what end up on dashboards.
Is 554 anchor items realistic? For ±0.01, at our sd(d), that is what the arithmetic says. Most teams do not need ±0.01, and the table above lets you buy the precision you actually need.
Does a deterministic metric avoid all of this? For the metric itself, yes. Exact match does not drift. But most people are not upgrading exact match, they are upgrading the thing they use because exact match was inadequate, and that thing has a version.
Open question
I have been assuming the anchor set is representative of production traffic, and everything above inherits that assumption. But an anchor set is frozen by construction, and traffic is not. Six months in, the anchor set describes a distribution that no longer arrives, so the offset you measured is the offset on old traffic, which may not be the offset on current traffic.
Refreshing the anchor set restores representativeness and destroys comparability with everything measured before the refresh, which is the same problem one level up.
I do not have a clean answer. The best I have managed is overlapping anchor sets, retiring and adding a slice at a time so consecutive versions share most of their items, which is the common-item nonequivalent-groups design from the test-equating literature (Kolen and Brennan, Test Equating, Scaling, and Linking). Whether that is worth the machinery for an internal quality metric, I genuinely do not know.
Top comments (0)