DEV Community: Maya Andersson

More eval traces will not stabilize your kappa. Stratify the ones you have

Maya Andersson — Tue, 09 Jun 2026 18:40:44 +0000

TL;DR: Our LLM-as-judge agreement (Cohen's kappa against human labels) swung between 0.41 and 0.63 week to week with no rubric change. First instinct was sample size, so we went from 50 weekly traces to 200. Variance barely moved. Then we stratified the 50 we already had, by score class and a couple of known failure dimensions, and the swing dropped more than quadrupling the sample did. Composition was the lever, not volume.

The symptom: kappa that will not sit still

The judge scored production traces against a 5-point rubric. Each week we hand-labeled a calibration set and computed kappa. It bounced: 0.55, then 0.42, then 0.61. Nothing in the rubric or the judge prompt had changed. A kappa that moves 0.2 on noise is useless as an early-warning signal, because you cannot tell a real judge regression from the wobble.

Why adding traces did almost nothing

Random sampling pulls mostly from the majority class. For us that was clean passes, the easy 5s. Kappa is driven by agreement on the rare, ambiguous classes (the 2s and 3s), and random sampling gives you only a handful of those no matter how big the sample gets. So 200 random traces was mostly more easy passes: more data, almost no new signal where it counts.

Sampling	n	kappa range over 4 weeks
Random	50	0.41 to 0.63
Random	200	0.43 to 0.61
Stratified	50	0.52 to 0.58

Fix step one: stratify by score class

Force every score class into the weekly set so the rare classes are actually estimable.

from sklearn.model_selection import train_test_split
# represent every judge score class in the weekly calibration set
cal, _ = train_test_split(
    traces, train_size=50, stratify=traces["score_class"], random_state=0
)

Fix step two: stratify by the failure dimensions you already know

Score class alone was not enough. We added two dimensions we had been burned by before (input length bucket and whether the trace was multi-turn) and stratified on the combination. The rare-and-hard cases now show up every week instead of randomly, so the kappa we compute is measuring the part of the distribution that actually drifts.

What I am still unsure about

Stratification needs you to know which dimensions matter. For a brand-new judge you do not know them yet, so you are stuck random-sampling until enough failures teach you the strata. I do not have a clean answer for that cold-start case. If you stratify a calibration set before you have failure data, what do you stratify on besides score class?

FAQ

How few traces can you get away with?
For us 50 stratified was stable. Below about 30 the rare classes had too few examples to estimate kappa at all.
Doesn't stratifying bias the estimate?
Yes, on purpose, toward the hard cases. We report both the stratified kappa (the early-warning number) and the raw kappa (the honest population number).
Which judge model?
A frontier model from a different family than the system under test. The cross-family part matters more than the exact model.

Calibration set size for LLM-as-judge: when 50 traces is enough and when 200 is mandatory

Maya Andersson — Thu, 04 Jun 2026 16:57:46 +0000

TL;DR. The human-labeled calibration set you use to validate an LLM-as-judge does not need a fixed size. It needs a size that depends on how balanced your labels are. For roughly balanced binary criteria with no heavy tail, 50 stratified traces will usually pin Cohen's kappa to within a tolerable band (in my runs, a 95 percent bootstrap interval on the order of plus or minus 0.10 to 0.15). The moment you have a rare-but-expensive category, say a safety violation that shows up in 6 percent of traces, 50 is not enough and you should plan for 200 or more, because the variance of kappa is dominated by the count of minority-class examples, not the total. Below I give the kappa formula and why it is sensitive to the marginal distribution, the sample-size intuition, Wilson confidence intervals for small-n per-class precision, and the stratified-sampling routine that keeps marginals stable week to week. Pasteable Python at the end.

1. Kappa, and why the marginal distribution is doing more work than you think

Cohen's kappa (Cohen, 1960) measures agreement between two raters corrected for the agreement you would expect by chance. Here the two raters are your human labeler and your LLM-as-judge. The formula is kappa = (p_o - p_e) / (1 - p_e), where p_o is observed agreement and p_e is chance agreement computed from the marginals. For a binary label, if the human marks "pass" with probability a and the judge with probability b, then p_e = a*b + (1-a)*(1-b).

The part people skim past is that p_e is a function of the label marginals, and that function is not linear. When the classes are balanced (a and b near 0.5), p_e sits near 0.5, the denominator is near 0.5, and kappa is well-behaved. When one class is rare (a near 0.95), p_e is pushed close to 1, the denominator collapses toward zero, and kappa becomes a ratio of two small numbers. Small numbers in a denominator are how you get instability. This is the origin of the kappa paradoxes: you can have 95 percent observed agreement and a kappa near zero, purely because the marginals are lopsided. It is not a bug in your judge. It is the chance-correction working as designed on a distribution where chance agreement is already very high. The practical consequence for sizing: a class-imbalanced set carries less information per trace, so you need more traces for the same precision on kappa. I want to be careful not to overclaim. Kappa is not broken on imbalanced data. It is higher-variance, and you pay for that variance with sample size.

A note on what I report. I treat kappa as descriptive, not as a hypothesis test. The interesting question is never "is kappa significantly greater than zero" (it almost always is, and that bar is meaningless). The question is "is kappa high enough, with a tight enough interval, that I trust this judge to stand in for a human." That is a question about the width of the interval, which is a question about n.

2. The case for 50 traces: balanced binary criteria, no heavy tail

If your label is a roughly balanced binary criterion, 50 stratified human labels are often enough to make a deployment decision. Two things have to hold: the criterion is genuinely binary and roughly balanced (each class in the 30 to 70 percent range), and there is no rare-but-costly tail you care about separately. Under those conditions the variance of kappa is at its most forgiving.

A concrete example from my own work. I had a 5-class quality scale (a 1-to-5 Likert a previous team had wired into the judge). Kappa with humans was 0.47, and the bootstrap interval on 80 examples was wide enough that I could not tell 0.47 from 0.35. The 5-class scale was the problem: it spread the marginals thin across five buckets, so most cells had tiny counts. I split it into three binary criteria (is it factually supported, is it relevant, is it complete) and re-labeled the same traces. On the "factually supported" criterion, which was close to balanced, kappa came out at 0.78 on 50 examples with an interval I was comfortable shipping on. Same traces, same judge prompt structure, very different statistical footing. The honest caveat: 50 works for "does this judge agree with us well enough on the common case." It does not work for "does this judge catch the rare bad thing."

3. When 200 becomes mandatory: heavy tails and the rare expensive class

The variance of kappa scales inversely with n, which everyone knows, but it also scales with the rarity of the minority class, which fewer people budget for. If the category you care about appears in 6 percent of traces, a 50-trace sample contains, in expectation, three examples of it. Three. Your estimate of the judge's recall on that category is being driven by three data points. This is where I insist on 200 or more. Not because 200 is magic, but because of what it does to the minority-class count: at a 6 percent base rate, 200 traces gives about 12 minority examples in expectation, 400 gives about 24. Pick the rarest class you care about, decide how many examples you need to estimate its precision and recall to a tolerable width, then back out the total n from the base rate. If you need 20 examples of a 6 percent class, you need roughly 20 / 0.06, over 300 traces of raw sampling, or you oversample the rare class deliberately and weight afterward.

This is the moment to mention the thing I keep repeating, phrased plainly. Quality detection without an uncertainty estimate around the metric does not actually catch the failure. If you report "the judge has 0.81 recall on hallucinations" from a sample with seven hallucinations in it, you have not measured recall. You have measured noise and rounded it to two decimal places. When the question is a paired comparison instead (you changed the judge prompt and want to know whether agreement improved on the same traces), McNemar's test is the right tool. It looks only at the discordant pairs and tests whether the split between the two kinds of disagreement is significant.

4. Wilson intervals: the per-class precision number you can trust on small n

Once you report per-class precision and recall (and on an imbalanced set you must, because aggregate kappa hides the minority class), you need a confidence interval on a proportion estimated from a small count. The normal approximation (p plus or minus 1.96 times the square root of p(1-p)/n) fails exactly when you need it most: when the count is small or the proportion is near 0 or 1, it produces intervals that run below zero or above one. Wilson (1927) gives the interval I default to. It centers on a value pulled slightly toward 0.5, derives its width from the score test, stays inside [0, 1], and has far better coverage near the boundaries. For "the judge flagged 9 traces as violations and 7 were real," the Wilson interval on that 7-of-9 precision is honest. The normal-approximation interval on the same 7-of-9 is not.

5. Stratified sampling across time windows: keeping the marginals stable

Everything above assumes the calibration set's label distribution resembles production. It will not, if you sample naively, because production drifts. I learned this once: I trained a judge for factual accuracy, got kappa 0.61 on the dev set, deployed it, and three weeks later kappa on a fresh sample was 0.39. The input distribution had shifted (more domain jargon than my calibration set contained). The kappa drop was not the judge getting worse. It was the calibration set no longer describing the job. The fix is to stratify across time windows and across whatever covariate moves your marginals: pull traces in weekly strata, sample within each week proportionally, oversample any rare class so its count is high enough within each window. This keeps the marginal distribution stable week to week and lets you watch for drift, because if this week's stratified sample shows a kappa outside last week's interval, that is a drift signal rather than sampling noise.

Pasteable Python

import numpy as np from sklearn.metrics import cohen_kappa_score

1. Cohen's kappa between human labels and judge labels.

human = np.array(["pass", "fail", "pass", "pass", "fail", "pass"]) judge = np.array(["pass", "fail", "pass", "fail", "fail", "pass"]) print(f"kappa = {cohen_kappa_score(human, judge):.3f}")

2. Wilson score interval for a proportion (per-class

precision/recall on small n). def wilson_interval(successes, n, z=1.96): if n == 0: return (0.0, 1.0) phat = successes / n denom = 1 + z**2 / n center = (phat + z**2 / (2 * n)) / denom half = (z / denom) * np.sqrt(phat * (1 - phat) / n + z**2 / (4 * n**2)) return (max(0.0, center - half), min(1.0, center + half)) low, high = wilson_interval(7, 9) print(f"precision 7/9 = 0.778, Wilson 95% CI = [{low:.3f}, {high:.3f}]")

3. Bootstrap the variance (and CI) of kappa at a given n.

def bootstrap_kappa(human, judge, n_boot=2000, seed=0): rng = np.random.default_rng(seed) n = len(human) human, judge = np.asarray(human), np.asarray(judge) estimates = np.empty(n_boot) for b in range(n_boot): idx = rng.integers(0, n, size=n) # resample the pairs, not the labels estimates[b] = cohen_kappa_score(human[idx], judge[idx]) return {"kappa": cohen_kappa_score(human, judge), "std": float(np.nanstd(estimates)), "ci95": (float(np.nanpercentile(estimates, 2.5)), float(np.nanpercentile(estimates, 97.5)))} rng = np.random.default_rng(1) h = rng.integers(0, 2, size=50) flip = rng.random(50) < 0.15 j = np.where(flip, 1 - h, h) res = bootstrap_kappa(h, j) print(f"n=50 kappa={res['kappa']:.3f} std={res['std']:.3f} 95% CI=[{res['ci95'][0]:.3f}, {res['ci95'][1]:.3f}]")

Two things to read off the bootstrap: the std is your standard error on kappa at this n, and the percentile CI is the band you should be quoting. Run it once at n=50 and once at n=200 on your own labels and you will see the interval shrink. That shrinkage, scaled by your minority-class rate, is the entire sizing decision.

References

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37 to 46.
Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158), 209 to 212.
McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2), 153 to 157.

FAQ

Is kappa the right metric at all? For nominal labels with two raters, kappa is a reasonable default. If your label is ordinal (a real 1-to-5 scale where the distance between buckets matters), weighted kappa or an intraclass correlation is more appropriate. My preference is to avoid ordinal judge scales and decompose into binary criteria.

Can I just label more data instead of doing any of this math? Yes, and if labeling is cheap for you, more data is the cleanest fix. The math matters when labels are expensive, which is the usual case for the rare-and-costly category.

Why bootstrap kappa instead of a closed-form variance? There is a closed-form asymptotic variance, but it is an asymptotic result and I do not trust it at the small n where I am actually operating. The bootstrap makes no large-sample assumption and surfaces the rare-class degeneracy directly.

What kappa value is good enough to ship a judge? There is no universal threshold and I am suspicious of the ones in circulation. It depends on the cost of the judge being wrong. For a low-stakes triage filter I have shipped in the high 0.6s. For anything where a missed positive is expensive, I want a higher kappa and a tight interval on the minority class specifically.

Open questions I want to test next

The relationship between bootstrap interval width and minority-class count: I have a working intuition you can predict the n you need from the rare-class rate alone, but I have not pinned down the constant across label types. Stratification under genuine drift: holding the marginals fixed makes the comparison legitimate but may hide the very change I should detect, and I do not have a principled way to do both at once. And whether any of this transfers to multi-judge ensembles, where pairwise kappa stops being the natural object and the right tool is probably closer to an intraclass correlation. If you have labeled data sitting around, the most useful thing you can do is run the bootstrap at two sizes on your own task and see where your interval lands. The number that matters is not the kappa. It is how much the kappa moves when you resample.

why Cohen's kappa drifts week to week (and what to do about it)

Maya Andersson — Tue, 02 Jun 2026 19:25:19 +0000

If your LLM-as-judge calibration kappa moves around week to week and you cannot explain it from labeller behavior, the usual cause is the marginal distribution of your calibration set, not the labellers.

Quick refresher. Cohen's kappa is:

kappa = (Po - Pe) / (1 - Pe)

Where Po is observed agreement and Pe is expected agreement by chance. Pe depends on the marginal distribution of the labels in your set.

If 70% of last week's traces were labelled "acceptable" by labeller A and 25% "good" and 5% "bad", Pe is one number. If this week's mix is 50/40/10, Pe shifts. The labellers can be doing exactly the same thing and your kappa value moves.

Three things that help:

Sample your calibration set across multiple time windows (rolling 4-week window, stratified by time bucket). Reduces the chance that one week's traffic pattern dominates Pe.
Report per-class precision and recall alongside kappa. Kappa is one summary number; the per-class metrics tell you where the labeller-LLM disagreement actually sits.
For very small calibration sets (under 100 traces), use Wilson confidence intervals around the per-class precision instead of treating kappa as a point estimate. The Wilson interval is robust to small samples; the normal-approximation interval is not.

References for the calibration-set design and the small-sample math are in Cohen (1960) "A coefficient of agreement for nominal scales" and Wilson (1927) "Probable inference, the law of succession, and statistical inference." Both are short reads.

Your LLM-as-judge eval set is too small. Here is the math

Maya Andersson — Tue, 26 May 2026 17:49:50 +0000

How many human-labeled examples do you need to calibrate an LLM-as-judge against humans on your task? The default answer most teams use is "enough," which usually means whatever they had time to label. That answer is wrong in a specific, mathematically tractable way.

The short version: if your judge has Cohen's kappa around 0.6 against humans and you want a 95% confidence interval no wider than 0.10, you need approximately 200 paired labels. If your judge has kappa around 0.4, you need approximately 400. Most production teams I have read about are using 50, which gives a CI width of 0.20 or wider at the same kappa range.

Method

Cohen's kappa (Cohen 1960) measures inter-rater agreement adjusted for chance. The classical interpretation thresholds (Landis & Koch 1977) treat 0.40 to 0.60 as "moderate" and 0.60 to 0.80 as "good."

The variance of an estimated kappa shrinks with sample size, but slower than linearly. For a fixed true kappa, doubling N narrows the CI by roughly sqrt(2). To halve the CI width, you need 4x the data.

Here is a bootstrap-CI calculation:

import numpy as np
from sklearn.metrics import cohen_kappa_score

def kappa_with_bootstrap_ci(judge_scores, human_scores,
                            n_resamples=2000, ci=0.95):
    """Returns (point_estimate, (low, high)) bootstrap CI."""
    paired = list(zip(judge_scores, human_scores))
    n = len(paired)

    point_estimate = cohen_kappa_score(judge_scores, human_scores)

    resampled_kappas = []
    rng = np.random.default_rng(42)
    for _ in range(n_resamples):
        idx = rng.integers(0, n, size=n)
        bs_pairs = [paired[i] for i in idx]
        bs_judge = [p[0] for p in bs_pairs]
        bs_human = [p[1] for p in bs_pairs]
        resampled_kappas.append(
            cohen_kappa_score(bs_judge, bs_human)
        )

    alpha = 1 - ci
    low = np.percentile(resampled_kappas, 100 * alpha / 2)
    high = np.percentile(resampled_kappas, 100 * (1 - alpha / 2))
    return point_estimate, (low, high)

For paired comparison between two judges on the same examples, McNemar's test is the right statistic (not a re-application of kappa). The implementation:

from statsmodels.stats.contingency_tables import mcnemar

def compare_judges(judge_a_scores, judge_b_scores, human_scores):
    """Returns McNemar exact test p-value for whether judge A
    and judge B differ in their agreement-with-human rate."""
    a_correct = [a == h for a, h in zip(judge_a_scores, human_scores)]
    b_correct = [b == h for b, h in zip(judge_b_scores, human_scores)]
    # 2x2 contingency: both right, A only, B only, both wrong
    both_right = sum(a and b for a, b in zip(a_correct, b_correct))
    a_only = sum(a and not b for a, b in zip(a_correct, b_correct))
    b_only = sum(not a and b for a, b in zip(a_correct, b_correct))
    both_wrong = sum(not a and not b for a, b in zip(a_correct, b_correct))
    table = [[both_right, a_only], [b_only, both_wrong]]
    return mcnemar(table, exact=True).pvalue

The bounded sample size problem

The CI width is the quantity that determines whether a kappa estimate is operationally useful. A point estimate of 0.65 with CI [0.45, 0.85] gives almost no information. A point estimate of 0.65 with CI [0.60, 0.70] tells you the judge is reliably "good."

For production drift detection, you need CIs tight enough that drift is distinguishable from sampling noise. CI width below 0.10 detects 0.10-point drops reliably; CI width 0.20 does not.

True kappa	N for CI width 0.10	N for CI width 0.20
0.3	approximately 450	approximately 115
0.5	approximately 250	approximately 65
0.7	approximately 150	approximately 40
0.9	approximately 50	approximately 15

These are Monte Carlo estimates, not closed-form derivations. The exact formula (Fleiss 1981) involves prevalence and bias terms.

What N to actually use

def recommend_n(target_kappa: float,
                target_ci_width: float = 0.1) -> int:
    """Lookup from Monte Carlo simulation; not a closed form."""
    if target_kappa >= 0.85:
        return max(50, int(40 / target_ci_width**2 * 0.5))
    elif target_kappa >= 0.65:
        return max(150, int(40 / target_ci_width**2 * 1.5))
    elif target_kappa >= 0.45:
        return max(250, int(40 / target_ci_width**2 * 2.5))
    else:
        return max(450, int(40 / target_ci_width**2 * 4.5))

If you do not know your judge's kappa yet, start with N=200 for initial calibration. Re-estimate the required N based on observed kappa and label more if you came in low.

Three production judges, three decisions

Judge A (refund agent factual accuracy). Initial N=200. Observed kappa 0.61 [CI 0.54, 0.68]. After 3 weeks in production, kappa on a fresh 200-example sample dropped to 0.39 [CI 0.30, 0.48]. Distribution shift on the input. The drop was detectable because both CIs were tight.

Judge B (customer-support tone scoring). Initial N=200, observed kappa 0.72 [CI 0.67, 0.78]. Stable across two months.

Judge C (code-review quality scoring). Initial N=200, observed kappa 0.31 [CI 0.22, 0.40]. Too low to use. Reverted to human-only review.

If I had used N=50, two of three decisions would have been ambiguous.

Limitations

Kappa is a single-criterion metric. Production judges often score multiple criteria; per-criterion kappa with separate CIs is the right approach.

Prevalence affects kappa variance. Stratified sampling helps. My Monte Carlo assumes balanced classes.

The bootstrap CI is approximate. For N less than 50, use Fleiss's closed form, or accept that you do not have enough data.

This is about agreement, not validity. A judge can have high kappa with humans who are themselves wrong. Sara Hooker's writing on benchmark validity is the relevant prior.

Open questions

The relationship between calibration set size and drift-detection sensitivity for production traces. My working hypothesis is sensitivity tracks 1 over sqrt(N), but I have not derived this formally.

The right cadence for re-labeling. Weekly works in practice; the closed-form relationship between re-labeling cadence and model-update cadence I have not seen written down.

Cross-judge agreement as a partial substitute for human labels. The published literature is thin. Farquhar et al. 2024 is close but is about hallucination detection, not judge calibration. Zheng et al. (LMSYS) hints at this direction but does not run the experiment systematically. If anyone has a citation, I would appreciate it.

The implication for benchmark validity. Most published LLM-as-judge benchmarks report kappa point estimates with sample sizes below what is required to detect 0.05 to 0.10-point differences between judges. The published rankings may be within sampling noise. The literature on this is not yet settled.