- Book: LLM Observability Pocket Guide: Picking the Right Tracing & Evals Tools for Your Team
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
Your golden eval set was good in March. It's December now. Half the queries you see in production don't look like anything in the eval set. The dashboard still shows 98% pass, and that number is a lie, because the test set you're measuring against stopped resembling the workload months ago.
This is eval drift. It's the quiet version of model regression: nothing breaks, the green checks keep landing, and you're flying blind.
How eval sets go stale
Three things move underneath you, usually at the same time.
Domain shift. New product surface, new user segment, new integration. A support-bot eval set built around billing and login questions in Q1 has nothing to say about the refund-automation workflow you launched in Q3. The eval set didn't get worse; the world it covers shrank.
Distribution shift. Same domain, different mix. Maybe enterprise traffic went from 10% to 45% of all queries because sales had a great quarter. Enterprise queries are longer, more multi-turn, and reference internal jargon. Your eval set is still 90% short single-turn questions because that's what March looked like.
Feature drift. The features you embedded against changed. You upgraded the embedding model. You switched chunking strategies in retrieval. You changed the system prompt and now responses condition on different context. The eval inputs are the same strings, but the vector representations (the things your retrieval and reranking actually score against) have shifted.
The first kind you usually notice (someone files a ticket). The second and third are the dangerous ones. Pass rate stays flat. User satisfaction craters. The two metrics tell different stories because they're measuring different populations.
A 60-line drift detector
You don't need a research-grade implementation. You need a number that goes up when production stops looking like your eval set. Maximum Mean Discrepancy (MMD) on embeddings is the right tool. It compares two distributions of vectors without assuming either is Gaussian, and the squared statistic has a clean interpretation: zero when the two samples come from the same distribution, larger when they don't.
Here's a working detector. Sentence-Transformers for embeddings, RBF kernel, ~60 lines:
# eval_drift.py
import numpy as np
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import rbf_kernel
MODEL = SentenceTransformer("all-MiniLM-L6-v2")
def embed(texts: list[str]) -> np.ndarray:
# normalize so kernel bandwidth is well-behaved across batches
return MODEL.encode(texts, normalize_embeddings=True, batch_size=64)
def mmd_rbf(x: np.ndarray, y: np.ndarray, gamma: float = 1.0) -> float:
# unbiased MMD^2 with an RBF kernel. zero = same distribution.
kxx = rbf_kernel(x, x, gamma=gamma)
kyy = rbf_kernel(y, y, gamma=gamma)
kxy = rbf_kernel(x, y, gamma=gamma)
n, m = len(x), len(y)
# drop diagonal terms for the unbiased estimator
np.fill_diagonal(kxx, 0); np.fill_diagonal(kyy, 0)
return (kxx.sum() / (n * (n - 1))
+ kyy.sum() / (m * (m - 1))
- 2 * kxy.mean())
def permutation_pvalue(x, y, gamma=1.0, n_perm=200) -> float:
# how often does a random split give an MMD this large?
pooled = np.vstack([x, y])
observed = mmd_rbf(x, y, gamma)
n = len(x)
count = 0
rng = np.random.default_rng(42)
for _ in range(n_perm):
rng.shuffle(pooled)
if mmd_rbf(pooled[:n], pooled[n:], gamma) >= observed:
count += 1
return (count + 1) / (n_perm + 1)
def detect_drift(eval_queries: list[str], prod_queries: list[str]):
x = embed(eval_queries)
y = embed(prod_queries)
# median heuristic for kernel bandwidth — works for normalized embeddings
from scipy.spatial.distance import pdist
sigma = np.median(pdist(np.vstack([x, y]))) or 1.0
gamma = 1.0 / (2 * sigma ** 2)
mmd2 = mmd_rbf(x, y, gamma)
pval = permutation_pvalue(x, y, gamma, n_perm=200)
return {"mmd2": float(mmd2), "p_value": float(pval), "n_eval": len(x), "n_prod": len(y)}
if __name__ == "__main__":
import json, sys
eval_q = json.load(open(sys.argv[1])) # golden set queries
prod_q = json.load(open(sys.argv[2])) # last 7 days of sampled prod queries
print(json.dumps(detect_drift(eval_q, prod_q), indent=2))
Sample run on a real-ish dataset:
$ python eval_drift.py golden.json prod_week.json
{
"mmd2": 0.0382,
"p_value": 0.005,
"n_eval": 500,
"n_prod": 1200
}
p-value below 0.05 with a non-trivial MMD² means the two distributions are statistically distinguishable. That's your signal: the golden set is no longer a fair sample of production. If you'd rather avoid kernels, the alternative is binning embedding coordinates and computing KL divergence on the histograms. Cheaper, noisier, easier to explain to a PM. MMD is the better default.
Two practical numbers to calibrate against. Run the detector on two random splits of your golden set. That gives you the noise floor: MMD² of roughly the same distribution. Drift becomes interesting when production-vs-eval is 3-5x the noise floor, not when it crosses some arbitrary 0.01 threshold.
Slice-level drift: surface where it moved
A single global MMD² number tells you something drifted. It doesn't tell you what. If your assistant handles billing, onboarding, refunds, API debugging, and password resets, the global number can be high because one slice moved, while the rest is fine. Refresh the whole eval set on that signal and you've over-corrected four slices to fix one.
Run the detector per slice. The cheapest way: tag every eval query and every sampled prod query with its intent, then loop:
slices = {}
for slice_name in {q["intent"] for q in prod_queries}:
eval_slice = [q["text"] for q in eval_queries if q["intent"] == slice_name]
prod_slice = [q["text"] for q in prod_queries if q["intent"] == slice_name]
if len(eval_slice) < 30 or len(prod_slice) < 30:
slices[slice_name] = {"status": "insufficient_samples"}
continue
slices[slice_name] = detect_drift(eval_slice, prod_slice)
Rank slices by MMD². The top of the list is your refresh target. You'll often see a pattern: one slice is at MMD² 0.08 with p=0.001, another at 0.04 with p=0.02, the rest sit at the noise floor. Refresh the top two. Leave the rest alone.
A second slice axis worth running: query length buckets (short/medium/long) and user segment (free/pro/enterprise). Drift often shows up there even when intent distribution looks stable. Long enterprise queries with retrieval-grounded answers are a different beast than short free-tier "how do I reset my password" lookups, and your eval set probably under-samples the former.
The slice with no production samples is also a signal, just the opposite one. If your eval set has a "legacy_v1_api" slice and prod hasn't sent one of those queries in three months, that slice is dead weight. Archive it; don't keep scoring against it.
The refresh protocol
Monthly. First Monday of the month. Block 90 minutes on the calendar.
The steps that work:
- Sample 500-2000 production queries from the last 30 days. Stratified by slice if you can, uniform random if you can't. Strip PII before anything else touches them.
- Run the drift detector globally and per slice. Save the numbers; you're building a trend line, not a one-shot signal.
- For each slice with MMD² > 3x the noise floor and p < 0.05, sample 30-50 new queries from that slice's production sample. Have a human (or a strong LLM as first pass + human review) write expected outputs.
- Append to the golden set; don't replace. New rows tagged with the month added and the slice. Old rows stay.
- Re-run your full eval suite on the appended set. Save the score. This becomes your new baseline going forward.
The append-don't-replace rule is the one that keeps the historical trend line readable. If you swap out 40% of the eval set each month, your "pass rate" graph is comparing different rulers each time. You can't tell whether a 2-point drop is the model getting worse or the eval set getting harder. Append, version, run the new model against both the old and new partitions, and you can decompose the change.
When to rewrite vs incrementally extend
Most months you append. Twice a year, maybe, you rewrite a slice from scratch. Rewrite when:
- Domain shifted hard enough that the expected outputs in the old slice are wrong, not just unrepresentative. If your refund policy changed, last March's golden answers are now incorrect, so don't keep scoring against them.
- You changed the model family (3.5 → 4 → 5, or moved to a different vendor) and the response style is structurally different. Old gold-standard outputs were written for a model with different verbosity defaults; the judge prompt scores them unfairly.
- The slice's MMD² has been creeping for three consecutive months despite incremental refreshes. That's a sign the slice definition itself has fragmented and needs to be split.
Even on a rewrite, keep the old slice frozen as slice_v1_archived. Re-run it quarterly. It's your regression canary. If slice_v1_archived suddenly tanks, something fundamental shifted in the model or the prompts that the new eval set didn't catch yet.
The gotcha: over-refreshing destroys baseline comparability
The biggest mistake teams make once they start measuring drift: they over-correct. Drift detected → full eval refresh → new baseline → forget the old one. Three months in, leadership asks "are we better than we were in Q2?" and nobody can answer because every Q2 number was measured against an eval set that no longer exists.
The fix is boring and works. Version the eval set (golden_2026_03, golden_2026_04, …). For every model release, run against the current month's set and the set from 6 months ago. Two numbers per release. The current number tells you about today's traffic. The 6-month-old number tells you whether you regressed against the baseline you originally validated against. If both go up, you're winning. If the current goes up and the old goes down, you're optimizing for new traffic at the cost of old, which is sometimes correct, but should be a conscious tradeoff, not an accident.
The other temptation: making the drift threshold too tight, so it fires every week. The detector's job is to flag meaningful distribution change, not noise. Calibrate against the noise floor of two random eval splits. Set the alert threshold at 3-5x that. Anything tighter and you'll refresh into instability, where each month's eval set is a slightly different population and no trend line means anything.
A stale golden set fails silently: green dashboards over a workload nobody validated. A well-refreshed one tells you the truth every month, and that's worth the 90 minutes.
Pick a slice this week. Sample 200 prod queries, run the detector, write down the MMD² and the noise floor from two random eval splits. You'll know within an afternoon whether you're flying with eyes open or measuring against a fossil.
What's the longest you've gone between eval-set refreshes, and what finally forced the rewrite?
If this was useful
The drift-detection workflow above is one piece of a larger eval system — sampling strategy, judge prompts, slice taxonomy, dashboard wiring. The Eval Strategies chapter of the LLM Observability Pocket Guide walks through the full pipeline, including how to pick between offline batch evals, online A/B, and continuous shadow scoring. If you're standing up evals from scratch or trying to make an existing setup actually trustworthy, it's the chapter to read first.
Top comments (0)