Your eval score moved from 82 to 79 overnight. Nobody changed the prompt. Nobody changed the model. You re-ran it and got 84. So you shipped, because 84 is up and to the right, and the standup was in ten minutes.
That is not an evaluation. That is a coin flip wearing a lab coat. If a unit test passed, failed, and passed again on the same code, you'd call it flaky and either fix it or quarantine it — you would never let it gate a release. LLM evals behave exactly like that flaky suite, and most teams read the noise as signal anyway.
TL;DR — An LLM eval is a test suite over a non-deterministic system graded by another non-deterministic system, on a set that quietly changes. Its real failure modes are the ones every flaky suite has: uncontrolled seeds, a drifting judge, a leaking or mutating dataset, and point scores reported without variance. Fix them the boring way — pin what you can, calibrate the judge, freeze and version the set, report distributions, gate on overlap — and "the model got worse" mostly turns back into "the measurement was noisy."
The score jitters because everything under it is non-deterministic
There are at least three independent random processes stacked in a single eval run. The model samples tokens with temperature. The judge — if you use LLM-as-judge — samples its verdict the same way. And the eval set you're scoring against is usually small, so a handful of items flipping swings the headline number by several points.
Stack three noise sources and you get a metric that moves on its own. Treating a 3-point delta as a regression, when your run-to-run standard deviation is 4 points, is the eval equivalent of chasing a race condition by adding print statements: you're reacting to jitter, not to change.
Pin the seeds you can; budget for the ones you can't
The first move with any flaky test is to remove the removable randomness. Same here. For scoring evals — factual accuracy, format compliance, pass/fail on a rubric — run the model at temperature = 0 and pin every sampling parameter you're allowed to pin. You will not get true determinism (kernels, batching, and hardware still wobble the logits), but you cut the variance hard.
For anything where you actually care about the spread — creativity, refusal behavior, robustness — do the opposite deliberately: fix a set of seeds and run each item across all of them. The point is that randomness is either controlled or measured. What you can't do is leave it uncontrolled and then read the average as if it were stable.
The LLM judge is a flaky dependency, so calibrate it
The moment your grader is itself a model, you've added a second system-under-test and started trusting its output without a test of its own. Judges drift across model versions, contradict themselves on re-run, and carry position and verbosity biases — the same answer scores differently depending on whether it was labeled A or B.
Treat the judge like any dependency you don't control: hold it to a golden set. Assemble a few dozen items you've hand-labeled, measure the judge's agreement with those human labels, and track that agreement number over time. If the judge only agrees with humans 70% of the time, your eval has a ±30% error bar baked in before the model under test says a single word. Calibrate the ruler before you measure the thing.
Version the eval set like source code, because it is
Here's the silent killer: someone "improves" the eval set. They fix a typo in a gold answer, drop three items that "weren't fair," add ten new hard cases. Now this week's 86 and last week's 81 are scored against different exams, and every comparison you make across that boundary is meaningless. That's not a better eval — it's a broken git history for your ground truth.
Eval data is code. Put it under version control, review changes to it in PRs, and stamp every result with the dataset hash it ran against. Never compare two scores that ran against different hashes without saying so out loud. And guard against the other direction — leakage — where eval items drift into training data or few-shot examples and the score climbs because the model has seen the answer key. A number that only goes up is often a number that's been contaminated.
Report the distribution, not the point score
A single number — "84" — throws away the one thing you needed: how much it moves when nothing changes. Run each eval N times (5 is enough to start), and report the mean with a spread — standard deviation or a confidence interval. That interval is your noise floor. It tells you the smallest change you're actually able to detect.
Once you have that, regressions get a real definition. A new prompt scoring 84 ± 3 against a baseline of 82 ± 3 has not improved anything — the intervals overlap; it's the same system. A drop to 74 ± 3 against 82 ± 3 is a genuine regression, because the intervals are clean apart. This is the same discipline as flaky-test triage: you don't chase a failure until you've shown it reproduces outside the noise band.
Gate merges on evals like tests, and quarantine the noisy ones
Evals only protect you if they run in CI and can actually block a merge. Wire them to the pipeline, set thresholds in terms of the confidence interval — "must not regress by more than the noise floor" — and fail the build when a change crosses that line. A dashboard nobody's merge depends on is a decoration, not a safety net.
And do the last flaky-test move: quarantine. When an eval is too noisy to gate on, don't average it into the headline number where it hides real movement — pull it into a watch list, tighten it, and only promote it back to gating once its variance is under control. Averaging a noisy eval into a clean one doesn't cancel the noise; it launders it into a number you'll trust by mistake.
The reframe
Stop treating a moved eval score as a verdict about the model's intelligence. It's a measurement, produced by a noisy pipeline you built, and measurements have error bars whether or not you print them. Almost everything that makes LLM evals feel unknowable is a discipline software testing already solved: control the randomness, calibrate the grader, version the fixtures, report the variance, gate on the signal.
Do that, and most "the model regressed" incidents resolve into "the measurement was noisy and we finally noticed." Your model is probably fine. Go fix the test suite around it.
Top comments (0)