LLM-as-judge disagrees with itself between runs

The flap

I had a faithfulness gate on merge: judge scores every case, the mean has to clear 0.80. One Tuesday it failed at 0.79. I re-ran the identical job, no code change, no prompt change, and it passed at 0.82. Ran it a third time: 0.80 exactly. Nothing in the repo had moved. The judge was disagreeing with itself.

A gate that returns a different verdict on the same inputs is worse than no gate. People stop believing the red, they re-run until it goes green, and now the check is a slot machine you pull until it pays out. The regression it was supposed to catch could sail through on the lucky pull. So before I trusted that gate again I had to make the judge reproducible enough to stand on.

Where the jitter comes from

Four sources, in the order they bit me.

Sampling temperature. A judge call is a generation. If temperature is above zero the model samples, and a borderline case lands on 4/5 one time and 3/5 the next. This is the biggest lever and the easiest to miss because most SDK defaults are not zero.

Model version drift. "gpt-4o" or "claude-latest" is a moving alias. The provider ships a new snapshot, your scores shift a few points overnight, and you blame your prompt. Pin the dated snapshot, not the floating name.

Prompt ambiguity. If your rubric says "rate helpfulness 1 to 5" without anchoring what a 3 versus a 4 means, the model resolves the ambiguity differently each call. Vague rubrics convert directly into variance.

Tie-breaking. When the judge is genuinely on the fence between two scores, tiny sampling noise decides, and that decision is exactly where your threshold tends to sit.

Make it reproducible enough to gate

You will not get bit-identical determinism from a hosted model. That is fine. The goal is not zero noise, it is noise small enough that a threshold crossing means a real change and not a coin flip. Five things got me there.

Temperature 0, and a seed where the provider supports one. This alone collapsed most of my flap. Seeds help further on providers that honor them, but do not assume a seed gives you exact reproducibility across a model update.

Pin the exact judge model and prompt version in the cache key. Same discipline as any eval cache: the score is only reusable if the input, the judge snapshot, and the rubric version all match. Bump the version string whenever you touch the rubric.

Average over k judged samples, or take majority vote. One call is a sample from a distribution. k calls and a mean (or a vote for pass/fail rubrics) shrink the variance of your estimate by roughly sqrt(k). I run k=5.

Quantize the score. If you gate on a continuous 0 to 1, every hundredth flaps. Round to a coarse grid (0.0, 0.25, 0.5, 0.75, 1.0) per case so sub-grid noise stops moving the aggregate.

Version the judge prompt as code. The rubric lives in the repo, gets a version string, and changes go through review. A judge prompt edited in a UI and not tracked is a silent score change you cannot bisect.

The gate that respects the noise band

The real fix is conceptual: stop treating one judged score as ground truth. Judge k times, keep the mean and the spread, and only fail when the mean is below the threshold by more than the noise you actually measured. If the mean sits inside the noise band around the threshold, that is not a regression, it is jitter, and failing on it is how you get a flapping gate.

import statistics
from typing import Callable

def stable_judge_score(
judge: Callable[[str, str], float],
output: str,
reference: str,
k: int = 5,
quantize_to: float = 0.25,
) -> tuple[float, float]:
"""Run the judge k times at temperature 0. Return (mean, stdev),
each raw score snapped to a coarse grid to kill sub-grid jitter."""
scores = []
for _ in range(k):
raw = judge(output, reference) # judge must be called at temperature 0
snapped = round(raw / quantize_to) * quantize_to
scores.append(snapped)
mean = statistics.fmean(scores)
stdev = statistics.pstdev(scores) if k > 1 else 0.0
return mean, stdev

def gate(mean: float, stdev: float, threshold: float = 0.80) -> bool:
"""Fail only when the mean is below threshold by more than the
observed noise. Inside the noise band counts as pass, not a flap."""
return mean >= (threshold - stdev)

if name == "main":
# toy judge: deterministic here, real one hits an LLM at temperature 0
def fake_judge(out: str, ref: str) -> float:
return 0.79 if "borderline" in out else 0.9

mean, stdev = stable_judge_score(fake_judge, "a borderline answer", "ref", k=5)
passed = gate(mean, stdev, threshold=0.80)
print(f"mean={mean:.3f} stdev={stdev:.3f} pass={passed}")
raise SystemExit(0 if passed else 1)   # this exit code is what CI reads

The raise SystemExit is the load-bearing line. That is what makes the branch protection rule refuse a real regression. Everything above it exists so that exit code means something. On my suite, moving from one raw judge call to k=5 with quantization took the run-to-run swing on that faithfulness metric from about 0.03 down to under 0.01, which was finally tight enough that a red meant a real drop and people stopped re-running to dodge it.

One caution on k: more samples cost more judge calls and more wall-clock, so I only spend the k on the cases near the threshold, and run the obviously-passing and obviously-failing cases once. The noise only matters where the decision is close.

What I'd check first

• Log temperature on the judge call. If it is not zero, nothing else you do about jitter matters until it is.
• Diff the judge model string between the run that passed and the run that failed. A floating alias silently swapped a snapshot on you more often than you would think.
• Measure the run-to-run stdev of your gated metric before you trust the gate. If the swing is wider than the margin your threshold sits on, you are gating on noise and the red is meaningless.