I Ran LLM-as-Judge 50 Times on the Same Output. It Failed More Than I'd Like to Admit.

Here's what a flaky eval looks like in practice.

I had a faithfulness gate on an AI agent merge. Every proposed change got scored by an LLM judge — 0.0 to 1.0, with 0.80 as the threshold. Clean setup. The kind of thing you see in a framework README and think "finally, someone put guardrails on this."

Then I looked at the distribution.

Same output, same input, same judge model. The scores landed everywhere from 0.62 to 0.91 across 50 runs. That output passed the gate about 68% of the time. Not because it changed — it never changed — but because the judge had a mood.

This is the eval problem nobody talks about honestly.

The Mechanic Nobody Explains

LLM-as-judge sounds simple: give a model a rubric, ask it to score, trust the number. Except LLMs don't work likerubric-checking machines. They work like very fluent pattern matchers with temperature and context and recency bias baked in.

When you ask an LLM to judge its own output (or output from a system that shares its training data), you're not getting an objective score. You're getting a probabilistic sample from a very wide distribution.

Three things cause the most noise:

Prompt sensitivity. Small wording changes in the judge prompt flip scores by 0.10-0.15. "Rate the following response on a scale of 1-10" gets different distributions than "Evaluate this response's quality from 0 to 1." The rubric you write isn't what the model reads.

Position preference. LLMs have documented ordering biases. When comparing two options, they tend to favor the first or the last depending on length and format. If your eval pairs outputs for comparison, the order leaks into the score.

Confidence instability. The same model asked the same question five times at temperature=0.1 still returns meaningfully different scores. Temperature isn't a dial from "random" to "deterministic" — it's a dial from "more random" to "less random."

What Actually Changes the Scores

I ran a structured experiment. Same 100 test cases, same judge model (GPT-4o), four different eval setups:

Setup	Score Range (p10–p90)	Pass Rate @ 0.80
Raw prompt, temp=0.1	0.61 – 0.89	71%
Few-shot examples (3)	0.69 – 0.84	79%
CoT + few-shot	0.73 – 0.86	83%
Ensemble (3 judges, median)	0.75 – 0.82	87%

The raw prompt setup had an 0.28 spread from p10 to p90. The ensemble compressed that to 0.07. That difference is whether your gate is a narrow corridor or a swinging door.

The few-shot improvement alone was the biggest single lever. Show the judge three examples with explicit reasoning and the score settles. Without them, the model is improvising its rubric interpretation on every call.

Chain-of-thought inside the judge prompt adds another 4-5 percentage points on pass-rate reliability. It forces the model to externalize its reasoning instead of jumping to a gut-feel number and reverse-justifying it.

The Code That Actually Works

Here's the eval setup I landed on after the experiment. Three judges, structured output, median aggregation.

import json
from openai import OpenAI

client = OpenAI()

JUDGE_SYSTEM_PROMPT = """You are an expert code reviewer evaluating AI agent outputs.
For each submission, you MUST output a JSON object with this exact structure:
{"score": float, "reasoning": string, "concerns": list[string]}

SCORING CRITERIA:
- 0.90–1.00: Production ready, no concerns
- 0.75–0.89: Minor issues, acceptable with changes
- 0.60–0.74: Significant issues, needs revision
- Below 0.60: Major problems, do not merge

Evaluate based on: correctness, safety, edge case handling, and code quality.
"""

FEWSHOT_EXAMPLES = [
    {
        "role": "user",
        "content": "Agent proposed: fn merge(arr: &[i32]) -> Vec<i32> { arr.to_vec() } on input [3,1,2]"
    },
    {
        "role": "assistant",
        "content": json.dumps({
            "score": 0.95,
            "reasoning": "Correctly implements sorting with proper ownership. Handles the single input gracefully.",
            "concerns": []
        })
    },
    {
        "role": "user",
        "content": "Agent proposed: fn merge(a: Option<i32>) -> i32 { a.unwrap() } on input None"
    },
    {
        "role": "assistant",
        "content": json.dumps({
            "score": 0.35,
            "reasoning": "Unwrap on None will panic. Major correctness issue.",
            "concerns": ["panic on None input", "no error handling"]
        })
    }
]

def judge_output(proposal: str, context: str, model: str = "gpt-4o") -> dict:
    messages = [
        {"role": "system", "content": JUDGE_SYSTEM_PROMPT},
        *FEWSHOT_EXAMPLES,
        {"role": "user", "content": f"Context: {context}\n\nAgent proposal: {proposal}"}
    ]

    response = client.chat.completions.create(
        model=model,
        messages=messages,
        temperature=0.1,
        response_format={"type": "json_object"}
    )

    return json.loads(response.choices[0].message.content)

def ensemble_judge(proposal: str, context: str, n_judges: int = 3) -> dict:
    results = [judge_output(proposal, context) for _ in range(n_judges)]
    scores = sorted([r["score"] for r in results])
    median_score = scores[len(scores) // 2]

    # Collect concerns across all judges
    all_concerns = set()
    for r in results:
        all_concerns.update(r.get("concerns", []))

    return {
        "score": median_score,
        "individual_scores": scores,
        "concerns": list(all_concerns),
        "passed": median_score >= 0.80
    }

The three-judge ensemble is the highest-leverage line in this whole setup. It's not elegant. It costs 3x the inference. But it converts your eval from "noisy signal" to "reliable gate."

The Meta-Problem

Here's what keeps me up at night: the judge is also an LLM. It has the same failure modes as the system it's evaluating. It hallucinates correctness. It overfits to style. It sometimes scores a completely broken output at 0.87 because it sounds confident.

You can't escape this loop by adding more LLM. You can only manage it.

The practical answer is: treat your eval like a flaky test suite. Run it multiple times. Track your false positive rate. Calibrate your threshold against human judgment on a sample set. And when the eval and the human disagree — which will happen — trust the human, then figure out why the eval failed, then fix the eval.

No eval is objective. The moment you think yours is, it starts costing you.

What I Learned

Running 50 eval iterations on the same input taught me three things I should have known going in:

1. Single-shot LLM judgment is not a reliable gate. It's a useful signal, not a final verdict. Treat it like a first-pass code review, not a CI check.

2. Few-shot examples are not optional. Without them, the judge improvises. With them, the variance drops significantly. Invest the time to build a good example set.

3. Temperature 0 doesn't mean stable. If you're running repeated evals, keep temperature above 0 but low (0.1-0.2) and average multiple runs. Determinism is an illusion at the inference layer.

LLM-as-judge is a genuinely useful pattern. But "useful" and "reliable" are different things, and confusing them is how you end up with production systems that pass their own tests and fail in the real world.