Who Grades the Grader? Your LLM Judge Is an Unvalidated Model in Production

Everybody's eval stack has the same load-bearing assumption nobody audits: that the model-as-judge is telling the truth.

You wrote deterministic checks for the easy stuff — schema valid, no PII, latency under budget. Then you hit the subjective stuff — "is this answer actually helpful," "did the agent follow the user's intent," "is this summary faithful to the source" — and you reached for an LLM judge, because what else are you going to do. Now a model grades your model. And here's the part that should keep you up at night: you never validated the grader. You're shipping or blocking releases based on a 0–10 score from a prompt you wrote in twenty minutes, and you have no idea if that score correlates with anything a human would agree with.

I've watched teams trust a green judge dashboard for months, then discover the judge was handing out 8s to answers users hated. The judge wasn't broken in an obvious way. It was just uncalibrated, and uncalibrated graders fail silently — which is the worst way to fail.

The judge is a model in production, so treat it like one

Say it plainly: your LLM judge is a non-deterministic model making consequential decisions in your release pipeline. That is the exact thing you spent the last year learning to distrust. Somehow when it's wearing a lab coat and called an "evaluator," people grant it authority they'd never give the agent itself.

Three ways judges quietly lie:

• Position bias. Swap the order of two candidate answers and the judge changes its winner. If A-vs-B and B-vs-A disagree more than ~10% of the time, your pairwise scores are partly coin flips.
• Verbosity bias. Longer, more confident answers score higher regardless of correctness. Your judge is grading prose, not truth.
• Self-preference. A judge from the same model family as the agent rates that family's outputs higher. If GPT grades GPT, you've got a conflict of interest with a number attached.

None of these show up on a dashboard that only plots the average score. They show up when you go looking — and most teams never look, because the judge produces a clean metric and clean metrics feel like ground truth.

Calibrate the judge against humans, then keep checking

The fix isn't "stop using LLM judges." They're genuinely useful and you can't human-label every run. The fix is to treat the judge as a system under test with its own ground-truth set. You need a labeled golden set — a few hundred examples scored by humans you trust — and you measure your judge's agreement with those humans. Cohen's kappa, not raw accuracy, because raw agreement is inflated when most answers are "fine."

Here's the calibration check I run before any judge is allowed to gate anything:

import { judge } from "./llm-judge";

type Labeled = { input: string; output: string; humanScore: number };

// Quadratic-weighted agreement: penalize big disagreements more than small ones.
function weightedAgreement(human: number[], model: number[], max = 10): number {
  let num = 0, den = 0;
  for (let i = 0; i < human.length; i++) {
    const w = ((human[i] - model[i]) ** 2) / (max ** 2);
    num += 1 - w;
    den += 1;
  }
  return num / den; // 1.0 = perfect, lower = drifting from humans
}

// Position-bias probe: judge must agree with itself when we flip the order.
async function positionBias(pairs: { a: string; b: string }[]): Promise<number> {
  let flips = 0;
  for (const { a, b } of pairs) {
    const fwd = await judge.compare(a, b);   // "a" | "b"
    const rev = await judge.compare(b, a);   // "a" | "b" (b is now first)
    const consistent = (fwd === "a" && rev === "b") || (fwd === "b" && rev === "a");
    if (!consistent) flips++;
  }
  return flips / pairs.length; // want this near 0
}

export async function certifyJudge(golden: Labeled[]) {
  const scored = await Promise.all(
    golden.map(async (g) => (await judge.score(g.input, g.output)).value),
  );
  const agreement = weightedAgreement(golden.map((g) => g.humanScore), scored);
  const bias = await positionBias(buildPairs(golden));

  const passed = agreement >= 0.85 && bias <= 0.1;
  if (!passed) {
    throw new Error(
      `Judge not certified: agreement=${agreement.toFixed(2)} (need >=0.85), ` +
      `positionBias=${bias.toFixed(2)} (need <=0.10). Do not gate releases with this judge.`,
    );
  }
  return { agreement, bias };
}

This runs in CI on a schedule, not just once. Judges drift the same way agents do — provider updates the underlying model, your prompt template gets edited, your data distribution shifts — and a judge that agreed with humans in March can quietly diverge by June. If you only calibrated once at the start, you don't have a calibrated judge; you have a historical artifact.

Calibration tells you that it's wrong. Traces tell you why.

Here's where the two halves of the workflow lock together, because a kappa of 0.6 is a smoke alarm, not a diagnosis.

agent-eval is what runs the scoring and the gate — it's the layer holding your deterministic checks, your model-as-judge, the golden set, and the certifyJudge step above. It's the thing that tells you the judge agreement dropped below 0.85 and refuses to let the release through. That's the signal. But a failing number with no context is just an argument waiting to happen — "the judge is wrong," "no, the agent regressed," and nobody can settle it.

That's the job of AgentLens: it captures the full trace behind every score — the exact prompt the judge saw, the candidate output, the resolved rubric, the judge's raw completion before you parsed a number out of it, and the agent's own tool-and-model steps that produced the answer in the first place. So when agent-eval flags that the judge handed a 9 to an answer humans scored 3, you open the AgentLens trace and see it: the judge rewarded a confident, verbose response that never grounded its central claim. Now it's not a vibe. You can see the verbosity bias in the raw text, fix the rubric to demand citations, and re-certify.

That's the loop. agent-eval scores and gates; AgentLens shows the trace so the score is debuggable. Without the trace, a bad judge score is unfalsifiable — you can't tell a judge problem from an agent problem, so you end up trusting the number you should be interrogating. With it, every disagreement between judge and human becomes a concrete, inspectable artifact instead of a meeting.

The uncomfortable takeaway

If you're using a model-as-judge and you can't state your judge's agreement with human labels as a number, you are not running evals. You're running a vibe check with extra steps and a false sense of rigor. The judge is the most trusted, least audited component in your entire pipeline — and "the LLM said it was good" is doing a lot of unexamined work in your release decisions.

Certify the judge. Re-certify on a schedule. Keep the traces so every score can be challenged. A grader you haven't validated isn't measuring quality — it's laundering an opinion into a metric, and your green dashboard is the receipt.

Comments

[Alex Shev]

The model-as-judge problem gets worse when teams give it dashboard authority too early. I like separating three things: deterministic gates for facts, judge disagreement for gray areas, and human spot checks on the judge itself. If the judge cannot be audited as a production dependency, it is just a confident opinion generator with metrics around it.