Stop Judging Every Run: Eval Sampling Is a Budget Decision, Not a Coverage One

There's a slide in every "LLM eval platform" pitch deck that says: score every response, catch every regression. It sounds responsible. It's also the fastest way to set a five-figure monthly bill on fire while measuring the wrong thing.

The mistake is treating "score every run" as a coverage requirement. It isn't. Coverage and sampling are two different axes, and conflating them is why teams end up paying a model to re-read 100% of their production traffic to produce a 7/10 that nobody reads.

Let me make the actual argument.

Evidence has an independence axis, not a cost axis

Before you can decide what to sample, you have to decide what kind of evidence you're collecting. This is where most eval thinking goes wrong: it ranks checks by cost (cheap regex vs. expensive GPT call) instead of by independence — how hard it is for the agent under test to forge the result.

That's the core doctrine behind agent-eval, and it splits into three tiers:

• Tier 1 — externally observable proof the agent can't forge. Did the output parse as valid JSON? Does the file it claims to have written actually exist on disk? Did the code compile? Did the tests pass? Did the run finish inside the timeout? Is the result non-empty? None of this is an opinion. The world either agrees or it doesn't.
• Tier 2 — statistical signal against a baseline the agent didn't author. Embedding similarity between the output and the original task. Length and repetition distributions. Did the diff actually change anything, or did the agent hand you back the input? The agent doesn't get to write the baseline, so it can't game the comparison.
• Tier 3 — model-as-judge. A shared-substrate opinion. Useful, but it's a signal, never a verdict.

Here's the part that reorganizes your whole pipeline: Tier 1 and Tier 2 are the real-time gate. They're deterministic, effectively free, and fast enough to sit in the hot path and block a bad run before it ships. Tier 3 is the opposite — metered, slow, non-deterministic — so it is offline-only by construction. You cannot put a model judge in the critical path without either lying about your latency or lying about your cost.

And there's a deeper reason Tier 3 stays offline: you can run Tier 1+2 over the agent's trajectory — its reasoning, its intermediate tool calls — but you cannot let a model judge that trajectory. A model grading another model's reasoning is circular: judge and judged share a substrate, so there's no independent ground truth. The judge may only inspect artifacts the judged agent didn't get to write.

Now the sampling decision writes itself

Once you accept that Tier 1+2 is the gate and Tier 3 is offline, sampling stops being about coverage. Your coverage is already 100% — every single run passes through the deterministic gate, because it costs ~$0 and runs in milliseconds. Most failures in production are boring and mechanical: stale data, a crash, malformed format, a hallucinated file path, an empty response. Tier 1+2 catches the overwhelming majority of them, alone.

So what's left for the judge? The subjective tail. "Is this summary actually faithful?" "Is the tone right?" "Did it answer the question the user meant?" That's the ~20% where deterministic checks genuinely can't reach — and it's the only place a metered model call earns its cost.

Which means the right sampling policy is not "sample 5% uniformly." It's: let Tier 1+2 gate everything, then send only the runs that passed the gate but live in the subjective tail to the judge. You're not sampling to save money on coverage you're giving up. You're sampling because you already have coverage, and the judge is for a different job.

Here's what that looks like in practice:

type Run = {
  id: string;
  task: string;
  output: string;
  trajectory: unknown[];
};

type GateResult =
  | { verdict: "blocked"; tier: 1 | 2; reason: string }
  | { verdict: "passed"; subjectiveRisk: number };

// Tier 1 + Tier 2: deterministic, ~$0, runs on EVERY run, in the hot path.
function gate(run: Run): GateResult {
  // Tier 1 — unforgeable proof
  if (run.output.trim().length === 0)
    return { verdict: "blocked", tier: 1, reason: "empty output" };
  try {
    JSON.parse(run.output);
  } catch {
    return { verdict: "blocked", tier: 1, reason: "invalid JSON" };
  }

  // Tier 2 — statistical signal vs a baseline the agent didn't author
  const sim = cosineSimilarity(embed(run.task), embed(run.output));
  if (sim < 0.35)
    return { verdict: "blocked", tier: 2, reason: `off-task (sim=${sim.toFixed(2)})` };

  // Passed the gate. Score how likely this run needs a human-ish opinion.
  // Borderline similarity == subjective territory == worth the judge.
  const subjectiveRisk = sim < 0.55 ? 1 : 0.1;
  return { verdict: "passed", subjectiveRisk };
}

// Tier 3 is OFFLINE. You only spend a model call on the ambiguous tail.
function shouldJudge(g: GateResult): boolean {
  return g.verdict === "passed" && Math.random() < g.subjectiveRisk;
}

Notice what the judge never sees: any run that Tier 1+2 already blocked, and the vast majority of clean passes. It sees the borderline middle — labeled, explicitly, as opinion, not evidence. That single framing keeps your green dashboard honest, because you never let a 7/10 masquerade as a gate result.

The judge is only as good as the trace you feed it

There's a load-bearing assumption in all of this: that Tier 1+2 has something real to score against, and that when a run is weird you can actually reconstruct why. That's not free either. If your only artifact is the final string the agent emitted, your "trace" is a single data point and you're back to guessing.

This is why eval and observability ship as a unit, not as separate purchases. agent-eval scores and gates the output — the tiers above, drift, hallucination. AgentLens captures the trace of how the agent got there: every model step and tool call, the resolved inputs, the raw outputs, the trajectory. Two things fall out of that:

1. Your Tier 1+2 checks now have unforgeable, agent-didn't-author data to score against — file writes, tool return values, resolved arguments — instead of just the agent's own summary of what it did.
2. When a sampled judge call flags something, you can actually open the trace and see the step that caused it, instead of re-running and hoping the non-determinism cooperates.

Without the trace, your judge is scoring fiction and your gate is scoring the agent's marketing copy about itself.

The takeaway

Stop asking "what percentage of runs should we score?" It's the wrong question. Score 100% of them with Tier 1+2, because it's deterministic, free, and it's your actual gate. Then sample the subjective tail into an offline judge, and label its output as opinion. Ship the 80% you can prove; reserve the metered model call for the 20% you can only have an argument about.

That's the difference between an eval strategy and a very expensive way to feel covered.

Comments

[Alexey Spinov]

The Tier 1+2-gates-everything / Tier 3-is-offline-opinion split is right, and "uniform 5%" deserved the kicking. I want to push on one line, because your whole allocation rests on it:

subjectiveRisk = sim < 0.55 ? 1 : 0.1

The judge's sampling rate is derived from sim — the same Tier 2 signal that already decided whether to block. So the judge looks hardest exactly where Tier 2 already said "I am doubtful", and least where Tier 2 said "I am confident". For any defect class Tier 2 cannot see, that is not two layers. It is one instrument consulted twice.

I ran your parameters (1.0 / 0.1, split at 0.55) against a budget-matched flat rate — flat set to the average rate your routing already spends, so nobody gets extra judge calls. The break-even falls out exactly clean: routing beats flat iff the defect class is over-represented in the low-sim band, i.e. share-of-class-in-low-band > borderline share of traffic.

Where that bet holds it is not close. 90% of a class in the low band beats flat by 4.8x at 10% borderline traffic, same spend. Your post is right and uniform sampling is worse than it looks.

Where it inverts is the class the judge exists for. A run that parses, stays on task, and is confidently wrong scores HIGH similarity by construction — so it draws the 0.1 rate, the least judge attention in the system:

borderline 10% of traffic:  routed P=0.100 (22 occurrences to 90% confidence)
                            flat   P=0.190 (11 occurrences)
borderline 20% of traffic:  routed P=0.100 (22)   flat P=0.280 (8)

Same spend, flat finds it 2-2.8x sooner. And it is structural rather than a threshold to tune: any monotone function of sim inherits sim's blind spots, so moving 0.55 or 0.1 slides the numbers without changing the shape.

The fix I would keep your allocation for: carve out a small flat stratum — 2-3% — sampled independently of sim, not to catch defects but to estimate the high-sim defect rate. Your routing is a bet, and a flat stratum is the only thing that tells you the bet is still paying. As written, if the high-sim band starts rotting, the mechanism that would notice is the one you sampled down.

Two things I did not measure and will not pretend I did: your borderline share (I swept it 5-40% rather than guess a number), and whether "confidently wrong" really lands high-sim on your embeddings. That second one is the premise of the entire argument, and it is testable on your existing traces in an afternoon.

So which is it in your data — is the high-sim band genuinely low-defect, or only low-observed-defect, because that is where you spend the fewest judge calls?

(Script: stdlib only, offline, closed-form probabilities rather than simulation; two runs byte-identical; sha256 335b7f02dbd56bcb.)