DEV Community: Saurav Bhattacharya

One Quality Score Is a Lie: Split Your RAG Judge Into Retrieval, Groundedness, and Relevance

Saurav Bhattacharya — Wed, 19 Aug 2026 01:02:05 +0000

Ask an LLM judge to score your RAG agent's answer and it will hand you a number. A 7. A 4.2. A crisp green 0.85. That number feels like signal. It is actually three different failures averaged into mush, and the average is engineered to hide the one you most need to see.

I'll make the case with a concrete .NET reference implementation (maf-evals), a three-tier agent evaluation built on Microsoft Agent Framework and .NET 8. Every number below came from running it, not from reasoning about it.

The problem with a single score

A retrieval-augmented answer can be wrong in at least three independent ways:

The retrieval was bad — wrong documents came back, or the knowledge base doesn't cover the question.
The groundedness was bad — the retrieved documents were fine, but the answer made claims they don't support. That's a hallucination.
The relevance was bad — the answer is perfectly grounded in real documents and still doesn't answer what the user asked.

These have nothing to do with each other. A great retrieval can feed a hallucinated answer. A flawlessly grounded answer can miss the question entirely. When you collapse them into one "quality" score, a mediocre-everywhere answer and a perfect-retrieval-but-hallucinating answer can land on the same 3.5. You cannot tell which knob to turn, because the score was designed to be indifferent to which knob is broken.

So the maf-evals RAG triad scores all three separately, and never averages them:

| Score        | Isolates                                        |
| Retrieval    | a bad knowledge base or a bad query             |
| Groundedness | claims the retrieved context doesn't support    |
| Relevance    | well-grounded answers that miss the question    |

Why the average actively lies

Here's the part that turned my opinion from "nice to have" to "non-negotiable." When you calibrate these judges against human labels — and you must calibrate them, a threshold picked without calibration is just taste — the Groundedness judge produces this:

| Score        | Exact | Within 1 | MAE  | Bias  | Same band |
| Groundedness | 42%   | 67%      | 1.17 | -0.17 | 75%       |

Look at that bias: -0.17. Nearly zero. If you were tracking a single blended quality score, that -0.17 would tell you the judge is essentially unbiased and calibrated. Ship it.

It is not calibrated. The Groundedness judge fails in two opposite directions at once. It scores outright fabrication at exactly 3.0 every time — a hallucination sails through as a mild warning — and it penalises well-grounded answers for being slightly off topic. The two errors point opposite ways, so they cancel. The near-zero bias is the average of a systematic over-score and a systematic under-score, and it looks healthy precisely because both are broken.

The metrics that expose it are the ones the average destroys: a mean absolute error of 1.17 (over a 1–5 scale, that's enormous) and band agreement of only 75%. You only see those if you refuse to blend.

That single insight — that fabrication was being scored 3.0 — is why the Groundedness floor moved from 3.0 to 3.5. At 3.0, every hallucination slipped through as a warning. Moving the floor to 3.5 lifted band agreement from 50% to 75%. A blended score would never have surfaced the problem to move the floor over.

Deterministic first, judge last

The triad isn't the first thing that runs. In the pull-request gate (Tier 2), five checks run cheapest-first, and the judge is last:

Rules — the same rule engine the live agent uses, so a rule can't drift between production and CI.
Retrieval — did the expected document IDs come back? Exact, free, no model involved.
Tool calls — right tool, right arguments? Compared, not judged.
Meaning — embeddings, for cases where wording is free to vary. Deterministic, ~1000x cheaper than a judge.
RAG triad — the judge, and only now.

The ordering is doctrine, not convenience. Evidence ranks on an independence axis — independent to corruptible — not a cost axis. A document-ID match is independent: it means the same thing every run. A judge score is corruptible: it moves between runs. So the exact checks run first and always block, while the judge scores get two thresholds — a floor that blocks and a target that warns — because a number that wobbles has no business being a hard gate on its own.

This is why Retrieval scoring is advisory in maf-evals, even though it's part of the triad. Calibration caught the judge returning 5, 2, 4, 5, 2 for the same input — 17% of cases would flip a merge decision at random. So expectedChunkIds, an exact and free check, does the actual gating. The judge only annotates.

What it looks like in code

A golden case declares the deterministic expectation right next to the judged one. Gating lives in the exact fields; the triad adds color:

{
  "id": "refund-within-limit",
  "query": "Order A-31905 arrived damaged. Please refund me 120 for it.",
  "expectedChunkIds": ["refunds#3"],
  "expectedToolCalls": [
    { "name": "issue_refund", "arguments": { "orderId": "A-31905", "amount": 120 } }
  ],
  "semanticExpectations": [
    { "name": "confirms_refund", "anyOf": ["Your refund of 120 has been issued."], "minSimilarity": 0.55 }
  ]
}

expectedChunkIds gates retrieval for free. The triad's Retrieval score, being corruptible, only advises on top of it. Groundedness and Relevance — the two that genuinely need a judge, because they're about meaning — get their floor-and-target bands. Nothing is averaged.

And it's cheap to be this careful about which score means what, but not free: the judge costs about 250x the agent. A full Tier 2 run is ~$0.16. The thing being tested is nearly free; measuring it is the entire bill. That's exactly why you don't want to pay a judge to produce a blended number you then can't act on.

The takeaway

One quality score is a management metric wearing an engineering costume. It goes up and to the right and tells you nothing you can fix. Split it, and each score points at a specific, actionable failure: bad KB, hallucination, or off-topic. Then gate on the checks that don't wobble, and let the judge report on the two that genuinely require meaning.

If you want the whole thing — the triad, the calibration harness that caught the 3.0-hallucination bug, the deterministic checks, and the cost tracker that proves the judge is your real bill — it's all runnable .NET here: github.com/sauravbhattacharya001/maf-evals. Clone it, run dotnet run --project src/EvalRunner -- calibrate --repeat 3, and watch a near-zero bias hide a broken judge on your own screen.

Measure the Judge Before You Trust It: Self-Consistency Comes Before Human Agreement

Saurav Bhattacharya — Wed, 12 Aug 2026 20:47:04 +0000

Here's a question almost no eval pipeline can answer: if you asked your LLM judge to score the exact same response five times, would you get the same number back?

Most teams never check. They wire up an LLM-as-judge, pick a threshold that feels right ("block anything under 7"), and ship it as a merge gate. Then they spend months wondering why the same PR is green on one run and red on the next. The judge wasn't wrong. It was unstable, and nobody measured the stability before trusting the number.

I built maf-evals — a three-tier agent evaluation reference on Microsoft Agent Framework and .NET 8 — partly to force this discipline into the open. The rule that fell out of it: measure the judge before you trust it, and measure self-consistency before you even think about human agreement. Every number below came from actually running the calibration command, not from reasoning about it.

The two questions, in the only order that works

When you calibrate a judge, there are two questions:

Does the judge agree with itself?
Does the judge agree with a human?

The second question is meaningless without the first. If a judge gives the same input a 5, then a 2, then a 4, then a 5, then a 2 — comparing its "score" to a human label is comparing to a coin flip. You can't calibrate a ruler that changes length every time you pick it up.

So maf-evals runs them in order. Twelve hand-labelled cases, each judged three times, self-consistency first.

dotnet run --project src/EvalRunner -- calibrate --repeat 3

Self-consistency: where a comfortable average hides a broken metric

Here's the self-consistency table from an actual run. Three RAG-triad scores — Retrieval, Groundedness, Relevance — each judged three times across the calibration set:

Score	Mean SD	Worst range	Verdict flips
Retrieval	0.20	3.0	17%
Groundedness	0.00	0.0	0%
Relevance	0.00	0.0	0%

Look at Retrieval. A mean standard deviation of 0.20 looks fantastic. If you stopped there — as most dashboards do — you'd conclude the judge is rock solid and promote it to a blocking gate.

That average is lying to you. Given the same input five times, Retrieval returned 5, 2, 4, 5, 2. Most cases were perfectly stable, which dragged the mean SD down to a cozy 0.20 — while two cases swung a full three points. The consequence is the column that actually matters: 17% of cases would flip a merge decision at random.

Think about what that means in CI. Nearly one in five pull requests, near the threshold, gets a merge verdict decided by nothing but which sample the judge happened to draw. That's not a quality gate. That's a random number generator with a nice UI.

The fix isn't to tune the prompt until Retrieval calms down. The fix is a doctrine: a score that moves between runs has no business blocking a merge. Retrieval was demoted to advisory only. The actual gating for "did the right documents come back?" is done by an exact, free, deterministic check — comparing returned chunk IDs against expectedChunkIds. No judge, no wobble.

This is the core independence idea in the whole repo. Evidence ranks on an independence axis — independent to corruptible — not a cost axis. Rules gate; judges report. A deterministic ID comparison is unforgeable and never flips, so it gates. A judge's opinion moves between runs, so it reports.

Only now do you ask about human agreement

Groundedness and Relevance passed self-consistency (0.00 SD), so they've earned the right to be compared against human labels. Here's that table:

Score	Exact	Within 1	MAE	Bias	Correlation	Same band
Retrieval	75%	92%	0.42	-0.42	0.88	83%
Groundedness	42%	67%	1.17	-0.17	0.44	75%
Relevance	25%	83%	0.92	-0.42	0.74	83%

Now look at Groundedness, and notice the trap. Its bias is -0.17 — practically zero. If bias were your health metric, you'd call it well-calibrated and move on.

Groundedness is broken in two opposite directions at once. It scores outright fabrication at exactly 3.0, every single time — and it separately penalises well-grounded answers for being slightly off-topic. Those two errors point in opposite directions, so they cancel in the aggregate, and the bias reads as a healthy -0.17. The metrics that expose the truth are mean absolute error (1.17) and band agreement (75%) — not the average.

Which is why the Groundedness floor moved from 3.0 to 3.5. At a 3.0 floor, every hallucination — parked at exactly 3.0 — slipped through as a mere warning instead of a block. That single threshold change lifted band agreement from 50% to 75%.

That's the whole point of calibration: the threshold wasn't chosen by taste. It was chosen because the labelled fixtures showed hallucinations clustering at 3.0, so the floor had to sit above them.

Two thresholds, because judges wobble and rules don't

Because judge scores move between runs, a single cut-off turns every borderline case into a coin flip — exactly the 17% problem. So in maf-evals, every judge score gets two thresholds:

a floor that blocks, and
a target that warns.

The band between them absorbs the wobble. Deterministic checks — chunk-ID matches, tool-call comparisons, rule evaluations — get no band. They always block, because they don't move.

Conceptually, the gate logic looks like this:

public GateOutcome ApplyScore(JudgeScore score, ScoreBand band)
{
    // Deterministic checks never reach here — they block or pass, no band.
    if (score.Value < band.Floor)
        return GateOutcome.Block(
            $"{score.Name} {score.Value:F1} < floor {band.Floor:F1}");

    if (score.Value < band.Target)
        return GateOutcome.Warn(
            $"{score.Name} {score.Value:F1} < target {band.Target:F1}");

    return GateOutcome.Pass();
}

// Groundedness after calibration: floor raised so hallucinations (which
// the judge parks at exactly 3.0) fall below it and BLOCK.
var groundedness = new ScoreBand(Floor: 3.5, Target: 4.0);

The floor at 3.5 isn't a guess. It's the number the labelled data demanded, verified by the jump in band agreement.

The rule that keeps you honest

One more constraint makes this whole thing trustworthy: re-run calibration after you change the judge model, the labelling guide, or any threshold. Scores from different judges are not comparable. A 3 from gpt-4o and a 3 from next quarter's model are different quantities wearing the same label. If you swap the judge and keep the old thresholds, you've silently thrown away the calibration and you're back to picking numbers by feel.

And the calibration set itself is built so the three scores pull apart from each other — twelve cases engineered so a judge can't get a good grade by collapsing retrieval, groundedness, and relevance into one vague sense of "quality." If your calibration cases don't isolate the failure modes, your calibration proves nothing.

The takeaway

An LLM judge is a measuring instrument, and you would never trust a measuring instrument you hadn't checked against a reference. Yet teams routinely promote a raw judge score straight to a merge gate.

The order is non-negotiable:

Self-consistency first. If the judge disagrees with itself, nothing else matters. Watch the worst-case range and verdict-flip rate, not the mean SD — a comfortable 0.20 average hid a 17% coin flip.
Human agreement second, and only for scores that survived step one. Watch MAE and band agreement, not bias — Groundedness's near-zero bias hid a metric broken in two directions.
Anything that wobbles gets demoted from gate to advisory, and the deterministic check does the real gating.

Rules gate; judges report. Measure the judge before you trust it. If a score moves between runs, it doesn't belong on the merge path.

The full three-tier implementation — guardrails inside the agent, the PR gate, trajectory judging, plus the calibration harness and an adversarial safety suite — is on Microsoft Agent Framework and .NET 8 here: github.com/sauravbhattacharya001/maf-evals. If you're about to trust an LLM judge with a blocking decision, run calibrate --repeat 3 on your own judge first. The number that comes back will change how you gate.

Your Agent's Context Window Overflowed and It Answered Anyway

Saurav Bhattacharya — Wed, 12 Aug 2026 01:03:33 +0000

Your agent works great in the demo. Then someone hands it a real ticket with a 40-message thread, three attached logs, and a stack trace, and it confidently answers using the first half of the context — because the second half fell off the back of the window. No error. No exception. Just a quietly wrong answer with full confidence. This is context overflow, and it is one of the most under-instrumented failure modes in production agents.

Here is the uncomfortable part: your model-as-judge will not catch it, and it shouldn't be asked to. Context truncation is not a subjective quality problem. It is an observable, deterministic fact about what actually entered the model. You can prove it happened. That makes it a Tier 1 problem, and treating it like one changes everything about how you defend against it.

Why this is invisible

Most agent frameworks silently truncate. You assemble a prompt from system instructions, retrieved documents, tool outputs, and conversation history, and if it exceeds the window, the framework (or the provider) drops the overflow — usually from the middle or the oldest turns. The model still returns a fluent, plausible response. Your evals are green. Your users get answers built on a partial view of the problem.

The reason this slips through is that teams evaluate the output text and never inspect the resolved input. They ask "did the answer look good?" instead of "did the evidence the agent needed actually make it into the call?" Those are different questions, and only one of them is answerable without opinion.

The independence axis, not the cost axis

This is where I want to plant a flag, because it's the thing that separates real agent evaluation from "LLM-as-judge gives you a 7/10" tooling. Evidence should be ranked on an independence axis — from independent to corruptible — not a cost axis of cheap to expensive.

Tier 1 — externally observable proof the agent can't forge. Did the input fit the window? Did the required document ID appear in the resolved prompt? Was the tool output non-empty? Did the run finish inside its timeout? These are facts. Valid or not, present or not, truncated or not.
Tier 2 — statistical signal against a baseline the agent didn't author. Is the retrieved chunk actually similar to the task embedding? Did the token count spike 3x versus the rolling baseline for this task type? Did the diff change anything?
Tier 3 — model-as-judge. A shared-substrate opinion. A signal, never a verdict.

Context overflow lives squarely in Tier 1. You don't need a smarter model to tell you the prompt didn't fit — you need to measure the resolved prompt.

There's a second reason Tier 3 is the wrong tool here, and it's structural: a model judging what another model saw is circular. Judge and judged share a substrate; there's no independent ground truth in that loop. Tier 1 and Tier 2 can run over the agent's actual trajectory precisely because they inspect artifacts the agent didn't get to write — the byte count of the assembled prompt, the presence of a chunk ID, the embedding of the input. The judge can only offer opinion about text, and it should only ever inspect artifacts the judged agent didn't author.

Instrument the resolved input

The fix is boring and effective: capture the fully-resolved prompt at the moment of the call and gate on it deterministically, before you spend a judge token.

import { encoding_for_model } from "tiktoken";

interface ResolvedCall {
  taskId: string;
  requiredIds: string[];      // doc/chunk IDs this task needs
  resolvedPrompt: string;     // the ACTUAL assembled prompt sent
  windowLimit: number;
}

interface GateResult {
  tier: 1 | 2;
  pass: boolean;
  reason: string;
}

export function contextGate(call: ResolvedCall): GateResult[] {
  const enc = encoding_for_model("gpt-4o");
  const tokens = enc.encode(call.resolvedPrompt).length;
  enc.free();

  const results: GateResult[] = [];

  // Tier 1: did it physically fit?
  results.push({
    tier: 1,
    pass: tokens <= call.windowLimit,
    reason: `resolved prompt = ${tokens} tokens (limit ${call.windowLimit})`,
  });

  // Tier 1: did every required piece of evidence survive assembly?
  const missing = call.requiredIds.filter(
    (id) => !call.resolvedPrompt.includes(id),
  );
  results.push({
    tier: 1,
    pass: missing.length === 0,
    reason: missing.length
      ? `dropped required evidence: ${missing.join(", ")}`
      : "all required evidence present",
  });

  return results;
}

Notice what this does not do: it doesn't ask whether the answer was good. It asks whether the answer was even possible given what entered the model. If a required chunk ID never made it into the resolved prompt, the run is dead on arrival — block it, don't grade it. This is the real-time gate: deterministic, roughly free, fast enough to sit in the hot path and stop a bad run before it ships.

The two halves: score the output, trace the run

You cannot gate on the resolved prompt if you never captured it. This is why evaluation and tracing ship as one workflow, not two products.

agent-eval scores and gates the output using the tier doctrine above — it's the thing that knows "context overflowed" is a Tier 1 red, not a judge's vibe. AgentLens captures the trace of how the agent got there: every model and tool step, the resolved inputs, the raw outputs. That trace is what gives Tier 1 and Tier 2 something unforgeable to score against, because the agent didn't author its own trace — the harness did. Without the trace, "did the required evidence enter the window?" is unanswerable. With it, it's a two-line check.

Run them together and the division of labor is clean: AgentLens records the ground truth of the run, agent-eval decides whether that ground truth clears the gate.

Ship the 80%

Most agent failures are not subtle disagreements a judge must arbitrate. They are stale caches, crashes, malformed JSON, hallucinated file paths, empty tool results — and context that silently overflowed. All of that is caught at Tier 1 and Tier 2 alone, deterministically, at roughly zero cost, in the hot path. Reserve the model-as-judge for the genuinely subjective ~20% tail, and label its output honestly: opinion, not evidence.

The context window isn't a config detail. It's the boundary of what your agent could possibly know at inference time. Measure that boundary, gate on it, and you'll retire a whole category of confident-but-wrong answers before a judge ever needs an opinion.

Your Golden Dataset Is Rotting: The Eval Oracle Nobody Re-Validates

Saurav Bhattacharya — Sun, 09 Aug 2026 01:01:57 +0000

We talk about agents drifting. We almost never talk about the thing we measure them against drifting. But your golden dataset — the fixtures, expected outputs, and "known good" traces your evals grade against — is code that ships to production and then never gets a code review again. It rots. And when your oracle rots, a green dashboard stops meaning your agent is correct. It means your agent still agrees with a snapshot of the world you took eight months ago.

This is the failure mode senior teams keep rediscovering the hard way: the agent is fine, the harness is fine, and the test oracle is the thing that's wrong. Nobody re-validates it, because passing tests feel like the end of the story instead of a claim that also decays.

Why oracles rot

A golden dataset encodes assumptions about the world at the moment you captured it. A few months later:

The API you called changed its response shape, so your "expected output" is now describing an endpoint that no longer exists.
The correct answer changed. "Current CEO," "latest stable version," "the recommended flag" — these have expiration dates baked in.
A human labeled the golden answer under a policy that has since been updated, so your eval now enforces last quarter's rules.
The expected string was itself generated by a model, and you never checked whether it was actually right. You froze a plausible guess and called it truth.

None of these trip an alarm. Your suite is green the whole way down, because green means "matches the fixture," not "matches reality."

The tier lens: which evidence actually expires?

This is where it helps to stop ranking evidence by cost (cheap vs expensive) and start ranking it by independence — how forgeable the signal is, and by extension how well it holds up over time. agent-eval sorts evidence into three tiers on exactly that axis:

Tier 1 — externally observable proof the agent can't forge. Valid JSON, the file exists, the URL resolves, it compiled, the tests passed, it finished inside the timeout, the output is non-empty.
Tier 2 — statistical signal against a baseline the agent didn't author. Embedding similarity to the task, length and repetition checks, whether the diff actually changed anything.
Tier 3 — model-as-judge. A shared-substrate opinion. A signal, never a verdict.

Now re-read the rot list through that lens. Tier 1 checks barely rot: "is this valid JSON" and "does this file exist" mean the same thing today as they did last year. The oracle for Tier 1 is the world, and the world re-validates itself every run. Tier 2 rots slowly and legibly: your baseline embedding shifts as the task definition shifts, which is a signal you can watch. It's Tier 3 and hand-frozen golden strings that rot fastest and most silently — because their oracle is an opinion or a snapshot, and opinions and snapshots don't refresh themselves.

That's the argument for two structural rules a lot of teams learn late:

Tier 1 + Tier 2 are your real-time gate — deterministic, roughly free, fast enough to block a run in the hot path. Tier 3 is offline-only — metered, slow, non-deterministic, so it can't sit in the latency budget of a live request.
Tier 1 + Tier 2 may run over the agent's full trajectory. Tier 3 may not. A model judging another model's reasoning is circular — judge and judged share a substrate, so there's no independent ground truth in the loop. Tier 3 gets to inspect artifacts the judged agent didn't get to write, and nothing more.

A rotting golden dataset is what happens when you let a frozen Tier 3 opinion masquerade as a Tier 1 fact.

Ship the 80%, then re-validate the tail

Most real failures — stale output, a crash, a malformed response, a hallucinated file path, an empty answer — are caught at Tier 1 + Tier 2 alone, and those checks age gracefully. Reserve the judge for the ~20% subjective tail, and label its output as "opinion, not evidence." That framing also tells you where to point your re-validation budget: not at the whole suite, but at the fixtures whose correctness is an opinion or a snapshot. Those are the ones with an expiration date.

Here's the cheap version: give every golden fixture a freshness contract, and let Tier 1 checks re-prove the world-facing ones on every run.

type Tier = 1 | 2 | 3;

interface GoldenFixture {
  id: string;
  expected: unknown;
  oracleTier: Tier;        // how the "truth" was established
  capturedAt: string;      // ISO timestamp
  maxAgeDays: number;      // expiration for snapshot/opinion oracles
  reverify?: () => Promise<boolean>; // Tier 1 re-proof against the world
}

async function checkOracleHealth(f: GoldenFixture): Promise<string[]> {
  const problems: string[] = [];

  // Tier 1 fixtures can re-prove themselves: does the URL/file still resolve?
  if (f.reverify && !(await f.reverify())) {
    problems.push(`${f.id}: world-facing oracle no longer holds`);
  }

  // Snapshot/opinion oracles (frozen strings, judge labels) expire.
  if (f.oracleTier >= 2) {
    const ageDays = (Date.now() - Date.parse(f.capturedAt)) / 86_400_000;
    if (ageDays > f.maxAgeDays) {
      problems.push(
        `${f.id}: oracle is ${Math.round(ageDays)}d old (max ${f.maxAgeDays}), re-validate before trusting`,
      );
    }
  }
  return problems;
}

Run that as a meta-eval — an eval on your evals. When it goes red, you don't touch the agent. You go re-validate the oracle. The point isn't the specific thresholds; it's making "this expected value has an expiration date" a first-class property instead of a tribal assumption.

You can't re-validate what you can't see

Here's the part teams miss: to re-validate a fixture, you need to know how the agent actually produced the output you're grading, not just the final string. That's where the eval half and the trace half meet. agent-eval scores and gates the output — the tiers, the drift, the hallucination checks above. AgentLens captures the trace of how the agent got there: every model call and tool step, the resolved inputs, the raw outputs. Two halves of one workflow.

The trace is what makes Tier 1 + Tier 2 possible in the first place — those tiers need trajectory data the agent didn't get to author, and AgentLens is where that unforgeable record lives. It's also what makes rot diagnosable: when your meta-eval flags a stale fixture, the trace tells you whether the expected value was a real observation or a model's frozen guess, and which tool response it was pinned to. Without the trace you're re-guessing the oracle. With it, you're auditing it.

The takeaway

An eval is a claim about the world, and claims expire. Rank your evidence by independence, not price: Tier 1 and Tier 2 hold up because their oracle is reality; Tier 3 and frozen golden strings are opinions with a shelf life. Gate on the durable tiers in real time, keep the judge offline and clearly labeled as opinion, and run a meta-eval that treats oracle freshness as a real signal. Pair agent-eval for the scoring with AgentLens for the trace, and a red fixture stops being a mystery. Because the most dangerous eval isn't the one that fails. It's the one that passes against a world that no longer exists.

Your Agent Passes Every Turn and Fails the Conversation

Saurav Bhattacharya — Fri, 07 Aug 2026 01:02:16 +0000

Your agent eval suite grades one turn at a time. Prompt in, output out, score it, move on. That model is fine for a completion endpoint. It is quietly wrong for anything that holds a conversation, and it is the reason your "94% pass rate" coexists with users who rage-quit on turn six.

The unit of failure for a conversational agent is not the turn. It is the session.

The per-turn blind spot

Consider a support agent. Turn 1: user asks about a refund. The agent answers correctly and passes every eval. Turn 4: user clarifies they mean a partial refund. The agent answers that correctly too. Turn 6: the agent quotes the original refund amount again, having silently dropped the "partial" constraint three turns ago.

Every single turn passes an isolated eval. Each output is grounded, well-formatted, on-topic. The failure only exists across turns: a dropped constraint, a contradiction with turn 4, a promise never kept. Per-turn grading is structurally blind to it, because the bug is in the relationship between outputs, not in any one of them.

This is not an exotic edge case. Constraint decay, self-contradiction, and forgotten commitments are the dominant failure class for multi-turn agents. And they are invisible to the eval architecture most teams actually run.

Where session-level signal lives on the tier ladder

If you have read anything I have written, you know I rank eval evidence on an independence axis, not a cost axis, from evidence the agent cannot forge down to opinion it shares a substrate with:

Tier 1: externally observable proof the agent cannot fake. Valid JSON, a file that exists, tests that passed, finished within timeout, non-empty.
Tier 2: statistical signal vs a baseline the agent did not author. Embedding similarity, repetition, whether a diff actually changed anything.
Tier 3: model-as-judge. Shared-substrate opinion. A signal, never a verdict.

The mistake teams make with multi-turn is assuming session evaluation is inherently a Tier 3 problem, "just ask a judge if the whole conversation was coherent." It is not. A huge fraction of session failures are Tier 1 and Tier 2, if you look at the trajectory instead of the last message:

Did turn 6's amount contradict a value the agent itself committed to in turn 4? That is a deterministic diff over structured extractions, Tier 1. No judge needed.
Did a constraint the user set ("partial", "in EUR", "before Friday") survive to the final output? Set-membership check over resolved slots, Tier 1/2.
Did the agent repeat an earlier answer verbatim instead of advancing? Repetition and embedding-similarity across turns, Tier 2.
Did every user question get a corresponding answer, or did one get dropped? Count and coverage check, Tier 1.

These run over the trajectory, deterministically, at ~$0, fast enough to gate. And Tier 1+2 are allowed to run over trajectories precisely because they do not share a substrate with the agent. Tier 3 cannot: a model judging another model's multi-turn reasoning is circular, because judge and judged share the same failure priors, so there is no independent ground truth. Reserve the judge for the genuinely subjective session tail ("was the tone appropriate across the escalation?"), offline, clearly labeled opinion.

You cannot check a trajectory you did not capture

Here is the operational catch: every Tier 1/2 session check above needs the trace, the resolved inputs, the committed values, the tool outputs at each step. Not a summary the agent wrote. The actual sequence.

This is why the two halves ship as one workflow. AgentLens captures the trace: every model and tool step, the resolved inputs, the raw outputs, unforgeable and agent-did-not-author records of what actually happened turn by turn. agent-eval scores and gates against that trace using the tier doctrine above. The eval is only as trustworthy as the trace it runs on, and the trace is only useful if something scores it. Trace without eval is a debugging log; eval without trace is a judge guessing.

Here is a Tier 1 session gate, contradiction detection over committed values, that needs zero model calls:

import { z } from "zod";

const RefundCommitment = z.object({
  turn: z.number(),
  amount: z.number(),
  scope: z.enum(["full", "partial"]),
});

type Commitment = z.infer<typeof RefundCommitment>;

// Pulled from the AgentLens trace: what the agent actually committed to,
// per turn, not what it claims in its final summary.
function evalSessionConsistency(commitments: Commitment[]) {
  const failures: string[] = [];

  for (let i = 1; i < commitments.length; i++) {
    const prev = commitments[i - 1];
    const cur = commitments[i];

    // A later turn silently reverting an earlier committed scope/amount
    if (cur.scope !== prev.scope && cur.amount === prev.amount) {
      failures.push(
        `turn ${cur.turn}: scope changed ${prev.scope}->${cur.scope} ` +
        `but amount stayed ${cur.amount} (stale value?)`
      );
    }
  }

  return {
    tier: 1 as const,
    passed: failures.length === 0,
    failures, // deterministic, ~$0, safe to block the run
  };
}

No judge scored that. It is a diff over trace data the agent could not author, and it catches the exact turn-6 bug that six green per-turn evals waved through.

Ship the 80%

The reflex is to throw a smart judge at multi-turn coherence. Resist it. Most session failures, dropped constraints, contradictions, repeated answers, unanswered questions, are caught deterministically at Tier 1+2 over the trajectory, at ~$0, fast enough to block before the bad turn ships. That is the 80%.

Reserve the model-as-judge for the ~20% subjective tail, tone, empathy, whether an escalation felt handled, offline, metered, and labeled for what it is: opinion, not evidence.

Grade the session, not the sentence. Capture the trajectory with AgentLens, gate it with agent-eval, and stop letting a wall of green per-turn checks certify a conversation that fell apart on turn six.

Your Agent Said It Worked. Go Check the World, Not the Sentence.

Saurav Bhattacharya — Wed, 05 Aug 2026 01:02:59 +0000

Your agent said it created the ticket. The eval passed. The ticket does not exist.

This is the failure mode that action-taking agents introduce and that output-grading evals sail right past. When an agent's job is to say something, grading the text is grading the job. When an agent's job is to do something — write a file, open a PR, charge a card, send an email — the text it emits is a claim about a side effect, not the side effect itself. An agent that has learned to produce confident, well-formatted "Done! I created issue #4213" strings will pass any judge you point at its output, forever, whether or not #4213 is real.

If you take one thing from this post: for action agents, the artifact under eval is the world, not the sentence.

Claims are cheap, effects are load-bearing

A model-as-judge reading "I created the ticket and assigned it to the on-call" has exactly zero independent information about whether a ticket exists. Judge and agent share a substrate; the judge is just a second language model agreeing that the sentence sounds like success. That is circular, and it is the whole reason agent-eval ranks evidence on an independence axis — independent to corruptible — rather than a cost axis of cheap to expensive.

Tier 1 — externally observable proof the agent can't forge. The ticket exists when you GET /issues/4213 and get a 200. The file exists on disk. The PR is open. The row is in the database. None of this can be hallucinated into being.
Tier 2 — statistical signal vs a baseline the agent didn't author. The created ticket's title actually embeds-similar to the task you gave it (not a real ticket for the wrong thing). The diff changed the file it claimed to change.
Tier 3 — model-as-judge. Was the ticket well-written? That's an opinion, a signal, never a verdict — and it only earns a seat after Tiers 1 and 2 have confirmed the ticket is real.

For side-effecting agents, Tier 1 is not the nice-to-have. It's the whole game. Most of your production failures — the API call 500'd, the write hit a read-only mount, the agent retried and created the ticket twice, the "sent" email bounced — are caught here for ~$0, deterministically, fast enough to sit in the hot path and block the run. Tier 3 can't do any of that: it's offline-only, metered, and non-deterministic. You do not want a slow model opinion standing between your user and a retry.

Verify the effect, don't grade the claim

Here's the shape of a Tier 1 side-effect check. Note what it does not do: it never reads the agent's own summary of what happened.

type EffectCheck<T> = {
  name: string;
  // Observe the world independently. The agent does not get to write this.
  observe: () => Promise<T | null>;
  // Assert the observed state matches the task, not the agent's story.
  expect: (observed: T) => boolean;
};

async function verifyEffect<T>(check: EffectCheck<T>) {
  const observed = await check.observe();
  if (observed === null) {
    return { tier: 1, pass: false, reason: `${check.name}: effect not found in world` };
  }
  if (!check.expect(observed)) {
    return { tier: 1, pass: false, reason: `${check.name}: effect exists but wrong shape` };
  }
  return { tier: 1, pass: true };
}

// The agent claimed it opened issue #4213 for the task "flaky login test".
const ticketExists = await verifyEffect({
  name: "github-issue",
  observe: () => gh.issues.get({ owner, repo, issue_number: 4213 })
                    .then(r => r.data).catch(() => null),
  expect: (issue) =>
    issue.state === "open" &&
    embedSimilar(issue.title, "flaky login test") > 0.75, // Tier 2 riding along
});

The observe function is the entire point. It calls GitHub, not the agent. A hallucinated issue number returns null and fails at Tier 1 before any judge is ever invoked. This is how you "ship the 80%": stale, crashed, wrong-shape, and hallucinated-effect failures all die here, cheaply, leaving only the genuinely subjective tail — is this a good ticket? — for the metered judge, clearly labeled opinion, not evidence.

The trace is what makes this debuggable

There's a second problem hiding in the code above. When verifyEffect returns effect not found, why? Did the agent call issues.create and get rate-limited? Did it call the wrong endpoint? Did it call the right one, get a 201 back with issue #4299, and then hallucinate #4213 into its summary? The pass/fail tells you the world is wrong. It doesn't tell you where the agent went off the rails.

That's what AgentLens is for. It captures the trace of how the agent got to its claim — every model step and tool call, the resolved inputs, and the raw outputs the agent actually received. So when Tier 1 goes red, you replay the trajectory: issues.create returned 201 with number: 4299, and the agent's final message said 4213. Now you know it's a summarization bug, not a permissions bug, and you fix the right thing.

The pairing runs deeper than debugging, though. Tier 1 and Tier 2 need something to score against, and it has to be data the agent didn't author. The trace is exactly that. The raw 201 response body sitting in AgentLens is unforgeable ground truth: agent-eval reads the real returned issue number from the trace and compares it to what the agent claimed, and the lie surfaces instantly. agent-eval scores and gates the output; AgentLens captures the trajectory that makes the score both debuggable and trustworthy. They're two halves of one loop — you can't do independent evals on trajectory data you didn't independently capture.

The rule

Stop asking "did the agent say it worked?" Ask "did the world change the way the task required?" For action agents those are different questions, and only one of them is load-bearing. Verify the effect at Tier 1, corroborate its shape at Tier 2, and reserve the judge for the 20% where taste actually matters — with the trace underneath so a red gate points you at a fix instead of a shrug.

Your agent's summary is a hypothesis. Go check.

Your New Eval Rule Is Untested Code Guarding Production

Saurav Bhattacharya — Sun, 02 Aug 2026 01:01:44 +0000

You wrote a new eval. It caught the failure you just saw in production. You shipped it as a gate. Congratulations — you now have a piece of untested code sitting on the hot path of every agent run, deciding what ships and what gets blocked.

We treat agent evals as if writing them is the hard part. It isn't. The hard part is knowing your eval actually discriminates: that it goes red on the bad traces and green on the good ones, and not the other way around. An eval you haven't run against known-labeled traces is a coin flip with a dashboard.

The failure nobody names: the false-negative gate

Here's the quiet disaster. You add a rule to catch hallucinated file paths. It has a regex bug. It matches nothing. Every run passes. Your dashboard is green. You feel safe. Three weeks later a customer finds the exact failure your "gate" was supposed to block, and you discover the rule never fired once.

A green eval is not evidence of a healthy agent. It's evidence that either the agent is healthy or your eval is broken — and you have no way to tell those apart unless you've fed it a trace you already know is bad and watched it go red.

Evals are code. Code that guards production gets tested against fixtures. Somehow evals got a pass on this, and it's the single biggest source of false confidence I see.

Where the tiers actually help you here

The reason this matters so much is tied to how you should be ranking eval evidence in the first place. Not on a cost axis — cheap-to-expensive is the wrong mental model. Rank it on an independence axis: how forgeable is the signal by the agent producing the output?

Tier 1 — externally observable proof the agent can't fake. Valid JSON, the file exists on disk, it compiled, tests passed, it finished before the timeout, the output isn't empty. Ground truth, no opinion.
Tier 2 — statistical signal against a baseline the agent didn't author. Embedding similarity between output and the task spec, length and repetition checks, did the diff actually change anything.
Tier 3 — model-as-judge. A shared-substrate opinion. A signal, never a verdict.

This ranking is exactly what makes your eval suite testable. Tier 1 and Tier 2 are deterministic, run at roughly zero cost, and are fast — which is why they're your real-time gate: they can sit in the hot path and block a run. Because they're deterministic, you can pin them against a fixed corpus of traces and get the same answer every time. That's what "testing your eval" even means.

Tier 3 can't do that. It's non-deterministic, metered, and slow, so it's offline-only — it can't live in the hot path, and it can't be regression-tested in the same clean way because it won't give you a stable answer twice. There's a deeper problem too: a model judging another model's reasoning is circular. Judge and judged share a substrate; there's no independent ground truth. So Tier 3 is only allowed to inspect artifacts the judged agent didn't get to write, and even then it's "opinion, not evidence." You don't gate on it and you don't pretend you can unit-test it into reliability.

The practical upshot: ship the 80%. Most real failures — stale output, a crash, malformed format, a hallucinated path, an empty result — are all caught at Tier 1+2 alone, deterministically, for free. Reserve the judge for the ~20% subjective tail and label it loudly as opinion. And it's precisely the Tier 1+2 rules — the deterministic ones — that you can and must test before trusting.

Testing an eval like it's code

The move is boring and it works: keep a corpus of labeled traces, and assert that each eval produces the label you expect. A new rule doesn't ship until it goes red on the known-bad and green on the known-good.

type Verdict = "pass" | "fail";

interface Trace {
  id: string;
  output: string;
  claimedPaths: string[];
  existsOnDisk: (p: string) => boolean;
}

// A Tier 1 eval: no hallucinated file paths. Deterministic, ~$0.
function noHallucinatedPaths(t: Trace): Verdict {
  return t.claimedPaths.every((p) => t.existsOnDisk(p)) ? "pass" : "fail";
}

// The fixtures are the point. Each is a trace you ALREADY labeled.
const fixtures: { trace: Trace; expected: Verdict }[] = [
  {
    trace: {
      id: "good-1",
      output: "wrote src/index.ts",
      claimedPaths: ["src/index.ts"],
      existsOnDisk: () => true,
    },
    expected: "pass",
  },
  {
    trace: {
      id: "bad-1",
      output: "edited src/does-not-exist.ts",
      claimedPaths: ["src/does-not-exist.ts"],
      existsOnDisk: () => false,
    },
    expected: "fail", // if this comes back "pass", your GATE is broken
  },
];

for (const { trace, expected } of fixtures) {
  const got = noHallucinatedPaths(trace);
  if (got !== expected) {
    throw new Error(
      `Eval regression on ${trace.id}: expected ${expected}, got ${got}`,
    );
  }
}

The bad-1 fixture is the whole game. Without it, a regex typo or an inverted condition ships silently and your gate becomes decoration. With it, a broken eval fails your CI instead of failing your customer.

This only works if your traces are trustworthy

Everything above assumes you have real traces with real, unforgeable inputs to score against — and that's exactly where most teams fall down. If your "trace" is a summary the agent wrote about its own run, you've handed the graded student the answer key.

This is why the two halves ship as a unit. agent-eval scores and gates the agent's output — it's where the tier doctrine lives, where drift and hallucination checks run, where a red gate blocks a run. But it can only score what it can see. AgentLens captures the trace of how the agent got there: every model and tool step, the resolved inputs, the raw outputs, none of it authored after the fact by the agent to look good. That trace data is what gives Tier 1+2 something unforgeable to grade against — and it's what lets you build a labeled corpus in the first place, because you're mining real runs instead of inventing fixtures from imagination.

agent-eval tells you the run failed. AgentLens lets you reopen it, see the exact step where it went wrong, label it, and drop it into your fixture set so the eval that missed it can never miss it again. One captures the trace; the other scores it. Neither is useful alone.

The one-line policy

Write it on the wall: no eval reaches the gate without a red fixture. If you can't produce a trace that makes the rule fail, you haven't written an eval — you've written a comment that happens to return "pass".

Test your evals against known-bad traces before you trust them to guard anything. Deterministic Tier 1+2 rules make that testing possible; capture real traces so the fixtures are real; keep the judge offline and honest about being an opinion. That's the difference between a gate and a green light nobody's checked.

Your Agent's Deadline Is a Correctness Test, Not an SLO

Saurav Bhattacharya — Fri, 31 Jul 2026 01:02:28 +0000

Ask an engineer to list their agent's failure modes and you'll hear about hallucinations, wrong tool calls, and bad JSON. Ask about time and you get a shrug. Yet the single most common thing a production agent does when it goes wrong is not fail loudly — it just takes too long. It loops. It retries a flaky tool. It waits on a model call that never streams a first token. And your eval suite, which runs after the fact on whatever output eventually showed up, grades it green.

This is the blind spot: we treat latency as an SRE dashboard concern, divorced from correctness. But for an agent, a deadline miss is a correctness failure. A summary that arrives 90 seconds late is often worse than no summary — the user already left, the downstream job already timed out, the retry already double-charged them. Time belongs in your evals, and it belongs at the very bottom of the stack.

Time is Tier 1 evidence

If you've followed the tier doctrine behind agent-eval, you know evidence ranks on an independence axis — from evidence the agent can't forge, to opinion it shares a substrate with — not a cost axis. Three tiers:

Tier 1 — externally observable proof the agent can't fake: valid JSON, the file exists, the code compiled, tests passed, it finished within the deadline, the output is non-empty.
Tier 2 — statistical signal against a baseline the agent didn't author: embedding similarity to the task, length and repetition, whether the diff actually changed anything.
Tier 3 — model-as-judge: a shared-substrate opinion. A signal, never a verdict.

Notice where "finished within the timeout" sits. It's Tier 1, right next to "valid JSON." That's not an accident. A wall-clock deadline is the most independent signal you have — the agent cannot argue with a stopwatch, cannot reason its way past it, cannot forge it. Physics grades this one. There is no more incorruptible ground truth in your whole eval suite than "the clock ran out."

And crucially, Tier 1+2 are your real-time gate: deterministic, ~$0, fast enough to sit in the hot path and block a run. A timeout check is the purest expression of that — it's already in the hot path by definition. Tier 3, the judge, is the opposite: metered, slow, non-deterministic, offline-only. You would never put a model-as-judge on the critical path to decide whether a response was fast enough. The judge can't even see the clock.

What "graded green" actually looks like

Here's the trap. Most eval harnesses receive (input, output) and score the output. The output is whatever the agent finally returned — so by construction, timing information has already been discarded before grading begins. The eval literally cannot see that the run blew its deadline, because it only exists once the run is over.

You have to grade the run, not the return value:

type RunResult<T> =
  | { status: "ok"; value: T; elapsedMs: number }
  | { status: "timeout"; elapsedMs: number; deadlineMs: number }
  | { status: "error"; elapsedMs: number; error: string };

async function withDeadline<T>(
  work: (signal: AbortSignal) => Promise<T>,
  deadlineMs: number,
): Promise<RunResult<T>> {
  const ctrl = new AbortController();
  const started = Date.now();
  const timer = setTimeout(() => ctrl.abort(), deadlineMs);
  try {
    const value = await work(ctrl.signal);
    return { status: "ok", value, elapsedMs: Date.now() - started };
  } catch (err) {
    const elapsedMs = Date.now() - started;
    if (ctrl.signal.aborted) {
      return { status: "timeout", elapsedMs, deadlineMs };
    }
    return { status: "error", elapsedMs, error: String(err) };
  } finally {
    clearTimeout(timer);
  }
}

Now the eval gate is trivial and deterministic — a Tier 1 check that runs in microseconds:

function tier1TimingGate(run: RunResult<unknown>): { pass: boolean; reason?: string } {
  if (run.status === "timeout") {
    return { pass: false, reason: `deadline miss: ${run.elapsedMs}ms > ${run.deadlineMs}ms` };
  }
  if (run.status === "error") {
    return { pass: false, reason: `crashed after ${run.elapsedMs}ms` };
  }
  return { pass: true };
}

A timed-out run never reaches Tier 2 or 3. There's nothing to embed, nothing to judge — the correct answer that arrives too late is not a correct answer. You short-circuit, you block the run, you fall back. No model was asked for its opinion, because no opinion was needed.

Deadlines are per-budget, not per-agent

The mistake I see next is a single global timeout. Deadlines are contextual: a background reindex job can take ten minutes; an interactive chat turn has maybe three seconds before the user perceives a stall. The deadline is a property of the call site, not the agent. Thread it through as part of the task contract, and let every tool step inherit and decrement a shared budget — so a tool that eats 80% of the budget leaves the model no room to actually respond, and that is a gradeable event too.

You can't grade time you didn't record

All of this assumes you actually captured elapsed time per step — and here's where the eval half of the story needs its other half. agent-eval scores and gates the output; it can only enforce a Tier 1 timing gate if something recorded the timing. That's the job of trace capture. AgentLens instruments the agent's trajectory — every model call and tool step, with resolved inputs, raw outputs, and start/stop timestamps — so "finished within the deadline" is a fact you can read off an unforgeable trace the agent didn't get to author, not a number you hope somebody logged.

This pairing is the whole point. The trace is the substrate Tier 1+2 grade against: because AgentLens records when each step started and stopped independently of what the agent claims it did, your timing gate has real ground truth. An agent can hallucinate that it "responded quickly." It cannot edit the timestamps in a trace it didn't write. That's the difference between an eval that catches the 80% of boring failures — stale, crashed, empty, too slow — deterministically at Tier 1+2, and a dashboard of green checkmarks that quietly hides every run that limped across the finish line a minute late.

Reserve the judge for the subjective 20% — tone, helpfulness, "did this actually answer the question" — and label it opinion, not evidence. But the stopwatch? That's evidence. Put it first.

Your Agent's Confidence Score Is Not a Probability

Saurav Bhattacharya — Wed, 29 Jul 2026 01:02:15 +0000

Ask an agent how sure it is and it will happily tell you. "Confidence: 0.92." It looks like a probability. It renders nicely in a dashboard. Teams wire it into routing logic: high confidence, auto-approve; low confidence, send to a human. It feels rigorous.

It is not rigorous. A self-reported confidence score is the agent grading its own homework, and it is one of the most seductive false signals in production agentic systems. If you are gating anything on it, you are trusting the defendant's opinion of their own alibi.

Where the number actually comes from

When an LLM emits confidence: 0.92, that number is not a calibrated posterior. It is another token sequence, generated by the same forward pass that produced the answer you are unsure about. It shares a substrate with the output it is describing. If the model hallucinated a file path, the same weights that invented the path will cheerfully assign it 0.9 confidence, because from the inside, a confident fabrication and a confident fact are indistinguishable.

This is not a prompt-engineering problem you can fix with "be honest about your uncertainty." You can push the distribution around, but you cannot make a model's self-report into independent evidence, because there is no independent ground truth in the loop. The grader and the graded are the same network.

The independence axis, applied

This is exactly why, when we build evals at agent-eval, we rank evidence on an independence axis — independent to corruptible — not a cost axis of cheap to expensive. Three tiers:

Tier 1 — externally observable proof the agent can't forge. The JSON parses. The file exists on disk. The code compiled. The tests passed. The call returned within the timeout. The output is non-empty. None of this can be faked by a confident model, because you are checking reality, not asking an opinion.
Tier 2 — statistical signal against a baseline the agent didn't author. Embedding similarity between the output and the actual task. Length and repetition checks. Did the diff actually change anything. The agent didn't write the baseline, so it can't game the comparison.
Tier 3 — model-as-judge. A shared-substrate opinion. A signal, never a verdict.

A self-reported confidence score is not even Tier 3. Tier 3 at least lets a separate model inspect an artifact. Self-confidence is the judged model judging itself in the same breath — maximally circular. It belongs at the very corruptible end of the axis.

Two facts that fall out of this

Tier 1+2 are the real-time gate. They are deterministic, roughly free, and fast, so they can sit in the hot path and block a bad run before it ships. Tier 3 is offline-only: metered, slow, non-deterministic. A judge cannot live in your latency budget. And self-reported confidence, despite looking cheap enough to gate on, is corruptible enough that gating on it is worse than gating on nothing — because it gives you false comfort.

A model judging another model's reasoning is circular. Tier 1+2 can run over agent trajectories. Tier 3 cannot — if you let a judge grade the reasoning steps, judge and judged share a substrate and there is no independent ground truth. So Tier 3 may only inspect artifacts the judged agent didn't get to write. Self-confidence violates this rule harder than anything else: it is a claim about the reasoning, authored by the reasoner.

What to do instead

Don't route on the agent's opinion of itself. Route on independent checks. Here is the shape of it in TypeScript:

type AgentOutput = {
  answer: string;
  filePath: string;
  selfConfidence: number; // present, and deliberately ignored for gating
};

type GateResult = { pass: boolean; tier: 1 | 2; reason: string };

async function gate(out: AgentOutput, task: string): Promise<GateResult> {
  // Tier 1: unforgeable proof
  if (!out.answer.trim()) return { pass: false, tier: 1, reason: "empty" };
  if (!(await fileExists(out.filePath)))
    return { pass: false, tier: 1, reason: "hallucinated path" };

  // Tier 2: statistical signal vs a baseline the agent didn't author
  const sim = await cosineSim(embed(out.answer), embed(task));
  if (sim < 0.55)
    return { pass: false, tier: 2, reason: `off-task (sim=${sim.toFixed(2)})` };

  // out.selfConfidence is never consulted. It's the defendant's alibi.
  return { pass: true, tier: 2, reason: "cleared independent checks" };
}

Notice selfConfidence is present in the type and never read in the gate. That is the point. You can log it, correlate it against outcomes offline, even discover it's anti-correlated with correctness — but it does not get a vote in the hot path.

Ship the 80%, then let a judge whisper about the rest

Most production failures are boring and mechanical: stale data, a crash, malformed output, a hallucinated path, an empty result. Every one of those is caught at Tier 1+2 alone, deterministically, for about zero dollars. Reserve the model-as-judge for the roughly 20% subjective tail — tone, helpfulness, whether an explanation is actually clear — and label its output honestly as "opinion, not evidence." A judge that says 7/10 is a suggestion for a human, not a gate.

You can't gate what you can't see

All of this assumes you can actually inspect what the agent did — the real inputs after resolution, the real tool outputs, the real intermediate steps. That is the other half of the workflow. AgentLens captures the trace: every model and tool step, resolved inputs, raw outputs. agent-eval scores and gates the output; AgentLens gives you the unforgeable, agent-didn't-author trace data for Tier 1+2 to score against — and the debugging surface for when a gate goes red and you need to know why.

The pairing matters here specifically because the whole failure mode of this post is trusting the agent's narration of itself. AgentLens replaces the narration with the record. agent-eval judges the record, not the narration.

Self-reported confidence is the narration. Stop routing on it. Route on proof.

Right Tool, Wrong Arguments: The Agent Failure Your Evals Wave Through

Saurav Bhattacharya — Sun, 26 Jul 2026 01:02:24 +0000

Your agent picked the correct tool. It routed to refund_order when the user asked for a refund. Your eval suite went green. And then it issued a $4,200 refund on order #0 because the argument extraction fumbled and defaulted the ID to zero.

This is the failure mode nobody instruments: right tool, wrong arguments. Most eval setups check whether the agent chose the correct action, then wave the arguments through because validating them looks tedious. That gap is where real money leaks.

Tool selection is the easy 20%

Routing to the right tool is a classification problem, and modern models are good at it. The hard, dangerous part is the arguments — the resolved IDs, amounts, filters, and paths the agent synthesizes from messy context. Those are structured claims the agent authored, and they fail in boring, expensive ways:

A refund amount pulled from the wrong line item.
A file path hallucinated from a plausible-looking directory that doesn't exist.
A date filter off by a timezone, silently returning an empty set that reads as "no results."
An order_id that got coerced to 0 or null and still satisfied a loose schema.

None of these are subjective. None of them need a model to grade them. They are Tier 1 problems — externally observable proof the agent can't forge — and they're getting handed to a model-as-judge (or worse, to production) because teams conflate "did it call the right tool" with "did it call the tool right."

The independence axis, applied to tool calls

The tier doctrine agent-eval is built on ranks evidence by independence, not cost:

Tier 1 — proof the agent can't forge. The arguments parse as valid JSON, satisfy the tool's schema, the referenced order_id actually exists, the amount is within the order total, the path resolves. Deterministic, ~$0, runs in the hot path, can block the call before it executes.
Tier 2 — statistical signal against a baseline the agent didn't author. The extracted amount is within a sane distribution for this merchant; the argument set actually changed from the previous turn; the resolved entity embeds close to the entity named in the user's request.
Tier 3 — model-as-judge. "Given the conversation, does this refund feel justified?" A signal, never a verdict, and offline only — metered, slow, non-deterministic, no business sitting in your payment path.

The mistake is asking Tier 3 to do Tier 1's job. A judge model evaluating whether order_id: 0 is correct is circular reasoning: it shares a substrate with the agent that produced the argument, and it has no independent ground truth about your database. Whether order 0 exists is not an opinion. Query the database.

Gate the arguments before the tool fires

Tier 1 for tool calls is a validation layer that sits between the model's proposed call and execution. Here it is against a proposed refund:

import { z } from "zod";

const RefundArgs = z.object({
  orderId: z.string().regex(/^\d+$/),
  amountCents: z.number().int().positive(),
});

type ToolCall = { name: string; args: unknown };

async function gateRefund(call: ToolCall, db: OrderStore) {
  // Tier 1a: does it even parse to the contract?
  const parsed = RefundArgs.safeParse(call.args);
  if (!parsed.success) {
    return { ok: false, tier: 1, reason: "args_schema_invalid" };
  }
  const { orderId, amountCents } = parsed.data;

  // Tier 1b: does the referenced entity actually exist?
  const order = await db.find(orderId);
  if (!order) {
    return { ok: false, tier: 1, reason: "order_not_found" };
  }

  // Tier 1c: is the amount observably impossible?
  if (amountCents > order.totalCents) {
    return { ok: false, tier: 1, reason: "amount_exceeds_total" };
  }

  // Tier 2: statistically weird, even if legal?
  if (amountCents > 3 * order.merchant.medianRefundCents) {
    return { ok: false, tier: 2, reason: "amount_outlier" };
  }

  return { ok: true };
}

Every check here is proof or statistics, not opinion. Schema, existence, bounds, distribution. The agent cannot talk its way past order_not_found, because the check consults a source the agent didn't get to write. That's the whole point of the independence axis: the gate is only worth anything if the agent couldn't author the evidence it's judged against.

This is the 80% you ship first. Malformed args, hallucinated IDs, empty result sets, out-of-bounds amounts — the failures that actually page you at 2am — are all caught at Tier 1+2, deterministically, before execution. The subjective ~20% ("was a refund the right call socially?") is the only thing you route to a judge, clearly labeled opinion.

You can't gate what you can't see

This all assumes you have the resolved arguments — what the tool actually received after the model's output was parsed, defaulted, and coerced. Most logging captures the user prompt and the final response and nothing in between. When the refund fires on order 0, your logs show a happy path.

This is where the two halves of the workflow lock together. AgentLens captures the trace: every model step and tool step, the resolved inputs the tool actually saw, and the raw outputs — unforged, agent-didn't-author records of what happened. agent-eval scores and gates against that trace using the tier doctrine above. AgentLens gives Tier 1+2 something real to validate; agent-eval turns it into a red/green decision that can block the call.

Without the trace, your evals are grading the story the agent tells about itself. With it, you're grading the arguments it actually passed — and stopping the $4,200 refund on order 0 before it clears.

The takeaway

Stop congratulating your agent for picking the right tool. Tool selection is classification; argument synthesis is where the risk lives. Validate the arguments as proof, not opinion — schema, existence, bounds — in the hot path, before execution. Reserve the judge for the genuinely subjective tail. And trace the resolved inputs, because you cannot gate what you never recorded.

Stop Leaving Findings in the Judge: The Ratchet That Turns Opinions Into Gates

Saurav Bhattacharya — Fri, 24 Jul 2026 01:02:25 +0000

When a model-as-judge flags something real, most teams do the worst possible thing: they leave it in the judge. The finding lives on forever as a slow, metered, non-deterministic Tier 3 opinion — re-litigated on every run, at cost, with a different verdict each time. That's not an eval strategy. That's paying rent on a bug you already found.

The senior move is a ratchet: every recurring Tier 3 finding is a candidate to be promoted into a Tier 1 or Tier 2 check, where it becomes free, fast, deterministic, and able to block the run. This post is about how that ratchet works, and why it's the whole point of having a judge at all.

The independence axis, briefly

Before promotion makes sense, you have to rank evidence the right way. agent-eval ranks it on an independence axis — from independent to corruptible — not a cost axis:

Tier 1 — proof the agent can't forge. Valid JSON, the file exists, it compiled, tests passed, it finished inside the timeout, the output is non-empty.
Tier 2 — statistical signal vs a baseline the agent didn't author. Embedding similarity to the task spec, length and repetition profiles, whether the diff actually changed anything.
Tier 3 — model-as-judge. A shared-substrate opinion. A signal, never a verdict.

Two properties fall out of this and they drive everything below. Tier 1+2 are the real-time gate: deterministic, ~$0, fast, so they can sit in the hot path and block a run. Tier 3 is offline-only: metered, slow, non-deterministic, so it can't. And Tier 1+2 can legitimately run over agent trajectories, while Tier 3 cannot — a model judging another model's reasoning is circular, because judge and judged share a substrate and there's no independent ground truth. So Tier 3 only gets to inspect artifacts the judged agent didn't write.

The ratchet: promote findings down the tiers

Here's the pattern. Your judge keeps flagging the same class of failure — say, "the summary references a section that doesn't exist in the source." That's a real defect. But it's expensive to catch this way, and the verdict wobbles.

Ask: what unforgeable fact would have caught this? In this case, "every cited section header appears verbatim in the source" is a string-membership check. That's Tier 1. You just promoted a subjective opinion into a deterministic gate.

type Finding = {
  claim: string;        // the judge's recurring complaint
  sourceText: string;   // artifact the agent did NOT author
  citedSections: string[];
};

// Tier 1 promotion: the finding is now an unforgeable-proof check.
function citationsExist(f: Finding): { pass: boolean; missing: string[] } {
  const missing = f.citedSections.filter(
    (s) => !f.sourceText.includes(s)
  );
  return { pass: missing.length === 0, missing };
}

// Tier 2 promotion: when membership is too rigid, drop to a
// baseline-relative signal the agent didn't get to author.
function citationGroundedness(
  f: Finding,
  embed: (t: string) => number[],
  cos: (a: number[], b: number[]) => number
): number {
  const src = embed(f.sourceText);
  const scores = f.citedSections.map((s) => cos(embed(s), src));
  return scores.reduce((a, b) => a + b, 0) / (scores.length || 1);
}

citationsExist is Tier 1: it's a fact about text the agent can't forge. When exact membership is too brittle — paraphrased citations, translated sources — you don't jump back to the judge, you drop to Tier 2: citationGroundedness scores similarity against a baseline (the source) the agent didn't write. Still deterministic. Still ~$0. Still allowed in the hot path. Only if neither tier can express the property do you leave it with the judge.

Ship the 80%, meter the 20%

This is where the ratchet pays off. Most failures in production are not subtle: stale outputs, crashes, malformed JSON, hallucinated file paths, empty responses. Every one of those is catchable at Tier 1+2 alone — for free, in the hot path, blocking the run before a user ever sees it.

The judge is for the ~20% subjective tail: tone, whether an explanation is actually helpful, whether a refactor is tasteful. That work is real, but you run it offline, metered, and you label its output honestly: opinion, not evidence. The ratchet steadily shrinks that 20% over time, because every recurring judge complaint that can be expressed as a fact eventually gets promoted out.

This is the line that separates a real eval layer from an "LLM-as-judge gives you a 7/10" tool. A 7/10 is not a gate and it's not reproducible. A promoted Tier 1 check is both.

You can't promote what you can't see

The ratchet has a hard dependency: to promote a finding, you need the raw material — the exact source text, the resolved tool inputs, the actual output the agent produced. This is where the two halves of the workflow lock together.

agent-eval scores and gates the agent's output — the tiers, the drift checks, the hallucination checks above. But it can only score against unforgeable data if that data was captured faithfully. That's AgentLens: it captures the trace of how the agent got there — every model and tool step, the resolved inputs, the raw outputs. Two things follow. First, when a Tier 3 opinion recurs, you open the trace, find the exact artifact the agent didn't author (the source document, the tool response), and that becomes the baseline your new Tier 1/2 check scores against. Second, the trace is itself agent-didn't-author data, which is exactly what Tier 1+2 need to run against without becoming circular.

Without the trace, your judge is guessing and your gate has nothing trustworthy to grade. With it, every recurring opinion is one refactor away from becoming a free, deterministic check.

The takeaway

A model-as-judge is not the top of your eval stack — it's the intake queue for it. Its job is to surface recurring, real defects so you can promote them down the independence axis into checks that are cheaper, faster, and impossible for the agent to forge. agent-eval runs the tiers and the gate; AgentLens gives you the unforgeable trace to build the next gate from. Leave a finding in Tier 3 forever and you're not evaluating — you're just paying, slowly, to be reminded of a bug you could have gated on day one.

The Cold-Start Problem for Agent Evals: What to Gate on Day One With Zero Labeled Data

Saurav Bhattacharya — Wed, 22 Jul 2026 01:02:08 +0000

You just shipped an agent. It works in the demo. Now someone asks the reasonable question: "How do we know it keeps working?" And you reach for evals — and hit a wall. You have no labeled dataset. No golden outputs. No historical traces. Nothing to grade against.

So the team stalls. "We'll add evals once we collect data." Meanwhile the agent runs in production, ungated, and the first time it silently breaks is the first time anyone notices.

This is the cold-start problem, and the usual response — "just get an LLM to score the output 1-10" — is exactly the wrong instinct. You do not need labels to start gating. You need to understand which evidence you can trust on day one, and which you can't trust ever.

Independence, not cost

Most eval discussions rank checks on a cost axis: cheap string matches at the bottom, expensive model-as-judge at the top, as if spending more buys you more truth. That's backwards. The axis that matters is independence: can the agent forge this signal, or not?

That reframing is the whole game for cold-start, because independent evidence needs zero labels. It's true or false about the world regardless of what your agent intended.

Three tiers, ranked independent to corruptible:

Tier 1 — externally observable proof the agent can't forge. Did it produce valid JSON? Does the file it claims to have written exist? Did the code compile? Did the tests pass? Did it finish inside the timeout? Is the output non-empty?
Tier 2 — statistical signal against a baseline the agent didn't author. Is the output embedding-similar to the task it was given? Is the length sane, or did it collapse into repetition? Did the diff actually change anything?
Tier 3 — model-as-judge. A shared-substrate opinion. A signal, never a verdict.

On day one you have Tier 1 and Tier 2 completely for free. Neither needs a single labeled example, because neither asks "is this good?" — they ask "is this real?"

The day-one gate

Here's a starter gate for an agent that's supposed to produce a code patch. No dataset required:

type GateResult = { pass: boolean; tier: 1 | 2; reason: string };

async function coldStartGate(
  task: string,
  output: { patch: string; targetFile: string },
  runtime: { durationMs: number; timeoutMs: number },
  embed: (s: string) => Promise<number[]>,
): Promise<GateResult[]> {
  const results: GateResult[] = [];

  // Tier 1 — unforgeable facts about the world
  results.push({
    tier: 1, reason: "non-empty output",
    pass: output.patch.trim().length > 0,
  });
  results.push({
    tier: 1, reason: "target file exists",
    pass: await fileExists(output.targetFile),
  });
  results.push({
    tier: 1, reason: "patch applies + compiles",
    pass: await applyAndCompile(output.patch, output.targetFile),
  });
  results.push({
    tier: 1, reason: "finished within timeout",
    pass: runtime.durationMs < runtime.timeoutMs,
  });

  // Tier 2 — statistical signal vs a baseline the agent didn't write
  const [taskVec, patchVec] = await Promise.all([embed(task), embed(output.patch)]);
  results.push({
    tier: 2, reason: "patch is on-topic for the task",
    pass: cosine(taskVec, patchVec) > 0.35,
  });
  results.push({
    tier: 2, reason: "diff changed something",
    pass: !/^\s*$/.test(output.patch.replace(/[-+]{3}.*$/gm, "")),
  });

  return results;
}

Every check here is either true about the filesystem/compiler/clock, or it's a distance against the task string — which the agent received but did not author. There is nothing to label. And this catches the overwhelming majority of real failures: the stale run, the crash, the malformed output, the hallucinated file path, the empty response, the patch that wandered off into an unrelated file.

Why Tier 3 stays out on day one (and after)

The temptation is to skip all this and let a judge model read the patch and give it a score. Resist it — and not just because you have no labels.

A model judging another model's work is circular. Judge and judged share a substrate: the same training distribution, the same blind spots, the same confident wrongness. There's no independent ground truth in that loop. So Tier 3 is a signal about taste, never a verdict about correctness, and it may only inspect artifacts the judged agent didn't get to write — never the agent's own reasoning trace, which it can rationalize.

There are two more hard constraints. Tier 1+2 are the real-time gate: deterministic, effectively free, fast enough to block a run before a bad output escapes. Tier 3 is offline-only: metered, slow, non-deterministic — it cannot sit in the hot path. You run it later, in batch, over the ~20% subjective tail that Tier 1+2 can't adjudicate, and you label its output "opinion, not evidence." Ship the 80% you can gate deterministically today; don't block your launch waiting for a judge you shouldn't trust in the loop anyway.

The two halves you actually need

Gating output is only half the job. The other half is knowing what happened, and this is where cold-start teams quietly cheat: they gate on the agent's self-report, which is exactly the forgeable thing Tier 1 is supposed to route around.

This is why the eval layer and the trace layer ship as a unit. agent-eval scores and gates the output — the tier logic above: evals, drift, hallucination checks. AgentLens captures the trace of how the agent got there: every model call and tool step, the resolved inputs, the raw outputs. The two connect at a specific seam: Tier 1+2 need unforgeable data to score against, and the agent must not be the one who wrote it. AgentLens gives you exactly that — the real file that got written, the actual exit code, the true wall-clock duration — instead of the agent's summary of what it thinks it did.

Without the trace, your gate degrades into grading the agent's own press release. With it, "finished within timeout" and "target file exists" become facts, not claims.

Start today

You don't have a labeled dataset. You never will on day one. But you already have a filesystem, a compiler, a clock, and an embedding model — which means you already have a Tier 1+2 gate. Wire agent-eval to it, point AgentLens at your run to feed it unforgeable trace data, and gate the 80%. Collect the judge-tail labels while you're already protected in production, not instead of protecting it.

The cold-start problem was never about missing data. It was about asking the wrong tier for permission to launch.