Your agent eval suite grades one turn at a time. Prompt in, output out, score it, move on. That model is fine for a completion endpoint. It is quietly wrong for anything that holds a conversation, and it is the reason your "94% pass rate" coexists with users who rage-quit on turn six.
The unit of failure for a conversational agent is not the turn. It is the session.
The per-turn blind spot
Consider a support agent. Turn 1: user asks about a refund. The agent answers correctly and passes every eval. Turn 4: user clarifies they mean a partial refund. The agent answers that correctly too. Turn 6: the agent quotes the original refund amount again, having silently dropped the "partial" constraint three turns ago.
Every single turn passes an isolated eval. Each output is grounded, well-formatted, on-topic. The failure only exists across turns: a dropped constraint, a contradiction with turn 4, a promise never kept. Per-turn grading is structurally blind to it, because the bug is in the relationship between outputs, not in any one of them.
This is not an exotic edge case. Constraint decay, self-contradiction, and forgotten commitments are the dominant failure class for multi-turn agents. And they are invisible to the eval architecture most teams actually run.
Where session-level signal lives on the tier ladder
If you have read anything I have written, you know I rank eval evidence on an independence axis, not a cost axis, from evidence the agent cannot forge down to opinion it shares a substrate with:
- Tier 1: externally observable proof the agent cannot fake. Valid JSON, a file that exists, tests that passed, finished within timeout, non-empty.
- Tier 2: statistical signal vs a baseline the agent did not author. Embedding similarity, repetition, whether a diff actually changed anything.
- Tier 3: model-as-judge. Shared-substrate opinion. A signal, never a verdict.
The mistake teams make with multi-turn is assuming session evaluation is inherently a Tier 3 problem, "just ask a judge if the whole conversation was coherent." It is not. A huge fraction of session failures are Tier 1 and Tier 2, if you look at the trajectory instead of the last message:
- Did turn 6's amount contradict a value the agent itself committed to in turn 4? That is a deterministic diff over structured extractions, Tier 1. No judge needed.
- Did a constraint the user set ("partial", "in EUR", "before Friday") survive to the final output? Set-membership check over resolved slots, Tier 1/2.
- Did the agent repeat an earlier answer verbatim instead of advancing? Repetition and embedding-similarity across turns, Tier 2.
- Did every user question get a corresponding answer, or did one get dropped? Count and coverage check, Tier 1.
These run over the trajectory, deterministically, at ~$0, fast enough to gate. And Tier 1+2 are allowed to run over trajectories precisely because they do not share a substrate with the agent. Tier 3 cannot: a model judging another model's multi-turn reasoning is circular, because judge and judged share the same failure priors, so there is no independent ground truth. Reserve the judge for the genuinely subjective session tail ("was the tone appropriate across the escalation?"), offline, clearly labeled opinion.
You cannot check a trajectory you did not capture
Here is the operational catch: every Tier 1/2 session check above needs the trace, the resolved inputs, the committed values, the tool outputs at each step. Not a summary the agent wrote. The actual sequence.
This is why the two halves ship as one workflow. AgentLens captures the trace: every model and tool step, the resolved inputs, the raw outputs, unforgeable and agent-did-not-author records of what actually happened turn by turn. agent-eval scores and gates against that trace using the tier doctrine above. The eval is only as trustworthy as the trace it runs on, and the trace is only useful if something scores it. Trace without eval is a debugging log; eval without trace is a judge guessing.
Here is a Tier 1 session gate, contradiction detection over committed values, that needs zero model calls:
import { z } from "zod";
const RefundCommitment = z.object({
turn: z.number(),
amount: z.number(),
scope: z.enum(["full", "partial"]),
});
type Commitment = z.infer<typeof RefundCommitment>;
// Pulled from the AgentLens trace: what the agent actually committed to,
// per turn, not what it claims in its final summary.
function evalSessionConsistency(commitments: Commitment[]) {
const failures: string[] = [];
for (let i = 1; i < commitments.length; i++) {
const prev = commitments[i - 1];
const cur = commitments[i];
// A later turn silently reverting an earlier committed scope/amount
if (cur.scope !== prev.scope && cur.amount === prev.amount) {
failures.push(
`turn ${cur.turn}: scope changed ${prev.scope}->${cur.scope} ` +
`but amount stayed ${cur.amount} (stale value?)`
);
}
}
return {
tier: 1 as const,
passed: failures.length === 0,
failures, // deterministic, ~$0, safe to block the run
};
}
No judge scored that. It is a diff over trace data the agent could not author, and it catches the exact turn-6 bug that six green per-turn evals waved through.
Ship the 80%
The reflex is to throw a smart judge at multi-turn coherence. Resist it. Most session failures, dropped constraints, contradictions, repeated answers, unanswered questions, are caught deterministically at Tier 1+2 over the trajectory, at ~$0, fast enough to block before the bad turn ships. That is the 80%.
Reserve the model-as-judge for the ~20% subjective tail, tone, empathy, whether an escalation felt handled, offline, metered, and labeled for what it is: opinion, not evidence.
Grade the session, not the sentence. Capture the trajectory with AgentLens, gate it with agent-eval, and stop letting a wall of green per-turn checks certify a conversation that fell apart on turn six.
Top comments (1)
Turn-level success is a weak metric for agents. The real failure often shows up across the conversation: accumulated assumptions, stale goals, missed reversals, and small confident choices that make the final state wrong.