Your Agent's Context Window Overflowed and It Answered Anyway

Your agent works great in the demo. Then someone hands it a real ticket with a 40-message thread, three attached logs, and a stack trace, and it confidently answers using the first half of the context — because the second half fell off the back of the window. No error. No exception. Just a quietly wrong answer with full confidence. This is context overflow, and it is one of the most under-instrumented failure modes in production agents.

Here is the uncomfortable part: your model-as-judge will not catch it, and it shouldn't be asked to. Context truncation is not a subjective quality problem. It is an observable, deterministic fact about what actually entered the model. You can prove it happened. That makes it a Tier 1 problem, and treating it like one changes everything about how you defend against it.

Why this is invisible

Most agent frameworks silently truncate. You assemble a prompt from system instructions, retrieved documents, tool outputs, and conversation history, and if it exceeds the window, the framework (or the provider) drops the overflow — usually from the middle or the oldest turns. The model still returns a fluent, plausible response. Your evals are green. Your users get answers built on a partial view of the problem.

The reason this slips through is that teams evaluate the output text and never inspect the resolved input. They ask "did the answer look good?" instead of "did the evidence the agent needed actually make it into the call?" Those are different questions, and only one of them is answerable without opinion.

The independence axis, not the cost axis

This is where I want to plant a flag, because it's the thing that separates real agent evaluation from "LLM-as-judge gives you a 7/10" tooling. Evidence should be ranked on an independence axis — from independent to corruptible — not a cost axis of cheap to expensive.

• Tier 1 — externally observable proof the agent can't forge. Did the input fit the window? Did the required document ID appear in the resolved prompt? Was the tool output non-empty? Did the run finish inside its timeout? These are facts. Valid or not, present or not, truncated or not.
• Tier 2 — statistical signal against a baseline the agent didn't author. Is the retrieved chunk actually similar to the task embedding? Did the token count spike 3x versus the rolling baseline for this task type? Did the diff change anything?
• Tier 3 — model-as-judge. A shared-substrate opinion. A signal, never a verdict.

Context overflow lives squarely in Tier 1. You don't need a smarter model to tell you the prompt didn't fit — you need to measure the resolved prompt.

There's a second reason Tier 3 is the wrong tool here, and it's structural: a model judging what another model saw is circular. Judge and judged share a substrate; there's no independent ground truth in that loop. Tier 1 and Tier 2 can run over the agent's actual trajectory precisely because they inspect artifacts the agent didn't get to write — the byte count of the assembled prompt, the presence of a chunk ID, the embedding of the input. The judge can only offer opinion about text, and it should only ever inspect artifacts the judged agent didn't author.

Instrument the resolved input

The fix is boring and effective: capture the fully-resolved prompt at the moment of the call and gate on it deterministically, before you spend a judge token.

import { encoding_for_model } from "tiktoken";

interface ResolvedCall {
  taskId: string;
  requiredIds: string[];      // doc/chunk IDs this task needs
  resolvedPrompt: string;     // the ACTUAL assembled prompt sent
  windowLimit: number;
}

interface GateResult {
  tier: 1 | 2;
  pass: boolean;
  reason: string;
}

export function contextGate(call: ResolvedCall): GateResult[] {
  const enc = encoding_for_model("gpt-4o");
  const tokens = enc.encode(call.resolvedPrompt).length;
  enc.free();

  const results: GateResult[] = [];

  // Tier 1: did it physically fit?
  results.push({
    tier: 1,
    pass: tokens <= call.windowLimit,
    reason: `resolved prompt = ${tokens} tokens (limit ${call.windowLimit})`,
  });

  // Tier 1: did every required piece of evidence survive assembly?
  const missing = call.requiredIds.filter(
    (id) => !call.resolvedPrompt.includes(id),
  );
  results.push({
    tier: 1,
    pass: missing.length === 0,
    reason: missing.length
      ? `dropped required evidence: ${missing.join(", ")}`
      : "all required evidence present",
  });

  return results;
}

Notice what this does not do: it doesn't ask whether the answer was good. It asks whether the answer was even possible given what entered the model. If a required chunk ID never made it into the resolved prompt, the run is dead on arrival — block it, don't grade it. This is the real-time gate: deterministic, roughly free, fast enough to sit in the hot path and stop a bad run before it ships.

The two halves: score the output, trace the run

You cannot gate on the resolved prompt if you never captured it. This is why evaluation and tracing ship as one workflow, not two products.

agent-eval scores and gates the output using the tier doctrine above — it's the thing that knows "context overflowed" is a Tier 1 red, not a judge's vibe. AgentLens captures the trace of how the agent got there: every model and tool step, the resolved inputs, the raw outputs. That trace is what gives Tier 1 and Tier 2 something unforgeable to score against, because the agent didn't author its own trace — the harness did. Without the trace, "did the required evidence enter the window?" is unanswerable. With it, it's a two-line check.

Run them together and the division of labor is clean: AgentLens records the ground truth of the run, agent-eval decides whether that ground truth clears the gate.

Ship the 80%

Most agent failures are not subtle disagreements a judge must arbitrate. They are stale caches, crashes, malformed JSON, hallucinated file paths, empty tool results — and context that silently overflowed. All of that is caught at Tier 1 and Tier 2 alone, deterministically, at roughly zero cost, in the hot path. Reserve the model-as-judge for the genuinely subjective ~20% tail, and label its output honestly: opinion, not evidence.

The context window isn't a config detail. It's the boundary of what your agent could possibly know at inference time. Measure that boundary, gate on it, and you'll retire a whole category of confident-but-wrong answers before a judge ever needs an opinion.