Your AI Agent's Context Window Is Full — What Should It Forget?

#typescript #agents #ai #node

Book: AI That Plans
The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

An agent that runs long enough fills its window. Tool results are the reason —
a single document fetch or query result can be thousands of tokens, and every
one of them is resent on every subsequent turn.

The naive fix is dropping the oldest messages. That works for a chat and
breaks an agent, because the oldest message is usually the task.

The question is not how much to drop. It is which parts are recoverable.

Rank the history by whether it can be reconstructed

Four tiers, and they are not equally expensive to lose.

Never drop — the task. The system prompt and the original request. Lose
these and the agent finishes something else. They should not even be in the
compaction path.

Never drop — irreversible facts. "Refund issued", "email sent", "ticket

4471 created". Dropping these is how an agent does something twice. These are

not chat history; they belong in structured state, not in the message list.

Compact — reasoning. Intermediate thinking about steps already completed.
The conclusions matter, the deliberation does not.

Drop or externalise — bulk tool results. The 4,000-token document the
agent read six turns ago. This is almost all of the pressure and almost none
of the value.

export type Retention = "pin" | "state" | "compact" | "evict";

export function classify(m: Msg): Retention {
  if (m.role === "system" || m.meta?.isOriginalTask) return "pin";
  if (m.meta?.effect === "irreversible") return "state";
  if (m.role === "tool" && (m.tokens ?? 0) > 500) return "evict";
  return "compact";
}

Tagging at write time rather than guessing at compaction time is what makes
this reliable. When a tool with a side effect returns, mark it — the
classifier should read a flag, not a string.

Evict the body, keep the receipt

The mistake in dropping a tool result is dropping the fact it happened. If the
call disappears entirely, the agent re-runs it, and you have built a loop.

function evict(m: ToolMsg): ToolMsg {
  return {
    ...m,
    content:
      `[result evicted — ${m.tokens} tokens]\n` +
      `tool: ${m.name}(${summariseArgs(m.args)})\n` +
      `outcome: ${m.summary}\n` +
      `retrieve with: recall_result("${m.id}")`,
    tokens: 60,
  };
}

Three things survive: that the call happened, what it broadly returned, and
how to get it back. The recall_result tool is real — a lookup into the run's
own store:

const recall = tool({
  name: "recall_result",
  schema: z.object({ resultId: z.string() }),
  async run({ resultId }, ctx) {
    const raw = await ctx.store.getResult(ctx.runId, resultId);
    return raw ?? "That result is no longer available.";
  },
});

Now eviction is reversible. The agent that genuinely needs the document back
can ask for it, and in practice it rarely does, which is the evidence the
eviction was correct.

Compact reasoning into a written summary

When pinned + state + recent still exceeds the target, summarise the middle:

async function compact(history: Msg[], ctx: Ctx): Promise<Msg[]> {
  const pinned = history.filter((m) => classify(m) === "pin");
  const recent = history.slice(-6);
  const middle = history.slice(pinned.length, -6);
  if (middle.length < 4) return history;

  const summary = await ctx.model.complete({
    system: COMPACT_PROMPT,
    messages: [{ role: "user", content: render(middle) }],
    maxTokens: 700,
  });

  return [
    ...pinned,
    { role: "user", content: `Summary of earlier work:\n${summary}`,
      meta: { compacted: middle.length } },
    ...recent,
  ];
}

The summariser prompt is where this succeeds or fails. It should be told
explicitly what to preserve:

Preserve: facts discovered, decisions made and why, actions already taken and
their outcomes, anything the user asked for that is not yet done, and
constraints the user stated.
Drop: deliberation, discarded approaches, restatements.
Write in the past tense. Do not add anything not present in the input.

Without that last line the summariser invents progress, and an agent that
believes it already refunded the order will not refund it.

Trigger on a budget, not on an error

Compacting when the API rejects the request is too late — the run has already
failed once and the latency is visible.

const LIMIT = 200_000;
const TARGET = 0.65;                       // compact at 65% of the window

if (estimateTokens(history) > LIMIT * TARGET) {
  history = await compact(history, ctx);
}

65% rather than 90% because the next turn adds a model response and a tool
result, and a large one can cross the remaining gap in a single step.

Use the provider's token-counting endpoint for the estimate where one exists —
character heuristics are wrong in exactly the case that matters, since JSON
tool results tokenise far worse than prose.

Compaction breaks your prompt cache

Worth knowing before you tune the threshold. Prompt caching works on a stable
prefix; rewriting the middle of the history invalidates everything after the
first changed token.

So the turn after a compaction is a full-price turn. That argues for
compacting rarely and hard rather than continuously trimming:

// good: one large compaction, cache rebuilds once
if (used > LIMIT * 0.65) history = await compact(history, ctx, { keep: 6 });

// bad: shaves a little each turn, cache never survives
if (used > LIMIT * 0.5) history = history.slice(1);

Eviction has the same property, which is a reason to evict in batches when you
compact rather than opportunistically per turn.

The structural fix: keep bulk out of the window

Everything above is damage control. The durable answer is not putting large
results in the window at all.

async function run(args, ctx) {
  const doc = await fetchDoc(args.id);
  const id = await ctx.store.putResult(ctx.runId, doc);   // full text, off-window

  return {
    resultId: id,
    title: doc.title,
    tokens: doc.tokens,
    excerpt: doc.text.slice(0, 800),
    note: "Call read_section(resultId, heading) for more.",
  };
}

A tool that returns a handle plus an excerpt, with a second tool to read
further. The agent pulls what it needs; the window holds a few hundred tokens
instead of four thousand.

Applied to the two or three highest-volume tools, this usually removes the
compaction problem rather than managing it.

What to watch

metrics.gauge("agent.window_used_pct", used / LIMIT, { tool: lastTool });
metrics.increment("agent.compaction", 1, { turns: history.length });
metrics.increment("agent.recall_result", 1);

The third one is the interesting number. Frequent recall_result calls mean
you are evicting things the agent still needs — either the summaries are too
thin, or that tool should be returning handles instead.

If this was useful

AI That Plans covers context
management for long-running agents — what to pin, what to move into state,
compaction that does not lose the task, and tools that keep bulk out of the
window.