Multi-Turn AI Conversations in TypeScript: What to Keep, What to Drop

#llm #ai #node #typescript

Book: AI That Answers
The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

The first version of chat history is messages.push(...). It works for a
demo, and it has two failure modes that arrive in order.

First the cost. Every turn resends the entire array, so a fifty-turn
conversation pays for turn one fifty times. Cost per turn climbs linearly and
nobody notices until the monthly total does something surprising.

Then the wall. You hit the context limit and something has to go. Whatever
truncates it at that moment — a slice, a while loop, an SDK helper — is making
a decision about what the AI remembers, and it is making it badly.

Cutting from the front breaks the conversation

The obvious cut is oldest-first:

while (estimateTokens(messages) > LIMIT) {
  messages.shift();
}

Three things go wrong, and only the third throws.

The system context is usually turn one, so the first thing dropped is the
instruction that shaped everything. If your system prompt lives in the array
rather than the system parameter, this deletes it.

The user's original goal is early. In a long debugging conversation the
task was stated at the top; forty turns of detail survive and the point does
not.

And a tool_use block can be dropped while its matching tool_result stays.
That one is a malformed request — a 400 from the API, on a conversation that
was working a minute ago, with an error that names neither the cause nor the
cure.

A window that is a type, not an array

export type Turn =
  | { role: "user"; content: string }
  | { role: "assistant"; content: ContentBlock[] }
  | { role: "user"; content: ToolResultBlockParam[] };

export class Window {
  private turns: Turn[] = [];
  private summary: string | null = null;

  constructor(
    private readonly system: string,
    private readonly budget: number,
    private readonly summarise: Summariser,
  ) {}

  add(t: Turn) { this.turns.push(t); }

  render(): { system: string; messages: MessageParam[] } {
    const head: MessageParam[] = this.summary
      ? [{ role: "user", content: `Earlier in this conversation:\n${this.summary}` },
         { role: "assistant", content: "Understood, continuing." }]
      : [];
    return { system: this.system, messages: [...head, ...this.turns] };
  }
}

Two decisions are already made by that shape.

The system prompt is not in the array. It goes in the system parameter,
so it cannot be evicted and it forms a stable cacheable prefix.

The summary is a user/assistant pair, not a bare user turn. Some models
handle a dangling user message oddly; a completed exchange reads as history.

Compaction that keeps the present

async compact(): Promise<void> {
  if (this.tokens() <= this.budget) return;

  const keep = this.safeTail(6);
  const older = this.turns.slice(0, this.turns.length - keep.length);
  if (older.length === 0) return;

  this.summary = await this.summarise(this.summary, older);
  this.turns = keep;
}

safeTail is the part that prevents the 400:

private safeTail(n: number): Turn[] {
  let start = Math.max(0, this.turns.length - n);
  // never begin on a tool_result whose tool_use was left behind
  while (start < this.turns.length && isToolResult(this.turns[start])) {
    start++;
  }
  return this.turns.slice(start);
}

Walk forward until the tail starts on a complete exchange. Four lines, and it
removes an entire class of production error that only appears in long
conversations, which means only real users find it.

Summarise for continuation, not for reading

A generic "summarise this conversation" gives you prose that reads nicely and
drops what the next turn needs.

const summarise: Summariser = async (prev, turns) => {
  const res = await client.messages.create({
    model: "claude-sonnet-5",
    max_tokens: 700,
    system:
      "You maintain a running brief so a conversation can continue after " +
      "older turns are removed. Output a list. Preserve: the user's goal, " +
      "decisions made and why, identifiers and values supplied, tools " +
      "already run and their outcomes, and anything the user asked you to " +
      "remember. Omit pleasantries and restatement. Under 250 words.",
    messages: [{
      role: "user",
      content: (prev ? `Existing brief:\n${prev}\n\n---\n\n` : "") + render(turns),
    }],
  });
  return textOf(res.content);
};

Feeding prev back in is what stops the tenth compaction being a summary of a
summary of a summary. Each pass revises one living document rather than
re-compressing its own output — the difference between a brief that stays
sharp and one that decays into "the user asked about several things."

"Tools already run and their outcomes" earns its line. Without it the model
re-runs a search it already did, three turns after the result was evicted.

Compact early, not at the wall

const window = new Window(SYSTEM, 0.6 * CONTEXT_LIMIT, summarise);

Sixty percent, not ninety-five. Two reasons.

You need headroom for the next turn, which might include a large tool
result. Compacting at 95% leaves nowhere for it, and you compact again
immediately.

And the summarisation call itself costs tokens. Triggering it at the ceiling
means doing expensive work at the worst moment, on the request a user is
waiting for.

Better still, compact between turns rather than during one:

res.on("finish", () => {
  void window.compact().catch((e) => logger.warn("compaction failed", { e }));
});

The user has their answer; the cleanup happens off the request path. If it
fails, the next turn just compacts inline — degraded, not broken.

What this costs and what it saves

Compaction is a model call, so it is not free. Against fifty turns of
resending the full history, it is not close: one summarisation replaces the
repeated cost of carrying forty evicted turns through every subsequent
request.

The number to watch is tokens per turn over conversation length. Without
compaction it climbs linearly. With it, it sawtooths — climbing, dropping at
each compaction, climbing again, and the average stays roughly flat.

metrics.histogram("chat.tokens_per_turn", tokens, { turnIndex });

If that line still climbs with the window in place, your summary is growing
instead of being replaced, which means the summariser is appending rather than
revising, and the prev handoff above is what fixes it.

The test that matters

it("never renders a tool_result without its tool_use", async () => {
  const w = new Window(SYSTEM, 500, fakeSummariser);
  for (const t of longConversationWithTools) w.add(t);
  await w.compact();

  const { messages } = w.render();
  for (const [i, m] of messages.entries()) {
    if (isToolResult(m)) expect(hasPrecedingToolUse(messages, i)).toBe(true);
  }
});

Injecting a fake summariser keeps this a fast unit test with no model call. It
catches the malformed-request bug before a user has a conversation long enough
to trigger it, which, in practice, is always a real user rather than anyone on
the team.

If this was useful

AI That Answers covers conversation
state properly — window budgets, compaction that keeps decisions, tool-pair
safety, and where the cost of a long chat actually comes from.