A chat only grows but the context window is fixed — context management is choosing what survives

#ai #chatgpt #llm #promptengineering

A context window is a hard token budget, and a conversation only ever grows. Left alone, every chat eventually crosses the limit and the request is rejected or silently truncated. Context management is the fix: before each request you token-budget the history and decide which turns to keep verbatim, which to drop, and which to compress. I built a small manager that runs in front of every model call. Here's how it works.

Count tokens, not characters

You can't budget what you can't measure, and it's tokens — not characters — that fill the window. Use the real tokenizer so your count matches the model's; a chars/4 approximation is fine for a first pass but undercounts code and non-English, so keep headroom. Count every message plus a small per-message overhead.

const countMsg = m => enc.encode(m.content).length + 4;  // ~4 tok overhead
const countAll = msgs => msgs.reduce((n, m) => n + countMsg(m), 0);

Budget for the reply, not just the input

The window holds input and output. Budget the history to window − reserveForReply − systemTokens so you never crowd out the answer. The system prompt is spent first and always; everything else competes for what's left. Forget the reply reserve and a full history can technically "fit" and still leave the model no room to respond.

Truncate-oldest: cheap and blunt

The simplest trimmer walks from newest to oldest, keeping messages while they fit and dropping the rest. It's stateless and cheap — but blunt: it will happily drop the early turn that held your requirements, and if the budget is tight enough, even the system prompt. Always keep the system prompt outside this loop.

function truncateOldest(history, budget){
  const kept = []; let used = 0;
  for (let i = history.length - 1; i >= 0; i--){
    const t = countMsg(history[i]);
    if (used + t > budget) break;         // older ones dropped
    kept.unshift(history[i]); used += t;
  }
  return kept;
}

Sliding window and summarize-old

A sliding window keeps the last N turns regardless of size — predictable and dead simple, but count-based not token-based, so a couple of long recent turns can still overflow and a short history wastes budget. Summarize-old is the smarter middle: keep the last K turns verbatim and fold everything older into one running summary with a cheap model. It preserves the gist of the whole conversation for a tiny token cost, trading verbatim detail for reach. Re-summarize incrementally as the tail grows so cost stays flat.

Pinned + recent — the workhorse

Reserve budget for pinned messages first — the system prompt and any flagged key facts — then fill the remainder with the newest turns. Pins are never dropped, so a requirement stated in turn 1 survives no matter how long the chat runs. That's the direct fix for truncate's worst failure: forget the stack you chose at the very start and the final "remind me what we picked" question becomes unanswerable — summarizing and pinning keep it, truncate and a plain window lose it.

Put it together

In production you combine the moves — pin the system prompt, summarize the old middle, keep a recent window, and verify the whole thing fits before sending, every turn. The tradeoff is the whole game: drop or over-compress and the model loses the detail it needs to stay coherent; keep everything and you blow the budget. This is distinct from the system prompt (what you pin) and from prompt caching (reuse for cost) — it's choosing what survives so the conversation still fits. Run the manager in front of every model call and the history can grow forever while the window never overflows.

async function manageContext(system, history, summarize){
  const budget = historyBudget(system);
  let msgs = await summarizeOld(history, 6, summarize);   // fold the old middle
  if (countAll(msgs) > budget) msgs = truncateOldest(msgs, budget);
  return [system, ...msgs];                                // system pinned, first
}