Don't Truncate Chat: Token-Aware Context Compressor

Long chats fail on free model endpoints because history grows faster than the context window, not because the model is weak. I compress old turns into one summary, keep recent turns verbatim, and send a smaller message array so the model still remembers names, decisions, and requirements.

Why long chats forget, contradict, and repeat

Every turn appends to the message array. Attention spreads across thousands of tokens. By turn 30 the model asks for a requirement I stated in turn 1, then contradicts it. That is context overflow. Free endpoints make it worse: they usually expose smaller windows than paid tiers, so overflow arrives earlier.

Research on long-context models, including Lost in the Middle, shows models underuse information buried in the middle of a long prompt. Dumping the full transcript is not more memory. It is more noise.

I see the same failure pattern every time:

• Turn 1: the user states a requirement.
• Turn 15: the model asks for that requirement again.
• Turn 30: the model contradicts turn 1.

Truncation vs a bigger window

Two naive fixes look tempting. Neither is enough on a free tier.

Truncation keeps only the last N messages. It is fast and cheap. It also drops facts: requirements, decisions, names, numbers. The model becomes fluent and amnesiac.

A bigger context window is rarely an option on free endpoints. I work with the window I have.

Compression sits between those two. Old turns become one summary message. Recent turns stay verbatim. The conversation fits the window, and the facts survive.

I follow three rules:

1. Count tokens before every request.
2. When the total crosses a threshold, summarize the oldest turns.
3. Keep the most recent turns untouched.

Count tokens, then compress at 70% of the window

Most free endpoints do not expose a token counter. OpenAI's token guide explains why counts differ by tokenizer, and tiktoken is the accurate path when I can use it. On a generic free endpoint I use a heuristic: about four characters per token. I only need a trigger, not an audit.

function estimateTokens(text) {
  return Math.ceil(text.length / 4);
}

function countMessages(messages) {
  return messages.reduce((sum, m) => sum + estimateTokens(m.content), 0);
}

I pick a fraction of the context window. 70% is a safe start. Below that, the model still has room to generate a response.

const WINDOW_SIZE = 8000;        // your endpoint's context window
const COMPRESS_AT = 0.7;         // compress when history exceeds 70%
const KEEP_RECENT = 6;           // keep the last 6 turns verbatim

function shouldCompress(messages) {
  const total = countMessages(messages);
  return total > WINDOW_SIZE * COMPRESS_AT;
}

Tune WINDOW_SIZE to the endpoint. Raise KEEP_RECENT if the chat is code-heavy or a planted fact keeps vanishing from the summary. Code uses more tokens per character than English, so the four-character heuristic will under-count a repo walkthrough. When that happens I lower COMPRESS_AT rather than pretending the estimate is exact.

Summarize old turns and rebuild the message array

I call the same model with a compression prompt. I ask for facts, not prose: names, decisions, numbers, requirements, open questions. I ban bullets and commentary so the summary stays dense. A chatty recap wastes the tokens I just tried to save.

const COMPRESS_PROMPT = `
Summarize the conversation so far.
Keep names, decisions, numbers, requirements, and open questions.
Output plain text. No bullets. No commentary.
`;

async function summarizeHistory(modelUrl, oldTurns) {
  const response = await fetch(modelUrl, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      messages: [
        { role: 'system', content: COMPRESS_PROMPT },
        ...oldTurns,
      ],
    }),
  });

  const data = await response.json();
  return data.choices[0].message.content;
}

One compression call costs tokens. That is the tradeoff: I spend a little once to save a lot on every future request.

Old turns become one system message. Recent turns stay in the array.

async function compressHistory(messages, modelUrl) {
  if (messages.length <= KEEP_RECENT) return messages;

  const splitAt = messages.length - KEEP_RECENT;
  const oldTurns = messages.slice(0, splitAt);
  const recentTurns = messages.slice(splitAt);

  const summary = await summarizeHistory(modelUrl, oldTurns);

  return [
    { role: 'system', content: `Summary of earlier conversation:\n${summary}` },
    ...recentTurns,
  ];
}

The summary carries facts. It does not carry verbatim wording. That is the point.

I wire this as middleware between the client and the model endpoint. Compression runs before the request is sent. The model never sees the oversized history.

app.post('/chat', async (req, res) => {
  let messages = req.body.messages;

  if (shouldCompress(messages)) {
    messages = await compressHistory(messages, MODEL_URL);
  }

  const response = await fetch(MODEL_URL, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ messages }),
  });

  const data = await response.json();
  res.json(data);
});

Verify with a planted fact, then ship behind a passing test

A compressor that loses facts is worse than truncation. I plant a fact, bury it under filler turns, then ask for it after compression.

const testMessages = [
  { role: 'user', content: 'My name is Ada. The budget is $500. The deadline is Friday.' },
  { role: 'assistant', content: 'Got it. $500 budget, Friday deadline.' },
];

for (let i = 0; i < 20; i++) {
  testMessages.push(
    { role: 'user', content: `Filler question ${i}: what is the status?` },
    { role: 'assistant', content: `Filler answer ${i}: all tasks are on track.` }
  );
}

testMessages.push({ role: 'user', content: 'What is my name and what is the deadline?' });

I run the compressed conversation through the model. If the answer contains "Ada" and "Friday", the compressor works. If not, I raise KEEP_RECENT or tighten the compression prompt. I add this check to the test suite and run it after every change. A regression here is silent. The model will not tell me it forgot. It will just answer wrong.

The compressor is middleware. It wraps every request to the model endpoint. It runs anywhere Node.js runs.

MonkeyCode is an open-source project that provides free model access and a free server tier. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free server is enough to run this middleware and test it against a real conversation.

I deploy in this order:

1. Set MODEL_URL and WINDOW_SIZE as environment variables.
2. Point the chat app at the middleware.
3. Run the planted-fact test against the deployed version.
4. Route real traffic only after that test passes.

Summarization is lossy. A summary can drop a number, a name, or a nuance. The longer the conversation, the more compression, the more loss. The token estimate is a heuristic: different languages compress differently, and code-heavy chats use more tokens per character than plain English. Compression also adds latency—one extra model call per compression event. For interactive chat that is acceptable. For high-frequency automation it may not be.

Skip this pattern when:

• The app needs exact recall of every message. Use a vector store instead.
• Chats stay shorter than ten turns. The overhead is not worth it.
• The team has a strict latency budget. The compression call adds a full round trip.

The context window is fixed. The conversation does not have to be. Compress the past. Keep the present.

If you are building a chat app on a free endpoint, ship this compressor next: paste the middleware, plant a fact, run the twenty-turn test, and only then point production traffic at it. That passing test is the difference between a model that remembers and one that repeats itself.