Stop Sending the Full Chat History With Every AI Request

Long-running AI conversations can quietly accumulate a lot of unnecessary context.

A common implementation simply appends every message and sends the entire array again:

const messages = [
  systemMessage,
  ...completeConversationHistory,
];

const response = await client.chat.completions.create({
  model: "your-model",
  messages,
});

This works initially, but the request grows after every turn. Token usage can increase, and additional input may also affect latency.

A more deliberate approach

Instead of keeping everything, build the request from the context the model currently needs:

const messages = [
  systemMessage,
  {
    role: "system",
    content: `Summary of earlier conversation: ${conversationSummary}`,
  },
  ...recentRelevantMessages,
];

const response = await client.chat.completions.create({
  model: "your-model",
  messages,
});

A practical context strategy might include:

• Keeping the original system instructions
• Keeping recent and relevant conversation turns
• Summarizing older exchanges
• Preserving tool results, identifiers, and decisions still required by the workflow
• Measuring input tokens before sending the request

Don’t truncate blindly

The smallest possible context is not always the best context.

Removing an important tool result, user constraint, or earlier decision can produce an incorrect response. Context optimization should therefore be tested against response quality—not only token count.

For more complex applications, retrieval can also be used to restore relevant information when it is needed instead of attaching the complete history to every request.

The main idea

Treat the model’s context window as an input you actively manage, not as permanent storage for the entire conversation.

Send enough information for a reliable answer, but avoid repeatedly sending content that no longer contributes to the current task.

APIHubRelay provides an OpenAI-compatible API gateway for working with multiple AI models:

https://apihubrelay.com/

How do you manage long AI conversations in production: summarization, truncation, retrieval, or a combination?