Multi-turn Dialogue: The `messages` Array is Your Agent’s Memory

Introduction

Large language model API endpoints are stateless by design. The LLM itself retains no memory of past conversations between separate HTTP requests. All conversational history, or what developers commonly call “memory”, must be manually passed into the model with every request via the messages array.

This article breaks down the underlying mechanics of multi-turn dialogue for LLM agents. It explains the stateless nature of LLM APIs, the role of each message role, core code logic for building conversation loops, context window and token limits, plus three practical strategies for memory management. We also cover common error codes and troubleshooting practices developers encounter in production LLM applications.

1. Stateless API: Memory Belongs to the Caller

Every single HTTP POST request sent to an LLM API is an isolated transaction. Once the model finishes generating a response, the server discards all session data.

The so-called context or conversation memory is nothing more than the complete conversation log stored in the messages array, delivered by your application on every new API call. Your calling application acts as the only memory store. The model’s context window only defines the maximum total token capacity of the content you send in a single request.

How the messages array grows over successive dialogue rounds:

1. Before round 1: [system]
2. Before round 2: [system, user1, assistant1]
3. Before round 3: [system, user1, assistant1, user2, assistant2]
4. Before round 4: [system, user1, assistant1, user2, assistant2, user3, assistant3]

The system, user, and assistant roles each carry distinct responsibilities:

• system: Sets persona, rules and global instructions once. It is fixed at the start of the array for the whole conversation.
• user: Records messages from human users. A new entry is appended for every new user input.
• assistant: Stores the model’s previous replies. In multi-turn workflows, you must reattach the full prior assistant response back into the array. The model cannot recall its previous outputs unless you feed them back.

This is the core secret of multi-turn dialogue. If you omit the previous assistant message, the model loses track of prior discussion, and will either throw errors or produce irrelevant, out-of-context replies.

2. Core Logic of Multi-turn Conversation

Abstract away streaming UI details, and a full multi-turn request cycle reduces to three simple operations:

// One full dialogue iteration
// Step 1: Append the new user input
$messages[] = ["role" => "user", "content" => $input];
// Step 2: Send the full message list to the model and get reply
$reply = await streamOnce($messages);
// Step 3: Append model reply back into history for next round
$messages[] = ["role" => "assistant", "content" => $reply->text];
// Next iteration repeats: array grows longer, memory accumulates until hitting context limits

A simple experiment demonstrates this principle. If you remove the third line which appends the assistant reply and run multiple rounds, the model will quickly go off-topic. From the model’s perspective, it never saw its earlier responses. That single line is the foundation for continuous multi-turn Agent conversations.

This minimal loop is the basis for nearly all chatbot and Agent implementations. Frameworks like LangChain, LlamaIndex and custom Agent systems all wrap this fundamental logic. The only difference lies in additional layers like memory compression, retrieval-augmented generation, and tool invocation handling.

3. Context Windows and Tokens: Why Conversation History Cannot Grow Indefinitely

Every LLM has a hard input limit known as the context window, measured in tokens. A token is the basic unit the model uses to parse text. As a rough estimation: one Chinese character equals 1–2 tokens, and one English word maps to roughly 1–2 tokens. Note that tokenization behavior varies between model vendors. Always refer to the usage object returned in API responses for precise counting.

Two key token metrics returned in API usage metadata:

• prompt_tokens: Total tokens in the input prompt for the current request, including all conversation history. This value controls both cost and context overflow checks.
• completion_tokens: Tokens generated by the model as output in this turn.

Billing is calculated per token. Input and output tokens often have separate pricing tiers. Longer conversations consume more tokens every round. Multi-turn dialogue is not purely a functional challenge; it creates predictable budget pressure for production applications.

What Happens on Context Overflow

When prompt_tokens approaches or exceeds the context window maximum, the request fails. OpenAI-compatible APIs commonly return HTTP status code 400. The error message includes phrases such as maximum context length or context_length_exceeded.

It helps to group common HTTP error codes for quick diagnosis:

• 401: Authentication / API key error
• 402: Insufficient account balance
• 429: Rate limiting and request throttling
• 400 + context length message: Context window overflow

A context overflow response is not a bug in the model. It signals your memory management logic needs adjustment.

4. Three Core Memory Management Strategies

When conversation history becomes too long, developers have three primary memory-handling approaches. These can be combined in hybrid Agent systems.

Strategy 1: Truncation (Keep System Prompt + N Most Recent Turns)

This approach retains the system instruction and the latest N rounds of dialogue, while discarding the oldest messages. Implementation uses a function that reads backwards from the end of the message array and slices the list.

• Pros: Simple to implement, low compute overhead.
• Cons: Early details and context are permanently lost. The model forgets earlier parts of the conversation.

This method is widely used for simple chatbots and low-complexity customer service agents. It works best when only recent context matters.

Strategy 2: Context Compression / Summarization

Older conversation segments are sent to the LLM and condensed into a short summary paragraph. The summary replaces the original long history in the messages array.

• Pros: Preserves core storyline while cutting token consumption far more effectively than plain truncation.
• Cons: Some fine-grained details may disappear during summarization. Critical facts can be omitted. This adds extra LLM calls and extra cost.

This is the preferred approach for long-running Agent workflows, research assistants, and document analysis agents that need persistent long-term context.

Strategy 3: Reset Conversation by Starting a Fresh Session

When the topic shifts, reset the messages array to contain only the original system prompt. Many “memory corruption” problems can be resolved by clearing history and starting a new session.

• Pros: Fast, zero risk of old context leakage.
• Cons: All prior conversation context is discarded. Users must reintroduce background information.

This is useful for separating unrelated tasks, or resetting state when an Agent enters an unrecoverable broken state.

5. Production Considerations for Multi-turn Agent Memory

The stateless design changes how developers think about Agent reliability. All state persistence, history pruning, token counting, and overflow mitigation must live on the application side.

For teams running multiple LLMs and managing many concurrent agent sessions, 4sapi, an API gateway, can centralize token usage tracking and request routing, simplifying memory telemetry across different model endpoints.

Developers should track token consumption continuously. Blindly appending messages every round without checking prompt_tokens will trigger periodic 400 overflow errors and unexpected cost spikes. Memory logic must run before every API request, not after failure occurs.

Developers should also distinguish between application memory and model context window. External vector databases can store long-term knowledge outside the messages array via RAG, retrieving relevant facts on demand. This offloads the context window and is a common advanced complement to the three memory strategies above.

6. Summary

The core principle is simple: LLM APIs carry no built-in conversational memory. The messages array passed with each request is the Agent’s entire memory store.

1. The system, user, and assistant roles work together to structure dialogue. Previous assistant replies must be fed back to maintain continuity.
2. Conversation loops repeatedly append user input and model responses to the messages array. The array grows until it hits the context window token cap.
3. Context window overflow returns HTTP 400 errors, requiring proactive memory management.
4. Three mainstream memory controls: truncation, summarization compression, and full session reset. These strategies can be mixed for Agent applications.

Understanding this stateless mechanism is foundational for anyone building chatbots, coding agents, research agents, or any multi-turn LLM application. Memory management is not an optional add-on; it is central to stable, cost-effective Agent design.

International access: https://4sapi.com
Domestic access: https://4sapi.cn