Taming Context Bloat: How to Scale AI Agent Memory Without Breaking the Token Bank

Stop dumping raw message arrays into LLMs and start using structured state with sliding windows.

The Bottleneck in Production

The most common mistake when deploying AI agents is treating chat history as an append-only log. In early prototypes, appending every user turn, tool response, and raw JSON blob directly into the messages array works fine.

In production, this pattern collapses after twenty turns. Token usage scales linearly with conversation depth, driving up API latency and inference costs. Worse, models experience "lost-in-the-middle" degradation, forgetting early constraints or crashing altogether due to token limit errors.

# The naive anti-pattern: unbounded list growth
messages.append({"role": "user", "content": user_input})
messages.append({"role": "assistant", "content": llm_response})
# 30 turns later: 15,000 tokens wasted on stale tool payloads
response = client.chat.completions.create(model="gpt-4o", messages=messages)

Dumping unpruned histories into your LLM turns your database into an expensive latency trap.

---

The System Architecture & Fix

The solution is decoupling ephemeral dialogue from persistent conversational state.

Instead of forcing the LLM to re-parse the entire conversation history on every turn to understand what happened ten minutes ago, we split context into two distinct layers:

1. Sliding Window Buffer: Retain only the last $N$ turns to preserve immediate conversational rhythm and flow.
2. Structured State JSON: Maintain a compact, deterministic key-value store of critical facts (e.g., active user goals, collected form values, resolved errors).

   Incoming User Turn
           │
           ▼
┌─────────────────────────────────────────┐
│            Context Assembler            │
│ ─────────────────────────────────────── │
│ 1. Static System Prompt (Identity)      │
│ 2. Current State JSON (Facts & Goals)   │
│ 3. Sliding Window Buffer (Last N Turns) │
└─────────────────────────────────────────┘
           │
           ▼
     LLM Inference (Bounded & Predictable)

This ensures your token payload stays flat whether a session lasts 3 turns or 300 turns.

---

The Implementation

Here is a lightweight context manager you can drop directly into your backend service pipeline.

from typing import Any, Dict, List

def build_bounded_context(
    system_prompt: str,
    raw_history: List[Dict[str, str]],
    state_payload: Dict[str, Any],
    max_turns: int = 6
) -> List[Dict[str, str]]:
    """Assemble a token-bounded context payload with structured state."""
    # Enforce strict sliding window on ephemeral chat history
    trimmed_history = raw_history[-max_turns:] if len(raw_history) > max_turns else raw_history

    # Inject current state directly as a system-level context injection
    state_injection = {
        "role": "system",
        "content": f"CURRENT_SESSION_STATE: {state_payload}"
    }

    return [{"role": "system", "content": system_prompt}, state_injection] + trimmed_history

This pattern provides deterministic context bounds. Your backend guarantees that the context size passed to the provider never exceeds your calculated budget:

If an agent needs to update persistent state (like a shipping address or user intent), extract that state asynchronously or via tool calls, store it in your database, and inject the clean JSON dictionary on the next invocation.

---

Production Lessons & Takeaways

• State belongs in your database, not the context window: Treat the LLM context like RAM and your database like a hard drive. Extract facts into structured fields instead of making the model re-read 40 turns of history.
• Keep $N$ between 4 and 8 turns: In practice, immediate dialogue context rarely requires more than the last 3-4 round trips. Anything older is usually noise.
• Make state updates idempotent: Update your structured state schema using explicit tool/function calls rather than free-form LLM summaries to avoid hallucinated state drift.

Comments

[anassBld]

The tiered-memory split is useful, but I think it needs one more boundary: conversation state and external-effect state cannot share the same truth model.

A summarized preference can be revised later. A tool write such as “payment sent” or “record deleted” needs immutable fields like source, authority, attempt_id, observed_at, and an explicit outcome_unknown state. Otherwise a timeout can leave the durable layer storing the agent's inference as fact, and the next compressed context makes that mistake look authoritative.

The hard case is two writes that disagree after a tool timeout: one run records success from model output while a later read-back finds no remote effect. In your architecture, which layer owns reconciliation, and what prevents the stale “success” memory from winning the next retrieval?