- Book: AI Agents Pocket Guide: Patterns for Building Autonomous Systems with LLMs
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
Your support agent has been chatting with a customer for forty turns. On turn three the customer said their account email is the work one, not the personal one. On turn thirty-eight the agent emails the reset link to the personal address anyway. The customer is annoyed. You pull the trace. The model never saw turn three. By turn thirty-eight that message had scrolled out of the context window, and nothing put it back.
This is the memory problem, and it does not go away when context windows get bigger. A 200K-token window still fills up in a long session, still costs you on every turn for tokens you reread, and still buries the one fact that mattered under forty turns of small talk. Bigger windows move the wall further out. They do not remove it.
You have three ways to deal with it. Rolling summarization, vector recall, and the hybrid that uses both. They are not interchangeable, and picking the wrong one shows up as either a fat bill or an agent that forgets the thing you told it to remember.
Why the naive approach falls over
The naive memory is "send the whole transcript every turn." It works for the first ten turns and then the math turns on you.
A chat model bills you for every input token on every call. If the conversation is T tokens after turn N, and each turn adds t tokens, then turn N costs you the sum of all prior turns. Across the whole session you pay roughly N² in cumulative input tokens, because each message gets reread once per remaining turn.
Put real numbers on it. A 50-turn session where each turn adds 400 tokens of conversation reaches 20,000 tokens by the end. The cumulative input billed across all 50 turns is around 500,000 tokens, not 20,000. You paid 25x for the privilege of rereading. That is the cost summarization attacks.
Rolling summarization
Keep the last few turns verbatim. Everything older gets folded into a running summary that the model rewrites as the conversation grows. The prompt you send is: summary, plus the recent verbatim turns, plus the new message.
def build_context(summary, recent_turns, new_msg):
parts = []
if summary:
parts.append(
{"role": "system",
"content": f"Conversation so far:\n{summary}"}
)
parts.extend(recent_turns)
parts.append({"role": "user", "content": new_msg})
return parts
When the verbatim window grows past its limit, you compress the oldest turns into the summary with a cheap model call.
def maybe_compress(summary, recent_turns,
model_call, keep=6):
if len(recent_turns) <= keep:
return summary, recent_turns
old = recent_turns[:-keep]
prompt = (
"Update the running summary with these turns. "
"Keep names, IDs, decisions, and open questions "
f"verbatim.\n\nSummary:\n{summary}\n\n"
f"Turns:\n{format_turns(old)}"
)
new_summary = model_call(prompt)
return new_summary, recent_turns[-keep:]
The win is bounded context. Your prompt size stops growing with N. A 200-turn session sends about the same number of tokens as a 20-turn one. The bill goes flat instead of quadratic.
The loss is detail. Summarization is lossy by design. The model decides what to keep, and it will drop the thing you needed. "Customer mentioned a second account" survives a summary three times. By the fourth rewrite it is gone, because nothing reinforced it. Summaries also drift: each rewrite is a copy of a copy, and small distortions compound.
Vector recall
Vector recall flips the storage model. You do not summarize anything. You embed every turn, store the vectors, and on each new turn you retrieve the k most similar past messages and inject only those.
def remember(store, embed, turn_id, text):
store.add(turn_id, embed(text), text)
def recall(store, embed, query, k=5):
hits = store.search(embed(query), k=k)
return [h.text for h in hits]
The prompt becomes: the k recalled turns, plus the recent verbatim turns, plus the new message. The recalled turns can be from turn 3 or turn 300; distance in the conversation does not matter, only semantic similarity to what the user just asked.
This is where recall beats summary. The turn-three account email is sitting in the store with its full text intact. When the customer asks about the reset on turn thirty-eight, "send reset link" embeds close to "my account email is the work one," and the original message comes back verbatim. A summary would have paraphrased or dropped it. The vector store kept the exact words.
Recall has its own failure mode. It only returns what the query is similar to. Ask a question that does not semantically resemble the fact you need, and the fact stays buried. Recall has no sense of recency or conversational flow either. It will happily skip the message from one turn ago because the message from turn five scored higher. And you now run an embedding model and a vector store, which is infrastructure the summary approach does not need.
When recall beats summary, and when it does not
The split is about what kind of memory the task needs.
Reach for summarization when the agent needs the gist of the conversation: the overall goal, the tone, the decisions made, where things stand. A planning agent, a long coding session, a multi-step research task. The narrative thread matters more than any single sentence. Summaries are cheap, need no extra infrastructure, and keep the model oriented.
Reach for vector recall when the agent needs specific facts pulled from a large history: the account ID from turn three, the constraint the user stated forty turns ago, the preference buried in a tangent. Support agents, personal assistants with months of history, anything where the exact words of an old message decide the next action. Recall keeps facts lossless; summaries blur them.
The honest answer for most production agents is both.
The hybrid, and the token-budget math
The hybrid runs a rolling summary for the narrative and vector recall for the facts. Your prompt has four parts: the running summary (gist), the top-k recalled turns (facts), the recent verbatim turns (flow), and the new message.
You have to budget tokens across those parts or you have rebuilt the unbounded prompt with extra steps. A workable split for an 8,000-token context allowance:
- Summary: 1,000 tokens (hard cap; compress when over)
- Recalled turns:
k=5at ~200 each, so 1,000 tokens - Recent verbatim: last 6 turns, ~2,400 tokens
- New message and headroom: the rest
That holds the prompt near 4,400 tokens regardless of whether the session is on turn 10 or turn 500. Flat cost, lossless facts, intact narrative. You pay for it with two extra moving parts: a summarizer call every few turns and an embed-plus-search on every turn.
A 40-line memory manager
Here is the whole thing. It keeps a rolling summary, embeds every turn into a store, recalls the relevant ones, and assembles a bounded prompt. Drop in any embed, any vector store with add/search, and any model_call.
from dataclasses import dataclass, field
@dataclass
class Memory:
embed: callable
store: object
model_call: callable
summary: str = ""
recent: list = field(default_factory=list)
keep: int = 6
k: int = 5
def add_turn(self, turn_id, role, text):
self.store.add(turn_id, self.embed(text), text)
self.recent.append({"role": role, "content": text})
if len(self.recent) > self.keep:
self._compress()
def _compress(self):
old = self.recent[:-self.keep]
prompt = (
"Update the running summary. Keep names, IDs, "
"and decisions verbatim.\n\n"
f"Summary:\n{self.summary}\n\n"
f"Turns:\n{old}"
)
self.summary = self.model_call(prompt)
self.recent = self.recent[-self.keep:]
def context_for(self, query):
hits = self.store.search(self.embed(query), k=self.k)
recalled = [h.text for h in hits]
msgs = []
if self.summary:
msgs.append({"role": "system",
"content": f"So far:\n{self.summary}"})
if recalled:
msgs.append({"role": "system",
"content": "Relevant past:\n"
+ "\n".join(recalled)})
msgs.extend(self.recent)
msgs.append({"role": "user", "content": query})
return msgs
Two things to watch. The recalled turns can duplicate what is already in recent, so dedupe by turn ID before you assemble the prompt if you care about the wasted tokens. And the summarizer call inside _compress should use a cheaper, faster model than your main agent; it is doing rote compression, not reasoning, and you call it often enough that the price difference adds up.
What to measure
Memory quality is hard to eyeball, so instrument it. Log, per turn: which recalled turn IDs came back, the prompt token count, and whether the summary was rewritten. Then watch two things over a week of real sessions.
First, prompt token count over session length. It should stay flat. If it climbs with turn count, a budget is leaking somewhere. Second, recall hit usefulness: sample sessions where the agent gave a wrong answer and check whether the fact it needed was in the store but not recalled. A high miss rate there means your k is too low or your embedding model is wrong for the domain, and that is the dial to turn before you touch anything else.
Memory is not a feature you finish. It is a budget you tune against real traffic, the same way you tune iteration caps and cost ceilings on the loop itself.
If you want the longer version of this, with the eviction policies, the dedupe logic, and how memory interacts with tool results in a real loop, that is one of the chapters in the AI Agents Pocket Guide. It covers the patterns this post sketches and the ones that only show up once an agent has been running for a few weeks.
Top comments (0)