Originally published on tamiz.pro.
You've spent weeks tuning your system prompt. You've tried chain-of-thought, ReAct, and tree-of-thought prompting. You've benchmarked GPT-4o against Claude 3.5 Sonnet and nothing clicks. Your agent still loses context, contradicts itself across turns, and feels like it's starting fresh every time a user comes back.
Here's the uncomfortable truth: your agent doesn't have a reasoning problem. It has a memory problem.
The models available today are remarkably capable at reasoning within the context window they're given. The gap between a "smart" agent and a "breaks after three turns" agent almost never traces back to the model's logical capabilities. It traces back to what the agent remembers, how it remembers it, and whether that memory survives the transition from demo to production.
This is a deep-dive into production-grade agent state — the architecture, patterns, and trade-offs that separate prototypes from systems that handle real users across real sessions.
1. The Reasoning Myth
Before we fix the memory problem, let's kill the reasoning narrative once and for all.
What the benchmarks actually measure
When we evaluate LLM reasoning, we're measuring something narrow: given a prompt and a constrained set of context tokens, how well does the model solve a defined problem? Chain-of-thought papers, MATH benchmarks, GPQA, LiveCodeBench — these are all stateless evaluations. The model sees the question, reasons through it, and produces an answer. Nothing persists. Nothing accumulates.
Real agent work is fundamentally different. An agent operates across multiple turns, multiple tools, and distributions of information that arrive incrementally. The reasoning challenge isn't "can the model think?" — it's "can the model think about what it already knows while also figuring out what to do next?"
The stateless model, the stateful world
Consider this conversation:
User (turn 1): "I need to book a flight from SFO to NYC next Tuesday for under $400."
Agent: Calls search tool → finds flights → returns results
User (turn 2): "Which one has the shortest layover?"
Agent: Calls another tool or reasons from prior results
User (turn 3): "Actually, change the destination to Newark."
Agent: ...what does it know about the original request?
At turn 3, the agent needs to reconcile a modified goal against previous tool results, intermediate conclusions, and user preferences expressed across turns. That's not a reasoning deficiency — that's a state management deficiency. No amount of prompt engineering on the model's reasoning ability fixes this. The information simply isn't there to reason about.
Why "just give it more context" doesn't work
You could throw every prior turn into the context window and hope the model tracks it. This fails in production for three reasons:
Context window is expensive. Every token you send costs money. A 10-turn conversation with tool results can easily consume 15,000–50,000 tokens per call. At scale, this is bankrupting.
Context window is noisy. Retrieving 40K tokens of conversation history doesn't mean the model attends to the right 40K tokens. Attention mechanisms dilute across long contexts. Critical details from turn 1 get buried.
Context window doesn't persist across sessions. When the user returns tomorrow, last week's conversation is gone unless you explicitly saved it somewhere and retrieved it again.
The solution isn't bigger windows. It's better memory architecture.
2. What Agent State Actually Is
Agent state is not a single thing. It's a composite of several distinct but interacting layers. Confusing these layers is the root cause of most production failures.
2.1 Working Memory (Episodic)
The short-lived, session-bound state that drives the current interaction. This includes:
- Current goal and sub-goals
- Tool call history (what was called, what returned)
- Intermediate conclusions and plans
- User intent as expressed so far in the conversation
This is typically held in the context window and refreshed every turn. It's fast, flexible, and ephemeral.
2.2 Semantic Memory (Declarative)
Longer-lived knowledge about the world that the agent needs to reference repeatedly:
- User profile and preferences ("Alice prefers aisle seats, travels light")
- Domain facts (company policies, product catalogs, API schemas)
- Learned procedures ("when booking flights, always check baggage allowance")
This lives outside the context window, usually in a database or vector store, and is retrieved on demand.
2.3 Procedural Memory (Skill-based)
The agent's repertoire of actions and their outcomes:
- Tool definitions and schemas
- Successful action sequences ("to book a flight: search → compare → select → book → confirm")
- Error recovery patterns ("if the booking fails with code 402, retry with different dates")
This is typically encoded in function/tool definitions but should also include learned heuristics that improve over time.
2.4 Source Memory (Provenance)
Where each piece of information came from — critical for trust and debugging:
- Which tool produced which fact?
- When was this information last updated?
- What's the confidence level?
Without source memory, agents confidently hallucinate details they "remembered" but can't verify. This is the difference between an agent that says "Based on your profile..." and one that says "You told me on March 3rd that you prefer..."
3. The Production Memory Architecture
A production-grade agent memory system has four layers, each serving a different latency and persistence profile.
Layer 1: The Context Buffer (Milliseconds)
The active working memory that gets injected into every LLM call. This is not raw conversation history — it's a curated buffer that the agent maintains and updates.
type ContextBuffer = {
// Current session state
sessionId: string;
turnCount: number;
// Active goal stack
currentGoal: GoalState;
subgoals: Subgoal[];
completedSubgoals: Subgoal[];
// Tool call ledger (compact, not raw logs)
toolLedger: ToolEvent[];
// Extracted facts for this turn
extractedFacts: Fact[];
// Conversation summary (rolling, for when we trim)
rollingSummary: string;
};
The key insight: the agent writes to this buffer as it works, not just receives it. After each tool call, the agent should update the buffer with structured results, not just dump raw output into the context.
Layer 2: The Retrieval Index (10–100ms)
Semantic and procedural memory lives here. When the agent needs to recall something beyond its current context, it queries this layer.
interface MemoryRetriever {
// Semantic recall — find relevant facts
recallSemantics(query: string, userId: string): Promise<Fact[]>;
// Episodic recall — find similar past interactions
recallEpisodes(similarTo: string, limit: number): Promise<Episode[]>;
// Procedural recall — find relevant tools/patterns
recallProcedures(taskType: string): Promise<ToolDefinition[]>;
}
Implementation options:
- Vector embeddings for semantic recall (pgvector, Weaviate, Pinecone)
- Embedding-based episode retrieval for episodic recall (hash the conversation, store embeddings of key moments)
- Metadata-filtered tool registry for procedural recall (tag tools by capability, not just name)
Layer 3: The Persistence Layer (Seconds)
Long-term storage with lifecycle management. This is where memories go to survive, age, and eventually get pruned.
interface MemoryStore {
// Write with TTL and priority
write(entry: MemoryEntry): Promise<void>;
// Compaction — merge redundant memories
compact(userId: string): Promise<number>;
// Expiry — remove stale memories
expire(maxAgeDays: number): Promise<number>;
// Audit trail for provenance
getProvenance(memoryId: string): Promise<ProvenanceRecord>;
}
interface MemoryEntry {
id: string;
type: 'fact' | 'preference' | 'event' | 'procedure';
content: string;
metadata: Record<string, unknown>;
source: SourceReference;
createdAt: Date;
expiresAt?: Date;
confidence: number; // 0–1, for uncertain memories
accessCount: number; // for recency-based eviction
}
Layer 4: The Forgetting Mechanism
This is the layer everyone skips. Memory without forgetting is a liability. Stale preferences, outdated goals, and redundant facts crowd out signal. A production system needs explicit forgetting:
- Temporal decay: Memories older than their useful lifespan get flagged for removal
- Confidence decay: Memories with low confidence scores that haven't been reinforced get evicted
- Competitive eviction: When storage thresholds are reached, the least-accessed, lowest-confidence memories are candidates for removal
- Explicit user correction: When a user says "I don't actually prefer that anymore," the system should actively supersede the old memory, not just accumulate a new one
4. The Agent Memory Loop
How does all of this work in practice? Here's the production-grade memory loop that runs on every agent turn:
┌─────────────────────────────────────────────────┐
│ TURN START │
│ │
│ 1. USER INPUT arrives │
│ ↓ │
│ 2. QUERY RETRIEVAL │
│ ┌─→ Semantic recall (facts, preferences) │
│ ├─→ Episodic recall (similar past turns) │
│ └─→ Procedural recall (relevant tools) │
│ ↓ │
│ 3. CONTEXT COMPOSING │
│ ┌─→ Current working memory (buffer) │
│ ├─→ Retrieved memories │
│ ├─→ Rolling summary (compressed history) │
│ └─→ System prompt + tool definitions │
│ ↓ │
│ 4. LLM CALL │
│ (agent reasons over composed context) │
│ ↓ │
│ 5. ACTION EXECUTION │
│ Tool calls → results → validate │
│ ↓ │
│ 6. MEMORY UPDATE │
│ ┌─→ Write new facts to persistence │
│ ├─→ Update working buffer │
│ ├─→ Compact/expire stale memories │
│ └─→ Update source provenance │
│ ↓ │
│ 7. RESPONSE │
│ Format and return to user │
└─────────────────────────────────────────────────┘
Step-by-step breakdown
Step 2 — Query Retrieval: Before the LLM sees anything, the system fetches relevant memories. The query for retrieval isn't the user's raw input — it's a meta-query generated from the user input plus current goals. This two-step process (generate query → retrieve memories) prevents irrelevant memories from polluting the retrieval.
async function buildRetrievalQuery(
userInput: string,
workingMemory: ContextBuffer
): Promise<string> {
// Use a lightweight model to generate a focused retrieval query
const queryGen = await llm.call({
model: 'fast-model', // cheaper, faster model for query generation
messages: [
{ role: 'system', content: RETRIEVAL_QUERY_SYSTEM_PROMPT },
{ role: 'user', content: `Goal: ${workingMemory.currentGoal.description}\nInput: ${userInput}` }
]
});
return queryGen.content;
}
Step 3 — Context Composing: This is where most agents fail. The context isn't just "concatenate everything." It's a structured assembly with priorities:
- System prompt (fixed, high priority)
- Current goal and subgoals (from working memory)
- Retrieved facts (ranked by relevance score)
- Tool definitions (only the ones relevant to current subgoals)
- Rolling summary (compressed history, max 500 tokens)
- Recent tool results (last 3–5 calls, full detail)
The rolling summary is critical. Instead of keeping raw conversation history, periodically (every 5–10 turns) compress the history into a concise summary using the LLM itself:
async function compressHistory(
rawHistory: Message[],
workingMemory: ContextBuffer
): Promise<string> {
const summary = await llm.call({
model: 'summary-model',
messages: [
{ role: 'system', content: COMPRESS_SYSTEM_PROMPT },
{ role: 'user', content: formatMessages(rawHistory) }
],
maxTokens: 500
});
return summary.content;
}
Step 6 — Memory Update: After the agent acts, the system extracts and persists new knowledge:
async function updateMemories(
turnResult: AgentTurnResult,
workingMemory: ContextBuffer,
memoryStore: MemoryStore
): Promise<void> {
// Extract new facts from the turn
const newFacts = await extractFacts(turnResult);
for (const fact of newFacts) {
await memoryStore.write({
id: crypto.randomUUID(),
type: classifyFact(fact),
content: fact.text,
metadata: fact.metadata,
source: { type: 'extraction', confidence: fact.confidence },
createdAt: new Date(),
confidence: fact.confidence
});
}
// Update working buffer
workingMemory.extractedFacts.push(...newFacts);
workingMemory.turnCount++;
// Periodic compaction
if (workingMemory.turnCount % 10 === 0) {
await memoryStore.compact(workingMemory.sessionId);
}
}
5. The Extraction Problem
The hardest part of agent memory isn't storage — it's knowing what to store. Raw conversation is noisy. You can't persist everything. You need to extract signal from the noise.
What to extract
| Signal Type | Example | Priority |
|---|---|---|
| User preferences | "I prefer morning flights" | High |
| Explicit facts | "My order number is ORD-12345" | High |
| Goal states | "Looking for a refund, not an exchange" | Medium |
| Tool outcomes | "Flight search returned 3 results under $400" | Medium |
| Contextual hints | Conversation tone, urgency signals | Low |
| Chatter | "Thanks!" "Got it" "No wait" | None |
Extraction pipeline
Raw Turn Output
↓
┌──────────────┐
│ Classifier │ Is this worth remembering?
└──────┬───────┘
↓ yes
┌──────────────┐
│ Extractor │ Pull out structured facts
└──────┬───────┘
↓
┌──────────────┐
│ Verifier │ Check against existing memories
│ (dedup) │ Avoid storing "SFO to NYC" when
└──────┬───────┘ "SFO to New York" already exists
↓
┌──────────────┐
│ Prioritizer │ Assign confidence, TTL, type
└──────┬───────┘
↓
┌──────────────┐
│ Persister │ Write to memory store
└──────────────┘
The extractor prompt
const EXTRACTION_PROMPT = `
You are extracting facts from a conversation between a user and an AI agent.
Extract ONLY the following types of information:
1. User preferences (travel, dietary, scheduling, etc.)
2. Explicitly stated facts (order numbers, dates, names)
3. Active goals and their status
4. Tool results that constrain future decisions
DO NOT extract:
- Casual conversation fillers
- Requests the agent already fulfilled in this turn
- Information already stored in existing memories (you'll be provided them)
Existing memories for this user:
{{existingMemories}}
Conversation:
{{conversation}}
Return a JSON array of facts. Each fact: {type, content, confidence (0-1), expiresAt (null if permanent)}.
`;
6. Provenance and Trust
An agent that can't cite its sources is an agent that will confidently hallucinate. Production systems need provenance tracking — every memory must be traceable to its origin.
Why provenance matters
Self-correction: When an agent says "Based on your previous request...", it should be able to point to which previous request. If the user corrects it, the system knows what to update.
Debugging: When an agent makes a wrong decision, you need to know whether it reasoned poorly or remembered poorly. These are different bugs requiring different fixes.
User trust: Users can detect when an agent is making things up. A system that says "You mentioned this on March 3rd" vs "I think you might have said..." builds different levels of trust.
Provenance schema
interface ProvenanceRecord {
memoryId: string;
sourceType: 'user-stated' | 'tool-result' | 'inferred' | 'system-injected';
sourceId: string; // references the original message, tool call, or system event
timestamp: Date;
confidence: number;
overwrittenBy?: string; // if this memory was superseded
}
When the agent retrieves a memory, the provenance record should be part of the retrieved context, not hidden metadata. The agent needs to know how it knows something, so it can express appropriate certainty.
7. Memory-Efficient Context Management
Even with good extraction and retrieval, you'll hit context limits. Here's how production systems handle it.
The sliding window with summary anchors
Instead of truncating conversation history at the beginning (losing the oldest, potentially critical context), use summary anchors:
[Summary of turns 1–12: "User booked flight SFO→NYC, order ORD-999. Prefers aisle seats."]
[Summary of turns 13–24: "User requested change to Newark. Refund initiated."]
[Turn 25: ...]
[Turn 26: ...]
[Turn 27: ...]
The summaries preserve the meaning of earlier turns without the token cost. You can implement this with periodic summarization triggered by turn count or context budget.
Context budgeting
interface ContextBudget {
totalTokens: number; // e.g., 128_000 for GPT-4o
reservedForSystem: number; // ~2,000 — prompt, tool defs
reservedForTools: number; // ~8,000 — relevant tool results
reservedForSummary: number; // ~1,000 — rolling summary
budgetForRetrieval: number; // remaining — dynamic based on retrieval score
}
Every turn, the context composer checks the budget and decides: how many retrieved memories can I include? If the budget is tight, only include memories above a relevance threshold.
8. Cross-Session Memory
The hardest case: the user returns days later. The working memory from last time is gone. How does the agent reconnect?
Session linking
interface SessionLink {
currentSessionId: string;
previousSessionId: string;
linkReason: 'same_user' | 'similar_goal' | 'referenced_context';
bridgeSummary: string; // "Last time, we were booking a flight to NYC..."
linkedAt: Date;
}
When a new session starts, the system checks for potential links:
- Identity link: Same user ID → load their full memory profile
- Goal link: Similar current goal to a recent session → load that session's bridge summary
- Context link: User mentions something from a past interaction → retrieve related memories
The bridge summary
The bridge summary is the single most important artifact for cross-session continuity. It's a 2–3 sentence condensation of what happened in the previous session, generated at session close:
"Last session: You were booking a flight from SFO to NYC for next Tuesday.
We found three options under $400. You were deciding between the United
red-eye and the Delta morning flight. The conversation ended before a
selection was made."
This bridge gets injected into the first turn of the next session, giving the agent immediate continuity without reconstructing the entire conversation.
9. Common Failure Modes
The echo chamber
When the agent's own outputs become part of its memory, it creates feedback loops. It "remembers" things it generated rather than things the user stated. Mitigation: source tagging. Every memory entry must carry a sourceType — and the agent's own reasoning outputs should never be written to long-term memory without user confirmation.
The hoarding problem
Agents that remember everything remember nothing useful. Without compaction and forgetting, the retrieval index becomes noisy and relevant memories get drowned out. Mitigation: TTL-based eviction and confidence-weighted pruning.
The false continuity
When the agent assumes continuity where none exists — mixing up conversations, attributing statements to the wrong session. Mitigation: session-scoped memory by default, with explicit cross-session linking only when verified.
The extraction blind spot
When the extractor misses important information because it's embedded in an indirect statement. "I'd prefer not to fly United" is a preference, but a naive extractor might miss it among the conversational noise. Mitigation: multi-pass extraction — first pass for explicit facts, second pass for implicit preferences, with different prompts for each.
10. Tooling and Implementation Patterns
The memory middleware pattern
Wrap your agent framework with a memory middleware layer that intercepts every turn:
class AgentWithMemory {
constructor(
private agent: BaseAgent,
private memoryStore: MemoryStore,
private retriever: MemoryRetriever,
private contextBuffer: ContextBuffer
) {}
async execute(userInput: string, sessionId: string): Promise<AgentResponse> {
// 1. Retrieve relevant memories
const retrievalQuery = await this.buildRetrievalQuery(userInput);
const memories = await this.retriever.recall(retrievalQuery, sessionId);
// 2. Compose enriched context
const enrichedContext = this.composeContext(
this.contextBuffer, memories, userInput
);
// 3. Execute agent
const response = await this.agent.execute(enrichedContext);
// 4. Update memories
await this.updateMemories(response, sessionId);
return response;
}
}
Integration with popular frameworks
| Framework | Memory Approach | Notes |
|---|---|---|
| LangGraph | StateGraph with persistent state | Best for complex multi-step agents; use Checkpointers for persistence |
| LangChain | ConversationBufferMemory, VectorStoreRetrieverMemory | Good primitives but requires manual composition |
| LlamaIndex | QueryEngine + ChatEngine with memory | Strong retrieval integration, weaker on state management |
| CrewAI | Agent memory via shared context | Simpler but less control over memory lifecycle |
| Custom (recommended for prod) | Hand-rolled middleware | Full control over every layer described above |
For production systems, the trend is toward custom middleware rather than out-of-the-box memory solutions. The frameworks provide building blocks, but the architecture described here — with explicit layers, provenance, compaction, and cross-session linking — requires orchestration that no framework provides today.
11. Measuring Memory Health
You can't improve what you don't measure. Track these metrics:
- Recall precision: Of the memories retrieved, how many were actually relevant to the agent's decision? (Measured via user feedback or automated evaluation)
- Memory retention rate: What percentage of extracted memories survive past their intended TTL?
- Context utilization: What percentage of the context window is actually used for useful information vs. noise?
- Cross-session continuity score: When users return, does the agent demonstrate awareness of prior interactions? (Automated or sampled manual evaluation)
- Memory compaction ratio: How many raw facts get merged or pruned during compaction? High ratios suggest poor initial extraction.
Frequently Asked Questions
Q: Can I just use a vector database and call it memory?
No. Vector databases give you retrieval, not memory. Memory requires extraction, provenance tracking, compaction, TTL management, and cross-session linking — none of which a vector store provides by default. A vector database is one component of a memory system, not the system itself.
Q: How much memory should I pre-fetch vs. on-demand?
Pre-fetch semantic memories (user preferences, domain facts) for every turn — the latency is acceptable and the relevance is high. Fetch episodic memories (past conversations) on-demand based on the retrieval query. Don't pre-fetch procedural memories (tool definitions) — load only the tools relevant to the current goal to save context tokens.
Q: What's the minimum viable memory system for a production agent?
Three things: (1) a structured context buffer that the agent writes to each turn, (2) a retrieval layer for semantic memories (user preferences + domain facts), and (3) a rolling summary mechanism to handle context window limits. Get these right before adding cross-session linking, provenance tracking, or compaction. Those are optimizations, not foundations.
The models have the reasoning. The missing piece is always the memory. Build the memory system, and the reasoning problems mostly solve themselves.
Top comments (0)