Originally published at heycc.cn. This is a mirrored copy — the canonical version is kept up to date at the source.
How AI Agent Memory Works in 2026: Context, Episodic, Semantic, and Procedural
A large language model is stateless. Each API call starts from zero, and everything the model "knows" about your task has to fit inside the prompt you send. An agent feels stateful — it remembers your name, the bug it fixed yesterday, your preference for tabs over spaces — only because something around the model is storing information and feeding the right slice back in at the right time. That something is the agent's memory system.
By mid-2026 there is finally enough convergence across vendors and research to draw a clean map. This guide separates two ideas people constantly conflate — the context window (short-term, in-prompt working memory) and long-term memory (the durable store outside the window) — then breaks long-term memory into the episodic / semantic / procedural taxonomy that LangChain's LangMem, Letta, and Mem0 have all adopted. It closes with the four techniques for keeping the active window healthy and a side-by-side of how the major frameworks actually implement memory.
Where these facts come from (read 2026-06-28): the figures and API surfaces below were read off the official Anthropic context-engineering and context-management pages, the Mem0 docs and 2025 system paper, the LangMem conceptual guide, and the Letta docs, all linked in Sources. The 84% figure, Mem0's add/update/delete surface, and LangMem's algorithm names are all version-sensitive, so the specifics below are a 2026-06-28 snapshot rather than fixed constants.
Short-term vs long-term: the core distinction
| Dimension | Short-term (context window) | Long-term memory |
|---|---|---|
| Where it lives | Inside the prompt, in-window | External store (files, vector DB, KV, graph) |
| Lifespan | One request / one session | Across sessions, indefinitely |
| Cost model | Billed per token, every turn | Storage + retrieval embedding cost |
| Capacity | Fixed window (finite) | Effectively unbounded |
| Failure mode | "Context rot," lost-in-the-middle | Stale facts, bad retrieval |
| Access | Automatic (model attends to all) | Must be retrieved and injected |
The window is fast and automatic but small and expensive. Long-term memory is large and cheap to keep but only helps if you retrieve the right item and inject it back into the window. Almost every design decision in agent memory is about moving information across this boundary efficiently.
Why the window isn't just "make it bigger"
Anthropic's context-engineering guidance frames context as a finite resource with diminishing marginal returns. It describes "context rot": as the number of tokens in the window increases, the model's ability to accurately recall any specific item decreases. Anthropic attributes this partly to the transformer architecture itself — its guidance notes that attention forms n² pairwise relationships across tokens, which become "stretched thin" across larger contexts, and that models have seen fewer training examples of very-long-range dependencies. The practical upshot is theirs, not mine: a bigger window does not linearly buy you more usable recall.
The "Lost in the Middle" study (Liu et al., TACL 2024; arXiv preprint 2023) put numbers on a related effect: model accuracy is highest when the relevant fact sits at the very start or end of the input and degrades sharply when it is buried in the middle — a U-shaped curve. The takeaway for builders is blunt: you cannot dump everything into a long prompt and assume the model will use it. Placement and curation matter as much as capacity. This is the same reason teams still reach for retrieval even with huge windows — see the trade-offs in RAG vs. Long Context.
The three types of long-term memory
The taxonomy borrowed from cognitive science — now shared across LangMem, Letta, and Mem0 — splits durable memory into three kinds. They differ in what they store and how they get used.
| Type | What it stores | Example | Typical write | Typical read |
|---|---|---|---|---|
| Semantic | Facts and relationships | "User's prod DB is Postgres 16" | Extract on the fly or in background | Vector similarity search |
| Episodic | Past experiences / interactions | A solved ticket kept as a few-shot example | Distill from a finished episode | Retrieve similar past episodes |
| Procedural | How to do tasks; rules & skills | "Always run tests before committing" | Reflect and rewrite instructions | Loaded into the system prompt |
Semantic memory is the knowledge layer: discrete facts and their relationships that ground future responses. LangMem represents these as either collections (unbounded searchable knowledge) or profiles (a strict, looked-up schema) and retrieves them by semantic similarity from a vector store.
Episodic memory records specific past interactions. LangMem captures "the full context of an interaction — the situation, the thought process that led to success, and why that approach worked," often distilling those episodes into few-shot examples. The agent learns by recalling "the last time this kind of thing came up, here's what worked."
Procedural memory is internalized know-how — generalized skills, rules, and behaviors. LangMem describes procedural memory as a combination of model weights, agent code, and the prompt, but its SDK "focus[es] on saving learned procedures as updated instructions in the agent's prompt." In other words, the agent rewrites its own operating manual over time.
Vector memory and the add/update/delete decision
Most semantic memory in production is backed by a vector database: text is embedded, stored, and later retrieved by cosine similarity to the query (so your embedding choice sets the ceiling on recall). But naive "embed everything, retrieve top-K" produces duplicate, contradictory, and stale memories. The interesting systems add a reasoning step.
Mem0 is a good reference design, though the algorithm itself changed shape partway through 2026. Its 2025 system paper describes the original add(): two LLM calls, an extraction pass that pulls salient facts from the conversation, then an update pass that retrieves the top-K semantically similar existing memories and asks an LLM, via function-calling, to pick one action per candidate — ADD (genuinely new fact), UPDATE (augment or correct an existing memory), DELETE (the new fact contradicts and obsoletes an old one), or NOOP (a repeat or irrelevant; change nothing).
Mem0's v2.0.0 release (April 16, 2026) replaced that with single-pass, ADD-only extraction: one LLM call pulls out new facts and appends them, with no automatic UPDATE/DELETE reconciliation against what's already stored. Mem0 reports this roughly halves extraction latency and lifts LoCoMo and LongMemEval scores by 20-plus points. The trade-off is that the store no longer self-corrects by default: deduplication on this path is exact-match (MD5 hash) only, and a maintainer closed a bug report confirming that semantically contradictory facts — two different stated names for the same person, in the reported case — are both kept as separate memories rather than reconciled. update() and delete() still exist, but as calls you make explicitly rather than a decision the pipeline makes for you. The open-source build has a second sharp edge on top of that: raw inserts (infer=False) skip extraction entirely, so mixing infer=False and infer=True writes for the same fact can duplicate it. "Coherent store" is now even more a property of the pipeline you build around Mem0 than of the database itself.
Mem0's Graph Memory goes further: during the add pipeline it extracts entities (proper nouns, quoted text, compound noun phrases) into a parallel collection and links memories that share an entity. At search time the query's entities are matched against that collection to boost the score of connected memories. This lets the agent reason across separate facts — "you mentioned Acme, and Acme's lead is Dana, who prefers Slack" — that pure vector similarity would miss because no single chunk contains the whole chain.
Managing the context window: four techniques
Long-term memory is only half the story. The other half is keeping the active window healthy on long-running tasks. Four techniques dominate in 2026, and the strongest agents combine them.
Compaction — When a conversation nears the window limit, summarize its contents and reinitialize a fresh window seeded with that summary. This is typically implemented by the agent, not by the platform: Anthropic notes that "In Claude Code… we implement this by passing the message history to the model to summarize and compress the most critical details." That makes it distinct from the server-side context editing in technique #2. The advice for the compaction prompt is to "start by maximizing recall… then iterate to improve precision by eliminating superfluous content" — get everything important into the summary first, then trim. (Recall-then-precision is guidance about how to write that summarization prompt, not a property of any automatic feature.)
Context editing — Programmatically clear stale tool calls and results from the window as you approach token limits. This is the one piece Anthropic ships as an automatic platform behavior: per its context-management post it "automatically clears stale tool calls and results from within the context window when approaching token limits." In a 100-turn web-search evaluation, Anthropic reports context editing cut token consumption by 84% and enabled workflows that would otherwise fail from context exhaustion. (Internal Anthropic eval, not an independent benchmark.)
Structured note-taking (agentic memory) — Instead of holding everything in-prompt, the agent regularly writes notes to a store outside the window and reads them back on demand. A running TODO file, a
progress.md, or scratchpad entries survive compaction and let the agent re-anchor after the window is reset. Anthropic shipped a file-based memory tool for exactly this, where the model issues CRUD-style file operations and the files live client-side (you host them).Sub-agents with clean context — Spin up a specialized sub-agent with its own fresh window to handle a focused task, and have it return only a condensed, distilled summary to the parent. The expensive intermediate reasoning — the dozens of tool calls and dead ends — never pollutes the main window; only the conclusion crosses back. This is how multi-agent research setups keep a coordinator's context lean while still doing deep work underneath.
Anthropic reports that combining the memory tool with context editing improved performance by 39% over baseline on an internal agentic-search evaluation — concrete (if vendor-supplied) evidence that window hygiene is not a micro-optimization.
Compact the resolved middle, not the live tail
One rule keeps compaction safe in practice: compress the resolved middle, keep the live tail verbatim. Closed-out tool calls, stack traces already acted on, and dead-end hypotheses are exactly what a summary should collapse; the most recent few turns and any unresolved decision should survive untouched, because a lossy summary of the live tail is where a model most often loses the thread and re-litigates a settled choice. Anthropic's recall-then-precision guidance covers writing that summary well — capture everything important first, then trim — and this guardrail covers where to aim it: summarize aggressively behind the cursor, never ahead of it.
Worth being precise about what the 84% figure above does and doesn't tell you: it measures token savings, not information loss. Neither Anthropic nor the memory-framework vendors publish a number isolating how much retrieval accuracy or recall drops specifically because of compaction — as opposed to long-term memory retrieval generally. The closest adjacent data point is Mem0's own benchmark, which reports roughly 37% multi-session retention and accuracy scores in the 3.7-4.0-out-of-5 range — but that measures long-horizon memory retrieval broadly, not a before/after compaction comparison. Until that specific number exists, "compress the resolved middle, keep the live tail verbatim" is a risk-reduction heuristic, not a guarantee backed by a measured recall figure.
The frameworks, compared
In 2026 the implementations have converged on the same vocabulary but diverge sharply on where the memory lives and who edits it.
| Framework | Memory model | Storage | Distinctive idea |
|---|---|---|---|
| Anthropic memory tool | File-based notes outside the window | Client-side (you host the files) | Model issues CRUD file ops; pairs with context editing + compaction |
| Letta (ex-MemGPT) | OS-style hierarchical, self-editing | Main context + recall + archival (vector) | Virtual-memory paging: the LLM pages info in/out and edits its own prompt |
| Mem0 | Extract-then-append fact store (ADD-only since v2.0.0, Apr 2026) | Vector DB + KV + built-in entity graph | Single-pass extraction; update()/delete() are explicit calls, not automatic reconciliation |
| LangMem / LangChain | Semantic / episodic / procedural API | Pluggable store (e.g. LangGraph store) | One SDK across all three memory types; "hot path" vs background extraction |
Anthropic's memory tool keeps it deliberately minimal: there is no server-side store, just a directory of files the model reads and writes through tool calls, paired with the compaction and context-editing techniques above. You own the persistence layer.
Letta (the company behind MemGPT) is the most explicit about the operating-system analogy. It splits memory into a small main context (working RAM the model sees every turn), recall storage (conversation history searchable on demand, like a disk cache), and archival storage (long-term vector knowledge, like cold storage). Crucially the agent itself decides when to page information out of main context into archival, page it back in, search recall, or rewrite its own system prompt — "virtual context management" lifted straight from virtual memory.
Mem0 is the production fact-store: single-pass extraction that appends new facts (see above for how the ADD-only default replaced the older LLM-mediated add/update/delete reconciliation), persisted across a vector DB, a key-value store, and a built-in entity graph. It now optimizes for low-latency ingestion over a self-correcting store — keeping memories deduplicated and non-contradictory is more the caller's job than the pipeline's.
LangMem gives you one SDK spanning all three memory types and lets you choose when extraction happens: synchronously in the "hot path" of a turn (immediate, but adds latency) or in a background job (cheaper per turn, eventually consistent). It is the most taxonomy-faithful of the four, exposing semantic, episodic, and procedural memory as first-class operations.
The pattern to take away: pick the framework by the question that bites hardest. Need a self-managing long-horizon agent? Letta's paging. Need a clean, deduplicated fact layer behind a chat product? Mem0. Want minimal infrastructure and full control of persistence? Anthropic's file tool. Want one API across all three memory types? LangMem. And regardless of framework, the four window-management techniques are what keep the active context from rotting out from under you.
How the claims were sourced
All version-sensitive claims here were read off official/primary sources on 2026-06-28: Anthropic's context-engineering and context-management pages (compaction is agent-implemented; context editing is the automatic, server-side clearing of stale tool calls; 84% and 39% are internal Anthropic evals, not independent benchmarks), Mem0's docs and 2025 system paper (the ADD/UPDATE/DELETE/NOOP function-calling surface and the entity-boost Graph Memory), the LangMem conceptual guide (semantic/episodic/procedural definitions and prompt-rewriting for procedural memory), and the Letta docs (main/recall/archival hierarchy and paging). "Lost in the Middle" is cited as TACL 2024 (volume 12, pp. 157–173) with the arXiv preprint dated 2023. Memory tooling ships releases on a near-weekly cadence, so the API names and the 84% / 39% figures here are a 2026-06-28 snapshot, not fixed constants.
Re-checked 2026-07-16: Mem0 shipped a v2.0.0 algorithm change on 2026-04-16 that replaced the two-phase extract-then-reconcile add() pipeline (ADD/UPDATE/DELETE/NOOP) described in its 2025 paper with single-pass, ADD-only extraction. The Mem0 sections above and the framework-comparison table were rewritten to match the current docs and migration guide; a maintainer-closed GitHub issue (mem0ai/mem0#4896) confirms the new default does not semantically dedupe contradictory facts. All other claims re-verified on this pass — the Anthropic context-engineering/context-management figures, the LangMem taxonomy, the Letta main/recall/archival hierarchy, and the Lost in the Middle citation — were unchanged from the original 2026-06-28 check.
Sources
- Anthropic — Effective context engineering for AI agents (compaction, context rot, recall-then-precision)
- Anthropic — Managing context on the Claude Developer Platform (context editing 84%, memory tool + editing 39%)
- Mem0 — Add Memory (current single-pass ADD-only pipeline)
- Mem0 — Migrating to the new memory algorithm (v2 → v3, ADD-only breaking changes)
- Mem0 — "Building Production-Ready AI Agents with Scalable Long-Term Memory" (2025 system paper; describes the original pre-v2.0.0 extract-then-reconcile design)
- Mem0 — Graph Memory overview (entity extraction + retrieval boost)
- mem0ai/mem0 GitHub issue #4896 — ADD-only architecture lacks semantic conflict resolution (closed, not planned)
- LangChain — Long-term memory in LLM applications (semantic / episodic / procedural)
- Letta — Memory and context concepts (main / recall / archival, MemGPT paging)
- Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" — TACL 2024 (ACL Anthology)
Top comments (0)