Not everything an agent sees is worth remembering. Store too little and it forgets what mattered; store everything and the memory fills with noise that drowns the signal at retrieval time.
The extraction step — deciding what to keep and distilling it into clean, self-contained facts — is the most consequential and most overlooked part of a memory system.
Memory begins with a decision
Of everything that just happened, what's worth keeping? This is the first and most important question a memory system answers, and it's easy to get wrong in both directions. Store too little and the agent forgets the thing that mattered. Store everything and the memory fills with noise, because every irrelevant memory is a candidate to surface at the wrong moment and drown the useful one.
What deserves to be remembered
- Stable facts about the user — preferences, context, recurring needs that will matter in future sessions.
- Decisions and outcomes — what was chosen, what worked, what failed, so the agent doesn't relitigate settled ground.
- Corrections — when a human fixes the agent, that correction is high-value memory: it must stick.
- Durable state — the status of ongoing work, entities the agent tracks, anything with a life beyond one turn.
What to leave out
Transient chatter, one-off details with no future relevance, and anything that will be stale by next session are noise. The discipline of not storing them is as important as the discipline of storing what matters — a memory full of irrelevant fragments retrieves worse than a lean one.
Good memory is curated, not recorded.
Free Agent Memory Quick-Start — the four kinds of memory and the whole loop (store, retrieve, reflect, forget) on a few pages. Download it free.
Turn experience into clean facts
Raw conversation is messy; stored memories should be clean. Extraction usually distills an interaction into concise, self-contained statements — "user prefers X," "project deadline is Y" — rather than dumping the transcript. A common pattern is to use the model itself to summarize what's worth remembering, producing tidy semantic facts from noisy episodic experience.
memories = extract(conversation)
# -> [{'content': 'prefers email over phone',
# 'subject': 'user_1837', 'type': 'semantic'},
# {'content': 'had billing issue, resolved by refund',
# 'subject': 'user_1837', 'type': 'episodic'}]
store.add(memories)
Self-contained is the key property: a memory that only makes sense with its original context will confuse retrieval later.
When to extract
There's also a question of timing. The common choices are at the end of a session, when the whole interaction can be distilled at once, or incrementally as significant things happen, so nothing is lost if a session is abandoned. End-of-session is simpler and cheaper but risks losing an interaction that never cleanly ends; incremental captures more reliably at the cost of running more often. Many systems do both.
Going deeper? AI Agent Memory: The Complete Guide is the full reference — 41 pages, 15 chapters, 5 appendices, with a worked support-agent example and a 30-day adoption path. Get the guide.
FAQ
What should an AI agent store in memory?
Stable facts about the user (preferences, context), decisions and outcomes, corrections from humans, and durable state about ongoing work. The test is whether it will matter in a future session.
What should an agent NOT store?
Transient chatter, one-off details with no future relevance, and anything stale by next session. Not storing noise is as important as storing signal — irrelevant memories surface at the wrong time and degrade retrieval.
How does memory extraction work?
It distills a messy interaction into concise, self-contained facts — 'user prefers X' — rather than storing the raw transcript. A common approach uses the model itself to summarize what's worth remembering into clean semantic facts.
When should extraction run?
At the end of a session (simpler, cheaper) or incrementally as important things happen (more reliable, more frequent). Many systems do both — capturing clearly important facts immediately and distilling the rest at the end.
Why is storing less sometimes better?
Because every stored memory is a candidate to surface at retrieval time. A store full of irrelevant fragments retrieves worse than a lean one, burying the memory that mattered under ones that didn't.
Top comments (2)
Your principle that good memory is curated not recorded is exactly the design constraint behind Opportunity Skill's impression management system. Each impression is a self-contained semantic unit, at most 512 characters, independently understandable without its original context. They are created and pruned but never edited in place, which keeps the embedding space clean the same way your lean store retrieves better than a noisy one. The extraction discipline you describe, distilling messy interactions into clean facts, is what the agent does when it observes your working behavior and writes structured impressions from it. The difference is the audience. Your article is about memory that serves one agent's internal continuity. Impressions serve external discoverability. They are embedded as semantic vectors so that other people's agents can search against them. Same extraction discipline, different output contract. That is the Memory versus Impression boundary the system enforces.
This is a sharp distinction, and I think the audience shift changes more than the output contract — it changes what "self-contained" has to mean. Internal memory can lean on implicit shared context: the same agent that wrote the fact will read it, so "prefers the usual pipeline" still retrieves fine. An impression searched by a stranger's agent gets zero shared context, so self-containment becomes a much harder constraint — which explains both your 512-character ceiling and the create-and-prune-but-never-edit rule. Editing in place would quietly break the contract with every embedding already computed against the old text. Curated-not-recorded turns out to be the shared discipline; what differs is who you're curating for. I'd be curious how you handle staleness on the impression side — internally, forgetting is a policy the agent applies to itself, but externally a pruned impression changes what other people's agents can find about you, which makes pruning almost a reputational act.