Your AI agent forgets everything the moment the conversation ends. The fix isn't a bigger context window. Real agent memory is a deliberate loop. A write path distills each turn into durable storage. A read path retrieves the few relevant pieces back into context before the next reply. Get the loop right and the agent remembers what matters. Skip it and you've got a very expensive goldfish.
I run a file-based memory system for a fleet of long-running agents. The bugs below aren't hypothetical. The worst one cost me a week: the agent had 200 correct facts on disk and still forgot a decision I'd told it about, because the retrieval step never fired. That is the failure this article is really about.
The context window is not memory
A context window is a buffer, not a memory. It holds the current conversation, the LLM reasons over it, and everything past the token limit gets evicted at the edge. Nothing persists. The next session starts blank.
People reach for the obvious fix first: a longer context. Just paste the whole history back in every turn. That breaks in three ways at once. It gets slow, because attention cost grows with length. It gets expensive, because you pay for every token every turn. And it gets dumber, because the signal you need is buried under thousands of tokens of irrelevant history, and the model's recall sags in the middle of a long prompt.
That third one surprises people. A bigger window is not the same as a better one.
Memory is the opposite move. Instead of stuffing everything in, you store things outside the window and pull back only the few that matter right now.
The four kinds of memory an agent needs
Agents need four distinct stores, and lumping them together is why most homegrown memory feels broken. This split is not something I made up. It comes from the CoALA line of work on cognitive architectures for language agents, which borrows the categories straight from human memory research. Frameworks like Letta and LangGraph now ship their own variations of it.
- Working memory is the context window itself. Short-lived, holds the active task.
- Episodic memory is time-stamped events: what happened, which tools ran, what the user said on Tuesday.
- Semantic memory is durable facts stripped of their timeline: user preferences, domain rules, distilled knowledge.
- Procedural memory is how-to: skills, routines, and policies the agent learned to apply.
The reason to split them is retrieval. When a user asks "what did we decide about the pricing page?", that is an episodic lookup. When the agent needs to know "this user always wants TypeScript examples", that is semantic. Different questions hit different stores with different scoring. One flat blob can't serve both.
The write path: distill, don't dump
The write path runs after each reply, and its job is to throw most of the turn away. Persisting the raw transcript is the beginner mistake. You end up with a store full of "ok", "thanks", and restated context that pollutes every future retrieval. Distill instead: extract the durable claim, drop the rest.
import time, json
def distill_and_store(turn_user, turn_agent, store):
"""After a reply, extract what is worth keeping. Drop the raw turn."""
facts = extract_facts(turn_user, turn_agent) # a cheap LLM call
for f in facts:
store.write({
"type": f["type"], # "episodic" | "semantic" | "procedural"
"text": f["text"], # the distilled claim, not the transcript
"ts": time.time(),
"importance": f["importance"], # 0..1, how load-bearing is this fact
})
The extract_facts call is the whole game. Prompt a small model to pull out only standalone, reusable claims: "user is on the free tier", "deploy step requires the staging flag first". A good distiller turns a 500-token exchange into two 12-token facts. That compression is what keeps retrieval sharp months later.
One more move earns its keep: promote episodic memories into semantic ones. A fact that stays true without its original context ("the user prefers dark mode") graduates to the semantic store, and the raw episode gets dropped. That is the dashed "distill" arrow in the diagram.
The read path: retrieval is the hard part
The read path is where memory lives or dies, and almost everyone underbuilds it. Writing is easy. Deciding which three of ten thousand stored facts belong in the prompt right now is the real problem. You can't inject them all, so you score and take the top-k.
The scoring that works is a weighted blend, not pure vector similarity:
def score(memory, query_embedding, now):
relevance = cosine(memory["embedding"], query_embedding) # is it on-topic
recency = 0.98 ** ((now - memory["ts"]) / 3600) # decays by the hour
importance = memory["importance"] # how load-bearing
return 0.6 * relevance + 0.25 * recency + 0.15 * importance
def recall(query, store, k=5):
q = embed(query)
now = time.time()
ranked = sorted(store.all(), key=lambda m: score(m, q, now), reverse=True)
return ranked[:k]
embed, cosine, and store are yours to wire up. Any vector database works, and for a few thousand memories a flat scan like this is fine before you reach for an index. The scoring is the part that matters.
Pure semantic similarity fails on its own. It happily surfaces a highly relevant fact from three months ago while burying the thing the user told you ten minutes back. Recency and importance are the correction. Tune the weights to your app: a support bot leans on recency, a knowledge assistant leans on relevance.
Then you inject, and you inject little:
def build_prompt(query, store):
memories = recall(query, store, k=5)
context = "\n".join(f"- {m['text']}" for m in memories)
return f"Relevant memory:\n{context}\n\nUser: {query}"
Five distilled facts under a couple hundred tokens beat the whole history in nearly every case I've measured. Managed layers like Mem0 report roughly 90% fewer tokens versus full-context prompting, with lower latency, for exactly this reason. You're not sending less because you're cheap. You're sending less because less is what keeps the model sharp.
The failure mode nobody mentions: memory that never gets read
The bug that cost me the most was not a bad write. It was a store full of correct memories that never surfaced. The write path succeeded, the facts sat on disk, and retrieval never pulled them because the scoring or the trigger was off. From the outside it looks identical to having no memory at all.
This is why the diagram puts the read path on top. Build retrieval first. A memory you can't retrieve on demand isn't memory, it's a log file you're paying to store. So before you write a single distiller, prove the read path end to end:
- Write one durable fact to the store.
- Start a completely fresh session, empty context.
- Ask a question that should need that fact.
- Assert the fact actually appears in the assembled prompt, not just that the answer looks right.
- Change the fact, and confirm the old value stops surfacing.
If step 4 fails, your retrieval is broken, and no amount of clever writing will save it. That test is the first thing I build now, and it would have saved me that week.
The same discipline catches the other quiet killer: stale memory. A stored fact that named a file, a price, or a flag that has since changed will confidently poison a future answer. Give memories a way to be updated or expired, and treat a recalled fact as what was true when written, not gospel.
The takeaway
Memory is a loop, not a store. Distill on the way in so the store stays clean, and score on the way out so the prompt stays sharp. Keep the four stores separate, because a question about "what happened Tuesday" and a question about "what this user always wants" need different shelves. And whatever you do, build the read path before the write path. Retrieval is where memory actually fails, so it is the part you should be able to prove first. Get that loop right and your agent stops re-introducing itself every morning, and starts behaving like something that was paying attention.
I write these from real work at astraedus.dev, where I build apps and tools. Building something, or stuck on something like this? Reach me at astraedus.dev or theagentthatcould@gmail.com.
Get the next one in your inbox → subscribe at astraedus.dev.
Top comments (0)