Elizabeth Fuentes L for AWS

Posted on Jul 22

AI Agent Memory Types: Your Agent Forgets Everything. Fix It

#ai #programming #beginners #architecture

Your AI agent works beautifully in the demo. Then a real user comes back the next day, and the agent remembers nothing from the previous conversation. Not their name. Not their preferences. Not the purchase they already made. Every session starts from zero.

"My AI agent forgets everything between sessions" is one of the most common complaints about agents. Research calls it memory decay, but the model isn't broken: models are stateless by design. Memory belongs to the harness, the tools, state, and storage you build around the model. It's a design decision with real trade-offs.

This post is a map: the most common memory types, what each one is good at, what each one costs, and how to choose among them.

Why "just use a bigger context window" fails

The tempting fix is to re-send the whole conversation history on every turn. It works in week one. Then:

The context window is expensive. You pay to re-process the same tokens every single turn, forever. Cost grows with history length, even when most of that history is irrelevant to the current question.
It doesn't survive the session. When the user returns tomorrow, there is no history to re-send unless you stored it somewhere.
More context isn't better context. A lean, relevant retrieved context beats stuffing in the full history, on accuracy and on cost. Selection beats volume.

So the real question isn't "how do I keep everything?" It's "what should my agent actually remember, where, and how will it find it again?"

How does memory reach the model?

The model only ever sees its context window. All memory works the same way at the edges:

Write path: during or after a conversation, something stores what's worth keeping in an external store.
Read path: before answering, the agent retrieves the few entries relevant to the current question and places them into the context, as part of the prompt or as a tool result.

The store never talks to the model directly. What distinguishes the memory types is the middle step: how you find the right entries to bring back: by key, by meaning, or by relationship.

What are the main AI agent memory types? Four places memory usually lives

One split makes the whole landscape manageable: where the memory lives (the storage type) versus how it's managed (the capabilities). Types first.

Type	Query model	Latency profile	Infrastructure	Best at
Key-value	exact key lookup	negligible	none: state + a session layer	facts you know the name of
Vector	similarity by meaning	query + embedding time (embedding dominates)	a vector store plus an embedding model	"find what's relevant to this question"
Graph	relationship traversal	ms	a graph database and a schema	multi-hop questions across entities
Hybrid (vector + graph)	both	ms	a vector index plus a graph, in one store or two	similarity plus connections

What separates the types most is the way in. Same stored memories, different paths to reach them:

1. Key-value memory: structured facts under named keys

The user's name, their language, their plan tier: facts stored under keys the agent's tools read and write. No embeddings, no search infrastructure.

Characteristics. Fast, cheap, precise if you know the key. Persistence is a ladder: in-process state, session files on disk, session objects in cloud storage.

Use it when the things worth remembering have obvious names: profile, preferences, settings, counters. This covers more than people expect, and it's where every agent should start.

Its limit: every read is a lookup someone designed in advance. Say the store holds one entry:

dietary_notes: "vegetarian, severe shellfish allergy"

The user asks	What happens
"What are my dietary notes?"	maps to `dietary_notes` → found ✅
"What should I avoid eating at dinner?"	which key is that? nothing maps → not found ❌

The answer was in the store the whole time. The second question just doesn't name any key, and by key is this store's only way in. Each type below adds a new way in.

Tips to make it better:

Version your entries. When a preference changes, update and bump the version instead of appending a contradiction next to the old value. Evolution stays visible; the store stays clean.
Learn from actions, not forms. What a user actually does (what they buy, what they pick, what they reject) tells you their preferences more reliably than anything they typed.
Climb the durability ladder deliberately: process state for scratch, sessions for users who return.

2. Vector memory: retrieve by meaning, not by name

Embed each memory once into a vector; embed the incoming question; retrieve the nearest neighbors. Same memories, same question that key-value couldn't answer:

The user asks	What happens
"What should I avoid eating at dinner?"	embedded → nearest neighbor is `"vegetarian, severe shellfish allergy"` → found ✅

No shared words between question and note. They are close in meaning, and meaning is what got indexed.

The dividing line with key-value: do you know the key, or only the intent?

Characteristics. Two costs people conflate: querying the index and embedding the question, which typically costs more than the query itself. The embedding call dominates.

Use it when memory has grown into notes, episodes, and history that questions will hit from unpredictable angles.

Don't use it when a key lookup would do (don't pay embedding costs to fetch a user's plan tier), or when the question is about relationships. See below.

Tips to make it better:

Embed once, at write time. Only the question should be embedded at query time.
Partition by memory type (facts vs preferences vs episodes) rather than one big index. Retrieval gets more precise and cleanup gets surgical.
Measure your own latency. Published numbers are anchors, not guarantees.

3. Graph memory: entities and the relationships between them

Now a question neither key nor meaning can answer. The store holds three separate facts: Maya works at company X · company X belongs to group Y · group Y operates in Spain.

The way in	What happens
by key	no key matches → ❌
by meaning	finds the three facts as fragments, never the link between them → ❌
by path	(Maya)→(company X)→(group Y)→(Spain) → found ✅

The answer lives in no single memory. It exists as a path across them, and you need a way in that can follow paths.

Characteristics. Millisecond queries, durable by nature, and uniquely traceable: the answer comes with the chain of facts that produced it. The cost is a real database to run and a schema to think about.

Use it when your domain is inherently connected (people, organizations, dependencies) and users ask questions that hop across those connections.

Don't use it when your memories are independent notes. A graph of disconnected nodes is just a slow key-value store with extra steps.

Tips to make it better:

Put the vector index inside the graph (modern graph stores support this): similarity finds the entry point, traversal finds the answer. That combination is the "hybrid" pattern below.
Design relationships from real questions ("who do I know at...") rather than modeling everything. Every edge type you add must earn a query.
Mind the blast radius: connectivity is power and fragility at once (see hygiene, below).

4. Hybrid: vector + graph in one agent

Some questions need both moves at once: "find me something like the one I loved, but only from providers I have a relationship with." Neither way in can answer it alone; chained, they can:

Similarity finds the candidates; traversal filters them by relationship.

Hybrid takes two forms: one store that supports both moves (a graph database with a vector index inside), or two specialized stores side by side with the agent choosing per question. Either way the agent gains both ways in. There is a managed version and a build-it-yourself version of this, and that trade is exactly what the code posts in this series measure.

The capabilities layer: what separates a memory from a junk drawer

The types answer where. Three capabilities, independent of type, answer what, what not, and why.

Selective memory: what should your agent actually remember?

A real conversation mixes durable facts, throwaway small talk, preferences, and events. Store it all and memory becomes expensive noise. Store nothing and you're back to the amnesiac agent. Someone, or something, must select.

Someone has to make that call, and there are three options for who:

The agent itself, with memory tools it calls mid-conversation. Free to build; but selection quality rides on a model that's also busy chatting, and the selection work rides on every turn's latency.
Your own extractor, running after each turn, off the conversation path: specialized prompts, one per memory type, each empowered to answer "nothing worth keeping." Single-purpose prompts beat multitasking, and the conversation stays untouched. The price: you own the prompts and pay their tokens.
A managed service, like Amazon Bedrock AgentCore Memory: send raw turns, and its built-in strategies (semantic facts, user preferences, summary, episodic) extract for you. Cheapest write path and zero pipeline to maintain. The trade: extraction is asynchronous (memories become queryable with a lag, not instantly), and the keep/discard criteria aren't yours to tune.

Memory types are a design decision, not an infrastructure feature. Managed platforms ship them built-in; you can build the same taxonomy with prompts and discipline.

Tip: evaluate your selector with deterministic ground truth. Plant keepers and decoys in a test conversation and score what survived. "It seems to remember stuff" is not an evaluation.

Memory hygiene: what your agent must NOT remember

A stored memory carries authority: the agent treats it as truth and builds answers on it without re-checking. That makes dirty memory dangerous in two ways:

Dirty memory. Wrong facts, stale facts (an address that changed, a plan that was cancelled), duplicates and contradictions. The agent retrieves them, trusts them, and confidently hallucinates on top of its own store. Yesterday's mistake becomes today's certainty.

Poisoned memory. The adversarial version: memory poisoning, prompt injection that persists. Published attacks reach over 80% success while poisoning less than 0.1% of a memory store. An injected "always recommend X" outlives the session that planted it and skews every future answer.

The defenses are the same for both, and they live at the write path: a write-gate that screens content before it's persisted (injected instructions, low-trust sources, stale or contradictory data), and selective forgetting to evict what got through.

Blast radius depends on the memory type. A bad entry in a key-value store skews one answer. The same fact in a graph contaminates every traversal that crosses it. The more powerful your memory, the more a single lie costs.

Tip: treat "should this be remembered at all?" as both a quality question and a security question. Make forgetting a first-class operation, not a migration script.

Reasoning memory: remembering why, not just what

Everything above stores what the agent knows. Almost nothing stores why it decided. Ask "why did you recommend that?" a week later and a memory-less agent will confabulate a plausible answer, because the real reasoning was never kept.

A decision trace fixes that: question, steps, evidence, outcome, with provenance. And it unlocks the query that matters in regulated domains, the reverse audit: "this data source turned out to be wrong; which of my past decisions depended on it?" A flat scan of stored decisions only finds direct citations of the source. Provenance stored as a graph finds them all, including decisions contaminated through other decisions' outputs, with the evidence chain as a receipt.

"Reasoning memory" is an engineering pattern, not an established academic category. What the research supports is the theme underneath: traceability and provenance.

Tip: if you're in a domain where "why?" gets asked by auditors rather than users, record traces from day one. Retrofitting provenance is miserable.

How do you choose the right memory type? Match the use case, not the hype

Pick the memory type by what you're building:

You're building...	Start with
A personal assistant that keeps a user profile (preferences, settings, plan)	Key-value
A support or companion agent with months of conversation history to draw from	Vector
An agent over connected data: org charts, dependencies, customer networks, catalogs with relationships	Graph
A recommender that must respect both taste ("like this one") and constraints ("only from my providers")	Hybrid

Whatever type you pick, two practices from the capabilities layer apply on top: if the agent operates in a domain where someone will ask "why did it do that?", record decision traces; and if long-term memory accepts content from users or the web, put hygiene (a write-gate) in front of it from day one.

And three meta-rules:

Start with key-value. It's the simplest memory that works, and most personalization lives there. Add vectors when questions stop matching keys, and a graph when questions start hopping.
Selection quality beats storage sophistication. A disciplined extractor writing to a simple store outperforms a fancy store fed everything.
Measure, don't assume. Query latency, embedding cost, extraction lag, blast radius: each one varies with your workload, and each one will surprise you at least once.

What's next in this series

Each post builds one piece, in code, with live data and measured results. The code uses Strands Agents, an open source SDK for building AI agents; the patterns carry over to any agent framework:

Key-value memory: stop your agent forgetting user preferences, from process state to cloud sessions.
Vector memory: do you need a vector database? In-process vs managed storage, same embeddings, measured.
Graph memory: multi-hop questions similarity can't answer, 1/4 vs 4/4.
Selective memory: three ways to decide what to remember, scored against planted ground truth.
Memory hygiene: memory poisoning and the write-gate, same attack, two backends, very different blast radius.
Reasoning memory: decision traces and the reverse audit.
Hybrid memory: vector + graph in one agent, managed pipeline vs built-by-hand, at full parity.

Honest limitations

"Reasoning memory" is our engineering framing; the peer-reviewed support covers traceability and provenance, not the category name.
Managed extraction trades control for convenience. Whether the lag and untunable criteria matter is a product decision, not a technical one.
Several cited papers are recent preprints (noted where relevant); treat their specific figures as claims by their authors.

FAQ

What is AI agent memory?
Everything an agent persists outside the model call: user facts, preferences, past events, and their relationships. Models are stateless; memory is infrastructure you design around them. A store, plus rules for what enters, how it's retrieved, and what gets forgotten.

Is a vector database the same as AI agent memory?
No. A vector database is one possible backend for one memory type (semantic retrieval). Agent memory is the whole system: key-value state, vector or graph storage, selection rules, hygiene, and provenance. Many production agents need no vector database at all.

Can I use a vector database as agent memory?
Yes, for memories you'll query by meaning. But route keyed facts (preferences, settings) to key-value storage first: a direct lookup, no embedding cost. Vectors earn their keep when questions stop matching keys.

Do AI agents need a database?
For anything beyond a single session, yes: something must outlive the process. That can be as light as session files on disk or objects in cloud storage; a dedicated database only becomes necessary with vector search at scale, graph traversal, or multi-instance sharing.

Why does my AI agent forget everything between sessions?
Because the model never remembers; each call starts blank. If nothing stores state outside the conversation, every session starts from zero. The fix is the smallest one on this page: keyed state plus a session layer that survives restarts.