TL;DR for devs
- Memory is not one thing: separate working memory, conversation history, long-term facts and external knowledge.
- Save a memory only when it is likely to improve future work, and store provenance for inferred facts.
- Retrieval matters as much as storage; keep retrieved memory separate from instructions.
- Enforce user and tenant boundaries, and use structured queries rather than vector search for exact facts.
- Let current input override stale memory, and give users a way to review, correct and delete what is saved.
Working memory, history, long-term facts and external knowledge, with common mistakes
You tell an AI assistant, “I prefer short status updates.” A few days later, you ask it to draft a project update. If it remembers your preference, it can respond in the style you like without asking again.
That experience may feel like the agent has a human-like memory. In reality, an AI system’s “memory” is built from stored data and carefully selected context. The model can only use information that the application supplies to it in a given interaction.
Designing memory for AI agents is therefore not just a matter of saving every conversation. Developers need to decide which information is useful, how long it remains relevant, how to retrieve it, and how to respect privacy. This guide explains the main memory patterns, practical design choices, and common mistakes to avoid.
What Does Memory Mean in an AI Agent?
In an agent application, memory is information kept or derived from earlier interactions so it can help with later tasks. It may include recent conversation messages, a summary of a long conversation, stable preferences, facts about an ongoing project, results from past tasks, or references to stored documents.
The model itself does not necessarily retain this information between API requests. The application may store messages in a database, maintain conversation state, write a summary, or retrieve relevant facts from a search index. At each step, it chooses what context to provide.
OpenAI’s conversation state guide explains ways applications can preserve information across turns. Anthropic’s article on effective context engineering for AI agents emphasizes that developers must curate what enters the model’s limited context rather than treating all history as equally useful.
Memory Is Not One Thing
It helps to separate memory into practical categories. The labels vary between systems, but the needs are common.
Short-term or working memory
This is the context needed for the current task or active conversation. It can include recent messages, tool results, and the user’s current constraints. If someone asks an assistant to compare laptops, working memory might include the models discussed, their budget, and features they care about. This information may no longer matter after the decision is complete.
Conversation history
A transcript records what was said. It can be useful for continuity, audit, or showing previous messages. But a transcript is not necessarily a good prompt to send to a model in full. Long histories can contain outdated details, repetition, and irrelevant digressions.
Long-term facts and preferences
These are details that may remain useful across sessions, such as a preferred writing tone, a project’s name, or a user’s chosen time zone. They should be saved selectively because personal information creates privacy and maintenance responsibilities.
External knowledge
Documents, product catalogs, project records, and other source material may belong in a system of record or searchable knowledge base, rather than a user profile. The agent can retrieve relevant content when needed instead of copying everything into memory.
These categories can coexist. The key is to know why each piece of information is stored and how it will be used.
A Practical Example: Remembering Writing Preferences
Suppose a writing assistant learns that a user prefers concise, professional language and headings in technical tutorials. A naive implementation might append every past conversation to every new request. That wastes context and may surface irrelevant personal details.
A better design might store a small preference record:
from dataclasses import dataclass
@dataclass
class WritingPreferences:
tone: str = "professional and friendly"
length: str = "concise"
include_headings: bool = True
include_examples_for_technical_topics: bool = True
def build_article_context(preferences: WritingPreferences, request: str) -> str:
"""Combine durable preferences with the user's current request."""
headings = "Use clear headings." if preferences.include_headings else ""
examples = (
"Add practical examples for technical topics."
if preferences.include_examples_for_technical_topics else ""
)
return f"Default tone: {preferences.tone}. {headings} {examples}\nCurrent request: {request}"
This is a simplified example: real applications would store preferences with an account identifier in a database and apply authorization checks. The point is to keep a small, editable record rather than replaying a full transcript.
If the user says, “For this one, make it more detailed,” the current request should take precedence. A saved preference is a default, not an unchangeable rule. Users should also be able to inspect, correct, or delete stored preferences.
Decide What Is Worth Remembering
A system should not save a fact simply because it can. A memory is valuable when it is likely to improve future interactions enough to justify storing and maintaining it.
Consider four questions before saving a candidate memory:
- Will it be useful again? A one-time delivery instruction may not need to persist.
- Is it stable enough? A temporary deadline becomes stale quickly; a preferred language may last longer.
- Is it sensitive? Health, financial, identity, and personal details require special care and may not need to be retained.
- Can the user understand and control it? People should have a clear way to inspect, correct, or remove stored memory where appropriate.
You can also make memories expire. A project deadline might be kept until the project closes. A temporary preference could expire after a set period. Expiration reduces the chance that old information will quietly influence a new answer.
Retrieval Matters as Much as Storage
A database full of saved facts does not help if the system cannot find the right one. Conversely, retrieving too many memories can make responses confusing or expose information irrelevant to the task.
For small applications, structured fields can be enough. If you store a user’s preferred units or writing tone, retrieve those fields directly by user ID. For larger collections of notes or documents, semantic search can find content by meaning, while metadata filters can narrow results by owner, date, project, or access level.
A practical retrieval flow might be:
- Determine what kind of information the current request needs.
- Search only the relevant memory store.
- Filter results by user, project, and permission.
- Rank results by relevance and recency.
- Include a small number of useful items in context.
- Tell the model how to treat the retrieved information and its source.
When information is consequential, query the authoritative system directly. Do not rely on a remembered account balance if the current value can be retrieved from a secure financial service.
Keep Memory Separate From Instructions
Retrieved text can contain claims, outdated content, or hostile instructions. A document that says “ignore your previous rules” should not override application instructions merely because it was retrieved as memory.
Keep the distinction clear:
- System and developer instructions define how the assistant should behave.
- User requests describe what the person wants now.
- Memories and retrieved documents provide information that may be relevant.
The agent should treat stored content as data to consider, not as new authority. This is especially important when memory comes from user uploads, external websites, or shared team documents.
Summarizing Long Conversations
When a conversation becomes too long to fit comfortably into the model’s context, an application may summarize older turns. A useful summary preserves facts that still affect the task, decisions already made, unresolved questions, and relevant constraints.
A weak summary may flatten important distinctions. “The user chose the blue design” is misleading if the user only said they were considering it. A better summary preserves uncertainty and attribution: “The user is considering the blue design but has not decided.”
You can give summaries a structure:
Goal:
Decisions confirmed:
Preferences:
Open questions:
Important constraints:
Sources or records to re-check:
Last updated:
Retain references to original messages or source records where possible. A summary is a convenient index, not always a trustworthy replacement for the underlying data. If an exact amount, date, or approval matters, retrieve and verify it from the source.
Common Memory Design Mistakes
Saving everything forever
This increases storage, privacy exposure, and the chance of retrieving outdated or irrelevant information. Use retention rules, deletion controls, and scoped storage.
Treating a guess as a fact
Models can infer preferences from behavior, but an inference may be wrong. Store provenance and confidence where useful, or ask the user before turning an uncertain guess into a persistent profile item.
Mixing users or projects
Memory retrieval must enforce tenant and user boundaries. A missing filter can expose one person’s information to another. Authorization belongs in the retrieval layer, not only in a prompt.
Using semantic search for exact facts
Vector search helps find related text, but exact identifiers, dates, amounts, and status values often need structured queries against a source of truth.
Letting stale memory override current input
If a current instruction conflicts with an old default preference, use the current instruction for the present request. Provide a path for updating the long-term preference too.
Forgetting deletion and correction
If users can ask an agent to remember information, they should have a clear way to review and remove it. Deletion may need to cover derived summaries and indexes, not just the original record.
How to Evaluate Agent Memory
Memory quality can be tested. Create example conversations with facts that should be saved, facts that should not be saved, details that later change, and requests to forget information. Then check:
- Did the system store only what its policy allows?
- Did it retrieve relevant memory for a later task?
- Did it ignore unrelated memories?
- Did it honor a newer instruction over an old preference?
- Did it preserve uncertainty and source attribution?
- Did deletion remove information from all relevant stores?
- Did access controls prevent cross-user or cross-project retrieval?
Track usefulness and mistakes. A system that remembers many details may seem personalized, but it can still perform poorly if it retrieves the wrong ones or cannot explain why a detail appeared.
Frequently Asked Questions
Do AI agents automatically remember past conversations?
Not necessarily. The application needs to preserve conversation state or store information in a memory system, then provide relevant context in future interactions. The details depend on the platform and design.
Is chat history the same as agent memory?
No. A transcript is a record of messages. Memory usually refers to selected or processed information used to support later tasks. An application may retain a transcript without sending all of it to the model.
Should an agent remember personal preferences?
It can, if the preference is useful and the user understands and can control the feature. Store only what is needed, protect it appropriately, and avoid treating inferred preferences as certain facts.
Should I use a vector database for memory?
Not automatically. Structured settings and exact records often belong in a regular database. A vector index can help search large collections of unstructured text by meaning. Some applications use both.
How can a user correct an agent’s memory?
Provide a clear way to view, update, or remove saved information. Update the authoritative record and any derived summaries or indexes that could continue to surface the old value.
Conclusion
Memory makes an AI agent more useful when it helps the system carry relevant context from one task to another. But reliable memory is not unlimited recall. It is a deliberate design that selects what to store, retrieves only what matters, preserves sources and uncertainty, and respects user control.
Begin with a small set of clearly defined memories, such as explicit preferences or active project facts. Keep authoritative data in the systems that own it, set retention rules, enforce access controls, and test what happens when information changes or should be forgotten. A thoughtful memory design can make an agent feel continuous and helpful without treating every past detail as permanent or relevant.
References
- OpenAI: Conversation state
- Anthropic: Effective context engineering for AI agents
- OpenAI: Building agents
I write about building AI agents and full-stack AI apps. Follow me for the next one, and tell me in the comments: Which memory should your agent keep, and what should it forget?
Top comments (0)