If you are building agentic AI systems then memory management quickly becomes one of the most confusing parts (You'll soon realize)
Not because the idea is hard but because there are too many ways to think about it
People mix up sessions, context, history, embeddings, user preferences, and logs. Everything starts getting labeled as “memory”
So I wanted to break this down in a simple way using a practical architecture that I’ve been using
The First Principle: Memory is Not One Thing 🤔
When we say “agent memory”, we are actually talking about multiple layers working together
At a high level, you can think of memory as two major categories
- Short-term memory
- Long-term memory
both serve very different purposes as shown in below diagram
Memory ID: The Logical Vault 🏦
Before diving into types of memory, there is one important concept
Memory ID
Think of it as a logical vault that holds everything related to a user or system . Inside this vault, memory is organized in a structured way
memoryId = the memory store / vault
├── short-term memory
└── actorId → sessionId → events
└── long-term memory
└── namespace → memory records
This abstraction is powerful because it decouples memory from the agent
Keep in mind that memory is not tied to a specific agent, it is tied to an actor
Short-Term Memory: Covers what Just Happened 🧩
Short-term memory is all about the current interaction
It captures the flow of a conversation or task in real time
At its core, it has a simple hierarchy
- Actor ID → identifies the user or system
- Session ID → one continuous interaction
- Events → the smallest unit of memory
An event can be anything like
- user prompt
- tool invocation
- assistant response So a session becomes a sequence of events
What About Branching? 🌿
Things get interesting when you introduce branching . Instead of having a single linear flow, you can fork memory into multiple branches
For example
- main conversation
- flight agent path (me going from Ottawa to NZ)
- hotel agent path (Finding a cool spot in Queentown)
Each branch can evolve independently while still being tied to the same session. This becomes very useful in multi-agent systems where different agents explore different reasoning paths
Long-Term Memory: Covers what Should Be Remembered 📚
Short-term memory is temporary
Long-term memory is intentional
This is where you decide what is worth keeping . Tt is not just one thing. Long-term memory has multiple types
1. Semantic Memory (Facts as Vectors) 🔎
This is your classic vector database layer . You store facts, embeddings, and retrievable knowledge
Examples
- user preferences
- structured facts
- extracted knowledge
A system like OpenSearch or any vector DB works well here
This enables similarity search and contextual recall
2. Episodic Memory (What Happened Over Time) 🕰️
This is about history . Instead of storing raw conversations forever, you summarize them
You keep
- summarized conversations
- logs
- historical patterns
Typically stored in something like S3 or object storage
This helps the agent recall past interactions without loading everything into context
3. Procedural Memory (How the System Behaves) ⚙️
This is often overlooked but very important . Procedural memory defines how the agent operates
It includes
- tool definitions
- policies
- system rules
- configurations
Think of this as the “operating manual” for the agent
Connecting Short-Term and Long-Term Memory 🔄
Now the real question
How do these two layers work together???
There are two main flows
Persist Flow (Short → Long)
Not everything from short-term memory should be stored
You apply strategies like
- Summarization
- Extraction
- Filtering
Only important signals get promoted to long-term memory
Recall Flow (Long → Short)
When a new request comes in, then agent retrieves relevant long-term memory and injects it into the current context
This is how personalization and continuity happen
Why Actor-Centric Memory Matters ? 👤
One subtle but important design choice . Memory is attached to actorId, not agent
This means
- one user can interact with multiple agents
- memory remains consistent across all of them
It also means your system is agent-agnostic
Whether you have
- 1 agent
- 10 agents
- or no agent (just APIs) The memory model still works
Practical Example 🛫
Let’s say Amar (user) is booking travel
Short-term memory tracks
- current booking session
- user inputs
- tool calls
Long-term memory stores
- Preference for window seat and business class
- Vegetarian meals
- Past trips
When the user comes back later
The system does not start from scratch, It recalls preferences and continues seamlessly
Key Takeaways I want you to remember from this blog 💡
Memory in agent systems is not just about storing chat history . It is about structuring information in a way that supports reasoning, recall and personalization
- Short-term memory handles the now
- Long-term memory handles the forever
The real power comes from how you connect the two
Top comments (0)