Hello there, it's been long since I posted my last blog. I have recently been learning a lot about AI agents and I thought of writing a small blog on how AI agent memory works. This is a very beginner-friendly blog. Someone who has little to no idea about how/why AI agentic memory works can get a simple mental model to understand this, and it can become an entry point to explore further.
Context, Context, Context
TL;DR Memory is a context to LLM
If you have been active in the world of LLMs, you probably have heard of this term "Context". It acts as an input to LLMs to produce a response. The context goes through the neural network and predicts the "token" as a response.
The more specific the context is, the more high-quality output you will get.
For example, if you ask a question like "How to fix a memory leak in my JavaScript code?". It may return a generic response like "ensure that you are freeing your memory after invoking the function". It can also give you a list of possible causes of memory leaks. It may or may not resonate with the specific problem you are dealing with.
On the other hand, if you give code snippets, add some additional details such as how it's getting called, what parameters the functions are expecting, what it returns, and so on. The chances are you will get a personalised, high-quality response in a one-shot prompt.
Sometimes context becomes the most important thing to LLMs to get a better outcome regardless of the LLM's intelligence.
LLMs have a "Context Window" which basically represents a capacity to process a number of contexts when generating the next token.
Although the context window is getting larger and larger, Gemini 3 Pro has up to a 1 Million Token context window. Since the core foundation of LLM is based on attention, it becomes harder for LLMs to consider and give the same attention to all the tokens when generating the next token as an output.
An earlier research paper "Lost in the Middle: How Language Models Use Long Contexts" shows that LLMs can struggle to retrieve relevant information when it is buried in the middle of a long context.
As a rule of thumb, the more in-depth and shorter/moderate context you give to your LLM, the more high-quality and accurate response you can get.
AI agent with memory
When expecting a desired output from LLMs, we typically give two types of prompts as context.
- System Prompt
- Query
System prompt is usually the persona or information of what the user might expect from you. It can also include details such as how to tailor the output, what key things you need to keep in mind, rules to follow, etc. This can help LLMs generate a personalised output.
In simple words, a query can be a question that the user expects your AI agent to answer.
For example, a company can have a customer support bot which can have a persona of a customer representative and is given a set of rules to follow while answering the question.
Now as you probably know, humans and AI both can generate a series of messages as humans interact. This can be as lengthy as possible, maybe until the user's query gets solved. He/She can keep asking a series of questions.
At a certain point, LLMs can easily hit the context limit and it becomes nearly impossible to send everything as context.
Context Window Challenges
The naive solution to solve this problem is to either send the last N messages to LLM or compress/compact the conversation, which includes generating a summary of the existing chat and sending it as a system prompt so LLM can continue the conversation with limited capacity.
There's a trade-off when using either of the solutions. LLM may lose important details and tend to hallucinate when asking specific questions.
This is also known as a Context Engineering problem at a higher level and the challenge is to provide the right context at the right time so LLM can give high-quality and relevant results and reduce the chances of hallucination.
This is usually a challenge that any developer has to take care of when building an AI agent. In fact, managing the context is one of the hardest problems when working with AI agents and there are many companies like SuperMemory, Mem0 which have invested both resources and time to solve this problem.
As you can see, this approach doesn't scale. You cannot compress everything. The summary can be hard to maintain after a certain point.
In order to solve this problem, there are different ways but this completely depends on what kind of challenges you are dealing with.
We will talk about these challenges at a high level first so it will help us understand the solution better.
I will consider two different examples to demonstrate the problems.
Building a personalised AI assistant that remembers almost everything about the user when having a conversation with him/her. Later, whenever the user asks a query about a specific event such as "when's my meeting with Gina?". It should be able to answer this question.
A Support chatbot, which helps the user resolve the query. For example, an e-commerce website can help users refund/cancel orders by sharing order details.
Dynamic Context ingestion
While these two problems can be solved by injecting dynamic context during inference time, also known as RAG (Retrieval Augmented Generation).
Usually when a script or an API requires fetching data for a specific user, it can fetch data from the database via user_id or any other reference.
For LLM, we have a RAG system which also uses a database but instead of storing the data as plain text it embeds that data in vectors before storing them.
Vector is basically a numeric array. It is a numeric representation of anything (usually text, image, audio, video) in a multi-dimensional space.
You probably wonder why we need to store something in the form of a vector? Why can't we just store this in plain form.
Well that's how the LLM works under the hood. It basically represents information as vectors in the space. It becomes easier and faster for a retrieval system to query the relevant chunks (vectors) using mathematical functions like cosine similarity.
In simple words, it will give you the similarity score or distance score between two given vectors. The less the distance, the more relevant it can be.
You can also associate metadata such as
user_id,tags, etc. to show relationships between users and vectors.
Putting everything together, LLM can use a vector database to fetch and retrieve the information on demand.
For example, when asked for the user's favourite food, LLM can look up similar information and give a response to the user. This way LLM does not have to pre-load everything initially.
RAG can help you load both personalised and dynamic context on demand. Many Agentic memory solutions are powered by RAG at the core.
While you don't need a RAG system if you are not required to scale and handle multi-relation information.
Markdown files as a memory
AI agents like OpenClaw use a naive memory system where it stores user-related information in a plain markdown file. It can update the markdown file. It basically logs everything in the markdown file.
Many agents don't require setting up a RAG system initially, a couple of markdown files can help when it is easy to maintain.
For example, OpenClaw has a SOUL.md file which represents the persona of the AI agent. It also maintains a series of memory log files to document everything about what happened that particular day.
Retrieval problem with large markdown files
As memory can grow linearly, it becomes hard to search through each file and figure out the relevant context. BM25 is one of the search techniques that can be used to search context in larger files.
In simple words, it looks for a matching keyword across the files and ensures it returns the relevant matches.
Agentic memory and beyond
While Agentic memory is a broader topic and it is one of the harder problems that many organizations are trying to solve, we have just scratched the surface of how you can get started setting up a memory for your AI agent.
There are advanced topics such as broad query search, re-ranking, de-duplication, hybrid search, etc.
I will write another article to cover all the topics one by one but the idea of this article was to make you familiar with how the overall memory system works with AI agents.
Thanks for reading this blog, if you have any questions please feel free to drop a comment.
Top comments (0)