I lost five days to an AI assistant that would not stop giving me the same wrong answer.
I was building an MLOps pipeline and needed to set up an Airflow DAG. Microsoft had just moved Airflow into Fabric, away from Azure, and there was almost no documentation on the new setup yet. Every time I corrected the assistant, it apologised, promised to fix it, and handed back a slightly reworded version of the same broken config. I opened fresh chats and re-explained the whole problem. Same loop, every time.
A plain documentation search would have been faster.
That experience stuck with me, and it turned into a project: reconstruct, from the ground up, why AI assistants forget, and what a system would actually need to remember properly. This is what I found.
Two failures wearing one costume
The first thing I got wrong was blaming hallucination. The real story was two separate problems.
One was a knowledge gap. The documentation genuinely did not exist yet, so no amount of memory would have helped. If the answer is not in the world, the assistant cannot find it.
The other was a memory gap, and that is the interesting one. The assistant could not hold my problem across the conversation. It could not remember what I had already ruled out. And it could not let my correction override its earlier answer. Even inside a single chat, it lost track. Across chats, it retained nothing at all.
Those are different failures, and telling them apart is the first honest step.
The method: break it before you read the solution
Instead of reading how memory systems are built and nodding along, I used a reconstruction approach. You start from the problem, imagine the simplest possible fix, then work out exactly how that fix breaks. Each failure points you to what the next design has to solve.
By the time you reach the real answer, it feels obvious, because you have personally hit every wall that made it necessary.
So here are the simple fixes, in order, and why each one falls apart.
Fix 1: just send the whole conversation every time
This is the obvious one. If the model forgets, resend everything.
It fails for three reasons, and only one of them is the one people expect.
It does not fit. Context windows are finite. Older models held only a few dozen back-and-forth messages before running out of room. For coding work it is worse, because pasted logs and stack traces use up space fast.
Making the window bigger is expensive. The attention mechanism compares every token against every other token, so the cost grows with the square of the length. Double the context, and you roughly quadruple the cost.
And here is the part that surprised me. Even when everything fits, the model often does not use it. Research shows models recall information at the start and end of the window much better than the middle. So the fact you need can be sitting right there and still get missed.
Bigger context is not the same as better memory.
Fix 2: keep a running summary
If you cannot keep everything, summarise it. Store a short version of the conversation and carry that forward. Most assistants that "remember" you work roughly this way.
The problem is that summarising throws away the specifics that later questions depend on.
There is a measurement in the MemGPT paper that made this click for me. They asked models questions about their own past conversations, working from a summary of earlier sessions. GPT-4 got them right 32 percent of the time. Roughly one in three.
The detail that really landed: GPT-4 scored lower than GPT-3.5 on the same test. A more capable model did worse.
That tells you the problem is not the model. If a smarter model does not help, you cannot wait for the next model to fix it. The gap is in the memory design, not the intelligence.
Fix 3: store everything and search for what is relevant
This is the grown-up version. Save all the past information in a database, and when a new question comes in, fetch the pieces most similar to it. This is retrieval, and it is genuinely useful. It powers a lot of what works today.
But similarity alone has real holes.
The right piece is often not in the top results. Sometimes the thing you need ranks well below the noise, and a system that only reads the top few never sees it.
One search is not enough for questions that need several facts combined. My Airflow problem needed the runtime version, the platform migration, and the approaches I had already tried, all at once. No single stored note held that answer. It had to be assembled across several, and a single lookup cannot do that.
And this is the big one for my case: similarity has no sense of time. When I told the assistant I had moved from Azure to Fabric, the old Azure setup stayed just as similar to my questions as before. So a similarity-only system keeps surfacing the outdated fact, with nothing to signal that it no longer applies. My correction never wins, because "similar" does not mean "current."
The two problems the papers did not cover
I studied four foundational papers for this. Two of the hardest problems showed up in none of them, and they only appeared once I picked a real domain, a coding assistant.
The first is leakage between users. If the system finds memories by similarity, and two people are debugging the same database error, what stops one person's private connection string from surfacing in the other's session? Nothing, if similarity is all you have. The content really is similar. And the failure is silent, because by the only measure the system uses, that memory is a perfect match.
The second is the sensitive stuff. When you debug, you paste config files, API keys, connection strings. A system built to "remember what is useful" will store exactly the things you would never want stored. In this kind of work, useful and sensitive are often the same information.
Neither of these is a retrieval bug or a model bug. They are decisions someone has to make on purpose, at the point where information gets stored.
What a real memory system actually needs
Once you have watched every simple fix break, the requirements stop feeling like a wishlist and start feeling forced by the evidence. A few of them:
Decide what to keep, rather than keeping everything or nothing. Rank memories on more than similarity, so recent and important things can win. Let a correction genuinely replace the old fact instead of sitting next to it. Delete on request, completely. Keep one user's memory away from another's. And know when it has not found the answer, so it can say so instead of looping, which is exactly what mine failed to do.
None of these name a specific solution. That comes later. They are the properties any honest attempt has to satisfy, and each one traces back to a failure I could point at.
What I actually took from this
The reconstruction approach was the real lesson, more than any single fact about memory.
Reading a finished design teaches you what. Breaking the simple version yourself teaches you why. And the why is the part you can defend when someone pushes back, because you watched the problem force each choice.
I am still early in this. Next I build the naive version for real, measure exactly where it falls over, and let those measurements shape the actual design. I will keep writing as I go.
If you have built anything with AI memory, I would genuinely like to hear how you handled the sensitive-data side. That is the piece I am least sure about.
Top comments (0)