Most AI assistants are extremely capable and remarkably forgetful.
Give one a complicated problem, and it can reason through it. Come back tomorrow and, depending on the system around it, you may need to explain who you are, what you were doing, and why FINAL_v12_ACTUALLY_FINAL.pdf apparently matters again.
The industry’s answer has mostly been to give the model an external memory.
Store conversations somewhere else.
Index them.
Retrieve the relevant pieces later.
Attach them to the next prompt.
It works.
But it also means the model itself still remembers nothing. We have essentially built a very smart employee who arrives every morning with amnesia and a very organized filing cabinet.
At Backboard, we wanted to test a different idea:
What if some of the memory lived inside the model itself?
We post-trained a model on a LOCOMO conversation set and scored 99.95% on the benchmark.
That number sounds impressive, but it also needs a very large asterisk. And that asterisk is the interesting part.
Memory Is an Architecture Decision
When people say an AI system “has memory,” they can mean very different things.
The most common approach looks something like this:
A conversation happens.
The system stores pieces of it.
Those pieces are embedded and indexed.
Later, when the user asks another question, the system searches for relevant information and includes it in the new request.
This is basically retrieval. The model has not remembered anything. The application has remembered something for the model.
That distinction matters because the memory system now becomes another piece of infrastructure you have to design, secure, operate, monitor, and pay for.
Where does the memory live?
How is it isolated between users?
What gets retrieved?
Who is allowed to retrieve it?
How long is it retained?
How do you delete it?
What happens when retrieval confidently finds exactly the wrong thing?
“Memory” sounds like a feature.
It is actually a collection of architectural decisions wearing a friendly name._
The Filing Cabinet Approach
Retrieval-based memory is useful for good reasons. It is easy to update, and it is flexible.
But every time the model needs that information, the system generally has to retrieve it and put it back into context.
Imagine having an employee who knows nothing about your company, but before every meeting you hand them the relevant sections of the employee handbook, customer history, policy documents, and notes from the last six meetings.
They can absolutely do useful work, but there is a lot of photocopying.
In AI terms, that photocopying is context.
And context costs tokens.
What If the Model Actually Learned Some of It?
There is another option: Instead of storing all long-term knowledge outside the model, you can teach some of it into the model through post-training.
The analogy is closer to what happens with an experienced employee.
At first, they look everything up.
Eventually, they know the terminology.
They know the recurring policies.
They know how the organization tends to work.
They stop checking the manual every time someone asks where the expense form lives.
Until recently, doing something similar with a model was expensive and specialized enough that it was not a realistic memory primitive for most companies.
That is changing.
Techniques such as LoRA make targeted post-training more efficient, while quantization can make adapted models smaller and easier to deploy.
So we wanted to know how far this idea could go.
Then We Got 99.95%
We post-trained a model on conversations from LOCOMO, a benchmark designed to test long-term conversational memory.
The resulting model scored 99.95%.
Essentially perfect recall.
At which point the obvious marketing strategy would have been to put the number in a 72-point font and quietly stop explaining.
Unfortunately, the explanation is the important part.
We trained the model on the same conversation material it was later evaluated on.
On purpose.
So no, this does not prove that we built a universally superior memory model.
It does not prove extraordinary generalization to conversations the model has never seen.
And it certainly does not mean everyone else should throw away their vector database before lunch.
What it demonstrates is something narrower and, I think, more useful:
If a model is taught a body of conversational memory, that memory can be represented parametrically with extremely high recall.
We were testing the ceiling of the mechanism. Not trying to disguise a training set as a magic trick.
Recall and Generalization Are Different Problems
This experiment also exposed something I think AI benchmarks need to handle more carefully. There are at least two different questions hiding inside “Does the model remember?”
- Can it recall information it has previously learned?
- Can it reason correctly over new conversations it was never trained on?
Those are not the same capability.
A model can be excellent at one and weaker at the other.
So we proposed an extension we call LOCOMO-Δ, where some conversations are held back from training.
That would make it possible to measure parametric recall separately from generalization.
Peak numbers are useful. Understanding what produced them is more useful.
This has become something of a recurring theme in our benchmark work.
Parametric Memory Changes the Economics
The interesting part for enterprise systems is not the benchmark. It is the architecture.
If stable information lives inside a model, the application does not necessarily need to retrieve and resend that information every time it is relevant. That changes the cost structure.
A retrieval-heavy system may repeatedly pay inference cost for the same policies, product knowledge, terminology, or recurring context.
Parametric memory shifts some of that work toward a teaching step. For knowledge that is queried again and again, that can become economically interesting.
It also changes deployment. An adapted model can potentially move with its knowledge.
Put it in a private environment. Run it on-prem. Quantize it and move it onto a capable workstation. Deploy it somewhere without continuous access to the original external memory service.
The memory becomes part of the artifact you deploy.
That is a very different architecture.
It Also Changes the Security Boundary
Another interesting property is external retrieval systems often keep many users’ or tenants’ memories inside shared infrastructure and rely on permissions, filters, metadata, and application logic to keep everything separated.
Those controls can be perfectly reasonable, but they are still controls that have to be implemented correctly.
With separate parametrically adapted models, tenant knowledge can instead reside in separate model weights.
There is no shared retrieval store containing that particular memory.
That gives the architecture a different isolation model.
Not automatically a perfect one. Not magically “secure.”
Just different.
And importantly, auditable in different ways.
Security architecture gets better when teams have multiple well-understood options instead of pretending one pattern is always correct.
Unfortunately, Models Are Bad Databases
There is a catch.
Several, actually.
Once information has been learned parametrically, editing it is not the same as updating a database row. Deleting one specific fact is harder. Auditing individual pieces of information is harder.
If a user invokes a right-to-deletion requirement, “we taught it to the weights” is not the kind of sentence that makes a privacy lawyer relax.
And some information changes constantly.
Yesterday’s transaction, this morning’s inventory, a user preference they changed five minutes ago, the current exchange rate, etc.
You probably do not want to retrain a model every time someone changes their shipping address.
Which leads to the part of this argument that tends to be less exciting on social media.
The Answer Is Both
I don’t think enterprise memory becomes:
RAG is dead. Everything goes into the model.
That would be replacing one overly broad architecture with another one.
The better distinction is about the kind of knowledge being stored.
Stable, high-value, recurring knowledge can be a good candidate for parametric memory.
Things like organizational terminology, durable policies, product knowledge, recurring workflows, or relatively stable tenant context.
Volatile, user-owned, or highly regulated information generally belongs somewhere external where it can be updated, inspected, audited, and deleted precisely.
Transactions.
Recent events.
Changing preferences.
Records governed by deletion requirements.
You use the memory mechanism that matches the information.
A shocking conclusion, I know:
Different data probably deserves different architecture.
AI Memory Is Becoming a Design Space
For a while, retrieval was effectively the default answer to long-term AI memory because it was the practical answer available.
That is starting to change.
As post-training becomes cheaper and model deployment becomes more flexible, memory stops being one pattern and becomes a design space.
Some memory can live in retrieval.
Some can live in application state.
Some can live in databases.
Some can live parametrically inside the model.
And a good system may use all of them.
The question is no longer:
How do we give the AI memory?
It is:
Which things should the model know, and which things should the system look up?
That is a much better architecture question.
And if the last few years of AI infrastructure have taught me anything, it is that the useful questions usually begin once the magic-sounding feature gets decomposed into boring engineering decisions.
Which is inconvenient, but also where most of the interesting work turns out to be.
This is a conversational remix of an article I published on Backboard’s blog. Read the original deep dive here.
Top comments (1)
The list of questions you put against retrieval memory is the right list, and the in-weights version does not escape it. It makes each one unanswerable.
Where does it live: in the weights. How is it isolated between users: it is not. What gets retrieved: unknown, and not loggable. How do you delete it: you retrain. Retrieval memory's operational surface looks ugly precisely because every one of those has a boring answer you can show someone. Erasure is a DELETE. Isolation is a tenant id. An audit trail is a log line.
I build under rules where a person can demand their data be removed and I have to prove it is gone. A vector store does that in an afternoon. A post-trained model cannot do it at all, and "we retrain quarterly" is a schedule, not a deletion mechanism.
None of which argues against the direction, only the scope. Memory about the domain, your product's vocabulary, the shape of your API, the things that are true for every user, looks like exactly the right thing to bake in, because there is nothing there to erase. Memory about a person looks like the case the filing cabinet was always for. Does the 99.95% survive splitting the benchmark along that line?