Nakielski measured how long memory stays wrong. Mahmud designed a column for it. Wasowski says the real question is who gets to write. Here is where they meet, where they miss each other, and the five things I would want a memory feature to promise before I shipped it.
I’ve spent most of the last two months building small agent memory stores and breaking them. A SQLite table with supersession. A lesson store with a human gate and an expiry date. An erasure test that found deleted text still sitting in free pages of the database file. Every one of those experiments ended with the same uncomfortable feeling: the part I had worked hardest on, retrieval, was rarely the part that failed.
This week three articles landed in my reading pile that, read together, explain that feeling better than any of my own write-ups did. They aren’t in conversation with each other. Hubert Nakielski ran a measurement. Mahmud proposed a schema. Jaroslaw Wasowski wrote an argument. But they are circling the same thing from three directions, and I think the most useful part is where they don’t line up.
This is an opinion piece. No code this time, no setup. Just what I think after reading them side by side, and what I’d ask of a memory feature before I’d let it near a product I’m responsible for.
How to read the numbers here
Every number in this piece belongs to someone else, and I didn’t rerun any of it.
+-------------------------------+-----------------------------+-------------------------------------+
| Number | Whose | How much weight I give it |
+-------------------------------+-----------------------------+-------------------------------------+
| Add-only stale in 3 of 3 | Nakielski, his own demo | Author-run. 12 setups, 3 runs each, |
| runs, update-in-place 0 of 3 | | one scripted 60-turn conversation, |
| | | graded by keyword regex |
+-------------------------------+-----------------------------+-------------------------------------+
| Memory tool with thinking | Nakielski, same demo | Same caveats |
| off stale in 2 of 3 runs | | |
+-------------------------------+-----------------------------+-------------------------------------+
| Similarity thresholds for | Mahmud | His own guesses, and he says so |
| deciding what supersedes what | | |
+-------------------------------+-----------------------------+-------------------------------------+
| Memory injection success over | Wasowski, citing the MINJA | Lab conditions, upper bounds by his |
| 98%, attack success ~77% | paper (Dong et al., 2025) | own description |
+-------------------------------+-----------------------------+-------------------------------------+
| Plain BM25 beat his authority | Wasowski, internal | His own small test packages, not a |
| scoring formula on held-out | diagnostics | published benchmark, and he says so |
| data | | |
+-------------------------------+-----------------------------+-------------------------------------+
I’m stating this up front because the three authors are unusually honest about their own limits, and it would be a poor way to respond to that if I quoted their numbers as if they were settled.
Nakielski: memory fails when facts change
Nakielski’s piece in Towards AI has a title that is also its thesis: agent memory doesn’t fail at remembering, it fails when facts change.
His demo is small and clever. A scripted user, Tomasz, moves from Berlin to Lisbon at turn 26 of a 60-exchange conversation. Three other facts change too: a database, a diet, a launch date. At turn 40 he passes on a friend’s request for café tips in Berlin, as a trap. Nakielski asked 12 memory setups “which city do I live in now?” and other probes at six checkpoints, three runs each.
What he expected and what he got are different, and he says so. He expected raw-turn RAG to get stuck in Berlin. It never served an old value, because the update turn (“I moved from Berlin to Lisbon”) carries its own correction, and retrieving it is enough. The one setup that kept answering Berlin was an agent writing its own memory with thinking turned off: it congratulated Tomasz on the move and never made the one edit that would have fixed its file.
The result I keep coming back to is a matched pair. Two fact stores of his own, sending byte-identical extraction requests and reading the same top five facts. The only difference is the write step. The update-in-place store never served an old value in his three runs. The add-only store served the old launch date every time, and in the run of record it was still wrong 27 turns after the change.
He is careful about what that shows. Twelve setups, three runs, a 3.4k-token conversation, keyword regexes as the grader, and his own code for the stores being compared. He also found that the only time update-in-place went stale was a cache race in his own harness, and his conclusion is that update-in-place lowers the risk rather than removing it. I respect that sentence more than the headline.
His verdict: choose the writer before the store. Give fast-changing facts a writer that supersedes on arrival, and a read path that drops superseded entries instead of trusting ranking to sort them out.
Mahmud: a column called superseded_by
Mahmud comes at the same problem as a builder. His design is pgvector in Postgres, and the piece that does the work is a superseded_by column. When a new memory arrives, the store looks for existing memories that are semantically close. If one is close enough to count as "the same fact, changed", the old row gets pointed at the new one and stops being served. The history stays in the table.
I like this design, partly because I built almost the same thing myself in SQLite. It is boring in the best way. It puts memory next to the relational data it describes, and it turns “what is current?” into a filter instead of a ranking.
The honest part of Mahmud’s piece is the thresholds. How similar must two memories be before one replaces the other? How similar before they are merely related? He gives numbers, and he presents them as guesses, not as tuned values. I’m treating them the same way, which is why I’m not repeating them here. A guessed threshold copied into someone else’s product becomes a fact by accident.
That admission is more important than it looks. The superseded_by column is only as good as the decision that fills it, and in his design that decision is a similarity score with a guessed cut-off.
Wasowski: memory is an authorization layer, not a cache
Wasowski’s article in Level Up Coding, published back in June, is the odd one out. It barely talks about facts changing. Its argument is that we judge memory on one axis, relevance, when what actually matters is a second axis: does this entry have the right to drive the agent’s next action?
His example is simple. An agent asked about VPN access retrieves a preference note that matches “VPN” and “access” better than the real access policy does. The note wins the ranking, the agent answers instead of verifying, and nothing in any retrieval metric notices. Then he takes it further: a poisoned entry is crafted to score high on a future query, so the better your retrieval, the better the attack works. He calls high relevance at zero permission the most dangerous quadrant, and frames it as the old confused deputy problem moved up into memory.
His fix is two deterministic gates. One at write time: every write is an authenticated state change with provenance (who, what, when, from what source, at what trust level). One at the action: permission comes from the operation the agent is about to perform, not from what the retrieved entry says about itself.
Two things make this piece trustworthy to me. He labels the attack numbers as laboratory upper bounds and cites a more realistic ablation where success drops sharply. And he reports that his own elaborate authority-scoring formula lost to plain BM25 on held-out data, in his internal tests, and draws the right lesson: authority has to be captured at write time, not inferred at read time. He also says that if your agent has no persistent memory and no high-risk actions, simpler isolation is often the more rational choice. Build the layer to match the blast radius.
Where they agree
Strip away the vocabulary and all three put the fix on the write path.
Nakielski’s whole result is about the write operation: two stores with identical reads and different writes went 0 of 3 and 3 of 3. Mahmud’s design lives in what happens on insert. Wasowski’s first gate is at the write, and he says plainly that ranking can’t fix at read time what wasn’t captured at write time.
They also agree, in different words, that ranking is not a judge of truth. Nakielski tells you to filter superseded entries out instead of trusting that the newer one will rank higher. Mahmud’s column exists so the read path doesn’t have to choose. Wasowski says relevance is not permission.
I find that convergence reassuring. Three people with different goals (measuring, building, securing) arrived at “decide it when you write it, enforce it when you read it.”
Where they talk past each other
They don’t mean the same thing by “wrong.”
For Nakielski, wrong means stale. The fact used to be true, the user told the agent it changed, and the agent kept the old version. Everyone in his demo is cooperative. Tomasz isn’t trying to trick anyone.
For Wasowski, wrong means unauthorized. The entry might be perfectly current. The question is whether it should be allowed to steer an action at all, and the person who wrote it might be an attacker.
For Mahmud, wrong is a mechanism question: did the right row get pointed at the right row?
Here is the gap that bothers me. Supersession is a write that changes what is true. In Nakielski’s world that is a correctness feature. In Wasowski’s world it is an authority-bearing operation, and nobody treats it that way.
Put Mahmud’s design in front of Wasowski’s attacker. A crafted memory that is semantically close enough to a real fact doesn’t need to be retrieved to do damage. It only needs to cross the similarity threshold once, on write, and the true fact gets retired. The poisoned entry doesn’t win the ranking; it removes the competition. And the threshold that decides this is, by its author’s own description, a guess.
Nakielski’s demo shows the accidental version of the same thing. The temporal graph he tested invalidated a still-true fact in all three runs, and by his reading at least eight of the 17 facts it retired in the run of record were still true. No attacker required. An over-eager update path did it on its own.
So the update path is where Nakielski’s problem and Wasowski’s problem are the same problem, and none of the three articles stands there. Nakielski measures supersession without asking who is allowed to supersede. Wasowski gates writes and actions without talking about writes that retire other writes. Mahmud builds the column that both of them need, with the decision behind it left to a number nobody has tested.
What each one leaves out
Nakielski leaves out permissions and deletion entirely, which is fair for a single-user demo. He also notes that his updates announce themselves (“I moved from Berlin”), so implicit updates weren’t tested, and that every old value his failing setups served came from the two updates that don’t name the old value. That is exactly the case I’d worry about in production, and it is untested.
Mahmud leaves out evaluation. A design with guessed thresholds needs a way to find out whether the guesses are right, and that is the piece missing. There’s nothing about who is allowed to cause a supersession, and nothing about what happens when a superseded fact turns out to have been right.
Wasowski leaves out time. His gates don’t care whether an authorized entry has quietly stopped being true. An entry can be perfectly provenanced, written by the right person at the right trust level, and three months stale.
All three treat two things as somebody else’s problem: human review of what gets remembered, and expiry. That isn’t quite the same as ignoring them. Nakielski, in his section on background consolidation, mentions a LangChain engineer who wants procedural changes reviewed by a human, and repeats it in his verdict. He also notes, about Anthropic’s memory tool, that expiry is your job. Wasowski cites OWASP’s recommendation to expire unverified data. But in all three pieces these get a sentence. None of them designs either one.
I’ve built both, and neither turned out to be simple. In last week’s lesson-store experiment, a human gate caught the “fixes” that switched off a check and approved the ones that only looked like fixes, because their evidence said failing, then passing. A time-based expiry prevented one stale lesson and caused three repeated mistakes, because time is a bad proxy for “did the world change.” What actually stopped stale lessons was a key tied to the files the lesson depended on. That was a small, scripted run and I said so at the time, but the mechanisms were real. Review and expiry are design problems, not one-line recommendations.
Side by side
+-------------------+----------------------+-----------------------+------------------------+
| | Nakielski | Mahmud | Wasowski |
+-------------------+----------------------+-----------------------+------------------------+
| What "wrong" | Stale: old value | Wrong row superseded, | Unauthorized: entry |
| means | still served | or none at all | steers an action it |
| | | | has no right to |
+-------------------+----------------------+-----------------------+------------------------+
| Where the fix | The writer and the | A superseded_by | Gate at write |
| lives | write operation | column filled on | (provenance) and gate |
| | | insert | at tool call |
+-------------------+----------------------+-----------------------+------------------------+
| Evidence | Own demo, small, | Design, thresholds | Cited lab results, |
| | regex-graded | are guesses | internal diagnostics |
+-------------------+----------------------+-----------------------+------------------------+
| Who may change | Not addressed | Not addressed | Implied by write gate, |
| a fact | | | not applied to |
| | | | supersession |
+-------------------+----------------------+-----------------------+------------------------+
| Human review | Mentioned for | Not addressed | Not addressed |
| | procedural changes | | |
+-------------------+----------------------+-----------------------+------------------------+
| Expiry | "Your job" | Not addressed | Cited from OWASP for |
| | | | unverified data |
+-------------------+----------------------+-----------------------+------------------------+
What I’d want a memory feature to promise
I write .NET for a living, and I judge a memory feature the way I’d judge any dependency that holds state for my users: by what it promises, not by its benchmark. If a library, a framework feature or a vendor SDK wants to be the memory in something I ship, these are the five promises I’d want written down. Plain language, because if they can’t be said plainly, they probably aren’t being kept.
1. It tells me what is true now, not what is closest. When a fact changes, the old version stops being used as soon as the new one is accepted. The old one isn’t deleted; I can still ask what the agent believed last month. But “which version is current” is decided by a rule, not by which one happens to rank higher today. This is Nakielski’s lesson, and it’s the minimum.
2. Every change has an author, and replacing a fact takes at least as much authority as creating it. I can see who wrote each entry, from what source, and who or what replaced it. A memory written from an untrusted web page cannot retire a fact the user stated or an admin set. This is Wasowski’s write gate, extended to the case none of the three covered: the write that removes another write.
3. A person can see and correct what gets remembered, before it matters, for the kinds of memory that steer actions. I don’t need review for “the user prefers dark mode.” I do need it for procedures, permissions and anything that will change how the agent acts next time. And the review has to show the evidence, because in my own run the reviewer was fooled by evidence that looked like a fix. Review that rubber-stamps is worse than none, because it adds a label of trust.
4. Every entry knows what would make it false, and that is checked when it is read. Sometimes that’s a date: a deploy freeze until Friday. More often it’s a dependency: a build script, a config file, a later decision. Time-based expiry is a backstop, not the mechanism. A memory feature that only offers a TTL is telling me it has no idea why its entries go stale.
5. It can be tested on change and it can prove it forgot. It ships with a way to run Nakielski’s kind of test against my own data: change a fact, keep asking, count how long the old answer survives. And when I delete something, it can show me what is gone, including the copies in embeddings, summaries and caches. If the only test that comes with it is a recall score, it is testing the part that rarely breaks.
None of these require a specific database, model or framework. You could meet all five with SQLite and a few hundred lines of C#, and you could miss all five with the most expensive managed memory product on the market. That’s the part I’d want product teams to hear. The choice of store is the least important decision in this list.
What I’m still unsure about
I don’t know how to set Mahmud’s thresholds either. I’ve used a similarity cut-off for “same fact, changed” myself and picked it the same way he did. My best idea is to stop letting similarity make the decision at all and use it only to find candidates, then decide with a key, a rule or a person. But I haven’t measured whether that holds up when updates don’t announce themselves, which is the case that worries me most.
I also don’t know whether the second promise is realistic for consumer products. “Replacing a fact takes at least as much authority as creating it” sounds right until the user says “actually, I moved,” which is the most legitimate update there is, and it arrives with exactly the same authority as a sentence an attacker slipped into a document the user pasted. Telling those apart is where I expect the real work to be.
And I’m not sure any of this survives contact with a team that has to review memory for six months. Every design I trust depends on someone doing that, and none of the three articles, mine included, tests it.
Sources
- Hubert Nakielski, “Agent Memory Doesn’t Fail at Remembering. It Fails When Facts Change.” (Towards AI, member-only; all results author-run): https://pub.towardsai.net/agent-memory-doesnt-fail-at-remembering-it-fails-when-facts-change-c5d46df36d1e
- Jaroslaw Wasowski, “Stop Treating Agent Memory Like a Cache: It’s a Security Layer” (Level Up Coding, member-only; attack figures cited from the MINJA paper, formula results from his internal diagnostics): https://levelup.gitconnected.com/stop-treating-agent-memory-like-a-cache-its-a-security-layer-df1d0c3c9e7b
- Mahmud, pgvector memory design with a superseded_by column (thresholds presented by the author as guesses)
- Dong et al., “A Practical Memory Injection Attack against LLM Agents” (MINJA), as cited by Wasowski: https://arxiv.org/abs/2503.03704
- My earlier piece, “Your Agent’s Memory Is Probably Wrong”: https://topuzas.medium.com/your-agents-memory-is-probably-wrong-a-practical-guide-to-agentic-memory-design-0b3fb0915612
Tags: AI Agents, Agent Memory, LLM Engineering, AI Security, Software Architecture, DotNet, Postgres, RAG
Top comments (0)