ChatGPT, Claude and Gemini all ship memory features now, and the pitch is the same: an assistant that knows you. Your preferences, your projects, your goals, carried across conversations.
OpenAI's own memory FAQ admits the awkward part. It says the previous saved-memories system "often became stale" and could hold contradicting memories at the same time. That is exactly the version of the problem a new benchmark called Memora is built to measure.
The paper behind it, "From Recall to Forgetting" (Uddin et al., Arizona State, University of Arizona, Genies, Findings of ACL 2026), argues that the way we test agent memory has a blind spot: we test recall, while in the authors' framing, real long-term memory is mostly revision.
The blind spot in how memory gets tested
Look at how existing long-term memory benchmarks are built:
- In LoCoMo, 94% of evaluation questions can be answered with evidence from at most two past sessions. In LongMemEval it is 85%. Across the major benchmarks, the average question needs about one session of history.
- Updates barely exist. LongMemEval allows at most two knowledge updates before evaluation, and PersonaMem handles changes across no more than three sessions.
So most memory scores measure something narrow: can the system dig up a fact that was stated once and never changed? Real use does not look like that. People correct themselves, change their minds, finish projects, cancel plans. Information gets updated, withdrawn and replaced, over weeks and months. A memory system that never forgets is not automatically impressive. It can be a liability with good recall.
What Memora does differently
Memora simulates 10 professional personas (engineers, researchers, designers, executives) across three timeline settings the authors call weekly, monthly and quarterly. Those labels refer to the span of the simulated history: about 155, 615 and 1,991 sessions per persona respectively, at roughly 16 turns per session.
The important part is what happens inside those sessions. In the quarterly setting, a persona's memory sees about 1,171 memory operations on average: 63% additions, 18% updates, 19% deletions. Preferences drift, activities repeat, goals progress. Every change is tracked in an explicit memory trace, so the benchmark knows exactly which facts are current and which are stale at question time.
Three task types get evaluated:
- Remembering: recall previously discussed facts.
- Reasoning: combine several memories to derive an answer (like computing budget status from logged expenses).
- Recommending: suggest things that match the user's current preferences, not their old ones.
Scoring uses a metric the authors call FAMA (Forgetting-Aware Memory Accuracy). Standard accuracy asks: did the response include the right information? FAMA also asks: did it avoid the outdated information? Each question comes with "must include" criteria and "must exclude" criteria, and using an invalidated memory costs you. Three LLM judges vote on each criterion, with 88.3% average agreement against human annotators.
They tested two setups. Four LLMs (GPT-5.2, Claude Sonnet 4.5, Gemini 3 Pro Preview, Qwen3-32B) reading conversation history directly, with and without reasoning tokens. And six dedicated memory agents (A-Mem, LangMem, Mem0, MemoBase, MemoryOS, Nemori) that store and retrieve memories across sessions, all answering through the same GPT-4o-mini backend. That last detail matters for how you read the results: the agents run on a smaller model than the LLMs reading raw history, so this is memory machinery on a fixed, modest backend, not a controlled memory-versus-no-memory comparison at equal model strength.
What they found
One note on the numbers below: the authors sum per-question FAMA across all three timeline settings, so the aggregated scores run to 300, not 100. Read them as relative scores, not percentages.
In this benchmark, memory systems are clearly better at remembering. On aggregated FAMA, memory agents nearly double the LLMs on recall tasks: 119.5 versus 65.7. Explicit memory stores do their core job.
LLMs match them on recommending anyway. 144.7 to 153.3 for plain LLMs against 138.4 for memory agents. The authors' explanation: recommendations tolerate approximate memory, because you can guess plausible preferences from persona cues without retrieving anything precise. So in this study, the task where memory should matter most is the one where it added the least.
Reasoning over memory is weak everywhere. Best average is 27.6, from the memory agents. LLMs sit at 12.4 to 13.9. In the authors' manual error analysis, every sampled reasoning error traced back to partial retrieval: the system had some of the pieces and answered anyway. One example from the paper: after logging several expenses against a monthly budget, agents said "within budget" instead of computing the remainder. Reasoning tokens made little difference.
The forgetting penalty grows as memory scales. This is the finding worth sitting with. For LLMs reading raw history, the FAMA penalty from stale-memory use shrinks on longer timelines (32.6 points in the weekly setting, 17.8 in the quarterly one), but for an unflattering reason: on long timelines the old conversations fall out of the context window entirely, so the model physically cannot reuse them. Memory agents show the opposite trend. Their penalty grows from 18.2 to 29.5. The more history the memory store accumulates, the more the agent leans on facts that stopped being true.
This changes rankings. In the monthly setting, MemoryOS (112.8) and A-Mem (112.0) both beat Nemori (105.4) on plain memory accuracy. But Nemori's forgetting penalty is much smaller (15.4 versus 28.4 and 29.5), so once FAMA counts stale memory, Nemori jumps ahead of both. In this result, the "worse" memory system gives better answers because it is better at letting go.
The error analysis makes it concrete. Of the recommendation errors the authors sampled, 64% came from outdated memory that was never invalidated. The paper's example: a user preferred non-fiction, later shifted to contemporary fiction, and the agent recommended a historical biography. It remembered. It just remembered the wrong version of the person.
What this means if you are building with memory
None of this is a reason to skip memory. It is a map of where the hard part actually is.
- Treat memory writes as versions, not appends. An update should supersede the old fact, and retrieval should prefer the current state. A flat, ever-growing list of "facts about the user" is exactly the shape this benchmark punishes.
- Test with corrections, not just recalls. If your eval set has no updates or deletions, you are measuring the easy half. Add "the user changed their mind" cases and check what the agent does with both versions.
- Score the exclusions. FAMA's core trick is cheap to copy: for every eval question, write down what the answer must include and what it must not bring up. Stale-memory use is invisible until you penalize it.
- Do not expect a memory layer to fix reasoning. Retrieval and consolidation came apart in every system tested, and the gap between them is the weakest point.
The honest caveats
The conversations are simulated by LLMs, not collected from real users. The authors argue this makes the benchmark a lower bound: real interaction adds ambiguity and contradiction, so a system that fails here will likely fail harder in the wild. Fair, but still synthetic.
As noted above, all six memory agents answered through GPT-4o-mini, so the agent numbers reflect the memory machinery more than the model. The scoring runs through LLM judges, though with three-way majority voting and a human agreement study behind it. Ten personas and three memory types (preferences, activities, goals) is also a narrow slice of what long-term memory could mean.
Even with all that, the direction of the result is hard to wave away: everyone is racing to give agents more memory, and Memora's results suggest that more memory, badly maintained, makes answers worse.
Sources: Uddin, Shubham, Blanco, Baral, Wang, "From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents", Findings of ACL 2026. Paper: https://aclanthology.org/2026.findings-acl.1337/ (preprint: https://arxiv.org/abs/2604.20006). Code and data: https://github.com/geniesinc/Memora. Product memory docs: https://help.openai.com/en/articles/8590148-memory-faq , https://support.claude.com/en/articles/11817273-use-claude-s-chat-search-and-memory-to-build-on-previous-context , https://support.google.com/gemini/answer/16598469
Written with AI assistance.
Top comments (7)
The Memora split is the right one — 18% updates and 19% deletions in the quarterly setting means revision is a third of memory work, yet benchmarks barely test it. I'd push one step further: deletion shouldn't be silent either. A memory that quietly discards a 'cancelled plan' without surfacing the change can hallucinate intent just as badly as one that remembers too much. Does the benchmark score whether the system announces what it forgot, or just whether the fact is gone?
It scores the answer, not whether the system announces the deletion. Sections 3.5 and 4 define criteria for what a response must include and which outdated facts it must exclude. So a good score doesn't prove the stored fact was erased, either. Your cancelled-plan example adds a separate test: does the assistant acknowledge the change and make clear that the plan is no longer active? I'd keep that distinct from simply not reusing it in a later answer.
That distinction is exactly right — scoring the answer and proving the erasure are two different claims, and a clean response can hide a stale store. Keeping the acknowledgement test separate from the non-reuse test feels like the honest split: one checks what the user is told, the other what the system still holds. Do you think the acknowledgement belongs in the benchmark itself, or as a separate transparency check alongside it?
I'd put it alongside the benchmark, with a separate score rather than fold it into FAMA. That would keep three checks visible: did the assistant acknowledge the change, did later answers stop using the old fact, and was the stored memory actually removed or invalidated? One correction to the split: non-reuse tests behavior, not what the store still holds. Checking the store needs its own inspection, and a reassuring acknowledgement shouldn't count as proof of either.
Stale facts are a cost too. They ride along in every prompt until someone prunes them, and the spend grows even when the answers look fine. A retrieval that returns only what is still true cuts both the errors and the tokens.
Yes, when stale facts get retrieved into the prompt, they take up space even if the answer happens to be right. I'd measure that separately from accuracy: how many invalidated facts reach the prompt, and how many input tokens they add. Memora doesn't report cost or latency, so it doesn't establish the net savings once the work of checking and updating memory is included.
The "must exclude" criteria in FAMA are what I'd want in every memory eval. Recalling a fact is easy to test; noticing it was withdrawn three months ago is what actually goes wrong in practice. "A liability with good recall" is a good way to put it.