ChatGPT, Claude and Gemini all ship memory features now, and the pitch is the same: an assistant that knows you. Your preferences, your projects, y...
For further actions, you may consider blocking this person and/or reporting abuse
The Memora split is the right one — 18% updates and 19% deletions in the quarterly setting means revision is a third of memory work, yet benchmarks barely test it. I'd push one step further: deletion shouldn't be silent either. A memory that quietly discards a 'cancelled plan' without surfacing the change can hallucinate intent just as badly as one that remembers too much. Does the benchmark score whether the system announces what it forgot, or just whether the fact is gone?
It scores the answer, not whether the system announces the deletion. Sections 3.5 and 4 define criteria for what a response must include and which outdated facts it must exclude. So a good score doesn't prove the stored fact was erased, either. Your cancelled-plan example adds a separate test: does the assistant acknowledge the change and make clear that the plan is no longer active? I'd keep that distinct from simply not reusing it in a later answer.
That distinction is exactly right — scoring the answer and proving the erasure are two different claims, and a clean response can hide a stale store. Keeping the acknowledgement test separate from the non-reuse test feels like the honest split: one checks what the user is told, the other what the system still holds. Do you think the acknowledgement belongs in the benchmark itself, or as a separate transparency check alongside it?
I'd put it alongside the benchmark, with a separate score rather than fold it into FAMA. That would keep three checks visible: did the assistant acknowledge the change, did later answers stop using the old fact, and was the stored memory actually removed or invalidated? One correction to the split: non-reuse tests behavior, not what the store still holds. Checking the store needs its own inspection, and a reassuring acknowledgement shouldn't count as proof of either.
Fair correction — non-reuse is behavior, and behavior can look clean while the store still holds the old fact, so it can't stand in for inspection. Your three-check split is the honest shape: acknowledgement, non-reuse, and store invalidation each scored on their own, so a good score on one can't launder a failure on another. The store check is the one I'd expect vendors to resist most — how would you inspect it without their cooperation, an export, or a probe that forces the old fact back out?
Without access to the store, I wouldn't claim to have verified erasure. I'd mark that check as unverified and report the behavioral results separately. Probes can expose reuse, but getting the old fact back doesn't tell you whether it came from memory, retained chat history, or inference; not getting it back doesn't prove deletion. A store-level claim needs inspectable evidence with a clear scope, including what an export or audit leaves out.
Stale facts are a cost too. They ride along in every prompt until someone prunes them, and the spend grows even when the answers look fine. A retrieval that returns only what is still true cuts both the errors and the tokens.
Yes, when stale facts get retrieved into the prompt, they take up space even if the answer happens to be right. I'd measure that separately from accuracy: how many invalidated facts reach the prompt, and how many input tokens they add. Memora doesn't report cost or latency, so it doesn't establish the net savings once the work of checking and updating memory is included.
The "must exclude" criteria in FAMA are what I'd want in every memory eval. Recalling a fact is easy to test; noticing it was withdrawn three months ago is what actually goes wrong in practice. "A liability with good recall" is a good way to put it.
Thanks. That's the useful shift: an answer can contain the right current fact and still get something wrong by adding the withdrawn one. Checking both what belongs and what no longer belongs makes that failure visible.
The Nemori result in the monthly setting is the one I would check. MemoryOS (112.8) and A-Mem (112.0) beat Nemori (105.4) on plain accuracy, but Nemori's forgetting penalty is 15.4 against 28.4 and 29.5, so the order flips under FAMA. That flip rests on about 13 points of penalty difference against 7 of accuracy. With three LLM judges at 88.3% agreement with human annotators, roughly one criterion in eight is graded differently from a human. Does the paper report agreement for the "must exclude" criteria separately from the "must include" ones, and any interval on the per-system scores? Without those, I would read the Nemori ordering as one run of one judge panel.
Appendix C.2 reports agreement by judge-consensus pattern, not separately for presence versus exclusion criteria. The 88.3% comes from 100 criteria sampled equally across four consensus patterns, so it isn't a benchmark-wide error rate or an uncertainty interval on the ranking. I couldn't find confidence intervals for the per-system scores. Figures 3 and 4 have error bars, but those show variability across timelines or tasks, not uncertainty on that monthly ordering. I agree the flip should be read as a reported result under this evaluation setup, not a statistically established ranking. That caveat belongs beside the comparison.
The Nemori flip is the finding that stuck with me. A system that forgets cleanly beat two stronger recall systems once stale answers counted against them.
Exactly. In that monthly comparison, the ranking changes once outdated information carries a penalty. I'd call it better avoidance of stale facts in the answers, though, rather than proof that the underlying store forgot cleanly. That's what makes the result useful without asking it to prove more than it measures.