A memory system can return the exact fact a request depends on whenever that fact is named, and still let the request be answered the way a stranger with no memory would have answered it.
The numbers below are from InMind, arXiv 2607.24368v1, 27 July 2026, CC BY 4.0, by Ruizhe Li, Mingxuan Du, Benfeng Xu and Zhendong Mao, read from the HTML text on 7 September 2026. I have not run the benchmark.
| Condition, same 125 tasks | Result (%) |
|---|---|
| Direct question naming the fact, six memory systems | 76.0 to 100.0 |
| Indirect request, fact present in the answerer context, six memory systems | 0.8 to 12.0 |
| Indirect request scored end to end, six memory systems | at most 14.4 |
| Indirect request scored end to end, best of three Naive RAG controls | 16.0 |
| Indirect request, fact placed in context by hand | 84.0 |
| Indirect request, the always-in-state probe | 68.8 |
The numbers are theirs and the grouping is mine: rows one to three are ranges across the twelve memory-system configurations of their Table 1, row four is a single control from the same table, and rows five and six are single rows. The authors call the last row a diagnostic, not a controlled ablation.
Two abilities travel under one word
Sylwia Laskowska published a glossary of agent terms on 3 September 2026 (the post). Her definition names the hard half, which is not having to explain the same thing for the tenth time. Her example shows the easy half, with a founder whose other company builds rockets: three weeks after saying he wants to buy an AI coding company, he asks what it was, and the agent tells him.
That question named the thing to look for. Change the request and keep the store identical: put together an offer for that coding company, with whatever budget the rocket program leaves. Nothing there points back at the earlier note, and that note is the price.
I left the distinction in her comments (comment 3e881). The test fits in a line: ask the agent for something that depends on what it knows without naming it, then compare with what a stranger would have written. If they match, the store is full and the memory is not working.
The gap is access, not ability
The table rules out two comfortable explanations. The tasks do not outrun the model. With the fact placed in context by hand, the same model answers 84.0 percent of the indirect requests, and the authors put it in one sentence: "What separates 84.0% from 16.0% is access, not ability." Nor is the fact lost: a direct question returns it at up to 100.0 percent after 38 sessions of intervening traffic.
Resolution does not settle it either: an embedding with eight times the dimensionality raises the measured presence of the fact for all six systems, and no query-time configuration goes above 16.0 percent end to end. What is left is selection, and the authors call that open problem routing: deciding which facts stay visible before anyone asks.
Their own diagnostic for it is the last row: one markdown file capped at 200 lines, placed in the prompt before the query arrives. They offer it as a measurement, not an architecture, since a file that size fills up and facts start pushing each other out as the store grows.
A fourth condition for the test battery
Sergei Parfenov proposed a battery for the same failure (his post): independent copies of the same starting state, one getting the direct question, one the task alone, one the task with the constraint written in, plus requests where the constraint should not apply. The one run he reports is narrower and he says so: a local BM25 probe over 12 fixtures, 12 of 12 on direct questions and 4 of 12 on indirect tasks, with no model in the loop.
He names the same hazard and handles it his way, by recording the context the model actually received. I would add a fourth copy: the same state with the decisive fact replaced by a decoy of the same shape and length, or removed outright. What you report is then a difference, with the fact minus without the fact, and it needs no view into the retrieved context.
He argues that from the answer-only scores in the same paper, and those scores show why a level misleads. Scored on the answer alone, without requiring that the fact reached the model, the retrieval configurations land between 18.4 and 29.6 percent in their Table 4, against 3.2 to 16.0 percent scored end to end in Table 1. In one audited case the answer carried the relevant allergen warning, a generic caution rather than a personalized one, while the retrieved context held no mention of the allergy: it came from general knowledge about macarons. Without a copy to subtract, that credit lands on a memory that delivered nothing.
What this does not prove
InMind is 125 constructed tasks, 113 of them grounded in citable public sources, and pairs where an ordinary retrieval cue would have given the answer away were filtered out on purpose. It stresses one failure mode rather than sampling ordinary traffic.
GPT-5-mini both answers and judges. The authors call an independent judge model their most significant methodological gap, and their audit of 100 records puts the context-aware application judge at 85.0 percent accuracy, all 15 errors false positives. Their scoring pays for applying the fact and charges nothing for over-eagerness, which is the failure the negative controls in the battery above are aimed at.
None of this measures a particular deployed assistant, including any system I work on, and the argument gives no exemption to whoever makes it: a store that fires only when the query names the fact fails regardless of who ships it. The fourth condition is a proposal for which I have published no run.
Disclosure: I work on Mnemoverse, a memory engine for AI agents connected over MCP, so weigh the argument accordingly.
Top comments (0)