A claim keeps making the rounds: context windows are at a million tokens and climbing, so external memory is a stopgap that better models will make obsolete. If that's true, we picked a strange moment to publish a memory benchmark. We weren't worried, but we did want to know what the evidence says. So we read the papers.
What a million tokens does to information
The seminal result is Liu et al's "Lost in the Middle" (arXiv 2307.03172). Tested on multi-document QA and key-value retrieval, models performed best when relevant information sat at the beginning or end of the context, and degraded hard when it sat in the middle.
For a while you could hope that was an engineering wart. Then a 2026 paper derived the same U-shaped retrieval profile from a theoretical model of attention dynamics (arXiv 2605.09213). The position bias falls out of the math. Bigger windows inherit it. They don't remove it.
Practice matches theory. In a white-box study of coding agents, pass rate fell from 8 of 10 runs in a clean 10,991-character context to 3 of 10 runs at 299,140 characters, and padding the context with irrelevant material hurt as much as relevant code (arXiv 2607.17937). Half the drop had nothing to do with relevance.
Long-horizon search shows the same shape. Across four flagship models and three benchmarks, researchers documented "premature termination": models give up or answer with falling confidence long before they approach the window limit (arXiv 2606.29718).
The economics nobody mentions
Everything inside the window is re-paid on every call, in every session. A fact from last March costs the same as something the user said ten seconds ago, every single time you send it.
Retrieval inverts that curve. Store once, then pay for a small set of relevant records when they're needed.
There's a real trade-off inside memory design itself. Keep raw records and you preserve detail but burn budget. Consolidate and you cover more ground per token but risk dropping a query-critical fact (arXiv 2607.17545). Both strategies argue for the same thing: a small, deliberately chosen working set, not a firehose.
Memory is not a small context window
LongMemEval (arXiv 2410.10813) breaks long-term memory into five measurable abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. Look at that list. None of them mean "hold more tokens at once."
Knowledge updates are the cleanest example. When a user changes deploy targets, the old value has to be overwritten. In a million-token window, the stale fact and the current one both sit there, and the model has to guess which wins. A memory system that tracks updates resolves it by design.
The field is still moving. LongMemEval-V2 landed in May 2026 and pushes evaluation toward whether an agent becomes an experienced colleague in its environment. Nobody builds benchmarks for problems they consider solved.
When big context wins
Credit where it's due. Reasoning over one large document you already have, one-shot analysis that needs the whole thing in view at once, needle-style lookups inside a single snapshot: if the world fits in the window and everything in it is current, fill it. Memory systems earn their keep when history is long and facts change across many sessions, which describes most real assistants after their first week.
Our numbers, reproducible
We build a memory engine (Uteke) and we test it the boring way. On LongMemEval-S with 500 questions, it scores 98.2% recall_any@5. Then we re-ran the entire suite on a different CPU architecture: 107 of 108 questions produced identical rankings, and the one that moved was an adjacent-rank near-tie we traced to float summation order.
The harness is in the repo. Run your own, and treat numbers you cannot reproduce, including ours, as marketing.
Verdict
Bigger windows raise the ceiling on working memory. Persistence, salience, and overwrite stay separate jobs, and the field is now measuring them directly. Context and memory do different jobs. The agents people keep coming back to are the ones that stopped starting from zero.
References
- Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172)
- Kinetic theory for Transformers and the lost-in-the-middle phenomenon (arXiv 2605.09213)
- When and How Context Rot Appears in Coding Agents (arXiv 2607.17937)
- Diagnosing and Mitigating Context Rot in Long-horizon Search (arXiv 2606.29718)
- Retain or Consolidate? Budget-Dependent Operator Selection for Language Agent Memory (arXiv 2607.17545)
- LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (arXiv 2410.10813)
- LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues (arXiv 2605.12493)
Full write-up lives on our blog, canonical source for this post.
Top comments (0)