The LongMemEval benchmark is a public test of long-term memory in chat assistants. It has 500 hand-written questions, and each one sits on top of a long history of past chat sessions between a user and an assistant. To answer a question, a system has to find the right facts in that history and reason over them. The facts may be spread across sessions, may have changed over time, or may not exist at all. The test was published in 2024 and comes in two common sizes. LongMemEval-S gives each question a history of roughly 115,000 tokens. LongMemEval-M uses far larger histories, well beyond what fits in any model's context window. It has become the standard way to compare memory systems for AI assistants.
At Open Agentic Brain & Memory (OpenAB), we report a LongMemEval score too. The OpenAB project page lists 94.2% on LongMemEval-S while reading 75% fewer tokens, measured over all 500 questions with a Claude reader. We are proud of that number, but we also want to be clear about what it means. A benchmark score answers one narrow question. Whether a memory layer will stop you from re-explaining your project to Claude Code, Cursor, Codex, Antigravity and Kilo Code every morning is a different question. This post covers both.
What LongMemEval actually measures
The benchmark groups its questions into five core memory abilities. These are worth knowing, because a single overall score hides which ones a system is good at.
Information extraction: recalling a specific detail the user mentioned once, sometimes deep inside a long session.
Multi-session reasoning: combining facts that appear in different conversations, such as counting or comparing things mentioned weeks apart.
Temporal reasoning: working out when something happened, or what was true at a given time.
Knowledge updates: noticing that a fact changed and answering with the latest version rather than the first one recorded.
Abstention: saying "I don't know" when the history doesn't contain the answer, instead of making one up.
A typical evaluation works like this. The memory system ingests the chat history. A reader model gets the question and whatever context the memory system retrieves. An automatic judge then scores the answer against a reference. Most papers report answer accuracy, and many also report a retrieval metric, such as whether the session containing the answer appears in the top five results.
What a high score does tell you
A strong LongMemEval-S result is real evidence of a few things.
Retrieval finds the right needle. At about 115,000 tokens per history, the haystack is big enough that naive approaches start to fail. A system that scores well is surfacing the relevant sessions reliably.
Updates are handled. The knowledge-update category punishes systems that store facts but never retire stale ones. For coding agents this is the difference between "we use Redis for the job queue" and "we dropped Redis for SQLite last month."
The system can stay quiet. Abstention questions reward a memory layer that doesn't pad the context with half-relevant material that pushes the model toward a confident wrong answer.
Token efficiency is measurable. Accuracy alone can be bought by stuffing more context into the prompt. Reporting accuracy alongside tokens read, as we do with our 75% reduction, shows whether a system is selecting context or just shoveling it.
That last point matters most to us. A memory layer that matches a full-context baseline while reading a fraction of the tokens is doing the job a memory layer is for.
What a LongMemEval score doesn't tell you
The benchmark was designed around personal-assistant chats: a user talking about trips, hobbies, purchases and plans. Coding work is a different shape, and several things developers care about are simply not measured.
It doesn't test cross-tool memory
LongMemEval assumes one assistant with one history. Real developers bounce between tools. You might settle a decision in Claude Code on Monday and open Cursor on Tuesday. The benchmark can't tell you whether Cursor knows what Claude Code learned, because that scenario isn't in it. For us this is the core problem: every session starts from zero, and no tool knows what the others learned.
It doesn't test project scoping
In a benchmark, every fact belongs to the same user and the same history. In a codebase, "always use pnpm" might apply to one repo and not to another. A memory system that does well on LongMemEval can still leak one project's conventions into another, or fail to surface the right repo's history when you open a folder. OpenAB tags each memory by project, found by walking up to the nearest .git. Opening a folder gives that repo's history first, while a map of up to 20 projects lives in the global CLAUDE.md, AGENTS.md and GEMINI.md, so recall can search one project or all of them.
It doesn't test latency or start-up cost
Benchmarks are run offline in batches. Nobody minds if retrieval takes a few seconds per question. In an editor, it matters a lot. OpenAB's start-up hook injects a briefing of at most 2,000 tokens in 84 ms (p50), and on-demand recall over 10,000 memories via MCP takes 28 ms (p50). None of that shows up in an accuracy number.
It doesn't test what you pay per session
LongMemEval measures whether the answer is right, not how much the context costs across a working week. For a coding agent, the token budget at session start is the thing you feel. With OpenAB, loading context for a 10,000-memory store takes 1,325 tokens instead of 1,131,367. On LoCoMo, a second long-conversation benchmark with 1,540 questions, we keep 97% of the full-conversation score while reading 18% of the tokens. Fewer tokens per session means you hit your Claude, Cursor or Codex limits later.
It doesn't test privacy or ownership
A benchmark is indifferent to where your memories live. You aren't. A hosted memory API might score well and still require an account, a vendor server and a copy of your project decisions on someone else's infrastructure. OpenAB has no account and no server of ours. Memories live in a local SQLite database. For sync, they are age-encrypted with a key kept only in your OS Keychain and stored as a Release asset in your own private GitHub repo. A leaked repo or token exposes nothing readable.
It doesn't test whether you can inspect and fix memory
When a memory system learns something wrong, you need to see it and correct it. No benchmark measures that. OpenAB ships a local control panel where you can read, edit, pin or delete every memory, and you can choose to start from past AI work or start fresh.
How to read memory benchmark claims
When you compare memory layers, including ours, these questions get more out of a headline number:
Which split? LongMemEval-S and LongMemEval-M are very different difficulties. A number without the split is hard to interpret.
Which reader model? A stronger reader raises scores on its own. Our 94.2% uses a Claude reader, and we say so.
How many tokens were read? Accuracy without token counts can hide a system that just passes everything through.
Accuracy or retrieval? "Answer found in top 5" and "answer correct" measure different things.
Is the harness public? Numbers you can rerun are worth more than numbers you have to trust.
Test memory on your own history
The most useful benchmark for your workflow is your own work. That is why OpenAB includes brain eval, which scores recall against your own coding history. You can see whether the memory layer finds the decisions and fixes you actually made, in the projects you actually work on.
Here is a concrete version. Say that last month, in Claude Code, you decided to drop Redis for SQLite in a side project and to always use pnpm. A good memory layer should:
Capture both decisions as memories tagged to that project.
Surface them in the start-up briefing when you open the repo in Cursor or Codex, without you pasting anything.
Prefer the SQLite decision over any older Redis notes, which is the knowledge-update problem in practice.
Stay out of the way when you open an unrelated repo.
Show up on your other laptop too, since at start-up OpenAB merges in what other machines learned, so the briefing is the same everywhere.
Under the hood, OpenAB uses hybrid search that combines keywords and meaning, along with outcome scores that rank memories by whether they helped. Tools can call brain_recall, brain_remember, brain_context and load-my-brain over MCP when they need more than the briefing.
Where benchmarks fit
LongMemEval is a good benchmark. It tests extraction, multi-session reasoning, temporal reasoning, updates and abstention in a way that is hard to fake, and a strong score is a reasonable sign that a retrieval pipeline works. But it is a test of personal-assistant memory in a single tool. It says nothing about cross-tool sharing, project scoping, start-up latency, token cost per session, privacy, or your ability to see and correct what the system remembers.
We use LongMemEval and LoCoMo to keep our retrieval honest, and we use brain eval and daily use to keep the rest honest. If you want to dig deeper, our blog has guides on AI memory layers, context engineering, long context vs RAG vs memory, agent hooks, and self-hosted memory privacy. OpenAB is free and open source under Apache-2.0, built by Rohith, Chandan and Sushmita. The source is on GitHub at rohithhn/Open-Agentic-Brain-and-Memory. Install it, run brain eval on your own history, and see whether your tools stop asking you to explain your project twice.
Top comments (0)