I changed one embedding model and my agent memory needed 8x fewer tokens for the same accuracy

goyohan0611-png — Fri, 30 Oct 2026 04:19:44 +0000

I build Turnbook, a local memory for coding agents. It saves every turn from Claude Code, Codex, Cursor, Copilot CLI, Gemini CLI and OpenCode word for word into one SQLite file on your PC, and when a question needs the past it hands the agent a short evidence packet. Saving calls no model, so it costs no tokens.

For the 0.10 release I swapped the search model from multilingual mpnet to EmbeddingGemma (4-bit ONNX, runs on CPU). I expected a small accuracy bump. What I got was mostly a token cut.

The setup

LongMemEval_S: 500 questions, each about a long chat history (~115k tokens). The memory searches the history, packs what it found into a bounded packet, and a reader model answers from the packet only.

Reader: gpt-5-mini
Judge: the benchmark's official gpt-4o prompt
All 500 questions count. One question the reader never answers (it spends its whole output budget thinking) counts as wrong.

Only two things change between rows: the search model, and how big a packet the reader gets.

Search model	Tokens the reader reads (median)	Accuracy
mpnet	3.4k	86.6%
mpnet	6.7k	89.6%
mpnet	13.2k	90.4%
mpnet	26.3k	91.8%
EmbeddingGemma	3.1k	91.8%

The old model reached 91.8% only when the reader got 26.3k tokens. EmbeddingGemma gets there with 3.1k.

At the same packet size, the paired comparison is 33 questions only Gemma gets right vs 7 only mpnet gets right (exact McNemar p = 0.00004). The biggest move was multi-session questions, the ones that need facts from several different conversations: 73.6% → 88.4%.

Why fewer tokens, not just higher accuracy

With a weaker search model, the right turn is often found but ranked 15th or 40th. The only way to get it in front of the reader is a bigger packet, so you pay for accuracy with tokens. A better search model puts the right turn near the top, so a small packet already contains it.

On LoCoMo conversations the re-ranker never saw, "answer turn among the first 10 found" went from 70.2% to 83.7%.

Other things that changed

Other languages. Meaning-only search, Korean questions, answer in the top 30: 67.5% → 86.9%. A Korean question finds an English conversation 87.8% of the time.
Memory use. The memory server went from ~772 MB to ~525 MB.
Speed. Search is a bit slower: ~40 ms vs ~26 ms per query on a GitHub runner. I think that's a fine trade, but it is a trade.

Caveats

The small learned re-ranker on top was trained on LoCoMo and LongMemEval together, so 91.8% is not a fully held-out score. On the 380 questions never used for tuning it is 91.1%.
EmbeddingGemma is under Google's Gemma Terms of Use. Turnbook downloads it on first use; it isn't shipped in the package.
Existing users' memories move to the new model in the background, at low priority; until that finishes, search keeps using the old model.

Every question, packet, answer and judge verdict is in the repo under research/runs, so you can check any row.

Try it

One command on Windows, macOS or Linux, no Python needed (see the README), or:

pip install turnbook && turnbook-install

It registers itself as an MCP server with the agents it finds. In Claude Code, Codex and Copilot CLI it also adds relevant memory to every message automatically. Apache-2.0.

https://github.com/goyohan0611-png/turnbook

I'd like to hear from anyone who has measured other embedding models for agent memory, especially at small packet sizes.

95% recall, 80% answers: what LongMemEval retrieval scores don't tell you

goyohan0611-png — Mon, 19 Oct 2026 02:35:25 +0000

Memory layers for AI agents love to quote LongMemEval. You will see 94%, 95%, 96%. Those numbers are real, but they usually measure retrieval: was one of the right sessions in the top 5 results? (recall_any@5). LongMemEval's own metric is different: did the model answer the question correctly?, graded by a GPT-4o judge.

I wanted to know how far apart those two numbers are, so I measured both on the same questions.

The setup

agentmemory reports 95.2% R@5 on LongMemEval_S, and to its credit it says plainly that this is retrieval recall and not QA accuracy. It also publishes the sessions its hybrid search (BM25 + embeddings) returned for each of the 500 questions.

So I took exactly those results, unchanged, pinned by commit and hash:

For every question, the top 5 sessions agentmemory returned, every turn in full.
The same answer model (gpt-5-mini), the same answer prompt.
The official LongMemEval judge (gpt-4o-2024-08-06).

Then I did the same with Graph-MIND, the memory layer I build, whose shipped retrieval returns a ~3.4k-token packet of individual turns.

Only the retrieval differs. Same questions, same reader, same judge.

The result

	answer accuracy (499 Qs)	never used for tuning (379)	text the reader sees per question
agentmemory (its top 5 sessions)	80.4%	78.6%	~16.5k tokens
Graph-MIND	89.0%	87.1%	~3.4k tokens

Graph-MIND alone was right on 64 questions and agentmemory alone on 21; exact McNemar p = 3.3e-6.

By question type (Graph-MIND / agentmemory):

multi-session: 80.8 / 65.8
temporal reasoning: 88.9 / 77.0
single-session user: 98.4 / 87.5
knowledge update: 94.4 / 90.3
preference: 75.0 / 71.4
abstention: 93.3 / 90.0
single-session assistant: 94.6 / 96.4 (agentmemory wins this one)

Why 95% recall becomes 80% answers

Three things happen between "a right session is in the top 5" and "the answer is right":

Any is not all. recall_any@5 counts a hit if one answer session is retrieved. Multi-session questions ("how many times did I...", "how long between...") need several. That is where the gap is widest: 15 points.
More text is not more signal. Five whole sessions are ~66k characters. The answer is a sentence or two in there. The reader has to find it, and on temporal questions also work out which of several dated mentions is the current one.
Rank is not position. A session ranked fifth still lands in the reader's context, but next to four sessions that look relevant and are not.

None of this makes agentmemory bad; 80% on LongMemEval with a small reader is respectable, and whole sessions are its natural unit. The point is that R@5 and QA accuracy are different measurements, and comparing one project's R@5 with another's QA (or with each other's R@5 on different units: sessions vs. turns vs. chunks) does not tell you which one answers better.

Caveats, because I build one of these

I picked the reader and the judge setup; it is the one every Graph-MIND number uses, and it is the official judge.
Graph-MIND was developed against 120 of these 500 questions. The untuned column (379 questions) is there for that reason; the gap holds.
I used agentmemory's published retrieval file, not a fresh run of its current code.
Whole sessions are agentmemory's best case here: its reader sees about five times more text.

Everything is reproducible: the script is research/rival_agentmemory.py, and every question, answer and judge verdict is in research/runs/graph-v9.6-rival-agentmemory-k5. If the method is wrong, I would like to know.

What Graph-MIND is, briefly

A local memory shared by the coding agents in your terminal (Claude Code, Codex, Cursor, Gemini CLI, Copilot CLI, OpenCode). It stores every turn verbatim in SQLite on your machine, with no model call at write time, and retrieves with word search plus a small local embedding model. Open source (AGPL), one-line install.

The takeaway, whichever memory you use: ask for answer accuracy, on the same reader and judge.

DEV Community: goyohan0611-png

I changed one embedding model and my agent memory needed 8x fewer tokens for the same accuracy

The setup

Why fewer tokens, not just higher accuracy

Other things that changed

Caveats

Try it

95% recall, 80% answers: what LongMemEval retrieval scores don't tell you

The setup

The result

Why 95% recall becomes 80% answers

Caveats, because I build one of these

What Graph-MIND is, briefly