I build Turnbook, a local memory for coding agents. It saves every turn from Claude Code, Codex, Cursor, Copilot CLI, Gemini CLI and OpenCode word for word into one SQLite file on your PC, and when a question needs the past it hands the agent a short evidence packet. Saving calls no model, so it costs no tokens.
For the 0.10 release I swapped the search model from multilingual mpnet to EmbeddingGemma (4-bit ONNX, runs on CPU). I expected a small accuracy bump. What I got was mostly a token cut.
The setup
LongMemEval_S: 500 questions, each about a long chat history (~115k tokens). The memory searches the history, packs what it found into a bounded packet, and a reader model answers from the packet only.
- Reader: gpt-5-mini
- Judge: the benchmark's official gpt-4o prompt
- All 500 questions count. One question the reader never answers (it spends its whole output budget thinking) counts as wrong.
Only two things change between rows: the search model, and how big a packet the reader gets.
| Search model | Tokens the reader reads (median) | Accuracy |
|---|---|---|
| mpnet | 3.4k | 86.6% |
| mpnet | 6.7k | 89.6% |
| mpnet | 13.2k | 90.4% |
| mpnet | 26.3k | 91.8% |
| EmbeddingGemma | 3.1k | 91.8% |
The old model reached 91.8% only when the reader got 26.3k tokens. EmbeddingGemma gets there with 3.1k.
At the same packet size, the paired comparison is 33 questions only Gemma gets right vs 7 only mpnet gets right (exact McNemar p = 0.00004). The biggest move was multi-session questions, the ones that need facts from several different conversations: 73.6% → 88.4%.
Why fewer tokens, not just higher accuracy
With a weaker search model, the right turn is often found but ranked 15th or 40th. The only way to get it in front of the reader is a bigger packet, so you pay for accuracy with tokens. A better search model puts the right turn near the top, so a small packet already contains it.
On LoCoMo conversations the re-ranker never saw, "answer turn among the first 10 found" went from 70.2% to 83.7%.
Other things that changed
- Other languages. Meaning-only search, Korean questions, answer in the top 30: 67.5% → 86.9%. A Korean question finds an English conversation 87.8% of the time.
- Memory use. The memory server went from ~772 MB to ~525 MB.
- Speed. Search is a bit slower: ~40 ms vs ~26 ms per query on a GitHub runner. I think that's a fine trade, but it is a trade.
Caveats
- The small learned re-ranker on top was trained on LoCoMo and LongMemEval together, so 91.8% is not a fully held-out score. On the 380 questions never used for tuning it is 91.1%.
- EmbeddingGemma is under Google's Gemma Terms of Use. Turnbook downloads it on first use; it isn't shipped in the package.
- Existing users' memories move to the new model in the background, at low priority; until that finishes, search keeps using the old model.
Every question, packet, answer and judge verdict is in the repo under research/runs, so you can check any row.
Try it
One command on Windows, macOS or Linux, no Python needed (see the README), or:
pip install turnbook && turnbook-install
It registers itself as an MCP server with the agents it finds. In Claude Code, Codex and Copilot CLI it also adds relevant memory to every message automatically. Apache-2.0.
https://github.com/goyohan0611-png/turnbook
I'd like to hear from anyone who has measured other embedding models for agent memory, especially at small packet sizes.
Top comments (0)