Short version: the hybrid retrieval mode we ship as the default scored recall@1 0.435 on LongMemEval_M. Plain vector search on the same run scored 0.739. The default was worse than the simpler option, and it took wiring a real embedder to find out.
The setup
mnemo is an MCP-native embedded memory store for agents. Rust core, DuckDB or Postgres backend, Apache-2.0. It exposes REMEMBER / RECALL / FORGET / SHARE as MCP tools.
Recall supports several strategies. The relevant two:
-
vector_only— dense retrieval, one arm -
auto— hybrid: dense + lexical (tantivy), fused with reciprocal rank fusion
auto was the default, on the usual reasoning: two arms beat one, fusion is robust, RRF is parameter-light.
The numbers
LongMemEval_M, real embedder (nomic-embed-text, 768-dim), n=23:
| strategy | recall@1 | MRR |
|---|---|---|
vector_only |
0.739 | 0.805 |
auto (RRF hybrid) |
0.435 | — |
A second configuration, ONNX MiniLM 384-dim, n=45, gave recall@1 0.689. I am labelling that one not currently CI-reproducible because our ort/ndarray pin is broken; it is indicative, not a claim. The honest version of a benchmark table includes the row you cannot yet re-run on demand.
Why I think it happened
RRF fuses by rank, not by score, which is the property that normally makes it safe. It also means a confidently-correct hit from the dense arm can be pulled down by a confidently-wrong hit from the lexical arm, because RRF has no way to know one arm is unreliable on this workload.
Long-conversation memory is close to the worst case for the lexical arm. The query and the stored memory frequently share almost no surface tokens — the whole point of the retrieval is that the user is referring to something obliquely. So the lexical arm ranks near-randomly, and fusion mixes a near-random ranking into a good one.
That is a hypothesis, not a proof. The sweep that would prove it (RRF k, arm weights, lexical-arm ablation) is not done.
The reproduction
git clone https://github.com/sattyamjjain/mnemo
cd mnemo
# the bench crate lives in bench/locomo
cargo run -p locomo --bin <the runner binary> -- --help
Run it with a local embedder rather than the Noop one. If your numbers disagree with mine, that is the useful outcome and I would like to see them.
What I actually changed my mind about
Not "hybrid retrieval is bad". Hybrid retrieval is good on workloads where both arms carry signal, which is most search.
What I changed my mind about is inheriting a default. auto was the default because it sounded like the safe choice, and it stayed the default through several releases without anyone putting a number against it. On this workload it cost roughly 30 points of recall@1.
If you are building retrieval into an agent, the cheap version of this post is: run your default and your simplest alternative on your own workload once, with the sample size written down, before you ship either.
Open question
Has anyone got RRF hybrid beating pure dense retrieval on a long-conversation memory benchmark with a local embedder? If so I want your k and your arm weights, because right now I cannot tell whether my default is wrong or my benchmark is too small to say.
Top comments (0)