Every memory vendor publishes the table they top. So we did the opposite: we published the one where we do not win.
I built an external memory system for AI agents (Claude Code, Cursor, any MCP client). Before asking anyone to use it, I ran it against LongMemEval_S — 500 questions, ~115k tokens per question, ~47 sessions each.
The numbers
Recall@1: 77.6%
Recall@3: 91.8%
Recall@5: 95.6% <- hybrid BM25+vector RRF
Recall@10: 97.4%
Environment: Apple M1, 8GB RAM. Embeddings: all-MiniLM-L6-v2 (23MB ONNX). Retrieval latency: 1.7s/query with zero LLM calls. Full run: 843 seconds.
The ablation
| Method | Recall@5 |
|---|---|
| Vector-only | 92.6% |
| BM25-only | 91.4% |
| Hybrid RRF (0.7/0.3) | 95.6% |
Where we do NOT win
- Memax: 96.9% recall@5 (self-reported, hosted stack)
- MemPalace: 96.6% (verbatim sessions, ChromaDB)
We are ~1 point behind on recall. We run locally on an M1 with no API calls; they require hosted infrastructure. Trade-offs are the honest unit of comparison.
The metric problem nobody talks about
The field freely mixes recall@5 (did retrieval surface the right evidence?) with QA accuracy (did the final answer match the judge?). They are not comparable, and vendors exploit the confusion. QA accuracy is dominated by the generator model: the same memory system scores wildly differently with GPT-4o vs Opus.
Knowledge updates: 100%
The category we care about most: knowledge-update questions (facts that change over time) hit 100% recall@5 (78/78) because conflict detection is first-class — an incoming memory is classified as new/update/duplicate/contradiction/irrelevant before it touches the store.
The honest pitch
The full report with sources, competitor numbers, and reproduction scripts: https://awareness.market/benchmarks
The scripts are public: github.com/everest-an/Awareness/tree/main/benchmarks/longmemeval
If your benchmark page only shows tables you win, what is it actually telling you?
Top comments (0)