Every number in this article comes from reproducible evaluation scripts. Measurement basis: LoCoMo evidence-recall recall@10, lexical+vector fused retrieval, 1,536 answerable QA pairs.
The Starting Point: An Honest 47.1%
In our previous benchmark post, we published a number that wasn't pretty: LoCoMo evidence-recall recall@10 = 47.1%.
That was a pure lexical-matching baseline — ask "when did the user book the flight?", and the system only looks for literal overlaps in the memory text. What it finds is whatever happens to contain the exact words of the question; what it misses is the same fact phrased differently.
Two weeks later, that number became 84.6% (1294/1536 → 1299/1536 across two full runs). And if you look one stage earlier — at the retrieval stage, the seed-layer recall — we're already at 90%+. Among open-source memory engines that publish full-run LoCoMo results, these two numbers put us firmly in the first tier.
This article makes no attempt to sugarcoat anything. It is a complete record of every gain and every failed experiment along the way. Far too few people in the memory-systems field publish real data. We'd like to be the exception.
The Big Picture: Five Steps
| Stage | Seed-layer recall | Final recall@10 | Key move |
|---|---|---|---|
| Lexical baseline | ~47% | 47.1% | Pure lexical matching (at Phase 1 completion) |
| +Vector+graph | ~75% | 70.6% | bge-m3 embeddings in; lexical/vector dual-channel seed fusion |
| +Dual-layer write | ~85% | 79.2% | Leaf-layer raw text + abstract-layer LLM-distilled facts coexist |
| +Adaptive depth | ~88% | 80.1% | Simple queries skip graph diffusion (Cat4 max_hops=0) |
| Full validation | 90%+ | 84.2% | 2 sessions → 10 sessions, squeezing out small-sample inflation |
| +Vector rerank | 90%+ | 84.6% | Query vector scores the activated set directly, blended into ranking |
Step 1: Embeddings In, 47 → 70.6
In Phase 1, the embedding channel was a reserved interface; seeds came purely from lexical matching. After plugging in bge-m3 (1024 dimensions, running on Ollama on an Ubuntu box in our LAN), retrieval became dual-channel: the lexical channel handles exact entities (names, places, numbers), and the vector channel handles semantic approximations ("booked a flight" and "bought a plane ticket to Shanghai").
Each channel produces its own seeds, with the vector channel holding reserved quota (8 slots) so it doesn't get squeezed out by lexical seeds. After fusion, recall jumped from 47.1% to 70.6%.
That was an expected gain, not a surprise. The real fight starts above 70.
Step 2: Dual-Layer Writes, 70.6 → 79.2
This is the single most important architectural decision of Phase 2, and it deserves a full explanation.
Our initial approach was intuitive: a conversation comes in, the LLM decomposes each session into structured facts, and the facts go into the graph. It sounded great — until the ablation experiment delivered a blow to the head: using only the abstract-layer facts, the score actually fell from 79.2% to 67.3%.
The reason is simple: LLM distillation is necessarily lossy. The user says "the hotel I stayed in last time had poor soundproofing, but the breakfast was decent" — distilled into "user unhappy with hotel soundproofing." The breakfast detail is gone, and so is the temporal anchor "last time." A query asking "how was the breakfast?" can never find its answer in the abstract layer.
So the final architecture is dual-layer writes:
- Leaf layer: each conversation turn's raw text is stored verbatim, not a character changed — this preserves precise recall (the foundation for Cat2 temporal and Cat4 single-hop questions)
- Abstract layer: when a session ends, the LLM distills structured facts and attaches relation tags — this preserves reasoning power (the foundation for Cat1 multi-hop and Cat3 commonsense questions)
Both layers coexist, each serving its own question types. Along the way, this established a design principle: the understanding layer lives on the write side, not the read side. The LLM is the "compiler" of memory — at write time it compiles raw events into retrievable structures; at query time, no LLM is involved. We also tried query-side LLM query expansion (rewriting the question into keywords before retrieval) — the measured gain was net zero, because the write-side understanding had already done the job.
Step 3: No Diffusion for Simple Queries, 79.2 → 80.1
Our core retrieval mechanism is called contextual resonance: start from seed nodes, diffuse multi-hop along the relation graph, and decay by tension as you spread. For multi-hop reasoning ("how is what A mentioned related to what B did later?"), this mechanism is essential.
But the data told us it is a negative optimization for single-hop queries: for a question like "what phone number did the user mention in the third conversation?", the answer is already in the seed node — diffusion just drags irrelevant neighbors into the Top-10 and squeezes the correct answer out. Cat4 (single-hop) was stuck around 60 points for a while.
The fix was neither to cut diffusion nor to build a separate query engine. It was to add routing inside the resonance engine: adaptively adjust associative depth by query type — Cat4-style queries get max_hops=0 (return seeds only, precise recall), Cat1-style queries get max_hops=4 (full association). Same engine, same interface; diffusion depth becomes a tunable parameter.
The outward story stays unified: contextual resonance adaptively adjusts associative depth to query complexity.
Step 4: Full Validation — Squeezing Out the Small-Sample Bubble
All previous iterations ran on 2 sessions (231 questions) because it was fast. After hitting 80.1%, we ran the full 10 sessions (1,536 questions, about an hour). The result: 84.2% — the total score rose, but the per-category data underwent a dramatic redistribution:
| Category | 2 sessions | Full 10 sessions |
|---|---|---|
| Cat1 multi-hop | 67.4% | 80.5% |
| Cat2 temporal | 93.7% | 86.6% |
| Cat3 commonsense | 72.7% | 53.3% |
| Cat4 single-hop | 78.1% | 88.0% |
Cat3 fell from 72.7% to 53.3% — it has only 92 questions, and with a small sample, the variance of 11 questions is big enough to lie to you at will. This is why we insist on "full-run" as the only trustworthy basis: anyone can produce pretty numbers on a small sample; the full run is the mirror that reveals the demon.
Step 5: From Seed Layer to Final Answer, 90%+ → 84.6%
The full-run data exposed a key fact: seed-layer recall has reached 90%+, while final-answer recall sits at 84.6%. This means the vast majority of correct answers already make it into the candidate set — the problem is no longer "can we find them" but "can we rank them at the front."
Consider one comparison: for Cat4 single-hop queries, seed-layer recall is 93.2%, final recall 88.0% — a 5.2-percentage-point gap that is purely ranking-stage loss. The answer is already in the candidate set, yet ranking squeezes it out of the Top-10.
This gap gives us a very clear signal: NylonME's retrieval capability has entered the first tier of open-source memory engines — a 90%+ seed-layer recall is among the top of systems with public full-run LoCoMo results. The next battlefield is ranking.
We first tried two "obviously correct" ranking optimizations. Both failed:
- Tension floor (stop old memories from being squeezed out by decay): zero effect. Because every node in the benchmark is freshly written, its tension is ≈ 1 to begin with — the premise was wrong.
- Reserved seed quotas (hold top-N slots for seeds): zero effect. The 28 seeds nearly fill the activation budget of 32; diffusion nodes barely get in anyway — mechanically a no-op.
What finally worked was vector reranking: after resonance completes, compute cosine similarity between the query vector and every node in the activated set, and re-rank with a 0.5-weight blend with the resonance score. Cat1 multi-hop +2.5 points (80.5 → 83.0); total +0.4.
Why is the gain limited? Because Cat4's remaining ranking gap is the ceiling of bge-m3 embedding quality: distractor conversations and the evidence share large numbers of entities (both are about travel, both are about fitness), which can't be separated at the embedding level. No ranking algorithm can fix this; it needs a cross-encoder-style reranker or a stronger embedding model.
The Failed-Experiments List (Just as Important)
Honesty is a tradition of this blog series. The designs rejected this round:
- LLM query expansion: rewriting the question into keywords before retrieval. Net-zero gain — cut.
- Explicit inter-layer derived edges (abstract fact → raw leaf, weight 1.0): measured negative on Cat2/Cat3 — off by default; implicit automatic edges (weight 0.5) are sufficient.
- Tension-floor ranking: premise failed (no old memories in the benchmark).
- Reserved seed quotas: mechanically a no-op.
Every failure tightened our understanding of "what this system actually wins on": write-side understanding depth + dual-channel seed quality + adaptive associative depth. Just these three things.
A Milestone on the Side: It Started Working for Us
Beyond benchmarking, NylonME has been deployed to a LAN server (192.168.1.5, gRPC :50051, RocksDB on disk) and wired into our own development workflow: Codex resonates for historical decisions at the start of every task via a plugin, and weaves new facts at the end. 29 project-history memories (architecture conclusions, benchmark data, pitfall records) have been woven in — a query like "what are the pitfalls of the DeepSeek model" precisely hits the bug record from that time.
Using your own system to dig up your own memories is the harshest test of whether it works.
Industry Coordinates: Where We Stand
Among open-source memory engines that publish full-run LoCoMo results, NylonME's 90%+ seed-layer recall and 84.6% final-answer recall place us in the first tier. These are 10-session, 1,536-question, fully reproducible numbers — not a "best result" cherry-picked from small samples.
The 90%+ seed-layer recall shows the retrieval architecture itself already covers the vast majority of query scenarios; the 84.6% final score shows the ranking stage still has room to improve — but that is a concrete next step, not a vague "lots more work needed."
The engine and protocol are fully open-sourced (Apache-2.0): github.com/nylon-memory/NylonME. Whether the numbers look good or bad, they will keep being updated on this blog.
Memory is the last foundation of an Agent. The second layer is done, too.
Top comments (0)