Keyword search ties a purpose-built memory layer on 200 LongMemEval questions

I build one of the systems in this test, so discount accordingly. Posting the numbers because they surprised me and because the boring result is the interesting one.

Setup: 200 questions from LongMemEval, three ways of giving a model memory over the same material. A collaborator ran all three arms with the same reader (Qwen 3.8 27B, Q4) and the same grading rubric.

1. A structured memory layer (MemStrata, my product): facts stored with provenance and validity, retrieved as structured evidence.
2. SQLite FTS5: plain keyword full-text search over the raw text. The baseline everyone skips.
3. An LLM wiki: the pattern where an LLM incrementally writes and maintains linked markdown pages, and answers from those pages.

Results:

MemStrata 188/200 94.0%
SQLite FTS5 184/200 92.0%
LLM wiki 163/200 81.5%

The 4-question gap between the memory layer and FTS5 is within noise at n=200. Call it a tie. The 25-question gap to the wiki is not noise.

Category detail, memory layer vs FTS5:

Knowledge update: 29/31 vs 31/31 (FTS5 wins outright)
Temporal reasoning: 50/53 vs 47/53
Multi-session: 48/53 vs 46/53
Preference: 12/12 vs 11/12
Single-session: tied on both subcategories

Two things I take from this:

First, keyword search is embarrassingly strong and most memory products are probably being compared against strawman baselines. On single-fact updates, where the current answer is stated verbatim somewhere, FTS5 is perfect and structure adds nothing.

Second, the wiki pattern lost on the thing it is supposed to be good at. It carries the highest maintenance cost of the three (an LLM rewriting pages on every new fact) and still landed 12.5 points behind, including on multi-session questions where a curated notes base should shine. My read: every rewrite is a chance to drop or mangle a fact, and the errors compound silently. That is an interpretation, not a measurement.

Disclosures: the wiki arm is our implementation of the pattern, not anyone's production system. One MemStrata answer was missing from the run output and was recovered, with the owner's authorization, before grading; scoring it as wrong would move the result by half a point. One reader model, one question set, n=200.

Question for anyone else testing memory layers: are you also finding FTS5 this hard to beat, and on which question types does structure actually earn its keep in your runs?