I've been building WOS, a long-term memory API for AI agents, and I want to write down the one thing that reframed how I think about the problem.
When people say "give the AI memory," it sounds like the work should mostly be search: store what happened, pull the relevant bits back later. And retrieval on its own is close to solved. On LongMemEval, the retrieval step finds the right memory well over 99% of the time.
The Real Bottleneck
But end-to-end accuracy—actually answering the question correctly using that memory—sits a lot lower, around 85 to 90%.
So the model has the right memory in hand and still gets the answer wrong. The gap isn't in finding the memory. It's in delivering it back in a form the model can actually use, especially for questions that span many sessions or depend on time ("what did I say last month", "has this changed since").
That gap is basically what WOS is built around.
Architectural Trade-offs
A few things that follow from it:
- Bring your own key: It's a retrieval API, not a model. You bring your own LLM key, and WOS only charges for the retrieval tokens. Your model, your data.
- Privacy & Control: It works across languages, not just English. It doesn't train on your data, and you can wipe a user's entire memory in one call.
- Tiered Performance: There's a lineup with different tradeoffs. The lightest tier (Tablet) does retrieval without calling a model at all, so it stays cheap and fast (~320ms). Heavier tiers add an LLM query layer for harder multi-session questions and score higher on the same benchmark.
One honest caveat: those tiers were measured with different reader models, so I wouldn't read the numbers as a clean apples-to-apples ranking. The papers say the same.
Benchmarks and write-ups are available at wontopos.com if you want the details.
Curious how others handle this. If you've given an agent long-term memory, where does it break for you—the retrieval step, or the part where the model has to use what it got back?
Top comments (0)