# I benchmarked AI agent memory in 2026 — and the numbers tell a different story than the marketing

How Mem0, Zep, Letta, MemPalace, and a local-first newcomer compare on LongMemEval, and what "96%" actually means.

---

Every AI agent memory vendor publishes a benchmark. Mem0 cites LoCoMo. Zep cites LongMemEval. Letta points at its own numbers. And in 2026, an independent test of Mem0's open-source edition scored 32.4% on LongMemEval while the vendor's managed platform self-reports 93.4%.

That gap isn't a conspiracy — it's the benchmark problem: vendors tune for their own harness, on their own data, with their own retrieval stack. And it means the numbers you see on marketing pages are, at best, an upper bound.

So I ran the thing myself, the honest way: through the real production retrieval pipeline of each system, on the public LongMemEval dataset (500 questions), and made the runner public so you can reproduce it.

The benchmark setup

• Dataset: LongMemEval_S — 500 human-curated questions, ~115k tokens per question
• Method: hybrid retrieval (BM25 full-text + semantic vector), Reciprocal Rank Fusion
• The key choice: I measured through the actual retrieval code path each system serves in production — not an independent re-implementation of the algorithm. A vendor's own Python harness that re-implements RRF in isolation proves the algorithm works; it doesn't prove their product does.

The scoreboard

System	LongMemEval R@5	Local / zero-API	Notes
Awareness (daemon path)	96.0%	✅	Reproducible runner, real production pipeline
MemPalace	96.6%	✅	R@5 only, raw verbatim storage
Mem0 (OSS, independent)	~32–49%	⚠️ graph = paid	Self-reports 93.4% on managed
Zep	63.8%	⚠️ SaaS-focused	Temporal knowledge graph
Letta / MemGPT	~83% (LoCoMo)	✅	Agent runtime, not a memory layer

Two things stand out.

First: the two loudest marketing brands (Mem0, Zep) are the two where the best features sit behind a paywall or a cloud meter. Mem0's graph memory — its most accurate mode — requires the $249/month Pro tier. Zep's full platform is SaaS from ~$25/month.

Second: a quiet local-first project — Awareness — lands at 96.0% R@5 on the exact same benchmark, through its real production pipeline, fully local, zero API calls, reproducible with one command.

What actually separates memory systems in 2026

Benchmarks measure retrieval accuracy. They say nothing about the things that make or break a memory layer in production:

1. Orchestration — who decides when to remember?

• Mem0/Zep: the developer manually calls add() / search() at the right moments. If you forget, memory doesn't happen.
• Letta: the LLM manages memory via function calls — but you must adopt Letta as your agent runtime.
• Awareness: writes workflow rules into your IDE config (CLAUDE.md, .cursor/rules/). The agent automatically calls awareness_init at session start, awareness_recall before work, awareness_record after changes. No manual orchestration, no runtime takeover.

This is the difference between a memory library and a memory system — and it's invisible in every benchmark.

2. What's stored — snippets or structure?

• Mem0 extracts short text snippets.
• Zep builds a temporal knowledge graph (excellent at "what changed when").
• Awareness stores structured knowledge across 13 categories: decisions, problem-solutions, workflows, pitfalls, skills, personal preferences, risks, action items. With conflict detection and deduplication built in.

A decision you made in March isn't a snippet floating in a vector index — it's a first-class object with category, confidence, and provenance.

3. Data residency — where does it live?

Every "memory" vendor quietly sends your conversations somewhere. In 2026, with EU AI Act compliance and data-sovereignty requirements becoming real, "the data never leaves my machine" is a hard requirement for a growing number of teams.

Awareness is local-first by default: npx @awareness.market/setup, no account, no API key, data stays on your machine as Markdown files. Cloud sync is optional.

Why I'm publishing this

I'm the maintainer of Awareness — a local-first memory layer for AI agents. Publishing a comparison where my own product's number is verifiable is the only honest way to enter this space.

So here's the reproducibility contract:

• The runner: run_f053_daemon_path.mjs — drives the real unifiedCascadeSearch per question, with an isolated index per question (no cross-question leakage).
• The results file: results_f053_daemon_path_n500_b999000000.json — 500 questions, 4.2s/question, R@5=96.0%.
• The full comparison: Awareness vs. Alternatives — factual side-by-side against Mem0, Zep, Letta, plain RAG, and vector DBs.

Run it on your machine. If my numbers don't reproduce, that's a bug — and I'd rather you find it than I hide it.

The takeaway

If you're choosing an agent memory layer in 2026:

1. Don't trust vendor benchmarks. Re-run on your data, or at minimum check whether the number comes from the production path or a re-implementation.
2. Benchmark accuracy ≠ production value. Orchestration, structure, and data residency matter more than ±1% on a leaderboard.
3. Local-first is a feature, not a compromise. 96.0% R@5 with zero API calls and zero data leaving your machine is now the bar, not the exception.

The memory layer is the cheapest thing to get right early and the most expensive to get wrong late. Measure honestly.