I wanted to build an Agent Memory System and blundered my way into 92% on LongMemEval

Shane Farkas — Sat, 11 Apr 2026 11:11:16 +0000

Like most users of AI agents like Claude Code, I have been frustrated by the agent memory problem. The models have gotten extremely good and no longer lose focus in one long conversation like they used to, but across sessions the memory is pretty spotty whether it's a conversation with an LLM where it recalls imperfect or irrelevant data from previous chats, or a new Claude Code session where I feel like it’s Groundhog Day onboarding a brand new employee who’s smart and talented but knows nothing about my world. So I started looking into the various memory systems. I tried a folder of markdown files, Obsidian vaults etc. but every AI memory system I tried had the same problem: dump text into a vector store, retrieve by cosine similarity, hope for the best.

It works fine for "what did we talk about last week?" but falls apart the moment you need real reasoning like when facts contradict each other, when the answer requires connecting information from three different conversations, or when the user asks "what changed since January?"

So I built Memento, a bitemporal knowledge graph memory system for AI agents. Then I put it through the best long-term memory benchmark I could find, LongMemEval. This is the story of what worked, what didn't, and what I learned along the way.

The Problem with Vector Store Memory

Most AI memory looks like: user says something, you embed it into a vector, store it, and later embed a query to find the nearest neighbors and put the results into a prompt. This is basically document search, and has no concept of entities eg when "John" and "John Smith" are two different chunks. It has no awareness of time, so a fact from January looks the same as one from yesterday. And it has no ability to detect when new information contradicts old information.

I wanted memory that could track entities, time, and contradictions.

What Memento Does Differently

Memento builds a knowledge graph from conversations. When you ingest text, it extracts entities and their properties (people, organizations, projects), then resolves them against existing entities — "John," "John Smith," and "the sales VP" become one node using tiered matching: exact, fuzzy, phonetic, then embedding similarity, then LLM tiebreaker. It detects contradictions — if John's title was "VP of Sales" and now it's "SVP of Sales," that gets flagged. It tracks time bitemporally, recording both when a fact was true in the world and when the system learned it. It also stores verbatim text as fallback — the raw conversation is always there via FTS5 and vector search, so extraction errors don't mean lost information.

When you recall information, it doesn't just search vectors. It traverses the graph: "What should I know before my meeting with John?" finds John's node, walks his relationships to Alpha Corp, finds Alpha Corp's pending acquisition, and composes a briefing — all within a token budget you specify.

   Agent / LLM
         │ (Query/Ingest)
         ▼
  Retrieval Engine  <───>  Ingestion Pipeline
         │                     │
         ▼                     ▼
    Temporal Knowledge Graph (SQLite)
         │
         ├── Consolidation Engine (decay, dedup, prune)
         ├── Verbatim Fallback (FTS5 + vector search)
         └── Privacy Layer (export, audit, hard delete)

Finding a Benchmark

I wanted to evaluate on solid examples. I found LongMemEval, a benchmark specifically designed for long-term conversational memory. It has 500 questions across five categories of increasing difficulty:

Single-session recall — facts stated by the user or assistant in one conversation. Preference tracking — applying user preferences revealed in past conversations. Multi-session reasoning — synthesizing information scattered across multiple conversations. Knowledge updates — returning the latest value when facts change over time. Temporal reasoning — understanding when events happened and their chronological order.

Each question comes with haystack sessions (the conversations containing the answer) and a reference answer. A GPT-4o judge compares the system's output against the reference. All runs used Claude Sonnet as the extraction and reasoning backbone, with GPT-4o as the eval judge (following the LongMemEval paper's methodology).Each question also has an abstention variant where the correct answer is "I don't know" which tests that the system doesn't hallucinate.

I ran against the oracle variant: evidence-only sessions with no distractors, 1–6 sessions per question. A clean test of whether the system can extract and reason over information it definitely has access to.

First Contact — 91.0%

I'd just open-sourced Memento and didn't have a benchmark harness yet. I wrote one, ran a 5-question smoke test (5/5, just checking the plumbing), then kicked off a full 500-question run.

But before that run, I'd already spotted two gaps.

The timestamp gap. Memento had no way to know when a conversation happened. For temporal reasoning questions like "which happened first, X or Y?" the system was guessing. Fix: pipe session dates into ingestion as timestamps and prepend [Conversation date: ...] headers to the text.

The verbatim search gap. Memento was ingesting each session as one big text block. If a user asked about a specific phrase, FTS5 was searching across entire concatenated sessions instead of individual turns. Fix: store each individual turn separately in the verbatim store for fine-grained keyword search, while still ingesting full sessions for entity extraction.

The first full run came back at 455/500 — 91.0% overall, 92.4% task-averaged.

Category	Accuracy
Single-session (assistant)	100.0%
Single-session (user)	94.3%
Single-session (preference)	93.3%
Temporal reasoning	91.0%
Knowledge update	91.0%
Multi-session	85.0%

Single-session questions were almost perfect. Multi-session at 85.0% was the weakest link.

The Retrieval Trap — 89.6%

The natural instinct: multi-session is weak, so give it more context. I widened the retrieval window — top_k from 10 to 20, conversation cap from 5 to 10, token budget from 4K to 8K. I also added some prompt improvements: don't ask clarifying questions for preference queries, enumerate before counting.

The full 500-question run came back: 89.6% overall, 91.0% task-averaged. A regression. Wider retrieval hurt single-session accuracy by 4–6 percentage points through context dilution, while only helping multi-session by +0.7%. When you dump 8K tokens of loosely related context into a prompt, the LLM gets confused and starts second-guessing itself.

Lesson: more retrieval is not better retrieval.

I reverted the retrieval widening but kept the prompt improvements, and ran again. 91.2% overall, 92.4% task-averaged — the best result yet. The prompt changes alone were pulling their weight.

Diminishing Returns — 86.0% → 90.8%

Next I fixed five bugs (conflict references, idempotent decay, recall entity depth, soft-delete for relationships, confirmation counts) and implemented two new features: adaptive retrieval, which classifies queries as "wide" (counting/enumeration) vs. "narrow" (single-fact recall) and adjusts parameters accordingly, and two-pass counting, which enumerates items first, then counts.

Quick validation on 25 questions after the bug fixes: 92.0%. A 50-question stratified sample with the new features: 92.0%. Looking good.

Full 500-question run: 86.0% overall, 88.8% task-averaged. A disaster. Two-pass counting helped roughly 5 multi-session questions but hurt 10+ temporal and knowledge-update questions. I also tested self-verification (having the LLM double-check its own answer), which dropped accuracy to 68%. Each additional LLM call is an opportunity to corrupt a correct answer.

I ripped out two-pass counting, kept adaptive retrieval only, and ran the final evaluation.

Final result: 454/500 — 90.8% overall, 92.2% task-averaged.

Category	Correct	Total	Accuracy
Single-session (assistant)	55	56	98.2%
Single-session (user)	68	70	97.1%
Single-session (preference)	28	30	93.3%
Temporal reasoning	119	133	89.5%
Knowledge update	69	78	88.5%
Multi-session	115	133	86.5%

Here's how all five full runs played out:

Run	Overall	Task-avg	What changed
v1	91.0%	92.4%	Baseline (timestamps + verbatim turns)
v2	89.6%	91.0%	Wider retrieval (regression)
v3	91.2%	92.4%	Revert widening, keep prompt improvements
v4	86.0%	88.8%	Two-pass counting (regression)
v5	90.8%	92.2%	Remove two-pass, adaptive retrieval only

The headline number went down from v1 to v5 (91.0% → 90.8%), but the category breakdown tells a different story. Single-session assistant dropped from 100% to 98.2% — likely noise on a 56-question sample — while single-session user climbed from 94.3% to 97.1% and multi-session improved from 85.0% to 86.5%. The task-averaged score stayed nearly identical at 92.2% vs. 92.4%. What really changed was robustness: adaptive retrieval made the system less brittle across query types, even if the aggregate number didn't move.

What I Learned

Knowledge graphs beat vector stores for structured memory. Entity resolution and temporal tracking make the difference. When you need to answer "what changed about John's role since January?" you need entities, timestamps, and contradiction detection — not cosine similarity.

Retrieval quality beats quantity. Focused retrieval with a well-crafted prompt consistently outperformed broad context. The 4K token budget with 10 results beat the 8K budget with 20 results every time.

Multi-pass generation is a trap. Each additional LLM call — self-verification, two-pass counting, chain-of-thought validation — is an opportunity to corrupt a correct answer. The simplest pipeline that works is the best pipeline.

Small samples lie. A 50-question sample scored 92% on the same configuration that scored 86% on 500 questions. Always run the full benchmark.

Try It

Memento works with any LLM provider (Claude, GPT, Gemini, Llama, Ollama) and any MCP client (Claude Desktop, Cursor, Claude Code, Cline, Windsurf).

# Pick your LLM provider:
pip install memento-memory[anthropic]   # Claude (ANTHROPIC_API_KEY)
pip install memento-memory[openai]      # GPT   (OPENAI_API_KEY)
pip install memento-memory[gemini]      # Gemini (GOOGLE_API_KEY)
pip install memento-memory[openai]      # Ollama (set MEMENTO_LLM_PROVIDER=ollama)

memento-mcp

Or use it as a Python library:

from memento import MemoryStore

store = MemoryStore()
store.ingest("John Smith is VP of Sales at Alpha Corp.")
store.ingest("Alpha Corp is acquiring Beta Inc.")

memory = store.recall("What should I know about John?")
print(memory.text)

The code is MIT-licensed: github.com/shane-farkas/memento-memory

DEV Community: Shane Farkas