"Should agent memory use a knowledge graph or retrieval?" is the wrong question, and the reason is worth more than the answer. Both are storage shapes. The decision that actually changes how your system fails is whether you store structure your system learned from use, or structure someone asserted about the world. Those two carry different truth guarantees, and merging them into one "memory graph" is how systems become confident without becoming checkable.
Here are the four terms that carry the whole argument.
A knowledge graph stores typed, directed relations between resolved entities. It asserts facts about the world and answers by traversal.
A learned graph, or associative memory, stores untyped weighted links that come from co-occurrence and use. It asserts nothing attributable and never merges identities.
Entity resolution is the decision that two mentions refer to the same thing. It is the load-bearing and most expensive assertion in any knowledge graph.
Provenance is the record of where a memory came from: the source link, the retrieval date, and the exact text that was read.
Disclosure before we go further: I work on Mnemoverse, which ships the learned side of this split, so I have a position. I have tried to make the trade-offs verifiable rather than flattering, including one place where the evidence goes against me.
The two structures fail in completely different ways
An asserted edge that is wrong is precisely wrong. If the system decided that "J. Smith", "Jane Smith" and "Jane A. Smith" are one person, and they are not, then every traversal crossing that merged node inherits the error. The system will state a falsehood with full confidence, and it will be attributable to a specific stored claim.
A learned edge that is wrong is never an explicit falsehood. It is a bias in what surfaces. That sounds safer, and in one narrow sense it is: no attributable claim, no identity merging. But it is not the same as "cannot mislead."
Two results make that concrete. Caliskan, Bryson and Narayanan showed that embeddings trained purely on language co-occurrence predict real occupational gender composition at rho = 0.90 and reproduce human bias effects (Science 356, 2017). Co-occurrence is not neutral just because nobody typed an explicit edge. And from psychology, the DRM paradigm shows that hearing bed, rest, tired produces confident false recall of sleep, a word that was never presented, in roughly 40 to 55 percent of trials (Roediger and McDermott, 1995). Associative structure alone is enough to manufacture a memory.
So the honest formulation for a learned graph is "no attributable claim, no identity merging," not "cannot be wrong."
Asserting identity is expensive, and the cost is structural
A knowledge graph earns real power. Traversal over typed relations answers questions plain retrieval handles badly. The price is entity resolution, and it is not an implementation detail you optimize away later.
The systems that do this at scale treat it with visible caution. Google's Knowledge Vault attached a calibrated confidence to every extracted triple rather than asserting it flat, and only a minority of triples reached high confidence (Dong et al., KDD 2014). Zep's Graphiti puts validity intervals on edges and runs entity resolution at ingest with an LLM in the loop (arXiv:2501.13956). Microsoft's GraphRAG builds an LLM-derived entity graph and says plainly in its own paper that indexing is expensive and the win shows up on global, corpus-level questions rather than needle lookup (arXiv:2404.16130).
None of them treats entity resolution as cheap or solved. Adding asserted structure means accepting that cost, and accepting that some merges will be precisely wrong.
A stored paraphrase is a stored misgrounding
Here is the shortcut almost every memory pipeline takes. At write time, an LLM extracts a summary of what happened, and the summary gets stored next to a pointer at the source. The retriever sees the summary. The pointer is there so someone can check later.
Nobody checks later.
A preregistered study of commercial legal AI tools measured hallucination rates of 17 to 33 percent, where hallucination includes misgrounding: the answer cites a source, and the source does not support the claim (Magesh et al., Journal of Empirical Legal Studies). A stored paraphrase beside a pointer is that failure mode by construction, and it is undetectable without re-reading the source.
The obvious fix, storing just the pointer, has its own decay. The Hypothes.is anchoring study looked at about 21,000 quote-anchored web annotations collected over roughly two years: 22 percent no longer matched their live page, 19 percent were permanently unrecoverable, and only about 12 percent of the broken ones could be re-attached through web archives (arXiv:1512.06195). A bare pointer is honest only at write time.
What works is storing the verbatim span that was read, together with the source link and the retrieval date. The span does three jobs at once. It is deterministic, because it is bytes from the source rather than a model's interpretation. It is ordinary text, so it embeds and searches like everything else. And it is exactly what a W3C Web Annotation TextQuoteSelector is: exact text with prefix and suffix, which re-locates the passage later and fails detectably when the document changed (W3C Web Annotation Data Model).
That last property is the whole point. Fuzzy anchoring reports failure. A paraphrase lies silently.
Anthropic's Citations API follows the same instinct: cited text is extracted directly with character indices, so interpretation happens at read time instead of being frozen into storage.
Provenance is the bridge, not a merge
If learned and asserted structure answer different questions with different guarantees, then the thing connecting them cannot be a merge. It has to be a record of consultation: what was read, from where, and when.
This is standardised prior art, not something to invent. Memento (RFC 7089) defines the original resource and the archived state with its datetime. Robust Links carry data-originalurl, data-versionurl and data-versiondate, so a reader can find a replacement snapshot if one archive dies. Wikipedia's citation templates even record drift as a state: url-status=deviated means the link is alive but no longer supports the claim it was cited for.
One limit worth stating plainly: at most 38.7 percent of composite web mementos are temporally coherent, and at most 17.9 percent are both coherent and complete, because embedded resources get captured at different moments (Ainsworth, Nelson, Van de Sompel). So promise the text span, not the whole page.
The part where the evidence goes against me
I would rather state this than have you find it.
The dominant memory architectures store interpretations, not raw spans, and they have a measurement on their side. Generative Agents stores natural-language experience records and synthesises reflections. RAPTOR stores recursive summaries as retrieval units. And on LOCOMO, a benchmark of question answering over long multi-session conversations, the best chunk-based RAG configuration reaches a judge score of J = 60.97 while LLM-extracted dense memories reach J = 66.88 (Table 2 of arXiv:2504.19413; note that this is the Mem0 team evaluating its own system).
The mechanism is not mysterious. A distilled fact is denser than the conversation around it, so the same retrieval budget lands on the answer more often.
That result should not be waved away, and I am not going to pretend it does not exist. Extraction buys benchmark points. What it costs is auditability, because an extracted memory is already a judgment and the system can only trust the extractor that made it. Storing the original wording and the original pointer gives up those points on purpose.
Make that trade explicitly. The one thing to never do is let an extractor's sentence silently become the stored fact.
A decision rule you can actually use
Not a verdict, a rule.
Use retrieval when the agent needs to find past evidence. This is the floor and it is the right default.
Add learned association when lexical retrieval keeps missing related memories that would have helped. Treat what it returns as ranking influence, not as truth.
Add an asserted knowledge graph when the agent must answer structural questions over resolved entities, and only when you accept entity resolution as a stored assertion you will sometimes have to correct.
Keep provenance across all three: source link, retrieval date, exact text read, and a dated snapshot where you can get one.
The two structures do not need to merge. They need to be connected by a record that can be checked. Treating them as one artifact is exactly how a memory system becomes confident without becoming checkable.
This is a shorter version of an article in the Mnemoverse library, where the full argument, the complete source list, and the related pieces on provenance and bitemporal memory live.
Top comments (0)