Everyone is building AI memory using graphs. They're all going to hit a wall. Most of them just don't know it yet.
The memory problem
LLMs are stateless. Every agent starts from zero. Your team makes a decision on Monday, by Wednesday the AI has forgotten it. You repeat yourself, re-explain context, burn tokens saying the same thing over and over.
The answer so far? Knowledge graphs. Nodes, edges, traversal. Looks great on a whiteboard. Falls apart in production.
Why graphs feel right
We think in relationships. "This connects to that." Graphs mirror how we talk about knowledge. It feels natural. But feeling natural and scaling are two different things. Whiteboards don't have to run at inference time.
Where it all breaks
The common take is that retrieval is expensive. Graph traversal is slow, relevance ranking is noisy. That's all true but it's not the real problem.
The real problem is keeping the graph up to date. Change a single node. Every relationship touching that node needs to be recomputed. But it doesn't stop there. Second-degree neighbors need their relationships recomputed too, because the indirect relationships have shifted. Nobody talks about this.
The update cost scales with the density of the graph, not just the size. A small change cascades through the entire neighborhood. The more knowledge you accumulate, the more expensive every single update becomes. Graphs were built for data that sits still. AI memory doesn't sit still. Decisions evolve. Context shifts. Knowledge gets superseded every day. Every update triggers a cascade.
You can't optimize your way out of a fundamental mismatch between the data structure and the use case.
What the LLM actually needs
The LLM doesn't care about your beautiful graph. It needs a tight, relevant, compressed chunk of context. That's it. Not a traversal through loosely connected nodes. Not a fuzzy "maybe related" result set. Just the right context, scoped correctly, in as few tokens as possible.
The AI memory problem isn't "how do we store knowledge in a connected way." It's "how do we get the right context to the right prompt at the right time." Most teams are so obsessed with elegant knowledge representation that they've lost sight of this.
Hierarchies win
Hierarchical data structures solve this cleanly. They keep the relationships. They categorize correctly. But unlike graphs, updates don't cascade. Changing an entry doesn't shift how other entries relate, because the structure is the hierarchy itself, not computed relationships between nodes.
Index-backed subtree lookups instead of graph traversal. Scoped retrieval instead of fuzzy matching. Deterministic results instead of "we found 47 maybe-relevant nodes." Retrieval stays flat as data grows. Update cost is constant. No recomputation.
What I built
I open sourced Lithium to prove this. Hierarchical versioned storage on PostgreSQL ltree. Three packages, 12 KB total. Scoped context retrieval. Deterministic. No wasted tokens.
It's free. Go build something.
npm install @lithium-ai/core @lithium-ai/postgres @lithium-ai/mcp
Top comments (0)