TL;DR: LLM agents forget everything because their "memory" is a sliding window. Sparse Distributed Memory (Kanerva, 1988) gives you a content-addressable long-term store with an astronomically large address space — and you can implement a working core in ~150 lines of pure Python, zero dependencies.
Why LLM agents forget everything
Context windows are the bottleneck. An agent that worked with you last Tuesday has no idea what you agreed on by Friday — unless you feed the whole transcript back, which is expensive and still hits the limit.
Vector databases are the usual fix, but they are approximate in a particular way: they measure similarity, not association. A vector DB can find "the chunk most like this query," but it does not naturally reconstruct a memory from a partial, noisy cue the way associative memory does.
The gap between "chatbot" and "partner" is memory: durable, cue-addressable, and quietly consolidated over time.
What is Sparse Distributed Memory (Kanerva, 1988)?
Sparse Distributed Memory is a mathematical model of associative memory from Pentti Kanerva's 1988 book. The core idea:
-
Huge address space: binary addresses of length n give
2^npossible locations. With n=1000 that is2^1000— more addresses than atoms in the observable universe. (Compare: a typical 1536-dim embedding vector.) -
Sparse physical storage: you cannot allocate
2^1000slots, so you allocate a few million hard locations at random and let each memory write to the neighborhood of its address. - Hamming distance + activation radius: an address "activates" every hard location within a radius r (by Hamming distance). Reading averages the contents of the activated locations; writing adds to them.
- Content-addressable: read with a noisy or partial address and you still land near the right neighborhood — this is what makes it work like a brain rather than a hash table.
Implementing SDM in Python
Here is a compact, dependency-free core: address generation, write, read, and a winner-take-all decode for noisy cues.
import random
from collections import defaultdict
class SparseDistributedMemory:
"""Kanerva SDM — hard locations, Hamming activation, distributed read/write."""
def __init__(self, n=1000, num_locations=100_000, radius=451, seed=42):
self.n = n # address bit length
self.radius = radius # activation radius (Hamming)
rng = random.Random(seed)
# hard locations: random binary addresses
self.locations = [rng.getrandbits(n) for _ in range(num_locations)]
# each hard location has an integer content vector (accumulator)
self.content = [0] * num_locations
def _activate(self, address):
hits = []
for i, loc in enumerate(self.locations):
if bin(loc ^ address).count("1") <= self.radius:
hits.append(i)
return hits
def write(self, address, pattern, strength=1):
"""Associate `pattern` (int bitmask) with `address`."""
for i in self._activate(address):
# accumulate: +1 where pattern has a 1-bit, -1 where it has a 0-bit
self.content[i] += strength if (pattern & 1) else -strength
pattern >>= 1
def read(self, address):
"""Return the average content vector of the activated neighborhood."""
hits = self._activate(address)
if not hits:
return None
sums = [0] * self.n
for i in hits:
c = self.content[i]
for b in range(self.n):
sums[b] += 1 if (c >> b) & 1 else -1
return [s / len(hits) for s in sums]
def decode(self, address, threshold=0.0):
"""Read + threshold into a clean binary pattern."""
vec = self.read(address)
if vec is None:
return None
out = 0
for b, v in enumerate(vec):
if v > threshold:
out |= (1 << b)
return out
This is intentionally simplified (production uses block addressing and accumulation weights), but it captures the mechanism: write spreads a pattern over a neighborhood; read averages the neighborhood back into an approximation.
Properties that matter for agents
- Zero external dependencies — pure Python, thread-safe per memory bank
- Predictive pre-activation — you can probe with a partial cue before committing
- Fractal compression — dense 100:1 patterns stored sparsely
- No RAG pipeline required for long-term recall
Integrating with a hippocampus (memory replay)
A raw SDM store is static. What makes memory feel like memory is consolidation:
- Idle-time replay: compress and replay past sessions 10–20× during idle, "steadier answers over time"
- Forgetting curve: Ebbinghaus-style decay for unimportant traces
- Schema consolidation: episodic → semantic → core, auto-triggered
- Archival pruning: demote cold traces to recoverable archive instead of hard-deleting
These are the mechanisms shipped in MeshCtx's Memory Engine v2 (FSRS spaced repetition + context markers + sleep-phase offline consolidation).
Results & benchmarks
Measured locally (2026-08-19, independent):
- LongMemEval (48 questions): strict EM 52–54% (4 samples: 24/25/26/25, oracle-subset methodology) · semantic judge 83.3% (40/48) — roughly 81–85% of GPT-4o-no-memory-full-context (60–64%)
- 16KB budget fairness: brain-region curated 33.3% vs brute-force truncation 25.0% (+8.3 pp), 4.5× fewer tokens
- Tool-output compression: 5008 B → 223 B (−95.5%), agent still completes the task
- Full regression: 3095 passed / 0 failed
Full open source
The complete framework — 17-region brain architecture, SDM memory engine, genetic-algorithm evolution engine (API-controlled), 5-model swarm review — is open core under AGPLv3, free for individual use:
- GitHub: https://github.com/LucyAndLuna2023/meshctx
- Site: https://meshctx.com
- Governance & telemetry details: https://meshctx.com/governance.html · https://meshctx.com/telemetry.html
Discussion
- When is SDM overkill? Short sessions with a tiny context — a list in RAM is fine.
- When is it essential? Long-running personal assistants, agents that accumulate a user's history across weeks, anything where "what did we agree on last Tuesday?" must not cost you the whole transcript.
I'd love feedback from people who have shipped associative-memory systems. What broke in production for you?
Cross-posted from the MeshCtx engineering log. Individual use is free (AGPLv3 open core).
Top comments (1)
The implementation of Sparse Distributed Memory as a solution for long-term memory in LLM agents is fascinating, especially the way it addresses the limitations of context windows and vector databases. The concept of a content-addressable memory system that can handle noisy or partial cues is a significant leap towards creating more intelligent and adaptable AI. One improvement idea could be to explore how different activation radii affect performance in various scenarios, which might yield insights into optimizing memory retrieval. If you’re looking for help refining this implementation or exploring its potential applications in real-world projects, I’d be glad to discuss a paid collaboration. What do you see as the next step for enhancing this model?