The naive version of agent memory is "save everything, retrieve what's relevant." I ran that version for three weeks. It failed in a way nobody warns you about: not with an error, but with a slow, silent rot.
By week three, recall quality had dropped noticeably. Not because the retrieval was broken — because the library itself was wrong.
This post is about the half of memory design that gets skipped: forgetting. I'll walk through the three ways a memory library rots, and the engineering we built to counter it.
The three kinds of rot
1. Duplicate pileup. The same decision gets restated across ten conversations and stored ten times. At recall time, ten hits eat ten slots of context, carrying one idea's worth of information.
2. Stale facts that outrank fresh ones. Three weeks ago, "database is Postgres." Last week, "migrated to SQLite." Both memories live in the library — and the old one often has higher weight, because it was recorded more times. The system confidently recalls the outdated decision.
3. Noise drowning signal. Chitchat, abandoned debugging paths, stack traces from problems long solved — all stored. Volume grows faster than value until the key decisions are buried.
All three share one root cause: nothing is ever wrong at write time. Memory rots because there's no maintenance step.
What we built: three mechanisms
I build a local-first memory layer (HyperMarrow — disclosure: it's my own product; I'm describing the engineering tradeoffs, not selling anything here). These are the parts that actually took time to get right.
1. Decay, not deletion
Every memory carries a lifecycle class:
- Regular memories decay in weight over time. If recall never hits them, they fade until they effectively stop participating in retrieval.
- Core memories are manually pinned — tech decisions, client constraints, anything that must never fade. They're exempt from decay and always eligible for recall.
Why decay instead of hard-delete? Because "stale-looking" is often just "not relevant yet." A memory about a weird network timeout looks useless for a month — until you hit the same timeout in a new project. Weight gives you a soft landing: faded but recoverable, re-boosted the next time it proves useful.
2. Consolidation as a batch job
Merging duplicates needs a global view (which memories overlap across the whole library?), so we never do it inline. It runs offline on a schedule, plus a threshold trigger when a single topic accumulates too many entries.
The merge criterion is semantic, not lexical: "database is PG" and "backend storage chose PostgreSQL" look nothing alike as strings. Rough candidate pairs come from embedding similarity, then a model pass confirms "is this the same decision?" Losers aren't deleted — they're marked superseded, with a pointer to the winner. Audit trails live in that pointer.
3. A dashboard, because rot is invisible
You can't maintain what you can't see. The minimum viable observability for a memory library:
- total size trend (healthy looks like: fast growth, then a plateau)
- which memories are aging out
- which ones get hit constantly
Decay curves make rot visible before it's expensive. The first time you watch a stale decision's weight fall below retrieval threshold, the whole design clicks.
The uncomfortable tradeoff
Forgetting has a cost, and it's worth saying plainly: a decay-heavy system will sometimes fade something you wanted. The pin (core memory) is the escape hatch, but pinning is manual, and manual means you'll forget to pin things.
Our current rule of thumb: pin decisions, let everything else earn its weight through recall. If a fact matters, it'll come up in conversation — and the act of restating it refreshes it.
Questions for you
How does your agent memory handle obsolescence? Have you built versioning for memories — supersede chains where only the latest version is recallable but history stays auditable? That's the piece I'm least satisfied with, and I'd genuinely like to hear how others solved it.
Top comments (0)