A memory layer is the one component in an agent stack where the tests can all pass while the system quietly rots.
That's not a testing-discipline problem. It's a shape problem. Every other part of the stack is a function: input in, output out, assert equality. A memory store is a relationship graph over time. Its failures aren't wrong values — they're stale values, ambiguous values, and values that used to be true. None of those show up in an assertion like expect(recall(q)).toEqual([...]).
Here are the six assertions I ended up automating, roughly in order of how early they catch something.
1. Supersession integrity
If entries can supersede one another, the graph has to be well-formed. Three checks:
- Reciprocal links: if
A.superseded_by = B, thenB.supersedescontainsA. A one-directional link means your history is unreadable in one of the two directions. - No cycles.
A → B → C → Ais always a bug, and it's the kind of bug that makes recall hang or return an arbitrary member of the cycle. - No orphan supersession:
superseded_bypointing at an id that was hard-deleted.
This one runs in milliseconds and has caught more real bugs for me than anything else on the list. It's a pure graph invariant, so it needs no fixture and no judgement.
2. Staleness ratio
Of the entries your recall path actually returns over a sample of real queries, what fraction are already marked superseded?
This is the closest thing to a single health number I've found. It should be near zero, and its trend matters more than its value. A store where this climbs from 2% to 15% over a month is telling you consolidation isn't running, or isn't looking at the right population.
The assertion isn't a magic threshold — it's a budget. Pick a number you're willing to defend, fail the test when you exceed it, and treat the failure as "your consolidation job is broken," not "the test is too strict."
3. Recall determinism
Same store state, same query, same result set — including the ordering.
This sounds trivial and it isn't, because ties happen constantly: embedding scores cluster, timestamps collide, and whatever sort your store uses is not guaranteed stable. Non-deterministic recall produces the worst class of bug: an agent that behaves differently on Tuesday for no visible reason, which you then misattribute to the model.
If you genuinely need variety, make it explicit and seeded. Don't let it fall out of an unstable sort.
4. Write-time dedupe
Write the same assertion twice with different wording, then assert that the store holds one canonical entry — not two — with both sources attached.
The interesting failure isn't the obvious duplicate. It's the near-duplicate that shouldn't be merged: "we use Postgres" versus "we use Postgres for the billing service only." A good test corpus contains both cases, and the assertion is that the first merges and the second does not.
That pair of tests is the only thing standing between you and an over-eager merge threshold that quietly averages two distinct decisions into one useless entry.
5. Conflict surfacing
Inject a deliberately contradictory pair, then assert that the recall path flags the conflict rather than silently picking one.
This is a behaviour test, not a data test, and it's the one most projects skip. An agent that confidently reads a stale decision is strictly worse than an agent that says "I have two conflicting notes, which is current?" The first one is fast and wrong; the second one costs you four seconds and a question.
Assert the flag exists. Assert the conflicting pair is named in the output. Don't assert the resolution, because resolution is domain-specific.
6. Decay monotonicity
All else equal, an entry that hasn't been referenced in six months must not outrank a fresh entry of the same type and comparable relevance.
All else is never actually equal, so test it as a controlled pair: two synthetic entries, identical type and text shape, different ages, identical access counts. The older one must not win. If it does, your decay function is decorative.
Building the corpus
Assertions are cheap; the fixture is the work. Two sources, in order of value:
Golden pairs from real sessions. Take twenty real queries you've actually run, and write down the memory entry a good recall would return. This is a slow, manual job and it's worth it — twenty honest pairs beat two hundred generated ones.
Synthetic invariants. The graph checks, the dedupe pair, the conflict pair, the decay pair. These need no domain knowledge and can be written in an afternoon.
Then run everything against a seeded fixture store, never production. Memory tests that read your real store are tests you'll disable within a week, because they'll fail for reasons that have nothing to do with the code you changed.
What a failure actually means
Map each assertion to a cause, or the test suite becomes noise:
| Assertion fails | Almost always means |
|---|---|
| Supersession integrity | A write path that bypassed the link bookkeeping |
| Staleness ratio | Consolidation isn't running, or isn't scoped to all entry types |
| Recall determinism | Unstable sort, or a tie broken by insertion order |
| Write-time dedupe | Merge threshold too loose (merged) or too tight (didn't) |
| Conflict surfacing | Recall returns text without checking validity windows |
| Decay monotonicity | Decay weight computed but not applied to ranking |
Six assertions, one afternoon to write, and the failure table is the actual deliverable — because the point isn't that the tests go green. It's that when a green suite turns red, you know which subsystem moved.
This is part of a series on building a local-first memory layer for coding agents. Part 1 covered the failure modes getting memory to work at all; part 2 covered consolidation — the stage that keeps a store from rotting:
- I gave my AI coding agents a local long-term memory layer — 8 things that broke
- Consolidation: the half of agent memory nobody builds
If you want to see how the four access paths (MCP, desktop, Python/Node SDK) are wired: https://hm.qianshi.cool/api/v2/dl?from=devto
Top comments (3)
I would add provenance preservation as a seventh invariant. When two memories merge or one supersedes another, the derived entry must keep the source set, trust labels, and audience limits of its inputs. Otherwise consolidation can turn an untrusted quote into an apparently internal instruction even while every graph link is valid. A useful fixture deliberately mixes trusted and untrusted sources, summarizes them, and asserts that retrieval still exposes the distinction at the action boundary.
Adding provenance preservation as invariant #7 is a sharper framing than what I had — my sixth invariant only covers trust labels at write time, not the merge path. The failure mode you describe is the nasty one: consolidation launders an untrusted quote into an apparently-internal instruction while every individual graph link stays valid, so integrity checks pass and the boundary still gets crossed.
The fixture you describe — mix trusted and untrusted sources, summarize, then assert the distinction survives retrieval at the action boundary — is exactly the regression test this needs. Honest status: we don't ship it yet. Right now trust labels propagate on supersede, but audience limits don't survive a three-way merge, which is precisely the hole you're pointing at. It's on the list ahead of any new features.
Disclosure: I build the memory layer this post describes, so weigh my agreement accordingly. The consolidation write path is where I'd want adversarial review most.
看到这个话题说两句：
这件事我有点切身体会，因为自己在做本地 AI 记忆相关的东西。
主流大模型每次对话都是独立上下文，关掉窗口它手里什么都不剩，这不是 bug 是设计。真正麻烦的是怎么把有用的记下来、把没用的忘掉。
我做的桌面端主打本地优先和长期记忆，数据不出本机，这点我特别在意。感兴趣可以搜一下智商藏不住，上面算是一点实践分享，不是广告。