A memory layer is the one component in an agent stack where the tests can all pass while the system quietly rots.
That's not a testing-discipline problem. It's a shape problem. Every other part of the stack is a function: input in, output out, assert equality. A memory store is a relationship graph over time. Its failures aren't wrong values — they're stale values, ambiguous values, and values that used to be true. None of those show up in an assertion like expect(recall(q)).toEqual([...]).
Here are the six assertions I ended up automating, roughly in order of how early they catch something.
1. Supersession integrity
If entries can supersede one another, the graph has to be well-formed. Three checks:
- Reciprocal links: if
A.superseded_by = B, thenB.supersedescontainsA. A one-directional link means your history is unreadable in one of the two directions. - No cycles.
A → B → C → Ais always a bug, and it's the kind of bug that makes recall hang or return an arbitrary member of the cycle. - No orphan supersession:
superseded_bypointing at an id that was hard-deleted.
This one runs in milliseconds and has caught more real bugs for me than anything else on the list. It's a pure graph invariant, so it needs no fixture and no judgement.
2. Staleness ratio
Of the entries your recall path actually returns over a sample of real queries, what fraction are already marked superseded?
This is the closest thing to a single health number I've found. It should be near zero, and its trend matters more than its value. A store where this climbs from 2% to 15% over a month is telling you consolidation isn't running, or isn't looking at the right population.
The assertion isn't a magic threshold — it's a budget. Pick a number you're willing to defend, fail the test when you exceed it, and treat the failure as "your consolidation job is broken," not "the test is too strict."
3. Recall determinism
Same store state, same query, same result set — including the ordering.
This sounds trivial and it isn't, because ties happen constantly: embedding scores cluster, timestamps collide, and whatever sort your store uses is not guaranteed stable. Non-deterministic recall produces the worst class of bug: an agent that behaves differently on Tuesday for no visible reason, which you then misattribute to the model.
If you genuinely need variety, make it explicit and seeded. Don't let it fall out of an unstable sort.
4. Write-time dedupe
Write the same assertion twice with different wording, then assert that the store holds one canonical entry — not two — with both sources attached.
The interesting failure isn't the obvious duplicate. It's the near-duplicate that shouldn't be merged: "we use Postgres" versus "we use Postgres for the billing service only." A good test corpus contains both cases, and the assertion is that the first merges and the second does not.
That pair of tests is the only thing standing between you and an over-eager merge threshold that quietly averages two distinct decisions into one useless entry.
5. Conflict surfacing
Inject a deliberately contradictory pair, then assert that the recall path flags the conflict rather than silently picking one.
This is a behaviour test, not a data test, and it's the one most projects skip. An agent that confidently reads a stale decision is strictly worse than an agent that says "I have two conflicting notes, which is current?" The first one is fast and wrong; the second one costs you four seconds and a question.
Assert the flag exists. Assert the conflicting pair is named in the output. Don't assert the resolution, because resolution is domain-specific.
6. Decay monotonicity
All else equal, an entry that hasn't been referenced in six months must not outrank a fresh entry of the same type and comparable relevance.
All else is never actually equal, so test it as a controlled pair: two synthetic entries, identical type and text shape, different ages, identical access counts. The older one must not win. If it does, your decay function is decorative.
Building the corpus
Assertions are cheap; the fixture is the work. Two sources, in order of value:
Golden pairs from real sessions. Take twenty real queries you've actually run, and write down the memory entry a good recall would return. This is a slow, manual job and it's worth it — twenty honest pairs beat two hundred generated ones.
Synthetic invariants. The graph checks, the dedupe pair, the conflict pair, the decay pair. These need no domain knowledge and can be written in an afternoon.
Then run everything against a seeded fixture store, never production. Memory tests that read your real store are tests you'll disable within a week, because they'll fail for reasons that have nothing to do with the code you changed.
What a failure actually means
Map each assertion to a cause, or the test suite becomes noise:
| Assertion fails | Almost always means |
|---|---|
| Supersession integrity | A write path that bypassed the link bookkeeping |
| Staleness ratio | Consolidation isn't running, or isn't scoped to all entry types |
| Recall determinism | Unstable sort, or a tie broken by insertion order |
| Write-time dedupe | Merge threshold too loose (merged) or too tight (didn't) |
| Conflict surfacing | Recall returns text without checking validity windows |
| Decay monotonicity | Decay weight computed but not applied to ranking |
Six assertions, one afternoon to write, and the failure table is the actual deliverable — because the point isn't that the tests go green. It's that when a green suite turns red, you know which subsystem moved.
This is part of a series on building a local-first memory layer for coding agents. Part 1 covered the failure modes getting memory to work at all; part 2 covered consolidation — the stage that keeps a store from rotting:
- I gave my AI coding agents a local long-term memory layer — 8 things that broke
- Consolidation: the half of agent memory nobody builds
If you want to see how the four access paths (MCP, desktop, Python/Node SDK) are wired: https://hm.qianshi.cool/api/v2/dl?from=devto
Top comments (0)