DEV Community: qianqiuwanzi

Testing an agent memory layer: the assertions that actually catch decay

qianqiuwanzi — Wed, 16 Sep 2026 06:28:57 +0000

A memory layer is the one component in an agent stack where the tests can all pass while the system quietly rots.

That's not a testing-discipline problem. It's a shape problem. Every other part of the stack is a function: input in, output out, assert equality. A memory store is a relationship graph over time. Its failures aren't wrong values — they're stale values, ambiguous values, and values that used to be true. None of those show up in an assertion like expect(recall(q)).toEqual([...]).

Here are the six assertions I ended up automating, roughly in order of how early they catch something.

1. Supersession integrity

If entries can supersede one another, the graph has to be well-formed. Three checks:

Reciprocal links: if A.superseded_by = B, then B.supersedes contains A. A one-directional link means your history is unreadable in one of the two directions.
No cycles. A → B → C → A is always a bug, and it's the kind of bug that makes recall hang or return an arbitrary member of the cycle.
No orphan supersession: superseded_by pointing at an id that was hard-deleted.

This one runs in milliseconds and has caught more real bugs for me than anything else on the list. It's a pure graph invariant, so it needs no fixture and no judgement.

2. Staleness ratio

Of the entries your recall path actually returns over a sample of real queries, what fraction are already marked superseded?

This is the closest thing to a single health number I've found. It should be near zero, and its trend matters more than its value. A store where this climbs from 2% to 15% over a month is telling you consolidation isn't running, or isn't looking at the right population.

The assertion isn't a magic threshold — it's a budget. Pick a number you're willing to defend, fail the test when you exceed it, and treat the failure as "your consolidation job is broken," not "the test is too strict."

3. Recall determinism

Same store state, same query, same result set — including the ordering.

This sounds trivial and it isn't, because ties happen constantly: embedding scores cluster, timestamps collide, and whatever sort your store uses is not guaranteed stable. Non-deterministic recall produces the worst class of bug: an agent that behaves differently on Tuesday for no visible reason, which you then misattribute to the model.

If you genuinely need variety, make it explicit and seeded. Don't let it fall out of an unstable sort.

4. Write-time dedupe

Write the same assertion twice with different wording, then assert that the store holds one canonical entry — not two — with both sources attached.

The interesting failure isn't the obvious duplicate. It's the near-duplicate that shouldn't be merged: "we use Postgres" versus "we use Postgres for the billing service only." A good test corpus contains both cases, and the assertion is that the first merges and the second does not.

That pair of tests is the only thing standing between you and an over-eager merge threshold that quietly averages two distinct decisions into one useless entry.

5. Conflict surfacing

Inject a deliberately contradictory pair, then assert that the recall path flags the conflict rather than silently picking one.

This is a behaviour test, not a data test, and it's the one most projects skip. An agent that confidently reads a stale decision is strictly worse than an agent that says "I have two conflicting notes, which is current?" The first one is fast and wrong; the second one costs you four seconds and a question.

Assert the flag exists. Assert the conflicting pair is named in the output. Don't assert the resolution, because resolution is domain-specific.

6. Decay monotonicity

All else equal, an entry that hasn't been referenced in six months must not outrank a fresh entry of the same type and comparable relevance.

All else is never actually equal, so test it as a controlled pair: two synthetic entries, identical type and text shape, different ages, identical access counts. The older one must not win. If it does, your decay function is decorative.

Building the corpus

Assertions are cheap; the fixture is the work. Two sources, in order of value:

Golden pairs from real sessions. Take twenty real queries you've actually run, and write down the memory entry a good recall would return. This is a slow, manual job and it's worth it — twenty honest pairs beat two hundred generated ones.

Synthetic invariants. The graph checks, the dedupe pair, the conflict pair, the decay pair. These need no domain knowledge and can be written in an afternoon.

Then run everything against a seeded fixture store, never production. Memory tests that read your real store are tests you'll disable within a week, because they'll fail for reasons that have nothing to do with the code you changed.

What a failure actually means

Map each assertion to a cause, or the test suite becomes noise:

Assertion fails	Almost always means
Supersession integrity	A write path that bypassed the link bookkeeping
Staleness ratio	Consolidation isn't running, or isn't scoped to all entry types
Recall determinism	Unstable sort, or a tie broken by insertion order
Write-time dedupe	Merge threshold too loose (merged) or too tight (didn't)
Conflict surfacing	Recall returns text without checking validity windows
Decay monotonicity	Decay weight computed but not applied to ranking

Six assertions, one afternoon to write, and the failure table is the actual deliverable — because the point isn't that the tests go green. It's that when a green suite turns red, you know which subsystem moved.

This is part of a series on building a local-first memory layer for coding agents. Part 1 covered the failure modes getting memory to work at all; part 2 covered consolidation — the stage that keeps a store from rotting:

If you want to see how the four access paths (MCP, desktop, Python/Node SDK) are wired: https://hm.qianshi.cool/api/v2/dl?from=devto

Consolidation: the half of agent memory nobody builds

qianqiuwanzi — Wed, 16 Sep 2026 03:59:07 +0000

Every agent memory project I've seen (including the first two versions of mine) builds two things: record and recall. You write something down, you read it back. Ship it.

Then, three weeks in, recall quality drops — and nobody can explain why, because nothing in the system got worse. Nothing in the system got cleaned, either.

That's the missing stage. Consolidation.

Why "just store everything" degrades

Three mechanisms, all of them boring and all of them certain:

Duplicates. The same fact gets recorded five times across five sessions, each with slightly different wording, because nothing at write time knew it already existed. Retrieval now returns five near-identical entries and burns its budget on one fact.

Silent contradiction. The store says using poetry for dependency management and switched to uv because CI install time — both true, one superseded. Nothing marks which one is current. The agent picks whichever one scores higher on embedding similarity that day. You get non-deterministic behaviour that looks like a model problem and is actually a data problem.

Unbounded growth. If nothing is ever pruned or compacted, a fixed retrieval budget has to cover an ever-larger candidate set. Recall doesn't fail loudly; it just gradually returns more generic, less useful context.

None of these are fixed by a better embedding model. They're fixed by a process that runs on the store itself.

The four operations

Consolidation is not one thing. It's four, and they have different failure modes.

1. Merge

Two entries that assert the same thing should become one, with both sources attached. The hard part is not detecting similarity — it's deciding when similarity means same assertion versus related but distinct. "We use Postgres" and "we use Postgres in the billing service only" are 0.9 similar and must not be merged.

Practical rule I use: merge automatically on high-confidence matches, and queue anything in the ambiguous band for review. Do not let the ambiguous band be auto-merged. That band is exactly where the information lives.

2. Supersede — never delete

When a decision is reversed, the old entry should remain, marked as superseded, linked to the entry that replaced it. Deleting it is the tempting option and it's the wrong one, because the reason a decision was reversed is the thing you'll need again in three months.

An entry needs, at minimum:

{
  "id": "mem_7f3a",
  "type": "decision",
  "text": "use uv instead of poetry for dependency resolution",
  "rationale": "poetry's lockfile resolution was too slow in monorepo CI",
  "created_at": "2026-04-11T09:12:00Z",
  "valid_from": "2026-04-11",
  "valid_until": null,
  "supersedes": ["mem_2c91"],
  "superseded_by": null,
  "sources": ["session/2026-04-11#turn-42"]
}

supersedes / superseded_by turn your memory store into a graph rather than a bag. It's what lets you answer "why is it like this now" instead of only "what is it now".

3. Decay

Old entries shouldn't be deleted; they should lose rank. Store a decay weight derived from age and access, and let retrieval use it as one signal among several. An entry that hasn't been relevant in six months but is still true should be cheap to keep and cheap to ignore.

Deletion is for garbage (duplicated, malformed, explicitly retracted). Everything else ages.

4. Conflict check

Two checks, at two different times:

At write time: does this new entry contradict something already stored? If yes, don't refuse the write — mark the pair and resolve it in the next consolidation pass.
At recall time: if two returned entries conflict, surface that. An agent that confidently reads a stale decision is worse than an agent that says "I have two conflicting notes here."

Design details that actually matter

Type your entries. Free-text blobs retrieve badly. fact / decision / pitfall retrieve under different policies, and mixing them means you always get an average of everything. "This project uses uv" (fact), "we chose uv because CI needed to be fast" (decision), and "poetry's lockfile has a path bug in monorepos" (pitfall) need different ranking.

Make consolidation idempotent and reviewable. It should produce a diff you can read, not an opaque rewrite. If a background process silently rewrites your project history and you can't audit it, you've built a system you can't trust — and you won't notice until it has already misled you a dozen times.

Run it out of band. Not in the request path. Consolidation is a batch job that runs between sessions; doing it inline adds latency to the exact moment you're waiting on the agent.

Anchor memories to something stable. Bind entries to a git remote or a project id, not an absolute path. Directory renames and moves are common and will otherwise silently detach the entire history.

Two things I got wrong

Over-merging first. My earliest merge threshold was too permissive and it collapsed distinct decisions into one averaged entry. Recovering the original two required going back to raw session logs. The information was gone from the store and only existed in the transcript. Aggressive merging feels like progress and quietly destroys the thing you built the system for.

Treating consolidation as a one-shot cleanup. I ran it once, was happy with the result, and made it manual. It needs to be scheduled. The store doesn't stay tidy; it's a garden, not a build artifact.

How to tell it's working

I deliberately won't quote numbers here, because my deployment is small and any percentage I gave you would be noise dressed as evidence. What I do track are proxies that are meaningful even at small scale:

Signal	What it tells you
Count of superseded entries	Whether reversals are being captured at all
Longest supersession chain	Whether you can reconstruct decision history
Share of recall results that are already superseded	Your staleness rate — should trend toward zero
Duplicate clusters merged per pass	Whether write-time filtering is too loose
Size of the ambiguous review queue	Whether you're guessing too much

If the staleness share is climbing, your consolidation isn't running or isn't looking at the right thing. That ratio is the closest thing to a health check I've found.

Where this sits in my setup

I run this as a local-first memory layer — everything on local disk, retrieval local, no cloud round-trip, and consolidation as a scheduled offline pass. The tradeoffs are real and I'll state them plainly: no cross-device sync (you move the files yourself), no team-shared memory, and it's Windows-only right now. macOS is planned, not shipped.

Access is over MCP, a desktop connection, or Python / Node SDKs depending on the client. And one honest caveat for anyone outside China: the trial signup uses a Chinese phone number or WeChat login, so it's realistically aimed at a China-based audience — I'd rather say that up front than have you hit a wall at the signup screen.

If you're building memory for an agent and you've reached the point where storing works but recall is degrading, consolidation is probably the missing piece. The two questions worth asking yourself: can I reconstruct why a decision changed? and do I know what fraction of what I hand the model is stale? If either answer is no, that's the next thing to build.

I write about this as I go — part 1 covered the eight failure modes I hit getting memory to work at all:

→ I gave my AI coding agents a local long-term memory layer — 8 things that broke

And if you want to look at the implementation: https://hm.qianshi.cool/api/v2/dl?from=devto

I gave my AI coding agents a local long-term memory layer — 8 things that broke

qianqiuwanzi — Wed, 16 Sep 2026 02:53:20 +0000

Every new chat window starts from zero.

I've been running coding agents (Claude Code, Cursor, Codex, and a rotating cast of others) long enough that this stopped being a minor annoyance and became the actual bottleneck. The agent isn't bad at the task. It's bad at remembering why the task looks the way it does — that we moved off approach A last Tuesday, that the flaky test is flaky for a known reason, that the config flag exists because of a specific incident.

So I spent a while trying to fix it. Then I spent longer fixing the fix.

What I tried first

1. Project rule files (CLAUDE.md, .cursorrules)

Works for static facts. Fails for anything that changes. The moment a decision gets revised, the rule file is stale and now actively misleading — and nothing tells you it's stale.

2. Pasting context manually

Reliable, until you forget. And you will forget, on the day you're moving fastest. Also: pasting 2000 tokens of background means those tokens are now competing with your actual question.

3. A very long system prompt

It grows. Then you start maintaining it like a codebase, except there's no test suite, so you can't tell what's still load-bearing. I removed a paragraph I was sure was useless and watched output quality drop.

The pattern in all three: I was asking the model — or the client — to be the memory. Neither of them is built for that.

The approach that actually held

Stop trying to make the model remember. Give it something to query.

Memory goes to disk, outside the context window. The agent gets tools to read and write it. Four responsibilities, kept separate:

record — accept a memory
recall — retrieve the relevant ones for the current task
consolidation — merge, deduplicate, and resolve conflicts over time
file-bridge — pull project files in as raw material

Keeping those four apart was the single most useful architectural decision, because every failure below lands squarely in one of them.

The 8 things that broke

1. The MCP server's lifetime is not yours

The MCP layer is a thin proxy; the actual memory service is a separate long-running process. If it isn't running, tool calls fail — and the error surfaces at the moment the agent needs memory most, which is exactly the wrong time to discover it. Treat the transport process as stateless and disposable, keep all state on disk, and make "service not running" a clear, actionable message rather than a stack trace.

2. "Remember everything" is worse than remembering nothing

My first version stored indiscriminately and dumped the top N results into context. Output got worse. Irrelevant memories don't just waste tokens — they actively mislead. Recall quality is a ranking problem, and ranking needs a relevance signal, not a recency sort.

3. Two agents writing at once

I run more than one agent. When two wrote to the same store concurrently, I got interleaved and occasionally contradictory records. Fixed with a single-writer discipline plus locking. Boring, unglamorous, non-optional.

4. Memories go stale and start contradicting each other

After a few weeks: "we use Postgres" and "we migrated off Postgres." Both true at their own timestamp, both useless to an agent that can't tell which is current. You need an explicit consolidation pass that detects conflict and supersedes — not just dedup by string similarity. This is the part I underestimated most.

5. Recall latency is on the critical path

Reading and ranking from disk on every turn adds visible delay. Cheap fix: maintain an index and impose a hard time budget on recall. If it can't rank in the budget, return fewer results rather than making the user wait.

6. Anything that reads project files will read .env

If a component ingests project files, it will eventually ingest a secret. Filter at the ingestion boundary, before it's written — not at read time, and not "later." Retrofitting a filter onto a store that already contains credentials is a genuinely bad afternoon.

7. Deciding what's worth ingesting is harder than reading it

The file-bridge is trivial to build and hard to tune. Reading a directory is easy. Knowing which files carry durable context — and which are generated noise — is the actual product.

8. Evaluation is the thing nobody budgets for

How do you know recall improved? I couldn't answer this for weeks, which meant "improvements" were vibes. What worked: keep a replay set of real past tasks and check whether the right memories surface for each. Without it, you're tuning blind.

Trade-offs I accepted on purpose

These are decisions, not oversights, so I'll state them plainly:

Decision	What it costs you
No cloud sync — memory is written to local disk only	Moving machines is manual
Retrieval runs locally, no cloud calls	No team-shared memory
Windows only right now — macOS is planned for a closed beta in Q1 2027	Bad news today if you're on macOS

The last one I'd change if I could. I can't yet.

What this is, if you want to try it

The product is HyperMarrow. It's a local long-term memory layer for coding agents — the four modules above, with four ways to connect (MCP server, direct desktop integration, Python SDK, Node SDK). It runs on Windows.

There's a 30-day full-feature trial, with no email/password to set up — sign-in is by WeChat QR or SMS code. I'll be straight with you, because you'll find out anyway: that also means it currently expects a Chinese phone number, so if you're outside China the download is mostly useless to you. The eight failure modes above are the transferable part.

→ https://hm.qianshi.cool/api/v2/dl?from=devto

If a different approach works better for you, the eight failure modes above are the useful part of this post — they're the same whether you build it yourself or adopt something. Steal the checklist.