How I measure whether my agent's memory works

Originally published at olund.dev.

I run my coding agents on top of a personal memory system I built: durable
facts live in a markdown file, an indexer embeds them with a local model
(bge-m3), and a recall command does semantic search over the result. Sessions
query it constantly. It looked healthy for months.

Then I asked it something I knew was in there: whether my system supports
Windows. The fact existed, word for word, in the durable file. recall
returned zero hits from it. Not ranked low - absent.

This post is the story of finding out why, and more usefully, the methods that
made every step provable instead of plausible. If you are building agent
memory, the specific bug will probably not be yours. The measurement traps
will be.

The bug: the text was never embedded

My indexer chunked the memory file by markdown section. One section had grown
to 904 tokens. The embedding library I use (fastembed) silently truncates
input at a default of 512 tokens unless you override it, and nothing in my
code did. Nothing upstream bounded chunk size either.

So the encoder saw the first 1687 characters of a 3090-character section and
threw away the rest - 45.4% of the text, containing three entire facts. No
error, no warning, no log line. The stored vector simply did not contain
their meaning. Retrieval was not failing to find them. There was nothing to
find.

The nasty property: this bug is monotone with growth. Every new fact appended
to the section pushed more content past the cut. The memory system degraded
because it was being used.

Proving it causally, not correlationally

"The section is long and recall is bad" is a correlation. Before touching any
code I wanted the mechanism pinned, because my first diagnosis was wrong (more
on that below). The probe that settled it:

query text	distance to the stored chunk
the full 3090-char section	0.0000
only the first 1687 chars	0.0000
only the discarded 1403-char tail	0.4247

The full section and its truncated prefix embed to the same vector - exact
zero distance. The tail contributes nothing at all. That is not what dilution
looks like; that is a step function. Correlating each fact's character offset
with its self-retrieval distance gave a Spearman rho of +0.952: a cliff at the
cut position, not a gradient.

This mattered because my original hypothesis was centroid dilution - one
embedding averaging over many facts, dominated by the numerous ones. Dilution
suggests fixes like re-ranking or splitting long sections more finely.
Truncation means data is missing from the index entirely, and no amount of
ranking cleverness can retrieve a vector that was never computed. Wrong
mechanism, wrong fix. The ten-minute probe was cheaper than shipping the
wrong repair.

The fix that looks right and the one that is

The one-line fix exists: the model genuinely supports an 8192-token window,
and a single .with_max_length(8192) restores it. I rejected it for three
reasons:

1. It trades a proven defect for an unproven one - the multi-fact centroid I could now actually measure. Even on a five-fact section that fits the window completely, querying any single fact's verbatim text returned its section at distance 0.21-0.34, against a 0.0000 floor for whole-chunk self-queries. Bundling costs retrieval sharpness even without truncation.
2. Quadratic attention on a 16x sequence, paid on every indexed chunk, for the benefit of exactly one oversized section.
3. It leaves the granularity wrong. Citations, quality reports, and injection payloads all wanted to address individual facts and could not.

So the fix was to make the fact the unit of embedding: the indexer now
parses the memory file's per-fact structure and emits one chunk per fact.
Every fact is 58-335 tokens, comfortably inside any window.

And separately - this is the part that prevents recurrence - the indexer now
counts every chunk's tokens against the encoder's actual window and warns on
offenders, and my health command reports the count. The window guard, not
the window raise, is what stops this bug class from coming back. Four wiki
documents are over the limit today; the detector names them, and that debt is
visible instead of silent.

Measuring the fix instead of arguing about it

The design had one open risk I could not reason away: per-fact chunks might
lose on broad queries ("what side projects am I working on"), where a big
section chunk plausibly wins by covering more topics. I had accepted this as
a trade-off in the design review.

Instead of accepting it, I measured it. Two indexes over the identical
corpus: arm A chunked by section (exactly reproducing the live store), arm B
chunked by fact. Twenty queries against both - memory-directed ones, broad
ones, and eight wiki-directed queries as a counter-direction check, because
an improvement that comes from crowding out the rest of the corpus is not an
improvement.

The accepted trade-off inverted:

• Rank-1 distance was never worse: 12 of 12 memory-directed queries.
• Memory facts present in a default top-5 rose from 9 to 27 across queries.
• All six broad queries improved or tied. The flagship broad query went from one memory hit at d=0.4763 to three hits led by d=0.3537.
• Four narrow probes went from zero memory chunks in the top 5 to rank 1.
• The eight wiki queries kept rank 1 unchanged, all eight.

The section chunk's assumed advantage on broad queries never existed. It was
an artifact of there being only two memory chunks in the index at all - of
course one of them "won broad queries"; there was nothing else to return.
This is my strongest argument for measuring over arguing: the risk I had
formally accepted in a design document was not real.

End to end, the six-probe acceptance set went from 0/6 correct to 6/6, and
every one of the twelve facts now self-retrieves at exact d=0.0000 - which
doubles as a standing proof that no fact is truncated anymore.

The two ways my "before" measurement almost lied

An honest before/after table requires the "before" arm to actually measure
the old behavior. Mine silently stopped doing that twice, and both failures
produced the same poisonous output: a table showing no change at all.

Trap one: the old store was auto-upgraded by the new binary. My index
store wipes and rebuilds itself when its schema version is outdated. The fix
bumped the schema. So when I pointed the new binary at the preserved
old store to get "before" numbers, the first query quietly rebuilt the
entire store with per-fact chunking - converting my before-arm into a second
after-arm. The before-arm has to be driven by the old binary, not just the
old data.

Trap two: the old binary overwrote the new one. I built that old binary
from a git worktree, but with the build cache directory shared with the main
checkout - which overwrote the release binary I was using for the "after"
column. Both columns were now measuring pre-fix code.

I caught it because of two smells: the after column exactly equaled the
before column, and a known-good result I had measured minutes earlier had
silently regressed. The rule I keep now: pin each arm to its own binary path,
and assert the two binaries differ before trusting any comparison between
them. "Before equals after, everywhere, exactly" is not a null result. It is
an instrument failure.

A third, related trap lives in the test suite. My mock embedder hashes the
full input text with no truncation, so every truncation test passes against
it - including on the broken code. The causal tests require the real encoder
(a 558MB model download I keep out of CI), so CI instead pins the structural
properties the fix guarantees: one chunk per fact, identity fields populated.
The proof runs by hand at the gate; the regression guard runs on every
commit; and I wrote down which is which, so a green CI run cannot be mistaken
for the causal proof.

What transfers

None of this needed special tooling - a CLI, a scratch directory for each
index, and discipline about what counts as evidence. The parts I would carry
to any agent-memory system:

• Prove mechanism before choosing a fix. The cheap causal probe (full vs. prefix vs. tail) killed a wrong diagnosis that would have led to a plausible, useless repair.
• Make missing data observable. The worst property of my bug was silence. The token-count guard turns the next occurrence into a warning with a filename in it. And "not measured" must render as not measured - my health check deliberately refuses to show a green tick when the data to judge is absent, because a green that certifies nothing is worse than no check.
• Measure accepted risks; they may not exist. The broad-query trade-off survived a design review as a reasonable-sounding caveat. It did not survive twenty queries.
• Distrust symmetric results. Identical before/after numbers mean your instrument broke, until proven otherwise.
• Claim what you measured, nothing more. Everything above is one corpus of 129 chunks, one embedding model, twenty queries, on my machine. It generalizes as a method, not as numbers.

The whole investigation - diagnosis, fix, and both measurement arms - is
recorded as an architecture decision record and a work plan in the project's
docs, which is also why I can write this post months of context later without
reconstructing anything from memory. But that discipline is a story for
another post.