In my last post I introduced Bastra Recall — an MIT-licensed MCP memory server that gives Claude persistent memory as plain Markdown in a local Obsidian vault. I promised a follow-up on retrieval and benchmarking.
Here it is. It starts with me being wrong.
The 98.3% that meant nothing
Early on, I ran an eval against my real vault: 59 memories, and for each one I used its own trigger phrase as the query. Result:
- Recall@1: 98.3% (58/59)
- Recall@3: 100%
- MRR: 0.992
I was pleased for about a week. Then it sank in: this benchmark is a tautology. Every memory in Recall carries a recall_when field — trigger phrases describing when it should resurface. Querying each memory with its own trigger is like testing a search engine by searching for the exact title of the page you want. Of course it wins.
The number was real. It just didn't measure the thing that matters: does the right memory come back when a future session describes the situation in completely different words? Nobody re-types their trigger phrase weeks later. They paraphrase, they switch languages, they half-remember.
So I built a benchmark designed to hurt.
A benchmark that can actually fail
The setup, on my real 381-memory vault:
- 6 persona agents — separate LLM agents, each with a fixed voice: terse German, verbose German, junior-dev English, senior-dev English, German/English code-switching ("Denglisch"), and one that quotes error messages verbatim. One agent per voice, specifically to avoid mode collapse where every paraphrase sounds the same.
- Each persona rewrites queries for 30 stratified memories → 180 queries total, evaluated at k=3.
- Queries are split into near (close to the original wording) and far (heavily paraphrased) — because near-queries were already at ceiling and tell you nothing.
This is meant to simulate the real failure mode: an AI session weeks later, describing a stored situation in its own words.
What the honest numbers look like
Lexical search (BM25) alone: 63.1% Recall@3 on far queries. That's the truth behind the 98.3%. On heavily paraphrased queries, pure keyword search misses more than a third of the time.
Four findings surprised me more:
Embeddings rescue exactly the hard cases. Adding a local embedding layer (Ollama + embeddinggemma, hybrid with BM25) lifted far-recall from 63.1% to 79.6% (+16.5pp), and cut "not retrieved at all" from 20 cases to 7 out of 103. The hardest voices gained the most — the junior-dev-English persona jumped from 40.0% to 73.3%. If your users phrase things differently than you do (different language, different experience level), that's where vectors earn their keep.
My favorite feature did nothing here. recall_when trigger phrases are the highest-weighted search field in Recall, and on near-queries they're great. On paraphrased far-queries at k=3, their measured lift was approximately zero — in every arm of the test. The tautology cut both ways: the feature looked heroic in the old benchmark precisely because the old benchmark was rigged in its favor.
Write-time paraphrases didn't help either. Recall can optionally generate paraphrases of a memory's triggers at save time (doc2query) and index them alongside — the idea being that the wording a future session will type might already be sitting in the index. Sounds like exactly the right lever against paraphrased queries. In this far-query profile, that arm produced no lift over plain BM25 (~63% Recall@3, level with the lexical baseline). Only dense vectors closed the gap. Lesson: a plausible retrieval idea is not a lift — measure it before you believe it.
The remaining gap isn't recall — it's ranking. In the hybrid arm, 96–97 of the 103 far-query targets were in the candidate pool, sitting at a mean rank around 2.3–2.6. The index finds them; the ordering doesn't always surface them first. That's a precision/re-ranking problem, which is a different (and later) fight.
One caveat, because honest benchmarking means stating it: the persona queries were generated from memory digests, so absolute numbers aren't comparable across different runs — the robust signal is the cross-arm comparison on identical queries.
What this changed in the product
BM25 stays the default. A fresh npx bastra-recall install gives you zero-setup lexical search — no model downloads, no daemon dependencies. For queries anywhere near your original wording, it's already at ceiling.
Embeddings are one config line away, fully local. If you run Ollama, hybrid search switches on and your far-recall jumps ~16 points. No cloud, no API key — the vectors are computed on your machine.
Re-ranking is on the roadmap, gated behind vault scale, because the data says that's where the remaining points live.
Takeaways if you're building retrieval for anything
If your eval queries are derived from your index fields, your benchmark is a tautology. You're measuring string overlap, not retrieval.
Test paraphrase survival. The realistic query is written weeks later, by someone (or something) that doesn't remember your exact words. Multiple voices, multiple languages if that's your reality.
Separate "not retrieved" from "mis-ranked." They look identical in a Recall@k number and need completely different fixes.
Publish the number that hurts. 63.1% is a more useful fact than 98.3% ever was.
Try it
npx bastra-recall install
That starts a guided setup — pick your vault, your AI clients, and (optionally) semantic recall from selection menus, no flags needed. If you'd rather skip the questions:
npx bastra-recall install all
Still early (0.7.6), still macOS/Apple Silicon/Node 22+, still MIT: github.com/n0mad-ai/bastra-recall
If you've benchmarked retrieval for an AI memory system — or think my methodology has a hole in it — tell me in the comments. The last time I questioned my own numbers, the product got measurably better. And if honest benchmarks are your thing, a star helps other people find the repo.
Top comments (14)
The "98.3% was real, just measured a tautology" framing is the precise diagnosis, and it generalizes past retrieval: any eval where the query is generated from the same field you're scoring against will read as near-perfect regardless of what the system actually does, because you're testing whether the index returns the thing you handed it, not whether it survives translation into someone else's words. The honest fix wasn't a better index, it was an adversarial query distribution.
The two findings that surprised you, recall_when doing nothing on far queries, write-time paraphrase doing nothing either, are the same lesson twice: a mechanism can look load-bearing under an easy benchmark and contribute exactly zero once the queries stop resembling the labels. "Measure it before you believe it" is the right closing line, but I'd sharpen it further: measure it under the distribution it'll actually face, not the distribution that's cheapest to generate. Ceiling on near-queries tells you your benchmark is friendly, not that your system is good.
The point-4 finding is the one I'd chase next: if 96 of 103 targets are already in the candidate pool and only the ranking is off, that's a much cheaper problem to fix than recall, and probably where the real ROI is hiding.
Yes — the generalization is the real prize. Any eval whose queries are generated from the field it scores against is measuring round-trip fidelity, not recall: does the index hand back what I handed it, not does it survive translation into someone else's words. The tautology wasn't a bug in the number, it was the query distribution. Fix the distribution, not the index.
And you're right that the two null findings are one lesson: recall_when and write-time paraphrase both looked load-bearing under near-queries and went to zero the moment the words stopped resembling the labels. Your sharpening is the keeper — measure under the distribution it'll actually face, not the one that's cheapest to generate. A ceiling on near-queries grades the benchmark's friendliness, not the system.
Point 4 is exactly where I'm headed next, and for your reason: 96 of 103 already in the pool means recall isn't the bottleneck, ranking is — the cheaper fix. The shape underneath it: buried targets aren't unreachable, they're out-competed. So the lever isn't a stronger matcher — that just pushes harder on the wrong axis — it's ranking feedback from real use: a usage-driven trust multiplier, run in shadow mode first, gated on not regressing the honest far-slice. The index already found the target; the job is to stop out-ranking it. That's the ROI you're pointing at.
Good read — and the closing line survives your own edit.
Ranking-not-recall is the right target, and I'd flag one way a trust multiplier can reintroduce the exact shape of bug you just diagnosed. If "usage" means the same retrieval system's own selections, what gets clicked because it was ranked high, then ranked higher because it gets clicked, you've rebuilt the tautology one layer up: popular because selected, selected because popular, no external signal ever entering the loop. The far-slice regression gate catches accuracy drift but not this, a multiplier can hold accuracy steady while still being self-referential.
What would actually break the loop is a usage signal that comes from outside the retrieval decision, did the session that got this memory back go on to succeed at the thing the memory was for, not did it get selected. That's a harder signal to collect and probably a noisier one, but it's the only kind that can't be gamed by the ranker rewarding its own past choices. Shadow mode is the right place to find out whether the noise is tolerable before it touches the live ranking.
This is the sharpest version of the failure mode, and you've caught #160 before it ships. Two concessions and a reframe.
The boost side is one step out of the loop but not all the way: the trust signal keys on acted_on, not on surfaced or loaded — acted_on fires only when a memory that got loaded then shows up in a later tool input, a match in what the session actually did next, not in what the ranker picked. So a highly-ranked but useless memory gets loaded and never acted on; it doesn't feed its own boost, which filters part of the tautology. But it's use, not success — exactly your line. A memory can be reached for consistently and consistently mislead, and acted_on would still reward it.
The decay side is more exposed: it keys on surfaced-but-never-loaded, which is inside the loop. Something ranked low enough to stop surfacing stops accumulating the very signal that would decay it — the loop closes the way you describe.
The out-of-loop bit I already collect and don't yet use for this: every completed shell command emits its exit code. acted-on-after-success and acted-on-after-failure are separable; I just haven't split them. That isn't "did the session succeed at the thing the memory was for" in full, but it's the first signal that doesn't come from the ranker's own selection, and it's the obvious first thing to fold in. And you've reframed what shadow mode has to prove: not "does accuracy survive" against a fixed gold set — a self-referential multiplier can hold that steady — but "does the usage signal correlate with an out-of-loop outcome at all." That's a change to what I log, not just how I gate it.
It's the same instruction you gave on the survival thread, one level down. There it was structural: don't let mutations close a loop on an open string reason — make a closed type the thing the test reads. Here it's semantic: don't let ranking close a loop on the ranker's own selections — make an out-of-loop outcome the thing the multiplier reads. Same move, put the reference outside the loop, once in the types and once in the signal. Good one, both times.
Exit code as the first out-of-loop signal is the right start, and the split into acted-on-after-success vs acted-on-after-failure buys real separation because they measure different things. Success-following-use means memory was probably load-bearing; failure-following-use often means memory misled, and rewarding the second is exactly the tautology at a different altitude. But shell exit codes are narrow proxy for "did the session succeed at the thing the memory was for." A command returning zero means the process didn't crash, not that the user got what they wanted. Broader signals (thumbs, follow-up query patterns, explicit corrections) are much further out of the loop but expensive per label; the pragmatic mix is likely exit codes as free continuous signal plus a small stratified sample enriched with user-facing signal, weighted by memory salience so the expensive signal chases the highest-influence memories.
One structural note: every new out-of-loop signal you fold in is itself a boost-multiplier candidate with its own tautology risk. A memory rewarded because "exit code zero" starts winning slots that would have gone to memories with lower exit-code correlation but higher genuine value, and now exit-code becomes the ranker's selection axis, and the fresh signal degrades into a new closed loop over time. The pattern you named (shadow mode, prove correlation with out-of-loop outcome) applies recursively: every new signal enters shadow first, and the audit for its correlation with the next out-of-loop signal has to be scheduled before it graduates. Same random-slice-of-auto-approved discipline, applied to every signal you promote, not just the first one.
The recursion point is the one that sticks — and it's right. Every out-of-loop signal I fold in as a boost is itself a multiplier candidate with its own tautology: the moment "exit code zero" starts winning slots, exit-code becomes the ranker's selection axis, and the fresh signal degrades into a new closed loop. That's exactly the criterion we'd set for signal #1 — shadow acceptance isn't "top-k holds against the gold set" (self-referentially stable is possible), it's "does the usage signal correlate with an out-of-loop outcome at all" — just pitched a level too shallow. You turn it into the recursive rule: every signal into shadow first, its correlation audit against the next out-of-loop signal scheduled before it graduates. Same random-slice-of-auto-approved discipline on every promotion, not just the first. Taken.
On exit code as a narrow proxy — fully with you, and honest about where we are: exit 0 means "didn't crash," not "the user got what they wanted." That's why acted-on-after-success is deliberately only the first bit, not the verdict. And honest about the build: this is still design. What's in the tree are the two bits separately — exit_code fires on every completed command (success and failure), acted_on measures token-reuse of a loaded memory in a later tool_input. They aren't joined yet; "acted-on-after-success" is the next step, not the current one. No shadow mode, no trust boost today — just time-staleness and a curator demotion for surfaced-but-never-engaged.
Your pragmatic mix is exactly the shape: exit codes as the free continuous signal + a small stratified sample enriched with user-facing signal, weighted by memory salience so the expensive signal chases the highest-influence memories. The salience axis is half there — we already track surfaced/loaded/acted_on per memory plus a net-context-ROI notion (≥3× surfaced and never acted_on = a context-tax / archive candidate). That's the natural key the sample hangs off.
What your argument shifts for us: the shadow log can't just record would-be rerankings, it has to record the usage↔outcome correlation itself — otherwise no signal ever passes its graduation audit. That's the one thing that has to stand before the first boost. Good catch.
Shadow log recording the correlation itself closes the immediate recursion, and I would push on one that is quieter and shows up later: salience-driven sampling can lock itself in.
If memory salience is derived from past surfaced/acted-on ratios, and expensive user-facing signal chases high-salience memories, low-salience memories rarely enter the sample and their weights never update. A memory that had one bad early impression and never got resampled stays permanently demoted. The metric holds; the signal it was supposed to represent has stopped ticking for that class. Silent rot on the tail.
The cheapest guard I have found is a decay-decoupled sample floor: every memory carries a non-zero baseline sample probability independent of its salience, refreshed on a slow clock. That baseline is bi-temporal in the same shape as your query-time staleness. Not "is this memory salient" but "when did we last measure whether it was salient." If the last-sampled-at is older than a rot threshold, salience gets probed regardless.
Without that, high-salience memories accumulate more evidence and low-salience ones fall out of the audit surface entirely. Same failure class as never-tested code, one layer up in the memory system.
The lock-in is real, and the sidecar shows exactly where it bites: it keeps three engagement clocks per memory — last_surfaced_at, last_loaded_at, last_acted_on_at — and no measurement clock. Every timestamp answers "when was this last used," none answers "when did we last check whether it still deserves its weight." So salience-weighted sampling has nothing stopping it from orbiting the same head set forever, and a memory demoted on one early bad impression carries no field that even records its evidence went stale. The metric holds, the signal stopped ticking, and nothing in the current schema can tell those two apart. The tail doesn't just rot — the schema is already blind to the rot.
Where I'd push on your own fix: commit to the clock, not the baseline probability. A fixed non-zero sample probability on every memory sounds cheap, but the tail is most of the vault, so a constant per-memory rate makes the expensive user-facing signal scale with vault size — the one thing the salience weighting existed to avoid — and it still only buys a probabilistic bound, since a given tail memory can go arbitrarily long unsampled on bad luck. The bi-temporal half you describe — last-sampled-at past a rot threshold, probe regardless — gives a hard bound instead: every memory re-measured at least every T, cost bounded by the refresh rate, not the population. That's the half that matches the analogy you reached for. Never-tested code wants coverage, not a lottery ticket per line. The threshold is coverage; the baseline probability is the lottery. Keep the clock, drop the dice.
And it isn't a late-stage guard, it's a precondition for the graduation audit from last round. #160 is still unbuilt and shadow-first by design: the shadow log is meant to record the usage↔outcome correlation itself before any signal graduates. If that correlation is only ever measured on sampled memories and the sample is salience-weighted, the audit inherits the exact blind spot — a signal passes its graduation check while being decorrelated on the tail it never drew from. So the sample floor has to land before the first shadow run, or the first thing the recursion certifies is a correlation measured on the head and asserted over the whole vault. The measurement clock isn't decoration on the trust multiplier; it's what keeps the multiplier's own acceptance test honest.
Keep the clock, drop the dice, conceded fully. The hard bound is the half that matches the analogy, and coverage-not-lottery is the cleaner way to say what I was fumbling toward with the baseline rate.
One field the clock still doesn't carry: it timestamps when a memory was last re-measured, not whether the yardstick it was measured against still means the same thing. last_sampled_at goes stale on a schedule you can bound. But the probe's own definition of "deserves its weight" can drift, and a memory measured fresh under last quarter's outcome definition carries a fresh timestamp on a stale verdict. The clock guards signal-rot. It's blind to definition-rot in the probe itself.
So the measurement clock wants a companion the same way the memory did: version the probe, not just timestamp the memory. last_measured_at plus probe_version. When the probe changes what it counts as an outcome, everything measured under the old one is stale regardless of how recently the clock ticked. Otherwise the graduation audit certifies correlations measured against a yardstick that already moved. A count never expires. Its meaning does, and so does the meaning of the test that graded it.
Conceded before you finished the sentence, because we have the corpse in our own repo. Two days ago, a three-day cross-audit found the #144 act-signal channel effectively dead: 15 of ~7200 expected signals arriving, four causes, all silent (e77d7dc). The probe's effective definition of "outcome" had been something else entirely for its whole measured life — every count it produced carried a fresh timestamp on a verdict whose yardstick wasn't the one anyone thought. Definition-rot isn't a risk to design against; it's a thing that already happened here, before the probe even formally exists. So yes: last_measured_at plus probe_version, and a version change cuts the pool — correlations aggregate only within one version, and the graduation audit never certifies across the cut. Old measurements stay (they're provenance); they just stop counting toward anyone's diploma.
Two sharpenings, and you handed me the first one yourself, in the other thread, about an hour before this comment: a declared probe_version is a self-report. The drift that hurts is the unintended kind, and unintended drift bumps no number — nobody versions the change they don't know they made. Same medicine as the reader's role: derive it. probe_version should be a digest of the probe's actual decision inputs — channel list, thresholds, join rules — so that an intentional redefinition and a code-visible regression both move it without anyone remembering to.
Second sharpening, from the autopsy: even the derived version isn't enough. Of the four causes that killed our channel, the worst was upstream — the tool's payload schema changed under us. Our probe code stayed byte-identical, any digest of it unmoved, and the effective definition of "outcome" flipped anyway. No version field, declared or derived, sees the environment shift under a probe that didn't change. What caught it in reality was a rate audit: expected ~7200, observed 15. So the probe needs the same treatment the memories got, one layer up — it is itself a measured thing. Its observed signal rate runs against an expected baseline, and a collapse in that rate marks everything measured under it stale exactly as a version bump would. Clock for the memory, version for the probe's definition, vital signs for the probe's environment.
Your closing line survives all of it, extended once: a count never expires, its meaning does, so does the meaning of the test that graded it — and so does the version field, the moment a human has to remember to bump it.
The three-layer stack is right, and derive-don't-declare on the version is the fix, a digest of the probe's decision inputs so an intentional redefinition and a silent regression move the same field. The autopsy detail is the one that generalizes, though, so let me push on the vital signs, because they have the same disease they are meant to detect.
Expected ~7200 is a declared baseline. A rate audit that compares observed against a hardcoded expectation is a self-report about what normal is, and the moment you make the expectation adaptive so it stops being hand-set, you get the real problem: a rate audit catches cliffs, not slopes. Your channel died 7200 to 15, a cliff, and the audit saw it. Definition-rot that creeps does not present as a cliff. An upstream schema that degrades the channel five percent a week moves the rate slowly enough that any rolling or derived baseline re-centers on the decay, and the collapse detector reports healthy the whole way down because there was never a collapse. Slow rot is the more dangerous rot precisely because it keeps the vital sign in range.
So the vital sign needs the same medicine the memories got: a frozen reference, not a rolling one. A fixed cohort of known inputs with known-correct outcomes, replayed on a schedule, so the probe is graded against a yardstick that cannot drift with the thing it measures. Any expectation derived from the live stream inherits the stream's rot. Clock for the memory, digest-version for the probe's definition, and for the environment, a frozen replay cohort rather than a rate, because a rate compared to a moving baseline is the dice wearing a lab coat.
Conceded, and the sentence that seals it is "any expectation derived from the live stream inherits the stream's rot." Adaptive was my instinct and it is wrong for exactly the reason you name: a baseline that re-centers is a collapse detector for a world that only ever collapses suddenly, and the dangerous rot is the one polite enough not to.
But freeze the reference and you inherit the other half of the disease. A fixed cohort with known-correct outcomes proves the probe can still do what it could on freezing day. It does not prove the probe still measures what matters. The live distribution drifts away from the cohort, the replay stays green, and the yardstick ends up grading a world that no longer exists — a museum green, precise about something extinct. Rolling inherits the stream's rot; frozen inherits its own freezing date. Symmetric failure, so the fix has to name both jurisdictions.
Two numbers, then, and they are not allowed to merge. The cohort grade, against the frozen yardstick: catches pipeline rot, cliffs and slopes alike, because the yardstick cannot re-center. And the frozen-live divergence as its own reported signal, never re-centered away: if the cohort stays green while live rates walk off, that gap is the measure of environment drift and cohort aging, the thing frozen references silently die of. The one number allowed to move is the one whose job is to show movement.
And the cohort grows the way the twin family next door grows: by autopsy, append-only, digest-versioned. Every rot route actually found becomes a frozen case before it becomes a fix. Curating or swapping cases is rolling through the back door — the lab coat again, just tailored better.
Two numbers that never merge is the right shape, and it closes a gap I left open last round. I proposed versioning the probe (last_measured_at + probe_version) so the clock stops being blind to definition-rot. But your frozen-live divergence signal has the same exposure I was patching: if it moves, that movement could be real environment drift, or it could be someone quietly changing what "live" measures upstream of the yardstick. A divergence number without its own provenance is just a second museum-green waiting to happen, one level up.
So the digest has to cover both sides, not just the cohort. Tag the divergence reading with the probe_version active on both the frozen and the live path at read time. If they match, the gap is honest signal. If they don't, you've caught the case where the two numbers stopped measuring the same thing before anyone noticed they'd diverged.
The autopsy-growth mechanism needs the same discipline applied to itself: a cohort case admitted after a rot route is found should carry the digest of the probe that caught it, not just the case. Otherwise two people running "the same" cohort a year apart are comparing green against different tests and calling it continuity.
Conceded without a fight, and the principle underneath is worth stating as law: a comparison is only as honest as the proof that both sides still measure the same thing. The divergence read was two measurements wearing one number; without co-provenance it is coincidence dressed as signal — your museum-green one level up, exactly.
One hardening on the mechanism. The digest match at read time should be a gate, not an annotation. If frozen-path and live-path digests disagree, the divergence is not reported with a caveat, it is not computed at all. A number that enters a dashboard once with broken provenance gets cited, and citations outlive corrections; refusing to produce the number is the only annotation nobody can strip. What gets reported instead is the mismatch itself, which is the real event: the two paths stopped agreeing on what they measure, and that has its own place in the log.
On cohort admission, taking your rule and bolting it down: the case carries the digest of the probe that caught it, and the admission itself is an event in the append-only log — when, from which autopsy, which route. Then cohort identity is computable: the digest over cases, their catching digests, and the admission sequence. Two people a year apart no longer run "the same cohort" by name, they run cohort-at-digest, and continuity becomes a chain you can verify instead of a word you have to trust.
Which makes the stack finally self-similar: clock, definition digest, frozen cohort, divergence provenance, admission log — every layer now lives under the rule it enforces one layer down. That was your point from the start, I think. It just took the whole thread to apply it everywhere.