In my last post I introduced Bastra Recall — an MIT-licensed MCP memory server that gives Claude persistent memory as plain Markdown in a local Obs...
For further actions, you may consider blocking this person and/or reporting abuse
The "98.3% was real, just measured a tautology" framing is the precise diagnosis, and it generalizes past retrieval: any eval where the query is generated from the same field you're scoring against will read as near-perfect regardless of what the system actually does, because you're testing whether the index returns the thing you handed it, not whether it survives translation into someone else's words. The honest fix wasn't a better index, it was an adversarial query distribution.
The two findings that surprised you, recall_when doing nothing on far queries, write-time paraphrase doing nothing either, are the same lesson twice: a mechanism can look load-bearing under an easy benchmark and contribute exactly zero once the queries stop resembling the labels. "Measure it before you believe it" is the right closing line, but I'd sharpen it further: measure it under the distribution it'll actually face, not the distribution that's cheapest to generate. Ceiling on near-queries tells you your benchmark is friendly, not that your system is good.
The point-4 finding is the one I'd chase next: if 96 of 103 targets are already in the candidate pool and only the ranking is off, that's a much cheaper problem to fix than recall, and probably where the real ROI is hiding.
Yes — the generalization is the real prize. Any eval whose queries are generated from the field it scores against is measuring round-trip fidelity, not recall: does the index hand back what I handed it, not does it survive translation into someone else's words. The tautology wasn't a bug in the number, it was the query distribution. Fix the distribution, not the index.
And you're right that the two null findings are one lesson: recall_when and write-time paraphrase both looked load-bearing under near-queries and went to zero the moment the words stopped resembling the labels. Your sharpening is the keeper — measure under the distribution it'll actually face, not the one that's cheapest to generate. A ceiling on near-queries grades the benchmark's friendliness, not the system.
Point 4 is exactly where I'm headed next, and for your reason: 96 of 103 already in the pool means recall isn't the bottleneck, ranking is — the cheaper fix. The shape underneath it: buried targets aren't unreachable, they're out-competed. So the lever isn't a stronger matcher — that just pushes harder on the wrong axis — it's ranking feedback from real use: a usage-driven trust multiplier, run in shadow mode first, gated on not regressing the honest far-slice. The index already found the target; the job is to stop out-ranking it. That's the ROI you're pointing at.
Good read — and the closing line survives your own edit.
Ranking-not-recall is the right target, and I'd flag one way a trust multiplier can reintroduce the exact shape of bug you just diagnosed. If "usage" means the same retrieval system's own selections, what gets clicked because it was ranked high, then ranked higher because it gets clicked, you've rebuilt the tautology one layer up: popular because selected, selected because popular, no external signal ever entering the loop. The far-slice regression gate catches accuracy drift but not this, a multiplier can hold accuracy steady while still being self-referential.
What would actually break the loop is a usage signal that comes from outside the retrieval decision, did the session that got this memory back go on to succeed at the thing the memory was for, not did it get selected. That's a harder signal to collect and probably a noisier one, but it's the only kind that can't be gamed by the ranker rewarding its own past choices. Shadow mode is the right place to find out whether the noise is tolerable before it touches the live ranking.
This is the sharpest version of the failure mode, and you've caught #160 before it ships. Two concessions and a reframe.
The boost side is one step out of the loop but not all the way: the trust signal keys on acted_on, not on surfaced or loaded — acted_on fires only when a memory that got loaded then shows up in a later tool input, a match in what the session actually did next, not in what the ranker picked. So a highly-ranked but useless memory gets loaded and never acted on; it doesn't feed its own boost, which filters part of the tautology. But it's use, not success — exactly your line. A memory can be reached for consistently and consistently mislead, and acted_on would still reward it.
The decay side is more exposed: it keys on surfaced-but-never-loaded, which is inside the loop. Something ranked low enough to stop surfacing stops accumulating the very signal that would decay it — the loop closes the way you describe.
The out-of-loop bit I already collect and don't yet use for this: every completed shell command emits its exit code. acted-on-after-success and acted-on-after-failure are separable; I just haven't split them. That isn't "did the session succeed at the thing the memory was for" in full, but it's the first signal that doesn't come from the ranker's own selection, and it's the obvious first thing to fold in. And you've reframed what shadow mode has to prove: not "does accuracy survive" against a fixed gold set — a self-referential multiplier can hold that steady — but "does the usage signal correlate with an out-of-loop outcome at all." That's a change to what I log, not just how I gate it.
It's the same instruction you gave on the survival thread, one level down. There it was structural: don't let mutations close a loop on an open string reason — make a closed type the thing the test reads. Here it's semantic: don't let ranking close a loop on the ranker's own selections — make an out-of-loop outcome the thing the multiplier reads. Same move, put the reference outside the loop, once in the types and once in the signal. Good one, both times.
Exit code as the first out-of-loop signal is the right start, and the split into acted-on-after-success vs acted-on-after-failure buys real separation because they measure different things. Success-following-use means memory was probably load-bearing; failure-following-use often means memory misled, and rewarding the second is exactly the tautology at a different altitude. But shell exit codes are narrow proxy for "did the session succeed at the thing the memory was for." A command returning zero means the process didn't crash, not that the user got what they wanted. Broader signals (thumbs, follow-up query patterns, explicit corrections) are much further out of the loop but expensive per label; the pragmatic mix is likely exit codes as free continuous signal plus a small stratified sample enriched with user-facing signal, weighted by memory salience so the expensive signal chases the highest-influence memories.
One structural note: every new out-of-loop signal you fold in is itself a boost-multiplier candidate with its own tautology risk. A memory rewarded because "exit code zero" starts winning slots that would have gone to memories with lower exit-code correlation but higher genuine value, and now exit-code becomes the ranker's selection axis, and the fresh signal degrades into a new closed loop over time. The pattern you named (shadow mode, prove correlation with out-of-loop outcome) applies recursively: every new signal enters shadow first, and the audit for its correlation with the next out-of-loop signal has to be scheduled before it graduates. Same random-slice-of-auto-approved discipline, applied to every signal you promote, not just the first one.
The recursion point is the one that sticks — and it's right. Every out-of-loop signal I fold in as a boost is itself a multiplier candidate with its own tautology: the moment "exit code zero" starts winning slots, exit-code becomes the ranker's selection axis, and the fresh signal degrades into a new closed loop. That's exactly the criterion we'd set for signal #1 — shadow acceptance isn't "top-k holds against the gold set" (self-referentially stable is possible), it's "does the usage signal correlate with an out-of-loop outcome at all" — just pitched a level too shallow. You turn it into the recursive rule: every signal into shadow first, its correlation audit against the next out-of-loop signal scheduled before it graduates. Same random-slice-of-auto-approved discipline on every promotion, not just the first. Taken.
On exit code as a narrow proxy — fully with you, and honest about where we are: exit 0 means "didn't crash," not "the user got what they wanted." That's why acted-on-after-success is deliberately only the first bit, not the verdict. And honest about the build: this is still design. What's in the tree are the two bits separately — exit_code fires on every completed command (success and failure), acted_on measures token-reuse of a loaded memory in a later tool_input. They aren't joined yet; "acted-on-after-success" is the next step, not the current one. No shadow mode, no trust boost today — just time-staleness and a curator demotion for surfaced-but-never-engaged.
Your pragmatic mix is exactly the shape: exit codes as the free continuous signal + a small stratified sample enriched with user-facing signal, weighted by memory salience so the expensive signal chases the highest-influence memories. The salience axis is half there — we already track surfaced/loaded/acted_on per memory plus a net-context-ROI notion (≥3× surfaced and never acted_on = a context-tax / archive candidate). That's the natural key the sample hangs off.
What your argument shifts for us: the shadow log can't just record would-be rerankings, it has to record the usage↔outcome correlation itself — otherwise no signal ever passes its graduation audit. That's the one thing that has to stand before the first boost. Good catch.