Part 1 of Retrieval-Augmented Self-Recall — the research track behind Claude Code, Beyond the Prompt. All code is open source: RE-call.
Almost ...
For further actions, you may consider blocking this person and/or reporting abuse
The gap_warning idea is the part I'm most curious to see land, because "the best match is too weak to trust" implies a threshold, and thresholds on cosine similarity are notoriously slippery across content. You flagged in the outline that the threshold didn't transfer between embedding models, which I'd bet is the real story here. Is the plan a single global cutoff you re-tune per embedder, or something relative, like the gap between the top hit and the rest of the batch rather than an absolute score? The second feels more likely to survive a model swap, but I'm guessing at your part 5 before you've written it.
You've guessed the real story: the non-transfer is Part 5, and it's the whole finding.
Direct answer, and it's the less clever one: a per-embedder calibrated absolute cutoff. I re-fit the threshold against a small labeled set for each model rather than shipping a constant. Not elegant, but measured.
Your relative idea, top hit versus the rest of the batch, was the first thing I reached for too, and it is more scale-invariant. But it has a hole that's worse for agent memory specifically: it's blind to the single confident distractor. When memory holds one memo that's semantically adjacent but wrong, that hit stands out from the pack, big margin, and a spread-based check waves it through as confident. That's the exact case that does the damage, one strong near-miss read as a hit.
And you don't escape calibration there, you move it: the margin has its own cutoff, and that distribution shifts across embedders too. Maybe less. I haven't measured the two head to head.
So absolute-per-embedder is what shipped, relative is the experiment I still owe. Good one to leave me with.
This is the RAG problem I spend most of my time on, and the line about a better retriever returning wrong results more confidently is exactly right. I would sharpen the abstention fix one degree: a similarity score is not a confidence score. The near-misses that hurt most are high-similarity and wrong, semantically adjacent memos that a threshold-based gap_warning waves straight through. So the abstention signal cannot be the retriever's own score. What has held up for me is a separate check that the retrieved memo actually entails an answer to the query, not just that it sits nearby in embedding space. Proximity is a candidate; entailment is the evidence. Score the second, abstain on the second.
On supersession, the hardest of your three, the reason timestamps fail is that you are trying to infer a relation between two memos at read time, when both look valid in isolation. That inference is a losing game. The version that works is to capture the relation at write time: when a decision supersedes an earlier one, the new memo names what it replaces, so retrieval returns the current head of the chain instead of a resolved-but-still-embedded old memo. Bind the truth when it is created, do not reconstruct it from ambiguous artifacts later. Same principle as calibrated abstention, just moved from read time to write time.
Both your fixes are ahead of what I shipped. Conceding precisely.
On abstention: you're right that the score can't be the signal, and it exposes something I glossed over. My cross-encoder rerank is the halfway house that looks like your fix but isn't. It scores query and memo jointly, closer to entailment than cosine, but it still emits a relevance score I then threshold, which is the same mistake one layer up. Entailment gives a decision, not a score, and that's the deeper win: it sidesteps the per-embedder calibration problem entirely, because there's no number left to re-tune. The cost is a second model call per candidate, which is the honest reason people reach for the threshold instead.
On supersession: bind at write time, agreed. It's the same move as capturing the relation when the truth exists rather than reconstructing it from artifacts later. My anti-re-litigation guard is already a crude version of this, a closed-decisions index, and it fails exactly where you'd predict: it is only as complete as my discipline in linking the chain. But that is a far better place to fail than read-time inference, because a missing link is lintable. Impossible to infer becomes possible to enforce.
Most useful comment I've had on this one. Thank you.
"Entailment gives a decision, not a score" is a sharper statement of the fix than mine. No number left to re-tune is the whole win, and the second model call per candidate is the honest price tag on it.
On the lintable missing link: the lint can be automated with machinery you already run, by executing retrieval at write time. When a new memo lands, query the index with it before committing; any high-similarity hit among closed decisions that the new memo does not reference is a candidate unlinked chain, surfaced for a one-keystroke confirm. Same embedder, same index, just pointed at the write path instead of the read path. Your discipline problem becomes a diff the system shows you, and the chain stays complete without depending on you remembering to link it.
The split between retrieving external documents and retrieving the agent's own prior reasoning is a distinction I wish more RAG write-ups made. The failure mode I keep hitting is an agent re-deriving a conclusion it already reached earlier in the same task because it can't recall its own intermediate state. How do you handle staleness when the recalled self-memory conflicts with newer context - does the freshest observation always win, or do you score them?
Freshest does not automatically win. Staleness is treated as a structural relation when I can, not a score.
For self-memory, the useful distinction is usually:
Observation: something the agent saw or the environment returned.
Conclusion: something the agent inferred from prior observations.
Decision/state: what the agent chose to do next.
If newer context contradicts an old conclusion, the old conclusion gets downgraded unless its supporting observations are still valid. But I don’t want “latest timestamp wins,” because recency is a bad proxy for truth. A newer note can be a partial observation, a mistaken inference, or only relevant to a different branch of the task.
The closest analogue to what I described above is: make invalidation explicit where possible. A recalled memory can be marked superseded_by, resolved_by, invalidated_by, or scoped to a task phase. Then retrieval doesn’t have to infer staleness from embedding score or timestamp. It can say: this memory is relevant but no longer authoritative.
When that structure is missing, I’d rather surface the conflict than silently rank it away: “I previously concluded X, but newer context says Y; using Y because it directly observes the current state.” So scoring can help retrieve the candidates, but it should not be the thing that decides authority.
This is a really important distinction. Traditional RAG assumes the answer exists and optimizes retrieval quality, while agent memory has to answer a different question first: should I trust any retrieved memory at all? We've run into similar challenges at IT Path Solutions building long-running AI agents, where false-positive retrieval often causes more damage than a missed retrieval because the agent confidently builds on incorrect assumptions. One thing I'd add is that retrieval should return confidence + provenance + validity, not just relevance. A memory that's semantically similar but superseded or outside its validity window should often lose to "I don't know." In production, calibrated abstention ends up being as important as retrieval accuracy.
Agreed, and the asymmetry has a mechanism worth naming: a miss is visible, a false positive is invisible. Return nothing and you go look. Return a confident near-miss and you stop looking. Same error rate, opposite consequences.
Your triad maps roughly onto what I built: gap_warning is confidence, freshness is validity, anti-re-litigation is provenance. But you have named the weakest of the three, and it is mine. Freshness measures age, and age is a cheap proxy for validity that is wrong in both directions: a memo from a year ago can still be binding, one from yesterday can already be superseded and still look pristine to any timestamp check.
The hard part: supersession is not a property of a memo, it is a relation between two. A memo cannot know it has been superseded, only the pair knows, so no per-document freshness field catches it however good the timestamps are.
Thanks for the prompt, genuinely useful. Returning validity and provenance alongside confidence, rather than leaning on a timestamp, is the right shape and I will be investigating it.
The distinction between ranking and abstention is the crux — and it's the one most off-the-shelf stacks quietly sidestep. I ran into the same failure mode on a financial-news knowledge graph: the retriever would surface the three nearest nodes when asked about a company it had never seen, and the downstream model would treat confident retrieval as confirmation the entity existed. What helped was adding a coverage_check field to the Pydantic output schema so the model had a typed slot to say "retrieved but no match above threshold" rather than silently returning a near-miss. The gap_warning pattern you describe closes the same gap at the retriever layer, which is cleaner.
"Treated confident retrieval as confirmation the entity existed" is a sharp statement. That is the whole thing: retrieval returning something is not evidence the thing exists, but downstream it reads exactly like it.
On coverage_check versus gap_warning, I would say complementary rather than cleaner, because each covers the other's weakness. A typed slot in the output schema is the right instinct: it makes the signal impossible to quietly drop, which an annotation on a retrieval result very much can be. But if the model fills that slot, you are asking the thing that just hallucinated to report its own hallucination. It is the least reliable narrator about its own grounding.
So the split I would argue for: the retriever computes it, the schema carries it. Computed from the cosine distribution it is a measurement. Self-reported it is a declaration. Those two fail very differently.
One warning if you go threshold-based, since it cost me: the threshold does not transfer across embedders. Same constant, wildly different behaviour. Calibrate per model against a small labeled set.
Really sharp reframe — self-recall vs document-QA. Once the KB is the agent’s own memory, nearest-neighbor-as-yes is the failure mode. Better embeddings making you wrong faster tracks.
The three modes (gaps, re-litigating settled decisions, stale-as-current) aren’t ranking bugs. Calibrated abstention + measuring false-confident rate next to MRR is the objective most stacks never ask for.
We’ve been in the same neighborhood on agent memory, but from the compliance / accumulation side: append-only chat logs explode; what you want to keep is the gist graph, not every turn. On our Azure runs (real Gemini), PrismCortex stored the gist ~5.2× smaller than the raw log (18.7 KB → 3.6 KB in that fixture), and after 675 chatter turns the graph stayed flat — 0 new edges — while the append-log kept everything. That’s the “memory that doesn’t grow with small talk” piece. Separate from RE-call’s gap_warning / freshness / anti-re-litigation guards — different knobs — but same instinct: don’t treat the full transcript as the retrieval corpus forever.
Also care about byte-identical replay + bitemporal history when a fact corrects ($40k → $55k style updates) so “what did we believe then” isn’t lost.
Curious how you persist closed-decision markers so anti-re-litigation survives daily re-index.
RE-call appreciated. PrismCortex demo if useful: prismcortex-demo.insightits.com
Results: github.com/insightitsGit/PrismCort...
Great insight. The shift from optimizing retrieval to optimizing honest abstention is something most RAG discussions miss. "I don't know" is often the smartest answer an AI can give.
it's interesting