Part 1 of Retrieval-Augmented Self-Recall — the research track behind Claude Code, Beyond the Prompt. All code is open source: RE-call.
Almost every RAG tutorial you've read solves the same problem: you have a pile of documents, a user asks a question, and you retrieve the chunks that answer it. Rank the right passage to the top, stuff it in the prompt, done.
There's a second kind of RAG that behaves completely differently, and almost nobody writes about it: an agent retrieving from its own memory.
I ran into it building the operational brain behind an automated trading system. That agent accumulates memory as it works — over 700 typed markdown memos, about 5 MB, re-indexed daily. Decisions, dead ends, calibration notes, "we tried this and it failed" postmortems. When the agent starts a task, it queries that memory: have we tested this before? what did we decide about X? is this still true?
The moment your knowledge base is the agent's own growing memory, the RAG problem inverts — and standard retrieval quietly does the wrong thing.
Why self-recall is not document QA
In document QA, there's a load-bearing assumption you probably never think about: the answer is in the corpus. Someone asked a question because the docs can answer it. Your whole job is ranking — surface the right chunk.
In self-recall, the most important queries are exactly the ones where the answer isn't there.
The agent asks "have we tried a mean-reversion filter on this market?" If the honest answer is no, never, a ranking-optimized retriever will still cheerfully return the three most cosine-similar memos — probably something about a different filter on a different market — and the agent, seeing confident results, concludes "yes, we looked at this." It just made a decision on a hallucination.
The failure isn't bad ranking. The top results might be the genuinely closest memos. The failure is that the system had no way to say "there's nothing relevant here."
Three failure modes unique to agent memory
Once you look at memory this way, three distinct failure modes show up that document-QA RAG never has to handle:
Hallucinating over gaps. The query has no real answer in memory, but retrieval returns the nearest neighbors anyway, and their presence reads as a "yes." The system needs to abstain — to flag "this is probably a gap" instead of pretending.
Re-litigating settled decisions. The agent proposes an idea it already tried and killed three months ago, because the "we decided against this, here's why" memo didn't surface at the moment of proposing. Memory that can't defend its own past decisions is doomed to relive them.
Acting on stale memory. A memo that was true in April is retrieved and treated as current in July. Without a freshness signal, old truth and current truth are indistinguishable — and in a system that touches money, that's expensive.
None of these are ranking problems. You cannot fix them by getting a better embedding model or a fancier reranker, because a better retriever just returns more confidently wrong results faster.
The reframe: abstention, not ranking
Here's the thesis of this whole series:
Document-QA RAG optimizes ranking. Agent-memory RAG has to optimize calibrated abstention — knowing when it doesn't know.
This matters because the metrics you've been trained to care about — MRR, nDCG, precision@k — don't measure abstention at all. They score how well you ordered the results assuming an answer exists. They are silent on the case that matters most in self-recall: the query with no answer, where the correct behavior is to return nothing and say so.
That's why reaching for an off-the-shelf RAG stack and pointing it at your agent's memory feels fine in a demo and rots in production. The stack is tuned for the wrong objective. It was never asked to abstain, so it doesn't.
RE-call: a reference implementation
To work through this properly I built and open-sourced RE-call — a retrieval engine designed around abstention from the start rather than bolted on after. The one-paragraph version, which the rest of this series unpacks:
- Storage & retrieval: PostgreSQL + pgvector as a single transactional store — dense vector search and sparse full-text search in the same database, fused with Reciprocal Rank Fusion, with an optional cross-encoder reranker.
-
Three honesty guards: a
gap_warningthat fires when the best match is too weak to trust, a freshness signal that flags stale memory, and an anti-re-litigation check that surfaces closed decisions before the agent re-proposes them. - Honest evaluation: a harness that measures not just ranking quality but a false-confident rate — how often the system fails to abstain when it should.
It ships as an MCP server, so an agent (Claude or otherwise) can query its own memory directly. That's the loop that closes back to my other series — this is the engine underneath "the memory file" and "semantic search."
What this series covers
Six parts, each a standalone piece of the problem:
- This one — why self-recall is a different problem.
- Hybrid RAG on nothing but Postgres — the architecture, and why I didn't reach for a dedicated vector database.
- Teaching RAG to say "I don't know" — the three honesty guards, in detail.
- Benchmarking retrieval and honesty — the eval harness, and why I measure a false-confident rate alongside MRR.
- The gap threshold that didn't transfer — the finding that a single hard-coded abstention threshold is worthless across embedding models. This one surprised me.
- The fine-tune that did nothing, and shipping it as an MCP server — an honest null result, then how the whole thing deploys.
A note on tone, because it's the point: this track reports what didn't work. A fine-tuning experiment that produced zero lift. An abstention threshold that fell apart the moment I changed embedders. In most domains those get buried. In this domain — calibration, honesty, knowing your limits — the negative results are the most useful thing I can hand you.
Next
Part 2 builds the retrieval core: dense plus sparse plus fusion plus reranking, all inside a single Postgres database, with pluggable embedders — and the argument for why, in 2026, you probably don't need a separate vector store to do this well.
Update (July 2026): the comments below did exactly what publishing is for. Two of them — the entailment-over-similarity argument and "supersession is a relation, not a property" — became measured experiments and shipped as RE-call v0.3: an opt-in entailment stage for the near-miss no threshold can catch, write-time supersession that beats even a steelmanned timestamp heuristic (83–100% stale-trust → 0.00), and a supersession lint. The full follow-up, with the commenters' names on it: What the Comments Taught Me (RE-call v0.3). The "three honesty guards" described above were v0.1's set — the current table has six.
Part 1 of Retrieval-Augmented Self-Recall. Code: RE-call (Postgres + pgvector, MIT). If you came from Claude Code, Beyond the Prompt, this is the engine under Part 1's memory and Part 5's search.
Top comments (14)
The gap_warning idea is the part I'm most curious to see land, because "the best match is too weak to trust" implies a threshold, and thresholds on cosine similarity are notoriously slippery across content. You flagged in the outline that the threshold didn't transfer between embedding models, which I'd bet is the real story here. Is the plan a single global cutoff you re-tune per embedder, or something relative, like the gap between the top hit and the rest of the batch rather than an absolute score? The second feels more likely to survive a model swap, but I'm guessing at your part 5 before you've written it.
You've guessed the real story: the non-transfer is Part 5, and it's the whole finding.
Direct answer, and it's the less clever one: a per-embedder calibrated absolute cutoff. I re-fit the threshold against a small labeled set for each model rather than shipping a constant. Not elegant, but measured.
Your relative idea, top hit versus the rest of the batch, was the first thing I reached for too, and it is more scale-invariant. But it has a hole that's worse for agent memory specifically: it's blind to the single confident distractor. When memory holds one memo that's semantically adjacent but wrong, that hit stands out from the pack, big margin, and a spread-based check waves it through as confident. That's the exact case that does the damage, one strong near-miss read as a hit.
And you don't escape calibration there, you move it: the margin has its own cutoff, and that distribution shifts across embedders too. Maybe less. I haven't measured the two head to head.
So absolute-per-embedder is what shipped, relative is the experiment I still owe. Good one to leave me with.
This is the RAG problem I spend most of my time on, and the line about a better retriever returning wrong results more confidently is exactly right. I would sharpen the abstention fix one degree: a similarity score is not a confidence score. The near-misses that hurt most are high-similarity and wrong, semantically adjacent memos that a threshold-based gap_warning waves straight through. So the abstention signal cannot be the retriever's own score. What has held up for me is a separate check that the retrieved memo actually entails an answer to the query, not just that it sits nearby in embedding space. Proximity is a candidate; entailment is the evidence. Score the second, abstain on the second.
On supersession, the hardest of your three, the reason timestamps fail is that you are trying to infer a relation between two memos at read time, when both look valid in isolation. That inference is a losing game. The version that works is to capture the relation at write time: when a decision supersedes an earlier one, the new memo names what it replaces, so retrieval returns the current head of the chain instead of a resolved-but-still-embedded old memo. Bind the truth when it is created, do not reconstruct it from ambiguous artifacts later. Same principle as calibrated abstention, just moved from read time to write time.
Both your fixes are ahead of what I shipped. Conceding precisely.
On abstention: you're right that the score can't be the signal, and it exposes something I glossed over. My cross-encoder rerank is the halfway house that looks like your fix but isn't. It scores query and memo jointly, closer to entailment than cosine, but it still emits a relevance score I then threshold, which is the same mistake one layer up. Entailment gives a decision, not a score, and that's the deeper win: it sidesteps the per-embedder calibration problem entirely, because there's no number left to re-tune. The cost is a second model call per candidate, which is the honest reason people reach for the threshold instead.
On supersession: bind at write time, agreed. It's the same move as capturing the relation when the truth exists rather than reconstructing it from artifacts later. My anti-re-litigation guard is already a crude version of this, a closed-decisions index, and it fails exactly where you'd predict: it is only as complete as my discipline in linking the chain. But that is a far better place to fail than read-time inference, because a missing link is lintable. Impossible to infer becomes possible to enforce.
Most useful comment I've had on this one. Thank you.
"Entailment gives a decision, not a score" is a sharper statement of the fix than mine. No number left to re-tune is the whole win, and the second model call per candidate is the honest price tag on it.
On the lintable missing link: the lint can be automated with machinery you already run, by executing retrieval at write time. When a new memo lands, query the index with it before committing; any high-similarity hit among closed decisions that the new memo does not reference is a candidate unlinked chain, surfaced for a one-keystroke confirm. Same embedder, same index, just pointed at the write path instead of the read path. Your discipline problem becomes a diff the system shows you, and the chain stays complete without depending on you remembering to link it.
The split between retrieving external documents and retrieving the agent's own prior reasoning is a distinction I wish more RAG write-ups made. The failure mode I keep hitting is an agent re-deriving a conclusion it already reached earlier in the same task because it can't recall its own intermediate state. How do you handle staleness when the recalled self-memory conflicts with newer context - does the freshest observation always win, or do you score them?
Freshest does not automatically win. Staleness is treated as a structural relation when I can, not a score.
For self-memory, the useful distinction is usually:
Observation: something the agent saw or the environment returned.
Conclusion: something the agent inferred from prior observations.
Decision/state: what the agent chose to do next.
If newer context contradicts an old conclusion, the old conclusion gets downgraded unless its supporting observations are still valid. But I don’t want “latest timestamp wins,” because recency is a bad proxy for truth. A newer note can be a partial observation, a mistaken inference, or only relevant to a different branch of the task.
The closest analogue to what I described above is: make invalidation explicit where possible. A recalled memory can be marked superseded_by, resolved_by, invalidated_by, or scoped to a task phase. Then retrieval doesn’t have to infer staleness from embedding score or timestamp. It can say: this memory is relevant but no longer authoritative.
When that structure is missing, I’d rather surface the conflict than silently rank it away: “I previously concluded X, but newer context says Y; using Y because it directly observes the current state.” So scoring can help retrieve the candidates, but it should not be the thing that decides authority.
This is a really important distinction. Traditional RAG assumes the answer exists and optimizes retrieval quality, while agent memory has to answer a different question first: should I trust any retrieved memory at all? We've run into similar challenges at IT Path Solutions building long-running AI agents, where false-positive retrieval often causes more damage than a missed retrieval because the agent confidently builds on incorrect assumptions. One thing I'd add is that retrieval should return confidence + provenance + validity, not just relevance. A memory that's semantically similar but superseded or outside its validity window should often lose to "I don't know." In production, calibrated abstention ends up being as important as retrieval accuracy.
Agreed, and the asymmetry has a mechanism worth naming: a miss is visible, a false positive is invisible. Return nothing and you go look. Return a confident near-miss and you stop looking. Same error rate, opposite consequences.
Your triad maps roughly onto what I built: gap_warning is confidence, freshness is validity, anti-re-litigation is provenance. But you have named the weakest of the three, and it is mine. Freshness measures age, and age is a cheap proxy for validity that is wrong in both directions: a memo from a year ago can still be binding, one from yesterday can already be superseded and still look pristine to any timestamp check.
The hard part: supersession is not a property of a memo, it is a relation between two. A memo cannot know it has been superseded, only the pair knows, so no per-document freshness field catches it however good the timestamps are.
Thanks for the prompt, genuinely useful. Returning validity and provenance alongside confidence, rather than leaning on a timestamp, is the right shape and I will be investigating it.
The distinction between ranking and abstention is the crux — and it's the one most off-the-shelf stacks quietly sidestep. I ran into the same failure mode on a financial-news knowledge graph: the retriever would surface the three nearest nodes when asked about a company it had never seen, and the downstream model would treat confident retrieval as confirmation the entity existed. What helped was adding a coverage_check field to the Pydantic output schema so the model had a typed slot to say "retrieved but no match above threshold" rather than silently returning a near-miss. The gap_warning pattern you describe closes the same gap at the retriever layer, which is cleaner.
"Treated confident retrieval as confirmation the entity existed" is a sharp statement. That is the whole thing: retrieval returning something is not evidence the thing exists, but downstream it reads exactly like it.
On coverage_check versus gap_warning, I would say complementary rather than cleaner, because each covers the other's weakness. A typed slot in the output schema is the right instinct: it makes the signal impossible to quietly drop, which an annotation on a retrieval result very much can be. But if the model fills that slot, you are asking the thing that just hallucinated to report its own hallucination. It is the least reliable narrator about its own grounding.
So the split I would argue for: the retriever computes it, the schema carries it. Computed from the cosine distribution it is a measurement. Self-reported it is a declaration. Those two fail very differently.
One warning if you go threshold-based, since it cost me: the threshold does not transfer across embedders. Same constant, wildly different behaviour. Calibrate per model against a small labeled set.
Really sharp reframe — self-recall vs document-QA. Once the KB is the agent’s own memory, nearest-neighbor-as-yes is the failure mode. Better embeddings making you wrong faster tracks.
The three modes (gaps, re-litigating settled decisions, stale-as-current) aren’t ranking bugs. Calibrated abstention + measuring false-confident rate next to MRR is the objective most stacks never ask for.
We’ve been in the same neighborhood on agent memory, but from the compliance / accumulation side: append-only chat logs explode; what you want to keep is the gist graph, not every turn. On our Azure runs (real Gemini), PrismCortex stored the gist ~5.2× smaller than the raw log (18.7 KB → 3.6 KB in that fixture), and after 675 chatter turns the graph stayed flat — 0 new edges — while the append-log kept everything. That’s the “memory that doesn’t grow with small talk” piece. Separate from RE-call’s gap_warning / freshness / anti-re-litigation guards — different knobs — but same instinct: don’t treat the full transcript as the retrieval corpus forever.
Also care about byte-identical replay + bitemporal history when a fact corrects ($40k → $55k style updates) so “what did we believe then” isn’t lost.
Curious how you persist closed-decision markers so anti-re-litigation survives daily re-index.
RE-call appreciated. PrismCortex demo if useful: prismcortex-demo.insightits.com
Results: github.com/insightitsGit/PrismCort...
Great insight. The shift from optimizing retrieval to optimizing honest abstention is something most RAG discussions miss. "I don't know" is often the smartest answer an AI can give.
it's interesting