Part 1 of Retrieval-Augmented Self-Recall — the research track behind Claude Code, Beyond the Prompt. All code is open source: RE-call.
Almost every RAG tutorial you've read solves the same problem: you have a pile of documents, a user asks a question, and you retrieve the chunks that answer it. Rank the right passage to the top, stuff it in the prompt, done.
There's a second kind of RAG that behaves completely differently, and almost nobody writes about it: an agent retrieving from its own memory.
I ran into it building the operational brain behind an automated trading system. That agent accumulates memory as it works — over 700 typed markdown memos, about 5 MB, re-indexed daily. Decisions, dead ends, calibration notes, "we tried this and it failed" postmortems. When the agent starts a task, it queries that memory: have we tested this before? what did we decide about X? is this still true?
The moment your knowledge base is the agent's own growing memory, the RAG problem inverts — and standard retrieval quietly does the wrong thing.
Why self-recall is not document QA
In document QA, there's a load-bearing assumption you probably never think about: the answer is in the corpus. Someone asked a question because the docs can answer it. Your whole job is ranking — surface the right chunk.
In self-recall, the most important queries are exactly the ones where the answer isn't there.
The agent asks "have we tried a mean-reversion filter on this market?" If the honest answer is no, never, a ranking-optimized retriever will still cheerfully return the three most cosine-similar memos — probably something about a different filter on a different market — and the agent, seeing confident results, concludes "yes, we looked at this." It just made a decision on a hallucination.
The failure isn't bad ranking. The top results might be the genuinely closest memos. The failure is that the system had no way to say "there's nothing relevant here."
Three failure modes unique to agent memory
Once you look at memory this way, three distinct failure modes show up that document-QA RAG never has to handle:
Hallucinating over gaps. The query has no real answer in memory, but retrieval returns the nearest neighbors anyway, and their presence reads as a "yes." The system needs to abstain — to flag "this is probably a gap" instead of pretending.
Re-litigating settled decisions. The agent proposes an idea it already tried and killed three months ago, because the "we decided against this, here's why" memo didn't surface at the moment of proposing. Memory that can't defend its own past decisions is doomed to relive them.
Acting on stale memory. A memo that was true in April is retrieved and treated as current in July. Without a freshness signal, old truth and current truth are indistinguishable — and in a system that touches money, that's expensive.
None of these are ranking problems. You cannot fix them by getting a better embedding model or a fancier reranker, because a better retriever just returns more confidently wrong results faster.
The reframe: abstention, not ranking
Here's the thesis of this whole series:
Document-QA RAG optimizes ranking. Agent-memory RAG has to optimize calibrated abstention — knowing when it doesn't know.
This matters because the metrics you've been trained to care about — MRR, nDCG, precision@k — don't measure abstention at all. They score how well you ordered the results assuming an answer exists. They are silent on the case that matters most in self-recall: the query with no answer, where the correct behavior is to return nothing and say so.
That's why reaching for an off-the-shelf RAG stack and pointing it at your agent's memory feels fine in a demo and rots in production. The stack is tuned for the wrong objective. It was never asked to abstain, so it doesn't.
RE-call: a reference implementation
To work through this properly I built and open-sourced RE-call — a retrieval engine designed around abstention from the start rather than bolted on after. The one-paragraph version, which the rest of this series unpacks:
- Storage & retrieval: PostgreSQL + pgvector as a single transactional store — dense vector search and sparse full-text search in the same database, fused with Reciprocal Rank Fusion, with an optional cross-encoder reranker.
-
Three honesty guards: a
gap_warningthat fires when the best match is too weak to trust, a freshness signal that flags stale memory, and an anti-re-litigation check that surfaces closed decisions before the agent re-proposes them. - Honest evaluation: a harness that measures not just ranking quality but a false-confident rate — how often the system fails to abstain when it should.
It ships as an MCP server, so an agent (Claude or otherwise) can query its own memory directly. That's the loop that closes back to my other series — this is the engine underneath "the memory file" and "semantic search."
What this series covers
Six parts, each a standalone piece of the problem:
- This one — why self-recall is a different problem.
- Hybrid RAG on nothing but Postgres — the architecture, and why I didn't reach for a dedicated vector database.
- Teaching RAG to say "I don't know" — the three honesty guards, in detail.
- Benchmarking retrieval and honesty — the eval harness, and why I measure a false-confident rate alongside MRR.
- The gap threshold that didn't transfer — the finding that a single hard-coded abstention threshold is worthless across embedding models. This one surprised me.
- The fine-tune that did nothing, and shipping it as an MCP server — an honest null result, then how the whole thing deploys.
A note on tone, because it's the point: this track reports what didn't work. A fine-tuning experiment that produced zero lift. An abstention threshold that fell apart the moment I changed embedders. In most domains those get buried. In this domain — calibration, honesty, knowing your limits — the negative results are the most useful thing I can hand you.
Next
Part 2 builds the retrieval core: dense plus sparse plus fusion plus reranking, all inside a single Postgres database, with pluggable embedders — and the argument for why, in 2026, you probably don't need a separate vector store to do this well.
Part 1 of Retrieval-Augmented Self-Recall. Code: RE-call (Postgres + pgvector, MIT). If you came from Claude Code, Beyond the Prompt, this is the engine under Part 1's memory and Part 5's search.
Top comments (2)
The distinction between ranking and abstention is the crux — and it's the one most off-the-shelf stacks quietly sidestep. I ran into the same failure mode on a financial-news knowledge graph: the retriever would surface the three nearest nodes when asked about a company it had never seen, and the downstream model would treat confident retrieval as confirmation the entity existed. What helped was adding a coverage_check field to the Pydantic output schema so the model had a typed slot to say "retrieved but no match above threshold" rather than silently returning a near-miss. The gap_warning pattern you describe closes the same gap at the retriever layer, which is cleaner.
"Treated confident retrieval as confirmation the entity existed" is a sharp statement. That is the whole thing: retrieval returning something is not evidence the thing exists, but downstream it reads exactly like it.
On coverage_check versus gap_warning, I would say complementary rather than cleaner, because each covers the other's weakness. A typed slot in the output schema is the right instinct: it makes the signal impossible to quietly drop, which an annotation on a retrieval result very much can be. But if the model fills that slot, you are asking the thing that just hallucinated to report its own hallucination. It is the least reliable narrator about its own grounding.
So the split I would argue for: the retriever computes it, the schema carries it. Computed from the cosine distribution it is a measurement. Self-reported it is a declaration. Those two fail very differently.
One warning if you go threshold-based, since it cost me: the threshold does not transfer across embedders. Same constant, wildly different behaviour. Calibrate per model against a small labeled set.