Part 2 of Retrieval-Augmented Self-Recall. Code: RE-call. Part 1: the self-recall problem.
Say "vector search" and the reflex is a dedicated vector database: Pinecone, Weaviate, Qdrant. I didn't install one. RE-call keeps the dense vectors, the full-text index, and the metadata you filter them by in a single Postgres.
Not as a shortcut, and not out of allergy to new infrastructure. Because for agent memory a separate vector store is the wrong shape, and it quietly costs you the one property this whole system depends on.
Here's the argument, and the retrieval pipeline it buys you.
Why not a dedicated vector DB
Agent memory has three properties that make Postgres the natural fit:
It's already relational. Memos have timestamps, source types, tags, decision status. That's structured metadata you want to filter and join on — exactly what a relational database is for. A separate vector store means keeping two systems in sync and losing transactional consistency between the vectors and the metadata.
pgvectorgives you real vector search inside Postgres. Approximate-nearest-neighbor cosine search, in the same database as your rows. And Postgres already ships full-text search. So you get dense and sparse retrieval in one transactional store — no sync layer, no second system to operate.The scale doesn't justify the complexity. My corpus is ~700 memos, about 5 MB, re-indexed daily. Even orders of magnitude larger, a read-mostly, latency-tolerant memory is nowhere near the regime where a distributed vector DB earns its operational cost. Reaching for one here is over-engineering.
One store, one source of truth, ops you already know. Now the pipeline.
The retrieval pipeline
RE-call retrieves in up to four stages. The first two run in parallel; the last two refine.
1. Dense retrieval
Embed the query, run a pgvector cosine-similarity search, take the top-k. This is semantic matching — it finds memos that mean the same thing as the query even with no shared words. Strong on concepts, weak on exact tokens.
2. Sparse retrieval
Run a Postgres full-text search (tsvector/tsquery) over the same corpus. This is lexical matching — it nails exact terms: a specific error code, a ticker, a piece of domain jargon, a proper noun. Strong on precision, blind to paraphrase.
3. Fusion with RRF
Now you have two ranked lists that disagree. You fuse them with Reciprocal Rank Fusion (k=60):
score(doc) = Σ 1 / (k + rank_in_list_i(doc))
Each document's score is the sum, across both lists, of one over its rank (plus a constant k). Documents that rank high in either list bubble up; documents high in both dominate.
The reason RRF specifically: dense cosine scores and full-text scores aren't on the same scale — you can't just add or average them without arbitrary normalization. RRF sidesteps that entirely by fusing on rank instead of raw score. k=60 is the well-established default and it's robust; you rarely need to tune it.
Why fuse at all? Because dense and sparse fail in opposite directions — one misses exact tokens, the other misses meaning. Combining them gives you the concept-matching of embeddings and the precision of keyword search. (If you read the applied series, this is the "hybrid beats either alone" lesson — here's the actual mechanism under it.)
4. Cross-encoder rerank (optional)
The fused shortlist can be reordered by a cross-encoder (ms-marco-MiniLM). Unlike the bi-encoder embeddings in stage 1 — which encode query and document separately and compare vectors — a cross-encoder encodes the query and a candidate together and scores their relevance jointly. It's meaningfully more accurate and meaningfully slower, so you only run it on the top handful of candidates, never the whole corpus.
Whether stage 4 is worth its cost turns out to depend heavily on your embedder — which is the subject of Part 4's benchmark.
One property of this stage matters more than I realized when I first drafted this: the cross-encoder reorders, but what it emits is still a score you end up thresholding somewhere downstream. That distinction — a score you tune versus a decision you can trust — comes back with force later in the series.
Pluggable embedders
RE-call treats the embedder as a swappable component, with three shipped:
-
HashingEmbedder— deterministic, offline, no model download. It exists so the test suite and CI can run vector retrieval with zero external dependencies. Weak, but reproducible. -
FastEmbed (
bge-small) — a strong local model, no API calls. Good default for privacy or air-gapped runs. -
Voyage (
voyage-3) — a cloud model, the strongest of the three.
Pluggability isn't just tidiness. It lets you test deterministically offline, run locally for privacy, or call the cloud for maximum quality — same pipeline, different tradeoff. And it sets up the single most important finding in this series: the embedder you choose changes how you have to calibrate everything downstream (Part 5). Hold that thought.
Is "just Postgres" actually enough?
Fair challenge, and I don't want to hand-wave it. The claim is backed by the harness, not vibes: of RE-call's 150-test suite, 49 integration tests run against a real pgvector container — no mock database — in CI. The retrieval you just read about is exercised against the real engine on every commit.
And to be honest about the boundary: when would you want a dedicated vector DB? Billions of vectors, sub-10 ms p99 under heavy concurrent QPS, distributed sharding across nodes. Agent memory is none of those. It's small, read-mostly, and perfectly happy with tens-of-milliseconds retrieval. Match the tool to the regime — and this regime is Postgres-shaped.
Next
Retrieval now returns the closest memos. But "closest" is not "relevant" — the closest match to a question with no answer is still just noise wearing a high similarity score. Part 3 is about the guards that let the system tell the difference: how RE-call learns to say "I don't know."
Part 2 of Retrieval-Augmented Self-Recall. Code: RE-call. If you came from Claude Code, Beyond the Prompt, this is the retrieval layer under Part 5's semantic search.
Top comments (0)