Phase 4: Retrieval Quality & Grounded Answers

From "Closest Match" to "Answer You Can Actually Trust"

---

The Story Starts: "Why Did It Confidently Lie to Me?"

👦 Nephew: Uncle, I tested our RAG system this week. I asked about a leave policy that doesn't exist in any of our documents. Instead of saying "I don't know," it made up a completely fake answer. Confidently. With a fake number of days!

👨‍🦳 Uncle: Welcome to the most common failure in RAG systems — and the exact reason Phase 4 exists. Tell me — in Phase 3, what did Top-K retrieval actually guarantee?

👦 Nephew: It... returns the 5 closest vectors?

👨‍🦳 Uncle: Say that sentence again, slowly, and notice what it does NOT say.

👦 Nephew: It doesn't say those 5 are actually... relevant. Just that they're the closest of whatever exists.

👨‍🦳 Uncle: Exactly the gap. Even if your database has zero chunks about leave policy, Top-K will still confidently hand back 5 chunks — probably about something completely unrelated, like office timings or dress code — because "closest" is a relative ranking, not a quality guarantee. The LLM then does what LLMs do: it takes whatever context it's given and tries its best to answer from it, even when that context is garbage. That's not the LLM's fault. That's a retrieval design flaw. Today we fix it.

---

Phase 4 Overview: Four Key Concepts

Phase 3: Storage, Indexing & Token Economics ✅
    ↓
Phase 4: Retrieval Quality & Grounded Answers ← WE ARE HERE
    │
    ├─ Step 1: Reranking
    │         (Retrieve cheap & wide, then score properly)
    │
    ├─ Step 2: Relevance Threshold / Abstention
    │         (Knowing when to say "I don't know")
    │
    ├─ Step 3: Grounded Prompt Assembly with Citations
    │         (Every claim traceable to a source)
    │
    └─ Step 4: Hybrid Search as a Safety Net
              (Catching what vector search misses)
    ↓
Phase 5: Evaluation & Guardrails at Scale

---

Step 1: Reranking — Retrieve Wide, Then Score Properly

Why "Closest" Isn't "Most Relevant"

👨‍🦳 Uncle: Let's go back to your resume example from Phase 2. Vector similarity is fast because it's approximate — that's the whole point of the ANN index we built in Phase 3. But "fast and approximate" means the ranking among your Top-20 candidates can genuinely be a bit sloppy at the edges.

Query: "What is the notice period for resignation?"

Vector search Top-5 (by cosine similarity):
1. "Notice period starts from submission date"        (0.89)
2. "Employees must inform HR before leaving"           (0.85)
3. "Resignation letters must be signed by manager"     (0.84)
4. "Office holiday calendar for 2025"                  (0.81)  ← WRONG, but close enough to sneak in
5. "Exit interview process overview"                   (0.79)

👦 Nephew: Wait — the holiday calendar chunk scored 0.81? That's completely unrelated!

👨‍🦳 Uncle: This happens more than beginners expect. Embedding models compress meaning into 1536 numbers — some unrelated chunks accidentally land "nearby" in that space due to shared vocabulary, formatting, or document structure, even when a human would instantly see they don't answer the question. Vector search is a wide net, not a precise judge.

The Fix: Two-Stage Retrieval

👨‍🦳 Uncle: The production pattern is always two stages, never one:

STAGE 1 — Retrieve (cheap, wide, approximate)
─────────────────────────────────────────
Vector search → Top-20 candidates
Fast (uses HNSW index from Phase 3)
Casts a wide net, some noise expected

STAGE 2 — Rerank (expensive, narrow, accurate)
─────────────────────────────────────────
A RERANKER model looks at the actual QUERY + each
CANDIDATE CHUNK together, and scores relevance directly
→ Keep only Top-5 after reranking
Slower per-item, but far more accurate

👦 Nephew: What makes a reranker more accurate than the embedding similarity we already had?

👨‍🦳 Uncle: Here's the key architectural difference, and it's worth understanding properly:

Embedding model (bi-encoder):
  Query  → [vector A]  ─┐
                          ├─ compare AFTER, separately
  Chunk  → [vector B]  ─┘
  The model NEVER sees query and chunk together.

Reranker (cross-encoder):
  [Query + Chunk] → fed into model TOGETHER
                   → model directly outputs a relevance score
  The model sees BOTH at once, so it can reason about
  how they actually relate — not just how "nearby" their
  independently-computed vectors happen to be.

👨‍🦳 Uncle: A cross-encoder is slower — you can't pre-compute anything, because it needs the query at scoring time — which is exactly why you never run it against millions of chunks directly. You run it only against the small Top-20 that vector search already narrowed down for you. Cheap net first, expensive judge second.

Code: Reranking with Cohere's Rerank API

npm install cohere-ai

const { CohereClient } = require("cohere-ai");
const cohere = new CohereClient({ token: process.env.COHERE_API_KEY });

async function rerankChunks(query, candidates, topN = 5) {
  // candidates = [{ chunk_text, metadata, similarity }, ...]  (Top-20 from vector search)

  const response = await cohere.rerank({
    model: "rerank-english-v3.0",
    query: query,
    documents: candidates.map(c => c.chunk_text),
    topN: topN,
  });

  // response.results gives back indices INTO our original array,
  // re-sorted by true relevance score
  return response.results.map(result => ({
    ...candidates[result.index],
    rerank_score: result.relevanceScore,
  }));
}

// Usage:
// const top20 = await vectorSearch(queryEmbedding, { limit: 20 });
// const top5 = await rerankChunks(userQuestion, top20, 5);

👦 Nephew: Why 20 candidates and not, say, 5 straight from vector search?

👨‍🦳 Uncle: Because if the true best chunk was sitting at position 8 in the approximate vector ranking (remember — approximate!), and you only ever pulled Top-5, the reranker never even gets a chance to see it. Retrieving wider (Top-20 or Top-30) before reranking gives the accurate judge enough candidates to actually find the right answer, even when the fast-but-approximate first pass ranked it imperfectly.

---

Step 2: Relevance Threshold — Knowing When to Say "I Don't Know"

👨‍🦳 Uncle: Now the fix for your original problem — the fake leave policy answer. Reranking alone doesn't solve it, because even after reranking, the system will still confidently hand back its "Top-5 best of what exists" — even if none of them are actually good.

👦 Nephew: So we need... a minimum bar? Like, "if nothing scores above X, don't even bother answering"?

👨‍🦳 Uncle: Precisely. This is called a relevance threshold, and it's one of the highest-leverage, most-skipped guardrails in RAG systems.

const RELEVANCE_THRESHOLD = 0.5; // tune per your reranker's score distribution

async function retrieveWithAbstention(query) {
  const top20 = await vectorSearch(query, { limit: 20 });
  const reranked = await rerankChunks(query, top20, 5);

  const bestScore = reranked[0]?.rerank_score ?? 0;

  if (bestScore < RELEVANCE_THRESHOLD) {
    return {
      hasRelevantContext: false,
      chunks: [],
    };
  }

  return {
    hasRelevantContext: true,
    chunks: reranked,
  };
}

Then, at prompt-assembly time:

async function answerQuestion(query) {
  const { hasRelevantContext, chunks } = await retrieveWithAbstention(query);

  if (!hasRelevantContext) {
    return "I don't have information about that in the available documents. Could you check with HR directly, or rephrase your question?";
  }

  // proceed to Step 3 — build a grounded prompt from `chunks`
}

👦 Nephew: How do I pick the actual threshold number? 0.5 feels arbitrary.

👨‍🦳 Uncle: It is somewhat empirical, and that's honest to say — you tune it against your own data. A practical way: run a batch of known "should answer" questions and known "should NOT be answerable" questions through your reranker, look at the score distributions for each group, and pick a threshold that sits in the gap between them. Revisit it periodically — as your document set grows, the distribution can shift.

Mental model — the bouncer at the door: Vector search invites 20 people who "sort of look like" they belong. The reranker checks IDs properly. The threshold is the bouncer's rule: "if literally nobody in this line actually qualifies, don't let anyone in — just close the door and say so." Without that bouncer, your system politely waves in whoever's closest to the front of the line, qualified or not.

---

Step 3: Grounded Prompt Assembly with Citations

The Problem with a Plain Context Dump

👨‍🦳 Uncle: Even with great chunks, how you hand them to the LLM matters. A lazy prompt looks like this:

Context:
Employees receive 30 days notice period.
Notice period starts from submission date.
Manager approval required for resignation.

Question: What is the notice period?

👦 Nephew: What's wrong with that? It has the right chunks.

👨‍🦳 Uncle: Nothing technically wrong — but if the LLM's answer is later questioned ("where did you get 30 days from?"), you have no way to trace the claim back to a specific source. In a compliance-sensitive domain — HR policy, legal, medical, financial — "trust me" isn't good enough. You need traceability.

The Fix: Numbered, Attributable Context

function buildGroundedPrompt(query, chunks) {
  const contextBlock = chunks
    .map((c, i) => `[${i + 1}] (Source: ${c.metadata.source_file}, ${c.metadata.department})\n${c.chunk_text}`)
    .join("\n\n");

  return `You are answering questions using ONLY the numbered sources below.

${contextBlock}

Instructions:
- Answer using ONLY the information in the sources above.
- After each claim, cite the source number in brackets, like [1].
- If the sources don't fully answer the question, say so explicitly —
  do not guess or use outside knowledge.

Question: ${query}

Answer:`;
}

Example LLM output with this prompt:

"The notice period is 30 days, starting from the date of submission [1][2].
Resignation also requires manager approval before it is finalized [3]."

👦 Nephew: Now I can map [1], [2], [3] straight back to the actual source_file and department in metadata!

👨‍🦳 Uncle: Exactly — and this is where Phase 3's metadata design finally pays off in the user-facing product. You can render those citations as clickable links back to the original document, which is the difference between "an AI said so" and "here's exactly where this comes from, go verify it yourself."

Code: Parsing Citations Back to Sources

function attachCitationSources(llmAnswer, chunks) {
  const citationPattern = /\[(\d+)\]/g;
  const usedIndices = new Set();
  let match;

  while ((match = citationPattern.exec(llmAnswer)) !== null) {
    usedIndices.add(parseInt(match[1], 10) - 1);
  }

  const sources = [...usedIndices].map(i => ({
    text: chunks[i]?.chunk_text,
    file: chunks[i]?.metadata?.source_file,
    department: chunks[i]?.metadata?.department,
  }));

  return { answer: llmAnswer, sources };
}

---

Step 4: Hybrid Search — Catching What Vector Search Misses

The Blind Spot: Exact Terms

👨‍🦳 Uncle: One more honest weakness. Say someone asks:

"What does error code ERR_4521 mean?"

👦 Nephew: Vector search should handle that fine, right? It's just text.

👨‍🦳 Uncle: Try it in your head using what you learned in Phase 2. Embedding models are trained to understand meaning and concepts — "leadership" relating to "team management." But "ERR_4521" isn't a concept — it's an exact, arbitrary identifier. The embedding model has no real semantic understanding of a specific error code string; it might embed it as "some kind of technical identifier" and miss the one chunk that specifically documents ERR_4521, especially if that chunk's surrounding language doesn't semantically resemble the question.

👦 Nephew: So the same weakness applies to product codes, order numbers, exact names...

👨‍🦳 Uncle: Exactly — anything where exact match matters more than meaning. This is precisely where old-fashioned keyword search still wins, and it's why production systems don't choose one or the other — they run both, and combine the results. That's hybrid search.

Architecture: Running Both Searches Together

User Query: "What does error code ERR_4521 mean?"
              ↓
        ┌─────┴─────┐
        ↓           ↓
  Vector Search   Keyword Search (BM25 / full-text)
  (semantic)      (exact term matching)
        ↓           ↓
   Top-10         Top-10
        └─────┬─────┘
              ↓
      Merge & Deduplicate
              ↓
      Rerank the COMBINED set (Step 1)
              ↓
         Final Top-5

Code: Postgres Full-Text Search Alongside pgvector

-- Add a full-text search column (once, at table setup time)
ALTER TABLE document_chunks
  ADD COLUMN chunk_tsv tsvector
  GENERATED ALWAYS AS (to_tsvector('english', chunk_text)) STORED;

CREATE INDEX idx_chunk_tsv ON document_chunks USING GIN (chunk_tsv);

-- Keyword search query
SELECT chunk_text, metadata,
       ts_rank(chunk_tsv, plainto_tsquery('english', $1)) AS keyword_score
FROM document_chunks
WHERE chunk_tsv @@ plainto_tsquery('english', $1)
ORDER BY keyword_score DESC
LIMIT 10;

Node.js: Combining Both Result Sets

async function hybridSearch(query, queryEmbedding) {
  const [vectorResults, keywordResults] = await Promise.all([
    db.query(
      `SELECT chunk_text, metadata, 1 - (embedding <=> $1) AS score
       FROM document_chunks ORDER BY embedding <=> $1 LIMIT 10`,
      [queryEmbedding]
    ),
    db.query(
      `SELECT chunk_text, metadata,
              ts_rank(chunk_tsv, plainto_tsquery('english', $1)) AS score
       FROM document_chunks
       WHERE chunk_tsv @@ plainto_tsquery('english', $1)
       ORDER BY score DESC LIMIT 10`,
      [query]
    ),
  ]);

  // Merge, de-duplicating by chunk_text, keeping the highest score seen
  const merged = new Map();
  for (const row of [...vectorResults.rows, ...keywordResults.rows]) {
    const key = row.chunk_text;
    if (!merged.has(key) || merged.get(key).score < row.score) {
      merged.set(key, row);
    }
  }

  const combined = [...merged.values()];
  return rerankChunks(query, combined, 5); // Step 1's reranker makes final sense of it
}

👦 Nephew: So hybrid search doesn't replace reranking — it feeds reranking a better, more complete set of candidates first?

👨‍🦳 Uncle: Exactly right. All four steps today form one pipeline, not four separate options: hybrid search casts the widest, smartest net (semantic and exact); reranking judges that combined set accurately; the relevance threshold decides whether any of it is good enough to answer from; and grounded prompting makes sure whatever answer comes out is traceable.

---

Complete Query-Time Flow, End to End

User Question
    ↓
Embed query (Phase 2)
    ↓
┌─────────────────────────────────────┐
│  HYBRID SEARCH (Step 4)              │
│  Vector search (Top-10)              │
│  + Keyword/BM25 search (Top-10)      │
│  → merge & deduplicate               │
└─────────────────────────────────────┘
    ↓
┌─────────────────────────────────────┐
│  RERANKING (Step 1)                  │
│  Cross-encoder scores combined set   │
│  → Top-5 by true relevance           │
└─────────────────────────────────────┘
    ↓
┌─────────────────────────────────────┐
│  RELEVANCE THRESHOLD (Step 2)        │
│  Best score < threshold?             │
│  → YES: return "I don't know"        │
│  → NO: continue                      │
└─────────────────────────────────────┘
    ↓
┌─────────────────────────────────────┐
│  GROUNDED PROMPT (Step 3)            │
│  Numbered, attributable context      │
│  → LLM generates cited answer        │
│  → Parse citations back to sources   │
└─────────────────────────────────────┘
    ↓
Final Answer + Verifiable Sources

---

Interview-Level Answers

Q1: Why do production RAG systems retrieve more chunks than they actually use?

👨‍🦳 Uncle: "Because the initial retrieval (vector search over an ANN index) is fast but approximate — the true best match isn't guaranteed to rank in the very top positions. By retrieving a wider candidate set (e.g., Top-20) and then applying a more accurate but slower reranking model, the system gets a chance to correctly identify the best matches that the initial approximate search may have under-ranked, before narrowing down to the final Top-K actually sent to the LLM."

Q2: What's the difference between a bi-encoder and a cross-encoder, and why does it matter for reranking?

"A bi-encoder — the model used for the original embeddings — encodes the query and each document independently into vectors, which are compared afterward using something like cosine similarity. This is fast and can be precomputed, which is why it's used for the initial large-scale search. A cross-encoder, used for reranking, takes the query and a candidate document together as a single input and directly outputs a relevance score, allowing it to reason about their relationship more accurately — at the cost of being too slow to run against millions of documents directly, which is why it's only applied to the small candidate set the bi-encoder already narrowed down."

Q3: How do you prevent a RAG system from hallucinating when no relevant document exists?

"By applying a relevance threshold after retrieval and reranking — if the best-scoring retrieved chunk falls below an empirically-tuned minimum relevance score, the system should explicitly abstain and respond that it doesn't have relevant information, rather than passing weak or unrelated context to the LLM and letting it generate a plausible-sounding but ungrounded answer. This threshold is typically tuned by examining reranker score distributions across known answerable versus known unanswerable questions."

Q4: Why combine vector search with keyword search instead of relying on embeddings alone?

"Embedding models excel at capturing semantic meaning and conceptual relationships, but they can underperform on exact-match needs like product codes, error codes, or specific identifiers, since these carry little inherent 'meaning' for the model to encode. Keyword-based search (e.g., BM25 or Postgres full-text search) reliably catches exact term matches. Running both searches and merging their results before reranking — hybrid search — combines semantic understanding with exact-match reliability, covering each other's blind spots."

Q5: What does "grounded" mean in the context of a RAG-generated answer?

"A grounded answer is one where every factual claim is explicitly traceable to a specific retrieved source, typically enforced by instructing the LLM to cite source numbers inline and restricting it to only use the provided context. This allows the system (and the end user) to verify exactly where each part of an answer came from, rather than trusting an unverifiable claim — which is especially critical in compliance-sensitive domains like HR, legal, or financial documentation."

---

The Complete Architecture So Far

PHASE 1: DOCUMENT INGESTION ✅
─────────────────────────────
PDF Upload → File Hash Check → Parse & Clean → Chunking
  → Deduplication → Store Chunk Text

PHASE 2: EMBEDDINGS & SEMANTIC SEARCH ✅
─────────────────────────────
Chunk Text → Tokenization → Embedding Layer → Vector
  → Cosine Similarity → Top-K Retrieval

PHASE 3: STORAGE, INDEXING & TOKEN ECONOMICS ✅
─────────────────────────────
Vector + Metadata → Postgres (pgvector) → HNSW/IVFFlat Index
  → Metadata Indexing (GIN/generated columns) → Pre-filtering
  → Token Economics (dedup, batching, caching)

PHASE 4: RETRIEVAL QUALITY & GROUNDED ANSWERS ← YOU ARE HERE
─────────────────────────────
Hybrid Search (vector + keyword) → merge candidates
  ↓
Reranking (cross-encoder) → true relevance scoring
  ↓
Relevance Threshold → abstain if nothing qualifies
  ↓
Grounded Prompt Assembly → numbered, cited context
  ↓
LLM Answer + Parsed Citations back to source documents

PHASE 5: EVALUATION & GUARDRAILS AT SCALE (next)
─────────────────────────────
Precision/Recall metrics → Groundedness scoring
  → Hallucination detection → Prompt injection defense
  → Cost & latency observability

---

Summary: What Phase 4 Solves

Problem	Phase 3	Phase 4
Fast search at scale	✅ ANN indexing	✅ Unchanged, still relies on it
Ranking accuracy among candidates	❌ Approximate only	✅ Reranking (cross-encoder)
Confidently answering with no basis	❌ Not addressed	✅ Relevance threshold / abstention
Traceable, verifiable answers	❌ Not addressed	✅ Grounded prompts with citations
Exact-match terms (codes, IDs)	❌ Semantic search misses these	✅ Hybrid search (vector + keyword)

---

Key Takeaways

• Vector search finds "closest," not "correct" — reranking closes that gap
• Bi-encoder (embeddings) = fast, precomputed, independent. Cross-encoder (reranker) = slow, accurate, sees query+chunk together
• Retrieve wide (Top-20), rerank narrow (Top-5) — never rerank straight from a Top-5 vector search
• A relevance threshold is what stops your system from confidently answering from garbage context
• Grounded prompting = numbered sources + citation instructions + parsing citations back to metadata
• Hybrid search (vector + keyword/BM25) catches exact-match terms that pure semantic search misses
• The four steps form one pipeline: hybrid search → rerank → threshold → grounded prompt — not four separate optional features

---

Next: Phase 5 — Evaluation & Guardrails at Scale

Now that you understand:

• Why "closest" and "correct" aren't the same thing
• How reranking and relevance thresholds turn approximate search into trustworthy retrieval
• How to make every answer traceable back to its source
• Why hybrid search exists alongside vector search, not instead of it

We'll implement Phase 5 in Node.js:

• Measuring retrieval quality with precision/recall against a test question set
• Automated groundedness/faithfulness scoring (does the answer actually match the cited sources?)
• Detecting prompt injection hidden inside retrieved documents
• Tracing and cost/latency observability for a production RAG API

Ready?

Remember: Less noise, more action. Phase 4 is where a RAG system stops guessing and starts knowing when it actually knows something.

Comments

[Nazar Boyko]

In hybridSearch the merge compares cosine similarity against ts_rank, and those two live on pretty different scales. Does that matter here, or does handing the whole merged set to the reranker make the merge order irrelevant anyway? Asking because the usual fix I see for this is reciprocal rank fusion, and I wonder if a cross-encoder afterwards just makes it unnecessary.

[Mads Hansen]

The distinction between relative ranking and an absolute quality bar is the key lesson here. One production refinement: calibrate abstention on the entire answer contract, not only the top reranker score. A high-scoring chunk can still support only part of a compound question. I would evaluate claim-level coverage, source authority, freshness, and contradiction, then return an answerable/partial/unanswerable state with the retrieval receipt. Also keep thresholds segmented by query class and corpus version; one global 0.5 can drift silently as documents and reranker versions change. A planted negative-question suite plus precision-recall curves for abstention makes the cutoff reviewable instead of a magic constant.

[James Sanderson]

The leave-policy example nails the trap - "closest" being a relative ranking with no floor is exactly why an empty corpus still hands back five confident chunks. Reranking sharpens the ordering, but does it solve the abstention case? If nothing relevant exists, a reranker still has to pick a top result. I've found you need an absolute score threshold (or an explicit "none of these clear the bar" escape hatch) on top of reranking, otherwise the system just reorders garbage instead of admitting it has nothing. Curious how you're handling the "I don't know" path in Phase 4.

[Kartik N V J K]

The point that Top-K is a relative ranking and not a quality guarantee is the crux, and it is why a relevance threshold with real abstention does more for trust than swapping in a bigger model. In practice I have found the reranker score alone is a shaky abstain signal, so pairing it with a grounding check on the drafted answer catches the confident-fake-number case better than a retrieval threshold by itself. When you reach Phase 5, will you measure abstention precision separately, so you can see how often it says "I don't know" on questions it actually could have answered?