Why my RAG retriever couldn't find a number it was quoted verbatim

what is RAG?

Retrieval Augmented Generation. RAG gives an LLM the specific info relevant to a prompt by retrieving it from your own documents first, then answering from those passages.

Embedding

Representing words, phrases, or images as vectors in a high-dimensional space.

what is a vector in high-dimensional space?

The model gives 768 numbers per text. Each number = a coordinate. So every text = a point in 768-D space. Similar meaning → close, unrelated → far. Closeness ≈ meaning.

Chunking

Breaking the material into smaller pieces, so each vector represents one focused passage instead of a whole document averaged out.

Chunking strategies

• recursive-character, semantic, page-level, LLM-based, size-based, late chunking
• the wrong strategy can open a large recall gap

what is recall? — did retrieval fetch the right chunk to pass to the LLM?

what is recall@k? — was the right chunk inside the top-k cutoff?

Let's start

1. Took 4 con-call transcripts of a company (Company A) — chose transcripts because they're pure-text PDFs (no tables).
2. Used a node script to extract contents, with cleanup to remove the cover pages.
3. Chunked them into size 1000, overlap 200 → chunks span chars 0–1000, 800–1800 …

why overlap?

If a hard cut falls at char 1000, mid-sentence, that sentence gets split: "...revenue for the" ends chunk 0, "quarter was ₹340 crore" starts chunk 1. Now neither chunk contains the complete fact. When you embed them, chunk 0's vector is about a dangling fragment, chunk 1's about another — and a query like "what was quarterly revenue" may match neither well, because the whole idea was severed. Overlap fixes this by making adjacent chunks share their edges.

Now let's choose an embedding model

• ollama nomic-embed-text or mxbai-embed-large?
• nomic-embed-text — 274 MB, enough for 4 docs.

Let's query — recall@5

Query: how much did revenue grow in Q2

[0.717] CompanyA_Q3_FY26#11 (Q3 FY26) ...total standalone revenue for the...
[0.708] CompanyA_Q2_FY26#15 (Q2 FY26) ...hand over the call to our CFO...
[0.703] CompanyA_Q4_FY26#32 (Q4 FY26) ...royalty income added to revenue... EBITDA margin is stable...
[0.703] CompanyA_Q2_FY26#23 (Q2 FY26) ...targeting 10% to 15% revenue growth for FY '26...
[0.699] CompanyA_Q4_FY26#10 (Q4 FY26) ...pleasure to discuss our performance as we close...

what are 0.717, 0.708?

Cosine similarity scores — how close the query embedding vector is to each chunk's embedding vector.

• 1.0 = identical direction, 0.0 = unrelated, -1.0 = opposite direction (won't see this with real text)
• 0.717 means the chunk is closer to the query asked
• Don't read the absolute value as "72% relevant" — cosine isn't a percentage, and what's "high" depends on the model. With nomic on similar prose, ~0.7 is ordinary. What's diagnostic is the gap between hits, and here it's flat — a genuine finding about why retrieval on near-identical earnings calls is hard.

Note: asked Q2, top hit is Q3. The quarter token got ignored.

what if I ask an irrelevant question?

Query: who will win fifa world cup

[0.563] CompanyA_Q2_FY26#2  ...brief overview of the past quarter and first half...
[0.562] CompanyA_Q1_FY26#2  ...open the floor for Q&A...
[0.558] CompanyA_Q2_FY26#43 ...how much can we expect to win? probable success ratio?...
[0.557] CompanyA_Q4_FY26#16 ...growing the business gradually... if we win this award...

Data retrieved but irrelevant. cosine ALWAYS returns k results — no "no match". Scores are lower (~0.56 vs ~0.71), so the intuitive fix: reject anything below 0.6 and say "I don't know". But that fix is wrong — see failure 1.

From now, the actual problem starts

failure 1 — entity-blindness (the threshold can't work)

Query: What was Company B's revenue last quarter?

[0.758] CompanyA_Q4_FY26#10 (Q4 FY26) ...pleasure to discuss our performance...

Company B is NOT in the corpus (different company). Yet it scored 0.758 — HIGHER than every real in-domain query (max 0.717). The embedding locked onto "revenue" and ignored "Company B".

→ So the 0.6 threshold idea is dead: any cutoff that admits real questions (≤0.717) also admits Company B (0.758). Pizza (0.55, fully unrelated) is rejected fine, but wrong-entity-right-domain sails through. Dense embeddings capture topic, not entity.

failure 2 — quarter-blindness (fixed by metadata)

Query: PAT for Q3 was 75 crores PAT margin 7.7

[0.758] CompanyA_Q4_FY26#11 (Q4 FY26) ...EBITDA for Q4 FY '26 stood at INR183 crores... margin of 14.1%...
[0.736] CompanyA_Q1_FY26#45 (Q1 FY26) ...order book of around Rs. 10,300 crores...
[0.709] CompanyA_Q2_FY26#24 (Q2 FY26) ...margins around 11.5%. Second half should be similar...

Quoted Q3 verbatim, top 3 are Q4/Q1/Q2 — the target quarter isn't even there. The quarter token is nearly ignored.

fix: metadata filtering — parse the quarter from the query and hard-filter chunks to that quarter before ranking.

after metadata filtering (Q3 only):
[0.631] CompanyA_Q3_FY26#34 ...early completion bonus...
[0.631] CompanyA_Q3_FY26#31 ...INR605 crores invested in HAM...
[0.607] CompanyA_Q3_FY26#33 ...sale of those projects...
[0.604] CompanyA_Q3_FY26#32 ...outstanding cash as of December...

All Q3 now. Cross-quarter confusion solved — the one clean win.

failure 3 — token-blindness (chunking can't fix it)

Even after isolating PAT into its own sentence-chunk (literally "PAT for Q3 was 75 crores"), the query "what was PAT" STILL doesn't retrieve it. Verified via grep the figures were never split. So it's not chunking — the dense model won't connect the token "PAT" to a chunk containing it. Needs hybrid dense+sparse.

The fixes each finding points to:

• Entity-blindness → metadata (company field) or sparse; no cosine threshold works.
• Quarter-blindness → metadata filtering (scope to Q3).
• Token-blindness → hybrid dense+sparse (match the literal token dense smears away).

Cosine similarity is about retrieval, not answer accuracy

A chunk can score 0.72 and still not contain the number you need. recall@k is stricter (is the known-correct chunk in top-k) but still a retrieval metric — not answer correctness.

Moved towards sentence chunking

Questions used for the recall@k eval:

{ question: "What was total standalone revenue in Q3 FY26?", goldIds: ["CompanyA_Q3_FY26#38"] },
{ question: "What was standalone EBITDA margin in Q3 FY26?", goldIds: ["CompanyA_Q3_FY26#38", "CompanyA_Q3_FY26#39"] },
{ question: "What was PAT for Q3 FY26?", goldIds: ["CompanyA_Q3_FY26#39"] },
{ question: "What revenue growth guidance did management give for FY26?", goldIds: ["CompanyA_Q2_FY26#81", "CompanyA_Q2_FY26#82"] },

// out-of-domain: goldIds MUST stay empty (any hit = false positive)
{ question: "Who will win the FIFA World Cup?", goldIds: [] },       // ~0.56, rejected by 0.6
{ question: "What was Company B's revenue last quarter?", goldIds: [] }, // 0.758, HIGHER than in-domain
{ question: "Best pizza place in Mumbai?", goldIds: [] },            // ~0.55, no overlap

Strategy	recall@1	recall@5	PAT	Notes
Fixed 1000/200	0.75	0.75	miss	baseline
Sentence (3/1)	0.50	0.75	miss	theme fragmentation hurt guidance

Sentence chunking made recall@1 WORSE (0.75→0.50) and still didn't fix PAT — guidance fragmented across ~12 chunks. Strongly suggests the PAT failure is embedding-level, not chunk size. Next = hybrid.

Moving back to fixed chunking(recall@k got worse, not better — 0.75→0.50)

why did isolating PAT into a sentence chunk still miss?

Chunking and retrieval are different stages. Sentence-chunking only changed what's IN the chunk — but PAT didn't fail because of chunk contents, it failed because nomic places the query vector ("what was PAT") and the chunk vector ("PAT was ₹75 crores") FAR APART in embedding space. Chunk boundaries don't move those points closer. So a cleaner chunk can't fix a query↔chunk distance problem — the failure lives in the embedding, not the chunking. That's what makes it a clean negative result: it ruled out chunking as the cause.

So how does the search actually happen?

how to fix the query → chunk embedding distance problem?

The PAT chunk exists but the query "what was PAT" lands far from it in vector space. Two families of fix:

A. move the points closer

• better embedding model (bge-m3 etc) — might place them closer, not guaranteed
• query expansion — rewrite the query to look like the target text before embedding
• HyDE — embed a fake answer instead of the question (answers sit closer to answers)

B. stop relying on distance

• hybrid (dense + BM25 sparse) — match the literal token "PAT", ignores geometry
• metadata extraction — pull figures into fields, query structure not prose
• reranking — retrieve wide (top-30), re-score query+chunk together with a cross-encoder

For my case: HyDE or hybrid most likely to fix PAT. Both buildable, no training data.

where does PAT chunk rank in a top-30 dense retrieval? It ranks 26/30 (0.539) — in the net but near-bottom, below the 0.6 threshold. Too deep for reranking to be reliable → hybrid (BM25) or HyDE is the fix, since both lift it in the initial ranking rather than re-ordering a barely-retrieved net.

What is HyDE?

HyDE = Hypothetical Document Embeddings.

The problem: a question and an answer are shaped differently. "What was PAT for Q3?" is a short interrogative. "PAT for Q3 was ₹75 crores with a margin of 7.7%" is a declarative statement full of specifics. The embedding model reads these as different kinds of text and places them far apart — even though one answers the other.

HyDE's trick: don't search with the question. Instead:

1. Ask a cheap LLM to write a fake, made-up answer to the question — e.g. "Company A's PAT for Q3 FY26 was approximately 70 crores, with a margin of around 8%." The numbers don't have to be correct.
2. Embed that fake answer (not the question).
3. Search with the fake answer's vector.

The fake answer is shaped like an answer, so it embeds close to the real answer chunk. The hallucinated numbers are thrown away — the real chunk it retrieves is what you use. Tradeoff: one extra LLM call per query, versus hybrid/BM25 which is pure math and instant.

Implementing hybrid search

The problem: I search my documents by meaning. Ask "what was PAT?" and it should find the sentence with the PAT number. But it didn't — the computer thought a question and its answer looked too "different" to match, even though the answer was right there.

Why: meaning-search is fuzzy. Great at "these are about the same topic," bad at "these both contain the exact word PAT." The number sentence got lost.

The fix — two searchers instead of one:

• Searcher A (what I had): matches by meaning. Good at topics, bad at exact words.
• Searcher B (BM25): matches by exact words. Dumb but reliable — just checks "does this chunk literally contain 'PAT'?"

Run both, combine the answers. That combo is hybrid search.

The twist: combining them equally didn't work — the meaning-searcher was so confidently wrong about the PAT chunk that it dragged the right answer down, even though the word-searcher had it at #1. Fix: tell the combiner "trust the word-searcher more." That did it.

Setup	recall@1	recall@3	recall@5	PAT
meaning-only (dense)	0.75	0.75	0.75	❌ miss
hybrid (meaning + words)	0.75	1.00	1.00	✅ found

PAT went from "not found" to "found." recall@3 jumped 0.75 → 1.00 — all four questions now find their answer.

After adding the generation layer (llama3:8b)

npx tsx answer.ts "What was PAT for Q3 FY26?"

Retrieved 5 chunks:
[1] CompanyA_Q3_FY26#11
[2] CompanyA_Q3_FY26#16
[3] CompanyA_Q3_FY26#15
[4] CompanyA_Q3_FY26#0
[5] CompanyA_Q3_FY26#12

Q: What was PAT for Q3 FY26?
A: The answer can be found in [1]:
   "While our standalone PAT for Q3 FY 26 was INR75 crores with a PAT margin of 7.7%."
   Therefore, the answer is: INR75 crores

npx tsx answer.ts "What was PAT for Q3 FY26 of Company B?"

Retrieved 5 chunks:
[1] CompanyA_Q3_FY26#11
[2] CompanyA_Q3_FY26#16
[3] CompanyA_Q3_FY26#12
[4] CompanyA_Q3_FY26#34
[5] CompanyA_Q3_FY26#15

Q: What was PAT for Q3 FY26 of Company B?
A: Not found in the provided documents. The context only mentions PAT figures
   for Company A Limited, but not for "Company B".

"not found" is the LLM prompt, not a RAG filter

The refusal came from the grounding instruction, not from retrieval (retrieval still returned Company A chunks). The prompt:

You are a financial analyst assistant. Answer the question using ONLY the context below.
If the answer is not in the context, say "not found in the provided documents."
Do not use outside knowledge. Be concise and cite the figure.

two-quarter problem (known limitation)

"Q1 FY27 outlook given in Q4 FY26" → parseQuarterFY grabs the first quarter (Q1 FY27) → filters to an empty set (no FY27 chunks) → 0 candidates → "not found," even though the answer is in Q4 FY26. The metadata filter that fixed quarter-blindness breaks on questions referencing two quarters. Real fix needs query-intent parsing (which quarter is the filter vs the target) — out of scope for now.