Here's the debugging loop I see constantly, and ran myself for 1 month.
Answer is wrong. Adjust the prompt. Slightly better. Swap the model. Different wrong. Add "only use the provided context" in bold. Still wrong. Conclude the model isn't good enough.
Every one of those changes is downstream of the actual failure.
If the right chunk never made it into the context window, no prompt fixes that. You are tuning the generator to compensate for a retriever nobody has ever scored.
Two components, two failure modes, and almost everyone debugs the wrong one — because the generator is the part you can see.
Split the system before you debug it
query → [RETRIEVER] → chunks → [GENERATOR] → answer
Ask one question and the whole thing decomposes:
Was the answer present in the retrieved chunks?
No → retrieval failure. The generator was never in a position to succeed. Prompting is irrelevant.
Yes, but the answer is still wrong → generation failure. Now prompting matters.
You cannot answer that question without labels, and getting them is less work than you think.
Twenty queries and an afternoon
You don't need a benchmark. You need twenty representative queries with hand-labelled relevant documents. Two hours, once.
python
eval_set.py — the highest-leverage file in your RAG project
EVAL = [
{
"q": "what's our refund window for annual plans",
"relevant": {"billing/refunds.md", "policies/annual-terms.md"},
},
{
"q": "how do we onboard a new enterprise client",
"relevant": {"process/enterprise-onboarding.md"},
},
# ... 18 more, drawn from real queries people actually asked
]
Draw these from your logs, not your imagination. Real queries are shaped differently from the ones you invent — shorter, more ambiguous, full of internal shorthand.
Now score:
python
def recall_at_k(retrieved, relevant, k):
got = set(retrieved[:k])
return len(got & relevant) / len(relevant) if relevant else 1.0
def precision_at_k(retrieved, relevant, k):
got = retrieved[:k]
return sum(1 for d in got if d in relevant) / k
def mrr(retrieved, relevant):
for i, d in enumerate(retrieved, 1):
if d in relevant:
return 1 / i
return 0.0
def evaluate(retriever, eval_set, k=8):
rs, ps, ms = [], [], []
for case in eval_set:
got = retriever(case["q"], k=k)
rs.append(recall_at_k(got, case["relevant"], k))
ps.append(precision_at_k(got, case["relevant"], k))
ms.append(mrr(got, case["relevant"]))
n = len(eval_set)
return {"recall@k": sum(rs)/n, "precision@k": sum(ps)/n, "mrr": sum(ms)/n}
How to read the three numbers — this is the part that actually tells you what to fix:
Symptom Diagnosis Fix
Low recall The right chunk isn't being found at all Chunking, or hybrid search — see below
High recall, low precision Found it, but buried in noise Rerank, or shrink the corpus
Decent precision, low MRR Right chunks present but ranked low Rerank
All three fine, answers still bad Genuine generation problem Now go tune the prompt
That last row is the point of the whole exercise. Most people start there and never leave.
Precision is a corpus problem before it's a model problem
The instinct when precision is bad is to reach for a better embedding model. Try this first, because it's free and usually larger:
Cut the corpus.
The model reads k chunks — eight, ten, twenty out of however many thousand you have. Every irrelevant document is another candidate for a fixed number of slots. Doubling the corpus without doubling relevant material strictly degrades precision.
I cut mine from 15 to 30 documents and precision went 4. That was a bigger improvement than any retrieval change I made, and it took an afternoon of reading rather than a week of engineering.
The four fixes, in the order I'd try them
- Chunk on structure, not character count
Fixed-size chunking splits mid-sentence, mid-table, mid-thought. The embedding then represents a fragment that means nothing on its own.
python
def chunk_by_heading(md: str, max_chars=1200):
"""Split on markdown headings; only sub-split if a section is huge."""
sections, cur = [], []
for line in md.splitlines():
if line.startswith("#") and cur:
sections.append("\n".join(cur)); cur = [line]
else:
cur.append(line)
if cur:
sections.append("\n".join(cur))
out = []
for s in sections:
if len(s) <= max_chars:
out.append(s)
else:
paras, buf = s.split("\n\n"), ""
for p in paras:
if len(buf) + len(p) > max_chars and buf:
out.append(buf); buf = p
else:
buf = f"{buf}\n\n{p}" if buf else p
if buf: out.append(buf)
return out
Prepend the document title and heading path to each chunk before embedding. A chunk that begins "the window is 30 days" is useless in isolation; Billing > Refunds > Annual plans — the window is 30 days is retrievable. This one change is usually the single biggest recall win available, and it costs nothing.
- Hybrid search
Dense embeddings are bad at exact tokens — error codes, SKUs, surnames, internal project names. Keyword search is bad at paraphrase. You need both.
python
def hybrid(query, k=8, alpha=0.6):
dense = {d: s for d, s in vector_search(query, k=k*3)}
sparse = {d: s for d, s in bm25_search(query, k=k*3)}
def norm(scores):
if not scores: return {}
lo, hi = min(scores.values()), max(scores.values())
rng = (hi - lo) or 1
return {d: (s - lo) / rng for d, s in scores.items()}
dn, sn = norm(dense), norm(sparse)
merged = {
d: alpha * dn.get(d, 0) + (1 - alpha) * sn.get(d, 0)
for d in set(dn) | set(sn)
}
return sorted(merged, key=merged.get, reverse=True)[:k]
Tune alpha against your eval set rather than by feel. On corpora full of internal jargon I've usually landed lower than expected — the keyword half does more work than you'd assume.
- Rerank
Retrieve 30, rerank, keep 8. A cross-encoder reads query and document together instead of comparing pre-computed vectors, so it's much more accurate and much slower — which is exactly why it belongs on a small candidate set.
Reranking fixes ranking, not recall. If the right chunk isn't in your 30, reranking cannot help. Check recall@30 before you reach for it, or you'll spend a week on the wrong component.
- Filter on metadata, especially freshness
This is the one that gets skipped and it's the one that bites.
Staleness is not a relevance property. A superseded pricing document is more semantically similar to "what's our pricing" than most current documents, because it's precisely about that. Your retriever will surface it enthusiastically and correctly.
python
def retrieve(query, k=8):
candidates = hybrid(query, k=k*4)
fresh = [d for d in candidates if store[d].meta.get("status") == "current"]
return rerank(query, fresh)[:k]
No embedding model detects that a document expired. That has to come from metadata, which has to come from somebody owning the document — which is the governance half of this and honestly the harder part.
Wire it to a diff
Same discipline as any test suite. The value isn't a score, it's did this change make things worse.
python
BASELINE = {"recall@k": 0.71, "precision@k": 0.44, "mrr": 0.62}
def check(new):
regressions = {
m: (BASELINE[m], new[m])
for m in BASELINE
if new[m] < BASELINE[m] - 0.03 # tolerance for noise
}
if regressions:
raise SystemExit(f"retrieval regression: {regressions}")
print("ok", new)
Run it on every chunking change, every embedding swap, every corpus addition. Especially every corpus addition — that's the change everyone assumes is monotonically good, and it's the one most likely to quietly cost you precision.
The 20-line version
[ ] 20 real queries from your logs, hand-labelled
[ ] recall@k, precision@k, MRR — before touching anything
[ ] Low recall → chunk on structure, prepend heading path, add keyword search
[ ] Low precision → cut the corpus before you swap the embedding model
[ ] Low MRR, decent recall → rerank a wide candidate set
[ ] Filter stale docs at retrieval; no model detects expiry
[ ] Diff against baseline on every change, corpus additions included
An afternoon of labelling tells you more than a month of prompt tweaking. And it tells you the one thing prompt tweaking never can: whether the answer was ever in the room.
[YOUR NAME] — I test AI stacks honestly, including the parts that don't work: AiStackGuru.
Top comments (0)