Fine-tuning teaches the shape, retrieval supplies the facts

Everyone building RAG right now is tuning their retriever — better reranking, deeper retrieval, smarter chunking. I ran six experiments to see how much that helps once retrieval is already good.

The short version: once the relevant passage was consistently reaching the context window, the retrieval-side changes I tried stopped moving answer quality — and a controlled test pointed at the reader, not the retriever, as the limiting factor. Here's how I got there, with numbers I can defend and the failures reported alongside the wins.

The setup: three systems, one model

I used Yu-Gi-Oh as the test domain on purpose: thousands of cards, intricate timing rules, dense proper nouns, and a base model that's weak on it out of the box (closed-book, the base model scores just 1.83/5 on correctness and 0.18/2 on groundedness — see below). That weakness is a feature — it lets the three systems separate instead of all scoring the same.

All three systems are the same google/gemma-2-2b-it. Only one lever changes at each step:

• System A — base. The model as shipped, closed-book. Adapter off.
• System B — fine-tune. A QLoRA adapter (rank 16, α 32, 4-bit NF4, all linear modules) trained on 2,683 teacher-distilled, judge-verified Yu-Gi-Oh Q&A pairs, early-stopped at the best validation checkpoint (~1 epoch; 3 epochs overfit). Still closed-book. (Validation perplexity was 3.87. That measures next-token fit on held-out text, not factual correctness or grounding — it's a training-health number, not evidence about answer quality, and I don't use it as such.)
• System C — fine-tune + retrieval. System B, but a hybrid retriever prepends real passages before it answers.

Keeping it one model with an adapter toggle matters: any score difference comes from the lever I moved, not from a different model underneath.

A three-layer way to read the results

Before the numbers, a framing that makes the rest precise. "RAG" is usually drawn as retrieval → answer, but there are three distinct layers, and they can fail independently:

1. Retrieval — does the relevant evidence make it into the retrieved set at all? (Measured by recall@k.)
2. Evidence utility — does the retrieved context actually contain sufficient, usable evidence for this question — not buried among distractors, not split across passages, not one fact among several competing ones?
3. Reader — can the model extract the right clause, reason over it, and produce the correct answer?

High recall only speaks to Layer 1. It says nothing about whether Layer 2 is sufficient or whether Layer 3 can use it. Keeping these separate is what lets the results below say something defensible.

Result 1: fine-tuning shifted the shape; retrieval drove the larger correctness gain

All three systems answer the same 60 held-out questions, each scored 0–10 by a blind, reference-grounded judge (rubric details later). The headline, with paired statistics:

System	Mean ± SE	vs. previous	95% CI of difference	p (t / Wilcoxon)
A — base, closed	3.98 ± 0.39	—	—	—
B — fine-tune, closed	5.25 ± 0.54	+1.27	[0.42, 2.17]	0.007 / 0.012
C — fine-tune + retrieval	8.05 ± 0.42	+2.80	[1.72, 3.87]	<0.001 / <0.001

Both step-ups are statistically significant on paired tests over the same 60 questions (these were the two comparisons I set out to make; treat them as strong exploratory evidence rather than a confirmatory benchmark).

The more interesting signal is where each jump comes from. Breaking the score into its four rubric components:

System	Correctness /5	Completeness /2	Groundedness /2	Clarity /1
A	1.83	0.97	0.18	1.00
B	2.35	1.03	0.87	1.00
C	3.85	1.55	1.65	1.00

Fine-tuning (A → B) moved groundedness the most (0.18 → 0.87) — the model learned to sound like a rulings answer and stopped confidently inventing card text. It also nudged correctness up (1.83 → 2.35). So fine-tuning did encode some facts — I'm not claiming it can't. But the larger correctness gain came from retrieval (B → C: 2.35 → 3.85), where real passages entered the prompt.

That's the first finding, stated carefully: in this setup, fine-tuning's main contribution was answer shape and grounding behavior, while retrieval contributed the larger factual-correctness gain. "Fine-tuning teaches the shape, retrieval supplies the facts" is a useful mnemonic for that pattern — but it's a metaphor for a difference in magnitude here, not a literal claim that fine-tuning cannot store facts.

Result 2: once retrieval was good, the retrieval-side changes I tried didn't move answer quality

System C averages 8.05/10, not 10 — the study reports 40 of 60 questions already at a perfect 10, so the lost points sit in a short tail. The retriever feeding it is already strong: hybrid dense (all-MiniLM-L6-v2) + BM25 fused with Reciprocal Rank Fusion, top-5 over 42,412 chunks. recall@5 = 0.933; the gold passage is at rank 1 about 72% of the time (recall@1 = 0.72) and in the top-5 for 93%. Recall does vary by source — rulings 0.96, card facts 0.93, archetype/lore lower on a small sub-sample — so "high recall" is an average, not uniform.

So I tried to buy back the tail from the retrieval side. Six experiments:

1. Cross-encoder reranking (retrieve top-20 → rerank → top-5). Result: no help, and slightly worse — 8.05 → 7.55 on the main set; on a balanced 60-question split the drop (8.25 → 7.37) was statistically significant.
2. Was the reranker mis-ranking? Result: no — recall@5 was identical (0.933) and gold stayed at rank 1 either way. The drop tracked the small reader reacting to reshuffled context, not a ranking defect.
3. Deeper retrieval (rerank a wider candidate pool). Result: recall rose (0.93 → 0.95 → 0.97) but the mean score did not move.
4. Chunk-repair (a card's effect text was split across 1000-char chunks; I reconstructed it with adjacent-chunk expansion). Result: it fixed the retrieval defect — the stranded clause was back in context — but the answer stayed incomplete; the reader didn't use the recovered clause. (This defect was rare: ~0.2% of cards.)
5. Failure re-diagnosis by question type. Result: the remaining tail is dominated by yes/no reasoning inversions — the model flips the answer with the correct passage present — rather than by missing facts.
6. Six in-model reader fixes (self-consistency, self-verification, quote-then-answer, and more), tested on two hard cards. Result: the in-model tricks didn't fix it; the one change that did was swapping in a stronger reader on the same context (next section).

The pattern across experiments 1–4: each change I tested improved (or held) a retrieval metric while producing no measurable end-to-end gain on this evaluation — and reranking actually cost a little. That is evidence of diminishing end-to-end returns from retrieval optimization in this high-recall regime, which is a narrower and more defensible statement than "retrieval stops mattering."

Why would more recall not help? Because recall is a Layer-1 metric. The tail failures here look like Layer-2/Layer-3 problems: the evidence is present but the reader mis-reads it (the yes/no inversions), or a recovered clause is ignored. High retrieval recall does not guarantee successful evidence utilization — and that gap is where the remaining points went.

The reader test — what it does and doesn't establish

Experiment 6 is the most informative, because it holds the retrieved context fixed and changes only the reader:

same question → same retrieved context → different reader → different answer.

On the two hardest cards (Blackwing FAM and Endymion), a stronger reader given the identical passages produced the complete, correct answer that the fine-tuned 2.6B reader kept under-delivering, including the exact effect clause it dropped.

Here's the honest scope. This is a controlled, same-context comparison — stronger evidence for a reader limitation than simply observing that RAG helps. But it was run on two cards, and the reference-grounded judge is stochastic run to run, so the study reports the result qualitatively (complete vs under-answered) rather than as fixed per-question scores. So the defensible conclusion is:

This comparison provides evidence that reader capability is a limiting factor for at least some of the remaining failures — not proof that the reader is the single binding constraint across the whole tail.

The failure taxonomy in experiment 5 (a tail dominated by reasoning inversions with the passage present) is consistent with that reading, but it's a diagnosis, not a controlled manipulation. The clean way to settle it is an experiment I did not run — see "The strongest next test."

What this suggests if you're building RAG

Scoped to settings resembling this one (a small reader, a fact-dense domain, already-high recall):

• If recall is already high, verify that added retrieval effort is actually changing answers before investing in it — in this study, reranking/deeper-retrieval/chunk-repair each moved retrieval metrics and not answer quality.
• Recall is necessary, not sufficient. A passage in the context window is not a fact in the answer; check evidence utility and reader behavior, not just recall@k.
• When retrieval is good and quality has plateaued, the higher-leverage move may be the reader — a stronger model, or RAG-aware training (the RAFT / preference-tuned line) — rather than a cleverer retriever.
• This does not say retrieval is unimportant: 5.25 → 8.05 was retrieval. It says retrieval's marginal return fell off once recall was already high here.

And a diagnostic workflow that generalizes better than any single number:

1. Measure retrieval recall (Layer 1)
2. Inspect evidence sufficiency on failures (Layer 2)
3. Measure end-to-end answer quality
4. Test with oracle / gold context (isolates the retrieval ceiling)
5. Compare readers on identical context (isolates the reader ceiling)
6. Decide whether retrieval or reader is the dominant limitation — then spend effort there

How the evaluation was designed to reduce judge bias

Every answer was scored by an LLM judge (gemini-3.1-flash-lite) that is reference-grounded (handed the gold answer plus the verbatim evidence and told to grade against that, not its own knowledge — which is what lets it fairly score a domain it wasn't trained on), blind (never shown which system produced an answer), pointwise (one answer at a time), and run over answers in shuffled order for extra blindness. The rubric sums to 10 (correctness 0–5, completeness 0–2, groundedness 0–2, clarity 0–1) with two guardrails: inventing a card detail forces groundedness to 0, and a correct refusal must outscore a confident wrong answer. Systems are compared paired (same 60 questions) with bootstrap 95% CIs cross-checked by paired t-test and Wilcoxon.

What this design does not yet establish, and I'd add before calling the measurement airtight: the judge temperature wasn't fixed, each answer was judged once (no repeated-judge consistency measured), and judge scores were never calibrated against human ratings. The study already notes the judge is stochastic per question; that variance is real and unquantified. Treat the 60-question means as reliable and individual per-question scores as noisy.

The whole study cost about $3.26 in GPU and API spend.

Limitations

The boundaries of what this experiment tested, and where I would not extend the conclusions:

1. Single specialized domain (Yu-Gi-Oh). Retrieval behavior and reader difficulty may differ substantially in legal, medical, financial, scientific-literature, enterprise-KB, multi-hop, or open-domain settings.
2. 60 held-out questions. Enough for the paired step-ups to reach significance; too small to characterize RAG in general or to pin a universal recall threshold.
3. LLM-as-judge, single pass, temperature unfixed, no human-agreement calibration.
4. Entity overlap not excluded. The held-out set is built from pages held out of training (a document-level split) and was decontaminated against training questions — so it tests generalization to unseen pages/questions. But the same cards/entities can appear across training and test pages, so this is not an entity-disjoint evaluation.
5. The reader finding rests on two cards in a controlled swap, reported qualitatively.
6. Recall ≠ evidence utility — I measured recall, and inferred utility failures from error analysis rather than measuring sufficiency directly.
7. One retriever/reranker configuration (MiniLM + BM25, RRF, one cross-encoder). Other configs may behave differently.
8. No oracle-context experiment was run, so the retrieval ceiling and reader ceiling aren't cleanly separated numerically.
9. Findings are strongest for small readers in a high-recall regime; they may not hold where recall is the actual bottleneck or where multi-hop reasoning dominates.
10. No human evaluation or cross-domain replication — both would be needed to call this more than a case study.

Takeaway

Fine-tuning changed how this model answered and improved grounding, but retrieval produced the larger factual-correctness gain. Once recall was already high, the retrieval-side changes I tested did not improve end-to-end answer quality, and a same-context reader swap on two hard cards pointed at reader capability as a limiting factor for at least some remaining failures.

These findings come from a 60-question evaluation in a single specialized domain with one model and one retriever, so they are best read as a controlled engineering case study — not a universal recall threshold and not proof that readers always dominate retrieval. The transferable lesson is diagnostic: once relevant evidence is consistently reaching the context window, measure whether the reader can actually use it before spending more effort optimizing retrieval.

The strongest next test

The cleanest way to turn the reader story from "suggested" into "demonstrated" is a 2×2 that separates the two ceilings — which I have not run:

	Actual retrieved context	Oracle / gold context
Weak reader (2.6B)	measured (System C)	isolates the retrieval ceiling
Strong reader	isolates residual reader gains	upper bound with perfect evidence

Comparing the columns shows how much quality is lost to imperfect evidence (retrieval ceiling); comparing the rows shows how much is lost to the reader (reader ceiling). Run across all 60 questions with fixed judge settings and repeated judging, it would replace the two-card qualitative result with a quantitative separation — and it's the experiment I'd prioritize before making any stronger claim.

---

Code, the 60-question eval set, the judge, and all six experiments: github.com/Ace-2504/short-answers-broken-rag. Live arena where all three systems answer and a live judge scores them: harman-ygo-slm.vercel.app.