Fine-Tuning and RAG: What a Dozen Failed Experiments Taught Me

The internet has a strong opinion about fine-tuning versus RAG, and most of it comes from people who never ran the experiment.

I ran about a dozen — fine-tuning LLMs, fine-tuning embedders, six flavors of RAG — on a real system that makes forward-looking predictions against noisy financial outcomes, with the kind of statistical rigor that kills your favorite result: walk-forward splits, permutation tests, multiple-comparison correction, pre-registered kill-gates. Most of the experiments died. The autopsies turned out to be far more useful than any hot take, because they all point at the same rule.

Here's that rule up front, so the rest of the post has somewhere to land:

Fine-tuning and RAG both operate on the input side — the vocabulary the model knows, and the context it can see. Neither changes whether the thing you're trying to predict is actually predictable from your data. Where the task is "know the words / find the fact / recall the knowledge," both work. Where the task is "predict a noisy outcome," neither manufactures a signal that isn't there — they just hand you more convincing ways to fool yourself.

Let me show you the bodies.

A note on why my results differ from the blog consensus

Almost every experiment below looked good on a naive train/test split. The failures only showed up under discipline: walk-forward with embargo, permutation tests for significance, Holm/BH-FDR correction because I was testing many cells, an SPY-counterfactual to check I wasn't just capturing market beta, and a binding calibration gate that a result had to clear no matter how good the P&L looked.

That gap is the whole story. If you evaluate fine-tuning or RAG on a single random split and read the headline metric, you will "discover" edges that evaporate the moment you test them honestly. The technique is not the hard part. The evaluation is.

Part 1 — Fine-tuning: four autopsies

Bigger model, better loss, worse decisions

I fine-tuned a 7B and a 14B open model on the same ~777 labeled examples for the same task. The 14B finished with a better evaluation loss (0.97 vs 1.01). It also made worse decisions on held-out data: 46.2% win rate versus the 7B's 68.4% — a 22-point gap, p=0.019.

Two lessons, both expensive to learn the hard way:

1. Generative loss does not predict downstream quality. The 14B was better at the thing loss measures (reproducing tokens) and worse at the thing I cared about (being right). If you pick your model by eval_loss, you will ship the worse one with confidence.
2. Small data punishes big models. With a few hundred to a few thousand examples, extra capacity doesn't find deeper structure — it memorizes the label generator, including its noise and mistakes. A smaller model's limited capacity is implicit regularization; it's forced to learn the simple, general pattern. Under ~3k examples, reach for the smaller model.

Fine-tuning a classifier learns the safe answer

Three separate attempts to fine-tune a model to make a directional call, three times the same outcome: the model converged to the safe, majority answer. It didn't learn a signal — it learned the distribution of the labels, which for a noisy target means "predict the common class and stop taking risks."

This is what fine-tuning does when there's no signal to find: it fits the label distribution beautifully and tells you nothing. The confident, low-loss model is not evidence the edge exists.

The tiny-transformer trap

I tokenized order-flow into sequences and trained a ~130k-parameter transformer on a direction target. It never converged — training loss sat exactly at the -log(0.5) coin-flip floor. Meanwhile a boring gradient-boosted model, on the same target with 12 hand-built features, reached AUC 0.71.

The signal was real and learnable; the architecture just couldn't touch it at that scale. "Use a transformer" is not a substitute for either signal or scale — a 130k-param model has nowhere near the capacity to discover from raw tokens what a GBM reads off 12 good features. Match the architecture to the data you actually have, not the one on the paper you're copying.

Fine-tuning an embedder: it depends entirely on your vocabulary

This one has a happy ending, and it's the most useful result of all because it tells you when fine-tuning pays. I domain-adapted a small embedder (all-MiniLM-L6-v2) on two corpora and measured retrieval quality (MRR, nDCG@10) on held-out queries:

• A "rich" corpus the base model already understood: base MRR 1.00 → fine-tuned MRR 1.00. Δ+0.00. Zero lift — and that's the correct result, because there was no headroom to recover.
• An "opaque-jargon" corpus, where concepts hid behind codenames the base model had never seen: base MRR 0.306 → fine-tuned 0.547. +0.24 MRR, +79% relative.

The conclusion writes itself: fine-tune the embedder only as much as your corpus's vocabulary diverges from what the base model already knows. Well-covered domain? You get nothing. Private jargon, internal codenames, a specialist vocabulary? That's exactly where fine-tuning earns its keep. (Full controlled study, open source, in RE-call.)

The fine-tuning through-line: fine-tuning is a vocabulary and behavior tool. It teaches words, formats, and styles the model didn't have. It does not manufacture predictive signal from a noisy target — and measured by loss instead of the downstream metric, it will actively mislead you.

Part 2 — RAG: the eight-times-falsified idea

"It fires, it just doesn't help"

The seductive idea: retrieve relevant history and inject it into the classifier's prompt so it decides with more context. I tested it eight to nine times — across two models, three embedding stacks, four retrieval targets (own history, sector peers, an outcome-supervised REPLUG-LSR adapter, and niche-conditional slicing), and three paradigms (similarity, outcome-supervised distillation, structured-feature extraction).

Every single time, the sanity check passed: RAG fired, changing 25–33% of the model's decisions. And every single time, those changed decisions were statistically indistinguishable from noise against the real forward outcome. ΔWR hovered around zero with confidence intervals straddling it, through round after round, under multiple-comparison correction.

The mechanism of failure is worth stating precisely: RAG amplified the score's magnitude without knowing when the retrieved past was relevant. Positive historical context pushed the model more bullish — even when the current situation had reversed. It added confidence, not correctness. "It fires" and "it helps" are completely different claims, and only the first one was ever true.

The apparent-alpha trap

One RAG variant looked like a winner. A structured-feature setup posted +11%/year of apparent alpha — the kind of number that gets a strategy shipped. Its ranking AUC was 0.486. Worse than a coin.

The +11% was regime exploitation, not skill. The model predicted the majority class more often, and the evaluation window happened to reward that class. A ranker with AUC below 0.5 has, by definition, no ability to tell winners from losers — so any P&L it produces is a property of the period, not the model.

This is the most dangerous failure in the whole post, because the metric that flatters you (P&L) and the metric that tells the truth (AUC/calibration) point in opposite directions. Make calibration a binding gate, not a footnote. A profitable backtest on a below-chance model is a mirage, and it will happily survive right up until it's live with real money.

Where RAG actually earned its place

Here's the part the "RAG is dead" crowd gets wrong: RAG didn't fail everywhere. It failed at prediction. It stayed on, permanently, for everything retrieval-shaped — the searchable knowledge base, pulling relevant context a human reads, giving the system access to facts it wasn't trained on.

That's the exact line the rest of my writing lives on. RAG to find code by meaning (semantic code search) works beautifully. RAG to give an agent a working memory (RE-call) works beautifully. RAG to make a hard forward-looking call better failed nine times. Same technique, opposite outcomes — because they're different problems wearing the same acronym.

The RAG through-line: RAG changes what the model sees, not whether the target is predictable. If the signal is in the data, retrieval helps you surface it. If it isn't, retrieval just gives the model more confident-looking ways to be wrong.

Part 3 — The rule that explains all of it

Line the failures up and they're the same shape:

• Fine-tuning operates on the model's vocabulary and behavior.
• RAG operates on the model's available context.
• Neither operates on the only question that decides a prediction task: is the target actually a function of the inputs?

So the decision isn't "fine-tune vs RAG." It's what kind of problem do you have?

• "The model doesn't know my words / format / domain." → Fine-tune. The embedder if it's a retrieval problem (and only if your vocabulary genuinely diverges — see the RE-call result); the LLM if it's a behavior or style problem. Measure on the downstream metric, never on loss.
• "The model can't access facts / code / memory it wasn't trained on." → RAG. This is retrieval's home turf and it's excellent at it.
• "The model should predict my noisy outcome better." → Neither will save you. No amount of fine-tuning or retrieval creates signal that isn't in the data — and both will hand you a convincing way to believe otherwise (a lower loss, a fatter backtest, a RAG that "fires"). Go find real signal, or accept there isn't any and stop paying to re-discover that.

And the meta-lesson, the one that made every result above trustworthy: nearly all of them looked good until the evaluation got honest. The bigger model won on loss. The dead RAG variant won on P&L. The naive splits won on the headline number. Walk-forward, permutation, multiple-comparison correction, and a binding calibration gate are what turned a pile of exciting-but-fake wins into a dozen reliable *no*s and two real *yes*es.

The techniques are easy. Knowing whether they worked is the entire job.

---

The retrieval side of all this — how to build RAG that actually works, and the full controlled study on when fine-tuning an embedder pays — is open source: RE-call. The applied side — RAG for code search and agent memory in day-to-day Claude Code — is my series Claude Code, Beyond the Prompt. If you're about to fine-tune or bolt on RAG, run the honest evaluation first — it's cheaper than the mirage.