Swapping every model in a RAG

In the comparison articles (part 1 and part 2) I put my own RAG against five other systems. The models were frozen there on purpose: same embedder, same generator everywhere, so that only the framework could make a difference. The 'Out of Scope' section admitted the two gaps that the design left open: one golden set on one topic, and no model variation at all.

This experiment inverts it. The pipeline is now the frozen part, a deliberately boring hand-rolled one, and the models are the only variables: three embedders, five generators, three judges, four topics. I am not looking for a winner here. If the behavior and the numbers survive swapping every model slot, then the conclusions belong to the system rather than to some vendor's checkpoint, and I can replace any model later without the evaluation story collapsing. And if they do not survive it, better to learn that now than after building more on top.

Hypothesis

The first change is the ruler itself. The golden set grew from 10 questions on one topic to 100 across four topics. With 10 questions, one lucky or unlucky answer moved the score by 10 points; with 100, the numbers get stable enough that comparing models means something at all. 20 of the 100 are refusal traps: their answer is verifiably NOT in the corpus, so the only correct answer is "the documents do not cover this".

Then, three questions going in:

• Do my numbers depend on the models I picked? Everything so far was measured with one embedder and one generator. If swapping models changes the story, the numbers describe those models, not my system. And if models do differ, which slot matters more: the embedder or the generator?
• Are the scores themselves model-independent? Every answer is scored by a Claude, a GPT and a Gemini judge, on purpose including the judge from the generator's own vendor. If a judge favors its own family, the scores are part of the problem.
• Does a RAG answer from the corpus or from the model's memory? The refusal traps test exactly this: a model that answers them is answering from memory. Several are bait, sitting right next to things the corpus DOES mention.

Development

One frozen pipeline: 200-word chunks with 20-word overlap, cosine similarity, top-5 to the generator, one shared answer prompt that demands answering only from the provided context. No frameworks, no reranking, and the production database untouched (a lesson from the comparison piece that I keep respecting).

flowchart LR
    Q[question] --> E["embedder<br/>(slot: 3 models)"]
    E --> R["cosine top-5<br/>(frozen)"]
    R --> G["generator<br/>(slot: 5 models)"]
    G --> A[answer + sources]
    A --> J1["judge: Claude<br/>(scores all)"]
    A --> J2["judge: GPT<br/>(scores all)"]
    A --> J3["judge: Gemini<br/>(scores all)"]

Four topics, chosen so that they are not four flavors of the same thing: the SEPA rulebooks from production (10 questions kept verbatim from the comparison golden set), open-source license texts plus the GNU GPL FAQ, two vendors' agent-building guides plus LangGraph docs, and WHO nutrition fact sheets. Nutrition is there deliberately as the topic where a model's pretrained opinions are strongest, so refusing to invent an answer costs the most.

Six pipelines; each one is an edge picking one embedder and one generator, with the same frozen retrieval between the two columns:

flowchart LR
    subgraph EMB[embedders]
        E1["text-embedding-3-small<br/>(production)"]
        E2[Voyage-4]
        E3[Gemini embedding]
    end
    subgraph GEN[generators]
        G1[Claude Sonnet]
        G2[Claude Haiku]
        G3[GPT-5.6 Terra]
        G4[DeepSeek]
        G5[Gemini Flash]
    end
    E1 -->|"p1 (baseline)"| G1
    E1 -->|p2| G2
    E1 -->|p3| G3
    E1 -->|p6| G4
    E2 -->|p4| G1
    E3 -.->|"p5 (whole stack)"| G5

Reading it: p2, p3 and p6 keep p1's embedder and change only the generator; p4 keeps p1's generator and changes only the embedder. So every comparison against the baseline isolates exactly one component (a test enforces this). The one exception is p5, which changes both at once, so it is flagged and never used for single-variable claims.

All three judges get an identical plain-JSON rubric, no vendor-specific structured output, so their scores stay comparable. Each grades two things per answer: factual correctness 0-100 against the reference, and a grounded true/false verdict: is every claim in the answer actually supported by the passages retrieval returned?

Note: the groundedness judge caught me before it caught any model. In its first version the judge received bare chunk texts, while the generator had seen the same passages labeled with rank and source filename. So judges kept marking legitimate "Sources:" citations as unsupported claims: 21 not-grounded verdicts, mostly artifact. After the
fix (the judge now sees exactly what the generator saw) 10 remained, all real. If the judge's input differs from the generator's input, you are measuring the difference between the two prompts, not the model.

Getting three new vendors into the run had its own friction: Voyage rate-limits hard until a payment method exists, and Gemini's OpenAI-compatible embeddings endpoint returns no usage data and no batch indices, which my provider layer had to learn the hard way. Also worth saying out loud, given the thesis of the comparison piece: running this experiment handed my corpus and questions to three vendors that never had them before.

Results

Mean correctness per pipeline (answerable questions only), as three judges see it, listed as Gemini judge / GPT judge / Claude judge, then the measured generation cost per 100 answers and mean latency:

• p1 oai-small + Sonnet: 89.3 / 88.2 / 90.1; $0.58, 2.6s
• p2 oai-small + Haiku: 88.5 / 87.1 / 93.4; $0.22, 2.4s
• p3 oai-small + GPT Terra: 88.1 / 86.5 / 90.2; $0.33, 1.6s
• p4 Voyage + Sonnet: 93.1 / 90.5 / 90.9; $0.59, 2.8s
• p5 all-Gemini: 94.3 / 93.4 / 93.6; $0.13, 2.0s
• p6 oai-small + DeepSeek: 87.4 / 85.7 / 88.6; $0.066, 1.1s

What the matrix actually says:

The corpus decides more than the model does. Of 80 answerable questions, 66 pass on all six pipelines and 3 fail on all six; only 11 depend on model choice at all. The largest spread between pipelines within one topic is 14 points, while the largest spread between topics within one pipeline is 20. Every pipeline finds licenses hard (topic means 74.6 to 81.5) and nutrition easy (93.0 to 99.5), for the same reasons.

The embedder moved scores more than the generator. p4 against p1 is a pure embedder swap with the generator held fixed, and it is worth between 0.8 and 3.8 points depending on the judge; under every judge that beats what the Sonnet-to-Terra generator swap moves. The component that almost never appears in benchmarks turned out to be the biggest single-variable lever in this matrix.

The two shared retrieval failures split cleanly. One question (the WHO saturated-fat limit, which exists verbatim in the corpus) scored 0 on every pipeline using the OpenAI or Voyage embedder and 100 on the Gemini embedder: a genuine embedder-quality difference. The other (the BSD-3-Clause question) went unanswered on all six pipelines, because
license texts never name themselves; the name lives only in the filename, which no chunk-text embedder can see. (16 of its 18 judge cells score it 0; the Claude judge twice credited the refusal as correct, a small judge-behavior story of its own.) No
model purchase fixes a corpus property. The fix for that class is contextual retrieval, measured in
the previous article.

Every model refused every unanswerable question. All six generators, DeepSeek included, refused all 20
questions whose answer is absent from the corpus, bait included, under all three judges. The refusal behavior comes from the prompt boundary, not from the model choice. Meaning, the cheapest generator in this matrix follows "answer only from context" as reliably as the most expensive one.

Right for the wrong reason exists, and groundedness sees it. 16 of 1440 answerable verdicts came back
not-grounded; 10 of those had scored 70 or above. The cleanest case: asked about charge sharing in a standard SCT, two pipelines answered correctly and scored 100, while the passage they retrieved states that principle for SCT Inst, a different scheme. The model knew the right answer from somewhere else. A correctness-only eval reports 100 and hides that the retrieval failed. Notably, zero of these cases happened on the two better embedders.

No consistent self-preference. Each judge does nudge its own family up a little, but all three agree on the ends of the ranking, the all-Gemini stack first and DeepSeek last, and the GPT judge actually scores its own family second-lowest. The one real disagreement: the Claude judge alone ranks Haiku first among generators, above Sonnet itself; the other two judges keep Sonnet ahead. Disagreement like this is a finding, so no number in this article merges scores from different judges.

Note: the whole experiment cost \$1.92 in generation, $0.07 in indexing, and \$9.57 in judging, metered at the config's list prices and re-checked against vendor pages before publishing. Two prices moved under my feet in those ten days: OpenAI started a promo on the judge model, and DeepSeek restructured its whole pricing into peak and off-peak tiers the day before the run, so p6's exact dollars are the least certain of the list (at most of the new
rates it gets cheaper still). At this volume the drift is cents. The durable point: evaluating the answers cost five times more than producing them, which is worth knowing before someone promises to LLM-judge everything in a production budget.

Conclusion

• Swapping the generator changed almost nothing. Six stacks from four vendors fail on the same questions and pass on the same ones; 66 of 80 outcomes are identical everywhere. Haiku costs 37% of Sonnet, DeepSeek 11%, and at those discounts the answers stay in the same band.
• Which makes budget-routing the generator a real option: once your own eval has proven the slot interchangeable, a router like OpenRouter can serve whatever is cheap this month, and the golden set catches any regression. I would not run the experiment itself through a router, since you never know which host actually served the tokens, but production serving after the experiment is exactly where one fits. Two costs to name: one more company sees your questions, and routers carry chat models only, so the embedder stays with its vendor either way.
• The embedder is the slot that still matters: up to 3.8 points between embedders with the generator fixed, and one question only the Gemini embedder solved. Pin it, version it, give it its own regression test.
• The failures no model fixed are corpus properties, like a license text that never names itself. Those get fixed in the index, not in the model catalog.
• All six generators refused all 20 trap questions, so the refusal discipline lives in the prompt and survives model swaps with everything else.
• This is also my answer to vendor benchmark charts. On my 100 questions the ranking depends on which judge grades it and which topic you slice, and the gaps between generators are about the size of the judges' disagreement over them. A leaderboard saying some model is two points better tells you nothing about your corpus. Running your own hundred questions costs a morning and, in my case, $11.56 end to end.
• Still a small ruler: 20 answerable questions per topic, one run per cell, so one question is worth 5 points of a topic mean. Honest rather than optimized; treat the deltas, not the decimals.

Out of Scope

• Prompt sensitivity: one shared answer prompt was a design constant, so nothing here says how much the numbers move when the prompt does.
• Contextual retrieval on the matrix corpora. Production already ships it (previous article); the matrix ran the plain pipeline on purpose, to stay comparable with the framework comparison. Re-running the matrix on contextual indexes is the natural phase 2, and the filename-blindness result predicts what it fixes. Prepending source names to chunk text is the cheap half-step in the same direction.
• Serving through a router (the OpenRouter point above) is argued here, not measured.

Sources

• GitHub repo: comparison/matrix/ (config, runners, golden sets), ADR-0021 (the experiment), ADR-0023 (contextual retrieval)
• Comparing RAGs, part 1 and part 2
• Improving the scores of a RAG (contextual retrieval, the fix for the failure class no embedder solved)
• Anthropic: Contextual Retrieval
• live demo