A RAG system answers a question in two steps: a retriever picks a handful of passages out of the documents, and a language model writes an answer from those passages. When the answer is wrong, one of the two failed, and most evaluations, mine included, only score the answer. This piece adds the missing measurement, whether the passage that holds the answer was among the passages the model received, and re-reads an earlier experiment with it.
The earlier experiment
The RAG in question indexes the two 2025 SEPA credit transfer rulebooks (the rules European banks follow for euro transfers, published by the EPC) in Postgres with pgvector. The text is split into 300-word chunks with a 50-word overlap, 484 chunks in total, and a question is answered from the 5 nearest chunks by cosine similarity. There is a live demo.
In the model matrix article I held that pipeline fixed and swapped only the models. Six pipelines, each one an embedder (the model that turns text into vectors for the search) paired with a generator (the model that writes the answer). Four pipelines share text-embedding-3-small as embedder, one uses voyage-4, one uses gemini-embedding-001; the generators are Claude Sonnet, Claude Haiku, GPT, Gemini Flash and DeepSeek. Each pipeline answered 80 questions across four small corpora (SEPA rules, software licenses, agent-building guides, WHO nutrition sheets), each question with a reference answer written by hand, and a separate LLM judge scored every answer against its reference from 0 to 100, pass at 70. The result: 66 questions passed on all six pipelines, 3 failed on all six, and only 11 depended on which models were used. I summarised it as "the corpus decides more than the model does".
Three readers pushed on that. Edward Izgorodin asked to split those 3 + 11 questions by whether the answer passage reached the top 5 at all, because a judge score cannot tell a retrieval miss from a generator miss, and "whether the passage was in the top 5 is a lookup, not a verdict". Mikhail ran the live demo by hand, got a refusal with zero citations on a question whose answer is in the rulebook, saw two rewordings answer it correctly, and could not tell from the UI which component had failed. kaziava described a negative test that passed after an embedder swap because the trap chunk stopped being retrieved and the model never saw the bait.
All three want the same number at the same point in the pipeline:
flowchart LR
Q["question<br/>(80 in the matrix, each with<br/>a reference answer)"] --> R["retriever<br/>embedder + cosine search<br/>(3 embedders)"]
R --> T["top 5 passages"]
T --> G["generator<br/>(6 pipelines = embedder x LLM)"] --> A["answer"]
A --> J["judge LLM<br/>answer vs reference<br/>score 0-100, pass at 70"]
T -. "new measurement:<br/>answer phrase in the top 5? yes / no" .-> L["lookup<br/>(regex, no model)"]
J --> V["verdict per pipeline"]
L --> VThe judge is a model reading a model; the lookup is a string search. Read together, they say which component failed.
Note: a few terms from the rulebooks appear below. SCT is the standard SEPA Credit Transfer scheme; SCT Inst is the instant version; the PSMB is the EPC's Payment Scheme Management Board, whose meeting rules are an annex in both rulebooks. None of the findings depend on knowing more than that.
Hypothesis
- Most of the 14 non-unanimous matrix questions would turn out to be retrieval misses. "The corpus decides" reads most naturally as "the passage was not there".
- Mikhail's refused question would be a generator failure: the passage in the top 5, the model too strict. Two rewordings answered it, and rewordings should not move a 484-chunk retriever much.
- Duplicated text between the two rulebooks would be a curiosity, a few pages of boilerplate.
The first held for about half the questions, the second turned out wrong, and the third was wrong by a wider margin than I would have guessed.
Development
The lookup
The matrix has six pipelines but only three retrievers, since retrieval depends on the embedder alone: the four pipelines on text-embedding-3-small receive the same five passages for a given question. So the 14 non-unanimous questions are 14 x 3 = 42 retrieval cells, and each cell either contains the answer passage or does not.
"Contains" is a regex: a phrase from the reference answer that the source text must hold, one per question, checked against the five retrieved passages. No model. Two anchors were wrong on the first pass and I corrected them by reading the passages: the charging rule is written as "shared principle" in the rulebook, not SHARE as in my reference answer, and "patent license" matched the GPL's patent clause when I was looking for Apache's. The anchor list is in the repo with the script.
The judge scores are the ones from the matrix, gpt as judge, pass bar 70.
Mikhail's runs
I read the stored production vectors and ran his exact questions through the same cosine search the live demo uses, k = 5, plus two variants: the same ranking with byte-identical chunks collapsed to one, and the hybrid path (vector plus Postgres full-text search, english configuration, the two rankings fused by reciprocal rank fusion). For each query I recorded the rank of the chunk that holds the answer, over all 484 chunks and not just the top 5.
Note: everything here reads the production database through a read-only connection and writes nothing to it. The live endpoint is plain vector retrieval, k = 5; dedup and hybrid are what it would do, not what it does.
Results
The 14 questions, by retrieval
Retrieved = rank of the first passage in the top 5 that holds the anchor phrase, per embedder. Scores are the judge's, pipelines p1 to p6.
| question | oai / voyage / gem | scores p1..p6 | what decided it |
|---|---|---|---|
| lic-bsd3-third-clause | none / none / none | 0,0,0,0,0,0 | retrieval, every embedder |
| sct-return-deadline | none / none / none | 0,0,0,0,0,0 | retrieval, every embedder |
| lic-mpl-larger-work | none / 2 / 4 | 0,0,0,0,0,0 | half retrieval: the definition was there, the permission clause was not |
| ag-langgraph-send | none / 1 / 1 | 0,0,0,100,100,0 | embedder |
| lic-apache-patent-retaliation | none / 3 / 4 | 0,0,0,100,100,0 | embedder |
| nut-saturated-fat-limit | none / none / 1 | 0,0,0,0,100,0 | embedder |
| sct-max-execution-time | none / none / 5 | 0,0,0,0,100,0 | embedder, plus a chunk cut before the number |
| lic-apache-patent | none / none / none | 0,0,100,0,0,65 | retrieval miss on every embedder, and one pass anyway |
| ag-aci | 1 / 1 / 1 | 100,100,93,100,100,55 | generator |
| ag-openai-guardrails | 1 / 1 / 2 | 86,71,50,71,100,72 | generator |
| ag-openai-when-agent | 4 / 3 / 1 | 98,65,95,100,100,0 | generator |
| lic-gpl-installation-info | 1 / 1 / 1 | 70,65,78,65,100,45 | generator |
| sct-charging-principle | 1 / 1 / 2 | 100,100,0,95,100,100 | generator, confused by a duplicate |
| sct-inquiry-reasons | 2 / 2 / 1 | 67,65,70,67,75,67 | judge threshold |
Six of the fourteen are decided by retrieval. For four of them the scores follow the embedder exactly: the passage is absent on the embedders that scored 0 and present on the ones that scored 100, with nothing in between. LangGraph's Send API and the Apache retaliation clause are found by voyage-4 and gemini-embedding-001 and not by the production embedder; the WHO saturated fat limit and the SCT execution time only by Gemini. Two more were never retrieved by anyone, so no generator could have answered them, and one of those is the BSD-3 question, where the license name exists only in the filename, which no embedder sees.
Five are the generator's, with the passage at rank 1 or 2 on every embedder. DeepSeek refused the OpenAI workflow question with the passage at rank 4 and the judge said so: "incorrectly declines to answer, since the context explicitly identifies workflows involving complex decisions, unstructured data, and brittle rule-based systems". The inquiry-reasons question has all six scores within 5 points of the pass bar, which is not a model difference, it is where I drew the line.
The Apache patent grant is the row Edward predicted and kaziava had already lived through. The grant clause was not in the top 5 on any embedder. Four generators refused, correctly. gpt answered "Yes. The Apache-2.0 license includes a patent grant, though the provided excerpt does not contain its specific terms." and the judge gave it 100, flagged grounded=false, because the yes/no was right. That is a pass for the wrong reason, a fact retrieved from the model's memory rather than the corpus, and the pass rate hides it completely where the lookup shows it in one column.
The charging-principle row is the one that sent me to the duplicate count. gpt scored 0 with the passage at rank 1 because it read "The available charging principle applies to SCT Inst only". The chunk it received was the SCT Inst rulebook's copy of the charging section; the SCT rulebook has the same paragraph, word for word, and it was not in the top 5 because its twin had taken the slot.
Mikhail's question
His original question, "Within how many calendar days can a PSMB member request a telephone meeting?", was refused twice on the live demo with zero citations. Against the stored vectors the passage that holds the rule is at rank 7. The top 5 were the general PSMB meeting pages, and 4 of those 5 slots were two duplicate pairs, so the generator saw three distinct passages and none of them had the number. A retrieval miss that the prompt turned into a refusal, exactly as he guessed from the outside.
flowchart LR
Q["'Within how many calendar days can a<br/>PSMB member request a telephone meeting?'"] --> R["vector search, k = 5"]
R --> T["top 5: general PSMB meeting pages<br/>(two duplicate pairs + one)<br/>rule passage at rank 7"]
T --> G["generator: no number in context"] --> A["'the sources do not contain information'"] --> U["UI: Evidence 0 passages<br/>(retrieved passages not shown)"]
R -. "collapse duplicates" .-> D["rule passage at rank 4"]
R -. "add full-text leg, fuse" .-> H["rule passage at rank 3"]
R -. "reword with 'written procedure'" .-> W["rule passage at rank 1"]His two rewordings, which add "instead of the written vote" or "after receiving a written communication with a proposed decision", put the same passage at rank 1. The words that carry the section are "written procedure"; his original question does not contain them, and the rulebook does not contain "telephone meeting" as a phrase either (it says "PSMB meeting by telephone"). The rule is the last sentence of a 300-word chunk about written votes, so the chunk as a whole is about written votes.
| query | vector, rank of the rule | duplicates collapsed | hybrid |
|---|---|---|---|
| original (telephone meeting, calendar days) | 7 | 4 | 3 |
| "... instead of the written vote?" | 1 | 1 | 1 |
| "... after receiving a written communication with a proposed decision?" | 1 | 1 | 1 |
| "Voting by written procedure" | 1 | 1 | 1 |
| "The communication shall be" (a 4-word quote) | 25 | 25 | 3 |
Either of the two fixes he proposed would have answered his original question on its own. Collapsing exact duplicates before building the context moves the rule from rank 7 to rank 4. The full-text leg alone had it at rank 1, and fusing it with the vector list puts it at rank 3. The 4-word quote is the case full-text exists for: vector rank 25, dedup leaves it there, full-text rank 1.
Note: in the previous piece the same hybrid path cost 2 to 3 questions in 20 on my golden set, in every language, because the lexical leg matched acronyms all over the corpus and the fusion trusted it. That measurement stands. Mikhail's questions are of a different kind, terse and section-specific, and there the lexical leg is the better retriever. The honest position is that the choice depends on the query, which is what he wrote in his comment about his own code-search ablation.
The duplicates
484 chunks in the index. 138 of them, 28.5%, are byte-identical to a chunk from the other rulebook. Both 2025 rulebooks bind the same 36-page "EPC Payment Scheme Management Rules" document as an annex, and my ingestion, which works one PDF at a time, indexed it twice. Every one of the 69 duplicate groups is one copy per rulebook.
flowchart LR
A["sct_rulebook_2025.pdf<br/>(248 chunks)"] -->|"from page 93:<br/>Scheme Management Rules, 36 pages"| X["chunker + embedder,<br/>one PDF at a time"]
B["sct_inst_rulebook_2025.pdf<br/>(236 chunks)"] -->|"from page 95:<br/>the same 36 pages"| X
X --> I["index: 484 chunks,<br/>138 of them in identical pairs (28.5%)"]
I --> T["a top 5 on a PSMB question:<br/>pair, pair, one single<br/>= 3 distinct passages"]This is why the duplicate pair took 2 of 5 slots in Mikhail's "voting by written procedure" query (in my reproduction, 4 of 5, two pairs), why gpt thought the charging rule was SCT Inst only, and why the previous piece's page labels undercounted one question: the label for the charging question names the SCT rulebook page, the retriever returns the identical SCT Inst page at rank 1 for all three embedders, and the eval scored it a miss. By passage the production embedder misses 3 of 20 English questions, not 4.
The fix Mikhail proposed is the right one: collapse identical text before building the context and keep both sources in the citation, since "this rule applies to both schemes" is information a payments engineer wants.
Conclusion
Whether the passage was retrieved is a lookup, and it changes how the matrix reads. Of the 14 questions that were not unanimous across six pipelines, 6 were decided by the embedder before any generator ran and a seventh got only half its passage, 5 by the generator with the passage in hand, 1 by where I put the pass bar, and 1 was a pass with the passage absent, which no judge score could have shown. The sentence "the corpus decides more than the model" survives, with a sharper meaning: it is mostly the retriever that decides, and the retriever is the embedder plus the chunker plus whatever the index contains twice.
Mikhail's refusal was a retrieval miss at rank 7, dressed as a refusal by the prompt. The UI showed him only the passages the generator cited, so he had to guess. Showing retrieved passages next to cited ones costs nothing and would have made his comment a one-liner.
A quarter of my index is duplicated, and I found out from a reader. Exact-duplicate collapsing goes in before the next measurement round.
Retrieval traps deserve their own verdict. kaziava's "test did not run", for a negative test whose trap chunk was not retrieved, is the third state my eval was missing. The Apache patent row is what it looks like when that state is missing: a correct answer from memory, a judge that agrees, and no line in any report.
Out of Scope
- Answer quality with duplicates collapsed, on the full golden set. This piece shows ranks for six queries; the recall@5 and judge numbers with dedup on are the next run.
- A query-type rule for when to fuse the lexical leg. Argued above from six queries, not measured.
- Paraphrase robustness across the 20 golden questions. Mikhail's three phrasings moved one passage from rank 7 to rank 1; how often that happens is its own measurement.
- The UI change. Recorded as a decision, not built.
Sources
- The comments: Edward Izgorodin, Mikhail and kaziava under Swapping every model in a RAG
- kaziava, The Negative Test That Passed for the Wrong Reason
- Swapping every model in a RAG (the matrix: 80 questions, 6 pipelines, 3 judges)
- Asking a RAG in the wrong language (hybrid at a cost of 2 to 3 in 20; the page labels)
- GitHub repo:
comparison/matrix/(pipelines, golden sets, judge scores) andcomparison/multilingual/run.py(the read-only store used for the reproductions) - EPC Payment Scheme Management Rules, EPC207-14 v5.0, annexed to both 2025 rulebooks
- live demo
Top comments (0)