DEV Community: Serhiy Kucherenko

The same question, asked three ways

Serhiy Kucherenko — Fri, 02 Oct 2026 17:10:02 +0000

A RAG system answers a question in two steps. A retriever turns the question into a vector, compares it with the vectors of every passage in the documents, and hands the five closest passages to a language model. The model writes the answer from those five. If the passage that holds the answer is not among the five, the model cannot answer, however good it is.

In the previous piece I measured that second condition directly and found a question whose answer sat at rank 7. The model only ever saw ranks 1 to 5, so it refused. A reader, Mikhail, had found that question by hand and then found that two rewordings of it, both containing the words of the section that holds the rule, put the same passage at rank 1. He asked for a new kind of test question: the answer is in the documents, but the question does not use the section's own words. This piece builds that test and runs it.

The earlier experiment, in one paragraph

The documents are the two SEPA credit transfer rulebooks from the European Payments Council, 484 passages of about 300 words each, stored as vectors from OpenAI's text-embedding-3-small. The test set is 20 questions whose answer page I labelled by reading the PDFs, plus Mikhail's. The live system retrieves the five closest passages by cosine similarity, nothing else. Two variants exist in the code but are not switched on: collapsing byte-identical passages before taking the top five (138 of the 484 passages are exact duplicates, because a 36-page annex is bound into both rulebooks), and a hybrid path that also runs a Postgres full-text search and merges the two rankings.

flowchart LR
    Q["one question<br/>3 phrasings"] --> E["embedder<br/>text-embedding-3-small"]
    E --> R["ranking of all 484 passages<br/>by cosine"]
    R --> M["where is the labelled page?<br/>rank 1 .. 484"]
    Q -. "words only" .-> F["full-text search<br/>Postgres, english"]
    F --> R2["ranking by ts_rank"]
    R2 --> M
    R --> D["drop exact duplicates"] --> M

Note: the measurement is the rank of the labelled page in the full ranking, not a pass or fail from a judge. Rank 5 or better means the model would have seen the passage. Rank 6 or worse means it would not, whatever it then answered.

Hypothesis

Three phrasings per question:

original: the question as it stands in the golden set, written by someone who knows the rulebook, so it shares the section's vocabulary.
plain: the same need as a customer or a new joiner would type it. "Bank" instead of PSP, "instant transfer" instead of SCT Inst, and none of the terms the section is built on. Each question has a list of forbidden terms and the runner refuses to start if the plain phrasing contains one.
terse: three to five words, what a person types into a search box. It may reuse section terms; here the variable is length.

Original	Plain	Terse
How long does the Beneficiary PSP have to send a Return in a standard SCT?	The account I paid to is closed. How many days does the receiving bank have to send the money back to my bank?	closed account money sent back when
Does the SCT scheme itself set a maximum transaction amount?	Is there a cap on how much I can send in one SEPA transfer?	SEPA transfer cap
Within how many calendar days can a PSMB member request a telephone meeting instead of the written vote? (Mikhail's rewording)	Within how many calendar days can a PSMB member request a telephone meeting? (his first question)	PSMB telephone meeting days

What I expected:

H1. The plain wording pushes several answers out of the top five, and the terse wording more. Four questions already sat at rank 4 or 5 with their original wording, so at least those are exposed.
H2. Dropping duplicates helps the plain and terse phrasings more than the originals, because it frees slots that duplicate passages otherwise take.
H3. In the language piece the hybrid path cost 3 questions in 20 on English questions, and I blamed passages matching on a single shared acronym. A rule that only counts a passage when it matches more than one query word should recover most of that.
H4. A rule that switches the lexical leg off when the query language differs from the corpus changes nothing here by construction; every phrasing is English.

The first held. The second and third did not.

Development

Everything runs against the stored production vectors over a read-only connection. Each phrasing is embedded once, compared with all 484 passage vectors, and the rank of the first labelled page is recorded. The full-text leg is the production query: plainto_tsquery with its terms joined by OR, ranked by ts_rank. The hybrid leg merges the top 20 of each list with reciprocal rank fusion, as the code does. Two extra legs implement the H3 rule with a threshold of 2 and of 3 distinct matched query words.

Mikhail's question is the twenty-first row. Its original is his rewording that answered on the live demo, its plain is the question he actually asked first and was refused.

Note: I wrote the plain and terse phrasings before the run and read them back against the labelled pages. Golden sets are the ground truth here, so changing one after seeing the ranks is not allowed; these are the phrasings the run used.

Results

Phrasing alone, vector retrieval

Phrasing	Answer in top 5	of 21	Mean reciprocal rank
original	0.81	17	0.61
plain	0.67	14	0.38
terse	0.43	9	0.27

The plain wording loses 3 of the 17 hits and gains none: the weekend-availability question falls from rank 2 to 46, the value-limits question from 4 to 8, and Mikhail's from 1 to 7. The terse wording loses 9 and gains 1. Eight of the 21 questions stay in the top five under all three phrasings, three are outside it under all three, and the remaining ten depend on the wording.

The plain wording is not uniformly worse, it is different. It moves the standard execution-time question from rank 5 to 2, the recall-deadlines question from 4 to 1, and the charging-principle question, a miss at 38 with the original wording, to 15. The embedder does not prefer the rulebook's vocabulary; it prefers whatever the passage happens to say, and the passage on Recall deadlines happens to say "Duplicate", "Fraudulent" and "sent".

The weekend question is the clearest loss. "Is the SCT Inst scheme available on weekends and public holidays?" finds the page that says "available 24 hours a day and on all Calendar Days of the year" at rank 2. "Can I send an instant transfer on a Sunday or on a bank holiday?" finds it at rank 46. The passage never says Sunday, weekend or holiday. It says "Calendar Days". A reader knows those are the same thing; the embedder puts them 44 places apart.

Dropping duplicates, on the full set

Phrasing	vector	duplicates dropped
original	17	17
plain	14	15
terse	9	10

One question flips, and it is Mikhail's, from rank 7 to 4 under both his first wording and the terse one. No other rank changes on the 21 originals, and every other move on the plain and terse phrasings is one to three places, all of them outside the top five already. The duplicate annex is 28.5% of the index, but it only crowds the top five when the question is about the annex, and none of the 20 golden questions is. H2 was wrong: dropping duplicates is the right fix for exactly one question in this set.

The hybrid path and the rule I promised

Leg	original	plain	terse
vector	17	14	9
hybrid (production shape)	14	8	11
hybrid, passage must match 2 words	14	8	10
hybrid, passage must match 3 words	14	8	12

On the original questions the hybrid path loses three: Recall deadlines (rank 4 to 7), value limits (4 to 10), Return deadline (2 to 7). They are the same three the language piece lost, so that measurement reproduces. On the plain questions it loses six more, because "bank", "money" and "transfer" match passages all over both rulebooks. On the terse questions it gains two, since a three-word query is a lexical job and the full-text leg does it well.

The minimum-words rule does nothing to the three lost questions. The rows are identical. I had expected the noise to be passages that share one acronym with the question, and a threshold of two would have removed them. That is not what the full-text leg returns for an English question:

flowchart LR
    Q["Does the SCT scheme itself<br/>set a maximum transaction<br/>amount? (6 query words)"] --> FT["full-text top 20"]
    FT --> N1["rank 1: SCT Inst p78<br/>obligations list<br/>matches 6 of 6 words"]
    FT --> N2["ranks 2 to 10<br/>match 5 of 6 words"]
    FT --> N3["ranks 11 to 20<br/>match 4 of 6 words"]
    FT -.-> G["labelled page, SCT p21<br/>Value Limits<br/>rank 93, matches 3 of 6"]
    style G fill:#FBEAE3,stroke:#993C1D

Every passage in the full-text top 20 matches four to six of the six query words. The labelled page matches three and sits at rank 93. The same shape holds for the other two lost questions: the top 20 match four or five words each, the answer matches four and sits at rank 16 and 32. A threshold of two or three removes nothing, and the fusion goes on trusting a list whose top is wrong.

So the loss on English questions is not single-word noise. It is the lexical leg ranking the right passage poorly, because a long question shares many ordinary words ("scheme", "set", "transaction", "amount") with many passages, and the one passage that answers it is not the one that repeats the most of them. The acronym explanation from the language piece was right for foreign-language questions, where the acronyms were the only words that could match, and wrong for English ones. H3 fails, and the rule I promised in the comments does not touch the problem it was meant to fix.

Mikhail's question, all legs

Phrasing	vector	duplicates dropped	full-text alone	hybrid
his rewording (original)	1	1	1	1
his first question (plain)	7	4	1	3
terse	7	4	1	4

This row reproduces the previous piece to the digit. It is also the only row in the set where the full-text leg has the answer at rank 1 under every phrasing: his first question has 9 words and the passage that holds the rule contains 8 of them, "PSMB", "telephone" and "meeting" among them. The lexical leg rescued his question because his question is the kind the lexical leg is good at. It does not follow that it rescues the others, and on this set it does not.

Conclusion

Wording moves retrieval more than any model swap did. A plain rewording of the same need costs 3 answers in 21; a search-box fragment costs 8 net, 9 lost and 1 gained. The model matrix piece found 11 of 80 questions depending on the model choice; here 10 of 21 depend on how the question is put, with the embedder held fixed.
The loss is not "rulebook words good, plain words bad". Plain wording improved three ranks and lost three hits. What decides it is whether the words the user chose happen to appear in the passage: "Sunday" against "Calendar Days" loses, "twice by mistake" against "Duplicate sending" wins.
Dropping exact duplicates is a fix for one question in this set, the one that is about the duplicated annex. It costs nothing and it is still worth doing, but it is not the reason the golden set misses.
The hybrid path helps short queries and hurts long ones, and the minimum-matched-words rule does not change that, because on English questions the full-text leg ranks the right passage poorly rather than letting noise in. That corrects my own explanation from the language piece.
The test type Mikhail asked for is worth keeping: same question, no section words, scored by the rank of the labelled page. It caught the weekend question, which passed every previous test in this series because every previous test used the rulebook's own words.

Out of Scope

Whether the model answers correctly from the plain phrasing when the passage is retrieved. This piece stops at the ranking.
The language-match rule (H4), which needs the Spanish and Russian questions from the language piece. Not run.
A better lexical ranker (BM25 instead of ts_rank, or field weights) as the fix the hybrid path actually needs. Argued from three questions, not built.
More phrasings per question. Three is enough to show the effect and too few to give a per-question stability number.

Sources

The runner and the phrasing set, in the payments-rag repo: comparison/paraphrase/run.py, comparison/paraphrase/golden/sepa-phrasings.yaml.
Page labels: evals/retrieval_golden_set.yaml, verified against the PDFs.
The answer was at rank 7 (the refused question, the duplicate annex).
Asking a RAG in the wrong language (hybrid at a cost of 2 to 3 in 20, the acronym explanation this piece corrects).
Swapping every model in a RAG (11 of 80 questions model-dependent).
EPC SCT and SCT Inst Rulebooks 2025 (the corpus).

The answer was at rank 7

Serhiy Kucherenko — Fri, 25 Sep 2026 07:00:00 +0000

A RAG system answers a question in two steps: a retriever picks a handful of passages out of the documents, and a language model writes an answer from those passages. When the answer is wrong, one of the two failed, and most evaluations, mine included, only score the answer. This piece adds the missing measurement, whether the passage that holds the answer was among the passages the model received, and re-reads an earlier experiment with it.

The earlier experiment

The RAG in question indexes the two 2025 SEPA credit transfer rulebooks (the rules European banks follow for euro transfers, published by the EPC) in Postgres with pgvector. The text is split into 300-word chunks with a 50-word overlap, 484 chunks in total, and a question is answered from the 5 nearest chunks by cosine similarity. There is a live demo.

In the model matrix article I held that pipeline fixed and swapped only the models. Six pipelines, each one an embedder (the model that turns text into vectors for the search) paired with a generator (the model that writes the answer). Four pipelines share text-embedding-3-small as embedder, one uses voyage-4, one uses gemini-embedding-001; the generators are Claude Sonnet, Claude Haiku, GPT, Gemini Flash and DeepSeek. Each pipeline answered 80 questions across four small corpora (SEPA rules, software licenses, agent-building guides, WHO nutrition sheets), each question with a reference answer written by hand, and a separate LLM judge scored every answer against its reference from 0 to 100, pass at 70. The result: 66 questions passed on all six pipelines, 3 failed on all six, and only 11 depended on which models were used. I summarised it as "the corpus decides more than the model does".

Three readers pushed on that. Edward Izgorodin asked to split those 3 + 11 questions by whether the answer passage reached the top 5 at all, because a judge score cannot tell a retrieval miss from a generator miss, and "whether the passage was in the top 5 is a lookup, not a verdict". Mikhail ran the live demo by hand, got a refusal with zero citations on a question whose answer is in the rulebook, saw two rewordings answer it correctly, and could not tell from the UI which component had failed. kaziava described a negative test that passed after an embedder swap because the trap chunk stopped being retrieved and the model never saw the bait.

All three want the same number at the same point in the pipeline:

flowchart LR
    Q["question<br/>(80 in the matrix, each with<br/>a reference answer)"] --> R["retriever<br/>embedder + cosine search<br/>(3 embedders)"]
    R --> T["top 5 passages"]
    T --> G["generator<br/>(6 pipelines = embedder x LLM)"] --> A["answer"]
    A --> J["judge LLM<br/>answer vs reference<br/>score 0-100, pass at 70"]
    T -. "new measurement:<br/>answer phrase in the top 5? yes / no" .-> L["lookup<br/>(regex, no model)"]
    J --> V["verdict per pipeline"]
    L --> V

The judge is a model reading a model; the lookup is a string search. Read together, they say which component failed.

Note: a few terms from the rulebooks appear below. SCT is the standard SEPA Credit Transfer scheme; SCT Inst is the instant version; the PSMB is the EPC's Payment Scheme Management Board, whose meeting rules are an annex in both rulebooks. None of the findings depend on knowing more than that.

Hypothesis

Most of the 14 non-unanimous matrix questions would turn out to be retrieval misses. "The corpus decides" reads most naturally as "the passage was not there".
Mikhail's refused question would be a generator failure: the passage in the top 5, the model too strict. Two rewordings answered it, and rewordings should not move a 484-chunk retriever much.
Duplicated text between the two rulebooks would be a curiosity, a few pages of boilerplate.

The first held for about half the questions, the second turned out wrong, and the third was wrong by a wider margin than I would have guessed.

Development

The lookup

The matrix has six pipelines but only three retrievers, since retrieval depends on the embedder alone: the four pipelines on text-embedding-3-small receive the same five passages for a given question. So the 14 non-unanimous questions are 14 x 3 = 42 retrieval cells, and each cell either contains the answer passage or does not.

"Contains" is a regex: a phrase from the reference answer that the source text must hold, one per question, checked against the five retrieved passages. No model. Two anchors were wrong on the first pass and I corrected them by reading the passages: the charging rule is written as "shared principle" in the rulebook, not SHARE as in my reference answer, and "patent license" matched the GPL's patent clause when I was looking for Apache's. The anchor list is in the repo with the script.

The judge scores are the ones from the matrix, gpt as judge, pass bar 70.

Mikhail's runs

I read the stored production vectors and ran his exact questions through the same cosine search the live demo uses, k = 5, plus two variants: the same ranking with byte-identical chunks collapsed to one, and the hybrid path (vector plus Postgres full-text search, english configuration, the two rankings fused by reciprocal rank fusion). For each query I recorded the rank of the chunk that holds the answer, over all 484 chunks and not just the top 5.

Note: everything here reads the production database through a read-only connection and writes nothing to it. The live endpoint is plain vector retrieval, k = 5; dedup and hybrid are what it would do, not what it does.

Results

The 14 questions, by retrieval

Retrieved = rank of the first passage in the top 5 that holds the anchor phrase, per embedder. Scores are the judge's, pipelines p1 to p6.

question	oai / voyage / gem	scores p1..p6	what decided it
lic-bsd3-third-clause	none / none / none	0,0,0,0,0,0	retrieval, every embedder
sct-return-deadline	none / none / none	0,0,0,0,0,0	retrieval, every embedder
lic-mpl-larger-work	none / 2 / 4	0,0,0,0,0,0	half retrieval: the definition was there, the permission clause was not
ag-langgraph-send	none / 1 / 1	0,0,0,100,100,0	embedder
lic-apache-patent-retaliation	none / 3 / 4	0,0,0,100,100,0	embedder
nut-saturated-fat-limit	none / none / 1	0,0,0,0,100,0	embedder
sct-max-execution-time	none / none / 5	0,0,0,0,100,0	embedder, plus a chunk cut before the number
lic-apache-patent	none / none / none	0,0,100,0,0,65	retrieval miss on every embedder, and one pass anyway
ag-aci	1 / 1 / 1	100,100,93,100,100,55	generator
ag-openai-guardrails	1 / 1 / 2	86,71,50,71,100,72	generator
ag-openai-when-agent	4 / 3 / 1	98,65,95,100,100,0	generator
lic-gpl-installation-info	1 / 1 / 1	70,65,78,65,100,45	generator
sct-charging-principle	1 / 1 / 2	100,100,0,95,100,100	generator, confused by a duplicate
sct-inquiry-reasons	2 / 2 / 1	67,65,70,67,75,67	judge threshold

Six of the fourteen are decided by retrieval. For four of them the scores follow the embedder exactly: the passage is absent on the embedders that scored 0 and present on the ones that scored 100, with nothing in between. LangGraph's Send API and the Apache retaliation clause are found by voyage-4 and gemini-embedding-001 and not by the production embedder; the WHO saturated fat limit and the SCT execution time only by Gemini. Two more were never retrieved by anyone, so no generator could have answered them, and one of those is the BSD-3 question, where the license name exists only in the filename, which no embedder sees.

Five are the generator's, with the passage at rank 1 or 2 on every embedder. DeepSeek refused the OpenAI workflow question with the passage at rank 4 and the judge said so: "incorrectly declines to answer, since the context explicitly identifies workflows involving complex decisions, unstructured data, and brittle rule-based systems". The inquiry-reasons question has all six scores within 5 points of the pass bar, which is not a model difference, it is where I drew the line.

The Apache patent grant is the row Edward predicted and kaziava had already lived through. The grant clause was not in the top 5 on any embedder. Four generators refused, correctly. gpt answered "Yes. The Apache-2.0 license includes a patent grant, though the provided excerpt does not contain its specific terms." and the judge gave it 100, flagged grounded=false, because the yes/no was right. That is a pass for the wrong reason, a fact retrieved from the model's memory rather than the corpus, and the pass rate hides it completely where the lookup shows it in one column.

The charging-principle row is the one that sent me to the duplicate count. gpt scored 0 with the passage at rank 1 because it read "The available charging principle applies to SCT Inst only". The chunk it received was the SCT Inst rulebook's copy of the charging section; the SCT rulebook has the same paragraph, word for word, and it was not in the top 5 because its twin had taken the slot.

Mikhail's question

His original question, "Within how many calendar days can a PSMB member request a telephone meeting?", was refused twice on the live demo with zero citations. Against the stored vectors the passage that holds the rule is at rank 7. The top 5 were the general PSMB meeting pages, and 4 of those 5 slots were two duplicate pairs, so the generator saw three distinct passages and none of them had the number. A retrieval miss that the prompt turned into a refusal, exactly as he guessed from the outside.

flowchart LR
    Q["'Within how many calendar days can a<br/>PSMB member request a telephone meeting?'"] --> R["vector search, k = 5"]
    R --> T["top 5: general PSMB meeting pages<br/>(two duplicate pairs + one)<br/>rule passage at rank 7"]
    T --> G["generator: no number in context"] --> A["'the sources do not contain information'"] --> U["UI: Evidence 0 passages<br/>(retrieved passages not shown)"]
    R -. "collapse duplicates" .-> D["rule passage at rank 4"]
    R -. "add full-text leg, fuse" .-> H["rule passage at rank 3"]
    R -. "reword with 'written procedure'" .-> W["rule passage at rank 1"]

His two rewordings, which add "instead of the written vote" or "after receiving a written communication with a proposed decision", put the same passage at rank 1. The words that carry the section are "written procedure"; his original question does not contain them, and the rulebook does not contain "telephone meeting" as a phrase either (it says "PSMB meeting by telephone"). The rule is the last sentence of a 300-word chunk about written votes, so the chunk as a whole is about written votes.

query	vector, rank of the rule	duplicates collapsed	hybrid
original (telephone meeting, calendar days)	7	4	3
"... instead of the written vote?"	1	1	1
"... after receiving a written communication with a proposed decision?"	1	1	1
"Voting by written procedure"	1	1	1
"The communication shall be" (a 4-word quote)	25	25	3

Either of the two fixes he proposed would have answered his original question on its own. Collapsing exact duplicates before building the context moves the rule from rank 7 to rank 4. The full-text leg alone had it at rank 1, and fusing it with the vector list puts it at rank 3. The 4-word quote is the case full-text exists for: vector rank 25, dedup leaves it there, full-text rank 1.

Note: in the previous piece the same hybrid path cost 2 to 3 questions in 20 on my golden set, in every language, because the lexical leg matched acronyms all over the corpus and the fusion trusted it. That measurement stands. Mikhail's questions are of a different kind, terse and section-specific, and there the lexical leg is the better retriever. The honest position is that the choice depends on the query, which is what he wrote in his comment about his own code-search ablation.

The duplicates

484 chunks in the index. 138 of them, 28.5%, are byte-identical to a chunk from the other rulebook. Both 2025 rulebooks bind the same 36-page "EPC Payment Scheme Management Rules" document as an annex, and my ingestion, which works one PDF at a time, indexed it twice. Every one of the 69 duplicate groups is one copy per rulebook.

flowchart LR
    A["sct_rulebook_2025.pdf<br/>(248 chunks)"] -->|"from page 93:<br/>Scheme Management Rules, 36 pages"| X["chunker + embedder,<br/>one PDF at a time"]
    B["sct_inst_rulebook_2025.pdf<br/>(236 chunks)"] -->|"from page 95:<br/>the same 36 pages"| X
    X --> I["index: 484 chunks,<br/>138 of them in identical pairs (28.5%)"]
    I --> T["a top 5 on a PSMB question:<br/>pair, pair, one single<br/>= 3 distinct passages"]

This is why the duplicate pair took 2 of 5 slots in Mikhail's "voting by written procedure" query (in my reproduction, 4 of 5, two pairs), why gpt thought the charging rule was SCT Inst only, and why the previous piece's page labels undercounted one question: the label for the charging question names the SCT rulebook page, the retriever returns the identical SCT Inst page at rank 1 for all three embedders, and the eval scored it a miss. By passage the production embedder misses 3 of 20 English questions, not 4.

The fix Mikhail proposed is the right one: collapse identical text before building the context and keep both sources in the citation, since "this rule applies to both schemes" is information a payments engineer wants.

Conclusion

Whether the passage was retrieved is a lookup, and it changes how the matrix reads. Of the 14 questions that were not unanimous across six pipelines, 6 were decided by the embedder before any generator ran and a seventh got only half its passage, 5 by the generator with the passage in hand, 1 by where I put the pass bar, and 1 was a pass with the passage absent, which no judge score could have shown. The sentence "the corpus decides more than the model" survives, with a sharper meaning: it is mostly the retriever that decides, and the retriever is the embedder plus the chunker plus whatever the index contains twice.

Mikhail's refusal was a retrieval miss at rank 7, dressed as a refusal by the prompt. The UI showed him only the passages the generator cited, so he had to guess. Showing retrieved passages next to cited ones costs nothing and would have made his comment a one-liner.

A quarter of my index is duplicated, and I found out from a reader. Exact-duplicate collapsing goes in before the next measurement round.

Retrieval traps deserve their own verdict. kaziava's "test did not run", for a negative test whose trap chunk was not retrieved, is the third state my eval was missing. The Apache patent row is what it looks like when that state is missing: a correct answer from memory, a judge that agrees, and no line in any report.

Out of Scope

Answer quality with duplicates collapsed, on the full golden set. This piece shows ranks for six queries; the recall@5 and judge numbers with dedup on are the next run.
A query-type rule for when to fuse the lexical leg. Argued above from six queries, not measured.
Paraphrase robustness across the 20 golden questions. Mikhail's three phrasings moved one passage from rank 7 to rank 1; how often that happens is its own measurement.
The UI change. Recorded as a decision, not built.

Sources

The comments: Edward Izgorodin, Mikhail and kaziava under Swapping every model in a RAG
kaziava, The Negative Test That Passed for the Wrong Reason
Swapping every model in a RAG (the matrix: 80 questions, 6 pipelines, 3 judges)
Asking a RAG in the wrong language (hybrid at a cost of 2 to 3 in 20; the page labels)
GitHub repo: comparison/matrix/ (pipelines, golden sets, judge scores) and comparison/multilingual/run.py (the read-only store used for the reproductions)
EPC Payment Scheme Management Rules, EPC207-14 v5.0, annexed to both 2025 rulebooks
live demo

Asking a RAG in the wrong language

Serhiy Kucherenko — Fri, 18 Sep 2026 12:11:20 +0000

Every article in this series so far assumed that the person asking speaks the language of the documents. The SEPA rulebooks my RAG indexes exist only in English, and so did every question in my golden set. But a payments engineer in Madrid or Kyiv asks in their own language, and I had never measured what that does to retrieval. I can read Spanish and Russian myself, so those two were the cheap place to start.

The question for this piece is narrow: what happens to the retriever when the question changes language and the corpus does not. It stops at whether the right page still lands in the top 5; answer quality and the generator wait for the next piece.

Hypothesis

Three things I expected going in:

Vector retrieval would lose something cross-lingually, since the embedder has to map a Spanish question next to an English paragraph, but not much. Vendors describe these models as multilingual.
Hybrid retrieval (vector plus Postgres full-text search, fused by reciprocal rank fusion) would quietly stop helping, because the full-text index is built with the english configuration. My guess was that a Spanish query would match nothing and the fusion would fall back to the vector ranking alone.
Translating the question to English first, then running the normal English pipeline, would recover most of the loss for the cost of one small LLM call.

The second one turned out wrong in an instructive way.

Development

The corpus is the production index untouched: 484 chunks from the two 2025 EPC rulebooks, with the contextual blurbs from the previous improvement round already in the stored vectors. The eval reads it read-only.

The golden set is the 20 answerable SEPA questions from the model matrix article, each now labelled with the page that answers it, and each translated into Spanish and Russian. Scheme and message names (SCT, SCT Inst, PSP, Recall, Return) stay in English in all three versions, because that is how practitioners actually talk about them. It also turned out to matter for what follows.

Every question, in each of its three languages, goes through four retrieval paths, and each path hands back a top 5 that is scored against the page labels:

Vector only: embed the question, take the 5 nearest chunks. Run once per embedder.
Full-text only: Postgres full-text search, the 5 best lexical matches. Run once per language configuration.
Hybrid: take the top 20 from vector and the top 20 from full-text, fuse the two rankings, keep the top 5. This is the production hybrid path with one change: the language configuration is a variable instead of the hardcoded english.
Translate first: Claude Haiku turns the Spanish or Russian question into English, then path 1 runs with the production embedder.

flowchart LR
    Q["question<br/>(en / es / ru)"] --> V["1. vector only"] --> T5a["top 5"]
    Q --> F["2. full-text only"] --> T5b["top 5"]
    Q --> H["3. hybrid: vector top-20 + full-text top-20,<br/>fused by reciprocal rank"] --> T5c["top 5"]
    Q --> TR["4. translate to English,<br/>then vector only"] --> T5d["top 5"]
    T5a & T5b & T5c & T5d --> S["scored against<br/>the page labels"]

The fusion in path 3 is reciprocal rank fusion, the same one production uses. Each chunk scores 1 / (60 + its rank) in every list it appears in, and the two scores are added. A chunk at rank 1 in one list and absent from the other gets 1/61; a chunk at rank 10 in both gets 2/70, which is more. So the fusion rewards agreement between the two lists and treats both lists as equally trustworthy. That second property is the one this piece is about.

The three embedders are the ones from the matrix piece: text-embedding-3-small (production, its vectors read straight from the database), voyage-4 and gemini-embedding-001. The full-text configurations tried were english, spanish, russian and simple (no stemming at all).

Two metrics. hit@5 against the page labels is the real one; with 20 questions every hit is worth 0.05. The second needs no labels at all: how much of the top 5 for the Spanish or Russian question overlaps with the top 5 for the same question in English. It says whether the retriever is even looking at the same pages.

Note: the production /ask endpoint runs plain vector retrieval. Hybrid exists behind an eval flag, since the round where I measured it landed at exactly the same recall as vector alone. So nothing below describes a live failure; it describes what turning that flag on would do.

Results

hit@5 over 20 questions, production embedder unless stated:

leg	en	es	ru
vector, `text-embedding-3-small`	0.80	0.70	0.65
vector, `gemini-embedding-001`	0.85	0.75	0.70
vector, `voyage-4`	0.70	0.70	0.60
full-text alone, `english` config	0.50	0.35	0.35
hybrid (vector + full-text), `english` config	0.65	0.60	0.55
hybrid, `spanish` config	0.65	0.60	0.60
translate to English, then vector	n/a	0.70	0.75

Top-5 overlap with the English question: 0.72 for Spanish, 0.63 for Russian on the production embedder; 0.81 and 0.73 on the Gemini embedder; 0.82 and 0.76 on Voyage.

The cross-lingual loss is real and small. Spanish costs the production embedder 2 questions in 20, Russian 3. English itself misses 4 of the 20; Spanish keeps all four of those misses and adds two, Russian keeps three of them and adds four, so the language penalty sits mostly on top of what English already gets wrong. Gemini also loses 2 and 3; Voyage starts lower in English and loses 0 and 2. Meaning, the embedder you already have is not the problem here, or at least not the biggest one.

Hybrid costs recall in every language, English included. Against plain vector it drops 0.80 to 0.65 in English, 0.70 to 0.60 in Spanish, 0.65 to 0.55 in Russian. In English it loses 3 hits and gains none; in Spanish it loses 2 and gains none; in Russian it loses 3 and gains 1. The English drop is new since the round where hybrid was at parity: contextual retrieval lifted the vector leg from 0.60 to 0.80 in between, and the full-text leg stayed where it was, so fusing them now dilutes a better ranking with a worse one.

The mechanism is where my second hypothesis broke. I expected a Spanish question to match nothing in an English full-text index. It matched 192 of 484 chunks. The production query rewrites the full-text search from AND to OR (a natural-language question shares only some words with a terse rulebook paragraph, so requiring every word matched nothing), and under OR any shared token is enough. For the question about Recall deadlines the english stemmer turned the Spanish text into this:

'en' | 'qué' | 'plazo' | 'debe' | 'enviars' | 'un' | 'recal' | 'de' | 'sct' | 'por' | 'duplicidad' | 'o' | 'error' | 'técnico' | 'frent' | 'fraud'

recal, sct, error and fraud survive translation, so 192 chunks match, ranked by how often they mention those tokens. The full-text leg returned all 20 of its 20 candidates on every one of the Spanish questions, and on 19 of the 20 Russian ones (one Russian question shared no token at all and matched zero, which is the behaviour I had expected everywhere). The top of that list for the Recall question was the SCT Inst page listing Recall reasons, the change log, and an annex, all of them about Recalls and none of them the page with the deadlines. Reciprocal rank fusion then weights that list equally with the vector list, so twenty plausible but wrong pages get the same say as the twenty the embedder picked.

flowchart LR
    Q["Spanish question<br/>¿En qué plazos debe<br/>enviarse un Recall...?"] --> TS["plainto_tsquery('english'),<br/>AND rewritten to OR<br/>recal | sct | error | fraud | plazo | ..."]
    TS --> FT["192 of 484 chunks match<br/>full-text top-20 by ts_rank:<br/>SCT Inst Recall p.40, change log p.8,<br/>annex p.134, ..."]
    Q --> VEC["vector top-20<br/>deadline page p.31 at rank 3"]
    FT --> RRF["RRF fusion<br/>both lists weighted equally"]
    VEC --> RRF
    RRF --> OUT["hybrid top-5<br/>five SCT Inst Recall pages,<br/>p.31 pushed out"]

For that question the vector leg alone had the deadline page at rank 3. After fusion the top 5 were five pages from the SCT Inst Recall section, and the deadline page was gone.

Switching the full-text configuration to spanish or russian changed nothing for Spanish or Russian questions (0.60 and 0.55 to 0.60, within one question). The documents are English, so a Spanish stemmer has nothing to stem on the index side; the only cross-language matches were the acronyms, and those match under any configuration. The fix is not in the stemmer.

Translating first helped Russian and did nothing for Spanish, which looked odd, since Spanish is the closer language. Claude Haiku turned the 40 questions into English for $0.0071 total, about $0.0002 and 0.7 seconds per query. Russian went from 0.65 to 0.75; Spanish stayed at 0.70 with exactly the same hits. Top-5 overlap with the English question rose to 0.87 and 0.81 for both, so translation did put the retriever back on the same pages even where the count did not move.

The explanation is in the ranks, not in the languages. The two questions that separate the Spanish and Russian translate legs are ones where the original English question itself only just makes it: the answer page sits at rank 4 for one and rank 5 for the other. A translation is a paraphrase, and any paraphrase can move a rank-5 page to rank 6. On the value-limits question the Russian back-translation came out word for word identical to my English original (rank 4, hit); the Spanish one came out as "establish a maximum amount per transaction" (rank 6, miss). On the execution-time question the Spanish back-translation dropped the word "credit" from "credit transfer", and the page went from rank 5 in English to rank 13. Russian also had more to recover: native Spanish already found two pages (settlement certainty, currency) at ranks 1 and 3 that native Russian had at rank 7. Meaning, translation buys back what the language cost, and then the fragile English questions cost it right back.

Note: I also wanted to say there were no false positives, and I cannot. The golden set carries 5 questions whose answer is verifiably not in the corpus (they are about direct debits, card fees, SWIFT messages and TARGET2 hours). I ran them in all three languages through the production answer path: 14 of 15 were refused correctly, in the language of the question. The 15th was the Russian version of the direct-debit refund question, which came back with "13 months" and a citation. Thirteen months is the window for recalling a SEPA Credit Transfer, a different payment instrument entirely; the English and Spanish versions of the same question were refused. One hallucination in fifteen, and it appeared only cross-lingually.

Where the same knob lives on other stacks, for anyone running one of the systems from the comparison with lexical search turned on (the comparison itself ran them vector-only):

Postgres: the text search configuration is the second argument of to_tsvector and plainto_tsquery; the index is built with one of them too, so a per-query change means either a second index or a sequential scan.
LlamaIndex: BM25Retriever takes stemmer=Stemmer.Stemmer("english") and language="english" for the stop-word list, both defaulting to English.
Haystack: the in-memory BM25 lowercases and tokenizes with the regex (?u)\b\w+\b, and that is all. No stemming, no stop words, no language to set, so a Spanish question matches an English corpus on exactly the shared tokens this piece is about.
LangChain: BM25Retriever takes a preprocess_func; the default splits on whitespace, and any stemming or stop-word handling is yours to write.
OpenAI file search: the docs say retrieval is "semantic and keyword search" and expose no language, tokenizer or keyword setting. NotebookLM exposes nothing at all.

In every one of these the lexical leg has a language baked in somewhere, and none of them will tell you when the question stops matching it.

Conclusion

Cross-lingual retrieval with the production embedder loses 2 to 3 questions in 20 against English. Small, consistent across three embedders, and sitting on top of the 4 questions English misses on its own.
Hybrid search is not neutral to language, and not for the reason I guessed. It does not fall back to vector when the query language changes; it fuses in a full list of confident lexical matches on shared acronyms, and with an OR-joined query that list is always full. It cost 2 to 3 questions in every language, English included.
The full-text language configuration is the wrong knob when the documents and the questions disagree. The right knob is fusion: skip or down-weight the lexical leg when the query language is not the corpus language, or require more than one matching term before a chunk counts.
Translating the question first is cheap ($0.0002, 0.7 s) and recovered Russian to within one question of English. Where it did not help, the reason was two English questions that already sit at rank 4 and 5, so a paraphrase pushes them out. Translation is a paraphrase.
The refusal boundary that held for all 20 trap questions in the matrix piece let one through here, in Russian only. Cross-lingual questions deserve their own refusal traps in the golden set.
Twenty questions means every hit is 0.05; I am reporting deltas of 2 and 3 questions and would not defend anything smaller. Honest rather than optimized.

Out of Scope

The generator, beyond the 15 refusal checks above. Whether Claude answers a Spanish question correctly from English passages, and in which language, is the next measurement.
A corpus that is itself not in English. That is a different problem: the chunker and the embedder both see different text, and it is the subject of the next piece. The obvious shortcut, translating the whole corpus into one language before indexing, is a document-translation problem at scale, where every sentence has to stay correct and nobody signs off on it. The tooling that exists for that is quality estimation, models like CometKiwi that score a translation without a reference, and even its authors warn about reading the scores as guarantees. A recent cross-lingual retrieval study (Goworek et al., 2025) found that dense retrievers trained for cross-lingual use "derive little benefit from document translation" and recommends multilingual embeddings over translation pipelines. So the honest position for now: do not translate the corpus, measure the embedder on it.
A fusion rule that is language-aware. Argued above, not implemented or measured.

Sources

GitHub repo: comparison/multilingual/ (runner, golden sets in three languages, page labels)
The illusion of improvement (hybrid at parity with vector, 0.60 = 0.60)
Improving the scores of a RAG (contextual retrieval, 0.60 to 0.80)
Swapping every model in a RAG (the three embedders and the 20 SEPA questions)
Postgres full-text search: text search configurations
LlamaIndex BM25Retriever (stemmer, language), Haystack InMemoryDocumentStore (bm25_tokenization_regex), LangChain BM25Retriever (preprocess_func), OpenAI file search
Goworek, Macmillan-Scott, Özyiğit: What Drives Cross-lingual Ranking? (2025)
CometKiwi and Pitfalls and Outlooks in Using COMET (translation quality estimation)
live demo

Swapping every model in a RAG

Serhiy Kucherenko — Mon, 14 Sep 2026 19:08:51 +0000

In the comparison articles (part 1 and part 2) I put my own RAG against five other systems. The models were frozen there on purpose: same embedder, same generator everywhere, so that only the framework could make a difference. The 'Out of Scope' section admitted the two gaps that the design left open: one golden set on one topic, and no model variation at all.

This experiment inverts it. The pipeline is now the frozen part, a deliberately boring hand-rolled one, and the models are the only variables: three embedders, five generators, three judges, four topics. I am not looking for a winner here. If the behavior and the numbers survive swapping every model slot, then the conclusions belong to the system rather than to some vendor's checkpoint, and I can replace any model later without the evaluation story collapsing. And if they do not survive it, better to learn that now than after building more on top.

Hypothesis

The first change is the ruler itself. The golden set grew from 10 questions on one topic to 100 across four topics. With 10 questions, one lucky or unlucky answer moved the score by 10 points; with 100, the numbers get stable enough that comparing models means something at all. 20 of the 100 are refusal traps: their answer is verifiably NOT in the corpus, so the only correct answer is "the documents do not cover this".

Then, three questions going in:

Do my numbers depend on the models I picked? Everything so far was measured with one embedder and one generator. If swapping models changes the story, the numbers describe those models, not my system. And if models do differ, which slot matters more: the embedder or the generator?
Are the scores themselves model-independent? Every answer is scored by a Claude, a GPT and a Gemini judge, on purpose including the judge from the generator's own vendor. If a judge favors its own family, the scores are part of the problem.
Does a RAG answer from the corpus or from the model's memory? The refusal traps test exactly this: a model that answers them is answering from memory. Several are bait, sitting right next to things the corpus DOES mention.

Development

One frozen pipeline: 200-word chunks with 20-word overlap, cosine similarity, top-5 to the generator, one shared answer prompt that demands answering only from the provided context. No frameworks, no reranking, and the production database untouched (a lesson from the comparison piece that I keep respecting).

flowchart LR
    Q[question] --> E["embedder<br/>(slot: 3 models)"]
    E --> R["cosine top-5<br/>(frozen)"]
    R --> G["generator<br/>(slot: 5 models)"]
    G --> A[answer + sources]
    A --> J1["judge: Claude<br/>(scores all)"]
    A --> J2["judge: GPT<br/>(scores all)"]
    A --> J3["judge: Gemini<br/>(scores all)"]

Four topics, chosen so that they are not four flavors of the same thing: the SEPA rulebooks from production (10 questions kept verbatim from the comparison golden set), open-source license texts plus the GNU GPL FAQ, two vendors' agent-building guides plus LangGraph docs, and WHO nutrition fact sheets. Nutrition is there deliberately as the topic where a model's pretrained opinions are strongest, so refusing to invent an answer costs the most.

Six pipelines; each one is an edge picking one embedder and one generator, with the same frozen retrieval between the two columns:

flowchart LR
    subgraph EMB[embedders]
        E1["text-embedding-3-small<br/>(production)"]
        E2[Voyage-4]
        E3[Gemini embedding]
    end
    subgraph GEN[generators]
        G1[Claude Sonnet]
        G2[Claude Haiku]
        G3[GPT-5.6 Terra]
        G4[DeepSeek]
        G5[Gemini Flash]
    end
    E1 -->|"p1 (baseline)"| G1
    E1 -->|p2| G2
    E1 -->|p3| G3
    E1 -->|p6| G4
    E2 -->|p4| G1
    E3 -.->|"p5 (whole stack)"| G5

Reading it: p2, p3 and p6 keep p1's embedder and change only the generator; p4 keeps p1's generator and changes only the embedder. So every comparison against the baseline isolates exactly one component (a test enforces this). The one exception is p5, which changes both at once, so it is flagged and never used for single-variable claims.

All three judges get an identical plain-JSON rubric, no vendor-specific structured output, so their scores stay comparable. Each grades two things per answer: factual correctness 0-100 against the reference, and a grounded true/false verdict: is every claim in the answer actually supported by the passages retrieval returned?

Note: the groundedness judge caught me before it caught any model. In its first version the judge received bare chunk texts, while the generator had seen the same passages labeled with rank and source filename. So judges kept marking legitimate "Sources:" citations as unsupported claims: 21 not-grounded verdicts, mostly artifact. After the
fix (the judge now sees exactly what the generator saw) 10 remained, all real. If the judge's input differs from the generator's input, you are measuring the difference between the two prompts, not the model.

Getting three new vendors into the run had its own friction: Voyage rate-limits hard until a payment method exists, and Gemini's OpenAI-compatible embeddings endpoint returns no usage data and no batch indices, which my provider layer had to learn the hard way. Also worth saying out loud, given the thesis of the comparison piece: running this experiment handed my corpus and questions to three vendors that never had them before.

Results

Mean correctness per pipeline (answerable questions only), as three judges see it, listed as Gemini judge / GPT judge / Claude judge, then the measured generation cost per 100 answers and mean latency:

p1 oai-small + Sonnet: 89.3 / 88.2 / 90.1; $0.58, 2.6s
p2 oai-small + Haiku: 88.5 / 87.1 / 93.4; $0.22, 2.4s
p3 oai-small + GPT Terra: 88.1 / 86.5 / 90.2; $0.33, 1.6s
p4 Voyage + Sonnet: 93.1 / 90.5 / 90.9; $0.59, 2.8s
p5 all-Gemini: 94.3 / 93.4 / 93.6; $0.13, 2.0s
p6 oai-small + DeepSeek: 87.4 / 85.7 / 88.6; $0.066, 1.1s

What the matrix actually says:

The corpus decides more than the model does. Of 80 answerable questions, 66 pass on all six pipelines and 3 fail on all six; only 11 depend on model choice at all. The largest spread between pipelines within one topic is 14 points, while the largest spread between topics within one pipeline is 20. Every pipeline finds licenses hard (topic means 74.6 to 81.5) and nutrition easy (93.0 to 99.5), for the same reasons.

The embedder moved scores more than the generator. p4 against p1 is a pure embedder swap with the generator held fixed, and it is worth between 0.8 and 3.8 points depending on the judge; under every judge that beats what the Sonnet-to-Terra generator swap moves. The component that almost never appears in benchmarks turned out to be the biggest single-variable lever in this matrix.

The two shared retrieval failures split cleanly. One question (the WHO saturated-fat limit, which exists verbatim in the corpus) scored 0 on every pipeline using the OpenAI or Voyage embedder and 100 on the Gemini embedder: a genuine embedder-quality difference. The other (the BSD-3-Clause question) went unanswered on all six pipelines, because
license texts never name themselves; the name lives only in the filename, which no chunk-text embedder can see. (16 of its 18 judge cells score it 0; the Claude judge twice credited the refusal as correct, a small judge-behavior story of its own.) No
model purchase fixes a corpus property. The fix for that class is contextual retrieval, measured in
the previous article.

Every model refused every unanswerable question. All six generators, DeepSeek included, refused all 20
questions whose answer is absent from the corpus, bait included, under all three judges. The refusal behavior comes from the prompt boundary, not from the model choice. Meaning, the cheapest generator in this matrix follows "answer only from context" as reliably as the most expensive one.

Right for the wrong reason exists, and groundedness sees it. 16 of 1440 answerable verdicts came back
not-grounded; 10 of those had scored 70 or above. The cleanest case: asked about charge sharing in a standard SCT, two pipelines answered correctly and scored 100, while the passage they retrieved states that principle for SCT Inst, a different scheme. The model knew the right answer from somewhere else. A correctness-only eval reports 100 and hides that the retrieval failed. Notably, zero of these cases happened on the two better embedders.

No consistent self-preference. Each judge does nudge its own family up a little, but all three agree on the ends of the ranking, the all-Gemini stack first and DeepSeek last, and the GPT judge actually scores its own family second-lowest. The one real disagreement: the Claude judge alone ranks Haiku first among generators, above Sonnet itself; the other two judges keep Sonnet ahead. Disagreement like this is a finding, so no number in this article merges scores from different judges.

Note: the whole experiment cost \$1.92 in generation, $0.07 in indexing, and \$9.57 in judging, metered at the config's list prices and re-checked against vendor pages before publishing. Two prices moved under my feet in those ten days: OpenAI started a promo on the judge model, and DeepSeek restructured its whole pricing into peak and off-peak tiers the day before the run, so p6's exact dollars are the least certain of the list (at most of the new
rates it gets cheaper still). At this volume the drift is cents. The durable point: evaluating the answers cost five times more than producing them, which is worth knowing before someone promises to LLM-judge everything in a production budget.

Conclusion

Swapping the generator changed almost nothing. Six stacks from four vendors fail on the same questions and pass on the same ones; 66 of 80 outcomes are identical everywhere. Haiku costs 37% of Sonnet, DeepSeek 11%, and at those discounts the answers stay in the same band.
Which makes budget-routing the generator a real option: once your own eval has proven the slot interchangeable, a router like OpenRouter can serve whatever is cheap this month, and the golden set catches any regression. I would not run the experiment itself through a router, since you never know which host actually served the tokens, but production serving after the experiment is exactly where one fits. Two costs to name: one more company sees your questions, and routers carry chat models only, so the embedder stays with its vendor either way.
The embedder is the slot that still matters: up to 3.8 points between embedders with the generator fixed, and one question only the Gemini embedder solved. Pin it, version it, give it its own regression test.
The failures no model fixed are corpus properties, like a license text that never names itself. Those get fixed in the index, not in the model catalog.
All six generators refused all 20 trap questions, so the refusal discipline lives in the prompt and survives model swaps with everything else.
This is also my answer to vendor benchmark charts. On my 100 questions the ranking depends on which judge grades it and which topic you slice, and the gaps between generators are about the size of the judges' disagreement over them. A leaderboard saying some model is two points better tells you nothing about your corpus. Running your own hundred questions costs a morning and, in my case, $11.56 end to end.
Still a small ruler: 20 answerable questions per topic, one run per cell, so one question is worth 5 points of a topic mean. Honest rather than optimized; treat the deltas, not the decimals.

Out of Scope

Prompt sensitivity: one shared answer prompt was a design constant, so nothing here says how much the numbers move when the prompt does.
Contextual retrieval on the matrix corpora. Production already ships it (previous article); the matrix ran the plain pipeline on purpose, to stay comparable with the framework comparison. Re-running the matrix on contextual indexes is the natural phase 2, and the filename-blindness result predicts what it fixes. Prepending source names to chunk text is the cheap half-step in the same direction.
Serving through a router (the OpenRouter point above) is argued here, not measured.

Sources

GitHub repo: comparison/matrix/ (config, runners, golden sets), ADR-0021 (the experiment), ADR-0023 (contextual retrieval)
Comparing RAGs, part 1 and part 2
Improving the scores of a RAG (contextual retrieval, the fix for the failure class no embedder solved)
Anthropic: Contextual Retrieval
live demo

Improving the scores of a RAG

Serhiy Kucherenko — Sat, 12 Sep 2026 11:28:33 +0000

In the previous articles (part 1 and part 2) I compared my own RAG against five other systems. One of its numbers bothered me since then: recall@5 = 0.60. In plain terms, for 4 questions out of 10 the page with the answer was actually retrieved, but ranked below the top 5 that the LLM gets to see. So the answers tended to avoid specifics ("settles instantly, see section 4.2.3" instead of "5 seconds"), and one question failed so badly that the judge scored the answer 0.

The cause was measured earlier, and it is quite illustrative. The same page (p26 of the SCT Inst rulebook, the one that says "5 seconds") ranks differently depending on nothing but the phrasing of the question:

"target maximum execution time", the spec's own words: rank 1
"how fast does an SCT Inst payment settle?", a user's words: rank 9

A user asks casually while the rulebook is written formally, so their embeddings end up far apart. The chunk size makes it worse: a ~300-word chunk embeds as an average of everything in it, meaning a single bare fact competes with the rest of its own chunk.

There are two known ways to close this gap, so I measured both.

Hypothesis

A vocabulary mismatch can be attacked in two places:

At query time: rewrite the casual question into several spec-vocabulary variants and search with all of them. Cheap, no re-indexing needed, and very popular (RAG tutorials call it multi-query or RAG-Fusion).
At index time: give every chunk back the context it lost when it was cut out of its document, so that a bare "5 seconds" embeds as what it actually is. This is Contextual Retrieval as described by Anthropic; they report large reductions in failed retrievals. Costs a one-time re-index.

Note: there is a third popular trick, HyDE, where you embed a hypothetical
answer instead of the question. I rejected it without testing: in a compliance-heavy domain, a hallucinated draft steering the retrieval is exactly the failure mode this project exists to avoid.

Both experiments follow the same rule as before: measure on the golden set first, and the production path changes only if the number justifies it. For reference, reranking passed through the same gate earlier: it lifted recall from 0.60 to 0.70 but costs about a minute per question, so it never shipped.

Development

Fix 1: multi-query

flowchart LR
    Q[casual question] --> RW["LLM rewriter<br/>(Haiku, ~$0.0001)"]
    RW --> V1[variant 1]
    RW --> V2[variant 2]
    RW --> V3[variant 3]
    Q --> R0[retrieve top-10]
    V1 --> R1[retrieve top-10]
    V2 --> R2[retrieve top-10]
    V3 --> R3[retrieve top-10]
    R0 --> F["reciprocal rank fusion<br/>(already in the codebase)"]
    R1 --> F
    R2 --> F
    R3 --> F
    F --> K[top-5 to the LLM]

An LLM rewrites the question into spec-vocabulary variants, each variant retrieves separately, and the ranked lists are merged with the same reciprocal-rank-fusion function that hybrid search already used. Nothing hallucinated can leak in this way: whatever the phrasing, retrieval only ever returns real corpus text.

The first surprise came before any quality result. Two identical eval runs returned 0.50 and 0.70. The reason was that the rewriter ran at the default temperature, so every run searched with different variants. If a retrieval mode has an LLM inside and the temperature is not pinned, the recall number is not reproducible. With temperature 0 the result became stable: 0.60, twice.

Which is... exactly the baseline. No improvement. The hits did move though: multi-query recovered the currency question (the one behind the judge-0 answer) and lost a different one, one for one. That trade was the most useful outcome of the experiment, because it showed the vocabulary gap is real and reachable, and also that rewriting alone cannot cash it in. A better phrasing still lands on the same diluted chunk embeddings.

I stopped there. With ten golden questions every hit is worth ±0.10, and tuning variant counts until the number goes up would be overfitting the eval, not improving retrieval.

Fix 2: contextual retrieval

For contrast, this is what indexing looked like until now. Each chunk is embedded exactly as stored:

flowchart LR
    PDF[rulebook PDF] --> CH[chunk ~300 words]
CH --> E["embed(chunk)"]
CH --> ST[store verbatim chunk]
E --> DB[(pgvector)]
ST --> DB

And with contextual retrieval, two LLM steps get inserted before the embedding. The same PDF feeds both paths: its full text produces a one-time summary, and that summary is the shared context for writing a short blurb per chunk:

flowchart LR
    PDF[rulebook PDF] --> CH[chunk ~300 words]
PDF -->|full text| S["LLM summary<br/>(once per doc)"]
S --> B["LLM blurb per chunk<br/>(one call per page)"]
CH --> B
B --> E["embed(blurb + chunk)"]
CH --> ST[store verbatim chunk]
E --> DB[(pgvector)]
ST --> DB

The blurb is one or two sentences situating each chunk: which rulebook it is from, which rule it belongs to, what its numbers are about. It is prepended only for the embedding. The stored text, the one that gets cited and shown as evidence, stays the verbatim spec passage. A bare "5 seconds" now embeds as "SCT Inst rulebook, target maximum execution time... 5 seconds", which is what a casual question is actually reaching for.

Two implementation choices kept it cheap and safe:

All chunks of a page get their blurbs in one call, against a one-time per-document summary. That is about $0.5 for the whole 484-chunk corpus, instead of the naive chunk-times-whole-document approach.
If a blurb call fails, that page just embeds bare chunks with a warning instead of aborting the run. In practice it never happened: 484 of 484 chunks got contextualized.

Results

Recall@5 per retrieval configuration, with the per-query cost of each:

plain index (baseline): 0.60
multi-query rewrites: 0.60, at ~1s + $0.0001 per query (and 0.50 to 0.70 until the temperature was pinned)
reranking, from an earlier experiment, for scale: 0.70, at about a minute per query
contextual index: 0.80, at zero per-query cost
contextual index plus multi-query: 0.80, the hits only reshuffle, no net gain

The contextual lift is strictly additive: every question the baseline got right, plus two it did not, including the one behind the only judge-0 answer. It also carries through end to end. The answer eval on the new index scores 96.3 mean with a 100% pass rate, up from 84.8 and 90%. Asked live, the formerly failing question now answers "SCT Inst payments are made in euro" and cites the rulebook pages.

One more detail: promoting the fix to production cost zero additional API dollars. The deploy tooling copies the chunks table, vectors included, from the local database to the cloud one, so production serves the exact index the numbers were measured on.

Note: the earlier six-system comparison scored this same system 84.8, fourth place out of six. On the new index it evals at 96.3, a hair under that comparison's winner (LlamaIndex, 96.5). This is not a re-ranking of the comparison: different index, different day, and the other five systems would deserve the same fix applied. But it does suggest the gap was never about the framework choice. It was about the index.

Conclusion

The cheap query-side trick did nothing, and the index-side fix did everything. Multi-query is what every tutorial reaches for first because it needs no re-indexing; on this corpus it moved recall not at all. Contextual retrieval, for about $0.5 one time, moved it from 0.60 to 0.80 with zero per-query cost.
The failed experiment still paid for itself. Its one gained question predicted what the real fix would deliver, and it surfaced that an unpinned LLM temperature makes an eval non-reproducible. A measured "no" is material, not waste.
Ten questions is a small ruler. Every hit is worth ±0.10, so I stopped tuning the moment the temptation appeared. The numbers above are honest rather than optimized.
Same discipline as always: nothing ships on vibes. Reranking measured well and stayed benched because of latency, multi-query measured flat and stayed benched, contextual retrieval measured +0.20 at zero query cost and shipped the same day.

Out of Scope

Sentence-window / small-to-big chunking, the other index-side fix from the same playbook; it stacks with this one but was not measured here
Re-running the full six-system comparison on contextual indexes (every system would need the same treatment to keep it fair)
BM25/hybrid contextualization: the blurbs currently improve only the vector side

Sources

GitHub repo, ADR-0022 (multi-query), ADR-0023 (contextual retrieval), the retrieval-quality playbook
live demo
Anthropic: Contextual Retrieval
HyDE paper: Gao et al., "Precise Zero-Shot Dense Retrieval without Relevance Labels" (the hypothetical-answer trick, rejected here)
The six-system comparison report

Comparing RAGs, Part 2: the benchmark

Serhiy Kucherenko — Mon, 07 Sep 2026 20:23:13 +0000

In Part 1, I laid out why "better" RAG isn't about raw accuracy: it's about whether your
data stays inside your own infrastructure or ends up as a standing index on someone else's. Six approaches got tested
head-to-head against the same 10-question golden set, payments-rag's own hand-rolled build, openai-file-search,
Google NotebookLM, and three library builds (Haystack, LlamaIndex, LangChain/LangGraph). Here's how each was
actually built, and what happened when they ran.

Development

The source code lies in a GitHub repo.
But the inner works across all six implementations can be summarized in the following sections.

System Design

This project consists of the following components:

Corpus: the raw PDFs (three SEPA/ISO 20022 rulebooks), the input every system indexes.
Indexing (one-time, per system): chunk the corpus, embed each chunk, store as vectors. payments-rag/Haystack/LlamaIndex/LangChain all do this themselves; openai-file-search and NotebookLM do it inside the vendor's own managed service instead.
Embedding: turns text (a question, or a chunk) into a vector. The same OpenAI model across every system except NotebookLM, whose embedding step is entirely internal/opaque.
Retrieval: given a question's embedding, find the closest stored chunks. pgvector similarity search for payments-rag; each framework's own equivalent for the rest.
Generation: question + retrieved chunks → an LLM → the answer. Claude for payments-rag, gpt-4o for most framework builds and openai-file-search, Gemini for NotebookLM.
Adapter (comparison-only, not part of any system's real architecture): a thin wrapper per system normalizing each one's very different calling convention into one common shape: question in, {answer, contexts, cost, latency} out. This is what makes the next layer possible across six otherwise-incompatible systems.
Judge/Eval (offline, never per-query): the normalized output gets scored two ways: RAGAS ( faithfulness/relevancy/precision/recall) and a cross-model judge (correctness vs. ground truth). Runs after the fact, on the fixed 10-question golden set.
Client vs. pipeline test client: a real user's question only ever touches payments-rag's own Embedder → Retriever → Generator chain and never reaches eval. The pipeline test client (the golden-set harness) is the only thing that reaches Adapter. It's also the only thing that talks to the other five systems.
Reaching the other five systems: four of them (openai-file-search, Haystack, LlamaIndex, LangChain) have a real API, so the test client calls them directly, same as it calls into payments-rag. NotebookLM doesn't, and it's reached through a Browser instead.

Indexing and Retrieving Text

1. payments-rag

2. openai-file-search

3. Google NotebookLM

4. Haystack

Haystack's sentence-based chunker treated a malformed PDF-extracted block as one giant "sentence," pushing it past
OpenAI's embedding size limit. As a consequence, Haystack silently dropped that chunk instead of raising an error,
quietly losing a third of the index. Fixed by switching to Haystack's own word-based chunking default.

5. LlamaIndex

LlamaIndex's default PDF reader leaked raw PDF metadata into extracted text and mangled some pages into unreadable
binary noise, both silently, without an error thrown. The noise sometimes outranked the real answer during retrieval.
Fixed by loading PDFs with pypdf directly, the same library Haystack already used cleanly.

6. LangChain / LangGraph

Results

LlamaIndex is the best in both judge and faithfulness scores.

OpenAI file search cost $0.3526 for the same ten questions, which is the most expensive system in the comparison
by a wide margin: over 8x pricier than the next-priciest option (Haystack, $0.0413), and roughly 13x pricier than
payments-rag's own hand-rolled approach ($0.0276). Likely because it stuffs more retrieved context into every request
than expected.

NotebookLM costs nothing in dollars, but it required 11 min of setup work of clicking
before the first question could even be asked. Meaning, it just moves the cost from money to time.

NotebookLM ties LlamaIndex for the highest answer-relevancy of all six, despite its other RAGAS metrics being
unusable. LangChain, the most popular framework by mindshare, placed second-worst, just above Haystack.
Haystack's remaining weakness is a retrieval bias toward one source document on standard-SCT questions, not a
grounding failure.

Conclusion

Some numbers worth pulling forward before the caveats:

own built payments-rag placed 4th of 6 on judge-scored accuracy (84.8), behind LlamaIndex (96.5), NotebookLM (92.5), and openai-file-search (90.2).
it was also the cheapest ($0.0276) and fastest (2.68s) system in the whole comparison. For instance, OpenAI markup caused it to cost about 8-13x, while NotebookLM took 11 minutes to set up.
silent bugs may creep simply because of how the systems are wired up as it happened to LlamaIndex and Haystack. In this case it was about clean corpus and a sane chunk size.

The obvious flaw of this stage is a small set of questions (or maybe topics of questions) of a golden set.
Additionally, the compared options could be expanded dimension-wise by combining different models
(both embedding, llm, judge).

However, this intel brings some clarity on why we might end up having RAGs everywhere as well as concerns about both
correctness and security. Building a working and useful RAG is one effort. Building one at scale is a different thing.
And building one for privacy is yet another problem. The only way to fully avoid exposure is self-hosting all
the layers: the embedding, the generation models, and the retrieval layer. payments-rag's own hand-rolled setup still
sends chunks to OpenAI and Anthropic at embed- and generation-time (see the exposure diagram in Part 1).

Out of Scope

Bigger and more diverse golden sets, additional metrics beyond RAGAS/judge
Testing every embedding/LLM/judge model combination for full experimental cleanliness: this run fixed one model set throughout
NotebookLM's paid Enterprise tier: investigated, doesn't change the conclusion (still no way to ask it a question programmatically)

Also, that was made on Python. I'd imagine someone doing it, say, on C++ or Rust (whatever is better) would make it
faster.

Sources

GitHub repo
live demo
RAGAS issue #2753: the scoring-library bug
NotebookLM Enterprise API docs: no query endpoint, any tier
NotebookLM privacy policy: training-data use
Haystack telemetry docs

Comparing RAGs, Part 1: the framework

Serhiy Kucherenko — Sat, 05 Sep 2026 20:25:59 +0000

In the past, I built my own RAG. The idea was to implement and debug concepts rather than read documentation. So far, a RAG is supposed to address this issue for a person or a company:

AS an engineer,
I WANT TO get accurate data on need
SO THAT I get the exact answer, with a citation, as fast as possible

This is the problem a RAG tries to solve. Since this is an engine that also includes a database, safety and scalability concerns apply. There are multiple options, and the problem might be: which option is the best?

Hypothesis

We could categorize solutions by usage and integration:

Manual personal interaction services Google's like NotebookLM
API services like OpenAI's file_search
Libraries like Haystack
Custom solution

Typically, if a system needs scale (imagine lots of people constantly surfing through documentation which could be indexed in a RAG), the choice lies between building an own solution using a library and a cloud solution from another company.

Warning: a claim like "no need for expertise because we have AI" means we have no way to verify if a clue detail was
missed (described at Judge Strictly Avoid Hallucinations).

Safety and Politics as concerns

Recent worldwide events have shown that depending on another company is not only a security vulnerability but also a political one. This matters because while it might be easier and work better initially to just dump everything into a cloud solution of another enterprise, this creates a single point of failure in many ways. Also tends to be quite expensive in the long run.

Maintaining an own solution is not cheap either: you will have an internal service with a team maintaining it, and with the pace the AI tech progresses, you risk falling behind already in a few months if not weeks.

Define "better"

So... which solution would work better, and how a RAG would benefit us?

Before comparing anything, it's worth being precise about what "better" even means here. It isn't rather who answers questions most accurately, as we will end up just comparing frontier models and context windows on a ten-question set. The real question is integration: can this run inside a system where data is private?

The Judge

The most helpful way is to use another LLM as a judge with strict scoring, to avoid any false positives, as these are the most dangerous silent killers of confidence. So a gpt-4o with text-embedding-3-small will do just fine as a different vendor from the Claude-generated answers this project produces, which avoids grading its own homework.

RAGAS

We will also use [RAGAS] https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/) 0.4.3 for evaluation. RAGAS was picked as the scoring method for the comparison itself specifically because it is a library, in-process, introduces no new recipient of corpus data, and has a documented telemetry kill switch.

This standard quartet of metrics will do:

Faithfulness: does the answer only say things the retrieved chunks actually support?
ResponseRelevancy: does the answer address the question asked, not something adjacent?
LLMContextPrecisionWithoutReference: of what got retrieved, how much was actually relevant?
LLMContextRecall: of what should have been retrieved, how much actually was?

Note: even the scoring library itself shipped broken. A plain pip install ragas crashes on import with ModuleNotFoundError: No module named 'langchain_community.chat_models.vertexai', caused by RAGAS's own code imports a path that newer versions of one of its own dependencies removed. Real, still-open bug on RAGAS's side
(issue #2753, fix still unmerged), not something this project did wrong. Worked around by pinning an older langchain-community.

Issue: Transient vs. Persistent Exposure

That's a second axis worth noting. Transient means a request goes out, a vendor's model processes it, and nothing about your corpus survives on their infrastructure afterward as a queryable asset.

Persistent means the opposite: your corpus (or an embedding of it) gets stored on someone else's infrastructure as a standing index that outlives the call that created it and keeps being a breach target indefinitely. OpenAI's file_search and Google's NotebookLM both fall into the second category: uploading your documents creates a managed index that exists independently of any single question you ask. That exposure means your private data may be breached or even secretly used as training by host companies.

Note: a RAG already accepts some transient exposure, as we will end up sending a chunk (100-1000 words, which is a typical size of a chunk).

sequenceDiagram
    participant You as Your infrastructure
    participant Embed as Embedding model (cloud API)
    participant LLM as Answering LLM (cloud API)
    Note over You, LLM: Embed-time (chunk text leaves your infra)
    You ->> Embed: chunk text (index + query)
    Embed -->> You: embedding vector
    Note over You: stored/searched locally (Postgres/pgvector)
    Note over You, LLM: Generation-time (chunk text leaves your infra again)
    You ->> LLM: chunk text (as prompt context)
    LLM -->> You: answer

The only way to avoid exposing your data completely is to host the embedding, the answering model, and the judging model yourself, at a cost of having weaker models and continuous maintenance.

Issue: Integration

Another issue is how easy it is to integrate a RAG into your knowledge base. Typically, a team has a separate ticketing platform, data, and a logging storage. Also, it has a company-wide context, dependencies on other teams, documentation, alerts. That presents a set of challenges when trying to find a necessary set of pieces of info. As well as an issue with integrating such a solution as RAG and maintaining the relevance of its data.

Besides, each team will have their unique and individual stuff like specified above. Meaning, having a single same corpus for all teams is not possible.

Note: As of August 2026 NotebookLM only works by hand, through its own web page, there's no way to script it or plug it into anything else
even on the paid enterprise tier. Google added an API in 2025, but it still can't be asked a question through it. Regardless, given the pace AI tech evolves, it is still included in this comparison to keep an eye on.

The six systems compared, and how each was queried

payments-rag: the project's own production path, Claude-generated answers over its own pgvector retriever.
openai-file-search: OpenAI Responses API, model="gpt-4o", tools=[{"type": "file_search", "vector_store_ids": [...]}], corpus PDFs uploaded once to a named OpenAI vector store
NotebookLM: queried through its web UI, answers and citation filenames captured by hand
Haystack: a RAG pipeline built with the deepset library instead of hand-rolling it with the same corpus and models. It is the framework that does the orchestration.

Note: Haystack ships with anonymous, opt-out telemetry (on by default), though it sends only component types (e.g., which retriever/store you used), no personal data. To turn it off: HAYSTACK_TELEMETRY_ENABLED=False.

LlamaIndex: the same idea, built with a different library, included to see whether "framework vs. framework" matters as much as "framework vs. hand-rolled"
LangChain/LangGraph: a RAG pipeline built with LangChain's components (loader, splitter, vector store, retriever), with LangGraph doing the orchestration instead of a hand-rolled function or the other frameworks' own pipeline objects

The comparison was based on answers of the same 10-question golden set that was used throughout the development of payments-rag.

Part 2 covers how each of these was actually built, and what broke along the way.

Publishing Monday.

How to hurt yourself with AI

Serhiy Kucherenko — Tue, 25 Aug 2026 14:56:39 +0000

Summary

34x more work than it should have been.

Timeline

I write technical articles, and I like experimenting with something new in each one. For one of them, the experiment was an AI pipeline: it would draft the piece and pass it through several rounds of automated review before I saw it.

That pipeline produced 1,948 words while processed 65,930 in total which is ~34x more.

The idea was to base the writing on careful review and a feedback loop. So I ended up reviewing in person and wasting 34x words/tokens more than what I could actually produce.

Root cause

I am not sure what the exact cause is. There are several candidate explanations, and I do not think any one of them is complete on its own:

Not enough context to judge whether a cut was a real improvement.
The AI outputs what is locally coherent, what "sounds right," with no guarantee that coherent and applicable are the same thing.
The AI was built to fill in gaps rather than flag them, and what it fills in with is often an assumption, usually one that turns out false.

What a human has that the AI does not is a felt sense of consequence: what a decision actually costs or breaks, in a specific world, learned from having been wrong before. "Experience" is the word for that, and it is overused to the point of meaning nothing, but I do not have a better one.

If I compress all of that into one line: the pipeline optimized for the wrong thing. It optimized for passing a check. It could not optimize for "is this piece actually good," because nothing in it could feel the cost of being wrong.

The aim behind building it was sound and I would keep it: automate whatever a computer can do so a human does not have to. What failed was assuming editorial judgment was one of those things.

The debt

Engineers already track several kinds of debt that are not financial.
One of them is getting real attention now: cognitive debt, which is what you take on when you outsource tracking why your own system does what it does to an LLM. Two recent sources define it the same way: a
MIT Media Lab study on AI-assisted writing, and getdx.com's framing for software teams specifically.

This pipeline is a worked example. I could tell you the ratio, the gate verdicts, the timeline. For a while I could not have told you, from memory, what any single piece actually argued.

The more AI gets pushed into everything, the more this particular kind of debt matters. On a prototype, nobody bothers tracking it. On something meant to hold up, not understanding what you built is a deal-breaker,
and "just ship it, iterate later" is the same old fallacy wearing an
AI-shaped coat: speed and understanding traded off as if they were opposites, when the trade was optional.

What changes

The intent behind the pipeline was to be the reviewer: read a draft,
judge it, hand back a correction, and only step in myself when the correction failed too. In practice, I was correcting constantly, sentence by sentence, round after round, and each correction only produced more issues.

That does not work, and it should not. Scale it up and the failure gets clear: imagine ten thousand articles like this shipping out daily, all with a human correcting sentence by sentence, all costing that person a 34x reading tax.

Going forward, AI in this workflow does small, bounded jobs: formatting, a lookup, a narrow research question with a checkable answer. Things that do not load much cognitively.

Question for you

What have you noticed, using AI or watching it get used badly? Two data points I found while researching this piece:

Different domain, same shape as this piece: a check that looked like it was working, until someone read closely enough to notice it wasn't.

Judge strictly to avoid hallucinations

Serhiy Kucherenko — Fri, 14 Aug 2026 13:13:20 +0000

Ten questions over the SEPA rulebooks. A second model grades each answer against a reference, zero to a hundred, and the run came back healthy.

Except for question five, which asked which currency SCT Inst payments are executed in. The reference answer is one word long.

A zero on a one-word question reads like a system that does not know the first thing about its own subject matter. That is not what happened.

It was doing what it was told

The production prompt is explicit about the case where retrieval comes up short:

Answer the question using ONLY the sources below. If they do not contain the
answer, say so.

Retrieval never put the euro page in the top five results. So the sources handed to the model genuinely did not contain the answer, and the model was under written instruction to say exactly that.

The grader took that response, compared it to the word "Euro", and returned a zero.

The grader cannot see the difference

The judge receives the question, the reference answer and the answer text. It does not receive the sources, and that is deliberate: a grader holding the retrieved chunks would end up grading the retrieval instead of the answer.

That choice has a cost. "Correct, given what it was handed" is not a judgement this grader can make. It has one axis, factual match against the reference, and everything that fails to match lands at the bottom together.

Which means a fabricated answer about euro payments would have scored exactly what the refusal scored. Zero, either way. The eval could not tell the two apart.

The saved run does not help either. Each question is stored as an id, a score and a one-line critique. The answer text itself is not kept, so nothing downstream can recover the distinction the grader could not draw.

The failure was upstream

If the refusal had been the problem, changing the answering model would have moved the number. It was not, and it did not.

Two separate experiments recovered that question, and neither one touched generation. Keyword search found the euro page instantly, because the page contains the actual word. Vector search matches on meaning rather than
words, and it ranked that page outside the top five. A cross-encoder reranker, kept to the eval path only, found it as well and lifted recall@5 from 0.60 to 0.70.

Neither is a free fix. Leaving keyword search on permanently left recall@5 exactly where it started, at 0.60, because it dropped a question the vectors had been getting right. The blind spot moved. It did not close. The reranker is the only arm that moved the number, and it is not in production.

Make a wrong answer cost something

Retrieval will keep missing pages. Improving that is a separate job, so what matters here is how the eval reacts when it happens.

The grader was not being harsh. My rubric just stops at zero, so both failures pile up there together. One of them wastes the reader's time. The other sends them into a payments rulebook holding a rule that does not exist. My eval scored those the same.

So the strictness belongs on the wrong answer. Meta's CRAG
benchmark, which I only found after this run, grades four outcomes instead of one axis: a correct answer scores 1, a useful answer with minor errors 0.5, a missing answer 0, and an incorrect answer minus one. "Missing" is defined there as the system replying "I don't know".

That minus sign is the part I was missing. Saying nothing costs you nothing. Being wrong costs you a point.

Nothing was fabricated in this run. The system declined, as instructed, and the number punished it for that. But the rubric that could not credit that refusal is the same rubric that would not have charged for a made-up answer, and a made-up answer is the one that reaches the reader.

Charge the model for being wrong. Let it say nothing for free.

The illusion of improvement

Serhiy Kucherenko — Sun, 09 Aug 2026 19:24:47 +0000

The problem

My RAG answers questions about SEPA payment rulebooks. Every page of those PDFs carries the same header and footer, and all of it lands inside the chunks that get embedded.

Two lines of letterhead on every one of 290 pages, sitting in the same vector as the sentence that actually answers something.

The hypothesis

Identical text in every chunk means a shared component in every embedding. I believed that component was flattening the similarity signal: distances on my test query all clustered tightly around 0.34 instead of spreading out.

Strip the repeated lines, and while I am in there, stop splitting mid-sentence. Sharper chunks, sharper vectors, better ranking.

It is a mechanism, and it is checkable. I would have bet on it.

What improved

The cleanup worked exactly as designed. Any line repeating on at least half a document's pages gets stripped, with digits masked first so page numbers cannot disguise a repeat. "www.epc-cep.eu 35" on one page and "www.epc-cep.eu 36" on the next both normalise to "www.epc-cep.eu #", counted once per page. That line recurs on all 290 pages, so it goes. The rulebook title and the "Date issued" line collapse the same way and go with it:

Boilerplate in chunk text: every chunk, to none.
Words on a typical page: 464, to 448.
Chunks in the corpus: 495, to 484.
Chunk boundaries: cut mid-sentence, to whole sentences.

Eleven chunks disappeared, and the reason is duller than it sounds. No chunk was pure boilerplate. The junk was about sixteen words per page, spread across every chunk on that page. Chunking runs per page at a 300-word target, so the only pages whose count changed were the eleven sitting just above that line: 315 words became 299, 306 became 290, and two chunks became one.

Everywhere else the text simply got cleaner without changing shape.

The reality check

Then I re-ran the query I had been watching. Same top pages, same order.

The one thing I could have seen with my own eyes did not happen. Nothing reordered.

That is the entire result, and it is worth being blunt about how weak an observation it is. There was no evaluation set yet. The test was me reading one query's results and forming an impression, and an impression cannot separate "no effect" from "an effect too small to notice".

Why the mechanism was wrong

Ranking does not compare chunks to each other. It compares each chunk to the query and sorts by that angle. The boilerplate was only ever in the chunks; the question you type carries none of it.

I have two explanations and they both fit. The junk was identical in every chunk, so removing it nudged them all in a similar direction instead of spreading them apart. It was also only about 3% of a page's words, which is not much of a shove to begin with. Either way, I predicted a reshuffle and got a nudge.

What I cannot do is tell those two apart. Separating them needs the per-result distances to more than two decimal places, and I never wrote them down. Even a 3% shift should have moved something in the third decimal. Whether it did is a question my own logs cannot answer.

The 0.34 floor was not noise the boilerplate added either. That is what a dense, homogeneous legal corpus looks like to an embedding model. The chunks really are that similar to each other, because rulebook pages really are that similar.

What I kept, and what I changed

The cleanup stayed. Clean chunks are what a citation displays against an exact page, and what the prompt feeds the model. The payoff is real. It is just not the payoff I built it for.

What actually changed was my process. Three days later I built a golden set of questions with verified answers, and it produced a retrieval baseline of recall@5 = 0.60, the number every retrieval change has had to argue with since.

One detail I have left alone deliberately. The comment at the top of the cleanup module still states the dead hypothesis as fact: boilerplate "drags every embedding toward the same noise, which flattens the similarity signal". It is wrong, it is still there, and it is a better reminder than anything I would write on purpose.

The hypothesis was reasonable, the mechanism was checkable, and the check said no. The next fix that sounds this obviously right gets measured before it gets believed.

Three small lessons from building a RAG by hand

Serhiy Kucherenko — Sun, 02 Aug 2026 19:16:41 +0000

A retrieval system over the SEPA payment rulebooks. Plain Python, Postgres, two LLM vendors. Three independent decisions from it are worth writing down, because in each case the decision was cheap and the thing it taught was not the thing it was made for.

1. The vector database never got installed

Every RAG guide starts in the same place. LangChain's own documentation
puts it plainly: to build RAG, you first need to create a vector store. Then the shortlist writes itself: Pinecone, Qdrant, Weaviate, Chroma.

The vectors live in Postgres instead, through pgvector.

Not because Postgres is faster. Because a new datastore is a new failure surface, with its own consistency model, its own operational habits and its own way of going wrong at 2am. Postgres was already there holding the documents, and I already knew how to debug it. Scaling was never the plan for a corpus of one rulebook.

The whole of the retrieval query is this:

SELECT id, source, text, page, embedding <=> %s::vector AS distance
FROM chunks
ORDER BY distance ASC
LIMIT %s

<=> is cosine distance: 0 is identical, 2 is opposite. The ::vector cast is not decoration. A bare Python list arrives as double precision[], which that operator refuses, so the vector travels as a text literal and is cast on arrival.

One operator, one cast, one index. That is the entire integration.

Convenience is not proof, though. Would the real vector database have bought back time worth having?

One question, end to end, takes about 2,823 ms. The vector search inside it takes 4.9 ms. Replacing pgvector with something infinitely fast would return 0.17% of the wait.

None of this says a vector database is wrong. It says the unfamiliar one would have been paid for in operations and returned a line nobody was waiting on.

2. You can only measure what you own

No LangChain, no LlamaIndex either. The pipeline is plain Python calling two LLM vendors and a Postgres.

The usual defence is YAGNI, and it holds: the flow is five fixed steps with no branching. Embed the question, search, build a prompt, call the model, parse the answer and its citations. Orchestrating five fixed steps is the easy part. A framework earns its keep on branching, retries and swappable backends, and none of those were in play.

But that is not the reason worth giving.

Every result worth publishing from this project came out of a layer a framework would have owned.

I wrote the chunker, so the repeated headers could be stripped from every page and the effect measured: retrieval did not move at all. I wrote the rank fusion, so it was visible that hybrid search scored exactly the same as dense-only, and that the corpus was too small to show a difference rather than the technique being wrong. I wrote the database call, so it could be timed alone and found to be 4.9 ms.

Three findings, all negative, all only visible from inside.

The cost is real and there is a file listing it: mature libraries exist for the chunker, the fusion, the metrics, the eval harness. That file is the off-ramp for when this stops being worth it.

You cannot instrument an abstraction you did not build.

3. Two dependencies called "model", two very different bills

The model this system was planned around was retired before the first commit. The decision record replacing it carries the same date as that commit. The project lost its LLM before it had a second file.

The swap itself was trivial. The responder is one environment variable:

LLM_MODEL: str = os.environ.get("LLM_MODEL", "claude-haiku-4-5")

One line, one deploy, done. Now the other model.

The embedding model cannot move like that. The table declares its vectors as 1536 dimensions and an insert guard rejects anything else, because vectors from different embedding models do not share a space. Changing it means re-embedding the entire corpus before a single query works again. An afternoon at this size. A migration at a real one.

Same word, "model". One is a config value. The other is infrastructure wearing a config value's clothes.

The vendor's deprecation schedule does not read your roadmap. Decide which of your dependencies is which before it decides for you.

The localhost trap: a 10-second database connection on Windows

Serhiy Kucherenko — Mon, 27 Jul 2026 15:41:36 +0000

Every question in the app took about 15 seconds. The model was fast. Retrieval was fast. Ten of those seconds were hiding somewhere else, and for days I couldn't say where.

The app is payments-rag, a RAG system I built over the SEPA payment rulebooks. You ask a question in plain English, it retrieves the relevant passages from Postgres + pgvector, sends them to Claude, and returns an answer with the exact rulebook page cited. Fast is not a goal; under 5 seconds is. On my Windows machine it was taking three times that, every single query, and the answers were correct, which somehow made it worse. Correct but slow doesn't scream "bug." It whispers "maybe that's just how it is."

For days I told myself it was just slow on Windows. Docker overhead, antivirus, something environmental, whatever. I checked the model latency three times before I thought to check the connection. That was the wrong order, and it cost me the better part of a week.

There's a reason this class of bug survives. An LLM app carries a built-in excuse: models are slow, everyone knows models are slow, so a 15-second answer doesn't trigger the same alarm a 15-second SQL query would. The LLM had a good alibi for the slowness. I believed it too early.

Measuring instead of guessing

Two cheap pieces of observability ended the mystery in minutes.

The first was a per-stage timer. I split each request into connect, retrieval, and generation, and logged the breakdown on every answer. The line read:

retrieval 2.1s · generation 3.4s · connect + overhead 10.2s

Retrieval plus generation was about 5.5 seconds. The other ten were sitting around the database connection, not in the model, not in the vector search.

The second was a health check: a small panel that pings each dependency and shows the round-trip time. For the database it read:

DB reachable · 10137 ms

Unambiguous. The connection itself took 10.1 seconds. Not the query, not the embedding, the TCP connect. The bug had been there for days behind a vague "it's just slow." The moment we measured, it took one screenshot to locate.

What was actually happening

The connection string said localhost. That one word triggered a chain of events.

On Windows, localhost is dual-stack: it resolves to both IPv6 ::1 and IPv4 127.0.0.1, and the OS prefers IPv6. So the Postgres client first tried to connect to ::1:5433.

Nothing was listening there. The Postgres in this setup runs in Docker, and the published port was bound on IPv4 only. The IPv4 loopback answers; ::1 does not.

And because no connect_timeout was set, the client sat on the dead IPv6 route until the OS gave up on its own schedule and retried over IPv4. That giving-up takes about 10 seconds. Then the IPv4 connection succeeds in milliseconds, the query runs fine, and the answer comes back correct. Every query paid the toll again.

Three things had to line up to make it hurt:

localhost is dual-stack, and Windows tries IPv6 first.
Docker published the container port on IPv4 only, so there was nothing on ::1 to answer.
No connect_timeout, so instead of failing fast and loud, the client waited out the full OS-level fallback on every single connection.

Remove any one of the three and the problem disappears, which is exactly why it survived for days. On Linux and macOS colleagues' setups the same code was quick.

The fix

One word. localhost becomes 127.0.0.1, which skips DNS ambiguity entirely and goes straight to the IPv4 loopback that Docker actually publishes:

# .env
DATABASE_URL=postgresql://user:pass@127.0.0.1:5433/mydb

And a timeout, so this class of bug can never hide as a silent hang again:

psycopg.connect(DATABASE_URL, connect_timeout=10)

A hang is worse than an error. An error tells you where it hurts; a hang just eats your latency budget and says nothing.

The before and after:

host	connect latency
`localhost` (IPv6 detour)	10,137 ms
`127.0.0.1` (direct IPv4)	27 ms

About 375× faster, one word changed.

One caveat so nobody over-applies this: the trap needs the specific combination above. Windows resolution order, a container port published on IPv4 only, no connect timeout. If your stack differs, your ten seconds are hiding somewhere else.

What I keep from this

localhost is not 127.0.0.1 once IPv6 is in play. For a container whose port is mapped to IPv4, prefer the literal address; it behaves identically on Linux and macOS, where loopback is loopback anyway.
Always set a connect timeout. Fail fast and loud.
Cheap observability pays for itself. A per-stage timer and a one-line health check are maybe an hour of work combined, and they turned a multi-day "it's just slow" into a five-minute fix.

The per-stage timer and the health check both stayed in the project permanently. They've earned their place: the same health view now checks all five dependencies on demand and every 10 minutes, so the next silent hang, wherever it comes from, gets a number attached to it before it gets a story.

Serhiy Kucherenko builds backend systems and LLM tooling. The project this story comes from, payments-rag, is open source: github.com/KucherenkoSerhiy/payments-rag.