The problem
My RAG answers questions about SEPA payment rulebooks. Every page of those PDFs carries the same header and footer, and all of it lands inside the chunks that get embedded.
Two lines of letterhead on every one of 290 pages, sitting in the same vector as the sentence that actually answers something.
The hypothesis
Identical text in every chunk means a shared component in every embedding. I believed that component was flattening the similarity signal: distances on my test query all clustered tightly around 0.34 instead of spreading out.
Strip the repeated lines, and while I am in there, stop splitting mid-sentence. Sharper chunks, sharper vectors, better ranking.
It is a mechanism, and it is checkable. I would have bet on it.
What improved
The cleanup worked exactly as designed. Any line repeating on at least half a document's pages gets stripped, with digits masked first so page numbers cannot disguise a repeat. "www.epc-cep.eu 35" on one page and "www.epc-cep.eu 36" on the next both normalise to "www.epc-cep.eu #", counted once per page. That line recurs on all 290 pages, so it goes. The rulebook title and the "Date issued" line collapse the same way and go with it:
- Boilerplate in chunk text: every chunk, to none.
- Words on a typical page: 464, to 448.
- Chunks in the corpus: 495, to 484.
- Chunk boundaries: cut mid-sentence, to whole sentences.
Eleven chunks disappeared, and the reason is duller than it sounds. No chunk was pure boilerplate. The junk was about sixteen words per page, spread across every chunk on that page. Chunking runs per page at a 300-word target, so the only pages whose count changed were the eleven sitting just above that line: 315 words became 299, 306 became 290, and two chunks became one.
Everywhere else the text simply got cleaner without changing shape.
The reality check
Then I re-ran the query I had been watching. Same top pages, same order.
The one thing I could have seen with my own eyes did not happen. Nothing reordered.
That is the entire result, and it is worth being blunt about how weak an observation it is. There was no evaluation set yet. The test was me reading one query's results and forming an impression, and an impression cannot separate "no effect" from "an effect too small to notice".
Why the mechanism was wrong
Ranking does not compare chunks to each other. It compares each chunk to the query and sorts by that angle. The boilerplate was only ever in the chunks; the question you type carries none of it.
I have two explanations and they both fit. The junk was identical in every chunk, so removing it nudged them all in a similar direction instead of spreading them apart. It was also only about 3% of a page's words, which is not much of a shove to begin with. Either way, I predicted a reshuffle and got a nudge.
What I cannot do is tell those two apart. Separating them needs the per-result distances to more than two decimal places, and I never wrote them down. Even a 3% shift should have moved something in the third decimal. Whether it did is a question my own logs cannot answer.
The 0.34 floor was not noise the boilerplate added either. That is what a dense, homogeneous legal corpus looks like to an embedding model. The chunks really are that similar to each other, because rulebook pages really are that similar.
What I kept, and what I changed
The cleanup stayed. Clean chunks are what a citation displays against an exact page, and what the prompt feeds the model. The payoff is real. It is just not the payoff I built it for.
What actually changed was my process. Three days later I built a golden set of questions with verified answers, and it produced a retrieval baseline of recall@5 = 0.60, the number every retrieval change has had to argue with since.
One detail I have left alone deliberately. The comment at the top of the cleanup module still states the dead hypothesis as fact: boilerplate "drags every embedding toward the same noise, which flattens the similarity signal". It is wrong, it is still there, and it is a better reminder than anything I would write on purpose.
The hypothesis was reasonable, the mechanism was checkable, and the check said no. The next fix that sounds this obviously right gets measured before it gets believed.
Top comments (0)