I mistook movement for improvement.
For three months, I changed our ranking algorithm every two weeks.
- BM25.
- Hybrid BM25 + vector search.
- A cross-encoder reranker.
- A different embedding model.
- Yet another embedding model.
Every change moved a few documents up or down the list.
None of them made the answers better.
The Same Wrong Answer, Wearing a Different Tie
The user asked:
"Why does the gateway reject service X?"
With BM25, the top result was a glossary page defining gateway. Relevant by keyword. Useless by intent.
With hybrid retrieval, the glossary page dropped to third. A Jira ticket mentioning service X moved to second. It described the symptom, not the cause.
With a cross-encoder reranker, the Jira ticket became the top result. The glossary page fell to fourth. The architecture document that actually explained the exception remained at seventh.
With a different embedding model, that architecture document finally moved to fifth.
I remember thinking we'd finally made progress.
We hadn't.
Every new algorithm felt like progress because something moved.
The problem was that nothing important moved.
The Document That Mattered Never Became a Candidate
The architecture document existed.
I knew it existed.
I had even written part of it.
It never appeared in the top results.
Not because it was ranked poorly.
Because it was never selected.
The ranking algorithm could only sort the candidates it received.
The document that actually explained the answer never entered that candidate set.
No ranking algorithm can promote a document that was excluded before ranking even began.
The Moment It Clicked
I had five search results on my screen.
BM25 returned them in one order.
The cross-encoder returned them in another.
A different embedding model shuffled them again.
They were still the same documents.
Not one of them answered the question.
Only then did I ask myself:
Why am I ranking these documents at all?
The Question That Changed Everything
I stopped asking:
How do I rank these documents better?
I started asking:
Why are these documents candidates in the first place?
That question led me backward.
Not to another ranking algorithm.
But to every previous question I had been trying to answer.
Why didn't a larger context window help?
Why did chunking make retrieval worse?
Why did search keep finding documents but miss the explanation?
I realized those weren't separate problems.
They were different symptoms of the same one.
I had been optimizing the last step of retrieval while the earlier ones were quietly determining the outcome.
The Hierarchy of Retrieval
I now think about retrieval as three different layers.
- Selection — What becomes a candidate at all?
- Assembly — How is information preserved, structured, and connected?
- Ranking — In what order should the candidates appear?
For months, I focused almost entirely on the third layer.
The first layer was already deciding whether the answer had any chance of being found.
Perfect ranking can only reorder what has already been selected.
It cannot retrieve what never became a candidate.
I kept polishing the order of documents that should never have been candidates.
I had mistaken ranking for retrieval.
In the next post, I'll look at a different question: why organizations rarely lose information—they lose the connections that once made that information meaningful.
Top comments (4)
the diagnostic that makes this hierarchy actionable: measure at the layer boundary. for each test question, check whether the gold document exists anywhere in the raw candidate pool before ranking touches it. that one rate splits every failure cleanly, low means selection is the ceiling and no reranker can save you, high with bad top-k means ranking is genuinely the problem. three months of shuffling algorithms is what happens when that number is invisible, because every reranker demo looks like progress against an unmeasured ceiling. selection bugs also hide in boring places, chunk boundaries cutting the explanation in half, filters silently excluding the doc type, the file never getting indexed at all.
I really like "measure at the layer boundary." That's a much cleaner way to separate the problems.
If the correct document never enters the candidate pool, ranking is almost irrelevant. The ceiling was already set by selection. And if the document is consistently in the candidate set but never reaches the top, then the reranker finally deserves the attention.
I also like your point that selection failures are often surprisingly mundane. We tend to blame embeddings or ranking, while the real cause is something much simpler: the document was never indexed, a filter excluded it, or the explanation was split apart during ingestion. By the time you look at ranking metrics, the failure has already happened upstream.
"I had mistaken ranking for retrieval."
That sentence just summarized about a year of my life. Great piece. Looking forward to the next one
I'm glad that line resonated.
It took me much longer than one sentence to arrive at it, so hearing that it captured a year of your experience is incredibly rewarding.
Thanks for reading. More to come.