TL;DR — Most RAG pipelines are tuned stage-by-stage — chunking, embeddings, re-ranking — using metrics like recall@k and MRR that only prove something relevant was retrieved, not that the generator actually used it correctly. Chunk size and embedding model interact nonlinearly, and re-rankers routinely mask chunking mistakes until they compound at generation time. The fix is to stop evaluating retrieval stages in isolation and instead measure claim-level groundedness in the final output.
Every RAG postmortem looks the same. Someone swaps the embedding model, recall@k goes up, everyone celebrates, and then the chatbot still hallucinates a policy that doesn't exist. The team re-checks retrieval, finds the right document sitting at rank two, and concludes the re-ranker needs tuning. Nobody asks the question that actually matters: did the model use the retrieved text, or did it just have the text nearby while it made something up anyway?
This is the core failure of how most teams evaluate RAG. Chunking, embedding, and re-ranking get scored as independent stages with independent metrics — chunk quality heuristics, cosine similarity, recall@k, MRR — and each stage can look healthy in isolation while the end-to-end system is quietly wrong. The metrics are proxies. Proxies drift from the thing they're supposed to represent, and in RAG they drift fast because the stages are coupled in ways that per-stage metrics can't see.
The proxy metric trap
Recall@k answers one question: is a relevant chunk somewhere in the top k results? That's a necessary condition for a good answer. It is nowhere near sufficient. A generator can receive the exactly correct chunk at rank one and still ignore it in favor of parametric memory, especially when the chunk is short, oddly phrased, or contradicts something the base model is confident about. A generator can also receive five irrelevant chunks and one correct one buried at rank four, and still produce a perfectly grounded answer because the model is good at filtering noise. Recall@k can't distinguish these cases. It measures what retrieval delivered, not what generation consumed.
Teams that optimize retrieval metrics in isolation are optimizing the wrong loop. They tune embeddings against a static labeled set of query-document pairs, ship the change, and then discover downstream answer quality didn't move — or got worse — because the labeled set never captured how the generator behaves when context is messy, redundant, or slightly off-topic. The eval set validates the retriever. It says nothing about the system.
Chunking is not a preprocessing step, it's a hyperparameter of the embedding model
Most chunking strategies are set once, early, using heuristics: fixed token windows, sentence boundaries, some overlap for safety. Then the team moves on and treats chunking as solved infrastructure. That's a mistake, because chunk size interacts directly with the effective range of the embedding model, and that interaction is nonlinear.
Every embedding model has a range of input lengths where it produces well-separated, semantically distinct vectors. Below that range, chunks are too sparse to carry meaning and embeddings cluster around generic topic centroids. Above that range, embeddings start to dilute — a chunk covering five different ideas produces a vector that's a blurry average of all five, indistinguishable from other blurry averages. This is semantic dilution, and it's invisible in a recall@k table because the test queries were probably written to match the chunking scheme that was already in place when the eval set was built.
Change your chunk size and you've silently changed the distribution of vectors the embedding model has to represent well. That changes what "similar" means in your vector index, which changes what the re-ranker sees, which changes what the generator gets. None of this shows up if you re-run the same static eval set, because the eval set encodes the old chunking assumptions.
Re-ranking hides chunking mistakes until it can't
Re-rankers get credit for improving precision, and they usually do improve the ordering of what's already in the candidate pool. But a re-ranker cannot rescue information that chunking destroyed. If a chunk boundary splits a critical caveat from the sentence it modifies — "refunds are available, except for custom orders" split across two chunks — no re-ranking model can reconstruct that relationship. It can only rank the fragments it's given.
This produces a specific and underappreciated failure mode: re-rankers make retrieval metrics look great while quietly papering over structural chunking damage. The correct fragment gets promoted to rank one, recall@k improves, and the generator still gets an incomplete claim because the fragment was incomplete from the moment it was created. The bug lives upstream, in a step nobody is currently measuring against generation outcomes.
What to evaluate instead: claim-level groundedness
If the system-level failure is "the answer isn't actually supported by what was retrieved," then the eval has to be defined at the level of the answer, not the level of the retrieval call. That means decomposing generated answers into individual claims and checking each one against the retrieved context — not against the whole corpus, against exactly what the pipeline surfaced for that query. A claim either traces back to a specific passage in the retrieved set, or it doesn't. If it doesn't, that's a failure regardless of how good recall@k looked.
This reframes the whole pipeline as one thing to optimize rather than three things to optimize separately. Groundedness scores expose chunking failures that recall never would, because a chunk can be "relevant" by embedding similarity and still be structurally unusable for grounding a specific claim. They expose re-ranker failures where the top result is topically right but doesn't actually contain the specific fact the answer needs. And critically, they let you A/B test chunk size, embedding model, and re-ranker together, as a system, instead of pretending you can hold two of the three constant and learn something meaningful about the third.
Building this eval is more work than running a recall@k script against a labeled set. You need a held-out set of real queries, a way to extract atomic claims from generated answers, and a grounding check — model-based or rule-based — that verifies each claim against the retrieved context rather than the general corpus. It's slower to build and it doesn't produce as clean a leaderboard number. But it's the only eval that actually correlates with whether users trust the system's answers.
The practical shift
Stop treating chunking, embedding choice, and re-ranking as three tuning knobs with three separate scoreboards. Treat them as one coupled system whose only real success criterion is: does the final claim trace back to the retrieved text. Run your recall@k and MRR checks if you like — they're cheap signals and they'll catch gross regressions. But don't let them stand in for the thing you actually care about. The gap between "we retrieved the right chunk" and "the model used the right chunk correctly" is where most production RAG systems quietly fail, and it's a gap that per-stage metrics were never designed to see.
Top comments (0)