Same query.
Same documents.
Same model.
And the RAG eval can still hand back a different Recall@8.
Not because the model is flaky. Because of an ORDER BY clause.
I didn't find this by watching a metric wobble. I found it reading the retrieval code, and realized the score would drift run to run even if the model never changed.
This came out of a habit I've adopted recently: I write the eval before the feature. Reviewing the retrieval pipeline behind my "Ask this Book" feature, I saw it: the retrieval layer wasn't deterministic.
Order isn't presentation. It's part of the input.
My RAG implementation is intentionally simple: plain PostgreSQL and .NET. Two retrieval strategies over the same table:
- semantic search using pgvector
- lexical search using PostgreSQL full-text search
The results are merged with Reciprocal Rank Fusion (RRF).
Here's the important part: RRF doesn't care about the retrieval scores. It only cares about rank.
If one retriever returns
A
B
C
instead of
B
A
C
RRF produces different fused scores. Different fused scores mean a different Top-K. Different Top-K means different Recall@K.
In RRF, order isn't a display detail. Order is data.
The bug
My lexical query ended like this:
ORDER BY score DESC
Looks perfectly reasonable. Except ts_rank_cd produces ties surprisingly often. Multiple chunks can have exactly the same score.
And SQL only guarantees the ordering you explicitly request. If multiple rows compare equal, PostgreSQL is free to return them in any order.
Nothing changed. Same database. Same query. Same model. Only the order of equally-ranked rows. Yet that's enough for RRF to assign different ranks, producing different fused scores and a different evaluation result.
The semantic retrieval had the same issue. Distance ties are much rarer than lexical ties, but "rare" isn't good enough for an evaluation pipeline.
The fix
The fix was almost embarrassingly small.
Before:
ORDER BY score DESC
After:
ORDER BY score DESC, id
A deterministic tie-breaker on both retrieval queries. Now equal-scoring rows always appear in the same order, RRF receives the same input every run, and the Top-K stays identical.
Notice what didn't happen. The retrieval didn't become better. It became reproducible.
Why this matters
We spend a lot of effort making the model deterministic during evaluation: temperature 0, fixed datasets, golden answers, reproducible prompts.
But it's easy to assume everything underneath the model is already deterministic. Often it isn't. Retrieval. Ranking. Sampling. Data loading. Any non-deterministic stage in the pipeline can quietly invalidate your eval.
A fluctuating eval isn't just annoying. It's dangerous. Eventually you stop trusting the number, even when it's pointing at a real problem.
The lesson I took away
Before debugging the model, debug determinism. An evaluation can only be as deterministic as the pipeline feeding it. Same query. Same rows. Same order. Only then can you trust what your eval is telling you.
I build TextStack, an open-source reader for technical books, in .NET. This is from the retrieval layer behind its "Ask this Book" feature. Code on github.com/mrviduus/textstack.
Top comments (2)
This is the kind of RAG failure mode that looks like an eval problem until you trace the retrieval layer carefully.
I like separating three sources of instability:
If those are mixed together, a failing eval tells you almost nothing. You need to know whether the answer changed because the model reasoned differently, because different chunks were retrieved, or because the same chunks arrived in a different order.
For production systems, I’d also log the retrieval fingerprint with every answer: index version, embedding model, query rewrite, filters, top-k ids, scores, and reranker version.
Without that, “RAG quality” becomes too vague to debug.
Yes, for me it was only the third one. Same IDs, same scores, just a different order.
Your fingerprint idea is good. If I log the top-k IDs per run, this bug is obvious right away. I will add it thanks.