Why Did That Chunk Win? 7 Questions I Ask Every Hybrid Search Result in .NET 10

My docs assistant got the question "refund fails with ERR-4012 after a partial refund". The top hit was a general page about issuing refunds. The actual ERR-4012 error page came third.

Was that a bug? I couldn't tell, and that bothered me more than the ranking did. I had a BM25 index and a vector index over the same chunks, fused with Reciprocal Rank Fusion (RRF), with an LLM reranker on top. A pipeline like that can be right for reasons you don't understand and wrong for reasons you can't see.

So I stopped staring at final lists and started asking every surprising result the same seven questions. This post is that list. It's built on Microsoft.Extensions.VectorData 10.11.0 and Microsoft.Extensions.AI 10.11.0, the code is trimmed to its shape, and the trimmed parts are marked.

Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10. The complete, working, tested hybrid search project behind this post is inside, with full source and an in-depth PDF guide. Join Tech Skill Builder

Question 1: Which leg found it, and at what rank?

A fused score on its own tells you nothing. The first thing I added was an explanation per hit: which retriever returned it, at which rank, with which raw score. Fusion keeps that as a list of contributions:

public sealed record Contribution(string Source, int Rank, double RawScore, double FusedPart);
public sealed record FusedHit(string Id, double Score, IReadOnlyList<Contribution> Contributions);

For the ERR-4012 question, the demo prints this:

refunds#issuing-a-refund                           rrf=0.0325  lexical #1 (12.83)  vector #2 (0.543)
refunds#refund-window                              rrf=0.0323  lexical #3 (5.66)  vector #1 (0.629)
errors#err-4012-amount-exceeds-refundable-balance  rrf=0.0315  lexical #2 (9.31)  vector #5 (0.200)

Now the mystery is gone. Both legs liked the general refund page. The error page was second for BM25, but only fifth for vectors, because to an embedding "ERR-4012" is just a short string near lots of refund text. Whether that's acceptable is a product decision, and the explanation is what lets you make it.

Question 2: Were raw scores ever added together?

If anything in the pipeline adds a BM25 score to a cosine similarity, even after min-max normalization, that's my first suspect. BM25 scores are unbounded and spiky, and cosine scores in a top list are often packed tightly together. Normalizing stretches the tight list and squashes the spiky one, so a good vector hit can end up worth zero.

RRF only looks at ranks. With 1-based ranks and k = 60, you can check the numbers above by hand: the refund page gets 1/61 + 1/62 = 0.0325. Being near the top of both lists beats being first in one list only:

// score(d) = sum over lists of weight / (k + rank(d))
public static IReadOnlyList<FusedHit> ReciprocalRank(IEnumerable<RankedList> lists, int k = DefaultK, int? top = null)
{
    // per list: skip duplicate ids (keep the best rank), add weight / (k + rank)
    // then sort with deterministic tie-breaks: score, best rank, number of lists, id
    // (full implementation in the complete project)
}

The tie-breaks aren't decoration. RRF produces exact ties, and without a fixed order your eval numbers wobble between runs.

Question 3: Did the analyzer see what the user typed?

When BM25 misses something it "obviously" should find, the scoring is rarely the problem. The tokenizer is. Split PAUSE_COLLECTION or RetryPolicy.MaxAttempts on every non-letter and the exact identifier disappears; the user's precise query turns into a handful of common words.

What I check:

• Compound tokens are kept whole and split into pieces, so both the exact form and the parts are searchable.
• The query goes through the same analyzer as the documents. Always. Including keywords that callers pass in directly.
• Stemming is light. Folding refunds to refund is fine. An aggressive stemmer that merges unrelated words invents matches in a technical corpus.

var terms = TextAnalyzer.Default.Analyze("RetryPolicy.MaxAttempts");
// includes the whole token and its pieces: retrypolicy.maxattempts, retrypolicy, retry, policy, maxattempts, attempt
// (the splitting and plural rules are in the complete project)

Question 4: Did both legs apply the same filter?

This is the one that can leak data. The question "what to do when ERR-5031 spikes" has a public answer and a staff-only incident runbook. The demo runs it for both audiences:

public:   sdk#configuring-retries, errors#err-5031-upstream-processor-unavailable, refunds#issuing-a-refund
internal: internal-incidents#err-5031-spikes, sdk#configuring-retries, errors#err-5031-upstream-processor-unavailable

That's correct. But if only the vector leg filtered by audience, BM25 would put the runbook into the public list and fusion would pass it along without complaint. So the filter is written once, as an expression, and both legs get the same one:

var filter = KnowledgeIndex.AccessFilter(request.Audience, request.Product);  // Expression<Func<KnowledgeChunk, bool>>
// vector leg: passed to the connector in the search options
// BM25 leg:  the same expression, compiled and applied per chunk
// (wiring omitted; a theory test checks that both legs return the same chunk counts per audience)

Question 5: Was the right chunk even in the candidate pool?

Fusion and reranking can only reorder what the legs returned. If each leg returns its top 10 and the right chunk sits at vector rank 14, nothing downstream can save it. I fetch 40 candidates per leg and keep 5.

The related trap is a leg you think is off. A leg with weight 0 should be dropped completely, not added with zero scores at the bottom of the list, or "disabled" retrievers keep quietly filling short result lists:

if (useLexical && Options.LexicalWeight > 0) lists.Add(new RankedList("lexical", lexicalHits, Options.LexicalWeight));
if (useVector  && Options.VectorWeight  > 0) lists.Add(new RankedList("vector",  vectorHits,  Options.VectorWeight));

Question 6: What did the reranker change, and what if it lied?

I rerank the top 12 fused candidates with an LLM and structured output. For "stop charging a customer every month" it graded the pausing and cancelling pages 3 out of 3, which is what I'd want. The questions are about the bad days:

• Does the model return grades (0 to 3) per candidate, while my code does the sorting? It does, and ties keep their fused order.
• Are candidates sent as short labels (c1 to c12), so an invented or garbled id is easy to spot? In the demo, a made-up id gets reported and ignored.
• Does invalid JSON, a timeout or an exception fall back to the fused list? Yes. Search never fails because the reranker did. Only caller cancellation propagates.

if (!response.TryGetResult(out var verdict) || verdict is null)
    return Fallback(fused, "model output was not valid JSON for the schema");
// validate labels and grades, sort by grade then fused rank (omitted here)

Question 7: Did a slice get worse while the average got better?

This is the one I would have missed without a golden set. On 36 answerable judged queries (12 exact lookups, 12 paraphrases, 12 mixed), recall@10 by slice looked like this:

mode             R@10 | exact  paraphrase  mixed
Lexical (BM25)  0.875 | 1.000       0.625  1.000
Vector          0.972 | 1.000       1.000  0.917
Hybrid (RRF)    0.972 | 1.000       0.917  1.000
Hybrid + rerank 0.986 | 1.000       0.958  1.000

Hybrid matches vectors overall, yet it's worse on paraphrases. Fusing in a keyword list with nothing useful to say pushes good vector hits down. So the CI gate checks every slice, not only the average, and the API answers GET /eval/gate with 200 or 422 plus reasons. One caveat: the corpus is synthetic and the project runs on offline stand-in models, so these numbers show how to measure, not how good any real embedding model is.

The checklist version

• [ ] Every hit can explain itself: leg, rank, raw score, fused part
• [ ] Scores are never added across legs; RRF uses ranks with deterministic ties
• [ ] One analyzer for documents and queries, compound tokens kept whole
• [ ] One filter expression for both legs, with a test that proves it
• [ ] Candidate depth is deeper than the final list; weight-zero legs are dropped
• [ ] The reranker grades, your code sorts, and every failure falls back to fusion
• [ ] The CI gate checks each query slice, not just the average

There's an eighth question I left out on purpose: what should the pipeline do when no chunk deserves to win? Both legs always return something, so the reranker picks the least-bad chunk and the model answers anyway. The project has an answer for that, calibrated so it doesn't break the day you change embedding models, and the PDF walks through it.

Get the complete project

Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10. This project is already inside: the full .NET 10 solution with the analyzer, the BM25 index, RRF fusion with explanations, the shared access filter, the guarded LLM reranker, the golden-set evaluator with its per-slice CI gate, the "not in the docs" answer, a minimal API, an 11-scene demo and 212 passing tests (190 unit, 22 integration), none of which needs an API key. Plus a PDF guide that goes much deeper than this post. You also get the other complete, tested .NET + AI projects in the membership, with new ones added regularly.

Join Tech Skill Builder for $5/month

Which of these seven questions would your RAG search fail first?

Comments

[Ahmet Özel]

Keeping the contribution list through fusion makes debugging much more actionable. I would extend that explanation with the corpus revision, analyzer version, and whether the reranker actually ran or fell back. Otherwise two identical contribution lists can still describe different systems after an index rebuild or a model timeout.

Your deterministic tie-breaks help here: an incident trace should be replayable against a frozen snapshot. For the paraphrase regression, that would let you distinguish a harmful lexical contribution from a changed candidate pool, instead of attributing both to the fusion formula.