A RAG system always gets something back from the index. Ask it about cooking and it returns the five nearest chunks of API docs, and the LLM then tries to be helpful. So: can a simple score threshold on retrieval tell "the docs cover this" from "they don't"?
Short answer: yes, mostly. On the FastAPI docs, the top-1 cosine score separated the two groups well (AUC 0.986), and a threshold of 0.40 refused 24 of 30 unanswerable questions while answering 29 of 30 answerable ones. The leftovers are near-domain questions, which no threshold fully fixes. Raw BM25 scores did worse, and BM25 almost never returns an empty result.
Setup
- Corpus: 152 files of the FastAPI docs, split into 2,808 heading-aware chunks with the heading path prefixed.
- Retrievers: MiniLM-L6-v2 cosine similarity, and BM25 over the same chunks.
- 30 answerable queries: hand-written questions, each matching one docs page (for example "How do I set a cookie on the response?").
- 30 unanswerable queries: 15 near-domain (pagination, rate limiting, gRPC, Terraform, Prometheus, Selenium...) and 15 unrelated (egg boiling, football, tides...). I checked by keyword search that the near-domain topics are not documented pages, though a few terms are mentioned in passing.
- Decision score: the top-1 chunk's score. Refuse when it is below the threshold.
Results
Top-1 cosine score by group:
| Group | Median | Min | Max |
|---|---|---|---|
| Answerable (30) | 0.619 | 0.393 | 0.818 |
| Unanswerable, near-domain (15) | 0.385 | 0.256 | 0.508 |
| Unanswerable, unrelated (15) | 0.218 | 0.175 | 0.298 |
How well each score separates answerable from unanswerable (AUC, 1.0 is perfect, 0.5 is chance):
| Signal | AUC |
|---|---|
| Cosine top-1 | 0.986 |
| BM25 top-1, stopwords removed | 0.927 |
| BM25 top-1, raw | 0.910 |
| Cosine gap between top-1 and top-2 | 0.649 |
Cosine threshold sweep:
| Threshold | Answerable kept | Near-domain refused | Unrelated refused | All unanswerable refused |
|---|---|---|---|---|
| 0.30 | 100% | 13% | 100% | 57% |
| 0.35 | 100% | 20% | 100% | 60% |
| 0.40 | 97% (29/30) | 60% | 100% | 80% (24/30) |
| 0.45 | 93% (28/30) | 73% | 100% | 87% (26/30) |
| 0.50 | 90% (27/30) | 93% | 100% | 97% (29/30) |
| 0.55 | 73% | 100% | 100% | 100% |
I also split each group in half, picked the threshold on one half and scored the other. Requiring every answerable calibration query to be kept gave a threshold of 0.393. On the held-out half it kept 15 of 15 answerable queries and refused 11 of 14 unanswerable ones (79%).
What the numbers mean
- Unrelated questions are easy. Their best score (0.298) is below the worst answerable score (0.393). Any threshold between 0.30 and 0.39 catches all of them.
- Near-domain questions are the real problem. Their scores run up to 0.508, which overlaps the lowest answerable queries. The highest-scoring one was "How do I expose Prometheus metrics from my service?" at 0.508, matched to the OpenTelemetry page. Arguably that page is a partial answer, so this one is a labeling grey zone as much as a miss.
- The cost of tightening is visible. Going from 0.40 to 0.50 refuses 5 more unanswerable queries but also loses 2 more answerable ones.
- BM25 is a weaker gate. Its raw scores depend on query length and term rarity, so one cutoff does not transfer. It also never returned an empty candidate list for any of the 60 queries, because words like "how" and "do" match something. With stopwords removed, only 5 of the 30 unanswerable queries came back empty. "Check for empty results" alone is not a refusal strategy.
- The top-1 vs top-2 gap is nearly useless here (AUC 0.649).
Caveats
- 60 queries is small. One query moves a percentage by about 3 points.
- I wrote both query sets myself, and the unanswerable ones are cleaner than real traffic.
- Thresholds are specific to this model and corpus. Recalibrate whenever you change the embedding model or re-chunk.
- This tests retrieval only. A model can still hallucinate when the retrieved chunks are on topic but do not contain the answer; that needs an answer-level check such as citation validation.
Copyable code
import numpy as np
def should_refuse(query_vec, chunk_matrix, threshold=0.40):
"""Vectors must be L2-normalised. Returns (refuse, top_score, top_index)."""
sims = chunk_matrix @ query_vec
top = int(np.argmax(sims))
return bool(sims[top] < threshold), float(sims[top]), top
REFUSAL = "I couldn't find this in the documentation."
Pick the threshold from your own labeled set: take the highest value that still answers at least 95-100% of your known-answerable queries, then check what it refuses.
Rule of thumb
- Build a set of at least 50 answerable and 50 unanswerable queries from your own logs, including near-domain ones.
- Use the top-1 cosine score, not BM25, as the gate, and set the threshold just under your lowest answerable score.
- Put the threshold in a test, so a model or chunking change that moves the score scale fails CI.
- Expect to refuse most, not all, of the near-domain questions. Treat the rest with answer-level checks.
The script and raw output are on the original post.
I run experiments like this every week at RecallRun. If you want to build RAG systems end to end, the AI Engineering course starts free: https://recallrun.dev/learn/ai-engineering/?utm_source=devto&utm_campaign=rag-idk
Top comments (1)
Really useful to see the near-domain group broken out separately, since that's where every threshold I've tried falls apart too. One thing that helped me there: when the top-1 score lands in the gray zone (say 0.38 to 0.45 in your numbers), have the model answer only if it can quote a sentence from the retrieved chunk that addresses the question, and refuse otherwise. It turns the borderline cases into a check on the text instead of the score. Did you look at whether the gap between top-1 and top-2 scores adds anything on top of top-1 alone?