Part 5 of Retrieval-Augmented Self-Recall. Code: RE-call. Part 4: the eval harness.
I shipped the gap_warning guard from Part 3 with a sensible-looking default: if the best cosine similarity is below 0.50, call it a probable gap and abstain. I tested it. It worked.
Then I swapped the embedding model. Every question that should have been refused came back as a confident answer.
No error. No crash. No failed test. The guard was still there, still running, still reporting that everything was fine. It had just quietly stopped being a guard.
Here's the mechanism, because it generalises well past my project, and there's a decent chance it's live in your RAG system right now.
Cosine similarity is not calibrated across models
The gap guard fires when best_cosine < threshold. I used 0.50. The problem is that 0.50 means completely different things to different embedders, because each model lays out its vector space with its own geometry. Same number, different meaning.
The harness measured the cosine distributions for answerable vs. unanswerable queries, per embedder. Look at what "similarity" actually ranges over:
| Embedder | Answerable cosine | Unanswerable cosine | Separable? | FCR @ 0.50 | FCR @ calibrated |
|---|---|---|---|---|---|
hashing-64 |
0.30 – 0.68 | 0.35 – 0.53 | no — overlap | 0.20* | — |
bge-small |
0.70 – 0.90 | 0.51 – 0.64 | yes, at ~0.70 | 1.00 | 0.00 |
voyage-3 |
0.53 – 0.70 | 0.09 – 0.32 | yes, at ~0.50 | 0.00 | 0.00 |
* misleadingly low: with overlapping distributions the 0.50 cut also wrongly flags answerable queries, and the error-minimizing threshold simply stops firing at all. No threshold works here.
Read across the rows and the whole story is there.
voyage-3: unanswerable queries score 0.09–0.32, answerable score 0.53–0.70. A threshold of 0.50 lands cleanly in the gap between them. FCR is 0.00. My default worked — by accident. Voyage's geometry just happens to put the boundary near 0.50.
bge-small: unanswerable queries score 0.51–0.64 — entirely above my 0.50 threshold. So the guard, which only fires below 0.50, never fires on them at all. Result: FCR 1.00. Every unanswerable query was confidently answered. The guard was switched off, and nothing told me. Recalibrate the threshold to ~0.70 and FCR drops to 0.00 — the distributions are separable, I was just cutting in the wrong place.
hashing-64: answerable (0.30–0.68) and unanswerable (0.35–0.53) overlap. No threshold separates them, because the embedder is too weak to distinguish "relevant" from "vaguely near." The right lesson here isn't "pick a better threshold" — it's "this embedder can't support abstention at all."
The lesson: never ship a hard-coded abstention threshold
A magic constant that "works" is the most dangerous kind of code, because it works right up until the context shifts — and then it fails silently, which is the worst possible failure mode for a safety guard. My 0.50 wasn't a good threshold that I'd validated. It was a coincidence that held for one embedder and collapsed the moment I changed one.
The fix is cheap and non-negotiable: calibrate the abstention threshold per embedding model, against a small labeled set. Twenty-odd queries — a handful answerable, a handful not — is enough to see where the two distributions actually sit and cut between them. Do NOT ship a constant. The writeup says it in one line: calibrate per embedding model against a small labeled set; do not ship a hard-coded constant.
Why this generalizes past RAG
Any decision that thresholds a similarity score inherits this exact trap:
- semantic caching ("is this query close enough to a cached one?")
- near-duplicate / dedup detection
- "is this document relevant enough to include?"
- clustering cutoffs, entity-matching thresholds
In every one of these, the threshold is a property of the (embedder, corpus) pair, not a universal constant. Swap the model and your carefully-chosen number is now cutting in the wrong place — and unless you're measuring the failure explicitly, you won't know.
Which is the meta-point, and the reason Part 4 mattered: this failure is invisible without the eval harness. The system ranks well, returns plausible results, and lies on gaps. You only catch it by measuring a false-confident rate per embedder. Ranking metrics would have shown me green the entire time.
Two footnotes from after this was drafted. First: when I trailed this finding in Part 1, a commenter guessed the mechanism before the post existed — and proposed a relative threshold (top hit versus the rest of the batch) instead of an absolute one. That experiment is still owed; my worry is that a spread-based check is blind to the single confident distractor. Second, and worse: there is a whole failure class no threshold can catch, by construction — the near-miss that scores high. That one needed a different kind of fix, and it's the subject of the follow-up post.
Next
If a better threshold helps, would a better embedding help more? The intuitive next move is to fine-tune the embedder on my domain. I did. The result was zero — and Part 6 is about why that was exactly the right outcome, and the one case where fine-tuning actually did move the needle.
Part 5 of Retrieval-Augmented Self-Recall. Code: RE-call. The "measure the failure that matters, not the one that flatters" discipline runs through Claude Code, Beyond the Prompt too.
Top comments (0)