Cross-Encoder Reranker Score Calibration: Why 0.5 Cutoffs Fail

Your support bot has an abstain gate: if the reranker's top score is below 0.5, answer "I couldn't find that in the docs." It worked for months. You bump the reranker checkpoint, and overnight the abstain rate goes from 4% to 31%. Retrieval didn't change. Recall@50 is identical. The gold passage is still sitting at rank 1 for almost every failing query — it just scores 0.38 now.

That's not a retrieval bug. That's cross-encoder reranker score calibration, and the uncomfortable part is that the number you thresholded on was never a probability of relevance in the first place. It was a per-query-arbitrary logit that happened to land near 0.5 on your eval set.

TL;DR

• Most rerankers are trained with a listwise softmax loss over a candidate group, which is invariant to adding any constant to every score in that group. The training signal never constrains the absolute level — only the ordering within one query's candidate list.
• So a score of 0.7 for query A and 0.7 for query B mean nothing in common. Fixed thresholds, score-weighted fusion, and "keep everything above X" top-k all silently break across query distributions.
• Diagnose it by comparing the ROC AUC of the raw top-1 score against within-query statistics — the rank1−rank2 margin, or the z-score of rank 1 against the candidate list. The within-query features usually win by a wide margin.
• The cheapest production fix: inject 3–5 fixed anchor passages into every rerank request and threshold on score(doc) − median(score(anchors)). This cancels exactly the per-query offset the loss left free.
• Keep the candidate list size constant. Softmax-normalized scores depend on N, so reranking top-20 sometimes and top-50 other times reintroduces the same incomparability you just removed.

Why aren't cross-encoder reranker scores comparable across queries?

Because the standard training objective is shift-invariant per query. Modern rerankers (the bge-reranker family, most jina/mxbai-style cross-encoders, and essentially every in-batch-negative recipe) are trained with cross-entropy over a group of one positive and k sampled negatives for the same query:

L(q) = -log( exp(s_pos) / Σ_j exp(s_j) )

Now add any constant c to every score in that group. exp(s+c) factors out of numerator and denominator, c cancels, and the loss is bit-identical. The gradient never sees it. The model is free to place query A's whole score distribution at −4 and query B's at +2, and the objective is perfectly happy.

What sets that offset in practice? Whatever surface features correlate with the query encoder's activations: query length, whether it's a keyword fragment or a full sentence, language, domain vocabulary, how close the phrasing is to the training distribution. None of these are "how relevant is the best document."

Pointwise BCE-trained rerankers (monoBERT-style) aren't safe either. They do supervise an absolute level, so they're better. But the target they're calibrated to is P(relevant | q, d, d ~ training negative sampler). Your candidate generator is not that sampler. If your first-stage retriever got better, the negatives at inference become harder than the ones seen in training, the whole score distribution slides down, and your threshold rots. Calibration is a property of the joint pipeline, not the reranker alone.

This is why at least one major hosted rerank API states in its docs that relevance scores are only meaningful within a single request. It's not hedging — it's the loss function.

What actually breaks in production?

Four things, roughly in order of how often I've seen them bite:

1. Abstain / "no answer found" gates. The failure is asymmetric and mean. Easy, well-phrased queries get high offsets and sail through. Terse, jargon-heavy, or non-English queries — exactly the ones where you most want retrieval to work — get low offsets and get refused. You ship a system that quietly serves your fluent users and stonewalls everyone else.

2. Score-weighted hybrid fusion. final = 0.7 * rerank + 0.3 * bm25_normalized assumes both terms live on a stable scale. The reranker term's per-query offset means the effective weight drifts query to query. (This is one reason rank-based fusion is the default in most stacks — it only consumes ordering.)

3. Adaptive top-k by threshold. "Send every passage above 0.6 to the model" sounds efficient. In reality your context length swings between 1 and 40 passages depending on the query's offset. That's unbounded cost variance, and on the high end you're padding the context with junk that measurably degrades answer quality.

4. Monitoring. A dashboard alerting on mean reranker score is an alert on your query mix, not your retrieval health. Every marketing campaign that shifts query phrasing will page you.

How do you diagnose miscalibration in your own reranker?

Compare how well the raw score separates answerable from unanswerable queries versus how well within-query statistics do. If the within-query features win, your scores carry an offset you're not modeling.

Take a labeled set of queries, run your real first-stage retriever, rerank a fixed number of candidates, and keep the raw logits — not the sigmoid. Sigmoid is monotone so it doesn't change ranking, but it compresses exactly the tails where your threshold lives, turning a 2-logit gap into a 0.01 probability gap.

import numpy as np
from scipy.special import logsumexp
from sklearn.metrics import roc_auc_score

# scores[i]: raw logits for query i's candidate list (fixed N, e.g. 50)
# y[i]:      1 if a gold passage is in that candidate list, else 0

def feats(row):
    s = np.sort(np.asarray(row))[::-1]
    return {
        "top1":    s[0],                                # absolute — the naive gate
        "margin":  s[0] - s[1],                         # within-query, shift-invariant
        "z":       (s[0] - s.mean()) / (s.std() + 1e-6),# within-query, scale-invariant too
        "lse_gap": s[0] - logsumexp(s[1:]),             # top vs. the rest of the list
    }

table = [feats(r) for r in scores]
for k in ("top1", "margin", "z", "lse_gap"):
    col = np.array([t[k] for t in table])
    print(f"{k:8s} AUC={roc_auc_score(y, col):.3f}")

The tell: top1 lags, margin and z are close together and clearly ahead. If instead top1 wins, congratulations — your reranker was trained pointwise on something close to your own query distribution, and a fixed threshold is defensible (until the next checkpoint).

Also plot top1 for answerable queries only, bucketed by query length or language. If those buckets have visibly different medians, you've found your offset.

How do you calibrate cross-encoder reranker scores without labels?

Give the model a per-query reference point. Inject a small fixed set of anchor passages — well-formed, fluent, and reliably off-topic for your domain — into every rerank request, then threshold on the difference. The anchors' scores estimate that query's "irrelevant" baseline, which is precisely the free constant the loss never pinned down.

import numpy as np
import torch
from sentence_transformers import CrossEncoder

model = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)

# Fluent, grammatical, and off-topic for THIS corpus. Freeze these forever;
# changing them invalidates every threshold downstream. Version them.
ANCHORS = [
    "The Baltic dry index rose 3% on stronger iron ore shipments out of Brazil.",
    "Sourdough starters need a 1:1:1 feeding ratio to stay active at room temperature.",
    "Regulation nine covers the offside rule and its 2005 amendment.",
    "Annual rainfall in the Atacama averages under 15 millimetres.",
]

N_CANDIDATES = 50  # keep this constant across every call

def calibrated(query, candidates):
    docs = candidates[:N_CANDIDATES]
    pairs = [(query, d) for d in docs] + [(query, a) for a in ANCHORS]
    # identity activation -> raw logits. The default sigmoid squashes the tails.
    logits = model.predict(pairs, activation_fct=torch.nn.Identity(),
                           batch_size=32, convert_to_numpy=True)
    cand, anch = logits[:len(docs)], logits[len(docs):]
    baseline = np.median(anch)          # median, not mean: one anchor may go stale
    return cand - baseline              # shift-invariant, comparable across queries

scores = calibrated(user_query, retrieved)
keep = [d for d, s in zip(retrieved, scores) if s > CALIBRATED_GATE]

Cost is four extra pairs on top of fifty — under 10% more reranker compute, and it batches into the same forward pass. Two rules keep it honest:

• Anchors must be fluent. Random character noise scores low for a degenerate reason (it's not text), not because it's irrelevant. You'd be measuring the wrong baseline.
• Anchors are part of your threshold contract. Swap a passage, retune the gate. Store the anchor set hash next to the threshold in config.

If you'd rather avoid the extra pairs entirely, margin or z from the diagnostic above are free — they use candidates you already scored. The tradeoff: margin conflates "nothing is relevant" with "twelve things are equally relevant," so it misfires on queries with many valid answers. Anchors don't have that failure mode. In a FAQ corpus with heavy near-duplication, use anchors.

What if you genuinely need a probability?

Then calibrate — but calibrate the right variable. Pooling every (query, doc) pair into one Platt or isotonic fit is the standard mistake: it averages over the per-query offset instead of removing it, so the fitted curve is only correct for the average query.

Fit the calibrator on the shift-invariant statistic instead:

from sklearn.isotonic import IsotonicRegression

# x: calibrated margins (anchor-adjusted top-1) from a labeled dev set
# y: 1 if that query was actually answerable from the candidate list
cal = IsotonicRegression(out_of_bounds="clip").fit(x, y)
p_answerable = cal.predict([calibrated_top1])[0]

Now p_answerable > 0.8 is a claim you can actually reason about, and it composes with other signals.

A useful complement under distribution shift: set the gate as a rolling quantile rather than a constant. "Abstain on the bottom 12% of queries by calibrated margin over the last 10k requests" holds your abstain rate fixed by construction. The assumption you're buying is that the true answerable rate is roughly stationary — which fails during incidents and product launches, so alert on the quantile value moving even while the rate stays pinned.

Does this apply to LLM rerankers too?

Yes, and worse. If you score passages with the log-prob of a "yes" token from Claude Sonnet 4.x or GPT-5.x, that log-prob is sensitive to the prompt template, the passage length, and the model's general verbosity priors — it's uncalibrated in every direction at once, and it moves when the provider updates the model behind a stable alias. Listwise LLM rerankers (RankGPT-style) don't even emit scores; they emit a permutation, so a within-list statistic is the only thing available. Anchor passages work there too, and they double as a sanity probe: if the model ranks an anchor about the Atacama above your real docs, the prompt is broken, not the corpus.

The short answer

Cross-encoder reranker scores are not calibrated probabilities and are not comparable across queries, because the listwise softmax loss they're trained with is invariant to a per-query constant offset — the objective only supervises ordering inside one candidate list. A fixed 0.5 cutoff therefore abstains at a rate that depends on query phrasing, length, and language rather than on whether an answer exists, and it silently rots on every checkpoint bump. Fix it by deciding on within-query statistics instead: the rank1−rank2 margin, a z-score against the candidate list, or — best — the top score minus the median score of a frozen set of off-topic anchor passages injected into every request. Keep the candidate count constant, fit any probability calibration on the shift-invariant statistic rather than on pooled raw scores, and store the threshold alongside the model version and anchor-set hash it was tuned against.