LLM Logprob Calibration: Why RLHF Makes Confidence Useless

You ship a router: if the model's confidence is below 0.85, escalate to a human reviewer. On your fine-tuned base model the gate fired on a healthy slice of traffic and caught most of the errors. You swap in a frontier instruction-tuned model, accuracy goes up several points — and the gate stops firing. Nearly every response comes back above 0.99, including the wrong ones.

Nothing is broken in your code. The model got better and its logprob calibration got worse, and those two facts are causally linked. RLHF and DPO are, mathematically, a learned sharpening of the output distribution. Sharpening is exactly what destroys calibration.

TL;DR

• Logprob calibration means: among all predictions the model assigns probability p, a fraction p should be correct. Base (pre-trained) LLMs are usually close to this on multiple-choice tasks. RLHF'd chat models are not.
• The closed-form optimum of the KL-regularized RLHF objective is π*(y|x) ∝ π_ref(y|x)·exp(r(x,y)/β). Exponentiating a reward is anti-temperature-scaling: it concentrates mass. The GPT-4 technical report showed calibration degrading by roughly an order of magnitude in ECE from the pre-trained to the post-RLHF model on MMLU.
• Use a single constrained answer token and renormalize over the label set. Never use mean sequence logprob as confidence — it mixes length, tokenization, and surface-form dispersion into your score.
• Fix it post-hoc with temperature scaling: one scalar T fit by NLL on a few hundred held-out labeled examples. Expect T > 1. Refit per model snapshot and per task distribution.
• No logprobs at all (Claude Opus 4.x / Sonnet 4.x Messages API)? Use k-sample self-consistency at temperature=1 and take the modal-vote frequency. It is a real empirical probability, and it beats asking the model "how confident are you?"

What does logprob calibration actually measure?

Calibration is not accuracy. A model that answers correctly 70% of the time and reports 0.70 on every prediction is perfectly calibrated and mediocre. A model that is right 95% of the time but reports 0.999 on everything is more accurate and useless for routing, because you cannot pick a threshold that separates its wins from its losses.

The standard metric is Expected Calibration Error: bin predictions by confidence, and in each bin take the gap between average confidence and empirical accuracy.

import numpy as np

def ece(confidences, correct, n_bins=15):
    conf = np.asarray(confidences, dtype=float)
    acc  = np.asarray(correct,  dtype=float)
    edges = np.linspace(0.0, 1.0, n_bins + 1)
    total, n = 0.0, len(conf)
    for lo, hi in zip(edges[:-1], edges[1:]):
        m = (conf > lo) & (conf <= hi)
        if m.sum() == 0:
            continue
        total += (m.sum() / n) * abs(acc[m].mean() - conf[m].mean())
    return total

Two things to watch. ECE with equal-width bins is degenerate when 95% of your predictions land in the top bin — you are measuring one number with a lot of ceremony. Use equal-mass bins, or report the reliability curve. And ECE ignores direction: over- and under-confidence cancel across bins. Log the signed gap per bin too.

Why does RLHF break logprob calibration?

Because sharpening the policy is the objective, not a side effect. The KL-regularized RLHF problem

max_π  E_{y~π}[ r(x,y) ] - β · KL(π(·|x) || π_ref(·|x))

has the closed-form solution

π*(y|x) = (1/Z(x)) · π_ref(y|x) · exp( r(x,y) / β )

That is the reference distribution reweighted by an exponentiated reward. In logit space you are adding r/β to every sequence's log-probability. Small β means aggressive reweighting; as β → 0 the policy collapses toward argmax r. Temperature scaling divides logits by T > 1 to flatten an overconfident model. RLHF does the opposite, with an input-dependent, learned amount of sharpening. DPO inherits this exactly — it is derived from the same optimum, with the reward reparameterized as the log-ratio of policy to reference.

Two more contributors compound it:

SFT trains against one-hot targets. For a question with fifty acceptable paraphrases, cross-entropy pushes mass toward the single curated continuation. The model learns that one surface form is the answer.

Preference data carries no probability supervision. A pairwise label says "B beat A." It never says "B is right 70% of the time." There is no gradient signal anywhere in post-training that rewards honest uncertainty, so nothing preserves it.

The result is a model whose ranking of answers improves while the magnitude of its probabilities becomes meaningless. That is the failure mode: your top-1 is more often right, and your top-1 probability tells you nothing about when it isn't.

How do you extract a usable probability from the API?

Constrain the answer to one token, then renormalize the logprobs over your label set. Do not read confidence off free-form prose.

import math
from openai import OpenAI

client = OpenAI()
LABELS = ["A", "B", "C", "D"]

def label_probs(question: str, model="gpt-5.1"):
    r = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system",
             "content": "Answer with exactly one character: A, B, C, or D. No other text."},
            {"role": "user", "content": question},
        ],
        max_completion_tokens=1,
        logprobs=True,
        top_logprobs=20,        # must exceed the label-set size with room to spare
    )
    top = r.choices[0].logprobs.content[0].top_logprobs
    raw = {t.token.strip(): t.logprob for t in top}

    # Renormalize over the label set only — the tail is not your problem.
    lps = {L: raw.get(L, -100.0) for L in LABELS}
    m = max(lps.values())
    z = sum(math.exp(v - m) for v in lps.values())
    return {L: math.exp(v - m) / z for L, v in lps.items()}

Three details that bite people:

• Renormalization is not optional. If the model leaks 3% of its mass onto "The" or a leading space, the raw probability of "A" understates the model's actual preference among the options you care about. Renormalize, then calibrate.
• Token identity matters. "A" and " A" are different tokens. Strip and merge, or your mass silently splits.
• top_logprobs truncates. If a label falls outside the returned top-k you get a floor value, not the truth. Keep the label set small and the k generous.

Why is sequence logprob a worse confidence signal than a single answer token?

Because a sequence probability answers a different question. P(y|x) for a 40-token answer is the probability of that exact string, not of the claim it expresses. Three confounds stack up:

Length. Sum-logprob is monotonically decreasing in length, so longer answers always look less confident. Mean-logprob overcorrects, rewarding a model for padding with high-probability filler tokens like the and ,.

Tokenization. The same semantic answer split into 3 tokens versus 5 tokens gets a different score. You are partly measuring the BPE merge table.

Surface-form dispersion. This is the subtle one, and it points the opposite direction from RLHF overconfidence. If the model is 90% sure the answer is Paris but spreads that mass across "Paris", "It's Paris", and "The capital is Paris", any single string's probability understates semantic confidence. Token-level scores are overconfident; sequence-level scores over many valid phrasings are underconfident. Mixing them produces noise, not a signal.

If you need free-form output and a confidence score, generate the answer and then score a separate one-token verification turn, or use self-consistency below. Do not divide one number by a token count and call it confidence.

How do you fix logprob calibration without retraining?

Temperature scaling — one scalar, fit by minimizing NLL on held-out labeled data. It is the same technique from Guo et al. (2017) for image classifiers, and it works here for the same reason: it rescales logits without changing their order, so accuracy is provably unchanged while ECE drops.

import numpy as np
from scipy.optimize import minimize_scalar

def fit_temperature(logits, y_true):
    """logits: (N, K) renormalized label logits.  y_true: (N,) int indices."""
    logits = np.asarray(logits, dtype=float)

    def nll(T):
        z = logits / T
        z -= z.max(axis=1, keepdims=True)
        logZ = np.log(np.exp(z).sum(axis=1))
        return float(-(z[np.arange(len(y_true)), y_true] - logZ).mean())

    res = minimize_scalar(nll, bounds=(0.05, 10.0), method="bounded")
    return res.x   # T > 1 => the model was overconfident

def apply_temperature(logits, T):
    z = np.asarray(logits, dtype=float) / T
    z -= z.max(axis=1, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=1, keepdims=True)

A few hundred labeled examples is usually enough to fit one parameter. Fit on a held-out split, evaluate ECE on a third split, and refit whenever the model snapshot, the prompt template, or the task distribution changes. Calibration does not transfer across any of those — a T fit on support-ticket triage will not hold on contract clause classification.

If a single T leaves structural bias (say, option A is systematically overweighted regardless of content), move to vector scaling: per-class weight and bias fit the same way. That does change the ranking, so re-check accuracy.

What do you do when the model has no logprobs (Claude Opus 4.x)?

Anthropic's Messages API does not return token logprobs, so the trick above is unavailable for Claude Opus 4.x and Sonnet 4.x. Use self-consistency instead: sample k answers at temperature=1 and use the modal vote's frequency as the probability.

That frequency is a genuine empirical estimate of the model's own answer distribution, and it is typically better calibrated than verbalized confidence — asking a model "rate your confidence 0-100" gets you a number generated by the same sharpened policy, and it clusters at 90 and 95 no matter what.

Two practical constraints. First, resolution is 1/k: at k=8 your confidence lives on an 8-point grid, so a 0.85 threshold and a 0.87 threshold are the same threshold. Pick k from the granularity you actually need. Second, cost — put the shared instructions and few-shot block behind a cache_control breakpoint so the k calls read the prefix from cache instead of reprocessing it:

system=[{
    "type": "text",
    "text": LONG_RUBRIC_AND_FEWSHOTS,
    "cache_control": {"type": "ephemeral"},
}]

Then fit temperature scaling on top of the vote frequencies exactly as above — treat log(count_i + 0.5) as pseudo-logits (the smoothing keeps zero-vote classes finite).

Where does this still fail?

Prompt-format sensitivity. Reordering multiple-choice options changes the probabilities. Calibrate on the exact template you ship, and if option position bias is large, average over shuffles before calibrating.

Distribution shift. Temperature scaling assumes your calibration set matches production. It does not fix out-of-domain overconfidence — a model confidently wrong about an unseen category stays confidently wrong, just slightly less loudly.

Silent recalibration on the provider's side. A model version bump can move T substantially. Pin the snapshot, and monitor ECE on a labeled canary slice as a production metric, not a one-time evaluation.

The short answer

RLHF makes LLM confidence useless because the KL-regularized objective it optimizes has a closed-form solution that exponentially reweights the reference distribution — a learned, input-dependent sharpening that is the mathematical inverse of temperature scaling. The model's ranking of candidate answers improves while the magnitudes of its probabilities collapse toward 1, so top-1 logprob stops separating correct from incorrect predictions. Fix it by constraining the answer to a single token, renormalizing over the label set, and fitting one temperature parameter by NLL on a few hundred held-out examples — refit per model snapshot and per task. When logprobs are unavailable, k-sample self-consistency at temperature=1 gives you a real empirical probability to calibrate instead.