You built a classifier on GPT-5.1. It hits 92% accuracy on your eval set. Ship it, and the 8% it gets wrong land in production with the same swagger as the 92% it gets right. So you add "confidence": 0-100 to the JSON schema, the model dutifully returns 95 on almost everything, and you now have a number that correlates with nothing. That number is theater. The signal you actually want is already in the API response — the token logprobs.
TL;DR
- Asking an LLM for a self-reported confidence score gives you verbalized confidence: badly calibrated, overconfident, and clustered on round numbers like 90/95/100.
- Token logprobs — the log-probability the model assigned to each token it emitted — are a real, per-decision confidence signal you can extract from the API for free.
- For a classifier, read the logprob of the label token,
exp()it to a probability, then apply temperature scaling on a held-out set to make it calibrated (low ECE). - Use the calibrated probability for selective prediction: auto-accept above a threshold, route the low-confidence tail to a human or a bigger model.
- Claude doesn't expose token logprobs, so approximate confidence there with self-consistency (sample N, measure agreement).
Why is asking an LLM for a confidence score a bad idea?
Because you get verbalized confidence, and verbalized confidence is not calibrated. When you prompt "rate your confidence 0-100," the model generates that number the same way it generates any other token: by pattern-matching what a confident-sounding assistant says. The result is systematically overconfident, spikes on human-friendly round numbers (90, 95, 99), and barely moves between an easy example and a genuinely ambiguous one.
Calibration has a precise meaning. A classifier is calibrated if, among all predictions it makes with confidence p, a fraction p are actually correct. A self-reported "95" that is right 70% of the time is not "pretty good" — it is a broken instrument. You measure this with Expected Calibration Error (ECE): bin predictions by stated confidence, and average the gap between confidence and empirical accuracy in each bin. Verbalized confidence routinely posts high ECE. It feels like a signal and behaves like noise.
What are token logprobs and how do I get them?
A token logprob is the natural log of the probability the model assigned to a specific token at the moment it sampled it. When the model emits positive for a sentiment label, it computed a distribution over the whole vocabulary first; the logprob tells you how much mass sat on the token it chose. exp(logprob) converts that back to a probability in [0, 1]. That probability is the model's internal confidence in that token — no self-report, no prompt engineering, no theater.
On the OpenAI-compatible chat completions endpoint you turn this on with two parameters. logprobs=True returns the chosen token's logprob; top_logprobs=n also returns the top n alternatives at each position, so you can see what the model almost said.
from openai import OpenAI
import math
client = OpenAI()
def classify(text):
resp = client.chat.completions.create(
model="gpt-5.1",
messages=[
{"role": "system",
"content": "Classify sentiment. Answer with exactly one "
"word: positive, negative, or neutral."},
{"role": "user", "content": text},
],
max_tokens=1,
temperature=0,
logprobs=True,
top_logprobs=5,
)
tok = resp.choices[0].logprobs.content[0]
label = tok.token.strip().lower()
prob = math.exp(tok.logprob) # confidence in THIS label
# full distribution over the alternatives the model considered
dist = {t.token.strip().lower(): math.exp(t.logprob)
for t in tok.top_logprobs}
return label, prob, dist
label, prob, dist = classify("shipping was slow but the product is great")
print(label, round(prob, 3), dist)
Two design choices make this clean. Force the label into a single token so one logprob captures the whole decision, and pin temperature=0 so decoding is deterministic while the logprobs still reflect the underlying distribution. top_logprobs is the bonus: when the model returns positive at 0.55 with negative at 0.42 right behind it, you know the example is a coin flip long before a human ever sees it.
How do I turn a raw logprob into a calibrated probability?
Read the label token's logprob, exponentiate it, then correct the model's overconfidence with temperature scaling — a single learned parameter fit on held-out data. Raw LLM logprobs are usually sharper than reality: the model puts 0.97 on answers that are right 85% of the time. Temperature scaling fixes the sharpness without touching accuracy.
The mechanics: collect logits (or logprobs) on a labeled held-out set, then find a scalar T that minimizes negative log-likelihood when you divide logits by T before the softmax. T > 1 softens overconfident distributions; T < 1 sharpens underconfident ones. Because it is one parameter, it needs only a few hundred labeled examples and cannot overfit its way into hurting your top-1 accuracy — the argmax is invariant to scaling.
import numpy as np
from scipy.optimize import minimize_scalar
# logits: (N, K) pre-softmax scores; here, reconstruct from top_logprobs.
# labels: (N,) correct class indices on a held-out set.
def fit_temperature(logits, labels):
def nll(T):
z = logits / T
z = z - z.max(axis=1, keepdims=True)
logp = z - np.log(np.exp(z).sum(axis=1, keepdims=True))
return -logp[np.arange(len(labels)), labels].mean()
return minimize_scalar(nll, bounds=(0.05, 10), method="bounded").x
T = fit_temperature(val_logits, val_labels) # e.g. 1.7
After fitting, apply the same T at inference. Verify it worked by recomputing ECE on a separate test split and plotting a reliability diagram — confidence on the x-axis, empirical accuracy on the y-axis. A calibrated model hugs the diagonal. If it does, your 0.9 predictions really are right ~90% of the time, which is the entire point.
What if my labels are more than one token?
Multi-token labels break the "one logprob = one decision" shortcut, so either avoid them or aggregate correctly. The cleanest fix is to design label surface forms that tokenize to a single token — pos / neg / neu, or route through a structured-output schema where the discriminating field is a single enum token. Then nothing changes.
When you can't avoid multi-token labels (refund_request, billing_dispute), the sequence probability is the product of per-token probabilities, so sum the logprobs. But raw sums penalize longer labels — a three-token class looks less likely than a one-token class purely from length. Use length-normalized log-probability (the mean per-token logprob) when comparing candidate labels of different lengths, and evaluate a small fixed set of candidates rather than trusting free-form generation. For a closed label set, scoring each candidate's normalized logprob and taking the argmax is more robust than reading whatever the model happened to decode first.
How do I get confidence from Claude, which has no logprobs?
Claude's API does not expose token logprobs, so you approximate confidence behaviorally with self-consistency: sample the same prompt N times at nonzero temperature and measure agreement. If Claude Sonnet 4.5 returns positive on 9 of 10 samples, treat 0.9 as the confidence estimate; a 6/4 split flags the same ambiguity a logprob split would. The agreement rate is a Monte Carlo estimate of the very distribution logprobs hand you directly — you're paying in tokens for what OpenAI returns free.
from collections import Counter
def claude_confidence(text, n=10):
votes = Counter()
for _ in range(n):
out = call_claude(text, temperature=0.8) # your wrapper
votes[out.strip().lower()] += 1
label, count = votes.most_common(1)[0]
return label, count / n
This costs N× the calls, so reserve it for decisions where the confidence actually gates an action. You can still temperature-scale the agreement rate against a held-out set to correct any residual bias. A lighter alternative is an explicit self-evaluation pass — a second call that judges the first — but that inherits the same verbalized-confidence weakness, so prefer sampling-based agreement when you can afford it.
How do I use calibrated confidence in production?
Turn it into selective prediction: pick a confidence threshold, auto-accept everything above it, and escalate the rest. This is where calibration pays for itself. With a trustworthy probability you can draw a risk-coverage curve — sort predictions by confidence, and for each coverage level (the fraction you auto-handle) read the error rate on that slice.
Concretely: if calibrated 0.9+ predictions cover 70% of your traffic at a 3% error rate, automate that 70% and route the low-confidence 30% to a stronger model or a human queue. You've converted a flat 92% accuracy into a tunable business dial — trade coverage for accuracy explicitly instead of eating a fixed error rate everywhere. The threshold is a product decision (how expensive is a wrong auto-accept?), and calibration is what makes that decision honest. Tiered routing falls out naturally: cheap model with logprob gating first, escalate the uncertain tail to the expensive model, and only the residual reaches a person.
The direct answer
Token logprobs beat asking your LLM how confident it is because a self-reported score is generated by the same next-token machinery as any other text — overconfident, clustered on round numbers, and uncorrelated with correctness — whereas a token logprob is the actual probability the model assigned to the token it chose. On the OpenAI-compatible endpoint, set logprobs=True with single-token labels, exp() the label token's logprob, and fit one temperature-scaling parameter on held-out data to drive ECE down. That gives you a calibrated per-decision probability you can threshold for selective prediction: auto-accept the confident majority, escalate the uncertain tail. For Claude, which exposes no logprobs, approximate the same signal with self-consistency sampling. Either way, stop trusting the model's self-graded confidence and start reading the number it already computed.
Top comments (0)