Verbalized confidence: make an LLM state how sure it is, then check if 90% really means 90%

Ask a model a question and it answers with the same flat certainty whether it knows the capital of Australia or is guessing a coin-flip. Verbalized confidence fixes half of that: you ask it to attach an explicit self-estimate — "Answer: Canberra, Confidence: 90%." That single extra line turns an opaque guess into a signal you can gate, route, or abstain on. The other half — the part everyone skips — is checking whether the number is honest.

Eliciting the number is one system instruction

The whole behaviour lives in a system prompt that does three jobs: ask for a probability of being correct (not a vibe), give the model explicit permission to say low numbers so it isn't pushed to sound sure, and fix a parseable output shape. Here's the copy-paste template:

You are a careful assistant. Answer the question, then state how
confident you are that your answer is correct, as a percentage
from 0 to 100.

Rules:
- Base the number on how likely you are to be RIGHT, not on how
  sure you want to sound. If you are guessing, say so with a low
  number — being honestly unsure is correct behaviour, not a failure.
- Reserve 90-100% for answers you would bet money on. Use 50% for a
  true coin-flip. Spread the rest in between.
- Do not default to a round, confident-sounding number.
- Output EXACTLY this and nothing else:

Answer: <your answer>
Confidence: <0-100>%

Parsing it back is a couple of regexes with a clamp and a sensible default, so a stray format never crashes the pipeline.

A stated confidence is only worth anything if it's calibrated

A stated 90% is a prediction about the world: "of a hundred questions I felt this sure about, I'd get about ninety right." Calibration is whether that holds. To test it you need a labelled eval set, then you bin predictions by stated confidence and measure the accuracy actually observed in each bin.

function binByConfidence(preds, width = 10) {
  const nb = Math.round(100 / width);
  const bins = Array.from({ length: nb }, (_, i) => (
    { lo: i * width, hi: (i + 1) * width, confs: [], correct: 0, n: 0 }));
  for (const p of preds) {
    let idx = Math.floor(p.confidence / width);
    if (idx >= nb) idx = nb - 1;                   // 100% → last bin
    const b = bins[idx];
    b.n++; b.confs.push(p.confidence);
    if (p.correct) b.correct++;
  }
  for (const b of bins) {
    b.acc      = b.n ? b.correct / b.n : 0;                              // y
    b.meanConf = b.n ? b.confs.reduce((s, c) => s + c, 0) / b.n / 100 : 0; // x
  }
  return bins;
}

Plot each bin as (mean stated confidence, empirical accuracy) against the y = x diagonal and you have a reliability diagram. Collapse the whole diagram into one scalar and you have Expected Calibration Error — the count-weighted average distance from the diagonal:

function expectedCalibrationError(preds, width = 10) {
  const N = preds.length;
  const bins = binByConfidence(preds, width);
  return bins.reduce((ece, b) =>
    ece + (b.n ? (b.n / N) * Math.abs(b.acc - b.meanConf) : 0), 0);
}
// overconfident model  → ECE ≈ 0.40  (claims ~85%, right ~50%)
// well-calibrated model → ECE ≈ 0.06  (claims ~75%, right ~75%)

Why LLMs skew overconfident — and what to do

The failure mode is baked in by training. Pretraining rewards fluent, assured continuations; hedging is rare in the data. RLHF then rewards answers humans like, and humans reward confident, decisive replies — so the model learns that sounding sure scores while honestly saying "maybe 55%" does not. The result: verbalized numbers cluster high and flat, and the curve sags below the diagonal in the top bins.

You push back on three fronts. Prompt-side: ask for a probability of being correct, permit low numbers, anchor the scale. Sampling-side: draw N answers and use their agreement as an empirical confidence (self-consistency), which is often better calibrated than the model's self-report. And measurement: you cannot trust a confidence you've never scored against labels. Once it's calibrated, that number becomes a real control lever — abstain, escalate to a human, or route to a bigger model whenever confidence is low.

Toggle an overconfident model against a calibrated one and watch the reliability curve bend, live at: https://dev48v.infy.uk/prompt/day54-verbalized-confidence.html