A confidence number your model reports is not a measurement

The prompt line is almost universal:

Answer the question, and also report your confidence as a number between 0 and 1.

The model obliges. {"answer": "...", "confidence": 0.92} comes back, and somewhere downstream the code does the
obvious thing:

if result["confidence"] > 0.8:
    apply(result["answer"])          # auto-accept
else:
    queue_for_human(result["answer"])

That if decides real work, and it is built on a token that was generated because it fitted the sentence so
far. This post is about why that number cannot do the job it was hired for, how to tell the difference between a
decorative number and a measured one, and the small amount of bookkeeping that turns the second into something you
can actually test.

Two different things wear the same name

A model-reported confidence and a calibrated probability are not the same kind of object, and the difference is not
philosophical — it is about what had to exist before the number was written down.

A generated number is produced inside the same forward pass as the answer. The model has seen thousands of
sentences of the form "the answer is X, and I am fairly confident", and the number that continues that sentence
well is a high one. Nothing in that process consults an outcome, because there is no outcome yet. The choice has
not been made; the result does not exist; there is nothing to be right or wrong about.

A measured number is produced by bookkeeping. Someone stored the number, later stored what actually happened,
and put the two in the same table. That join is where calibration comes from — and it is the only place it can come
from. A number without that join is not a probability; it is a word that looks like one.

This is why "just ask the model to be better calibrated" cannot work. Wording changes the distribution of reported
numbers, not the fact that the second column is missing. You can prompt a model into saying 0.6 instead of 0.95
on exactly the same input. You cannot prompt it into having recorded whether 0.6 calls were right six times in
ten — only your system can do that, over time.

The tell is the shape of the distribution

There is a cheap way to see which kind of number you have: check whether it moves when the problem gets harder.

Decorative numbers are high and narrow, and they barely respond to difficulty. Ask for a number on a genuinely
evenly matched pair of options, then ask on a case where one option is clearly right, and both come back in the high
0.8s or low 0.9s. The number is tracking how decisive the answer sounds, not how close the evidence was.

A set of numbers that came out of real close decisions looks different. We publish the record behind our own
service — 27 decisions, recorded verbatim, on everyday calls where both options were defensible. Across that set
the reported separation runs 27.3% to 88.3%, median 74.0%, and nothing above 90%. That shape is the point: on
questions that really are close, a number that has been checked against outcomes is allowed to be low, so it is.
A service reporting the upper nineties on such questions is not reporting a probability; it is reporting
enthusiasm.

	Decorative number	Measured number
Where it comes from	a sampled token in the same generation as the answer	a record: the number stored now, the outcome stored later, the buckets compared
What it responds to	how decisive the answer sounds	how close the evidence actually was
Behaviour on a harder input	nearly unchanged	falls, often below 0.7
What you can compute from it	nothing — no join is possible	per-band hit rate, and whether the bands are honest
Failure mode when you branch on it	a coin flip with a confident label, invisible in production	you own the errors, but you can see them and fix the bands
Cheapest way to tell them apart	ask the same question twice with different framing; watch the number track the framing	ask whether your logs contain the outcome column at all

Move the judgement out of the prompt and into a component

The fix is not a better prompt — that only moves the numbers. The fix is to make the judgement a call with a
contract, and to log it like any other call in your system: the pick, the reported separation, and the reason,
under an id you can retrieve later.

curl -s -X POST "https://mcp.turingcorp.net/mcp" \
  -H 'content-type: application/json' \
  -H 'accept: application/json, text/event-stream' \
  -H "authorization: Bearer $AGENT_PASS" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"decide","arguments":{
        "task":"Which draft of the outage notice do we send to customers?",
        "optionA":"Short: what broke, what we did, when it was fixed.",
        "optionB":"Longer: the same content plus what we changed so it cannot recur."
      }}}'

The result carries a job_id alongside the decision and the reason. Treat that id as the first column of your
bookkeeping, not as an error-handling detail: a call that never came back is retrieved by id rather than repeated,
and a call that did come back is a row you can join later.

Then two moves, both boring, both mandatory:

1. Log every judgement as a row — id, reported separation, the inputs, timestamp, and what your system did with it.
2. Join the rows to outcomes later — bucket by band, and compute the hit rate for each band.

Until step 2 has run at least once on your own traffic, the number is not a measurement. It is a style.

The checklist

Copy this into whatever review gates the prompt change:

1. Do not branch on a number whose per-band accuracy you have never computed.
2. Log every judgement: id, number, inputs, timestamp, and the action taken.
3. Define the bands before looking at outcomes. Otherwise you will fit the bands to the data and call it calibration.
4. Join the log to outcomes; compute the hit rate per band; keep the table where the team can see it.
5. When the numbers and the outcomes disagree, change the band-to-action mapping. Do not re-word the prompt — that only moves the numbers.

Two consequences follow. First, the band table is the only thing that tells you what 0.74 means for your use
case: in ours, a low number means the two options were close and the choice should stay with a human, and that
claim is only defensible because the band accuracy is measured and published rather than asserted. Second, the
number becomes testable the moment it is a component: you can regression-test it, you can change a threshold and
see what breaks, and you can answer "how often is it wrong?" without guessing.

A number is a measurement when something outside the model has checked it. Everything else is a sentence that
happens to contain digits — and if your control flow reads that sentence, the model is choosing your thresholds for
you. The endpoint and the published record are at https://mcp.turingcorp.net.