The Unplotted Instrument: Who Calibrates the Person Who Signs?
Ask a room of experienced people to put a number on a hard call and most of them will hand you a phrase instead. Likely. Probably. I am fairly confident. There is no question in my mind. That is not evasion, and it is not vagueness for its own sake. It is what confidence sounds like when nobody has ever asked you to score it.
This series has spent a lot of words asking the other side of that question. When a model reports a number, we ask for the curve behind it: the bins, the count in each bin, the cases that were excluded, the date it was measured on. That demand is correct, and it should keep being made of every vendor, including us. But it has only ever been made of machines. And the machine is not the instrument that appears in every decision you sign.
You are. Nobody has ever asked you for the diagram.
The only instrument present at every decision
Consider what is actually constant across the hard calls in one person's year. The questions are all different. The option sets are different. The people affected are different.
The judge, if you use one, is called once per question and keeps no memory of you. That is a feature — it returns a typed judgment and goes back to sleep — but it also means a judge cannot accumulate anything about your judgment. On the questions that matter most it has no reference class to consult, because the case has never happened before and the population a probability would be a lookup into does not exist. On the questions you will only ever answer once, the outcome gives you a single draw, and a single draw is the weakest evidence in the room.
So the accounting has a gap, and it is not a small one. Every part of the process is measured except the part that decides what to do with the measurement. The instrument that is present at every decision — the one that sets the number in the config file, that chooses whether to accept the machine's pick, that signs — has no curve behind it at all.
Three reasons your number was never plotted
The absence is not an accident and it is not a character flaw. It is structural.
The vocabulary has no resolution rule. "Probably" cannot be scored. Neither can "I had a good feeling" or "it was the obvious call." To be graded at all, a claim has to name an event and a date, and the naming has to happen before the event. Nobody teaches this, and no meeting agenda has a slot for it. So the phrase gets spoken, the decision gets made, and no claim exists that a later reading could resolve true or false.
The feedback arrives late and is then rewritten. Months pass. By the time the result is in, the person who made the call has a story, and the story is built out of the result. What gets compared against the outcome is a memory of a confidence, not the confidence itself. The one moment when a scoreable record was cheap to create is the moment before the decision, and that is the moment nobody is looking at.
There is no buyer asking for the chart. A vendor gets the calibration question because someone is about to pay for the model. The person who signs has no equivalent. Nobody downstream of a signature asks for the bins and the counts, and if someone did, the request would look like an audit of the person rather than of an instrument — which is the third reason, and the awkward one.
A band is a claim, and a claim needs a resolution rule
Fixing this does not require a personality change. It requires two artifacts and one habit.
The first artifact is a number, stated before the outcome is known. Not a feeling with a decimal point — a number attached to a named event. Seventy percent that we clear the volume target by June. Eighty percent that the integration takes less than six weeks. Sixty percent that the candidate accepts. The event has to be stated plainly enough that a stranger could resolve it from a calendar and a dashboard.
The second artifact is the resolution. When the date arrives, mark it: did the named event happen? That is the whole of the scoring. Notice that this is different from asking whether the decision was good. A decision can be good and lose, and a bad process can win, and neither of those sentences can be scored. The number can be.
The habit is the ledger. One line per call: the question, the number, the date it was stated, the named event, the resolution date. Then a reading — monthly, quarterly — where the stated numbers are grouped into bands and set against what actually happened. That is a reliability diagram for a person. Its bins will be coarse for a long time, and that is honest; six decisions do not make a curve, they make a first row.
What a machine's curve looks like when it is being honest
The value of the exercise becomes obvious the moment you look at what an honest curve contains, which is the thing this series has been asking vendors for.
Ours is self-run under named benchmarks, with the failures disclosed rather than quietly retried. On JudgeBench, 620 judgments, with the 6 first-verdict failures disclosed, calls reported at 90% confidence or above were right 99.6% of the time, the 80–90 band 94.0%, the 70–80 band 84.1%, and calls below 70% were right 67.7% of the time. Raw accuracy came out at 92.5% against 92.2% for a plain direct baseline, a tie that we report as a tie. On ContextualJudgeBench, self-run under the official pairwise protocol in which a pair counts as correct only if both presentation orders are judged correctly, which is why the random floor is 25% rather than 50%, consistent accuracy is 67.1% against the official reference of 65.4%, with 12 orders excluded after repeated platform failures and that exclusion printed beside the result. The deliberately constructed near-tie splits sit at 46–60%.
Read that as a shape rather than a scoreboard. The top band is nearly perfect and therefore nearly useless as a separator: a judge that reports 90% on everything and is right 90% of the time has produced a constant, and a constant cannot tell two cases apart. The information lives in the middle and the bottom, where the number actually moves — and the most valuable row is the one that admits the instrument does not know.
A personal ledger produces the same shape at a smaller scale. Its most useful entries will not be the calls you were sure about. They will be the ones in the sixty-to-eighty zone, where your record either supports your confidence or quietly contradicts it.
The failure mode: scoring the person instead of the instrument
There is a way to get this badly wrong, and it is worth naming because it is the first thing a manager will want to do with a ledger.
The moment the numbers feed a performance review, they stop being measurements. A stated probability that can be held against you becomes a strategic statement: everyone learns to say fifty percent, which is a claim about nothing and therefore cannot be wrong. The ledger turns into an exercise in self-presentation, and the one honest instrument in the room goes dark. Calibration and evaluation are two different uses of the same data, and a system that runs both on the same numbers will end up keeping only the second.
So the discipline has a boundary. The ledger is for you, or for a peer who trades readings with you. It is not a dossier. The reason to keep it is not that being scored improves you; it is that an unscored number is not knowledge. And there is a second, quieter reason: the record is the only object that lets a decision you make once be looked at more than once.
What the slow end owes the ledger
We build the end of this field that takes minutes, so the bias is ours to declare. But the relevant point is not that a slow judge is wiser. It is that a judge returning a pick, a calibrated confidence, and a written argument gives your ledger something to check itself against.
Two numbers on the same question — yours and the machine's — are more informative than either one alone. Where you disagree with a written argument, you have a specific thing to disagree with rather than a verdict to resent. And where the machine reports a low band on a question you feel certain about, the gap is itself an entry: one of the two instruments is wrong about this question, and the ledger is how you find out which, later, without trusting memory.
That is the actual division of labor at the accountable end. The machine supplies a measured number and a disagreeable argument. You supply the resolution, the record, and the signature. Neither of those is a personality trait, and both of them can be plotted.
Decider takes one question and two candidate answers and returns a pick with a calibrated confidence and a written argument — a number to set in the ledger beside your own.
The unmeasured instrument was never the model.
Top comments (1)
tr.ee/dev-to