A language model will tell you it is 95 % sure of almost anything. I spent a week turning that into a number you could actually route work on, and then tested it against a model built to do the same thing. The result was a split decision, and the most useful thing I learned came from neither of them.
The setting is a mortgage servicing desk. Under Regulation X, when a borrower writes in, what the letter is starts a clock: a notice of error gets thirty days, a request for the identity of the loan's owner gets ten, a payoff request gets seven. Misfile a letter and you miss a federal deadline. So the triage decision — which of five kinds is this — is worth automating and expensive to get wrong, which makes it a good place to ask the question everyone building on LLMs eventually has to answer: when can I let this thing decide by itself?
Step one: the number in the JSON is not a number
The first version asked a model to classify the letter and return a confidence. It returned 0.95 or higher on essentially everything, including the letters it got wrong. That is not a bug in the model. It was asked to generate a plausible-looking confidence field, and it did. Nothing measured anything.
The fix is to stop asking the model to say a probability and start reading one. For a fixed set of answers you can put the options in the prompt as A, B, C, look at the next-token distribution at the position where the answer goes, and renormalise over just those letters. One forward pass, no decoding, and what comes back is a real distribution over the allowed answers.
That helped, and then it didn't. Small models have position bias — they like A — so I scored every question under five rotations of the option order and averaged. The averaged number looked reasonable and carried almost no information: AUROC 0.64, barely better than a coin flip at separating right answers from wrong ones.
The reason turned out to be the averaging. Each individual rotation was nearly one-hot: 0.999 for one option, nothing for the rest. Averaging five of those gives you the fraction of orderings that agreed — a vote share with six possible values — and throws away every margin. Averaging the log-probabilities instead keeps them. AUROC went from 0.64 to 0.76 on the same data, same model, same prompts. The only change was doing the mean in the right space.
Step two: ordered is not calibrated
Now the number ranked correctly — higher meant more likely right — but it still wasn't a probability. When it said 0.9 it was right about 75 % of the time. Ranking and calibration are different properties, and the standard fix is boring and effective: divide the logits by a temperature fit on held-out data. Expected calibration error dropped from 0.15 to 0.06. When it said 0.9 it was now right about nine times in ten.
That fix has a catch: it assumes the next batch of letters looks like the batch you fit the temperature on. So the last layer is conformal prediction, which makes a promise that does not depend on the model being good. Calibrate a threshold on held-out examples, and for a new letter output the set of labels that clear it. With exchangeable data the true label is in that set at least 95 % of the time, whatever the model is doing. Sets of size one are the ones you auto-route; everything else goes to a human. The coverage knob becomes a queue-length knob, with a stated error rate attached.
At that point the local pipeline — a 4-billion-parameter model on one consumer GPU, no API key — was auto-routing 45 % of real borrower correspondence with an 8.8 % error rate among the routed letters.
Step three: someone shipped the whole layer
Then TypeSafe released Jev, a model that answers typed questions and returns distributions directly. One request, one response, calibrated probabilities. Exactly the thing I had just assembled out of logprobs and arithmetic.
I wired it in as a second decision layer behind the same interface, sharing the same option descriptions, so that switching between them changed the decision layer and nothing else. Then I ran both over 42 synthetic letters written to exercise the tricky cases and 300 real mortgage complaints pulled from a public mirror of the CFPB complaint database — prose nobody on the project had written, angrier and longer and vaguer than anything I would have thought to compose.
The calibration claim held up completely. Jev arrived with what I had built by hand: AUROC 0.74 against my 0.64 raw, and its confidence bands were already ordered out of the box — the letters it scored under 0.70 were right about half the time, the ones it scored 0.99 were right every time. It needed a temperature of 1.7 where my local pipeline needed 6. One request at 338 ms against sixteen local forward passes at a second.
And it was eleven points less accurate on the real letters. 197 out of 300 against the local model's 228.
Step four: the part that was actually worth finding
The two models were not making the same mistakes. They were making opposite ones.
Eighty-five of Jev's errors were letters my labels called covered that it filed as "not covered" — long complaints describing a servicing failure without ever using the word "error". The local model erred the other way, pulling twenty-nine letters into "notice of error" that belonged outside the rule.
Then I checked something else. The labels for that set had been written by me and independently checked by a second labeller on a random sixty. We agreed 78 % of the time, κ = 0.61 — and of our thirteen disagreements, ten were on exactly that line. Two models and two people, all failing on the same boundary.
When that happens, the boundary is the defect. Not the model.
So I rewrote the definition. Not by looking at which letters anyone got wrong — that is fitting to your test set — but from the labelling policy I had written down before any of this ran. Where the old text said "a written notice asserting that the servicer made an error", the new text lists what a servicing failure actually looks like: a payment misapplied or credited late, a fee said to be unjustified, escrow items left unpaid, information lost in a transfer between servicers — and states that it counts whether or not the letter uses the word "error", and whether or not it asks for anything in return. Lending decisions — refinance applications, assumptions — are named as the thing that is not servicing.
One prompt change, nothing else:
| on 300 real letters | before | after |
|---|---|---|
| Jev, accuracy | 197 (66 %) | 273 (91 %) |
| Jev, AUROC | 0.74 | 0.87 |
| Jev, auto-routed at 95 % coverage | 58 letters, 4 wrong | 237 letters, 8 wrong |
| local 4B, accuracy | 228 (76 %) | 218 (73 %) |
| local 4B, auto-routed at 95 % coverage | 136 letters, 12 wrong | 62 letters, 0 wrong |
Seventy-six letters, from writing the definition down properly. No amount of calibration would have found it, because nothing was miscalibrated — both models were confidently applying a rule nobody had stated clearly.
The thing I did not expect
The same change cost the small model ten letters.
This was the second time in two rounds. Earlier, I had noticed the flag questions — "is this request overbroad?", "is it duplicative?" — were leaning toward yes, which is what binary questions do when only the affirmative is described. TypeSafe's own format has a slot for describing both outcomes, so I filled it in for both providers. Spurious flags fell from 3 to 1 for Jev and rose from 3 to 7 for the local model.
Richer instructions, twice, helped the purpose-built model and hurt the 4B. It is not a coincidence: a small model scoring multiple-choice options degrades as the options grow, whether they grow by adding a negation or by adding detail. How much prompt a model can use is a property of the model, and "write a better prompt" is advice with a size dependency nobody mentions.
I kept the shared definitions anyway. Per-provider prompts would make every future comparison meaningless, the sharpened wording is what the desk's policy actually says, and the small model's calibration improved even as its accuracy fell — it now routes 62 letters at 95 % coverage and gets none of them wrong. Ten letters is the measured price of having one prompt.
What I would tell you if you are about to do this
The confidence in the JSON is decoration. If you need a probability, read one — from logprobs, or from a model that returns them.
Average in log space. Averaging near-one-hot distributions destroys exactly the information you are trying to collect.
Ranking and calibration are different problems and need different fixes. Check both.
Conformal prediction is the only layer that survived a provider change. Its guarantee never depended on the model being good, so when I swapped the decision layer it read the new distributions unchanged. If you build one thing from this list, build that.
Before you blame the model, check whether two careful humans agree. If they do not, you are measuring the rubric.
One caveat belongs next to that 91 %. The new definitions encode my own labelling policy, and the labels were written by one person applying it — so part of the jump is the model finally being told the rubric it is graded against. Our two labellers agreed only 78 % of the time, which means a single labeller's policy has a ceiling and 91 % is close enough to it that the remaining nine points are as likely to be the labels as the model. Reporting the number without that sentence would be a lie of omission.
The desk, the evaluation harness, the labelled sets and every number above are in comedianhhh/servicing-desk; the round-by-round write-up, including the failures that are not in this article, is in backend/evals/README.md.
Top comments (0)