DEV Community: Alan Li

I built a calibration layer by hand. Then a model shipped with one.

Alan Li — Tue, 22 Sep 2026 22:16:28 +0000

A language model will tell you it is 95 % sure of almost anything. I spent a week turning that into a number you could actually route work on, and then tested it against a model built to do the same thing. The result was a split decision, and the most useful thing I learned came from neither of them.

The setting is a mortgage servicing desk. Under Regulation X, when a borrower writes in, what the letter is starts a clock: a notice of error gets thirty days, a request for the identity of the loan's owner gets ten, a payoff request gets seven. Misfile a letter and you miss a federal deadline. So the triage decision — which of five kinds is this — is worth automating and expensive to get wrong, which makes it a good place to ask the question everyone building on LLMs eventually has to answer: when can I let this thing decide by itself?

Step one: the number in the JSON is not a number

The first version asked a model to classify the letter and return a confidence. It returned 0.95 or higher on essentially everything, including the letters it got wrong. That is not a bug in the model. It was asked to generate a plausible-looking confidence field, and it did. Nothing measured anything.

The fix is to stop asking the model to say a probability and start reading one. For a fixed set of answers you can put the options in the prompt as A, B, C, look at the next-token distribution at the position where the answer goes, and renormalise over just those letters. One forward pass, no decoding, and what comes back is a real distribution over the allowed answers.

That helped, and then it didn't. Small models have position bias — they like A — so I scored every question under five rotations of the option order and averaged. The averaged number looked reasonable and carried almost no information: AUROC 0.64, barely better than a coin flip at separating right answers from wrong ones.

The reason turned out to be the averaging. Each individual rotation was nearly one-hot: 0.999 for one option, nothing for the rest. Averaging five of those gives you the fraction of orderings that agreed — a vote share with six possible values — and throws away every margin. Averaging the log-probabilities instead keeps them. AUROC went from 0.64 to 0.76 on the same data, same model, same prompts. The only change was doing the mean in the right space.

Step two: ordered is not calibrated

Now the number ranked correctly — higher meant more likely right — but it still wasn't a probability. When it said 0.9 it was right about 75 % of the time. Ranking and calibration are different properties, and the standard fix is boring and effective: divide the logits by a temperature fit on held-out data. Expected calibration error dropped from 0.15 to 0.06. When it said 0.9 it was now right about nine times in ten.

That fix has a catch: it assumes the next batch of letters looks like the batch you fit the temperature on. So the last layer is conformal prediction, which makes a promise that does not depend on the model being good. Calibrate a threshold on held-out examples, and for a new letter output the set of labels that clear it. With exchangeable data the true label is in that set at least 95 % of the time, whatever the model is doing. Sets of size one are the ones you auto-route; everything else goes to a human. The coverage knob becomes a queue-length knob, with a stated error rate attached.

At that point the local pipeline — a 4-billion-parameter model on one consumer GPU, no API key — was auto-routing 45 % of real borrower correspondence with an 8.8 % error rate among the routed letters.

Step three: someone shipped the whole layer

Then TypeSafe released Jev, a model that answers typed questions and returns distributions directly. One request, one response, calibrated probabilities. Exactly the thing I had just assembled out of logprobs and arithmetic.

I wired it in as a second decision layer behind the same interface, sharing the same option descriptions, so that switching between them changed the decision layer and nothing else. Then I ran both over 42 synthetic letters written to exercise the tricky cases and 300 real mortgage complaints pulled from a public mirror of the CFPB complaint database — prose nobody on the project had written, angrier and longer and vaguer than anything I would have thought to compose.

The calibration claim held up completely. Jev arrived with what I had built by hand: AUROC 0.74 against my 0.64 raw, and its confidence bands were already ordered out of the box — the letters it scored under 0.70 were right about half the time, the ones it scored 0.99 were right every time. It needed a temperature of 1.7 where my local pipeline needed 6. One request at 338 ms against sixteen local forward passes at a second.

And it was eleven points less accurate on the real letters. 197 out of 300 against the local model's 228.

Step four: the part that was actually worth finding

The two models were not making the same mistakes. They were making opposite ones.

Eighty-five of Jev's errors were letters my labels called covered that it filed as "not covered" — long complaints describing a servicing failure without ever using the word "error". The local model erred the other way, pulling twenty-nine letters into "notice of error" that belonged outside the rule.

Then I checked something else. The labels for that set had been written by me and independently checked by a second labeller on a random sixty. We agreed 78 % of the time, κ = 0.61 — and of our thirteen disagreements, ten were on exactly that line. Two models and two people, all failing on the same boundary.

When that happens, the boundary is the defect. Not the model.

So I rewrote the definition. Not by looking at which letters anyone got wrong — that is fitting to your test set — but from the labelling policy I had written down before any of this ran. Where the old text said "a written notice asserting that the servicer made an error", the new text lists what a servicing failure actually looks like: a payment misapplied or credited late, a fee said to be unjustified, escrow items left unpaid, information lost in a transfer between servicers — and states that it counts whether or not the letter uses the word "error", and whether or not it asks for anything in return. Lending decisions — refinance applications, assumptions — are named as the thing that is not servicing.

One prompt change, nothing else:

on 300 real letters	before	after
Jev, accuracy	197 (66 %)	273 (91 %)
Jev, AUROC	0.74	0.87
Jev, auto-routed at 95 % coverage	58 letters, 4 wrong	237 letters, 8 wrong
local 4B, accuracy	228 (76 %)	218 (73 %)
local 4B, auto-routed at 95 % coverage	136 letters, 12 wrong	62 letters, 0 wrong

Seventy-six letters, from writing the definition down properly. No amount of calibration would have found it, because nothing was miscalibrated — both models were confidently applying a rule nobody had stated clearly.

The thing I did not expect

The same change cost the small model ten letters.

This was the second time in two rounds. Earlier, I had noticed the flag questions — "is this request overbroad?", "is it duplicative?" — were leaning toward yes, which is what binary questions do when only the affirmative is described. TypeSafe's own format has a slot for describing both outcomes, so I filled it in for both providers. Spurious flags fell from 3 to 1 for Jev and rose from 3 to 7 for the local model.

Richer instructions, twice, helped the purpose-built model and hurt the 4B. It is not a coincidence: a small model scoring multiple-choice options degrades as the options grow, whether they grow by adding a negation or by adding detail. How much prompt a model can use is a property of the model, and "write a better prompt" is advice with a size dependency nobody mentions.

I kept the shared definitions anyway. Per-provider prompts would make every future comparison meaningless, the sharpened wording is what the desk's policy actually says, and the small model's calibration improved even as its accuracy fell — it now routes 62 letters at 95 % coverage and gets none of them wrong. Ten letters is the measured price of having one prompt.

What I would tell you if you are about to do this

The confidence in the JSON is decoration. If you need a probability, read one — from logprobs, or from a model that returns them.

Average in log space. Averaging near-one-hot distributions destroys exactly the information you are trying to collect.

Ranking and calibration are different problems and need different fixes. Check both.

Conformal prediction is the only layer that survived a provider change. Its guarantee never depended on the model being good, so when I swapped the decision layer it read the new distributions unchanged. If you build one thing from this list, build that.

Before you blame the model, check whether two careful humans agree. If they do not, you are measuring the rubric.

One caveat belongs next to that 91 %. The new definitions encode my own labelling policy, and the labels were written by one person applying it — so part of the jump is the model finally being told the rubric it is graded against. Our two labellers agreed only 78 % of the time, which means a single labeller's policy has a ceiling and 91 % is close enough to it that the remaining nine points are as likely to be the labels as the model. Reporting the number without that sentence would be a lie of omission.

The desk, the evaluation harness, the labelled sets and every number above are in comedianhhh/servicing-desk; the round-by-round write-up, including the failures that are not in this article, is in backend/evals/README.md.

Read isn't heard: detecting when streaming TTS actually finishes

Alan Li — Tue, 22 Sep 2026 21:11:58 +0000

Our voice agent kept reopening the mic before it had finished talking. The last word was still coming out of the speaker when the app started listening for the user again.

I spent a good part of this year on a 3D AI companion for phones: you talk, an LLM answers, ElevenLabs streams the voice back as Ogg/Opus, and a character on screen says it with lip sync, gaze and expression. Getting the first word out fast was the easy part. This bug took three rewrites, and it came down to one question:

When has the audio actually been heard?

That one moment drives everything after it:

the microphone reopens, so the user can answer
the subtitle comes down
the mouth closes and the body goes back to idle
the next turn is allowed to start

Too early, and the mic catches the tail of the character's own voice, the classic way a voice agent ends up treating its own words as a new user turn. Too late, and the user is talking to a character that isn't listening yet. Neither shows up in the logs.

Wrong answer 1: estimate it from the text

The first version guessed. Count the words, weight the punctuation:

float duration = words * 0.33f       // seconds per word
               + longPauses * 0.5f   // . ! ?
               + shortPauses * 0.2f; // , ; :

Start a timer, end the turn at duration + 1s.

That holds in a demo and drifts everywhere else. Voices read at different speeds. We sent Chinese at 1.2× speed, so it needed its own correction. And LLM text is full of numbers and abbreviations the voice model reads however it likes. Padding just moves the failure: too little clips long replies, too much leaves a dead gap after short ones.

The estimate still earns its place as a progress bar before any audio arrives. It can't decide when the turn ends.

Wrong answer 2: done when the stream is done

Next: stop guessing, count samples.

// main thread (DownloadHandlerScript.ReceiveData): each decoded chunk
totalDecoded += pcm.Length;

// PCMReaderCallback: each block Unity pulls from the clip
totalRead += samplesRead;

bool finished = downloadComplete && totalRead >= totalDecoded;

This kills a trap on its own: the HTTP stream closing means nothing. TTS servers generate faster than real time, so a ten-second reply can be fully downloaded while the character is on its second sentence. OpenAI's Realtime API documents the same gap. conversation.item.truncate exists for audio "sent to the client but not yet played". And two weeks ago Pipecat, one of the most widely used voice-agent frameworks, had an issue filed for logging "Bot stopped speaking" about four seconds before playback ended.

Counting read against decoded is much better, and progress becomes honest too. But on some phones the last syllable was still getting cut. A 0.15-second grace period helped. A magic number that works on some phones and not others means the model is wrong.

Turning point: stop reading logs, record the audio

Every log line said "done." So I built a capture you can arm for one session, in editor and development builds only. It writes three files:

the raw Ogg bytes, as downloaded
the decoded PCM, as WAV
what came out of the AudioSource, recorded in OnAudioFilterRead, as WAV

Line them up in any audio editor. If a word is in (1) but not (2), it's the decoder. If it's in (2) but not (3), it's timing. Ours was in (2) and missing from (3).

Wrong answer 3: done when Unity has read every sample

A streaming AudioClip is fed by a PCMReaderCallback. When that callback takes your last sample, the sample has not been played. It has entered a pipeline:

your buffer → PCMReaderCallback → AudioSource (+ filters) → mixer → DSP buffers → OS / device → speaker
                    ↑                                                                      ↑
            "all samples read"                                                    "user hears it"

Unity reads streamed clips ahead, in blocks. People have measured how far ahead. How much audio is in flight behind the reader depends on the platform, which is exactly why a fixed grace period worked on one phone and not another.

The same gap exists outside Unity. In the browser, an AudioWorklet consuming your last sample isn't playback either (see AudioContext.outputLatency). On native iOS and Android, the output buffer sits after your render callback.

What held up: follow the last frame to the output

Each utterance is a session with explicit states:

State	Meaning
Buffering	clip is playing, no data yet (silence)
Playing	real samples are flowing
Draining	every decoded sample has been read; waiting for it to be heard
Completed / Cancelled / Failed	terminal, and whichever comes first wins

The session exposes a PlaybackCompleted task that the turn logic awaits. Cancel and fail resolve it too, so nothing waits forever on audio that will never play. LiveKit Agents hit that exact deadlock this year, when wait_for_playout() ignored interruption.

Ending the turn takes three steps. The counters cross threads, so they use Interlocked.

1. Mark the last frame at the reader. When the download is complete and the reader has consumed every decoded sample, record the frame index of the final sample and move to Draining. Short replies can get here straight from Buffering.

// PCMReaderCallback (a Unity audio thread)
int frames = data.Length / channels;
long frameStart = Interlocked.Add(ref readerFrames, frames) - frames;
int got = buffer.Read(data);
totalRead += got;

if (downloadComplete && totalRead >= totalDecoded && state is Buffering or Playing)
{
    Interlocked.Exchange(ref finalFrame, frameStart + got / channels);
    state = Draining;
}

2. Confirm it after the AudioSource. OnAudioFilterRead inserts a filter into this AudioSource's own chain. It sees the source's output, already resampled to the output rate and channel layout, before the mixer sums it. Count frames there too. When that count passes finalFrame, the last sample has made it through the source.

void OnAudioFilterRead(float[] data, int channels)
{
    long outFrames = Interlocked.Add(ref outputFrames, data.Length / channels);
    long last = Interlocked.Read(ref finalFrame);
    if (last > 0 && outFrames >= last && !tailConfirmed)
    {
        tailConfirmedAt = AudioSettings.dspTime;
        tailConfirmed = true;
    }
}

One caveat: the reader counts frames at the clip's rate, and the filter counts at AudioSettings.outputSampleRate. Our clip is 48 kHz Opus. If your output rate differs, or you change pitch, scale first: finalFrame * outputRate / clipRate.

3. Wait out Unity's mixer buffer. Unity will tell you its size:

AudioSettings.GetDSPBufferSize(out int length, out int count);
double drain = (double)length * count / AudioSettings.outputSampleRate;
endAt = tailConfirmedAt + drain;   // e.g. 1024 × 4 / 48000 ≈ 85 ms

When AudioSettings.dspTime passes endAt, stop the source, settle the session, reopen the mic.

Treat this as a lower bound. It covers Unity's buffering, not the OS mixer or a Bluetooth headset, which can add a few hundred milliseconds. For our purpose that was fine: with headphones on, the mic can't hear the tail anyway.

The mobile catch. On some phones, OnAudioFilterRead was late, and in some cases never fired, once the stream went quiet. If step 2 waits forever, the subtitle stays up forever. So step 2 has a 2-second timeout. By then every decoded sample has been handed to Unity, so the timeout moment becomes the confirmation point and the drain is added on top. The worst case is a couple of seconds of silence, not a stuck turn. It logs a warning, so you can see which devices take that path.

Two more that bit us

A fixed ring buffer drops audio without telling you. The first buffer was a 20-second ring. The stream outruns playback, so a long reply could fill it. Write returned how many samples it accepted, and nothing checked. The replacement starts at 10 seconds, doubles as needed, and is capped at 120 seconds of unplayed audio (5,760,000 samples at 48 kHz mono). Past the cap it reports overflow and fails the session. A reply with a missing middle is worse than an error you can see.

Callbacks from a dead session. The user interrupts, a new reply starts, and a chunk from the old reply lands a moment later. If it writes into shared state, you get two sentences glued together. Every callback carries its session and checks it's still current. And callers cancel only the session they own, instead of a global "stop everything" that used to kill audio from other features.

void OnChunk(Session s, byte[] chunk)
{
    if (!ReferenceEquals(current, s) || s.IsTerminal) return;
    // decode, write
}

The buffer and the session state machine are unit-tested without a device. The tail detection is verified with the three-file capture.

How others handle it

LiveKit Agents makes the audio sink report the end itself: the sink calls on_playback_finished() with a playback position. That's the same "confirm at the output" idea.
OpenAI's realtime console counts samples inside an AudioWorklet. That's the browser version of step 2, without a drain step.
Pipecat decides the bot stopped speaking when its output queue goes idle for a short window. That is why the issue above sees the event arrive early.

If you're building this

Don't end a turn on a timer or on the stream closing.
"Read by the audio callback" is not "heard." Confirm after the source, then add the buffer.
Build the capture before you need it.

References: Unity AudioClip.Create · ElevenLabs stream speech · RFC 7845 (Ogg Opus) · RFC 3533 (Ogg) · Voice AI & Voice Agents primer