One afternoon my own dictation tool started typing sentences I had not said.
Not garbage. Fluent, confident, plausible English. If you have used Whisper, you may already know where this is going.
Whisper does not do "nothing"
Speech models like Whisper are trained to produce text. Hand one a clip of an empty room, and it will very often give you words anyway, a phrase it has seen a thousand times, or something that sounds like a polite closing line. This is well known, and the usual defence is simple: don't send it silence.
In YazSes (an offline hold-to-talk dictation tool I maintain) the first defence is an energy gate. When you release the key, it computes the mean absolute level of the recording. If that is below a threshold, the clip is dropped and Whisper never sees it.
[accessibility]
vad_threshold = 0.01 # the default
My own config said 0.0005.
My room's noise floor, measured, was around 0.0039. The gate sat almost eight times below the room. Every time I pressed the key without speaking, or paused before talking, the fan and the street went straight through to Whisper, and Whisper wrote me some prose.
Three things that went wrong at once
It is tempting to blame "a bad config value". But three separate things had to fail for me to not notice for a while.
1. The value was below the noise. I don't actually know how 0.0005 got into my config. It does not matter much. Any threshold that is not measured against the room it runs in is a guess.
2. The warning that should have caught it was silent. yazses doctor --mic measured the room and compared it with the threshold. In exactly this case, it printed:
[OK] Mic level: ambient 0.0010 under vad_threshold 0.0005
Read that twice. 0.0010 is not under 0.0005.
The warning only fired when the room was "not silent", and "silent" was judged against a fixed floor of 0.002 that had nothing to do with my gate. So for any threshold below 0.002, the warning was switched off across exactly the band where it was needed. The one check designed for this situation reported success. (Fixed back in v2.25.1. It now says, plainly, that silence will pass the gate and be transcribed as invented words.)
3. The calibrator measured the wrong thing. There was a mic-level command to pick a threshold for you. It recorded one clip and assumed you were speaking in it. If you weren't, it calibrated to the room. In four empty-room recordings it measured 0.0036 to 0.0050 and recommended 0.002 to 0.0025: below the noise, every single time.
How to calibrate a gate properly
The fix is boring, and that is the point.
Measure the room and the voice separately. Since v2.36.0, yazses mic-level records a few seconds of silence first, then a few seconds of you talking. Now there are two numbers, and the gate must sit between them:
lo = max(_MIN_THRESHOLD, room * 1.5) # comfortably above the room
hi = speech * 0.5 # comfortably below your voice
Refuse when the numbers are too close. If hi < lo, the voice and the room are less than 3x apart. At that point any threshold you pick will either let noise through or eat quiet words. The honest output is not a number. It is "I can't calibrate this, move the mic or lower the gain." So it refuses.
That 3x is not tuned, by the way. It falls out of the two margins (1.5 / 0.5). For reference, across 1,646 real dictation events, speech had a median level of 0.0394 and clips that passed the gate but produced no text had a median of 0.0069, about 5.7x apart. Plenty of room, if you measure it.
The next release goes one step further (already on main): it aims the gate at twice the room level instead of halfway between room and voice, and it refuses recordings that are too short to say anything (under one second of room or two seconds of speech). That came from a bug report that mic-level --set picked a different threshold on every run.
What I took away
- A threshold is meaningless without the noise it is meant to reject. Measure the noise, in place, on the real device.
- One sample cannot tell you whether anyone spoke in it. If your calibration depends on the user doing something, record that something separately.
- When two measurements are too close, refuse. A confident wrong number is worse than "I can't".
- Test your warnings in the case they exist for. My warning was fine in every case except the one it was written for.
If you dictate with Whisper (any tool, not just mine) and you see phantom text, check your silence gate before you blame the model.
yazses mic-level # measures room, then voice, and suggests a threshold
yazses mic-level --set # writes it to your config
YazSes is offline, open source (Apache-2.0), and runs on Linux, macOS and Windows: github.com/MSKazemi/yazses. How the gate works in detail: silent audio discarding.
I wrote this with help from an AI assistant for drafting and editing; the incident, the numbers and the conclusions are mine, and I checked each fact against the code and changelog.
Top comments (0)