A user emailed me a screenshot of their feedback report. Under "Communication," my scoring model had written: "Closed the answer with 'Thanks for watching,' which feels scripted and out of place in a live conversation."
They never said it. They paused for about four seconds, decided they were done, and stopped talking. Whisper filled the silence for them, and then an LLM graded them on the words Whisper made up.
This post covers Whisper hallucinations on silence: how often they happened in my pipeline (9.2% of clips), why the built-in no_speech_threshold didn't catch them, and the config change that got the count from 183 to 4.
TL;DR
- Whisper hallucinates text on silence. Trailing or leading quiet in a clip gets filled with phrases like "Thanks for watching!", "Thank you.", or "Subtitles by the Amara.org community".
- In my audit of 2,000 voice clips, 183 (9.2%) had hallucinated text. Clips with 3+ seconds of trailing silence hit 23%. Clips without it hit 3%.
-
no_speech_thresholdalone won't save you. A segment only gets skipped ifno_speech_probis high andavg_logprobis low. Confident hallucinations pass. -
The fix that worked:
vad_filter=True(Silero VAD) withspeech_pad_ms=400,condition_on_previous_text=False, plus a small guard on known phrases. 183 dropped to 4. - Never feed raw ASR output to an LLM judge without telling it the transcript may contain artifacts.
What was the pipeline?
I run Preterview, a voice interview practice app (full disclosure: I built it). You talk to an AI interviewer, and afterward you get a written report scored from your transcript. Live turn-taking runs on a streaming ASR, but the report uses a second, slower pass: every answer clip goes through faster-whisper with large-v3 on a single GPU, and the cleaned transcript goes to Claude for scoring.
That second pass is where accuracy matters, so I thought it was the safe part. It was the part lying to me.
How often does Whisper hallucinate on silence?
In my data, about 1 clip in 11. I pulled 2,000 answer clips from one week of sessions and flagged anything matching a list of known hallucination phrases, plus any segment with compression_ratio > 2.4 (a decent signal for repetition loops). That flagged 196 clips. I listened to all 196. It took an afternoon and a lot of coffee.
13 were real: people genuinely saying "thank you" at the end. The other 183 were invented.
| Hallucination type | Clips |
|---|---|
| "Thanks for watching!" / "Thank you for watching." | 71 |
| Standalone "Thank you." | 58 |
| Repetition loops (same phrase 3+ times) | 27 |
| Other ("Please subscribe", random short phrases) | 18 |
| "Subtitles by the Amara.org community" | 9 |
| Total | 183 / 2,000 (9.2%) |
The location pattern was the giveaway. 164 of the 183 sat in the first or last segment of the clip, right next to at least two seconds of silence. When I split by trailing silence:
- Clips with 3+ seconds of trailing silence: 141 of 612 hallucinated (23%)
- Everything else: 42 of 1,388 (3%)
Interview answers are basically designed to trigger this. People think before they speak, and they trail off when they finish. My recorder kept capturing a few seconds after the last word so it wouldn't clip anyone. That safety margin was the hallucination fuel.
Why do Whisper hallucinations happen on silence?
Whisper's decoder is a language model, and it was trained on a huge amount of weakly labeled web audio paired with captions. When the audio gives it almost nothing, the decoder falls back on its prior: what text usually appears over quiet, music, or a fade-out in its training data? Video outros. "Thanks for watching." Caption credits.
It also always works on 30-second windows. A short clip gets padded, so the model is constantly looking at stretches of nothing and being asked to produce text for them. Most of the time it emits nothing or a no-speech token. Sometimes it confidently writes a sign-off.
The repetition loops have a separate cause. With condition_on_previous_text=True (the default), the previous segment's text gets fed in as a prompt. Once one segment goes wrong, the next one is conditioned on the wrong text, and you get "I think that I think that I think that" until the window ends.
Why didn't no_speech_threshold catch it?
This one cost me the most time. I assumed no_speech_threshold=0.6 meant "drop anything Whisper thinks is 60%+ likely to be silence." It doesn't. Here's the logic from openai-whisper's transcribe.py (faster-whisper does the same thing):
if no_speech_threshold is not None:
should_skip = result.no_speech_prob > no_speech_threshold
if logprob_threshold is not None and result.avg_logprob > logprob_threshold:
# don't skip if the logprob is high enough, despite the no_speech_prob
should_skip = False
A segment only gets skipped if no-speech probability is high and the average log-probability of the decoded text is low. If the model is confident about its made-up text, the segment survives.
And "Thanks for watching" is something Whisper is very confident about. Across my 183 hallucinations, the median no_speech_prob was 0.71, comfortably above the 0.6 threshold. But 129 of them had avg_logprob above -1.0, so the skip got overruled.
What I tried before the real fix:
-
Lower
no_speech_thresholdto 0.4: 183 became 176. Almost nothing, because the logprob veto still applied. -
Set
logprob_threshold=None(skip on no-speech alone): 183 became 61, but 37 clips lost real speech. Quiet, soft-spoken answers got deleted. Trading invented words for deleted real ones is worse for a scoring product.
Thresholds were the wrong layer. The problem was that silence reached the decoder at all.
What actually fixed the Whisper hallucinations?
Stop sending silence to the model. faster-whisper ships a built-in Silero VAD filter that strips non-speech regions before decoding. Here's the config I landed on:
from faster_whisper import WhisperModel
model = WhisperModel("large-v3", device="cuda", compute_type="float16")
KNOWN_GHOSTS = {
"thanks for watching", "thank you for watching", "thank you",
"please subscribe", "subtitles by the amara.org community",
}
def transcribe_answer(path: str) -> str:
segments, _ = model.transcribe(
path,
beam_size=5,
vad_filter=True,
vad_parameters=dict(min_silence_duration_ms=500, speech_pad_ms=400),
condition_on_previous_text=False,
)
kept = []
for seg in segments:
norm = seg.text.strip().lower().rstrip(".!")
if norm in KNOWN_GHOSTS and seg.no_speech_prob > 0.5:
continue # confident ghost in a quiet region
kept.append(seg.text.strip())
return " ".join(kept)
I added changes one at a time and re-ran the same 2,000 clips after each:
| Change | Hallucinations | Real speech lost |
|---|---|---|
| Baseline | 183 | 0 |
condition_on_previous_text=False |
159 | 0 |
+ vad_filter=True (default padding) |
22 | 11 clips |
+ speech_pad_ms=400
|
24 | 3 clips |
| + known-phrase guard | 4 | 3 clips + 1 real "thank you" |
Turning off conditioning killed the loops (27 down to 3). VAD did the heavy lifting on everything else.
The VAD had its own failure mode: with default padding, 11 clips lost their first word. Soft openers like "So," or "Um, yeah" got classified as noise. Bumping speech_pad_ms to 400 brought that to 3 and let 2 hallucinations back in. I took that trade.
The phrase guard is deliberately narrow. It only drops a known phrase when the segment's own no_speech_prob is above 0.5. Of the 13 real "thank you"s, 12 had no_speech_prob under 0.2 and survived. One person said it so quietly that it got dropped. I can live with that.
What were the last 4 hallucinations?
Two were "Thank you." in clips with loud keyboard noise, where VAD called the typing speech. One was a lone "you." One was a three-time "um" loop that survived because conditioning wasn't the cause. I stopped there. Going from 4 to 0 would mean more aggressive VAD, and that eats real words.
What else changed besides the ASR config?
Two things, both cheap.
The scorer prompt now knows transcripts are machine-made. I added a line telling Claude the transcript came from automatic speech recognition, may contain artifacts like sign-offs or repeated phrases, and must never penalize those. Even with perfect ASR config, a judge that trusts its input blindly is one bad clip away from writing nonsense about a real person.
Hallucination rate is now a metric. A nightly job runs the phrase list and the compression-ratio check over the day's transcripts and alerts if the flagged rate goes above 1%. It sits around 0.3%. If a model upgrade or a recorder change pushes it back up, I'll know before a user emails me a screenshot.
A bonus I didn't expect: VAD made transcription faster. Skipping silent regions cut average GPU time per clip from 0.84s to 0.61s. The fix paid for itself.
Why does Whisper say "Thanks for watching" on silence?
Whisper says "Thanks for watching" on silence because its decoder is a language model trained on web video captions, and when the audio carries no speech it falls back to the text that most often appears over quiet endings in that data: video outros and caption credits. The default no_speech_threshold doesn't stop it, since a segment is only skipped when no-speech probability is high and decoding confidence is low, and these phrases decode with high confidence. In my pipeline that meant 183 of 2,000 clips (9.2%) contained invented text. The reliable fix is to remove silence before decoding: enable vad_filter=True in faster-whisper with some speech padding, set condition_on_previous_text=False to stop repetition loops, and add a narrow guard for known phrases in high no-speech segments. That took my count from 183 to 4.
Written by the developer behind Preterview, an interview prep platform.
Top comments (0)