jidonglab

Posted on Sep 23

Whisper Hallucinates on Silence: 187 Phantom Transcripts in a Week

#ai #llm #machinelearning #python

The report said the candidate paused mid-answer about database sharding and then said, clearly, "Thanks for watching!"

They didn't. They were thinking. Nobody was talking for about six seconds, and Whisper filled that silence with a YouTube outro. Then my LLM interviewer read the transcript and replied, "Glad you enjoyed it! Let's keep going."

Whisper hallucinates on silence. It's a known quirk, but I didn't grasp how often it happened until I counted. I found 187 phantom transcripts in one week of production audio, and the obvious fix (turning up no_speech_threshold) barely helped. This post covers why that setting fails, what I measured on 400 test clips, and the setup that finally worked.

TL;DR

Whisper hallucinates on silence because it learned from web audio with subtitles. When it gets silence or room noise, it tends to output common subtitle filler like "Thank you.", "Thanks for watching!", or just "you".
no_speech_threshold alone won't fix it. A segment only gets skipped when no_speech_prob is high and avg_logprob is low. A confident hallucination has a high logprob, so it goes through.
On 400 non-speech clips, default faster-whisper large-v3 produced text on 112 (28%). Lowering no_speech_threshold to 0.3 still left 97.
Adding a VAD gate (vad_filter=True) dropped that to 9. VAD plus a small phantom-phrase filter got it to 2.
In production, phantom turns fell from 5.5% to 0.4%, and GPU decode time dropped 19% because silence never reaches the decoder.

Where did the phantom transcripts come from?

They came from long thinking pauses in a voice interview app. I run Preterview, a voice mock-interview platform that talks with you, scores your answers, and writes up a report (full disclosure: I built it). Candidates pause a lot. Good candidates pause even more, because they're thinking. Each pause gets chunked, sent to Whisper, and anything that comes back goes into both the interviewer LLM and the scoring prompt.

So a hallucinated "Thank you." isn't only messy text in a log. It becomes something the candidate "said" that the scorer reads. One report marked a candidate down for "abruptly ending the answer." All they had done was breathe.

How often does Whisper hallucinate on silence in production?

It happened in 5.5% of turns in my case. To count it, I flagged every turn where the transcript had text but the client-side audio level never went above the speech floor. Then I checked all flagged turns by hand.

One week, 3,412 transcribed turns, 187 confirmed phantoms:

Phantom output	Count
"Thank you."	71
"Thanks for watching!"	38
"you"	29
Repetition loops ("I'm going to I'm going to...")	11
"Subtitles by the Amara.org community" and similar credits	6
Other short filler	32

Caveat: this is a lower bound. My flag only catches turns that were entirely silent. If a phantom "Thank you." got added to the end of a real answer, the audio level check doesn't see it. I know those exist because I saw some while checking the others. I just didn't count them.

Why doesn't no_speech_threshold stop Whisper hallucinations?

Because it isn't the only condition. The skip logic has an override. Here's the logic from Whisper's transcribe loop, paraphrased (faster-whisper does the same thing):

should_skip = result.no_speech_prob > no_speech_threshold

if log_prob_threshold is not None and result.avg_logprob > log_prob_threshold:
    # confident output wins, even if the model thinks there's no speech
    should_skip = False

Look at the second condition. Whisper will output text over silence as long as it's confident about that text. And "Thank you." is about the most confident thing it can say, because it saw that phrase at the end of countless subtitled videos. The model can believe there's no speech (no_speech_prob 0.8) and still output its favorite outro with an avg_logprob of -0.3. The logprob check wins, and the segment is kept.

So changing no_speech_threshold only changes one of the two checks. The override still applies.

What actually fixes Whisper hallucinations on silence?

A voice activity detector in front of Whisper, so silence never reaches the decoder. Before changing anything in production, I built a test set: 400 clips of 2-8 seconds with no speech (my room, a laptop fan, keyboard clatter, breathing, AC hum, a door closing down the hall). I also recorded 400 short clips with real speech, including one-word answers, to see what each fix threw away.

Same model for every run: faster-whisper large-v3, float16, one GPU.

Config	Phantoms (400 silent clips)	Real speech lost (400 speech clips)
Defaults	112	0
`no_speech_threshold=0.3`	97	1
`condition_on_previous_text=False`	104	0
`log_prob_threshold=None` (disables override)	41	6
`vad_filter=True`	9	3
VAD + phantom filter	2	3

Here's what I took from it:

Lowering the threshold did almost nothing. From 112 to 97. The logprob override kept most of the phantoms.

Disabling the override worked, but it cost real answers. With log_prob_threshold=None, skipping depends only on no_speech_prob. Phantoms dropped to 41, but 6 real answers disappeared. They were all short and quiet: "Yes.", "No.", "Right." In an interview, losing a candidate's "No." is worse than a fake "Thank you."

condition_on_previous_text=False isn't a silence fix. It barely changed the count. What it does fix is the repetition loops, where one hallucination gets fed back as context and the model keeps repeating it. I keep it off for that reason.

VAD did most of the work. Silero VAD (included in faster-whisper) cuts non-speech out before decoding. If Whisper gets no audio, it has nothing to hallucinate over.

What does the final configuration look like?

VAD first, conditioning off, then a narrow filter for the known phantom phrases:

from faster_whisper import WhisperModel

model = WhisperModel("large-v3", device="cuda", compute_type="float16")

PHANTOMS = {
    "thank you.", "thank you", "thanks for watching!",
    "thanks for watching.", "you", "bye.",
}

def transcribe_turn(audio):
    segments, _ = model.transcribe(
        audio,
        vad_filter=True,
        vad_parameters={"min_silence_duration_ms": 500},
        condition_on_previous_text=False,
    )
    kept = []
    for seg in segments:
        text = seg.text.strip()
        # only drop a known phantom when the model itself doubts there was speech
        if text.lower() in PHANTOMS and seg.no_speech_prob > 0.3:
            continue
        if seg.compression_ratio > 2.4:  # repetition loop
            continue
        kept.append(text)
    return " ".join(kept)

The filter depends on no_speech_prob on purpose. Real candidates do say "Thank you." at the end of an interview, and I don't want to delete that. The phrase only gets dropped when it matches the list and the model had doubts about speech being there. Across my 400 speech clips, that rule removed zero real phrases.

The 3 real answers that VAD lost were single syllables under about a quarter second. I didn't tune them back in. Instead, the interviewer treats an empty turn as "still thinking" and waits, so if a "yeah" gets dropped, the candidate hears a short pause instead of a made-up reply. That's an acceptable failure.

What changed in production?

The week after I shipped it: 3,690 turns, 14 phantom transcripts, down from 5.5% to 0.4%. Most of the remaining 14 were loud non-speech sounds (a cough, a chair scraping) that VAD let through as speech. Whisper then decided they were "you."

I also got something I hadn't planned for: GPU decode seconds fell 19%. Candidates spend a lot of an interview in silence, and none of it is decoded anymore.

Worth noting what didn't change. The scoring prompt still gets the occasional "you." I added a line to it telling the model that one-word fragments may be transcription artifacts, not answers. That's a patch on top of the fix, not the fix itself.

So why does Whisper hallucinate on silence, and how do you stop it?

Whisper hallucinates on silence because it learned from subtitled web audio, so with no speech to transcribe it outputs the most likely subtitle filler: "Thank you.", "Thanks for watching!", "you". Raising no_speech_threshold doesn't stop it, because a segment is only skipped when no_speech_prob is high and avg_logprob is low, and confident hallucinations have high logprobs. What works is removing silence before Whisper ever sees it: set vad_filter=True in faster-whisper, turn off condition_on_previous_text to stop repetition loops, and drop known phantom phrases only when no_speech_prob is also high. On my 400 silent test clips, that took phantoms from 112 to 2.

Written by the developer behind Preterview, an interview prep platform.

DEV Community