Practical Techniques to Improve Faster-Whisper Recognition Accuracy — Use Recent Conversations, Not Dictionaries, for initial_prompt

📝 Originally published (in Japanese) at forge.workstyle.tech.

Techniques That Actually Improved Accuracy When Running Faster-Whisper as STT for Real-Time Voice AI (Avatars/Voicebots)

While running faster-whisper as the STT component in a real-time voice interaction system (avatar/voicebot), I’ve compiled practical accuracy-improvement techniques that showed real-world effectiveness. These are not batch-style single-shot transcriptions, but insights specifically for integrating into conversational systems.

---

1. initial_prompt — The Most Effective Yet Most Misused Feature

Whisper’s initial_prompt acts as a lexical bias for the decoder by providing the "transcription that came just before this audio." When speech is ambiguous, the model tends to "fall back" to spellings present in the prompt. This is the most powerful countermeasure against misrecognizing proper nouns (e.g., "社員数" → "シャインズ").

Common Mistake: Passing a Dictionary of Proper Nouns

It’s tempting to think: "Just pass a list of company names and product names." But in real-world operation, this approach breaks down.

• Vocabulary varies per tenant (customer company) — the dictionary grows endlessly. Even with 20–30 entries, it can’t cover all in-field proper nouns.
• Ongoing operational cost for registration and updates.
• Long prompts have side effects (explained later).

What Actually Worked: Pass the Avatar’s Most Recent Utterance

In a conversational system, there’s a better vocabulary source than a dictionary: the text of your own avatar’s recent spoken output via TTS.

• Scripts and responses already contain proper nouns in correct spelling.
• As the conversation progresses, vocabulary relevant to the current topic is automatically included in the hint. Zero registration effort.
• Implementation is simple: maintain a deque of recent utterances and pass the last N characters.

class SttHint:
    def __init__(self, max_chars=160):
        self._items = deque()
        self._max = max_chars
    def add(self, text):          # Only add avatar speech
        ...
    def text(self):
        return "".join(self._items)[-self._max:]

segments, info = model.transcribe(
    audio, language="ja",
    initial_prompt=(hint.text() or None),
)

Two Critical Warnings

• Never include user-side STT results in the prompt. Once a misrecognition sneaks in, subsequent recognition gets pulled into that error and becomes fixed. Only include text whose spelling is guaranteed — your own utterances or scripts.
• Keep length between 150–200 characters. Longer prompts can cause whisper to hallucinate the prompt itself as heard audio, creating false positives.

---

2. Input Audio Quality — Half the Battle Is Won Before STT

In systems receiving microphone audio via WebRTC, the default Opus bitrate (~30 kbps) quietly matters. When migrating from an uncompressed PCM-over-WebSocket setup to WebRTC, we saw increased misrecognition in edge cases (e.g., overlapping speech).

• In the answer SDP, specify maxaveragebitrate=128000;useinbandfec=1 to control the sender’s encoder.
• In-band FEC also helps with packet loss over TURN relays.
• In systems using AEC (Acoustic Echo Cancellation), be aware that near-end suppression during double-talk can distort word onsets. This cannot be fixed at the STT stage, so combine this section’s improvements with Section 1’s hints and Section 5’s LLM correction.

---

3. Cutting Off Low-Confidence Results (avg_logprob) Is Dangerous

The intuitive idea — "Discard low-confidence results to reduce noise" — was invalidated by real measurements. Even genuine speech often has avg_logprob as low as -0.8. Setting a threshold causes valid utterances to be dropped, leading to the worst symptom: silent non-response (invisible to users).

• Do not apply thresholding. Instead, log the confidence continuously — it’s invaluable for diagnostics.
• Reject noise not by confidence, but via contextual judgment in downstream LLM/app logic.

---

4. Pathologies of Non-Speech Input — Silence and Pure Tones Are "Slow"

Feeding sine waves or silence into whisper causes it to repeatedly fall back to temperature sampling, resulting in 5–8 seconds for a 3-second input (measured). Real speech in the same environment completes in under 1 second.

• Using sine waves or silence for health checks or warm-ups skews latency measurements. Use real speech samples.
• Conversely, if you observe such slowness in production, it’s a sign that non-speech is leaking in — check VAD thresholds and microphone paths.

It’s worth enabling vad_filter=True (with Silero VAD) to remove silent segments. This improves both speed and reduces hallucinations.

---

5. Don’t Rely on STT Alone — LLM Post-Correction Is the Final Line of Defense

No matter how well you tune STT, certain errors — like misheard proper nouns during double-talk — will slip through. In conversational systems, the most cost-effective move is to tell the LLM that the input is an STT transcript.

[Interpretation of Misheard Input]
- Input is a speech recognition transcript and may contain phonetically similar mishearings (e.g., "社員数" → "シャインズ")
- If a word feels unnatural in context, search reference material or recent conversation for a phonetically close word and interpret accordingly.
- Begin response with: "Regarding your question about ◯◯..."
- If no phonetically close word is found and a neologism appears, do not fabricate an explanation. Instead, ask for clarification.

The last line is crucial. Without it, the LLM will confidently invent product descriptions for unknown words like "シャインズ" (observed in real systems). Think of it as explicitly granting the model the same contextual reading ability humans use when parsing garbled chat input.

---

6. Operational Trap: "False Degradation" Right After Deployment

• Faster-Whisper models often use lazy loading, adding tens of seconds of load time on the first request (especially on GPU).
• If co-located processes (e.g., TTS pre-warming) monopolize GPU or event loops, /transcribe responses can be delayed by tens of seconds — indistinguishable from code regression.
• Solution: After deployment, verify load completion logs before measuring performance. Embed warm-up confirmation in E2E tests. "Failures right after release" should first suspect environment issues.

---

Summary

Technique	Effect	Cost
Use avatar’s recent utterance as initial_prompt (no dictionary)	Most effective against proper noun errors	Tens of lines of code, zero operational overhead
Never include user STT results in prompt	Prevents error fixation	Design decision only
Opus 128kbps + in-band FEC	Improves edge-case robustness	A few lines in SDP munging
Avoid confidence-based filtering	Prevents silent non-response	Actually, remove it
Enable vad_filter + warm-up/measure with real speech	Improves speed and reduces hallucinations	Configuration only
Tell LLM input is STT + ask for clarification on neologisms	Final correction of misheard terms	A few lines in prompt

Rather than chasing STT-only benchmarks, optimize the entire pipeline — from input quality → hinting → recognition → LLM correction — to "sandwich" errors out of existence. This approach delivers noticeable accuracy gains faster.