📝 Originally published (in Japanese) at forge.workstyle.tech.
Building an Emotion-Expressive TTS Model: How Quality Gates Can Backfire
I was working on an automated pipeline to generate training data for an emotion-expressive TTS model. For each of 12 emotions (joy, sadness, anger, fear, etc.), I prepared several audio clips with the target emotion applied. Naturally, this required quality control. I used Whisper to transcribe the generated audio and only kept clips that matched the script exactly.
The resulting model ended up sounding flat and monotone.
The issue turned out to be the quality control process itself.
Switching Emotion Styles Still Sounds Like the Same Voice
This model has distinct "styles" for each emotion—joy style, sadness style, etc.—and can synthesize speech while switching between them. However, when switching styles, the perceived difference in the audio is minimal.
The numbers made this painfully clear. I measured the cosine similarity between each emotion-style synthesis and a neutral style. If the emotion was properly applied, the similarity should decrease (i.e., the value should be smaller).
Bulk-generated corpus: cos 0.77–0.94
A well-made individual: cos 0.164
A cosine similarity near 0.9 means that even when using the "joy" style, the output sounds almost identical to the neutral voice. The emotion styles were effectively non-functional.
Honestly, when I first listened to the samples, I thought, "Eh, it's fine." It wasn't until the numbers showed 0.9 that I realized something was seriously wrong.
Three Root Causes
Two of these were configuration issues, but the third is the real culprit.
1. Speaker CFG Settings Cause Interjections to Sound Like a Different Person
When scripts started with interjections like "Waa!" or "Eh!", the beginning of the clip would sound like a completely different voice. The parameter controlling fidelity to the reference audio was breaking speaker consistency when emotion was applied.
2. Emojis Get Read as Audio
I had included emojis in the script as emotion markers, but they were either being read aloud ("happy face") or triggering unintended sound effects. These markers should have been placed outside the text.
3. Whisper-Only Validation Favors Flat Takes
This was the main issue.
Validation Rejects Emotion-Rich Audio
The pipeline generates multiple candidate clips and keeps only those that pass Whisper validation. Here's what happens:
-
Emotion-rich audio tends to have:
- Trembling voice (fear)
- Extended or rising pitch at the end (joy)
- Volume distortion (anger)
- Fading endings (sadness) All of these make recognition harder for Whisper, causing the transcription to deviate from the script and fail validation.
Flat, monotone takes, on the other hand, are clear and easy to recognize. They pass validation effortlessly.
So when filtering clips based on "does it match the script?", emotion-sparse takes are disproportionately selected. The stricter the gate, the stronger this bias becomes. The better the gate works, the flatter the corpus becomes.
The goal (training an emotionally expressive voice) and the method (selecting based on script fidelity) were directly at odds. And since every component was functioning correctly, no errors were thrown—making the issue hard to detect.
Split Validation into Two Stages: Filtering and Ranking
The solution was to split the selection process into two stages.
# Before: Take the first valid clip
for seed in seeds:
wav = gen(text, ref, seed)
if judge(text, whisper(wav)).ok:
return wav # ← First valid = flattest take
# After: Collect all valid clips, then pick the most emotionally expressive
candidates = []
for seed in seeds:
wav = gen(text, ref, seed)
if judge(text, whisper(wav)).ok: # ① Filtering
candidates.append((wav, style_distance(wav, neutral_ref)))
if not candidates:
return None
return max(candidates, key=lambda c: c[1])[0] # ② Rank by style distance
Stage ① is just a quality filter—it doesn't determine ranking. Stage ② uses a separate metric to rank candidates.
How to Measure "Emotion Applied"
To measure whether emotion is applied, we use style embedding distance between the synthesized clip and a neutral version of the same speaker. Since the TTS model can extract style vectors from audio, we leverage that.
def style_distance(wav, neutral_wav):
a = extract_style_vector(wav) # Use TTS's embedding extraction API
b = extract_style_vector(neutral_wav)
cos = np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
return 1.0 - cos # Larger = more distant
If embedding extraction isn't available, you can approximate using combinations of:
- Median F0
- F0 range (in semitones)
- RMS variance
The key idea is to measure relative deviation from neutral, not absolute values. Trying to define absolute thresholds like "joy should sound bright" forces you to adjust thresholds per speaker. Relative distance avoids this dependency.
Don’t Use a Single Threshold for Filtering
The acceptance criteria now use two conditions in an OR:
def accept(text, transcript):
v = judge_transcript(text, transcript)
tail = trailing_elongation_mismatch(text, transcript) # Unscripted elongation
return (v.ratio >= 0.82 and tail <= 2) or (v.ratio >= 0.70 and tail == 0)
We allow slightly lower transcription accuracy if there’s no unscripted elongation. This tolerates minor recognition errors due to emotional expression but strictly rejects cases where the model "invents" elongated endings.
The reason we can’t relax the elongation check is that the model learns the habit of elongating endings if such clips sneak into the corpus. Early on, the model started rendering "Konnichiwa" as "Konnichiwaaa"—and it turned out the corpus included clips with elongated endings ([[where-did-the-elongated-ending-come-from|Where did the elongated ending come from?]]).
⚠️ Important: The elongation check must be performed on raw Whisper transcriptions. Japanese normalization drops long vowel symbols, so comparing normalized strings won’t detect elongations. The acceptance criteria use different string inputs for transcription accuracy vs. elongation detection.
Emotion Anchor Method
A further refinement was needed. The idea is to first create a strong "anchor" for each emotion.
1. For each emotion, generate multiple clips from out-of-corpus "high-emotion" sentences using different seeds
2. From the valid clips, select the one with maximum style distance → register as the anchor
3. Use this anchor as the reference when generating the main corpus
4. For main corpus lines, again select the clip with maximum style distance from valid candidates
In code:
# Stage 1: Finalize emotion anchors
ANCHOR_TEXTS = {
"joy": "やった、ついにできましたね！本当に、本当に嬉しいです！",
"fear": "怖い、怖いです。どうしよう。",
"sadness": "もう、どうにもならないんです……。",
...
}
anchors = {}
for emo, text in ANCHOR_TEXTS.items():
cands = []
for seed in ANCHOR_SEEDS: # Try multiple seeds
wav = gen(text, base_ref, seed) # Use long reference for speaker consistency
if accept(text, whisper(wav)):
cands.append((wav, style_distance(wav, neutral)))
anchors[emo] = max(cands, key=lambda c: c[1])[0] # Most emotionally expressive
register_voice(f"anchor_{emo}", anchors[emo])
# Stage 2: Generate main corpus using anchors as reference
for emo, lines in CORPUS.items():
for line in lines:
cands = [w for w in (gen(line, anchors[emo], s) for s in SEEDS)
if accept(line, whisper(w))]
clip = max(cands, key=lambda w: style_distance(w, neutral))
save(clip, line, group=emo)
The idea is to first lock in a canonical example of "this voice, when expressing this emotion, sounds like this." Then use that anchor as a reference when mass-producing the corpus. The bulk generation failed because it tried to simultaneously optimize emotion expression and speaker identity in one shot. With anchors, emotion is inherited from the reference prosody, while speaker identity is preserved by the anchor itself.
A key detail is that anchor sentences are not from the main corpus. If you reuse corpus sentences as anchors, you end up with an imbalanced dataset where one sentence is extremely expressive.
Anchor method: cos 0.45–0.61
Bulk generation: cos ~0.8
Pitfalls Encountered
Trembling emotions need different anchor settings. For emotions like fear where the voice trembles, setting a high fidelity parameter caused the first and second halves of the clip to sound like different people. You need to adjust settings per emotion type. Increasing fidelity reduces emotional expression, so you must decide per emotion whether to prioritize speaker consistency or emotional expression.
Stuttering written in the script sounds unnatural. Phrases like "da, dare desu ka" ("who, who are you?") or "ya, yada" ("no, no") cause the TTS to render them unnaturally. Rewriting as repeated words—"Kowai, kowai desu" ("scary, scary")—produces more natural results. This is the opposite of how you'd coach a human actor.
Reference audio has length limits. Clips up to 55 seconds work, but at 110 seconds the GPU throws an assertion error. With sequential generation, memory fragmentation causes crashes even at 55 seconds on smaller GPU slices. If using long references, set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and allocate a sufficiently large GPU slice.
When Filtering and Ranking Use the Same Metric, Bias Occurs
I think this failure pattern isn't limited to TTS.
- In code generation, selecting only for "tests pass" favors simplistic implementations.
- In summarization, selecting for "high overlap with source" favors extractive summaries.
- In image generation, selecting for "prompt fidelity" favors bland compositions.
In all cases, the quality check is technically correct, but it’s the wrong criterion for ranking. The filter should only eliminate bad candidates; ranking should use a separate metric aligned with the ultimate goal.
The insidious part is that this bias produces no errors. Every component works correctly, resulting in a perfectly flat dataset. You only notice when you listen to the final output and think, "Something’s off," then backtrack to find the root cause.
This reinforces a mundane truth: even automated pipelines need a final sanity check on the actual output. But in this case, even listening wasn’t enough—I almost dismissed the flatness as acceptable until the cosine similarity of 0.9 revealed the truth. Both intuition and numbers are essential.
Series: Scaling Practical Voices from Diffusion TTS
A record of designing voices from single captions, manufacturing training corpora, and mass-producing role-specific voices.
This article is Part 2: Manufacturing.
← Previous: [[screening-voices-by-metrics-not-ears|"Narrator-like voice" selected from 24 candidates by machine]]
→ Next: [[speaking-style-is-baked-into-the-corpus|You can't change speaking rate after training]]
Full series (18 parts)
- [[diffusion-tts-too-slow-for-conversation|TTS chosen for quality was too slow for conversation]]
- [[deterministic-voice-gacha-and-design-ledger|Voice gacha with a design ledger]]
- [[screening-voices-by-metrics-not-ears|"Narrator-like voice" selected from 24 candidates by machine]] 4. [[quality-gate-selection-bias-flat-takes|Quality gates favor flat takes over expressive ones]] ← you're here
- [[speaking-style-is-baked-into-the-corpus|You can't change speaking rate after training]]
- [[tts-changes-recording-room-every-time|TTS changes "recording room" every generation]]
- [[one-rough-clip-ruins-the-whole-style|One rough clip ruins the entire style]]
- [[where-did-the-elongated-ending-come-from|Where did the elongated ending "konnichiwaaa" come from?]]
- [[the-character-that-broke-the-tts-input|The character that broke TTS input: "slightly" became "shomo"]]
- [[hallucination-guard-that-never-fired|Hallucination guard that never fired when it should have]]
- [[three-chars-became-a-verbal-tic|Quality gate-approved "3 characters" became a verbal tic]]
- [[measuring-factory-defects-as-product-traits|Treating fixable defects as product traits]]
- [[defects-invisible-to-transcription|Some defects are invisible to transcription]]
- [[70-minutes-lost-to-a-network-blink|70 minutes of training material lost to a network blink]]
- [[ja-vs-JP-babbling-model|How "ja" became "JP" and broke the babbling model]]
- [[four-registration-paths-one-exit|Four registration paths, zero management UI]]
- [[who-is-rolling-back-whom|Deployments kept rolling back each other's work]]
- [[chasing-unmeasured-targets-with-thresholds|Chasing unmeasured targets with thresholds guarantees failure]]
All insights are consolidated in [[Diffusion TTS Manufacturing Pipeline to Scale Practical Voices]].
Top comments (0)