TTS That Changes "Recording Room" Every Time You Generate — Style Falls Apart Unless You Match the Acoustics of Your Source Material

📝 Originally published (in Japanese) at forge.workstyle.tech.

Why Generated TTS Sounds Different Every Time

The training corpus for my voice model was generated using another TTS system. Same speaker settings, same model, same server. Yet the audio quality varies from clip to clip.

I first noticed this when switching emotional styles. When changing from a "joy" style to a "sadness" style, not only does the tone change, but the entire sound quality changes. One clip sounds like it was recorded up close, while the other sounds like it was recorded from a slightly farther distance. It feels like the same speaker is talking from different rooms.

The root cause was the training material. Each clip in the corpus had different frequency characteristics depending on the generation conditions.

Why Does the Sound Quality Change?

This corpus was generated using different reference audio (anchors) for each emotion ([[quality-gate-selection-bias-flat-takes|The stricter the quality gate, the more flat readings survive]]).

joy text     → generated with joy anchor
sadness text → generated with sadness anchor
anger text   → generated with anger anchor

Since the reference audio differs, the TTS not only mimics the tone but also the acoustic characteristics of the reference. Because each anchor was generated separately, they each have slightly different spectral shapes. As a result, the sound quality shifts between emotional groups.

Extreme performances (screams, laughter, etc.) have an even wider dynamic range. When trained together, the model learns not just the style but also things like "this style sounds distant."

This wouldn’t happen with human recordings, since they’re all recorded with the same microphone in the same room. But with generated audio, the TTS has no concept of recording environment, so changing conditions changes the channel characteristics.

Matching LTAS

The solution was to align all clips to a common frequency characteristic before training.

LTAS (Long-Term Average Spectrum) represents the average frequency distribution of the entire audio. It reflects speaker characteristics and room acoustics. We align all clips to this.

Here’s the process:

1. Compute the LTAS from the entire corpus or a reference clip group to get a reference spectrum
2. Compute the LTAS for each clip and calculate the difference from the reference
3. Apply EQ to each clip to cancel out the difference
4. Finally, normalize the RMS

def match_ltas(wav, ref_ltas, max_gain_db=10.0):
    """Align clip LTAS to reference (±max_gain_db clip)"""
    spec = stft(wav)
    ltas = np.mean(np.abs(spec), axis=1)              # Average over time

    gain_db = 20 * np.log10((ref_ltas + eps) / (ltas + eps))
    gain_db = np.clip(gain_db, -max_gain_db, max_gain_db)   # Prevent extreme correction
    gain    = 10 ** (gain_db / 20.0)

    spec = spec * gain[:, None]
    return istft(spec)

Why Clip at ±10dB?

We set a limit on the correction amount. Without it, we’d artificially boost frequency bands that don’t exist in the original clip. Those bands contain no signal, only noise, so amplifying them just increases noise.

High frequencies are especially problematic—some generated clips have almost no content above 8kHz. Trying to match the reference would introduce hiss. Clipping at ±10dB prevents this kind of breakdown.

Zero-Phase Processing

When applying EQ, we avoid shifting the phase. Standard filters introduce group delay, causing time shifts at different frequencies. While subtle to the ear, this is undesirable for training material.

We either manipulate only the STFT magnitude while keeping the phase intact, or apply the filter in both forward and backward directions to cancel phase rotation (like filtfilt). In this case, we chose the former.

Finally, Normalize RMS

After aligning the frequency characteristics, we also normalize the volume.

def normalize_rms(wav, target_dbfs=-20.0):
    rms = np.sqrt(np.mean(wav ** 2))
    gain = 10 ** (target_dbfs / 20.0) / (rms + 1e-9)
    return np.clip(wav * gain, -1.0, 1.0)

We normalize to −20dBFS. The key is using RMS normalization instead of peak normalization, since peak normalization would be skewed by sudden loud sounds (like the start of a scream), causing inconsistent overall volume.

Results

The sudden change in sound quality when switching emotional styles disappeared. That was our primary goal.

But there was an unexpected side effect: extreme performance clips became trainable.

Before normalization, training with scream clips tended to degrade the entire model. Their high dynamic range and distinct spectra made it hard for the model to treat them as part of the same voice.

After normalization, the model started learning the temporal structure of screams—like rapid pitch sweeps. By removing volume and spectral differences, the remaining "movement" information reached the model.

So while normalization seems like "discarding information," it actually removes irrelevant information (channel characteristics) to highlight the important stuff (prosodic movement).

How Much Should We Align?

Overdoing it causes other problems.

Speaker identity can vanish. LTAS includes vocal tract characteristics, so aggressive alignment can dilute the speaker’s individuality. Since we’re working with a single-speaker corpus here, it wasn’t an issue—but with multiple speakers, caution is needed.

Emotional acoustic features can also be lost. Anger tends to have stronger high frequencies, while sadness is weaker. Completely aligning everything would erase these differences. The ±10dB limit helps here too—strong corrections are avoided, so emotional differences remain intact.

Choosing the reference is also critical. We used the average of the entire corpus, but another approach is to use "the best clips" as the reference. Using the average can be problematic if many clips are of poor quality, dragging the reference down.

Implementation Notes

Process all clips in bulk just before training. If you normalize one clip at a time during generation, you’ll be processing before the reference spectrum is even defined. Normalize everything together after alignment.

Keep the original clips. Store the normalization process so it can be redone. You might want to tweak parameters (max dB, target RMS) and retrain.

Listen to the normalized audio. Even if the numbers look aligned, listening might reveal noise or unnatural artifacts. Clips where the correction hits the limit need special attention.

# Record clips where correction reached the limit
clipped = np.sum(np.abs(gain_db_raw) >= max_gain_db) / len(gain_db_raw)
if clipped > 0.3:
    logger.warning(f"{clip_id}: Over 30% of correction hit the limit (clip too far from reference)")

If many clips hit this limit, they might be too different from the rest—consider excluding them rather than forcing alignment.

Summary

• Generated audio has no "recording environment," so changing conditions changes channel characteristics—a problem unique to TTS and not seen in human recordings
• Switching emotional references changes sound quality, causing noticeable jumps when switching styles
• LTAS matching + RMS normalization aligns frequency characteristics—limit corrections to ±10dB and avoid phase shifts
• Alignment highlights the important information—extreme performance styles now train properly due to preserved temporal structure
• Don’t over-align—speaker identity and emotional frequency cues can vanish. Limits are essential for quality

---

Series: Mass-Producing Practical Voices from Diffusion TTS

A record of designing voices from a single caption line, manufacturing training corpora, and mass-producing role-specific practical voices. This article is Part 2: Manufacturing.

← Previous: [[speaking-style-is-baked-into-the-corpus|You Can’t Change Speaking Rate After Training]]
→ Next: [[one-rough-clip-ruins-the-whole-style|One Poor Clip Can Ruin an Entire Style]]

Full Series (18 Parts)

1. [[diffusion-tts-too-slow-for-conversation|The TTS I Chose for Audio Quality Was Too Slow for Conversation]]
2. [[deterministic-voice-gacha-and-design-ledger|Rolling the Dice for Voice Selection]]
3. [[screening-voices-by-metrics-not-ears|Letting Machines Choose "Narrator-like Voices" from 24 Candidates]]
4. [[quality-gate-selection-bias-flat-takes|The Stricter the Quality Gate, the More Flat Readings Survive]]
5. [[speaking-style-is-baked-into-the-corpus|You Can’t Change Speaking Rate After Training]] 6. [[tts-changes-recording-room-every-time|TTS Changes the "Recording Room" Every Time You Generate Audio]] ← Now Reading
6. [[one-rough-clip-ruins-the-whole-style|One Poor Clip Can Ruin an Entire Style]]
7. [[where-did-the-elongated-ending-come-from|Where Did the AI’s Habit of Stretching "Konnichiwa" Come From?]]
8. [[the-character-that-broke-the-tts-input|How "Slightly" Became "Shomo" — The Allowed Character List Was Deleting Japanese]]
9. [[hallucination-guard-that-never-fired|The Hallucination Guard Code Only Worked When It Shouldn’t Have]]
10. [[three-chars-became-a-verbal-tic|The Quality Gate-Approved "3 Characters" Became the Model’s Verbal Tic]]
11. [[measuring-factory-defects-as-product-traits|Dropping Candidates for Fixable Defects]]
12. [[defects-invisible-to-transcription|Some Defects Can’t Be Found Through Transcription]]
13. [[70-minutes-lost-to-a-network-blink|70 Minutes of Training Material Lost to a Network Blink]]
14. [[ja-vs-JP-babbling-model|How Writing "ja" as "JP" Created a Babbling Model]]
15. [[four-registration-paths-one-exit|Four Registration Paths, Zero Management Screens]]
16. [[who-is-rolling-back-whom|Deployments Kept Rolling Back Each Other’s Work]]
17. [[chasing-unmeasured-targets-with-thresholds|Chasing Unmeasured Targets with Thresholds Always Fails]]

All insights from this series are compiled in [[Diffusion TTS Manufacturing Pipeline for Practical Voices]].