📝 Originally published (in Japanese) at forge.workstyle.tech.
I Found a Diffusion TTS That Generates Voices from Just a Caption
"Calm adult female narrator voice. Slow, warm, and trustworthy delivery,
carefully reading long sentences."
Pass this description, and it will speak in exactly that voice. You can generate any Japanese voice without preparing even one second of speaker audio. Moreover, the same caption and random seed will always produce the same voice.
I thought this could be useful for interactive avatars—eliminating the hassle of preparing voices for each character.
In short: it didn’t work for conversations. But it was too valuable to discard, so I repurposed it.
Benchmark: 2.5x Slower on the Same GPU
Our existing system used a pre-trained TTS (Style-Bert-VITS2-based). We synthesized the same 7.5-second sentence on the same GPU slice to compare.
| Engine | Conditions | Generation Time | RTF |
|---|---|---|---|
| Diffusion TTS 40 steps | 2g.20gb | 1.9–2.1s | 0.25–0.28 |
| Diffusion TTS 24 steps | 2g.20gb | 1.4–2.0s | 0.19–0.27 |
| Diffusion TTS 16 steps | 2g.20gb | 1.1–1.4s | 0.15–0.19 |
| Pre-trained Model | 2g.20gb | 0.74–0.81s | 0.115–0.127 |
RTF (Real Time Factor) = generation time ÷ audio length. Lower is faster.
At 40 steps, it’s 2.5× slower. Even cutting steps to 16 still leaves a 1.5× gap. And below 16 steps, quality collapses (6 steps was immediately rejected on listening).
On a smaller GPU slice, the gap widens further.
| Engine | Conditions | Generation Time | RTF |
|---|---|---|---|
| Diffusion TTS 40 steps | 1g.10gb | 3.4–4.0s | 0.46–0.54 |
| Diffusion TTS 16 steps | 1g.10gb | 2.0s | 0.27 |
In conversation, audio is synthesized and played sentence-by-sentence. The delay until the first sentence plays determines the user experience. A 1–2 second lag per turn is prohibitive.
Upgrading GPUs Didn’t Solve It
I considered allocating more GPU resources—from 2g.20gb to 7g.80gb (4× the compute).
But measurements showed a fixed overhead of ~1.1 seconds. Even cutting steps to 12, generation time bottomed out at 1.1–1.4 seconds. This is due to model loading and text processing—parts that don’t benefit from parallelism.
So even with 4× the resources, the fastest possible generation would be ~1.0–1.2 seconds—still slower than the pre-trained model’s 0.8 seconds. The cost multiplies. We decided not to adopt it.
A useful takeaway: measure fixed overhead first. By reducing steps to the point where performance stops improving, you can estimate how much of the bottleneck is parallelizable.
Splitting the Workflow
Speed was the only downside. But diffusion TTS offers something pre-trained models can’t: generating voices from captions alone. Pre-trained models can only reproduce voices seen during training.
So we split the pipeline:
[Design Phase] Diffusion TTS ── caption + seed → voice ── Training Corpus (~200 clips)
↓
Training
↓
[Runtime] Pre-trained Model ────────────────── Voice heard by users
During design, diffusion TTS creates “this kind of voice.” We record ~200 lines with that voice to build a training corpus, then train a lightweight model for runtime use.
Since diffusion TTS runs only once per voice, its slowness isn’t a problem. Generating one voice takes ~70 minutes, but it’s a batch process.
Conditions for This Split to Work
Deterministic generation. The same caption and seed must always produce the same voice. Without this, we couldn’t reproduce the designed voice, and every retraining would yield a different speaker.
Thanks to this property, we only need to store the design ledger to recreate the model. Even if we lose the trained model file, we can regenerate the exact voice from the caption and seed. (We once lost the driver script, but recovered the voice by restoring the caption and seed from logs: [[deterministic-voice-gacha-and-design-ledger|Voice Gacha and the Design Ledger]]).
Audio quality must survive training. Since generated audio becomes training data, diffusion TTS’s audio quality sets the ceiling. We verified this by actually training and listening—no issues here.
Speaker identity must remain consistent. Over 200 clips, the voice must stay consistent. This was tricky. Starting sentences with exclamations sometimes changed the speaker at the beginning, and strong emotional expressions could break speaker identity. We solved this by carefully managing reference audio—but that’s another story.
Side Effect: Captions Control Prosody Too
After splitting the workflow, we discovered a constraint: captions don’t just define timbre—they also dictate speech rate and intonation.
We tried designing a male counselor voice with this caption:
A calm, gentle male voice. Low and slow, enveloping the listener with reassurance.
The result was criticized as “too low and too slow,” so we revised the caption:
A calm, gentle male voice. A moderately low voice that reassures the listener,
spoken at a natural pace with careful, warm delivery.
F0 rose from 105–159Hz to 136–159Hz—good. But speech rate jumped from 5.5–6.7 to 7.3–7.7 syllables per second.
Even adding “a bit slower” brought it only to 6.4–7.1. F0 and speech rate are coupled in the caption—you can’t adjust one independently.
After testing five variants, we concluded: separating pitch and tempo via caption alone is difficult. Raising pitch inevitably increases tempo.
Moreover, speech rate cannot be changed after synthesis (see [[speaking-style-is-baked-into-the-corpus|Speech Rate Is Baked Into the Corpus]] for measurements). This means the combination of pitch and tempo is fixed at caption-writing time, with no post-hoc adjustment.
Our new workflow: decide upfront, generate multiple candidates, and compare. Since generating one candidate takes only a few minutes, running 5 variants × 12 seeds to build a comparison table was feasible.
Summary
- Measure RTF on the same GPU and same text. Published specs aren’t comparable due to differing conditions.
- Measure fixed overhead first. Reduce steps to the point where performance plateaus; this reveals how much of the bottleneck is parallelizable.
- Even if slower, different capabilities can justify splitting roles. “Once at design time” vs. “every turn at runtime” tolerates vastly different latencies.
- Deterministic generation has intrinsic value. A design ledger lets you recreate the model from captions and seeds.
- Empirically verify what captions can control and what can’t be changed later. In our case, speech rate was “controllable at design time but unchangeable later.”
Series: Mass-Producing Practical Voices from Diffusion TTS
A record of designing voices from single-line captions, manufacturing training corpora, and mass-producing role-specific practical voices. This article is Part 1: Design.
(This article is the entry point.)
→ Next: [[deterministic-voice-gacha-and-design-ledger|Voice Gacha and the Design Ledger]]
Full 18-part series
1. [[diffusion-tts-too-slow-for-conversation|The TTS We Chose for Quality Was Too Slow for Conversation]] ← You are here
- [[deterministic-voice-gacha-and-design-ledger|Voice Gacha and the Design Ledger]]
- [[screening-voices-by-metrics-not-ears|“Narrator-like Voice” — Selecting from 24 Candidates Using Metrics, Not Ears]]
- [[quality-gate-selection-bias-flat-takes|The Stricter the Quality Gate, the More Monotone Voices Survive]]
- [[speaking-style-is-baked-into-the-corpus|Speech Rate Cannot Be Changed After Training]]
- [[tts-changes-recording-room-every-time|This TTS Changes the “Recording Room” Every Time]]
- [[one-rough-clip-ruins-the-whole-style|One Rough Clip Spoils the Entire Style]]
- [[where-did-the-elongated-ending-come-from|Why Does the AI Say “Kon-nichiwaaa”?]]
- [[the-character-that-broke-the-tts-input|“Slightly” Becomes “Shomo” — The Permissible Character List Was Trimming Japanese]]
- [[hallucination-guard-that-never-fired|A Hallucination Guard That Only Worked When It Shouldn’t]]
- [[three-chars-became-a-verbal-tic|A Quality Gate-Approved “3-Character” Phrase Became the Model’s Verbal Tick]]
- [[measuring-factory-defects-as-product-traits|We Were Rejecting Candidates Based on Fixable Flaws]]
- [[defects-invisible-to-transcription|Some Flaws Are Invisible to Transcription]]
- [[70-minutes-lost-to-a-network-blink|70 Minutes of Training Data Lost to a Network Blink]]
- [[ja-vs-JP-babbling-model|Changing “ja” to “JP” Made the Babbling Model Work]]
- [[four-registration-paths-one-exit|Four Registration Paths, Zero Management Screens]]
- [[who-is-rolling-back-whom|Deployments Kept Rolling Back Each Other’s Work]]
- [[chasing-unmeasured-targets-with-thresholds|Chasing Unmeasured Targets with Thresholds Always Fails]]
All insights are consolidated in [[拡散TTSから実用ボイスを量産する製造パイプライン|Mass-Producing Practical Voices from Diffusion TTS: Manufacturing Pipeline]].
Top comments (0)