From 'ja' to 'JP': How a Jabbering Model Emerges - A Catalog of Pitfalls in Speech Learning

📝 Originally published (in Japanese) at forge.workstyle.tech.

Here are the pitfalls I encountered while training nearly 20 voice synthesis models. All of them resulted in incorrect outcomes without any errors, and it took a significant amount of time to identify the root causes.

1. Using "JP" leads to Japanese being trained as English

There's a parameter for passing the language to the training API. I wrote it straightforwardly:

resp = await client.post(f"{TRAINER_URL}/train", json={
    "model_id": model_id,
    "language": "JP",          # ← This is the issue
    "training_audio": clips,
})

The training completed successfully without any errors or warnings, and the loss decreased.

However, when I tested the resulting model, it only produced babbling sounds – meaningless sequences of sounds that didn't form valid Japanese phonemes.

Reading the trainer's code revealed the reason:

if language.startswith("ja"):
    phonemes = japanese_g2p(text)
else:
    phonemes = english_g2p(text)      # ← This is where it ends up

"JP".startswith("ja") returns False because it's case-sensitive and doesn't match. As a result, Japanese text was processed through the English G2P, leaving the phoneme sequence almost empty during training.

Even with empty phonemes, the training process continues, and the loss decreases (since the model converges to producing average sounds without learning anything). No abnormalities appear anywhere in the pipeline.

Using "ja" fixes the issue. ISO 639-1 specifies lowercase letters, and "JP" is a country code (ISO 3166), which I had confused.

Lesson learned: When passing language codes, log the phoneme sequence once after passing it to visually inspect it. If it's empty or looks like English, you'll know immediately. Discovering this after starting training is too costly.

2. curl --data @file causes OOM errors with 280MB payloads

When manually triggering training with large datasets, I sometimes use curl:

curl -X POST http://trainer:8000/train \
  -H "Content-Type: application/json" \
  --data @payload.json          # 280MB

This results in an OOM error. curl reads the entire file into memory with --data @file and allocates additional buffers to handle it as JSON. A 280MB file can consume several GB.

Sending the request from within a Pod using Python works:

import json, urllib.request
req = urllib.request.Request(
    "http://trainer:8000/train",
    data=open("payload.json", "rb").read(),
    headers={"Content-Type": "application/json"},
)
urllib.request.urlopen(req, timeout=3600)

This still reads the file but avoids curl's excessive buffering, making it manageable. Ideally, implementing streaming uploads or avoiding base64 encoding of audio in JSON is the proper solution, but for manual operations, the above Python approach suffices.

Lesson learned: Always test large payload handling procedures with expected sizes. "Just use curl" only works for payloads up to a few dozen MB.

3. CUDA assertion errors recur during continuous training

After training multiple models consecutively, this error occurs at some point:

SBV2 train 失敗 (rc=-6): ...device-side assertions.
Exception raised from c10_cuda_check_implementation at
  ../c10/cuda/CUDAException.cpp:43

rc=-6 indicates a SIGABRT, meaning a CUDA device-side assertion failure.

Setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True helped by allowing the memory allocator to reuse fragmented segments, which improved the situation.

env:
  - name: PYTORCH_CUDA_ALLOC_CONF
    value: "expandable_segments:True"   # Mitigates NVML assert during continuous training

However, this wasn't a complete fix. Even with this setting, the error reoccurred after approximately 15 consecutive training sessions. Restarting the Pod resolves the issue.

This setting only prolongs the time until failure; eventually, the same state will occur after extended operation. For large-scale training, restarting the service every 10 or so sessions became the operational norm.

kubectl rollout restart deployment/tts-trainer -n <ns>
kubectl rollout status  deployment/tts-trainer -n <ns> --timeout=600s

⚠️ Restarting kills any ongoing training, so time it when the queue is empty. In this case, restarting during "clip generation" (which uses a separate GPU service) avoided interrupting active jobs.

Lesson learned: For GPU-dependent issues, even if a setting seems to fix it, track how many sessions it takes to recur. Knowing it recurs after 15 sessions allows for operational adjustments like restarting every 10 sessions.

4. Reference audio length has an upper limit

Assuming longer zero-shot reference audio improves quality, I extended it until hitting a wall:

55 seconds → Works
110 seconds → Fails with NVML assert

More problematic is that smaller GPU slices fail even at 55 seconds due to memory fragmentation during sequential generation. Even lengths that work in isolation fail during mass production.

Two solutions:

• Allocate larger GPU slices for mass production
• Use expandable_segments:True (helps with fragmentation)

Lesson learned: "Works in isolation" doesn't mean "works consecutively." Conducting a small-scale rehearsal with the same number of sessions as production reveals such degradation.

5. Reference audio registration is lost on Pod restart

Reference audio stored in a temporary directory is lost on Pod restart. Unnoticed during training, the next job fails with "reference not found."

volumes:
  - name: voices
    emptyDir: {}          # ← Lost on Pod restart

Lesson learned: For "upload and register" resources, verify persistence of storage upfront. During development, emptyDir works fine until the Pod restarts, making the issue unnoticed.

6. Shared images cause duplicate workers in different services

A backend with training workers and another service shared the same container image. As a result, workers started in both, causing port conflicts for distributed training.

# Explicitly disable workers in the non-training service
env:
  - name: VOICE_DESIGN_WORKER_ENABLED
    value: "false"

Lesson learned: When sharing images, explicitly specify what not to run in each deployment via environment variables. Defaulting to starting everything leads to accidents when sharing images.

7. Speaking style is baked into the corpus and can't be changed later

This is more of a design characteristic than a pitfall, but it's crucial to know.

Intonation, speaking speed, and emotional style are determined by the training corpus content, not adjustable at synthesis time. Testing synthesis API parameters for speaking speed showed no effect on output.

length=1.00 → Speed 5.3
length=1.15 → Speed 5.2
length=1.35 → Speed 5.1
length=0.85 → Speed 5.2      ← Within error margin; ineffective

The interface accepts parameters, but the implementation ignores them. "Acceptable" and "effective" are different; empirical testing is necessary.

Thus, use cases (narrator, call center, host, etc.) must be decided during manufacturing (speaking style can't be changed after training). For different styles with the same voice, create another model with the same design values (caption + seed) but varying speaking style. Each takes 1-2 hours, making it operationally feasible.

Lesson learned: For parameters assumed adjustable at runtime, empirically test their effectiveness. If ineffective, design with the assumption of deciding them upfront. Proceeding under false assumptions leads to unrecoverable situations.

---

Looking at these seven pitfalls, a common thread emerges: none of them produce errors.

The language code training completes, speaking speed parameters are accepted, duplicate workers partially succeed, and reference audio loss only surfaces in the next job. They appear to work but don't produce the intended results.

To reduce such issues, verify whether the intended content is produced, not just whether the process completes. Log phoneme sequences, measure speaking speeds, log worker enablement status at startup – each takes minutes but saves hours later.

---

Series: Mass-producing practical voices from diffusion TTS

This series documents designing voices from a single caption, manufacturing training corpora, and mass-producing role-specific practical voices. This article is Part 4: Operations.

← Previous: 70 minutes of training material lost to a network blink

→ Next: Four registration paths, zero management screens

Full series (18 articles)

1. The TTS chosen for quality was too slow for conversation
2. Voice gacha
3. Letting a machine select "narrator-like voices" from 24 candidates
4. The stricter the quality gate, the more monotone voices survive
5. Speaking style can't be changed after training
6. TTS that changes "recording location" every generation
7. One rough clip ruins the entire style
8. Where did the elongated ending come from?
9. Allowed character list was trimming Japanese
10. Hallucination guard that never fired
11. Three characters became a verbal tic
12. Fixable defects were disqualifying candidates
13. Defects invisible to transcription
14. 70 minutes of training material lost to a network blink 15. Writing "ja" as "JP" creates a babbling model ← You are here
15. Four registration paths, zero management screens
16. Deployments kept overwriting each other's work
17. Chasing unmeasured metrics with thresholds always fails

The insights are compiled in the notebook: Mass-producing practical voices from diffusion TTS manufacturing pipeline.