DEV Community

orca_forge
orca_forge

Posted on Originally published at forge.workstyle.tech

How 'Shōshō' Became 'Shomo' — Permission Character List Was Trimming Japanese

#tts #unicode

📝 Originally published (in Japanese) at forge.workstyle.tech.

I received a report about the avatar for the inquiry desk:

"しょうしょうおまちください" becomes "しょもおまちください".

Since I had just retrained the voice model multiple times, I first suspected the model. To cut to the chase, the model, parameters, and cache were all normal, but the input text passed to TTS was corrupted.

Debugging from the downstream

I'll list my suspicions in the order I checked them. This order itself is a lesson learned.

Model: I traced the database to see which model the inquiry site was using. It was correctly using the latest trained model.

Cache: There was a TTS cache table, so I checked if it was returning old audio. The entries were all from a different provider and over a month old, so they were unrelated.

Synthesis parameters: The runtime was using style_weight=2.0 and sdp_ratio=0.8. My validation used 1.0 / 0.4, so I thought this might be the cause. I tested various combinations:

style_weight 1.0 / 2.0 / 3.0 / 4.0   → all ratio 1.00
sdp_ratio    0.2 / 0.4 / 0.6 / 0.8   → all ratio 1.00
All 12 styles × weight 2.0             → all ratio 1.00
Enter fullscreen mode Exit fullscreen mode

Emotional styles: I synthesized "少々お待ちください。" with all 12 styles, and they were all normal.

Pipeline: The voice pipeline had been moved to a separate service, so I checked if it was synthesizing independently. Synthesis was performed on the backend, and the pipeline was the same.

After several hours, everything checked out.

Printing the preprocessed output once

The only thing left was the input text.

>>> _clean_tts_text('少々お待ちください。')
'少お待ちください。'
Enter fullscreen mode Exit fullscreen mode

The "々" character was missing. And when this corrupted text was synthesized, it sounded like this:

Input '少々お待ちください' → Heard '少々お待ちください' (normal)
Input '少お待ちください'    → Heard 'ショーをお待ちください'   ← this
Enter fullscreen mode Exit fullscreen mode

"ショーを" was heard as "しょも". I did the 5-minute check last.

Cause: "々" is not in the CJK Unified Ideographs range

The preprocessor had a whitelist to remove emojis, emoticons, and special characters.

# TTS preprocessing: remove unnecessary symbols and emoticons
_TTS_ALLOWED_RE = re.compile(
    r"[^぀-ゟ"   # Hiragana
    r"゠-ヿ"     # Katakana
    r"一-鿿"     # CJK Unified Ideographs
    r"ｦ-ﾟ"     # Half-width Katakana
    r"a-zA-Zａ-ｚＡ-Ｚ"
    r"0-9０-９"
    r"、。！？,.ー"
    r"\s"
    r"]"
)
Enter fullscreen mode Exit fullscreen mode

The intention is clear, and the implementation is straightforward. The problem is that 一-鿿 (CJK Unified Ideographs) does not include "々". "々" is 々, which is in the CJK Symbols and Punctuation block. It's classified as a symbol, not a kanji character.

For Japanese speakers, "々" is considered a kanji character, but Unicode classifies it differently.

I found 7 missing characters

Since I found one missing character, there must be others. I checked all characters used in Japanese that are outside the CJK Unified Ideographs.

Character Example Result Impact
々 少々・日々 少・日 "しょも"
〆 〆切 切 "きり"
〇 〇月〇日 月日 Dates disappear
𠮷 (CJK Extension A) 𠮷野家 野家 Proper nouns break
髙 﨑 (Compatibility Ideographs) 﨑山 山 Names break
〜 10〜20分 1020分 Numbers become something else
: 3:30 330 Times become something else

At the inquiry desk, "﨑山さま" becoming "山さま" is quite serious. "10〜20分" becoming "せんにじゅっぷん" is similarly problematic.

On the other hand, % & 「」 are also removed, but this is intentional as they're unnecessary for reading. It's not about keeping everything.

Keeping symbols didn't fix it

I straightforwardly added 〜 and : to the allowlist. It got worse.

'午後330に開始します'   → Heard '午後330に…'          (incorrect number)
'午後3:30に開始します'  → Heard '5も30、30に開始します'  ← worse when kept
Enter fullscreen mode Exit fullscreen mode

TTS couldn't interpret : as a time and produced garbage like "ごも". 〜 was treated as a comma, not "から".

Removing changes the meaning, keeping makes it unreadable. Neither was correct.

Opening up to Japanese

The correct solution was a third option: convert to Japanese at the preprocessing stage.

_TTS_CLEAN_PATTERNS = [
    # ⚠️ Symbols with numerical meaning: neither remove nor keep, but "open to Japanese".
    # Removing turns "10〜20分" into "1020分", and keeping makes TTS unreadable,
    # e.g., "3:30" becomes garbage like "5も30、30" (verified).
    (re.compile(r"(\d)\s*[〜～~]\s*(\d)"), r"\1から\2"),   # 10〜20 → 10から20
    (re.compile(r"(\d{1,2})\s*[:：]\s*(\d{2})"), r"\1時\2分"),      # 3:30 → 3時30分
    ...
]
Enter fullscreen mode Exit fullscreen mode

⚠️ Place substitutions before removal. Otherwise, 10〜20 becomes 1020 first, and the substitution target disappears.

And add necessary characters to the allowlist.

r"一-鿿"     # CJK Unified Ideographs
# ⚠️ Characters necessary for Japanese reading but outside CJK Unified Ideographs.
# In practice, "少々お待ちください" became "少お待ちください",
# and was pronounced as "ショーをお待ちください".
r"々〆〻"  # 々 〆 〻 (repeating characters, abbreviations)
r"〇"           # 〇 (Chinese numeral zero)
r"㐀-䶿"    # CJK Extension A (variant characters, names)
r"豈-﫿"    # CJK Compatibility Ideographs (e.g., 髙 﨑 for names)
Enter fullscreen mode Exit fullscreen mode

Verified with actual audio:

'少々お待ちください'          → '少々お待ちください'       ✅
'髙橋・﨑山'                 → '髙橋・﨑山'              ✅
'10から20分ほどかかります'     → '10から20分ほどかかります'  ✅
'午後3時30分に開始します'      → '午後3時30分に開始します'   ✅
'受付は9時00分から17時00分です' → '9時0分から17時0分'        ✅
Enter fullscreen mode Exit fullscreen mode

⚠️ Wave dashes have another pitfall. 〜 (U+301C) and ～ (U+FF5E) are different characters, and which one is used varies by environment. Including only one will miss the other. I included both.

Also found: Pronunciation dictionary wasn't applied

During debugging, I discovered that the pronunciation dictionary wasn't applied at all for this inquiry site. The dictionary is scoped per project, and all 44 existing entries were tied to different projects.

I tested with the inquiry voice:

Notation Correct Reading Actual Reading
液冷 エキレイ できげ
主な オモナ オーナー
従量課金 ジューリョーカキン 重量価値
行っています オコナッテイマス 言っています

"行っています" becoming "言っています" is frequent in customer service phrases and completely changes the meaning.

Of the 44 entries, excluding 10 for company product names and personal names, 34 were general terms (technical terms and Japanese words with split kun/on readings) needed across all sites. I deployed these to 3 other projects.

Lessons learned

Check the input first. I spent hours eliminating the model, parameters, cache, and pipeline, only to finish by printing the preprocessed output once. The order was backwards.

Whitelists decide what to remove, not what to allow. If you list what to remove, unexpected characters pass through. Listing what to allow means any oversight immediately causes omissions. For languages with many character types like Japanese, comprehensive whitelists are difficult.

When it seems like a remove-or-keep choice, there's a third option. Symbols were caught between "removing changes meaning" and "keeping makes unreadable," but there was the option to open to Japanese. I was stuck thinking preprocessing was "where unnecessary things are removed," not "where form is changed while preserving meaning."

Check all characters in the same category. When I found "々", I investigated other characters that might be missing for the same reason. I found 7. If I had stopped at fixing one, the next report would have been about variant characters in names.

Series: Mass-producing practical voices from diffusion TTS

This is a record of designing voices from a single caption line, creating training corpora, and mass-producing role-specific practical voices. This article is Part 3: Quality Gates.

← Previous: [[where-did-the-elongated-ending-come-from|Where did the AI's habit of saying "こんにちわー" come from?]]

→ Next: [[hallucination-guard-that-never-fired|The hallucination guard code only failed when there was a hallucination]]

All 18 articles in the series

  1. [[diffusion-tts-too-slow-for-conversation|The TTS chosen for sound quality was too slow for conversation]]
  2. [[deterministic-voice-gacha-and-design-ledger|Drawing voices like gacha]]
  3. [[screening-voices-by-metrics-not-ears|Having a machine select "narrator-like voices" from 24 candidates]]
  4. [[quality-gate-selection-bias-flat-takes|The stricter the quality gate, the more monotone voices survive]]
  5. [[speaking-style-is-baked-into-the-corpus|Speaking speed can't be changed after training]]
  6. [[tts-changes-recording-room-every-time|TTS that sounds like it's recorded in a different room every time]]
  7. [[one-rough-clip-ruins-the-whole-style|One rough clip makes the entire style sound hoarse]]
  8. [[where-did-the-elongated-ending-come-from|Where did the AI's habit of saying "こんにちわー" come from?]] 9. [[the-character-that-broke-the-tts-input|"少々" becoming "しょも" — The allowlist was cutting Japanese characters]] ← You are here
  9. [[hallucination-guard-that-never-fired|The hallucination guard code only failed when there was a hallucination]]
  10. [[three-chars-became-a-verbal-tic|The "3 characters" the quality gate allowed became the model's verbal tic]]
  11. [[measuring-factory-defects-as-product-traits|Eliminating candidates over fixable defects]]
  12. [[defects-invisible-to-transcription|Defects transcription can't catch]]
  13. [[70-minutes-lost-to-a-network-blink|70 minutes of training material lost to a network blink]]
  14. [[ja-vs-JP-babbling-model|From "ja" to "JP": Creating a babbling model]]
  15. [[four-registration-paths-one-exit|Four registration paths, zero management screens]]
  16. [[who-is-rolling-back-whom|Deploying while overwriting each other's work]]
  17. [[chasing-unmeasured-targets-with-thresholds|Chasing unmeasurable targets with thresholds always fails]]

The insights are summarized in [[Manufacturing Pipeline for Mass-Producing Practical Voices from Diffusion TTS]].

Top comments (0)