DEV Community

orca_forge
orca_forge

Posted on Edited on Originally published at forge.workstyle.tech

Where Did the AI Learn to Elongate Its "Konnichiwa"?

#tts #whisper

πŸ“ Originally published (in Japanese) at forge.workstyle.tech.

The Elongated Ending that AI Learned

When I made my trained voice model say "こんにけは" (konnichiwa), it came out as "こんにけわぁ" (konnichiwaa) with an elongated ending. There was no indication of elongation in the script.

The issue was pointed out as follows:

When I say "konnichiwa," it comes out as "konnichiwaa" with an accent at the end. It sounds like something is mixed in.

The comment about "something being mixed in" was accurate, and indeed, something was mixed in. The training corpus contained audio clips with elongated endings.

The problem was that the mechanism to detect this was fundamentally flawed.

Script Comparison was Being Done

In corpus generation, I was using Whisper to transcribe the audio read by TTS and comparing it to the script.

def _kana(s: str) -> str:
    # Convert katakana to hiragana
    return "".join(chr(ord(c) - 0x60) if "γ‚‘" <= c <= "γƒΆ" else c for c in s)

_PUNCT_RE  = re.compile(r"[γ€γ€‚οΌοΌŸ!?…・\sγ€Œγ€γƒΌγ€œ,\.]")
_REPEAT_RE = re.compile(r"(.)\1+")

def _collapse(s):
    return _REPEAT_RE.sub(r"\1", _PUNCT_RE.sub("", s or ""))

def judge_transcript(script_text, transcript, ...):
    a = _kana(_collapse(script_text))
    b = _kana(_collapse(transcript))
    sm = difflib.SequenceMatcher(None, a, b)
    ...
Enter fullscreen mode Exit fullscreen mode

I was normalizing the text before comparing. This is a straightforward implementation.

Look at _PUNCT_RE. The characters being removed include γƒΌ (the long sound symbol). And _REPEAT_RE compresses consecutive identical characters into one.

Script: こんにけは
Transcript: こんにけわー

After normalization:
  Script β†’ こんにけは
  Transcript β†’ こんにけわ      ← "γƒΌ" is removed
Enter fullscreen mode Exit fullscreen mode

The matching rate is high, with only a one-character difference between "は" and "わ". The information about the elongated ending is discarded during the normalization step.

The same thing happens with consecutive vowels.

Transcript: こんにけわあ  β†’  _REPEAT_RE compresses consecutive "あ" β†’  こんにけわ
Enter fullscreen mode Exit fullscreen mode

In other words, this verification will never detect elongated endings. The normalization that was written to ignore long sounds is working in the same way in situations where we want to detect them.

Normalization is correct in terms of content matching. If we want to see the consistency of the content, ignoring long sounds is the right thing to do. The problem was that there were two purposes, but only one normalization.

The transcript

Judging with Raw Transcripts

I separated the judgment of content matching and elongated ending detection. The latter uses the raw transcript string.

_TAIL_LONG_RE = re.compile(r"[ーぁ-γ‚“]$")

def trailing_elongation_mismatch(script_text: str, raw_transcript: str) -> bool:
    """Detects elongated endings not present in the script.

    ⚠️ Pass the raw transcript from Whisper to raw_transcript.
    Normalization discards long sounds, so it cannot be detected after normalization.
    """
    script = (script_text or "").rstrip("γ€‚γ€οΌοΌŸ!? ")
    trans  = (raw_transcript or "").rstrip("γ€‚γ€οΌοΌŸ!? ")
    if not script or not trans:
        return False

    # Check if the last character of the script is elongated in the transcript
    tail_script = script[-1]
    # Long sound symbol is not in the script but is in the transcript
    if "γƒΌ" not in script and trans.endswith("γƒΌ"):
        return True
    # Same vowel is duplicated in the transcript but not in the script
    if len(trans) > len(script) and trans[len(script)-1:].startswith(tail_script):
        extra = trans[len(script):]
        if extra and all(c in "γγƒγ…γ‡γ‰γ‚γ„γ†γˆγŠγƒΌ" for c in extra):
            return True
    return False
Enter fullscreen mode Exit fullscreen mode

The caller now sees the two judgments separately.

res  = judge_transcript(text, tr["text"])                  # Content matching (with normalization)
tail = trailing_elongation_mismatch(text, tr["text"])      # Elongated ending detection (with raw string)

if tail:
    continue          # If elongated, re-draw immediately (even if content matches)
if res.ok:
    save(wav)
Enter fullscreen mode Exit fullscreen mode

Elongated endings are not allowed, even if the content matches. If the content matching rate is high, but there is an elongated ending, it is not used as training material. If this is relaxed, the model will learn the elongation habit.

Designing Tolerance

However, if it's completely zero, the yield will drop. In emotional speech, some elongation will occur naturally.

Ultimately, I used a combination of two conditions.

def accept(text, transcript):
    v    = judge_transcript(text, transcript)
    tail = trailing_elongation_mismatch(text, transcript)
    return (v.ratio >= 0.82 and tail <= 2) or (v.ratio >= 0.70 and tail == 0)
Enter fullscreen mode Exit fullscreen mode
  • If the content matches well (0.82 or higher), up to 2 elongated endings are allowed
  • If the content matching rate is slightly lower (0.70 or higher), no elongated endings are allowed

This is a shape that ensures clips with both suspicious content and elongated endings are discarded. While allowing some recognition degradation due to emotional expression, the elongation habit is not allowed to pass through.

Also Hit on the Caption Side

Another thing that worked was writing it in the caption at generation time.

When designing voices by role, I added the following to the caption:

A clear and easy-to-listen female announcer voice that reads news scripts accurately.
The tone is calm and intelligent, with clear pronunciation of the ending.
Enter fullscreen mode Exit fullscreen mode

The last part, "with clear pronunciation of the ending", is what did it.

The effect was clear. Out of 120 clips generated (24 candidates Γ— 5 probe texts), there were no elongated endings. Before the quality gate could filter them out, the generated speech no longer had elongated endings.

Role-inclusive caption (with clear ending) β†’ Elongated ending 0/120
Enter fullscreen mode Exit fullscreen mode

The gate is a mechanism to "discard bad ones", but discarding them reduces the yield. If it's possible to prevent them from being generated in the first place, that's cheaper. For TTS that can be instructed by caption, it's worth writing the items being checked by the quality gate into the caption as well.

Relationship with Speaking Style

As I pursued this phenomenon, I saw a deeper structure. The desirability of elongation varies by purpose.

  • Narrators, announcers, call centers β†’ Want endings to be clear
  • VTubers, streaming β†’ Elongation sounds more natural

So, I defined "speaking styles" by purpose and changed the script and quality gate strictness for each speaking style. For business-style speaking, I applied strict elongation gates, and for casual styles, I relaxed them.

The important thing is that speaking style is baked into the corpus and cannot be changed at synthesis time (Speaking style is baked into the corpus). If you want both "clear endings" and "elongated endings" with the same voice, you need to bake two corpora with the same design values (caption + seed) but different speaking styles. I'm actually doing this.

Summary

Normalization should be split by purpose. Normalization for content matching and normalization for detecting specific anomalies are different. Trying to combine them into one will cause one or the other to fail.

Be aware of the information being discarded. Adding γƒΌ to _PUNCT_RE was a correct decision, but the information was needed for a judgment that came later. Writing "what is being discarded" in the comments of the normalization code helps the next person to notice.

It's cheaper to prevent generation in the first place. If it's possible to instruct the generator, it's worth writing the items being checked by the quality gate into the caption.

Symptoms appear in the model's behavior. Even if you look at the corpus data, you might just see "a few clips with elongated endings" and not think it's abnormal. It's only after training and making the model speak that the habit appears. Checking the output after training is necessary, in addition to checking the dataset.

Series: Mass-producing Practical Voices from Diffusion TTS

This is a record of designing voices from a one-line caption, manufacturing training corpora, and mass-producing practical voices by role. This article is part of Part 3: Quality Gates.

← Previous: One rough clip ruins the whole style
β†’ Next: The character that broke the TTS input

Series (18 articles)

  1. Diffusion TTS was too slow for conversation
  2. Voice gacha and design ledger
  3. Screening voices by metrics, not ears
  4. Quality gate selection bias and flat takes
  5. Speaking style is baked into the corpus
  6. TTS changes recording room every time
  7. One rough clip ruins the whole style 8. Where did the elongated ending come from? ← This article
  8. The character that broke the TTS input
  9. Hallucination guard that never fired
  10. Three characters became a verbal tic
  11. Measuring factory defects as product traits
  12. Defects invisible to transcription
  13. 70 minutes lost to a network blink
  14. From "ja" to "JP" and babbling models
  15. Four registration paths, one exit
  16. Who is rolling back whom?
  17. Chasing unmeasured targets with thresholds

The notes that led to these insights are summarized in Mass-producing Practical Voices from Diffusion TTS.

Top comments (0)