đź“ť Originally published (in Japanese) at forge.workstyle.tech.
oneme timing refers to temporal information for each phoneme—for example, /m/ in "ma" corresponding to 0–80ms from playback start, and /a/ to 80–200ms. In this example, the duration of each phoneme is 80ms and 120ms, respectively.
Target: Developers building products handling multiple TTS and choosing a lip-sync method. own*Input/Output: Input is playback audio or timestamped phoneme information; output is mouth shapes and opening values for a VRM.
**What you'll learn*: Why we chose not to adopt phoneme timing, and the conditions under which we would change that decision.
Separating "desired expression" from "available information"
There are limits to methods that create mouths based solely on volume and spectrum. In our current method based on RMS and band features, it is difficult to stably represent the closure of bilabials, the nasal /n/, the geminate consonant /tsu/, and voiceless vowels.
For instance, the lip closure required for /m/, /b/, and /p/ is hard to handle with a control that simply opens the mouth when the volume is high. Similarly, when a vowel becomes voiceless (like the /u/ at the end of "desu" or "masu" where vocal fold vibration stops), determining intent from sound intensity is insufficient.
If we know the phonemes and their durations, we can explicitly design these movements. However, having phoneme information is not the same as having timing information that matches the product's playback audio. Furthermore, simply receiving the information doesn't result in a natural mouth. Mapping to mouth shapes and handling transitions between them requires separate processing.
In this case, I decided not to adopt phoneme timing. This wasn't because I dismissed the value of phoneme-based methods, but because the necessary information and operational conditions were not met.
Could not receive durations from the TTS being used
The pipeline of the TTS we use includes a process to determine the Japanese reading and convert it into a phoneme sequence. A phoneme sequence represents, example, "ma" as /m/ and /a/. However, the output we use does not include timing per phoneme. The length returned by the backend was the total length of the clip.
Even if we know the total length, the question remains how to distribute that time among the phonemes. Simply dividing them equally offers no guarantee they will match the actually generated audio.
This is not to claim that obtaining timing is impossible for every usage of that TTS. It is a constraint: in the path we currently connected, we could not receive the information necessary for adoption.
The conditions for changing the decision are clear: being able to retrieve timestamped phonemes matching the actual playback audio from the API or a separate alignment process. Alignment involves, for example, matching the phoneme sequence of "ma" with the synthesized audio to determine where /m/ and /a/ occur in the sound. This mapping must be verified for the target speaker and text. If these conditions are met, the phoneme-based method becomes a concrete candidate.
TTS swapping a prerequisite for the product
The product is designed to allow the TTS to be swapped. In other products, users connect their own cloud TTS.
If mouth control is heavily dependent on specific TTS internal information, the available features and quality will vary between providers that return that information and those that do not. Simply being able to play audio is not enough to provide the same lip-sync.
What we chose was a method that analyzes the audio being played itself. By using audio as a common input, we can distance ourselves from whether or not each TTS provides phoneme information. There is also no need to align text and audio.
That said, using audio input doesn't mean quality will be consistent. If the speaker's voice quality or volume distribution changes, vowel estimation and opening adjustments will need to be rechecked. What can be standardized is the entry point; the guarantee of visual appearance is not automatically standardized.
If the used TTS is limited and we can convert each provider's timestamped phonemes into a common format, the situation changes. A configuration that uses the phoneme-based method for supported TTSs and falls back to audio analysis for others is a de-facto candidate. However, the condition is that this quality difference must be acceptable and verifiable as a product.
Didn't want to drive synchronization with uncertain durations
In this design decision, we weighed the lack of synchronization between sound and mouth more heavily than the fine details of mouth shapes. Even if the shapes become more detailed, if the mouth closes before the sound or opens after the pronunciation has finished, it is hard to call that an improvement.
This is not a general rule that synchronization is always the most important factor in every application. It was the axis of quality we wanted to prioritize for this conversational avatar.
If we move the mouth by crudely estimating durations from a phoneme sequence, we decide synchronization based on that uncertain estimation. While direct audio analysis methods can also have processing or rendering delays, at least it avoids bringing the problem of matching a time distribution created separately from the reading to the playback audio.
If we adopt the phoneme method, checking the timestamp format is not enough. We want to verify if the sound and mouth align in the browser, including start, stop, and resume. We also need a design that does not treat the time received from the network and the actual playback position as the same thing.
Top comments (0)