📝 Originally published (in Japanese) at forge.workstyle.tech.
Target Audience: Developers working on web front-end, audio processing, and avatar development who need to choose a lip-sync method.
Input and Output: Takes playback audio or time-stamped phonemes (e.g., "a" from 0–180ms, "i" from 180–360ms) as input and animates the mouth of a VRM avatar in the browser.
What You’ll Gain: Guidance on selecting the right method for your use case, along with implementation, evaluation, and visual tuning strategies.
Start by Defining the Required Mouth Information
In lip-sync, it’s important to distinguish between mouth movement, shaping it into vowel-like forms, and representing closures based on phonemes. The required information and post-implementation quality checks differ for each.
This series focuses on browser-based avatars using Three.js and @pixiv/three-vrm. It plays TTS-synthesized audio via <audio> and applies weights to VRM’s aa / ih / ou / ee / oh morph targets.
The choice of method depends on your use case, not just the tools at hand.
| Use Case / Condition | Initial Method Choice | What You Get | Limitations to Check |
|---|---|---|---|
| Show speech presence and volume variations | Volume only | Mouth opening/closing based on audio | No distinction between vowel shapes |
| Enhance mouth shape changes in dialogue avatars with replaceable TTS | Bandwidth + Shape | Vowel-like changes estimated from audio | Dependency on speaker and audio conditions, closure representation |
| Prioritize lip closures and voiceless sounds for accurate pronunciation | Phonemes | Mouth shaping based on time-stamped phonemes | Timing acquisition, synchronization, VRM mapping |
| Pre-verify synthesized audio | Prioritize phonemes | Control based on alignment (e.g., matching "a" segments with phoneme timestamps) | Phoneme quality and production workload |
| Focus on long sustained sounds | Start with volume-only, evaluate shape estimation separately | Follows mouth opening | Long-term averaging may smooth out deviations |
No method guarantees naturalness on its own. Specifically, "Bandwidth + Shape" differs from directly assigning frequency bands to vowels. This series uses a method that compares deviation patterns between bands. For example, if the long-term average of past sounds is −30dB and the current sound is −20dB, the deviation is +10dB, and such fluctuations are compared across multiple bands.
Volume Only: Start Here if Mouth Opening/Closing is Your Goal
In the volume-based method, RMS is calculated from the audio waveform and mapped to mouth openness. Since vowel types aren’t estimated, it’s a lightweight way to react to speech.
Even at this stage, visual tuning is necessary. If the input volume threshold is too low, the mouth stays open; if weak sounds open the mouth too little, it looks unnatural despite speech.
Start with Part 1: VRM Lip-Sync Using RMS. If adjusting openness becomes tricky, refer to Part 5 for saturation rate (e.g., percentage of frames where weight hits 1.0) and percentiles (e.g., 25th percentile of weights sorted in ascending order). These concepts are useful even before introducing vowel classification.
If opening/closing alone meets your needs, consider it complete. Adding vowel estimation aims to achieve necessary mouth shape changes.
Bandwidth + Shape: Create Vowel-Like Changes from Audio
Initially, frequency band energy was directly assigned to vowels. However, in the evaluated audio, the assumed bands reacted strongly to different vowels.
| Old Band Assignment | Most Prominent Vowel in Current Measurement |
|---|---|
aa: 250–700Hz |
"e" |
oh: 700–1200Hz |
"a" |
ou: 1200–2000Hz |
"a" |
ee: 2000–3000Hz |
"e" |
ih: 3000–5000Hz |
"e" |
This was observed with the current speaker and sustained vowels (e.g., "aah"). It’s not a general claim about vowel-band correspondence.
Absolute band levels mix vowel differences with spectral slope (higher frequencies weaken), voice quality, and volume. The new implementation calculates deviations from long-term averages per band and compares them to vowel templates (e.g., mid-range drops and high-range rises for "i"). Uniform fluctuations across all bands are removed to separate volume changes from shape differences.
The failure reasons are covered in Part 2, and classification using deviations and cosine similarity in Part 4. Cosine similarity measures shape match (e.g., 1 if deviation series are proportional and aligned). The implementation shift focuses on inter-band relationships, not single-band magnitudes.
However, long-term averaging assumes sustained sounds. If the same vowel is prolonged, the average approaches that sound, weakening deviations for classification. This method is designed for alternating vowels, so evaluate separately for sustained sound-heavy use cases.
Evaluate Metrics Alongside Audio Creation Methods
When comparing methods, use metrics measured under the same conditions. Evaluation harnesses, for example, input vowel sequences and compare ground truth labels with estimation results.
| Evaluation Harness | Old Implementation | New Implementation |
|---|---|---|
| Sustained vowels, fixed order | 14.0% | 59.6% |
| Sustained vowels, rotated order | 14.4% | 57.5% |
| 120ms segments, random order | 12.7% | 71.3% |
Each row compares old and new implementations under the same harness. Don’t cherry-pick numbers from different rows to claim improvement.
Specifically, 71.3% comes from 120ms segments in random order, testing short speech-like conditions, not continuous speech with consonants.
This distinction affects adoption decisions. Improvements in vowel sequences don’t guarantee natural mouth movement in full conversations. Sustained vowels, however, serve as stress tests for long-term average tracking.
Part 3 covers evaluation data creation and pitfalls in speech order and duration measurements. Before refining classification, ensure tests match your use case.
Tune Visuals Separately from Classification
Even with correct vowel selection, if weights stick to the upper limit, the mouth looks overly exaggerated. Conversely, insufficient opening in weak segments makes speech appear mumbled, even with good averages.
Part 5 breaks down post-classification issues into input volume thresholds, saturation, VRM mouth shape visual strength, and lower percentiles of max weights.
Key here is not to overuse amplification as a tuning knob. To lift the distribution’s bottom, adjust the curve from volume to openness. If specific mouth shapes are weak, inspect the model’s visuals.
The only constant without measurement backing was per-viseme (e.g., aa for "a") adjustments. The article emphasizes not treating empirically determined values as universal truths.
Phonemes: Choose if Expression and Timing Are Critical
With phonemes and durations, you can design movements for bilabial closures (e.g., closing lips before "ma"), nasal sounds ("n"), geminates ("tt"), and voiceless vowels (e.g., in "suki"). These are unstable in RMS or bandwidth-based methods.
However, phoneme sequences alone don’t determine timing. Time-stamped phonemes and synchronization with actual audio playback are needed.
In this product, the TTS output lacked per-phoneme durations, and TTS replacement was assumed, so this method wasn’t adopted. Using pre-trained models via external services also had unverified latency budgets. Part 6 explains this decision and conditions for adoption.
Browser Audio Paths Are Part of the Design
The setup captures the playback stream via createMediaStreamSource and analyzes it with AnalyserNode, without connecting it to destination. This prevents AEC reference signal issues in echo cancellation. AEC, for example, uses speaker output as a reference to remove echoed sounds from the mic.
Even if mouth movement is correct, changing the audio path can break interaction, rendering the product incomplete. Verify input distribution, classification, synchronization, and visuals while preserving the actual playback path.
First, define the required expressions for your use case and choose a method that provides that information. Focus on openness distribution for volume-only, evaluation audio validity for bandwidth + shape, and timing info/sync for phonemes. This order makes it easier to identify which layer to revisit if issues persist.
Top comments (0)