📝 Originally published (in Japanese) at forge.workstyle.tech.
Target: Developers who have improved the classification results of VRM avatars in browsers but still feel that the mouth movements are unnatural.
Input and Output: The input is the volume and vowel estimation obtained from the reproduced audio, and the output is the weight of aa / ih / ou / ee / oh.
What you can gain from this article: The procedure for adjusting the saturation rate and percentile to distinguish between overly open and too weak mouth movements. The saturation rate is, for example, the percentage of frames that reach the upper limit, such as 20% if 20 frames out of 100 frames reach the upper limit. The percentile is, for example, the value at the 25% point, which is the value at the 25% position when the values are arranged in ascending order.
Even if the classification is correct, the mouth may not look natural
When the vowel classification was improved, I thought the lip sync was close to completion. However, in the actual machine, I continued to receive feedback, such as "overemphasized", "still too open", "small 'e'", and "overall too soft". When I investigated each of these, I found that the causes were in different places.
The target is a system that reproduces audio synthesized by TTS in a browser and moves the VRM mouth based on the audio. Vowel estimation selects the type of mouth based on the shape of the spectrum, and volume determines the size of the opening.
Even with this division of labor, there are things to be evaluated. Did the vowels match? Did the weights move within an appropriate range? Did the model look natural? The classification rate only tells us part of this.
"Overemphasized" is first checked for saturation
The first feedback I received was "overemphasized 'a' and 'i'". I suspected the coefficients for each vowel. However, in reality, the amplification coefficient was large, and 48.1% of the frames had a weight that was stuck at the upper limit of 1.0.
Once the upper limit is reached, even if there is a difference in the input, there is no difference in the output. It looks like the mouth is moving with the voice, but in reality, it is repeating the maximum shape.
In this state, adjusting the appearance of each vowel would hide the common problem with individual coefficients. What should be looked at first is not which mouth is larger, but how much of the output is reaching the upper limit.
The next feedback was "still too open". I thought that reducing the amplification further would be effective, but this time there was a problem with the criteria for converting volume to opening. The actual RMS central value of the TTS audio was 0.214, while the criteria were 0.15. The criteria were low compared to the input distribution, and 58.5% of the frames were saturated.
Even with the same "too open", it is necessary to distinguish between saturation due to amplification and saturation that occurs in the volume normalization stage. The difference in size that is lost in the middle cannot be recovered even if the coefficient is lowered later.
| TTS audio frame RMS | Measured value |
|---|---|
| 10% point | 0.017 |
| 25% point | 0.024 |
| Median | 0.214 |
| 75% point | 0.355 |
| 90% point | 0.403 |
| 99% point | 0.572 |
The criteria value should be compared with the actual distribution of the audio being played, rather than the imagined loudness. This table is an observation of the TTS output this time, and it is not a standard value that can be applied directly to other speakers or TTS.
"Small 'e'" was not a classification problem
Next, I was told that the "e" was too small. I thought that the classification might be biased towards aa, but the discrimination rate for "e" in short vowel sequences was 72%.
Here, short vowel sequences refer to evaluation audio created by cutting sustained vowels into 120ms pieces. Since it does not include consonants or continuous speech, this number cannot be read as the discrimination rate in actual speech. Nevertheless, it became a basis for investigating the drawing side before adjusting the classification alone.
The cause was that the visual strength of each VRM blend shape was different. Even with the same weight, the mouth may not appear to be equally noticeable. Therefore, I added a correction to the appearance.
{ aa: 0.8, oh: 1.0, ou: 1.0, ee: 1.4, ih: 0.85 }
This is the only constant used in this adjustment that does not have any measurement backing. It was decided based on feedback from the actual machine, and it is not a correction value that can be applied to other VRMs.
After adding the correction, the discrimination rate changed from 71.4% to 71.2%, and the "e" changed from 72% to 76%. However, this does not prove the visual validity of the correction value. Confirming the impact on classification and quantitatively evaluating the appearance are separate things.
The true nature of "overall soft" was not the average but the lower 25%
Finally, there was the impression that "the overall mouth movement is too soft". This makes you want to move the mouth more, so it's natural to think of increasing the amplification.
However, during the adjustment process, I tried "increasing the amplification", but it didn't work. The average maximum weight increases, but the saturation also returns.
| Amplification | Average maximum weight | Saturation at 1.0 |
|---|---|---|
| 1.00 | 0.537 | 3.5% |
| 1.30 | 0.666 | 18.1% |
| 1.50 | 0.732 | 32.2% |
Looking only at the average, it seems that increasing the amplification is effective. However, this time, the "overall soft" was not a problem where the entire mouth movement was uniformly small. When looking at the distribution of maximum weights per frame, the 25th percentile was only 0.375. The weak mouth movements were not visible enough.
So, I changed the mapping from volume to opening. I changed it from a straight line that uses the normalized volume as is to a square root with an exponent of 0.5.
| Curve exponent | Average maximum weight | 25th percentile | Saturation at 1.0 |
|---|---|---|---|
| 1.00 (straight line) | 0.537 | 0.375 | 3.5% |
| 0.70 | 0.597 | 0.465 | 4.3% |
| 0.50 | 0.647 | 0.531 | 6.1% |
| 0.35 | 0.692 | 0.575 | 8.8% |
The square root does not change the position of the upper limit, but relatively strengthens the small normalized input. This is an adjustment that changes the distribution of openings while maintaining the order of voice sizes. However, the size difference itself is compressed, so it's not a process that "doesn't change the strength at all".
This time, with an exponent of 0.50, the 25th percentile rose to 0.531. It was a more suitable adjustment than increasing the amplification to raise the overall level. However, the saturation increased to 6.1%, and changing the curve does not eliminate saturation.
Adjust the order according to the cause
In the implementation, first, take the volume distribution and check if the normalization criteria match the input. Next, look at the saturation rate of the final weight. If the opening is weak, check the average and lower percentile together. Then, adjust the remaining visual differences for each mouth in the actual machine.
It's also desirable to have consistent measurement conditions. Whether to include silent periods, or which audio interval to use, changes the meaning of the percentile. Even if the average and lower percentile are lined up from different intervals, it's not guaranteed that the same problem is being looked at.
Indicators cannot replace visual inspection. On the other hand, having indicators makes it possible to avoid pushing all visual feedback into amplification coefficients. Being able to decompose the impression of "large" or "small" into saturation, input criteria, blend shape, and distribution bottom was the most helpful thing in this adjustment.
Top comments (0)