π Originally published (in Japanese) at forge.workstyle.tech.
Target Audience: Developers implementing voice analysis or avatar lip-syncing in browsers.
Input and Output: Labeled TTS audio is input into a vowel estimator, and the accuracy rate is obtained from the correct answers and estimated results.
What You'll Gain from This Article: The reason why the order and length of audio affect the evaluation results in a stateful estimator, and how to verify this.
While trying to improve the vowel estimator, I found myself writing code to accommodate the quirks of the evaluator. The most significant discovery this time was that the measurement method was creating the answer.
The target is a system that estimates vowels from TTS audio played in a browser and moves the VRM avatar's aa / ih / ou / ee / oh. The timing is not adjusted with the text, and the audio itself is analyzed during playback.
The estimator uses the deviation from the long-term average of the current level for each frequency band. This average is updated every time audio is received. In other words, even if it's the same sound, the input features change depending on what was heard before. Simply preparing evaluation audio does not fix the measurement conditions.
Start with Audio that Can Create Correct Answers
For evaluation, I synthesized sustained vowels like "γγΌγΌγΌ" and "γγΌγΌγΌ" using our company's Style-Bert-VITS2. The data consists of 3 speakers Γ 5 vowels. Since the pronunciation content is known, the audio and correct answer labels can be matched.
However, the fact that I was able to synthesize "γ" and the fact that I obtained a suitable "γ" for evaluation are different. Before listening, I mechanically checked the length, RMS, peak, voicing rate, fundamental frequency, and formant. The voicing rate is the percentage of frames judged as voiced, such as 80% if 80 frames out of 100 were judged as voiced. If silent or abnormal outputs are mixed in, the estimator will be scored for the data's malfunction rather than its own performance.
In this verification, it was also necessary to suspect the inspection side. For example, something that was judged as clipping just by looking at the peak was actually peak normalization when confirmed. The fact that the threshold was touched did not necessarily mean that the waveform was crushed.
Formant estimation using LPC also returned unnatural values as formants when the speaker had a high fundamental frequency, picking up harmonics. LPC is a method of applying a model that predicts the next sample by multiplying the previous samples by coefficients and adding them to the audio. Therefore, I stopped using the estimated formant as the basis for band design and directly examined the spectrum. This is not a conclusion about LPC in general, but a problem that occurred in the verification of this evaluation audio and estimation conditions.
Additionally, to avoid results that only work for the speaker used in the template design, I confirmed the validity using leave-one-speaker-out, where the speaker is removed from the design and evaluation. The template is a sample used for comparison, such as a list of deviations for each band of "γ". In leave-one-speaker-out, for example, a sample is created using two out of three speakers, and the remaining speaker is used for evaluation, rotating the roles.
Fixed Order Made the First Vowel Unfavorable
In the initial evaluation, the vowels were always arranged in the order of a, i, u, e, o. This seems like a neat and comparable input. However, this order conflicted with the initialization of the long-term average.
The estimator uses the average from the beginning to make it usable for input, and it follows the input more quickly at the start. If "γ" comes first, the average quickly learns the spectrum of "γ". Since the average is subtracted from the current "γ", the deviation necessary for classification becomes smaller.
Subsequent vowels are compared to the average, which includes the previous vowels. Only the first vowel was measured as the difference from the standard that strongly included itself.
To investigate this bias, I introduced a rotation of the order, changing the first vowel. For example, a, i, u, e, o is shifted to i, u, e, o, a. What's important is not just shuffling, but also making the initial state of the estimator uniform and measuring the condition where each vowel becomes the first vowel.
In JavaScript, the part that creates the order can be written briefly:
const vowels = ["a", "i", "u", "e", "o"];
const orders = vowels.map((_, start) => [
...vowels.slice(start),
...vowels.slice(0, start),
]);
for (const order of orders) {
estimator.reset();
evaluateSequence(order, estimator);
}
evaluateSequence is a process that plays the audio in the specified order, records the correct answer label and estimated result for each interval, and does not reset in the middle of the concatenated interval. This is because the condition itself, where the long-term average is continuously updated, is being evaluated.
This problem is not a matter of a specific vowel being inherently difficult to distinguish. The combination of the fixed order and initialization gave unfavorable conditions to a specific label.
The Longer It Is, the More the Features Are Absorbed into the Average
Another flaw was that the vowels were stretched to about 1.2 seconds for measurement.
The deviation from the long-term average assumes that the input is replaced. If the same vowel is maintained, the average approaches that vowel, and the deviation disappears. It may seem easier to measure when the audio is stable, but this was a harsh condition for this estimator.
Therefore, I cut the sustained vowels to 120ms and evaluated them in a random order. This is a series of tests that see how the vowels change at a speed close to speech, with the vowels replacing each other. I did not synthesize continuous speech, including consonants.
The evaluation results are always compared within the same harness, old implementation and new implementation side by side. Here, the harness refers to the evaluation system that plays the audio in a specified order, compares the estimated results with the correct answer labels, and outputs the accuracy rate.
| Evaluation Condition | Old Implementation | New Implementation |
|---|---|---|
| Sustained Vowel, Approximately 1.2 Seconds, Fixed Order | 14.0% | 59.6% |
| Sustained Vowel, Order Rotation | 14.4% | 57.5% |
| 120ms, Random Order | 12.7% | 71.3% |
The old implementation directly assigns bands to vowels, while the new implementation compares the deviation patterns between bands. The change to the new implementation was confirmed to be an improvement in every harness. On the other hand, if the evaluation conditions change, the numbers attached to the implementation also change.
Therefore, it is not possible to call 71.3% the "accuracy in actual speech". Accurately, it is the accuracy rate in the series of TTS sustained vowels cut to 120ms and randomly arranged this time. The boundaries of the cut-out vowels are different from the boundaries of actual speech, which include consonants and changes in articulation.
Incorrect Measurements Led to Plausible Countermeasures
The flaws in the measurement changed not only the numbers reported but also the code written next.
When the performance of "γ" was poor, I hypothesized that the classification was being pulled towards aa. I then wrote countermeasure code to remove common components from the template. However, the effect was limited to +0.1%, and I withdrew it.
At this point, I was looking too much for the cause inside the classifier. I should have confirmed earlier what the average was learning at the start of the evaluation and whether the first vowel was not at a disadvantage.
The removal of common components that I withdrew here and the "centering" that will be dealt with next, which subtracts the average value of all bands from each vector, should not be treated as the same correction. Centering is a process that, for example, subtracts the average of 4 from [2, 4, 6] to get [-2, 0, 2]. Even if the explanation of the process is similar, the components to be removed and the phenomena to be addressed are different.
Evaluation Conditions Should Also Be Retained as Part of the Results
The sustained vowel test is not worthless. If the purpose is to output the same sound for a long time, it is necessary to investigate the behavior where the deviation is lost. This long vowel series also served as a stress test that exposed the dependence on the long-term average.
The problem lies in the fact that the results were reported as the overall strength of the purpose.
In future comparisons, I will retain not only the speaker and audio but also the cut-off length, vocalization order, average reset position, and scoring interval together. Even with the same material, if these factors are different, it becomes a different test. Before fixing the classifier, I will confirm what the measurement is disadvantageous for. This procedure has reduced the time spent writing ineffective countermeasure code.
Top comments (0)