VRM Lip Sync with RMS — Minimal Implementation with 'aa' and Conditions for a Natural Look

📝 Originally published (in Japanese) at forge.workstyle.tech.

Target Audience: Developers using Three.js and @pixiv/three-vrm in the browser who want to animate an avatar's mouth in sync with audio.
Input/Output: Takes a playing audio stream as input and outputs the weight for the VRM aa expression.
What You'll Gain: A minimal implementation using RMS and methods to distinguish between overheating (opening too wide) and insufficient movement for quiet voices.

Start by Syncing Mouth Open/Closed with Sound, Not Vowels

In the system I'm working on, TTS audio synthesized on the server is played via an <audio> tag in the browser, and the VRM avatar's mouth is animated to match that audio.

VRM includes visemes like aa / ih / ou / ee / oh. For example, aa represents the mouth shape for the vowel "a". However, you don't need to set up a vowel classifier from the start. Simply animating aa based on audio intensity is enough to create a behavior where the mouth opens while speaking and closes during silence.

The goal here isn't to reproduce specific vowels, but to map sound to mouth opening/closing. Even the vowel "i" will open with an "a" shape. In exchange for accepting this limitation, you can skip the process of estimating pronunciation timing from text. Since you are using the actual playing audio, you don’t need to synchronize it with the text either.

At this stage, you want to verify: Does the mouth move when sound is present? Does it close when the sound stops? Is there a visible difference between loud and quiet parts? Get this foundation right before adding more vowel shapes.

Branch Off the Playback Path for Analysis Only

Use the Web Audio API's AnalyserNode for audio analysis. In this setup, pass the playing stream to createMediaStreamSource.

The critical part is not connecting the analysis node to the destination. Leave audio playback to the existing <audio> element. In this system, if the analysis side also outputs audio, it breaks the relationship with the AEC (Acoustic Echo Cancellation) reference signal, causing the echo cancellation to fail. AEC is a process that removes echoes from the microphone by referencing the audio being played from the speakers. This is a constraint of our playback setup.

// stream is the MediaStream extracted from the playing audio.
// context is an AudioContext that has been resumed based on user interaction.
const source = context.createMediaStreamSource(stream);
const analyser = context.createAnalyser();

source.connect(analyser);
// Do NOT connect analyser to destination.

const samples = new Float32Array(analyser.fftSize);

Extracting the stream depends on the implementation of the playback side, so I've only shown the receiving end here. Integrate this as code that observes existing playback, not as code that starts new audio playback.

Converting RMS to Mouth Weight

RMS (Root Mean Square) is calculated by squaring each sample of the waveform, averaging them, and taking the square root. Simply averaging a waveform that oscillates between positive and negative values causes them to cancel out. RMS allows you to measure the signal strength over a specific interval.

Note that RMS is not inherently the same as human-perceived volume. Here, we use it as an input feature to drive mouth opening.

Call the following function from your existing render loop. floor is the lower bound used to close the mouth, and reference is the RMS value corresponding to the maximum opening limit (ensure reference > floor, determined from actual audio). follow is a coefficient representing the speed of tracking.

let opening = 0;

function updateMouth({ playing, floor, reference, follow }) {
  analyser.getFloatTimeDomainData(samples);

  const power = samples.reduce((sum, x) => sum + x * x, 0);
  const rms = Math.sqrt(power / samples.length);

  const level = Math.min(
    1,
    Math.max(0, (rms - floor) / (reference - floor))
  );
  const target = playing ? Math.sqrt(level) : 0;

  opening += (target - opening) * follow;
  vrm.expressionManager.setValue("aa", opening);
}

In this example, we apply a square root to the normalized volume. This helps ensure movement is visible even for small inputs; I’ll explain why later.

A smaller follow value makes the animation smoother but delays the opening and closing. Since a fixed coefficient changes the effective tracking speed depending on the rendering cycle, it’s often more practical to calculate it based on elapsed time in production implementations. Be careful to prioritize smoothness without losing sync with the audio.

Also, set the expression before running the standard VRM update process. If other processes are manipulating other visemes, manage those weights as well. Even if you only update aa, it won't look like a minimal implementation if previous values for ih or others remain active.

Explicitly close aa when playback ends, and disconnect the analysis nodes during cleanup. In setups where the render-side updates stop when the audio stops, you need this termination logic to prevent the mouth from remaining open.

What Determines Naturalness? First, the Volume Baseline

Once implemented, the first thing to check is not the amplification level, but the distribution of the input audio.

In our TTS output, the median frame RMS was 0.214, the 25th percentile was 0.024, and the 90th percentile was 0.403. The 25th percentile is the value at the 25% position when all frame values are sorted in ascending order; these values are called percentiles. There is a significant range between quiet and loud segments. Tuning only for an average voice won’t determine the overall opening behavior.

In on-device tuning, our volume baseline was set to 0.15, which resulted in 58.5% of frames being saturated. In this context, saturation means the input exceeds the baseline, causing the target opening to hit the upper limit of 1.0. When pointed out that the mouth was "opening too wide," we needed to check if the baseline matched the actual TTS before simply lowering the amplification.

Note that these values are observations from our specific TTS and processing conditions. They do not imply that the same baseline will work for other audio. This is why reference in the code should not be distributed as a fixed "correct" answer, but should be verified with the audio you intend to use.

Adjusting Small Movements with Curves, Not Amplification

When you suppress excessive opening, you may find that quiet sections appear to have no movement. If you then increase the overall amplification, you push the loud parts up as well, increasing the number of segments hitting the upper limit.

In on-device tuning that includes downstream viseme estimation, changing the mapping from volume to opening from linear to square root resulted in the following changes. The "Max Weight" in the table refers to the largest weight among aa / ih / ou / ee / oh for each frame.

Volume-to-Opening Curve	Average Max Weight	Bottom 25%	Saturation at Max 1.0
Linear	0.537	0.375	3.5%
Square Root	0.647	0.531	6.1%

These results are not from evaluating the aa-only code above. However, the focus of tuning is common: use percentiles to catch "weakness in small movements" that averages might miss, and use the curve to boost them.

Changing to a square root raises small inputs before reaching the limit, without changing the input value that corresponds to the maximum. It preserves the relative order of input sizes but compresses the difference in opening. As the table shows, saturation increases, so you need to verify both the visual appearance and the saturation rate.

Defining the Scope for Using aa Alone

If the goal is simply mouth opening/closing that reacts to audio, RMS is a great way to start small. However, it has limitations if you want to convey specific lip shapes.

Lip closure for bilabial sounds (the motion of closing the lips before "m"), the nasal sound "n", the geminate consonant "tsu", and devoiced vowels (e.g., when the vowel in "su" of "suki" does not involve vocal cord vibration) cannot be reliably expressed using this RMS/band-based method. Closing the mouth during silence and creating the necessary closure at the correct time for pronunciation are separate challenges.

Start by aligning sound with opening/closing using RMS, and tune the volume baseline, saturation, and movement in quiet sections. If you need to distinguish lip shapes, add a separate estimator to define "which shape" on top of "how much" the volume determines.