📝 Originally published (in Japanese) at forge.workstyle.tech.
Three Traps in Building a Fully Automated AI Avatar Streaming Pipeline
When building a continuously running AI avatar streaming pipeline, you often encounter bugs that are hard to pinpoint: "It's running, but something feels off." The avatar isn't crashing or throwing errors, but its mouth is out of sync with the audio, the drift worsens over time, or a 90-second recording mysteriously becomes only 6 seconds long.
This article summarizes three traps we fell into while building such a pipeline—all stemming from handling video and audio through separate paths. Each manifests differently:
- Lip Sync Issues — We were trying to fix two distinct types of desync with a single adjustment knob
-
Progressive Drift — Accumulated delays in
setIntervalfiring - Recording Cutting Off Prematurely — MediaRecorder stops the video stream when the audio track runs dry
The overarching lesson: In media processing, failures don’t throw exceptions—they return as silent discrepancies, and separating measurable parts from immeasurable ones is key to solving these bugs.
The Base Pipeline Architecture
Our streaming pipeline looked like this:
Browser Page
├ Video: CDP screencast → JPEG stream → Named pipe
└ Audio: Web Audio tap → Named pipe
↓
ffmpeg (mux 2 inputs) → RTMP
We render a 3D avatar in a headless browser, extract video and audio separately, then multiplex (mux) them into a single stream downstream (ffmpeg or MediaRecorder). Video and audio take separate paths to reach downstream components—this structure is the root cause of all the problems we’ll discuss. Since muxing requires aligning timestamps from both tracks, if one lags or stops, it directly breaks synchronization or halts recording.
Our first decision was the zero point—when to start writing. We chose to begin writing video as soon as the first audio chunk arrives. We based this on the principle that humans are more sensitive to audio dropouts than video frame skips. A missing first frame is less noticeable than a missing audio onset.
Trap #1: Stop Trying to Fix "Mouth Sync" by Feel
The first feedback we got was: "The mouth movements don’t match the voice." We tweaked values, re-streamed, and checked visually. Still felt off. Tweaked again. After several rounds, we realized: we were trying to fix two entirely different types of desync with a single adjustment knob.
| Type of Desync | Root Cause | How to Measure | Our Value |
|---|---|---|---|
| Mechanical A/V Offset | Time difference due to separate mux paths for video and audio | Measurable (embed markers and auto-measure) | Measured 6ms (≈ zero) |
| Perceived Lip Sync | Delay from when audio is analyzed in-page to when the avatar’s mouth moves | Not measurable—determined by human perception | Delay audio by 0.10 seconds |
Mechanical Offset: Measure Using Embedded Markers
To measure the offset, we embed "simultaneous events" in both video and audio, then analyze their positions in the final recording. The difference is the offset.
On the page side, we emit a calibration marker: a beep and a full-screen flash at the same time.
// Calibration marker: emit sound and flash simultaneously
function emitCalibrationMark() {
// Sound: short beep
const osc = ctx.createOscillator();
osc.frequency.value = 1000;
osc.connect(dest);
osc.start(); osc.stop(ctx.currentTime + 0.05);
// Flash: make screen white
document.body.style.background = '#fff';
setTimeout(() => { document.body.style.background = ''; }, 50);
}
After recording, we analyze the output with ffmpeg.
# Audio side: detect silence breaks = beep positions
ffmpeg -i out.mp4 -af silencedetect=n=-40dB:d=0.03 -f null - 2>&1
# Video side: detect scene changes = flash positions
ffmpeg -i out.mp4 -vf "select='gt(scene,0.5)',showinfo" -f null - 2>&1
The time difference between the two is the offset. In our environment, it was 6ms—effectively zero. We didn’t apply any itsoffset correction in ffmpeg.
The key insight wasn’t that it was 6ms—it was that we could measure it. If the environment changes, we can re-measure. Once confirmed, we removed the mechanical offset from further discussion.
Perceived Lip Sync: Only Humans Can Decide
Even after confirming the mechanical offset was zero, the mouth still felt "too early." This led us to the second issue.
The avatar’s mouth movement is driven by analyzing the audio within the page. This introduces inherent tracking latency—the mouth appears to lead the audio slightly. This isn’t a pipeline issue; it’s internal to the page. And crucially, there’s no numerical "correct" value—whether it "looks right" depends on human perception.
Our fix: add a delay node to the audio sent to the capture pipeline and tweak it visually.
const delay = ctx.createDelay();
delay.delayTime.value = 0.10; // Determined by visual inspection
tapSource.connect(delay).connect(dest);
We settled on 0.10 seconds. Too small and the mouth feels early; too large and the audio lags. This calibration is baked into the output video, so it works the same on YouTube or Twitch. We don’t need to re-adjust per platform. Also, this is unrelated to the 15–30 second delivery delay to viewers—mixing those up breaks the discussion.
Trap #2: Progressive Drift — Never Use setInterval for 30fps Framing
After calibrating the initial offset, a new synchronization bug emerged:
- At stream start: mouth and audio are in sync
- After 5 minutes: slight drift
- After 15 minutes: obvious desync
The key clue: it gets worse over time.
"Initially Off" vs. "Progressively Off" Are Different Problems
| Symptom | Root Cause Family | Fix |
|---|---|---|
| Initially off by a fixed amount | Fixed offset (processing delay differences) | Correct with a constant (e.g., itsoffset or delay node) |
| Drift increases over time | Accumulated error (clock drift, timer lag) | No constant can fix this. Must stop accumulation. |
Fixed offsets are solved by measuring and applying a constant—exactly what we did with the 0.10-second calibration. But no constant adjustment fixes progressive drift. You can tweak it to sync temporarily, but it drifts again. Once you find yourself "tweaking it again," you should suspect accumulated error.
The Culprit: setInterval Frame Pacing
We received screencast video and normalized it to 30fps on the Node side before passing it to ffmpeg. The pacing looked like this:
// Intended: write 1 frame every 33.33ms
setInterval(() => {
writeFrame(latestFrame);
}, 1000 / 30);
But setInterval doesn’t mean "fire every 33.33ms"—it means "fire at least 33.33ms after the previous call, when the event loop is free." If the loop is busy, it fires late. And critically, late firings are never made up.
Even if each delay averages 1ms, at 30fps that’s 30ms per second, 1.8 seconds per minute, and 27 seconds over 15 minutes. Audio flows in real time on a separate path. Video falls behind due to missing frames. This is the true cause of drifting lip sync.
The Fix: Calculate "Expected Frame Count" from Elapsed Time
Instead of "writing one frame every interval," we switch to: "Check the wall clock, calculate how many frames should have been written, and write any missing ones."
const FPS = 30;
const startedAt = Date.now();
let written = 0;
setInterval(() => {
const elapsed = Date.now() - startedAt;
const shouldHave = Math.floor(elapsed * FPS / 1000); // Expected frame count
while (written < shouldHave) {
writeFrame(latestFrame);
written++;
}
}, 1000 / FPS);
Even if the timer fires late, the next interval catches up by writing multiple frames. Error resets each time—no accumulation. You can keep using setInterval; the key is to trust elapsed time, not timer firings.
General Principle: Count Time, Not Events
This pattern applies broadly to timer usage:
// Don't do this
count++; // Count firings
elapsed += intervalMs; // Accumulate intervals
// Do this
const elapsed = Date.now() - startedAt; // Take time difference
The former assumes timers are accurate—a flawed assumption in browsers and Node. Event loop congestion, GC, background tab throttling, system load—there are countless reasons for delays. The longer a system runs, the more this gap grows. Short tests (like 90 seconds) won’t catch it. We only noticed after running full show lengths.
Trap #3: A 90-Second Recording That’s Only 6 Seconds Long
The third bug occurred in a separate path: recording screen and audio with MediaRecorder to archive content. We ran it for 90 seconds, but the resulting file was only 6 seconds long.
It reproduced every time. No matter what, ~6 seconds. And upon inspection, only the avatar’s greeting segment (TTS) was recorded.
The root cause, in short:
When no real audio samples flow into
MediaStreamAudioDestinationNode,MediaRecorderstops writing video frames too. If the audio track halts, the video stream halts.
We were only recording the 6 seconds when TTS was playing. The remaining 84 seconds were effectively silent—not just no sound, but no audio track at all.
Initial Suspicions (All Wrong)
We went down a rabbit hole:
1. CPU rendering was too slow. We were using CPU rendering (SwiftShader) and only getting 4fps. "Frames are dropping because rendering can’t keep up" seemed plausible. The numbers even loosely matched: 4fps ≈ 1/10 real time. This misled us for days. Plausible hypotheses can derail investigations—especially when they align loosely with observations.
2. Codec was too heavy. We tried forcing VP8. No change.
3. Capture method issue. We suspected canvas.captureStream() contention with the main thread. While this was a real issue elsewhere, it wasn’t the cause here.
The Breakthrough
The key clue: the bug persisted even after switching to GPU rendering. We were getting 57fps, yet the recording still cut off after a few seconds. That shattered the "slow rendering" hypothesis.
What remained: Why was only 6 seconds recorded? Re-examining the recording and avatar logs showed perfect alignment: only the TTS-speaking segment was recorded.
Why This Happens
MediaRecorder muxes video and audio tracks into one file. Since muxing requires aligning timestamps across tracks, if the audio timeline stops, the muxer can’t advance the video either.
In our setup, audio was assembled via Web Audio API and output through MediaStreamAudioDestinationNode. The critical detail: this node doesn’t necessarily emit silent samples when the source is silent. If the audio timeline doesn’t progress, the muxer can’t write video frames—even if they’re coming in. Result: video frames arrive, but the file doesn’t grow. Only when audio (TTS) plays do both timelines advance, leaving only that segment in the file.
The Fix: Keep the Audio Timeline Alive with Silent Samples
We connected a ConstantSourceNode outputting 0 continuously to the audio graph.
const ctx = new AudioContext();
const dest = ctx.createMediaStreamDestination();
// Keep timeline alive with a constant 0
const keepalive = ctx.createConstantSource();
keepalive.offset.value = 0;
keepalive.connect(dest);
keepalive.start();
// Real audio (e.g., TTS) also connects to dest
ttsSourceNode.connect(dest);
This ensures the audio timeline always advances, even during silence. Output remains unchanged (we’re just adding 0), but the audio track’s timeline never stalls. Now a 90-second run produces a 90-second file.
We also delayed starting the recorder by 500ms to avoid instability when the audio graph isn’t fully set up.
Six Lessons Drawn from Three Traps
Though these bugs occurred in different places, their underlying lessons converge.
1. Stabilize what you can measure first. Lip sync confusion stemmed from mixing "mechanical offset (measurable)" and "perceived lag (not measurable)." When you find yourself tweaking by feel, ask: Is there a measurable component hidden in here? Fixing that first often shortens the rest of the process dramatically.
2. Use symptom evolution to classify root causes. "Initially off" suggests fixed offset; "progressively worse" suggests accumulated error. If tweaking a constant fixes it temporarily but it drifts again, check whether the problem grows over time—it’s faster than endlessly re-tuning.
3. Don’t trust timer firings—trust elapsed time. setInterval isn’t accurate, and delays aren’t recovered. Instead of counting firings, calculate "expected amount" from wall time and catch up. Accumulated error only appears in long-running systems.
4. Beware of hypotheses that "fit the numbers." The 4fps ≈ 1/10 real-time coincidence was accidental. A plausible story can waste days. The real breakthrough came when we removed a variable entirely (GPU vs CPU) and saw the bug persist. Sometimes, breaking your hypothesis is faster than validating it.
5. In media processing, "nothing" and "silence" are not the same. Whether an empty audio track emits silence or is truly empty changes downstream behavior. Confirm how downstream interprets "no data"—empty arrays vs null, 0 vs undefined, silence vs missing track.
6. Monitor output duration as a key metric. The 6-second bug didn’t appear in fps logs, error traces, or CPU usage. The only detectable signal was comparing actual file duration to intended runtime. Now, our validation criteria include: "Did an 89-second run produce an 89-second file?"
Building fully automated systems means constantly dealing with silent failures—media APIs rarely crash; they return subtle drift instead. That’s why measuring what you can, trusting time over timers, and monitoring duration matter. These unglamorous practices become the scaffolding that keeps things running.
Top comments (0)