When an AI Avatar Keeps Replying to Its Own Voice — Writing 1,657 Lines of Band-Aids, Then Throwing Them All Away

📝 Originally published (in Japanese) at forge.workstyle.tech.

When building an AI avatar that can converse through voice, you'll inevitably encounter this scenario:

The avatar greets the user. The microphone picks up the voice from the speaker. Speech recognition transcribes it as "user speech." The avatar responds to its own greeting. The microphone picks up the response again—the avatar starts asking and answering itself.

We fought this "self-echo problem" for a long time, piling on symptomatic fixes, until we finally scrapped them all and rebuilt the architecture from scratch. We removed 1,657 lines of code from the frontend. This article chronicles the entire ordeal and almost every pitfall we encountered along the way.

The Quagmire of Symptomatic Fixes

Initially, we had a straightforward setup. The browser opens the microphone, uses VAD (Voice Activity Detection) to segment the input, sends it to STT (Speech-to-Text), synthesizes the response with TTS (Text-to-Speech), and plays it back via Web Audio—all within the browser.

When self-echoes occurred, we added symptomatic fixes. Each one seemed reasonable in isolation:

• Echo window: Discard recognition results similar to the avatar's speech for N seconds after it speaks
• Text matching: If the bigram match rate with the avatar's recent speech is ≥0.75, discard as "echo"
• Hallucination vocabulary list: Block common false positives STT produces during silence
• Duplicate question discard: Treat identical phrases within 30 seconds as repeats and discard
• Hold-type barge-in: Don't stop playback immediately when the user starts speaking (as the echo would stop its own playback). Interrupt only after STT confirmation

These fixes mostly stopped the self-questioning. However, each fix created new problems:

Text matching discards the most natural user behavior—"asking about what the avatar just said"—because it's too similar to the recent speech. Hold-type barge-in delays interruptions by 2.1–2.5 seconds, ruining conversational flow. The echo window also swallows genuine speech. Every adjustment to the fixes broke something else, leading to a state where "echoes disappeared, but real questions did too."

Turning Point: All Fixes Stemmed from One Constraint

One day, while reviewing the list of fixes, I realized they weren't independent solutions but links in a causal chain:

Unable to acoustically eliminate self-echo
  → Can't stop playback immediately when user speaks (avatar's voice stops itself)
    → Requires hold-type barge-in (2+ second delay)
      → Need to determine if confirmed text is an echo
        → Requires text matching, echo window, vocabulary list, duplicate discard
          → Ends up discarding real questions too

The root cause was just one thing: The browser can't acoustically remove its own speaker output from the microphone input.

This raised a question: Browsers have built-in AEC (Acoustic Echo Cancellation). Why wasn't getUserMedia({ audio: { echoCancellation: true } }) working?

The answer lies in how AEC works. Echo cancellation requires a reference signal—what to cancel. Browser AEC can reference only the browser's own audio playback paths (e.g., <audio> elements or WebRTC receive tracks). Since we were playing TTS audio through a custom Web Audio graph, this path wasn't a reliable reference for AEC. The AEC was sitting idle, unaware of what to cancel.

The Cure: Move Audio Pipeline to Server, Use WebRTC for Sound

Identifying the constraint dictated the solution:

1. Move VAD, STT, turn management, and TTS to a single server-side pipeline (using Pipecat)
2. Browser just sends microphone input via WebRTC
3. Avatar audio is returned as a remote track in the same WebRTC session and played via <audio>

The key is point 3. Remote track playback becomes an official reference signal for browser AEC. The browser now knows exactly what to cancel—the avatar's voice.

The results were dramatic:

• Even with the speaker volume maxed, 99 seconds of continuous avatar speech produced zero false user turn detections—without text matching, echo windows, or vocabulary lists
• Barge-in latency dropped from 2.1–2.5 seconds to 4–6 milliseconds
• The chain of symptomatic fixes became entirely unnecessary once the root constraint was removed

...If only the story ended here. But the real challenges began during the transition. It was a minefield. Below are all the traps we encountered, in order.

Trap 1: AEC Isn't Free—Double Talk Distorts User Voice

Soon after switching to the new setup, users reported increased recognition errors. "Employee count" became "shines."

Investigation revealed the mechanism. In the old setup, AEC was effectively disabled, so user audio was mixed but undistorted. In the new setup, AEC cancels echoes but distorts the user's voice (especially at word starts) when they speak simultaneously with the avatar—a fundamental trade-off called near-end suppression. Zero self-echo came at a cost.

We implemented a three-layer solution:

1. At the input: Set microphone Opus bitrate to 128kbps and enable in-band FEC (by modifying the fmtp line in the answer SDP). The old setup sent uncompressed PCM, so we needed to match the information density
2. STT: Pass the avatar's recent speech text as initial_prompt to faster-whisper. Crucially, don't create a dictionary. Company-specific dictionaries grow endlessly. Instead, pass the avatar's recent speech text directly—scripts and responses already contain proper nouns in correct spelling, requiring zero registration. Never mix user STT results (one misrecognition could contaminate the prompt). Limit prompt length to 160 characters (longer prompts cause whisper to "hear" the prompt text)
3. LLM: Add instructions to the system prompt: "Input is speech-to-text transcription and may contain phonetic errors. Interpret unnatural words as phonetic matches and confirm with 'Regarding ◯◯...' at the start. If no match is found, ask for clarification without inventing explanations"

Layer 3 was surprisingly effective. Just as humans can interpret messy chat input through context, LLMs can handle phonetic errors in speech. But without "don't invent explanations," the LLM would confidently fabricate product details for unknown words like "shines."

Trap 2: "Not Responding" ≠ "Not Hearing"

Users reported "no response for a while after starting a presentation." Server logs showed complete silence during those periods—the user's voice wasn't being ignored, it wasn't reaching the server.

Logs from the model we'd added for estimating backchannel timing (monitoring microphone audio at 10Hz) proved invaluable. The response probability was a flat 0.000—no acoustic energy was arriving. Multiple culprits were found:

• Microphone track started as enabled=false (sending silence until the mic button was pressed). Users assumed they could speak anytime. Design expectations and implementation were misaligned → Changed to auto-enable on ready
• AEC/AGC convergence time. Echo cancellers learn only when the avatar is actually speaking. During the first utterance of a session, unconverged AEC swallowed double-talk audio entirely → Added a short greeting during the intro animation to train AEC before the main content. Also disabled autoGainControl (gain ramp-up made initial speech sound too quiet; server-side whisper handles volume fluctuations well)
• Added a level meter for the actual transmitted stream to the UI for faster debugging. Instantly distinguishes between "mic off," "eaten by preprocessing," and "server-side issues"

Lesson: For audio "no response" issues, you need observability into where the sound is dying. The 10Hz input logs and level meter made subsequent investigations exponentially faster.

Trap 3: Headless Environment Triple Threat (Streaming Renderer Edition)

This avatar also streams to YouTube/Twitch. Streaming uses a headless Chromium instance on a cloud GPU, rendering the avatar page and streaming canvas/audio via ffmpeg to RTMP. Switching to the new setup triggered three consecutive traps:

Trap 1: Headless environments have no microphone. The connection process called getUserMedia at the start, which threw an exception in the mic-less renderer, killing the connection and leaving the stream silent (while video kept alive with silent keep-alive—extra nasty). → Streaming pages now send a synthetic silent track created with WebAudio's createMediaStreamDestination. The SDP contract matches the microphone, so no server changes were needed.

Trap 2: Destinations without input may stop frame generation. When the silent track's RTP packets ceased, the server's audio reader errored and disconnected. → Added a ConstantSource node set to 0 to keep sending "actual silent samples."

Trap 3 (The biggest trap): Chromium doesn't route WebRTC remote MediaStreams to WebAudio until a media element consumes them. The streaming design tapped the avatar's remote track with createMediaStreamSource for capture. But without this, the stream was silent. It worked in normal pages because <audio> playback was already happening for AEC reference. The streaming page lacked this crucial step. → Solution: Attach the srcObject to a muted <audio> element and call play() (no sound needed—just "triggering" playback is the goal).

This known behavior requires a one-line fix if you know it, but without knowledge, you get the frustrating "server is speaking, connection is alive, but stream is silent" situation.

Trap 4: Framework Defaults Can Kill You—Idle Timeout

During streaming, we encountered "9 minutes of silence after greetings and chat responses, with no closing remarks." Logs revealed the culprit: Pipecat's idle timeout (default 300 seconds).

For conversational use, "kill pipelines idle for 5 minutes" is a sensible default. But streaming involves permanent microphone silence and legitimate silence when there's no topic. Five minutes after the last utterance, the pipeline committed suicide with Idle pipeline detected, cancelling, and subsequent closing remarks/viewer chats vanished into the void.

→ Disabled idle timeout for streaming routes (cancel_on_idle_timeout=False). Left dialog routes at default (effective for abandoned session cleanup).

Framework defaults become weapons when use cases change. It's worth auditing default assumptions for each workload.

Trap 5: Monitor Billing Yourself

Testing streaming on pay-as-you-go cloud GPUs taught another lesson. Failed avatar connections sometimes didn't trigger "start billing," preventing automatic time-based shutdown. Fortunately, a separate 25-minute safety timeout existed, but it was nerve-wracking until we realized this.

Now, after each test, we API-verify that the cloud instance list is empty and added a safety timer to the test sessions that force-checks and kills remaining instances after N minutes. When testing pay-as-you-go infrastructure, trust the framework's cleanup but manually confirm the final state for peace of mind.

Trap 6: Write Test Harnesses from Logs, Not Implementations

To reduce browser testing, we built an E2E harness that establishes WebRTC connections, plays pre-recorded WAV files at normal speed, and machine-judges message contracts (4 scenarios, 24 checks). This was hugely successful, catching almost all "messages not arriving/out of order" issues. Human testing was limited to self-echo, voice naturalness, and lip-sync.

However, we made one painful mistake. We wrote a test for the "continue" function (resuming explanations) based on our implementation, not real behavior. The harness passed, but real devices failed. The issue? Pausing could be triggered not just by the stop button but also by questions—a flow the scenario didn't replicate. Now, scenarios are written from actual utterance sequences in device logs.

Beware of false regressions too. Running the harness when TTS model pre-warming monopolizes the GPU causes STT to time out after 30 seconds. Post-deploy "it's broken!" reports should first suspect environmental factors (warm-up, quotas, concurrent builds).

Withdrawal: Deleting 1,657 Lines

After validating the new setup across five outputs (admin panel, embedded widget, desktop app, presentations, live streaming), we deleted the old setup and symptomatic fixes:

• Browser-side VAD/STT
• Echo guard, echo window, hallucination vocabulary, duplicate discard, hold-type barge-in
• Browser playback queue and its tied subtitle/expression/page transition sync mechanisms
• WS reconnection/heartbeat mechanisms (consolidated into WebRTC)

-1,657 lines from the frontend alone. Major components shrank from 957→639, 500→363, and 487→367 lines.

Deletion had traps too. Old setup practices were often incompatible with the new principles. For example, the old stopAudio() included lip-sync termination and reconnected the playback queue each time. The new setup has no reconnections, so the same call caused "mouth stays still forever" bugs. During migration/deletion, ask what each call assumes one by one.

Conclusion: When Symptomatic Fixes Pile Up, Hunt the Root Constraint

Key takeaways from this journey:

1. If you have >3 symptomatic fixes, diagram whether they stem from one constraint. If they're linked, pruning branches won't stop growth. Uproot the constraint to kill them all
2. Browser AEC lives and dies by the reference signal. Custom Web Audio playback often isn't a valid reference. WebRTC remote tracks + <audio> playback are officially recognized
3. AEC distorts user voice during double talk. The cost of canceling one problem appears elsewhere. STT vocabulary hints should use "your own recent speech"—zero registration cost, high effectiveness
4. Distinguish "not responding" from "not hearing" with observability (continuous input level logs, transmitted stream meters)
5. Headless environments are a different world: No microphone, no-input nodes stop, remote streams need consumers for WebAudio
6. Framework defaults become weapons when use cases change (idle timeout)
7. For pay-as-you-go testing, manually confirm termination
8. Write test scenarios from device logs, not implementations. Automate everything machine-judgable; rely on human ears/eyes only for subjective tests

Voice AI is easy to demo but hard to productize. Most of that gap comes from audio physics and browser realities like those described here. Hopefully, this shortcuts the journey for others in the same swamp.