My smartphone fingers were breaking the echo cancellation — speech cutting out due to non-verbal sounds

📝 Originally published (in Japanese) at forge.workstyle.tech.

The Avatar that Stopped Working on iPhone Safari

The voice dialogue avatar that worked fine on PC had an issue on iPhone Safari.

The avatar seems to be reacting to its own speech.
It also seems to be reacting when the speaker's volume changes while operating the smartphone.

What the Logs Showed

Looking at the judgment records, a pattern emerged.

13:39:52.843  Speech start detected (VAD)
13:39:52.848  Bot stopped speaking   ← The avatar was stopped 5 milliseconds later

All five instances showed the same pattern. The decisive factor was that no transcription was output.

The VAD judged that "speech has started" and stopped the avatar's speech as an interrupt. However, the sound was not recognized as a word, and no turn occurred. There was also a period where the avatar started speaking and was stopped repeatedly at 2-second intervals.

Mechanism

The user's guess was " sudden decrease in volume". To clarify further, when you cover the speaker with your finger, the sound's loudness and resonance change. The browser's echo canceller learns the relationship between the "sound sent to the speaker" and the "sound returned to the microphone" and cancels it out. If this relationship changes suddenly, it cannot be completely cancelled out, and the residual sound leaks out.

The VAD sees this as human speech. The reason there was no transcription output was that it was not a word.

The reason it didn't happen on PC was not because the countermeasures were effective, but because the browser's AEC was canceling out the echo during playback.

The Gate I Created is Not Effective for This Symptom

I had already implemented a gate as a countermeasure against self-echo, which drops speech immediately after the sound finishes. However, this is not effective. By design, speech during playback is allowed to pass through. This is done to prevent interrupts from being killed.

Self-echo    … After the sound finishes (AEC tail) → Gate's responsibility
Incorrect interrupt    … During playback                   → Outside gate's responsibility

Echo and incorrect interrupts occur at different times. A different countermeasure is needed.

How the Industry Handles This

I investigated how web services that keep the microphone open for conversation handle this issue.

LiveKit's publicly available numbers were specific. They don't trigger an interrupt immediately, but instead look at the first few hundred milliseconds of speech (the rise, sustain, and rhythm of the waveform) before making a judgment.

• Listen to 216ms of audio on average before interrupting
• This rejects 51% of VAD-based interrupts
• The judgment model's inference itself is less than 30ms

In other words, delaying the judgment by 200-300ms to examine the audio's characteristics was the answer.

On the other hand, no one has a silver bullet for echo itself. Deepgram's official documentation only says "leave it to the browser's standard", and there is no mention of mobile or speakerphones. Depending on the client-side AEC is a common premise.

Implementation: Make VAD Less Sensitive During Playback

Since all observed misfires were short sounds (no transcription output), we can filter by duration.

During the avatar's speech, increase the continuous time required to recognize speech from 0.2 seconds to 0.5 seconds
Do not change the normal time ＝ The response time of the turn remains unchanged. Only the interrupt becomes less sensitive

The framework we were using had a mechanism for interrupt strategy, but waiting for the transcription to be ready would delay the interrupt by about 1.6 seconds (1.2 seconds of silence waiting + speech recognition). This contradicts the requirements for responsiveness, so we didn't adopt it.

⚠️ The VAD parameter setting API could not be used. Because it initializes the internal state, calling it while the user is interrupting and speaking will cut off the speech. Moreover, the timing you want to call it (when the avatar's speech stops) is precisely that moment. We changed it to directly overwrite the threshold frame count.

⚠️ This attribute does not exist immediately after creating the object. It is born when the sampling rate is decided (at the time the pipeline starts). We were testing with a fake object, so we didn't notice it, and it crashed the moment we added the item to test with the real thing. Until then, we were suppressing exceptions and doing nothing.

Results

                    Before countermeasures    After countermeasures
Interrupts during playback    5 instances      0 instances
Self-echo disposal      0 instances      1 instance (speech 7ms after playback ended)
Real speech           ―        3 instances all passed (5.0 seconds / 10.5 seconds / 17.2 seconds later)

We also left evidence of the countermeasures being effective in the logs.

Countermeasure against incorrect interrupts: increase the speech judgment time during playback from 0.2 seconds to 0.5 seconds

This line not being output means the countermeasures are not working. Since we were once deceived by this, we made sure to output both the evidence that it worked and the fact that it didn't work.

Generalizable Points

"Audio input" and "speech" are different events, and VAD can only judge the former.

VAD cuts speech intervals based on audio energy and features. It cannot determine whether a person intentionally spoke. Therefore, coughs, noise, machine contact sounds, and echo residuals all become "speech".

There are two directions for countermeasures.

1. Cut by time (ignore sounds that don't continue for a certain time) — deterministic, light, and delayed by several hundred milliseconds
2. Cut by characteristics (look at waveforms or rhythms) — high accuracy, requires a model

The difficulty level also changes depending on the environment. Desktop PCs with headsets are easy, while smartphones with speakers are difficult. The device moves, the way it's held changes, and the speaker is covered with a finger — the echo canceller's premise that "the sound path is constant" does not hold for mobile devices.

"Since it worked on PC, it's fine" is not applicable to voice.

---

Series: Making the Voice Dialogue Avatar Answer Correctly

This article is the last part of Part 1: Stopping the Sound.

← Previous: The gate that never fired
→ Next: The three kinds of "it"

Series of 8 articles

Part 1: Stopping the Sound

1. There were two types of events with the same name
2. The gate that never fired
3. A finger on the speaker was breaking the echo canceller ← Now here

Part 2: Understanding Words

1. The three kinds of "it"
2. One line at the end of a huge prompt was ignored four times
3. Apology words were poisoning the search

Part 3: Judging

1. Not all speech is a question
2. I thought I was measuring something, but it was actually something else

The notes that led to this insight are summarized in The Quality of the Voice Dialogue Avatar's Response.