DEV Community

orca_forge
orca_forge

Posted on Originally published at forge.workstyle.tech

Echo Issue Went Unnoticed as It Never Triggered

#pipecat

📝 Originally published (in Japanese) at forge.workstyle.tech.

The Voice Dialogue Avatar Was Responding to Its Own Utterances 

🤖 (filler) "Yes, I'll check, please wait a moment."
STT: [Yes, I'll check, please wait a moment.] Confidence -0.306
🤖 "I'll check, please wait a moment."
Enter fullscreen mode Exit fullscreen mode

It's not fabrication, but correct transcription, so it can't be dropped due to confidence. The confidence threshold is intentionally disabled (the true identity of low confidence is surrounding environmental noise, which can lower the confidence of real utterances to -0.826).

There was no defense on the server side, and only the browser's echo canceller was relied upon.

Made According to Convention

Investigation revealed that this issue has a established solution: discard input for a certain period of time after utterance playback ends.
300ms was insufficient, and 500-800ms was required (AEC tail).

Input during playback is not discarded. Discarding it would prevent interruption, which is the responsibility of the echo canceller.

I implemented this and wrote tests.

OK Utterance starts immediately after playback ends (self-echo) → Discarded
OK Utterance starts 600ms after playback ends (within AEC tail) → Discarded
OK Utterance starts outside the window (real response) → Passed
OK Utterance starts during playback (barge-in) → Passed
OK All 3 intermediate results are discarded
...
8/8 Passed
Enter fullscreen mode Exit fullscreen mode

I also ran it in a container with the real library. 8/8. Deployed.

Zero Incidents in Actual Machine

A user verified it on their smartphone and collected judgment logs.

echo-gap breakdown:
  (empty)

Discarded count: 0
Enter fullscreen mode Exit fullscreen mode

The conversation was successful. The avatar uttered 18 times, and the user started speaking 17 times. Yet,
the gate judgment was never reached.

The setting was enabled. The pipeline was entered. The file contents matched the source hash.

The cause was that the frame type being observed was different. The frame type that VAD outputs for utterance start and the frame type that the user aggregator outputs were different, and I was looking at the latter. The aggregator is downstream of this gate. Things created downstream do not flow upstream.

The gate was operating as if it didn't exist.

Why the Tests Were Green

It was only natural. The tests were using frames I created and passed through.

await gate.process_frame(UserStartedSpeakingFrame(), DOWNSTREAM)
Enter fullscreen mode Exit fullscreen mode

It's only natural that what I created and passed through would be processed, and the fact that the actual frame type is different wouldn't appear in this test.

I rewrote the test and added the following to the beginning:

def _transport_emits(frame_cls) -> bool:
    """Whether the input transport actually outputs this frame."""
    import inspect
    from pipecat.transports import base_input
    return f"{frame_cls.__name__}(" in inspect.getsource(base_input)

check(f"{UserStart.__name__} is the type that transport.input() actually outputs",
      _transport_emits(UserStart))
Enter fullscreen mode Exit fullscreen mode

I checked the library source code. This became the first item.

Stepped on the Same Issue Again on the Same Day

I added another countermeasure (raising the VAD threshold during playback) to the revised version. The tests passed. This time, I was using a fake VAD object.

class _FakeVAD:
    """Minimum VAD with only `_vad_start_frames` (same attribute name as the real one)."""
    def __init__(self, frames: int = 4):
        self._vad_start_frames = frames
Enter fullscreen mode Exit fullscreen mode

The moment I added an item to verify that "it works with the real VAD", it failed.

AttributeError: '_vad_start_frames'
Enter fullscreen mode Exit fullscreen mode

The real VAD doesn't have that attribute initially. It's created when the sampling rate is decided (at pipeline startup). My code was swallowing the exception, so it was "fixed" without actually being fixed.

except Exception:  # noqa: BLE001
    pass          # ← This creates "it worked without actually working"
Enter fullscreen mode Exit fullscreen mode

I made sure to log when swallowing exceptions. Both the evidence of it working (info) and the fact that it didn't work (warning, only once).

Effect of the Countermeasure

The revised version with the correct type finally made the gate work.

echo-gap 7ms within window → Discarded as self-echo
echo-gap 5037ms outside window → Passed (real utterance)
echo-gap 10454ms outside window → Passed
echo-gap 17231ms outside window → Passed
Enter fullscreen mode Exit fullscreen mode

Utterance that started 7 milliseconds after playback ended. Clearly its own voice.

Generalizable Points

"Implemented" and "working" are different, and the difference can't be covered by tests.

It's only natural that tests pass with the input I prepared, but if the actual input is different, the tests will all be green while the actual machine does nothing. I stepped on the same issue 4 times that day.

I was able to detect it because I was counting how many times it fired in the actual machine.

  • Number of judgments entered (all, including passed ones)
  • Number of discards
  • Evidence of countermeasures being effective

⚠️ Don't just count the number of discards. That would make it impossible to judge whether the window width is reasonable, as the "leaked" part outside the window would be invisible in principle. Also record the passed ones.

And when swallowing exceptions, always log them. except: pass is a setup that makes it look like it's working when it's not.

Series: Until the Voice Dialogue Avatar Responds Correctly

This article is part of Part 1: Stopping the Sound.

← Previous: There were two types of events with the same name
→ Next: The smartphone's finger was breaking the echo canceller

Series of 8 articles

Part 1: Stopping the Sound

  1. There were two types of events with the same name
  2. The self-echo countermeasure was never working ← Now here
  3. The smartphone's finger was breaking the echo canceller

Part 2: Understanding Language

  1. "It", "this page", and "earlier" were different things
  2. The one line at the end of the huge prompt was ignored all 4 times
  3. Apology words were contaminating the next search

Part 3: Judging

  1. Not all utterances are questions
  2. I was measuring something different from what I thought

The notes that led to this insight are summarized in Improving the Response Quality of Voice Dialogue Avatars.

Top comments (0)