Two Events with the Same Name: An Audio Pipeline Mix-Up I Fell for Three Times

📝 Originally published (in Japanese) at forge.workstyle.tech.

Reported issue for the voice conversation avatar embedded in our website:

The transition to the next page happens before the bot finishes saying "I'll show you around," causing the utterance to be cut off.

We had already implemented a mechanism to wait for the end of the utterance before transitioning. We were just waiting for the wrong signal.

There are two meanings of "Speaking Finished"

In the framework we were using, BotStoppedSpeakingFrame signaled the end of the bot's speech.
However, this frame was being emitted from two different places.

Source	Meaning
Brain (Response Generation)	Text generation is complete. Audio is still playing.
Output Transport	Actual playback is complete.

Text-to-speech synthesis is much faster than playback. So if we treated the "text complete" signal as "playback complete," the screen changed the moment the text was generated. The audio was still playing.

The tricky part: both frame sources used the same class. Type checking alone couldn't distinguish between them.

The Fix Made It Worse

I changed the logic to: "If there is still unplayed audio remaining, don't treat it as playback complete." essentially checking the remaining amount of scheduled audio.

Result: Transitions stopped happening entirely.

tts_sentence  14 times
tts_drained    0 times      ← The "playback complete" signal never fired

The culprit was rounding errors. The expected byte count and the actual byte count didn't match perfectly, leaving a tiny remainder. The code kept interpreting this remainder as "still playing."

Ironically, the original reason we started using this frame as a signal was to work around this exact rounding issue that prevented the signal from firing. I fell into the same hole, this time as a side effect of the workaround.

Distinguish by Source, Not State

Instead of looking at remaining bytes, we should have distinguished who created the frame.

The output transport sends paired frames upstream and downstream, embedding each other's IDs in the frames. Raw frames created by the "Brain" don't have this. This is a reliable discriminator.

_unknown = object()
_sibling = getattr(frame, "broadcast_sibling_id", _unknown)
if _sibling is None:
    # Raw frame = Text is done. Audio is still playing.
    return

⚠️ In versions where the attribute doesn't exist, fall back to the original behavior (emit the signal). Failing silently when you can't determine the source is the worst outcome (that's exactly what caused the transitions to stop).

Round 3: The Same Thing Was Happening at "Speaking Start"

To handle self-echos, I added a gate to "drop utterances that start immediately after the avatar finishes speaking." On real hardware, the gate's logs showed zero hits. The gate was enabled and present in the pipeline.

Turns out there were also two types of "speech start" frames.

Source	Where it's created
User Aggregator	Declaration of turn start. Downstream of the gate
Input Transport	VAD detection. Upstream of the gate

I was looking for the first one. Frames created downstream never flow back upstream. The gate was effectively non-existent.

I had hit the "speaking finished" variant twice, but I failed to check if "speaking started" had the same trap.

Unit Tests Couldn't Catch It

All three times, unit tests were green. Naturally, tests use frames I created myself. They don't reflect the actual types flowing through the pipeline in production.

To prevent this, I added this check at the top of my tests:

def _transport_emits(frame_cls) -> bool:
    """Does the input transport actually emit this frame?"""
    import inspect
    from pipecat.transports import base_input
    return f"{frame_cls.__name__}(" in inspect.getsource(base_input)

This inspects the library's source code. If a library update changes the frame types, this test will fail.

Generalized Lessons

Events with the same name but different meanings are common in frameworks. "Start," "End," and "Complete" change meaning depending on the layer. "End" at the application layer means logical completion; "End" at the transport layer means physical completion.

Two key lessons for distinguishing them:

1. Read the source code of the component emitting the event. Just grep for FrameName(. If there are multiple emission sites, they likely have different meanings.
2. Check the direction. In pipeline-based frameworks, you won't receive frames created downstream of you. "Not receiving it" and "It wasn't emitted" are different problems.

And the biggest takeaway: count how many times it fires on real hardware. In round 3, I only noticed because the log count was zero. If I hadn't been tracking the count, I would have endlessly attributed the "gate isn't working" issue to other causes.

---

Series: Making the Voice Avatar Actually Answer

This is a record of the 3 days I spent fixing the quality of responses from a voice conversation avatar embedded in a website.
This article is Part 1: Stopping the Audio.

→ Next: The self-echo gate never fired once

All 8 articles in the series

Part 1: Stopping the Audio

1. There were two frames with the same name ← You are here
2. The self-echo gate never fired once
3. A finger on the speaker broke the echo canceller

Part 2: Decoding Words

1. "That," "This page," and "The earlier one" were three different things
2. One line at the end of a huge prompt was ignored 4 times in a row
3. Apology phrases were poisoning the next search

Part 3: Distinguishing

1. Not every utterance is a question
2. I thought I was measuring this, but it was actually that

The notes that served as the basis for these insights are summarized in Voice Conversation Avatar Response Quality.