Barge-in for eight AI voices: what it took to let people interrupt a live AI group call

I build AI Group Call, a mobile app where you put two to eight AI voices on one live voice call and talk an idea through with them. A host, a skeptic, a couple of specialists, or models you pick (Claude, GPT, Gemini, Grok) as seats at the table.

The hardest part was not getting eight voices to talk. It was letting a human interrupt them.

In a one-on-one voice assistant, barge-in is mostly solved for you. In a room of eight, every mistake is louder: an agent that hears its own voice will answer itself, a missed "wait, stop" means three more people talk over you, and a cough can derail the whole conversation. Here is what ended up working.

The setup

Each call has:

• One transcription session for your mic (a realtime Whisper model).
• One realtime speech session per AI participant. These are stateless: every turn, the server sends the shared transcript in the turn instructions, so all agents share context without sharing a session.
• A small, fast director model that decides who speaks next and in what order, with a round-robin fallback.

A server-side "conductor" owns the floor. Only one voice speaks at a time, and the conductor decides when that changes.

Lesson 1: don't wait for the transcriber to tell you someone is talking

My first version used transcription events to detect that you had started speaking. The problem: transcription deltas lag speech by seconds, and the transcriber I use has no server-side voice activity detection. Interruptions felt dead. You would say "hang on" and the agent would finish its paragraph.

The fix was to run my own energy-based VAD on the raw PCM coming from the phone:

• RMS energy per chunk against an adaptive noise floor.
• Agent audio is held within about 150 ms of a likely voice.
• A short probe (around 700 ms) confirms it was really speech, then the floor is handed to you.
• Transcriber speech events are OR'd into the same signal, so either source can trigger it.

Holding first and confirming second is the key. Pausing an agent for half a second by mistake costs almost nothing. Talking over a human costs a lot.

Lesson 2: the agents will hear themselves

The first time I tested on Android, the agents started answering each other in a loop. I was playing their audio on the media stream, so the hardware echo canceller had no reference signal and cancelled nothing. The mic picked up the agents and fed them straight back in.

Running the whole call in communication mode, with speakerphone forced, fixed most of it (plus a small software gain, because the voice stream is quieter). But "most" is not enough when eight voices are involved, so echo protection is now layered, and each layer is tested on its own:

1. Capture. Platform voice processing on (iOS voice processing, Android communication-mode capture). A build that silently loses this still compiles and ships, so the patch that enables it runs first at install time and fails loudly if the audio library version changes.
2. Client gate. While agent audio is actually playing, the app only forwards mic frames that are clearly louder than a rolling noise floor and sustained (about 300 ms). A short pre-roll goes out with an accepted burst so your first word is not clipped.
3. Server VAD. Same idea, independently. With the floor open, about 170 ms of voice is enough to count as speech. Over a speaking agent it needs 300 ms or more of clearly louder, sustained speech. There is also an echo history: if several barges in a minute turn out to be echo, bare voice stops being trusted to interrupt for a while.
4. Transcript. If a transcript is mostly made of the agents' own recent words (70% or more), it is dropped as echo. And agents never receive audio at all, only text, so there is no path for them to hear the room directly.

Every barge decision logs a single line with its RMS, how long it was voiced, and who held the floor. When something regresses, it shows up in the logs instead of in a one-star review.

Lesson 3: not every interruption is a turn

Once interruptions worked, a new problem appeared. People say "yeah" and "mm-hmm" while listening. With instant barge-in, every "yeah" killed the speaking agent, and the room would then respond to "yeah" as if it were a point.

So an interruption now gets a verdict. The speaking agent still stops instantly, but the transcript that arrives a second later decides what it was:

• Backchannel ("yeah", "mm-hmm", "sí"): the turn goes back to the agent, who continues from where it stopped.
• Noise (no words within 2.5 seconds): same, hand it back.
• Stop ("please stop", "shut up"): the room holds until you say something with content.
• Anything else: a real turn. The director picks who answers.

Before this, "please stop" earned you another round of opinions. That is funny once.

Lesson 4: pick the director for latency, not just accuracy

The director runs on every turn, so its latency is dead air. I benchmarked candidates on real transcripts: several models got every pick right, but one did it at about 0.5 s median while another took nearly 2 s and blew the time budget on about a quarter of turns. Accuracy was a tie; speed decided it.

Even the fast one has bad days, so the director call is hedged, not retried: after 1.2 seconds of silence, a fallback model starts in parallel and the first valid answer wins.

What it adds up to

On a call, you can cut in at any moment, the room goes quiet, and whoever is best placed answers you. The agents build on each other instead of taking turns reading out separate answers, which is the whole point: it is closer to an LLM council you can talk to than to asking three chatbots the same question.

If you are building anything multi-voice, my short list would be:

• Run your own VAD on raw audio. Do not wait for transcripts.
• Hold first, confirm second.
• Assume echo will get through one layer, and add another.
• Classify interruptions before you act on them.
• Measure your turn-taking model's p50 and p90, not just its accuracy.

If you want to hear it, AI Group Call is on iOS and Android with three free minutes: aigroupcall.app. Happy to answer questions about any of this in the comments.