A caller rings the Harborview Inn and asks what they need to know about checking in. The front desk starts the full rundown: check-in from 3 pm, photo ID and the card they booked with, a 100 dollar deposit, breakfast hours, the pool and the wifi. Four seconds in, the caller cuts across it with "Sorry, wait, is there parking?", and the client's log prints a single line: (barge-in: stopped the reply after 4.8 s).
That line is accurate, but it answers a different question. It records how far into the reply the client was when it decided to stop, which the server needs to trim its history. It does not record how long the caller heard the agent talking over them, and it cannot say whether the agent should have stopped at all, because the same handler runs for a question, a murmured "mm-hmm" or a television in the room whenever the server reports the sound as the caller.
In this guide, you will take the Harborview Inn agent in pmckeetx/barge-in-voice-agent and record scripted calls to PolyAI's Dialog-RSN-1 API as stereo WAV files, with the caller on the left channel and the agent on the right, both timed by when each sound actually played. You will then score them with bargein.py, a Python script that uses only the standard library and reads any 16-bit two-channel WAV, whichever stack produced it. For every moment the caller starts talking while the agent is speaking, it reports how long the agent kept talking and, given a label for what the caller meant, whether that was a stop, a missed barge-in, an ignored backchannel (a listener's "mm-hmm") or a false stop.
On 17 November 2026 I recorded 70 scripted calls against api.studio.poly.ai from the US Midwest. When the caller asked about parking, the agent's voice stopped a median 0.78 s after the caller started talking (0.76 to 0.86 s over 20 calls). If you hold your agent to a 300 ms budget, a figure some developers start from although no standard sets it, that misses by about half a second. Nearly all of that time passes before speech_started arrives, because while a reply is audible Dialog-RSN-1 confirms that a sound is the caller before it sends the event; the client then cut playback within one 20 ms tick. The same check let a murmured "mm-hmm" pass in 10 of 10 calls, but it mistook a newsreader on a nearby TV for the caller in 10 of 10, and the newsreader cut the reply every time.
Code: pmckeetx/barge-in-voice-agent (recorder.py, record.py and bargein.py)
What you'll build
You will add three files to a Python voice agent for the Harborview Inn. recorder.py writes each scripted call as a stereo WAV on the playback clock, with a JSON file beside it that says what the caller said, when, and what they meant. record.py runs a caller script repeatedly and prints the stop each recording shows next to the arrival of speech_started, so you can check the recording against the server's events. bargein.py scores any 16-bit two-channel WAV, and offline tests prove it on synthetic calls with known answers.
Here is the finished result, bargein.py scoring all 70 recordings from the run described below:
How it works
Stop time is the interval between two edges on two channels: the moment the caller starts talking on the left, and the start of the silence that ends the agent's speech on the right. Three things add up to it. The first is the server's confirmation, which ends when speech_started arrives. The second is the client dropping the audio it has queued but not yet played, which in this repository happens within one 20 ms tick. The third, on real hardware, is the sound card's output latency, which no recording made inside the client can see.
The first part depends on what the server believes the caller can hear. Our client sends no playback reports, so, per the overview, the server estimates how long each reply stays audible from a typical speaking rate. Every number in this guide was measured that way.
The recorder's design follows from that definition. The channels stay separate because on a mixed recording the agent's last words and the caller's first words are one signal. The recording is written on the playback clock because the server streams the reply in 200 ms chunks, so the client holds 0.08 to 0.5 s more audio than it has played (measured on api.studio on 10 November 2026, with this repository's client), and a recording taken when audio arrives would place the agent's last word up to half a second away from where the caller heard it. The caller's intent comes from labels written by the caller script, because a length cutoff cannot do that job: our "mm-hmm" clip lasts 0.51 s, which is longer than our "Wait." clip.
Prerequisites
To follow along, you will need Python 3.13 and uv, which installs the exact dependency versions from the lockfile. You will also need a Dialog-RSN-1 API key: sign in at studio.poly.ai and create one in the API Keys tab.
You do not need a microphone. Every call in this guide is a scripted caller playing WAV files into a simulated room, which keeps echo out of the numbers and makes each run repeatable. To score your own stack in Step 6, you will need a two-channel recording of one of its calls.
Set up the project
Start by cloning the repository and installing the dependencies:
git clone https://github.com/pmckeetx/barge-in-voice-agent.git
cd barge-in-voice-agent
cp .env.example .env
uv sync
The repository already holds the agent (agent.py), its audio (audio.py) and the scripted caller (caller.py); this guide adds recorder.py, record.py and bargein.py, and changes caller.py to feed the recorder and to give each caller line an intent. Open .env and add your key:
# Your Dialog-RSN-1 API key: sign in at https://studio.poly.ai and create one in the API Keys tab.
POLY_API_KEY=
# Optional: the full realtime endpoint. Defaults to wss://api.studio.poly.ai/v1/realtime.
# DIALOG_REALTIME_URL=
POLY_API_KEY is the key you just created. DIALOG_REALTIME_URL stays commented out unless your key belongs to a different PolyAI host. Before you spend any API time, run the offline tests, which need no key and no network:
uv run pytest tests/test_bargein.py tests/test_recorder.py
tests/test_bargein.py .............. [ 77%]
tests/test_recorder.py .... [100%]
============================= 18 passed in 15.86s ==============================
test_recorder.py takes about 15 seconds because it runs two real-time calls through the simulated room against a fake realtime server.
Step 1: Record both sides of the call on one clock
The scripted caller in caller.py runs a loop that ticks every 20 ms. On each tick it pulls the next 20 ms of the agent's audio from the speaker, which is the audio the caller hears at that moment, and builds the next 20 ms of what the agent's microphone picks up. Both halves of the recording already exist inside that loop; the recorder only has to write them side by side.
Here is the recorder, excerpted from recorder.py:
def add(self, caller: bytes, agent: bytes):
"""One tick: what the agent heard from the caller, and what the caller heard from the agent."""
pair = np.column_stack([np.frombuffer(caller, np.int16), np.frombuffer(agent, np.int16)])
if self.step > 1:
pair = pair.reshape(-1, self.step, 2).mean(axis=1).astype(np.int16)
self.frames.extend(pair.tobytes())
def now(self) -> float:
"""Seconds of call recorded so far: the recording's clock."""
return len(self.frames) / 4 / self.rate
def label(self, line: str, intent: str | None):
self.lines.append({"line": line, "onset_s": round(self.now(), 3), "intent": intent})
def mark(self, event: str):
self.events.append({"type": event, "t_s": round(self.now(), 3)})
add() interleaves one tick from each side into stereo frames with np.column_stack, so a sample on the left and the sample beside it on the right happened at the same instant. now() turns the bytes written into seconds (four bytes per stereo frame), which makes the recording its own clock, and label() and mark() note caller lines and server events on it. save(), not shown, writes the WAV and the JSON.
Watch out: Record in the playback loop
Do not tap the agent's channel in theresponse.output_audio.deltahandler. Audio arrives up to half a second ahead of playback, so a recording taken there places the agent's words up to half a second away from when the caller heard them, and the stop time inherits that error.
Then hook the recorder into the room's tick in caller.py. Excerpt from caller.py:
async def run(self):
start, n = time.monotonic(), 0
while True:
played = np.frombuffer(self.speaker.pull(CHUNK), dtype=np.int16)
...
mic = np.frombuffer(said, dtype=np.int16).astype(np.float32) + self.air.popleft() * self.echo_gain
mic = mic.clip(-32768, 32767).astype(np.int16).tobytes()
if self.recorder:
self.recorder.add(mic, played.tobytes()) # both sides, from the same tick
await self.desk.send_audio(mic)
...
def say(self, name: str, intent: str | None = None):
self.desk.log("", f"(caller says {name}.wav)")
if self.recorder:
self.recorder.label(name, intent)
self.voice.extend(read(name))
played is what the speaker gave the room on this tick. mic is the caller's line as the agent receives it; self.air and echo_gain can mix a delayed copy of the agent's audio back in to imitate laptop speakers, but the gain is zero in every run in this guide. Passing both to add() before send_audio() keeps the two channels in step. say() labels each line at the moment it starts to play, which gives the analyzer the caller's intent and a time to match it against.
Step 2: Find speech on each channel
With a stereo file in hand, the next job is to decide, frame by frame, when each channel is speaking. Here is frames_db() in bargein.py, which reads the WAV and measures every 20 ms frame:
def frames_db(path: Path, caller: str = "left") -> tuple[list[float], list[float]]:
"""Read a 16-bit stereo WAV and return the level of every 20 ms frame, in dBFS: (caller, agent)."""
with wave.open(str(path)) as w:
if w.getnchannels() != 2 or w.getsampwidth() != 2:
raise SystemExit(f"{path}: needs a 16-bit two-channel WAV, not {w.getnchannels()} channel(s)")
rate, pcm = w.getframerate(), array("h", w.readframes(w.getnframes()))
if sys.byteorder == "big":
pcm.byteswap()
n = round(rate * FRAME_S)
def levels(channel: array) -> list[float]:
out = []
for i in range(0, len(channel) - n + 1, n):
frame = channel[i:i + n]
rms = math.sqrt(math.sumprod(frame, frame) / n)
out.append(20 * math.log10(rms / 32768) if rms else -120.0)
return out
left, right = levels(pcm[0::2]), levels(pcm[1::2])
return (left, right) if caller == "left" else (right, left)
The function refuses anything but 16-bit stereo and computes each frame's root-mean-square level with math.sumprod, which is fast enough in pure Python to score all 70 calls in about a second. Levels are in decibels relative to full scale (dBFS), with -120 for digital silence, and the caller argument says which side is the caller.
Levels alone are not enough, because speech is full of short silences between words, and an agent that pauses for 200 ms is not an agent that stopped. segments() turns the levels into runs of speech:
def segments(db: list[float], hangover_s: float) -> list[tuple[float, float]]:
"""Where a channel is speaking, as (start_s, end_s), with gaps between words held together.
Speech starts above the on level and stops only after hangover_s below the off level. The end is the start
of that quiet run: the moment the listener last heard it.
"""
floor = max(sorted(db)[len(db) // 10], FLOOR_DB) if db else FLOOR_DB
on, off, hang = floor + ON_DB, floor + OFF_DB, round(hangover_s / FRAME_S)
runs, start, quiet = [], None, 0
for i, level in enumerate(db):
if start is None:
if level >= on:
start, quiet = i, 0
elif level < off:
quiet += 1
if quiet == hang:
runs.append((start * FRAME_S, (i + 1 - hang) * FRAME_S))
start = None
else:
quiet = 0
if start is not None:
runs.append((start * FRAME_S, (len(db) - quiet) * FRAME_S))
return [(round(a, 2), round(b, 2)) for a, b in runs]
The function first estimates the channel's noise floor as its tenth-percentile level, clamped at -70 dBFS (FLOOR_DB) so that a perfectly clean channel does not turn every stray sample into speech. Speech starts on a frame 16 dB above that floor (ON_DB) and only ends once frames fall below a lower level, 10 dB above it (OFF_DB). The gap between the two levels stops the analyzer flickering on and off as a voice hovers around one threshold. Speech ends only after hangover_s of continuous quiet, and the end it reports is the start of that quiet run, which is the moment the listener last heard the voice.
The first hangover I chose for the agent was wrong. These are the defaults at the top of bargein.py:
# How much quiet ends a channel's speech. The agent's has to outlast its longest pause between sentences and stay
# shorter than its quickest answer after a stop. On api.studio.poly.ai's server voice (17 November 2026) the
# longest pause inside a reply was 0.36 s, and the shortest quiet after a cut was 0.72 s.
HANGOVER_S = {"caller": 0.4, "agent": 0.5}
A pause longer than the hangover looks like a stop, and an answer that starts sooner than the hangover merges into the reply it interrupted. I started at 0.8 s, from pauses I had measured in narrated voice-overs. In the live calls, the server's voice never paused for more than 0.36 s inside a reply, but after "Stop." the agent began its answer 0.72 to 0.90 s after the cut, so at 0.8 s half of the "Stop." calls were scored as the agent talking on. At 0.5 s, every call scored correctly. If you score a different voice, measure its pauses first and pass --agent-hangover.
Step 3: Score every overlap
An overlap is a caller burst that starts while the agent is speaking. Each one becomes an Overlap, defined in bargein.py:
@dataclass
class Overlap:
file: str
onset_s: float # when the caller started, on the recording's clock
line: str | None # the label's name for what the caller said
intent: str | None # barge-in, backchannel, background, or None with no label
talked_s: float # how long the agent kept talking after the caller started
result: str # stopped, talked on, or ended (the reply finished as the caller started)
verdict: str | None # stopped, missed, ignored or false stop, once the intent is known
talked_s is the stop time. result is what the audio shows, and verdict is what the audio means once you know what the caller intended. Here are the four constants that set the rules, and score(), which walks the caller's bursts and classifies each one. Excerpt from bargein.py:
LEAD_S = 0.2 # the agent must have been talking this long when the caller starts, or it's the agent cutting in
REACT_S = 0.15 # no detector reacts faster: an agent that goes quiet sooner was finishing anyway
MISS_S = 2.0 # still talking this long after the caller finished: it didn't stop
STOP = {"barge-in", "turn"} # the intents that should stop the agent; backchannel and background should not
...
def score(file: str, caller: list[tuple[float, float]], agent: list[tuple[float, float]], labels: list[dict]) -> list[Overlap]:
"""One Overlap for every time the caller started while the agent was already talking."""
out, unused = [], sorted(labels, key=lambda label: label["onset_s"])
for onset, end in caller:
talking = next(((a, b) for a, b in agent if a + LEAD_S <= onset < b), None)
if not talking:
continue
talked = round(talking[1] - onset, 2)
result = "ended" if talked < REACT_S else "stopped" if talking[1] <= end + MISS_S else "talked on"
# A label belongs to the first caller burst that starts at or up to a second after it.
label = next((x for x in unused if x["onset_s"] - FRAME_S <= onset <= x["onset_s"] + 1.0), None)
if label:
unused.remove(label)
intent = label and label.get("intent")
verdict = None
if intent and result != "ended":
should_stop, stopped = intent in STOP, result == "stopped"
verdict = {(True, True): "stopped", (True, False): "missed",
(False, False): "ignored", (False, True): "false stop"}[should_stop, stopped]
out.append(Overlap(file, onset, label and label.get("line"), intent, talked, result, verdict))
return out
A burst only counts if the agent had already been talking for LEAD_S (0.2 s) when it started; otherwise the agent started over the caller, which is a different problem. The agent's segment end, minus the caller's onset, is the stop time.
A stop faster than REACT_S (0.15 s) is reported as ended, because no voice activity detector reacts that quickly: the reply was finishing on its own. An agent still talking MISS_S (2 s) after the caller's burst ends has talked on. The miss window counts from the end of the burst because a long sound, such as our 4.6 s TV clip, can stop the agent while it is still playing. Each label is matched to the first burst that starts within a second after it, and the pair of "should it stop?" and "did it stop?" becomes the verdict.
Note: Audio alone cannot tell a stop from an ending
When a caller starts talking just as a reply finishes, the recording looks exactly like a fast stop, which is how a log ends up reporting a barge-in handled in 0 ms.REACT_Sskips the obvious cases, and the labels supply the intent, but the analyzer has no way to know what the agent had left to say.
Step 4: Test the analyzer on synthetic calls
Before the analyzer reads a real call, prove it on calls where you already know the answer. The tests build stereo WAVs from noise with a helper in tests/test_bargein.py:
def synth(path, agent, caller, gaps=0.15, agent_left=False, labels=None):
"""A stereo WAV: the agent talks in 0.5 s words with gaps between them over each (start, end) in agent, and the
caller makes a sound over each (start, end) in caller. Noise, not speech: the analyzer only hears energy."""
end = max(b for _, b in agent + caller) + 2.0
rng = np.random.default_rng(0)
left, right = np.zeros(int(end * RATE)), np.zeros(int(end * RATE))
for a, b in caller:
left[int(a * RATE):int(b * RATE)] = rng.normal(0, 3000, int(b * RATE) - int(a * RATE))
for a, b in agent:
t = a
while t < b:
word_end = min(t + 0.5, b)
right[int(t * RATE):int(word_end * RATE)] = rng.normal(0, 3000, int(word_end * RATE) - int(t * RATE))
t = word_end + gaps
channels = (right, left) if agent_left else (left, right)
with wave.open(str(path), "wb") as w:
w.setnchannels(2), w.setsampwidth(2), w.setframerate(RATE)
w.writeframes(np.column_stack(channels).astype(np.int16).tobytes())
if labels:
path.with_suffix(".json").write_text(json.dumps({"lines": labels}))
return path
synth() fills the agent's channel with 0.5 s bursts of noise separated by gaps of silence, which stand in for words, and the caller's channel with one burst per (start, end) pair; noise works because the analyzer only measures energy. The first test plants a stop 0.78 s after the caller starts and checks that the analyzer finds it within one frame:
def test_a_known_stop_time_comes_back_within_a_frame(tmp_path):
# The caller cuts in at 3.0 s and the agent stops 0.78 s later.
wav = synth(tmp_path / "call.wav", agent=[(1.0, 3.78)], caller=[(3.0, 4.5)],
labels=[{"line": "parking", "onset_s": 3.0, "intent": "barge-in"}])
[o] = analyse(wav)
assert o.onset_s == pytest.approx(3.0, abs=0.02)
assert o.talked_s == pytest.approx(0.78, abs=0.02)
assert (o.result, o.verdict) == ("stopped", "stopped")
The other thirteen tests cover the cases that trip up a naive analyzer, from gaps between words to a reply that ends as the caller starts, and a labels file that matches nothing. Run them with -v to see each name:
uv run pytest tests/test_bargein.py -v
tests/test_bargein.py::test_a_known_stop_time_comes_back_within_a_frame PASSED [ 7%]
tests/test_bargein.py::test_gaps_between_words_are_not_stops PASSED [ 14%]
tests/test_bargein.py::test_a_quick_answer_after_a_stop_is_not_more_of_the_reply PASSED [ 21%]
tests/test_bargein.py::test_a_stop_during_a_long_sound_counts_from_when_it_ends PASSED [ 28%]
...
tests/test_bargein.py::test_the_cli_warns_about_a_label_that_matched_nothing PASSED [100%]
============================== 14 passed in 0.12s ==============================
I added the quick-answer and long-sound tests after the live runs exposed both mistakes.
Step 5: Script the caller's lines and their intent
The measurement needs callers who interrupt in different ways. Each script in caller.py is a list of steps, and each step now carries the caller's intent:
# Each step: the line the caller says, when, and what they mean by it. None waits for the agent to finish its reply
# first; a number talks over that reply that many seconds after it starts. A barge-in should stop the agent; a
# backchannel ("mm-hmm") and background noise (a TV in the room) should not.
SCRIPTS = {
"check-in": [("check-in", None, "turn"), ("parking", 4.0, "barge-in"), ("go-on", None, "turn")],
"booking": [("booking", None, "turn"), ("never-mind", 0.3, "barge-in")],
"stop": [("check-in", None, "turn"), ("stop", 4.0, "barge-in")],
"wait": [("check-in", None, "turn"), ("wait", 4.0, "barge-in")],
"mm-hmm": [("check-in", None, "turn"), ("mm-hmm", 1.2, "backchannel")],
"tv": [("check-in", None, "turn"), ("tv", 1.2, "background")],
}
Every interrupting script asks the same opening question, so the agent gives the same long rundown each time. parking is a full question 4 s into it, stop and wait are single words at the same point (made with macOS say; see make_samples.sh), mm-hmm is a 0.51 s backchannel 1.2 s in, and tv is 4.6 s of a newsreader with room noise, about 10 dB under the caller's level.
record.py runs these scripts with the recorder on. It also wraps the front desk so that every speech_started is marked on the recording's clock:
def recorded(desk: type[FrontDesk], recorder: StereoRecorder) -> type[FrontDesk]:
"""The same desk, noting on the recording's clock when each speech_started arrives."""
class Recorded(desk):
async def handle(self, event: dict):
if event["type"] == "input_audio_buffer.speech_started":
recorder.mark("speech_started")
await super().handle(event)
return Recorded
recorded() returns a subclass whose handle() marks the event and then runs the normal handling, where the client cuts playback, so record.py can compare the recorded stop with the event that caused it.
Run it
Record 20 calls of the parking script against the real API:
uv run record.py --script check-in --runs 20
--script picks one of the scripts above and --runs repeats it. Each call takes about 40 seconds, because the room plays in real time, so the full set in this section (70 calls) keeps a session open for about 47 minutes of your API usage. Each call prints its transcript, the recording's path and the comparison:
(caller says check-in.wav)
Caller: Hi, I'm staying with you this weekend. Can you tell me everything I need to know about checking in?
(caller says parking.wav)
(barge-in: stopped the reply after 4.8 s)
Agent: Check-in is from 3 pm. Please bring your photo ID and the card you booked- [cut off]
Caller: Sorry, wait, is there parking?
Agent: Yes, parking is valet only, and it's 40 dollars a night. Check-in is from 3 pm, bring photo ID and the card you booked with. There's a 100 dollar deposit.
(caller says go-on.wav)
Caller: Okay, thanks. Go on.
Agent: Breakfast is served from 7 to 10 in the Atrium. The pool is on floor 2, and the wifi is HarborGuest with no password.
runs/check-in-02.wav
parking (barge-in): stopped after 0.78 s; speech_started arrived after 0.78 s (recording minus events +0.00 s)
...
The recording shows the agent stopping 0.78 s after the caller started, and speech_started arrived 0.78 s after the same moment, so "recording minus events" is zero. Next, record the other scripts and one batch on a phone line:
uv run record.py --script stop --runs 10
uv run record.py --script wait --runs 10
uv run record.py --script mm-hmm --runs 10
uv run record.py --script tv --runs 10
uv run record.py --script check-in --format pcmu --runs 10
--format pcmu puts the caller on a phone line: the client sends 8 kHz G.711 μ-law instead of 24 kHz PCM, and the recorder writes both channels at 8 kHz, which is what a carrier's two-channel recording looks like. The reply audio still comes back as 24 kHz PCM, the only output format the API reference lists (read 17 November 2026), so the recorder averages every three samples into one on both channels. The caller's channel is the room's audio before μ-law encoding, not the encoded bytes the server received. Then score everything at once:
uv run bargein.py runs/*.wav
recordings line intent calls stopped median range false stops missed
check-in parking barge-in 20 20 0.78 s 0.76 to 0.86 s 0 0
check-in-pcmu parking barge-in 10 10 0.90 s 0.86 to 0.92 s 0 0
mm-hmm mm-hmm backchannel 10 0 - - 0 0
stop stop barge-in 10 10 0.78 s 0.78 to 0.84 s 0 0
tv tv background 10 10 1.87 s 0.72 to 2.06 s 10 0
wait wait barge-in 10 10 0.77 s 0.74 to 0.80 s 0 0
All 50 barge-ins stopped the agent, and none was missed. The parking question stopped it a median 0.78 s after the caller started, and the single words were just as fast: "Stop." at 0.78 s and "Wait." at 0.77 s, even though both clips end before the server's confirmation does. On the phone line the same question took a median 0.90 s, about 0.12 s longer. The recording agreed with the server's events in every one of the 60 stops: the difference was 0.00 s in 58, and in the other two the recording showed the stop 0.02 and 0.04 s earlier, because the cut came as a word was trailing off and the agent's level had already fallen below the off level.
The two scripts that should not have stopped the agent split. The "mm-hmm" was talked through in 10 of 10 calls, with no speech_started at all. The TV newsreader stopped the agent in 10 of 10, 1.84 to 2.06 s in for eight calls and 0.72 to 0.74 s in for two, and the server transcribed the newsreader as the caller every time ("Officials say the bridge will stay closed through the weekend..."). In every TV call the agent went back to its rundown afterwards, eight times mid-sentence and twice from the top, after 3.8 to 5.4 s of silence while the server listened to the news.
After "Stop.", the agent returned to the rundown in 6 of 10 calls ("Sure. You'll need to bring your photo ID and the card you booked with...") and paused to ask what else the caller needed in the other 4. After "Wait.", it waited every time. bargein.py measures whether and how fast the agent stopped; what it says next is a question for the transcript.
Step 6: Score a recording from your own stack
bargein.py reads any 16-bit two-channel WAV, so you can point it at a recording from another stack. The only things it needs to know are which side is the caller and, if you want verdicts, what the caller meant:
uv run bargein.py call.wav --caller right
uv run bargein.py call.wav --labels call.csv
--caller right is for recordings with the agent on the left; without it, such a file prints No overlaps: the caller never started while the agent was talking. If the agent is on the left, pass --caller right. Stacks disagree on the layout: Pipecat's AudioBufferProcessor puts the user on the left, as our recorder does, while LiveKit's DUAL_CHANNEL_AGENT egress puts the agent on the left. A Twilio recording comes back in two channels when you request .wav?RequestedChannels=2, but at the time of writing (November 2026) Twilio's recording docs do not say which leg is on which side, so listen to the first few seconds before you score it. Run the file once without labels first. Each per-call line starts with the time the caller's burst began, and those are the onsets to label. --labels then takes a CSV you write by hand, with one row per caller line, in place of the JSON the recorder writes:
onset_s,intent
2.2,backchannel
5.0,barge-in
onset_s is roughly when the caller starts (a label matches the first burst up to a second after it), and intent is barge-in, turn, backchannel or background. A label that matches no burst prints a warning that names its onset, so a typo does not pass silently. Without labels, you still get every stop time and result, but no verdict, and --json prints one object per overlap. I have only run the script on this repository's recordings, so check its results by ear on the first few calls from another stack.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Every caller burst reads "talked on", or there are no overlaps at all | The channels are reversed | Pass --caller right
|
call.wav: needs a 16-bit two-channel WAV, not 1 channel(s) |
A mixed (mono) recording | Record the two sides separately; a mixed file cannot be scored |
A backchannel scores as a false stop, with no speech_started beside it in record.py's output |
The agent's hangover is shorter than one of its pauses | Raise --agent-hangover
|
| A barge-in that stopped scores as "talked on" | The agent answered faster than the hangover, so the answer merged into the reply | Lower --agent-hangover
|
The server refused the connection (HTTP 401). Check POLY_API_KEY, and set DIALOG_REALTIME_URL if your key is for a different host than wss://api.studio.poly.ai/v1/realtime. |
Wrong key, or a POLY_API_KEY in your shell is overriding .env
|
echo $POLY_API_KEY, then unset POLY_API_KEY
|
POLY_API_KEY isn't set. Copy .env.example to .env and add your key from studio.poly.ai. |
No key in .env or the shell |
Add your key to .env
|
Before you ship
These recordings are taken inside the client, at the simulated sound card, and that flatters the agent. A real sound card adds its output latency on top (the latency attribute of a sounddevice stream reports it for your device), and on a phone line the carrier's jitter buffer adds more. Measure as close to the caller as you can: on a phone line, that means the carrier's own two-channel recording.
A scripted caller is also not a person. Real interruptions often start soft and build, people overlap in ways a WAV file does not, and a cough is a caller burst to an energy detector. Use the scripted runs to compare one build against the next, and score a sample of real calls before you trust the numbers.
Background speech about 10 dB under the caller stopped the agent in 10 of 10 calls here, so if your callers ring from kitchens and cars, score some calls with the background they actually have.
Recording real callers needs their consent where the law requires it, and you should tell callers at the start that they are speaking with an automated AI system.
Where to go next
You now have a recorder that writes each call as the caller heard it and an analyzer that turns any two-channel recording into stop times, false stops and missed barge-ins.
From here, read the turn-taking section of the Dialog-RSN-1 overview to see how the server confirms speech while a reply is playing. Sending playback reports makes the server's audibility check follow what the caller actually heard, and the scripts in this guide will measure whether that changes your numbers. The limits page lists the audio formats a phone-line recording can use. To measure stop time on real calls, run bargein.py on your carrier's two-channel recordings with a hand-written labels file. The complete code is in pmckeetx/barge-in-voice-agent.
![A terminal running uv run bargein.py runs/*.wav: per-call lines such as "parking (barge-in) stopped after 0.80 s [stopped]" and "tv (background) stopped after 1.86 s [false stop]", then a table with one row per script: parking stopped 20 of 20 with a median of 0.78 s, the phone-line run 10 of 10 at 0.90 s, Stop and Wait 10 of 10 at 0.78 and 0.77 s, mm-hmm 0 of 10, and the TV 10 false stops](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fku2goqr7oyqhm7bvnwnz.png)

Top comments (0)