DEV Community: Peter McKee

Start and end a voice agent call in Python without clipping a word

Peter McKee — Thu, 12 Nov 2026 22:07:53 +0000

Every phone call has two fragile boundaries: the pickup and the hang-up. In real-time voice agents, these edges are where conversations lose words. A caller starts speaking the millisecond they hear the line connect, often well before your client application has finished handshaking with the upstream speech API. Minutes later, as the conversation wraps up, the agent begins saying goodbye, only for the client to assume the exchange is over and abruptly sever the socket mid-sentence.

To see how words get lost at the edges, look at this scripted call from the companion repository, pmckeetx/barge-in-voice-agent, connecting to PolyAI's Dialog-RSN-1 API. Here, an early client implementation handles a routine hotel booking inquiry, stumbling at both ends:

        (caller says pickup.wav)
        (session ready: sent 0.00 s of held audio, now live)
Caller: Yes, I'm calling about my booking for Friday. The surname's Okafor.
        (look_up_booking returned for Okafor)
Agent:  Let me pull up your booking, the system can take a moment to load.
Agent:  I have your booking here, Ms. Okafor. You're in room 214 for two nights, checking in on Friday. Is there anything else you need to know?
        (caller says thats-all.wav)
Caller: Great, that's everything I needed. Thanks very much.
        (end_call: Caller confirmed they had everything they needed and said goodbye.)
        (hangs up)
first line: lost 1 of 12 words at the start (session ready 0.26 s after pickup); goodbye: played 1.0 s of "You're welcome, Ms. Okafor. Have a wonderful stay with us. Goodbye.", then hung up ...

Notice the damage on both sides of the exchange. The agent heard "Yes, I'm calling about..." because the caller's initial "Hi" landed during the quarter-second window before the session was ready, and the client discarded it. At the end of the call, the agent composed a polite farewell, but the client slammed the line down after a static one-second sleep while the server was still streaming audio chunks over the wire. There isn't even an Agent: transcript line for the goodbye because the call ended before the reply did.

Neither failure is the model's fault. The speech recognizer is working fine, and the language model knows what to say. The breakdown happens entirely in client-side lifecycle management: the client assumes the call hasn't started until the network is ready, and assumes the call is finished before playback concludes.

To fix this, we can equip our front-desk agent (the Harborview Inn from the repository) with two changes: a circular audio buffer that starts capturing frames the moment the sound card activates, flushing them the instant the session is configured; and an event-driven hang-up routine that tracks the server's response lifecycle and drains the playback buffer before tearing down the connection.

When benchmarked against the live API on 10 November 2026 across 10 runs each, the naive client clipped the caller's first word on every single call and severed every goodbye. The buffered, event-driven client captured every word at both ends.

Code: pmckeetx/barge-in-voice-agent (call.py and pickup.py)

What you'll build

You will build call.py, a voice agent client that listens from the exact instant the line opens and refuses to hang up until its farewell has finished playing through the speakers. Under the hood, it:

Holds up to 5 seconds of caller audio in memory while waiting for the session handshake.
Exposes an end_call tool configured with a spoken goodbye argument.
Waits up to 8 seconds for that goodbye to fully stream and play.
Cancels the hang-up and keeps the line open if the caller talks over the goodbye.

To verify these behaviors reliably, you'll also use pickup.py, a test harness that feeds scripted caller audio into the pipeline from the very first frame without requiring a live microphone. Here are the old client and the new one on the same scripted call:

How it works

To understand why calls clip, we have to look closely at the mechanics of the session lifecycle.

A call begins with an initialization dance across the network. The client opens a WebSocket, the server answers with session.created once accepted, the client replies with session.update specifying tools and audio configurations, and the server confirms with session.updated. Until that final confirmation arrives, the session runs on its defaults (which on 10 November 2026 meant raw 24 kHz PCM input, per the quickstart).

In an isolated connection probe, that handshake is fast: session.updated arrived a median 234 ms after the connection began (measured over 20 runs on api.studio from the US Midwest). In the 20 scripted calls further down, sessions were ready 0.23 to 2.83 seconds after pickup (median 0.66 s), for reasons I didn't pin down. A caller saying "Hi, yes..." gets those first words out inside that window, and routing calls over mobile networks or SIP trunks only widens the gap.

The end of the call presents the inverse problem. When the model decides the interaction is complete, it triggers end_call. Rather than sending plain text, the server speaks the tool's goodbye argument directly. The tool invocation and the outgoing audio stream arrive simultaneously: in 10 test calls, the first audio chunk arrived a median 46 ms before the client received the end_call tool event.

The server then streams the farewell audio in 200 ms chunks at a natural speaking cadence, keeping the client's local buffer between 100 ms and 500 ms ahead of real-time playback. The final response.done event only arrives after the last chunk is dispatched (a median 436 ms later). Because the client only ever holds about half a second of audio at a time, a naive asyncio.sleep(1.0) hangs up a second into a goodbye that runs about 4 seconds: by then, only 1.35 to 1.59 s of it had reached the client in the probe calls.

Reliable orchestration comes down to two explicit checkpoints: wait for session.updated before routing live frames upstream, and wait for response.done followed by speaker drainage before closing the socket.

Our implementation builds directly on FrontDesk in agent.py, a base class providing booking lookups, a core listen() event loop, logging utilities, and barge-in handling. Call in call.py wraps this foundation to reinforce both edges.

Prerequisites

To run the project, ensure you have Python 3.13 and uv installed. You will also need a Dialog-RSN-1 API key from the API Keys tab in studio.poly.ai.

The scripted test suite requires no hardware. If you plan to test call.py interactively, use headphones alongside your microphone to prevent local echo from feeding back into the agent.

Set up the project

Clone the companion repository and enter the project directory:

git clone https://github.com/pmckeetx/barge-in-voice-agent.git
cd barge-in-voice-agent

Initialize your environment variables from the template:

cp .env.example .env

# Your Dialog-RSN-1 API key: sign in at https://studio.poly.ai and create one in the API Keys tab.
POLY_API_KEY=

# Optional: the full realtime endpoint. Defaults to wss://api.studio.poly.ai/v1/realtime.
# DIALOG_REALTIME_URL=

Add your API key to POLY_API_KEY. Leave DIALOG_REALTIME_URL commented out unless your key is for a different PolyAI host. Next, install dependencies and execute the offline mock suite:

uv sync
uv run pytest

...
tests/test_agent.py ......                                               [ 28%]
tests/test_audio.py ....                                                 [ 47%]
tests/test_call.py .........                                             [ 90%]
tests/test_live.py ss                                                    [100%]

======================== 19 passed, 2 skipped in 12.77s ========================

The offline tests use a simulated event server. To run the two skipped integration tests against live infrastructure, run POLYAI_LIVE=1 uv run pytest -m live. Note that this executes two real calls lasting around 70 seconds and consumes concurrent session quota.

Step 1: Start listening at pickup

The primary mistake in naive voice clients is tying hardware capture to network readiness. The sound card is local and ready immediately; the remote session takes multiple network round trips to establish.

To prevent dropping words during that window, instantiate Call before opening the WebSocket, capturing all inbound audio into an in-memory ring buffer immediately:

FRAME_S = 0.02  # the sound card and the scripted caller both hand over 20 ms frames
BUFFER_CAP_S = 5.0  # the most of the caller's audio we hold while the session gets ready
HANG_UP_CAP_S = 8.0  # the longest we wait for a goodbye to finish before hanging up anyway
...
class Call(FrontDesk):
    ...
    def __init__(self, ws, speaker: Speaker, echo: EchoCanceller | None = None):
        super().__init__(ws, speaker, echo)
        self.held: deque[bytes] = deque(maxlen=round(BUFFER_CAP_S / FRAME_S))  # past the cap, the oldest frame goes
        self.live = False  # True once the held audio has gone out and frames can go straight to the server
        self.goodbye: dict | None = None  # the end_call call, while its goodbye is being spoken
        self.goodbye_spoken = asyncio.Event()
        self.hanging_up: asyncio.Task | None = None
        self.hung_up = asyncio.Event()

    async def send_audio(self, pcm: bytes):
        if self.hung_up.is_set():
            return
        if self.live:
            await super().send_audio(pcm)
        else:
            self.held.append(pcm)

The held queue stores 20 ms frames with a maximum size of 250 entries (5 seconds total). If the connection stalls for 10 seconds, the oldest frames drop off the back, so the agent hears the caller's last five seconds, not the stalest. Inside send_audio(), frames accumulate in held until self.live flips to True. Once live, frames route straight to FrontDesk.send_audio(), which encodes them into input_audio_buffer.append events. Once the call is marked as hung up, any trailing mic input is dropped safely.

With the buffer ready, we invert our startup sequence: activate the local microphone stream first, and initiate the remote connection second. Here's call() in call.py:

async def call(speakers: bool):
    """A live call on your microphone. The microphone opens first, then the connection."""
    speaker = Speaker()
    echo = EchoCanceller() if speakers else None
    if echo:
        speaker.on_played = echo.played
    desk = Call(None, speaker, echo)
    with SoundCard(speaker) as card:  # pickup: from here on, everything the caller says is kept

        async def stream_mic():
            while True:
                await desk.send_audio(await card.queue.get())

        mic = asyncio.create_task(stream_mic())
        print("Line open. Start talking whenever you like; say goodbye when you're done.")
        async with connect() as ws:
            desk.ws = ws
            await desk.start()
            await desk.listen()  # returns when the hang-up closes the socket
        mic.cancel()

Opening SoundCard inside a context manager begins capturing audio instantly. The background stream_mic task pumps frames into desk.send_audio() before connect() is even dispatched. When the connection establishes, desk.ws is assigned, start() negotiates the session, and listen() takes over until teardown.

Step 2: Send the held audio once the session is configured

Buffering is only half the battle; how and when you flush that buffer determines whether the agent understands the speech.

Warning: Any audio forwarded before session.update takes effect is processed under the session's default audio format (24 kHz PCM). In testing on 10 November 2026, sending 8 kHz μ-law telephony audio prior to session.updated failed silently in 5 out of 5 runs: no speech turn triggered, no transcript was generated, and no server error was raised.

Here is start() handling the configuration handshake and flushing the buffered speech:

    async def start(self):
        await self.ws.recv()  # session.created: the session is open, but still on its default settings
        await self.send({
            "type": "session.update",
            "session": {
                "type": "realtime",
                "instructions": CALL_INSTRUCTIONS,
                "tools": TOOLS + [END_CALL],
                "audio": {"input": {"format": {"type": "audio/pcm", "rate": 24000}, "turn_detection": {"type": "server_vad"}}},
            },
        })
        updated = json.loads(await self.ws.recv())
        if updated["type"] != "session.updated":
            raise RuntimeError(updated.get("error", {}).get("message", f"unexpected {updated['type']}"))
        held = len(self.held)
        while self.held:  # frames that arrive during the flush join the end of the queue, so order holds
            await super().send_audio(self.held.popleft())
        self.live = True
        self.log("", f"(session ready: sent {held * FRAME_S:.2f} s of held audio, now live)")

The method waits for session.created, dispatches session.update with instructions, schemas, and codecs, and explicitly waits for session.updated. If the server rejects the configuration, it yields an error rather than an updated event; checking for this explicitly prevents pumping audio into a misconfigured session.

Waiting for session.updated costs about 50 ms, and with the 24 kHz PCM this front desk sends, it saves no words: 24 kHz audio sent before session.update was heard in full in 5 of 5 runs, and so was μ-law sent between session.update and session.updated. The wait is there for a refused update, and for the day you switch to phone audio.

When releasing the buffer, frames flush sequentially as fast as the socket can transmit them. You might worry that dumping multiple seconds of speech into the server in one burst would break turn detection or voice activity detection (VAD). In practice, it didn't. In 30 runs on 10 November 2026, with the line sent at real time or with 1.5 s or 3 s of it in one burst, all 12 words were transcribed every time, each as a single turn. The server catches up, and audio_end_ms, where turn detection puts the end of the caller's speech, came out the same as for audio sent at real time.

One curious quirk emerged during testing: the caller's surname ("Okafor") was transcribed as "O'For" in 6 out of 10 scripted calls with held audio and in none of the 10 without it, while the probes spelled it right in all 30 bursts. I can't explain it yet. Every other word came through, but if names matter on your calls, test your own audio.

Notice also how frame ordering is preserved during the flush: new microphone frames entering send_audio() append to the tail of held while the loop runs. Because there is no await between draining the final frame and setting self.live = True, no incoming live frame can bypass the queue.

Step 3: Give the agent an end_call tool with a spoken goodbye

To end the call naturally, the agent must be able to say goodbye and trigger the termination signal within the same conversational turn. Here's the tool, from call.py:

END_CALL = {
    "type": "function",
    "name": "end_call",
    "description": "Hang up. Use it when the caller has everything they need and is saying goodbye.",
    "parameters": {
        "type": "object",
        "properties": {
            "goodbye": {
                "type": "string",
                "poly_spoken": True,  # the server speaks this argument, in the same response as the call
                "description": "The last thing you say, as you hang up: thank the caller by name, wish them a good stay, and say goodbye.",
            },
            "reason": {"type": "string", "description": "Why the call is ending, for the log."},
        },
        "required": ["goodbye", "reason"],
    },
}
CALL_INSTRUCTIONS = INSTRUCTIONS + "\nWhen the caller has everything they need, end the call with end_call."

The poly_spoken: True property on goodbye makes this work. It tells the server to say that argument aloud while the tool runs, in the same response as the call.

Attempting this via prompt engineering (such as instructing the model to "say goodbye, then invoke the tool") fails (probed on 23 October 2026): the model says the line, finishes its turn, and the tool call never comes. Making goodbye a required parameter is equally important; per the Dialog-RSN-1 documentation, the model often skips an optional poly_spoken argument.

In 10 probe calls on 10 November 2026, the model called end_call with a spoken goodbye every time, and the goodbye and the call came back in one response. In the scripted calls further down, the goodbyes ran 3.6 to 5.1 seconds. Because the session closes immediately afterward, the client does not need to send a function_call_output back to the model under normal conditions.

Step 4: Hang up when the goodbye has played

Replacing a brittle asyncio.sleep() requires listening for the exact server events that denote completion. In call.py, handle() monitors this progression:

    async def handle(self, event: dict):
        match event["type"]:
            case "response.function_call_arguments.done" if event["name"] == "end_call":
                self.goodbye = event
                self.log("", f"(end_call: {json.loads(event['arguments']).get('reason', '')})")
                self.hanging_up = asyncio.create_task(self.hang_up_after_goodbye())
                return  # not a booking lookup: FrontDesk's tool runner never sees it
            case "response.done" if self.goodbye and event["response"]["id"] == self.goodbye["response_id"]:
                if event["response"]["status"] == "completed":
                    self.goodbye_spoken.set()  # the server has sent the whole goodbye
                else:
                    await self.keep_line_open()
        await super().handle(event)

When end_call arrives, handle() stores the event details and spawns the teardown task. It intercepts the event before the default tool runner attempts to process it as a database lookup. When the matching response.done event arrives with a completed status, it sets the goodbye_spoken event. Any non-completed status indicates an interruption, transitioning us into step 5.

The teardown task then coordinates the physical exit:

    async def hang_up_after_goodbye(self):
        try:
            async with asyncio.timeout(HANG_UP_CAP_S):
                await self.goodbye_spoken.wait()  # every word of the goodbye has arrived
                await self.speaker.drain()  # and the speaker has played the last of it
        except TimeoutError:
            self.log("", f"(the goodbye didn't finish within {HANG_UP_CAP_S:.0f} s: hanging up anyway)")
        await self.hang_up()
    ...
    async def hang_up(self):
        self.hung_up.set()
        self.log("", "(hangs up)")
        await self.ws.close()

This ensures the client waits for the entire goodbye to be received (goodbye_spoken.wait()), and then waits for the local output hardware to play the remaining audio buffer (self.speaker.drain()).

Wrapping the shutdown in an 8-second asyncio.timeout prevents deadlocks if a network packet drops or the server hangs. The Dialog-RSN-1 API bills connected time until the WebSocket closes, per the limits and pricing page, so the cap also bounds what a stuck call costs.

Contrast this approach with the naive implementation:

    async def hang_up_after_goodbye(self):
        await asyncio.sleep(1.0)
        await self.hang_up()

Closing the connection after an arbitrary 1-second pause truncates the goodbye while most of it is still on its way. In the scripted runs, the caller heard a median 1.0 s of each goodbye.

One alternative you might consider is sleeping briefly and then checking whether the speaker buffer is empty. When I replayed the timing of the 10 probe goodbyes, this played every one in full, because the server never let the local queue run dry. But it waits on the wrong thing: an empty speaker queue only indicates that the client has played whatever it has received so far. If a network hiccup introduces a 100 ms transmission gap, an empty-buffer check will mistakenly conclude the goodbye is over and hang up mid-word.

Step 5: Keep the line open if the caller talks over the goodbye

Real callers don't always respect the end of a script. A customer will often chime in with "Wait, one more question!" right as the agent begins saying goodbye.

Whenever a user speaks over model audio, the server issues a cancellation. In 5 out of 5 tests where the caller interrupted half a second into the farewell, the server marked response.done as cancelled about 0.8 s after the caller started talking.

Rather than terminating the call, Call catches this event and recovers:

    async def keep_line_open(self):
        """The caller talked over the goodbye, so the server cancelled it. Stay on the line and tell the model."""
        self.hanging_up.cancel()
        call, self.goodbye = self.goodbye, None
        self.log("", "(the caller talked over the goodbye: staying on the line)")
        await self.send({
            "type": "conversation.item.create",
            "item": {"type": "function_call_output", "call_id": call["call_id"], "output": json.dumps(
                {"hung_up": False, "why": "The caller started talking over your goodbye, so the call is still open."})},
        })
        # No response.create: the caller's new turn gets its own reply.

The method cancels the pending hang-up task, clears the goodbye state, and posts a function_call_output explaining to the model that the call remains open because the caller interrupted. We avoid dispatching response.create here because the caller's new turn is already being processed by server VAD and will trigger its own response.

Here is the resulting flow during an interruption, run via uv run pickup.py --talk-over:

...
Caller: Great, that's everything I needed. Thanks very much.
        (end_call: Caller confirmed they had everything they needed and said goodbye.)
        (caller says one-more-thing.wav)
        (barge-in: stopped the reply after 1.3 s)
        (the caller talked over the goodbye: staying on the line)
Agent:  Thank you, Ms-  [cut off]
Caller: Oh wait, one more thing. Is breakfast included?
Agent:  I don't have information about that. Breakfast is served from 7 to 10 in the Atrium. Is there anything else you need?
        (caller says thats-all.wav)
Caller: Great, that's everything I needed. Thanks very much.
        (end_call: Caller confirmed they had everything they needed and said goodbye.)
Agent:  Thank you, Ms. Okafor. We hope you have a wonderful stay. Goodbye.
        (hangs up)
...

The [cut off] line tracks the actual audio heard by the caller, and it's also what the server keeps in conversation history, thanks to the front desk's barge-in handling. The agent answers the follow-up question and only closes the call once the customer says goodbye a second time.

Step 6: Test both edges on a server that answers late

Relying on unit tests against instantaneous local mocks can blind you to race conditions. To stress-test these timing edges, the repository includes a configurable mock server (FakeRealtime) equipped with ready_after (to simulate network delays during session setup) and pace (to stream replies the way api.studio did on 10 November 2026: 300 ms up front, then real time, with response.done 0.4 s after the last chunk). It can also pause a reply partway through. Excerpt from the reply loop in tests/fake_realtime.py:

                if n == 0 and fc:  # the goodbye's first audio arrives just ahead of the call itself
                    await send({"type": "response.function_call_arguments.done", "response_id": rid, "item_id": fc["id"],
                                "call_id": fc["call_id"], "name": "end_call", "arguments": fc["arguments"]})
                if self.stall and (n + 1) * 100 == self.stall[0]:
                    await asyncio.sleep(self.stall[1])  # the stream pauses, as on a slow network
                if not self.pace:
                    await asyncio.sleep(0.005)
                elif n >= LEAD_MS // 100 - 1:
                    await asyncio.sleep(0.1)

With a pause in the stream simulated, we can write an explicit test comparing Call against DrainCall (the naive implementation that hangs up as soon as its audio queue empties) in the presence of a 500 ms stream stall:

@pytest.mark.parametrize("client", [Call, DrainCall])
async def test_a_pause_in_the_stream_isnt_the_end_of_the_goodbye(monkeypatch, client):
    from fake_realtime import FakeRealtime

    async with FakeRealtime([("goodbye", GOODBYE, 3000)], pace=True, stall=(1500, 0.5)) as fake:
        speaker = Speaker()
        desk, listener = await open_call(fake, monkeypatch, speaker, client)
        card = asyncio.create_task(play(speaker))
        await caller_says(desk)
        await asyncio.wait_for(desk.hung_up.wait(), 6)
        card.cancel()

        played = speaker.played["item_1"] // BYTES_PER_MS
        if client is Call:
            assert played == 3000  # it waited for response.done, so the pause didn't matter
        else:
            assert played < 2000  # the speaker ran dry during the pause, and that looked like the end

In this scenario, Call waits for response.done and plays the full 3,000 ms of audio despite the mid-stream stall. DrainCall encounters the buffer starvation at 1,500 ms, assumes playback has concluded, and terminates early.

You can run the full suite covering the buffer ring, timeouts, and barge-in recovery locally:

uv run pytest tests/test_call.py -v

tests/test_call.py::test_caller_audio_from_pickup_reaches_the_server_in_order[Call] PASSED [ 11%]
tests/test_call.py::test_caller_audio_from_pickup_reaches_the_server_in_order[NaiveCall] PASSED [ 22%]
tests/test_call.py::test_buffer_keeps_the_last_five_seconds PASSED       [ 33%]
tests/test_call.py::test_hang_up_waits_until_the_goodbye_has_played PASSED [ 44%]
tests/test_call.py::test_a_fixed_sleep_cuts_the_goodbye PASSED           [ 55%]
tests/test_call.py::test_a_pause_in_the_stream_isnt_the_end_of_the_goodbye[Call] PASSED [ 66%]
tests/test_call.py::test_a_pause_in_the_stream_isnt_the_end_of_the_goodbye[DrainCall] PASSED [ 77%]
tests/test_call.py::test_hang_up_gives_up_when_the_goodbye_never_finishes PASSED [ 88%]
tests/test_call.py::test_talking_over_the_goodbye_keeps_the_line_open PASSED [100%]

============================== 9 passed in 11.95s ==============================

Run it

With the pipeline hardened, execute the test client against the live API:

uv run pickup.py

        (caller says pickup.wav)
        (session ready: sent 0.76 s of held audio, now live)
Caller: Hi, yes, I'm calling about my booking for Friday. The surname's Okafor.
        (look_up_booking returned for Okafor)
Agent:  Let me pull up your booking, the system can take a moment to load.
Agent:  I have your booking here, Ms. Okafor. You're in room 214 for two nights, checking in Friday. Is there anything else you need to know before your arrival?
        (caller says thats-all.wav)
Caller: Great, that's everything I needed. Thanks very much.
        (end_call: Caller confirmed they had all the information they needed.)
Agent:  You're welcome, Ms. Okafor. We look forward to having you stay with us. Have a wonderful day.
        (hangs up)
first line: lost 0 of 12 words at the start (session ready 0.76 s after pickup); goodbye: played 4.9 s of "You're welcome, Ms. Okafor. ...", all of it

This time, the client buffered 0.76 seconds of speech while the session negotiated. The agent heard "Hi, yes" cleanly, and the goodbye played for its full 4.9 seconds before teardown.

To compare performance across multiple calls, use --before and --runs:

uv run pickup.py --before --runs 10
uv run pickup.py --runs 10

10 runs: first words lost in 10 (1 to 1 words, median 1); session ready a median 0.36 s after pickup; goodbye played in full in 0, median 1.0 s played
10 runs: first words lost in 0 (0 to 0 words, median 0); session ready a median 0.83 s after pickup; goodbye played in full in 10, median 4.8 s played

In live testing against api.studio on 10 November 2026, from the US Midwest, the naive client lost the caller's first word in all 10 calls and played a median 1.0 s of each goodbye. Call lost no words and played every goodbye in full, a median 4.8 s.

To test the system yourself with real hardware, run uv run call.py with your headphones connected (or add --speakers to enable software echo cancellation). Start speaking immediately when the prompt appears, inquire about your reservation, and say goodbye when finished.

Troubleshooting

Symptom	Likely cause	Fix
The transcript starts a few words into the caller's first sentence	The client drops audio, or opens the microphone, until the session is ready	Open the microphone first and hold its frames until `session.updated`
The caller's first words vanish on phone audio, with no error	8 kHz μ-law sent before `session.update` is read as 24 kHz PCM	Hold the audio until `session.updated`
The agent answers something the caller said long ago	The buffer has no cap and the connection was slow	Cap it; `call.py` keeps the last 5 s
The goodbye is cut after a second	A fixed sleep before the hang-up	Wait for the goodbye's `response.done`, then drain the speaker
The call never hangs up after the goodbye	Nothing caps the wait	Wrap it in `asyncio.timeout(8)`
No goodbye and no `end_call`	The prompt says "say goodbye, then call end_call"	Make the goodbye a required `poly_spoken` argument
`The server refused the connection (HTTP 401). Check POLY_API_KEY, and set DIALOG_REALTIME_URL if your key is for a different host than wss://api.studio.poly.ai/v1/realtime.`	Wrong key, or a `POLY_API_KEY` in your shell is overriding `.env`	`echo $POLY_API_KEY`, then `unset POLY_API_KEY`
`POLY_API_KEY isn't set. Copy .env.example to .env and add your key from studio.poly.ai.`	No key in `.env` or the shell	Add your key to `.env`
`The server refused the connection (HTTP 429). Check POLY_API_KEY, ...`	Your account's concurrent sessions are all in use (2 on the sign-up credit), maybe by a run you left open	Wait for a session to end and try again; the message's advice about the key doesn't apply to a 429

Before you ship

Over a carrier line such as Twilio Media Streams, your application doesn't control the caller's speaker, so draining a local buffer tells you nothing about the carrier's playback. Use the carrier's own signal instead: Twilio sends a mark message back once the audio before it has finished playing, so you'd hang up on the returned mark, still with a cap. I haven't run this code on a Twilio line, so treat that as the place to start.

You could remove the start-up gap by opening the WebSocket when the phone starts ringing rather than on answer. That holds a session open for the whole ring, which bills every second of it and ties up one of your concurrent sessions (2 on the sign-up credit) on calls nobody picks up. The in-memory buffer is far cheaper.

Log every hang-up that hits the 8-second cap. The goodbyes I measured ran 3.6 to 5.1 seconds, so a capped one almost always means an upstream stall or a dropped socket.

Finally, tell callers at the start that they're speaking with an automated AI system, and get their consent before you record where the law requires it.

Where to go next

With these mechanics in place, your voice agent cleanly captures speech from the instant the line connects, handles barge-in interruptions gracefully, and waits for every syllable of its goodbye to finish playing before closing down.

From here, the natural next step is deploying the buffer pattern against a live telephony stream like Twilio or Telnyx, where 8 kHz μ-law audio sent too early is lost without an error. You can also explore PolyAI's API reference to apply poly_spoken properties to other slow tools, so your agent isn't silent while it queries external systems. The complete code is ready in pmckeetx/barge-in-voice-agent.

Stop your voice agent going quiet after it refunds a caller

Peter McKee — Thu, 12 Nov 2026 22:07:28 +0000

A caller rings a kitchen shop about a cast iron skillet that arrived cracked, hoping to get their money back. The voice agent politely replies, "Let me process that refund for you now." While the payment provider processes the transaction in the background, everyday life intervenes: the caller murmurs a quiet "mm-hmm," the evening news buzzes on a nearby television, or the caller suddenly remembers something and shouts, "Wait, wrong card!"

Real-time voice agents inhabit a messy world of background noise and spontaneous interruptions. When I set out to test these scenarios, I wanted to answer two practical questions: Which of these everyday sounds might trick an agent into issuing a duplicate refund? And second, what causes an agent to process a refund successfully, yet drop into complete silence without ever telling the caller it worked?

To find out, I built an automated refund desk on PolyAI's Dialog-RSN-1 API (pmckeetx/refund-voice-agent) and ran a scripted caller through 75 measured calls. Passive backchannels turned out to be completely harmless: a soft "mm-hmm" spoken over the agent never changed the outcome across 20 test runs. The real breakdown happened when the caller asked a natural follow-up question over the holding line: "Sorry, did that go through?"

Here is what that interaction looked like on the starting client implementation on 10 November 2026. This naive client suppresses follow-up agent responses once the caller speaks, and it mistakenly keys refunds on ephemeral tool calls rather than the underlying order (run as caller.py --before in the repository, producing R-4417-1):

        (caller says refund.wav)
Caller: Hi, I need a refund on order four thousand four hundred seventeen. The cast iron skillet arrived cracked.
        (issue_refund 4417 to 4242: refunded)
        (caller says did-it-go-through.wav)
        (barge-in: stopped the reply after 1.7 s)
Agent:  Let me process that refund-  [cut off]
Caller: Sorry, did that go through?
Agent:  It's still processing. I'll let you know as soon as it goes through.

refund_id  order  amount  card  status   told  calls
R-4417-1   4417    32.00  4242  pending  no    1

The money was sent, but the call fell silent. Across 10 out of 10 test runs, the agent promised to confirm the refund as soon as it finished, but never spoke another word. In the real world, this customer wakes up tomorrow, assumes the request was lost, and calls back to demand a refund they have already received.

The bug is subtle. The language model answered the caller's immediate question before the payment gateway returned. When the tool result finally arrived, the client saw that the user had spoken in the interim, decided not to interrupt with an unprompted reply, and abandoned the confirmation entirely.

Solving this requires a client-side ledger that tracks both the state of the transaction and whether the caller has actually heard the confirmation. By keying refunds directly to the order ID, duplicate tool calls resolve to the existing transaction, and card switches update the pending record instead of adding a second charge. Whenever the gateway finishes and no subsequent response has started, the client explicitly prompts the model for confirmation, and automated tests verify that the ledger only marks a refund told when the customer has heard the outcome.

What you'll build

In this guide, you will build a terminal-based voice agent for a fictional store, Copperline Kitchen Supply. This agent enforces strict financial idempotency: it pays exactly one refund per order regardless of how many times the model triggers the tool. If the caller blurts out "wait, wrong card" while the transaction is still pending, the agent diverts the pending refund to the new card rather than creating a duplicate payout.

The agent also ensures every transaction is accompanied by an audible confirmation, even if the caller interrupts with a question mid-process or talks directly over the agent's first attempt to speak. The project includes a scripted caller that simulates acoustic environments and phone calls in real time, allowing you to observe edge cases without touching a physical microphone, along with a test suite that reconciles your SQLite ledger against the audio heard by the caller.

How it works

When a caller requests a refund on Dialog-RSN-1, the model emits an issue_refund call. The client receives the response.function_call_arguments.done event before the audible holding line begins playing; across 72 test runs, the spoken holding audio lagged the event by a median of 0.12 seconds. The client kicks off the background refund immediately upon receiving that event.

Even if the caller barges in and cancels the spoken holding phrase, the tool execution itself has already been dispatched. In 10 test runs where the caller interrupted the agent mid-sentence, the tool call remained safely recorded in the server's conversation history every time.

This means transaction integrity rests entirely on the client, where three failure points emerge. The model might trigger the refund tool more than once, which the ledger must recognize as the same refund. The caller might speak while the gateway is processing, so the client has to decide when to ask for a confirmation. And the confirmation itself might be cut off by an interruption, leaving a completed refund the caller never heard.

Under the hood, barge_in.py manages audio interruptions: it halts outbound audio as soon as the server detects incoming speech, truncates conversation history to what was actually played out loud, and logs cut-off replies. We will leave that module as-is to focus on the heart of the system: the ledger and turn coordinator.

Prerequisites

To follow along, you will need Python 3.13 and uv, which locks down dependencies to match this environment. You will also need an active Dialog-RSN-1 API key, which you can generate under the API Keys tab at studio.poly.ai.

Calls run $0.02 per connected minute plus tokens, new accounts receive launch credits, and sign-up tiers support two concurrent sessions (limits and pricing, verified 10 November 2026). Each automated test call in this walkthrough finishes in under a minute.

The scripted testing suite operates entirely without a physical microphone or speaker setup. However, if you would like to test the live voice agent interactively using agent.py, keep a microphone and a pair of headphones ready.

Set up the project

Start by cloning the repository and generating your local environment configuration:

git clone https://github.com/pmckeetx/refund-voice-agent.git
cd refund-voice-agent
cp .env.example .env

Open .env in your editor and supply your API key:

# Your Dialog-RSN-1 API key: sign in at https://studio.poly.ai and create one in the API Keys tab.
POLY_API_KEY=

# Optional: the full realtime endpoint. Defaults to wss://api.studio.poly.ai/v1/realtime.
# DIALOG_REALTIME_URL=

Leave DIALOG_REALTIME_URL commented out unless your key is for a different PolyAI host. Next, install project dependencies and execute the offline test suite, which runs without requiring network access or API credentials:

uv sync
uv run pytest

...
tests/test_agent.py ......                                               [ 20%]
tests/test_audio.py ....                                                 [ 33%]
tests/test_ledger.py ........                                            [ 60%]
tests/test_live.py sss                                                   [ 70%]
tests/test_refunds.py .........                                          [100%]

======================== 27 passed, 3 skipped in 22.07s ========================

The three skipped tests represent live calls. When enabled, they run scripted audio against the real-time API using your credentials, taking roughly 90 seconds:

POLYAI_LIVE=1 uv run pytest -m live -v

...
tests/test_live.py::test_one_refund_and_the_caller_heard_it[mm-hmm] PASSED [ 33%]
tests/test_live.py::test_one_refund_and_the_caller_heard_it[did-it-go-through] PASSED [ 66%]
tests/test_live.py::test_one_refund_and_the_caller_heard_it[missed-confirmation] PASSED [100%]

================= 3 passed, 27 deselected in 84.89s (0:01:24) ==================

Setting POLYAI_LIVE=1 activates live integration, while -m live limits test collection to those scenarios. If you encounter an HTTP 401 error, verify that an existing POLY_API_KEY in your shell environment is not overriding your .env value.

Step 1: Give every refund a key

Our ledger consists of a single SQLite table, and it uses order_id as its primary key, making it structurally impossible to insert multiple refunds against the same order.

Here is the schema defined in ledger.py:

SCHEMA = """
CREATE TABLE IF NOT EXISTS refunds (
    order_id     TEXT PRIMARY KEY,                -- the key: what the refund is for, not who asked
    refund_id    TEXT NOT NULL UNIQUE,
    amount_cents INTEGER NOT NULL,
    card_last4   TEXT NOT NULL,
    status       TEXT NOT NULL DEFAULT 'pending', -- pending until the payment provider settles it
    session_id   TEXT NOT NULL,                   -- the call that asked first
    call_ids     TEXT NOT NULL,                   -- every tool call that asked for it, as a JSON list
    accepted     INTEGER NOT NULL DEFAULT 0,      -- has the payment provider accepted it, to this card?
    told         INTEGER NOT NULL DEFAULT 0       -- has the caller heard a reply that names its card?
)
"""

It is tempting to key transactions by the tool call's call_id because each call generates a unique identifier. However, that design is precisely why duplicate payouts happen: if the model decides to invoke the tool a second time, it generates a fresh call_id, and a naive insert creates a duplicate record. A refund is an action on an order, meaning the order must serve as the primary key. We still record session_id and all associated call_ids as an audit trail, while status, accepted, and told track provider state and user comprehension.

Warning: Don't key a side effect on call_id. It identifies the request, not the thing being done, so a repeat of the same request looks new.

Writing to this ledger is handled by a single transaction method, Ledger.refund():

    def refund(self, order_id: str, card_last4: str, session_id: str, call_id: str) -> tuple[str, dict]:
        """Record a refund for an order. Returns what happened, and the row as it now stands."""
        amount = ORDERS[order_id]["amount_cents"]
        self.db.execute("BEGIN IMMEDIATE")  # the check and the write happen under one lock
        try:
            added = self.db.execute(
                "INSERT INTO refunds (order_id, refund_id, amount_cents, card_last4, session_id, call_ids) "
                "VALUES (?, ?, ?, ?, ?, ?) ON CONFLICT (order_id) DO NOTHING",
                (order_id, f"R-{order_id}", amount, card_last4, session_id, json.dumps([call_id])),
            ).rowcount
            if added:
                outcome = "refunded"
            else:
                row = self.row(order_id)
                self.db.execute("UPDATE refunds SET call_ids = ? WHERE order_id = ?",
                                (json.dumps(json.loads(row["call_ids"]) + [call_id]), order_id))
                if row["card_last4"] == card_last4:
                    outcome = "already_refunded"
                elif row["status"] == "pending":
                    self.db.execute("UPDATE refunds SET card_last4 = ?, accepted = 0, told = 0 WHERE order_id = ?",
                                    (card_last4, order_id))
                    outcome = "card_changed"
                else:
                    outcome = "settled"  # the money has gone: moving it is a new refund and a reversal, not an update
            self.db.execute("COMMIT")
        except BaseException:
            self.db.execute("ROLLBACK")
            raise
        return outcome, self.row(order_id)

The write uses SQLite's ON CONFLICT (order_id) DO NOTHING, so the database itself decides whether this is the first refund for the order. Evaluating rowcount reveals whether a new row was added or an existing record caught the conflict. Relying on an in-memory dictionary or simple checks invites race conditions the moment two tasks execute concurrently. Wrapping the block in BEGIN IMMEDIATE guarantees that the conflict check, read, and subsequent update occur under the same write lock, and a test in tests/test_ledger.py races two connections at one order to prove it.

When an order row already exists, the method determines the appropriate business response. A repeat request on the same card is marked already_refunded. A switch to a different card while the status is still pending updates the target card in place and resets both accepted and told, because both the provider and the customer need to hear about the new card. If the refund has already cleared, the method returns settled, as modifying settled funds requires an explicit reversal and new transaction. Notice also that the refund amount is sourced directly from internal store records (ORDERS), never from model parameters.

Step 2: Answer a repeat call from the ledger

Because the language model only understands tool outcomes through returned values, our tool response must state the ledger's ground truth directly.

The tool definition is declared in agent.py:

TOOLS = [{
    "type": "function",
    "name": "issue_refund",
    "description": "Refund an order in full to one of the caller's cards.",
    "parameters": {
        "type": "object",
        "properties": {
            "order_id": {"type": "string", "description": "The order number, like 4417."},
            "card_last4": {"type": "string", "description": "The last four digits of the card to refund to."},
            "say_while_working": {
                "type": "string",
                "poly_spoken": True,  # the server speaks this argument while the tool runs
                "description": "A sentence to say while the refund goes through, like 'Let me process that refund for you now.'",
            },
        },
        "required": ["order_id", "card_last4", "say_while_working"],
    },
}]

The say_while_working parameter is tagged with "poly_spoken": True, instructing the server to speak this phrase aloud while the tool runs in the background. We make this field required on purpose. As highlighted in the Dialog-RSN-1 overview, models tend to omit optional spoken arguments, which leaves callers waiting in awkward silence while the external payment gateway completes its work.

The execution handler validates the inputs, consults the ledger, and formats the response:

    async def issue_refund(self, args: dict, call_id: str) -> dict:
        order_id, card = str(args.get("order_id", "")).strip(), str(args.get("card_last4", "")).strip()
        if order_id not in ORDERS:
            return {"status": "error", "note": f"There's no order {order_id} for this caller. Ask them to check the number."}
        if card not in CARDS:
            return {"status": "error", "note": f"The caller has no card ending {card}. Ask which card they want."}
        outcome, row = self.ledger.refund(order_id, card, self.session_id, call_id)
        self.log("", f"(issue_refund {order_id} to {card}: {outcome})")
        if not row["accepted"]:  # new, a new card, or a repeat of one the provider never accepted
            await submit_refund(row)
            self.ledger.mark_accepted(order_id)
        return {"status": outcome, "order_id": order_id, "refund_id": row["refund_id"], "note": self.note_for(outcome, row)}

Any transaction that has not yet been accepted (whether it is brand new, transferred to a replacement card, or retried after the provider failed the first time) is forwarded to the payment gateway through submit_refund(), mocked here as a four-second delay. Subsequent calls for an already accepted refund return immediately. In production, each request includes provider_key(row) as an idempotency key so the payment provider can reject duplicates on its end.

The model learns what to say next via note_for():

    def note_for(self, outcome: str, row: dict) -> str:
        """What the model is told, worded from the ledger's row, so its next line says what actually happened."""
        if outcome == "refunded":
            return f"Refunded {describe(row)}. Tell the caller it's done."
        if outcome == "already_refunded":
            when = "earlier in this call" if row["session_id"] == self.session_id else "on an earlier call"
            return f"This refund was already issued {when}: {describe(row)}. Tell the caller it's done; don't refund again."
        if outcome == "card_changed":
            return f"Still one refund, now {describe(row)} instead. Tell the caller."
        return f"This refund already settled, {describe(row)}, and can't be moved. Tell the caller a person will call back."

The helper function describe(row) converts the database entry into natural phrases like "32 dollars to the Visa ending 4242," anchoring the agent's statements directly to the database state. If an order was previously refunded, the note indicates whether the event occurred in the current session or a previous call, providing clear context when a customer calls back.

Across 10 test calls where callers asked "Sorry, did that go through?", the model never triggered issue_refund a second time. However, when the caller interrupted with "Wait, wrong card. Can you put it on my Mastercard instead?", the model reliably invoked issue_refund with the new details in 5 out of 5 runs. While a naive client would have charged the customer twice, our ledger flagged the transition as card_changed, kept a single database row, and prompted the agent to confirm: "That's done. I've put the 32 dollars back to your Mastercard ending 8812."

Step 3: Confirm once the result is in

Now let us revisit the silence bug from our opening example.

In the initial implementation, the client skipped sending response.create after receiving a tool result if the caller had spoken in the meantime. The reasoning seemed sound: avoid talking over a caller who has already moved on to a new question. While that works for read-only database queries, it is disastrous for mutations. When the caller asked "did that go through?", the model answered "It's still processing." When the gateway finally finished, the client saw that the user had spoken, suppressed the follow-up response, and never closed the conversational loop.

To fix this, the client must trigger a reply whenever no new turn has started since the tool result entered history. The event handler tracks response counters, records which responses contained tool calls, and logs mapping identifiers:

    async def handle(self, event: dict):
        match event["type"]:
            case "response.created":
                self.responses += 1
            ...
            case "response.function_call_arguments.done":
                self.with_calls.add(self.responses)
                self.spawn(self.run_tool(event))  # the call is final: start the refund before the holding line plays
        await super().handle(event)

The tool executes inside a dedicated task started as soon as argument streaming completes. When finished, run_tool() appends the outcome to the history and coordinates the response:

    async def run_tool(self, call: dict):
        try:
            result = await self.issue_refund(json.loads(call["arguments"]), call["call_id"])
        except Exception as error:  # every call gets an output, or the model waits for it forever
            self.log("", f"(issue_refund failed: {error})")
            result = {"status": "error", "note": "The refund system failed, so the refund isn't confirmed. Don't call "
                      "issue_refund again. Tell the caller the team will email them once it's sorted."}
        since = self.responses
        await self.send({
            "type": "conversation.item.create",
            "item": {"type": "function_call_output", "call_id": call["call_id"], "output": json.dumps(result)},
        })
        if result.get("refund_id"):
            self.owed.setdefault(result["order_id"], since)  # from the first result the caller hasn't heard
            self.reminded.pop(result["order_id"], None)
        await self.ask_for_reply(since)

    async def ask_for_reply(self, since: int):
        """Ask the model to speak about the result, unless a reply has already started since it went into history."""
        while self.responding:  # one response at a time: let the holding line, or the answer to the caller, finish
            await asyncio.sleep(0.05)
        if self.responses == since:
            await self.send({"type": "response.create"})  # sent while the caller talks, it's held for their turn

A tool output event alone does not trigger speech. Appending function_call_output merely updates the conversation log; an explicit response.create event is required to prompt an agent response.

The since variable captures the response count at the exact moment the output was recorded, and owed preserves this marker until the caller hears the result. The ask_for_reply() method waits for any active turn to conclude (such as the holding line or the answer to an interruption), avoiding conversation_already_has_active_response errors. If no newer turn has begun, the client issues response.create. If the caller is speaking when that event arrives, the API reference says the server holds it and answers their turn with a single reply, which is what happened in all 5 mm-hmm-gap runs on 10 November.

Running the original problem scenario through the updated client demonstrates the fix:

...
Caller: Sorry, did that go through?
Agent:  It's still processing. I'll let you know as soon as it goes through.
Agent:  Yes, it's done. I've refunded 32 dollars back to your Visa ending 4242.

refund_id  order  amount  card  status   told  calls
R-4417     4417    32.00  4242  pending  yes   1

The caller heard the confirmation in 5 out of 5 runs, compared to 0 out of 10 on the older client.

Handling exceptions inside run_tool() protects this flow. Every tool invocation receives an explicit payload so the model is never left waiting indefinitely. In a run on 10 November with the payment provider down, the caller was promptly told: "The refund system failed, so it's not confirmed yet. Our team will email you once it's sorted." The row remained stored with accepted and told set to zero, an actionable record for support staff, and if the caller asks again, step 2's check submits it again.

Step 4: Tell the caller what the ledger says

Just because the server sent an audio stream does not mean the user actually heard it. An interruption can cut a sentence off halfway through.

To bridge this gap, the client sets the ledger's told flag only when a reply the caller actually heard names the refunded card:

    async def heard(self, item_id: str, text: str, whole: bool):
        await super().heard(item_id, text, whole)
        for order_id, since in list(self.owed.items()):
            n = self.response_of.get(item_id, 0)
            if n <= since or n in self.with_calls:
                continue  # a holding line, or a reply that started before the result was in history
            row = self.ledger.row(order_id)
            if mentions(text, row["card_last4"]):
                self.ledger.mark_told(order_id)
                del self.owed[order_id]
            elif self.reminded.get(order_id, 0) < 2:  # twice at most, so a model that won't say it can't loop
                self.spawn(self.remind(row))

The heard() hook fires after every spoken reply, receiving the exact text delivered to the caller: either the full sentence or the truncated fragment captured when an interruption occurred. Holding lines and earlier responses are filtered out, and mentions() confirms the specific last four card digits made it across the wire.

If an interruption truncates the confirmation, remind() puts the ledger's word into history and asks for a reply:

    async def remind(self, row: dict):
        """The refund is real, but the caller didn't hear it. Put the ledger's word in history and ask for a reply."""
        self.reminded[row["order_id"]] = self.reminded.get(row["order_id"], 0) + 1
        note = (f"Refund {row['refund_id']} is done: {describe(row)}. The caller hasn't heard that yet. In your next "
                "reply, tell them it's done, with the amount and the card's last four digits.")
        self.log("", f"(reminder: {row['refund_id']} not heard yet)")
        since = self.responses
        await self.send({"type": "conversation.item.create",
                         "item": {"type": "message", "role": "system", "content": [{"type": "input_text", "text": note}]}})
        await self.ask_for_reply(since)

In early iterations, I tried injecting this reminder purely as a passive role: system message. When a caller interrupted the confirmation, the system message landed in the history before the agent replied to the caller's new turn. Across 10 out of 10 runs, the model answered the caller's immediate question and completely ignored the passive instruction to confirm the refund.

What proved reliable was requesting a dedicated response once that turn finished. For instance, when a caller interrupted 1.2 seconds into the confirmation with "Okay, thanks. How long does it take to show up?", the system handled the conversational detour before circling back:

Agent:  I've refunded 32 dollars back to your Visa-  [cut off]
        (reminder: R-4417 not heard yet)
Caller: Okay, thanks. How long does it take to show up?
Agent:  I don't have information about that. Would you like me to transfer you to someone who can tell you how long it takes?
        (reminder: R-4417 not heard yet)
Agent:  It's done, 32 dollars back to your Visa ending 4242. I don't have information about how long it takes to show up. Would you like me to transfer you to someone who can tell you?

That second response successfully confirmed the refund in 5 out of 5 runs. Generating the reminder directly from ledger data avoids the ambiguity of truncated transcript text, which often lacks both the card digits and confirmation status.

Step 5: Test the ledger against what the caller heard

Every conversational failure we have examined boils down to a discrepancy between two sources of truth: the database records and the audio that reached the caller's ear.

We can capture that invariant in a single evaluation function inside ledger.py:

def mismatches(rows: list[dict], heard: list[str]) -> list[str]:
    """Where the ledger and the caller disagree. heard is every reply the caller heard, holding lines left out."""
    problems = [f"order {order} refunded {n} times" for order, n in Counter(r["order_id"] for r in rows).items() if n > 1]
    for row in rows:
        if not any(mentions(reply, row["card_last4"]) for reply in heard):
            problems.append(f"{row['refund_id']} is in the ledger, but the caller never heard it was done")
        elif not row["told"]:
            problems.append(f"{row['refund_id']} was heard, but the ledger says it wasn't")
    confirmations = [reply for reply in heard if any(mentions(reply, card) for card in CARDS)]
    if confirmations and not any(mentions(confirmations[-1], row["card_last4"]) for row in rows):
        problems.append(f"the caller last heard {confirmations[-1]!r}, but no refund in the ledger goes to that card")
    return problems

An empty list signifies a fully reconciled call. The assertions catch duplicate payouts, unconfirmed transactions, and mismatched card details. The heard log originates from the test harness, which discards intermediate holding lines so phrases like "let me refund your Visa ending 4242" spoken before execution cannot masquerade as completed confirmations.

Offline integration tests simulate calls against a mock server in tests/fake_realtime.py. The mock synchronizes against caller audio duration, streaming responses at normal conversational cadences while allowing test suites to run ten times faster than real time.

Here is how we verify idempotent re-issuance in tests/test_refunds.py:

async def test_a_reissued_call_is_the_same_refund(monkeypatch, tmp_path):
    steps = [("tool", "issue_refund", VISA, LINE, 2000), ("tool", "issue_refund", VISA, "Let me check on that.", 1500),
             ("say", DONE, 2500)]
    async with FakeRealtime(steps, ["I need a refund on order 4417.", "Sorry, did that go through?"]) as fake:
        room, ledger = await call(fake, monkeypatch, tmp_path, "did-it-go-through")
    outputs = [json.loads(e["item"]["output"]) for e in fake.sent("conversation.item.create")
               if e["item"]["type"] == "function_call_output"]
    assert sorted(o["status"] for o in outputs) == ["already_refunded", "refunded"]  # one refund, and one repeat
    assert len(ledger.rows()) == 1
    assert_ledger_matches_heard(room, ledger)

The steps script a repeat call that the real model didn't make on 10 November, because a test that only covers what one model did on one day isn't much of a test. Comparing it against a client keyed naively on call_id shows why conversational testing alone is insufficient:

uv run pytest tests/test_refunds.py -k "reissued or keyed_on" -s -v

...
tests/test_refunds.py::test_a_reissued_call_is_the_same_refund         (caller says refund.wav)
Caller: I need a refund on order 4417.
        (issue_refund 4417 to 4242: refunded)
        (caller says did-it-go-through.wav)
        (barge-in: stopped the reply after 1.7 s)
Agent:  Let me process that refund for yo-  [cut off]
Caller: Sorry, did that go through?
        (issue_refund 4417 to 4242: already_refunded)
Agent:  Let me check on that.
Agent:  That's done: 32 dollars back to your Visa ending 4242.
PASSED
tests/test_refunds.py::test_a_ledger_keyed_on_call_id_pays_out_twice         (caller says refund.wav)
Caller: (caller)
        (issue_refund 4417 to 4242: refunded)
        (caller says did-it-go-through.wav)
        (barge-in: stopped the reply after 1.7 s)
Agent:  Let me process that refund for yo-  [cut off]
Caller: (caller)
        (issue_refund 4417 to 4242: refunded)
Agent:  Let me check on that.
Agent:  That's done: 32 dollars back to your Visa ending 4242.
PASSED

In both runs, the spoken dialogue sounds completely natural. However, on the call-keyed implementation, mismatches() identifies ["order 4417 refunded 2 times"]. Listening to the audio alone would never reveal the duplicate charge.

Our test suite applies this reconciliation across multiple edge cases, covering mid-flight questions, ignored reminders, card changes, and upstream payment errors.

Run it

You can execute specific interaction scripts against a live server using the scripted caller:

uv run caller.py --script did-it-go-through

The script streams the interaction in real time and prints the resulting ledger status upon completion. Use --script to test different conversational branches: mm-hmm and tv play a backchannel and a newsreader under the holding line, wrong-card and did-it-go-through test direct interruptions, mm-hmm-gap says "mm-hmm" just after the line while the refund is still running, and missed-confirmation talks over the final confirmation. Passing --before runs these scenarios against the legacy client for direct comparison.

To benchmark consistency over time, measure.py executes batches against isolated ledgers:

uv run measure.py --script did-it-go-through --runs 5 --audit

did-it-go-through, 5 runs on api.studio.poly.ai
...
Caller: Sorry, did that go through?
Agent:  It's still processing. I'll let you know as soon as it goes through.
Agent:  Yes, it's done. I've refunded 32 dollars back to your Visa ending 4242.
RUN 5: {"speech_started": 2, "speech_stopped": 2, "cut_after_s": 0.74, ... "calls_in_history": ["function_call"], "rows": [["4417", "4242", "told"]], "mismatches": [], ...}
5 runs: more than one call 0, refunded twice 0, no refund 0, caller never told 0, any mismatch 0
reply cut a median 0.76 s after the caller started (0.74 to 0.79, 5 runs)

The --audit flag queries the server using conversation.item.retrieve to verify that tool invocations persist in remote history.

Here are the aggregated results from test runs conducted on api.studio.poly.ai on 10 November 2026:

Script	Runs	What happened	Caller never told
`mm-hmm`, 0.8 s into the holding line	10	Nothing: no `speech_started`, the line played on	0
`mm-hmm-confirmation`, 1.2 s into the confirmation	10	Nothing: the confirmation played to the end	0
`mm-hmm-gap`, after the line, refund still running	5	A turn ("Mm-hmm."), answered by the confirmation	0
`tv`, 0.8 s into the holding line	10	Not a barge-in, but transcribed as the caller's turn every time	0
`wrong-card`, 0.5 s into the line	5	Line cut, a second `issue_refund` to 8812, one row	0
`did-it-go-through`, 0.8 s into the line	10	Line cut, "still processing", no second call	10
`did-it-go-through`, fixed client	5	"still processing", then "Yes, it's done"	0
`missed-confirmation`	5	Confirmation cut, the reminder ignored	5
`missed-confirmation`, fixed client	5	The second ask told them	0
`did-it-go-through --before`	5	As above, and one row: the model never asked twice	5
`mm-hmm-gap --before`	5	As above	0

Not a single run resulted in a duplicate refund. On the legacy client running did-it-go-through, the holding line was cut a median 0.76 seconds after the caller started talking, timed on the client, so that includes a network round trip.

To test the system yourself with a live microphone, start the interactive service:

uv run agent.py

Ask for a refund on order 4417, interrupt the agent while it works to ask if the transaction went through, and press Ctrl+C to inspect the final ledger summary.

Troubleshooting

Symptom	Likely cause	Fix
The agent says "still processing" and never confirms	`response.create` skipped because the caller spoke while the tool ran	Ask unless a reply has started since the result (step 3)
The caller talked over the confirmation and never heard it again	A reminder note with no `response.create` after it	Ask for a reply once the caller's turn has been answered (step 4)
Two rows for one order, with different `call_id`s	The ledger is keyed on `call_id`	Key on the order and keep `call_id`s as a column
`sqlite3.IntegrityError: UNIQUE constraint failed: refunds.order_id`	A plain `INSERT` on the keyed table	`INSERT ... ON CONFLICT (order_id) DO NOTHING`, then read the row
An `error` event with code `conversation_already_has_active_response`	`response.create` sent while a response was still open	Wait for `response.done` before sending it
`The server refused the connection (HTTP 401).`	Wrong key, or a `POLY_API_KEY` in your shell beats `.env`	`echo $POLY_API_KEY`, then `unset POLY_API_KEY`

Before you ship

While this implementation uses a simulated four-second sleep to represent the payment provider, a production deployment should pass provider_key(row) (a composite of the refund ID and card details) directly to the gateway as an idempotency key. This ensures payment providers like Stripe deduplicate repeated requests upstream. Stripe's v1 API (consulted 10 November 2026) replays the first response for a key, raises an error if the same key comes back with different parameters, and can prune keys once they're 24 hours old. Because of those limitations, your database ledger remains your primary source of truth, with provider keys serving as secondary protection. Writing the ledger entry before initiating network calls guarantees that an unexpected crash leaves an actionable record for reconciliation rather than untracked money movement. A card change that lands while the first submission is still with the provider needs that first request cancelled, which the stand-in doesn't model.

Our example assumes full refunds keyed strictly by order ID. If your store supports partial refunds, expand the key to include line item identifiers or amount markers to prevent distinct partial claims from blocking one another. High-value transactions should similarly pass through an explicit human review step before hitting the payment gateway, which fits naturally into the ledger's pending state.

Any order marked as untold when a caller hangs up should automatically queue an asynchronous confirmation via SMS or email. During our testing, background television dialogue was transcribed as direct caller input in all 10 runs; while our agent handled it gracefully, tools that trigger destructive actions should always prompt for explicit voice confirmation before running. Finally, have the agent say it's an AI at the beginning of the call, and if you record calls, tell the caller and, where the law requires it, get their consent.

Where to go next

You now have a refund desk where the database ledger is the source of truth: orders refund once regardless of model repetition, card switches update pending transactions safely, and confirmations are delivered even when callers interrupt mid-flight.

To expand this setup, try introducing an order cancellation tool like cancel_order with its own idempotency constraints within the same SQLite database. For deeper dives into real-time voice architectures, consult the Dialog-RSN-1 overview on turn-taking mechanics, along with the API reference on managing barge-in events and response lifecycles. All source code is available in the pmckeetx/refund-voice-agent repository, and you can obtain an API key at studio.poly.ai.