How does an AI voice receptionist work?
A voice receptionist is a loop with a hard real-time budget. The carrier streams the caller's audio to your server, speech-to-text turns it into words, a language model decides what to say or which tool to call, and text-to-speech streams the answer back before the caller says "hello?" a second time.
I run Cuanto Labs, a software studio in Fort Lauderdale, Florida. A few weeks ago we put our own phone agent on a real number. It answers, screens, takes messages and hands calls to a person. This post covers the architecture and the bugs that cost us the most time, because most voice-agent write-ups stop at the diagram. It describes components, not products.
| Component | Job | What to test |
|---|---|---|
| Telephony carrier | Carries the call and streams audio both ways | Audio direction, webhook verification, spend caps |
| Speech-to-text | Turns caller audio into words | Accents, noise, phone numbers spoken aloud |
| Language model | Decides the reply or the tool call | Time to first token, tool-use accuracy |
| Text-to-speech | Turns the reply into audio | Naturalness on a phone line, second languages |
| Agent framework | Runs the frame pipeline between them | Interruptions, voice activity detection |
We used an open source agent framework for voice activity detection, interruption handling and partial transcripts, so we did not own that plumbing.
Why use WebSocket media instead of SIP?
Use WebSocket media when your agent runs behind a home or office router. SIP needs public UDP ports for signalling and audio, while a WebSocket stream is ordinary TCP, so it passes through a tunnel with no port forwarding. If your server has a public IP address, revisit SIP, which is a good system.
We evaluated SIP first, but our agent runs behind a residential router, so the carrier's WebSocket media stream was the easier path.
How do you keep a voice agent from doing something dangerous?
Give the model a short, fixed list of tools and keep dangerous arguments out of its reach. Our agent can call exactly three tools: transfer to a human, take a message, and end the call. The transfer destination is fixed in the server environment, so the model never supplies a number.
That is the cheapest safety decision in the build: a prompt injection from the caller cannot redirect a call to a premium-rate number if the model has no argument to put it in.
We also run a small, fast classifier before answering, to catch spam and robocalls, and on each turn, to decide whether to escalate to a person. It has a hard timeout, because a slow check must never stall the call.
Which language model works best on a phone call?
Pick the model by time to first token, not by benchmark rank. On a phone call, quality that arrives late is quality the caller never hears. In our tests a small fast model answered in well under a second, while a larger model took more than a second and a half, and that gap decided it.
A fast model is not free. One otherwise capable model mangled messy phone-number extraction in half of our runs, because callers say numbers like "seven eight six, uh, four one three, zero four one six." Treat number capture as something you verify by reading it back, never as something the model got right.
How should you choose a text-to-speech voice?
Choose by listening, because the measurable metrics tie. We compared six voices on the same script at telephone audio quality. Time to first audio and word error rate on a round-trip transcription test were effectively tied. A blind human-likeness rating was the only measure that separated them.
A second language costs you. In a Hebrew test, replies took several seconds, against well under a second in English. Plan the greeting and filler phrases accordingly.
What bugs make a voice agent go silent?
Three bugs cost us days, and all three were configuration, not code. The first left the agent unable to hear, the second left it silent in the recording, and the third made webhook signature checks fail, even though the logs looked healthy.
The agent that could not hear
The bot greeted the caller, then never responded. We had asked the carrier for both audio directions, which sends the bot's own voice back to us as a media message tagged as outbound. Our framework's stock serializer did not filter by direction, so the bot's own speech fed the speech recognizer and the voice activity detector. The bot kept "listening" to itself.
The fix was a small serializer subclass that drops every frame not tagged as inbound. Five lines of code, two days of confusion.
The agent that was silent in the recording
Calls connected and transcripts looked fine, but the bot's channel in the recording was empty. One carrier option routes the bot's audio to a bridged call leg. Our call had no bridged leg, so the audio went nowhere. Pointing the option at the call itself fixed it.
We caught it by comparing the two channels of the recording, not the logs. A pipeline can log "audio sent" faithfully while the carrier discards it. Listen to a real recorded call after any change to stream parameters.
The webhook that verified against the wrong key
Webhook signatures should be verified against the public key the carrier serves from its own key endpoint. A key we had copied into our secrets file looked right, was a different length, and did not match. If signature verification passes in tests and fails in production, or is quietly disabled, check the key source first.
What else belongs on a voice agent checklist?
Four smaller items belong on every checklist before launch. They are the ones we hit in the first week of running real calls, and each one is cheap to handle once at build time and expensive to find later, in production, with a caller waiting.
- Carrier command payloads can require base64 encoding for state fields, and call identifiers can contain characters that need URL-encoding in paths.
- A 403 when creating a call can mean the call-control application was set inactive. Read the response body, which says so.
- The first call after a restart can time out while models and sockets warm up. Place a warm-up call in your deploy script.
- Telephony is not chat. Set an outbound allowlist where empty means deny everything, an hourly call cap, and a daily spend cap on the carrier profile.
How do you test a voice agent without a phone?
Write a fake carrier. Ours is a WebSocket client that speaks the carrier's media protocol and plays synthesized caller audio, so the full pipeline runs in continuous integration with no real phone numbers, no carrier charges and no cost beyond speech processing.
For real-audio checks, one of our numbers answers with a listener persona, and a test asserts that both call logs contain user turns. If either side has zero user turns, you have a one-way audio bug like the two above.
What does Florida law say about recording a voice agent's calls?
Florida generally requires the consent of all parties to record a phone call. Our greeting states that the call is recorded, and the disclosure is spoken in the caller's language. Have an attorney confirm the wording for your business, since this post is engineering experience and not legal advice.
If your agent talks to the public, also decide up front when it admits it is an AI (ours does when asked directly) and keep a human fallback for urgent or sensitive calls. The agent should never triage an emergency. It should tell the caller to dial 911 and stop.
What would we tell a client?
The parts are not exotic. What decides whether it works is a short list: pick the model by latency, keep dangerous arguments out of the model's reach, listen to real recorded calls, and build the spend caps and kill switches before launch, not after.
We build scoped voice agents for Florida businesses at a fixed price from $5K, typically in 2 to 6 weeks, and the code and accounts transfer to you. Details and the questions to ask any vendor are on the AI voice receptionist page. For prices, see what an AI voice receptionist costs in Florida.
Written by Eliran Yihye, founder of Cuanto Labs, on October 15, 2026.
Originally published at cuantolabs.com.
Top comments (0)