A production AI voice agent is not a chatbot with a microphone. It's a phone system that transcribes the caller, reasons with an LLM, speaks back fast enough to feel human, and writes the outcome into a calendar or CRM. I build these for dental, healthcare, fitness, and mortgage businesses, including agents that book real appointments after hours.
Most people evaluate a voice agent the wrong way. They listen to a two-minute demo, hear a pleasant voice, and assume the hard part is the model. The hard part is everything around the model: the phone network, the speech pipeline, the calendar, the CRM, and the half-second of silence that makes a caller hang up.
I work on more than ten live AI platforms, including VoiceCake, Rawk.ai, and Dynaris. This is the production checklist I use before an agent is allowed to sit on a real phone number.
What a production voice agent actually is
An inbound agent answers a missed or after-hours call, identifies intent, collects the facts it needs, and either completes the job or hands off cleanly. An outbound agent places follow-up or reminder calls with the same discipline. Both are phone systems first. The LLM is one component.
A working stack usually looks like this:
Telephony: Twilio or a comparable SIP/PSTN layer for a real number, recording policy, and failover path.
Speech-to-text: Deepgram or equivalent, tuned for the accent mix and vocabulary of the business, provider names, insurance terms, and service codes.
Reasoning: an LLM behind LangChain or LangGraph, so the conversation can branch, retry, and recover instead of following a rigid tree.
Text-to-speech: ElevenLabs or similar, with a barge-in so the caller can interrupt.
Orchestration: Vapi or Retell AI when they fit, custom glue when they don't.
Systems of record: the calendar, CRM, and notification tools the staff already trusts.
If the agent can't write a confirmed appointment into the same calendar a human receptionist uses, it's not a booking agent. It's a voicemail with extra steps.
The latency budget is the product.
Callers don't forgive lag the way chat users do. A 900ms gap after every sentence feels like a bad connection. People talk over the agent, the agent talks over them, and the call collapses into noise.
On Rawk.ai, a live voice-agent builder platform, I led a latency pass that dropped end-to-end response time from roughly 900ms to roughly 320ms. That's not a marketing number. It's the difference between a call that feels robotic and one that feels human.
A practical budget, measured from the end of the caller's utterance to the start of audible speech:
STT partials and endpointing: keep streaming, don't wait for a perfect transcript before thinking.
LLM time-to-first-token: stream the reply, never buffer a full paragraph.
TTS time-to-first-byte: start audio as soon as the first clause is ready.
Tool calls: prefetch calendar availability before you need it, cache what doesn't change mid-call.
Network: colocate STT, LLM, and TTS as close as the providers allow, extra regions add tens of milliseconds you can't buy back with a better prompt.
Sub-400ms is the range I aim for on client builds. You won't hit it on day one if the agent is running three sequential tool calls against a cold calendar API. That's an architecture problem, not a prompt problem.
Conversation design that survives a real caller
Production callers don't follow the happy path. They give a date without a time. They change their name spelling twice. They ask a question the prompt never mentioned. There's a crying child in the background.
Collect facts in a stable order. For appointment booking, the agent needs a small, explicit slot list: who's calling, what service, which location if there's more than one, preferred window, and a reachable callback number. Confirm each high-stakes field out loud before writing it. Don't invent a slot the calendar didn't offer.
Define failure, not just success. Every agent needs an escape hatch: transfer to a human, take a message, or schedule a callback. Infinite retry loops are how callers decide the business is closed forever. On Dynaris, the front desk isn't voice-only, chat and email sit in the same customer thread, so a failed call can continue as a message without losing context.
Isolate tools per workspace. Multi-location and multi-tenant systems can't share tool servers casually. Dynaris runs voice agents and tools as separate MCP-based services so each workspace has its own tool surface. That's a security and correctness requirement, not a nice-to-have.
CRM and calendar wiring is the booking
VoiceCake is a live 24/7 inbound platform for dental, healthcare, fitness, and mortgage clients. The proof that matters isn't that the agent can talk. It's that a dental engagement books real appointments. That only happens when the agent can read open slots, write the booking, and trigger the same confirmation the staff already sends.
Read availability from the system of record, not from a spreadsheet export.
Write the appointment with the same fields a receptionist would: patient, provider, duration, notes, source.
Send confirmation on the channel the business already uses, SMS, email, or both.
Log the call outcome even when nothing was booked, so follow-up isn't guesswork.
If the CRM is HubSpot, Salesforce, Zoho, or something custom, the same rule applies. This work exists because a voice agent that can't write a record just creates a second, worse inbox.
A ship checklist I actually use
Latency on live PSTN, not a browser demo: measure p50 and p95 of time-to-first-audio.
Barge-in: the caller can interrupt without the agent finishing a paragraph.
Booking write-path: create, reschedule, and cancel against the real calendar.
Identity: name, phone, and location disambiguation when two records look similar.
Handoff: warm transfer or a complete message, never a dead air drop.
After-hours vs in-hours behavior: different prompts, same system of record.
Recording and retention matched to the industry. Healthcare work, like OptimateMD.health, needs data-handling designed in from architecture, not bolted on.
Observability: transcripts, tool-call traces, and a way to replay a failed call without guessing.
When you shouldn't build one yet
If the business has no calendar of record, no one who owns the front desk process, or a phone tree that already loses callers before a human answers, a voice agent will amplify the mess. Fix the workflow, then automate it. Same stance I take on custom software generally: ship the smallest system that owns the outcome.
If you do have missed calls, after-hours demand, or a receptionist drowning in routine booking, a production voice agent is one of the few AI products that shows up on the P&L in weeks rather than quarters.
Originally published at keencraft.tech/blog/production-ai-voice-agents
Top comments (0)