Last quarter, we ran our production voice AI receptionist — Loquent — across four different TTS engines simultaneously, split-testing real patient calls at dental and healthcare clinics. The results surprised us: the most "natural sounding" engine in demos performed the worst with actual patients.
Why We Ran This Test
At Autor, we've been running Loquent in production for over a year now. It handles thousands of automated calls per month for healthcare and dental clinics across Canada — booking appointments, answering insurance questions, handling after-hours triage. The voice is the product. If patients don't trust the voice, they hang up, and the clinic loses a booking.
When we first built Loquent, we picked our TTS engine the way most teams do: we generated a few sample clips, played them for ourselves, and went with the one that sounded best in a quiet office. That worked fine until we started digging into our call analytics and noticed something weird. Our completion rate — the percentage of calls where patients actually finished the full interaction instead of hanging up or asking for a human — was hovering around 74%. Good, but not great. We suspected the voice itself was part of the problem.
So we designed a proper A/B test. Not a demo comparison. A production comparison on live calls.
The Setup
We tested four TTS engines across 12,247 calls over 8 weeks. Each engine handled roughly equal volume, randomly assigned at call start. All other variables stayed constant: same prompts, same Anthropic Claude backbone for conversation, same Twilio infrastructure, same clinics.
The four engines:
- Engine A: ElevenLabs (Turbo v2.5) — our existing production engine
- Engine B: OpenAI TTS (tts-1-hd) — the model most teams default to
- Engine C: Deepgram Aura — optimized for real-time, low-latency use cases
- Engine D: A newer entrant we'd been evaluating (under NDA, so I can't name it)
We measured five things:
- Completion rate — did the patient finish the full call flow?
- Time-to-first-hang-up — how far into the call patients dropped if they did hang up
- Human transfer requests — how often patients explicitly asked for a real person
- Repeat-caller behavior — did patients who called back engage or immediately ask for a human?
- Post-call survey scores — a subset of clinics ran a 1-question SMS survey: "How was your experience? Reply 1-5"
We did not measure "naturalness" or "voice quality" in isolation. We measured whether patients trusted the voice enough to complete their task.
The Results
Here's what we found, and it's not what the marketing pages of any of these vendors would predict.
Completion rates:
- Engine C (Deepgram Aura): 81.2%
- Engine A (ElevenLabs): 78.4%
- Engine D (unnamed): 76.1%
- Engine B (OpenAI TTS): 71.8%
Deepgram won on completion rate by a meaningful margin. But the interesting part is why.
Latency was the dominant factor, not voice quality. When we dug into the data, the correlation between response latency and hang-up rate was stronger than any voice quality metric. Engine C had a median time-to-first-byte of 180ms. Engine A was at 320ms. Engine B sat around 480ms. Engine D was 410ms.
In a voice call, latency feels like hesitation. And when you're a patient calling a clinic, hesitation from an AI voice triggers the uncanny valley harder than a slightly robotic tone ever will. Patients interpreted fast responses as confidence and slow responses as something being wrong.
The "best sounding" engine had the worst outcomes. OpenAI's tts-1-hd produces objectively beautiful speech. In blind audio comparisons, our team ranked it first every time. But in production, it had the lowest completion rate. The higher latency meant longer pauses in conversation, and patients read those pauses as the system struggling to understand them. Several patients in the survey comments said things like "it seemed confused" — when the AI wasn't confused at all, it was just waiting for audio to generate.
Human transfer requests told a different story:
- Engine A (ElevenLabs): 14.2%
- Engine B (OpenAI TTS): 19.7%
- Engine C (Deepgram Aura): 12.8%
- Engine D (unnamed): 16.3%
Engine C again. But the gap between A and C was smaller here. Patients who stayed on the call long enough to interact mostly stuck with it regardless of engine. The damage was being done in the first 3-5 seconds.
The first response is everything. We isolated the first-turn drop-off rate — patients who hung up after hearing just the greeting and one response. The spread was dramatic:
- Engine C: 4.1% first-turn drop-off
- Engine A: 6.8%
- Engine D: 8.9%
- Engine B: 11.3%
Nearly 1 in 9 patients hung up after the first exchange with Engine B. With Engine C, it was 1 in 24. Same greeting, same words, same AI — different voice engine, different outcome.
Repeat callers validated the pattern. Patients who had previously interacted with Engine C were 2.3x more likely to engage with the AI on their next call versus asking for a human immediately. Engine B repeat callers asked for a human 34% of the time on their second call. First impressions stuck.
What We Changed
We moved Loquent's primary TTS to Deepgram Aura for all production healthcare clients. But we didn't just swap engines and call it done. The test taught us three architectural things:
First, we implemented adaptive engine selection. For complex responses where voice quality matters more — reading back appointment details, explaining insurance coverage — we route to ElevenLabs. For fast conversational turns — confirmations, short answers, greetings — we use Deepgram. This hybrid approach got us to 83.6% completion rate, higher than any single engine alone.
Second, we added latency budgeting to our voice pipeline. Every component now has a time budget: LLM inference gets 800ms, TTS gets 250ms, network overhead gets 150ms. If any component is trending over budget, we fall back to a faster path. We pre-generate common responses (greetings, "one moment please" fillers, confirmations) so the first turn is always instant.
Third, we started measuring voice trust, not voice quality. We retired our internal "does this sound good?" reviews and replaced them with production metrics: completion rate, first-turn drop-off, and repeat-caller engagement. The voice that sounds best in a meeting room is not the voice that performs best on a phone call from a noisy waiting room.
Key Findings
Latency beats quality in production voice AI. A 300ms difference in time-to-first-byte correlated with a 10-percentage-point swing in call completion. Patients interpret speed as competence and delays as confusion.
The first 3 seconds determine the entire call. First-turn drop-off varied by nearly 3x across engines. If you're optimizing your voice AI, start with the greeting response time, not the voice timbre.
Demo performance inversely correlated with production performance. The engine our team unanimously preferred in blind tests had the worst patient outcomes. Always test in production, not in a conference room.
Hybrid TTS routing outperforms any single engine. Using fast engines for conversational turns and higher-quality engines for complex information delivery gave us better results than committing to one engine for everything.
Trust is measurable, and it compounds. Repeat-caller behavior is the most honest signal. Patients who had a good first experience came back willing to engage. Patients who didn't came back demanding a human. Your TTS choice has a retention effect.
What This Means for Builders
If you're building voice AI for healthcare or any high-trust domain, stop optimizing for how the voice sounds in your headphones. Optimize for how fast it responds when a nervous patient calls at 7am about a toothache. That's the test that matters.
We're considering publishing our full latency benchmarking methodology as an open-source tool (we teased this in a previous post). If there's interest, we'll prioritize it.
If you're building something similar, we'd love to hear about it. Reach out at hello@autor.ca or visit autor.ca.
Top comments (0)