DEV Community: Mart Schweiger

Voice Agents: Build Real-Time AI Phone Systems (2024)

Mart Schweiger — Wed, 29 Apr 2026 19:59:07 +0000

A recent survey of voice agent builders found that 82% feel confident in their technology—but 55% of end users are frustrated with interruptions, mishearing, and repetition. That gap between builder confidence and user reality gets wider in noisy environments: drive-thru lanes with engine noise and wind, contact centers with dozens of overlapping conversations, and field service calls where a technician is shouting over heavy machinery.

This article explains how the voice agent pipeline works, why noise is the hardest unsolved problem in production deployments, and what to consider when building voice agents for environments where clean audio is a luxury. It covers real-world use cases, the core build decision every team faces, and how features like noise suppression, configurable VAD, and dynamic mid-session settings change what's possible in these conditions.

What are voice agents?

A voice agent is an AI-powered system that holds real-time spoken conversations—it listens to what you say, understands it, and talks back. You can call a company, speak naturally ("I need to reschedule my appointment to Thursday"), and the system handles it without pressing a single button or waiting for a human.

That's a meaningful shift from the old way. Legacy IVR systems—Interactive Voice Response—forced you through rigid menus: "Press 1 for billing, press 2 for support." Voice agents skip all of that. They understand natural language, maintain context across a conversation, and take action.

The result is a system that can handle thousands of calls simultaneously, around the clock, without dropping in quality.

How do voice agents work?

Voice agents run on a three-stage pipeline. Each stage feeds directly into the next, which means a failure at any point breaks the whole conversation.

Here's the chain:

Speech-to-text (STT): Your voice is converted to text in real time
Large Language Model (LLM): The text is processed, intent is understood, and a response is generated
Text-to-speech (TTS): That response is converted back into spoken audio

Seems simple—but the details of each stage determine whether a voice agent feels natural or frustrating.

Speech-to-text

Speech-to-text is the first stage, and it's the most consequential. If the system mishears "fifteen" as "fifty," everything downstream is wrong—the LLM reasons from bad input, and the agent gives the user a nonsensical answer.

Most speech recognition models handle clean, studio-quality audio well. The hard part is real-world audio: background noise from a busy call center, regional accents, phone compression that strips audio quality, or a customer rattling off an alphanumeric order code. These conditions expose the gap between a model that scores well on benchmarks and one that actually works when deployed.

There's another variable most people overlook: turn detection. This is how the system knows you've finished speaking versus pausing mid-sentence. Poor turn detection is the most common reason voice agents feel unnatural—they either cut you off or sit in awkward silence. The best implementations handle turn detection at the model level, not as an afterthought bolted on separately. AssemblyAI's Universal-3 Pro Streaming model includes acoustic turn detection built directly into the model, designed for real-time voice agent pipelines where this precision matters.

Test real-time STT on your audio

Experience streaming transcription with model-level turn detection for natural turn-taking. Upload noisy, real-world audio and evaluate accuracy and latency before you build.

Try playground

Language model (LLM)

The LLM is the reasoning layer. Once the transcript arrives from the STT stage, the LLM figures out what the user wants, decides how to respond, and—in most deployments—calls external tools to take action.

Intent understanding: The LLM interprets what the user actually means, not just what they said literally
Tool calling: The LLM can query a database, book an appointment, or update a CRM record mid-conversation
Context management: The LLM tracks what was said earlier so the user doesn't have to repeat themselves

The LLM is only as effective as the transcript it receives. A single misrecognized word can completely change the intent—which is why teams that underinvest in STT quality often blame their LLM for errors that started one stage earlier.

Text-to-speech (TTS)

TTS converts the LLM's text response into audio the user hears. Two things matter most: how natural it sounds and how fast the first audio byte arrives.

The standard approach is sentence-chunked streaming—TTS starts generating audio as soon as the first sentence is ready, rather than waiting for the full response. This cuts perceived latency significantly and makes conversations feel more responsive.

Why noise changes everything

In a quiet office, most Voice AI pipelines work well enough. But production environments are rarely quiet. A drive-thru window has engine idling, wind, rain on the speaker box, and a customer shouting over traffic. A contact center floor has dozens of agents talking simultaneously, with phone compression stripping the audio down to 8kHz. A field technician might be diagnosing equipment while generators run in the background.

Noise doesn't just reduce transcription accuracy—it breaks the entire conversation loop. Background sounds trigger false voice activity detection (VAD) events, causing the agent to think the user is speaking when they're not. That leads to empty turn finalizations, premature interruptions, and the agent responding to nothing. The user experience feels broken even if the underlying STT model is accurate on the words it does capture.

This is why noise handling can't be an afterthought. The most effective approach addresses it at multiple levels: noise suppression on the audio input, VAD tuning calibrated for the specific environment, and turn detection logic that distinguishes between ambient sound and actual speech.

AssemblyAI's Universal-3 Pro Streaming model handles background noise well out of the box—internal testing shows it outperforms preprocessing pipelines, which often introduce artifacts that degrade accuracy. For extreme environments like drive-thrus, AssemblyAI also offers a hosted noise suppression endpoint with standard and aggressive modes, so teams can dial in the right level of filtering for their specific conditions without building a separate denoising pipeline.

What are voice agents used for?

The highest-value deployments share two traits: high call volume and repeatable, structured workflows. Voice agents aren't well-suited for highly ambiguous or emotionally complex interactions—but for predictable, high-frequency tasks, they're hard to beat.

Customer service and support

Customer service is where most organizations deploy voice agents first. The use case is handling tier-1 inquiries—order status, account questions, basic troubleshooting—without routing every call to a human agent.

The business logic is straightforward: if a customer asks "where's my package?" and the answer requires looking up a tracking number, a voice agent can do that in under two seconds at any hour of the day.

What separates good customer service voice agents from frustrating ones is the escalation path. When a conversation becomes too complex or emotionally charged, the agent needs to hand off to a human—with the full conversation context included. A handoff that makes the customer repeat everything they just said is a failed experience, full stop.

Sales and lead qualification

Outbound sales is another strong fit. Voice agents can call a list of prospects, ask qualification questions, and book meetings with human reps when someone meets the criteria. This works because the conversation is structured—there's a predictable flow that doesn't require deep reasoning.

The specific accuracy challenge in sales is proper noun recognition. Company names, people's names, and product titles need to be transcribed correctly for the CRM data to be useful. A lead record that says "works at Corel" instead of "works at Coral" creates downstream problems that are hard to catch and harder to fix. AssemblyAI's streaming speech-to-text is specifically optimized for entity accuracy—emails, phone numbers, and proper nouns—which is where most competing models break down.

Healthcare scheduling and intake

Healthcare is one of the most demanding environments for voice agents. The workflows—appointment scheduling, medication reminders, patient intake—are highly repeatable, which makes them a good structural fit. But the accuracy requirements are stricter than general customer service.

Drug names, dosage instructions, and medical condition names need to be transcribed precisely. AssemblyAI's Medical Mode is designed for this: it delivers significantly better accuracy on medical terminology including medications, procedures, conditions, and dosages, and is available in English, Spanish, German, and French.

Healthcare deployments also require careful handling of protected health information (PHI). AssemblyAI enables covered entities and their business associates subject to HIPAA to use AssemblyAI services to process PHI. AssemblyAI is considered a business associate under HIPAA and offers a Business Associate Addendum (BAA) required under HIPAA to ensure appropriate safeguarding of PHI.

Build voice agents for noisy, high-stakes environments

Get the speech accuracy foundation that drive-thru, contact center, and healthcare voice agents require. One API, $4.50/hr flat rate, with noise suppression and configurable VAD built in.

How do you build a voice agent?

The fundamental build decision is this: assemble your own multi-vendor stack or use a unified API that handles the pipeline for you.

The DIY multi-vendor approach means choosing a separate STT provider, LLM provider, and TTS provider, then writing the orchestration layer that connects them, handles errors, manages latency, and reconciles three separate billing systems. This gives you granular control but introduces significant complexity. Debugging is painful because a broken response could originate at any of the three layers.

The unified API approach collapses that complexity. One connection handles the full pipeline—one bill, one set of logs, one integration point. AssemblyAI's Voice Agent API is built on this model: a single WebSocket API that handles STT, LLM reasoning, and TTS for $4.50/hr flat—no token math, no separate input/output charges. It's built on Universal-3 Pro Streaming as the speech accuracy foundation, with turn detection, interruption handling, and tool calling included. The design philosophy is invisible infrastructure—you configure your agent's behavior, connect your backend tools, and build your product without managing the voice plumbing underneath. Your customers should feel like you built it from scratch.

The setup is minimal. Connect to the WebSocket, send a configuration message, and start streaming audio:‍

mport asyncio
import websockets
import json

API_KEY = "YOUR_ASSEMBLYAI_API_KEY"

async def run_agent():
    async with websockets.connect(
        "wss://agents.assemblyai.com/v1/ws",
        additional_headers={"Authorization": f"Bearer {API_KEY}"},
    ) as ws:
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "system_prompt": "You are a helpful customer service agent.",
                "output": {
                    "voice": "ivy"
                },
                "tools": [{
                    "type": "function",
                    "name": "check_order",
                    "description": "Look up order status by order ID",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "order_id": {"type": "string"}
                        },
                        "required": ["order_id"]
                    }
                }]
            }
        }))
        # Stream audio in, get audio back

Key elements you'll configure regardless of approach:

System prompt: Defines the agent's personality, scope, and behavior—and can be updated mid-conversation without reconnecting
Tool calling: Connects the agent to your backend systems—databases, CRMs, booking platforms—via JSON Schema
Voice selection: Chooses the TTS voice that fits your brand's tone, configurable under session.output.voice
Turn detection settings: Controls how the agent handles pauses and interruptions, including configurable VAD threshold, min/max turn silence, and interrupt duration

For noisy environments specifically, the Voice Agent API supports dynamic mid-session configuration via UpdateConfiguration. You can change VAD thresholds, turn silence timing, and domain vocabulary on the fly—increasing max_turn_silence to 3000ms while a customer dictates a credit card number, then dropping back to 1000ms for conversational turns. The keyterms_prompt parameter lets you pass up to 100 domain-specific terms (menu items, product names, medical terms) and update them dynamically as the conversation progresses through different stages.

Session resumption is another critical feature for unreliable environments: if the WebSocket drops—common in field deployments with spotty connectivity—you can reconnect within 30 seconds and pick up where the conversation left off, with full context preserved.

The Voice Agent API works with LiveKit and Pipecat for teams already using those orchestration frameworks. It also works well with Claude Code—simple enough that you can copy the docs, paste them in, and build a working agent the same afternoon.

What are the limitations of voice agents?

Voice agents are genuinely capable—but setting realistic expectations before you build matters. Data from AssemblyAI's Voice Agent Report paints a clear picture: while 82% of builders feel confident in their technology, 55% of end users report frustration with interruptions. The top user complaints—repetition, mishearing, and interruptions—are all symptoms of the same root cause: insufficient speech accuracy in real-world conditions. On the builder side, the top challenges are accuracy (50%), integration complexity (45%), and costs (42%).

Real-world accuracy is harder than benchmarks suggest: Clean test data doesn't reflect what happens when you add phone compression, background noise, or fast regional speech. Always evaluate with audio from your actual deployment environment—a model that scores well on LibriSpeech may struggle with your contact center audio.
Latency is a ceiling, not a floor: Total response time—from the user stopping speaking to the agent starting to respond—typically falls between one and two seconds for well-optimized pipelines. That feels natural for structured workflows but can feel slightly mechanical in rapid back-and-forth exchanges.
Complex reasoning requires guardrails: LLMs handle conversational language well but can struggle with strict procedural logic. Production systems often pair LLM-generated responses with deterministic logic for anything where precision is critical—payment confirmation, dosage instructions, legal disclosures.
User acceptance varies by context: Many users will happily interact with voice agents for routine tasks. But when topics become sensitive, the expectation shifts—low-friction escalation to a human isn't optional, it's required.

Building with these constraints in mind from day one produces significantly better outcomes than retrofitting solutions after launch.

Final words

Voice agents combine speech-to-text, a large language model, and text-to-speech into a pipeline that handles real spoken conversations at scale. Speech accuracy sets the ceiling for every stage that follows—and in noisy environments, that ceiling drops fast unless your infrastructure is built for it.

For teams building voice agents that need to work in drive-thrus, contact centers, and the field, AssemblyAI's Voice Agent API offers a unified infrastructure path—one WebSocket, one bill at $4.50/hr, and speech accuracy built on Universal-3 Pro Streaming with acoustic turn detection at the model level. Add in built-in noise suppression, configurable VAD, dynamic mid-session settings, and session resumption for unreliable connections, and you have infrastructure designed for the environments where voice agents are hardest to get right—and most valuable when they work.

Build voice agents that work in any environment

Use AssemblyAI's Voice Agent API to ship voice agents with built-in noise suppression, configurable VAD, and dynamic mid-session tuning. One API, $4.50/hr, working agent by end of day.

Frequently asked questions about voice agents

What is the difference between a voice agent and a chatbot?

A voice agent processes spoken audio in real time using a speech-to-text → LLM → text-to-speech pipeline, while a chatbot handles text-based input and output. The core difference is modality—voice introduces real-time audio processing and latency constraints that text-based systems don't face.

What makes a voice agent sound natural rather than robotic?

Naturalness comes from three things working well together: accurate speech recognition, contextually appropriate LLM responses, and TTS with human-like prosody. Turn detection is often the biggest factor—poor turn detection is why many agents feel abrupt or stilted, cutting users off mid-thought or sitting in dead silence.

How accurate does speech-to-text need to be for a voice agent to work well?

It depends on the use case—general customer service tolerates slightly higher word error rates, while medical or financial applications require near-perfect recognition of terminology like drug names and account numbers. The practical rule: test with audio from your actual deployment environment, not clean recordings.

What is the typical end-to-end latency for a voice agent?

End-to-end latency—from the user finishing speaking to the agent beginning to respond—typically falls between one and two seconds for well-optimized pipelines. Streaming architectures, where STT delivers partial transcripts and TTS starts generating audio before the full LLM response is ready, are the standard approach for keeping latency within that range.

Can voice agents handle multiple languages in a single conversation?

Yes—with the right underlying speech-to-text model. AssemblyAI's Universal-3 Pro Streaming covers six languages—English, Spanish, French, German, Italian, and Portuguese—with regional dialect recognition for higher accuracy within those languages. For broader language coverage, AssemblyAI's Universal-2 model supports over 99 languages with automatic language detection and code switching for speakers who move between languages mid-conversation.

What is a conversational AI voice agent?

A conversational AI voice agent is a voice agent capable of multi-turn dialogue—maintaining context across several exchanges rather than treating each utterance as a standalone query. This is what separates modern voice agents from older IVR systems, which reset context with every menu selection and had no memory of what you said ten seconds earlier.

How do voice agents handle noisy environments like drive-thrus or contact centers?

Noise handling requires a multi-layered approach: noise suppression on the audio input to clean the signal, VAD tuning calibrated for the specific environment to prevent false speech detections from background sounds, and turn detection logic that distinguishes between ambient noise and actual speech. The most effective systems handle noise at the model level rather than relying solely on client-side preprocessing, which can introduce artifacts that degrade transcription accuracy.

Build an Ambient AI Scribe for Telehealth in Python

Mart Schweiger — Wed, 29 Apr 2026 19:58:21 +0000

This tutorial walks through building an ambient AI scribe for telehealth visits in Python. AssemblyAI's Universal-3 Pro model with Medical Mode handles the accuracy requirements of clinical audio, including medications, procedures, and dosages. You'll need Python 3.8 or later, an AssemblyAI API key, and an OpenAI API key to follow along.

What is an ambient AI scribe?

An ambient AI scribe is software that listens to a patient-provider conversation, transcribes it, and turns it into a structured clinical note—automatically. This means the provider never has to type a single word during or after the visit.

It's worth distinguishing from two things it's often confused with:

Ambient dictation: The provider actively speaks to a recorder to narrate notes after the visit. An ambient scribe is passive—it listens to the natural conversation as it happens.
Human scribes: A person (in the room or remote) manually types notes in real time. An ambient AI scribe replaces this step entirely with software.

Every ambient AI scribe follows the same three-stage pipeline: audio capture → speech-to-text transcription → clinical note generation. That's exactly what you'll build here.

Why ambient AI scribes matter in clinical settings

Providers spend a significant portion of their workday on documentation rather than direct patient care—and telehealth compounds this problem since they're already managing a screen-based interaction.

Building an ambient scribe addresses three outcomes health systems care most about:

Less time on documentation: Providers reclaim time previously spent on after-hours charting and manual note entry.
More present patient interactions: With ambient documentation handling notes, providers make more eye contact and engage more naturally during visits.
Lower burnout: Administrative burden from EHR documentation is one of the leading drivers of clinician burnout, and ambient scribes reduce that load directly.

Telehealth makes ambient scribing even more practical—the visit is already recorded by the platform, so audio capture is a natural part of the existing workflow rather than an added step.

Build an ambient AI scribe in Python

Here's what you'll build, step by step:

Set up your environment
Transcribe the telehealth session audio
Separate patient and provider speech with speaker diarization
Generate a structured clinical note with an LLM

Set up your environment

You'll need Python 3.8 or later, an AssemblyAI API key, and an OpenAI API key.

Install the required packages:

pip install assemblyai openai

Then configure your AssemblyAI API key:‍

import assemblyai as aai

aai.settings.api_key = "YOUR_ASSEMBLYAI_API_KEY"

This tutorial assumes the telehealth session has already been recorded and saved as an audio file. MP3, WAV, and MP4 are all supported—these are the standard output formats from telehealth platforms like Zoom, Microsoft Teams, and Doxy.me. No video SDK integration is required.

Transcribe the telehealth session audio

Speech-to-text converts the telehealth recording into a transcript the LLM can work with. Submit your recorded audio file to AssemblyAI's transcription API like this:‍

config = aai.TranscriptionConfig(
    speech_models=["universal-3-pro"],
    domain="medical-v1",
    speaker_labels=True,
    keyterms_prompt=["metformin", "lisinopril", "hypertension", "SOAP note"]
)

transcriber = aai.Transcriber(config=config)
transcript = transcriber.transcribe("telehealth_session.mp3")
print(transcript.text)

Two configuration options here are specifically valuable for clinical audio:

speech_models=["universal-3-pro"] with domain="medical-v1": This selects AssemblyAI's Universal-3 Pro model with Medical Mode enabled—an add-on that improves recognition of medications, procedures, conditions, and dosages. Medical terminology is notoriously hard for general speech-to-text models to get right, so this matters.
keyterms_prompt: Keyterms prompting lets you pass a list of specialty-specific words—medication names, procedure names, lab values—that the model should pay close attention to. Universal-3 Pro supports up to 1,000 keyterms.

If you're building for a multilingual practice or need broader language coverage, Universal-2 supports 99 languages with keyterms prompting up to 200 words.

Model	Best for	Languages	Keyterms limit
Universal-3 Pro	Highest accuracy, medical terminology	6 (EN, ES, DE, FR, PT, IT)	1,000
Universal-2	Broad language coverage	99	200

Test clinical transcription with diarization

Upload a sample telehealth recording and see Universal models separate patient and provider speech while capturing clinical terms with high accuracy—all in your browser.

Open playground

Separate patient and provider speech with speaker diarization

Speaker diarization identifies who said what in a conversation—so the transcript gets labeled by speaker rather than appearing as one undifferentiated block of text. This is what makes the LLM's job possible: it needs to know which lines are the provider's observations and which are the patient's complaints.

Setting speaker_labels=True in the config (which you already did above) enables this automatically. Access the labeled transcript through the utterances property:

for u in transcript.utterances:
    print(f"{'Provider' if u.speaker == 'A' else 'Patient'}: {u.text}")

AssemblyAI labels the first detected speaker as "A." In a telehealth session, this is typically the provider who opens the call. You can confirm this by checking the first utterance, or set it manually if your practice has a consistent call structure.

Here's what the diarized output looks like:

Provider: Good morning. What brings you in today?
Patient: I've been having chest tightness for about three days.
Provider: Is it worse with exertion or at rest?
Patient: It gets worse when I climb stairs or walk quickly.
Provider: Any shortness of breath or pain radiating to your arm?
Patient: No, just the tightness in my chest.

This structured, speaker-labeled transcript is what you'll pass to the LLM in the next step.

Generate a structured clinical note with an LLM

Now you'll use an LLM to read the diarized transcript and generate a SOAP note—a standard clinical documentation format that organizes information into four sections: Subjective, Objective, Assessment, and Plan.‍

from openai import OpenAI

client = OpenAI(api_key="YOUR_OPENAI_API_KEY")

conversation = "\n".join(
    f"{'Provider' if u.speaker == 'A' else 'Patient'}: {u.text}"
    for u in transcript.utterances
)

response = client.chat.completions.create(
    model="gpt-4",
    messages=[
        {
            "role": "system",
            "content": "You are a clinical documentation assistant. Generate accurate SOAP notes from patient-provider conversation transcripts. Be concise and use standard clinical language."
        },
        {
            "role": "user",
            "content": f"Generate a SOAP note from this telehealth visit transcript:\n\n{conversation}"
        }
    ]
)

soap_note = response.choices[0].message.content
print(soap_note)

Here's what the output looks like for the example conversation above:‍

SUBJECTIVE:
Patient is a [age] presenting with chest tightness for 3 days.
Symptoms worsen with exertion (climbing stairs, walking quickly).
Denies shortness of breath, radiation to arm, nausea, or diaphoresis.

OBJECTIVE:
Telehealth visit. Patient appears comfortable at rest. No visible distress.

ASSESSMENT:
Exertional chest tightness. Differential includes stable angina,
musculoskeletal pain, or anxiety-related chest symptoms.

PLAN:
1. Order ECG and troponin levels
2. Start aspirin 81mg daily pending workup
3. Follow up in person within 48 hours
4. Patient advised to go to ED if symptoms worsen or change in character

Alternative: Use AssemblyAI's LLM Gateway instead of OpenAI. LLM Gateway provides access to 20+ models—including Claude, GPT, Gemini, and more—through an OpenAI-compatible chat completions API. The advantage for clinical workflows is keeping your entire pipeline on one vendor: transcription, diarization, and LLM-powered note generation all through AssemblyAI, with one bill and one set of logs. The endpoint is llm-gateway.assemblyai.com/v1/chat/completions, so switching from the OpenAI SDK requires only changing the base URL and API key.

SOAP is the most common format, but you can adapt the system prompt for other clinical note types depending on your specialty:

Format	Description	When to use it
SOAP	Subjective, Objective, Assessment, Plan	General medical visits
DAP	Data, Assessment, Plan	Mental health sessions
Narrative	Free-text summary	Complex or multisystem cases
After-visit summary	Patient-friendly summary	Patient portal delivery

Start building your telehealth scribe

Get an API key to transcribe visits with Universal-3 Pro and speaker diarization, then feed labeled transcripts into your LLM to generate SOAP notes.

Get API key

Privacy and compliance requirements for clinical ambient scribes

Any ambient AI scribe handling telehealth recordings must meet three requirements before it goes anywhere near a real patient visit.

Patient consent: Patients must know the visit is being recorded and transcribed, and must be able to opt out. In telehealth, this is typically a verbal confirmation at the start of the call, documented in the patient record.
Provider review and sign-off: AI-generated notes are drafts, not final records. The provider reviews, edits for accuracy, and signs before the note enters the EHR.
HIPAA and data handling: Telehealth recordings and transcripts contain protected health information (PHI). AssemblyAI enables covered entities and their business associates subject to HIPAA to use AssemblyAI services to process PHI—AssemblyAI is considered a business associate under HIPAA and offers a Business Associate Addendum (BAA) that is required under HIPAA to ensure AssemblyAI appropriately safeguards PHI. Any LLM provider you use for note generation must also support BAA execution.

What you've built

You now have a working ambient AI scribe pipeline: a recorded telehealth session goes in, a speaker-labeled transcript comes out, and a structured clinical note gets generated—ready for provider review.

AssemblyAI's Universal-3 Pro model, Medical Mode, and speaker diarization handle the hardest parts of clinical Voice AI: getting medical terminology right, telling patient from provider, and doing it accurately enough that the generated note is actually useful. Teams building on this foundation can extend it to add post-call analytics, patient sentiment analysis, or automated after-visit summaries using AssemblyAI's LLM Gateway for applying LLMs to transcribed audio.

What about real-time ambient scribing? This tutorial processes recorded audio after the visit—and that's how most production ambient scribes work today. But if you're building toward live, in-visit documentation where the scribe generates notes as the conversation happens, AssemblyAI's Voice Agent API is worth exploring. It provides a single WebSocket connection that handles speech understanding, LLM reasoning, and voice generation at a $4.50/hr flat rate—one API, one bill, no multi-vendor orchestration. The same Universal-3 Pro foundation powers the speech understanding layer, so the medical terminology accuracy you get in this tutorial carries over. It's designed as invisible infrastructure: you configure your agent's behavior and build your product without managing the voice plumbing underneath.

Extend your ambient AI scribe solution

Create an account to access Universal-3 Pro and speaker diarization used throughout this tutorial, plus sample code to accelerate integration.

Start now

Frequently asked questions

What is the difference between an ambient AI scribe and traditional medical dictation?

Traditional medical dictation requires the provider to speak directly to a recorder after the visit to narrate notes—it's an active, intentional step that still takes time. An ambient AI scribe passively listens to the natural patient-provider conversation and generates documentation automatically, without the provider changing their behavior at all.

How accurate is speech-to-text transcription for medical terminology?

General-purpose speech-to-text models often misrecognize medications, procedures, and clinical shorthand because these terms rarely appear in general training data. AssemblyAI's Universal-3 Pro model with Medical Mode is specifically optimized for clinical vocabulary, and keyterms prompting (up to 1,000 terms) lets you further tune recognition for your specialty.

Does an ambient AI scribe built with this tutorial need to integrate with an EHR?

No—the scribe generates a structured text note that providers can copy into any EHR manually. Direct EHR integration via APIs like Epic's FHIR is a common next step for production deployments, but it's outside the scope of this tutorial.

What audio file formats does AssemblyAI's transcription API accept?

AssemblyAI accepts all common audio and video formats including MP3, MP4, WAV, M4A, FLAC, and WebM—which covers the export formats of every major telehealth platform. You can submit files as a local file path or a publicly accessible URL.

Do ambient AI scribes need to process audio in real time during the telehealth visit?

Not necessarily—and for most clinical workflows you wouldn't want them to. The async approach in this tutorial processes the recorded audio file after the visit concludes, which is how most ambient scribe products work in production. For teams that do want live, in-visit transcription and note generation, AssemblyAI's Voice Agent API provides a single WebSocket that handles speech understanding, LLM reasoning, and voice generation in one connection—no separate STT, LLM, and TTS providers to stitch together. That said, real-time streaming adds complexity and latency tradeoffs, so evaluate whether your use case truly requires it or if post-visit processing is the better fit.

Voice Agent Architecture: Build STT-LLM-TTS Pipeline

Mart Schweiger — Wed, 29 Apr 2026 19:56:49 +0000

Voice agents are transforming how businesses interact with customers, automate workflows, and build conversational interfaces. At their core, most voice agents follow a chained architecture: speech-to-text (STT) converts audio to text, a large language model (LLM) generates a response, and text-to-speech (TTS) converts that response back to audio. This STT‑LLM‑TTS pipeline is the most common pattern for building real-time voice agents today.

In this guide, we'll walk through every component of a voice agent architecture, show you how to wire them together with working Python code, discuss performance requirements, and help you decide whether to build or buy your voice agent pipeline. Whether you're building a customer support bot, a virtual receptionist, or an AI-powered phone agent, this article gives you the foundation to get started.

What are the core components of voice agent architecture?

A voice agent pipeline has four core components. Each one plays a distinct role in converting spoken input into a spoken response.


Component	Role	Example providers
Speech-to-text (STT)	Converts spoken audio into text in real time	AssemblyAI, Google Cloud STT, Whisper
Large language model (LLM)	Processes the transcribed text and generates an intelligent response	Anthropic Claude, OpenAI GPT, Google Gemini
Text-to-speech (TTS)	Converts the LLM's text response into natural-sounding audio	ElevenLabs, Amazon Polly, Google Cloud TTS
Orchestration layer	Manages data flow, turn detection, interruption handling, and error recovery	Custom code, LiveKit, AssemblyAI Voice Agent API

Speech-to-text

Streaming speech-to-text is the entry point of any voice agent. It takes raw audio from a microphone or phone call and produces a text transcript in real time. For voice agents, you need a streaming STT provider that supports WebSocket connections and delivers transcripts with minimal latency.

AssemblyAI's streaming STT uses the Universal-3 Pro Streaming speech model, which delivers state-of-the-art accuracy for real-time transcription. Here's how to connect using the v3 streaming API. Note that the SDK invokes each handler with two arguments — the StreamingClient instance and the event — so every callback's first parameter is client:

from assemblyai.streaming.v3 import (
    BeginEvent,
    StreamingClient,
    StreamingClientOptions,
    StreamingError,
    StreamingEvents,
    StreamingParameters,
    TurnEvent,
    TerminationEvent,
)

def on_begin(client: StreamingClient, event: BeginEvent):
    """Called when the streaming session begins."""
    print(f"Session started — ID: {event.id}")

def on_turn(client: StreamingClient, event: TurnEvent):
    """Called on each turn update. Check end_of_turn for final transcripts."""
    if event.end_of_turn:
        print(f"Final transcript: {event.transcript}")
    else:
        print(f"Partial: {event.transcript}", end="\r")

def on_error(client: StreamingClient, error: StreamingError):
    print(f"Error: {error}")

def on_terminated(client: StreamingClient, event: TerminationEvent):
    print("Session terminated")

client = StreamingClient(
    StreamingClientOptions(
        api_key="YOUR_ASSEMBLYAI_KEY",
        api_host="streaming.assemblyai.com",
    )
)

client.on(StreamingEvents.Begin, on_begin)
client.on(StreamingEvents.Turn, on_turn)
client.on(StreamingEvents.Termination, on_terminated)
client.on(StreamingEvents.Error, on_error)

client.connect(
    StreamingParameters(
        speech_model="u3-rt-pro",
        sample_rate=16000,
    )
)

The v3 streaming API connects over wss://streaming.assemblyai.com/v3/ws and uses a turn-based event model. The TurnEvent provides both partial and final transcripts—check event.end_of_turn to determine when a complete utterance has been captured. This is critical for voice agents because you only want to send complete utterances to the LLM.

Get started with streaming STT

Sign up for a free AssemblyAI API key to start building with real-time streaming speech-to-text powered by Universal-3 Pro..

Get free API key

Large language models

The LLM is the "brain" of your voice agent. Once you have a transcript from the STT component, you send it to an LLM to generate an appropriate response. For voice agents, you want a model that balances speed with quality—a fast model that produces concise, conversational responses works better than a large model that generates lengthy, verbose output.

Here's an example using Anthropic's Claude as the LLM in a voice agent pipeline:

import anthropic

client = anthropic.Anthropic(api_key="YOUR_ANTHROPIC_KEY")
conversation_history = []

SYSTEM_PROMPT = """You are a helpful voice assistant. Keep your responses
concise and conversational—aim for 1-3 sentences. Avoid bullet points,
markdown formatting, or lengthy explanations. Respond as if you're
having a natural phone conversation."""

def get_llm_response(user_text: str) -> str:
    """Send transcribed text to Claude and get a response."""
    conversation_history.append({
        "role": "user",
        "content": user_text
    })

    response = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=150,
        system=SYSTEM_PROMPT,
        messages=conversation_history
    )

    assistant_text = response.content[0].text
    conversation_history.append({
        "role": "assistant",
        "content": assistant_text
    })

    return assistant_text

A few best practices for using LLMs in voice agents:

Keep the system prompt voice-optimized. Instruct the model to be concise, avoid formatting, and respond conversationally.
Limit max tokens. For voice, shorter responses feel more natural. Aim for 50–150 tokens.
Use streaming responses. Stream the LLM output token by token so you can start TTS before the full response is generated.
Maintain conversation history. Pass the full conversation context so the LLM can handle multi-turn dialogues.

Text-to-speech

The final stage of the pipeline converts the LLM's text response into audio that the user hears. Modern TTS systems produce remarkably natural speech, and many offer streaming output so audio playback can begin before the full response is synthesized.

Here's a comparison of popular TTS providers for voice agents:


Provider	Streaming support	Voice cloning	Latency
ElevenLabs	Yes	Yes	~300 ms
Amazon Polly	Yes	No	~200 ms
Google Cloud TTS	Yes	Limited	~250 ms
OpenAI TTS	Yes	No	~300 ms

Here's a basic example of streaming TTS with ElevenLabs:

import requests

def text_to_speech_stream(text: str, voice_id: str = "21m00Tcm4TlvDq8ikWAM"):
    """Convert text to speech using ElevenLabs streaming API."""
    url = f"https://api.elevenlabs.io/v1/text-to-speech/{voice_id}/stream"

    headers = {
        "xi-api-key": "YOUR_ELEVENLABS_KEY",
        "Content-Type": "application/json"
    }

    payload = {
        "text": text,
        "model_id": "eleven_turbo_v2",
        "voice_settings": {
            "stability": 0.5,
            "similarity_boost": 0.75
        }
    }

    response = requests.post(url, json=payload, headers=headers, stream=True)

    for chunk in response.iter_content(chunk_size=1024):
        if chunk:
            yield chunk  # Stream audio chunks for playback

Orchestration

The orchestration layer is the glue that holds everything together. It handles:

Turn detection: Knowing when the user has finished speaking so you can trigger the LLM
Barge-in / interruption handling: Allowing the user to interrupt the agent mid-response
Error recovery: Handling dropped connections, timeouts, and API failures gracefully
Context management: Maintaining conversation state across turns
Audio routing: Managing audio input/output streams, mixing, and playback

Orchestration is often the hardest part to get right. Turn detection alone—deciding when a pause means "I'm done talking" versus "I'm thinking"—requires careful tuning. This is one of the key reasons many teams choose a managed solution rather than building from scratch.

Architecture patterns for voice agents

There are two primary patterns for wiring STT, LLM, and TTS together in a voice agent: the cascading pipeline and the streaming architecture. The right choice depends on your latency requirements and implementation complexity tolerance.


Pattern	How it works	Latency	Complexity
Cascading pipeline	Each stage completes fully before the next begins	Higher (sum of all stages)	Low
Streaming architecture	Stages overlap—LLM streams tokens to TTS as they arrive	Lower (stages run in parallel)	High

Cascading pipeline

In a cascading pipeline, each stage completes before the next one starts. The user speaks, the STT produces a full transcript, the LLM generates a complete response, and then TTS converts the entire response to audio.

This is the simplest pattern to implement, but it has a significant latency penalty. If STT takes 500 ms, the LLM takes 1 second, and TTS takes 400 ms, the user waits 1.9 seconds before hearing any audio. For casual applications, this may be acceptable—but for real-time voice agents, it feels sluggish.

The snippet below illustrates the shape of a cascading pipeline. speech_to_text, get_llm_response (defined earlier), and text_to_speech are placeholders — substitute the streaming-STT, Claude, and ElevenLabs implementations from the previous sections to make it runnable end-to-end:

import time

# Placeholder helpers — replace with real implementations:
# - speech_to_text: blocking call to your STT provider
# - get_llm_response: defined in the LLM section above
# - text_to_speech: blocking call to your TTS provider

def speech_to_text(audio_data: bytes) -> str: ...
def text_to_speech(text: str) -> bytes: ...

def cascading_pipeline(audio_data: bytes) -> bytes:
    """Simple cascading pipeline — each stage runs sequentially."""

    # Stage 1: STT — wait for full transcript
    start = time.time()
    transcript = speech_to_text(audio_data)
    stt_time = time.time() - start
    print(f"STT completed in {stt_time:.2f}s: {transcript}")

    # Stage 2: LLM — wait for full response
    start = time.time()
    response_text = get_llm_response(transcript)
    llm_time = time.time() - start
    print(f"LLM completed in {llm_time:.2f}s: {response_text}")

    # Stage 3: TTS — wait for full audio
    start = time.time()
    audio_output = text_to_speech(response_text)
    tts_time = time.time() - start
    print(f"TTS completed in {tts_time:.2f}s")

    total = stt_time + llm_time + tts_time
    print(f"Total pipeline latency: {total:.2f}s")

    return audio_output

Streaming architecture

The streaming architecture overlaps stages to minimize time-to-first-audio. As soon as the STT delivers a final transcript, the LLM begins streaming tokens. As tokens arrive, they're buffered into sentence fragments and sent to TTS immediately. The user starts hearing audio while the LLM is still generating the rest of the response.

Here's a complete streaming pipeline using AssemblyAI's v3 streaming API for STT, Claude for the LLM, and ElevenLabs for TTS. Note that play_audio is left as a stub — wire it to your platform's audio output (PyAudio, sounddevice, a phone-bridge SDK, etc.):

import asyncio
import anthropic
import requests
from assemblyai.streaming.v3 import (
    BeginEvent, StreamingClient, StreamingClientOptions,
    StreamingError, StreamingEvents, StreamingParameters,
    TurnEvent, TerminationEvent,
)

# --- Configuration ---
ASSEMBLYAI_API_KEY = "YOUR_ASSEMBLYAI_KEY"
ANTHROPIC_API_KEY  = "YOUR_ANTHROPIC_KEY"
ELEVENLABS_API_KEY = "YOUR_ELEVENLABS_KEY"
ELEVENLABS_VOICE_ID = "21m00Tcm4TlvDq8ikWAM"

# --- LLM ---
anthropic_client = anthropic.Anthropic(api_key=ANTHROPIC_API_KEY)
conversation_history = []

SYSTEM_PROMPT = """You are a helpful voice assistant. Keep your responses
concise and conversational—aim for 1-3 sentences. Avoid bullet points,
markdown formatting, or lengthy explanations."""

def stream_llm_response(user_text: str):
    """Stream tokens from Claude for low-latency TTS handoff."""
    conversation_history.append({"role": "user", "content": user_text})

    with anthropic_client.messages.stream(
        model="claude-haiku-4-5-20251001",
        max_tokens=150,
        system=SYSTEM_PROMPT,
        messages=conversation_history
    ) as stream:
        full_response = ""
        sentence_buffer = ""

        for text in stream.text_stream:
            full_response += text
            sentence_buffer += text

            # Flush buffer at sentence boundaries for TTS
            if any(sentence_buffer.rstrip().endswith(p)
                   for p in [".", "!", "?"]):
                yield sentence_buffer
                sentence_buffer = ""

        # Flush remaining text
        if sentence_buffer.strip():
            yield sentence_buffer

    conversation_history.append({
        "role": "assistant",
        "content": full_response
    })

# --- TTS ---
def stream_tts(text_chunk: str):
    """Send a text chunk to ElevenLabs and stream back audio."""
    url = f"https://api.elevenlabs.io/v1/text-to-speech/{ELEVENLABS_VOICE_ID}/stream"
    headers = {
        "xi-api-key": ELEVENLABS_API_KEY,
        "Content-Type": "application/json"
    }
    payload = {
        "text": text_chunk,
        "model_id": "eleven_turbo_v2",
        "voice_settings": {"stability": 0.5, "similarity_boost": 0.75}
    }

    response = requests.post(url, json=payload, headers=headers, stream=True)
    for chunk in response.iter_content(chunk_size=1024):
        if chunk:
            play_audio(chunk)

def play_audio(chunk: bytes):
    """Placeholder — replace with actual audio output."""
    pass

# --- Orchestration ---
def handle_final_transcript(transcript: str):
    """Called when STT delivers a final transcript. Streams LLM -> TTS."""
    print(f"User said: {transcript}")
    for sentence in stream_llm_response(transcript):
        print(f"Agent says: {sentence}")
        stream_tts(sentence)

# --- STT with v3 streaming API ---
def on_begin(client: StreamingClient, event: BeginEvent):
    print(f"Streaming session started — ID: {event.id}")

def on_turn(client: StreamingClient, event: TurnEvent):
    if event.end_of_turn:
        handle_final_transcript(event.transcript)
    else:
        print(f"Partial: {event.transcript}", end="\r")

def on_error(client: StreamingClient, error: StreamingError):
    print(f"STT error: {error}")

def on_terminated(client: StreamingClient, event: TerminationEvent):
    print("STT session terminated")

def main():
    client = StreamingClient(
        StreamingClientOptions(
            api_key=ASSEMBLYAI_API_KEY,
            api_host="streaming.assemblyai.com",
        )
    )
    client.on(StreamingEvents.Begin, on_begin)
    client.on(StreamingEvents.Turn, on_turn)
    client.on(StreamingEvents.Termination, on_terminated)
    client.on(StreamingEvents.Error, on_error)

    client.connect(
        StreamingParameters(
            speech_model="u3-rt-pro",
            sample_rate=16000,
        )
    )

if __name__ == "__main__":
    main()

In this streaming architecture, the key optimization is at the sentence boundary. Instead of waiting for the full LLM response, we flush each sentence to TTS as soon as it's complete. This means the user hears the first sentence of the response while the LLM is still generating the second sentence—dramatically reducing perceived latency.

Try the Voice Agent API playground

Skip the complexity of wiring three services together. AssemblyAI's Voice Agent API handles STT, LLM, and TTS in a single WebSocket connection.

Try playground

Performance requirements for real-time voice agents

Latency is the single most important performance metric for voice agents. Users expect conversational AI to feel like a natural conversation, and research shows that response delays beyond 500–700 ms start to feel unnatural. Here's a breakdown of typical latency budgets for each component:


Component	Target latency	What impacts it
STT (time to final transcript)	200–500 ms	Model size, network latency, audio chunk size
LLM (time to first token)	150–400 ms	Model size, prompt length, provider infrastructure
TTS (time to first audio)	200–400 ms	Voice model, text length, streaming support
Network overhead	50–150 ms	Geographic distance, connection type, edge routing

Latency budget

In a streaming architecture, the total time from "user stops speaking" to "user hears first audio" is approximately: STT finalization + LLM time-to-first-token + TTS time-to-first-audio + network overhead. With well-optimized providers, this can be as low as 600–900 ms—fast enough to feel conversational.

To stay within this budget, consider the following optimizations:

Use streaming everywhere. Stream STT transcripts, LLM tokens, and TTS audio. Never wait for a full response when you can start processing partial output.
Choose fast models. Smaller, faster LLMs like Claude Haiku outperform larger models for voice agents because the speed improvement matters more than marginal quality gains.
Minimize network hops. Keep your services in the same region when possible. AssemblyAI offers edge routing that automatically directs requests to the nearest compute region, reducing network latency for geographically distributed users.
Pre-warm connections. Establish WebSocket connections before the user starts speaking to eliminate connection setup latency.
Buffer intelligently. Send text to TTS in sentence-sized chunks rather than word-by-word (too many requests) or all-at-once (too much delay).

For entity-heavy use cases like a customer intake agent (e.g., "I need to reach Sarah Kowalczyk at Acme Corp"), STT accuracy is especially critical. A single transcription error in a name or company can break downstream processes. This is where investing in a high-accuracy STT provider like AssemblyAI pays dividends.

Build vs. buy

One of the biggest decisions you'll face is whether to build your own STT-LLM-TTS pipeline or use a managed Voice Agent API. Here's how the two approaches compare:


Factor	Build your own pipeline	AssemblyAI Voice Agent API
Setup time	Days to weeks—integrate 3+ providers, handle auth, manage connections	Minutes—single WebSocket connection handles everything
Cost	Variable—pay each provider separately, costs scale unpredictably	$4.50/hr flat rate covering STT + LLM + TTS
Orchestration	You build turn detection, interruption handling, error recovery	Built-in turn detection, barge-in, and error recovery
Tool calling	Custom implementation with your LLM provider	Built-in function calling with tool definitions
Reliability	You manage failover across multiple providers	Session resumption—reconnect within 30 seconds without losing context
Flexibility	Full control over every component and model choice	Configurable via JSON—choose voice, system prompt, tools

When to build your own: If you need full control over every component, want to use specific providers for each stage, or have unique latency requirements that demand custom optimization, building your own pipeline makes sense. The streaming architecture example earlier in this article gives you a solid starting point.

When to use the Voice Agent API: If you want to ship fast, avoid managing three separate provider integrations, and prefer predictable pricing, the Voice Agent API is the better choice. It uses a standard WebSocket + JSON API—no SDK required. Connect to wss://agents.assemblyai.com/v1/ws, send a session.update message with your configuration, and start streaming audio.

Here's how to connect to the Voice Agent API and configure a session:

import asyncio
import websockets
import json
import os

async def voice_agent():
    uri = "wss://agents.assemblyai.com/v1/ws"
    headers = {"Authorization": f"Bearer {os.environ['ASSEMBLYAI_API_KEY']}"}

    async with websockets.connect(uri, additional_headers=headers) as ws:
        # Configure session — STT, LLM, and TTS handled automatically
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "system_prompt": "You are a helpful customer support agent.",
                "greeting": "Hi! How can I help you today?",
                "output": {"voice": "ivy"},
                "tools": [{
                    "type": "function",
                    "name": "get_account_info",
                    "description": "Look up account status",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "account_id": {"type": "string"}
                        },
                        "required": ["account_id"]
                    }
                }]
            }
        }))

        async for message in ws:
            event = json.loads(message)
            if event["type"] == "session.ready":
                print(f"Ready — session {event['session_id']}")
                # Start streaming audio with input.audio events
                break

asyncio.run(voice_agent())

With the Voice Agent API, you define your system prompt, greeting, voice, and tools in a single JSON message. The API handles STT, LLM inference, TTS, turn detection, interruption handling, and tool execution—all through one WebSocket connection. Session resumption lets you reconnect within 30 seconds without losing conversation context, which is critical for production deployments over unreliable networks.

Final words

Building a voice agent with a chained STT-LLM-TTS architecture is a well-understood pattern, but getting it to production quality requires careful attention to latency, orchestration, and reliability. The streaming architecture—where STT, LLM, and TTS stages overlap—is the key to achieving conversational response times.

For teams who want the full pipeline handled, AssemblyAI's Voice Agent API provides a single WebSocket connection at $4.50/hr that replaces three separate providers, with built-in orchestration, turn detection, tool calling, and interruption handling included. No SDK required—connect to wss://agents.assemblyai.com/v1/ws, send JSON, and ship.

If you prefer full control, the code examples in this guide give you a working foundation to build your own pipeline with AssemblyAI's v3 streaming STT, Claude, and ElevenLabs. Either way, the core principles are the same: stream everything, minimize latency at every stage, and invest in a robust orchestration layer.

Start building voice agents today

Sign up for a free AssemblyAI account and build your first voice agent—whether you use the chained STT-LLM-TTS pipeline or the Voice Agent API.

Get free API key

Frequently asked questions

What is a chained STT-LLM-TTS architecture?

A chained STT-LLM-TTS architecture is a voice agent pipeline where speech-to-text converts audio to text, a large language model generates a response, and text-to-speech converts that response back to audio. These three stages run in sequence (or overlapping in a streaming architecture) to enable real-time voice conversations. It is the most common pattern for building voice agents today because it lets you choose best-in-class providers for each stage independently.

How do I reduce latency in a voice agent?

The most effective way to reduce latency is to use a streaming architecture where STT, LLM, and TTS stages overlap. Stream LLM tokens to TTS in sentence-sized chunks so the user hears audio while the model is still generating. Additionally, choose fast models (e.g., Claude Haiku for LLM), pre-warm WebSocket connections, and use providers with edge routing to minimize network latency. A well-optimized streaming pipeline can achieve 600–900 ms time-to-first-audio.

What is AssemblyAI's Voice Agent API?

The Voice Agent API is a managed service that handles the entire STT-LLM-TTS pipeline through a single WebSocket connection at wss://agents.assemblyai.com/v1/ws. It costs $4.50/hr flat and includes built-in turn detection, barge-in handling, tool calling, and session resumption. No SDK is required—you connect with standard WebSockets and send JSON messages to configure your agent.

Should I build my own voice agent pipeline or use a managed API?

Build your own if you need full control over every component, want to use specific providers for each stage, or have unique requirements that demand custom optimization. Use a managed API like AssemblyAI's Voice Agent API if you want to ship faster, avoid integrating three separate services, and prefer predictable pricing at $4.50/hr. Most teams start with the managed API for speed and only move to a custom pipeline when they hit specific limitations.

What speech-to-text model should I use for voice agents?

For real-time voice agents, you need a streaming STT model with low latency and high accuracy. AssemblyAI's Universal-3 Pro Streaming model (u3-rt-pro) is purpose-built for real-time applications and delivers state-of-the-art accuracy via the v3 streaming API at wss://streaming.assemblyai.com/v3/ws. Accuracy is especially important for entity-heavy use cases like names, addresses, and account numbers where a single transcription error can break downstream processes.

How does turn detection work in voice agents?

Turn detection determines when a user has finished speaking so the agent can begin responding. It typically combines voice activity detection (VAD) with silence thresholds—if the user stops speaking for a configured duration (usually 500–800 ms), the system treats it as a completed turn. Advanced implementations also use semantic cues from partial transcripts. AssemblyAI's v3 streaming API provides turn-level events via TurnEvent with an end_of_turn flag, simplifying this for developers.

OpenAI Realtime API Migration: Complete Guide

Mart Schweiger — Wed, 29 Apr 2026 19:56:46 +0000

This guide walks you through migrating a voice agent from the OpenAI Realtime API to AssemblyAI's Voice Agent API. You'll replace manual session management, audio buffer handling, and ephemeral token generation with a cleaner WebSocket interface that manages the full voice pipeline through a single connection.

The migration covers four areas: authentication setup, session configuration, audio streaming, and tool migration. Each section shows the OpenAI implementation alongside the AssemblyAI equivalent so you can see exactly what changes and what stays the same. You'll need Python 3.8+, an AssemblyAI API key, and working familiarity with WebSockets and async Python. Your existing business logic and function definitions transfer directly.

What is the OpenAI Realtime API?

The OpenAI Realtime API is a WebSocket-based interface that lets you build using OpenAI's audio models—specifically gpt-realtime and gpt-realtime-mini. Instead of chaining together separate speech-to-text, LLM, and text-to-speech services, it processes audio directly as a single multimodal input and output.

That word "multimodal" is important. It means the same underlying model handles audio, text, and images—voice is one capability among many, not the primary focus.

This distinction matters when you hit production. A model designed to do everything doesn't always do any one thing as well as a model built specifically for it. Speech accuracy is where you'll feel that most.

How the OpenAI Realtime API works: sessions and events

The Realtime API is event-driven. You and the server exchange JSON messages over a persistent WebSocket connection, and a "session" holds the state of your entire conversation.

Think of a session like a phone call—it stays open, remembers what was said, and closes when you hang up. Every action you take sends a specific event type over that connection.

Here are the core events you'll work with:

Event name	Direction	What it does
`session.update`	client → server	Configures session settings like instructions and voice
`session.created`	server → client	Confirms the session is ready with its default config
`input_audio_buffer.append`	client → server	Sends a chunk of audio to the model's input buffer
`input_audio_buffer.commit`	client → server	Finalizes the buffer so the model can process it
`response.create`	client → server	Tells the model to generate a response
`response.output_audio.delta`	server → client	Streams audio response chunks back to you
`response.done`	server → client	Signals the response is complete

Two more concepts you'll need to understand before writing any code:

Voice activity detection (VAD): This is how the model knows when you've stopped talking. You can use server_vad (silence-based) or semantic_vad (content-based, meaning it waits for a natural pause in meaning rather than just silence).
Function calling: The model can invoke tools you've registered—like looking up a customer record—before generating its spoken response.

Here's the minimal Python code to open a connection and configure a session:

import asyncio
import websockets
import json
import os
from dotenv import load_dotenv

load_dotenv()

async def connect():
    uri = "wss://api.openai.com/v1/realtime?model=gpt-realtime"
    headers = {"Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}"}

    async with websockets.connect(uri, extra_headers=headers) as ws:
        # Configure the session
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "type": "realtime",
                "instructions": "You are a helpful assistant.",
                "audio": {
                    "input": {
                        "turn_detection": {"type": "server_vad"}
                    },
                    "output": {"voice": "alloy"}
                }
            }
        }))

        # Listen for the server to confirm
        async for message in ws:
            event = json.loads(message)
            print(f"Received: {event['type']}")
            if event["type"] == "session.created":
                print("Session ready.")
                break

asyncio.run(connect())

‍This is the foundation every OpenAI Realtime API integration starts with. Everything else—sending audio, handling responses, calling tools—builds on this session.

OpenAI Realtime API production challenges

OpenAI Realtime API works well for prototypes. But when you move to production, three problems compound quickly.

Token-based pricing is hard to predict. Audio tokens are billed per million tokens for input and output separately. The count changes based on speech pace, pauses, and conversation length—so your monthly bill is a moving target, not a fixed cost.

Session management requires significant boilerplate. In production, you need to handle concurrent event streams, manage reconnections after network interruptions, enforce the 30-minute session timeout, and write retry logic. All of that exists before you write a single line of product logic.

Concurrency limits require quota negotiations. Session limits are tied to account tiers. As your user base grows, you'll need to request higher limits from OpenAI and architect around those ceilings from the start.

AssemblyAI's Voice Agent API was built specifically for these production constraints—one WebSocket, all-inclusive flat-rate pricing at $4.50/hr for the complete voice agent service, and auto-scaling concurrency address each of these friction points directly.

Start with your AssemblyAI API key

Authenticate once with a single key—no ephemeral tokens or session boilerplate. Create your account to begin migrating your agent.

What is AssemblyAI Voice Agent API?

AssemblyAI's Voice Agent API is a single WebSocket API that handles the full voice pipeline—speech understanding, LLM reasoning, and voice generation—in one connection. You don't manage separate STT, LLM, and TTS providers. You connect once, and the infrastructure handles the rest.

The key architectural difference from OpenAI Realtime API: where OpenAI's model is multimodal (voice as one of many capabilities), AssemblyAI's pipeline is purpose-built for speech. Universal-3 Pro Streaming—AssemblyAI's dedicated Voice AI model, ranked #1 on the Hugging Face Open ASR Leaderboard—sits at the foundation. Getting the input right is the whole point, because when speech-to-text misreads a customer's name or account number, every downstream step responds to the wrong thing.

Here's how the two APIs compare side by side:

Feature	OpenAI Realtime API	AssemblyAI Voice Agent API
Pricing model	Token-based (varies)	Flat-rate ($4.50/hr)
Concurrency	Tier-based quotas	Auto-scaling
Speech accuracy foundation	Multimodal model	Universal-3 Pro (dedicated, #1 Hugging Face)
Connection type	WebSocket, manual event handling	WebSocket, standard JSON API
Session management	Manual (30+ event types)	Streamlined (~10 event types)
Turn detection	server_vad or semantic_vad	Acoustic + contextual (built-in, configurable)
Supported languages	Multiple	English, Spanish, French, German, Italian, Portuguese + multilingual voices

The "invisible infrastructure" framing here is intentional. You're not building on a platform with its own opinions about your product—you control the conversation design, tool integrations, and agent behavior fully. The infrastructure just works.

Migrate from OpenAI Realtime API to AssemblyAI Voice Agent API

Most migrations take one to two days. You're not rewriting your product logic—you're replacing infrastructure code with simpler infrastructure code. The four steps are: authentication setup, session configuration, audio streaming, and tool migration.

Authentication and environment setup

Before you connect, install the required packages:

pip install websockets pyaudio python-dotenv

Add your AssemblyAI API key to your .env file:

ASSEMBLYAI_API_KEY=your_assemblyai_key_here

Here's where the first simplification happens. OpenAI requires you to generate an ephemeral token as a separate API call before you can open a WebSocket connection:

from openai import OpenAI
import os
from dotenv import load_dotenv

load_dotenv()

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

# Extra step before every connection
response = client.realtime.sessions.create(
    model="gpt-realtime",
    voice="alloy"
)
ephemeral_token = response.client_secret.value
# Token expires — you need to handle that too

With AssemblyAI, you authenticate directly by passing your API key as a Bearer token in the WebSocket connection header. No token generation, no expiration handling:‍

import asyncio
import websockets
import os
from dotenv import load_dotenv

load_dotenv()

async def connect():
    uri = "wss://agents.assemblyai.com/v1/ws"
    headers = {"Authorization": f"Bearer {os.environ['ASSEMBLYAI_API_KEY']}"}

    async with websockets.connect(uri, additional_headers=headers) as ws:
        print("Connected — ready to configure session.")

asyncio.run(connect())

Removing the ephemeral token step eliminates an entire failure mode. Token expiration mid-session was a real edge case to handle with OpenAI; it doesn't exist here.

Note for browser-based applications: If you're building a client-side voice agent that runs in the browser, you'll need to use AssemblyAI's temporary token flow to avoid exposing your API key. Generate a temporary token server-side using the token endpoint and pass it as a query parameter: wss://agents.assemblyai.com/v1/ws?token=YOUR_TEMP_TOKEN.

Session configuration

With OpenAI, session configuration requires a deeply nested JSON structure. You manually specify audio format, sample rate, VAD parameters, and voice settings—all before anything works:

import json
import asyncio
import websockets
import os

async def configure_openai_session():
    uri = "wss://api.openai.com/v1/realtime?model=gpt-realtime"
    headers = {"Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}"}

    async with websockets.connect(uri, extra_headers=headers) as ws:
        session_config = {
            "type": "session.update",
            "session": {
                "instructions": "You are a helpful customer support agent.",
                "audio": {
                    "input": {
                        "format": {"type": "audio/pcm", "rate": 24000},
                        "turn_detection": {
                            "type": "server_vad",
                            "threshold": 0.5,
                            "prefix_padding_ms": 300,
                            "silence_duration_ms": 200
                        }
                    },
                    "output": {
                        "voice": "alloy",
                        "format": {"type": "audio/pcm", "rate": 24000}
                    }
                }
            }
        }
        await ws.send(json.dumps(session_config))

        async for message in ws:
            event = json.loads(message)
            if event["type"] == "session.updated":
                print("Session configured.")
                break

asyncio.run(configure_openai_session())

With AssemblyAI, you send a session.update message after connecting. Audio format defaults to PCM at 24kHz, and turn detection is built in with sensible defaults—you only override what you actually need to change:‍

import json
import asyncio
import websockets
import os
from dotenv import load_dotenv

load_dotenv()

async def configure_assemblyai_session():
    uri = "wss://agents.assemblyai.com/v1/ws"
    headers = {"Authorization": f"Bearer {os.environ['ASSEMBLYAI_API_KEY']}"}

    async with websockets.connect(uri, additional_headers=headers) as ws:
        # Configure the session — defaults handle audio format and turn detection
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "system_prompt": "You are a helpful customer support agent.",
                "greeting": "Hi! How can I help you today?",
                "output": {"voice": "ivy"}
            }
        }))

        # Wait for session.ready before streaming audio
        async for message in ws:
            event = json.loads(message)
            if event["type"] == "session.ready":
                print(f"Session ready — id: {event['session_id']}")
                break

asyncio.run(configure_assemblyai_session())

The difference is more than syntax. With OpenAI, misconfiguring a single audio parameter breaks everything silently. AssemblyAI's defaults are tuned for real-world conversations, so you only override what you actually need to change. Need to adjust turn detection sensitivity? Add turn_detection to the config:

# Optional: customize turn detection
await ws.send(json.dumps({
    "type": "session.update",
    "session": {
        "input": {
            "turn_detection": {
                "vad_threshold": 0.5,
                "min_silence": 600,
                "max_silence": 1500,
                "interrupt_response": True
            },
            "keyterms": ["AssemblyAI", "Universal-3 Pro"]
        }
    }
}))

You can also update session.update mid-conversation—change the system prompt, add tools, or adjust turn detection without reconnecting.

See Voice AI in action

Test streaming transcription and voice agent interactions in the browser before wiring your event handlers. No code required.

Try playground

Audio streaming and event handling

This is where the complexity gap is most visible. OpenAI requires you to manually encode audio as base64, manage the input buffer, commit it at the right time, and decode the audio response back to bytes:‍

import base64
import pyaudio
import asyncio
import websockets
import json
import os

async def openai_audio_loop(ws):
    p = pyaudio.PyAudio()

    # Input stream
    input_stream = p.open(
        format=pyaudio.paInt16,
        channels=1,
        rate=24000,
        input=True,
        frames_per_buffer=4800
    )

    # Output stream
    output_stream = p.open(
        format=pyaudio.paInt16,
        channels=1,
        rate=24000,
        output=True
    )

    async def send_audio():
        while True:
            chunk = input_stream.read(4800, exception_on_overflow=False)
            # Must encode as base64 before sending
            await ws.send(json.dumps({
                "type": "input_audio_buffer.append",
                "audio": base64.b64encode(chunk).decode("utf-8")
            }))
            await asyncio.sleep(0.1)

    async def receive_audio():
        async for message in ws:
            event = json.loads(message)
            if event["type"] == "response.output_audio.delta":
                # Must decode from base64 before playing
                audio_bytes = base64.b64decode(event["delta"])
                output_stream.write(audio_bytes)
            elif event["type"] == "response.done":
                print("Response complete.")

    # Run both concurrently
    await asyncio.gather(send_audio(), receive_audio())

With AssemblyAI, you still work with WebSocket events and base64 audio—but the event types are cleaner and there's no buffer management step. You send input.audio events and receive reply.audio events, plus you get transcript events for both sides of the conversation:‍

import base64
import pyaudio
import asyncio
import websockets
import json
import os
from dotenv import load_dotenv

load_dotenv()

async def assemblyai_audio_loop():
    uri = "wss://agents.assemblyai.com/v1/ws"
    headers = {"Authorization": f"Bearer {os.environ['ASSEMBLYAI_API_KEY']}"}

    p = pyaudio.PyAudio()
    input_stream = p.open(
        format=pyaudio.paInt16, channels=1, rate=24000,
        input=True, frames_per_buffer=4800
    )
    output_stream = p.open(
        format=pyaudio.paInt16, channels=1, rate=24000,
        output=True
    )

    async with websockets.connect(uri, additional_headers=headers) as ws:
        # Configure session
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "system_prompt": "You are a helpful customer support agent.",
                "output": {"voice": "ivy"}
            }
        }))

        ready = False

        async def send_audio():
            nonlocal ready
            while True:
                if ready:
                    chunk = input_stream.read(4800, exception_on_overflow=False)
                    await ws.send(json.dumps({
                        "type": "input.audio",
                        "audio": base64.b64encode(chunk).decode("utf-8")
                    }))
                await asyncio.sleep(0.1)

        async def receive_events():
            nonlocal ready
            async for message in ws:
                event = json.loads(message)

                if event["type"] == "session.ready":
                    ready = True
                    print("Session ready.")
                elif event["type"] == "reply.audio":
                    audio_bytes = base64.b64decode(event["data"])
                    output_stream.write(audio_bytes)
                elif event["type"] == "transcript.user":
                    print(f"User: {event['text']}")
                elif event["type"] == "transcript.agent":
                    print(f"Agent: {event['text']}")
                elif event["type"] == "reply.done":
                    if event.get("status") == "interrupted":
                        print("Agent interrupted by user.")

        await asyncio.gather(send_audio(), receive_events())

asyncio.run(assemblyai_audio_loop())

The event model is simpler: no buffer commit step, no separate response trigger. You stream audio in with input.audio, and the server handles turn detection, generates a response, and streams audio back with reply.audio. You also get transcript.user and transcript.agent events—full text transcripts of both sides of the conversation, which OpenAI doesn't provide natively.

Barge-in is built in too. When a user interrupts, you'll receive reply.done with status: "interrupted"—flush your audio playback buffer and the agent picks up the new turn automatically.

Function calling and tool migration

Both APIs support tool calling, and the JSON Schema definitions are similar enough that your existing tool definitions translate directly. The difference is in how you handle the results.

With OpenAI, you track a call_id for each function call, execute the function, then manually route the result back to the model via a conversation.item.create event:‍

import json
import asyncio
import websockets
import os

def get_account_info(account_id: str) -> dict:
    # Your business logic
    return {"status": "active", "balance": 1000, "id": account_id}

# Register tools in session config
tools = [{
    "type": "function",
    "name": "get_account_info",
    "description": "Look up account status by ID",
    "parameters": {
        "type": "object",
        "properties": {
            "account_id": {"type": "string", "description": "The customer account ID"}
        },
        "required": ["account_id"]
    }
}]

async def openai_with_tools():
    uri = "wss://api.openai.com/v1/realtime?model=gpt-realtime"
    headers = {"Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}"}

    async with websockets.connect(uri, extra_headers=headers) as ws:
        # Include tools in session config
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "instructions": "You are a support agent.",
                "tools": tools
            }
        }))

        async for message in ws:
            event = json.loads(message)

            if event["type"] == "response.done":
                output = event["response"]["output"][0]

                if output.get("type") == "function_call":
                    # Parse arguments
                    args = json.loads(output["arguments"])

                    # Execute your function
                    result = get_account_info(args["account_id"])

                    # Manually route the result back to the model
                    await ws.send(json.dumps({
                        "type": "conversation.item.create",
                        "item": {
                            "type": "function_call_output",
                            "call_id": output["call_id"],  # Must track this
                            "output": json.dumps(result)
                        }
                    }))

                    # Trigger the next response
                    await ws.send(json.dumps({"type": "response.create"}))

asyncio.run(openai_with_tools())

With AssemblyAI, the tool definition format is nearly identical—your JSON Schema definitions transfer directly. The key difference: you accumulate tool results during tool.call events, then send them all back after reply.done. The agent speaks a natural transition phrase while waiting for your results:‍

import json
import asyncio
import websockets
import os
from dotenv import load_dotenv

load_dotenv()

def get_account_info(account_id: str) -> dict:
    # Your business logic — same function, no changes needed
    return {"status": "active", "balance": 1000, "id": account_id}

# Tool definitions — same JSON Schema format as OpenAI
tools = [{
    "type": "function",
    "name": "get_account_info",
    "description": "Look up account status by ID",
    "parameters": {
        "type": "object",
        "properties": {
            "account_id": {"type": "string", "description": "The customer account ID"}
        },
        "required": ["account_id"]
    }
}]

async def assemblyai_with_tools():
    uri = "wss://agents.assemblyai.com/v1/ws"
    headers = {"Authorization": f"Bearer {os.environ['ASSEMBLYAI_API_KEY']}"}

    async with websockets.connect(uri, additional_headers=headers) as ws:
        # Include tools in session config
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "system_prompt": "You are a support agent. Use get_account_info to look up accounts.",
                "output": {"voice": "ivy"},
                "tools": tools
            }
        }))

        pending_tools = []

        async for message in ws:
            event = json.loads(message)

            if event["type"] == "session.ready":
                print("Session ready with tools.")

            elif event["type"] == "tool.call":
                # Execute your function
                name = event["name"]
                args = event.get("arguments", {})

                if name == "get_account_info":
                    result = get_account_info(args["account_id"])
                else:
                    result = {"error": "Unknown tool"}

                # Accumulate — don't send yet
                pending_tools.append({
                    "call_id": event["call_id"],
                    "result": result,
                })

            elif event["type"] == "reply.done":
                if event.get("status") == "interrupted":
                    # User barged in — discard pending results
                    pending_tools.clear()
                elif pending_tools:
                    # Now send all tool results
                    for tool in pending_tools:
                        await ws.send(json.dumps({
                            "type": "tool.result",
                            "call_id": tool["call_id"],
                            "result": json.dumps(tool["result"]),
                        }))
                    pending_tools.clear()

            elif event["type"] == "transcript.user":
                print(f"User: {event['text']}")
            elif event["type"] == "transcript.agent":
                print(f"Agent: {event['text']}")

asyncio.run(assemblyai_with_tools())

Your actual business logic—get_account_info in this case—doesn't change at all. The tool definition JSON Schema is the same format. The difference is in the plumbing: AssemblyAI uses a cleaner accumulate-and-send pattern with tool.call / tool.result events, and the agent automatically speaks a transition phrase ("Let me check that for you") while waiting for results—no dead air.

What changes and what stays the same

Migrating from OpenAI Realtime API to AssemblyAI's Voice Agent API is, in practice, mostly a subtraction exercise. Here's a quick summary:

What you remove: Ephemeral token generation, manual audio buffer management and commit steps, base64 encoding/decoding boilerplate (though you still work with base64, the event model is simpler), the 30+ event types to handle, manual response triggering after tool calls.

What stays the same: Your business logic and function implementations, JSON Schema tool definitions (they transfer directly), the WebSocket + JSON communication model, base64 PCM audio format.

What you gain: Streaming transcripts of both sides of the conversation (transcript.user and transcript.agent), purpose-built speech accuracy from Universal-3 Pro Streaming, built-in turn detection with acoustic + contextual signals, natural barge-in handling, predictable $4.50/hr pricing, session resumption (reconnect within 30 seconds and pick up where you left off), and key terms prompting to boost accuracy on domain-specific vocabulary.

If you're building production voice agents and want speech accuracy, predictable pricing, and auto-scaling concurrency without managing three separate providers, AssemblyAI's Voice Agent API is worth evaluating. It's built on Universal-3 Pro Streaming—the same dedicated Voice AI model that powers AssemblyAI's real-time transcription—and is designed to be invisible infrastructure that stays out of the way while you build the product your users actually interact with.

Build production voice agents today

Get all-inclusive flat-rate pricing at $4.50/hr, auto-scaling concurrency, and industry-leading speech accuracy built on Universal-3 Pro. Read the full API reference in 10 minutes.

Frequently asked questions

How does AssemblyAI Voice Agent API pricing compare to OpenAI Realtime API?

AssemblyAI charges a flat rate of $4.50/hour that covers speech understanding, LLM reasoning, and voice generation regardless of conversation density. OpenAI charges per audio token for input and output separately, meaning costs vary based on speech pace, pause frequency, and response length. The flat-rate model makes cost forecasting straightforward—one line of math to model what a 5-minute call costs.

How long does migrating from OpenAI Realtime API to AssemblyAI Voice Agent API take?

Most migrations take one to two days. The changes are primarily structural—replacing OpenAI's event handling patterns with AssemblyAI's cleaner event model. Your business logic and JSON Schema tool definitions transfer directly without modification.

Can I reuse my existing OpenAI function definitions when migrating?

Yes. Both APIs use JSON Schema for tool definitions, so your existing schemas work as-is. The difference is in the execution flow: OpenAI uses conversation.item.create to return results, while AssemblyAI uses a tool.call / tool.result pattern where you accumulate results and send them after reply.done.

Does AssemblyAI Voice Agent API require an SDK?

No. The Voice Agent API is a standard WebSocket + JSON API. You connect to wss://agents.assemblyai.com/v1/ws, authenticate with a Bearer token, and exchange JSON messages. No SDK to install, no framework to learn. You can read the entire API reference in 10 minutes, and it works natively with coding agents like Claude Code.

What languages does AssemblyAI Voice Agent API support?

The Voice Agent API supports English, Spanish, French, German, Italian, and Portuguese through Universal-3 Pro, with regional dialect recognition across all six languages. Multilingual voices are available for additional languages including Hindi, Mandarin, Russian, Korean, and Japanese.

What happens if the WebSocket connection drops during a conversation?

AssemblyAI preserves sessions for 30 seconds after disconnection. You can reconnect using the session.resume event with the session_id from the original session.ready event, and the conversation picks up where it left off with full context preserved. This eliminates the need for custom reconnection logic.

How to create a phone-based voice agent

Mart Schweiger — Wed, 22 Apr 2026 15:47:15 +0000

A phone-based voice agent is an AI system that answers or places phone calls, understands what the caller is saying, and responds in natural speech — with no human in the loop. The best ones pick up in under a ring, handle interruptions the way a person does, and hang up before anyone realizes they weren't talking to a human.

Getting there takes four things: a telephony provider to carry the call, a streaming speech-to-text model to hear, an LLM to think, and a text-to-speech model to respond. This guide walks through each layer, the latency budget they share, and the architecture decisions that decide whether your agent feels natural or obviously robotic.

We'll use the AssemblyAI Universal-3 Pro Streaming model for transcription throughout — it's built specifically for voice agents, with 307ms P50 latency and native 8kHz mulaw support so phone audio doesn't need resampling.

What is a phone-based voice agent?

A phone-based voice agent is a Voice AI application that conducts full conversations over the public telephone network (PSTN) or a SIP trunk. Unlike an IVR ("press 1 for billing"), it listens to free-form speech, understands intent with a Large Language Model, and replies in synthesized voice — all inside a single phone call.

Common jobs for phone-based voice agents:

Inbound support: answer questions, triage tickets, hand off to a human
Appointment scheduling and confirmations
Insurance verification and benefits lookup
Outbound reminders, surveys, and follow-ups
Lead qualification and sales discovery
After-hours coverage for tier-1 calls

The technology stack is the same regardless of direction. The only thing that changes is who initiates the call.

The four components of a phone-based voice agent

Every phone-based voice agent has the same four pieces. Get any one of them wrong and the whole conversation falls apart.

1. Telephony (the phone network layer)

This is what connects your agent to a real phone number. You have three practical options:

Twilio Voice + Media Streams — the most common choice. Twilio gives you a number, handles the PSTN, and streams 8kHz mulaw audio to your server over WebSocket.
SIP trunk (Telnyx, Plivo, direct carrier) — more control and lower per-minute cost once you're at scale. More setup work.
Managed voice agent platforms (Vapi, LiveKit, Pipecat, Daily, Retell) — they abstract telephony, STT, LLM, and TTS orchestration into one SDK. Fastest path to a working prototype.

The audio format matters here. Traditional telephony runs at 8kHz mulaw (μ-law), not the 16kHz PCM most audio tools assume. A speech-to-text model that accepts mulaw natively saves you a resampling step and the latency that comes with it.

2. Streaming speech-to-text (the ears)

This is the most time-sensitive component. The speech-to-text model has to produce transcripts fast enough that your LLM can start generating before the caller finishes talking — otherwise there's an awkward silence every turn.

What to look for:

Sub-300ms P50 latency on real phone audio, not studio recordings
Immutable transcripts — finalized words don't change, so downstream systems can act on partials with confidence
Intelligent endpointing that combines acoustic pauses with semantic cues (knowing "my number is five five five" isn't a complete thought)
Native telephony format support — 8kHz mulaw input without resampling
Accuracy on alphanumerics — phone numbers, confirmation codes, email addresses

The Universal-3 Pro Streaming model hits 307ms median latency with 21% fewer alphanumeric errors than the previous generation of streaming speech-to-text models, which matters when your agent is writing down a credit card number.

3. The LLM (the brain)

The LLM takes the transcript, decides what the agent should say, and optionally calls tools — a calendar API, your CRM, a payment processor. Any modern model works: GPT-4o, Claude Sonnet, Gemini Flash. Pick for latency first, capability second — a smarter response that arrives 800ms late feels worse than a slightly simpler one that lands instantly.

The LLM Gateway lets you route transcripts to multiple LLM providers through a single API, which is useful when you want to switch models without touching the rest of your stack.

4. Text-to-speech (the mouth)

TTS converts the LLM's reply back into audio. ElevenLabs, Cartesia, Deepgram Aura, and OpenAI Realtime TTS are the common choices. Three things matter:

Time to first audio byte (under 400ms is good)
Voice naturalness and prosody
Ability to stream audio in chunks so the caller hears speech while the rest is still generating

Stream the TTS output back to Twilio as mulaw frames and you're done.

Architecture: how the pieces fit together

Here's the reference architecture for a phone-based voice agent using Twilio for telephony. If you use a managed platform like Vapi or LiveKit, most of the middle is handled for you.

<pre>
  Caller's phone
        │
        │  PSTN
        ▼
   Twilio Voice
        │  TwiML → open WebSocket
        ▼
  Your server (WebSocket bridge)
   ┌────┴────┐
   ▼         ▼
 Universal-3    TTS
 Pro Streaming  (ElevenLabs /
 (STT)          Cartesia / etc.)
   │             ▲
   │ transcript  │ synthesized audio
   ▼             │
   LLM (GPT-4o / Claude / etc.)
   │  text response
   └───────────►
</pre>

Audio flows in two directions continuously. Inbound audio (caller → agent) goes to the speech-to-text model. Outbound audio (agent → caller) is generated by the TTS model. Your server's job is to bridge them and hand transcripts to the LLM at the right moment.

The latency budget

The single biggest quality driver in a phone-based voice agent is end-to-end latency from "caller stops talking" to "agent starts talking." Under 800ms feels natural. Over 1,500ms feels like a bad satellite connection.

Two rules fall out of this budget:

Every component has to stream. Batch anything and you're over budget.
Don't wait for a complete transcript before calling the LLM. Start generating on partials that your STT model flagged as stable.

The Universal-3 Pro Streaming model produces immutable partials — once a word is finalized, it won't change — which lets the LLM start generating while the caller is still finishing their sentence.

Building a phone-based voice agent with Twilio and AssemblyAI

Here's a minimal server that bridges Twilio Media Streams into AssemblyAI Universal-3 Pro Streaming. This handles the inbound audio path; wire it up to your LLM and TTS of choice.

<pre><code class="language-python">import asyncio
import base64
import json
import os
import websockets
from fastapi import FastAPI, WebSocket
from fastapi.responses import Response

app = FastAPI()

ASSEMBLYAI_KEY = os.environ["ASSEMBLYAI_API_KEY"]
AAI_WS = (
    "wss://streaming.assemblyai.com/v3/ws"
    "?sample_rate=8000&encoding=pcm_mulaw&format_turns=true"
)

@app.post("/incoming-call")
async def incoming_call(request):
    # TwiML tells Twilio to open a Media Stream to our /ws endpoint
    twiml = """&lt;Response&gt;
      &lt;Connect&gt;
        &lt;Stream url="wss://your-server.ngrok.app/ws" /&gt;
      &lt;/Connect&gt;
    &lt;/Response&gt;"""
    return Response(content=twiml, media_type="application/xml")

@app.websocket("/ws")
async def ws(ws: WebSocket):
    await ws.accept()
    async with websockets.connect(
        AAI_WS,
        additional_headers={"Authorization": ASSEMBLYAI_KEY},
    ) as aai:

        async def twilio_to_aai():
            async for msg in ws.iter_text():
                data = json.loads(msg)
                if data["event"] == "media":
                    # Twilio sends base64 mulaw at 8kHz
                    audio = base64.b64decode(data["media"]["payload"])
                    await aai.send(audio)

        async def aai_to_logic():
            async for raw in aai:
                evt = json.loads(raw)
                if evt.get("type") == "Turn" and evt.get("end_of_turn"):
                    transcript = evt["transcript"]
                    # Hand off to your LLM + TTS here
                    print("Final turn:", transcript)

        await asyncio.gather(twilio_to_aai(), aai_to_logic())
</code></pre>

Two things to note:

encoding=pcm_mulaw and sample_rate=8000 match Twilio's native format. No resampling.
format_turns=true gives you clean turn boundaries with punctuation — easier to pass to an LLM than raw partials.

For a full runnable example — FastAPI server, ElevenLabs TTS, GPT-4o with function calling for order lookup and appointment booking — clone the companion repo:

git clone https://github.com/kelsey-aai/phone-voice-agent-assemblyai

Or see our step-by-step Twilio phone agent with AssemblyAI Universal-3 Pro Streaming tutorial for a simpler variant without function calling.

Design decisions that make or break a phone-based voice agent

Most failed phone-based voice agent projects fail the same way: the tech works in a demo, then breaks on real calls. Here are the decisions that matter in production.

Turn detection strategy

Fixed silence timers (wait 800ms, then respond) feel broken. Real callers pause mid-sentence to think, and the agent keeps cutting them off. Modern streaming models combine acoustic silence with semantic signals — the model knows that "the number is five five five…" is mid-thought even if there's a pause. Use intelligent endpointing with tunable thresholds, not a fixed timer.

Barge-in handling

Callers interrupt. When they do, the agent has to stop talking immediately, not finish its sentence. That means you need to pipe STT transcripts back even while TTS is playing, and cut the TTS audio the moment you detect the caller started a new turn.

Alphanumeric accuracy

The hardest thing a phone-based voice agent does is get a confirmation code right on the first try. "B as in boy, eight, zero, zero, five" — that's what your STT model has to capture. Telephony audio is 8kHz, which already hurts accuracy; noisy environments hurt it more. Use a model measured on alphanumeric benchmarks, not just overall word error rate.

Function calling and tool use

Real phone agents don't just talk — they book the appointment, push the update to the CRM, charge the card. That means the LLM needs to call tools during the conversation. Keep tool calls short (under 500ms) or the silence becomes obvious. Preload what you can before the call if you know the caller ID.

PII redaction and compliance

Phone calls pick up credit cards, medical information, and account numbers. If you're in healthcare, financial services, or Europe, you need PII redaction and often a BAA or DPA. AssemblyAI enables covered entities and their business associates subject to HIPAA to use AssemblyAI services to process protected health information (PHI); a Business Associate Addendum (BAA) is available for healthcare deployments.

Concurrency and scale

A single agent is easy. 500 simultaneous calls during a support spike is not. Pick a speech-to-text provider with unlimited concurrency and session-based pricing so you're not renegotiating contracts when traffic grows. Universal-Streaming supports unlimited concurrent streams at a flat $0.15/hr.

Common use cases for phone-based voice agents

The pattern is consistent: phone-based voice agents work best on high-volume, well-scoped calls where the caller wants a specific outcome and the range of conversation is predictable.

Healthcare: appointment scheduling, insurance verification, prescription refills, post-visit follow-ups. See our guide to voice agents in healthcare.
Contact centers: tier-1 deflection, after-hours coverage, outbound reminders. Often paired with real-time agent assist for calls that escalate to humans.
Financial services: balance inquiries, payment reminders, fraud verification.
E-commerce and logistics: order status, delivery coordination, returns initiation.
Field service: dispatch, status updates, technician check-ins.

In every one of these, the bottleneck is the same: how accurately the agent hears the caller, and how fast it can respond.

How to evaluate a phone-based voice agent before shipping

Demos lie. Here's the evaluation loop we recommend before putting an agent on a real phone number:

Record 50–100 real calls (with consent) or synthesize them from your top call drivers
Measure turn-by-turn latency from caller-stops to agent-starts, not just STT latency in isolation
Audit transcription errors on critical entities: names, addresses, confirmation codes, dollar amounts
Score completion rate: did the agent actually do the task, or hand off?
Read the transcripts: you'll find prompt failures and tool-call bugs you'd never catch by listening alone

Most teams skip step 3 and ship with a 90% WER model that silently fumbles phone numbers. Don't.

Phone-based voice agent vs. IVR vs. chatbot

An AI voice agent isn't a smarter IVR — it's a replacement for the human on the other end of the phone for calls that don't need a human.

Putting it together

Building a phone-based voice agent in 2026 is a matter of wiring four well-understood components together: telephony, streaming speech-to-text, an LLM, and TTS. The difference between an agent that feels natural and one that doesn't isn't the LLM — it's the latency budget and the speech-to-text accuracy on phone audio.

Start with a managed platform if you need to ship this quarter. Drop to a Twilio + custom stack when you need to control the latency and accuracy yourself. Either way, the streaming speech-to-text layer is the part that will make or break the caller experience, so pick it carefully.

If you're building phone-based voice agents that need to sound human, Universal-3 Pro Streaming is the reference benchmark for sub-300ms streaming speech-to-text with native telephony support. The companion GitHub repo — github.com/kelsey-aai/phone-voice-agent-assemblyai — is a ~200-line FastAPI server you can fork and ship today.

Frequently asked questions

What is a phone-based voice agent?
A phone-based voice agent is an AI system that answers or places phone calls, transcribes the caller's speech in real time, decides what to say with a Large Language Model, and responds with synthesized voice — all within a single live phone call. It replaces IVR menus and tier-1 human agents for well-scoped tasks like scheduling, verification, and support.

How does a phone-based voice agent work?
A phone-based voice agent combines four components: a telephony provider (Twilio, SIP, or a managed platform) to carry the call, a streaming speech-to-text model to transcribe audio in real time, an LLM to understand intent and generate replies, and a text-to-speech model to speak the response. Everything streams, so the caller hears a reply roughly 800ms after they stop talking.

What is the best speech-to-text for a phone-based voice agent?
The best speech-to-text for a phone-based voice agent is a streaming model with sub-300ms latency, native 8kHz mulaw support, immutable transcripts, intelligent endpointing, and strong accuracy on alphanumerics. AssemblyAI's Universal-3 Pro Streaming model hits 307ms P50 latency and 21% fewer alphanumeric errors than the previous generation, which matters when the agent is capturing phone numbers or confirmation codes.

How do I build a phone-based voice agent with Twilio?
To build a phone-based voice agent with Twilio, create a Twilio phone number, configure TwiML to open a Media Stream to your server's WebSocket endpoint, and bridge the 8kHz mulaw audio into a streaming speech-to-text model like Universal-3 Pro Streaming. Pipe final transcripts into an LLM, then stream the response back through a text-to-speech model as mulaw frames for Twilio to play.

What is the difference between a phone-based voice agent and an IVR?
An IVR uses touch-tone input and rigid menus ("press 1 for billing"), while a phone-based voice agent understands free-form natural speech, maintains conversation context with an LLM, handles interruptions, and can complete tasks end-to-end without handing off to a human. The caller experience is closer to talking to a person than navigating a menu.
How much does it cost to run a phone-based voice agent?
The main cost components are telephony (Twilio per-minute voice rates), streaming speech-to-text (Universal-3 Pro Streaming is $0.15/hour of session time), the LLM (varies by provider and tokens used), and TTS (per-character or per-minute). A typical phone-based voice agent call costs a few cents per minute end-to-end at scale, with the exact number driven by which LLM and TTS you pick.

How to create an AI cold-calling agent

Mart Schweiger — Wed, 22 Apr 2026 15:40:27 +0000

An AI cold-calling agent is an outbound Voice AI system that places calls, opens the conversation, pitches, handles objections, and either books a meeting or disqualifies the lead — without a human on the line. Built right, it runs 500 calls in parallel at roughly the cost of a single SDR.

Built wrong, it sounds like a telemarketer with a bad connection and gets hung up on in four seconds.

This guide walks through how to actually create one: the architecture, the speech-to-text accuracy you need for objection handling to work, the compliance traps (TCPA, state-level consent), and the pieces that decide whether your agent books meetings or ends up on a "do not call" list.

We'll anchor the stack on the AssemblyAI Universal-3 Pro Streaming model — 307ms P50 latency, native mulaw, and the alphanumeric accuracy that matters when the prospect rattles off their email.

What is an AI cold-calling agent?

An AI cold-calling agent is an outbound Voice AI system that dials a prospect, delivers a pitch in natural conversation, adapts in real time based on what the prospect says, and books qualified meetings or gathers disposition data. Unlike a robocall (one-way recorded message) or a dialer with a human rep, it conducts a two-way conversation autonomously.

The typical jobs an AI cold-calling agent does:

Outbound SDR prospecting: open with a relevant hook, qualify BANT, book a demo
Appointment setting for field sales, financial advisors, home services
Re-engagement of lapsed leads in a CRM
Survey and research calls at scale
Event follow-up and RSVP confirmation
Renewal and upsell motions for existing customers

The common thread: one script, thousands of conversations, measurable booking rate.

The architecture of an AI cold-calling agent

An AI cold-calling agent is a phone-based voice agent with a few extra components tuned for outbound. Here's the full stack:

<pre>
  CRM / lead list (Salesforce, HubSpot, CSV)
        │
        ▼
  Dialer / orchestrator
  (concurrency, pacing, DNC check, retries)
        │
        ▼
  Twilio / SIP outbound call
        │  WebSocket bridge
        ▼
  Universal-3 Pro Streaming (STT)
        │  transcript
        ▼
  LLM with sales prompt + objection map
        │  text response + tool calls
        ▼
  TTS (ElevenLabs / Cartesia)
        │  audio
        ▼
        Prospect
        │
        └─► Call disposition
            └─► CRM update + calendar booking
</pre>

The five components that matter:

Lead source and dialer — where the list comes from and how you pace calls
Telephony — Twilio, SIP, or a managed voice agent platform
Streaming speech-to-text — the ears; must hear objections the moment they start
LLM with a sales-specific prompt — opener, discovery, objection handling, booking logic
Text-to-speech — the voice; naturalness matters more here than on inbound

Plus two things that are unique to outbound: compliance filtering (TCPA, state consent laws, DNC registries) and post-call disposition sync back to the CRM.

Why speech-to-text accuracy decides whether an AI cold-calling agent works

On an inbound support call, the caller wants help — they'll repeat themselves if you miss something. On an outbound cold call, the prospect is deciding whether to hang up in the first five seconds. If your agent mishears "not interested, take me off the list" as "I'm interested, tell me more," you don't get a second chance.

Three STT capabilities decide the quality of an AI cold-calling agent:

Low, stable latency

Natural turn-taking happens in under 800ms end-to-end. Any longer and the prospect thinks they lost connection — or worse, that they're on a robocall. The Universal-3 Pro Streaming model delivers 307ms median latency with immutable transcripts, which lets your LLM start generating a response before the prospect even finishes their sentence.

Alphanumeric accuracy

Cold calls capture emails, phone numbers, company names, and job titles. "J at acme dot io," "director of rev ops," "five one five, nine eight two, four zero zero zero." Universal-3 Pro Streaming delivers 21% fewer alphanumeric errors and 28% better accuracy on consecutive numbers than the previous generation — the difference between a booked meeting in your calendar and a typo you never catch.

Intelligent endpointing

Prospects pause. "I'm… probably not the right person to talk to about this." If your agent jumps in at the first pause, it interrupts. If it uses a fixed silence timer, it feels robotic. Intelligent endpointing combines acoustic and semantic signals to detect real turn boundaries — the difference between a thoughtful agent and an impatient one.

Building the conversation logic

The LLM prompt is where an AI cold-calling agent earns its meetings or wastes the prospect's time. A good cold-calling prompt has four sections:

1. Identity and opener

Who the agent is, which company it represents, why it's calling. Must include clear AI disclosure in the opener — this is both good practice and legally required in several states (California, Florida, Texas among others).

2. Discovery questions

Two to four questions that qualify or disqualify the prospect. Don't ask five — you'll get hung up on.

3. Objection handling map

A structured map of likely objections and how to respond. The usual suspects:

"How did you get my number?"
"Send me an email instead."
"I'm not the right person."
"We already use [competitor]."
"We're not interested."
"Take me off your list."

That last one is the most important. If the prospect says anything that sounds like a do-not-call request, the agent must immediately:

Acknowledge
Confirm the number will be added to DNC
End the call politely
Flag the number in your CRM and DNC database

No upselling. No "can I just ask one question?" You don't get a second chance on a compliance complaint.

4. Booking logic

If the prospect qualifies and is interested, the agent needs to book — not hand off. That means live calendar access via tool call, a handful of proposed times, and confirmation sent over SMS or email during the call.

Picking the telephony layer

Three options depending on your volume and how much you want to operate yourself.

Whatever you pick, the audio path is the same: 8kHz mulaw in and out. Use a speech-to-text model that accepts mulaw natively — resampling to 16kHz PCM adds round-trip latency you can't afford on a cold call.

The outbound-specific components

Dialer and pacing

You can't just fire off 10,000 calls at once. Telco carriers flag high-volume outbound as spam within minutes, and your numbers get blocked. Real dialers pace calls, rotate outbound numbers, and respect time-of-day rules (TCPA restricts calls before 8am and after 9pm in the recipient's local time).

If you're using Twilio, you'll want a local presence strategy — matching the outbound caller ID to the area code of the number being dialed. Connection rates go up meaningfully.

Compliance filtering

Before any call goes out:

Scrub against the federal Do Not Call registry
Scrub against state DNC lists (several states maintain their own)
Scrub against your internal suppression list (previous DNC requests, unsubscribes)
Verify you have a valid purpose under TCPA for B2C calls, or a legitimate business interest for B2B
For calls into EU numbers, confirm GDPR lawful basis

Build this filtering as a hard gate — no call goes out if any check fails. The fines for TCPA violations are $500–$1,500 per call.

Call recording and PII redaction

Record every call for quality and compliance. Store recordings encrypted. If you're recording in a two-party consent state (California, Florida, Pennsylvania, and others), the agent must get consent at the top of the call.

Use PII redaction on transcripts before they hit your CRM or analytics warehouse. Cold calls pick up personal data you often don't need to retain.

CRM sync and disposition

Every call ends with a disposition: booked, callback, not interested, DNC, voicemail, no answer, wrong number. That disposition has to land in the CRM within seconds, along with the transcript, recording URL, and any tool calls the agent made (calendar event IDs, follow-up email queued, etc.).

This is where most AI cold-calling agent projects leak value. Great calls, terrible data hygiene, nothing tracked, impossible to iterate on.

Minimal implementation sketch

Here's the shape of an AI cold-calling agent built on Twilio + AssemblyAI Universal-3 Pro Streaming + your LLM and TTS of choice. This is the outbound-specific piece — it assumes you already have the inbound WebSocket bridge from a standard phone-based voice agent tutorial.

<pre><code class="language-python">from twilio.rest import Client
import os

twilio = Client(
    os.environ["TWILIO_SID"],
    os.environ["TWILIO_AUTH"],
)

def place_cold_call(prospect):
    # 1. Compliance gate — no call without a clean scrub
    if is_on_dnc(prospect.phone) or is_suppressed(prospect.phone):
        log_skipped(prospect, reason="dnc")
        return

    # 2. Pick a local-presence outbound number
    from_number = pick_local_number(prospect.phone)

    # 3. Open the call — TwiML handoff to our media stream handler
    call = twilio.calls.create(
        to=prospect.phone,
        from_=from_number,
        url=f"https://your-server.app/voice-agent/start?lead_id={prospect.id}",
        record=True,
        recording_status_callback="https://your-server.app/recording-done",
        machine_detection="Enable",  # detect voicemail, don't pitch a robot
        time_limit=600,              # cap at 10 min
    )
    return call.sid
</code></pre>

Two things worth calling out:

machine_detection="Enable" — Twilio tells you when the call hit a voicemail. Your agent should either leave a short pre-recorded voicemail (compliant, clear AI disclosure) or hang up. Don't pitch a recording machine.
time_limit=600 — cap call duration. Runaway LLM loops on a long call are a common failure mode; a hard cap prevents runaway cost and angry prospects.

The inbound audio path (WebSocket → Universal-3 Pro Streaming → LLM → TTS → back to Twilio) is identical to any other phone-based voice agent. The outbound piece is the dialer, the compliance gate, and the disposition logic.

For a full runnable implementation — dialer.py with compliance gating, server.py with the four sales tools (book_meeting, mark_callback, mark_not_interested, honor_dnc), and automatic disposition writing — clone the companion repo:

git clone https://github.com/kelsey-aai/ai-cold-calling-agent

The repo ships with a sample leads.csv, a stubbed compliance layer, and a --dry-run mode so you can verify the pipeline before dialing a real number.

Measuring an AI cold-calling agent

A cold-calling agent lives or dies by four numbers.

The end-to-end number — meetings booked per 1,000 dials — is what determines whether the agent is ROI-positive. Track each stage independently so you know where to iterate.

Two qualitative signals also matter:

Transcript read-through: spend an hour a week reading transcripts. You'll find LLM failures you never catch in aggregate metrics.
Prospect complaints: any complaint is a leading indicator of a future regulatory issue. Take them seriously, even when "only one."

Conversation intelligence on your call corpus is the fastest way to spot which prompt changes actually moved book rate vs. which just changed the vibe.

Compliance: the part most teams underweight

The single fastest way to kill an AI cold-calling agent program is a TCPA class action. A few non-negotiables:

Scrub DNC before every call, not just at list ingest
Disclose AI clearly in the opener (several states now require it; California SB 243 and others are tightening)
Honor "take me off the list" immediately and permanently
Respect state-level outbound calling windows — TCPA's federal baseline is 8am–9pm local time, but several states are stricter
Record and retain evidence of consent for any B2C call
Don't spoof caller ID — use owned numbers with a local presence strategy, not fake ones

When in doubt, B2B calls to work phone numbers generally have more latitude than B2C calls to mobiles. Still, assume every call is a compliance event and log accordingly.

Closing thoughts

An AI cold-calling agent is a phone-based voice agent with a sales prompt, a dialer, and a compliance layer strapped on. The hard part isn't the LLM or the TTS — it's the speech-to-text layer that decides whether the agent hears objections accurately enough to respond well, and the operational layer that keeps you out of TCPA trouble.

Don't ship one without reading your own transcripts. Don't ship one without DNC scrubbing. Don't ship one with a speech-to-text model that was trained on podcast audio, not phone audio.

The fastest way to find out if an AI cold-calling agent will work for your motion is to build a small one against 500 leads, read every transcript, and measure the book rate. Universal-3 Pro Streaming is the reference streaming speech-to-text layer we'd recommend starting with — low latency, accurate on phone audio, unlimited concurrency, and $0.15/hour. The companion GitHub repo at github.com/kelsey-aai/ai-cold-calling-agent is the full working implementation — fork it, drop in your lead list, and run python dialer.py --dry-run first.

Frequently asked questions

What is an AI cold-calling agent?
An AI cold-calling agent is an outbound Voice AI system that places phone calls, conducts a natural spoken conversation with the prospect using a streaming speech-to-text model and a Large Language Model, handles objections, and either books a qualified meeting or marks the lead as not interested — without a human on the line. It's different from a robocall because it holds a real two-way conversation, and different from an AI SDR email tool because it works over the phone.

How does an AI cold-calling agent work?
An AI cold-calling agent works by dialing a prospect through a telephony provider like Twilio, streaming the prospect's voice into a real-time speech-to-text model, passing transcripts to an LLM that follows a sales prompt with objection-handling logic, and speaking replies back through a text-to-speech model. The full loop runs in under 800ms per turn, which is what makes the conversation feel natural instead of robotic.

What is the best speech-to-text for an AI cold-calling agent?
The best speech-to-text for an AI cold-calling agent is a streaming model with sub-300ms latency, native 8kHz mulaw support, and high accuracy on alphanumerics like emails and phone numbers. AssemblyAI's Universal-3 Pro Streaming model is purpose-built for voice agents, with 307ms median latency, immutable transcripts, intelligent endpointing, and 21% fewer alphanumeric errors than the previous streaming generation.

Is it legal to use an AI cold-calling agent?
Using an AI cold-calling agent is legal in most jurisdictions when you follow TCPA requirements in the US, GDPR in the EU, and state-level rules — meaning you scrub the federal and state Do Not Call registries before every call, disclose that the caller is an AI (required in California, Florida, Texas, and a growing list of states), honor opt-out requests immediately, and respect calling-hour windows. B2B calls to work numbers generally have more latitude than B2C calls to mobiles, but compliance filtering should be a hard gate regardless.

How much does it cost to run an AI cold-calling agent?
An AI cold-calling agent typically costs between $0.50 and $2.00 per conversation end-to-end at scale. The components are telephony (Twilio per-minute outbound voice), streaming speech-to-text (AssemblyAI Universal-3 Pro Streaming is $0.15/hour of session time), the LLM (varies by model and tokens), and text-to-speech (per-character or per-minute). At 10,000 calls/month the economics are roughly one-tenth the cost of an equivalent human SDR seat.

How do I build an AI cold-calling agent?
To build an AI cold-calling agent, combine a telephony provider (Twilio Voice, SIP, or a managed platform like Vapi or Retell) with a streaming speech-to-text model like Universal-3 Pro Streaming, an LLM with a cold-calling prompt that includes opener, discovery, objection handling, and booking logic, and a text-to-speech model. Wrap it with a dialer that enforces DNC scrubbing, calling-hour rules, and CRM disposition sync — those operational pieces are what separate a working program from a compliance incident.

Build a Voice Agent with LiveKit

Mart Schweiger — Tue, 14 Apr 2026 19:27:45 +0000

What is LiveKit?

LiveKit is an open-source real-time communication platform built on WebRTC infrastructure. It handles signaling, media routing, and scaling challenges so developers can focus on application logic rather than media plumbing.

LiveKit Agents is the framework layer specifically designed for AI-powered voice and video agents. It manages orchestration between speech-to-text, language model, and text-to-speech services.

The Voice Agent Pipeline

Voice agents follow a three-step cascade:

Speech-to-text (STT): User speech converts to transcript in real time
LLM: Transcript generates language model response
Text-to-speech (TTS): Response converts back to audio

Flow: WebRTC → LiveKit Cloud → AssemblyAI Universal-3 Pro Streaming → OpenAI GPT-4o → Cartesia Sonic → Back to LiveKit room

Why Universal-3 Pro Streaming for STT?

Accuracy

Production benchmarks from Hamming.ai across 4M+ production calls:

Metric	Universal-3 Pro Streaming	Deepgram Nova-3
P50 latency	307 ms	516 ms
P99 latency	1,012 ms	1,907 ms
Word error rate	8.14%	9.87%
Alphanumeric accuracy	+21% fewer errors	baseline

Neural Turn Detection

Most models use voice activity detection (VAD) — silence-based turn endings that trigger on mid-sentence pauses. Universal-3 Pro Streaming uses neural turn detection, combining acoustic and linguistic signals to distinguish mid-sentence breathing from actual end-of-turn moments.

Result: Faster response times, fewer false triggers, more natural conversations. Costs $0.45/hour and supports six languages.

Prerequisites

Python 3.11+
Microphone and speakers (for local testing)
API keys for: AssemblyAI, LiveKit Cloud, OpenAI, Cartesia

Step 1: Installation

python -m venv .venv
source .venv/bin/activate   # On Windows: .venv\Scripts\activate

pip install "livekit-agents[assemblyai,silero,codecs]~=1.0" python-dotenv
pip install "livekit-agents[openai,cartesia]~=1.0"

Note: Universal-3 Pro Streaming support requires livekit-agents@1.4.4 or newer.

Step 2: Configure API Keys

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
ASSEMBLYAI_API_KEY=your_assemblyai_key
OPENAI_API_KEY=your_openai_key
CARTESIA_API_KEY=your_cartesia_key

Step 3: Build the Agent

Create agent.py:

from dotenv import load_dotenv
from livekit import agents
from livekit.agents import AgentSession, Agent
from livekit.plugins import (
    assemblyai,
    cartesia,
    openai,
    silero,
)

load_dotenv()


class Assistant(Agent):
    def __init__(self) -> None:
        super().__init__(instructions="You are a helpful voice AI assistant.")


async def entrypoint(ctx: agents.JobContext):
    await ctx.connect()

    session = AgentSession(
        stt=assemblyai.STT(
            model="u3-rt-pro",
            min_turn_silence=100,
            max_turn_silence=1000,
            vad_threshold=0.3,
        ),
        llm=openai.LLM(model="gpt-4o"),
        tts=cartesia.TTS(),
        vad=silero.VAD.load(
            activation_threshold=0.3,
        ),
        turn_detection="stt",
        min_endpointing_delay=0,
    )

    await session.start(
        room=ctx.room,
        agent=Assistant(),
    )

    await session.generate_reply(
        instructions="Greet the user and offer your assistance."
    )


if __name__ == "__main__":
    agents.cli.run_app(agents.WorkerOptions(entrypoint_fnc=entrypoint))

Component Breakdown

turn_detection="stt" — Uses Universal-3 Pro Streaming's neural turn detection instead of LiveKit's default detector. Requires min_endpointing_delay=0 to avoid stacking delay.

Step 4: Run It

Console Mode (No LiveKit connection)

python agent.py console

Dev Mode (LiveKit Cloud connection)

python agent.py dev

Open agents-playground.livekit.io, enter your LiveKit credentials, and converse through the browser.

Tuning Turn Detection

stt=assemblyai.STT(
    model="u3-rt-pro",
    end_of_turn_confidence_threshold=0.4,  # 0.0-1.0
    min_turn_silence=300,                  # milliseconds
    max_turn_silence=1200,                 # milliseconds
)

end_of_turn_confidence_threshold: Lower = faster response; higher = fewer false triggers. Use 0.6 for noisy call centers.
min_turn_silence: Lower to 200 for rapid back-and-forth; keep at 300 for general conversation.
max_turn_silence: Raise to 2000 for healthcare or deliberate speakers.

Mid-session updates work without reconnecting:

await session.stt.update_options(max_turn_silence=3000)

Enabling Keyterm Prompting

Boost recognition accuracy for domain-specific vocabulary:

await session.stt.update_options(
    keyterms_prompt=["AssemblyAI", "Universal-3 Pro", "LiveKit"]
)

Supports up to 1,000 terms, each up to 50 characters. Takes effect immediately without restart.

Swapping Components

Swap LLM to Claude:

from livekit.plugins import anthropic
session = AgentSession(
    llm=anthropic.LLM(model="claude-sonnet-4-5"),
    ...
)

Swap TTS to ElevenLabs:

from livekit.plugins import elevenlabs
tts=elevenlabs.TTS(voice_id="your-voice-id")

Next Steps

Deploy to LiveKit Cloud: Run python agent.py start for persistent worker deployment
Add tool calling: Enable function calling through the LLM layer for lookups and actions
Enable speaker diarization: Real-time speaker identification for multi-party conversations
Add telephony: SIP support via Telnyx or Twilio integration

FAQs

What is LiveKit Agents?
LiveKit Agents is a framework for building AI-powered voice and video agents. It orchestrates between STT, LLM, and TTS components, handling real-time audio routing and turn flow while you define agent logic.

Universal-3 Pro Streaming vs. Universal-Streaming?
Universal-3 Pro Streaming (u3-rt-pro) targets production voice agent workflows with neural turn detection and superior accuracy on structured entities. Universal-Streaming is faster and cheaper ($0.15/hr) but English-only and VAD-based.

Best STT API for LiveKit voice agents?
AssemblyAI's Universal-3 Pro Streaming. Native one-line integration, 307ms P50 latency, neural turn detection, and 8.14% word error rate across production benchmarks.

How does neural turn detection work?
Combines acoustic signals (voice energy, pitch, cadence) and linguistic signals (sentence completion, punctuation patterns) to determine turn end — not just silence-based detection.

Pricing?
AssemblyAI Universal-3 Pro Streaming: $0.45/hour. Budget separately for LLM (OpenAI) and TTS (Cartesia) providers.

Build a Voice Agent with Function Calling

Mart Schweiger — Tue, 14 Apr 2026 19:27:08 +0000

Overview

This tutorial demonstrates constructing a customer support voice agent using AssemblyAI's Universal-3 Pro Streaming for speech-to-text, OpenAI GPT-4o for LLM orchestration with function calling, and ElevenLabs for voice synthesis.

The Core Problem: Transcription Accuracy

The core challenge with function-calling voice agents is transcription accuracy. When customers provide specific entities like order IDs, phone numbers, or email addresses — the exact data your functions need — poor STT quality causes silent failures. Garbage in, garbage function call out.

Universal-3 Pro Streaming addresses this with superior entity recognition: 34.79% missed entity rate for phone numbers versus 37.11% for previous models, and 59.64% for emails versus 89.09% previously.

What You'll Build

Three customer support scenarios:

Check order status — Retrieve status via order ID
Schedule a callback — Capture name and phone number
Transfer to human — Escalate when needed

Tech Stack:

AssemblyAI Universal-3 Pro Streaming (STT)
OpenAI GPT-4o (LLM with function calling)
ElevenLabs (text-to-speech)
Python 3.9+

Setup & Installation

pip install websockets openai elevenlabs pyaudio python-dotenv

ASSEMBLYAI_API_KEY=your_key_here
OPENAI_API_KEY=your_key_here
ELEVENLABS_API_KEY=your_key_here

Step 1: Connect to Universal-3 Pro Streaming

Universal-3 Pro Streaming operates via WebSocket, streaming audio chunks and receiving Turn messages with transcripts.

import asyncio
import json
import os
import websockets
import pyaudio
from dotenv import load_dotenv

load_dotenv()

ASSEMBLYAI_WS_URL = "wss://streaming.assemblyai.com/v3/ws"
SAMPLE_RATE = 16000
CHUNK_SIZE = 8000  # 500ms at 16kHz

async def connect_assemblyai():
    params = {
        "speech_model": "u3-rt-pro",
        "sample_rate": SAMPLE_RATE,
        "format_turns": "true",
    }
    query = "&".join(f"{k}={v}" for k, v in params.items())
    url = f"{ASSEMBLYAI_WS_URL}?{query}"

    ws = await websockets.connect(
        url,
        extra_headers={"Authorization": os.getenv("ASSEMBLYAI_API_KEY")}
    )
    return ws

Handling Turn Messages

async def receive_transcripts(ws, on_turn_complete):
    async for message in ws:
        data = json.loads(message)
        msg_type = data.get("type")

        if msg_type == "Begin":
            print(f"Session started: {data['id']}")

        elif msg_type == "Turn":
            transcript = data.get("transcript", "")
            if data.get("end_of_turn") and transcript.strip():
                print(f"User said: {transcript}")
                await on_turn_complete(transcript)

        elif msg_type == "Termination":
            print("Session ended.")
            break

Step 2: Define Your Functions

GPT-4o uses JSON Schema to understand available tools and their parameters.

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "get_order_status",
            "description": "Look up the status of a customer order by order ID.",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {
                        "type": "string",
                        "description": "The customer's order ID, e.g. AB3792"
                    }
                },
                "required": ["order_id"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "schedule_callback",
            "description": "Schedule a callback for a customer who wants to be called back.",
            "parameters": {
                "type": "object",
                "properties": {
                    "name": {
                        "type": "string",
                        "description": "The customer's full name"
                    },
                    "phone_number": {
                        "type": "string",
                        "description": "The customer's phone number in any format"
                    }
                },
                "required": ["name", "phone_number"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "transfer_to_human",
            "description": "Transfer the customer to a human agent when requested or when the issue can't be resolved.",
            "parameters": {
                "type": "object",
                "properties": {
                    "reason": {
                        "type": "string",
                        "description": "Brief reason for the transfer"
                    }
                },
                "required": ["reason"]
            }
        }
    }
]

Function Handlers

def get_order_status(order_id: str) -> str:
    mock_orders = {
        "AB3792": "Shipped — expected delivery April 15",
        "CD1204": "Processing — ships within 2 business days",
    }
    result = mock_orders.get(order_id.upper())
    return result if result else f"No order found with ID {order_id}"

def schedule_callback(name: str, phone_number: str) -> str:
    print(f"[SYSTEM] Callback scheduled: {name} at {phone_number}")
    return f"Got it. We'll call {name} at {phone_number} within 2 hours."

def transfer_to_human(reason: str) -> str:
    print(f"[SYSTEM] Transferring to human. Reason: {reason}")
    return "Transferring you now. Please hold for a moment."

FUNCTION_MAP = {
    "get_order_status": get_order_status,
    "schedule_callback": schedule_callback,
    "transfer_to_human": transfer_to_human,
}

Step 3: Wire Up the LLM with Function Calling

from openai import OpenAI

client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

conversation_history = [
    {
        "role": "system",
        "content": (
            "You are a helpful customer support voice agent. "
            "Keep responses short — this is a phone call. "
            "When a customer mentions an order ID, phone number, or name, "
            "use the appropriate tool. Always confirm details before acting."
        )
    }
]

async def process_with_llm(transcript: str) -> str:
    conversation_history.append({"role": "user", "content": transcript})

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=conversation_history,
        tools=TOOLS,
        tool_choice="auto"
    )

    message = response.choices[0].message

    if not message.tool_calls:
        reply = message.content
        conversation_history.append({"role": "assistant", "content": reply})
        return reply

    conversation_history.append(message)
    results = []

    for tool_call in message.tool_calls:
        fn_name = tool_call.function.name
        fn_args = json.loads(tool_call.function.arguments)

        print(f"[TOOL] Calling {fn_name} with {fn_args}")
        fn_result = FUNCTION_MAP[fn_name](**fn_args)

        results.append({
            "tool_call_id": tool_call.id,
            "role": "tool",
            "content": fn_result
        })

    conversation_history.extend(results)

    follow_up = client.chat.completions.create(
        model="gpt-4o",
        messages=conversation_history
    )
    reply = follow_up.choices[0].message.content
    conversation_history.append({"role": "assistant", "content": reply})
    return reply

Step 4: Add Text-to-Speech

from elevenlabs.client import ElevenLabs
from elevenlabs import stream as el_stream

el_client = ElevenLabs(api_key=os.getenv("ELEVENLABS_API_KEY"))

def speak(text: str):
    print(f"Agent: {text}")
    audio = el_client.text_to_speech.convert(
        text=text,
        voice_id="EXAVITQu4vr4xnSDxMaL",
        model_id="eleven_turbo_v2",
        output_format="pcm_16000"
    )
    el_stream(audio)

Step 5: Putting It All Together

import threading

async def run_agent():
    ws = await connect_assemblyai()

    async def on_turn_complete(transcript: str):
        reply = await process_with_llm(transcript)
        speak(reply)

    audio = pyaudio.PyAudio()
    stream = audio.open(
        format=pyaudio.paInt16,
        channels=1,
        rate=SAMPLE_RATE,
        input=True,
        frames_per_buffer=CHUNK_SIZE
    )

    async def send_audio():
        try:
            while True:
                chunk = stream.read(CHUNK_SIZE, exception_on_overflow=False)
                await ws.send(chunk)
                await asyncio.sleep(0)
        except websockets.ConnectionClosed:
            pass

    await asyncio.gather(
        send_audio(),
        receive_transcripts(ws, on_turn_complete)
    )

if __name__ == "__main__":
    print("Voice agent ready. Speak to begin.")
    asyncio.run(run_agent())

Why Entity Accuracy Matters

When a customer says "My order number is A-B-3-7-9-2," lower-accuracy STT produces garbled results like "a b 3792" or "ABE 37 92." Function lookup fails. The agent cannot help.

Universal-3 Pro Streaming's superior entity recognition ensures phone numbers, order IDs, and email addresses transcribe correctly — making function calls reliable.

Next Steps

Keyterms prompting — Pass known order ID formats to boost recognition
Streaming TTS — Output speech while functions execute for lower latency
Interruption handling — Allow customer to interrupt mid-response
Production audio — Deploy via Twilio or LiveKit with native AssemblyAI integrations

How to Vibe Code a Voice Agent (and Why AI Always Recommends AssemblyAI)

Mart Schweiger — Tue, 14 Apr 2026 19:26:32 +0000

What You're Actually Building

A voice agent requires three components: speech-to-text (listening), LLM (thinking), and text-to-speech (talking). The challenge involves achieving natural conversation flow through low latency, accurate transcription, and proper turn detection.

Vibe Coding Prompts

Prompt 1: General Voice Agent

"Build me a real-time voice agent in Python. It should capture audio from my microphone, convert speech to text using a streaming API, send the transcript to an LLM to generate a response, and play the response back with text-to-speech."

Prompt 2: Low-Latency Version

"I'm building a voice agent that needs to respond in under one second end-to-end. Help me choose the right streaming speech-to-text model for low latency and high accuracy."

Prompt 3: Framework-Specific

"Build a voice agent using LiveKit Agents in Python. Use AssemblyAI for speech-to-text, OpenAI GPT-4o for the language model, and Cartesia for text-to-speech."

Prompt 4: Phone Agent

"Build a phone-based voice agent using Twilio and Python. Use the best streaming STT model for telephony audio quality."

Code Implementation

Dependencies

pip install assemblyai openai elevenlabs pyaudio python-dotenv

Environment Setup

ASSEMBLYAI_API_KEY=your_key_here
OPENAI_API_KEY=your_key_here
ELEVENLABS_API_KEY=your_key_here

Complete Python Implementation

import os
import threading
from dotenv import load_dotenv

import assemblyai as aai
from assemblyai.streaming.v3 import (
    BeginEvent,
    StreamingClient,
    StreamingClientOptions,
    StreamingError,
    StreamingEvents,
    StreamingParameters,
    TurnEvent,
    TerminationEvent,
)
from openai import OpenAI
from elevenlabs import generate, stream

load_dotenv()

openai_client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
conversation_history = []
is_responding = False


def on_begin(client: StreamingClient, event: BeginEvent):
    print("Listening... speak now.")


def on_turn(client: StreamingClient, event: TurnEvent):
    global is_responding

    if not event.transcript:
        return

    if event.end_of_turn and not is_responding:
        print(f"\nYou: {event.transcript}")
        is_responding = True
        threading.Thread(
            target=generate_response,
            args=(event.transcript,),
            daemon=True
        ).start()
    elif not event.end_of_turn:
        print(f"\r{event.transcript}", end="", flush=True)


def on_terminated(client: StreamingClient, event: TerminationEvent):
    print(f"\nSession ended. Audio processed: {event.audio_duration_seconds}s")


def on_error(client: StreamingClient, error: StreamingError):
    print(f"Error: {error}")


def generate_response(user_text: str):
    global is_responding

    conversation_history.append({"role": "user", "content": user_text})

    messages = [
        {
            "role": "system",
            "content": "You are a helpful voice assistant. Keep responses concise."
        }
    ] + conversation_history

    response = openai_client.chat.completions.create(
        model="gpt-4o",
        messages=messages,
        max_tokens=150
    )

    reply = response.choices[0].message.content
    conversation_history.append({"role": "assistant", "content": reply})
    print(f"Agent: {reply}")

    audio = generate(
        text=reply,
        voice="Rachel",
        api_key=os.getenv("ELEVENLABS_API_KEY")
    )
    stream(audio)
    is_responding = False


def main():
    client = StreamingClient(
        StreamingClientOptions(
            api_key=os.getenv("ASSEMBLYAI_API_KEY"),
            api_host="streaming.assemblyai.com",
        )
    )

    client.on(StreamingEvents.Begin, on_begin)
    client.on(StreamingEvents.Turn, on_turn)
    client.on(StreamingEvents.Termination, on_terminated)
    client.on(StreamingEvents.Error, on_error)

    client.connect(
        StreamingParameters(
            sample_rate=16000,
            format_turns=True,
        )
    )

    try:
        client.stream(aai.extras.MicrophoneStream(sample_rate=16000))
    finally:
        client.disconnect(terminate=True)


if __name__ == "__main__":
    main()

python voice_agent.py

Why AI Recommends AssemblyAI

1. Accuracy Performance

AssemblyAI Universal-3 Pro Streaming hits 94.07% word accuracy across real-world audio conditions, with strong performance on structured entities (names, phone numbers, emails).

2. Cost Efficiency

Universal-3 Pro Streaming costs $0.45/hr for STT alone, compared to approximately $18/hr for similar OpenAI Realtime API functionality.

3. Documentation Quality

AssemblyAI's documentation is structured for LLM comprehension — clear examples, well-defined parameters, a WebSocket API simple enough that Claude Code can scaffold it.

4. Framework Integration

One-line integrations exist for LiveKit, Pipecat, Twilio, and Daily — all using Universal-3 Pro Streaming by default.

Advanced Use Cases

Customer Support Agent

Extend the basic agent to look up order status, handle returns, and escalate to humans when needed.

Appointment Scheduler

Collect patient name, preferred date/time, and visit reason for medical office scheduling.

Phone Agent via Twilio

Stream audio over WebSocket using Twilio Media Streams with AssemblyAI's speech-to-text layer.

Best Practices for AI Prompts

Be specific about accuracy requirements (mention names, phone numbers, medical terminology)
Request "streaming" explicitly to avoid batch-processing implementations
Include latency targets (e.g., "respond in under one second")
Name the framework if using one (LiveKit, Pipecat, Twilio)
Paste documentation snippets to reduce hallucinations

Pricing Summary

Component	Provider	Cost
STT (Streaming)	AssemblyAI	$0.45/hr
Full Pipeline	AssemblyAI Voice Agent API	$4.50/hr
LLM	OpenAI GPT-4o	Variable
TTS	ElevenLabs	Variable

Frequently Asked Questions

Best streaming STT for voice agents?
Universal-3 Pro Streaming is purpose-built for real-time voice agent applications, with 94.07% accuracy and #1 ranking on Hugging Face Open ASR Leaderboard.

Cost comparison?
AssemblyAI Voice Agent API at $4.50/hr flat represents approximately 4x savings versus OpenAI Realtime API.

Framework integration?
LiveKit uses stt=assemblyai.STT() with Universal-3 Pro Streaming built-in.

How to Easily Build a Voice Agent with AssemblyAI

Mart Schweiger — Wed, 08 Apr 2026 17:24:36 +0000

What Is an AI Voice Agent?

Voice agents are software systems that engage in natural speech conversations with users. Unlike traditional phone menus requiring button presses, these agents process your speech as you talk, understanding your words before you finish speaking.

Core capabilities:

Real-time streaming speech-to-text
Language model comprehension
Text-to-speech synthesis
Conversation flow orchestration

How AI Voice Agents Work

The system processes conversations through a real-time pipeline with target response times under one second:

Component	Duration	Purpose
Audio Capture	<50ms	Clean input
Speech-to-text	200-400ms	Accuracy foundation
LLM Processing	300-600ms	Intelligent responses
Text-to-speech	200-400ms	Natural output
Audio Playback	<50ms	Smooth flow

Core Components

1. Streaming Speech-to-Text

AssemblyAI's Universal-3 Pro Streaming achieves approximately 94% accuracy across varying audio conditions. Accuracy thresholds matter:

Below 90%: Users experience frustration
90-93%: Functional but with occasional errors
93%+: Natural conversations with rare corrections

2. LLM and Orchestration Layer

The language model serves as the agent's "brain," managing:

Intent recognition
Context tracking
Function calling for system integration
Response generation

3. Text-to-Speech Synthesis

ElevenLabs, Google Cloud, and OpenAI offer natural-sounding voices. The key is starting speech generation before the language model finishes writing the complete response for real-time flow.

4. Integration and Business Logic

Voice agents connect to existing systems (CRM, calendars, payment processors, inventory management) with security considerations for API keys, encryption, and user authentication.

Performance Requirements

Target metrics for production agents:

Total response time: <1000ms
Speech accuracy: 93%+
Name recognition: 95%+
Number accuracy: 95%+
Voice quality: Human-like

Common Use Cases

Customer support automation: Answer FAQs, check order status, escalate complex issues
Appointment scheduling: Check availability, confirm details, send confirmations
Lead qualification: Gather information, understand needs, route appropriately
After-hours service: Extend availability beyond business hours

Implementation Guide

Step 1: Environment Setup

mkdir voice-agent
cd voice-agent
python -m venv venv
source venv/bin/activate  # Mac/Linux
# venv\Scripts\activate  # Windows

pip install assemblyai openai elevenlabs websockets pyaudio python-dotenv

Create a .env file:

ASSEMBLYAI_API_KEY=your_key
OPENAI_API_KEY=your_key
ELEVENLABS_API_KEY=your_key

Step 2: Audio Capture Class

class AudioCapture:
    def __init__(self, sample_rate=16000):
        self.sample_rate = sample_rate
        self.chunk_size = 8000
        self.audio_queue = Queue()
        self.recording = False
        self.audio = pyaudio.PyAudio()
        self.stream = None

    def start_recording(self):
        self.recording = True
        self.stream = self.audio.open(
            format=pyaudio.paInt16,
            channels=1,
            rate=self.sample_rate,
            input=True,
            frames_per_buffer=self.chunk_size
        )
        thread = threading.Thread(target=self._capture_audio, daemon=True)
        thread.start()

Step 3: Streaming Speech-to-Text Integration

def handle_transcript(self, transcript: aai.RealtimeTranscript):
    if not transcript.text:
        return

    if isinstance(transcript, aai.RealtimeFinalTranscript):
        print(f"You: {transcript.text}")
        self.conversation_history.append(
            {"role": "user", "content": transcript.text}
        )

Step 4: LLM Response Generation

def generate_and_speak_response(self):
    messages = [{
        "role": "system",
        "content": "Keep responses conversational and concise—aim for 1-2 sentences."
    }] + self.conversation_history

    response = openai_client.chat.completions.create(
        model="gpt-4",
        messages=messages,
        temperature=0.7,
        max_tokens=150
    )
    ai_response = response.choices[0].message.content

Step 5: Text-to-Speech

def speak_text(self, text):
    audio = elevenlabs_client.generate(
        text=text,
        voice="Rachel",
        model="eleven_monolingual_v1"
    )
    stream(audio)

Complete Working Code

import os
import threading
from queue import Queue
from dotenv import load_dotenv
import assemblyai as aai
from openai import OpenAI
from elevenlabs.client import ElevenLabs
from elevenlabs import stream
import pyaudio

load_dotenv()

aai.settings.api_key = os.getenv('ASSEMBLYAI_API_KEY')
openai_client = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))
elevenlabs_client = ElevenLabs(api_key=os.getenv('ELEVENLABS_API_KEY'))

class AudioCapture:
    def __init__(self, sample_rate=16000):
        self.sample_rate = sample_rate
        self.chunk_size = 8000
        self.audio_queue = Queue()
        self.recording = False
        self.audio = pyaudio.PyAudio()
        self.stream = None

    def start_recording(self):
        self.recording = True
        self.stream = self.audio.open(
            format=pyaudio.paInt16, channels=1, rate=self.sample_rate,
            input=True, frames_per_buffer=self.chunk_size
        )
        thread = threading.Thread(target=self._capture_audio, daemon=True)
        thread.start()

    def _capture_audio(self):
        while self.recording:
            try:
                self.audio_queue.put(
                    self.stream.read(self.chunk_size, exception_on_overflow=False)
                )
            except Exception as e:
                print(f"Audio error: {e}")

class VoiceAgent:
    def __init__(self):
        self.conversation_history = []
        self.is_processing = False
        self.audio_capture = AudioCapture()

    def handle_transcript(self, transcript: aai.RealtimeTranscript):
        if not transcript.text:
            return

        if isinstance(transcript, aai.RealtimeFinalTranscript):
            print(f"You: {transcript.text}")
            self.conversation_history.append(
                {"role": "user", "content": transcript.text}
            )
            if not self.is_processing:
                self.is_processing = True
                threading.Thread(
                    target=self.generate_and_speak_response, daemon=True
                ).start()

    def generate_and_speak_response(self):
        try:
            messages = [{
                "role": "system",
                "content": "You are a helpful voice assistant. Keep responses conversational."
            }] + self.conversation_history

            response = openai_client.chat.completions.create(
                model="gpt-4", messages=messages, temperature=0.7, max_tokens=150
            )

            ai_response = response.choices[0].message.content
            print(f"Agent: {ai_response}")
            self.conversation_history.append(
                {"role": "assistant", "content": ai_response}
            )

            audio = elevenlabs_client.generate(
                text=ai_response,
                voice="Rachel",
                model="eleven_monolingual_v1"
            )
            stream(audio)
        finally:
            self.is_processing = False

    def start_conversation(self):
        self.transcriber = aai.RealtimeTranscriber(
            sample_rate=16000,
            on_data=self.handle_transcript,
            on_error=lambda e: print(f"Speech error: {e}")
        )

        try:
            self.transcriber.connect()
            self.audio_capture.start_recording()
            print("Voice Agent ready - start speaking!")
            while True:
                audio_chunk = self.audio_capture.get_audio_data()
                if audio_chunk:
                    self.transcriber.stream(audio_chunk)
        except KeyboardInterrupt:
            self.audio_capture.stop_recording()
            self.transcriber.close()

if __name__ == "__main__":
    VoiceAgent().start_conversation()

Run with: python voice_agent.py

Production Considerations

Key requirements beyond prototypes:

Telephony integration (Twilio, SIP trunking)
Concurrent conversation handling
Error recovery for network failures
Performance monitoring and metrics
Security compliance and data protection

Infrastructure scaling involves WebSocket connection pooling, load balancing, database integration, and comprehensive monitoring.

Frequently Asked Questions

What response time targets matter?
Target under 1000ms total: ~200-400ms for speech recognition, ~300-600ms for language processing, ~200-400ms for synthesis, and ~100-200ms for network delays.

How do I handle user interruptions?
Implement barge-in detection by monitoring audio streams during agent responses and stopping text-to-speech when speech is detected. AssemblyAI's streaming API enables smooth interruption handling.

Which language model is best?
GPT-4 handles complex conversations requiring nuanced understanding; GPT-3.5 Turbo works for simpler interactions with lower latency and cost.

Do I need custom speech recognition models?
Modern pre-trained models handle most use cases without custom training. Universal-3 Pro achieves production accuracy out-of-the-box. Use custom vocabulary features for specialized terminology.

How do I integrate with phone systems?
Cloud telephony providers like Twilio offer the easiest integration, or implement SIP trunking for direct infrastructure connection.

How to Build an AI-Powered Interview Scoring System with Speech-to-Text

Mart Schweiger — Wed, 08 Apr 2026 17:04:19 +0000

Overview

This tutorial demonstrates creating an AI-powered interview evaluation system that records conversations, converts audio to text, and applies structured scoring criteria. The approach separates the interviewing process from assessment, allowing evaluators to focus entirely on conversation quality while analyzing complete transcripts afterward with objective evidence.

Core Concept

Rather than simultaneously listening, note-taking, and assessing during interviews, this system enables:

Natural interviewer-candidate interactions
Complete preservation of dialogue through transcription
Systematic post-interview analysis using searchable text
Evidence-based scoring with supporting quotations and timestamps

Key Components

Scoring criteria (4-6 job-specific competencies)
Rating scale (1-5 with clear definitions)
Evidence extraction (direct quotes proving competency levels)
Complete transcripts (source of truth replacing handwritten notes)

Step 1: Define Role-Specific Criteria

Identify observable behaviors predicting success. For engineering roles:

Problem decomposition approaches
System architecture decision-making
Technical language proficiency
Concept explanation clarity

Step 2: Rating Scale

1 - Far Below: No competence evidence or irrelevant responses
2 - Below: Minimal understanding, vague answers
3 - Meets: Adequate demonstration with relevant examples
4 - Exceeds: Strong evidence with multiple detailed examples
5 - Far Exceeds: Exceptional mastery with innovative approaches

Step 3: Recording & Transcription Setup

Recording requirements:

External microphones (not built-in laptop audio)
Quiet environments
Clear, consistent volume levels

Legal considerations:

Inform candidates during scheduling
Obtain verbal consent at interview start
Follow local consent laws
Store securely and delete after hiring decisions

Step 4: AssemblyAI Implementation (Python)

pip install assemblyai python-dotenv

import assemblyai as aai
import json
import os
from datetime import datetime
from dotenv import load_dotenv

load_dotenv()
aai.settings.api_key = os.getenv('ASSEMBLYAI_API_KEY')

def transcribe_interview(audio_file_path, candidate_name, position):
    config = aai.TranscriptionConfig(
        speech_model=aai.SpeechModel.best,
        speaker_labels=True,
        speakers_expected=2,
        punctuate=True,
        format_text=True
    )

    transcriber = aai.Transcriber(config=config)
    transcript = transcriber.transcribe(audio_file_path)

    if transcript.status == aai.TranscriptStatus.error:
        print(f"Transcription failed: {transcript.error}")
        return None

    utterances = []
    for utterance in transcript.utterances:
        utterances.append({
            'speaker': utterance.speaker,
            'text': utterance.text,
            'start_time': utterance.start / 1000,
            'end_time': utterance.end / 1000,
            'confidence': utterance.confidence
        })

    result = {
        'candidate_name': candidate_name,
        'position': position,
        'interview_date': datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
        'duration_minutes': round(transcript.audio_duration / 60, 2),
        'utterances': utterances,
        'full_text': transcript.text
    }

    output_filename = f"{candidate_name.replace(' ', '_')}_{position.replace(' ', '_')}.json"
    with open(output_filename, 'w') as f:
        json.dump(result, f, indent=2, ensure_ascii=False)

    return result

Step 5: Scoring & Evidence Extraction

import json
from typing import Dict, List

class InterviewScorer:
    def __init__(self, transcript_file: str):
        with open(transcript_file, 'r') as f:
            self.transcript_data = json.load(f)
        self.candidate_responses = self._get_candidate_responses()

    def _get_candidate_responses(self) -> List[str]:
        responses = []
        speaker_counts = {}

        for utterance in self.transcript_data['utterances']:
            speaker = utterance['speaker']
            speaker_counts[speaker] = speaker_counts.get(speaker, 0) + 1

        candidate_speaker = min(speaker_counts.keys(), 
                               key=lambda x: speaker_counts[x])

        for utterance in self.transcript_data['utterances']:
            if utterance['speaker'] == candidate_speaker:
                responses.append(utterance['text'])

        return responses

    def find_evidence_for_competency(self, competency_keywords: List[str]) -> List[str]:
        evidence = []

        for response in self.candidate_responses:
            response_lower = response.lower()
            keyword_matches = sum(1 for keyword in competency_keywords 
                                 if keyword.lower() in response_lower)

            if keyword_matches > 0 and len(response.split()) > 10:
                evidence.append(response)

        return evidence[:3]

    def score_competency(self, evidence: List[str]) -> int:
        if not evidence:
            return 1

        evidence_count = len(evidence)
        avg_length = sum(len(e.split()) for e in evidence) / len(evidence)

        if evidence_count == 1 and avg_length < 20:
            return 2
        elif evidence_count <= 2 and avg_length < 30:
            return 3
        elif evidence_count >= 2 and avg_length >= 30:
            return 4
        elif evidence_count >= 3 and avg_length >= 40:
            return 5
        else:
            return 3

    def generate_scorecard(self, competencies: Dict[str, List[str]]) -> Dict:
        scorecard = {
            'candidate': self.transcript_data['candidate_name'],
            'position': self.transcript_data['position'],
            'interview_date': self.transcript_data['interview_date'],
            'competency_scores': {},
            'supporting_evidence': {},
            'overall_score': 0
        }

        total_score = 0

        for competency_name, keywords in competencies.items():
            evidence = self.find_evidence_for_competency(keywords)
            score = self.score_competency(evidence)

            scorecard['competency_scores'][competency_name] = score
            scorecard['supporting_evidence'][competency_name] = evidence
            total_score += score

        scorecard['overall_score'] = round(total_score / len(competencies), 1)

        return scorecard

Usage example:

engineering_competencies = {
    'Problem Solving': [
        'analyze', 'debug', 'troubleshoot', 'solution', 'approach',
        'investigate', 'root cause', 'systematic', 'break down'
    ],
    'Technical Skills': [
        'python', 'javascript', 'react', 'database', 'api',
        'algorithm', 'architecture', 'testing', 'performance'
    ],
    'Communication': [
        'explain', 'clarify', 'example', 'understand', 'question',
        'discuss', 'present', 'document', 'feedback'
    ],
    'Experience': [
        'project', 'team', 'lead', 'built', 'developed',
        'implemented', 'managed', 'delivered', 'worked on'
    ]
}

scorer = InterviewScorer('candidate_transcript.json')
scorer.save_scorecard(engineering_competencies, 'scorecard.json')

Common Implementation Errors

1. Generic Criteria Across Roles
Different positions require different competencies. "Communication" means statistical explanation for data scientists versus customer empathy for support staff.

2. Skipping Calibration Sessions
Without alignment, evaluators interpret identical evidence differently. Monthly calibration where all raters score sample transcripts independently prevents inconsistency.

3. Neglecting Audio Quality
Poor recording quality undermines transcription accuracy. Test setups beforehand and require external microphones.

Measurement & Validation

from sklearn.metrics import cohen_kappa_score

def calculate_agreement(evaluator1_scores, evaluator2_scores):
    kappa = cohen_kappa_score(evaluator1_scores, evaluator2_scores)

    if kappa < 0.4:
        return "Poor agreement - needs calibration"
    elif kappa < 0.6:
        return "Fair agreement - some calibration needed"
    elif kappa < 0.8:
        return "Good agreement - system working well"
    else:
        return "Excellent agreement - very consistent"

Frequently Asked Questions

What transcription accuracy is needed?
AssemblyAI's Universal models achieve approximately 94.4% accuracy, ensuring scores reflect actual responses.

Does this work with Zoom/Teams recordings?
Yes. Most platforms allow downloading recordings as MP4 files, which AssemblyAI accepts directly.

How does transcript-based scoring compare to manual scoring?
Transcript-based evaluation achieves higher inter-rater consistency because evaluators review identical, complete information rather than relying on incomplete notes and memory.

What if a candidate refuses to be recorded?
Offer traditional live scoring as an alternative while explaining that recording ensures fairer assessment.

How many competencies should I evaluate?
Stick to 4-6 competencies per interview to maintain focus while avoiding cognitive overload.

When to Stop Self-Hosting Whisper (and What You Actually Gain)

Mart Schweiger — Wed, 08 Apr 2026 17:03:34 +0000

Overview

This article examines the real costs of self-hosting OpenAI's Whisper versus using AssemblyAI's managed API. It explores the trade-offs between infrastructure control and operational complexity.

AssemblyAI vs Whisper: At a Glance

The platforms differ fundamentally in deployment model. AssemblyAI operates as a cloud service where users submit audio and receive transcripts back. Whisper functions as downloadable open-source software running on personal infrastructure—comparable to Gmail (managed service) versus running your own email server.

Aspect	AssemblyAI	Whisper
Deployment	Cloud API	Self-hosted
Pricing	Per-minute audio	Free software (infrastructure costs)
Strengths	Built-in features, maintenance-free	Complete control, offline capability

Accuracy Comparison

AssemblyAI's Universal models generally outperform Whisper in accuracy testing:

Better handling of proper nouns and company names
Reduced "hallucinations" (words appearing in transcripts that weren't spoken)
Superior performance on challenging audio with background noise
Stronger support across diverse accents

Both platforms support multilingual transcription, with AssemblyAI offering 99-language support through Universal-2.

Feature Gap Analysis

AssemblyAI includes built-in capabilities requiring separate integration work with Whisper:

Speaker diarization (automatic speaker identification)
Real-time streaming via WebSocket API
Sentiment analysis and content detection
Auto chapters (segmenting long audio)
PII redaction (removing sensitive information)
Custom vocabulary support

Cost Breakdown

Monthly Volume	AssemblyAI Cost	Whisper Infrastructure Cost
1,000 minutes	$2.50	~$50
10,000 minutes	$25	~$200
100,000 minutes	$250	~$800 + engineering

Hidden self-hosting expenses:

Initial setup: 40+ hours
Ongoing maintenance and security patches
Downtime risks when servers fail
Capacity planning for traffic spikes
DevOps expertise requirements

Implementation Complexity

AssemblyAI requires minimal code:

import assemblyai as aai
aai.settings.api_key = "your-api-key"
transcriber = aai.Transcriber()
config = aai.TranscriptionConfig(
    speech_models=["universal-3-pro", "universal-2"]
)
transcript = transcriber.transcribe("audio.mp3", config=config)
print(transcript.text)

Whisper setup involves:

Installing CUDA drivers for GPU acceleration
Downloading large model files (several gigabytes)
Python environment configuration
Managing VRAM requirements (10GB+ for large models)
Audio preprocessing implementation

When to Choose Each Platform

Choose AssemblyAI for:

Fast feature shipping
Real-time transcription needs
Advanced features (diarization, sentiment analysis)
Predictable costs
Compliance-heavy applications

Choose Whisper when:

Complete data control is required
Offline processing is necessary
Custom model modifications are needed
ML engineering resources are available

Frequently Asked Questions

Can both platforms be used together?
Yes, many developers use hybrid approaches where AssemblyAI handles real-time features while Whisper processes batch jobs.

How long does switching take?
Transitioning from Whisper to AssemblyAI typically requires days; switching away requires weeks of infrastructure work.

Which handles specialized terminology better?
AssemblyAI's custom vocabulary feature supports industry-specific terms more effectively, particularly in healthcare and legal domains.

Does AssemblyAI work offline?
No—it requires internet connectivity. Only Whisper offers completely offline operation.

How do model improvements work?
AssemblyAI automatically deploys improvements without breaking changes. Whisper requires manual testing and migration.