Every vendor documents its own path and calls it "how to build a voice agent". Read three of them and you have three incompatible mental models, none of which tells you which one you should be using. That is the actual first decision, and it is an architecture decision rather than a vendor one: where does reasoning live, and where do tools run?
A useful way to group current production voice-agent designs is into three shapes. A single realtime model that hears speech, decides what to do and speaks back. A full-duplex conversational model that handles the conversation and delegates the thinking to a backend you choose. And a chained pipeline where speech-to-text, an agent and text-to-speech are separate stages you own. Plenty of real systems are hybrids of these, so treat the grouping as a way to reason about the tradeoffs rather than a closed taxonomy. Each shape is a legitimate production choice, each fails in a different way, and the components you need around them are almost identical.
None of them wins outright. Which one is right depends on where your business logic already lives, which is why the decision belongs to you rather than to whichever vendor page you happened to read first.
Architecture 1: a single realtime model
Speech goes in, speech comes out, and the same model interprets audio, decides what to do, calls your tools and generates the reply. In OpenAI's Agents SDK a common setup pairs a RealtimeAgent with a RealtimeSession: the session connects over WebRTC in a browser or WebSocket from a server and handles audio turns, tools, interruptions and handoffs, while the agent carries tools, handoffs, guardrails and business logic. Underneath that, at the Realtime API itself, the central abstraction is the realtime session and the conversation events flowing through it. Google's Live API is the same shape over a stateful WebSocket connection, either server-to-server (your backend relays audio) or client-to-server (the browser connects directly, which Google notes generally offers better streaming performance because it skips the relay through your backend, but should use ephemeral tokens rather than API keys in production).
What you get for that is one model, one session, the fewest moving parts, and a model that hears tone and hesitation instead of a flattened transcript.
Where it breaks. Long tool calls are the first problem. The conversation is inside the session, so a tool that takes several seconds risks dead air unless the platform can say something while it waits. Both vendors have an answer, and they are different. OpenAI uses preambles, the short spoken updates that tell the caller work is happening, and its prompting guide notes gpt-realtime-2 generates them by default. Gemini handles it with non-blocking function declarations. A tool declared with behavior: NON_BLOCKING runs asynchronously, and the function response tells the model what to do with the result: surface it immediately, wait until the conversation is idle, or keep it silent for later use. On the current generation this is the default rather than an opt-in. Gemini 3.8 Live defaults to asynchronous execution, keeps synchronous BLOCKING available for backwards compatibility, and supports function scheduling. Gemini 3.8 Live Extended Thinking only accepts non-blocking declarations, returns a hard error for synchronous mode, and does not support scheduling configurations at all, which is worth knowing before you design around it. One practical note: Google's current documentation disagrees on the spelling of the immediate scheduling value, using INTERRUPT in the Live tools guide and INTERRUPTED on the 3.8 model and capabilities pages, so use whichever the SDK or schema you deploy actually accepts.
Policy is the next one. Burying complex business rules in the voice prompt makes those rules harder to test, version and audit, and it increases the amount of policy the conversational model has to carry alongside the instructions that keep its speech natural. The problem people notice last is auditability. Intermediate text exists, but what you are reading is the model's own transcription of a decision it has already made, not a text step you can inspect and gate before it becomes speech.
Architecture 2: full-duplex with a delegated backend
OpenAI's GPT-Live is the clearest implementation of this shape. The voice model does one job: hold the conversation, listen while speaking (that is the full-duplex part), and decide when to ask a backend for help. The backend does the reasoning and the tool work, and it is a component you choose. OpenAI calls sending work to the backend delegation and documents two modes:
- Responses delegation: an OpenAI-hosted model runs the backend reasoning; GPT-Live supplies conversation context and manages the calls; your application still runs its own function tools.
- Client delegation: you connect your own agent, workflow or service, with whatever model and harness you already run, and return results to GPT-Live.
The docs are explicit about the division of labour: keep speaking behaviour in the live model's prompt and business rules in the backend prompt. Your application checks permissions, obtains confirmations, runs the functions that touch your systems and saves task progress.
Two things follow from that, and both need designing rather than assuming. The caller can keep talking while backend work runs, which is the entire point, but it also means an interruption does not automatically cancel backend work. Your application decides whether to finish or cancel it, so a caller who says "no, cancel that" while the backend is mid-refund needs that path wired deliberately. The other is more encouraging: this is a migration path rather than a rebuild. If you already have a text agent with tools and business logic, you can put a voice interface in front of it without moving any of that into a speech prompt.
Where it breaks. You now have two prompts, two failure surfaces and a delegation boundary to observe. The backend can become a major term in the latency budget and should be measured separately.
Architecture 3: a chained pipeline
Speech-to-text, then your agent as ordinary text, then text-to-speech. OpenAI documents it as the path to choose when you want to inspect or transform text between stages; examples of orchestration frameworks include LiveKit Agents and Pipecat, both actively released (livekit-agents 1.8.2 on 15 September 2026, pipecat-ai 1.11.0 on 18 September 2026).
Every stage is inspectable and replaceable: you can store the transcript, run policy checks before the agent answers, call internal systems, and only synthesise speech once the workflow reaches an approved answer. With a clean interface between stages, you can often swap the STT provider without changing the agent's core business logic. Much of an existing text agent, its tools and its policy logic can be reused, though voice still needs its own evaluation and its own handling for turn-taking, interruptions, audible latency and speech. For workflows that require inspectable intermediate text, explicit policy checks or approval before speech is generated, a chained pipeline offers a clearer control surface.
The cost is that you own turn-taking. Knowing when the user has finished speaking is the hardest unglamorous problem in voice, and the frameworks have specialised for it: LiveKit defaults to its turn-detector model for turn completion, which predicts the end of a turn from the meaning of the speech as well as its acoustics, with VAD-only, STT endpointing, realtime-model and manual modes available instead, plus explicit handling for false interruptions, where the framework detects speech and interrupts the agent but the transcription comes back empty. Pipecat splits the problem in two: a turn starts on VAD, or on an arriving transcription when VAD missed quiet speech, with an optional minimum-word strategy on top, and the turn end defaults to its Smart Turn model with a fixed speech timeout as the simpler alternative. You also pay latency at every stage boundary, and the agent reads a transcript rather than hearing the caller.
The components every voice agent needs
Whichever shape you choose, the same list applies. Many production failure modes come from missing one of these components rather than from choosing the wrong high-level architecture.
| Component | What it must do | Notes that apply across vendors |
|---|---|---|
| Transport | Carry audio and events | WebRTC for browsers and mobile, WebSockets for server-side audio, SIP for telephony. OpenAI documents all three for its voice APIs; Gemini Live is WebSocket, with WebRTC available through partner integrations |
| Credentials | Never ship a key to a client | The mechanism is per platform. Realtime API: ephemeral client secrets minted server-side via POST /v1/realtime/client_secrets, or the unified interface where your server brokers the browser's connection. GPT-Live: the project API key stays on a trusted server that sets up the session. Gemini: ephemeral tokens |
| Turn detection | Decide when the caller has finished | Server VAD (silence-based) or semantic detection (model decides from the words). OpenAI exposes server_vad, tuned with threshold, prefix_padding_ms and silence_duration_ms, and semantic_vad, tuned with an optional eagerness value of low, medium, high or auto; frameworks add turn-detector models and endpointing |
| Interruption handling | Stop instantly and forget what was not heard | Who does this depends on the transport. On Realtime WebRTC and SIP connections the server owns the output buffer and truncates unplayed audio automatically. On a WebSocket connection the client owns playback, so it must stop it and send conversation.item.truncate with an audio_end_ms, or the model believes it said things the caller never heard. Note that this trims the audio: OpenAI's docs state the model cannot precisely align transcript to audio, so the transcript is not a reliable record of exactly what was heard |
| Tool execution | Run work, with permissions | Tools may be local functions, or remote MCP servers the platform calls for you. The Realtime API's MCP tool takes a server_label plus either server_url for a remote server or tunnel_id for a local one reached through Secure MCP Tunnel, with optional allowed_tools and require_approval. connector_id is deprecated for models released after 1 September 2026, though existing models keep connector support. Decide deliberately which tools need approval |
| Backend or agent boundary | Hold business rules | Whether it is a delegated backend or the agent stage of a pipeline, business rules belong here, not in the voice prompt |
| Session state and resumption | Survive a dropped connection | Calls drop. Decide what a reconnect restores and what it must not replay |
| Transcripts | Produce a record | Needed for evaluation, disputes and improvement. Check what each API exposes and what has to be enabled. GPT-Live exposes input and output transcript delta events in its event stream. On the Realtime API, assistant output audio has transcript events, while input audio transcription must be configured when you need it |
| Cost accounting | Attribute spend per call | Voice sessions can be billed by duration (GPT-Live is billed per second, and backend model and tool usage is billed separately), so per-call cost is a different calculation from text |
| Evaluation | Test the conversation and the task | See below |
Testing the conversation and the task
OpenAI's evaluation guide makes a distinction worth borrowing: test both the conversation and the completed task. For a booking assistant, listen to the confirmation and check the correct appointment was saved. For GPT-Live specifically, their guide separates task and tool outcomes, conversational timing, speech and language, and session reliability. It also recommends adding complexity in stages: synthetic speech for repeatable single-turn tests, then replayed human recordings, then an independent simulated caller for multi-turn conversations with interruptions and corrections. If you measure latency, change one factor at a time and keep the caller, recording, backend model, prompt, transport, audio cadence and grader fixed when comparing frontend models. Measure acknowledgements like "I'm checking" separately from the answer the caller actually needs.
Choosing
| If this is true | Lean toward | Because |
|---|---|---|
| You already have a text agent with tools and business rules | Full-duplex with delegation, or a chained pipeline | Both reuse the agent you have built with the least rework |
| Tool calls are fast and the domain is narrow | Single realtime model | Fewest moving parts in the voice path, and the model hears the caller directly. Tool implementations, permissions and guardrails still live in your code, but conversational policy, tool coordination and realtime-agent behaviour need careful design |
| Tool calls are slow, or work must continue while the caller talks | Full-duplex with delegation | It is the shape built for backend work overlapping speech; check your platform's cancel semantics |
| Every decision must be inspectable or approved before it is spoken | Chained pipeline | Text between stages is the audit surface |
| Telephony is the primary channel | Any architecture, provided the chosen stack has a SIP or telephony gateway | Native transport support varies by provider. OpenAI documents SIP for its voice APIs; Gemini Live itself is WebSocket and reaches telephony through partner integrations. Check this first, because it constrains the rest |
| You need to switch languages mid-conversation | Verify per model | Language support and whether the language can be set explicitly vary by model family |
| You need retrieval over private documents | Any, with retrieval in the backend or agent stage | Voice changes the interaction layer, not the retrieval architecture. Intent routing and authorization-aware retrieval still belong behind the voice layer. JarvisBitz Engineering works on this class of production AI systems |
Production considerations
Never ship a long-lived API key in browser or mobile code. The replacement differs by vendor. OpenAI mints ephemeral client secrets server-side for untrusted clients (POST /v1/realtime/client_secrets). Google recommends ephemeral tokens rather than standard API keys for production client-to-server Live API applications, and its own documentation marks the ephemeral-token feature as Preview, so recheck its status on the day you publish or ship. Server-to-server architectures are a separate case: there the backend holds the credential and the client never sees it, which is why the server-to-server shape remains the simpler security story.
Reconnects. Design what happens when the connection drops mid-sentence: what the user hears, what the agent believes was said, whether a partially completed task resumes or restarts.
Observability at the boundaries. Record the stages your application can actually see. For a delegated architecture that means delegation receipt, backend request start, first useful result, tool start and end, result submission, audio arrival and client playback. For a pipeline it means the same discipline at each stage. Compare median and 95th percentile across similar calls rather than averages.
Keep the voice prompt about voice. In every architecture, the prompt that controls speaking should control speaking: pace, verbosity, when to acknowledge, when to hand off. Business rules, policy and tool workflows belong in the backend or agent layer where they can be tested as text.
Assume the model list will change. Model identifiers and feature availability have changed quickly in 2026. Pin versions in code, and re-read the model page before a release rather than trusting a blog post, including this one.
When not to build a voice agent
Voice is the right interface less often than the demos suggest.
- The task is a form with four fields. A form is faster for the user and cheaper for you.
- The answer has to be exact and remembered. An address, a reference number, a dosage: the caller will ask you to repeat it and then want it in writing anyway.
- Your knowledge base cannot answer the questions in text. Voice will not fix that. Get retrieval right first, because a voice layer over a system that does not know the answers is just a faster way to disappoint people.
- Nobody can staff the escape path. Define an escape path for cases the agent cannot safely or reliably complete. Depending on the product, that may be a human transfer, a callback, a support ticket or another channel. Without one, the agent becomes a wall.
- Nobody will review the conversations. Voice systems fail in ways dashboards do not show. Review a representative sample of interactions, subject to the product's consent, privacy and retention requirements, and treat automated evaluation as a complement to that review rather than a replacement for it. Without either, the system drifts and nobody notices.
So the decision comes back to where reasoning and tools live, and all three answers are defensible. Pick the one that matches where your business logic already sits, then go through the component list, because that list is what separates a voice agent that survives contact with real callers from one that only demos well.
If you are building one now, take the component table and mark which of the ten your current design has a real answer for. Whatever you cannot answer for is where to start.
Drafted with AI assistance and reviewed, edited and approved by the author.
Top comments (0)