Why a talking chatbot and a real voice agent are not the same thing
A voice AI agent looks deceptively simple from the outside. You speak. It listens. It thinks. It responds. But underneath that simple exchange sits a real-time distributed system juggling audio streams, speech detection, transcription, reasoning, tool execution, state, speech synthesis, interruptions, latency budgets, and — sometimes — a handoff to a human.
This post breaks down what's actually happening under the hood, and why the hardest problems in voice AI today have less to do with the language model and more to do with the runtime around it.
Not a chatbot with a microphone
A text chatbot is fundamentally a request-response system: a message comes in, the model reasons, a response goes out. A voice agent has to operate continuously in time. Audio comes in as a stream, and the system must detect speech, figure out whether the user is actually finished talking, transcribe, reason, possibly call a tool, generate a response, synthesize it into audio, and play it back — all while still listening.
That last part matters more than it sounds. The user might interrupt with "wait," or "no, I meant tomorrow," or "stop, that's not what I asked." A good voice agent has to notice that, stop itself mid-sentence, hold onto the relevant context, and pick up from the new input. This is called barge-in, and modern voice runtimes treat it — along with turn detection — as a first-class architectural concern, not an afterthought.
Who has the floor?
Human conversation feels effortless partly because our brains are constantly solving one question: who should be speaking right now? Voice agents have to solve the same problem computationally, and it's harder than it looks.
Consider someone saying "I'd like to book an appointment..." and then pausing. Are they done, or mid-thought? If they continue with "...for tomorrow afternoon," the pause meant nothing. But if they'd said the whole sentence and stopped, the agent should respond. Detecting speech isn't the same as detecting a finished thought — and that distinction is the root of two related but different technologies: VAD and turn detection.
VAD: is there speech?
Voice Activity Detection (VAD) answers one narrow question: does this audio currently contain speech? It analyzes incoming audio frames and estimates whether what it's hearing is a human voice versus silence, a keyboard, a fan, traffic, or a door slamming.
Its limitation is important: VAD doesn't understand what is being said. It can tell you "there appears to be speech," but it can't tell you "the user has finished expressing their thought." That's a different problem entirely — which is why modern systems pair VAD with endpointing or model-based turn detection rather than relying on it alone.
Endpointing: how long is long enough?
Once VAD detects speech turning into silence, the system has to decide how much silence means "the user is done." Is 200 milliseconds enough? 500ms? A full second? There's no universal answer, and getting it wrong in either direction breaks the experience.
Wait too little, and the agent jumps in on "I need to book—" before the user finishes. Wait too long, and a simple exchange starts to feel sluggish and robotic. This is the endpointing problem, and while a simplified version just runs a silence timer against a fixed threshold, production systems increasingly use models that weigh both the acoustic signal and the semantic content of the utterance to decide whether a thought is actually complete. LiveKit's documentation on turn detection is a good example of this hybrid approach in practice.
Backchannels: the "yeah" problem
Humans constantly emit small sounds while listening — "yeah," "uh-huh," "okay," "right." These aren't new conversational turns; they're backchannels, signals that the listener is still engaged. If an agent mid-sentence hears "yeah" and interprets it as an interruption, it'll stop talking unnecessarily and the conversation starts to feel broken.
So the real classification problem an interruption system faces isn't just "is there speech," it's: is this a genuine interruption, a backchannel, background noise, or an echo of the agent's own voice? That's a meaningfully harder problem than measuring volume, and it's an active area of development — LiveKit's adaptive interruption handling is one example of a system built specifically to make this distinction.
Barge-in: where voice agents become conversational
Barge-in — letting the user interrupt the agent mid-response — is the moment a voice agent starts to feel genuinely conversational rather than scripted. But implementing it well requires tight coordination across several components at once. When the agent is announcing an appointment confirmation and the user cuts in with "wait!", the system has to detect the incoming speech, confirm it's a real interruption, halt text-to-speech playback, flush any queued audio, capture and transcribe the new input, preserve the conversation state, decide what happens to whatever operation was in flight, and generate the next response — all within a fraction of a second.
This is why a production voice agent is, at its core, a real-time event-driven system rather than a simple pipeline.
The echo problem nobody thinks about
There's a subtler issue lurking in the audio layer: the agent's own voice, played through the user's speaker, can travel back into the microphone. Without proper handling, the system ends up hearing itself — and can effectively interrupt its own response. Solving this requires acoustic echo cancellation, noise suppression, gain control, and careful audio routing.
It's easy to overlook, but if the audio pipeline is weak, the AI can look bad even when the underlying model is excellent. The audio layer is, in a very practical sense, part of the agent's intelligence.
Why streaming changes everything
A naive voice pipeline waits for each stage to fully complete before starting the next: the user finishes speaking, transcription completes, the LLM finishes generating, speech synthesis completes, and only then does audio play. That creates a lot of dead air.
Production pipelines instead treat each handoff — STT to LLM, LLM to TTS — as a streaming interface. Partial transcripts feed the LLM as they arrive; partial LLM output feeds text-to-speech as it's generated; audio plays in chunks rather than waiting for the whole response. LiveKit describes this distinction explicitly, framing streaming across every stage boundary as a defining trait of production-quality pipelines. The effect on perceived latency is substantial — a system can feel instantaneous even when the total compute time hasn't changed much, simply because nothing is waiting on anything else to fully finish.
Latency isn't one number
Claims like "our voice agent has 800ms latency" don't mean much without more context, because a single voice interaction is made up of several latency contributors stacked together: audio capture, VAD and turn detection, speech-to-text, LLM reasoning, tool execution, text-to-speech, network transport, and playback.
Teams building serious voice systems typically track each of these separately — time to speech detection, time to end-of-turn, STT latency, LLM time-to-first-token, TTS time-to-first-audio, tool latency, and end-to-end latency — because a single blended number hides which part of the pipeline is actually slow.
The agent runtime: the part hiding behind "AI voice agent"
Everything described so far — turn detection, interruption handling, streaming, state, tools — needs to be coordinated by something. That something is the agent runtime: the system responsible for state, memory, policies, tool orchestration, turn management, recovery, interruptions, and observability, all wrapped around the STT/LLM/TTS core.
Frameworks like LiveKit Agents exist specifically to provide these abstractions — the realtime audio pipeline, turn detection, interruption handling, tool calling, and orchestration — so teams don't have to build them from scratch. The LLM provides the reasoning; the runtime is what makes that reasoning usable in a live, real-time environment.
Tools turn a voice bot into an agent
A voice assistant that can only answer questions is limited. An agent can act — check availability, book an appointment, update a CRM record. But that requires more than the LLM generating plausible-looking JSON for a function call. The runtime has to validate the arguments, execute the tool, handle errors, enforce permissions, update state, decide whether to retry, and determine what the user should actually hear as a result. Tool orchestration is where a voice bot starts to become a genuine agent.
State and memory are not the same thing
These two concepts get conflated constantly, but they answer different questions.
State is: what is happening right now in this conversation? Current intent, current step, information collected so far.
Memory is: what information should persist across a longer period — the customer's name, their preferred branch, their last appointment date?
Neither should simply mean "stuff everything into the prompt." Treating state and memory as distinct engineering concerns — with their own storage, lifecycle, and update logic — is what lets an agent handle interruptions, tool failures, human handoffs, dropped calls, and changes of mind without losing the thread of the conversation.
What happens when a tool fails?
This is where demos turn into production systems. If a booking API times out, the worst possible response is "Your appointment is booked!" when it isn't. A better system says "I'm having trouble confirming that right now, let me try again" or escalates to a human. The runtime needs explicit policies for timeouts, retries, fallback paths, and human handoff.
There's a subtler trap here too: what if the booking actually succeeded, but the response timed out? Blindly retrying in that case could create a duplicate booking. This is why idempotency — ensuring an operation can be safely retried without side effects — becomes a serious design concern. Production voice agents need to reason about failure at the workflow level, not just the model level.
Interruptions during tool calls are even harder
Imagine the user says "book Friday at 3," the agent starts the booking, and mid-call the user says "actually, make it 4." What happens to the operation already in flight? There are a few real options: cancel it if cancellation is supported, let it complete and modify afterward, temporarily disallow interruption for irreversible actions, or require explicit confirmation before executing anything irreversible.
The underlying principle is that not every part of an agent should be interruptible — and deciding which parts aren't is a deliberate design choice, not a default.
Human handoff is part of the agent, not a failure of it
A production agent needs to know when not to be autonomous. Common triggers include repeated misunderstanding, low confidence, sensitive requests, an upset customer, tool failures, or an explicit request to speak to a person.
A good handoff isn't just "I'll transfer you" followed by dumping a confused human into a blank conversation. The human agent should receive a summary: who the customer is, what they wanted, what's already been tried, and what the recommended next action is. The AI doesn't disappear at the handoff — it prepares the human to pick up where it left off.
Observability: knowing why a call actually failed
When something goes wrong, "call unsuccessful" isn't a useful answer. A proper trace breaks a call down by component — how many turns, how many false endpoints, average STT and LLM latency, which tool calls succeeded or failed and how long they took, TTS latency, and the final outcome. With that level of detail, an engineer can actually ask useful questions: why was this call slow, why are users getting interrupted incorrectly, which tool is failing most often. Observability is what turns a voice agent from a mysterious black box into an engineering system that can be debugged and improved.
The voice agent as a state machine
One useful way to think about the whole runtime is as a state machine cycling through idle, listening, thinking, optionally executing a tool, speaking, and — if the user interrupts — back to listening via a barge-in path. That loop, running continuously, is a better mental model than the simple "input → LLM → output" picture most people start with.
Managed platforms vs. building your own runtime
This all creates a real architectural decision: build the runtime yourself, or use a managed voice platform that provides much of it out of the box?
Managed platforms typically handle audio transport, VAD, and turn detection for you, and provide integrations for STT, LLM, and TTS providers — trading flexibility for speed. Building your own runtime means owning state, memory, recovery, and observability directly, at the cost of more engineering effort. Neither is inherently better: shipping an appointment-booking agent quickly favors a managed platform, while building differentiated voice infrastructure favors owning more of the stack. Frameworks like LiveKit sit deliberately close to the runtime layer, giving developers an open framework and letting them choose their own AI providers rather than locking them in.
The real engineering trade-off
The most important question in voice AI today isn't "which LLM is best?" It's: which runtime architecture gives the best combination of latency, reliability, flexibility, and cost for this specific workload?
Optimizing for latency means streaming STT, LLM, and TTS, using preemptive generation, and tuning turn detection to be fast. Optimizing for reliability means explicit state, typed tools, retries, timeouts, idempotency, fallback paths, and human escalation. Optimizing for cost means smaller models where they're sufficient, model routing, shorter prompts, and caching. Optimizing for control means open frameworks, owned infrastructure, and provider abstraction. There's no single optimal architecture — only an optimal one for a given workload.
Why voice AI is becoming a runtime problem
Voice AI's early questions were about the model: can it talk, can it understand speech, can it hold a conversation? The questions that matter now are about everything around the model: can it interrupt naturally, recover from errors, execute tools safely, maintain state, run with low latency, scale, hand off to a human, and be observed when something breaks?
The LLM is still critical — but it's no longer the whole product. The interesting engineering work has moved to the runtime.
The takeaway
A production voice agent isn't "microphone → LLM → speaker." It's audio infrastructure, VAD, turn detection, speech-to-text, an agent runtime with state and memory, the LLM itself, tool orchestration, failure recovery, speech synthesis, barge-in handling, observability, and human escalation — all cooperating in real time.
Building a convincing voice demo is relatively easy. Building a voice agent that can handle ten thousand messy, real-world conversations without breaking is a completely different engineering problem — and that's where the interesting work is happening now.
One sentence to remember: a voice agent isn't an LLM that can talk. It's a real-time control system that happens to use an LLM.
Further reading: LiveKit's Agents documentation covers realtime voice agent architecture, turn detection, and orchestration in more depth; their turn detection and adaptive interruption handling docs go deeper into endpointing and backchannel detection specifically; NVIDIA has also published on barge-in architectures using continuous ASR during TTS playback.
Top comments (0)