How AI Calling Agents Work: STT, LLM and TTS in One Call

#ai #llm #python #architecture

A voice AI agent looks like one thing on the phone, but a single call is three systems passing data to each other: speech-to-text (STT), a language model (LLM) and text-to-speech (TTS). Most quality problems come from how those three are connected, not from any one of them.

This post walks through the pipeline, shows a small runnable skeleton in Python, and lists the design choices that matter in production.

The three stages

STT turns the caller's audio into text. Streaming STT sends partial transcripts while the caller is still speaking, and a final transcript when they stop.
LLM reads the transcript plus conversation context and produces a reply, token by token.
TTS turns reply text into audio that is played back to the caller.

Around these sit telephony (the audio transport), turn detection (deciding when the caller has finished) and business logic (tools, CRM lookups, escalation to a human).

The naive version and why it feels slow

The simplest design runs the stages one after another: wait for the full transcript, wait for the full LLM reply, then synthesize the whole reply. Each stage waits for the previous one to finish, so the caller hears silence for the sum of all three. On a phone call that gap feels like the line has dropped.

Stream between stages

The fix is to overlap the stages. Start speaking as soon as the first sentence of the reply is complete, instead of waiting for the entire reply.

Here is a minimal skeleton. The three functions are stand-ins for real STT, LLM and TTS clients. The structure is the point:

import asyncio
import re

SENTENCE_END = re.compile(r"(?<=[.!?])\s+")


async def transcribe(audio_chunks):
    """Stand-in for a streaming STT. Yields (text, is_final)."""
    words = ["what", "are", "your", "support", "hours"]
    for i, w in enumerate(words):
        await asyncio.sleep(0.01)
        yield " ".join(words[: i + 1]), i == len(words) - 1


async def llm_stream(prompt):
    """Stand-in for a streaming LLM. Yields tokens."""
    reply = "We are open from nine to six. Monday to Saturday. Anything else?"
    for tok in reply.split(" "):
        await asyncio.sleep(0.005)
        yield tok + " "


async def synthesize(sentence):
    """Stand-in for TTS. Returns audio bytes for one sentence."""
    await asyncio.sleep(0.01)
    return sentence.encode()


async def speak(audio_out, audio):
    await audio_out.put(audio)


async def handle_turn(audio_chunks, audio_out):
    final_text = ""
    async for text, is_final in transcribe(audio_chunks):
        if is_final:
            final_text = text

    buffer = ""
    async for token in llm_stream(final_text):
        buffer += token
        parts = SENTENCE_END.split(buffer)
        # every part except the last is a complete sentence
        for sentence in parts[:-1]:
            await speak(audio_out, await synthesize(sentence))
        buffer = parts[-1]
    if buffer.strip():
        await speak(audio_out, await synthesize(buffer.strip()))

Running it prints the reply one sentence at a time:

We are open from nine to six.
Monday to Saturday.
Anything else?

The first sentence reaches TTS while the LLM is still generating the rest. In a real system the caller starts hearing audio after the first sentence, not after the whole reply.

Design choices that matter

Turn detection. Cutting the caller off is worse than a small delay. Use the STT's end-of-speech signal plus a short silence window, and tune that window against real calls.

Barge-in. Callers interrupt. When new speech arrives while the agent is talking, stop playback, cancel the pending LLM and TTS work, and start a new turn.

Context and tools. Keep the prompt small. Fetch data (order status, appointments) with tool calls, and have the agent say a short filler line while a slow tool runs.

Fallbacks. Decide in advance what happens when STT confidence is low, an API times out, or the caller asks for a human. A clear handoff beats a confident wrong answer.

Measure it. Log timestamps per stage for every turn so you can see where time goes. Without per-stage numbers, you are guessing.

Summary

A calling agent is a streaming pipeline, not three separate calls. Overlap the stages, handle interruptions, and measure each stage. The model matters, but the plumbing decides whether the call feels natural.

We build conversational voice agents at VRM AI Technology. If you are planning a Generative AI project, see our generative AI development work too.

DEV Community