DEV Community

Deependra Rajput
Deependra Rajput

Posted on

Lessons from Running AI Voice Calls at Scale

A chatbot request takes a few seconds. A phone call takes minutes, and during those minutes it holds a telephony connection, a speech-to-text stream, LLM calls and text-to-speech output, all of which have their own limits and failure modes.

Here are the lessons that matter most once an AI voice system goes from a handful of test calls to thousands a day.

1. A call is a session, not a request

Web backends are built around short requests. Voice calls are long-lived sessions with state: who's on the line, what's been said, which step of the conversation you're in. Keep that state outside the process handling the audio (Redis works well), so a worker restart or a reconnect doesn't lose the conversation.

2. Limit concurrent calls explicitly

Your telephony provider, your speech services and your LLM provider all cap how much you can run at once. For outbound campaigns, enforce a per-campaign limit before you dial:

async def acquire_call_slot(redis, campaign_id: str, limit: int) -> bool:
    key = f"active_calls:{campaign_id}"
    active = await redis.incr(key)
    if active > limit:
        await redis.decr(key)
        return False
    return True


async def release_call_slot(redis, campaign_id: str) -> None:
    await redis.decr(f"active_calls:{campaign_id}")
Enter fullscreen mode Exit fullscreen mode

Release the slot when the call ends, whatever the reason. Then put calls that can't get a slot back in the queue instead of dropping them.

3. Make webhook handling idempotent

Telephony providers retry webhooks when they don't get a fast response, so the same "call answered" or "call ended" event can arrive twice. Processing it twice means double billing, duplicate records or a confused conversation state.

async def handle_call_event(event, redis) -> None:
    first_time = await redis.set(f"event:{event.id}", 1, nx=True, ex=86_400)
    if not first_time:
        return  # provider retried an event we already processed
    await process(event)
Enter fullscreen mode Exit fullscreen mode

Respond to the webhook quickly, then do the slow work in the background.

4. Budget latency per turn

On a call, silence feels broken after about a second. Each turn adds up: speech-to-text finishing, the LLM's first token, and text-to-speech starting to play. What helps most:

  • Stream everything. Start speaking the first sentence while the rest is still generating.
  • Keep prompts short. Every extra token of context adds time to the first token.
  • Cache the predictable parts. Greetings and common answers don't need a fresh generation every time.

5. Plan for the unhappy paths

Voicemail, no answer, the caller hanging up mid-sentence, background noise, someone saying "stop calling me". Each needs a defined behaviour and a logged outcome. The happy path is maybe half the work.

6. Log outcomes, not just errors

For every call, record how it ended, how long it lasted and where in the flow it stopped. Those outcomes, more than error logs, show where conversations go wrong and what to fix next.

Closing thoughts

Voice AI is mostly distributed-systems engineering with a language model in the middle. Get state, concurrency, idempotency and latency right, and the AI part gets much easier to improve.

Have you built anything with voice AI? I'd love to compare notes in the comments.

Top comments (0)