How We Got Voice AI Response Times Under 1.8 Seconds on Real Phone Calls

Originally published at parvejshah.com/blog/architecting-sub-18s-voice-ai-pipelines by Parvej Shah.

There's a specific type of frustration that's hard to explain unless you've experienced it. You call a business. An automated voice picks up. You ask your question. And then — silence. Not a brief pause. A real silence. Long enough that you start wondering if the call dropped, long enough that you pull the phone away from your ear to check the signal bars.

That silence is what we were trying to eliminate when building the telephony dispatcher for Minions.AI, a voice-based service dispatch platform for trade contractors.

In human conversation, the natural gap between one person finishing a sentence and the other beginning a response is around 200 to 300 milliseconds. Anything beyond 600ms starts to feel awkward. At 2,500ms — which was where the original prototype sat — callers would repeat themselves, raise their voice, or hang up. The call experience was technically functional and practically unusable.

The Sequential Pipeline Problem

The first design was a completely natural one: record audio, run transcription, generate a response, synthesize speech, play it back. Each stage waited for the previous one to finish. The latency budget looked like this:

Stage	Time
Voice Activity Detection (end-of-turn)	800ms
Speech-to-Text transcription	400ms
LLM generation (full response)	1,200ms
Text-to-Speech synthesis	500ms
Total	~2,900ms

That math is catastrophic for a phone call. And it gets worse in real conditions: cellular networks introduce jitter, LLM response times have variance, TTS output buffering adds overhead.

The solution wasn't to make each stage faster in isolation. It was to stop treating them as stages at all.

Replacing Stages with Streams

The rewrite changed the mental model from a sequential pipeline to an overlapping set of event-driven streams. Nothing waits for anything it doesn't strictly have to.

Neural VAD instead of silence timers. The original design waited for 800ms of audio silence before assuming the caller had finished speaking. We replaced this with a WebRTC-compatible neural Voice Activity Detection model running on 20ms audio frames. It detects speech completion at the prosodic level — reading the natural falling intonation of a completed sentence — rather than just measuring decibels.

interface VADConfig {
  frameSizeMs: 20;
  positiveSpeechThreshold: 0.65;
  negativeSpeechThreshold: 0.35;
  minSilenceDurationMs: 280;    // down from 800ms
  prefixPaddingFrames: 3;
}

function handleIncomingAudioFrame(frame: Buffer, vad: NeuralVAD) {
  const isSpeech = vad.process(frame);

  // Immediate barge-in handling
  if (isSpeech && currentAgentState === "SPEAKING") {
    audioOutputBuffer.clear();
    llmAbortController.abort();
    transitionToState("LISTENING");
  }
}

This dropped end-of-turn detection from 800ms to 280ms without causing false cutoffs when callers paused mid-thought.

Overlapping LLM token generation with TTS synthesis. We stopped waiting for the full LLM completion before initiating text-to-speech. As the LLM streams tokens, a boundary detector splits on sentence clauses (periods, commas, clause breaks) and dispatches the first clause to the TTS engine immediately.

async function streamToTTS(
  tokenStream: AsyncIterable<string>,
  ttsEngine: StreamingTTSClient,
  audioSink: AudioStreamSink
) {
  let buffer = "";
  const sentenceEndPattern = /[.!?,;:]\s+/;

  for await (const token of tokenStream) {
    buffer += token;
    const match = buffer.match(sentenceEndPattern);

    if (match && match.index !== undefined) {
      const clause = buffer.slice(0, match.index + 1).trim();
      buffer = buffer.slice(match.index + match[0].length);

      if (clause.length > 0) {
        // Synthesize and stream the first clause immediately
        const audioChunk = await ttsEngine.synthesizeClause(clause);
        await audioSink.enqueue(audioChunk);
      }
    }
  }

  // Flush any remaining text in buffer
  if (buffer.trim().length > 0) {
    const finalAudio = await ttsEngine.synthesizeClause(buffer.trim());
    await audioSink.enqueue(finalAudio);
  }
}

The first audio chunk starts synthesizing while the LLM is still generating the second half of the response. The caller hears the first word of the answer within ~600ms of the LLM receiving the prompt, rather than waiting 1,200ms for the full sentence to generate.

Speculative Tool Pre-fetching

The largest latency spike occurred whenever the agent needed to call an external tool — checking technician availability in the CRM or looking up an address in the dispatch database. Standard tool-calling waits for the LLM to output a tool_call token sequence, executes the function, appends the result to the context, and prompts the LLM again. That roundtrip routinely added 1,100ms.

We implemented speculative pre-fetching: while the user is still speaking and the interim transcription suggests a booking intent (e.g., "Do you have anyone available on Thursday..."), a background worker queries the technician availability API with predicted date parameters before the utterance completes.

// Interim transcript listener for speculative pre-fetch
sttStream.on("interim_transcript", (partialText: string) => {
  const intentPrediction = fastIntentClassifier(partialText);

  if (intentPrediction.type === "CHECK_AVAILABILITY" && intentPrediction.confidence > 0.85) {
    // Pre-warm the cache before the caller finishes speaking
    technicianScheduleCache.prefetch({
      date: intentPrediction.extractedDate ?? getNextBusinessDay(),
      serviceType: intentPrediction.serviceType,
    });
  }
});

By the time the LLM formally issues the tool call, the database result is already warm in Redis cache. Tool execution time dropped from 380ms to 4ms.

The Result: Sub-1.8s in Real Conditions

Here is the latency budget before and after the architecture change:

Stage	Before	After	Delta
Voice Activity Detection (end-of-turn)	800ms	280ms	-520ms
Speech-to-Text	400ms	190ms (streaming)	-210ms
LLM Time-to-First-Token	1,200ms (full completion)	310ms (first token)	-890ms
TTS First Audio Chunk	500ms (full audio)	220ms (first clause)	-280ms
Network Jitter Buffer	0ms	180ms (added for stability)	+180ms
Total Response Latency	~2,900ms	~1,180ms – 1,450ms	~55% faster

Under real cellular phone call conditions (varying 4G/5G latency, background noise, slight packet loss), the median round-trip time consistently stays under 1.8 seconds.

At 1.4 to 1.8 seconds, the interaction stops feeling like a command-and-response terminal and starts feeling like a conversation. Callers don't talk over the agent, they don't repeat themselves, and appointment conversion rates increased by over 30% compared to the sequential prototype.

---

Parvej Shah is a Lead Full-Stack Web Developer & Platform Architect based in Dhaka, Bangladesh. Explore full architecture case studies and production code at parvejshah.com.