How We Built a Sub-500ms Real-Time AI Voice Agent with TypeScript, Twilio & WebSockets

When building conversational AI phone agents for real-world production environments (such as after-hours dental clinic receptionists or real estate lead qualifiers), latency is everything.

In human conversation, the natural pause between speakers averages 200ms to 400ms. If an AI telephone system takes 1.5 to 3 seconds to respond:

• Callers talk over the AI (conversational collision).
• Users assume the line disconnected and hang up.
• Trust evaporates instantly.

In this tutorial, we share the exact streaming architecture we engineered at Hikari Webworks to achieve sub-500ms end-to-end voice latency in production.

---

🏗️ The Latency Budget Breakdown

Sequential HTTP request-response cycles (Record audio -> Send POST -> Wait for LLM -> Generate MP3 -> Play audio) add at least 2.5 seconds of artificial delay.

To achieve sub-500ms, the entire pipeline must operate over full-duplex bi-directional WebSockets:

[Phone Caller] 
      │ (8kHz raw audio stream over PSTN/SIP)
      ▼
[Twilio Media Stream WebSocket]
      │
      ├── (80ms) ──► Deepgram Nova-2 (Live Streaming STT)
      │
      ├── (140ms) ─► Streaming LLM (Groq Llama 3.3 70B / Claude 3.5 Haiku)
      │
      ├── (90ms) ──► Ultra-Fast TTS (Cartesia Sonic / ElevenLabs Turbo)
      │
      └── (40ms) ──► Raw Mu-Law Audio Chunk Streaming back to Caller

Total Turnaround Time: ~350ms - 480ms (Sub-500ms Real-Time Loop)

---

1. Setting Up the Twilio WebSocket Stream in TypeScript

When a call connects, Twilio opens a WebSocket connection and streams 8kHz 16-bit Mu-law audio packets encoded in base64:

// server/voice-agent.ts
import { WebSocketServer, WebSocket } from 'ws';

interface TwilioMediaMessage {
  event: 'connected' | 'start' | 'media' | 'stop' | 'mark' | 'clear';
  streamSid?: string;
  media?: {
    payload: string; // Base64 encoded 8kHz mu-law audio
    timestamp: string;
    chunk: string;
  };
}

export function initializeVoiceServer(port: number = 8080) {
  const wss = new WebSocketServer({ port });

  wss.on('connection', (ws: WebSocket) => {
    let streamSid = '';

    ws.on('message', async (data: string) => {
      const msg: TwilioMediaMessage = JSON.parse(data);

      if (msg.event === 'start' && msg.streamSid) {
        streamSid = msg.streamSid;
        console.log(`[Twilio] Call stream active: ${streamSid}`);
      }

      if (msg.event === 'media' && msg.media) {
        // Decode raw audio frame and stream immediately to Speech-to-Text
        const rawChunk = Buffer.from(msg.media.payload, 'base64');
        liveSttStream.send(rawChunk);
      }

      if (msg.event === 'stop') {
        console.log(`[Twilio] Call ended: ${streamSid}`);
      }
    });
  });

  console.log(`AI Voice Gateway listening on port ${port}`);
}

---

2. Solving the Barge-In (Interruption) Problem

One of the most critical engineering challenges in voice AI is handling human interruptions. If the AI is in the middle of speaking and the user says "Wait, how much does that cost?", the AI must shut up instantly.

To achieve this:

1. Client-side Voice Activity Detection (VAD) detects user speech onset in < 50ms.
2. A clear signal is immediately dispatched to Twilio to purge the pending audio playback buffer.
3. In-flight text-to-speech generation is aborted.

// Handling real-time human barge-in
function handleUserInterruption(ws: WebSocket, streamSid: string) {
  // Purge Twilio's audio queue immediately
  const clearPayload = JSON.stringify({
    event: 'clear',
    streamSid: streamSid
  });

  ws.send(clearPayload);

  // Abort pending TTS stream chunk generator
  ttsStreamController.abort();
}

---

3. Streaming Structured Tool Execution (Real Estate Lead Example)

In our Hikari Real Estate AI Solution, the voice agent qualifies buyer budget and timeline, books an executive site tour, and dispatches project brochures over WhatsApp in under 2 seconds:

// Define structured function calling schemas
const qualificationTools = [
  {
    type: "function",
    function: {
      name: "confirm_site_visit_and_send_whatsapp",
      description: "Confirms property site visit and immediately sends location & e-brochure over WhatsApp.",
      parameters: {
        type: "object",
        properties: {
          buyerName: { type: "string" },
          phoneNumber: { type: "string" },
          unitPreference: { type: "string", enum: ["2BHK", "3BHK", "Villa"] },
          budgetLakhs: { type: "number" },
          visitDateTime: { type: "string" }
        },
        required: ["buyerName", "phoneNumber", "unitPreference", "visitDateTime"]
      }
    }
  }
];

---

📊 Production Performance Metrics

Across over 50,000 live conversations handled across our AI Phone Agent Deployments and Enterprise AI Operations:

1. Sub-60s Speed-to-Lead: Inbound ad leads receive a voice qualification call in under 45 seconds.
2. 4.2x Qualification Rate: Real-time speech triage recovers 38% of previously lost after-hours inquiries.
3. Zero Hallucination: Strict deterministic function calling schemas prevent inaccurate answers.

---

🚀 Key Takeaways & Resources

• Do not build production voice agents using sequential HTTP requests.
• Use full-duplex bi-directional WebSockets with 8kHz Mu-law audio.
• Always implement instantaneous buffer purging for natural interruption handling.

To test live interactive ROI benchmarks and voice latency cost estimators, explore our Web & AI Cost Calculator or connect directly with our engineering team at Hikari Webworks.