DEV Community

amitbhakt
amitbhakt

Posted on

How to Build a Real-Time WhatsApp Voice AI Agent in 90 Seconds

#ai #whatsapp #python #webrtc

Connecting a real-time voice AI agent to WhatsApp usually sounds simple until you actually try to build it.

Within minutes, you run into the reality of Meta's telephony plumbing:

  • Digging through Meta Business Manager for obscure IDs.
  • Manually spinning up tunnels and pasting verification tokens.
  • Enabling voice calling flags via Graph API endpoints.
  • Handling WebRTC SDP offer/answer exchanges and SRTP audio.
  • Diagnosing weird 5-second ringing delays before a call even connects.

I got tired of repeating this setup for every voice experiment, so I built an open-source CLI called WAVoice to collapse the entire setup into a single command.

Here is how to get a live WhatsApp AI agent answering calls in under 90 seconds.

What You Need

You only need 3 credentials:

  1. Meta System User Access Token (with whatsapp_business_messaging and whatsapp_business_management permissions)
  2. Meta App Secret (from your Meta App Basic Settings)
  3. Google Gemini API Key (or your own custom LLM key)

Step 1: Run the Setup Command

If you have uv installed, run:

uvx wavoice init
Enter fullscreen mode Exit fullscreen mode

(Or via pip: pip install wavoice && wavoice init)

The wizard prompts for your 3 keys and automatically:

  • Starts an ephemeral Cloudflare Quick Tunnel (public HTTPS URL, zero accounts needed).
  • Auto-discovers your WABA ID and Phone Number ID from your access token.
  • Programmatically turns on WhatsApp Voice Calling via Graph API.
  • Configures webhook subscriptions.
  • Launches the local WebRTC runtime. 
✓ Found WhatsApp Business Account (WABA ID): 2349713195497707
✓ Found Phone Number: +1 555-188-2279 (Verified)
✓ WhatsApp Voice Calling is ENABLED
🚀 WhatsApp Voice Server is live!
Enter fullscreen mode Exit fullscreen mode

Step 2: Test the Call

Open WhatsApp on your phone and tap the call button on your business number.

Your AI voice agent answers immediately. By default, it runs Gemini 2.0 Flash Live with sub-second (~300ms) speech-to-speech response times.

How It Works

WAVoice decouples the telephony networking from your AI logic:

  1. Telephony Layer (Automated): Cloudflare tunnel, Meta webhooks, WebRTC SDP handshake, and SRTP audio decryption.
  2. AI Layer (Pluggable):
    • Mode A (Out-of-the-box): Native Gemini Live S2S (~300ms latency, zero code required).
    • Mode B (Custom Pipeline): Pipecat cascaded stack (STT -> LLM -> TTS).

Using Custom Models (Deepgram, OpenAI, Cartesia)

If you prefer a custom stack (Deepgram STT -> GPT-4o -> Cartesia TTS), WAVoice integrates natively with Pipecat:

import os
from wavoice import WhatsAppVoiceApp
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.services.deepgram import DeepgramSTTService
from pipecat.services.openai import OpenAILLMService
from pipecat.services.cartesia import CartesiaTTSService

app = WhatsAppVoiceApp()

@app.on_call
async def handle_call(transport, call_info):
    caller = getattr(call_info, "from_", "caller")
    print(f"Call received from {caller}")

    pipeline = Pipeline([
        transport.input(),    # Raw WebRTC audio in from WhatsApp
        DeepgramSTTService(api_key=os.environ["DEEPGRAM_API_KEY"]),
        OpenAILLMService(api_key=os.environ["OPENAI_API_KEY"], model="gpt-4o-mini"),
        CartesiaTTSService(api_key=os.environ["CARTESIA_API_KEY"], voice_id="..."),
        transport.output(),   # Raw WebRTC audio out to WhatsApp
    ])

    task = PipelineTask(pipeline=pipeline, params=PipelineParams(allow_interruptions=True))
    runner = PipelineRunner()
    await runner.run(task)

if __name__ == "__main__":
    app.run()
Enter fullscreen mode Exit fullscreen mode

Technical Note: Fixing the 5-Second Ringing Delay

During testing, callers were experiencing 5 to 7 seconds of ringing before the bot picked up.

Looking into aiortc and aioice, we found that macOS creates virtual tunnel interfaces (utun, common with VPNs and Docker). aioice was attempting STUN discovery on all interfaces sequentially, timing out for exactly 5,000ms on unreachable virtual interfaces before returning candidates.

By patching candidate gathering to query the kernel routing table for the primary routable interface (socket.connect(('8.8.8.8', 80))), candidate gathering dropped from 5,004ms down to ~15ms, eliminating the ringing delay completely.

Links & Code

Feedback and PRs are welcome!

Top comments (0)