Let any AI agent answer WhatsApp voice notes in 10 minutes

Denegra — Thu, 05 Nov 2026 21:17:56 +0000

Many people on WhatsApp don't type. They hold the mic button and talk. If your AI agent only understands text, those users get nothing back.

In this tutorial you'll connect an AI agent to WhatsApp so that:

a typed message gets a text reply, and
a voice note gets a voice note reply.

The agent can be anything that answers an HTTP POST: a LangGraph graph, a CrewAI flow, an n8n workflow or a ten-line FastAPI app. We'll use the last one so you can see every moving part, then swap in your real agent at the end.

This is about voice notes, the recorded messages WhatsApp users send with the mic button, not live WhatsApp calls.

The glue is agent-voice-bridge, a small open-source FastAPI gateway (Apache 2.0) we wrote for this. We looked for an existing project that combines all three: any agent over HTTP, both Meta and Twilio, and voice note in → voice note out. We didn't find one, so we built it.

How it works

WhatsApp user ──► Twilio ──► agent-voice-bridge ──► your agent
                                │    ▲                  │
                    speech-to-text  text-to-speech      │
                                │    │                  │
WhatsApp user ◄── Twilio ◄──────┴────┴──── reply_text ◄─┘

Twilio sends the incoming WhatsApp message to the bridge.
If it's a voice note, the bridge downloads and transcribes it.
The bridge POSTs one JSON body to your agent.
Your agent answers with {"reply_text": "..."}.
The bridge sends the reply back: synthesised as a voice note if the user spoke, as text if they typed.

Your agent never sees a Twilio or Meta payload. The same agent also works on Meta's WhatsApp Cloud API without changes.

What you need

Docker and Python 3.11+
A free Twilio account (the WhatsApp Sandbox is free for testing)
An OpenAI API key, or an Azure OpenAI resource with a Whisper and a TTS deployment
A tunnel to give your laptop a public HTTPS URL: ngrok or cloudflared
WhatsApp on your phone

Step 1: Write the agent (2 minutes)

This is the whole agent. It just echoes what it heard, which is perfect for checking the pipes.

# echo_agent.py
from fastapi import FastAPI

app = FastAPI()


@app.post("/agent")
async def agent(msg: dict) -> dict:
    return {"reply_text": f"You said ({msg['message_type']}): {msg['content']}"}

pip install fastapi uvicorn
uvicorn echo_agent:app --port 9000

The bridge will send it a body like this:

{
  "sender_id": "whatsapp:+905551112233",
  "message_type": "voice",
  "content": "What time does the store open tomorrow?",
  "channel": "twilio"
}

sender_id is stable per person, so use it as your conversation or session key.

Step 2: Start a tunnel (1 minute)

In a second terminal:

ngrok http 8000

Copy the https://…ngrok-free.app URL. That's your public URL.

Step 3: Join the Twilio WhatsApp Sandbox (2 minutes)

In the Twilio Console, go to Messaging → Try it out → Send a WhatsApp message.
The page shows a sandbox number and a phrase like join two-words. Send that phrase from your WhatsApp to the sandbox number.
Note your Account SID and Auth Token from the Console home page.

Step 4: Configure and run the bridge (3 minutes)

git clone https://github.com/denegra-ai/agent-voice-bridge.git
cd agent-voice-bridge
python3 scripts/configure.py

The setup wizard asks only what your choices need:

Agent URL: http://host.docker.internal:9000/agent (the bridge runs in Docker, your echo agent on your laptop)
Public URL: the ngrok URL from step 2
Channel: Twilio, with your Account SID and Auth Token
Speech providers: OpenAI (or Azure OpenAI) for both speech-to-text and text-to-speech

Keys are typed hidden and checked against the real service before anything is saved. The wizard writes .env and prints the webhook URL you need next.

docker compose up -d --build
curl http://localhost:8000/health
# {"status":"ok","version":"0.1.2","channels":["twilio"]}

Step 5: Point Twilio at the bridge (1 minute)

On the same Twilio page, open Sandbox settings and set When a message comes in to:

https://<your-ngrok-url>/webhook/twilio

Method POST, then Save.

Step 6: Talk to it

Send a text to the sandbox number. You get:

You said (text): hello

Now hold the mic button and say something. A few seconds later you get a voice note back that reads out "You said (voice): …" with your transcript.

Swap in your real agent

Point AGENT_WEBHOOK_URL at anything that accepts the JSON above and returns reply_text. For example, a LangGraph graph with a checkpointer:

from fastapi import FastAPI
from my_agent import graph  # your compiled LangGraph graph

app = FastAPI()


@app.post("/agent")
async def agent(msg: dict) -> dict:
    result = await graph.ainvoke(
        {"messages": [("user", msg["content"])]},
        config={"configurable": {"thread_id": msg["sender_id"]}},
    )
    return {"reply_text": result["messages"][-1].content}

In n8n it's a Webhook node (respond using "Respond to Webhook") and a Respond to Webhook node that returns {"reply_text": "..."} at the end of your workflow.

If your agent needs a token, set AGENT_AUTH_TOKEN and the bridge sends it as Authorization: Bearer ….

Things the bridge handles for you

Timeouts. Twilio and Meta expect a fast webhook response. The bridge acknowledges immediately and does the slow work (transcription, your agent, speech synthesis) in the background, then replies through the provider's API. Your agent can take 20 seconds.
No double answers. The agent call is retried only when your agent cannot have seen the message (connection errors, 429/502/503/504). A 500 or a read timeout is not retried.
Security. Twilio's X-Twilio-Signature and Meta's X-Hub-Signature-256 are checked. Message content is never logged. Generated audio sits at an unguessable URL and is deleted after an hour.
Voice note vs. audio file. OpenAI and Azure OpenAI return Ogg/Opus, which WhatsApp shows as a real voice note. ElevenLabs returns MP3, which arrives as an audio attachment.
Fallbacks. If text-to-speech fails, the user still gets the answer as text.

Going further

Production: the sandbox is for testing. Register your own WhatsApp sender in Twilio, or use Meta's WhatsApp Cloud API directly. Step-by-step guides are in the repo: Twilio, Meta.
Other providers: set STT_PROVIDER=deepgram or TTS_PROVIDER=elevenlabs in .env.
Scaling: webhook dedupe is in memory, so run one process per instance for now.

A note on what's been tested: we've run the full voice round trip on a real phone through the Twilio sandbox, once with Azure OpenAI for speech and once with Deepgram (speech-to-text) and ElevenLabs (text-to-speech). The plain OpenAI option uses the same code as Azure OpenAI but hasn't had its own phone test yet. Meta's Cloud API path is verified up to the signed webhook and the agent call; we haven't yet done a full round trip on a phone through Meta. If you try it, we'd love to hear how it goes in the issues.

The repo is here: https://github.com/denegra-ai/agent-voice-bridge. Stars, issues and PRs are welcome.

agent-voice-bridge is open source and backed by Denegra.

DEV Community: Denegra