DEV Community

VoiceDeveloper
VoiceDeveloper

Posted on

ElevenLabs + Twilio: Building Voice Call Applications

#webdev #javascript #tutorial #ai

Why combine ElevenLabs and Twilio?

If you’ve ever wanted to give your users a truly personal voice experience—think automated support lines, interactive storytelling, or dynamic notifications—pairing ElevenLabs’ state‑of‑the‑art text‑to‑speech (TTS) with Twilio’s programmable voice platform is a match made in the cloud.

  • ElevenLabs provides ultra‑realistic voice cloning and a fast REST API that can turn any string into a high‑quality audio file in seconds.
  • Twilio lets you make, receive, and control phone calls programmatically, delivering the audio to any PSTN or SIP endpoint.

Together, you can generate a custom voice on the fly and pipe it straight into a live call, all with just a few lines of code.

What you’ll need

Item Why it matters
ElevenLabs API key Authenticates your TTS requests. Sign up and grab a key from the dashboard.
Twilio account & phone number Required to place outbound calls and host TwiML.
A server (Flask, Express, etc.) Hosts the webhook that Twilio hits to fetch your generated audio.
Python 3.8+ (or Node.js) The examples below use Python, but the same concepts apply in any language.

Quick tip: If you haven’t tried ElevenLabs yet, you can sign up through this affiliate link: https://try.elevenlabs.io/kr07zfuqn1bp – it’s the same product, just a convenient way to get started.

Generating speech with ElevenLabs

ElevenLabs exposes a simple /v1/text-to-speech/{voice_id} endpoint. Here’s a minimal Python function that sends a prompt and returns a URL to the generated MP3.

import requests
import os

ELEVEN_API_KEY = os.getenv("ELEVEN_API_KEY")
VOICE_ID = "EXAVITQu4vr4xnSDxMaL"   # default “Rachel” voice; replace with your cloned voice ID

def generate_speech(text: str) -> str:
    url = f"https://api.elevenlabs.io/v1/text-to-speech/{VOICE_ID}"
    headers = {
        "xi-api-key": ELEVEN_API_KEY,
        "Content-Type": "application/json"
    }
    payload = {
        "text": text,
        "model_id": "eleven_monolingual_v1",
        "voice_settings": {
            "stability": 0.75,
            "similarity_boost": 0.85
        }
    }

    resp = requests.post(url, json=payload, headers=headers)
    resp.raise_for_status()
    # The API returns raw audio bytes; we’ll store them in a temporary file.
    filename = f"/tmp/{hash(text)}.mp3"
    with open(filename, "wb") as f:
        f.write(resp.content)
    return f"file://{filename}"
Enter fullscreen mode Exit fullscreen mode

A few things to note:

  1. Voice cloning – If you’ve uploaded a custom voice to ElevenLabs, replace VOICE_ID with that ID. The same endpoint works for both built‑in and cloned voices.
  2. Stability & similarity – Tweak these numbers to make the voice sound more expressive or more faithful to the source speaker.
  3. File handling – In production you’d probably upload the MP3 to a CDN (S3, Cloudflare R2, etc.) and return a public URL. Twilio can only stream from an HTTP(S) address, not a file:// path.

Serving the audio to Twilio

Twilio expects a publicly reachable URL that returns a media file with a supported MIME type (audio/mpeg, audio/wav, etc.). A quick Flask route can bridge the gap:

from flask import Flask, request, send_file, abort
import uuid

app = Flask(__name__)

# In‑memory map: request_id → local file path
audio_store = {}

@app.route("/speak", methods=["POST"])
def speak():
    data = request.json
    if not data or "text" not in data:
        abort(400, "Missing 'text' field")

    # Generate speech and keep a reference
    audio_url = generate_speech(data["text"])
    request_id = str(uuid.uuid4())
    audio_store[request_id] = audio_url.replace("file://", "")

    # Return a TwiML response that tells Twilio where to fetch the audio
    twiml = f"""<?xml version="1.0" encoding="UTF-8"?>
<Response>
    <Play>{request.host_url}audio/{request_id}</Play>
</Response>"""
    return twiml, 200, {"Content-Type": "application/xml"}

@app.route("/audio/<uid>")
def serve_audio(uid):
    path = audio_store.get(uid)
    if not path or not os.path.exists(path):
        abort(404)
    return send_file(path, mimetype="audio/mpeg")
Enter fullscreen mode Exit fullscreen mode

When Twilio hits /speak with a JSON payload like {"text":"Hello, this is a test!"}, the endpoint:

  1. Calls ElevenLabs to synthesize the voice.
  2. Stores the resulting MP3 locally (or you could upload it to a CDN).
  3. Returns TwiML that instructs Twilio to <Play> the generated file.

Wiring it up with Twilio

1. Create an outbound call webhook

In the Twilio Console, go to Phone Numbers → Manage → Active Numbers, select your number, and set the Voice & Fax → A CALL COMES IN webhook to point at your /speak endpoint, e.g.:

https://my‑app.example.com/speak
Enter fullscreen mode Exit fullscreen mode

Make sure the request method is POST (the Flask route expects JSON).

2. Trigger a call from your code

You can launch a call from anywhere—your backend, a CLI script, or even a Zapier action. Here’s a simple Python snippet using twilio-python:

from twilio.rest import Client
import os

TWILIO_SID = os.getenv("TWILIO_ACCOUNT_SID")
TWILIO_TOKEN = os.getenv("TWILIO_AUTH_TOKEN")
client = Client(TWILIO_SID, TWILIO_TOKEN)

call = client.calls.create(
    to="+15551234567",                # Destination number
    from_="+15557654321",             # Your Twilio number
    url="https://my-app.example.com/speak",  # TwiML webhook
    method="POST",
    # Optional: pass the text you want spoken as a query param or body
    # Twilio will POST it to your endpoint; you can read it there.
)
print(f"Call SID: {call.sid}")
Enter fullscreen mode Exit fullscreen mode

When the call connects, Twilio will request the /speak endpoint, which in turn will ask ElevenLabs to generate the audio and then play it back to the callee.

Going further: streaming instead of pre‑saving

If you need near‑real‑time responses (e.g., a chatbot that replies instantly), you can stream the ElevenLabs audio directly to Twilio using Twilio Media Streams. The idea is:

  1. Open a WebSocket from Twilio to your server (<StartStream> TwiML).
  2. Pipe the binary audio chunks from ElevenLabs into that socket.

While the full implementation is beyond the scope of this article, the pattern looks like this:

<Response>
  <StartStream name="myStream" url="wss://my-app.example.com/stream"/>
  <Pause length="1"/> <!-- give the stream a moment to connect -->
  <Say voice="alice">Listening...</Say>
</Response>
Enter fullscreen mode Exit fullscreen mode

Your WebSocket handler would receive the text you want spoken, call ElevenLabs with the streaming endpoint (/v1/text-to-speech/{voice_id}/stream), and forward each chunk back to Twilio. This yields sub‑second latency and feels truly conversational.

Tips & best practices

Tip Why it matters
Cache generated audio Many prompts (e.g., “Your verification code is 123456”) repeat often. Store the MP3 in a CDN for faster playback and lower API costs.
Set appropriate voice stability Higher stability reduces artifacts but can sound robotic. Play with the stability and similarity_boost parameters to hit the sweet spot for your use case.
Handle errors gracefully If ElevenLabs throttles you or returns an error, fall back to Twilio’s built‑in <Say> with a default voice so the call never hangs.
Secure your endpoints Use basic auth or signed JWTs for the /speak route. You don’t want strangers spamming your ElevenLabs quota.
Monitor usage Both Twilio and ElevenLabs bill per minute of audio. Log the length of each generated clip and set alerts if you cross thresholds.

Putting it all together

Below is a minimal end‑to‑end flow diagram:

User → (Dial) Twilio → (POST) /speak → generate_speech() → ElevenLabs API
          ↑                                          |
          |                                          v
          ←——————— <Play> audio URL ←——————— (store & serve MP3)
Enter fullscreen mode Exit fullscreen mode

With the Flask app, the Twilio webhook, and the tiny twilio-python client snippet, you have a fully functional voice‑AI pipeline that can be extended to:

  • IVR menus that sound like your brand’s CEO.
  • Appointment reminders with a custom regional accent.
  • Dynamic storytelling where each user choice generates a fresh narration.

All of this is possible with a few dozen lines of code and the powerful combination of ElevenLabs and Twilio.

Ready to give your callers a human‑like voice?

The barrier to building high‑quality voice applications has never been lower. Grab an API key, spin up a tiny Flask (or Express) service, and start experimenting. If you’re curious about how realistic voice cloning can boost engagement, try ElevenLabs today via this link: https://try.elevenlabs.io/kr07zfuqn1bp. Happy coding!

Top comments (0)