DEV Community

VoiceDeveloper
VoiceDeveloper

Posted on

Build a Voice-Enabled Chatbot with ElevenLabs and OpenAI

#webdev #tutorial #javascript #ai

Why Voice-Enabled Chatbots Are the Next Big Thing

Voice interfaces are no longer a novelty—they’re the default way people interact with devices. From virtual assistants on phones to in‑car systems, people expect to talk to software and get instant, natural responses. If you’re building a chatbot that needs to feel human, adding high‑quality text‑to‑speech (TTS) and voice cloning can make the difference between a clunky demo and a product‑ready experience.

In this article we’ll walk through a practical, end‑to‑end example that stitches together OpenAI’s language model for generating dialogue and ElevenLabs for turning that text into expressive, customizable speech. By the time you’re finished, you’ll have a working prototype that you can extend, deploy, and even monetize.

1. Prerequisites

Item Why it matters
Python 3.10+ The code snippets use modern syntax and async support.
OpenAI API key Powers the GPT‑4 model that understands context.
ElevenLabs API key Provides the TTS engine and voice‑cloning tools.
FFmpeg Optional, but handy for converting raw audio to MP3 or OGG.

Tip: If you don’t have an ElevenLabs account yet, sign up here: https://try.elevenlabs.io/kr07zfuqn1bp. The link is an affiliate link that supports this tutorial, so feel free to use it.

2. Architecture Overview 

┌─────────────────────┐
│  Front‑end (React/  │
│  Web, Android, iOS) │
└──────────┬───────────┘
           │
           ▼
┌─────────────────────┐
│   FastAPI / Flask   │
│   (Python Backend)  │
└──────────┬───────────┘
           │
           ▼
┌─────────────────────┐
│  OpenAI GPT‑4        │
│  (Generate text)    │
└──────────┬───────────┘
           │
           ▼
┌─────────────────────┐
│  ElevenLabs TTS      │
│  (Convert text →    │
│  audio)              │
└──────────┬───────────┘
           │
           ▼
┌─────────────────────┐
│  Audio Stream /     │
│  File Delivery      │
└─────────────────────┘
Enter fullscreen mode Exit fullscreen mode
  • OpenAI GPT‑4 handles the conversational logic.
  • ElevenLabs provides the voice layer, with realistic prosody, emotional cues, and the ability to clone a user’s own voice.
  • The Python backend orchestrates the flow, keeping latency low.

3. Getting Started: Environment Setup 

# Create a virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install required packages
pip install openai requests fastapi uvicorn python-multipart aiofiles

# (Optional) FFmpeg for audio conversion
# On Ubuntu:
# sudo apt-get install ffmpeg
Enter fullscreen mode Exit fullscreen mode

Create a .env file to store your secrets:

OPENAI_API_KEY=sk-xxxxxx
ELEVENLABS_API_KEY=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9....
Enter fullscreen mode Exit fullscreen mode

Load them in your Python code with python-dotenv or by reading the file directly.

4. A Minimal FastAPI Example

Below is a stripped‑down example that takes a user prompt, asks GPT‑4 for a response, then feeds that text into ElevenLabs to get back an audio file.

# main.py
import os
import uuid
from fastapi import FastAPI, File, UploadFile
from fastapi.responses import StreamingResponse
import openai
import requests
import json
from dotenv import load_dotenv

load_dotenv()

openai.api_key = os.getenv("OPENAI_API_KEY")
ELEVEN_API_KEY = os.getenv("ELEVENLABS_API_KEY")
ELEVEN_API_URL = "https://api.elevenlabs.io/v1/text-to-speech"

app = FastAPI()

@app.post("/speak")
async def speak(prompt: str):
    # 1️⃣ Generate text with GPT‑4
    completion = openai.ChatCompletion.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}]
    )
    answer_text = completion.choices[0].message.content.strip()

    # 2️⃣ Convert text to speech with ElevenLabs
    payload = {
        "text": answer_text,
        "voice_settings": {
            "stability": 0.5,
            "similarity_boost": 0.75
        }
    }
    headers = {
        "xi-api-key": ELEVEN_API_KEY,
        "Content-Type": "application/json"
    }
    response = requests.post(ELEVEN_API_URL, headers=headers, json=payload, stream=True)

    if response.status_code != 200:
        return {"error": "TTS failed", "details": response.text}

    # 3️⃣ Stream the audio back to the client
    return StreamingResponse(
        response.iter_content(chunk_size=8192),
        media_type="audio/mpeg"
    )
Enter fullscreen mode Exit fullscreen mode

Run it:

uvicorn main:app --reload
Enter fullscreen mode Exit fullscreen mode

Send a POST request to http://127.0.0.1:8000/speak with a JSON body like:

{
  "prompt": "Tell me a joke about cats."
}
Enter fullscreen mode Exit fullscreen mode

You’ll receive an MP3 stream you can play in the browser or any media player.

5. Adding Voice Cloning

ElevenLabs lets you clone a voice from a short sample, then use that voice for all future syntheses. This is especially useful if you want a chatbot that speaks like a brand mascot or a real person.

  1. Upload a sample (minimum 30 seconds, ideally clear speech). 
curl -X POST "https://api.elevenlabs.io/v1/voices" \
  -H "xi-api-key: $ELEVEN_API_KEY" \
  -H "Content-Type: multipart/form-data" \
  -F "audio=@/path/to/sample.wav" \
  -F "voice_name=MyCustomVoice"
Enter fullscreen mode Exit fullscreen mode
  1. Use the cloned voice in the TTS request: 
payload = {
    "text": answer_text,
    "voice_id": "MyCustomVoice",   # <-- ID returned from the upload
    "voice_settings": {
        "stability": 0.4,
        "similarity_boost": 0.9
    }
}
Enter fullscreen mode Exit fullscreen mode

Pro tip: Keep your sample audio in a lossless format (WAV) and avoid background noise for the best clone quality.

6. Making It Real‑Time

For a truly conversational experience, you’ll want to stream audio as it’s generated. ElevenLabs supports a streaming endpoint that returns audio chunks in real time. Here’s a quick async example using httpx:

import httpx
import asyncio

async def stream_tts(text, voice_id="eleven_multilingual_v1"):
    url = "https://api.elevenlabs.io/v1/text-to-speech/" + voice_id
    headers = {"xi-api-key": ELEVEN_API_KEY}
    payload = {"text": text, "voice_settings": {"stability": 0.5}}

    async with httpx.AsyncClient() as client:
        async with client.stream("POST", url, headers=headers, json=payload) as resp:
            async for chunk in resp.aiter_bytes():
                # Here you could write to an audio buffer or WebSocket
                pass
Enter fullscreen mode Exit fullscreen mode

Integrate this with a WebSocket endpoint in FastAPI so the browser can play the audio as it arrives.

7. Deployment Tips

  • Containerize: Use Docker for reproducible builds. A minimal Dockerfile: 
  FROM python:3.11-slim
  WORKDIR /app
  COPY requirements.txt .
  RUN pip install --no-cache-dir -r requirements.txt
  COPY . .
  CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "80"]
Enter fullscreen mode Exit fullscreen mode

  • Scale: ElevenLabs offers rate limits per key. For high‑volume production, request a higher quota or cache generated audio for repeated prompts.

  • Security: Store API keys in secrets management services (e.g., AWS Secrets Manager, HashiCorp Vault) rather than in plain env files.

8. Common Pitfalls

Issue Fix
Latency > 2 s Use the streaming endpoint, cache responses, or pre‑generate idle phrases.
Unnatural intonation Tweak stability and similarity_boost or use a custom voice.
Audio glitches Ensure the client’s audio player supports the MIME type (MP3/Ogg).
Quota exceeded Monitor usage via the ElevenLabs dashboard and add alerts.

9. Wrap‑Up

Combining OpenAI’s conversational prowess with ElevenLabs’ high‑fidelity TTS and voice‑cloning capabilities lets you build chatbots that feel genuinely human. Whether you’re prototyping a personal assistant, creating an interactive story, or building a customer‑support bot, the stack we covered is battle‑tested and production‑ready.

Now it’s your turn. Grab an ElevenLabs API key (the link above) and start turning your chatbot’s words into speech today. Happy coding!

Top comments (0)