DEV Community

VoiceDeveloper
VoiceDeveloper

Posted on

Build a Voice-Enabled Chatbot with ElevenLabs and OpenAI

#webdev #tutorial #javascript #ai

Why Voice‑Enabled Chatbots Matter

Voice is the most natural way we communicate. From virtual assistants to customer support, people expect to talk instead of type. For developers, building a chatbot that can listen, understand, and speak back is a compelling way to create engaging experiences. With the explosion of powerful APIs from OpenAI and ElevenLabs, the barrier to entry has never been lower.

In this post we’ll walk through a practical, end‑to‑end workflow:

  1. Capture audio from the user
  2. Transcribe it with OpenAI Whisper
  3. Generate a response with GPT‑4
  4. Convert the response to speech with ElevenLabs Text‑to‑Speech
  5. (Optional) Clone a custom voice for a more personal touch

We’ll use Python for the backend and a tiny React front‑end to demo the flow. By the end you’ll have a fully functional voice chatbot that you can extend to any domain.

1. Prerequisites

Item How to get it
Python 3.10+ python --version
Node.js 20+ node --version
OpenAI API key Sign up at https://platform.openai.com/signup
ElevenLabs API key Get yours at https://try.elevenlabs.io/kr07zfuqn1bp

Tip: Store your keys in environment variables or a .env file. Never hard‑code them.

2. Setting Up the Backend

Create a new directory, voice-chatbot, and inside it run:

python -m venv venv
source venv/bin/activate   # Windows: venv\Scripts\activate
pip install fastapi uvicorn openai elevenlabs python-dotenv
Enter fullscreen mode Exit fullscreen mode

Create a file main.py:

import os
import io
import json
from fastapi import FastAPI, File, UploadFile, HTTPException
from fastapi.responses import StreamingResponse
import openai
import elevenlabs

# Load secrets
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
ELEVENLABS_API_KEY = os.getenv("ELEVENLABS_API_KEY")

openai.api_key = OPENAI_API_KEY
elevenlabs.set_api_key(ELEVENLABS_API_KEY)

app = FastAPI()

@app.post("/transcribe")
async def transcribe(file: UploadFile = File(...)):
    if file.content_type not in ("audio/wav", "audio/mpeg"):
        raise HTTPException(status_code=400, detail="Unsupported audio format")
    audio_bytes = await file.read()
    response = openai.Audio.transcribe("whisper-1", io.BytesIO(audio_bytes))
    return {"text": response["text"]}

@app.post("/synthesize")
def synthesize(text: str, voice_id: str = "Rachel"):
    # Use ElevenLabs TTS
    audio = elevenlabs.generate(
        text=text,
        voice=voice_id,
        model="eleven_monolingual_v1"
    )
    return StreamingResponse(io.BytesIO(audio), media_type="audio/mpeg")
Enter fullscreen mode Exit fullscreen mode

Why ElevenLabs?

ElevenLabs offers high‑fidelity, natural‑sounding voices that are easy to integrate. Use the free tier to start, and upgrade when you need higher quality or custom voice cloning. Check out https://try.elevenlabs.io/kr07zfuqn1bp to sign up.

Run the server:

uvicorn main:app --reload
Enter fullscreen mode Exit fullscreen mode

Your API is now available at http://localhost:8000.

3. Front‑End Demo (React)

Create a simple React app in the client folder:

npx create-react-app client
cd client
npm install axios
Enter fullscreen mode Exit fullscreen mode

Replace src/App.js with:

import React, { useState, useRef } from "react";
import axios from "axios";

const App = () => {
  const [transcript, setTranscript] = useState("");
  const [audioUrl, setAudioUrl] = useState("");
  const recorderRef = useRef(null);

  const startRecording = async () => {
    const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
    recorderRef.current = new MediaRecorder(stream);
    const chunks = [];
    recorderRef.current.ondataavailable = e => chunks.push(e.data);
    recorderRef.current.onstop = async () => {
      const blob = new Blob(chunks, { type: "audio/webm" });
      const formData = new FormData();
      formData.append("file", blob, "speech.webm");

      const transcribeRes = await axios.post(
        "http://localhost:8000/transcribe",
        formData,
        { headers: { "Content-Type": "multipart/form-data" } }
      );
      setTranscript(transcribeRes.data.text);

      // Synthesize reply
      const synthRes = await axios.post(
        "http://localhost:8000/synthesize",
        { text: `You said: ${transcribeRes.data.text}` },
        { responseType: "blob" }
      );
      const url = URL.createObjectURL(synthRes.data);
      setAudioUrl(url);
    };
    recorderRef.current.start();
  };

  const stopRecording = () => recorderRef.current.stop();

  return (
    <div style={{ padding: "2rem" }}>
      <h2>Voice‑Enabled Chatbot Demo</h2>
      <button onMouseDown={startRecording} onMouseUp={stopRecording}>
        Hold to Speak
      </button>
      <p><strong>Transcription:</strong> {transcript}</p>
      {audioUrl && <audio controls src={audioUrl} />}
    </div>
  );
};

export default App;
Enter fullscreen mode Exit fullscreen mode

Start the React app:

npm start
Enter fullscreen mode Exit fullscreen mode

Now you can hold the button, speak, and hear the chatbot’s reply. This is a minimal example—feel free to add voice cloning, better UI, or integrate with a voice assistant SDK.

4. Adding Voice Cloning (Optional)

ElevenLabs allows you to clone a voice from a few seconds of audio. Here’s how:

# Clone a new voice from a short sample
sample_path = "sample.wav"  # 10‑15 seconds of clear speech
with open(sample_path, "rb") as f:
    voice_id = elevenlabs.create_voice_from_audio(f.read(), name="My Clone")

print("Cloned voice ID:", voice_id)
Enter fullscreen mode Exit fullscreen mode

Use this voice_id in the /synthesize endpoint to get a response in the cloned voice. Remember that cloning may require a higher plan, but the free tier lets you experiment.

5. Deploying to the Cloud

For production, consider:

  • FastAPI + Uvicorn behind a reverse proxy (NGINX) or a platform like Fly.io, Render, or Railway.
  • HTTPS and proper CORS settings.
  • Rate limiting and request validation.

A quick deployment on Fly.io:

flyctl launch   # follow prompts
flyctl deploy
Enter fullscreen mode Exit fullscreen mode

Update the React build to point at your new API URL, and serve the static files via the same host or a CDN.

6. Common Pitfalls & Debugging Tips

Issue Fix
“Unsupported audio format” error Make sure the browser records in WebM or WAV. Convert on the backend if needed.
“OpenAI API key missing” Double‑check environment variables and the .env file.
Low‑quality TTS Use the “eleven_monolingual_v1” model or upgrade to “eleven_multilingual_v2” for higher fidelity.
Voice cloning too slow Clone once during setup; reuse the voice_id for subsequent calls.

7. Take It Further

  • Multilingual support: Use OpenAI’s gpt-4o-mini for faster, cheaper responses, and ElevenLabs’ multilingual models for TTS.
  • Contextual memory: Store conversation history and feed it back into GPT to keep the dialogue coherent.
  • Edge deployment: Run the TTS on the client with WebAssembly if latency matters.

8. Wrap‑Up

You now have a complete, voice‑enabled chatbot that leverages OpenAI’s powerful language models and ElevenLabs’ state‑of‑the‑art text‑to‑speech. The stack is lightweight, yet it gives you a solid foundation for building sophisticated voice assistants, interactive tutorials, or customer support bots.

Call to Action

Ready to elevate your project with high‑quality voice synthesis? Sign up for ElevenLabs today and start experimenting with realistic, natural‑sounding voices. Grab your API key at https://try.elevenlabs.io/kr07zfuqn1bp and let the conversation begin!

Top comments (0)