VoiceDeveloper

Posted on Oct 11

Build a Voice-Enabled Chatbot with ElevenLabs and OpenAI

#ai #javascript #tutorial #webdev

Why Voice‑Enabled Chatbots Matter

Voice assistants are no longer a niche feature—customers expect a natural, hands‑free interface on phones, smart speakers, and even in‑car infotainment systems.

Building a chatbot that can hear a user’s question, think with GPT‑4, and speak a polished reply is a great way to showcase modern AI and give your product a competitive edge.

Below, I’ll walk you through the whole stack: from capturing audio, to sending text to OpenAI, to generating speech with ElevenLabs.

1. Capture the User’s Voice

If you’re targeting web or mobile, the simplest way to get a user’s voice is the browser’s Web Speech API (or a mobile library like react-native-audio).

For the sake of this tutorial, we’ll assume you already have a front‑end that turns speech into a UTF‑8 string and sends it to your backend.

// Example: Capture and send voice to the server
const recognition = new SpeechRecognition();
recognition.lang = 'en-US';
recognition.onresult = async event => {
  const transcript = event.results[0][0].transcript;
  const response = await fetch('/api/chat', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ message: transcript })
  });
  const data = await response.json();
  // Play the returned audio blob
  const audio = new Audio(data.audioUrl);
  audio.play();
};
recognition.start();

2. Ask OpenAI for a Text Answer

On your server, forward the user’s text to the OpenAI API and get a natural language reply.

Below is a minimal Python example using openai:

import openai

openai.api_key = "YOUR_OPENAI_KEY"

def get_gpt_response(prompt: str) -> str:
    response = openai.ChatCompletion.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.7,
        max_tokens=256
    )
    return response.choices[0].message.content.strip()

This function is all you need to turn a user’s spoken question into a GPT‑generated answer.

3. Turn Text Into Speech with ElevenLabs

Here’s where ElevenLabs shines. Their Text‑to‑Speech engine offers high‑fidelity voices, fine‑grained control over prosody, and an easy‑to‑use REST API.

I recommend using their ElevenLabs service for production‑grade voice output.

You can sign up and get started immediately at https://try.elevenlabs.io/kr07zfuqn1bp.

Why ElevenLabs?

Feature	ElevenLabs	Other TTS (e.g., OpenAI Whisper)
Voice quality	Ultra‑natural, multi‑accent support	Good, but less natural
Voice cloning	100% voice‑matching, 1‑minute sample	Limited or no cloning
API simplicity	One‑call, JSON response	Multiple steps or SDK required
Pricing	Pay‑as‑you‑go, free tier	Often higher per‑second cost

4. Putting It All Together

Below is a full Python Flask endpoint that ties everything together: receive the user’s text, ask GPT‑4 for a reply, synthesize the answer with ElevenLabs, and stream the audio back to the client.

from flask import Flask, request, jsonify, send_file
import openai
import requests
import io

app = Flask(__name__)
openai.api_key = "YOUR_OPENAI_KEY"
ELEVENLABS_API_KEY = "YOUR_ELEVENLABS_KEY"

def synthesize(text: str) -> bytes:
    headers = {"xi-api-key": ELEVENLABS_API_KEY, "Content-Type": "application/json"}
    data = {"text": text, "voice_settings": {"stability": 0.5, "similarity_boost": 0.75}}
    response = requests.post(
        "https://api.elevenlabs.io/v1/text-to-speech/eleven_monolingual_v1",
        json=data,
        headers=headers,
    )
    response.raise_for_status()
    return response.content  # raw audio bytes (mp3)

@app.route("/api/chat", methods=["POST"])
def chat():
    user_input = request.json.get("message")
    if not user_input:
        return jsonify({"error": "No message provided"}), 400

    # 1. Get GPT response
    gpt_answer = get_gpt_response(user_input)

    # 2. Convert text to speech
    audio_bytes = synthesize(gpt_answer)

    # 3. Send audio back as a downloadable blob
    return send_file(
        io.BytesIO(audio_bytes),
        mimetype="audio/mpeg",
        as_attachment=False,
        download_name="response.mp3",
    )

if __name__ == "__main__":
    app.run(debug=True)

What this does:

Receive the user’s spoken text.
Ask GPT‑4 for an answer.
Synthesize that answer into an MP3 with ElevenLabs.
Return the audio to the front‑end for playback.

5. Adding Voice Cloning

If you want your chatbot to sound like a specific person (e.g., a brand spokesperson), ElevenLabs’ voice cloning feature lets you create a custom voice from just a few minutes of audio.

Upload a short clip, generate a voice ID, and then use that ID in the voice_settings payload.

def clone_voice(audio_file_path: str) -> str:
    with open(audio_file_path, "rb") as f:
        files = {"file": f}
        headers = {"xi-api-key": ELEVENLABS_API_KEY}
        response = requests.post(
            "https://api.elevenlabs.io/v1/voices",
            files=files,
            headers=headers
        )
    response.raise_for_status()
    return response.json()["voice_id"]

Use the returned voice_id in the synthesize step:

data = {"text": text, "voice_id": custom_voice_id}

6. Deployment & Scaling Tips

Tip	Why it matters
Use async workers	GPT and TTS calls can take 1–2 s each; async queues (Celery, RQ, or FastAPI async) keep the API responsive.
Cache common prompts	If the same question appears often, cache the GPT or TTS response to cut costs.
Serve audio from a CDN	Streaming large MP3s directly from your server can be slow; upload to CloudFront or Cloudflare.
Handle rate limits	Both OpenAI and ElevenLabs enforce limits; back‑off and retry logic prevents 429 errors.

7. Wrap‑Up

You now have a fully functional voice‑enabled chatbot:

Speech → Text on the client.
Text → GPT‑4 for intelligent replies.
Text → ElevenLabs for crystal‑clear speech.
Play the result back to the user.

Whether you’re building a customer‑support bot, an educational tutor, or a smart‑home assistant, this stack gives you the best of both worlds—OpenAI’s reasoning and ElevenLabs’ voice quality.

Call to Action

Ready to add lifelike voices to your chatbot?

Sign up for ElevenLabs today and get started with their powerful API at https://try.elevenlabs.io/kr07zfuqn1bp.

Happy coding!

DEV Community