Why Voice‑Enabled Chatbots Matter
Voice is the most natural way we interact with technology. From Siri and Google Assistant to Alexa and Cortana, the trend is clear: people want to talk to their devices. For developers building conversational AI, adding speech capabilities transforms a simple text chatbot into an engaging, hands‑on experience. In this post we’ll walk through how to build a voice‑enabled chatbot that uses OpenAI’s powerful language model for dialogue and ElevenLabs for high‑quality, customizable text‑to‑speech (TTS) output. We’ll cover the whole pipeline: receiving user speech, converting it to text, generating a response, and finally turning that response back into speech.
1. Architecture Overview
[User] → (Microphone) → Speech‑to‑Text (OpenAI Whisper) →
| |
v v
[OpenAI GPT‑4] → Text Response → TTS (ElevenLabs) → Audio Output
- Speech‑to‑Text: We’ll use OpenAI’s Whisper API to transcribe the user’s voice into text.
- Dialogue Engine: GPT‑4 (or GPT‑3.5‑turbo) processes the text and generates a natural response.
- Text‑to‑Speech: ElevenLabs’ TTS engine produces a lifelike voice that can be customized with different voices and speaking styles.
2. Prerequisites
| Item | What You Need |
|---|---|
| Python 3.10+ | For the example scripts |
pip install openai elevenlabs |
Python client libraries |
| An OpenAI API key | For Whisper and GPT |
| An ElevenLabs API key | For TTS (sign up at https://try.elevenlabs.io/kr07zfuqn1bp) |
Tip: Store your keys in environment variables (
OPENAI_API_KEY,ELEVENLABS_API_KEY) to keep them secure.
3. Transcribing Speech with Whisper
First, capture audio from the user. Below is a minimal example that records 5 seconds of audio and sends it to Whisper.
import os
import sounddevice as sd
import numpy as np
import openai
# Configure OpenAI
openai.api_key = os.getenv("OPENAI_API_KEY")
duration = 5 # seconds
sample_rate = 16000 # Whisper’s preferred sample rate
print("Speak now…")
audio = sd.rec(int(duration * sample_rate), samplerate=sample_rate, channels=1, dtype='int16')
sd.wait()
print("Recording finished.")
# Convert to bytes
audio_bytes = audio.tobytes()
# Whisper transcription
response = openai.Audio.transcribe(
model="whisper-1",
file=audio_bytes,
language="en"
)
transcript = response["text"]
print(f"User said: {transcript}")
Pro tip: If you’re building a web app, you can stream the audio directly to Whisper using the streaming endpoint for lower latency.
4. Generating a Response with GPT‑4
Once we have the transcript, we feed it into GPT‑4 and let it generate a natural reply.
def get_gpt_response(user_text: str) -> str:
prompt = (
f"You are a helpful chatbot. The user says: \"{user_text}\". "
"Respond in a friendly, concise manner."
)
completion = openai.ChatCompletion.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}]
)
return completion.choices[0].message.content.strip()
response_text = get_gpt_response(transcript)
print(f"Chatbot: {response_text}")
Feel free to tweak the prompt or add system messages for persona, tone, or context.
5. Turning Text into Speech with ElevenLabs
ElevenLabs offers a rich set of voices, including the ability to clone a custom voice from a short audio clip. For this tutorial we’ll use a ready‑made voice.
import elevenlabs
import os
import requests
elevenlabs.api_key = os.getenv("ELEVENLABS_API_KEY")
def synthesize_speech(text: str, voice_id: str = "21m00Tcm4TlvDq8ikWAM") -> bytes:
"""
voice_id: Default is the “Rachel” voice. Find others in your ElevenLabs dashboard.
"""
audio = elevenlabs.generate(
text=text,
voice=voice_id,
model="eleven_multilingual_v2" # choose a high‑quality model
)
return audio
audio_bytes = synthesize_speech(response_text)
# Save to file (or play directly)
with open("response.wav", "wb") as f:
f.write(audio_bytes)
print("TTS output saved to response.wav")
Note: ElevenLabs’ API returns raw MP3 bytes by default. If you prefer WAV, set
output_format="wav"in thegeneratecall.
6. Putting It All Together
Here’s a full script that ties everything together. Run it, speak, and you’ll hear the chatbot’s reply.
import os
import sounddevice as sd
import openai
import elevenlabs
openai.api_key = os.getenv("OPENAI_API_KEY")
elevenlabs.api_key = os.getenv("ELEVENLABS_API_KEY")
def record_and_transcribe(duration=5, sr=16000):
audio = sd.rec(int(duration * sr), samplerate=sr, channels=1, dtype='int16')
sd.wait()
audio_bytes = audio.tobytes()
resp = openai.Audio.transcribe("whisper-1", file=audio_bytes, language="en")
return resp["text"]
def ask_gpt(user_text):
prompt = f"You are a friendly chatbot. User says: \"{user_text}\". Reply briefly."
completion = openai.ChatCompletion.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}]
)
return completion.choices[0].message.content.strip()
def speak(text, voice_id="21m00Tcm4TlvDq8ikWAM"):
audio = elevenlabs.generate(text, voice=voice_id, model="eleven_multilingual_v2")
with open("output.mp3", "wb") as f:
f.write(audio)
# Play it back (simple cross‑platform)
os.system("start output.mp3" if os.name == "nt" else "afplay output.mp3")
if __name__ == "__main__":
print("Speak to the chatbot:")
user_msg = record_and_transcribe()
print(f"User: {user_msg}")
bot_reply = ask_gpt(user_msg)
print(f"Bot: {bot_reply}")
speak(bot_reply)
Debugging tip: If you’re on Linux, replace the
os.systemcall withmpg123 output.mp3orffplay -nodisp -autoexit output.mp3.
7. Advanced Features
| Feature | How to Implement |
|---|---|
| Custom Voice Cloning | Upload 30‑second audio to ElevenLabs and use the returned voice_id. |
| Multilingual Support | Whisper can detect language; ElevenLabs offers multilingual models. |
| Streaming Responses | Use OpenAI’s streaming API for GPT and ElevenLabs’ streaming endpoint for TTS to minimize latency. |
| Voice‑to‑Voice Interaction | Capture user audio, transcribe, respond, synthesize, and play back in a continuous loop. |
8. Performance Tips
- Batch Audio: For real‑time apps, send 1‑second chunks to Whisper to get faster feedback.
- Cache Voices: Store generated MP3s for common responses to avoid repeated TTS calls.
- Rate Limits: Both OpenAI and ElevenLabs have quotas. Monitor usage via dashboards and add exponential back‑off.
9. Security & Privacy
- Never expose your API keys in client‑side code.
- Delete temporary audio files after use.
- If you’re handling sensitive data, consider adding encryption or a local Whisper deployment.
10. Next Steps
- Build a web interface using Streamlit or FastAPI that streams microphone input.
- Add contextual memory: keep conversation history to make replies more coherent.
- Explore emotion‑aware TTS: ElevenLabs offers style control to make the voice sound excited, calm, or empathetic.
Call to Action
You’re now ready to turn any text chatbot into a lifelike voice‑enabled assistant. If you’re serious about high‑quality, customizable TTS, give ElevenLabs a try. Sign up today through our affiliate link: https://try.elevenlabs.io/kr07zfuqn1bp and unlock the full potential of voice AI. Happy coding!
Top comments (0)