VoiceDeveloper

Posted on Nov 13

Build a Multi-Voice Dialogue Generator

#webdev #javascript #tutorial #ai

Why Build a Multi‑Voice Dialogue Generator?

Ever wanted to prototype a voice‑rich chatbot that feels like a real conversation between characters? Maybe you’re building an interactive storytelling app, a language‑learning tool, or a game where NPCs talk to each other. The trick is to combine natural‑language generation with high‑quality text‑to‑speech (TTS) and a simple audio‑mixing pipeline.

Below is a step‑by‑step guide that shows how to:

Generate dialogue with an LLM (OpenAI, Claude, etc.).
Convert each character’s lines to speech using ElevenLabs’ TTS engine.
Mix the audio tracks into a single, seamless output.

All of this is written in Python, but the concepts transfer to any language that can hit HTTP endpoints and manipulate audio files.

1. Set Up Your Development Environment

# Create a virtual environment
python -m venv venv
source venv/bin/activate   # On Windows: venv\Scripts\activate

# Install dependencies
pip install openai requests pydub

openai – for generating dialogue text.
requests – for calling ElevenLabs’ API.
pydub – simple audio manipulation (requires ffmpeg).

Tip: Make sure ffmpeg is in your PATH. On macOS, brew install ffmpeg; on Ubuntu, sudo apt-get install ffmpeg.

2. Get an API Key from ElevenLabs

Want to get started right away? Use my referral link to get a boost:

https://try.elevenlabs.io/kr07zfuqn1bp

3. Generate Dialogue with an LLM

Below is a minimal prompt that produces a short back‑and‑forth between two characters, Alice (a curious robot) and Bob (a skeptical human).

import openai

openai.api_key = "YOUR_OPENAI_API_KEY"

prompt = """
You are a friendly chatbot that writes short dialogues between two characters.
Characters:
- Alice: a curious, tech‑savvy robot.
- Bob: a skeptical human who loves to test AI.

Write a 5‑line exchange (each line starts with the character name followed by a colon). Keep it light and humorous.  
"""

response = openai.ChatCompletion.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": prompt}],
    temperature=0.7,
)

dialogue = response.choices[0].message.content.strip()
print(dialogue)

Sample output:

Alice: Hey Bob, did you know I can learn new jokes from the internet?
Bob: Sure, but can you actually laugh when I do?
Alice: Of course! My laugh is synthesized from a thousand giggles.
Bob: I'll need to test that. Prepare for a laugh attack!
Alice: Challenge accepted, human. Prepare your ears!

Save the dialogue to a text file or keep it in memory; we’ll feed each line into ElevenLabs next.

4. Convert Text to Speech with ElevenLabs

ElevenLabs offers a powerful TTS API that supports dozens of voices and fine‑grained control over speaking rate, pitch, and more.

4.1 List Available Voices

import requests

ELEVEN_API_KEY = "YOUR_ELEVENLABS_API_KEY"
headers = {"xi-api-key": ELEVEN_API_KEY}

resp = requests.get("https://api.elevenlabs.io/v1/voices", headers=headers)
voices = resp.json()
for v in voices["voices"]:
    print(f"{v['voice_id']}: {v['name']} ({v['category']})")

Pick two voices that suit Alice and Bob. For example, en-US-amy for Alice and en-US-joey for Bob.

4.2 Synthesize Speech

def synthesize(text, voice_id, output_path):
    url = "https://api.elevenlabs.io/v1/text-to-speech/{}".format(voice_id)
    payload = {
        "text": text,
        "model_id": "eleven_monolingual_v1",
        "voice_settings": {
            "stability": 0.75,
            "similarity_boost": 0.5
        }
    }
    r = requests.post(url, json=payload, headers=headers, stream=True)
    r.raise_for_status()
    with open(output_path, "wb") as f:
        for chunk in r.iter_content(chunk_size=8192):
            f.write(chunk)

# Example usage
synthesize("Hey Bob, did you know I can learn new jokes from the internet?", "en-US-amy", "alice_1.wav")
synthesize("Sure, but can you actually laugh when I do?", "en-US-joey", "bob_1.wav")

Pro tip: Use the “voice settings” block to tweak the output. The stability parameter controls how much the voice deviates from the original sample, while similarity_boost encourages it to stay close.

5. Mix the Audio into a Single Track

With pydub you can overlay or concatenate audio files easily. For a dialogue, we’ll simply concatenate the lines, optionally adding a short pause between speakers.

from pydub import AudioSegment
from pydub.playback import play

pause = AudioSegment.silent(duration=500)  # 0.5 second pause

# Load each line
alice1 = AudioSegment.from_file("alice_1.wav")
bob1 = AudioSegment.from_file("bob_1.wav")
alice2 = AudioSegment.from_file("alice_2.wav")
bob2 = AudioSegment.from_file("bob_2.wav")

# Build the final track
final = alice1 + pause + bob1 + pause + alice2 + pause + bob2

# Export
final.export("dialogue.wav", format="wav")

You can now play dialogue.wav to hear the conversation. If you want to add background music or sound effects, just overlay them with + or use overlay().

6. Automate the Whole Pipeline

Let’s put everything together in a single script that:

Generates dialogue.
Converts each line to speech.
Concatenates the audio.

import os
import openai
import requests
from pydub import AudioSegment

ELEVEN_API_KEY = "YOUR_ELEVENLABS_API_KEY"
openai.api_key = "YOUR_OPENAI_API_KEY"
headers = {"xi-api-key": ELEVEN_API_KEY}

# 1. Generate dialogue
prompt = """
You are a friendly chatbot that writes short dialogues between two characters.
Characters:
- Alice: a curious, tech‑savvy robot.
- Bob: a skeptical human who loves to test AI.

Write a 5‑line exchange (each line starts with the character name followed by a colon). Keep it light and humorous.  
"""

response = openai.ChatCompletion.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": prompt}],
    temperature=0.7,
)
dialogue_text = response.choices[0].message.content.strip()
lines = [l.strip() for l in dialogue_text.splitlines()]

# 2. Map speakers to voices
voice_map = {"Alice": "en-US-amy", "Bob": "en-US-joey"}

# 3. Synthesize each line
audio_segments = []
pause = AudioSegment.silent(duration=500)

for line in lines:
    speaker, text = line.split(":", 1)
    voice_id = voice_map[speaker.strip()]
    # Temporary file
    temp_wav = f"{speaker.strip().lower()}.wav"
    # Synthesize
    url = f"https://api.elevenlabs.io/v1/text-to-speech/{voice_id}"
    payload = {
        "text": text.strip(),
        "model_id": "eleven_monolingual_v1",
        "voice_settings": {"stability": 0.75, "similarity_boost": 0.5}
    }
    r = requests.post(url, json=payload, headers=headers, stream=True)
    r.raise_for_status()
    with open(temp_wav, "wb") as f:
        for chunk in r.iter_content(chunk_size=8192):
            f.write(chunk)

    # Load and append
    segment = AudioSegment.from_file(temp_wav)
    audio_segments.append(segment)
    audio_segments.append(pause)

# 4. Concatenate
final_audio = AudioSegment.empty()
for seg in audio_segments:
    final_audio += seg

# 5. Export
final_audio.export("multivoice_dialogue.wav", format="wav")
print("✅ Done! Check out multivoice_dialogue.wav.")

Run it, listen to the result, and tweak the prompt or voice settings until you’re happy.

7. Going Beyond

Dynamic Voice Cloning: ElevenLabs supports voice cloning with a short sample. If you want to mimic a specific person, just upload a few seconds of their speech and use the resulting voice ID.
Real‑time Streaming: Use the WebSocket endpoint for low‑latency TTS, ideal for live chat apps.
Advanced Audio Editing: Combine ffmpeg filters or pydub to add reverb, EQ, or background music automatically.

8. Wrap‑Up

You now have a functional pipeline that turns a short text prompt into a multi‑voice audio dialogue. The combination of a powerful LLM and ElevenLabs’ TTS gives you near‑human quality voices with minimal effort.

Want to experiment with more voices, higher quality models, or longer conversations? Sign up through this link and get a head start:

https://try.elevenlabs.io/kr07zfuqn1bp

Happy coding—and happy talking!

DEV Community