DEV Community

VoiceDeveloper
VoiceDeveloper

Posted on

Create an AI Voice Assistant with ElevenLabs and Node.js

#webdev #javascript #tutorial #ai

Why ElevenLabs for Your Voice Assistant?

If you’ve ever wanted a real‑time voice assistant that sounds natural, you’ve probably tried the usual suspects—Google TTS, Amazon Polly, or even open‑source models. They’re decent, but they often sound robotic or lack the nuance you need for a polished user experience. That’s where ElevenLabs shines. Their neural text‑to‑speech engine produces expressive, human‑like speech and even supports voice cloning, which means you can give your assistant a unique personality with just a few minutes of audio.

In this post we’ll walk through a complete Node.js setup:

  1. Grab an API key from ElevenLabs (via the affiliate link below)
  2. Create a tiny Express server that accepts plain‑text prompts
  3. Call the ElevenLabs TTS endpoint and stream the resulting MP3 back to the client
  4. (Optional) Upload a custom voice for cloning your own voice or a brand‑specific one

By the end you’ll have a working AI voice assistant that you can plug into a chat UI, a smart speaker, or any other interactive app.

👉 Get your free ElevenLabs API key here: https://try.elevenlabs.io/kr07zfuqn1bp

Prerequisites

What you need Why
Node.js ≥ 18 Built‑in fetch support makes HTTP calls a breeze.
npm or yarn For installing dependencies.
ElevenLabs API key Required to authenticate every request.
A small audio clip (optional) If you want to create a custom voice for cloning.

1. Set Up the Project 

mkdir ai-voice-assistant
cd ai-voice-assistant
npm init -y
npm install express dotenv
Enter fullscreen mode Exit fullscreen mode

Create a .env file to keep your secret safe:

ELEVENLABS_API_KEY=your_api_key_here
Enter fullscreen mode Exit fullscreen mode

Tip: Never commit .env to version control. Add it to .gitignore.

2. Build the Express Server

Create index.js and paste the following:

import express from 'express';
import fs from 'fs';
import { pipeline } from 'stream';
import { promisify } from 'util';
import dotenv from 'dotenv';

dotenv.config();

const app = express();
app.use(express.json());

const ELEVENLABS_API_KEY = process.env.ELEVENLABS_API_KEY;
const ELEVENLABS_TTS_URL = 'https://api.elevenlabs.io/v1/text-to-speech';

// Helper to call ElevenLabs TTS
async function synthesize(text, voiceId = 'EXAVITQu4vr4xnSDxMaL') {
  const response = await fetch(`${ELEVENLABS_TTS_URL}/${voiceId}`, {
    method: 'POST',
    headers: {
      'xi-api-key': ELEVENLABS_API_KEY,
      'Content-Type': 'application/json',
      Accept: 'audio/mpeg',
    },
    body: JSON.stringify({
      text,
      model_id: 'eleven_monolingual_v1',
      voice_settings: {
        stability: 0.75,
        similarity_boost: 0.75,
      },
    }),
  });

  if (!response.ok) {
    const err = await response.text();
    throw new Error(`ElevenLabs error: ${err}`);
  }

  return response.body; // This is a readable stream of the MP3
}

// Route: POST /speak { "text": "Hello world!" }
app.post('/speak', async (req, res) => {
  const { text, voiceId } = req.body;
  if (!text) return res.status(400).json({ error: 'Missing "text" in body.' });

  try {
    const audioStream = await synthesize(text, voiceId);
    res.setHeader('Content-Type', 'audio/mpeg');
    // Stream the MP3 directly to the client
    const pipe = promisify(pipeline);
    await pipe(audioStream, res);
  } catch (err) {
    console.error(err);
    res.status(500).json({ error: err.message });
  }
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => console.log(`🚀 Server listening on http://localhost:${PORT}`));
Enter fullscreen mode Exit fullscreen mode

How It Works

  • /speak receives JSON with a text field (and optionally a voiceId).
  • It calls synthesize(), which POSTs to ElevenLabs’ /text-to-speech/{voice_id} endpoint.
  • The API returns an audio/mpeg stream that we pipe straight to the HTTP response, avoiding temporary file writes.

You can now start the server:

node index.js
Enter fullscreen mode Exit fullscreen mode

Test it with curl:

curl -X POST http://localhost:3000/speak \
  -H "Content-Type: application/json" \
  -d '{"text":"Hey there, I’m your new AI assistant!"}' \
  --output hello.mp3
Enter fullscreen mode Exit fullscreen mode

Play hello.mp3—you should hear a smooth, natural voice.

3. Adding a Custom Voice (Voice Cloning)

ElevenLabs lets you clone a voice by uploading a short recording (≈30 seconds). The workflow is:

  1. Create a voice via the /v1/voices/add endpoint.
  2. Upload your audio (WAV or MP3).
  3. Use the returned voice_id in the TTS calls above.

Here’s a quick Node snippet to do it:

async function createCustomVoice(name, audioFilePath) {
  const audioBuffer = await fs.promises.readFile(audioFilePath);
  const response = await fetch('https://api.elevenlabs.io/v1/voices/add', {
    method: 'POST',
    headers: {
      'xi-api-key': ELEVENLABS_API_KEY,
    },
    body: JSON.stringify({
      name,
      files: [
        {
          filename: audioFilePath,
          data: audioBuffer.toString('base64'),
        },
      ],
    }),
  });

  if (!response.ok) {
    const err = await response.text();
    throw new Error(`Voice creation failed: ${err}`);
  }

  const data = await response.json();
  return data.voice_id; // Store this for later TTS calls
}

// Example usage:
(async () => {
  const voiceId = await createCustomVoice('MyBrandVoice', './my-brand-sample.wav');
  console.log('Created voice ID:', voiceId);
})();
Enter fullscreen mode Exit fullscreen mode

Note: Voice cloning is a premium feature. If you’re just experimenting, stick with the default voices (e.g., EXAVITQu4vr4xnSDxMaL).

4. Hooking It Up to a Front‑End

A minimal HTML page can fetch the /speak endpoint and play the result using the <audio> element:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>AI Voice Assistant Demo</title>
</head>
<body>
  <textarea id="prompt" rows="4" cols="50">What’s the weather like today?</textarea><br>
  <button id="talk">Speak</button>
  <audio id="player" controls></audio>

  <script>
    document.getElementById('talk').onclick = async () => {
      const text = document.getElementById('prompt').value;
      const res = await fetch('/speak', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ text })
      });
      const blob = await res.blob();
      const url = URL.createObjectURL(blob);
      const player = document.getElementById('player');
      player.src = url;
      player.play();
    };
  </script>
</body>
</html>
Enter fullscreen mode Exit fullscreen mode

Serve this static file from Express (or any web server) and you have a click‑to‑talk demo in under a minute.

5. Production Tips

Concern Recommendation
Latency Cache frequently used prompts or pre‑generate audio for static responses.
Security Never expose your ElevenLabs API key to the client. Keep all calls server‑side.
Scalability Use a queue (e.g., BullMQ) if you expect bursts of TTS requests.
Rate limits ElevenLabs allows a generous quota, but monitor usage via their dashboard.
Voice consistency Store the voice_id you generate for a custom voice in a database and reuse it across sessions.

6. Wrap‑Up

You now have a fully functional AI voice assistant powered by ElevenLabs and Node.js. The core pieces are:

  • An Express endpoint that streams MP3 audio directly from ElevenLabs.
  • Optional voice cloning for brand‑specific personalities.
  • A lightweight front‑end that turns any text into spoken output.

Because the TTS happens on the server, you keep your API key safe, and you can easily extend the service to handle multi‑modal interactions (e.g., speech‑to‑text with Whisper, intent parsing with LLMs, etc.).

Ready to give your app a human voice? Grab your ElevenLabs API key now and start building: https://try.elevenlabs.io/kr07zfuqn1bp

Happy coding! 🚀

Top comments (0)