DEV Community

VoiceDeveloper
VoiceDeveloper

Posted on

Create an AI Voice Assistant with ElevenLabs and Node.js

#webdev #tutorial #javascript #ai

Overview

If you’ve ever wanted to give your app a friendly voice that sounds like a real person, you’re in the right place. In this tutorial we’ll build a simple AI voice assistant using ElevenLabs’ cutting‑edge text‑to‑speech (TTS) API and a Node.js backend. By the end you’ll be able to send a string of text to the API, receive high‑quality audio, and stream it back to a web client—all with just a few lines of code.

Why ElevenLabs?

Their neural TTS models produce natural‑sounding speech, support multiple languages, and even let you clone a voice from a short recording. The service is developer‑friendly, offers generous free tier limits, and integrates nicely with standard HTTP clients.

Let’s dive in.

Prerequisites

  • Node.js 18+ (for native fetch support) or any recent version with a fetch polyfill.
  • npm or yarn for package management.
  • An ElevenLabs API key – you can sign up and grab one from the dashboard at the link below.
  • Basic familiarity with Express (or any HTTP framework) and front‑end JavaScript.

Getting an API Key from ElevenLabs

  1. Visit the ElevenLabs sign‑up page: https://try.elevenlabs.io/kr07zfuqn1bp.
  2. Complete the registration and navigate to the API section of your account.
  3. Copy the API key – you’ll need it as an environment variable. 
# .env file (keep this out of version control!)
ELEVENLABS_API_KEY=your_api_key_here
Enter fullscreen mode Exit fullscreen mode

Setting Up the Project

Create a fresh folder and initialize a Node project:

mkdir ai-voice-assistant
cd ai-voice-assistant
npm init -y
npm install express dotenv
Enter fullscreen mode Exit fullscreen mode

We'll use the built‑in fetch (Node 18+) to talk to ElevenLabs, so no extra HTTP client is required.

Create server.js:

// server.js
import express from 'express';
import dotenv from 'dotenv';
dotenv.config();

const app = express();
app.use(express.json());

const ELEVEN_API_KEY = process.env.ELEVENLABS_API_KEY;
const ELEVEN_TTS_ENDPOINT = 'https://api.elevenlabs.io/v1/text-to-speech';

// Helper to call ElevenLabs TTS
async function synthesizeSpeech(text, voiceId = 'EXAVITQu4vr4xnSDxMaL') {
  const response = await fetch(`${ELEVEN_TTS_ENDPOINT}/${voiceId}`, {
    method: 'POST',
    headers: {
      'xi-api-key': ELEVEN_API_KEY,
      'Content-Type': 'application/json',
      Accept: 'audio/mpeg',
    },
    body: JSON.stringify({
      text,
      model_id: 'eleven_monolingual_v1', // default model
      voice_settings: {
        stability: 0.75,
        similarity_boost: 0.85,
      },
    }),
  });

  if (!response.ok) {
    const err = await response.text();
    throw new Error(`ElevenLabs error: ${response.status} – ${err}`);
  }

  // Returns a Buffer containing the MP3 audio
  return Buffer.from(await response.arrayBuffer());
}
Enter fullscreen mode Exit fullscreen mode

Exposing a Simple API Endpoint

Add a route that receives a JSON payload { "text": "Hello world!" } and returns the generated MP3:

app.post('/api/speak', async (req, res) => {
  const { text, voiceId } = req.body;
  if (!text) {
    return res.status(400).json({ error: 'Missing "text" in request body' });
  }

  try {
    const audioBuffer = await synthesizeSpeech(text, voiceId);
    res.set({
      'Content-Type': 'audio/mpeg',
      'Content-Length': audioBuffer.length,
    });
    res.send(audioBuffer);
  } catch (err) {
    console.error(err);
    res.status(500).json({ error: err.message });
  }
});
Enter fullscreen mode Exit fullscreen mode

Finally, start the server:

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => console.log(`🚀 Server listening on http://localhost:${PORT}`));
Enter fullscreen mode Exit fullscreen mode

Run it with:

node server.js
Enter fullscreen mode Exit fullscreen mode

You now have a tiny micro‑service that turns any text into speech using ElevenLabs.

Streaming Audio Directly to the Browser

Instead of downloading the whole file first, you can pipe the response straight to the client. This reduces latency and feels more “assistant‑like”.

Update the route to use a ReadableStream:

app.post('/api/speak-stream', async (req, res) => {
  const { text, voiceId } = req.body;
  if (!text) return res.status(400).json({ error: 'Missing text' });

  try {
    const response = await fetch(`${ELEVEN_TTS_ENDPOINT}/${voiceId || 'EXAVITQu4vr4xnSDxMaL'}`, {
      method: 'POST',
      headers: {
        'xi-api-key': ELEVEN_API_KEY,
        'Content-Type': 'application/json',
        Accept: 'audio/mpeg',
      },
      body: JSON.stringify({
        text,
        model_id: 'eleven_monolingual_v1',
        voice_settings: { stability: 0.75, similarity_boost: 0.85 },
      }),
    });

    if (!response.ok) throw new Error(`ElevenLabs error ${response.status}`);

    // Forward the streaming body directly
    res.setHeader('Content-Type', 'audio/mpeg');
    response.body.pipeTo(new WritableStream({
      write(chunk) {
        res.write(chunk);
      },
      close() {
        res.end();
      },
    }));
  } catch (err) {
    console.error(err);
    res.status(500).json({ error: err.message });
  }
});
Enter fullscreen mode Exit fullscreen mode

On the front‑end you can now fetch the endpoint and pipe the result into an Audio element:

<button id="talkBtn">Speak</button>
<audio id="assistantAudio" controls></audio>

<script>
document.getElementById('talkBtn').addEventListener('click', async () => {
  const resp = await fetch('/api/speak', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ text: 'Hello, I am your AI assistant!' })
  });
  const blob = await resp.blob();
  const url = URL.createObjectURL(blob);
  const audio = document.getElementById('assistantAudio');
  audio.src = url;
  audio.play();
});
</script>
Enter fullscreen mode Exit fullscreen mode

That’s a functional voice assistant in under 100 lines of code!

Adding Voice Cloning (Optional)

ElevenLabs also lets you clone a voice from a short recording (as little as 10 seconds). Here’s a quick walkthrough:

  1. Upload a sample (WAV/MP3) to the /v1/voices/add endpoint.
  2. Grab the returned voice_id and reuse it in the TTS calls above. 
async function cloneVoice(samplePath, name = 'MyClone') {
  const form = new FormData();
  form.append('name', name);
  form.append('files', await fetch(samplePath).then(r => r.blob()), 'sample.wav');

  const resp = await fetch('https://api.elevenlabs.io/v1/voices/add', {
    method: 'POST',
    headers: { 'xi-api-key': ELEVEN_API_KEY },
    body: form,
  });

  if (!resp.ok) throw new Error('Cloning failed');
  const data = await resp.json();
  return data.voice_id; // store this for later TTS calls
}
Enter fullscreen mode Exit fullscreen mode

Once you have a voice_id, pass it to synthesizeSpeech or the /api/speak endpoint. Your assistant can now speak with a custom persona—perfect for brand‑specific bots or personal projects.

Debugging Tips

Symptom Likely Cause Fix
401 Unauthorized Wrong or missing API key Verify ELEVENLABS_API_KEY in .env and restart the server
Empty audio file Accept header not set to audio/mpeg Ensure the header is present in the request
Slow response Large text payload or low stability value Split long passages into chunks; experiment with stability and similarity_boost settings

Wrap‑up

We’ve built a minimal yet extensible AI voice assistant using ElevenLabs and Node.js. The core steps are:

  1. Obtain an API key from ElevenLabs.
  2. Set up a simple Express server that forwards text to the TTS endpoint.
  3. Stream the resulting MP3 back to the browser (or any client).
  4. (Optional) Clone a custom voice for a personal touch.

All of this runs on standard HTTP, so you can swap Express for Fastify, NestJS, or even a serverless function with minimal changes.

Next Steps

  • Add speech‑to‑text (STT) via a service like OpenAI Whisper or Google Speech API to make the assistant truly conversational.
  • Persist user preferences (voice choice, language) in a database.
  • Deploy to Vercel, Render, or Railway for a public URL you can share with friends.

Ready to give your app a voice that sounds human? Grab your ElevenLabs API key now and start experimenting: https://try.elevenlabs.io/kr07zfuqn1bp.

Happy coding! 🚀

Top comments (0)