Build an AI Voice Agent in TypeScript — Cloud or 100% Local, One Config Swap

You can build an AI voice agent in TypeScript with LLMRTC — an open-source SDK that streams your microphone over WebRTC through a speech-to-text → LLM → text-to-speech pipeline, then swaps OpenAI for a fully local stack by changing config, not code.

That last part is the whole point of this tutorial. Every ranking guide for this locks you to one vendor: OpenAI's realtime models, one STT service, one telephony platform.

Here you'll build the agent once, run it on OpenAI, then run the same agent 100% offline — no keys, no cloud. And we'll cover the step almost every tutorial skips: what happens when a real user, behind a corporate firewall, tries to connect.

We ran this whole thing live before publishing. Every number below is measured on our own machine, not a spec sheet.

TL;DR: To build a real-time AI voice agent in TypeScript, use LLMRTC — an open-source SDK that streams audio over WebRTC through an STT→LLM→TTS pipeline. Unlike vendor-locked tutorials, LLMRTC swaps LLM/STT/TTS providers by config, so the same agent runs on OpenAI or fully local (Ollama + Faster-Whisper + Piper). For real users behind NAT/firewalls, add a TURN server (Open Relay is free).

What you're building

You're building a two-way voice conversation in the browser: you speak, an AI agent answers out loud, and you can cut it off mid-sentence.

The shape is simple. Your browser captures the microphone and sends audio to a Node backend over WebRTC. The backend runs the pipeline — speech-to-text, then an LLM, then text-to-speech — and streams the agent's voice back over the same connection.

Three stages do the work:

• STT (speech-to-text) turns your spoken audio into text.
• LLM reads that text and generates a reply.
• TTS (text-to-speech) turns the reply back into audio.

Streaming ties it together. Playback starts before the full reply is generated, so the agent feels responsive instead of walkie-talkie slow.

The feature that makes it feel human is barge-in. When you start talking over the agent, server-side voice-activity detection (VAD) hears you and cancels the agent's speech instantly — just like interrupting a person. LLMRTC handles VAD and barge-in on the server, so you don't hand-roll it.

Why LLMRTC

LLMRTC is the right base here for four concrete reasons — and one deliberate trade-off we'll name up front.

It's TypeScript-native, end to end. The backend and the browser client are both TypeScript. No Python service to stand up beside your Node app, which is where most open-source voice stacks send you.

It's provider-agnostic by config. OpenAI is one option among many for the LLM, STT, and TTS layers. You swap providers by editing a config object — not by rewriting your app. That's the swap we'll demonstrate live in a later section.

It's Apache 2.0, with no platform. There's no cloud control plane you rent, no per-minute platform fee, no dashboard you're forced through. You run the backend yourself, and it's feature-complete today.

It's built by an infrastructure team, and it's open. LLMRTC is built and maintained by Metered, which has operated production WebRTC infrastructure (TURN, STUN, signalling) for a decade. Full disclosure so you know where it comes from: the SDK is Apache 2.0 on GitHub, and it's genuinely free.

Setup

You need three things before the first line of code: Node.js 20+, FFmpeg, and the three LLMRTC packages.

Check your Node version first — LLMRTC requires Node 20 or newer:

node --version   # v20.x or higher

Install FFmpeg. LLMRTC uses it to convert streaming TTS audio, so it's required, not optional:

# macOS
brew install ffmpeg
# Debian/Ubuntu
sudo apt install ffmpeg
# Windows
choco install ffmpeg

Now create the project as an ES module and install the SDK. The three packages split cleanly: -backend runs the pipeline, -web-client runs in the browser, and -core holds the shared types.

npm init -y
npm pkg set type=module
npm install @llmrtc/llmrtc-backend @llmrtc/llmrtc-web-client @llmrtc/llmrtc-core
npm install -D tsx typescript @types/node

For the cloud stack you'll need one OpenAI API key. Put it in a .env file — never commit it:

# .env
OPENAI_API_KEY=sk-...

That's the entire setup. No accounts, no CLI login, no media server. Next: the backend.

The backend (about 40 lines)

The whole backend is one file. It configures a provider set, hands it to an LLMRTCServer, and starts listening.

Here's the complete server.ts we ran — including both the cloud and local provider sets, selected by an env var:

import 'dotenv/config';
import {
  LLMRTCServer,
  OpenAILLMProvider,
  OpenAIWhisperProvider,
  OpenAITTSProvider,
  OllamaLLMProvider,
  FasterWhisperProvider,
  PiperTTSProvider,
  type ConversationProviders,
} from '@llmrtc/llmrtc-backend';

// The entire cloud -> local swap lives in this one block.
// STACK=cloud (default) uses OpenAI; STACK=local runs 100% on your machine.

const STACK = process.env.STACK ?? 'cloud';

const providers: ConversationProviders =
  STACK === 'local'
    ? {
        llm: new OllamaLLMProvider({ model: 'llama3.2' }), // Ollama on :11434
        stt: new FasterWhisperProvider(), // faster-whisper server on :9000
        tts: new PiperTTSProvider({ voice: 'en_US-amy-medium' }), // Piper on :5002
      }
    : {
        llm: new OpenAILLMProvider({
          apiKey: process.env.OPENAI_API_KEY!,
          model: 'gpt-5.6-terra',
        }),
        stt: new OpenAIWhisperProvider({ apiKey: process.env.OPENAI_API_KEY! }),
        tts: new OpenAITTSProvider({
          apiKey: process.env.OPENAI_API_KEY!,
          voice: 'alloy',
        }),
      };

const server = new LLMRTCServer({
  providers,
  systemPrompt: `You are a helpful voice assistant.
Keep responses concise and conversational.
Respond in 1-2 sentences when possible.`,
  streamingTTS: true,
  port: 8787,
});

await server.start();
console.log(`[${STACK}] voice agent running on ws://localhost:8787`);

Read it top to bottom and it explains itself.

The providers object is the agent's brain. Each layer — llm, stt, tts — is a provider instance, and swapping a layer means swapping one line. Keep that in mind; it's the payoff later.

The systemPrompt shapes the agent's personality. We ask for short, conversational replies because long monologues feel wrong in a voice UI.

streamingTTS: true is what makes it feel live — audio starts flowing before the reply is fully written. And port: 8787 is where the browser client will connect.

One call — await server.start() — and the pipeline is live. That's the backend, done.

The browser client

The browser side captures the mic, plays the agent's voice, and listens to a handful of events. Here are the parts that matter — the full client, with the on-page latency panel, is in the companion repo.

Create the client and point it at the backend:

import { LLMRTCWebClient } from 'https://esm.sh/@llmrtc/llmrtc-web-client@1.2.0';

const client = new LLMRTCWebClient({ signallingUrl: 'ws://localhost:8787' });

Starting the conversation is a click handler: start the client, grab the mic, and share it.

$('talk').onclick = async () => {
  $('talk').disabled = true;
  await client.start();
  const micStream = await navigator.mediaDevices.getUserMedia({ audio: true });
  client.shareAudio(micStream);
  $('talk').textContent = 'Listening — just speak';
};

From there, the client emits events you subscribe to: transcript (what you said), llmChunk (the reply streaming in), ttsStart and ttsComplete (agent speaking), and ttsCancelled (barge-in fired).

Gotcha #1: keep one audio element alive for the whole session

This one cost us real time, so here's the fix straight up. The agent's TTS audio arrives on one persistent WebRTC track per session — not a fresh track per turn.

A naive ttsCancelled handler that tears the audio element down — audio.pause(); audio.srcObject = null, roughly the shape a quick read of the docs suggests — silences every turn after the first barge-in. The symptom is nasty because it looks like a logic bug, not an audio bug: transcripts keep flowing, the agent keeps "replying," but there's no sound.

The fix is to keep a single Audio element alive for the session and never destroy it on cancel. On barge-in, just let the server's cancel stop the stream:

// ONE persistent element for the session: the TTS audio arrives on a single
// WebRTC track. Destroying the element on ttsCancelled (as a naive reading of
// the docs suggests) silences every later turn — the gotcha we hit live.
let currentAudio = null;
client.on('ttsTrack', (stream) => {
  console.log('[ttsTrack] track event fired');
  currentAudio = new Audio();
  currentAudio.srcObject = stream;
  currentAudio.play();
});

On the next ttsStart, if the element is paused, call play() again — don't rebuild it. That single decision is the difference between a demo that works once and one that survives a real back-and-forth.

Run it: our measured numbers

Now the fun part — running it and watching real latency. Install, add your key, start the backend, and serve the client:

npm install
cp .env.example .env   # put your OPENAI_API_KEY in .env
npm run cloud          # backend on ws://localhost:8787
npm run serve          # client on http://localhost:3000 (separate terminal)

Open http://localhost:3000, click Start talking, allow the mic, and speak. You'll see your words appear as a transcript, the agent's reply stream in, then hear it out loud. Talk over it mid-reply and it stops — that's barge-in.

We instrumented the client with performance.now() between SDK events and ran two multi-turn sessions on the cloud stack. Here's exactly what we got.

Measured on our run, 2026-07-14 — MacBook Pro M1 Pro, 16 GB RAM, residential network. OpenAI GPT + Whisper STT + OpenAI TTS, streamingTTS: true. These are our numbers on our hardware, not an LLMRTC or Metered performance guarantee:

Turn	Speech end → transcript	→ first LLM token	→ agent audio	Barge-in
S1-T1	1,267 ms	2,832 ms	2,832 ms	—
S1-T2	1,841 ms	2,700 ms	2,751 ms	—
S2-T1	1,414 ms	3,619 ms	3,870 ms	interrupted ✓
S3-T1	1,214 ms	2,578 ms	3,014 ms	cut ≤1 ms after speech-start ✓
S3-T2	1,531 ms	2,295 ms	2,598 ms	interrupted again ✓

End to end, we measured roughly 2.5–3.9 seconds from the end of our speech to the agent's first audio on the default buffered pipeline.

Here's the insight that matters: STT dominates. Buffered Whisper alone accounts for 1.2–1.8 seconds of that — the single biggest slice. The LLM and TTS are not your bottleneck; transcription is.

Barge-in was effectively instant. The server cancels TTS the moment its VAD detects speech, and the ttsCancelled event reached our client within about 1 ms of the speechStart event — the perceived cut is really just VAD detection time.

Want it faster? LLMRTC's docs describe two paths we did not benchmark here: streamingSTT with a streaming STT provider (so transcription overlaps your speech instead of waiting for you to finish), and an experimental realtimeSpeech relay mode. If latency is your priority, start there — see the LLMRTC docs.

The payoff: go 100% local

Here's the moment that no vendor-locked tutorial can show you. Take the exact same agent and run it fully offline — no OpenAI key, no cloud — by changing configuration only.

Remember the providers block in server.ts? The local branch is the only thing that changes:

const providers: ConversationProviders = {
  llm: new OllamaLLMProvider({ model: 'llama3.2' }),        // Ollama on :11434
  stt: new FasterWhisperProvider(),                         // faster-whisper on :9000
  tts: new PiperTTSProvider({ voice: 'en_US-amy-medium' }), // Piper on :5002
};

Same LLMRTCServer, same systemPrompt, same browser client — different brain. Your application code doesn't move a line. That's provider-agnostic, demonstrated instead of promised.

To run it, start the three local services and flip the env var:

brew install ollama && ollama serve     # LLM on :11434
ollama pull llama3.2                    # ~2 GB

docker run -d --name faster-whisper -p 9001:8000 fedirz/faster-whisper-server:latest-cpu
docker run -d --name piper -p 5099:5000 \
  -e MODEL_DOWNLOAD_LINK="https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_US/amy/medium/en_US-amy-medium.onnx?download=true" \
  artibex/piper-http

npx tsx local-bridge.ts   # /asr on :9000 + /api/tts on :5002 (separate terminal)
npm run local
npm run serve

Now the numbers. Measured on our run, 2026-07-14 — same M1 Pro (16 GB), fully local: Ollama llama3.2 + faster-whisper-small + Piper. Again, our machine, not a guarantee:

Turn	Speech end → transcript	→ first LLM token	→ agent audio	Barge-in
L-T1	9,070 ms	12,207 ms	12,337 ms	—
L-T2	9,318 ms	9,615 ms	11,624 ms	—
L-T3	9,257 ms	9,539 ms	10,810 ms	—
L-T4	8,855 ms	9,218 ms	10,547 ms	cut/speech-start gap 7 ms ✓
L-T6	10,781 ms	11,828 ms	12,576 ms	—

Fully local, we measured about 10.5–12.6 seconds end to end on a 16 GB laptop — and once again, STT is the story. The buffered faster-whisper-small model took roughly 9–11 seconds of every turn.

Barge-in behaved identically to cloud (a 7 ms cut in our run). The Piper TTS container ran x86-emulated on our Apple Silicon and still wasn't the bottleneck.

So here's the honest cloud-vs-local trade, stated plainly. Cloud gets you 2.5–3.9 s and costs API dollars.

Local gets you privacy and zero per-request cost, but 10.5–12.6 s on a laptop — the STT model is the lever, and a GPU or a smaller streaming STT model is where you'd claw the seconds back. Pick per use case; the code doesn't care.

Ship it to real users: NAT and TURN

Your agent works on localhost. That's exactly why it will break for real users — and this is the section nearly every voice-agent tutorial leaves out.

On localhost there's no network to cross, so WebRTC connects trivially. Real users sit behind home routers and corporate firewalls that block direct peer connections. And voice-agent media is almost entirely relay traffic in production — so when the direct path fails, the connection has nowhere to go.

The fix is a TURN server: a relay that carries the media when a direct path is impossible. This isn't a Metered opinion — it's LLMRTC's own docs, which state plainly that "for production use, WebRTC requires a TURN server to ensure reliable connections for users behind NAT/firewalls" and recommend Open Relay, a free global TURN network with 20 GB of monthly TURN usage at no cost (llmrtc.org, 2026-07-14).

or you can use the paid Metered TURN servers, if you are looking for a paid service

"Production-ready" in the title; the one thing that breaks in production, omitted. If you want the networking primer it skips, this STUN vs TURN vs ICE explainer is a solid starting point.

or if you need a list of WebRTC ICE servers, then for your app

Wiring TURN into LLMRTC is a config option — pass iceServers to the web client:

const client = new LLMRTCWebClient({
  signallingUrl: 'wss://your-backend.example.com',
  iceServers: [
    // Copy your STUN + TURN URLs and credentials from the Open Relay
    // dashboard: https://www.metered.ca/tools/openrelay/
    { urls: 'stun:<your-open-relay-stun-url>' },
    {
      urls: 'turn:<your-open-relay-turn-url>',
      username: '<your-username>',
      credential: '<your-credential>',
    },
  ],
});

Open Relay's 20 GB/month free tier is enough to test and ship a small agent. When you outgrow it — more concurrent users, region pinning, dynamic credentials, human support — Metered's paid TURN service scales the same relay up to production volumes. Either way, don't ship a voice agent without one.

Frequently Asked Questions

How do I build a real-time voice AI agent in TypeScript?

Use LLMRTC: install @llmrtc/llmrtc-backend and @llmrtc/llmrtc-web-client on Node 20+ (with FFmpeg). The backend runs an STT→LLM→TTS pipeline configured with provider objects and a system prompt; the browser client connects over WebRTC and manages the microphone and audio playback. Start both, and you have a working voice agent. (llmrtc.org, 2026-07-14)

Can I swap providers or run the agent locally?

Yes — LLMRTC is provider-agnostic by config. Start on OpenAI, then switch the LLM to local Ollama or LM Studio, speech-to-text to Faster-Whisper, and text-to-speech to Piper — the same agent, now fully offline and free per request. You change only the provider configuration, not your application code. (llmrtc.org, 2026-07-14)

Do I need a TURN server for a production voice agent?

Yes — voice-agent media is nearly all relay traffic, and users behind NAT or firewalls can't connect without one. LLMRTC's docs say WebRTC "requires a TURN server ... for users behind NAT/firewalls" and recommend Open Relay (free for 20 GB/month); configure it in your ICE servers before you ship. (llmrtc.org, 2026-07-14)

What does it cost to run?

LLMRTC itself is free (Apache 2.0, no paid tier), and a fully local stack — Ollama, Faster-Whisper, Piper — costs nothing per request. Open Relay gives you 20 GB/month of free TURN. Your only spend is optional cloud provider API usage if you choose OpenAI, Anthropic, or another hosted model. (llmrtc.org, 2026-07-14)

Wrapping up

You just built an AI voice agent in TypeScript that captures the mic, streams a reply over WebRTC, and handles barge-in — then ran the same agent 100% locally by changing one config block. That's provider freedom you can prove, not a bullet point.

The two things to carry forward: STT is your latency bottleneck (we measured it on both stacks), and you need a TURN server before real users behind firewalls can connect — LLMRTC's own docs say so, and most tutorials pretend otherwise.

Where to go next:

• Install it: npm install @llmrtc/llmrtc-backend @llmrtc/llmrtc-web-client @llmrtc/llmrtc-core
• Read the docs at llmrtc.org — start with streamingSTT if you want lower latency.
• Star the repo on GitHub if this saved you an afternoon.
• Add agent orchestration (events, tools, multi-step logic) — our guide on building a WebSocket server in Node.js is the natural next step for wiring agent events.

Get the complete app

The full, runnable demo — server.ts, the browser client, and local-bridge.ts — is on GitHub:

git clone <repo-url-pending>
cd llmrtc-voice-agent-demo
npm install

---

About the author: This tutorial was written and tested by the A.L. Every latency number here was measured on my own hardware on 2026-07-14, not taken from a spec sheet.

Comments

[alakkadshaw]

Thank you for reading. I hope you like the article

[Luis]

I was impressed by the provider-agnostic design of LLMRTC, which allows for seamless swapping of LLM, STT, and TTS providers via a simple config change. The fact that it's TypeScript-native, end to end, is also a significant advantage, eliminating the need for a separate Python service. I've worked on similar projects where vendor lock-in was a major concern, and this approach would have saved us a lot of headaches. Have you considered exploring other use cases for LLMRTC, such as integrating it with existing telephony platforms or expanding its capabilities to support multi-language support?