Everybody can now build the best voice agent into their own product. The #1 loop is open
source, and the benchmark that says so is public.
- Demo, no keys: todoforai.github.io/voiceloop
-
Library: github.com/todoforai/voiceloop —
npm i @todoforai/voiceloop - Bench: github.com/todoforai/voice-agent-bench
Why we built it
We wanted the fluid JARVIS feel in the browser for TODOforAI — you talk, it answers within a
second, you interrupt it mid-sentence and it just stops. We could not find a stack that did
this properly. Closed APIs were close but not ours; the open frameworks talked over the user,
or worse, heard their own voice through the speakers and cut themselves off.
It is 2026. This should be a solved problem. So we solved it and published the whole thing:
the loop, the numbers, and the rig that produced the numbers.
What voiceloop is
A zero-dependency JavaScript library that runs the full loop in the browser:
VAD → STT → LLM → TTS, with the hard parts already handled.
- Real barge-in — triggers on transcribed novel words, not mic energy, so the agent's own voice leaking into the mic never cuts it off.
- Self-echo filtering — a word-match filter compares what the mic hears against what the agent is currently saying. 0 self-interruptions in 30 echo-coupled turns with echo cancellation fully off.
- First audio under a second — TTS speaks sentence 1 while the LLM is still writing sentence 2, and the LLM call starts speculatively during your end-of-turn pause.
- Local-first — Silero VAD and Piper TTS run as WASM in the tab. Free, no cloud round-trip.
- Serialized turns — rapid-fire turns, tool results, holds and replays can never talk over each other. Locked in by 178 tests.
Everything is pluggable: any OpenAI-compatible LLM, four STT providers (Web Speech,
ElevenLabs Scribe, Deepgram Flux, Speechmatics), swappable TTS (Piper local, ElevenLabs
cloud, or your own).
import { VoiceAgent, unlockAudio } from '@todoforai/voiceloop';
const agent = new VoiceAgent({
llmUrl: '/api/chat/completions', // any OpenAI-compatible endpoint, behind your proxy
model: 'claude-haiku-4-5',
persona: 'You are a friendly cooking assistant.',
onEvent: (e) => { if (e.type === 'assistant') render(e.text); },
});
button.onclick = async () => { unlockAudio(); await agent.start(); };
That is the entire integration.
The numbers
Latency claims in voice AI are usually self-reported and unreproducible. We did not want to
add another one, so we built voice-agent-bench:
a black-box rig. A scripted "person" (byte-identical pre-generated speech) talks into a
virtual mic, the agent's speaker output is recorded, and every score is derived from the
audio alone. No integration needed — any agent that makes sound can be measured, including
closed ones.
Every system gets the same scripted conversations and, where the system allows it, the same
fixed mock LLM (300ms TTFT), so the comparison isolates the voice loop from the model. 5
conversations × 6 turns pooled, n=30, median and p95 — single runs jitter by ±300ms and are
not worth printing.
Smalltalk — same mock LLM, 5×6 turns pooled
| configuration | voice→voice | p95 | barge-in stop | stalls |
|---|---|---|---|---|
| OpenAI Realtime (speech-to-speech, own LLM) * | 866ms | 1644 | 429ms | 20 |
| voiceloop · deepgram + ElevenLabs flash | 862ms | 1067 | 944ms | 16 |
| voiceloop · deepgram + Piper (free, local TTS) | 974ms | 1287 | 1463ms | 19 |
| Pipecat 1.8.1 · deepgram + EL flash | 1046ms | 3573 | 542ms | 14 |
| ElevenLabs ConvAI | 1454ms | 1632 | 1042ms | 8 |
| voiceloop · EL Scribe + EL flash | 1562ms | 1855 | 1566ms | 12 |
| voiceloop · Speechmatics + EL flash | 1706ms | 2069 | 1046ms | 17 |
| voiceloop · webspeech + Piper (zero-key) | 2113ms | 2607 | 1257ms | 30 |
Across scenarios
| system | clean | hesitation | talked through user | echo | cut itself |
|---|---|---|---|---|---|
| OpenAI Realtime * | 870 | 1290 | 0 (yields 130ms) | 790 | 17/30 |
| voiceloop · deepgram + EL flash | 860 | 1400 | 0 (420ms) | 930 | 0 |
| voiceloop · deepgram + Piper | 970 | 1400 | 0 | — | — |
| Pipecat | 1050 | 1290 | 2 (200ms) | 1320 | 20/30 |
| ElevenLabs ConvAI | 1450 | 1810 | 0 (490ms) | 1410 | 0 |
* Realtime is speech-to-speech and can't use the fixed mock LLM, so its row isn't fully
apples-to-apples.
How to read it
Clean audio: voiceloop with Deepgram Flux + ElevenLabs flash is the fastest configuration
we measured, at 862ms median — and its p95 (1067ms) is the tightest in the table by a wide
margin. Pipecat's p95 of 3573ms on the same providers means one turn in twenty takes over
three seconds. The free, fully local Piper path lands at 974ms with no cloud TTS at all.
Echo is the failure that separates the stacks. Feed each system its own voice back
through the mic (−15dB, 30ms delay, no AEC — what a laptop with the speakers on actually
does) and the other fast stacks hear themselves as the user and cut their own replies:
Pipecat on 20 of 30 turns, OpenAI Realtime on 17. voiceloop cut itself zero times and
ran echo-coupled turns at 930ms — parity with clean. Word-level echo filtering costs no
latency once it classifies correctly.
Hesitation: a user who pauses mid-sentence should not be talked over. Every stack except
Pipecat backs off; voiceloop enters 2 of 30 hesitation turns and yields within 420ms.
The zero-key default is honest about its cost. Browser Web Speech + Piper needs no
account anywhere and runs the demo, but it is ~1.2s slower to close a turn than cloud STT
(2113ms). Pick a pipeline STT provider for the numbers above.
Full per-scenario tables, methodology and reproduction steps:
results/RESULTS.md.
Hundreds of configurations, so you don't have to
The rows above are the survivors. Behind them are hundreds of runs across STT providers, TTS
engines, VAD thresholds, end-of-turn debounces, barge-in minimum lengths, prefetch stability
windows and echo-match thresholds. Every knob that mattered is exposed in
src/tuning.js with the
default set to what won on the bench. The edge cases you would otherwise discover one
production bug at a time — the agent interrupting itself, tool calls firing on a sentence
the user was still amending, a hung tool stalling the next turn — are already handled and
regression-tested.
Try it, use it, beat it
- Try: todoforai.github.io/voiceloop — 20 seconds, no keys, runs in your tab.
-
Use:
npm i @todoforai/voiceloop— MIT, zero dependencies. Point it at any OpenAI-compatible endpoint and wire in your tools. - Beat it: the bench is black-box and public. If your stack does better, add it — ADDING_A_SUT.md is the contract. We will print the row.
This is the voice loop inside TODOforAI's JARVIS; the integration overhead between the
library and the product is nil, which is exactly the point. Everybody should have the best
voice loop. Star it, share it, contribute — let's keep the best one open source.
Top comments (1)
Good to know