DEV Community

Cover image for Wake word + voice commands in the browser: a full offline pipeline
VoxRT
VoxRT

Posted on

Wake word + voice commands in the browser: a full offline pipeline

#javascript #webdev #machinelearning #tutorial

Voice interfaces in the browser have always had the same friction: Web Speech API is patchy across browsers, ships your audio to a vendor, and often needs network to respond. If you want a private, offline pipeline that works the same everywhere, you have had to wire it yourself. That got easier when WebAssembly SIMD128 shipped in every modern browser.

A few weeks ago I wrote a piece on adding a wake word to a web app in five minutes. This is the natural next step. A wake word alone tells you when the user wants attention. To do anything useful, you also need to know what they said. This post wires up the second half: a 14-command keyword spotter that activates on wake and dispatches commands to your app. Everything runs on-device, in about 1.75 MB of assets total, at roughly 1% of one CPU core on a modern laptop.

What you will need

  • A modern browser: Chrome/Edge 91+, Firefox 89+, Safari 16.4+. WASM SIMD128 is required.
  • A microphone.
  • Any static file server on HTTPS or localhost. Browsers block getUserMedia on plain HTTP.

The pattern: cheap wake, activated commands

You could run keyword spotting continuously and skip the wake word. Don't. Two reasons.

First, wake-word models are trained for one job: sit in a hot loop for hours, fire on a single phrase, ignore everything else. They are tiny (100 KB of weights, ~275 KB total bundle) and hardened against false triggers from radio, TV, and background conversation. Command spotters are heavier (~1.47 MB with a 14-word vocabulary) and, running always-on, will occasionally fire when someone in the room says "play" or "next" without meaning to talk to your app.

Second, the split matches how users actually interact. A wake word says "I am about to talk to you." A command says what to do. Gating the command recognizer on the wake cuts false triggers and CPU when the user is not addressing your app.

The pattern:

[Microphone]
     ↓
[Wake word engine] ── always on, ~275 KB, ~0.16% RTF
     ↓ (fires on "Hey Assistant")
[Listening window ~3 seconds]
     ↓
[KWS engine] ── activated during window, ~1.47 MB, ~0.97% RTF
     ↓
[Command: play / pause / next / ...]
     ↓
[Your app]
Enter fullscreen mode Exit fullscreen mode

Both engines read the same 16 kHz mono PCM stream from the microphone. The wake engine sees every frame. The KWS engine only sees frames inside a listening window opened by the wake trigger. When a command fires (or the window times out), listening closes and the pipeline goes quiet again.

Step 1: install the SDKs 

npm install @voxrt/wake-word-browser @voxrt/kws-browser
Enter fullscreen mode Exit fullscreen mode

Or via CDN if you skip the bundler:

import initWake, { WakeWordEngine } from
  "https://unpkg.com/@voxrt/wake-word-browser@0.1.1/voxrt-wake-word-browser.js";
import initKws, { KwsEngine } from
  "https://unpkg.com/@voxrt/kws-browser@0.1.0/voxrt-kws-browser.js";
Enter fullscreen mode Exit fullscreen mode

Both packages ship two things: a WebAssembly runtime (.js + .wasm) and a .vxrt model file. The model files download separately at runtime and get memory-mapped by the engine.

Step 2: HTML shell

Minimum needed to show what is happening:

<!DOCTYPE html>
<html>
<head><meta charset="utf-8"><title>Voice control demo</title></head>
<body>
  <button id="start">Start listening</button>
  <p id="status">Idle</p>
  <p>Last command: <span id="last">none</span></p>
  <script type="module" src="./app.js"></script>
</body>
</html>
Enter fullscreen mode Exit fullscreen mode

Serve over HTTPS or from localhost.

Step 3: initialize both engines

Create app.js:

import initWake, { WakeWordEngine } from
  "https://unpkg.com/@voxrt/wake-word-browser@0.1.1/voxrt-wake-word-browser.js";
import initKws, { KwsEngine } from
  "https://unpkg.com/@voxrt/kws-browser@0.1.0/voxrt-kws-browser.js";

await Promise.all([initWake(), initKws()]);

const [wakeBytes, kwsBytes] = await Promise.all([
  fetch("https://unpkg.com/@voxrt/wake-word-browser@0.1.1/voxrt_wake_word.vxrt")
    .then(r => r.arrayBuffer()).then(b => new Uint8Array(b)),
  fetch("https://unpkg.com/@voxrt/kws-browser@0.1.0/voxrt_kws.vxrt")
    .then(r => r.arrayBuffer()).then(b => new Uint8Array(b)),
]);

const wake = WakeWordEngine.fromBytes(wakeBytes);
const kws = KwsEngine.fromBytes(kwsBytes);
Enter fullscreen mode Exit fullscreen mode

Two engines, two model files, both loaded before we touch the microphone. Downloads happen in parallel. On a warm cache the whole init is under 100 ms.

Default thresholds are 0.9 in sigmoid space for both. Lower them to increase sensitivity (more false accepts). Raise them to be pickier. We will tune later if needed.

Step 4: capture microphone audio

Both engines want 16 kHz mono. We ask AudioContext for it and verify, because Firefox and Safari sometimes ignore the hint:

document.getElementById("start").onclick = async () => {
  const stream = await navigator.mediaDevices.getUserMedia({audio: true});
  const audioContext = new AudioContext({sampleRate: 16000});
  if (audioContext.sampleRate !== 16000) {
    console.warn(`Got ${audioContext.sampleRate} Hz. Need 16000. Resample or reject.`);
    return;
  }
  const source = audioContext.createMediaStreamSource(stream);
  const processor = audioContext.createScriptProcessor(512, 1, 1);
  const mute = audioContext.createGain();
  mute.gain.value = 0;

  // detection loop goes here (Step 5)

  source.connect(processor);
  processor.connect(mute).connect(audioContext.destination);
  document.getElementById("status").textContent = "Listening for wake word";
};
Enter fullscreen mode Exit fullscreen mode

Two gotchas from the wake-word post still apply here:

  • ScriptProcessorNode is deprecated. Fine for a demo, swap for AudioWorkletNode in production.
  • The muted GainNode between processor and destination is a Web Audio quirk. Without connecting the processor to the graph, audioprocess never fires. Muting the gain prevents the microphone from being routed straight to the speakers.

Step 5: detection loop with a listening window

Here is the actual logic. The wake engine sees every frame. On a wake hit, we open a 3-second window during which frames also go to the KWS engine. First command detection closes the window early.

const LISTEN_WINDOW_MS = 3000;

const pcmBuffer = new Int16Array(512);
let listeningUntil = 0; // performance.now() timestamp; 0 means closed

processor.addEventListener("audioprocess", (e) => {
  const inputFloat = e.inputBuffer.getChannelData(0);

  // Float32 [-1, 1] to Int16
  for (let i = 0; i < inputFloat.length; i++) {
    const s = Math.max(-1, Math.min(1, inputFloat[i]));
    pcmBuffer[i] = s < 0 ? s * 0x8000 : s * 0x7fff;
  }

  // Wake word always runs
  for (const hit of wake.pushPcmI16(pcmBuffer)) {
    console.log(`Wake at ${hit.timestampSec.toFixed(3)}s, score ${hit.score.toFixed(3)}`);
    listeningUntil = performance.now() + LISTEN_WINDOW_MS;
    document.getElementById("status").textContent = "Listening for command...";
  }

  // KWS only runs while the listening window is open
  if (performance.now() < listeningUntil) {
    for (const command of kws.pushPcmI16(pcmBuffer)) {
      if (command.class === "hey_vox" || command.class === "voxrt") {
        // Also in KWS vocab. Treat as a second wake: re-arm the window, do not dispatch.
        listeningUntil = performance.now() + LISTEN_WINDOW_MS;
        continue;
      }
      console.log(`Command "${command.class}" score ${command.score.toFixed(3)}`);
      document.getElementById("last").textContent = command.class;
      dispatch(command.class);
      listeningUntil = 0; // close window after first command
      document.getElementById("status").textContent = "Listening for wake word";
    }
  } else if (listeningUntil !== 0) {
    // window expired without a command
    listeningUntil = 0;
    document.getElementById("status").textContent = "Listening for wake word";
  }
});

function dispatch(command) {
  switch (command) {
    case "play":  /* start media playback */  break;
    case "pause": /* pause media playback */  break;
    case "next":  /* skip to next item */     break;
    case "previous": /* previous item */      break;
    case "up":    /* volume up */             break;
    case "down":  /* volume down */           break;
    case "on":    /* enable something */      break;
    case "off":   /* disable something */     break;
    case "yes":   /* confirm */               break;
    case "no":    /* dismiss */               break;
    case "cancel": /* cancel current action */ break;
    case "back":  /* go back / navigate up */ break;
  }
}
Enter fullscreen mode Exit fullscreen mode

One note on the vocabulary overlap: voxrt and hey_vox are both in KWS's 14-word set. Once the listening window is open, the user can say them and KWS will fire. The filter at the top of the loop treats those two as a "second wake" (re-arm the window) rather than a command. Without that filter, saying "hey vox" mid-listening would silently close the window on a no-op dispatch. Either behaviour is defensible, but the reader should see the choice.

That is the whole runtime. Say "Hey Assistant", then within 3 seconds say one of the 14 supported commands, and dispatch fires. Open DevTools (F12, Console) to see the timing and scores.

Numbers

Measured on a MacBook Pro M4 running Chrome, single-threaded WASM with SIMD128:

  • Wake word: RTF 0.16% (625× real time), always on
  • KWS: RTF 0.97% (103× real time), only during listening windows
  • Combined footprint: ~1% of one CPU core when actively listening, closer to 0.2% between windows

Model quality:

  • Wake word ("Hey Assistant"): ROC AUC 0.9966, precision 0.993 and recall 0.982 at threshold 0.9 on the reference test split (11,656 utterances)
  • KWS (14 commands): F1 macro 0.9671 at threshold 0.9 on a speaker-disjoint test set. Per-class ROC AUC is ≥0.99 for every command

Assets shipped to the browser:

  • Wake-word runtime + model: ~275 KB
  • KWS runtime + model: ~1.47 MB
  • Total pipeline: ~1.75 MB

That is smaller than the average site's hero image.

The 14-word vocabulary

Fixed in v0.1.0: yes, no, cancel, play, pause, next, previous, up, down, back, on, off, voxrt, hey_vox.

It covers three classes of interaction:

  • Confirmation and cancellation: yes, no, cancel, back
  • Media control: play, pause, next, previous
  • State toggles and navigation: up, down, on, off

Those 12 cover most "wake, then dispatch" flows: a recipe app in the kitchen, an accessibility layer for hands-free navigation, a dashboard that toggles panels by voice, a car UI. The remaining two (voxrt, hey_vox) exist as alternate wake-style phrases if you want a second entry point.

If you need product-specific words (custom brand names, unique verbs, domain jargon), the vocabulary is currently fixed at these 14. Custom command sets are a paid tier: help@voxrt.com.

Where this fits

Two-stage voice control fits a specific shape of use case:

  • Hands are busy or dirty. Kitchen recipe apps, workshop tool controllers, medical stations.
  • Accessibility. Users with motor limitations get an entry-level voice control layer that does not depend on Apple or Google account state.
  • In-car dashboards. Voice shortcuts on top of a touch UI, without shipping cabin audio to a cloud provider.
  • Kiosks and public terminals. No microphone stream leaves the machine.

If you need continuous transcription (arbitrary sentences, not fixed commands), you want streaming ASR instead. That is not in browser yet: our streaming ASR runtime is Android, iOS, and Linux only. In-browser ASR is a real question we get and the honest answer today is "not shipped."

Wrap up

Full pipeline, on-device, in ~1.75 MB. Two SDKs, one shared .vxrt runtime, no cloud, no per-request cost. Wake word optimized for always-on, command spotter activated on demand.

Repos and packages:

If you build something with this, the vocabulary limits (14 words, English) will be the first thing that pinches. That is next on our roadmap.

Top comments (0)