Building Offline Voice Control with a Custom Wake Word and ONNX

Cloud speech recognition is useful, but many voice-control products do not need to transcribe every sentence. If the interaction starts with a wake word and continues with a small list of fixed commands, local keyword spotting can be a simpler solution.

This article shows a practical architecture for:

• Detecting a custom wake word locally
• Recognizing multiple fixed commands with one model
• Keeping microphone audio on the device
• Deploying the same workflow with ONNX or TFLite
• Improving false-trigger performance with a second training round

The two-stage architecture

A useful design is to separate wake-up from command recognition:

Microphone
    ↓
Wake-word model
    ↓
Short command window
    ↓
Multi-keyword model
    ↓
Application action

For example:

Wake word: Hey device
Command: Turn on the light

The wake-word model can run continuously with a conservative threshold. After it detects the wake word, the application enables the command model for a short period.

This avoids running a larger command recognizer continuously and makes the interaction state easier to control.

Why not use speech recognition for the wake word?

The Web Speech API is convenient for prototypes, but it is not a reliable always-on wake-word detector across mobile browsers. Browser support for SpeechRecognition and its continuous mode varies, and some implementations may disconnect or use a remote recognition service.

For a local browser implementation, a better approach is:

1. Request microphone access after a user gesture.
2. Read the audio with getUserMedia().
3. Convert the stream into the frame format expected by the model.
4. Run local keyword-spotting inference.
5. Start ASR or show the command interface only after the wake word is detected.

The browser still has permission and lifecycle limitations, but the keyword detector does not depend on SpeechRecognition returning a transcript.

One model for multiple fixed commands

Suppose a device needs these commands:

next
previous
stop
open menu
close menu

A multi-keyword model can return a score for each label. The application maps the selected label to an action:

const actions = {
  next: () => nextPage(),
  previous: () => previousPage(),
  stop: () => stopPlayback(),
  "open menu": () => openMenu(),
  "close menu": () => closeMenu(),
};

function handleKeyword(label) {
  actions[label]?.();
}

This works well when the vocabulary is known in advance. It is not a replacement for open-ended speech recognition. A request such as “Set the living-room light to 37 percent” needs ASR and intent parsing because the words and parameters are not fixed.

False triggers: R0 and R1

Getting the target phrase recognized is only the first part of the problem. A real device also has to reject:

• Normal conversation
• Music and television audio
• Fans and mechanical noise
• Similar-sounding phrases
• Microphone or enclosure noise

The first model can be treated as an R0 model. After testing it, collect the audio clips that incorrectly trigger it. These hard negatives become part of the next training round, often called R1.

R0 model
   ↓
Scan difficult negative audio
   ↓
Collect false-trigger clips
   ↓
Retrain with hard negatives
   ↓
R1 model

R1 is a training process, not a second model that must run at runtime. The goal is to teach the model which confusing sounds should be rejected while preserving recall for the real phrase.

Local inference and privacy

Local keyword spotting is useful when a product needs:

• Low latency
• Offline operation
• Predictable fixed-command behavior
• Lower cloud and network dependency
• Better control over microphone privacy

The model file is only one part of the system. The runtime also needs audio capture, framing, feature extraction, threshold handling, cooldown logic and application callbacks.

For this reason, always validate the complete pipeline rather than testing only whether the model file can be loaded.

ONNX and TFLite deployment

ONNX is convenient for desktop, Python, Android and browser runtimes that support ONNX Runtime. TFLite or TFLite Micro is commonly used on constrained embedded devices.

Before moving a model to another runtime, verify:

• Sample rate
• PCM format
• Frame length and hop size
• Mel-feature configuration
• Input tensor shape
• Output label order
• Quantization settings
• Detection threshold

If preprocessing changes, recall can drop even when the model file is unchanged.

A practical testing checklist

Test the complete system with the final microphone and target environment:

1. Test the wake word with multiple speakers.
2. Test different distances and speaking speeds.
3. Play ordinary conversation and background audio.
4. Measure false triggers per hour.
5. Measure recall separately from false-trigger rate.
6. Add difficult negative samples for the next training round.
7. Test cooldown and repeated-trigger behavior.
8. Verify that every recognized command maps to the intended safe action.

Do not optimize only for recall. A model that triggers too often is not practical for a product, even if it rarely misses the target phrase.

Open-source inference example

The onnx-wakeword project contains open-source inference code and examples for running ONNX wake-word and keyword-spotting models in local applications.

It can be used to test the inference path before integrating a model into a larger product. The final result should still be tested with the actual microphone, operating system and hardware.

Conclusion

For a product with one wake word and a limited command vocabulary, a two-stage local design is often easier to control than continuously running full ASR:

Wake word → command window → fixed keyword → local action

The approach can provide low latency, offline operation and a small deployment footprint across ONNX- and TFLite-compatible platforms.

You can read the longer architecture guide on Voicute, or try the local inference code in the onnx-wakeword repository.

The accompanying German voice-control demonstration is available on YouTube.