Building a Small Multilingual Offline Voice-Control Pipeline with ONNX

voicute — Mon, 12 Oct 2026 19:44:34 +0000

Many products do not need a full speech-to-text system to support voice control.

A smart switch may only need:

one wake phrase;
“turn the light on”;
“turn the light off”;
“brighter”;
“dimmer”.

For this kind of fixed vocabulary, keyword spotting (KWS) can be a better fit than running a general ASR model all the time.

This article explains the pipeline we use for small offline keyword models, what KWS can and cannot do, and how the same model can move from a browser test to an embedded product.

KWS is not ASR

ASR answers:

What did the user say?

KWS answers:

Did one of the phrases I care about occur in this audio stream?

The processing paths are different:

ASR:
audio → speech-to-text → text matching / intent parsing → action

KWS:
audio → acoustic features → keyword model → command id → action

If the command set is small and predefined, KWS avoids a large transcription model and a second text-matching layer.

KWS is not a replacement for free-form speech. If users need to say things such as “set the living-room light to 50 percent”, the product will usually need ASR and intent parsing for the room and numeric value.

The runtime pipeline

Our offline runtime follows this shape:

microphone frames
      ↓
audio buffering
      ↓
Mel features
      ↓
keyword model
      ↓
matched keyword outputs
      ↓
consecutive-frame and false-trigger logic
      ↓
application callback

The model is only one part of a usable always-listening system. A raw score should not automatically trigger a product action. The runtime also needs temporal confirmation, cooldown handling, background-energy checks, and application-level state handling.

Why use a small model?

The model is expected to run continuously while the product is waiting for a command. Size and compute affect:

RAM and flash requirements;
battery life;
wake-up latency;
the cost of keeping the main processor active;
whether the model fits on a microcontroller.

The demo models in our open runtime are designed for small-footprint deployment. The repository includes ONNX paths for Python, Web and Android, plus an INT8 TFLite path for ESP32-S3.

The exact result still depends on the audio front end and target device. A desktop result should not be presented as an ESP32 result.

Multi-keyword models

A product may need a wake phrase and several commands. Instead of loading a separate model for every phrase, a multi-keyword model can expose several outputs from one inference pipeline:

"Hey Lamp"       → WAKE
"Turn it on"     → LIGHT_ON
"Turn it off"    → LIGHT_OFF
"Brighter"       → BRIGHTER
"Dimmer"         → DIMMER

This does not remove the need for testing. Similar-sounding phrases can interfere with one another, and each output can have a different recall/false-trigger trade-off.

What we test before shipping

For a real product, “the model triggered once in a demo” is not enough. We measure at least:

true-trigger recall;
false activations per hour;
distance and speaking volume;
different speakers and accents;
TV, music, fan and room noise;
model size and device memory;
end-to-end latency and power impact.

Recall and false activation rate must be considered together. Lowering a threshold can make a model easier to trigger while also increasing false activations.

From a phrase to a device test

For teams without a speech ML group, a practical validation workflow is:

define the exact fixed phrases;
generate a first model;
test it with real speakers before downloading;
run it on the target microphone and device;
collect misses and false activations;
add real-voice samples or hard negatives when needed.

Voicute is the online training and testing part of this workflow. The onnx-wakeword repository contains the open offline runtime and platform examples.

The important distinction is that online testing reduces the risk of downloading the wrong model; it does not replace validation on the final hardware.

When this architecture is a good fit

Use a small KWS pipeline when:

the vocabulary is fixed;
the device needs local or offline operation;
the device is battery-powered or resource-constrained;
low latency matters;
the product does not want to send continuous audio to a cloud service.

Use ASR and intent parsing when:

users can express the same intent in many ways;
commands contain open values, numbers or entities;
the product is closer to a conversational interface.

Disclosure: I’m the creator of Voicute. The model-generation service is commercial, while the offline inference runtime and integration examples are open source. The claims above should be validated on the reader’s own microphone, language and target device.

I built a portable keyword spotting engine — started with Chinese, now supporting English

voicute — Tue, 11 Aug 2026 04:16:05 +0000

I built a portable keyword spotting engine — started with Chinese, now supporting English

I wanted offline voice commands for a few Python projects. Nothing fancy — just a single file I could drop in and run.

What it does

One Python file — wakeword_engine.py, copy it into your project
Small model — ~135KB for 3 keywords, <5ms inference
Multi-keyword — one model detects 2-10 keywords in a single pass
Fully offline — no cloud, no internet, ONNX Runtime under the hood
5-layer noise filtering — consecutive frame check, background suppression, cooldown, burst lock, energy jump detection

Why Chinese first

Most KWS engines optimize for English. I started with Chinese instead. It's a tonal language with lots of single-syllable words — harder for keyword spotting. No pre-trained English embeddings to lean on either.

The upside: the architecture (causal TCN on mel spectrograms) doesn't depend on any language-specific pretrained model. Once Chinese worked, English was almost free. Just needed the training data.

Usage

pip install onnxruntime numpy pyaudio

from wakeword_engine import WakeWordEngine

engine = WakeWordEngine()
engine.load('model_info.json', 'melspectrogram.onnx')
engine.set_L1(True)
engine.start(lambda word, prob, info: print(f'{word} {prob:.0%}'))

What's working now

Platform	Status
Python	✅ mature
Web (ONNX Runtime Web)	✅ working
Android (Java)	✅ working
ESP32	🚧 early

Wyoming protocol integration for Home Assistant.

Repo: github.com/voicute/onnx-wakeword — MIT license.

DEV Community: voicute

Building a Small Multilingual Offline Voice-Control Pipeline with ONNX

KWS is not ASR

The runtime pipeline

Why use a small model?

Multi-keyword models

What we test before shipping

From a phrase to a device test

When this architecture is a good fit

I built a portable keyword spotting engine — started with Chinese, now supporting English

I built a portable keyword spotting engine — started with Chinese, now supporting English

What it does

Why Chinese first

Usage

What's working now

Next