Building a Small Multilingual Offline Voice-Control Pipeline with ONNX

Many products do not need a full speech-to-text system to support voice control.

A smart switch may only need:

• one wake phrase;
• “turn the light on”;
• “turn the light off”;
• “brighter”;
• “dimmer”.

For this kind of fixed vocabulary, keyword spotting (KWS) can be a better fit than running a general ASR model all the time.

This article explains the pipeline we use for small offline keyword models, what KWS can and cannot do, and how the same model can move from a browser test to an embedded product.

KWS is not ASR

ASR answers:

What did the user say?

KWS answers:

Did one of the phrases I care about occur in this audio stream?

The processing paths are different:

ASR:
audio → speech-to-text → text matching / intent parsing → action

KWS:
audio → acoustic features → keyword model → command id → action

If the command set is small and predefined, KWS avoids a large transcription model and a second text-matching layer.

KWS is not a replacement for free-form speech. If users need to say things such as “set the living-room light to 50 percent”, the product will usually need ASR and intent parsing for the room and numeric value.

The runtime pipeline

Our offline runtime follows this shape:

microphone frames
      ↓
audio buffering
      ↓
Mel features
      ↓
keyword model
      ↓
matched keyword outputs
      ↓
consecutive-frame and false-trigger logic
      ↓
application callback

The model is only one part of a usable always-listening system. A raw score should not automatically trigger a product action. The runtime also needs temporal confirmation, cooldown handling, background-energy checks, and application-level state handling.

Why use a small model?

The model is expected to run continuously while the product is waiting for a command. Size and compute affect:

• RAM and flash requirements;
• battery life;
• wake-up latency;
• the cost of keeping the main processor active;
• whether the model fits on a microcontroller.

The demo models in our open runtime are designed for small-footprint deployment. The repository includes ONNX paths for Python, Web and Android, plus an INT8 TFLite path for ESP32-S3.

The exact result still depends on the audio front end and target device. A desktop result should not be presented as an ESP32 result.

Multi-keyword models

A product may need a wake phrase and several commands. Instead of loading a separate model for every phrase, a multi-keyword model can expose several outputs from one inference pipeline:

"Hey Lamp"       → WAKE
"Turn it on"     → LIGHT_ON
"Turn it off"    → LIGHT_OFF
"Brighter"       → BRIGHTER
"Dimmer"         → DIMMER

This does not remove the need for testing. Similar-sounding phrases can interfere with one another, and each output can have a different recall/false-trigger trade-off.

What we test before shipping

For a real product, “the model triggered once in a demo” is not enough. We measure at least:

1. true-trigger recall;
2. false activations per hour;
3. distance and speaking volume;
4. different speakers and accents;
5. TV, music, fan and room noise;
6. model size and device memory;
7. end-to-end latency and power impact.

Recall and false activation rate must be considered together. Lowering a threshold can make a model easier to trigger while also increasing false activations.

From a phrase to a device test

For teams without a speech ML group, a practical validation workflow is:

1. define the exact fixed phrases;
2. generate a first model;
3. test it with real speakers before downloading;
4. run it on the target microphone and device;
5. collect misses and false activations;
6. add real-voice samples or hard negatives when needed.

Voicute is the online training and testing part of this workflow. The onnx-wakeword repository contains the open offline runtime and platform examples.

The important distinction is that online testing reduces the risk of downloading the wrong model; it does not replace validation on the final hardware.

When this architecture is a good fit

Use a small KWS pipeline when:

• the vocabulary is fixed;
• the device needs local or offline operation;
• the device is battery-powered or resource-constrained;
• low latency matters;
• the product does not want to send continuous audio to a cloud service.

Use ASR and intent parsing when:

• users can express the same intent in many ways;
• commands contain open values, numbers or entities;
• the product is closer to a conversational interface.

Disclosure: I’m the creator of Voicute. The model-generation service is commercial, while the offline inference runtime and integration examples are open source. The claims above should be validated on the reader’s own microphone, language and target device.