Build a Multi-Keyword Offline Voice Controller with ONNX and TFLite

Many voice interfaces do not need to transcribe arbitrary speech. A lamp, robot, appliance, kiosk, or media controller may only need a small vocabulary such as:

turn on the light
turn off the light
next
previous
stop

For this kind of product, a multi-keyword model can be a simpler fit than a full cloud ASR pipeline. The model answers a constrained question: which known command, if any, is present in the current audio window?

This article shows the engineering shape of that system, including model metadata, local inference, command callbacks, and the tests that matter before connecting voice input to a real action.

1. KWS is not general speech recognition

Keyword spotting (KWS) detects a known phrase or a small set of known phrases. Automatic speech recognition (ASR) produces a transcript for open-ended speech.

Product requirement	Suitable first tool
Start an interaction with one phrase	Single-keyword KWS
Recognize 2–10 fixed commands	Multi-keyword KWS
Understand arbitrary sentences	ASR plus intent parsing
Verify who is speaking	Speaker verification

A typical two-stage voice controller looks like this:

always-on wake word
        ↓
short command window
        ↓
multi-keyword model
        ↓
command label
        ↓
safe application action

The wake word and the command vocabulary are separate concerns. Keeping the command list fixed makes the application behavior easier to test and keeps the system local.

2. One model, several command labels

A multi-keyword model exposes a fixed output order. For example:

{
  "model_type": "multi_keyword",
  "keywords": ["turn on the light", "turn off the light", "next", "stop"],
  "model_file": "commands.onnx",
  "mel_time": 98,
  "n_mels": 32,
  "cons_frames": 2
}

The keywords array is part of the interface. Its order must stay aligned with the model output:

output[0] → turn on the light
output[1] → turn off the light
output[2] → next
output[3] → stop

The application should map a recognized label to an action instead of embedding business logic inside the model.

3. A minimal application callback

The exact API differs between Python, Android, Web, and embedded runtimes, but the application boundary can remain simple:

def on_keyword(word, probability, info):
    if word == "turn on the light":
        lights.set_power(True)
    elif word == "turn off the light":
        lights.set_power(False)
    elif word == "next":
        player.next()
    elif word == "stop":
        player.stop()

In a real product, add a cooldown and make repeated events safe. Voice detection should not directly bypass authorization or confirmation for destructive actions such as deleting data, unlocking a door, or purchasing something.

4. The local inference pipeline

The model file is only one part of a working detector:

microphone
   ↓
mono PCM audio
   ↓
sample-rate conversion and framing
   ↓
log-Mel feature extraction
   ↓
ONNX or TFLite inference
   ↓
threshold and consecutive-frame logic
   ↓
command callback

The runtime preprocessing must match the training and export configuration. A model can appear correct in one demo and lose recall if the target application changes the sample rate, frame shape, Mel settings, or input normalization.

The open-source onnx-wakeword repository contains the local inference examples. The same general model concept can be delivered as ONNX for desktop, Web, Android, and Python integrations, or as TFLite/TFLite Micro for constrained embedded targets.

5. ONNX and TFLite are deployment choices

Use the runtime that matches the target platform:

Target	Typical format or runtime	Important boundary
Web	ONNX Runtime Web	Browser microphone permission and compatibility still matter
Android	ONNX Runtime in the open demo	The current demo is not an Android TFLite SDK
Python / desktop	ONNX Runtime	Audio device and operating-system setup must be tested
ESP32-S3	INT8 TFLite Micro	The public hardware example is tied to its tested audio board

“The model is portable” does not mean that every audio device or board behaves identically. The final microphone, enclosure, operating system, and power budget need their own test.

6. Testing commands as a group

Testing only one successful utterance per command is not enough. For each command, collect:

• positive examples from multiple speakers;
• different distances, speeds, and volumes;
• similar-sounding phrases;
• ordinary conversation and silence;
• television, music, fans, and device playback;
• the microphone and enclosure intended for the product.

Measure at least:

recall = detected intended utterances / total intended utterances
false accepts per hour = unwanted activations / hours of negative audio

Also check confusion between commands. A model can have good recall for next while incorrectly confusing next with stop in the same environment.

7. When a second training round is useful

If the baseline model triggers on a recurring sound, keep the false-trigger clips as hard negatives for a later training round:

base model
   ↓
scan representative negative audio
   ↓
collect false-trigger clips
   ↓
retrain with hard negatives
   ↓
repeat held-out evaluation

Do not reuse every clip as training data. Keep a separate held-out set so that the second evaluation still measures generalization.

8. Generate before you integrate

The practical workflow for a fixed-command product is:

1. Define the wake word and command vocabulary.
2. Generate a Basic model for the target language.
3. Test the phrases with real voices before downloading.
4. Download the ONNX or TFLite artifact and its metadata.
5. Integrate the matching local runtime.
6. Measure recall, false accepts, latency, memory, and power on the target device.

Voicute supports custom keywords in Chinese, English, Spanish, Japanese, German, French, and Hindi. The online test is a screening step; it is not a production guarantee for every microphone or enclosure.

Conclusion

For a product with a small, known command vocabulary, one multi-keyword model can be cleaner than one detector per command or a full ASR service. The model recognizes the vocabulary; the application owns the action policy; the final device test decides whether the result is ready to ship.

The open inference examples are available in voicute/onnx-wakeword. You can generate and test your own fixed command vocabulary at Voicute.

Disclosure: I am the creator of Voicute. The model-generation service is commercial, while the local inference examples are open source. Results depend on the language, speakers, microphone, environment, and runtime configuration.