Add Simple Offline Voice Commands with One Multi-Keyword Model

You do not always need a full speech-to-text service to add voice control to a project.

If your app, robot, web tool, or small device only needs commands such as:

turn on the light
turn off the light
next
previous
stop

then a small keyword-spotting model can be a practical starting point. It listens for a known vocabulary, returns a command label, and lets your application decide what to do next.

This article shows the complete shape of that workflow: define a command list, generate one multi-keyword model, run it locally, and map the result to safe application actions.

KWS or ASR?

Keyword spotting (KWS) and automatic speech recognition (ASR) solve different problems.

What your project needs	A sensible first choice
Detect one wake phrase	Single-keyword KWS
Recognize 2–10 known commands	Multi-keyword KWS
Transcribe arbitrary speech	ASR
Understand open-ended intent	ASR plus intent parsing
Verify who is speaking	Speaker verification

KWS does not try to transcribe everything the user says. It answers a smaller question: “Is one of my known phrases present in this audio window?” That narrower problem is useful when you want local processing, a small vocabulary, and predictable callbacks.

The architecture

A common voice-control flow has two stages:

always-on wake word
        ↓
short command window
        ↓
multi-keyword model
        ↓
command label
        ↓
application action

The wake word and the command list are separate. The wake word starts the interaction; the multi-keyword model identifies the command. For a very small prototype, you can also run the command model directly and add a wake-word gate later.

One model, several command labels

A multi-keyword model contains a fixed list of phrases. The application uses the same list to interpret the model output:

{
  "model_type": "multi_keyword",
  "keywords": ["turn on the light", "turn off the light", "next", "stop"],
  "model_file": "commands.onnx",
  "mel_time": 98,
  "n_mels": 32,
  "cons_frames": 2
}

The order matters:

output[0] → turn on the light
output[1] → turn off the light
output[2] → next
output[3] → stop

Treat the metadata as part of the model interface. Do not change the command order in the app without also changing the model metadata.

Map a label to an application action

The model should return a stable event. Business logic should remain in the application layer:

def on_keyword(word, probability, info):
    if word == "turn on the light":
        lights.set_power(True)
    elif word == "turn off the light":
        lights.set_power(False)
    elif word == "next":
        player.next()
    elif word == "stop":
        player.stop()

In a real app, also add:

• a confidence threshold;
• consecutive-frame confirmation;
• a cooldown so one utterance does not trigger several callbacks;
• a clear “unknown” or “no command” path;
• authorization or confirmation for sensitive actions.

A voice detector should not directly unlock a door, delete data, make a purchase, or perform another destructive operation based on one uncertain frame.

What the local pipeline actually contains

The model file is only one part of the system:

microphone
   ↓
mono PCM audio
   ↓
sample-rate conversion and framing
   ↓
log-Mel feature extraction
   ↓
ONNX or TFLite inference
   ↓
threshold and temporal filtering
   ↓
command callback

The runtime preprocessing must match the model configuration. Sample rate, frame size, Mel settings, tensor shape, normalization, threshold, and cooldown all affect the result. Copying only the ONNX file into an unrelated audio pipeline is not enough.

The open-source onnx-wakeword repository includes local inference examples and demo projects. Use the runtime that matches your target:

Target	Typical path	Boundary to check
Python / desktop	ONNX Runtime	Audio device and OS setup
Web	ONNX Runtime Web	Microphone permission and browser support
Android	ONNX Runtime in the demo	App lifecycle, permissions, and battery
Raspberry Pi	ONNX Runtime or TFLite	OS architecture and audio device
ESP32-S3	INT8 TFLite Micro	Tested board and audio driver

ONNX and TFLite are deployment formats, not guarantees that every board or microphone behaves identically.

Generate a model without building a training pipeline

For a small personal project, the shortest workflow is:

1. Write down the exact command phrases.
2. Keep the list small and make commands easy to distinguish.
3. Generate a Basic multi-keyword model.
4. Test the phrases with real voices before downloading.
5. Download the ONNX or TFLite artifact and its metadata.
6. Connect the model to your application callback.
7. Measure it on the final microphone and in the final environment.

Voicute lets you enter custom keywords, generate a model, and test it before you integrate it. It supports Chinese, English, Spanish, Japanese, German, French, and Hindi. The online test is a screening step, not a production guarantee for every speaker, enclosure, or noise condition.

Test commands as a group

One successful recording per command is not enough. Test each phrase with:

• several speakers and accents;
• different distances, speeds, and volumes;
• similar-sounding phrases;
• silence and ordinary conversation;
• fans, television, music, and device playback;
• the microphone and enclosure intended for the product.

Track at least these two measurements:

recall = detected intended utterances / total intended utterances
false accepts per hour = unwanted activations / hours of negative audio

Also check command confusion. A model can detect next reliably while occasionally returning stop for a similar phrase.

If a recurring background sound causes false activations, keep those clips as hard negatives for a later training round. Preserve a separate held-out set so that the second evaluation still measures generalization.

When multi-keyword is a good fit

Use one multi-keyword model when the vocabulary is known in advance and several commands share the same audio pipeline. It can be easier to package and maintain than one independent model per command.

Use ASR instead when users can say arbitrary sentences, numbers, names, or changing parameters. For example, “turn the light on” is a fixed command candidate; “set the bedroom light to 37 percent” usually requires ASR and intent parsing unless you deliberately reduce it to fixed options.

Conclusion

For a small voice-controlled project, the practical path is often:

fixed command list
→ multi-keyword model
→ local inference
→ stable label
→ application callback

The model recognizes the vocabulary. The application owns the action policy. The final device test decides whether the result is ready to ship.

Start with the open-source onnx-wakeword examples, then generate and test your own multi-keyword model with Voicute.

Disclosure: I am the creator of Voicute. The model-generation service is commercial, while the local inference examples are open source. Results depend on the language, speakers, microphone, environment, and runtime configuration.