DEV Community

voicute
voicute

Posted on

Multilingual Command Words on ESP32: A TFLite Alternative to MultiNet

#hardware #iot #machinelearning

ESP32 voice control is often discussed as if the hardware determines the language. In practice, the bigger limitation is usually the available acoustic model and its training pipeline.

An embedded command-word system needs three separate pieces:

  1. A wake-up or start-of-interaction detector.
  2. A small command-word model.
  3. Firmware logic that maps the model output to device actions.

A TFLite Micro workflow makes it possible to keep the runtime on the ESP32 while changing the language model.

The supported language set

The current Voicute workflow supports custom keyword models in:

  • Chinese
  • English
  • Japanese
  • French
  • German
  • Spanish
  • Hindi

The important detail is that the language is in the trained model, not in the command list alone. Translating turn on the light into another language does not create an acoustic model for that language.

One model, multiple keywords

For a selected language, a Multi-Keyword model can recognize several fixed commands in one inference pass.

For example:

1 -> lights on
2 -> lights off
3 -> brighter
4 -> darker
Enter fullscreen mode Exit fullscreen mode

The runtime returns a command ID. The ESP32 application decides what that ID means.

This gives a MultiNet-like command recognition shape without requiring the product to use the same model family or language coverage.

Why this matters on ESP32

A full ASR system is often too large or power-hungry for a small always-on product. A fixed command model can stay compact and offline:

  • No network connection
  • No cloud speech API
  • Lower latency
  • Predictable command vocabulary
  • Easier privacy story

The trade-off is that the device only recognizes the commands included in the model. This is a feature for appliances and embedded controls, not a limitation to hide.

The embedded audio contract

The model and firmware must agree on:

  • Sample rate
  • Channel count
  • Frame size
  • Feature extraction
  • Quantization
  • Output labels
  • Threshold and smoothing behavior

The common embedded input is 16 kHz, 16-bit, mono PCM. If the board supplies a different format, convert it before inference and test the conversion path itself.

A practical architecture 

WakeNet or another wake detector
        |
        v
16 kHz mono audio
        |
        v
Multi-keyword INT8 TFLite model
        |
        v
command ID
        |
        v
ESP-IDF application action
Enter fullscreen mode Exit fullscreen mode

If the language or command set changes, retrain/export the model. The firmware-side command-ID interface can remain stable.

What to test

For each language, test more than clean studio audio:

  • Different speakers and accents
  • Quiet and noisy rooms
  • Music and television
  • Similar command phrases
  • Short and long distances
  • The final microphone and enclosure

Recall alone is not enough. Track false triggers per hour and confusion between commands.

Final takeaway

The useful comparison is not “which framework has the most preset words?” It is:

  • Can I train the language I need?
  • Can one small model recognize my fixed command set?
  • Can I export it to the runtime on my target board?
  • Can I test it before shipping?

For an open offline implementation, see voicute/onnx-wakeword. For online custom model generation and testing, see Voicute.

Top comments (0)