DEV Community

voicute
voicute

Posted on

Custom Wake Word on ESP32-S3 with INT8 TFLite Micro

#hardware #iot #machinelearning

How to Run a Custom Wake Word on ESP32-S3 with INT8 TFLite Micro

A custom wake word is easy to demonstrate in a browser and much harder to ship on a real device.

The microphone, enclosure, gain, background noise, memory budget and audio pipeline all affect the result. For an ESP32-S3 product, I prefer to treat the model and the embedded audio pipeline as one system.

This post shows the practical path:

  1. Train or generate a custom wake-word model.
  2. Export a compact INT8 TFLite model.
  3. Run it locally with TFLite Micro.
  4. Test it on the final microphone and enclosure.

Why ESP32-S3?

The ESP32-S3 is a useful starting point for embedded keyword spotting because it has more compute and memory headroom than smaller ESP32 variants. Boards differ substantially, though. A model that works on an ESP32-S3 development kit is not automatically validated on an ESP32-C3 or a custom board.

For a microcontroller target, the model format is only one part of the problem. The firmware also needs to reproduce:

  • 16 kHz mono PCM input
  • The same window and frame configuration
  • The same feature extraction
  • The same quantization assumptions
  • The same model output mapping

Changing one of these can reduce recall even when the model file itself is unchanged.

The deployment path

The public ESP32 example is available here:

https://github.com/voicute/onnx-wakeword/tree/main/esp32

A basic ESP-IDF workflow looks like this:

git clone https://github.com/voicute/onnx-wakeword.git
cd onnx-wakeword/esp32/examples/esp32s3_hmi_devkit

idf.py set-target esp32s3
idf.py build
idf.py -p COM6 flash monitor
Enter fullscreen mode Exit fullscreen mode

The model and output head are a matched pair. Replace them together. Do not copy a model from one training/export configuration and keep the head from another configuration.

Training a custom phrase

There are two reasonable approaches:

  • Use an online training workflow when the goal is fast product validation.
  • Build a self-hosted training pipeline when you need complete control over data, augmentation and hard negatives.

For a small product team, online training is usually the faster first step. The important part is not just whether the model detects the phrase in a quiet demo. It must be tested with:

  • The final microphone
  • Different speaking distances
  • Different speakers
  • TV, music and speech in the background
  • Similar-sounding phrases
  • Device vibration and enclosure noise

Measuring the real result

I would record at least:

  • Recall on positive utterances
  • False triggers per hour
  • Detection latency
  • Flash and RAM usage
  • Behavior after repeated commands

A browser score is useful for iteration, but it is not an embedded benchmark.

When should you use ONNX instead?

ONNX is a good fit for Linux, Android, Python or browser runtimes with a full ONNX Runtime implementation. For direct inference inside constrained ESP32 firmware, INT8 TFLite Micro is usually the more natural deployment target.

The same trained keyword concept can therefore have different exports for different devices. Keep the evaluation set and acceptance criteria consistent across platforms.

Final takeaway

The shortest path to a custom ESP32 wake word is not “find a model file and flash it.” It is:

phrase
  -> training
  -> target-device audio test
  -> matched INT8 export
  -> ESP-IDF integration
  -> false-trigger evaluation
Enter fullscreen mode Exit fullscreen mode

If you want to try the open-source inference component and examples, start with onnx-wakeword. For online custom model training and testing, see Voicute.

Top comments (0)