Building a fully offline voice assistant on Raspberry Pi 4

What we are building

A voice assistant that runs entirely on a Raspberry Pi 4. No cloud, no API keys, no per-minute pricing. The user says "Hey Assistant", the device wakes up, and then handles whatever was said next.

Two interaction paths:

• The user says one of 14 fixed commands (turn on lights, play music, stop, and so on). The device recognises it instantly and runs the action.
• The user says something outside that set (read me the news, what time does the store open). The device transcribes the speech into text and processes the open-ended query.

Three models work together. A wake-word detector listens to the microphone 24/7. It is cheap and always on. A keyword spotter takes over for the next couple of seconds and tries to match a fixed vocabulary. And a streaming ASR transcribes open-ended speech if the keyword spotter finds no match. It is heavier but only runs when actually needed.

All three install with three pip install calls and run on a $35 Raspberry Pi 4 without an internet connection. This article walks through the full Python code, shows equivalent snippets in Node.js, Go and C, and explains why splitting this into three SDKs on one runtime keeps the always-on cost low without giving up open-ended speech.

Why the triage pattern (WW then KWS then ASR)

The core idea is to avoid running the heaviest model until it is actually needed.

The wake-word detector is the lightest piece of the pipeline. Our model is about 100 KB and uses about 5.3% of one A53 core on a $15 Raspberry Pi Zero 2 W (24/7 sustained, per our earlier benchmarks, link below). On a Pi 4 the cost drops further because the Cortex-A72 cores are faster and run in 64-bit mode. That means the wake-word can listen to the environment continuously without stealing meaningful CPU from anything else on the device.

But the wake-word only knows one thing. Did the trigger phrase ("Hey Assistant" in our case) get said. To understand what the user actually wants, you need another model.

The obvious option is to fire up ASR as soon as the wake-word triggers. That works but it is overkill for most real requests. In a voice assistant, users usually say one of a small set of commands. Turn on lights. Set a timer. Play music. Running full open-vocabulary ASR on that is like using a regex when strcmp would do.

KWS (keyword spotter) solves this cheaply. Our model processes 16 kHz mono audio in 32 ms frames and classifies each into one of 14 fixed command classes. It does not transcribe arbitrary speech, but it does quickly answer "yes, this is command X" or "no match".

The triage logic looks like this.

1. WW listening state. Wake-word runs continuously, waiting for "Hey Assistant".
2. WW triggered. Switch to KWS. Give the user about 2 seconds to say a command.
3. KWS matched. Run the command action (turn on lights, stop, and so on). Go back to state 1.
4. KWS did not match. Fall back to ASR. Transcribe the open-ended speech until silence or a timeout.
5. ASR finished. Process the text (route to a local LLM, a parser, or send it to your backend). Go back to state 1.

In compute terms this is a switch with branches for known cases and a default for everything else. WW costs almost nothing all the time. KWS runs only after a wake-word (typically 1 to 2 seconds). ASR runs only when the command missed the KWS vocabulary. In practice, most assistant requests fall inside a small command vocabulary, though the exact fraction depends heavily on your product.

Hardware

Minimum setup:

• Raspberry Pi 4 (4GB). Four gigabytes of RAM give comfortable headroom for the ASR model, which keeps its weights in memory. The 2GB variant will boot the whole pipeline but leaves little room for anything else running on the device. Pi 5 and Pi 3B+ also work, but we tested on Pi 4 since that is the most common Pi model developers reach for right now.
• A USB microphone ($10 to $15, any class-compliant device) or an I2S mic HAT such as the ReSpeaker 2-Mics Pi HAT (about $10) for cleaner audio.
• A 32GB microSD card with 64-bit Raspberry Pi OS (aarch64). The 64-bit build is required. Our Linux SDKs ship aarch64 wheels only, and the 32-bit armhf variant is not supported.
• Optional: a speaker or headphones if you want to add a TTS response. TTS is in development. This tutorial does not cover it.

Wake-word alone (without the rest of the pipeline) works even on a Pi Zero 2 W. We tested that in an earlier article, wake-word on a $15 Pi Zero 2 W. Adding KWS and ASR bumps the compute budget, so this tutorial uses a Pi 4.

Installing the three SDKs

Each model ships as a separate package.

pip install voxrt-wake-word voxrt-kws voxrt-asr

The three libraries share the same VoxRT runtime under the hood. That runtime is a Rust inference engine with a C ABI. We do not ship a single voxrt-all package, because many integrators only need part of the pipeline (wake-word only for a kiosk device, ASR only for a dictation app) and carrying weights they never load makes no sense.

Model weights download separately as files.

# Wake-word "Hey Assistant"
curl -LO https://github.com/VoxRT/voxrt-wake-word-models/releases/download/v0.1.0/voxrt_wake_word.vxrt

# Keyword spotter (14 commands)
curl -LO https://github.com/VoxRT/voxrt-kws-models/releases/download/v0.1.0/voxrt_kws.vxrt

# Streaming ASR (medium PC model)
curl -LO https://github.com/VoxRT/voxrt-asr-models/releases/download/v0.1.2/streaming_medium_pc.vxrt

(Check the exact versions in each repo README. They update periodically.)

The Node.js, Go and C install options are covered in the "Same pipeline in other languages" section below.

Code: the triage pipeline in Python

The full example breaks into three parts.

1. Microphone capture. Shared across all three SDKs.
2. State machine. Controls which engine gets audio at any given moment.
3. Wiring. The event loop that puts it all together.

Audio capture

We use sounddevice (a PortAudio wrapper), which is the standard way to read a microphone in Python on Linux. It provides a callback-based API. The PortAudio audio thread calls our function with each new chunk of samples. We push them into a queue and the main thread drains it.

import sounddevice as sd
import numpy as np
from queue import Queue

SAMPLE_RATE = 16000   # all three models expect 16 kHz mono
CHUNK_FRAMES = 512    # 32 ms, matches WW and KWS internal chunk size

audio_queue: "Queue[np.ndarray]" = Queue(maxsize=64)

def on_audio(indata, frames, time_info, status):
    if status:
        print(f"audio status: {status}")
    # indata is (frames, 1) int16. Copy the buffer since sounddevice
    # reuses the same backing storage between callbacks.
    audio_queue.put(indata.copy())

stream = sd.InputStream(
    samplerate=SAMPLE_RATE,
    channels=1,
    dtype="int16",
    blocksize=CHUNK_FRAMES,
    callback=on_audio,
)
stream.start()

After this call, 32 ms blocks of int16 samples land in audio_queue steadily. The next question is which engine handles them at any given moment.

State machine

We keep one state enum and a main loop that pulls chunks off the queue and routes them into the right engine.

from enum import Enum

class State(Enum):
    IDLE = 1            # only WW is listening
    KWS_CHECK = 2       # WW fired, KWS is checking for 2 seconds
    ASR_TRANSCRIBE = 3  # KWS missed, ASR is transcribing

state = State.IDLE

Initialise the three engines

from voxrt_wake_word import WakeWordEngine
from voxrt_kws import KwsEngine
from voxrt_asr import AsrStreamingEngine, DecodeMode

ww = WakeWordEngine.from_path("voxrt_wake_word.vxrt")
ww.threshold = 0.9
ww.cooldown_frames = 100  # ignore 100 frames (~3.2 s) after a detection

kws = KwsEngine.from_path(
    "voxrt_kws.vxrt",
    threshold=0.9,
    consecutive_frames_required=3,
    cooldown_frames=25,
)
print(f"KWS commands: {kws.class_names}")

asr = AsrStreamingEngine.from_path(
    "streaming_medium_pc.vxrt",
    mode=DecodeMode.RNNT,
)

The main event loop

This is where the triage happens. Broken down by state:

import time
from queue import Empty

KWS_WINDOW_SEC = 2.0    # window after WW for a KWS attempt
ASR_SILENCE_SEC = 1.5   # silence at the end of an ASR utterance


def handle_command(name):
    print(f"running fixed command: {name}")
    # wire this into GPIO, home automation API, whatever you need


def handle_open_query(text):
    print(f"open-ended query: {text}")
    # send this to a local LLM, a parser, wherever


kws_check_deadline = 0.0
asr_buffer_f32 = []
asr_last_speech_time = 0.0

try:
    while stream.active:
        try:
            chunk_i16 = audio_queue.get(timeout=0.1)
        except Empty:
            continue

        samples_i16 = chunk_i16.flatten().tolist()

        if state == State.IDLE:
            # WW listens all the time.
            for det in ww.push_pcm_i16(samples_i16):
                print(f"[WW] wake detected, score={det.score:.3f}")
                state = State.KWS_CHECK
                kws_check_deadline = time.time() + KWS_WINDOW_SEC

        elif state == State.KWS_CHECK:
            # Try to match a fixed command in the next 2 seconds.
            matched = False
            for det in kws.push_pcm_i16(samples_i16):
                print(f"[KWS] command: {det.class_name} score={det.score:.3f}")
                handle_command(det.class_name)
                state = State.IDLE
                matched = True
                break

            if not matched and time.time() > kws_check_deadline:
                print("[KWS] no match, falling back to ASR")
                state = State.ASR_TRANSCRIBE
                asr_buffer_f32 = []
                asr_last_speech_time = time.time()

        elif state == State.ASR_TRANSCRIBE:
            # ASR expects f32 samples. Convert from int16.
            f32 = [s / 32768.0 for s in samples_i16]
            asr_buffer_f32.extend(f32)

            # ASR wants 200 ms chunks (3200 samples).
            while len(asr_buffer_f32) >= 3200:
                chunk_200ms = asr_buffer_f32[:3200]
                asr_buffer_f32 = asr_buffer_f32[3200:]
                partial = asr.push_audio(chunk_200ms)
                if partial:
                    print(f"[ASR partial] {partial}")
                    asr_last_speech_time = time.time()

            # If silence lasts more than 1.5 seconds, finalise.
            if time.time() - asr_last_speech_time > ASR_SILENCE_SEC:
                final = asr.stop()
                print(f"[ASR final] {final}")
                handle_open_query(final)
                # asr.reset() keeps the loaded weights and resets the streaming
                # state. Cheaper than re-loading ~150 MB via from_path().
                # If your voxrt-asr version does not expose reset(), fall back
                # to AsrStreamingEngine.from_path(...) here, but expect a
                # multi-second stall while weights re-load.
                asr.reset()
                state = State.IDLE
finally:
    stream.stop()
    stream.close()

One key detail. WW and KWS want int16 samples. ASR wants float32 normalised to [-1.0, 1.0]. The conversion is one line (s / 32768.0) but it is easy to forget. Chunk sizes also differ. 32 ms for WW and KWS. 200 ms for ASR (matching the ASR model's internal attention window).

This is not "one API for everything". It is "consistent integration pattern (from_path then push_something then read the result), with input format matched to each model's requirements". Wake-word and KWS have the same shape internally (both are event detectors on short chunks), so they share a signature. ASR works differently (it is a streaming decoder), so it has a different signature.

Same pipeline in other languages

Full quickstart examples for Node.js, Go and C are in the repos:

• github.com/VoxRT/voxrt-wake-word-linux/tree/main/examples/{nodejs,go,c}/
• github.com/VoxRT/voxrt-kws-linux/tree/main/examples/{nodejs,go,c}/
• github.com/VoxRT/voxrt-asr-linux/tree/main/examples/{nodejs,go,c}/

Below are the critical snippets. How push_pcm_i16 and push_audio look in each language. Follow the repo links for the full runnable examples.

Node.js

const { WakeWordEngine } = require("@voxrt/wake-word");
const { KwsEngine } = require("@voxrt/kws");
const { AsrStreamingEngine, DecodeMode } = require("@voxrt/asr");

const ww = WakeWordEngine.fromPath("voxrt_wake_word.vxrt");
ww.threshold = 0.9;

// mic capture via any ALSA/PortAudio Node binding available on Pi.
mic.on("data", (buffer) => {
  // Buffer is a slice of a larger ArrayBuffer. Use byteOffset and
  // byteLength to build a view over just the samples we received.
  const samples = new Int16Array(
    buffer.buffer,
    buffer.byteOffset,
    buffer.length / 2,
  );
  for (const det of ww.pushPcmI16(samples)) {
    console.log(`[WW] score=${det.score}`);
    // switch state to KWS_CHECK here
  }
});

Go

import (
    "log"

    wakeword "github.com/VoxRT/voxrt-wake-word-linux/go"
    // kws     "github.com/VoxRT/voxrt-kws-linux/go"
    // asr     "github.com/VoxRT/voxrt-asr-linux/go"
)

engine, err := wakeword.OpenFromPath("voxrt_wake_word.vxrt")
if err != nil {
    log.Fatal(err)
}
defer engine.Close()

_ = engine.SetThreshold(0.9)
_ = engine.SetCooldownFrames(100)

// mic capture via any ALSA/PortAudio Go binding available on Pi.
for chunk := range micChan { // chunk is []int16 of length 512
    for _, d := range engine.PushPcmI16(chunk) {
        log.Printf("[WW] score=%.3f", d.Score)
        // switch state here
    }
}

C

#include <stdint.h>
#include <stddef.h>
#include <stdio.h>

#include <voxrt_wake_word.h>

// load_vxrt_model() mmaps the model file and returns (bytes, len).
// See examples/c/alsa-mic-quickstart/main.c in the repo for the exact
// implementation, about 20 lines around open() + fstat() + mmap().
// alsa_read_chunk() is a thin wrapper over snd_pcm_readi().

const uint8_t *model_bytes = NULL;
size_t model_len = 0;
if (load_vxrt_model("voxrt_wake_word.vxrt", &model_bytes, &model_len) != 0) {
    fprintf(stderr, "failed to mmap the wake-word model file\n");
    return 1;
}

voxrt_wake_word_t *engine = NULL;
voxrt_status_t rc = voxrt_wake_word_create(model_bytes, model_len, &engine);
if (rc != VOXRT_OK) {
    fprintf(stderr, "voxrt_wake_word_create failed: %d\n", (int)rc);
    return 2;
}
voxrt_wake_word_set_threshold(engine, 0.9f);
voxrt_wake_word_set_cooldown_frames(engine, 100);

// ALSA capture, 16 kHz mono int16, 512 samples per chunk.
int16_t chunk[512];
voxrt_wake_word_detection_t dets[8];
while (alsa_read_chunk(pcm_handle, chunk, 512) == 0) {
    size_t written = 0;
    rc = voxrt_wake_word_push_pcm_i16(engine, chunk, 512, dets, 8, &written);
    // VOXRT_ERR_BUFFER_TOO_SMALL just means more detections would fit;
    // the ones we got are still valid.
    if (rc != VOXRT_OK && rc != VOXRT_ERR_BUFFER_TOO_SMALL) {
        fprintf(stderr, "push failed: %d\n", (int)rc);
        break;
    }
    for (size_t i = 0; i < written; i++) {
        printf("[WW] score=%.3f\n", dets[i].score);
        // switch state here
    }
}

voxrt_wake_word_destroy(engine);
// remember to munmap(model_bytes, model_len)

Each SDK's quickstart file already does this work. It opens the mic, feeds the engine, and prints detections. Our triage loop is the state machine on top of that.

Expected performance on Pi 4

Important disclaimer. The numbers below extrapolate from benchmarks on other hardware. We have not run the full triage loop on our Pi 4 yet. Your numbers should land in this range but you will only know exactly after you measure on your own board.

• IDLE state (WW only). About 1 to 2% of one CPU core. Extrapolated from 5.3% on Pi Zero 2 W (A53 at 1 GHz running the default 32-bit Raspberry Pi OS). Pi 4 uses a 64-bit A72 at 1.5 GHz with better IPC and more efficient NEON when the OS runs in aarch64 mode.
• KWS_CHECK state (2 seconds after a wake). About 15 to 25% CPU during the burst. Extrapolated from 16% RTF on Snapdragon 662, but Pi 4's A72 is a generation older than the Kryo 260 Gold cores in SD662, so expect somewhat higher RTF. KWS only runs in short windows.
• ASR_TRANSCRIBE state. Roughly 40 to 70% CPU during active transcription. ASR is the heaviest of the three, and NEON-specialised int8 kernels tuned for mobile Snapdragons will not extract the same efficiency on A72. Real-time streaming (RTF under 1.0) should still be achievable on Pi 4 aarch64, but the headroom is the tightest of the three. Measure on your device before assuming.

RAM footprint with all three engines initialised: around 200 to 300 MB. ASR dominates with its roughly 150 MB of weights. That is why we recommend Pi 4 with 4 GB. The 2 GB variant also works but leaves little room for the rest of your software.

If we run real Pi 4 numbers after publish, we will do a follow-up article with actual measurements.

What this article does not cover

• TTS (speaking back). In development. A separate on-device voice-cloning model. Will get its own article when it ships.
• Custom wake-word phrase. We currently only ship "Hey Assistant" as a pre-trained model. Training your own trigger phrase requires a separate training pipeline that we have not released publicly yet.
• Speaker verification / voice biometrics. Not included. The models do not tell you who is speaking.
• Multi-language support. Wake-word and KWS are English-only at the moment. The ASR model is multilingual in general, but we have only tested it on English.

Everything above is on the VoxRT roadmap. If any of it is critical for your use case, mention it in the comments so we can prioritise.

Repos and resources

• Wake-word Linux SDK: github.com/VoxRT/voxrt-wake-word-linux
• KWS Linux SDK: github.com/VoxRT/voxrt-kws-linux
• ASR Linux SDK: github.com/VoxRT/voxrt-asr-linux
• All SDKs: github.com/VoxRT
• Our earlier on-device voice articles: dev.to/voxrtio

If you are building anything voice-driven and want the audio to stay on the device, all three SDKs are on GitHub with the documentation in the README. Full quickstart examples in Python, Node.js, Go and C are in every repo.