Whistle: Local Speech-to-Text in 16.9 MB, No Cloud Required

OpenAI's smallest Whisper model, tiny, has 39 million parameters and ships as a roughly 75 MB file in whisper.cpp's ggml format. Whistle's speech-to-text model is 16.9 MB. That size is small enough to bundle inside a desktop installer, a browser extension, or a Raspberry Pi image without anyone noticing the download.

The size matters because of what it enables. Every voice note, meeting recording, or support call you send to a cloud transcription API leaves your infrastructure. It also gets billed per minute. A model that fits in a few megabytes and runs on a CPU removes both problems. You still need to handle accuracy, latency, and integration yourself, and the sections below cover each.

Why a 16.9 MB Model Changes Your Architecture

Cloud speech-to-text forces a specific architecture. You capture audio, upload it, wait, then receive text. That flow brings network latency, retry logic, API key management, and a data processing agreement your legal team has to review.

A local model removes those pieces. Here are three cases where that changes the design:

• Field apps with no connectivity. Inspection tools, medical intake forms, and warehouse scanners often run where Wi-Fi is unreliable. A model bundled with the app transcribes offline and syncs only the text later.
• Regulated data. If you handle health records under HIPAA or personal data under GDPR, keeping raw audio on the device shrinks your compliance surface. You never transmit the most sensitive artifact.
• High-volume, low-value audio. Transcribing short voice commands at scale through a paid API adds up linearly with usage. A local model costs you only the CPU cycles.

Size also matters for distribution, not just inference. A 16.9 MB asset can ship inside a mobile app bundle or an Electron app. It fits in a Docker layer without bloating CI caches. Compare that with Whisper base at around 142 MB, or larger models that run into gigabytes.

Takeaway: List every place your product currently uploads audio. Mark each one as either "needs best-possible accuracy" or "needs privacy/offline/cost control." The second group is your candidate list for a local model.

Preparing Audio Correctly

Most bad local transcription results come from bad input rather than a bad model. Small speech models are typically trained on 16 kHz mono audio, and feeding them 48 kHz stereo from a browser's MediaRecorder degrades results or fails outright.

Normalize everything with ffmpeg before inference:

ffmpeg -i input.webm -ar 16000 -ac 1 -c:a pcm_s16le output.wav

The flags do the following:

• -ar 16000 resamples to 16 kHz.
• -ac 1 downmixes to mono.
• pcm_s16le writes uncompressed 16-bit little-endian PCM, which most inference code reads directly.

Check the input format Whistle's documentation specifies and match it exactly. If you're capturing live audio, do the resampling in the capture pipeline instead of writing temporary files. The Web Audio API's AudioContext accepts a sampleRate option. On the Python side, sounddevice lets you set samplerate=16000 at capture time.

Two more preprocessing steps pay off with small models:

1. Trim silence. Voice activity detection, such as the webrtcvad Python package, cuts dead air. Less audio means faster inference and fewer hallucinated words in quiet stretches.
2. Chunk long recordings. Split audio into segments of a few seconds to roughly 30 seconds at silence boundaries. Small models handle short, clean segments far better than hour-long files.

Takeaway: Add a single normalization function to your pipeline today that converts every input to 16 kHz mono PCM. Log the input format so you can catch mismatches early.

Wrapping the Model as a Local Service

Don't call the model directly from five places in your codebase. Wrap it once behind a small interface. That way you can swap Whistle for whisper.cpp or Vosk later without touching application code.

Here is a minimal Python pattern using FastAPI. Replace transcribe_file with the actual call from Whistle's README:

from fastapi import FastAPI, UploadFile
import tempfile, subprocess

app = FastAPI()

def transcribe_file(path: str) -> str:
    # Replace with Whistle's documented inference call
    raise NotImplementedError

@app.post("/transcribe")
async def transcribe(file: UploadFile):
    with tempfile.NamedTemporaryFile(suffix=".wav") as out:
        raw = await file.read()
        subprocess.run(
            ["ffmpeg", "-y", "-i", "pipe:0", "-ar", "16000",
             "-ac", "1", "-c:a", "pcm_s16le", out.name],
            input=raw, check=True, capture_output=True,
        )
        return {"text": transcribe_file(out.name)}

Run it with uvicorn app:app --host 127.0.0.1 --port 8000. Binding to 127.0.0.1 matters: the service never listens on a public interface, so audio never leaves the machine.

Load the model once at startup, not per request. With a small model, load time is short, but repeating it on every call still adds measurable latency under load.

Takeaway: Put the transcription behind a single function or local HTTP endpoint with a stable audio in, text out contract. Bind it to localhost.

Measuring Accuracy Before You Commit

Small models trade accuracy for size. That trade is usually fine for voice commands and rough notes, and often not fine for legal transcripts or names-heavy medical dictation. Don't guess which side you're on. Measure it.

The standard metric is word error rate (WER). The Python package jiwer computes it in one line:

from jiwer import wer
print(wer(reference_text, model_output))

Build an evaluation set from your own domain:

1. Collect 50 to 100 short clips that represent real usage, including accents, background noise, and domain jargon.
2. Hand-transcribe them as ground truth.
3. Run Whistle, whisper.cpp with ggml-tiny.bin, and your current cloud provider against the same clips.
4. Compare WER and per-clip latency on the hardware you'll actually deploy to.

This comparison tells you whether a model roughly a quarter the size of Whisper tiny is accurate enough for your use case. Public benchmarks run on audiobooks and read speech can't tell you that. Your users' audio can.

If accuracy falls short only in specific cases, use a hybrid approach. Run locally by default, and fall back to a larger model or a cloud API only when the user explicitly opts in or a confidence threshold fails.

Takeaway: Never ship a speech model without a WER number measured on your own audio. A spreadsheet with 50 clips beats any vendor benchmark.

Start today by recording ten real clips from your product's actual use case. Normalize them with the ffmpeg command above, then run them through Whistle and whisper.cpp's tiny model side by side. Within an hour you'll know whether local transcription is viable for your app, without sending a single byte of audio to anyone's server.