On 29 October 2026 Hugging Face (via Mattt) released two open Swift packages aimed at Apple platforms: AnySpeechRecognizer and AnyDecisionModel. The first gives you a SpeechAnalyzer-shaped API so you can keep the same capture and session code while swapping the underlying recognizer. The second returns typed yes/no, choice or score results with probabilities instead of forcing you to parse free-form generation.
What AnySpeechRecognizer actually ships
It implements the shape of Apple’s newer SpeechAnalyzer API. Default backend is Apple’s own Speech framework. You can also point it at Core ML conversions of NVIDIA Parakeet (derived from Parakeet + Moondream work) or Alibaba’s Qwen3-ASR 0.6B / 1.7B.
Parakeet v3 covers roughly 25 European languages. It supports both live streaming and file transcription, returns word-level timing and confidence, and can do speaker diarization for up to four speakers when paired with the right backend. That is immediately useful for meeting or interview transcripts where you need “who said what” without a separate diarization stage.
Qwen3-ASR adds more languages (including Mandarin, Cantonese, Japanese, Korean and 22 Chinese dialects) but the current integration returns finalized segments of up to ~40 seconds. It does not expose the same timing/confidence surface and is described as slower, so better suited to offline file jobs than live calls.
Model sizes are practical for on-device use: Parakeet around 600 MB plus a ~590 MB live encoder; Qwen3-ASR in the 0.9–2.4 GB range depending on variant. Assets land in the normal Hugging Face cache so they can be shared with other tools outside the sandbox.
A one-line mental model for the migration:
// before
import Speech
// after
import AnySpeechRecognizer
// then pick the engine (Apple / Parakeet / Qwen3-ASR) at session creation
You still own VAD, resampling, turn detection and the decision of when a partial becomes final. The package just gives you a consistent recognizer surface.
Typed decisions instead of “generate then parse”
AnyDecisionModel is the complementary piece. Instead of prompting a model to emit “billing” or “yes” and then regex-ing the output, you ask a structured question and get a typed result plus the probability mass over the allowed answers.
Supported shapes:
- Yes/no → probability
- Choice among options (can map to a Swift enum) → chosen case + per-option probabilities + confidence
- Ordinal score → expected level + distribution
For the MLX next-token backend the package builds a prompt that includes one-token labels for every allowed answer and reads the probabilities from a single forward pass. That is far cheaper than full generation when you only need a classification. Other backends include MLX models with a trained decision head (e.g. Cloudflare Clef), Core AI, llama.cpp GGUF, TypeSafe Jev / System One, and Ollama (local or cloud).
Raw LM probabilities are usually overconfident, so the local backends accept a Calibration (temperature + bias) that you can fit on your own data. Results are batched: a hosted backend can take hundreds of questions in one request; local models process groups of 16 while sharing session state.
Why the pattern matters for real-time call tools
Most production STT still leaves the machine. That adds 50–200 ms of network RTT on top of your WASAPI / Core Audio buffering, forces you to send raw PCM off-box, and fails when the network is degraded. A pluggable local first-pass changes the design:
- Capture → VAD → short chunks (1–4 s) → local recognizer (Parakeet-style for European languages, or whatever fits the domain).
- Optional diarization so you know which speaker is which.
- Structured decision model for “is this a question?”, “which skill is being tested?”, “confidence high enough to act?”, etc., without a second full LLM call.
- Escalate only the low-confidence or domain-specific turns to a larger cloud model.
On Windows the same architecture is already reachable through Ollama’s recent LFM3-audio / LFM3-omni support or NVIDIA’s Nemotron transcription models. You keep the capture, VAD and overlay logic; you just swap the HTTP endpoint or the local runtime. The Swift packages show that the “shared API + multiple engines” idea is shipping on Apple silicon today; the equivalent Windows surface is the OpenAI-compatible /v1/audio/transcriptions plus a small decision head or a second small model for classification.
Latency budget example for a live interview assist:
- WASAPI exclusive or event-driven capture: 10–20 ms buffer
- VAD: 20–40 ms
- Local 0.6–2B ASR on a modern GPU or NPU: 50–150 ms for a 2 s chunk
- Decision model single forward pass: <30 ms
- Total under 250 ms is realistic before you even hit the network
Tradeoffs are the usual ones: larger models give better WER and more languages but raise memory and first-token latency; live streaming models expose partials and timestamps while file-oriented ones do not; diarization quality drops above four speakers or with heavy overlap. Test on your actual hardware and audio (background noise, accents, code-switching) — the published numbers are machine-specific.
Practical next steps
If you are on Apple platforms:
# packages are on GitHub; models pull from the Hub
# see mattt/AnySpeechRecognizer and mattt/AnyDecisionModel
On Windows the equivalent experiment is:
ollama pull lfm3-audio
# or the Nemotron transcription checkpoint of your choice
# then stream short PCM chunks to the local OpenAI-compatible endpoint
Keep the recognizer behind a narrow interface (transcribe(chunk) → Result{text, words, speakers, confidence}) so you can A/B Apple Speech vs Parakeet vs a local GGUF without touching the rest of the pipeline. The same interface works for the decision step: decide(state, question) → TypedAnswer.
That is the useful part of the 29 October drop: not another giant model, but a clean separation between “how I capture and process audio” and “which engine turns it into text or a decision.”
I'm building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project's code and logs.
Top comments (0)