How to Run Whisper Locally for Real-Time Speech-to-Text (faster-whisper)

Run Whisper locally for real-time speech-to-text

For live translation and voice apps, you want speech-to-text that's fast and free per use — which
means running Whisper locally with faster-whisper. It pairs perfectly with APIVAI for the
translation step: Whisper turns audio into text locally, APIVAI (GPT-5.5) translates it cheaply over
an OpenAI-compatible call, and you keep latency low and per-second cost at zero for the audio part.

This guide gets local Whisper running for real-time use.

Install

pip install faster-whisper

faster-whisper is a fast reimplementation of OpenAI's Whisper using CTranslate2. It runs on GPU
(CUDA) or CPU.

Choose a model size

Model	Speed	Accuracy	Use
tiny / base	fastest	lower	quick captions, weak hardware
small	fast	good	recommended for live
medium	slower	better	accuracy over speed
large-v3	slowest	best	offline/high-accuracy jobs

For live captioning, small on a GPU is the sweet spot.

Transcribe in near real time

from faster_whisper import WhisperModel

model = WhisperModel("small", device="cuda", compute_type="float16")  # device="cpu", compute_type="int8" for CPU

def transcribe(audio_path: str, lang: str = "zh") -> str:
    segments, _ = model.transcribe(audio_path, language=lang, vad_filter=True, beam_size=1)
    return "".join(s.text for s in segments).strip()

• Capture short audio chunks (1–3s) from the mic/OBS monitor and feed them in.
• vad_filter=True skips silence so you don't transcribe noise.
• beam_size=1 favors speed for live use.

Hook it to translation (APIVAI)

from openai import OpenAI
ai = OpenAI(api_key="YOUR_APIVAI_API_KEY", base_url="https://api.apivai.com/v1")

def translate(text, target="Spanish"):
    r = ai.chat.completions.create(model="gpt-5.5", messages=[
        {"role": "system", "content": f"Translate to natural {target}. Output only the translation."},
        {"role": "user", "content": text}])
    return r.choices[0].message.content.strip()

print(translate(transcribe("chunk.wav")))

Performance tips

• GPU (float16) is much faster than CPU; on CPU use int8 and a smaller model.
• Pre-load the model once and reuse it; don't reload per chunk.
• Keep chunks short; longer audio adds latency.

FAQ

Is local Whisper free? Yes — faster-whisper runs on your own hardware; there's no per-call
cost. You only pay for the translation step (e.g. GPT-5.5 via APIVAI), which is cheap.

Does APIVAI provide Whisper? No — APIVAI provides the OpenAI/Anthropic-compatible chat models
(the translation step). Run Whisper locally for speech-to-text.

Which model size for live captions? small on a GPU — a good balance of speed and accuracy.

What latency can I expect? With small + short chunks, transcription is sub-second; end-to-end
(STT → translate → caption) is typically ~1–2s.

Get started

Install faster-whisper, run the transcribe loop, and send the text to APIVAI for translation.
Examples: APIVAI examples repo.

Comments

[Toshius Klay]

Nice writeup. One thing worth adding for anyone wiring this up for real time: the jump from "Whisper runs locally" to "Whisper is accurate in real time" is mostly about chunking. faster-whisper is quick enough, but if you feed it fixed-size windows you get word truncation at the boundaries and timestamp drift over long sessions. Gating chunks with a VAD (only transcribe actual speech segments, with a little overlap) fixed most of that for me.

The other thing that bit me: local Whisper is great on clean single-speaker dictation and falls off hard on overlapping speakers and strong accents, which is exactly where people assume more compute will save them and it doesn't. Are you running any diarization on top for multi-speaker audio, or keeping it strictly single-voice dictation?