DEV Community

Cover image for Whisper large-v3 VRAM Requirements: Why It Won't Fit on a 5GB GPU (and What We Tried Instead)
Arsen Apostolov
Arsen Apostolov

Posted on

Whisper large-v3 VRAM Requirements: Why It Won't Fit on a 5GB GPU (and What We Tried Instead)

#machinelearning #whisper #gpu #homelab

TL;DR

whisper-large-v3 OOMs on a 5GB GPU (Quadro P2000) at float16, int8_float16, and full int8 — before serving a single request. Root cause is architecture overhead (32-layer encoder-decoder, activations, CUDA context), not just weight size. Fine-tuned whisper-tiny → base → small → small-v2 on Common Voice Bulgarian instead: held-out WER improved from 88.2% → 32.7% across escalating model size, but never closed the gap to large-v3's 27.3%. A community large-v3-turbo Bulgarian fine-tune claiming 9.97% WER on FLEURS scored 31.2% on our own held-out set — same ballpark as our own model, not the win the model card implied. Built a real dual-GPU nginx failover (P2000 = fine-tune, 3090 = large-v3) that worked correctly on deploy, then failed a real spontaneous-speech test badly enough to roll back to large-v3-only within ~5 seconds. Core finding: Common Voice read-aloud WER does not predict real assistant-use transcription quality.

The setup

ardi has one RTX 3090 (24GB) doing LLM inference work, and a Quadro P2000 (5GB) that's sat idle for about two years. Jarvis, a self-hosted assistant, depends on Whisper for speech-to-text — testing showed only large-v3 handles Bulgarian well; smaller stock checkpoints are fine for English, not for a lower-resource language. large-v3 sits permanently loaded on the 3090, the same card needed for local LLM serving.

Question: can the idle P2000 take Bulgarian transcription off the 3090's hands via a Bulgarian-specific fine-tune small enough to fit 5GB?

(One naming note so the rest of this makes sense: the container running here is whisper-asr-webservice wrapping faster-whisper — not the separate WhisperX project, despite what I've been calling it internally for months.)

Attempt 1: does large-v3 just fit?

Tested large-v3 on the P2000 at three precisions:

float16        -> OOM
int8_float16   -> OOM
int8           -> OOM
Enter fullscreen mode Exit fullscreen mode

Horizontal bar chart: VRAM used on the P2000 by whisper-tiny (0.32GB), the deployed small-v2 fine-tune (1.4GB), and a community turbo-bg model (4.15GB), all under a 5.0GB ceiling — versus large-v3 as a red hatched bar breaking through the ceiling, labeled OOM, needs ~8GB

All three OOM before serving a request. Not a quantized-weight-size problem — the encoder-decoder's non-weight overhead (32 layers, activations, CUDA context) exceeds 5GB regardless of precision. whisper-tiny loads at 318MB with no issue, ruling out a driver/compatibility problem. medium (769M params) was the practical ceiling for raw model size — 3.87GB used, 1.2GB headroom — but a generic multilingual medium isn't good enough for Bulgarian on its own.

Attempts 2–4: escalating fine-tunes

Fine-tuned on Mozilla Common Voice Bulgarian, on the 3090, via HuggingFace transformers Seq2SeqTrainer. Evaluated on the same 150 held-out test clips (never seen in training) for every model:

Grouped bar chart: WER by model, zero-shot vs fine-tuned, tiny through small-v2, with a dashed reference line at large-v3's 27.3%. Fine-tuned WER descends from 68.7% to 32.7% across the four models, never reaching the reference line

small-v2 = same architecture as small, retrained on train+other combined (6,739 rows vs 4,952) for 5 epochs. Validation WER by epoch: 32.17 → 28.99 → 28.21 → 28.21 → 28.44 — flattened, then rose at epoch 5 (overfitting), so load_best_model_at_end correctly kept the epoch 3/4 checkpoint rather than the final one. No more clean Bulgarian Common Voice data exists beyond train+other, so this is the practical ceiling for this data/model-size combination.

Two gotchas caught along the way:

# 1. CUDA_VISIBLE_DEVICES alone doesn't guarantee GPU index matches
# nvidia-smi's PCI-bus order -- a run silently landed on the P2000
# instead of the intended 3090 until:
export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=1

# 2. ardi's root disk (already at a tight 90% baseline) filled to 100%
# mid-training from accumulated dataset/HF caches -- silent SIGKILL,
# no traceback. Fixed by pointing the cache at a bigger volume instead
# of the system disk:
export HF_HOME=/backup/hf-cache
export CACHE_DIR=/backup/whisper-bg-tiny-data
Enter fullscreen mode Exit fullscreen mode

Neither is the interesting part of this story, but both cost real debugging time — worth checking explicitly on any shared multi-GPU box.

Attempt 5: the community shortcut that didn't reproduce

Searched Hugging Face for an existing Bulgarian ASR fine-tune before pushing further on limited training data. Found sam8000/whisper-large-v3-turbo-bulgarian-bulgaria — a fine-tune of large-v3-turbo (same 32-layer encoder as full large-v3, decoder pruned from 32 to 4 layers), claiming 9.97% WER on the FLEURS Bulgarian benchmark.

Converted to CTranslate2, it does fit the P2000 — 4.1–4.2GB used, ~900MB headroom — tight but real (the third bar in the VRAM chart above). Evaluated on the same held-out Common Voice test set used for every model above:

sam8000/whisper-large-v3-turbo-bulgarian-bulgaria: 31.2% WER
our own small-v2 (fine-tuned):                     32.7% WER
Enter fullscreen mode Exit fullscreen mode

Statistically the same result, not the dramatic win the model card implied. The 9.97% FLEURS number isn't fake — it just doesn't transfer to a different eval set with different preprocessing/normalization. Always re-measure a candidate on your own eval, apples to apples, before trusting a model card's headline number.

The part that worked: dual-GPU failover

Built a real deployment: two whisper containers (P2000 = small-v2, 3090 = large-v3 unchanged) behind an nginx sidecar using proxy_next_upstream for automatic failover. One detail that shapes what "failover" means here: whisper-asr-webservice loads its model eagerly at process boot, not per-request — so this isn't a live per-call fallback, it's "is this backend up or down," decided once at startup.

Deployed live, confirmed it actually worked — routing correct. The old standalone production container was kept stopped, not deleted, for the entire session — the eventual rollback was a container start, not a rebuild.

The test that actually mattered

Real spontaneous speech — describing colors and objects out loud, not Common Voice-style read sentences. Verdict: "quite, quite, quite weak." Noticeably worse than the 32.7% benchmark WER suggested for casual listening. Rolled back to large-v3-only production immediately — ~5 seconds, because the old container was never torn down.

What we deliberately didn't do next

  • Didn't publish a GitHub repo for the fine-tuned checkpoints — the result isn't good enough to ship as a "solution."
  • Didn't chase a 6th fine-tune attempt (medium-size, more data augmentation) — diminishing returns were already visible in the epoch curve, and the deeper problem (domain mismatch between read-aloud and spontaneous speech) wouldn't be fixed by more of the same data.
  • Didn't keep the dual-GPU stack running "just in case" — production reverted to exactly its pre-session state, P2000 idle again.

The actual finding

Common Voice is people reading prepared text aloud in clean conditions — a different domain from spontaneous conversational speech directed at an assistant (prosody, hesitation, mic quality, vocabulary). A benchmark WER on read-aloud speech didn't predict real assistant-use quality here, for either our own fine-tune or a community model claiming a much better number on a different benchmark. This generalizes past Bulgarian and past Whisper: eval-set domain match matters more than the headline metric.

Full narrative version — the charts, the physical GPU install photo, the "why I still don't have a use for this card" ending — on Medium.

Every VRAM ceiling and WER number above was measured via HomeLab Monitor — MIT licensed, one container, the same tool that's priced every benchmark in this series.

Curious if anyone's gotten a Bulgarian (or other lower-resource-language) Whisper fine-tune to hold up on real spontaneous speech, not just a read-aloud benchmark — and what closed the gap if so.

Top comments (1)

Collapse
 
elbokazqc profile image
elboKazQC •

Your core finding, that read-aloud WER doesn't predict spontaneous-speech quality, is exactly why I've refused to publish a WER number for Quebec French dictation even after promising one three times. The public FR corpora lean toward read-aloud France French, so picking the test sentences is already a form of tuning, and it says nothing about someone dictating code with FR/EN code-switching mid-sentence.

On "what closed the gap" for my lower-resource case (Quebec French plus dev jargon): I never went the fine-tune route, the clean data isn't there either. I steered at decode time instead, faster-whisper's hotwords plus a French initial_prompt that literally tells the model to keep English words in English and not translate "commit" into something French, then a small deterministic regex layer for the handful of mistakes it makes the same way every time. No training, no GPU bill, and it hits the domain mismatch where it actually shows up for me: vocabulary, not acoustics. Same medium-is-the-ceiling story otherwise, medium int8 on CPU and large-v3 float16 only when CUDA is present.

Did prompt and hotwords steering ever hold up for you on Bulgarian, or does a lower-resource language really need the fine-tune to close it?