Whisper large-v3 VRAM Requirements: Why It Won't Fit on a 5GB GPU (and What We Tried Instead)

TL;DR

whisper-large-v3 OOMs on a 5GB GPU (Quadro P2000) at float16, int8_float16, and full int8 — before serving a single request. Root cause is architecture overhead (32-layer encoder-decoder, activations, CUDA context), not just weight size. Fine-tuned whisper-tiny → base → small → small-v2 on Common Voice Bulgarian instead: held-out WER improved from 88.2% → 32.7% across escalating model size, but never closed the gap to large-v3's 27.3%. A community large-v3-turbo Bulgarian fine-tune claiming 9.97% WER on FLEURS scored 31.2% on our own held-out set — same ballpark as our own model, not the win the model card implied. Built a real dual-GPU nginx failover (P2000 = fine-tune, 3090 = large-v3) that worked correctly on deploy, then failed a real spontaneous-speech test badly enough to roll back to large-v3-only within ~5 seconds. Core finding: Common Voice read-aloud WER does not predict real assistant-use transcription quality.

The setup

ardi has one RTX 3090 (24GB) doing LLM inference work, and a Quadro P2000 (5GB) that's sat idle for about two years. Jarvis, a self-hosted assistant, depends on Whisper for speech-to-text — testing showed only large-v3 handles Bulgarian well; smaller stock checkpoints are fine for English, not for a lower-resource language. large-v3 sits permanently loaded on the 3090, the same card needed for local LLM serving.

Question: can the idle P2000 take Bulgarian transcription off the 3090's hands via a Bulgarian-specific fine-tune small enough to fit 5GB?

(One naming note so the rest of this makes sense: the container running here is whisper-asr-webservice wrapping faster-whisper — not the separate WhisperX project, despite what I've been calling it internally for months.)

Attempt 1: does large-v3 just fit?

Tested large-v3 on the P2000 at three precisions:

float16        -> OOM
int8_float16   -> OOM
int8           -> OOM

All three OOM before serving a request. Not a quantized-weight-size problem — the encoder-decoder's non-weight overhead (32 layers, activations, CUDA context) exceeds 5GB regardless of precision. whisper-tiny loads at 318MB with no issue, ruling out a driver/compatibility problem. medium (769M params) was the practical ceiling for raw model size — 3.87GB used, 1.2GB headroom — but a generic multilingual medium isn't good enough for Bulgarian on its own.

Attempts 2–4: escalating fine-tunes

Fine-tuned on Mozilla Common Voice Bulgarian, on the 3090, via HuggingFace transformers Seq2SeqTrainer. Evaluated on the same 150 held-out test clips (never seen in training) for every model:

small-v2 = same architecture as small, retrained on train+other combined (6,739 rows vs 4,952) for 5 epochs. Validation WER by epoch: 32.17 → 28.99 → 28.21 → 28.21 → 28.44 — flattened, then rose at epoch 5 (overfitting), so load_best_model_at_end correctly kept the epoch 3/4 checkpoint rather than the final one. No more clean Bulgarian Common Voice data exists beyond train+other, so this is the practical ceiling for this data/model-size combination.

Two gotchas caught along the way:

# 1. CUDA_VISIBLE_DEVICES alone doesn't guarantee GPU index matches
# nvidia-smi's PCI-bus order -- a run silently landed on the P2000
# instead of the intended 3090 until:
export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=1

# 2. ardi's root disk (already at a tight 90% baseline) filled to 100%
# mid-training from accumulated dataset/HF caches -- silent SIGKILL,
# no traceback. Fixed by pointing the cache at a bigger volume instead
# of the system disk:
export HF_HOME=/backup/hf-cache
export CACHE_DIR=/backup/whisper-bg-tiny-data

Neither is the interesting part of this story, but both cost real debugging time — worth checking explicitly on any shared multi-GPU box.

Attempt 5: the community shortcut that didn't reproduce

Searched Hugging Face for an existing Bulgarian ASR fine-tune before pushing further on limited training data. Found sam8000/whisper-large-v3-turbo-bulgarian-bulgaria — a fine-tune of large-v3-turbo (same 32-layer encoder as full large-v3, decoder pruned from 32 to 4 layers), claiming 9.97% WER on the FLEURS Bulgarian benchmark.

Converted to CTranslate2, it does fit the P2000 — 4.1–4.2GB used, ~900MB headroom — tight but real (the third bar in the VRAM chart above). Evaluated on the same held-out Common Voice test set used for every model above:

sam8000/whisper-large-v3-turbo-bulgarian-bulgaria: 31.2% WER
our own small-v2 (fine-tuned):                     32.7% WER

Statistically the same result, not the dramatic win the model card implied. The 9.97% FLEURS number isn't fake — it just doesn't transfer to a different eval set with different preprocessing/normalization. Always re-measure a candidate on your own eval, apples to apples, before trusting a model card's headline number.

The part that worked: dual-GPU failover

Built a real deployment: two whisper containers (P2000 = small-v2, 3090 = large-v3 unchanged) behind an nginx sidecar using proxy_next_upstream for automatic failover. One detail that shapes what "failover" means here: whisper-asr-webservice loads its model eagerly at process boot, not per-request — so this isn't a live per-call fallback, it's "is this backend up or down," decided once at startup.

Deployed live, confirmed it actually worked — routing correct. The old standalone production container was kept stopped, not deleted, for the entire session — the eventual rollback was a container start, not a rebuild.

The test that actually mattered

Real spontaneous speech — describing colors and objects out loud, not Common Voice-style read sentences. Verdict: "quite, quite, quite weak." Noticeably worse than the 32.7% benchmark WER suggested for casual listening. Rolled back to large-v3-only production immediately — ~5 seconds, because the old container was never torn down.

What we deliberately didn't do next

• Didn't publish a GitHub repo for the fine-tuned checkpoints — the result isn't good enough to ship as a "solution."
• Didn't chase a 6th fine-tune attempt (medium-size, more data augmentation) — diminishing returns were already visible in the epoch curve, and the deeper problem (domain mismatch between read-aloud and spontaneous speech) wouldn't be fixed by more of the same data.
• Didn't keep the dual-GPU stack running "just in case" — production reverted to exactly its pre-session state, P2000 idle again.

The actual finding

Common Voice is people reading prepared text aloud in clean conditions — a different domain from spontaneous conversational speech directed at an assistant (prosody, hesitation, mic quality, vocabulary). A benchmark WER on read-aloud speech didn't predict real assistant-use quality here, for either our own fine-tune or a community model claiming a much better number on a different benchmark. This generalizes past Bulgarian and past Whisper: eval-set domain match matters more than the headline metric.

Full narrative version — the charts, the physical GPU install photo, the "why I still don't have a use for this card" ending — on Medium.

---

Every VRAM ceiling and WER number above was measured via HomeLab Monitor — MIT licensed, one container, the same tool that's priced every benchmark in this series.

Curious if anyone's gotten a Bulgarian (or other lower-resource-language) Whisper fine-tune to hold up on real spontaneous speech, not just a read-aloud benchmark — and what closed the gap if so.