The Local LLM Lab — Part 13
One controlled experiment. One number. One verdict.
The local-AI crowd has a habit: buy the biggest GPU. 24 GB. 32 GB. 48 GB. The reasoning is always the same — "models are big, you need VRAM."
I stopped believing that reasoning when I looked at my own telemetry. I've been running a Whisper transcription service on an RTX 3090 (24 GB) for a few months now. Over the last 30 days I pulled the full VRAM history: every two hours, what the card was holding, what it spiked to, and whether anything ever spilled into system RAM.
The service used 7.9 GB on average. 10.4 GB at peak. Never more. In 30 days.
This is the story of a GPU that is 2.3× bigger than the thing running on it — and what that means for what you should actually buy.
The setup
- Card: NVIDIA RTX 3090, 24 GB (24,576 MB), power limit 260 W
- Service: Whisper ASR webservice (WhisperX, large-v3), resident Docker container, 24/7
- Telemetry: 30 days of VRAM / power / utilisation at 2-hour resolution, plus per-caller attribution
- Other residents: a TTS service (~5.3 GB, added partway through the window), an embedding model (negligible), a photo-tagging ML container (negligible)
The 3090 was never sized for Whisper. Whisper is the small, quiet service in the corner of the stack — a transcription endpoint that my own tools call when I record a voice memo, a meeting, or a podcast clip. The big chat models live on a second machine. The TTS service is the one other real tenant; it arrived partway through the measurement window, which is why the "one service" story below is about the Whisper service's own working set, not the card's total.
That's the important part. Most "how much VRAM do I need" answers on the internet are about the model on paper, not the service in production. Paper VRAM is the weights. Production VRAM is weights plus context plus overhead plus whatever the framework decides to allocate. The difference is the whole article.
The number
Average VRAM: 7.9 GB. Peak: 10.4 GB. Spills to RAM: zero.
| Metric (30 days) | Value |
|---|---|
| Total VRAM | 24,576 MB |
| Average VRAM | 7,871 MB (~7.9 GB) |
| Peak VRAM | 10,366 MB (~10.4 GB) |
| Peak / Total | 42% |
| Spills to system RAM | 0 |
| GPU utilisation | ~0% (bursts to 1–7%) |
| Power at rest (model resident) | ~40 W |
| Energy attributed to the service (30d) | 21.73 kWh / 5.67 BGN |
The shape of the trace is boring, and that's the point. It's a flat line at ~7.9 GB with short, sharp spikes to ~10.4 GB when audio comes in, and a few longer elevated stretches when there's a queue. It never trends upward. It never approaches the ceiling. It never spills.
The card sits at ~40 W with the model resident — that's the "keep it warm" cost, not the work cost. The work (the spikes) is short enough that the 30-day total attributed to the service is 21.73 kWh, which at my tariff is 5.67 BGN. The full electricity breakdown is in Part 5 of this series.
What the VRAM is actually holding
People treat VRAM like a single bucket. It isn't. In this service, the ~10.4 GB peak is:
- Model weights. Whisper large-v3 is 1.55B parameters. In fp16 that's ~3.1 GB. This is the "paper VRAM" number everyone quotes.
- Audio context and feature buffers. The encoder processes audio in chunks; long files and queued files hold more buffer. This is where 3.1 GB becomes 7.9 GB on average.
- Framework overhead. CUDA context, cuDNN workspaces, allocator fragmentation. Usually 1–2 GB, invisible in any spec sheet.
- The small helper models. WhisperX runs a VAD (voice-activity detection) and an alignment model alongside the main model. Small, but real.
The lesson: a model's "VRAM requirement" is its weights. A service's VRAM requirement is 2–3× its weights. When someone says "large-v3 needs 3 GB," they're telling you the floor, not the working set.
Who is actually calling it
The attribution over the last 7 days is a nice detail, because it shows this is a shared service, not a single app:
| Caller | Seconds (7d) |
|---|---|
| whisper-proxy (my own tools) | 3,350 |
| Langfuse MinIO (observability stack) | 1,080 |
| Langfuse worker | 720 |
| CI runner | 300 |
| Warmup job | 280 |
My own tools are the main tenant; the LLM-observability stack is a secondary one — it transcribes the audio attached to the traces it ingests. Even with multiple callers hitting the endpoint, the peak stays at 10.4 GB. The second and third callers add latency, not VRAM, because they queue behind the first.
The counterargument: headroom is insurance
The fair objection: VRAM headroom is insurance. You want room for longer context, bigger batches, a bigger model next quarter. A card that's exactly sized for today's workload is a card that's undersized next month.
True. But the measurement says what the insurance was worth over 30 days of real use: the service used 42% of the card at its absolute peak, 32% on average, and never came close to the ceiling. Insurance that is never claimed is still a cost — in this case, the delta between a 16 GB card and a 24 GB card, plus the power and thermals of running a card that's 2.3× the working set.
The honest version of the headroom argument is: size for the biggest model you will actually run, plus ~50% headroom. Not the biggest model that exists. Not the biggest model the card could run.
The contrast: the other machine
This is where the 3090's real job shows up. The big model — a 27B chat model on vLLM — runs on a second machine with two 3090s. That machine is sized for its workload: a 27B model in fp8 needs well over 24 GB, so two cards, no spills, no compromise.
The hub 3090 is not sized for Whisper. It's sized for "whatever I might want to run on the hub someday." That's a different purchase decision, and it's the one most of us make without measuring. The measurement says: for the workload that actually lives on this card, a 16 GB card is enough for Whisper alone. 24 GB is only justified if a 20B+ chat model is going to share the card — and if that's the plan, you're sizing for the chat model, not for Whisper, and Whisper should be the tenant that fits around it.
The card today: two tenants, still not full
Three weeks into the window a TTS service joined the card. It holds ~5.3 GB. Right now the card's total is ~13.3 GB of 24 GB — 55% at the current moment, still zero spills, still ~2 GB of headroom under memory pressure.
That's the honest current state: two AI services (Whisper + TTS) on one 24 GB card, and the card is not full. The "you need big VRAM" advice would have me buying more. The measurement says the card has room for another tenant of similar size before it gets close to the ceiling. The 24 GB is doing its job as headroom — but it's headroom for a second or third small service, not because Whisper needs it.
What I would buy now
If I were building this stack today, from scratch:
- The hub card (Whisper + TTS + embeddings): 24 GB is the right call, but only because of the second tenant. Whisper alone peaks at 10.4 GB; with the TTS service it's ~13.3 GB. A 16 GB card would be tight but workable for just Whisper; 24 GB is comfortable for two.
- The chat card(s): sized for the chat model. 27B in fp8 → two 24 GB cards, or one 48 GB card if you can find one at a sane price.
- The rule: the GPU is sized for the biggest model you will actually run, plus 50% headroom. Everything smaller is a tenant.
The 3090 remains a great card — it's the best price-per-GB on the used market, and that's exactly why it's everywhere in home stacks. But "best price-per-GB" is an argument for the chat machine, and for a hub that will host a couple of small services, not for the transcription service alone.
Verdict
A 24 GB GPU running a Whisper service for 30 days: 7.9 GB average, 10.4 GB peak, zero spills, 42% of the card at peak. Add a 5.3 GB TTS service and the card is still only 55% full. The "you need big VRAM" advice is true for the model you're afraid of running, and false for the service you're actually running.
Right-size the GPU to the workload you measured, not the workload you imagine. The meter and the VRAM counter don't care about your imagination.
The Local LLM Lab is a series of measured experiments on a home GPU stack. Every number in this article was pulled from 30 days of telemetry on the card, not estimated. The series continues with the cost of keeping all of this alive — the full electricity bill for the whole machine, not just the GPU.
Top comments (0)