I Put Three AI Services on One 24 GB GPU. It Used 17.8. Here's What Fits.
The Local LLM Lab — Part 9
One controlled experiment. One number. One verdict.
The "how much VRAM do I need" advice on the internet comes in one flavour: a spec sheet. "27B in FP8 needs 28 GB. 70B in Q4 needs 42 GB. Buy a 48 GB card."
That's the answer for one model. Nobody tells you what happens when you try to run your actual stack — a transcription service, an embedding model, a small chat model — on one card. The answers you find are guesses, because nobody has measured a multi-tenant GPU.
I have. For the last week my RTX 3090 (24 GB) has been running three AI services at once: a Whisper ASR webservice, a small TTS voice model, and an embedding model for a RAG pipeline. I pulled the full VRAM history and the per-tenant attribution. This is what the card is actually holding, and what the number says about sizing a home GPU.
The setup
- Card: NVIDIA RTX 3090, 24 GB (24,576 MB), power limit 390 W
-
Tenants (all resident, all Docker):
- Whisper ASR webservice (WhisperX, large-v3) — a transcription endpoint my own tools call for voice memos, meetings, podcast clips
- A small TTS voice model — text-to-speech for the same assistant stack
- nomic-embed-text (Ollama) — embeddings for the RAG pipeline
- Telemetry: 7 days of VRAM / power / utilisation at 2-hour resolution, plus per-caller attribution
- Everything else on the box: 55 containers, 92 systemd units, a 56-thread Xeon, 64 GB of RAM. The GPU sees only the three tenants above.
The three services were chosen because that is the actual shape of a home AI stack: one big-ish resident model (ASR), one medium resident model (TTS), one tiny resident model (embeddings). No chat model. The 27B chat model lives on a second machine with two 3090s — that's a different article.
The number
The card is holding 17.8 GB of its 24 GB — 72% — with three models resident, and the peak over the week was 23.4 GB. It never spilled a byte to system RAM.
| Tenant | Avg VRAM (7d) | Peak VRAM (7d) |
|---|---|---|
| Whisper ASR (large-v3) | 7,871 MB (~7.7 GB) | 10,558 MB (~10.3 GB) |
| TTS voice model | 4,876 MB (~4.7 GB) | 4,964 MB (~4.8 GB) |
| nomic-embed-text | 4,633 MB (~4.5 GB) | 4,668 MB (~4.6 GB) |
| Total (measured, 7d) | ~17,380 MB (~17.0 GB) | 23,379 MB (~22.8 GB) |
The measured current state at the time of writing: 17,775 MB used of 24,576 MB — 72% of the card, with the three tenants sitting at 7.87 GB + 4.96 GB + 4.67 GB = 17.5 GB, plus the CUDA context overhead of the card itself.
The peak of 23.4 GB is the interesting number. It happened when the Whisper service received a batch of long audio files at the same time the TTS model was rendering a long response. The two big tenants spiked together, the embedding model held its 4.6 GB, and the card came within 1.2 GB of the ceiling. It held. No spill, no OOM, no crash — because the peak was a few seconds, not a sustained load.
What each tenant is actually holding
The "paper VRAM" of a model is its weights. The measured VRAM is weights plus working set plus framework overhead. Here's the gap, per tenant:
Whisper large-v3. The weights in FP16 are ~3.1 GB. The measured average is 7.7 GB — 2.5× the weights. The difference is the encoder's audio feature buffers (long files and queued files hold more), the VAD and alignment models WhisperX runs alongside, and the CUDA context. The peak of 10.3 GB is a batch of long files in flight.
The TTS voice model. Weights ~4.5 GB, measured 4.7–4.8 GB. This one is close to its paper number because TTS is stateless between requests: no long context, no queue buffers. It's the most "honest" tenant on the card.
nomic-embed-text. A 137M-parameter embedding model. Paper weights ~0.3 GB. Measured 4.5–4.7 GB. That's the Ollama runtime, its CUDA context, and the KV space it pre-allocates for embedding batches. A model that "needs 300 MB" needs 4.6 GB when it's a resident service. That's the number people never quote.
The pattern: the measured VRAM of a resident service is 1.2–2.5× the weight size, and the multiplier is biggest for the models with the most state. If you're sizing a card, budget for the measured number, not the weights.
Who is calling it
The 7-day attribution shows why the Whisper tenant is the biggest consumer of time on the card, even though the TTS model is the bigger spike contributor:
| Caller | Seconds on the card (7d) |
|---|---|
| My own assistant core (via the whisper proxy) | 4,970 |
| Whisper proxy (shared endpoint) | 920 |
| Langfuse worker (observability stack, transcribes attached audio) | 910 |
| Langfuse MinIO (object storage, transcribes on ingest) | 860 |
| Whisper warmup (keep-alive job) | 360 |
| GitHub runner (CI jobs that transcribe test fixtures) | 360 |
The card is a shared service, not a single app. Six different callers hit the Whisper endpoint over the week. The TTS and embedding tenants have one caller each (the assistant stack). The attribution is what turns "the GPU is busy" into "here is who is busy" — and it's the difference between a dashboard you glance at and one you can act on.
The counterargument: 72% is a lot
The fair objection: a card at 72% of its VRAM with a 1.2 GB headroom at peak is a card that will fail the day a bigger file comes in. Headroom is insurance, and 12% is thin insurance.
True. But the measurement says what the insurance was worth over a week of real use: the peak was a 3-second event, and the card absorbed it. The sustained state is ~17 GB, which leaves 7.5 GB of headroom — 31% of the card. The 23.4 GB peak is the worst case that actually happened, not the worst case that is possible.
The honest version of the headroom argument: size for the biggest sustained working set you will actually run, plus 50% headroom. Not for the biggest spike you can imagine. If your stack's sustained working set is 17 GB, a 24 GB card is 1.4× sustained — enough. If you add a 7B chat model to the same card (sustained ~8 GB), the sustained working set becomes 25 GB and the 24 GB card is done. That's the actual failure mode: not a spike, but a new tenant.
What I would buy now
If I were building this stack today:
- The hub card (ASR + TTS + embeddings): 24 GB. Sustained working set 17 GB, peak 23.4 GB. A 16 GB card is not enough — the TTS model alone would crowd the Whisper service. A 32 GB card would be comfortable but costs more and runs hotter for a workload that fits.
- The chat card(s): sized for the chat model. The 27B in FP8 needs two 24 GB cards (or one 48 GB card). It does not share the hub card.
- The rule: one card per class of workload — resident services on one, the big chat model on another. The measured working set of each class, plus 50% headroom, is the size.
The 24 GB 3090 is still the best price-per-GB on the used market, and that's why it's everywhere in home stacks. But "best price-per-GB" is an argument for the chat machine. For the hub, the argument is "fits the measured working set of my resident services with room for one more tenant" — and for my stack, that's 24 GB, not 16 and not 48.
Verdict
Three AI services on one 24 GB GPU: 17.8 GB sustained, 23.4 GB peak, zero spills over 7 days. The measured VRAM of a resident service is 1.2–2.5× its weights, and the peak is a few seconds, not a sustained load. Size the card for the sustained working set plus 50% headroom, and keep the big chat model on its own card. The spec sheet tells you what one model needs. The meter tells you what your stack needs.
The Local LLM Lab is a series of measured experiments on a home GPU stack. Every number in this article was pulled from 7 days of VRAM telemetry on the card, not estimated. Next: who is actually eating the GPU — the caller-level attribution that turns "the GPU is busy" into a list of names.
Top comments (0)