The Local LLM Lab — Part 10
One controlled experiment. One number. One verdict.
Every homelab has a moment like this: the GPU is at 100%, the fan is screaming, and you have no idea which of the twenty services on the box is doing it. You open nvidia-smi, it shows one process, you close it, and the mystery continues.
The problem is that nvidia-smi answers "which process is using the GPU" — and in a Docker stack, the process is the container runtime, not the service. The answer you actually want is "which service is using the GPU, and how much time did it take, and who called it."
I built the answer into my homelab monitor: caller→server attribution. Every sample of GPU utilisation is tagged with the service that owns the process, and for shared services (a transcription endpoint, an embedding endpoint) it's tagged with the caller — the thing that made the request. Over the last 7 days the attribution logged 1,055 samples. This is what it says about who eats a home GPU.
The setup
- Card: NVIDIA RTX 3090, 24 GB, power limit 390 W
-
Shared services on the card:
- Whisper ASR webservice — a transcription endpoint with an nginx proxy in front (my tools, the observability stack, CI, and a warmup job all hit the proxy)
- nomic-embed-text (Ollama) — an embedding endpoint used by the RAG pipeline
- Attribution: every GPU sample over 7 days is tagged with (a) the service owning the process and (b) the caller for shared endpoints. Total: 1,055 samples, ~1.1 million seconds of GPU time attributed.
The key design decision: the caller is recorded from the request, not from the process. When my assistant core sends an audio file to the whisper proxy, the sample is tagged jarvis2-core → whisper-proxy, not nginx → whisper. That's the difference between "the GPU is busy" and "my assistant is the one using it."
The number
Over 7 days, 47% of the GPU's logged time went to one caller: my own assistant core (4,970 seconds of 10,555 total attributed seconds).
| Caller | Seconds (7d) | Server | % of total |
|---|---|---|---|
| jarvis2-core (my assistant) | 4,970 | whisper-proxy | 47.1% |
| whisper-proxy (shared endpoint, direct) | 920 | whisper service | 8.7% |
| langfuse worker (observability) | 910 | whisper service | 8.6% |
| langfuse minio (object storage) | 860 | whisper service | 8.1% |
| whisper-warmup (keep-alive job) | 360 | whisper service | 3.4% |
| gh-runner (CI) | 360 | whisper-proxy | 3.4% |
| jarvis-server + watchtower | 410 | whisper-proxy / service | 3.9% |
| firefly importer | 80 | whisper service | 0.8% |
| ollama-embed callers (CI, assistant, watchtower) | 210 | ollama-embed | 2.0% |
| UptimeKuma (health checks) | 20 | whisper service | 0.2% |
The single biggest consumer of the GPU is not a model, not a benchmark, not a training run. It's my assistant transcribing the audio it records. Voice memos, meeting recordings, the audio attached to the traces the observability stack ingests — all of it flows through the Whisper endpoint, and all of it is attributed to the caller that sent it.
What the attribution changes
Without caller attribution, the 7-day story of this GPU is: "the Whisper service used 10,555 seconds of GPU time." That's a number. It doesn't tell you anything you can act on.
With caller attribution, the story is:
- My assistant is the tenant. 47% of the GPU's time is my own tooling. The service exists for it, and it uses it more than everything else combined. That's the design working as intended — and it's also the reason the service is resident 24/7 instead of on-demand.
- The observability stack is the second tenant. Langfuse's worker and MinIO together take 17% of the GPU's time. They transcribe the audio attached to the LLM traces they store. That's a cost nobody budgets for: the tracing stack is using the GPU to do its job, on top of the LLM it's tracing.
- CI is a real, small tenant. The GitHub runner takes 3.4% — test fixtures that get transcribed during CI runs. It's small, but it's unexpected small. Nobody thinks about CI touching the GPU.
- The warmup job is 3.4%. A keep-alive job that pings the endpoint every few minutes so the model stays warm. That's the "keep it warm" cost made visible — 3.4% of the GPU's week, doing nothing but staying ready.
The verdict from the numbers: the GPU is not "busy" in the way the alarm implies. It is busy for specific, named reasons, and the reasons add up to a stack that is working, not failing. The alarm that says "GPU at 100%" is a symptom. The attribution is the diagnosis.
The counterargument: attribution is overhead
The fair objection: tagging every GPU sample with a caller is extra bookkeeping, and in a hot loop it's overhead. The attribution has to be cheap enough that it doesn't slow the inference it's measuring.
The implementation I use is cheap by design: the tag is set at the request boundary, not at the sample boundary. When a request comes in, the caller is recorded once and attached to every sample the request produces. The per-sample cost is a field read, not a lookup. Over 1,055 samples the overhead is below the resolution of the power meter — you cannot measure it against a 2-hour bucket.
The other objection: attribution only works for services that go through a request boundary. A model that's loaded and used in-process (a training run, a direct script) has no caller to tag. For those, the service-level tag is the best you can do, and that's still better than nothing. The attribution is a spectrum: caller-level for request-driven services, service-level for the rest.
What I would do now
If I were building a homelab monitor today, the GPU panel would have three layers:
-
The card level: utilisation, VRAM, power, temperature. The thing
nvidia-smishows. - The service level: which container owns the process, and how much VRAM and time it holds. The thing Docker's stats show, but per-GPU.
- The caller level: for shared endpoints, which client made the request that produced each sample. The thing nothing shows, and the thing that answers "who is using my GPU."
The third layer is the one that turns a dashboard into a tool. When the GPU spikes, you don't open nvidia-smi and guess. You look at the attribution and see the name. When a service misbehaves (a runaway job, a retry storm), the caller list shows it in the same view. The number that matters is not "the GPU is at 100%" — it's "jarvis2-core took 4,970 seconds this week, and here's the request that did it."
Verdict
One week of caller-level GPU attribution: 47% of the card's time went to my own assistant, 17% to the observability stack, 3.4% to CI, 3.4% to a warmup job that does nothing but stay ready. The GPU was never "mysteriously busy." It was busy for named reasons, and the attribution is the only view that shows them. If your homelab GPU spikes and you don't know why, the fix is not a faster card — it's a caller tag on every sample.
The Local LLM Lab is a series of measured experiments on a home GPU stack. Every number in this article was pulled from 7 days of caller-level attribution on the card, not estimated. Next: the 55 containers and 92 services that share this box with the GPU — what actually eats RAM when the machine has 64 GB of it.
Top comments (0)