DEV Community: Arsen Apostolov

Patzer v7.19: the tactic trainer that shows your move and explains why the better one wins

Arsen Apostolov — Mon, 19 Oct 2026 08:06:43 +0000

Last week I wrote about Patzer — a self-hosted Chess.com alternative that runs in one Docker container (the original post). Since then, the feature I've been most proud of shipped: a tactic trainer that explains your own mistakes.

The gap in puzzle trainers

The classic puzzle trainer — Lichess puzzles, Chess.com tactics — gives you a position and asks for the best move. Miss it, and you see the solution. But you never see why yours was wrong: why the engine's move wins and your move loses. You memorize that Qh5 is the move, not that Qd3 leaves your queen hanging to a fork.

v7.19.0 closes that gap.

What's new

Your move, on the board. In Train → From your games, Patzer builds puzzles from your actual Chess.com or Lichess games. The move you played in that game is now drawn as a red arrow on the puzzle position, so you no longer have to work out from "you played Qd3" which move that was.

The coach explains the difference. Once the puzzle is over — solved, or the solution shown, never before — your AI coach explains in plain words why the engine's move beats the one you played. It's the same engine-checked explanation as in Game Review, in your language, pitched at your audience setting.

The design decision from the original post is what makes this cheap to run: the LLM never analyses. The server works out the facts — what each move allows, hangs or wins, the opponent's best answer — and the model only turns that into coaching. A small local model (Ollama or vLLM) is enough, and a hallucinated piece is impossible.

Puzzles come back. A puzzle you don't solve at the first try comes back the next day, marked Review — the same simple daily queue as the opening trainer. Solved at the first try on a later day, it's done; missed again, it's back tomorrow.

Puzzle of the day, with a streak. One puzzle from the Lichess set near your puzzle rating, the same one all day, with a plain counter of the days in a row you did it.

Play from here. In Game Review, every move now has a "Play from here against Patzer" button. You play your own colour, pick the bot's strength and the clock, and the game is saved like any bot game — your own games become training positions.

Full list in the release notes, including a fix for the saved AI model dropdown when the model isn't on the server:

Still one container:

docker run -d -p 8800:8800 -v patzer-data:/app/data --name patzer ghcr.io/SikamikanikoBG/patzer:latest

MIT licensed, English-first with Bulgarian, Spanish, German and Russian out of the box. If you're running Patzer, pull the latest tag and try the trainer on your own games — that's the part I'd want you to try first.

54 Containers, 93 Services, 64 GB of RAM: What Actually Eats It

Arsen Apostolov — Mon, 19 Oct 2026 07:11:05 +0000

The Local LLM Lab — Part 11

One controlled experiment. One number. One verdict.

A home server has a question the cloud never asks: who is eating the RAM?

In the cloud, memory is a line item on a bill. At home, it's a finite pool — 64 GB on my box — shared by every container, every service, every daemon, and the one GPU that runs the AI. When the pool runs dry, something gets killed, and the thing that gets killed is usually the thing you least expected.

I ran a memory audit on my homelab: 54 Docker containers, 93 running systemd units, one RTX 3090, 64 GB of RAM. I pulled the per-service memory history over 7 days, the current per-process state, and the kernel's non-reclaimable share. This is what the 64 GB is actually holding, and what the number says about how much RAM a homelab really needs.

The setup

Machine: HP Z840 Workstation, 2× Intel Xeon E5-2680 v4 (28 cores, 56 threads), 64 GB RAM, one RTX 3090
Workload: 54 Docker containers, 93 running systemd units, all up
Telemetry: 7 days of per-service memory at 2-hour resolution, plus the current per-process state and the kernel's non-reclaimable memory
GPU tenants (for context): Whisper ASR, a TTS model, an embedding model — see the VRAM piece for what the card holds

The 64 GB is the honest number. It's not a "big" machine by data-centre standards, but it's a real workstation, and it runs a real stack: photo storage, music, a RAG pipeline, an LLM observability stack, a CI runner, a media server, a few self-hosted apps, and the AI services on the GPU.

The number

The box is using 41.6 GB of its 64 GB — 65% — and the peak over the week was 15.9 GB of application memory, with the rest held by the kernel and the GPU's system-side buffers.

Layer	RAM (current)	Share of 64 GB
Kernel (non-reclaimable)	1,754 MB (~1.7 GB)	2.7%
GPU tenants (system-side buffers + CUDA context)	~2.0 GB	3.1%
Top application (Whisper ASR webservice)	7,870 MB (~7.7 GB)	12.2%
TTS voice model (omnivoice)	5,100 MB (~5.0 GB)	7.9%
LLM observability stack (web + worker + ClickHouse)	~2.3 GB	3.6%
Photo storage server (Immich: server + postgres + ML)	~1.0 GB	1.6%
RAG pipeline (Qdrant + Ollama + Chroma)	~0.7 GB	1.1%
The other 49 containers + 93 services	~22.7 GB	35.3%
Total in use	41,622 MB (~40.7 GB)	65%

The "other 49 containers + 93 services" line is the one that surprises people. It's 35% of the RAM, and it's not one thing — it's a long tail. The biggest individual entries in that tail:

Service	RAM (current)	What it is
ClickHouse (Langfuse storage)	1,082 MB	The tracing database
Open WebUI (chat UI)	777 MB	The chat interface
Langfuse web (LLM observability)	721 MB	The tracing UI
Immich server (photo storage)	562 MB	The photo library backend
Immich postgres (photo metadata)	536 MB	The photo database
Qdrant (vector DB)	483 MB	The RAG index
Langfuse worker (trace processing)	346 MB	The trace worker
Botpress (chatbot)	292 MB	The chatbot framework
n8n (workflow automation)	279 MB	The automation engine
Flowise (LLM builder)	255 MB	The LLM workflow builder
homelab-monitor (this monitor)	233 MB	The dashboard itself
jarvis2-core (my assistant)	229 MB	The assistant core
jarvis-server	213 MB	The server host
jarvis2-voiceline (voice pipeline)	199 MB	The voice line
Immich machine-learning (face/photo ML)	177 MB	The photo ML worker
Langfuse minio (trace object storage)	164 MB	The trace storage
searxng (meta-search)	111 MB	The search engine
UptimeKuma (health checks)	103 MB	The uptime monitor
MySQL (8.0, two instances)	34 MB + 164 MB	The databases
GitHub runner (CI)	58 MB	The CI agent
Postgres (three instances)	15 MB + 20 MB + 536 MB	The databases

The pattern: no single service is the hog. The RAM is eaten by a long tail of small, resident services, each holding 10–1,000 MB, and the sum is 22.7 GB. The GPU tenants hold the most per service (7.7 GB and 5.0 GB), but they're two services. The long tail is forty-nine, and it's still a third of the box.

What the peak says

The 7-day peak of 15.9 GB of application memory is the number that matters for sizing. It happened when the photo server was running a batch of thumbnail generation at the same time the CI runner was building a project and the observability stack was ingesting a burst of traces. Three things spiking at once, on a 64 GB box, and the peak was 15.9 GB of application memory — 25% of the pool.

The headroom at peak was 48 GB. That's the insurance. The box has never come close to the ceiling, and the events log over the week shows zero OOM kills, zero memory threshold crossings — the only alerts were eight disk I/O spikes, which are storage, not memory.

The honest reading: 64 GB is comfortable for this stack. 32 GB would be tight but workable. 16 GB would not. The peak of 15.9 GB is the worst case that actually happened, and it left 48 GB on the table. If I were buying RAM for this box today, I'd buy 64 GB and stop — not because it's the minimum, but because it's the size where the peak is a non-event.

The counterargument: RAM is cheap, don't sweat it

The fair objection: RAM is the cheapest part of a server. 64 GB is a few hundred euros. Why spend an article on it?

Because the RAM question is where homelab sizing goes wrong. People size the GPU (the expensive, scarce part) and under-size the RAM (the cheap, abundant part), and then the box OOMs the one service that matters — usually the database or the photo server — at the worst moment. The GPU is the part everyone benchmarks. The RAM is the part nobody measures, and it's the part that fails.

The measurement says the failure mode is not "one big service." It's the long tail. Fifty small services, each holding 10–100 MB, add up to a third of the box. The fix is not "buy more RAM" (though that helps). The fix is knowing the tail is real, so when the box gets tight you know which of the fifty to trim, not which of the two to kill.

What I would do now

If I were building this box today:

RAM: 64 GB. The measured peak is 15.9 GB of application memory; 64 GB is 4× peak. 32 GB would be 2× peak — workable, but the photo server and the CI runner would start fighting.
The rule: size RAM for the peak of the long tail, not for the biggest single service. The biggest service is 7.7 GB. The tail is 22.7 GB. The tail is the size.
The audit: a per-service memory panel, current and 7-day peak, with the kernel share separated out. The number that matters is not "the box is at 65%" — it's "the peak was 15.9 GB, and here are the ten services that were up at the peak."

The homelab monitor I use has exactly that panel. The 7-day peak per service, the current per-process state, and the kernel's non-reclaimable share, in one view. It's the difference between "the box is full" and "here's what's full, and here's what to trim."

Verdict

54 containers, 93 services, 64 GB of RAM: 41.6 GB in use (65%), 15.9 GB peak over the week, zero OOM kills. The RAM is not eaten by one big service — it's eaten by a long tail of fifty small, resident services that add up to 22.7 GB. The GPU tenants hold the most per service, but the tail holds a third of the box by itself. Size the RAM for the peak of the tail, not for the biggest single service, and the box stops being a guessing game.

The Local LLM Lab is a series of measured experiments on a home GPU stack. Every number in this article was pulled from 7 days of per-service memory telemetry on the box, not estimated. Next: the one failed service on a 93-service stack — and how the uptime checks keep the rest alive.

Your homelab, legible to AI agents — a read-only MCP server that rides along with your monitor

Arsen Apostolov — Sun, 18 Oct 2026 08:43:30 +0000

i keep a monitoring container on my homelab — it reads nvidia-smi, the docker api, systemd and /proc, and turns it into one page. the boring kind of dashboard: what's using the gpu, what it costs, what's filling the disk.

last month i started running agents on the box, and i hit a wall that i think most people will hit: the agent couldn't see my homelab. it could run commands, but it couldn't see — which model is holding the 3090, why the temperature is climbing, what the month cost in energy. i'd have to paste screen output into the chat by hand, and the agent was guessing about the rest.

the data was already there. the monitor was reading all of it, every few seconds, into a sqlite database. what was missing was just a standard way to hand it over.

so i added a read-only MCP server to the same container. it comes up on port 9810, and the whole setup is one line:

claude mcp add --transport http homelab http://YOUR-HUB:9810/mcp

nineteen named tools, all read-only: hosts, containers, services, per-card gpu state and which process is on which card, per-process ram, the model servers, installed models, energy cost, benchmark runs, disk treemaps, history, alerts. that's the same coverage the dashboard has — the server is just the same database with an MCP face on it.

what this actually gets you:

"which model is eating vram right now, and on which card?" — it reads the per-card split, not one pooled number. on my 3x3090 box a 63gb model shows as 22.5 / 22.1 / 18.8.
"what did the last week of inference cost me?" — it has the power integration, so it's a measured number, not a guess.
"is anything down?" — the uptime checks and alerts are tools too.
"what's filling the disk on the pi?" — the treemap, on demand, over the same ssh connection everything else uses.

a few things i'm deliberately not doing:

no writes. both directions are question and answer. the agent can look around the lab, it can't restart anything. (the dashboard itself has controls, but the mcp surface is read-only on purpose — an agent that can see your lab is already trusted, and i didn't want to make it responsible.)
no extra install on the remote boxes. the monitor already polls them over ssh with nothing persisted, and the mcp server just reads what the monitor has.
no cloud. the server is on your lan, same as the dashboard. point a local ollama agent at it and nothing leaves the house.

the monitor is open source, mit, one container: github.com/SikamikanikoBG/homelab-monitor — and the mcp part is on by default (enable_mcp=0 turns it off). if you run a homelab and you're pointing agents at it, i'd be curious what you'd ask it first.

My GPU Was Busy. I Finally Found Out Who Was Using It.

Arsen Apostolov — Mon, 12 Oct 2026 07:10:05 +0000

The Local LLM Lab — Part 10

One controlled experiment. One number. One verdict.

Every homelab has a moment like this: the GPU is at 100%, the fan is screaming, and you have no idea which of the twenty services on the box is doing it. You open nvidia-smi, it shows one process, you close it, and the mystery continues.

The problem is that nvidia-smi answers "which process is using the GPU" — and in a Docker stack, the process is the container runtime, not the service. The answer you actually want is "which service is using the GPU, and how much time did it take, and who called it."

I built the answer into my homelab monitor: caller→server attribution. Every sample of GPU utilisation is tagged with the service that owns the process, and for shared services (a transcription endpoint, an embedding endpoint) it's tagged with the caller — the thing that made the request. Over the last 7 days the attribution logged 1,055 samples. This is what it says about who eats a home GPU.

The setup

Card: NVIDIA RTX 3090, 24 GB, power limit 390 W
Shared services on the card:
- Whisper ASR webservice — a transcription endpoint with an nginx proxy in front (my tools, the observability stack, CI, and a warmup job all hit the proxy)
- nomic-embed-text (Ollama) — an embedding endpoint used by the RAG pipeline
Attribution: every GPU sample over 7 days is tagged with (a) the service owning the process and (b) the caller for shared endpoints. Total: 1,055 samples, ~1.1 million seconds of GPU time attributed.

The key design decision: the caller is recorded from the request, not from the process. When my assistant core sends an audio file to the whisper proxy, the sample is tagged jarvis2-core → whisper-proxy, not nginx → whisper. That's the difference between "the GPU is busy" and "my assistant is the one using it."

The number

Over 7 days, 47% of the GPU's logged time went to one caller: my own assistant core (4,970 seconds of 10,555 total attributed seconds).

Caller	Seconds (7d)	Server	% of total
jarvis2-core (my assistant)	4,970	whisper-proxy	47.1%
whisper-proxy (shared endpoint, direct)	920	whisper service	8.7%
langfuse worker (observability)	910	whisper service	8.6%
langfuse minio (object storage)	860	whisper service	8.1%
whisper-warmup (keep-alive job)	360	whisper service	3.4%
gh-runner (CI)	360	whisper-proxy	3.4%
jarvis-server + watchtower	410	whisper-proxy / service	3.9%
firefly importer	80	whisper service	0.8%
ollama-embed callers (CI, assistant, watchtower)	210	ollama-embed	2.0%
UptimeKuma (health checks)	20	whisper service	0.2%

The single biggest consumer of the GPU is not a model, not a benchmark, not a training run. It's my assistant transcribing the audio it records. Voice memos, meeting recordings, the audio attached to the traces the observability stack ingests — all of it flows through the Whisper endpoint, and all of it is attributed to the caller that sent it.

What the attribution changes

Without caller attribution, the 7-day story of this GPU is: "the Whisper service used 10,555 seconds of GPU time." That's a number. It doesn't tell you anything you can act on.

With caller attribution, the story is:

My assistant is the tenant. 47% of the GPU's time is my own tooling. The service exists for it, and it uses it more than everything else combined. That's the design working as intended — and it's also the reason the service is resident 24/7 instead of on-demand.
The observability stack is the second tenant. Langfuse's worker and MinIO together take 17% of the GPU's time. They transcribe the audio attached to the LLM traces they store. That's a cost nobody budgets for: the tracing stack is using the GPU to do its job, on top of the LLM it's tracing.
CI is a real, small tenant. The GitHub runner takes 3.4% — test fixtures that get transcribed during CI runs. It's small, but it's unexpected small. Nobody thinks about CI touching the GPU.
The warmup job is 3.4%. A keep-alive job that pings the endpoint every few minutes so the model stays warm. That's the "keep it warm" cost made visible — 3.4% of the GPU's week, doing nothing but staying ready.

The verdict from the numbers: the GPU is not "busy" in the way the alarm implies. It is busy for specific, named reasons, and the reasons add up to a stack that is working, not failing. The alarm that says "GPU at 100%" is a symptom. The attribution is the diagnosis.

The counterargument: attribution is overhead

The fair objection: tagging every GPU sample with a caller is extra bookkeeping, and in a hot loop it's overhead. The attribution has to be cheap enough that it doesn't slow the inference it's measuring.

The implementation I use is cheap by design: the tag is set at the request boundary, not at the sample boundary. When a request comes in, the caller is recorded once and attached to every sample the request produces. The per-sample cost is a field read, not a lookup. Over 1,055 samples the overhead is below the resolution of the power meter — you cannot measure it against a 2-hour bucket.

The other objection: attribution only works for services that go through a request boundary. A model that's loaded and used in-process (a training run, a direct script) has no caller to tag. For those, the service-level tag is the best you can do, and that's still better than nothing. The attribution is a spectrum: caller-level for request-driven services, service-level for the rest.

What I would do now

If I were building a homelab monitor today, the GPU panel would have three layers:

The card level: utilisation, VRAM, power, temperature. The thing nvidia-smi shows.
The service level: which container owns the process, and how much VRAM and time it holds. The thing Docker's stats show, but per-GPU.
The caller level: for shared endpoints, which client made the request that produced each sample. The thing nothing shows, and the thing that answers "who is using my GPU."

The third layer is the one that turns a dashboard into a tool. When the GPU spikes, you don't open nvidia-smi and guess. You look at the attribution and see the name. When a service misbehaves (a runaway job, a retry storm), the caller list shows it in the same view. The number that matters is not "the GPU is at 100%" — it's "jarvis2-core took 4,970 seconds this week, and here's the request that did it."

Verdict

One week of caller-level GPU attribution: 47% of the card's time went to my own assistant, 17% to the observability stack, 3.4% to CI, 3.4% to a warmup job that does nothing but stay ready. The GPU was never "mysteriously busy." It was busy for named reasons, and the attribution is the only view that shows them. If your homelab GPU spikes and you don't know why, the fix is not a faster card — it's a caller tag on every sample.

The Local LLM Lab is a series of measured experiments on a home GPU stack. Every number in this article was pulled from 7 days of caller-level attribution on the card, not estimated. Next: the 55 containers and 92 services that share this box with the GPU — what actually eats RAM when the machine has 64 GB of it.

I built my family a private Chess.com — one Docker container, Stockfish plus your own LLM

Arsen Apostolov — Mon, 12 Oct 2026 03:55:29 +0000

My kids play chess on Chess.com. It is a great product, but the account, the cloud and the ads are not something I want to hand to a nine-year-old. So I built Patzer: a self-hosted, open-source alternative that runs on a Pi, a NAS or an old laptop.

What it does:

Game Review — pull your public Chess.com or Lichess games, or paste a PGN. Bundled Stockfish classifies every move (Brilliant / Great / Inaccuracy / Mistake / Blunder), gives accuracy, estimated Elo and an eval graph, plus a "what's the threat?" probe that answers in plain words.
Play vs Bot — Stockfish at seven named tiers from Kid to Stockfish max, all standard time controls, with kid-mode blunder warnings.
Play vs Friend — real-time PvP between profiles on the same server over WebSocket: draw offers, takebacks, one-click rematch.
AI Coach — point it at your own Ollama or vLLM host. The coach explains why a move works or fails, shows the better move, and connects the mistake to your own recent games ("4 of your last 10 games: leaving a piece unprotected"). It never contradicts the engine: an answer that praises a blunder is caught before you see it.
Family-ready — multi-user with an admin console, per-profile language, kid mode, "horsey" piece names for the youngest profiles.

The design choice I keep coming back to: the LLM never analyses. The server works out the facts — what a move allows, hangs or wins, the opponent's best answer, your recurring mistakes — and the model only turns that into coaching. So a small local model is enough, and a hallucinated piece is impossible.

One container:

docker run -d -p 8800:8800 -v patzer-data:/app/data \
  --name patzer ghcr.io/SikamikanikoBG/patzer:latest

Everything runs on your LAN. Games, profiles and settings never leave home. MIT licensed, English-first with Bulgarian, Spanish, German and Russian out of the box.

Repo: https://github.com/SikamikanikoBG/patzer

I Put Three AI Services on One 24 GB GPU. It Used 17.8. Here's What Fits.

Arsen Apostolov — Mon, 12 Oct 2026 03:49:09 +0000

I Put Three AI Services on One 24 GB GPU. It Used 17.8. Here's What Fits.

The Local LLM Lab — Part 9

One controlled experiment. One number. One verdict.

The "how much VRAM do I need" advice on the internet comes in one flavour: a spec sheet. "27B in FP8 needs 28 GB. 70B in Q4 needs 42 GB. Buy a 48 GB card."

That's the answer for one model. Nobody tells you what happens when you try to run your actual stack — a transcription service, an embedding model, a small chat model — on one card. The answers you find are guesses, because nobody has measured a multi-tenant GPU.

I have. For the last week my RTX 3090 (24 GB) has been running three AI services at once: a Whisper ASR webservice, a small TTS voice model, and an embedding model for a RAG pipeline. I pulled the full VRAM history and the per-tenant attribution. This is what the card is actually holding, and what the number says about sizing a home GPU.

The setup

Card: NVIDIA RTX 3090, 24 GB (24,576 MB), power limit 390 W
Tenants (all resident, all Docker):
- Whisper ASR webservice (WhisperX, large-v3) — a transcription endpoint my own tools call for voice memos, meetings, podcast clips
- A small TTS voice model — text-to-speech for the same assistant stack
- nomic-embed-text (Ollama) — embeddings for the RAG pipeline
Telemetry: 7 days of VRAM / power / utilisation at 2-hour resolution, plus per-caller attribution
Everything else on the box: 55 containers, 92 systemd units, a 56-thread Xeon, 64 GB of RAM. The GPU sees only the three tenants above.

The three services were chosen because that is the actual shape of a home AI stack: one big-ish resident model (ASR), one medium resident model (TTS), one tiny resident model (embeddings). No chat model. The 27B chat model lives on a second machine with two 3090s — that's a different article.

The number

The card is holding 17.8 GB of its 24 GB — 72% — with three models resident, and the peak over the week was 23.4 GB. It never spilled a byte to system RAM.

Tenant	Avg VRAM (7d)	Peak VRAM (7d)
Whisper ASR (large-v3)	7,871 MB (~7.7 GB)	10,558 MB (~10.3 GB)
TTS voice model	4,876 MB (~4.7 GB)	4,964 MB (~4.8 GB)
nomic-embed-text	4,633 MB (~4.5 GB)	4,668 MB (~4.6 GB)
Total (measured, 7d)	~17,380 MB (~17.0 GB)	23,379 MB (~22.8 GB)

The measured current state at the time of writing: 17,775 MB used of 24,576 MB — 72% of the card, with the three tenants sitting at 7.87 GB + 4.96 GB + 4.67 GB = 17.5 GB, plus the CUDA context overhead of the card itself.

The peak of 23.4 GB is the interesting number. It happened when the Whisper service received a batch of long audio files at the same time the TTS model was rendering a long response. The two big tenants spiked together, the embedding model held its 4.6 GB, and the card came within 1.2 GB of the ceiling. It held. No spill, no OOM, no crash — because the peak was a few seconds, not a sustained load.

What each tenant is actually holding

The "paper VRAM" of a model is its weights. The measured VRAM is weights plus working set plus framework overhead. Here's the gap, per tenant:

Whisper large-v3. The weights in FP16 are ~3.1 GB. The measured average is 7.7 GB — 2.5× the weights. The difference is the encoder's audio feature buffers (long files and queued files hold more), the VAD and alignment models WhisperX runs alongside, and the CUDA context. The peak of 10.3 GB is a batch of long files in flight.

The TTS voice model. Weights ~4.5 GB, measured 4.7–4.8 GB. This one is close to its paper number because TTS is stateless between requests: no long context, no queue buffers. It's the most "honest" tenant on the card.

nomic-embed-text. A 137M-parameter embedding model. Paper weights ~0.3 GB. Measured 4.5–4.7 GB. That's the Ollama runtime, its CUDA context, and the KV space it pre-allocates for embedding batches. A model that "needs 300 MB" needs 4.6 GB when it's a resident service. That's the number people never quote.

The pattern: the measured VRAM of a resident service is 1.2–2.5× the weight size, and the multiplier is biggest for the models with the most state. If you're sizing a card, budget for the measured number, not the weights.

Who is calling it

The 7-day attribution shows why the Whisper tenant is the biggest consumer of time on the card, even though the TTS model is the bigger spike contributor:

Caller	Seconds on the card (7d)
My own assistant core (via the whisper proxy)	4,970
Whisper proxy (shared endpoint)	920
Langfuse worker (observability stack, transcribes attached audio)	910
Langfuse MinIO (object storage, transcribes on ingest)	860
Whisper warmup (keep-alive job)	360
GitHub runner (CI jobs that transcribe test fixtures)	360

The card is a shared service, not a single app. Six different callers hit the Whisper endpoint over the week. The TTS and embedding tenants have one caller each (the assistant stack). The attribution is what turns "the GPU is busy" into "here is who is busy" — and it's the difference between a dashboard you glance at and one you can act on.

The counterargument: 72% is a lot

The fair objection: a card at 72% of its VRAM with a 1.2 GB headroom at peak is a card that will fail the day a bigger file comes in. Headroom is insurance, and 12% is thin insurance.

True. But the measurement says what the insurance was worth over a week of real use: the peak was a 3-second event, and the card absorbed it. The sustained state is ~17 GB, which leaves 7.5 GB of headroom — 31% of the card. The 23.4 GB peak is the worst case that actually happened, not the worst case that is possible.

The honest version of the headroom argument: size for the biggest sustained working set you will actually run, plus 50% headroom. Not for the biggest spike you can imagine. If your stack's sustained working set is 17 GB, a 24 GB card is 1.4× sustained — enough. If you add a 7B chat model to the same card (sustained ~8 GB), the sustained working set becomes 25 GB and the 24 GB card is done. That's the actual failure mode: not a spike, but a new tenant.

What I would buy now

If I were building this stack today:

The hub card (ASR + TTS + embeddings): 24 GB. Sustained working set 17 GB, peak 23.4 GB. A 16 GB card is not enough — the TTS model alone would crowd the Whisper service. A 32 GB card would be comfortable but costs more and runs hotter for a workload that fits.
The chat card(s): sized for the chat model. The 27B in FP8 needs two 24 GB cards (or one 48 GB card). It does not share the hub card.
The rule: one card per class of workload — resident services on one, the big chat model on another. The measured working set of each class, plus 50% headroom, is the size.

The 24 GB 3090 is still the best price-per-GB on the used market, and that's why it's everywhere in home stacks. But "best price-per-GB" is an argument for the chat machine. For the hub, the argument is "fits the measured working set of my resident services with room for one more tenant" — and for my stack, that's 24 GB, not 16 and not 48.

Verdict

Three AI services on one 24 GB GPU: 17.8 GB sustained, 23.4 GB peak, zero spills over 7 days. The measured VRAM of a resident service is 1.2–2.5× its weights, and the peak is a few seconds, not a sustained load. Size the card for the sustained working set plus 50% headroom, and keep the big chat model on its own card. The spec sheet tells you what one model needs. The meter tells you what your stack needs.

The Local LLM Lab is a series of measured experiments on a home GPU stack. Every number in this article was pulled from 7 days of VRAM telemetry on the card, not estimated. Next: who is actually eating the GPU — the caller-level attribution that turns "the GPU is busy" into a list of names.

Qwen3.8-Flash-Next (125B) on Three RTX 3090s at 80 Tokens/s: Teaching llama.cpp Which Experts Matter

Arsen Apostolov — Sat, 26 Sep 2026 16:02:31 +0000

TL;DR: Qwen3.8-Flash-Next is a 125B mixture-of-experts model: on paper it beats the Qwen3.8-27B I run in production everywhere, by +16.5 points on agentic coding. It does not fit in the 72 GB of VRAM my three RTX 3090s have, and stock llama.cpp, spilling a quarter of the experts into system RAM, runs it at 23 tokens/second. Three changes get it to 80 tokens/second, entirely in VRAM:

Sort the experts by how often they're used. They are wildly unequal: the busiest 25% handle 52% of the work.
Store them at three precisions. Popular experts keep more bits and rare ones get squeezed harder, so the whole model fits on the GPUs.
Patch llama.cpp (~400 lines) so one routing decision drives three expert tables at once, then add speculative decoding on top.

Quality stays close to the 8-bit reference: +1.9% perplexity, 91% same next token, GSM8K 95.5% (the 27B scores 95–96.5% on the same harness). There's also a 256k-context profile that finds a random code hidden 173,000 tokens deep. The trade-offs are real, and I've listed them. Everything is open: patches, tools, raw measurements.

Update, 26 Sep 2026, after the first day behind my assistant: the 80 tok/s holds for short prompts. In real agent use, with ~25k-token prompts, sampling and thinking on, the model decodes at 30–45 tok/s. My first deployment also had a caching mistake that cost ~45 s before every answer; a server config change fixed it (now 0.7 s). Details in the "real-world update" section below.

the use case

My box, vader, runs Qwen3.8-27B around the clock: two RTX 3090s, vLLM, about 135 tokens/second. It's the first local model I actually trust with agent work, and it does most of the work my assistant does.

Then Qwen shipped Qwen3.8-Flash-Next, the preview of their Qwen4 architecture. Their own model card puts it ahead of my 27B on almost everything I care about:

benchmark (Qwen's card)	Flash-Next	3.8-27B	gap
DeepSWE (agentic coding)	58.7	42.2	+16.5
JobBench (professional tasks)	55.7	33.4	+22.3
SWE-bench Multilingual	81.0	73.8	+7.2
Toolathlon (tool use)	73.5	67.1	+6.4
HLE	35.9	30.8	+5.1
GPQA Diamond	91.7	89.2	+2.5
IFBench	81.3	79.5	+1.8

So the goal was simple to state: the smarter model, on the same three cards, with decode and time-to-first-token that still feel interactive. No new hardware. I told my AI engineer (Claude Code, more on that at the end) that "no" was not an acceptable answer.

the box

GPUs: 3x RTX 3090, 24 GB each: 72 GB of VRAM. No NVLink, and no PCIe peer-to-peer (the board has no Resizable BAR), so the cards can't talk to each other directly.
CPU: dual Xeon E5-2660 v4, 56 threads.
RAM: 60 GB. Only one 32 GB stick per CPU socket, so it's single-channel, and that matters a lot below.
Disk: one 1 TB NVMe.

a small dictionary before we start

If you know what a KV cache is, skip this. If you don't, it's all you need for the rest of the article.

Parameters / weights. The numbers a model learned. "125B" means 125 billion of them. Each one normally takes 2 bytes, so 125B parameters is ~250 GB, far more than 72 GB of VRAM.
VRAM. The GPU's own memory. It's about 20–50x faster than system RAM on this box. A model runs fast only if what it needs for each word lives in VRAM.
Quantization. Storing each weight with fewer bits (8, 4, 3…) instead of 16. Like saving a photo as a smaller JPEG: smaller file, a little less detail. "Q8_0", "Q4_K", "IQ3_S", "MXFP4" are llama.cpp's names for different bit-budgets and recipes.
Mixture of experts (MoE). Instead of one giant network, each layer has many small "expert" networks, 512 of them here, and a router picks the best 10 for each word. So the model knows 125B parameters' worth of things but only computes with about 6B per word. That's why it can be fast, and also why it's huge.
Token. A word or word-piece. Decode speed (tokens/second) is how fast the answer streams out.
Prefill / time to first token (TTFT). Before answering, the model reads your whole prompt. That's prefill, and TTFT is how long you wait before the first word appears.
Context / KV cache. How much text the model can hold in mind at once ("256k context" is about 500 pages), and the memory it uses to remember it.
n-gram embedding. New in this model: a 51 GB lookup table indexed by 2–3 word phrases. It's only ever looked up, a few rows per word, so it can live on the NVMe drive.
Speculative decoding / MTP. A small built-in "draft head" guesses the next few words, and the big model checks all the guesses in one pass. Correct guesses are free words.
KL divergence (KLD). How far the compressed model's word probabilities drift from the original's. 0 means identical; lower is better. It's the most sensitive quality meter there is.
llama.cpp / GGUF. The inference engine I patched, and its model-file format.

the problem, in one picture

In 16-bit the model is 360 GB. Even the popular 4-bit build (unsloth's UD-Q4_K_XL) is 111 GB: 72 GB of experts, 27 GB of n-gram table (which can stay on disk) and a few GB of everything else. The experts alone don't fit in 72 GB of VRAM once the rest of the model and the context need room too. So llama.cpp does the sensible thing: it puts the overflow experts in system RAM and computes them on the CPU.

On this machine that means 23 tokens/second. With one memory stick per socket, the CPU reads those experts about ten times slower than a GPU would. And for every word, the work bounces between the GPUs and the CPU dozens of times, so each GPU waits.

step 1: the obvious setup, and the first trap

The first run with stock settings also showed prefill all over the place (47 to 306 tokens/second on the same 4k prompt) and decode sagging to 10 tok/s. Thread states told the story: the main thread sat in D (waiting on disk). llama.cpp memory-maps the model file, loading the GPU part streams 70+ GB through the page cache, and the CPU-side experts kept getting evicted and re-read from NVMe.

--load-mode none loads the CPU-side weights into RAM and keeps them there. Also, don't download a 188 GB file while you benchmark: my download filled the page cache and pushed the server into swap. With both fixed, prefill doubled (658–792 tok/s) and decode steadied at 23–26 tok/s.

Speculative decoding (MTP) barely helped: 24–29 tok/s. The reason matters for everything that follows. A draft of 5 words means the big model checks 6 words at once. Each word picks 10 experts, so one check can touch up to 60 different experts per layer instead of 10, and the slow ones in CPU RAM get hit harder. As long as experts live in CPU RAM, nothing else will make this fast.

step 2: the experts are not equal

Here's the observation the whole project rests on. llama.cpp's calibration data (the importance matrix unsloth publishes with their quants) records how often each of the 24,576 experts was picked on real text:

In the average layer, the busiest 25% of experts handle 52% of the tokens, and the busiest 80% handle 95%. Stock llama.cpp can only place whole layers of experts: a layer's 512 experts are one tensor, all on the GPU or all on the CPU. It offloads a quarter of the experts, including popular ones, so a quarter of the expert work lands on the slow path.

step 3: split every layer into hot and cold

The first patch splits every layer's expert tensor in two at load time:

hot experts go to the GPUs;
cold experts go to CPU RAM;
the router is permuted so its choices land in the right half.

With the same 19% of expert bytes on the CPU, the cold experts now serve only 4.5% of routed tokens instead of ~19%, about 4x less slow-path traffic.

Getting it right took four bugs, each instructive enough to list in its own section below. The result is 30–32 tok/s with correct answers. That's +30%, but it's not the leap it should be, so I profiled it:

where one decoded word spends its time
kernel launches	~4,300
dense weights (attention, DeltaNet, shared expert), 8-bit	~4.9 ms
routed experts	~2.5 ms
~3,500 tiny element-wise kernels	~6–7 ms
GPU↔CPU synchronisations	~390

The CPU was barely reading any experts anymore; it was the round trips that cost. Every layer handed work to the CPU and waited for the answer. Then a cheap experiment: drop the cold experts entirely (quality garbage, speed only) and see how fast it goes fully on GPU: 52 tok/s, and 86–88 tok/s with speculative decoding. That was the target. The question became: how do you fit every expert in VRAM without wrecking quality?

step 4: three shelves, all on the GPU

The idea: if some experts do most of the work, give them more bits and squeeze the rarely-used ones harder. Everything still fits in VRAM, and most tokens still see a well-preserved expert. Think of a library that keeps its bestsellers in hardcover and prints the rarely-borrowed titles as compact paperbacks, so the whole collection fits in the building.

Two pieces make it work.

An offline re-packer (tools/expert_tiers.py). It starts from the 8-bit model (188 GB) and uses the importance matrix for both how often each expert is used and which inputs matter inside it. A greedy planner fills a VRAM budget: each byte goes to the expert where it removes the most expected error. Every expert is then re-quantized with llama.cpp's own quantizers, and the file is written with the experts reordered hottest-first as three tensors per layer. One wrinkle: the down-projection matrices have rows 640 wide, which the fancy 2–3-bit formats can't handle (they need multiples of 256). So they get their own ladder: IQ4_NL / MXFP4 instead of IQ4_XS / IQ3.

A patched llama.cpp (patches/). mul_mat_id, the operation that runs "each word through its chosen experts", learns to take an id range. Each shelf's matrix multiply sees the router's full choice list, computes only the ids that fall on its shelf, and writes zeros for the rest. The three results simply add up. That meant touching the CPU kernels, three CUDA paths (decode, prefill, and the fused gate+up+activation kernel) and the loader. On top sits Qwen's multi-token-prediction draft head, from a not-yet-merged llama.cpp PR, re-quantized to 4 bits so it fits too.

Result, in one chart:

the four bugs, briefly

For people who'll try this, and because two of them are delightful.

Duplicate experts crash the CUDA MoE kernel. My first version pointed "not on this shelf" at expert 0. A word with two cold experts then had expert 0 twice, and CUDA's expert-grouping helper assumes each expert appears at most once per word. It silently dropped a slot, and a later kernel read an unwritten index. Fixed by teaching the kernels a real "skip".
A flag stored where a precision setting lives. I marked the skip-capable nodes in op_params[0], which llama.cpp already uses for the accumulator precision. The flag was quietly overwritten. It moved to slot 7, with a magic value.
The unsigned ternary. ids ? ids[i] : blockIdx.x, where blockIdx.x is unsigned. C++ promotes the whole expression to unsigned, so my -1 ("skip") became 4,294,967,295 and the kernel read 4 billion rows past the buffer. compute-sanitizer found it in one run.
Fused kernels hide the tag. For decode, CUDA fuses gate, up and activation into one kernel whose "destination" is the activation node, not the matrix multiply that carries my id-range tag. The first id-range build printed //////////. The kernel now looks up the tag on its source node.

how much quality did it cost?

The honest meter is KL divergence against the 8-bit model over 24,576 tokens of Wikipedia text:

model	expert size	fits in VRAM	perplexity vs 8-bit	KLD	same top token
unsloth UD-Q4_K_XL (stock)	71.7 GiB	no, 25% in RAM	+0.9%	0.045	93.7%
tiered, 128k profile	53.0 GiB	yes	+1.9%	0.091	91.1%
uniform IQ3_S / MXFP4, same size	52.2 GiB	yes	+3.0%	0.095	91.0%
tiered, 256k profile	49.5 GiB	yes	+3.2%	0.113	90.1%

What I take from it:

Fitting in VRAM costs quality, and there's no way around that. 20 GB less expert weight roughly doubles the KL divergence of the stock 4-bit build. That's physics, not a bug.
At equal size, tiering beats uniform: perplexity drift 1.9% vs 3.0%. But the gain is smaller than my planner predicted: its error model is crude, and at this size the total byte count dominates. There's more to win with a better tier recipe.
On a task, it holds up. GSM8K (200 grade-school maths questions, greedy, no thinking) scores 95.5%. The 27B's published number on the same harness is 95.0–96.5%. GSM8K is saturated, so it mainly proves nothing broke. The model card's gains on agentic work are where the upgrade should show.

speed vs context, and the 256k profile

Both profiles start at ~80 tokens/second on a short prompt and slow down as the prompt grows. The per-word attention and indexer work grows with context, and the draft head's guesses get a little worse too. Typical numbers, averaging two runs:

prompt	128k profile: decode / time to first token	256k profile: decode / time to first token
4k tokens	82 tok/s / 7–9 s	67 tok/s / 7–8 s
15k tokens	66 tok/s / 18–22 s	58 tok/s / 19–22 s
30k tokens	58 tok/s / 35 s	54 tok/s / 38 s
61k tokens	53 tok/s / 76–79 s	51 tok/s / 82 s
95k / 122k tokens	39 tok/s / 140 s	35 tok/s / 190 s

Prefill runs at 540–860 tokens/second. That's the weak spot next to the 27B's ~1,300, and it's where I'd look next: it's also the part that makes a 120k-token prompt a three-minute wait.

The 256k profile needed two more compromises. The KV cache drops to 8-bit, and experts shrink to 49.5 GiB, because the sparse-attention indexer's scratch memory grows with context and pushed a card out of memory at ~200k on the first try. After rebalancing layers across the cards, it read 173,692 tokens of Wikipedia with a random 10-character code hidden at 45% depth, and returned the code: exactly right (6BC5XYE8FS). A second run with the code at 90% depth of 112,711 tokens was also exact. It took 6.1 minutes (472 tokens/second) to read that much. Long-context prefill on this architecture is still the slow part.

what it costs to run

Measured at the cards (nvidia-smi, 5 Hz, three 1,024-token generations per profile): the three 3090s draw ~540 W together while decoding and ~124 W at rest with the model loaded. That's 7.5–8.4 joules per token, 2.1–2.3 kWh per million tokens. On my tariff (0.30 BGN/kWh day, 0.18 night) a million generated tokens cost 0.63–0.70 BGN in daytime, 0.38–0.42 BGN at night, about $0.35–0.40. The 27B costs 0.21–0.34 BGN for the same million, so the bigger brain costs roughly twice as much per word, all three cards included. It's still coffee money.

real-world update: day one behind my assistant

Benchmarks are one request with a short prompt and greedy decoding. My assistant is none of those things, so after a day of it running Jarvis, here is what the server logs say about real requests:

	a real assistant turn	the benchmark
prompt	~25,000 tokens (system prompt, tools, memory)	41 tokens
sampling	temperature 0.7, thinking on	greedy
tokens per verify step (draft acceptance)	2.6 (39%)	3.4 (62%)
decode	41–45 tok/s (30–45 is what it feels like)	80 tok/s

Decode: 30–45 tok/s, not 80. Two things compound. The prompt is deep, and every decoded word pays attention over it; the context chart above already shows ~50 tok/s at 60k. And the draft head guesses sampled text, tool-call JSON and reasoning worse than greedy prose, so speculative decoding saves less. That's the honest number for agent work on this box. For comparison, the 27B benchmarks at ~135 tok/s on this box, so the upgrade is intelligence, not speed.

Time to first token: the mistake was mine. My assistant runs several roles: chat, planner, classifier, triage, background jobs. Each has its own ~25k-token prompt prefix. I deployed the server with one slot (-np 1), so every time the role changed, the cached prefix was thrown away and 25,000 tokens were re-read from scratch: ~45 seconds before every answer. The fix is four slots sharing one KV pool, so every role keeps its prefix warm:

returning to a 14k-token prompt after another role ran	before (`-np 1`)	after (`-np 4 -kvu`)
time to first token	20–45 s	0.6–0.7 s

It isn't free: four slots need a little more memory, so the shared pool dropped from 128k to 96k tokens (128k with four slots ran out of VRAM). A single request can still use all 96k.

Schedule the background jobs apart. A model at 30–45 tok/s with 25k-token prompts spends minutes per agent task. My assistant had 13 scheduled jobs between 06:30 and 09:00 on Mondays, four of them at the same minute. I've spread them 30 minutes apart (05:00 to 11:30) so they stop queueing behind each other. I'll report after a week whether that's enough.

Images and video work too, once the vision projector is loaded (--mmproj, on the GPU with the most free memory: 4 s per photo, against 61 s on the CPU) and the container has ffmpeg for video (a 5-second clip in 6.4 s). The repo has the exact launcher.

what I'd not claim

It's single-user. Every speed number here is one request at a time, and real agent turns decode at 30–45 tok/s, not 80. The 27B on vLLM batches many users far better.
The 27B is still faster. About 135 vs 80 tok/s decode, and roughly 1,300 vs 540–860 tok/s prefill. The 125B is smarter, not quicker.
It uses all three cards. You can't also keep the 27B running.
The quality numbers are wikitext + GSM8K. They say the compression is gentle. They don't prove the agentic gains survive intact; that needs agent benchmarks I haven't run yet.
The tier planner is a heuristic. The measurements say it helps; a calibrated one would help more.

reproduce it

Everything is in SikamikanikoBG/qwen38-flash-next-3x3090: both llama.cpp patches, the re-packer, the benchmark scripts, and every raw number behind the charts. The short version:

# llama.cpp at 81bc6b8 + MTP PR #28243 + the tiering patch
git apply patches/0001-qwen4exp-mtp-pr28243.patch patches/0002-tiered-experts-hot-cold-split.patch
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 && cmake --build build -j

# re-pack the 8-bit model into three precision shelves (53 GiB of experts)
python tools/expert_tiers.py write --src Qwen3.8-Flash-Next-Q8_0-00001-of-00006.gguf \
  --imatrix imatrix_unsloth.gguf --budget-gib 53 \
  --gu Q6_K,IQ4_XS,IQ3_XXS --dn Q8_0,IQ4_NL,MXFP4 --out fn-tier53.gguf

# serve: all layers on GPU, 4 slots sharing a 96k KV pool, vision, MTP drafting 4 tokens
llama-server -m fn-tier53-00001-of-00002.gguf -ngl 99 -ts 16,16,16 -c 98304 -np 4 -kvu \
  -b 1024 -ub 256 -fa on --jinja \
  --mmproj mmproj-F16.gguf -mmdev CUDA1 --image-max-tokens 1024 \
  -md mtp-shared-iq4.gguf --spec-type draft-mtp --spec-draft-n-max 4

credits

Qwen, for the model and an architecture that rewards this kind of work.
unsloth, for the GGUFs, the MTP draft files and the importance matrix.
llama.cpp and ggml, plus the authors of the Qwen4Exp MTP PR and the sparse-attention decay issue.
syv-ai/qwen38-27b-rtx3090, whose rigor on the 27B set the bar for measuring this.

This project, from the patches and the tools to the measurements and this write-up, was done with **Claude Code* acting as my AI engineer, working on my hardware under my direction. The numbers are from vader, the raw data is in the repo, and the mistakes are both of ours.*

Qwen3.8-27B on One RTX 3090 vs Two: +20% Decode, +14% Cold Prefill, and 3x on Cached Prompts

Arsen Apostolov — Wed, 23 Sep 2026 03:27:37 +0000

TL;DR: Qwen3.8-27B — the dense 27.8B that dropped on August 14 — fits on a single RTX 3090 at W4A16 and decodes at ~125–155 tokens/sec. Splitting it across two cards with tensor parallelism buys roughly 20% more decode (noisy: +13% in one run, +22% in the next) and 14% faster prefill on average on a cold prompt (23% at the top of the ladder: an 8,600-token prompt takes 8.5 s to first token on one card and 6.9 s on two). On a prompt the server has already seen, the pair answers 2–3x faster — but that is the prefix cache, not the cards, and the script below measures the two separately. At the API list price of $3.00 per million output tokens, one million tokens cost 0.21–0.34 BGN of electricity, about $0.12–0.20, on the box that serves them.

the box

vader is the machine I run this on: three RTX 3090s in an ASUS Z10PE-D8 WS, dual Xeon E5-2660 v4 (56 threads), 60 GB of RAM, Ubuntu 26.04. It has been my inference box since the Whisper experiments, and since July it has also carried the power-limiting work (the meters post is where those numbers live). Two of the three cards are in this test; the third sits idle.

The model under test is the one that is currently loaded on it:

	single-card	TP2
container	`qwen38-27b-single-1`	`qwen38-27b-tp2-single-1`
image	`ghcr.io/syv-ai/qwen38-27b-rtx3090`	same, `:latest`
weights	Qwen3.8-27B-W4A16-AutoRound-fast	same
cards	1x 3090 (22.6 GB VRAM)	2x 3090 (21.7 GB each)
max context	131,072	262,144
port	18021	18022

Both are vLLM, both have been up two days, both report healthy, and both run with prefix caching on — which is why every number below comes in two flavours, cold and warm. The W4A16 quantization is what puts a 27.8B dense model inside one 24 GB card — 22.6 GB of it, with the KV cache budget for the rest of the 128k window coming out of what is left.

the method

No benchmark harness, no vllm bench — a 150-line script (bench_qwen38_v2.py) that sends a real chat-completion request with a prompt of a given size, streams the answer, and times it. For each prompt size it runs three times and takes the median:

prompt sizes: calibrated so the server reports ~600, ~2,400, ~4,700, ~9,300 prompt tokens (the script probes the tokenizer first, then scales the prompt to hit the target)
output: max_tokens=256, temperature=0
measured: time to first SSE event, total time, and the server-reported token counts from stream_options.include_usage — not my own chunk counting
cold vs warm: every cold request carries a unique first line, so no cached prefix can match it; the warm pass repeats one identical request after a warm-up call

That last bullet is the one that decides whether a benchmark means anything. With prefix caching on — and it is on by default in vLLM — timing the same prompt twice does not measure prefill the second time, it measures a cache lookup. The tell is arithmetic: 8,600 tokens "prefilled" in 0.56 s is 15,000 tokens/sec, roughly ten times what two 3090s can actually push through a 27B model. If your TTFT numbers look like that, you are timing the cache. So every cold request here is salted with a unique first line, and the warm pass is run deliberately and reported in its own column.

One caveat: this is a single-request benchmark. vLLM's continuous batching is where production throughput lives, and I did not hammer it with concurrent requests. What I measured is the latency you feel as the one person using the box — which, for a homelab, is most of the time.

decode: the second card buys 15–30%, noisily

The single card holds 123–154 tokens/sec across the ladder; the TP2 pair runs 146–193. A 27B dense model at W4A16 on one 3090 is memory-bandwidth-bound in a way that does not care much about prompt size, because the KV cache for these sizes is small next to the weights. The pair is faster at every size but one (at 4,700 tokens the two came out level), and the average gain was +22% in this run against +13% in a run an hour earlier. Single-stream decode on these cards moves by about 10% between runs an hour apart, so I am not going to print a decimal: call it 15–30%, and expect the lower end.

That is a real number and a small one. For a 27B model, tensor parallelism across two cards is not the 2x you get from adding compute — the all-reduce between the cards eats most of it.

prefill: cold is the cards, warm is the cache

Two columns, two different questions.

Cold — a prompt the server has never seen — is prefill, and prefill is compute-bound: the single card does 1,100–1,220 tokens/sec of it, the pair 1,230–1,360. Time to first token climbs from 0.53 s to 8.5 s on one card as the prompt goes from 600 to 9,300 tokens, and from 0.50 s to 6.9 s on two. The gap grows with the prompt: 6% at 600 tokens, 23% at 9,300, 14% on average. That is what the second card buys you on a fresh long prompt.

Warm — the same prompt again — is the prefix cache doing its job, and here the pair really is 2–3x faster: 0.5–0.75 s flat across the ladder against 0.5–1.7 s on the single card, whose smaller KV pool and slower re-read of the uncached tail show at every size above 600 tokens. It is a cache number rather than a prefill number, and it is also the one you feel most often, because every follow-up turn in a chat is a warm request.

The routing rule that falls out: a long fresh prompt goes to the pair and saves about a fifth of its wait; a conversation stays on whichever lane it started on, so its prefix stays cached. For prompts under about 2,000 tokens the single card is within noise of the pair on both counts and draws one card's power.

what it costs

The box draws 620 W at the wall with both containers loaded and serving — 247 W on one TP2 card, 328 W on the other, 45 W on the single-card 3090 sitting at P8 with its weights resident. Those are the monitor's readings from the day the containers went up, not a measurement taken during these runs; the governor from the meters post is power-capping it, holding 79°C.

At the API's list price — $3.00 per million output tokens, DeepInfra's rate for Qwen3.8-27B as listed on llm-stats — one million generated tokens takes about 6,667 seconds at 150 tokens/sec. At 620 W that is 1.15 kWh. On the Bulgarian dual-rate tariff that is 0.34 BGN by day (0.30 BGN/kWh) or 0.21 BGN by night (0.18 BGN/kWh) — roughly $0.20 or $0.12. The API price is 15x to 25x the electricity, depending on when the box runs.

I am not going to pretend that makes the API a bad deal. The API does not cost you 620 W of heat in your room, it does not need a 60 GB machine, it does not need you to keep a W4A16 quant on a 24 GB card, and its $3.00 buys you their datacenter's power, cooling and idle capacity, not yours. But for a box that is already on and already paying its standby tax, every token it serves is almost pure margin. The electricity argument for local inference is not "it is cheaper than the API" in any total-cost sense — it is "the marginal cost of a token on hardware you already run is an order of magnitude below the list price."

the verdict

Qwen3.8-27B is the first 27B-class dense model I have run that I would put in front of real work on this box, and the numbers say why:

It fits. W4A16 on one 3090, 22.6 GB, with room for a 128k context window. The TP2 variant doubles the context to 262k by splitting the weights.
It is fast enough to feel like an API. 125–190 tokens/sec decode is faster than most hosted endpoints feel in a chat UI. The part you notice is a cold long prompt: several seconds on either lane.
The two-card setup is not 2x. It is roughly +20% decode and +14% cold prefill, and 2–3x on warm follow-ups because the pair keeps a bigger cache. Buy the second card for the context window and the cache, not for raw speed.
The marginal cost is 0.21–0.34 BGN, about $0.12–0.20, per million tokens against a $3.00 list price, on hardware that is already drawing its standby power.
Benchmark with the cache in mind. If a prefill number looks like 15,000 tokens/sec on consumer cards, it is the cache. Salt every request, or turn prefix caching off, and report which one you did.

The script, the raw JSON and the charts are in the repo: qwen38-bench. If you run a different quant or a different card, I want your numbers — the interesting part of this post is the shape of the curve, and the shape changes with the hardware.

vader: 3x RTX 3090 (two used here), dual Xeon E5-2660 v4, 60 GB RAM, Ubuntu 26.04, vLLM, W4A16-AutoRound-fast quantization.

I Benchmarked Qwen3.8-27B on Three RTX 3090s. The Second Card Bought Me 13% More Decode — and a 3x Faster Prefill

Arsen Apostolov — Sun, 20 Sep 2026 12:14:36 +0000

What Does a Local LLM Actually Cost per Month? I Read the Meters.

Arsen Apostolov — Sun, 20 Sep 2026 07:59:22 +0000

What Does a Local LLM Actually Cost per Month? I Read the Meters.

The Local LLM Lab — Part 5

One controlled experiment. One number. One verdict.

The question nobody answers in the local-LLM hype is the boring one: what does the electricity bill say?

Not "how many tokens per second." Not "how many GB of VRAM." The bill. The one that arrives on the first of the month and is the only number that actually matters for the person paying for the machine.

I've been running a local AI stack on an RTX 3090 for about three months now — Whisper transcription as a permanent service, an embedding model for a RAG pipeline, and a 27B chat model on a second machine. I instrumented the GPU with a power meter, set up a dual-rate tariff (day 0.30 BGN/kWh, night 0.18 BGN/kWh), and let the meter run.

This is what the last 30 days actually cost.

The setup

GPU: NVIDIA RTX 3090, 24 GB, power limit 260 W, running WhisperX (a Whisper ASR webservice) as a permanent resident
Tariff: dual-rate — 0.30 BGN/kWh day, 0.18 BGN/kWh night (22:00–06:00)
What's on the GPU 24/7:
- Whisper ASR webservice — avg 7.9 GB VRAM, peak 10.3 GB, resident
- nomic-embed-text (Ollama) — 308 MB VRAM, resident
- immich ML — negligible
What's on the second machine (Vader): Qwen3.8-27B via vLLM, two RTX 3090s, on demand

The Whisper service is the interesting one. It's not a chatbot. It's a transcription endpoint that my own tools call whenever I record a voice memo, a meeting, or a podcast clip. It sits there all day, at rest, drawing power, waiting for audio.

That's the honest shape of a "local AI stack" — not a GPU that's 100% utilised 24/7, but a GPU that's 0% utilised 99% of the time and 100% for a few seconds when it's actually doing work.

The number

The last 30 days cost €2.00 in electricity.

That's the entire GPU. Not the whole machine. Just the GPU, measured at the card, over 30 days, with the Whisper service resident 24/7 and the embedding model resident 24/7.

Breakdown by service:

Service	Avg W	Energy (kWh)	Cost (€)
Whisper ASR (3090)	22 W	13.24	1.76
nomic-embed-text (Ollama)	3 W	1.75	0.23
immich ML	~0 W	0.01	0.00
Total	~25 W	15.03	2.00

The Whisper service is the big one, but even it is only 22 W on average. That's not a GPU under load. That's a GPU that's mostly idle with a model resident in VRAM, spiking to ~120 W for a few seconds when it transcribes something, and sitting at ~35 W the rest of the time.

The embedding model costs 23 cents a month. That's less than a coffee.

Why the number is so small (and why that's the point)

People who argue "local LLMs are expensive" usually mean one of two things:

The hardware cost amortised over time. A 3090 costs €800–1,200 used. Amortised over 3 years, that's ~€25/month. That's real, and it's the dominant cost, not the electricity.
A GPU that's pegged at 100% utilisation 24/7. That's a training rig, not an inference stack. My GPU is at 0% utilisation on average. The 30-day average power draw is ~25 W. A 3090 under sustained LLM inference at 260 W would burn ~187 kWh/month and cost ~€23/month. That's a different machine doing a different job.

The honest framing for a home inference stack is: the GPU is a mostly-idle appliance that costs a few euros a month to keep warm, and spikes when you actually use it.

The €2.00 number is the "keep it warm" cost. The spikes are the "actually use it" cost, and they're short enough that they barely move the monthly total.

What the power trace actually looks like

I pulled the 30-day power history at 2-hour resolution. The shape is:

Baseline: ~35 W, 24/7 (Whisper resident, idle)
Spikes: ~120 W, for a few seconds to a few minutes, when audio comes in
Occasional higher spikes: ~50–54 W sustained for longer stretches when there's a queue of transcriptions
One outlier: a single 2-hour bucket at 121 W (a batch of long audio files)

The 2-hour buckets that show 35 W are the honest baseline. The 2-hour buckets that show 121 W are the work. The average of all of them is 22 W for the Whisper service.

That's the shape of a local inference stack: a low baseline with short, sharp spikes. Not a sustained load.

The comparison that matters

Here's the comparison that actually answers "is local worth it":

Scenario	Monthly electricity cost	Notes
This stack (Whisper + embed, 24/7 resident)	€2.00	Measured, 30 days
3090 at 100% util, 24/7 (training rig)	~€23	Hypothetical, sustained load
API transcription (Whisper API, ~10 hrs audio/month)	~€14–29	OpenAI Whisper API pricing
API LLM (GPT-4o, ~1M tokens/month)	~€14	List price

The local stack costs less than a tenth of the equivalent API spend, even before you factor in that the API spend is per-use and the local spend is a flat "keep it warm" cost that doesn't scale with usage.

The electricity is not the cost. The hardware is the cost. But the electricity is small enough that it stops being the argument against.

What this means for the "is local worth it" question

The electricity bill is not the reason to choose cloud over local. It's not even close. A 3090 running a resident inference stack costs ~€2/month in electricity. The hardware amortisation is ~€25/month. The cloud API equivalent is ~€14–29/month for comparable usage, and it scales with usage while the local cost doesn't.

The real cost of local is the upfront hardware and the time you spend keeping it running. The electricity is a rounding error.

If you're already paying for a GPU for other reasons (gaming, rendering, a second machine), the marginal cost of adding a local inference stack is essentially the electricity: a few euros a month.

Verdict

A local LLM inference stack on a 3090 costs ~€2/month in electricity. The hardware is the real cost, not the power. The electricity bill is small enough that it should stop being an argument in the cloud-vs-local debate.

If your objection to local is "the electricity," the meter says otherwise. The meter says: a few euros a month, mostly idle, spikes when you use it.

That's not a cost problem. That's a hardware-cost problem, and it's a one-time problem, not a recurring one.

How I measured this

Every number in this article came out of my homelab monitor, not an estimate: a small self-hosted dashboard that reads the GPU's power draw at the card, tracks per-service VRAM, and prices the energy against my tariff. It's open source, and it exposes the same data through a read-only MCP server, so an AI agent can pull the numbers for you instead of you SSH-ing in to run nvidia-smi by hand.

GitHub: SikamikanikoBG/homelab-monitor
Docs: sikamikanikobg.github.io/homelab-monitor

If you want to run your own "what does my stack cost" experiment, that's the tooling I'd point you at.

The Local LLM Lab is a series of measured experiments on a home GPU stack. Every number in this article was pulled from a power meter on the card, not estimated. The next piece prices the same stack by € per 1,000 correct answers — the number that actually matters when the model is doing your work.

Qwen3.8-27B on 2 RTX 3090: My First Local Model I Actually Trust

Arsen Apostolov — Tue, 08 Sep 2026 05:45:03 +0000

For years "local LLM" meant a toy — a chatbot you played with on weekends, too slow or too dumb to trust with real work. This month, for the first time, I stopped worrying. My assistant, Jarvis, runs its daily tasks on a 27B model on my own hardware. On the tasks I actually run, it feels Sonnet-like — on some, better. Tests continue.

What it does — not chat, work

Read ~500 Jira emails and built a live tracker of 20+ demands: status, UAT dates, declared FTE benefits, what's closed and why.
A compliance table for the EU AI Act's image-marking rules — provider vs. deployer, the metadata-vs-visible-label split, with dates. It caught a term the legal team was misusing.
A net-worth rollup from my own books, and a gold-position sizing question that came back as a plan: 5–10% of liquid, DCA'd over 2–3 tranches.
The homelab itself: benchmarking models, watching the boxes, writing this post.

The point isn't any single task. It's that I stopped being careful. With an API, every prompt is a small cost and a small privacy decision. Locally, it's just a prompt.

The setup

        ┌────────────────────────────────┐
        │        ArDi (the hub)          │
        │  56-core · 64GB · 52 Docker    │
        │  Jarvis 2 core · Whisper ASR   │
        │  embeddings · homelab-monitor  │
        └───────────────┬────────────────┘
                        │  Tailscale
        ┌───────────────▼────────────────┐
        │       Vader (inference)        │
        │  vLLM · Qwen3.8-27B (262K ctx) │
        │  2× RTX 3090 · 48GB VRAM       │
        │  tensor-parallel · port 8010   │
        └────────────────────────────────┘

Qwen3.8-27B is dense 27B, Apache 2.0, 262K context, vision + video. I picked a dense 27B over a bigger MoE on purpose: one model that fits in 48 GB and is fast, not a 2.4T monster that needs a cluster. Under load Vader draws ~685 W at 80 °C. It's a server, not a laptop.

If you're going to do this

Four things I'd tell my past self:

VRAM is the budget, not FLOPs. A 27B dense model at Q8/Q4 fits two 24 GB cards with headroom. That's why I went dense: the math is simple — weights + KV cache ≤ 48 GB. A 2.4T MoE would need a cluster and a mortgage.
Split the box. Inference on its own machine, everything else (the assistant core, ASR, embeddings, the monitor) on the hub, talking over Tailscale. When the GPU pegs at 100%, the rest of the house keeps working. One box doing everything is where "local AI" dies.
vLLM over Ollama for serving. Ollama is fine for tinkering; vLLM with tensor parallelism is what makes 48 GB feel like one big GPU and what gets you real tokens/sec. Port 8010, OpenAI-compatible API — your existing clients just point at it.
Measure the power, or you're guessing. "It's a 685 W box" only matters if you can attribute that draw to the process and price it. I built homelab-monitor for exactly that — VRAM and power per process, energy priced at your tariff. The whole hub ran 4.97 kWh this week: 1.30 BGN / 7 days. If you run more than one box, it's the thing I'd reach for first.

The honest part

Tests continue. The hardest agentic chains still make me double-check. But for the daily 80%, it's reliable enough that I stopped caring. A local model you have to think about is a toy. One you don't is infrastructure.

Drafted with AI assistance. Hardware, cost and task data are from my own homelab, measured live.

Does a Second GPU Increase Ollama's Context Window? (Quadro P2000 + RTX 3090 Tested)

Arsen Apostolov — Thu, 09 Jul 2026 12:37:52 +0000

TL;DR

Short version: no. I dropped a much older GPU (Quadro P2000, 5GB, Pascal, 2016) next to an RTX 3090 (24GB, Ampere) on the same box, ran the same context-length ladder (8K→128K) through Ollama and vLLM on qwen3-coder:30B-A3B, and got zero extra usable context in either engine — and a 74% decode-speed hit for the trouble. Ollama hits the identical Chunk too big wall at ctx=65536 whether the P2000 is there or not. vLLM refuses tensor-parallel across the two cards entirely — not a VRAM problem, a flat compute-capability rejection (Minimum capability: 75. Current capability: 61.) that fails in 40 seconds, before any memory profiling. And the one real, measured effect of adding the P2000 to Ollama: decode speed goes from 76 → 19.5 tok/s at ctx=49152 once the P2000 gets pulled in as an actual compute device.

Full narrative version — the two-stage collapse, the prompt-cache validation bug caught mid-sweep, the CUDA13-silently-drops-Pascal finding — is on Medium.## The setup

ardi (dual Xeon E5-2680 v4, 128GB RAM, openSUSE Leap) has a Quadro P2000 sitting in a second slot next to the RTX 3090 this whole series has run on so far. Same model as phase 1 (qwen3-coder:30B-A3B), same box, four legs: {Ollama, vLLM} × {3090 only, 3090+P2000 tandem}, priced through HomeLab Monitor against real GPU power draw.

Ollama: same wall, extra tax

ctx	3090 only decode tok/s	tandem decode tok/s	P2000 VRAM (tandem)
8,192	124.3	122.0	6 MB / 0%
24,576	108.2	70.0	62 MB / 0%
32,768	99.4	61.0	62 MB / 0%
49,152	75.7	19.5	3,580 MB / 55%
65,536	fatal: `Chunk too big`	fatal: identical `Chunk too big`	—

Two separate costs, not one: decode already falls behind at ctx=24576 while the P2000 is still basically idle (62MB, 0% util) — some scheduling overhead just from having a second visible device. Then the real collapse hits at ctx=49152, when the P2000 actually gets pulled into the compute path (3.58GB, 55% util) and decode craters to 19.5 tok/s. Same context ceiling either way, worse speed the whole way there.

vLLM: doesn't even get to try

Expected failure mode going in: tensor-parallel splits the ~17GB AWQ checkpoint roughly in half, and the P2000's 5GB doesn't hold its ~8.5GB share. Actual failure, at ctx=8192, in 40 seconds, before any memory profiling:

ValueError: The quantization method auto_awq is not supported for the current GPU.
Minimum capability: 75. Current capability: 61.

AWQ's Marlin kernel needs compute capability 7.5+ (Turing and later). The P2000 is 6.1 (Pascal). Not a close VRAM call — a flat architectural exclusion, decided before capacity is even checked.

Bonus finding: Ollama's own CUDA13 build almost drops the P2000

Boot log, before any of the above:

skipping CUDA device — compute capability not in compiled architectures
device="Quadro P2000" cc=610
archs="[750 800 860 870 890 900 1000 1030 1100 1200 1210]"

Falls back to a legacy cuda_v12 runtime that does support Pascal — so it works, just via a path most people wouldn't notice without reading boot logs. This 2016 card is now old enough that modern quantized-inference stacks are starting to architecturally step around it, not just outrun it.

What wasn't the point of this one

Not claiming a second GPU is never worth it — a matched pair, or a smaller-but-newer card, is a different setup entirely. This was specifically: does this 5GB Pascal card, next to this 3090, on these two engines, buy anything. Check compute capability against your quantization scheme before you do the VRAM math — it can end the conversation first.

Every number above priced through HomeLab Monitor — open source, MIT licensed — against ardi's real GPU power draw. Full write-up with all four charts and the mid-sweep debugging on Medium.What's the oldest card you've tried to tandem into a rig — did it actually pull weight, or did you just assume it was?