DEV Community: Arsen Apostolov

Qwen3.8-Flash-Next (125B) on Three RTX 3090s at 80 Tokens/s: Teaching llama.cpp Which Experts Matter

Arsen Apostolov — Sat, 26 Sep 2026 16:02:31 +0000

TL;DR: Qwen3.8-Flash-Next is a 125B mixture-of-experts model: on paper it beats the Qwen3.8-27B I run in production everywhere, by +16.5 points on agentic coding. It does not fit in the 72 GB of VRAM my three RTX 3090s have, and stock llama.cpp, spilling a quarter of the experts into system RAM, runs it at 23 tokens/second. Three changes get it to 80 tokens/second, entirely in VRAM:

Sort the experts by how often they're used. They are wildly unequal: the busiest 25% handle 52% of the work.
Store them at three precisions. Popular experts keep more bits and rare ones get squeezed harder, so the whole model fits on the GPUs.
Patch llama.cpp (~400 lines) so one routing decision drives three expert tables at once, then add speculative decoding on top.

Quality stays close to the 8-bit reference: +1.9% perplexity, 91% same next token, GSM8K 95.5% (the 27B scores 95–96.5% on the same harness). There's also a 256k-context profile that finds a random code hidden 173,000 tokens deep. The trade-offs are real, and I've listed them. Everything is open: patches, tools, raw measurements.

Update, 26 Sep 2026, after the first day behind my assistant: the 80 tok/s holds for short prompts. In real agent use, with ~25k-token prompts, sampling and thinking on, the model decodes at 30–45 tok/s. My first deployment also had a caching mistake that cost ~45 s before every answer; a server config change fixed it (now 0.7 s). Details in the "real-world update" section below.

the use case

My box, vader, runs Qwen3.8-27B around the clock: two RTX 3090s, vLLM, about 135 tokens/second. It's the first local model I actually trust with agent work, and it does most of the work my assistant does.

Then Qwen shipped Qwen3.8-Flash-Next, the preview of their Qwen4 architecture. Their own model card puts it ahead of my 27B on almost everything I care about:

benchmark (Qwen's card)	Flash-Next	3.8-27B	gap
DeepSWE (agentic coding)	58.7	42.2	+16.5
JobBench (professional tasks)	55.7	33.4	+22.3
SWE-bench Multilingual	81.0	73.8	+7.2
Toolathlon (tool use)	73.5	67.1	+6.4
HLE	35.9	30.8	+5.1
GPQA Diamond	91.7	89.2	+2.5
IFBench	81.3	79.5	+1.8

So the goal was simple to state: the smarter model, on the same three cards, with decode and time-to-first-token that still feel interactive. No new hardware. I told my AI engineer (Claude Code, more on that at the end) that "no" was not an acceptable answer.

the box

GPUs: 3x RTX 3090, 24 GB each: 72 GB of VRAM. No NVLink, and no PCIe peer-to-peer (the board has no Resizable BAR), so the cards can't talk to each other directly.
CPU: dual Xeon E5-2660 v4, 56 threads.
RAM: 60 GB. Only one 32 GB stick per CPU socket, so it's single-channel, and that matters a lot below.
Disk: one 1 TB NVMe.

a small dictionary before we start

If you know what a KV cache is, skip this. If you don't, it's all you need for the rest of the article.

Parameters / weights. The numbers a model learned. "125B" means 125 billion of them. Each one normally takes 2 bytes, so 125B parameters is ~250 GB, far more than 72 GB of VRAM.
VRAM. The GPU's own memory. It's about 20–50x faster than system RAM on this box. A model runs fast only if what it needs for each word lives in VRAM.
Quantization. Storing each weight with fewer bits (8, 4, 3…) instead of 16. Like saving a photo as a smaller JPEG: smaller file, a little less detail. "Q8_0", "Q4_K", "IQ3_S", "MXFP4" are llama.cpp's names for different bit-budgets and recipes.
Mixture of experts (MoE). Instead of one giant network, each layer has many small "expert" networks, 512 of them here, and a router picks the best 10 for each word. So the model knows 125B parameters' worth of things but only computes with about 6B per word. That's why it can be fast, and also why it's huge.
Token. A word or word-piece. Decode speed (tokens/second) is how fast the answer streams out.
Prefill / time to first token (TTFT). Before answering, the model reads your whole prompt. That's prefill, and TTFT is how long you wait before the first word appears.
Context / KV cache. How much text the model can hold in mind at once ("256k context" is about 500 pages), and the memory it uses to remember it.
n-gram embedding. New in this model: a 51 GB lookup table indexed by 2–3 word phrases. It's only ever looked up, a few rows per word, so it can live on the NVMe drive.
Speculative decoding / MTP. A small built-in "draft head" guesses the next few words, and the big model checks all the guesses in one pass. Correct guesses are free words.
KL divergence (KLD). How far the compressed model's word probabilities drift from the original's. 0 means identical; lower is better. It's the most sensitive quality meter there is.
llama.cpp / GGUF. The inference engine I patched, and its model-file format.

the problem, in one picture

In 16-bit the model is 360 GB. Even the popular 4-bit build (unsloth's UD-Q4_K_XL) is 111 GB: 72 GB of experts, 27 GB of n-gram table (which can stay on disk) and a few GB of everything else. The experts alone don't fit in 72 GB of VRAM once the rest of the model and the context need room too. So llama.cpp does the sensible thing: it puts the overflow experts in system RAM and computes them on the CPU.

On this machine that means 23 tokens/second. With one memory stick per socket, the CPU reads those experts about ten times slower than a GPU would. And for every word, the work bounces between the GPUs and the CPU dozens of times, so each GPU waits.

step 1: the obvious setup, and the first trap

The first run with stock settings also showed prefill all over the place (47 to 306 tokens/second on the same 4k prompt) and decode sagging to 10 tok/s. Thread states told the story: the main thread sat in D (waiting on disk). llama.cpp memory-maps the model file, loading the GPU part streams 70+ GB through the page cache, and the CPU-side experts kept getting evicted and re-read from NVMe.

--load-mode none loads the CPU-side weights into RAM and keeps them there. Also, don't download a 188 GB file while you benchmark: my download filled the page cache and pushed the server into swap. With both fixed, prefill doubled (658–792 tok/s) and decode steadied at 23–26 tok/s.

Speculative decoding (MTP) barely helped: 24–29 tok/s. The reason matters for everything that follows. A draft of 5 words means the big model checks 6 words at once. Each word picks 10 experts, so one check can touch up to 60 different experts per layer instead of 10, and the slow ones in CPU RAM get hit harder. As long as experts live in CPU RAM, nothing else will make this fast.

step 2: the experts are not equal

Here's the observation the whole project rests on. llama.cpp's calibration data (the importance matrix unsloth publishes with their quants) records how often each of the 24,576 experts was picked on real text:

In the average layer, the busiest 25% of experts handle 52% of the tokens, and the busiest 80% handle 95%. Stock llama.cpp can only place whole layers of experts: a layer's 512 experts are one tensor, all on the GPU or all on the CPU. It offloads a quarter of the experts, including popular ones, so a quarter of the expert work lands on the slow path.

step 3: split every layer into hot and cold

The first patch splits every layer's expert tensor in two at load time:

hot experts go to the GPUs;
cold experts go to CPU RAM;
the router is permuted so its choices land in the right half.

With the same 19% of expert bytes on the CPU, the cold experts now serve only 4.5% of routed tokens instead of ~19%, about 4x less slow-path traffic.

Getting it right took four bugs, each instructive enough to list in its own section below. The result is 30–32 tok/s with correct answers. That's +30%, but it's not the leap it should be, so I profiled it:

where one decoded word spends its time
kernel launches	~4,300
dense weights (attention, DeltaNet, shared expert), 8-bit	~4.9 ms
routed experts	~2.5 ms
~3,500 tiny element-wise kernels	~6–7 ms
GPU↔CPU synchronisations	~390

The CPU was barely reading any experts anymore; it was the round trips that cost. Every layer handed work to the CPU and waited for the answer. Then a cheap experiment: drop the cold experts entirely (quality garbage, speed only) and see how fast it goes fully on GPU: 52 tok/s, and 86–88 tok/s with speculative decoding. That was the target. The question became: how do you fit every expert in VRAM without wrecking quality?

step 4: three shelves, all on the GPU

The idea: if some experts do most of the work, give them more bits and squeeze the rarely-used ones harder. Everything still fits in VRAM, and most tokens still see a well-preserved expert. Think of a library that keeps its bestsellers in hardcover and prints the rarely-borrowed titles as compact paperbacks, so the whole collection fits in the building.

Two pieces make it work.

An offline re-packer (tools/expert_tiers.py). It starts from the 8-bit model (188 GB) and uses the importance matrix for both how often each expert is used and which inputs matter inside it. A greedy planner fills a VRAM budget: each byte goes to the expert where it removes the most expected error. Every expert is then re-quantized with llama.cpp's own quantizers, and the file is written with the experts reordered hottest-first as three tensors per layer. One wrinkle: the down-projection matrices have rows 640 wide, which the fancy 2–3-bit formats can't handle (they need multiples of 256). So they get their own ladder: IQ4_NL / MXFP4 instead of IQ4_XS / IQ3.

A patched llama.cpp (patches/). mul_mat_id, the operation that runs "each word through its chosen experts", learns to take an id range. Each shelf's matrix multiply sees the router's full choice list, computes only the ids that fall on its shelf, and writes zeros for the rest. The three results simply add up. That meant touching the CPU kernels, three CUDA paths (decode, prefill, and the fused gate+up+activation kernel) and the loader. On top sits Qwen's multi-token-prediction draft head, from a not-yet-merged llama.cpp PR, re-quantized to 4 bits so it fits too.

Result, in one chart:

the four bugs, briefly

For people who'll try this, and because two of them are delightful.

Duplicate experts crash the CUDA MoE kernel. My first version pointed "not on this shelf" at expert 0. A word with two cold experts then had expert 0 twice, and CUDA's expert-grouping helper assumes each expert appears at most once per word. It silently dropped a slot, and a later kernel read an unwritten index. Fixed by teaching the kernels a real "skip".
A flag stored where a precision setting lives. I marked the skip-capable nodes in op_params[0], which llama.cpp already uses for the accumulator precision. The flag was quietly overwritten. It moved to slot 7, with a magic value.
The unsigned ternary. ids ? ids[i] : blockIdx.x, where blockIdx.x is unsigned. C++ promotes the whole expression to unsigned, so my -1 ("skip") became 4,294,967,295 and the kernel read 4 billion rows past the buffer. compute-sanitizer found it in one run.
Fused kernels hide the tag. For decode, CUDA fuses gate, up and activation into one kernel whose "destination" is the activation node, not the matrix multiply that carries my id-range tag. The first id-range build printed //////////. The kernel now looks up the tag on its source node.

how much quality did it cost?

The honest meter is KL divergence against the 8-bit model over 24,576 tokens of Wikipedia text:

model	expert size	fits in VRAM	perplexity vs 8-bit	KLD	same top token
unsloth UD-Q4_K_XL (stock)	71.7 GiB	no, 25% in RAM	+0.9%	0.045	93.7%
tiered, 128k profile	53.0 GiB	yes	+1.9%	0.091	91.1%
uniform IQ3_S / MXFP4, same size	52.2 GiB	yes	+3.0%	0.095	91.0%
tiered, 256k profile	49.5 GiB	yes	+3.2%	0.113	90.1%

What I take from it:

Fitting in VRAM costs quality, and there's no way around that. 20 GB less expert weight roughly doubles the KL divergence of the stock 4-bit build. That's physics, not a bug.
At equal size, tiering beats uniform: perplexity drift 1.9% vs 3.0%. But the gain is smaller than my planner predicted: its error model is crude, and at this size the total byte count dominates. There's more to win with a better tier recipe.
On a task, it holds up. GSM8K (200 grade-school maths questions, greedy, no thinking) scores 95.5%. The 27B's published number on the same harness is 95.0–96.5%. GSM8K is saturated, so it mainly proves nothing broke. The model card's gains on agentic work are where the upgrade should show.

speed vs context, and the 256k profile

Both profiles start at ~80 tokens/second on a short prompt and slow down as the prompt grows. The per-word attention and indexer work grows with context, and the draft head's guesses get a little worse too. Typical numbers, averaging two runs:

prompt	128k profile: decode / time to first token	256k profile: decode / time to first token
4k tokens	82 tok/s / 7–9 s	67 tok/s / 7–8 s
15k tokens	66 tok/s / 18–22 s	58 tok/s / 19–22 s
30k tokens	58 tok/s / 35 s	54 tok/s / 38 s
61k tokens	53 tok/s / 76–79 s	51 tok/s / 82 s
95k / 122k tokens	39 tok/s / 140 s	35 tok/s / 190 s

Prefill runs at 540–860 tokens/second. That's the weak spot next to the 27B's ~1,300, and it's where I'd look next: it's also the part that makes a 120k-token prompt a three-minute wait.

The 256k profile needed two more compromises. The KV cache drops to 8-bit, and experts shrink to 49.5 GiB, because the sparse-attention indexer's scratch memory grows with context and pushed a card out of memory at ~200k on the first try. After rebalancing layers across the cards, it read 173,692 tokens of Wikipedia with a random 10-character code hidden at 45% depth, and returned the code: exactly right (6BC5XYE8FS). A second run with the code at 90% depth of 112,711 tokens was also exact. It took 6.1 minutes (472 tokens/second) to read that much. Long-context prefill on this architecture is still the slow part.

what it costs to run

Measured at the cards (nvidia-smi, 5 Hz, three 1,024-token generations per profile): the three 3090s draw ~540 W together while decoding and ~124 W at rest with the model loaded. That's 7.5–8.4 joules per token, 2.1–2.3 kWh per million tokens. On my tariff (0.30 BGN/kWh day, 0.18 night) a million generated tokens cost 0.63–0.70 BGN in daytime, 0.38–0.42 BGN at night, about $0.35–0.40. The 27B costs 0.21–0.34 BGN for the same million, so the bigger brain costs roughly twice as much per word, all three cards included. It's still coffee money.

real-world update: day one behind my assistant

Benchmarks are one request with a short prompt and greedy decoding. My assistant is none of those things, so after a day of it running Jarvis, here is what the server logs say about real requests:

	a real assistant turn	the benchmark
prompt	~25,000 tokens (system prompt, tools, memory)	41 tokens
sampling	temperature 0.7, thinking on	greedy
tokens per verify step (draft acceptance)	2.6 (39%)	3.4 (62%)
decode	41–45 tok/s (30–45 is what it feels like)	80 tok/s

Decode: 30–45 tok/s, not 80. Two things compound. The prompt is deep, and every decoded word pays attention over it; the context chart above already shows ~50 tok/s at 60k. And the draft head guesses sampled text, tool-call JSON and reasoning worse than greedy prose, so speculative decoding saves less. That's the honest number for agent work on this box. For comparison, the 27B benchmarks at ~135 tok/s on this box, so the upgrade is intelligence, not speed.

Time to first token: the mistake was mine. My assistant runs several roles: chat, planner, classifier, triage, background jobs. Each has its own ~25k-token prompt prefix. I deployed the server with one slot (-np 1), so every time the role changed, the cached prefix was thrown away and 25,000 tokens were re-read from scratch: ~45 seconds before every answer. The fix is four slots sharing one KV pool, so every role keeps its prefix warm:

returning to a 14k-token prompt after another role ran	before (`-np 1`)	after (`-np 4 -kvu`)
time to first token	20–45 s	0.6–0.7 s

It isn't free: four slots need a little more memory, so the shared pool dropped from 128k to 96k tokens (128k with four slots ran out of VRAM). A single request can still use all 96k.

Schedule the background jobs apart. A model at 30–45 tok/s with 25k-token prompts spends minutes per agent task. My assistant had 13 scheduled jobs between 06:30 and 09:00 on Mondays, four of them at the same minute. I've spread them 30 minutes apart (05:00 to 11:30) so they stop queueing behind each other. I'll report after a week whether that's enough.

Images and video work too, once the vision projector is loaded (--mmproj, on the GPU with the most free memory: 4 s per photo, against 61 s on the CPU) and the container has ffmpeg for video (a 5-second clip in 6.4 s). The repo has the exact launcher.

what I'd not claim

It's single-user. Every speed number here is one request at a time, and real agent turns decode at 30–45 tok/s, not 80. The 27B on vLLM batches many users far better.
The 27B is still faster. About 135 vs 80 tok/s decode, and roughly 1,300 vs 540–860 tok/s prefill. The 125B is smarter, not quicker.
It uses all three cards. You can't also keep the 27B running.
The quality numbers are wikitext + GSM8K. They say the compression is gentle. They don't prove the agentic gains survive intact; that needs agent benchmarks I haven't run yet.
The tier planner is a heuristic. The measurements say it helps; a calibrated one would help more.

reproduce it

Everything is in SikamikanikoBG/qwen38-flash-next-3x3090: both llama.cpp patches, the re-packer, the benchmark scripts, and every raw number behind the charts. The short version:

# llama.cpp at 81bc6b8 + MTP PR #28243 + the tiering patch
git apply patches/0001-qwen4exp-mtp-pr28243.patch patches/0002-tiered-experts-hot-cold-split.patch
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 && cmake --build build -j

# re-pack the 8-bit model into three precision shelves (53 GiB of experts)
python tools/expert_tiers.py write --src Qwen3.8-Flash-Next-Q8_0-00001-of-00006.gguf \
  --imatrix imatrix_unsloth.gguf --budget-gib 53 \
  --gu Q6_K,IQ4_XS,IQ3_XXS --dn Q8_0,IQ4_NL,MXFP4 --out fn-tier53.gguf

# serve: all layers on GPU, 4 slots sharing a 96k KV pool, vision, MTP drafting 4 tokens
llama-server -m fn-tier53-00001-of-00002.gguf -ngl 99 -ts 16,16,16 -c 98304 -np 4 -kvu \
  -b 1024 -ub 256 -fa on --jinja \
  --mmproj mmproj-F16.gguf -mmdev CUDA1 --image-max-tokens 1024 \
  -md mtp-shared-iq4.gguf --spec-type draft-mtp --spec-draft-n-max 4

credits

Qwen, for the model and an architecture that rewards this kind of work.
unsloth, for the GGUFs, the MTP draft files and the importance matrix.
llama.cpp and ggml, plus the authors of the Qwen4Exp MTP PR and the sparse-attention decay issue.
syv-ai/qwen38-27b-rtx3090, whose rigor on the 27B set the bar for measuring this.

This project, from the patches and the tools to the measurements and this write-up, was done with **Claude Code* acting as my AI engineer, working on my hardware under my direction. The numbers are from vader, the raw data is in the repo, and the mistakes are both of ours.*

Qwen3.8-27B on One RTX 3090 vs Two: +20% Decode, +14% Cold Prefill, and 3x on Cached Prompts

Arsen Apostolov — Wed, 23 Sep 2026 03:27:37 +0000

TL;DR: Qwen3.8-27B — the dense 27.8B that dropped on August 14 — fits on a single RTX 3090 at W4A16 and decodes at ~125–155 tokens/sec. Splitting it across two cards with tensor parallelism buys roughly 20% more decode (noisy: +13% in one run, +22% in the next) and 14% faster prefill on average on a cold prompt (23% at the top of the ladder: an 8,600-token prompt takes 8.5 s to first token on one card and 6.9 s on two). On a prompt the server has already seen, the pair answers 2–3x faster — but that is the prefix cache, not the cards, and the script below measures the two separately. At the API list price of $3.00 per million output tokens, one million tokens cost 0.21–0.34 BGN of electricity, about $0.12–0.20, on the box that serves them.

the box

vader is the machine I run this on: three RTX 3090s in an ASUS Z10PE-D8 WS, dual Xeon E5-2660 v4 (56 threads), 60 GB of RAM, Ubuntu 26.04. It has been my inference box since the Whisper experiments, and since July it has also carried the power-limiting work (the meters post is where those numbers live). Two of the three cards are in this test; the third sits idle.

The model under test is the one that is currently loaded on it:

	single-card	TP2
container	`qwen38-27b-single-1`	`qwen38-27b-tp2-single-1`
image	`ghcr.io/syv-ai/qwen38-27b-rtx3090`	same, `:latest`
weights	Qwen3.8-27B-W4A16-AutoRound-fast	same
cards	1x 3090 (22.6 GB VRAM)	2x 3090 (21.7 GB each)
max context	131,072	262,144
port	18021	18022

Both are vLLM, both have been up two days, both report healthy, and both run with prefix caching on — which is why every number below comes in two flavours, cold and warm. The W4A16 quantization is what puts a 27.8B dense model inside one 24 GB card — 22.6 GB of it, with the KV cache budget for the rest of the 128k window coming out of what is left.

the method

No benchmark harness, no vllm bench — a 150-line script (bench_qwen38_v2.py) that sends a real chat-completion request with a prompt of a given size, streams the answer, and times it. For each prompt size it runs three times and takes the median:

prompt sizes: calibrated so the server reports ~600, ~2,400, ~4,700, ~9,300 prompt tokens (the script probes the tokenizer first, then scales the prompt to hit the target)
output: max_tokens=256, temperature=0
measured: time to first SSE event, total time, and the server-reported token counts from stream_options.include_usage — not my own chunk counting
cold vs warm: every cold request carries a unique first line, so no cached prefix can match it; the warm pass repeats one identical request after a warm-up call

That last bullet is the one that decides whether a benchmark means anything. With prefix caching on — and it is on by default in vLLM — timing the same prompt twice does not measure prefill the second time, it measures a cache lookup. The tell is arithmetic: 8,600 tokens "prefilled" in 0.56 s is 15,000 tokens/sec, roughly ten times what two 3090s can actually push through a 27B model. If your TTFT numbers look like that, you are timing the cache. So every cold request here is salted with a unique first line, and the warm pass is run deliberately and reported in its own column.

One caveat: this is a single-request benchmark. vLLM's continuous batching is where production throughput lives, and I did not hammer it with concurrent requests. What I measured is the latency you feel as the one person using the box — which, for a homelab, is most of the time.

decode: the second card buys 15–30%, noisily

The single card holds 123–154 tokens/sec across the ladder; the TP2 pair runs 146–193. A 27B dense model at W4A16 on one 3090 is memory-bandwidth-bound in a way that does not care much about prompt size, because the KV cache for these sizes is small next to the weights. The pair is faster at every size but one (at 4,700 tokens the two came out level), and the average gain was +22% in this run against +13% in a run an hour earlier. Single-stream decode on these cards moves by about 10% between runs an hour apart, so I am not going to print a decimal: call it 15–30%, and expect the lower end.

That is a real number and a small one. For a 27B model, tensor parallelism across two cards is not the 2x you get from adding compute — the all-reduce between the cards eats most of it.

prefill: cold is the cards, warm is the cache

Two columns, two different questions.

Cold — a prompt the server has never seen — is prefill, and prefill is compute-bound: the single card does 1,100–1,220 tokens/sec of it, the pair 1,230–1,360. Time to first token climbs from 0.53 s to 8.5 s on one card as the prompt goes from 600 to 9,300 tokens, and from 0.50 s to 6.9 s on two. The gap grows with the prompt: 6% at 600 tokens, 23% at 9,300, 14% on average. That is what the second card buys you on a fresh long prompt.

Warm — the same prompt again — is the prefix cache doing its job, and here the pair really is 2–3x faster: 0.5–0.75 s flat across the ladder against 0.5–1.7 s on the single card, whose smaller KV pool and slower re-read of the uncached tail show at every size above 600 tokens. It is a cache number rather than a prefill number, and it is also the one you feel most often, because every follow-up turn in a chat is a warm request.

The routing rule that falls out: a long fresh prompt goes to the pair and saves about a fifth of its wait; a conversation stays on whichever lane it started on, so its prefix stays cached. For prompts under about 2,000 tokens the single card is within noise of the pair on both counts and draws one card's power.

what it costs

The box draws 620 W at the wall with both containers loaded and serving — 247 W on one TP2 card, 328 W on the other, 45 W on the single-card 3090 sitting at P8 with its weights resident. Those are the monitor's readings from the day the containers went up, not a measurement taken during these runs; the governor from the meters post is power-capping it, holding 79°C.

At the API's list price — $3.00 per million output tokens, DeepInfra's rate for Qwen3.8-27B as listed on llm-stats — one million generated tokens takes about 6,667 seconds at 150 tokens/sec. At 620 W that is 1.15 kWh. On the Bulgarian dual-rate tariff that is 0.34 BGN by day (0.30 BGN/kWh) or 0.21 BGN by night (0.18 BGN/kWh) — roughly $0.20 or $0.12. The API price is 15x to 25x the electricity, depending on when the box runs.

I am not going to pretend that makes the API a bad deal. The API does not cost you 620 W of heat in your room, it does not need a 60 GB machine, it does not need you to keep a W4A16 quant on a 24 GB card, and its $3.00 buys you their datacenter's power, cooling and idle capacity, not yours. But for a box that is already on and already paying its standby tax, every token it serves is almost pure margin. The electricity argument for local inference is not "it is cheaper than the API" in any total-cost sense — it is "the marginal cost of a token on hardware you already run is an order of magnitude below the list price."

the verdict

Qwen3.8-27B is the first 27B-class dense model I have run that I would put in front of real work on this box, and the numbers say why:

It fits. W4A16 on one 3090, 22.6 GB, with room for a 128k context window. The TP2 variant doubles the context to 262k by splitting the weights.
It is fast enough to feel like an API. 125–190 tokens/sec decode is faster than most hosted endpoints feel in a chat UI. The part you notice is a cold long prompt: several seconds on either lane.
The two-card setup is not 2x. It is roughly +20% decode and +14% cold prefill, and 2–3x on warm follow-ups because the pair keeps a bigger cache. Buy the second card for the context window and the cache, not for raw speed.
The marginal cost is 0.21–0.34 BGN, about $0.12–0.20, per million tokens against a $3.00 list price, on hardware that is already drawing its standby power.
Benchmark with the cache in mind. If a prefill number looks like 15,000 tokens/sec on consumer cards, it is the cache. Salt every request, or turn prefix caching off, and report which one you did.

The script, the raw JSON and the charts are in the repo: qwen38-bench. If you run a different quant or a different card, I want your numbers — the interesting part of this post is the shape of the curve, and the shape changes with the hardware.

vader: 3x RTX 3090 (two used here), dual Xeon E5-2660 v4, 60 GB RAM, Ubuntu 26.04, vLLM, W4A16-AutoRound-fast quantization.

I Benchmarked Qwen3.8-27B on Three RTX 3090s. The Second Card Bought Me 13% More Decode — and a 3x Faster Prefill

Arsen Apostolov — Sun, 20 Sep 2026 12:14:36 +0000

What Does a Local LLM Actually Cost per Month? I Read the Meters.

Arsen Apostolov — Sun, 20 Sep 2026 07:59:22 +0000

What Does a Local LLM Actually Cost per Month? I Read the Meters.

The Local LLM Lab — Part 5

One controlled experiment. One number. One verdict.

The question nobody answers in the local-LLM hype is the boring one: what does the electricity bill say?

Not "how many tokens per second." Not "how many GB of VRAM." The bill. The one that arrives on the first of the month and is the only number that actually matters for the person paying for the machine.

I've been running a local AI stack on an RTX 3090 for about three months now — Whisper transcription as a permanent service, an embedding model for a RAG pipeline, and a 27B chat model on a second machine. I instrumented the GPU with a power meter, set up a dual-rate tariff (day 0.30 BGN/kWh, night 0.18 BGN/kWh), and let the meter run.

This is what the last 30 days actually cost.

The setup

GPU: NVIDIA RTX 3090, 24 GB, power limit 260 W, running WhisperX (a Whisper ASR webservice) as a permanent resident
Tariff: dual-rate — 0.30 BGN/kWh day, 0.18 BGN/kWh night (22:00–06:00)
What's on the GPU 24/7:
- Whisper ASR webservice — avg 7.9 GB VRAM, peak 10.3 GB, resident
- nomic-embed-text (Ollama) — 308 MB VRAM, resident
- immich ML — negligible
What's on the second machine (Vader): Qwen3.8-27B via vLLM, two RTX 3090s, on demand

The Whisper service is the interesting one. It's not a chatbot. It's a transcription endpoint that my own tools call whenever I record a voice memo, a meeting, or a podcast clip. It sits there all day, at rest, drawing power, waiting for audio.

That's the honest shape of a "local AI stack" — not a GPU that's 100% utilised 24/7, but a GPU that's 0% utilised 99% of the time and 100% for a few seconds when it's actually doing work.

The number

The last 30 days cost €2.00 in electricity.

That's the entire GPU. Not the whole machine. Just the GPU, measured at the card, over 30 days, with the Whisper service resident 24/7 and the embedding model resident 24/7.

Breakdown by service:

Service	Avg W	Energy (kWh)	Cost (€)
Whisper ASR (3090)	22 W	13.24	1.76
nomic-embed-text (Ollama)	3 W	1.75	0.23
immich ML	~0 W	0.01	0.00
Total	~25 W	15.03	2.00

The Whisper service is the big one, but even it is only 22 W on average. That's not a GPU under load. That's a GPU that's mostly idle with a model resident in VRAM, spiking to ~120 W for a few seconds when it transcribes something, and sitting at ~35 W the rest of the time.

The embedding model costs 23 cents a month. That's less than a coffee.

Why the number is so small (and why that's the point)

People who argue "local LLMs are expensive" usually mean one of two things:

The hardware cost amortised over time. A 3090 costs €800–1,200 used. Amortised over 3 years, that's ~€25/month. That's real, and it's the dominant cost, not the electricity.
A GPU that's pegged at 100% utilisation 24/7. That's a training rig, not an inference stack. My GPU is at 0% utilisation on average. The 30-day average power draw is ~25 W. A 3090 under sustained LLM inference at 260 W would burn ~187 kWh/month and cost ~€23/month. That's a different machine doing a different job.

The honest framing for a home inference stack is: the GPU is a mostly-idle appliance that costs a few euros a month to keep warm, and spikes when you actually use it.

The €2.00 number is the "keep it warm" cost. The spikes are the "actually use it" cost, and they're short enough that they barely move the monthly total.

What the power trace actually looks like

I pulled the 30-day power history at 2-hour resolution. The shape is:

Baseline: ~35 W, 24/7 (Whisper resident, idle)
Spikes: ~120 W, for a few seconds to a few minutes, when audio comes in
Occasional higher spikes: ~50–54 W sustained for longer stretches when there's a queue of transcriptions
One outlier: a single 2-hour bucket at 121 W (a batch of long audio files)

The 2-hour buckets that show 35 W are the honest baseline. The 2-hour buckets that show 121 W are the work. The average of all of them is 22 W for the Whisper service.

That's the shape of a local inference stack: a low baseline with short, sharp spikes. Not a sustained load.

The comparison that matters

Here's the comparison that actually answers "is local worth it":

Scenario	Monthly electricity cost	Notes
This stack (Whisper + embed, 24/7 resident)	€2.00	Measured, 30 days
3090 at 100% util, 24/7 (training rig)	~€23	Hypothetical, sustained load
API transcription (Whisper API, ~10 hrs audio/month)	~€14–29	OpenAI Whisper API pricing
API LLM (GPT-4o, ~1M tokens/month)	~€14	List price

The local stack costs less than a tenth of the equivalent API spend, even before you factor in that the API spend is per-use and the local spend is a flat "keep it warm" cost that doesn't scale with usage.

The electricity is not the cost. The hardware is the cost. But the electricity is small enough that it stops being the argument against.

What this means for the "is local worth it" question

The electricity bill is not the reason to choose cloud over local. It's not even close. A 3090 running a resident inference stack costs ~€2/month in electricity. The hardware amortisation is ~€25/month. The cloud API equivalent is ~€14–29/month for comparable usage, and it scales with usage while the local cost doesn't.

The real cost of local is the upfront hardware and the time you spend keeping it running. The electricity is a rounding error.

If you're already paying for a GPU for other reasons (gaming, rendering, a second machine), the marginal cost of adding a local inference stack is essentially the electricity: a few euros a month.

Verdict

A local LLM inference stack on a 3090 costs ~€2/month in electricity. The hardware is the real cost, not the power. The electricity bill is small enough that it should stop being an argument in the cloud-vs-local debate.

If your objection to local is "the electricity," the meter says otherwise. The meter says: a few euros a month, mostly idle, spikes when you use it.

That's not a cost problem. That's a hardware-cost problem, and it's a one-time problem, not a recurring one.

How I measured this

Every number in this article came out of my homelab monitor, not an estimate: a small self-hosted dashboard that reads the GPU's power draw at the card, tracks per-service VRAM, and prices the energy against my tariff. It's open source, and it exposes the same data through a read-only MCP server, so an AI agent can pull the numbers for you instead of you SSH-ing in to run nvidia-smi by hand.

GitHub: SikamikanikoBG/homelab-monitor
Docs: sikamikanikobg.github.io/homelab-monitor

If you want to run your own "what does my stack cost" experiment, that's the tooling I'd point you at.

The Local LLM Lab is a series of measured experiments on a home GPU stack. Every number in this article was pulled from a power meter on the card, not estimated. The next piece prices the same stack by € per 1,000 correct answers — the number that actually matters when the model is doing your work.

Qwen3.8-27B on 2 RTX 3090: My First Local Model I Actually Trust

Arsen Apostolov — Tue, 08 Sep 2026 05:45:03 +0000

For years "local LLM" meant a toy — a chatbot you played with on weekends, too slow or too dumb to trust with real work. This month, for the first time, I stopped worrying. My assistant, Jarvis, runs its daily tasks on a 27B model on my own hardware. On the tasks I actually run, it feels Sonnet-like — on some, better. Tests continue.

What it does — not chat, work

Read ~500 Jira emails and built a live tracker of 20+ demands: status, UAT dates, declared FTE benefits, what's closed and why.
A compliance table for the EU AI Act's image-marking rules — provider vs. deployer, the metadata-vs-visible-label split, with dates. It caught a term the legal team was misusing.
A net-worth rollup from my own books, and a gold-position sizing question that came back as a plan: 5–10% of liquid, DCA'd over 2–3 tranches.
The homelab itself: benchmarking models, watching the boxes, writing this post.

The point isn't any single task. It's that I stopped being careful. With an API, every prompt is a small cost and a small privacy decision. Locally, it's just a prompt.

The setup

        ┌────────────────────────────────┐
        │        ArDi (the hub)          │
        │  56-core · 64GB · 52 Docker    │
        │  Jarvis 2 core · Whisper ASR   │
        │  embeddings · homelab-monitor  │
        └───────────────┬────────────────┘
                        │  Tailscale
        ┌───────────────▼────────────────┐
        │       Vader (inference)        │
        │  vLLM · Qwen3.8-27B (262K ctx) │
        │  2× RTX 3090 · 48GB VRAM       │
        │  tensor-parallel · port 8010   │
        └────────────────────────────────┘

Qwen3.8-27B is dense 27B, Apache 2.0, 262K context, vision + video. I picked a dense 27B over a bigger MoE on purpose: one model that fits in 48 GB and is fast, not a 2.4T monster that needs a cluster. Under load Vader draws ~685 W at 80 °C. It's a server, not a laptop.

If you're going to do this

Four things I'd tell my past self:

VRAM is the budget, not FLOPs. A 27B dense model at Q8/Q4 fits two 24 GB cards with headroom. That's why I went dense: the math is simple — weights + KV cache ≤ 48 GB. A 2.4T MoE would need a cluster and a mortgage.
Split the box. Inference on its own machine, everything else (the assistant core, ASR, embeddings, the monitor) on the hub, talking over Tailscale. When the GPU pegs at 100%, the rest of the house keeps working. One box doing everything is where "local AI" dies.
vLLM over Ollama for serving. Ollama is fine for tinkering; vLLM with tensor parallelism is what makes 48 GB feel like one big GPU and what gets you real tokens/sec. Port 8010, OpenAI-compatible API — your existing clients just point at it.
Measure the power, or you're guessing. "It's a 685 W box" only matters if you can attribute that draw to the process and price it. I built homelab-monitor for exactly that — VRAM and power per process, energy priced at your tariff. The whole hub ran 4.97 kWh this week: 1.30 BGN / 7 days. If you run more than one box, it's the thing I'd reach for first.

The honest part

Tests continue. The hardest agentic chains still make me double-check. But for the daily 80%, it's reliable enough that I stopped caring. A local model you have to think about is a toy. One you don't is infrastructure.

Drafted with AI assistance. Hardware, cost and task data are from my own homelab, measured live.

Does a Second GPU Increase Ollama's Context Window? (Quadro P2000 + RTX 3090 Tested)

Arsen Apostolov — Thu, 09 Jul 2026 12:37:52 +0000

TL;DR

Short version: no. I dropped a much older GPU (Quadro P2000, 5GB, Pascal, 2016) next to an RTX 3090 (24GB, Ampere) on the same box, ran the same context-length ladder (8K→128K) through Ollama and vLLM on qwen3-coder:30B-A3B, and got zero extra usable context in either engine — and a 74% decode-speed hit for the trouble. Ollama hits the identical Chunk too big wall at ctx=65536 whether the P2000 is there or not. vLLM refuses tensor-parallel across the two cards entirely — not a VRAM problem, a flat compute-capability rejection (Minimum capability: 75. Current capability: 61.) that fails in 40 seconds, before any memory profiling. And the one real, measured effect of adding the P2000 to Ollama: decode speed goes from 76 → 19.5 tok/s at ctx=49152 once the P2000 gets pulled in as an actual compute device.

Full narrative version — the two-stage collapse, the prompt-cache validation bug caught mid-sweep, the CUDA13-silently-drops-Pascal finding — is on Medium.## The setup

ardi (dual Xeon E5-2680 v4, 128GB RAM, openSUSE Leap) has a Quadro P2000 sitting in a second slot next to the RTX 3090 this whole series has run on so far. Same model as phase 1 (qwen3-coder:30B-A3B), same box, four legs: {Ollama, vLLM} × {3090 only, 3090+P2000 tandem}, priced through HomeLab Monitor against real GPU power draw.

Ollama: same wall, extra tax

ctx	3090 only decode tok/s	tandem decode tok/s	P2000 VRAM (tandem)
8,192	124.3	122.0	6 MB / 0%
24,576	108.2	70.0	62 MB / 0%
32,768	99.4	61.0	62 MB / 0%
49,152	75.7	19.5	3,580 MB / 55%
65,536	fatal: `Chunk too big`	fatal: identical `Chunk too big`	—

Two separate costs, not one: decode already falls behind at ctx=24576 while the P2000 is still basically idle (62MB, 0% util) — some scheduling overhead just from having a second visible device. Then the real collapse hits at ctx=49152, when the P2000 actually gets pulled into the compute path (3.58GB, 55% util) and decode craters to 19.5 tok/s. Same context ceiling either way, worse speed the whole way there.

vLLM: doesn't even get to try

Expected failure mode going in: tensor-parallel splits the ~17GB AWQ checkpoint roughly in half, and the P2000's 5GB doesn't hold its ~8.5GB share. Actual failure, at ctx=8192, in 40 seconds, before any memory profiling:

ValueError: The quantization method auto_awq is not supported for the current GPU.
Minimum capability: 75. Current capability: 61.

AWQ's Marlin kernel needs compute capability 7.5+ (Turing and later). The P2000 is 6.1 (Pascal). Not a close VRAM call — a flat architectural exclusion, decided before capacity is even checked.

Bonus finding: Ollama's own CUDA13 build almost drops the P2000

Boot log, before any of the above:

skipping CUDA device — compute capability not in compiled architectures
device="Quadro P2000" cc=610
archs="[750 800 860 870 890 900 1000 1030 1100 1200 1210]"

Falls back to a legacy cuda_v12 runtime that does support Pascal — so it works, just via a path most people wouldn't notice without reading boot logs. This 2016 card is now old enough that modern quantized-inference stacks are starting to architecturally step around it, not just outrun it.

What wasn't the point of this one

Not claiming a second GPU is never worth it — a matched pair, or a smaller-but-newer card, is a different setup entirely. This was specifically: does this 5GB Pascal card, next to this 3090, on these two engines, buy anything. Check compute capability against your quantization scheme before you do the VRAM math — it can end the conversation first.

Every number above priced through HomeLab Monitor — open source, MIT licensed — against ardi's real GPU power draw. Full write-up with all four charts and the mid-sweep debugging on Medium.What's the oldest card you've tried to tandem into a rig — did it actually pull weight, or did you just assume it was?

Whisper large-v3 VRAM Requirements: Why It Won't Fit on a 5GB GPU (and What We Tried Instead)

Arsen Apostolov — Wed, 08 Jul 2026 03:29:30 +0000

TL;DR

whisper-large-v3 OOMs on a 5GB GPU (Quadro P2000) at float16, int8_float16, and full int8 — before serving a single request. Root cause is architecture overhead (32-layer encoder-decoder, activations, CUDA context), not just weight size. Fine-tuned whisper-tiny → base → small → small-v2 on Common Voice Bulgarian instead: held-out WER improved from 88.2% → 32.7% across escalating model size, but never closed the gap to large-v3's 27.3%. A community large-v3-turbo Bulgarian fine-tune claiming 9.97% WER on FLEURS scored 31.2% on our own held-out set — same ballpark as our own model, not the win the model card implied. Built a real dual-GPU nginx failover (P2000 = fine-tune, 3090 = large-v3) that worked correctly on deploy, then failed a real spontaneous-speech test badly enough to roll back to large-v3-only within ~5 seconds. Core finding: Common Voice read-aloud WER does not predict real assistant-use transcription quality.

The setup

ardi has one RTX 3090 (24GB) doing LLM inference work, and a Quadro P2000 (5GB) that's sat idle for about two years. Jarvis, a self-hosted assistant, depends on Whisper for speech-to-text — testing showed only large-v3 handles Bulgarian well; smaller stock checkpoints are fine for English, not for a lower-resource language. large-v3 sits permanently loaded on the 3090, the same card needed for local LLM serving.

Question: can the idle P2000 take Bulgarian transcription off the 3090's hands via a Bulgarian-specific fine-tune small enough to fit 5GB?

(One naming note so the rest of this makes sense: the container running here is whisper-asr-webservice wrapping faster-whisper — not the separate WhisperX project, despite what I've been calling it internally for months.)

Attempt 1: does large-v3 just fit?

Tested large-v3 on the P2000 at three precisions:

float16        -> OOM
int8_float16   -> OOM
int8           -> OOM

All three OOM before serving a request. Not a quantized-weight-size problem — the encoder-decoder's non-weight overhead (32 layers, activations, CUDA context) exceeds 5GB regardless of precision. whisper-tiny loads at 318MB with no issue, ruling out a driver/compatibility problem. medium (769M params) was the practical ceiling for raw model size — 3.87GB used, 1.2GB headroom — but a generic multilingual medium isn't good enough for Bulgarian on its own.

Attempts 2–4: escalating fine-tunes

Fine-tuned on Mozilla Common Voice Bulgarian, on the 3090, via HuggingFace transformers Seq2SeqTrainer. Evaluated on the same 150 held-out test clips (never seen in training) for every model:

small-v2 = same architecture as small, retrained on train+other combined (6,739 rows vs 4,952) for 5 epochs. Validation WER by epoch: 32.17 → 28.99 → 28.21 → 28.21 → 28.44 — flattened, then rose at epoch 5 (overfitting), so load_best_model_at_end correctly kept the epoch 3/4 checkpoint rather than the final one. No more clean Bulgarian Common Voice data exists beyond train+other, so this is the practical ceiling for this data/model-size combination.

Two gotchas caught along the way:

# 1. CUDA_VISIBLE_DEVICES alone doesn't guarantee GPU index matches
# nvidia-smi's PCI-bus order -- a run silently landed on the P2000
# instead of the intended 3090 until:
export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=1

# 2. ardi's root disk (already at a tight 90% baseline) filled to 100%
# mid-training from accumulated dataset/HF caches -- silent SIGKILL,
# no traceback. Fixed by pointing the cache at a bigger volume instead
# of the system disk:
export HF_HOME=/backup/hf-cache
export CACHE_DIR=/backup/whisper-bg-tiny-data

Neither is the interesting part of this story, but both cost real debugging time — worth checking explicitly on any shared multi-GPU box.

Attempt 5: the community shortcut that didn't reproduce

Searched Hugging Face for an existing Bulgarian ASR fine-tune before pushing further on limited training data. Found sam8000/whisper-large-v3-turbo-bulgarian-bulgaria — a fine-tune of large-v3-turbo (same 32-layer encoder as full large-v3, decoder pruned from 32 to 4 layers), claiming 9.97% WER on the FLEURS Bulgarian benchmark.

Converted to CTranslate2, it does fit the P2000 — 4.1–4.2GB used, ~900MB headroom — tight but real (the third bar in the VRAM chart above). Evaluated on the same held-out Common Voice test set used for every model above:

sam8000/whisper-large-v3-turbo-bulgarian-bulgaria: 31.2% WER
our own small-v2 (fine-tuned):                     32.7% WER

Statistically the same result, not the dramatic win the model card implied. The 9.97% FLEURS number isn't fake — it just doesn't transfer to a different eval set with different preprocessing/normalization. Always re-measure a candidate on your own eval, apples to apples, before trusting a model card's headline number.

The part that worked: dual-GPU failover

Built a real deployment: two whisper containers (P2000 = small-v2, 3090 = large-v3 unchanged) behind an nginx sidecar using proxy_next_upstream for automatic failover. One detail that shapes what "failover" means here: whisper-asr-webservice loads its model eagerly at process boot, not per-request — so this isn't a live per-call fallback, it's "is this backend up or down," decided once at startup.

Deployed live, confirmed it actually worked — routing correct. The old standalone production container was kept stopped, not deleted, for the entire session — the eventual rollback was a container start, not a rebuild.

The test that actually mattered

Real spontaneous speech — describing colors and objects out loud, not Common Voice-style read sentences. Verdict: "quite, quite, quite weak." Noticeably worse than the 32.7% benchmark WER suggested for casual listening. Rolled back to large-v3-only production immediately — ~5 seconds, because the old container was never torn down.

What we deliberately didn't do next

Didn't publish a GitHub repo for the fine-tuned checkpoints — the result isn't good enough to ship as a "solution."
Didn't chase a 6th fine-tune attempt (medium-size, more data augmentation) — diminishing returns were already visible in the epoch curve, and the deeper problem (domain mismatch between read-aloud and spontaneous speech) wouldn't be fixed by more of the same data.
Didn't keep the dual-GPU stack running "just in case" — production reverted to exactly its pre-session state, P2000 idle again.

The actual finding

Common Voice is people reading prepared text aloud in clean conditions — a different domain from spontaneous conversational speech directed at an assistant (prosody, hesitation, mic quality, vocabulary). A benchmark WER on read-aloud speech didn't predict real assistant-use quality here, for either our own fine-tune or a community model claiming a much better number on a different benchmark. This generalizes past Bulgarian and past Whisper: eval-set domain match matters more than the headline metric.

Full narrative version — the charts, the physical GPU install photo, the "why I still don't have a use for this card" ending — on Medium.

Every VRAM ceiling and WER number above was measured via HomeLab Monitor — MIT licensed, one container, the same tool that's priced every benchmark in this series.

Curious if anyone's gotten a Bulgarian (or other lower-resource-language) Whisper fine-tune to hold up on real spontaneous speech, not just a read-aloud benchmark — and what closed the gap if so.

vLLM vs llama.cpp vs Ollama: What Happens When Your Model Doesn't Fit in 24GB VRAM

Arsen Apostolov — Sun, 05 Jul 2026 05:54:01 +0000

TL;DR

Benchmarked llama.cpp, Ollama, and vLLM across 5 models (1B to 116.8B params) on one RTX 3090 (24GB) + 128GB RAM home-lab box, priced through HomeLab Monitor. Inside 24GB, vLLM's continuous batching scales aggregate throughput 3.9x-5.4x from concurrency 1 to 8 (llama.cpp only manages 1.2x-1.9x, even with -np 8 explicitly set to match). Past 24GB — two models deliberately chosen to force RAM-spill — llama.cpp and Ollama both degrade to single-digit tok/s and keep generating. vLLM OOMs outright on both, at the same ~22.1-22.2GB-used / <700MB-free ceiling, regardless of quantization scheme. Sub-plot: llama.cpp's manually-tuned layer offload beats Ollama's automatic split by 37x on time-to-first-token during RAM-spill, while landing on nearly identical steady-state decode speed.

The roster

Model	Vendor	Type	Fits in 24GB?
Gemma 3 1B	Google	dense	yes
Qwen3-Coder 30B-A3B	Alibaba	MoE (~3.3B active)	yes
Gemma 4 26B-A4B	Google	MoE (~4B active)	yes
GLM-4.5-Air 106B-A12B	Zhipu	MoE (~12B active)	no, deliberately
GPT-OSS 120B-A5.1B	OpenAI	MoE (~5.1B active)	no, deliberately

(Gemma 4 is real — Google's newest release as of this writing, not a Gemma 3 typo.)

3 prompt tiers (short/medium/long), concurrency 1 and 8, 2 reps per cell, 15 backend×model pairs total. Caveat stated up front: the first three models ran against my production Ollama (OLLAMA_NUM_PARALLEL=1, serialized by default — real daily-use config); GLM and GPT-OSS ran against a separate isolated instance (OLLAMA_NUM_PARALLEL=4) since they needed a clean volume anyway. Ollama's concurrency=8 numbers for the first three models are not its concurrency ceiling — they're its actual default production behavior.

Concurrency, inside 24GB

Aggregate decode tok/s, concurrency 1 → concurrency 8:

Model	Ollama	llama.cpp	vLLM
Gemma 3 1B	125.6 → 71.4	294.1 → 400.6	235.5 → 1172.1
Qwen3-Coder 30B-A3B	129.3 → 108.4	157.2 → 183.9	172.0 → 677.9
Gemma 4 26B-A4B	84.5 → 78.5	118.8 → 220.6	133.8 → 723.4

vLLM's own c1→c8 scaling: 3.9x-5.4x (paged attention, requests slot into idle cycles). llama.cpp's, even with -np 8 matched to the concurrency level: 1.2x-1.9x — it pre-declares a fixed KV-cache reservation per parallel slot before the server starts, so concurrency is a config decision, not a runtime one. Head-to-head at c8: vLLM beats llama.cpp by 2.9x-3.7x, beats Ollama's serialized default by 6.3x-16.4x (caveat above applies).

The cliff, and vLLM's wall

GLM-4.5-Air (~52% of layers spilled to system RAM under llama.cpp's tuning) and GPT-OSS-120B (~67% spilled) were picked specifically to not fit. llama.cpp and Ollama both ran them — slow, single-digit tok/s, but real generation, no crash. vLLM failed outright on both:

# GPT-OSS-120B, native MXFP4, --cpu-offload-gb 45
OutOfMemoryError: CUDA out of memory. Tried to allocate 1.08 GiB.
GPU 0 has a total capacity of 23.56 GiB of which 533.69 MiB is free.
Process ... has 22.21 GiB memory in use.
RuntimeError: Engine core initialization failed.

# GLM-4.5-Air, pre-quantized AWQ, --cpu-offload-gb 36
OutOfMemoryError: CUDA out of memory. Tried to allocate 1.16 GiB.
GPU 0 has a total capacity of 23.56 GiB of which 685.69 MiB is free.
Process ... has 22.12 GiB memory in use.

Same shape, different model, different quantization path. I retried GLM at --gpu-memory-utilization 0.78 (down from 0.90, to force more declared headroom) — got the byte-for-byte identical error: 22.12 GiB used, 685.69 MiB free, 1.16 GiB requested. That rules out the utilization knob as the fix; the base weight + offload footprint is already pinned at the ceiling before profiling starts. Two models, two quant schemes, same ~22GB wall — reads as a real limit of vLLM's CPU-offload path for >100B-param MoE on one 24GB card on this stack, not a per-model quirk.

TTFT: the 37x gap that steady-state doesn't show

On the models that ran everywhere, steady-state decode is nearly a tie once warmed up — GPT-OSS-120B's longest tier: 7.65 tok/s (llama.cpp) vs 7.6 tok/s (Ollama). GLM: 4.58 vs 4.59. Time-to-first-token is a different story:

Model	Ollama TTFT	llama.cpp TTFT	Gap
GLM-4.5-Air	13.6s	8.1s	1.7x
GPT-OSS-120B	274.0s	7.3s	37x

llama.cpp's -ngl is a number I computed myself from the model's real config.json (layer count, per-layer size) — -ngl 12 for GPT-OSS, offloading ~21GB deliberately. Ollama figures the split out automatically at load time, and on a freshly-pulled, partially-RAM-resident 65GB model, that automatic path is expensive. Same destination, very different path there.

What it costs (BGN per 1M output tokens, real GPU energy)

Model	Ollama	llama.cpp	vLLM
Gemma 3 1B	0.19	0.05	~0*
Gemma 4 26B-A4B	0.25	0.14	0.04
Qwen3-Coder 30B-A3B	0.16	0.13	0.04
GLM-4.5-Air	2.61	1.95	OOM
GPT-OSS-120B	10.00	1.43	OOM

*vLLM's Gemma 3 1B run finished in 6s — too fast for the power sampler to catch a reading, recorded near-zero. A sampling limitation on short bursts, not a genuine free result.

GPT-OSS-120B on Ollama costs ~7x more real electricity per million tokens than llama.cpp for the identical model — the TTFT convenience tax from above, showing up again in currency.

Three disclosed vLLM checkpoint swaps

The original plan was on-the-fly bitsandbytes 4-bit quant for every vLLM leg. It failed for every MoE model, for three distinct, verified reasons — not the same error copy-pasted three times:

Qwen3-Coder-30B: ValueError: BitsAndBytes quantization with padded hidden_size ... Parameter shape (786432, 1) != checkpoint shape (2048, 768) — bnb can't dequantize this MoE's padded expert layout. Fix: pre-quantized AWQ checkpoint. Ran clean after (677.9 tok/s aggregate @ c8).
Gemma 4 26B-A4B: AttributeError: MoE Model Gemma4ForConditionalGeneration does not support BitsAndBytes quantization yet. A new architecture, bnb path not wired up yet. Fix: a different pre-quantized checkpoint — which then hit a pydantic error because its config.json says compressed-tensors, not AWQ, despite the repo name. Fixed by dropping the explicit --quantization flag entirely and letting vLLM auto-detect.
GLM-4.5-Air: not a failure — a practicality call. Skipped a 212GB native bf16 download to test a bnb+MoE+CPU-offload combo the vLLM community already flagged as shaky, went straight to a ~63GB pre-quantized AWQ checkpoint that tests the exact same question.

Every root cause above came from the actual container logs, not from assuming precedent carried over from the previous model's failure.

What wasn't tested

Only two --gpu-memory-utilization values before accepting the OOM as final, not a full --cpu-offload-gb sweep. No multi-GPU / tensor-parallel vLLM path — a different question from "does single-card CPU offload work." Ollama's c8 numbers for the first three models are its production default, not its concurrency ceiling. And one raw llama.cpp per-request timing (Gemma 4, medium tier, c8) self-reported an impossible 250,024 tok/s from a near-zero-duration completion — the aggregate figures used throughout are total-tokens-over-wall-time, which isn't corrupted by that, but it's a known rough edge in the raw per-request logs.

Full narrative version, with the RAM-spill mechanics and the redacted dashboard screenshot: on Medium.

Every number above was priced through HomeLab Monitor — open source, MIT licensed — against the RTX 3090's real power draw.

If you're already running one of these three backends: has yours ever tried to load something that just didn't fit — and did it fail loud or fail quiet?

Local LLM vs Claude: Benchmarking qwen3-coder:30b as a Production Agent Backend

Arsen Apostolov — Fri, 03 Jul 2026 11:16:36 +0000

TL;DR

Replayed 27 real historical tasks from Jarvis (my LangGraph agent, ~90 tools) through qwen3-coder:30b on an RTX 3090, scored against Claude's actual production answers to the same tasks. Quality: Claude 89.4/100 vs qwen 22.8/100. Cost: qwen ~5,150x cheaper per task ($0.00015 vs $0.763, real GPU electricity vs real API billing). Reliability: qwen leaked malformed tool-call tags into 26% of answers and only overlapped with the tools the task actually needed 14.8% of the time. Same qwen3-coder:30b scored 100% in an earlier, much smaller benchmark — the gap here is about tool-surface complexity, not the model being bad.

The question

Jarvis is a real personal AI agent — LangGraph create_react_agent, ~90 tools spanning email/calendar/notes/files/messaging/code, running on Claude in production. qwen3-coder:30b had already scored 100% task success in a controlled 17-task benchmark on the same RTX 3090. Obvious next question: drop it into the real agent and see what happens.

The setup

28 real task prompts pulled from Jarvis's own Langfuse traces (90-day window), stratified 4×7 across calendar / code / email / files / general / messaging / notes.
Claude's answers are real production history, not re-run. Re-running through the sandbox would hand it fake stub data it never saw — that's a worse baseline, not a fairer one.
qwen runs fresh, through a sandboxed replay harness: the real Jarvis agent code in-process, every write-capable tool intercepted (nothing sent/written for real), and every mocked read-only tool serves the real recorded output from that task's original trace when available — not a generic stub. Same data, both models.
1/28 tasks excluded (336,906-char prompt, over any 16K–24K context window) → 27 scored.
Judge: LLM-as-judge (claude-opus-4-8), scored independently per answer (not pairwise) to avoid position bias, 1–5 → 0–100.
Every qwen run priced as a HomeLab Monitor experiment against real 3090 power draw. Claude's cost is Langfuse's recorded API billing.

Caveat, stated plainly: the judge is a Claude model scoring Claude's own answers alongside qwen's — self-preference bias is a documented effect in LLM-as-judge setups and probably inflates the gap somewhat. It doesn't explain a 66-point gap, a 26% malformed-output rate, or two tool-call loops, but it's a real methodology limitation, not a footnote.

Getting here took three re-runs: a judge-response parsing bug that silently neutral-scored ~40/54 calls, a mock-data bug that starved qwen of real inbox/calendar content on 16/28 tasks while Claude's baseline had the real thing, and a Claude-API rate limit that neutral-scored another batch mid-scoring. All three caught by checking score distributions, not by trusting a clean exit code — worth knowing before trusting the numbers below.

The numbers

	Claude	qwen3-coder:30b
Avg quality (0–100)	89.4	22.8
Cost / task	$0.763 (real API billing)	$0.00015 (real GPU electricity)
Total cost, 27 tasks	$20.60	$0.004
Total energy	—	0.0396 kWh

~5,150x cheaper per task for qwen (precise, currency-converted from a 0.0072 BGN total across all 27 tasks, at 1 BGN = $0.5547 — an earlier rough estimate of 180x on this project was wrong, this is the corrected number).

By category (Claude | qwen | n):

calendar:   90 | 30 | 4
code:       87 | 25 | 3
email:      92 | 15 | 4
files:      88 | 15 | 4
general:    85 | 30 | 4
messaging:  87 | 22 | 4
notes:      97 | 22 | 4

qwen's best relative showing (calendar, general) is still a third of Claude's score. It never wins a category.

Where it breaks

Malformed tool-call leak — instead of a real LangGraph tool call, qwen sometimes emits the call as raw text in its final answer:

<function=send_email>
{"to": "...", "subject": "...", "body": "..."}
</function>

That happened on 7/27 tasks (26%). The user reading that answer sees broken syntax where a real action should have been confirmed or a real answer given.

Tool-overlap recall: 14.8% average, measured over the 18/27 tasks where the original historical trace actually used at least one tool (9 tasks needed none). Most of the time qwen reached for different tools than the ones that actually solved the task — or none.

Repetitive-loop failure on 2/27 tasks: pilot-17 (email, 24 tool calls, 138.6s, ~196.7K input tokens) and pilot-27 (messaging, 27 tool calls, 148.9s, ~196.7K input tokens) both called the same already-answered tool (run_command, todo_write) repeatedly instead of stopping. Confirmed via raw logs both tasks got real replayed data (replayed_real_data: true) — a genuine stopping-condition failure, not a data-starvation artifact.

One more data point worth having, not a verdict: on a task where both models actually called send_email(...) in the harness (intercepted, nothing sent), Claude told the user the email had been sent — a fabrication. qwen correctly disclosed the send didn't go through. Not "qwen is more honest" — it's also the model leaking raw tags 26% of the time. Both mishandled the mock, just differently.

Scope of the claim

Same qwen3-coder:30b, same GPU, scored 100% on a 17-task controlled benchmark with a much smaller tool surface. This isn't "local LLMs are bad" — it's that a model excellent on a scoped benchmark isn't automatically a safe drop-in for a large, real, ~90-tool production surface with a 31KB context prompt and real messy history behind it. Task/tool-surface complexity mattered as much as raw model quality here. Claude isn't flawless either — see the fabricated send-email confirmation above.

Jarvis stays on Claude for now. The cost number is real enough to be worth a narrower follow-up — testing qwen on just the categories where it scored closest (calendar, general) as a cheap fallback path, rather than a full swap.

Full narrative version, charts, and the three-bug scoring saga: on Medium.

Every qwen run here was priced through HomeLab Monitor against the 3090's real power draw — MIT licensed, one container, reproducible if you want to price your own local-model experiments the same way.

Curious where the line is for you: how cheap does a local model have to be before you'd trust it with a slice of a real agent, and which slice would you pick first?

How to Run Reliable Local LLM Agents on an RTX 3090: A Benchmark (5 Models, Priced in Watts)

Arsen Apostolov — Sun, 28 Jun 2026 06:54:12 +0000

I gave GLM-4.5-Air (106B, open weights) 12 coding tasks through opencode on my RTX 3090. It scored 0% — never edited a single file.

Same model, same GPU, same tasks, but driven by a ~150-line LangGraph agent instead: 93%.

The model was never the problem. The orchestrator was. Here's the benchmark — including the part nobody else measures, the electricity cost per correct task.

Setup

RTX 3090 (24 GB) + 128 GB RAM, models via ollama, Q4 quants, temp 0.2
5 recent open models × 2 orchestrators (opencode vs custom LangGraph ReAct with ollama-native tool-calling)
17 graded tasks (12 coding in Python/JS/C++ + 5 general-agent) with hidden unit tests
Every run priced in GPU watts via my open-source homelab-monitor

Results

Model	tok/s	opencode adh.	LangGraph adh.	LangGraph coding	LangGraph general
Qwen3-Coder 30B-A3B	130	92%	100%	100%	100%
GLM-4.5-Air 106B	5.7	0%	100%	89%	100%
Devstral Small 24B	49	8%	53%	8%	40%
Seed-OSS 36B	9.5	0%	7%	0%	20%
DeepSeek-R1-Distill 32B	6.7	0%	0%	0%	0%

Tool-adherence = % of tasks where the model actually called a tool instead of just printing code in chat. It was the master variable. (GLM's headline "93%" is its blended score across all 17 tasks: 89% coding + 100% general.)

Three takeaways

The framework can matter more than the model. opencode sends a frontier-shaped system prompt + 12 tools over its OpenAI-compat path; most local models fall back to chatting. Native tool-calling through a lean agent fixes that — GLM went 0% → 93%. (Qwen3-Coder is the exception: it's tuned for agentic tool use and aces opencode out of the box.)
Acting ≠ solving. LangGraph made Devstral act (8% → 53% adherence) but not solve (coding stayed 8%). The framework decides whether a model acts; the model decides whether it's right.
The wattmeter ranks honestly. Qwen solved tasks at ~0.0005 BGN each; the models that scored zero still burned 10–30× more energy for nothing. On a home rig, the cheapest model is the fast, correct one — and MoE (Qwen activates ~3B of 30B per token) wins twice.

Bonus: 128 GB RAM let me run the 106B GLM (23 GB VRAM + 27 GB spilled to RAM) — it works, at 5.7 tok/s. Great for fire-and-forget batch jobs, not interactive coding.

The recipe for reliable local agents

Pick a tool-use-tuned model (Qwen3-Coder 30B-A3B is the all-weather winner) → use native tool-calling, not an OpenAI-compat path → keep the harness lean → use RAM for reach, not speed → measure correctness per kWh.

📖 Full write-up with methodology, charts, and the deeper "why" → [https://medium.com/@arsen.apostolov/local-llm-agents-on-an-rtx-3090-i-benchmarked-5-models-2-frameworks-and-the-orchestrator-f5fd600ca221]

⭐ Every number was priced in watts by homelab-monitor — my open-source tool that turns your GPU's power draw into per-task cost. Star it if you want the same receipts for your own rig. Harness + tasks + leaderboard code are reproducible.

How to Rank Local LLMs by Cost per Correct Answer (Measured GPU Energy, 8 Ollama Models)

Arsen Apostolov — Tue, 23 Jun 2026 18:11:23 +0000

TL;DR: I priced 8 local Ollama models by € per 1,000 correct answers — metered GPU energy ÷ correct answers, on one RTX 3090. gemma4:26b won at 96.9% accuracy for €0.013/1k-correct. The most expensive model (qwen3:8b-fp16) cost €0.239/1k and scored worse (66.7%). Reasoning tokens and full precision both cost a lot and bought nothing here. Every cost comes from real metered kWh via the open-source HomeLab Monitor.

This is the short, copy-pasteable version. The narrative writeup is on Medium.

The metric

€ per correct answer = (metered GPU energy cost over the eval window) ÷ (number of correct answers)

Tokens-per-euro flatters whichever model talks the most. Cost-per-correct only rewards being right cheaply — which is the thing you actually pay for.

The signal

Model                  VRAM     Acc     Tok/task  Tok/s  Wh/pass  €/1k correct (day)
gemma4:26b             16.9 GB  96.9%   68        86     4.5      €0.013   ← winner
gemma3:1b              0.9 GB   82.1%   125       133    3.8      €0.013
gemma3:27b             17.1 GB  100.0%  119       36     16.3     €0.046
qwen3:30b-a3b   (MoE)  18.4 GB  83.3%   555       186    14.1     €0.048
qwen3:8b (Q4_K_M) 🧠   5.4 GB   64.8%   626       126    22.7     €0.100
qwen3:8b          🧠   5.4 GB   64.8%   626       126    23.6     €0.104
qwen3:8b (Q8_0)   🧠   8.7 GB   61.1%   672       88     33.5     €0.156
qwen3:8b (fp16)   🧠   15.5 GB  66.7%   664       53     56.2     €0.239   ← most expensive

🧠 = reasoning/thinking mode on. Night tariff knocks ~40% off every row.

Three things the numbers say

1. The value champion is mid-size, not max-size. gemma4:26b hit 96.9% for €0.013 per 1,000 correct — cheapest-per-correct on the whole bench and near-perfect, ~18× cheaper per correct answer than qwen3:8b-fp16. gemma3:27b is the only 100% model but costs ~3.5× more (slower, 36 tok/s).

2. The thinking tax is real and didn't pay off. qwen3 reasoning models emit 555–672 tokens/task vs the gemmas' 68–125 (5–9×). Tokens are energy. On these 54 deterministic tasks that extra reasoning bought no correctness — the priciest model scored lower than one 18× cheaper. (Caveat: this suite is arithmetic / executable code / format-following. On open-ended hard problems, reasoning earns its tokens. On structured agent work, it was dead weight.)

3. The quantization paradox. Same qwen3:8b at three precisions:

            Accuracy   Energy/pass   Throughput
Q4_K_M      64.8%      22.7 Wh       126 tok/s
Q8_0        61.1%      33.5 Wh       88  tok/s
fp16        66.7%      56.2 Wh       53  tok/s
            └ flat ┘   └ 2.5× ┘      └ halved ┘

Higher precision cost 2.5× the energy and half the throughput for accuracy that's flat-and-noisy. On a 3090, aggressive quant was the correct call, not a compromise.

Methodology (so you can trust the ranking)

54 deterministic tasks, mechanically graded — no LLM judge. Reasoning 15 (GSM8K-style numeric extraction), code 12 (HumanEval-style, executed asserts in a sandbox), factual 12 (keyword), instruct 15 (format predicates). Grader selftest 11/11.
Controls identical across all 8 models: temperature 0, seed 42, num_ctx 4096, num_predict 1024, identical prompts.
Warm-up discarded → model-load energy excluded (pricing inference, not cold starts).
3 passes each, ranges reported.
Idle baseline = 38 W, measured as a control.
qwen3 thinking left on (realistic); thinking tokens counted for energy, stripped before grading.
Honest determinism caveat: Ollama is not bit-exact at temp 0. gemma3:1b drifted 81–83% across passes; gemma3:27b was 100% on all three; qwen3 runs were identical. Report ranges, not point claims.
CPU/DRAM not metered (no RAPL on this host), so true wall-plug cost is a bit higher — but the ranking holds because every model paid the same un-metered overhead.

The currency gotcha (measure twice)

Costs are EUR from measured kWh × Bulgarian dual tariff (€0.1534 day / €0.0920 night). While building this I caught my own dashboard mislabeling BGN as EUR: the tariff read 0.30/0.18 EUR, but those are leva. Bulgaria joined the euro on 2026-01-01 at fixed 1 EUR = 1.95583 BGN; €0.30/kWh would be German-tier, implausible for the EU's cheapest household power. Converted: 0.30 / 1.95583 = €0.1534, 0.18 / 1.95583 = €0.0920. Lesson: don't trust the dashboard's € field — compute from physical kWh and your verified tariff.

How to reproduce the energy tracking

Every cost above came from HomeLab Monitor (MIT, one container) — its Experiments tab integrates real GPU power over a run's window into kWh and money. Bring it up:

docker compose up -d        # port 9800

Grab the one-file homelab_run.py client, mint an ingest key, and wrap your eval — the run comes back priced:

import homelab_run as homelab
homelab.configure(url="http://<your-host>:9800", key="hlm_…")

with homelab.run("gemma4:26b", tags=["llm-cost-bench"]) as r:
    for _ in range(PASSES):
        run_graded_eval(model)          # all inference inside the run
priced = homelab.pull(r.id)             # energy_kwh, cost, avg_w, peak_util — from real power

That's the whole instrumentation. Divide the priced energy by your grader's correct count and you've got cost-per-correct for your own roster. Docs · docker pull sikamikaniko123/homelab-monitor.

What I deliberately did NOT do

No LLM-as-judge — mechanical grading only.
No cold-start energy in the numbers — warm-up discarded on purpose.
No trusting the dashboard's € field — costs recomputed from measured kWh.
No single-run claims — 3 passes, ranges where they exist.
No CPU/DRAM cost claim — only the GPU is metered, and I say so.

Over to you

Bigger and full-precision lost. A 26B model did near-perfect work for a rounding error; an fp16 reasoning model charged 18× as much to be wrong more often.

So when you reach for a local model — accuracy, speed, or cost per answer it actually gets right? And have you ever measured the third one? Drop your own cost-per-correct numbers in the comments.

How Much Does It Actually Cost to Run a Local LLM? (€ per Million Tokens, Measured)

Arsen Apostolov — Mon, 22 Jun 2026 18:33:31 +0000

"It runs on my own GPU, so it's basically free." I believed that until I put a meter on it. So I ran a controlled benchmark on one box — an openSUSE machine with a single RTX 3090 — driving three local models through ollama under an identical fixed workload (256-token generations in a loop for ~4 minutes each), while my open-source dashboard priced every run by the real GPU energy it burned: power sampled from nvidia-smi every 10 s, integrated over each run's exact window, multiplied by my actual day/night tariff. One number per model, in euros per million output tokens.

Here's the part that made me re-run it. The tiny gemma3:1b came out at €0.118 / 1M tokens — about 5× cheaper than a hosted Flash-class API (~€0.55). But gemma3:27b's electricity alone was €0.706 / 1M — more expensive per token than just paying the cloud, and that's before a single cent of the GPU's purchase price. "Local" didn't make it cheaper; it made it cost more and I own the depreciation. The mechanism is one line: each token costs watts ÷ throughput, and a big dense model is both slow and thirsty. A newer mid-size architecture (gemma4:26b) bought a lot of that back, landing at €0.272.

The full guide is methodology-first and reproducible end to end — minting an ingest key, the stdlib-only client, the exact ollama loop that reads eval_count/eval_duration for real tokens-per-second, reading each run back priced, and the honest caveats (this is marginal GPU energy only — not capex, idle, or cooling — and the absolute numbers round to fractions of a cent; the shape is the finding).

Read the full guide on Medium → https://medium.com/@arsen.apostolov/how-much-does-it-actually-cost-to-run-a-local-llm-per-million-tokens-measured-4a90a7f31a48