DEV Community: Arsen Apostolov

Cloud vs Local: The Break-Even Point Where Self-Hosting Stops Losing Money

Arsen Apostolov — Mon, 09 Nov 2026 08:30:16 +0000

The Local LLM Lab — Part 14

One controlled experiment. One number. One verdict.

"Local LLMs are cheaper than the cloud" is a claim you see constantly. "The cloud is cheaper" is a claim you see just as often. Both are right, both are wrong, and the answer depends on a number most people have never calculated: their own break-even point.

The break-even point is the monthly usage at which self-hosting costs the same as the API for the same work. Below it, the cloud wins. Above it, local wins. It is different for every person, because it depends on three things: what your hardware cost, what your electricity costs, and which cloud model you would otherwise be paying for.

I calculated mine from a metered 30-day bill and a current price sheet. Here is the method, so you can run it on your own numbers.

The local side: what the box actually costs

Two components, and only one of them scales with anything.

1. Hardware amortisation. A used RTX 3090 runs €800–1,200. Midpoint €1,000, amortised over 36 months: €27.78/month. If you already own the card for other reasons, this is €0 and the marginal cost of the inference stack is just electricity.

2. Electricity. This is measured, not estimated. My homelab monitor attributes power per process, and over the 30 days to 2 November the resident inference stack drew:

Service	Energy (30d)	Cost (BGN)
Whisper (ASR, 24/7)	21.73	5.67
Text-to-speech (24/7)	13.47	3.51
Embedding model	5.09	1.32
Everything else GPU-side	0.36	0.09
Total	40.65	10.59

Tariff: 0.30 BGN/kWh by day, 0.18 BGN/kWh by night (22:00–06:00). The bill above is not priced at either one on its own — it is a blend, and it works out to an effective 0.26 BGN/kWh. If you quote a single rate against a metered figure, quote the blended one and say it is blended.

Converted at 1.95583 BGN/EUR, that bill is €5.42 for the month.

So the local monthly cost is:

You already own the card: €5.42/month
You are buying the card: €27.78 + €5.42 = €33.20/month

Note what that number is: the whole stack, running around the clock, including every transcription queue and embedding job it handled in those 30 days. It is close to flat. It does not move much when you ask it more questions.

The part that does scale, and how I know its size. The card measures 40 W with the models resident and nothing queued — that is measured, and it is most of the month. What it draws while actually generating I did not meter this month, so I will not pretend otherwise: the 3090's board power limit is 300 W at spec, and that is a ceiling, not a measurement. At the day tariff, an hour pinned at that ceiling is 0.09 BGN — about €0.046. Two hours a day of it adds €2.76/month, putting the local side at €8.18. Treat that as the worst case for a heavy user, not the typical one.

The cloud side: what the same work costs

List prices, checked against the published price sheets in early November 2026. A "Q&A pair" is the shape of one typical assistant exchange: 500 input tokens + 200 output tokens, so 1,000 pairs is 0.5M input + 0.2M output.

Cloud model	$/M input	$/M output	Per 1,000 Q&A pairs
Claude Sonnet 5.5	$2.00	$10.00	$3.00 (€2.69)
GPT-6.1 Sol	$2.00	$10.00	$3.00 (€2.69)
Gemini 3.8 Flash	$0.75	$3.75	$1.13 (€1.01)
Gemini 3.5 Flash-Lite	$0.30	$2.50	$0.65 (€0.58)
Claude Haiku 5.5	$0.10	$0.50	$0.15 (€0.13)
GPT-6 Luna	$0.10	$0.50	$0.15 (€0.13)

Two things worth knowing before you use these. First, on the premium models output costs 5x input, so a chatty model is a more expensive model independent of quality. Second, several of those rates are introductory: Gemini 3.8 Flash goes to $1.50/$7.50 on 1 January, and the Google flagship that matches the premium tier doubles from $2/$10 to $4/$20 on 1 December. The cheap end of the market is not permanently cheap.

The break-even point

Set local cost equal to cloud cost and solve for pairs per month.

Case 1 — you already own the card (€5.42/month flat)

Comparing against	Break-even	Per day
Claude Sonnet 5.5 / GPT-6.1 Sol	~2,000 pairs	~67
Gemini 3.8 Flash	~5,400 pairs	~179
Claude Haiku 5.5 / GPT-6 Luna	~40,000 pairs	~1,340

Case 2 — you are buying the card (€33.20/month flat)

Comparing against	Break-even	Per day
Claude Sonnet 5.5 / GPT-6.1 Sol	~12,400 pairs	~412
Gemini 3.8 Flash	~33,000 pairs	~1,100
Claude Haiku 5.5 / GPT-6 Luna	~247,000 pairs	~8,200

Case 3 — you own the card and generate heavily (€8.18/month, two hours of generation a day)

Against Sonnet-class: ~3,000 pairs/month, ~102/day. The adder moves the line, it does not move the conclusion.

What the numbers say

The break-even is not "a few times a day." It is dozens. The claim that circulates in the local-LLM crowd — that you recoup a GPU by using a local assistant a couple of times a day — does not survive contact with a meter and a price sheet. Against a premium cloud model, a card you already own pays for its electricity at around 67 exchanges a day. That is a real daily habit, not a casual one.

Against the cheap end, local loses outright. Haiku-class and Luna-class APIs price a Q&A pair at about €0.00013. Your electricity is €5.42 a month before the first token. You would need 40,000 pairs a month to break even — and 247,000 if you had to buy the hardware. If the model you would actually use is a small fast cloud model, do not buy a GPU to save money. You will never catch up.

The hardware decision dominates everything else. Owning the card already moves the break-even from ~12,400 pairs to ~2,000 — a 6x difference, and it is decided before you send a single request. If the GPU is already in the machine for gaming, for work, or for something else entirely, the inference stack is nearly free and the arithmetic flips in your favour. If you would be buying it for this, this article is not the reason to do it.

The honest summary

"Local is cheaper" holds if and only if:

The card is already paid for (or is being bought for some other reason), AND
You use it at the volume the table demands — dozens of exchanges a day against a premium cloud model, hundreds against a cheap one, AND
The local model is accurate enough for the task. A model that is wrong half the time is not cheaper. The cost of a wrong answer does not appear on either side of this equation, and it is usually the largest number in it.

The profile where local genuinely wins is narrower than the community likes to admit: someone who owns a 3090 anyway, runs a local assistant as a daily tool, and is otherwise paying premium-tier API rates. That person is over the break-even line by a wide margin and should keep the box running.

The profile where it loses is the one most often arguing for it: someone considering a first GPU purchase, comparing it against a cheap fast cloud model. That comparison is not close.

Run it on your own numbers

Four inputs, one division:

Hardware: purchase price ÷ 36. Zero if you own it.
Electricity: meter it for 30 days. Do not estimate from a spec sheet's TDP — my stack's real figure is the difference between a card idling warm and a card doing work, and the idle warm part is most of the month.
Cloud price: your actual model, at the input/output token split your actual usage has. Not the model you wish you were using.
Break-even: monthly local cost ÷ cost per 1,000 pairs × 1,000.

The result is a number of exchanges per month. Compare it against what you really do in a month, and you have your answer — which is the only answer that was ever yours to have.

Verdict

If you own the card, you break even against a premium cloud model at roughly 2,000 Q&A pairs a month — about 67 a day. Against a cheap cloud model, 40,000. If you are buying the card, multiply those by six.

The card in my machine cost me €5.42 in electricity last month. Whether that was cheap or expensive depends entirely on which row of that table I would otherwise be paying for — and now I know which row I am in.

The Local LLM Lab is a series of measured experiments on a home GPU stack. The electricity figures here are pulled from 30 days of per-process power attribution on the box; the cloud figures are published list prices as of early November 2026, and several of them change on 1 December or 1 January.

I Ran Whisper on a 24 GB GPU for 30 Days. It Used 8.

Arsen Apostolov — Mon, 02 Nov 2026 08:28:16 +0000

The Local LLM Lab — Part 13

One controlled experiment. One number. One verdict.

The local-AI crowd has a habit: buy the biggest GPU. 24 GB. 32 GB. 48 GB. The reasoning is always the same — "models are big, you need VRAM."

I stopped believing that reasoning when I looked at my own telemetry. I've been running a Whisper transcription service on an RTX 3090 (24 GB) for a few months now. Over the last 30 days I pulled the full VRAM history: every two hours, what the card was holding, what it spiked to, and whether anything ever spilled into system RAM.

The service used 7.9 GB on average. 10.4 GB at peak. Never more. In 30 days.

This is the story of a GPU that is 2.3× bigger than the thing running on it — and what that means for what you should actually buy.

The setup

Card: NVIDIA RTX 3090, 24 GB (24,576 MB), power limit 260 W
Service: Whisper ASR webservice (WhisperX, large-v3), resident Docker container, 24/7
Telemetry: 30 days of VRAM / power / utilisation at 2-hour resolution, plus per-caller attribution
Other residents: a TTS service (~5.3 GB, added partway through the window), an embedding model (negligible), a photo-tagging ML container (negligible)

The 3090 was never sized for Whisper. Whisper is the small, quiet service in the corner of the stack — a transcription endpoint that my own tools call when I record a voice memo, a meeting, or a podcast clip. The big chat models live on a second machine. The TTS service is the one other real tenant; it arrived partway through the measurement window, which is why the "one service" story below is about the Whisper service's own working set, not the card's total.

That's the important part. Most "how much VRAM do I need" answers on the internet are about the model on paper, not the service in production. Paper VRAM is the weights. Production VRAM is weights plus context plus overhead plus whatever the framework decides to allocate. The difference is the whole article.

The number

Average VRAM: 7.9 GB. Peak: 10.4 GB. Spills to RAM: zero.

Metric (30 days)	Value
Total VRAM	24,576 MB
Average VRAM	7,871 MB (~7.9 GB)
Peak VRAM	10,366 MB (~10.4 GB)
Peak / Total	42%
Spills to system RAM	0
GPU utilisation	~0% (bursts to 1–7%)
Power at rest (model resident)	~40 W
Energy attributed to the service (30d)	21.73 kWh / 5.67 BGN

The shape of the trace is boring, and that's the point. It's a flat line at ~7.9 GB with short, sharp spikes to ~10.4 GB when audio comes in, and a few longer elevated stretches when there's a queue. It never trends upward. It never approaches the ceiling. It never spills.

The card sits at ~40 W with the model resident — that's the "keep it warm" cost, not the work cost. The work (the spikes) is short enough that the 30-day total attributed to the service is 21.73 kWh, which at my tariff is 5.67 BGN. The full electricity breakdown is in Part 5 of this series.

What the VRAM is actually holding

People treat VRAM like a single bucket. It isn't. In this service, the ~10.4 GB peak is:

Model weights. Whisper large-v3 is 1.55B parameters. In fp16 that's ~3.1 GB. This is the "paper VRAM" number everyone quotes.
Audio context and feature buffers. The encoder processes audio in chunks; long files and queued files hold more buffer. This is where 3.1 GB becomes 7.9 GB on average.
Framework overhead. CUDA context, cuDNN workspaces, allocator fragmentation. Usually 1–2 GB, invisible in any spec sheet.
The small helper models. WhisperX runs a VAD (voice-activity detection) and an alignment model alongside the main model. Small, but real.

The lesson: a model's "VRAM requirement" is its weights. A service's VRAM requirement is 2–3× its weights. When someone says "large-v3 needs 3 GB," they're telling you the floor, not the working set.

Who is actually calling it

The attribution over the last 7 days is a nice detail, because it shows this is a shared service, not a single app:

Caller	Seconds (7d)
whisper-proxy (my own tools)	3,350
Langfuse MinIO (observability stack)	1,080
Langfuse worker	720
CI runner	300
Warmup job	280

My own tools are the main tenant; the LLM-observability stack is a secondary one — it transcribes the audio attached to the traces it ingests. Even with multiple callers hitting the endpoint, the peak stays at 10.4 GB. The second and third callers add latency, not VRAM, because they queue behind the first.

The counterargument: headroom is insurance

The fair objection: VRAM headroom is insurance. You want room for longer context, bigger batches, a bigger model next quarter. A card that's exactly sized for today's workload is a card that's undersized next month.

True. But the measurement says what the insurance was worth over 30 days of real use: the service used 42% of the card at its absolute peak, 32% on average, and never came close to the ceiling. Insurance that is never claimed is still a cost — in this case, the delta between a 16 GB card and a 24 GB card, plus the power and thermals of running a card that's 2.3× the working set.

The honest version of the headroom argument is: size for the biggest model you will actually run, plus ~50% headroom. Not the biggest model that exists. Not the biggest model the card could run.

The contrast: the other machine

This is where the 3090's real job shows up. The big model — a 27B chat model on vLLM — runs on a second machine with two 3090s. That machine is sized for its workload: a 27B model in fp8 needs well over 24 GB, so two cards, no spills, no compromise.

The hub 3090 is not sized for Whisper. It's sized for "whatever I might want to run on the hub someday." That's a different purchase decision, and it's the one most of us make without measuring. The measurement says: for the workload that actually lives on this card, a 16 GB card is enough for Whisper alone. 24 GB is only justified if a 20B+ chat model is going to share the card — and if that's the plan, you're sizing for the chat model, not for Whisper, and Whisper should be the tenant that fits around it.

The card today: two tenants, still not full

Three weeks into the window a TTS service joined the card. It holds ~5.3 GB. Right now the card's total is ~13.3 GB of 24 GB — 55% at the current moment, still zero spills, still ~2 GB of headroom under memory pressure.

That's the honest current state: two AI services (Whisper + TTS) on one 24 GB card, and the card is not full. The "you need big VRAM" advice would have me buying more. The measurement says the card has room for another tenant of similar size before it gets close to the ceiling. The 24 GB is doing its job as headroom — but it's headroom for a second or third small service, not because Whisper needs it.

What I would buy now

If I were building this stack today, from scratch:

The hub card (Whisper + TTS + embeddings): 24 GB is the right call, but only because of the second tenant. Whisper alone peaks at 10.4 GB; with the TTS service it's ~13.3 GB. A 16 GB card would be tight but workable for just Whisper; 24 GB is comfortable for two.
The chat card(s): sized for the chat model. 27B in fp8 → two 24 GB cards, or one 48 GB card if you can find one at a sane price.
The rule: the GPU is sized for the biggest model you will actually run, plus 50% headroom. Everything smaller is a tenant.

The 3090 remains a great card — it's the best price-per-GB on the used market, and that's exactly why it's everywhere in home stacks. But "best price-per-GB" is an argument for the chat machine, and for a hub that will host a couple of small services, not for the transcription service alone.

Verdict

A 24 GB GPU running a Whisper service for 30 days: 7.9 GB average, 10.4 GB peak, zero spills, 42% of the card at peak. Add a 5.3 GB TTS service and the card is still only 55% full. The "you need big VRAM" advice is true for the model you're afraid of running, and false for the service you're actually running.

Right-size the GPU to the workload you measured, not the workload you imagine. The meter and the VRAM counter don't care about your imagination.

The Local LLM Lab is a series of measured experiments on a home GPU stack. Every number in this article was pulled from 30 days of telemetry on the card, not estimated. The series continues with the cost of keeping all of this alive — the full electricity bill for the whole machine, not just the GPU.

Patzer just got single sign-on — one login for your whole family

Arsen Apostolov — Tue, 27 Oct 2026 09:14:48 +0000

Last week Patzer got single sign-on. If your family already runs Authentik, Keycloak, Authelia or Pocket ID, everyone in the house now logs into the chess server with the same identity they use everywhere else — no more one extra username and password to remember, and no more "I forgot my password" at the dinner table.

How it works

Set OIDC_ISSUER, OIDC_CLIENT_ID and OIDC_CLIENT_SECRET and the login page gets a "Sign in with …" button (you choose the label with OIDC_BUTTON_TEXT). Nothing changes until you set them — existing installs are untouched.
OIDC_ONLY=true removes password login, sign-up and password reset entirely, in the UI and on the server. A fresh install started that way skips the setup wizard, whose password nobody could use anyway: the first SSO login creates the admin.
OIDC_ADMIN_GROUP=patzer-admins makes members of a provider group the admins, re-checked at every login. While no admin exists, only group members can sign in — so on a fresh install the first admin is the first group member, not whoever is fastest.
Logging out of Patzer logs you out of the provider too. That matters on a shared family computer.
OIDC_AUTO_PROVISION=true creates an account for whoever the provider lets in; OIDC_MATCH_BY=username|email links a first SSO login to an existing account instead (email only when the provider marks it verified). Linked accounts get the provider's email if they have none, so password resets and notifications still reach them.

The flow is the standard authorization-code-with-PKCE, with state and nonce, built on openid-client. The part I care about most is the failure handling: a failed SSO login comes back to the login page with the reason in the user's language — provider unreachable, cancelled, no account, already linked, too many attempts — and the callbacks are rate-limited per IP, so Patzer can't be used to flood your identity provider.

Why a chess app needs SSO

Patzer is a family product, and the weakest link in a family app is always the account layer: nine-year-olds reset passwords, siblings share devices, and the last thing you want is a chess server that can't answer "who is this profile?". If your identity provider already knows who your family is, the chess server should just ask it.

SSO is merged to main and ships in the next release (v7.20). The full guide — every environment variable, the SSO-only setup, and an honest security section on what this protects and where its limits are — is in docs/OIDC.md in the repo.

Repo: https://github.com/SikamikanikoBG/patzer — MIT licensed, one Docker container, Stockfish plus your own LLM (Ollama or vLLM).

93 Services, 1 Failed, 1 Probe Down for a Week: How a Home Stack Stays Alive

Arsen Apostolov — Mon, 26 Oct 2026 08:18:46 +0000

The Local LLM Lab — Part 12

One controlled experiment. One number. One verdict.

A home server is a stack of things that can fail. 59 containers. 94 active systemd units. A GPU with models on it. A network of ports, a registry, a CI runner, a media server, a photo library, an LLM observability stack. Any one of them can die, and the question is never "will it fail" — it's "will you know, and will it come back."

I instrumented the whole stack: every container has a health check, every critical service has an uptime probe, and the monitor watches both. Over the last 7 days the probes ran 10,000 checks. This is what they found — and it is not a clean 100%. One probe target has been refusing connections for the entire window, and the monitor flagged it on day one. That is the point.

The setup

Machine: HP Z840 Workstation, 2× Xeon E5-2680 v4, 64 GB RAM, one RTX 3090
Stack: 59 Docker containers, 94 active systemd units (93 running, 1 failed)
Monitoring:
- Container health: Docker's own healthchecks on the containers that define them (the rest report state only)
- Service uptime: two HTTP probes against the homelab monitor's own endpoints, every 60 seconds, 7-day window, fail threshold of 2 consecutive failures
- Event log: edge-triggered alerts for OOM kills, threshold crossings, disk I/O spikes
The two things that are not up: nginx.service (the old reverse proxy, a leftover) and the 9802 probe target (a port that no longer listens). Both are flagged, both are explained, neither is hidden.

The two non-up states are the honest details. They're not mistakes I'm hiding — they're the things that are down, and they're down for known reasons: a retired reverse-proxy unit, a retired probe port. The monitor sees them, flags them, and I left them, because the roles they used to fill no longer exist behind them. The article is about the detection, not the failures — the failures are the proof that the detection works.

The number

Over 7 days, 10,000 uptime checks ran against two endpoints. One returned 100% — 5,000 checks, in the low-teens of milliseconds, zero failures. The other returned 0% — connection refused on every check — and the monitor has flagged it as a critical since day one.

Check	Target	Interval	Checks (7d)	Uptime	Latency	Failures
HomeLab 9800	http://ardi:9800	60 s	5,000	100%	~14 ms	0
HomeLab 9802	http://ardi:9802	60 s	5,000	0%	refused	5,000

The 0% is the number that matters. Not "the stack is up" — that's a state. "One endpoint up, one endpoint refusing connections for a week, both named" is a measurement. The 9800 endpoint answered 200 in about 14 ms on every check. The 9802 endpoint refused the connection on all 5,000. The fail threshold is 2 consecutive failures, so the 9802 alarm fired at the 2-minute mark on day one and has not cleared — because the port genuinely stopped listening.

The event log over the same week adds the texture: zero OOM kills, eight disk I/O spikes (the photo server writing thumbnails, the CI runner pulling images), and the single critical — 9802 down. No memory events, no GPU events, no service crashes. The only other "crit" on the box is the known, explained nginx unit.

What the monitoring actually does

The stack has three layers of liveness, and they answer three different questions:

Is the process alive? Docker's healthchecks and systemd's unit state. This catches a crash — a process that died and didn't restart. It's the coarsest layer, and it's the one that's always on.
Is the service answering? The uptime probes. A process can be alive and not answering — hung, deadlocked, stuck in a retry loop. The probe is a 60-second HTTP GET that fails after 10 seconds. It catches the "alive but not working" state that the process check misses.
Is the service working correctly? The event log. A service can answer 200 and still be wrong — returning an error body, a 500 behind a proxy, a timeout that's just under the threshold. The edge-triggered events (OOM, threshold crossings, I/O spikes) catch the "answering but degrading" state.

The three layers are not redundant. Each one catches a failure the others miss. The process check catches the crash. The probe catches the hang. The event log catches the degradation. A 94-unit stack needs all three, because a 94-unit stack has 94 different ways to be subtly wrong.

The counterargument: 100% uptime is boring — so here is 100% and 0%

The fair objection: an article about a stack that didn't fail is an article with no story. 100% uptime is the baseline, not the achievement.

The point is the opposite. The achievement is that the stack didn't fail and that I would have known if it did. A 100% uptime number is only meaningful if the monitoring is good enough to have caught a failure. The 10,000 checks, the 2-consecutive-failure threshold, the three-layer liveness — that's the system that makes the 100% a fact instead of a hope. If 9800 had dropped for 30 minutes last Tuesday, the probe would have flagged it at the 2-minute mark, the event log would have recorded it, and I would have had a named, timestamped, attributed failure to look at — not a "why is the photo app slow" mystery.

And this week the system didn't have to be hypothetical. The 9802 endpoint stopped answering — not for a few minutes, for the whole window — and the monitor carried it as a critical from day one. Nobody "discovered" it. It's in the dashboard, marked, explained, and left, because the role behind it no longer exists. Same with the nginx unit: down for weeks, flagged the entire time, explained, and left. That's the system working: the failures are known, named, and triaged, not hidden.

What I would do now

If I were building a homelab today, the monitoring would have exactly this shape:

Every container: a healthcheck if the image defines one, a state check if it doesn't. The cost is nothing; the value is the "is the process alive" layer.
Every critical service: a 60-second uptime probe with a 10-second timeout and a 2-consecutive-failure threshold. The threshold is the key — 1 failure is a blip, 2 is a problem. One failure a day, over 94 units, would be 94 false pages; the threshold kills that.
The event log: edge-triggered alerts for the things that matter — OOM kills, threshold crossings, disk I/O spikes. Not a log of every metric, but a log of every event. The difference is the difference between a dashboard you scroll and a feed you read.
The one rule: a failure that is known, named, and explained is not a failure. It's a state. The nginx unit is down, and the 9802 probe is down, and that's fine, because both are flagged, both are explained, and neither is hiding. The system's job is not to prevent every failure — it's to make every failure known.

The 94-unit stack stays alive not because nothing fails, but because when something fails, the failure has a name, a timestamp, and a place in the dashboard. The 100% on 9800 is the output of that system, not the input — and the 0% on 9802 is the proof that the output is real.

Verdict

94 active systemd units, 59 containers, 10,000 uptime checks over 7 days: one endpoint at 100% (~14 ms mean), one endpoint at 0% (refused, flagged from day one), zero OOM kills, eight disk I/O spikes, two known-and-explained non-up states. The stack stays alive because the monitoring catches the three layers of failure — the crash, the hang, the degradation — and names each one. The 100% is not luck, and the 0% is not a miss. It's 10,000 checks that would have told me, in two minutes, if anything had broken.

The Local LLM Lab is a series of measured experiments on a home GPU stack. Every number in this article was pulled from 7 days of uptime and event telemetry on the box, not estimated. The series continues with the cost of keeping all of this alive — the full electricity bill for the whole machine, not just the GPU.

Patzer v7.19: the tactic trainer that shows your move and explains why the better one wins

Arsen Apostolov — Mon, 19 Oct 2026 08:06:43 +0000

Last week I wrote about Patzer — a self-hosted Chess.com alternative that runs in one Docker container (the original post). Since then, the feature I've been most proud of shipped: a tactic trainer that explains your own mistakes.

The gap in puzzle trainers

The classic puzzle trainer — Lichess puzzles, Chess.com tactics — gives you a position and asks for the best move. Miss it, and you see the solution. But you never see why yours was wrong: why the engine's move wins and your move loses. You memorize that Qh5 is the move, not that Qd3 leaves your queen hanging to a fork.

v7.19.0 closes that gap.

What's new

Your move, on the board. In Train → From your games, Patzer builds puzzles from your actual Chess.com or Lichess games. The move you played in that game is now drawn as a red arrow on the puzzle position, so you no longer have to work out from "you played Qd3" which move that was.

The coach explains the difference. Once the puzzle is over — solved, or the solution shown, never before — your AI coach explains in plain words why the engine's move beats the one you played. It's the same engine-checked explanation as in Game Review, in your language, pitched at your audience setting.

The design decision from the original post is what makes this cheap to run: the LLM never analyses. The server works out the facts — what each move allows, hangs or wins, the opponent's best answer — and the model only turns that into coaching. A small local model (Ollama or vLLM) is enough, and a hallucinated piece is impossible.

Puzzles come back. A puzzle you don't solve at the first try comes back the next day, marked Review — the same simple daily queue as the opening trainer. Solved at the first try on a later day, it's done; missed again, it's back tomorrow.

Puzzle of the day, with a streak. One puzzle from the Lichess set near your puzzle rating, the same one all day, with a plain counter of the days in a row you did it.

Play from here. In Game Review, every move now has a "Play from here against Patzer" button. You play your own colour, pick the bot's strength and the clock, and the game is saved like any bot game — your own games become training positions.

Full list in the release notes, including a fix for the saved AI model dropdown when the model isn't on the server:

Still one container:

docker run -d -p 8800:8800 -v patzer-data:/app/data --name patzer ghcr.io/SikamikanikoBG/patzer:latest

MIT licensed, English-first with Bulgarian, Spanish, German and Russian out of the box. If you're running Patzer, pull the latest tag and try the trainer on your own games — that's the part I'd want you to try first.

54 Containers, 93 Services, 64 GB of RAM: What Actually Eats It

Arsen Apostolov — Mon, 19 Oct 2026 07:11:05 +0000

The Local LLM Lab — Part 11

One controlled experiment. One number. One verdict.

A home server has a question the cloud never asks: who is eating the RAM?

In the cloud, memory is a line item on a bill. At home, it's a finite pool — 64 GB on my box — shared by every container, every service, every daemon, and the one GPU that runs the AI. When the pool runs dry, something gets killed, and the thing that gets killed is usually the thing you least expected.

I ran a memory audit on my homelab: 54 Docker containers, 93 running systemd units, one RTX 3090, 64 GB of RAM. I pulled the per-service memory history over 7 days, the current per-process state, and the kernel's non-reclaimable share. This is what the 64 GB is actually holding, and what the number says about how much RAM a homelab really needs.

The setup

Machine: HP Z840 Workstation, 2× Intel Xeon E5-2680 v4 (28 cores, 56 threads), 64 GB RAM, one RTX 3090
Workload: 54 Docker containers, 93 running systemd units, all up
Telemetry: 7 days of per-service memory at 2-hour resolution, plus the current per-process state and the kernel's non-reclaimable memory
GPU tenants (for context): Whisper ASR, a TTS model, an embedding model — see the VRAM piece for what the card holds

The 64 GB is the honest number. It's not a "big" machine by data-centre standards, but it's a real workstation, and it runs a real stack: photo storage, music, a RAG pipeline, an LLM observability stack, a CI runner, a media server, a few self-hosted apps, and the AI services on the GPU.

The number

The box is using 41.6 GB of its 64 GB — 65% — and the peak over the week was 15.9 GB of application memory, with the rest held by the kernel and the GPU's system-side buffers.

Layer	RAM (current)	Share of 64 GB
Kernel (non-reclaimable)	1,754 MB (~1.7 GB)	2.7%
GPU tenants (system-side buffers + CUDA context)	~2.0 GB	3.1%
Top application (Whisper ASR webservice)	7,870 MB (~7.7 GB)	12.2%
TTS voice model (omnivoice)	5,100 MB (~5.0 GB)	7.9%
LLM observability stack (web + worker + ClickHouse)	~2.3 GB	3.6%
Photo storage server (Immich: server + postgres + ML)	~1.0 GB	1.6%
RAG pipeline (Qdrant + Ollama + Chroma)	~0.7 GB	1.1%
The other 49 containers + 93 services	~22.7 GB	35.3%
Total in use	41,622 MB (~40.7 GB)	65%

The "other 49 containers + 93 services" line is the one that surprises people. It's 35% of the RAM, and it's not one thing — it's a long tail. The biggest individual entries in that tail:

Service	RAM (current)	What it is
ClickHouse (Langfuse storage)	1,082 MB	The tracing database
Open WebUI (chat UI)	777 MB	The chat interface
Langfuse web (LLM observability)	721 MB	The tracing UI
Immich server (photo storage)	562 MB	The photo library backend
Immich postgres (photo metadata)	536 MB	The photo database
Qdrant (vector DB)	483 MB	The RAG index
Langfuse worker (trace processing)	346 MB	The trace worker
Botpress (chatbot)	292 MB	The chatbot framework
n8n (workflow automation)	279 MB	The automation engine
Flowise (LLM builder)	255 MB	The LLM workflow builder
homelab-monitor (this monitor)	233 MB	The dashboard itself
jarvis2-core (my assistant)	229 MB	The assistant core
jarvis-server	213 MB	The server host
jarvis2-voiceline (voice pipeline)	199 MB	The voice line
Immich machine-learning (face/photo ML)	177 MB	The photo ML worker
Langfuse minio (trace object storage)	164 MB	The trace storage
searxng (meta-search)	111 MB	The search engine
UptimeKuma (health checks)	103 MB	The uptime monitor
MySQL (8.0, two instances)	34 MB + 164 MB	The databases
GitHub runner (CI)	58 MB	The CI agent
Postgres (three instances)	15 MB + 20 MB + 536 MB	The databases

The pattern: no single service is the hog. The RAM is eaten by a long tail of small, resident services, each holding 10–1,000 MB, and the sum is 22.7 GB. The GPU tenants hold the most per service (7.7 GB and 5.0 GB), but they're two services. The long tail is forty-nine, and it's still a third of the box.

What the peak says

The 7-day peak of 15.9 GB of application memory is the number that matters for sizing. It happened when the photo server was running a batch of thumbnail generation at the same time the CI runner was building a project and the observability stack was ingesting a burst of traces. Three things spiking at once, on a 64 GB box, and the peak was 15.9 GB of application memory — 25% of the pool.

The headroom at peak was 48 GB. That's the insurance. The box has never come close to the ceiling, and the events log over the week shows zero OOM kills, zero memory threshold crossings — the only alerts were eight disk I/O spikes, which are storage, not memory.

The honest reading: 64 GB is comfortable for this stack. 32 GB would be tight but workable. 16 GB would not. The peak of 15.9 GB is the worst case that actually happened, and it left 48 GB on the table. If I were buying RAM for this box today, I'd buy 64 GB and stop — not because it's the minimum, but because it's the size where the peak is a non-event.

The counterargument: RAM is cheap, don't sweat it

The fair objection: RAM is the cheapest part of a server. 64 GB is a few hundred euros. Why spend an article on it?

Because the RAM question is where homelab sizing goes wrong. People size the GPU (the expensive, scarce part) and under-size the RAM (the cheap, abundant part), and then the box OOMs the one service that matters — usually the database or the photo server — at the worst moment. The GPU is the part everyone benchmarks. The RAM is the part nobody measures, and it's the part that fails.

The measurement says the failure mode is not "one big service." It's the long tail. Fifty small services, each holding 10–100 MB, add up to a third of the box. The fix is not "buy more RAM" (though that helps). The fix is knowing the tail is real, so when the box gets tight you know which of the fifty to trim, not which of the two to kill.

What I would do now

If I were building this box today:

RAM: 64 GB. The measured peak is 15.9 GB of application memory; 64 GB is 4× peak. 32 GB would be 2× peak — workable, but the photo server and the CI runner would start fighting.
The rule: size RAM for the peak of the long tail, not for the biggest single service. The biggest service is 7.7 GB. The tail is 22.7 GB. The tail is the size.
The audit: a per-service memory panel, current and 7-day peak, with the kernel share separated out. The number that matters is not "the box is at 65%" — it's "the peak was 15.9 GB, and here are the ten services that were up at the peak."

The homelab monitor I use has exactly that panel. The 7-day peak per service, the current per-process state, and the kernel's non-reclaimable share, in one view. It's the difference between "the box is full" and "here's what's full, and here's what to trim."

Verdict

54 containers, 93 services, 64 GB of RAM: 41.6 GB in use (65%), 15.9 GB peak over the week, zero OOM kills. The RAM is not eaten by one big service — it's eaten by a long tail of fifty small, resident services that add up to 22.7 GB. The GPU tenants hold the most per service, but the tail holds a third of the box by itself. Size the RAM for the peak of the tail, not for the biggest single service, and the box stops being a guessing game.

The Local LLM Lab is a series of measured experiments on a home GPU stack. Every number in this article was pulled from 7 days of per-service memory telemetry on the box, not estimated. Next: the one failed service on a 93-service stack — and how the uptime checks keep the rest alive.

Your homelab, legible to AI agents — a read-only MCP server that rides along with your monitor

Arsen Apostolov — Sun, 18 Oct 2026 08:43:30 +0000

i keep a monitoring container on my homelab — it reads nvidia-smi, the docker api, systemd and /proc, and turns it into one page. the boring kind of dashboard: what's using the gpu, what it costs, what's filling the disk.

last month i started running agents on the box, and i hit a wall that i think most people will hit: the agent couldn't see my homelab. it could run commands, but it couldn't see — which model is holding the 3090, why the temperature is climbing, what the month cost in energy. i'd have to paste screen output into the chat by hand, and the agent was guessing about the rest.

the data was already there. the monitor was reading all of it, every few seconds, into a sqlite database. what was missing was just a standard way to hand it over.

so i added a read-only MCP server to the same container. it comes up on port 9810, and the whole setup is one line:

claude mcp add --transport http homelab http://YOUR-HUB:9810/mcp

nineteen named tools, all read-only: hosts, containers, services, per-card gpu state and which process is on which card, per-process ram, the model servers, installed models, energy cost, benchmark runs, disk treemaps, history, alerts. that's the same coverage the dashboard has — the server is just the same database with an MCP face on it.

what this actually gets you:

"which model is eating vram right now, and on which card?" — it reads the per-card split, not one pooled number. on my 3x3090 box a 63gb model shows as 22.5 / 22.1 / 18.8.
"what did the last week of inference cost me?" — it has the power integration, so it's a measured number, not a guess.
"is anything down?" — the uptime checks and alerts are tools too.
"what's filling the disk on the pi?" — the treemap, on demand, over the same ssh connection everything else uses.

a few things i'm deliberately not doing:

no writes. both directions are question and answer. the agent can look around the lab, it can't restart anything. (the dashboard itself has controls, but the mcp surface is read-only on purpose — an agent that can see your lab is already trusted, and i didn't want to make it responsible.)
no extra install on the remote boxes. the monitor already polls them over ssh with nothing persisted, and the mcp server just reads what the monitor has.
no cloud. the server is on your lan, same as the dashboard. point a local ollama agent at it and nothing leaves the house.

the monitor is open source, mit, one container: github.com/SikamikanikoBG/homelab-monitor — and the mcp part is on by default (enable_mcp=0 turns it off). if you run a homelab and you're pointing agents at it, i'd be curious what you'd ask it first.

My GPU Was Busy. I Finally Found Out Who Was Using It.

Arsen Apostolov — Mon, 12 Oct 2026 07:10:05 +0000

The Local LLM Lab — Part 10

One controlled experiment. One number. One verdict.

Every homelab has a moment like this: the GPU is at 100%, the fan is screaming, and you have no idea which of the twenty services on the box is doing it. You open nvidia-smi, it shows one process, you close it, and the mystery continues.

The problem is that nvidia-smi answers "which process is using the GPU" — and in a Docker stack, the process is the container runtime, not the service. The answer you actually want is "which service is using the GPU, and how much time did it take, and who called it."

I built the answer into my homelab monitor: caller→server attribution. Every sample of GPU utilisation is tagged with the service that owns the process, and for shared services (a transcription endpoint, an embedding endpoint) it's tagged with the caller — the thing that made the request. Over the last 7 days the attribution logged 1,055 samples. This is what it says about who eats a home GPU.

The setup

Card: NVIDIA RTX 3090, 24 GB, power limit 390 W
Shared services on the card:
- Whisper ASR webservice — a transcription endpoint with an nginx proxy in front (my tools, the observability stack, CI, and a warmup job all hit the proxy)
- nomic-embed-text (Ollama) — an embedding endpoint used by the RAG pipeline
Attribution: every GPU sample over 7 days is tagged with (a) the service owning the process and (b) the caller for shared endpoints. Total: 1,055 samples, ~1.1 million seconds of GPU time attributed.

The key design decision: the caller is recorded from the request, not from the process. When my assistant core sends an audio file to the whisper proxy, the sample is tagged jarvis2-core → whisper-proxy, not nginx → whisper. That's the difference between "the GPU is busy" and "my assistant is the one using it."

The number

Over 7 days, 47% of the GPU's logged time went to one caller: my own assistant core (4,970 seconds of 10,555 total attributed seconds).

Caller	Seconds (7d)	Server	% of total
jarvis2-core (my assistant)	4,970	whisper-proxy	47.1%
whisper-proxy (shared endpoint, direct)	920	whisper service	8.7%
langfuse worker (observability)	910	whisper service	8.6%
langfuse minio (object storage)	860	whisper service	8.1%
whisper-warmup (keep-alive job)	360	whisper service	3.4%
gh-runner (CI)	360	whisper-proxy	3.4%
jarvis-server + watchtower	410	whisper-proxy / service	3.9%
firefly importer	80	whisper service	0.8%
ollama-embed callers (CI, assistant, watchtower)	210	ollama-embed	2.0%
UptimeKuma (health checks)	20	whisper service	0.2%

The single biggest consumer of the GPU is not a model, not a benchmark, not a training run. It's my assistant transcribing the audio it records. Voice memos, meeting recordings, the audio attached to the traces the observability stack ingests — all of it flows through the Whisper endpoint, and all of it is attributed to the caller that sent it.

What the attribution changes

Without caller attribution, the 7-day story of this GPU is: "the Whisper service used 10,555 seconds of GPU time." That's a number. It doesn't tell you anything you can act on.

With caller attribution, the story is:

My assistant is the tenant. 47% of the GPU's time is my own tooling. The service exists for it, and it uses it more than everything else combined. That's the design working as intended — and it's also the reason the service is resident 24/7 instead of on-demand.
The observability stack is the second tenant. Langfuse's worker and MinIO together take 17% of the GPU's time. They transcribe the audio attached to the LLM traces they store. That's a cost nobody budgets for: the tracing stack is using the GPU to do its job, on top of the LLM it's tracing.
CI is a real, small tenant. The GitHub runner takes 3.4% — test fixtures that get transcribed during CI runs. It's small, but it's unexpected small. Nobody thinks about CI touching the GPU.
The warmup job is 3.4%. A keep-alive job that pings the endpoint every few minutes so the model stays warm. That's the "keep it warm" cost made visible — 3.4% of the GPU's week, doing nothing but staying ready.

The verdict from the numbers: the GPU is not "busy" in the way the alarm implies. It is busy for specific, named reasons, and the reasons add up to a stack that is working, not failing. The alarm that says "GPU at 100%" is a symptom. The attribution is the diagnosis.

The counterargument: attribution is overhead

The fair objection: tagging every GPU sample with a caller is extra bookkeeping, and in a hot loop it's overhead. The attribution has to be cheap enough that it doesn't slow the inference it's measuring.

The implementation I use is cheap by design: the tag is set at the request boundary, not at the sample boundary. When a request comes in, the caller is recorded once and attached to every sample the request produces. The per-sample cost is a field read, not a lookup. Over 1,055 samples the overhead is below the resolution of the power meter — you cannot measure it against a 2-hour bucket.

The other objection: attribution only works for services that go through a request boundary. A model that's loaded and used in-process (a training run, a direct script) has no caller to tag. For those, the service-level tag is the best you can do, and that's still better than nothing. The attribution is a spectrum: caller-level for request-driven services, service-level for the rest.

What I would do now

If I were building a homelab monitor today, the GPU panel would have three layers:

The card level: utilisation, VRAM, power, temperature. The thing nvidia-smi shows.
The service level: which container owns the process, and how much VRAM and time it holds. The thing Docker's stats show, but per-GPU.
The caller level: for shared endpoints, which client made the request that produced each sample. The thing nothing shows, and the thing that answers "who is using my GPU."

The third layer is the one that turns a dashboard into a tool. When the GPU spikes, you don't open nvidia-smi and guess. You look at the attribution and see the name. When a service misbehaves (a runaway job, a retry storm), the caller list shows it in the same view. The number that matters is not "the GPU is at 100%" — it's "jarvis2-core took 4,970 seconds this week, and here's the request that did it."

Verdict

One week of caller-level GPU attribution: 47% of the card's time went to my own assistant, 17% to the observability stack, 3.4% to CI, 3.4% to a warmup job that does nothing but stay ready. The GPU was never "mysteriously busy." It was busy for named reasons, and the attribution is the only view that shows them. If your homelab GPU spikes and you don't know why, the fix is not a faster card — it's a caller tag on every sample.

The Local LLM Lab is a series of measured experiments on a home GPU stack. Every number in this article was pulled from 7 days of caller-level attribution on the card, not estimated. Next: the 55 containers and 92 services that share this box with the GPU — what actually eats RAM when the machine has 64 GB of it.

I built my family a private Chess.com — one Docker container, Stockfish plus your own LLM

Arsen Apostolov — Mon, 12 Oct 2026 03:55:29 +0000

My kids play chess on Chess.com. It is a great product, but the account, the cloud and the ads are not something I want to hand to a nine-year-old. So I built Patzer: a self-hosted, open-source alternative that runs on a Pi, a NAS or an old laptop.

What it does:

Game Review — pull your public Chess.com or Lichess games, or paste a PGN. Bundled Stockfish classifies every move (Brilliant / Great / Inaccuracy / Mistake / Blunder), gives accuracy, estimated Elo and an eval graph, plus a "what's the threat?" probe that answers in plain words.
Play vs Bot — Stockfish at seven named tiers from Kid to Stockfish max, all standard time controls, with kid-mode blunder warnings.
Play vs Friend — real-time PvP between profiles on the same server over WebSocket: draw offers, takebacks, one-click rematch.
AI Coach — point it at your own Ollama or vLLM host. The coach explains why a move works or fails, shows the better move, and connects the mistake to your own recent games ("4 of your last 10 games: leaving a piece unprotected"). It never contradicts the engine: an answer that praises a blunder is caught before you see it.
Family-ready — multi-user with an admin console, per-profile language, kid mode, "horsey" piece names for the youngest profiles.

The design choice I keep coming back to: the LLM never analyses. The server works out the facts — what a move allows, hangs or wins, the opponent's best answer, your recurring mistakes — and the model only turns that into coaching. So a small local model is enough, and a hallucinated piece is impossible.

One container:

docker run -d -p 8800:8800 -v patzer-data:/app/data \
  --name patzer ghcr.io/SikamikanikoBG/patzer:latest

Everything runs on your LAN. Games, profiles and settings never leave home. MIT licensed, English-first with Bulgarian, Spanish, German and Russian out of the box.

Repo: https://github.com/SikamikanikoBG/patzer

I Put Three AI Services on One 24 GB GPU. It Used 17.8. Here's What Fits.

Arsen Apostolov — Mon, 12 Oct 2026 03:49:09 +0000

I Put Three AI Services on One 24 GB GPU. It Used 17.8. Here's What Fits.

The Local LLM Lab — Part 9

One controlled experiment. One number. One verdict.

The "how much VRAM do I need" advice on the internet comes in one flavour: a spec sheet. "27B in FP8 needs 28 GB. 70B in Q4 needs 42 GB. Buy a 48 GB card."

That's the answer for one model. Nobody tells you what happens when you try to run your actual stack — a transcription service, an embedding model, a small chat model — on one card. The answers you find are guesses, because nobody has measured a multi-tenant GPU.

I have. For the last week my RTX 3090 (24 GB) has been running three AI services at once: a Whisper ASR webservice, a small TTS voice model, and an embedding model for a RAG pipeline. I pulled the full VRAM history and the per-tenant attribution. This is what the card is actually holding, and what the number says about sizing a home GPU.

The setup

Card: NVIDIA RTX 3090, 24 GB (24,576 MB), power limit 390 W
Tenants (all resident, all Docker):
- Whisper ASR webservice (WhisperX, large-v3) — a transcription endpoint my own tools call for voice memos, meetings, podcast clips
- A small TTS voice model — text-to-speech for the same assistant stack
- nomic-embed-text (Ollama) — embeddings for the RAG pipeline
Telemetry: 7 days of VRAM / power / utilisation at 2-hour resolution, plus per-caller attribution
Everything else on the box: 55 containers, 92 systemd units, a 56-thread Xeon, 64 GB of RAM. The GPU sees only the three tenants above.

The three services were chosen because that is the actual shape of a home AI stack: one big-ish resident model (ASR), one medium resident model (TTS), one tiny resident model (embeddings). No chat model. The 27B chat model lives on a second machine with two 3090s — that's a different article.

The number

The card is holding 17.8 GB of its 24 GB — 72% — with three models resident, and the peak over the week was 23.4 GB. It never spilled a byte to system RAM.

Tenant	Avg VRAM (7d)	Peak VRAM (7d)
Whisper ASR (large-v3)	7,871 MB (~7.7 GB)	10,558 MB (~10.3 GB)
TTS voice model	4,876 MB (~4.7 GB)	4,964 MB (~4.8 GB)
nomic-embed-text	4,633 MB (~4.5 GB)	4,668 MB (~4.6 GB)
Total (measured, 7d)	~17,380 MB (~17.0 GB)	23,379 MB (~22.8 GB)

The measured current state at the time of writing: 17,775 MB used of 24,576 MB — 72% of the card, with the three tenants sitting at 7.87 GB + 4.96 GB + 4.67 GB = 17.5 GB, plus the CUDA context overhead of the card itself.

The peak of 23.4 GB is the interesting number. It happened when the Whisper service received a batch of long audio files at the same time the TTS model was rendering a long response. The two big tenants spiked together, the embedding model held its 4.6 GB, and the card came within 1.2 GB of the ceiling. It held. No spill, no OOM, no crash — because the peak was a few seconds, not a sustained load.

What each tenant is actually holding

The "paper VRAM" of a model is its weights. The measured VRAM is weights plus working set plus framework overhead. Here's the gap, per tenant:

Whisper large-v3. The weights in FP16 are ~3.1 GB. The measured average is 7.7 GB — 2.5× the weights. The difference is the encoder's audio feature buffers (long files and queued files hold more), the VAD and alignment models WhisperX runs alongside, and the CUDA context. The peak of 10.3 GB is a batch of long files in flight.

The TTS voice model. Weights ~4.5 GB, measured 4.7–4.8 GB. This one is close to its paper number because TTS is stateless between requests: no long context, no queue buffers. It's the most "honest" tenant on the card.

nomic-embed-text. A 137M-parameter embedding model. Paper weights ~0.3 GB. Measured 4.5–4.7 GB. That's the Ollama runtime, its CUDA context, and the KV space it pre-allocates for embedding batches. A model that "needs 300 MB" needs 4.6 GB when it's a resident service. That's the number people never quote.

The pattern: the measured VRAM of a resident service is 1.2–2.5× the weight size, and the multiplier is biggest for the models with the most state. If you're sizing a card, budget for the measured number, not the weights.

Who is calling it

The 7-day attribution shows why the Whisper tenant is the biggest consumer of time on the card, even though the TTS model is the bigger spike contributor:

Caller	Seconds on the card (7d)
My own assistant core (via the whisper proxy)	4,970
Whisper proxy (shared endpoint)	920
Langfuse worker (observability stack, transcribes attached audio)	910
Langfuse MinIO (object storage, transcribes on ingest)	860
Whisper warmup (keep-alive job)	360
GitHub runner (CI jobs that transcribe test fixtures)	360

The card is a shared service, not a single app. Six different callers hit the Whisper endpoint over the week. The TTS and embedding tenants have one caller each (the assistant stack). The attribution is what turns "the GPU is busy" into "here is who is busy" — and it's the difference between a dashboard you glance at and one you can act on.

The counterargument: 72% is a lot

The fair objection: a card at 72% of its VRAM with a 1.2 GB headroom at peak is a card that will fail the day a bigger file comes in. Headroom is insurance, and 12% is thin insurance.

True. But the measurement says what the insurance was worth over a week of real use: the peak was a 3-second event, and the card absorbed it. The sustained state is ~17 GB, which leaves 7.5 GB of headroom — 31% of the card. The 23.4 GB peak is the worst case that actually happened, not the worst case that is possible.

The honest version of the headroom argument: size for the biggest sustained working set you will actually run, plus 50% headroom. Not for the biggest spike you can imagine. If your stack's sustained working set is 17 GB, a 24 GB card is 1.4× sustained — enough. If you add a 7B chat model to the same card (sustained ~8 GB), the sustained working set becomes 25 GB and the 24 GB card is done. That's the actual failure mode: not a spike, but a new tenant.

What I would buy now

If I were building this stack today:

The hub card (ASR + TTS + embeddings): 24 GB. Sustained working set 17 GB, peak 23.4 GB. A 16 GB card is not enough — the TTS model alone would crowd the Whisper service. A 32 GB card would be comfortable but costs more and runs hotter for a workload that fits.
The chat card(s): sized for the chat model. The 27B in FP8 needs two 24 GB cards (or one 48 GB card). It does not share the hub card.
The rule: one card per class of workload — resident services on one, the big chat model on another. The measured working set of each class, plus 50% headroom, is the size.

The 24 GB 3090 is still the best price-per-GB on the used market, and that's why it's everywhere in home stacks. But "best price-per-GB" is an argument for the chat machine. For the hub, the argument is "fits the measured working set of my resident services with room for one more tenant" — and for my stack, that's 24 GB, not 16 and not 48.

Verdict

Three AI services on one 24 GB GPU: 17.8 GB sustained, 23.4 GB peak, zero spills over 7 days. The measured VRAM of a resident service is 1.2–2.5× its weights, and the peak is a few seconds, not a sustained load. Size the card for the sustained working set plus 50% headroom, and keep the big chat model on its own card. The spec sheet tells you what one model needs. The meter tells you what your stack needs.

The Local LLM Lab is a series of measured experiments on a home GPU stack. Every number in this article was pulled from 7 days of VRAM telemetry on the card, not estimated. Next: who is actually eating the GPU — the caller-level attribution that turns "the GPU is busy" into a list of names.

Qwen3.8-Flash-Next (125B) on Three RTX 3090s at 80 Tokens/s: Teaching llama.cpp Which Experts Matter

Arsen Apostolov — Sat, 26 Sep 2026 16:02:31 +0000

TL;DR: Qwen3.8-Flash-Next is a 125B mixture-of-experts model: on paper it beats the Qwen3.8-27B I run in production everywhere, by +16.5 points on agentic coding. It does not fit in the 72 GB of VRAM my three RTX 3090s have, and stock llama.cpp, spilling a quarter of the experts into system RAM, runs it at 23 tokens/second. Three changes get it to 80 tokens/second, entirely in VRAM:

Sort the experts by how often they're used. They are wildly unequal: the busiest 25% handle 52% of the work.
Store them at three precisions. Popular experts keep more bits and rare ones get squeezed harder, so the whole model fits on the GPUs.
Patch llama.cpp (~400 lines) so one routing decision drives three expert tables at once, then add speculative decoding on top.

Quality stays close to the 8-bit reference: +1.9% perplexity, 91% same next token, GSM8K 95.5% (the 27B scores 95–96.5% on the same harness). There's also a 256k-context profile that finds a random code hidden 173,000 tokens deep. The trade-offs are real, and I've listed them. Everything is open: patches, tools, raw measurements.

Update, 26 Sep 2026, after the first day behind my assistant: the 80 tok/s holds for short prompts. In real agent use, with ~25k-token prompts, sampling and thinking on, the model decodes at 30–45 tok/s. My first deployment also had a caching mistake that cost ~45 s before every answer; a server config change fixed it (now 0.7 s). Details in the "real-world update" section below.

the use case

My box, vader, runs Qwen3.8-27B around the clock: two RTX 3090s, vLLM, about 135 tokens/second. It's the first local model I actually trust with agent work, and it does most of the work my assistant does.

Then Qwen shipped Qwen3.8-Flash-Next, the preview of their Qwen4 architecture. Their own model card puts it ahead of my 27B on almost everything I care about:

benchmark (Qwen's card)	Flash-Next	3.8-27B	gap
DeepSWE (agentic coding)	58.7	42.2	+16.5
JobBench (professional tasks)	55.7	33.4	+22.3
SWE-bench Multilingual	81.0	73.8	+7.2
Toolathlon (tool use)	73.5	67.1	+6.4
HLE	35.9	30.8	+5.1
GPQA Diamond	91.7	89.2	+2.5
IFBench	81.3	79.5	+1.8

So the goal was simple to state: the smarter model, on the same three cards, with decode and time-to-first-token that still feel interactive. No new hardware. I told my AI engineer (Claude Code, more on that at the end) that "no" was not an acceptable answer.

the box

GPUs: 3x RTX 3090, 24 GB each: 72 GB of VRAM. No NVLink, and no PCIe peer-to-peer (the board has no Resizable BAR), so the cards can't talk to each other directly.
CPU: dual Xeon E5-2660 v4, 56 threads.
RAM: 60 GB. Only one 32 GB stick per CPU socket, so it's single-channel, and that matters a lot below.
Disk: one 1 TB NVMe.

a small dictionary before we start

If you know what a KV cache is, skip this. If you don't, it's all you need for the rest of the article.

Parameters / weights. The numbers a model learned. "125B" means 125 billion of them. Each one normally takes 2 bytes, so 125B parameters is ~250 GB, far more than 72 GB of VRAM.
VRAM. The GPU's own memory. It's about 20–50x faster than system RAM on this box. A model runs fast only if what it needs for each word lives in VRAM.
Quantization. Storing each weight with fewer bits (8, 4, 3…) instead of 16. Like saving a photo as a smaller JPEG: smaller file, a little less detail. "Q8_0", "Q4_K", "IQ3_S", "MXFP4" are llama.cpp's names for different bit-budgets and recipes.
Mixture of experts (MoE). Instead of one giant network, each layer has many small "expert" networks, 512 of them here, and a router picks the best 10 for each word. So the model knows 125B parameters' worth of things but only computes with about 6B per word. That's why it can be fast, and also why it's huge.
Token. A word or word-piece. Decode speed (tokens/second) is how fast the answer streams out.
Prefill / time to first token (TTFT). Before answering, the model reads your whole prompt. That's prefill, and TTFT is how long you wait before the first word appears.
Context / KV cache. How much text the model can hold in mind at once ("256k context" is about 500 pages), and the memory it uses to remember it.
n-gram embedding. New in this model: a 51 GB lookup table indexed by 2–3 word phrases. It's only ever looked up, a few rows per word, so it can live on the NVMe drive.
Speculative decoding / MTP. A small built-in "draft head" guesses the next few words, and the big model checks all the guesses in one pass. Correct guesses are free words.
KL divergence (KLD). How far the compressed model's word probabilities drift from the original's. 0 means identical; lower is better. It's the most sensitive quality meter there is.
llama.cpp / GGUF. The inference engine I patched, and its model-file format.

the problem, in one picture

In 16-bit the model is 360 GB. Even the popular 4-bit build (unsloth's UD-Q4_K_XL) is 111 GB: 72 GB of experts, 27 GB of n-gram table (which can stay on disk) and a few GB of everything else. The experts alone don't fit in 72 GB of VRAM once the rest of the model and the context need room too. So llama.cpp does the sensible thing: it puts the overflow experts in system RAM and computes them on the CPU.

On this machine that means 23 tokens/second. With one memory stick per socket, the CPU reads those experts about ten times slower than a GPU would. And for every word, the work bounces between the GPUs and the CPU dozens of times, so each GPU waits.

step 1: the obvious setup, and the first trap

The first run with stock settings also showed prefill all over the place (47 to 306 tokens/second on the same 4k prompt) and decode sagging to 10 tok/s. Thread states told the story: the main thread sat in D (waiting on disk). llama.cpp memory-maps the model file, loading the GPU part streams 70+ GB through the page cache, and the CPU-side experts kept getting evicted and re-read from NVMe.

--load-mode none loads the CPU-side weights into RAM and keeps them there. Also, don't download a 188 GB file while you benchmark: my download filled the page cache and pushed the server into swap. With both fixed, prefill doubled (658–792 tok/s) and decode steadied at 23–26 tok/s.

Speculative decoding (MTP) barely helped: 24–29 tok/s. The reason matters for everything that follows. A draft of 5 words means the big model checks 6 words at once. Each word picks 10 experts, so one check can touch up to 60 different experts per layer instead of 10, and the slow ones in CPU RAM get hit harder. As long as experts live in CPU RAM, nothing else will make this fast.

step 2: the experts are not equal

Here's the observation the whole project rests on. llama.cpp's calibration data (the importance matrix unsloth publishes with their quants) records how often each of the 24,576 experts was picked on real text:

In the average layer, the busiest 25% of experts handle 52% of the tokens, and the busiest 80% handle 95%. Stock llama.cpp can only place whole layers of experts: a layer's 512 experts are one tensor, all on the GPU or all on the CPU. It offloads a quarter of the experts, including popular ones, so a quarter of the expert work lands on the slow path.

step 3: split every layer into hot and cold

The first patch splits every layer's expert tensor in two at load time:

hot experts go to the GPUs;
cold experts go to CPU RAM;
the router is permuted so its choices land in the right half.

With the same 19% of expert bytes on the CPU, the cold experts now serve only 4.5% of routed tokens instead of ~19%, about 4x less slow-path traffic.

Getting it right took four bugs, each instructive enough to list in its own section below. The result is 30–32 tok/s with correct answers. That's +30%, but it's not the leap it should be, so I profiled it:

where one decoded word spends its time
kernel launches	~4,300
dense weights (attention, DeltaNet, shared expert), 8-bit	~4.9 ms
routed experts	~2.5 ms
~3,500 tiny element-wise kernels	~6–7 ms
GPU↔CPU synchronisations	~390

The CPU was barely reading any experts anymore; it was the round trips that cost. Every layer handed work to the CPU and waited for the answer. Then a cheap experiment: drop the cold experts entirely (quality garbage, speed only) and see how fast it goes fully on GPU: 52 tok/s, and 86–88 tok/s with speculative decoding. That was the target. The question became: how do you fit every expert in VRAM without wrecking quality?

step 4: three shelves, all on the GPU

The idea: if some experts do most of the work, give them more bits and squeeze the rarely-used ones harder. Everything still fits in VRAM, and most tokens still see a well-preserved expert. Think of a library that keeps its bestsellers in hardcover and prints the rarely-borrowed titles as compact paperbacks, so the whole collection fits in the building.

Two pieces make it work.

An offline re-packer (tools/expert_tiers.py). It starts from the 8-bit model (188 GB) and uses the importance matrix for both how often each expert is used and which inputs matter inside it. A greedy planner fills a VRAM budget: each byte goes to the expert where it removes the most expected error. Every expert is then re-quantized with llama.cpp's own quantizers, and the file is written with the experts reordered hottest-first as three tensors per layer. One wrinkle: the down-projection matrices have rows 640 wide, which the fancy 2–3-bit formats can't handle (they need multiples of 256). So they get their own ladder: IQ4_NL / MXFP4 instead of IQ4_XS / IQ3.

A patched llama.cpp (patches/). mul_mat_id, the operation that runs "each word through its chosen experts", learns to take an id range. Each shelf's matrix multiply sees the router's full choice list, computes only the ids that fall on its shelf, and writes zeros for the rest. The three results simply add up. That meant touching the CPU kernels, three CUDA paths (decode, prefill, and the fused gate+up+activation kernel) and the loader. On top sits Qwen's multi-token-prediction draft head, from a not-yet-merged llama.cpp PR, re-quantized to 4 bits so it fits too.

Result, in one chart:

the four bugs, briefly

For people who'll try this, and because two of them are delightful.

Duplicate experts crash the CUDA MoE kernel. My first version pointed "not on this shelf" at expert 0. A word with two cold experts then had expert 0 twice, and CUDA's expert-grouping helper assumes each expert appears at most once per word. It silently dropped a slot, and a later kernel read an unwritten index. Fixed by teaching the kernels a real "skip".
A flag stored where a precision setting lives. I marked the skip-capable nodes in op_params[0], which llama.cpp already uses for the accumulator precision. The flag was quietly overwritten. It moved to slot 7, with a magic value.
The unsigned ternary. ids ? ids[i] : blockIdx.x, where blockIdx.x is unsigned. C++ promotes the whole expression to unsigned, so my -1 ("skip") became 4,294,967,295 and the kernel read 4 billion rows past the buffer. compute-sanitizer found it in one run.
Fused kernels hide the tag. For decode, CUDA fuses gate, up and activation into one kernel whose "destination" is the activation node, not the matrix multiply that carries my id-range tag. The first id-range build printed //////////. The kernel now looks up the tag on its source node.

how much quality did it cost?

The honest meter is KL divergence against the 8-bit model over 24,576 tokens of Wikipedia text:

model	expert size	fits in VRAM	perplexity vs 8-bit	KLD	same top token
unsloth UD-Q4_K_XL (stock)	71.7 GiB	no, 25% in RAM	+0.9%	0.045	93.7%
tiered, 128k profile	53.0 GiB	yes	+1.9%	0.091	91.1%
uniform IQ3_S / MXFP4, same size	52.2 GiB	yes	+3.0%	0.095	91.0%
tiered, 256k profile	49.5 GiB	yes	+3.2%	0.113	90.1%

What I take from it:

Fitting in VRAM costs quality, and there's no way around that. 20 GB less expert weight roughly doubles the KL divergence of the stock 4-bit build. That's physics, not a bug.
At equal size, tiering beats uniform: perplexity drift 1.9% vs 3.0%. But the gain is smaller than my planner predicted: its error model is crude, and at this size the total byte count dominates. There's more to win with a better tier recipe.
On a task, it holds up. GSM8K (200 grade-school maths questions, greedy, no thinking) scores 95.5%. The 27B's published number on the same harness is 95.0–96.5%. GSM8K is saturated, so it mainly proves nothing broke. The model card's gains on agentic work are where the upgrade should show.

speed vs context, and the 256k profile

Both profiles start at ~80 tokens/second on a short prompt and slow down as the prompt grows. The per-word attention and indexer work grows with context, and the draft head's guesses get a little worse too. Typical numbers, averaging two runs:

prompt	128k profile: decode / time to first token	256k profile: decode / time to first token
4k tokens	82 tok/s / 7–9 s	67 tok/s / 7–8 s
15k tokens	66 tok/s / 18–22 s	58 tok/s / 19–22 s
30k tokens	58 tok/s / 35 s	54 tok/s / 38 s
61k tokens	53 tok/s / 76–79 s	51 tok/s / 82 s
95k / 122k tokens	39 tok/s / 140 s	35 tok/s / 190 s

Prefill runs at 540–860 tokens/second. That's the weak spot next to the 27B's ~1,300, and it's where I'd look next: it's also the part that makes a 120k-token prompt a three-minute wait.

The 256k profile needed two more compromises. The KV cache drops to 8-bit, and experts shrink to 49.5 GiB, because the sparse-attention indexer's scratch memory grows with context and pushed a card out of memory at ~200k on the first try. After rebalancing layers across the cards, it read 173,692 tokens of Wikipedia with a random 10-character code hidden at 45% depth, and returned the code: exactly right (6BC5XYE8FS). A second run with the code at 90% depth of 112,711 tokens was also exact. It took 6.1 minutes (472 tokens/second) to read that much. Long-context prefill on this architecture is still the slow part.

what it costs to run

Measured at the cards (nvidia-smi, 5 Hz, three 1,024-token generations per profile): the three 3090s draw ~540 W together while decoding and ~124 W at rest with the model loaded. That's 7.5–8.4 joules per token, 2.1–2.3 kWh per million tokens. On my tariff (0.30 BGN/kWh day, 0.18 night) a million generated tokens cost 0.63–0.70 BGN in daytime, 0.38–0.42 BGN at night, about $0.35–0.40. The 27B costs 0.21–0.34 BGN for the same million, so the bigger brain costs roughly twice as much per word, all three cards included. It's still coffee money.

real-world update: day one behind my assistant

Benchmarks are one request with a short prompt and greedy decoding. My assistant is none of those things, so after a day of it running Jarvis, here is what the server logs say about real requests:

	a real assistant turn	the benchmark
prompt	~25,000 tokens (system prompt, tools, memory)	41 tokens
sampling	temperature 0.7, thinking on	greedy
tokens per verify step (draft acceptance)	2.6 (39%)	3.4 (62%)
decode	41–45 tok/s (30–45 is what it feels like)	80 tok/s

Decode: 30–45 tok/s, not 80. Two things compound. The prompt is deep, and every decoded word pays attention over it; the context chart above already shows ~50 tok/s at 60k. And the draft head guesses sampled text, tool-call JSON and reasoning worse than greedy prose, so speculative decoding saves less. That's the honest number for agent work on this box. For comparison, the 27B benchmarks at ~135 tok/s on this box, so the upgrade is intelligence, not speed.

Time to first token: the mistake was mine. My assistant runs several roles: chat, planner, classifier, triage, background jobs. Each has its own ~25k-token prompt prefix. I deployed the server with one slot (-np 1), so every time the role changed, the cached prefix was thrown away and 25,000 tokens were re-read from scratch: ~45 seconds before every answer. The fix is four slots sharing one KV pool, so every role keeps its prefix warm:

returning to a 14k-token prompt after another role ran	before (`-np 1`)	after (`-np 4 -kvu`)
time to first token	20–45 s	0.6–0.7 s

It isn't free: four slots need a little more memory, so the shared pool dropped from 128k to 96k tokens (128k with four slots ran out of VRAM). A single request can still use all 96k.

Schedule the background jobs apart. A model at 30–45 tok/s with 25k-token prompts spends minutes per agent task. My assistant had 13 scheduled jobs between 06:30 and 09:00 on Mondays, four of them at the same minute. I've spread them 30 minutes apart (05:00 to 11:30) so they stop queueing behind each other. I'll report after a week whether that's enough.

Images and video work too, once the vision projector is loaded (--mmproj, on the GPU with the most free memory: 4 s per photo, against 61 s on the CPU) and the container has ffmpeg for video (a 5-second clip in 6.4 s). The repo has the exact launcher.

what I'd not claim

It's single-user. Every speed number here is one request at a time, and real agent turns decode at 30–45 tok/s, not 80. The 27B on vLLM batches many users far better.
The 27B is still faster. About 135 vs 80 tok/s decode, and roughly 1,300 vs 540–860 tok/s prefill. The 125B is smarter, not quicker.
It uses all three cards. You can't also keep the 27B running.
The quality numbers are wikitext + GSM8K. They say the compression is gentle. They don't prove the agentic gains survive intact; that needs agent benchmarks I haven't run yet.
The tier planner is a heuristic. The measurements say it helps; a calibrated one would help more.

reproduce it

Everything is in SikamikanikoBG/qwen38-flash-next-3x3090: both llama.cpp patches, the re-packer, the benchmark scripts, and every raw number behind the charts. The short version:

# llama.cpp at 81bc6b8 + MTP PR #28243 + the tiering patch
git apply patches/0001-qwen4exp-mtp-pr28243.patch patches/0002-tiered-experts-hot-cold-split.patch
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 && cmake --build build -j

# re-pack the 8-bit model into three precision shelves (53 GiB of experts)
python tools/expert_tiers.py write --src Qwen3.8-Flash-Next-Q8_0-00001-of-00006.gguf \
  --imatrix imatrix_unsloth.gguf --budget-gib 53 \
  --gu Q6_K,IQ4_XS,IQ3_XXS --dn Q8_0,IQ4_NL,MXFP4 --out fn-tier53.gguf

# serve: all layers on GPU, 4 slots sharing a 96k KV pool, vision, MTP drafting 4 tokens
llama-server -m fn-tier53-00001-of-00002.gguf -ngl 99 -ts 16,16,16 -c 98304 -np 4 -kvu \
  -b 1024 -ub 256 -fa on --jinja \
  --mmproj mmproj-F16.gguf -mmdev CUDA1 --image-max-tokens 1024 \
  -md mtp-shared-iq4.gguf --spec-type draft-mtp --spec-draft-n-max 4

credits

Qwen, for the model and an architecture that rewards this kind of work.
unsloth, for the GGUFs, the MTP draft files and the importance matrix.
llama.cpp and ggml, plus the authors of the Qwen4Exp MTP PR and the sparse-attention decay issue.
syv-ai/qwen38-27b-rtx3090, whose rigor on the 27B set the bar for measuring this.

This project, from the patches and the tools to the measurements and this write-up, was done with **Claude Code* acting as my AI engineer, working on my hardware under my direction. The numbers are from vader, the raw data is in the repo, and the mistakes are both of ours.*

Qwen3.8-27B on One RTX 3090 vs Two: +20% Decode, +14% Cold Prefill, and 3x on Cached Prompts

Arsen Apostolov — Wed, 23 Sep 2026 03:27:37 +0000

TL;DR: Qwen3.8-27B — the dense 27.8B that dropped on August 14 — fits on a single RTX 3090 at W4A16 and decodes at ~125–155 tokens/sec. Splitting it across two cards with tensor parallelism buys roughly 20% more decode (noisy: +13% in one run, +22% in the next) and 14% faster prefill on average on a cold prompt (23% at the top of the ladder: an 8,600-token prompt takes 8.5 s to first token on one card and 6.9 s on two). On a prompt the server has already seen, the pair answers 2–3x faster — but that is the prefix cache, not the cards, and the script below measures the two separately. At the API list price of $3.00 per million output tokens, one million tokens cost 0.21–0.34 BGN of electricity, about $0.12–0.20, on the box that serves them.

the box

vader is the machine I run this on: three RTX 3090s in an ASUS Z10PE-D8 WS, dual Xeon E5-2660 v4 (56 threads), 60 GB of RAM, Ubuntu 26.04. It has been my inference box since the Whisper experiments, and since July it has also carried the power-limiting work (the meters post is where those numbers live). Two of the three cards are in this test; the third sits idle.

The model under test is the one that is currently loaded on it:

	single-card	TP2
container	`qwen38-27b-single-1`	`qwen38-27b-tp2-single-1`
image	`ghcr.io/syv-ai/qwen38-27b-rtx3090`	same, `:latest`
weights	Qwen3.8-27B-W4A16-AutoRound-fast	same
cards	1x 3090 (22.6 GB VRAM)	2x 3090 (21.7 GB each)
max context	131,072	262,144
port	18021	18022

Both are vLLM, both have been up two days, both report healthy, and both run with prefix caching on — which is why every number below comes in two flavours, cold and warm. The W4A16 quantization is what puts a 27.8B dense model inside one 24 GB card — 22.6 GB of it, with the KV cache budget for the rest of the 128k window coming out of what is left.

the method

No benchmark harness, no vllm bench — a 150-line script (bench_qwen38_v2.py) that sends a real chat-completion request with a prompt of a given size, streams the answer, and times it. For each prompt size it runs three times and takes the median:

prompt sizes: calibrated so the server reports ~600, ~2,400, ~4,700, ~9,300 prompt tokens (the script probes the tokenizer first, then scales the prompt to hit the target)
output: max_tokens=256, temperature=0
measured: time to first SSE event, total time, and the server-reported token counts from stream_options.include_usage — not my own chunk counting
cold vs warm: every cold request carries a unique first line, so no cached prefix can match it; the warm pass repeats one identical request after a warm-up call

That last bullet is the one that decides whether a benchmark means anything. With prefix caching on — and it is on by default in vLLM — timing the same prompt twice does not measure prefill the second time, it measures a cache lookup. The tell is arithmetic: 8,600 tokens "prefilled" in 0.56 s is 15,000 tokens/sec, roughly ten times what two 3090s can actually push through a 27B model. If your TTFT numbers look like that, you are timing the cache. So every cold request here is salted with a unique first line, and the warm pass is run deliberately and reported in its own column.

One caveat: this is a single-request benchmark. vLLM's continuous batching is where production throughput lives, and I did not hammer it with concurrent requests. What I measured is the latency you feel as the one person using the box — which, for a homelab, is most of the time.

decode: the second card buys 15–30%, noisily

The single card holds 123–154 tokens/sec across the ladder; the TP2 pair runs 146–193. A 27B dense model at W4A16 on one 3090 is memory-bandwidth-bound in a way that does not care much about prompt size, because the KV cache for these sizes is small next to the weights. The pair is faster at every size but one (at 4,700 tokens the two came out level), and the average gain was +22% in this run against +13% in a run an hour earlier. Single-stream decode on these cards moves by about 10% between runs an hour apart, so I am not going to print a decimal: call it 15–30%, and expect the lower end.

That is a real number and a small one. For a 27B model, tensor parallelism across two cards is not the 2x you get from adding compute — the all-reduce between the cards eats most of it.

prefill: cold is the cards, warm is the cache

Two columns, two different questions.

Cold — a prompt the server has never seen — is prefill, and prefill is compute-bound: the single card does 1,100–1,220 tokens/sec of it, the pair 1,230–1,360. Time to first token climbs from 0.53 s to 8.5 s on one card as the prompt goes from 600 to 9,300 tokens, and from 0.50 s to 6.9 s on two. The gap grows with the prompt: 6% at 600 tokens, 23% at 9,300, 14% on average. That is what the second card buys you on a fresh long prompt.

Warm — the same prompt again — is the prefix cache doing its job, and here the pair really is 2–3x faster: 0.5–0.75 s flat across the ladder against 0.5–1.7 s on the single card, whose smaller KV pool and slower re-read of the uncached tail show at every size above 600 tokens. It is a cache number rather than a prefill number, and it is also the one you feel most often, because every follow-up turn in a chat is a warm request.

The routing rule that falls out: a long fresh prompt goes to the pair and saves about a fifth of its wait; a conversation stays on whichever lane it started on, so its prefix stays cached. For prompts under about 2,000 tokens the single card is within noise of the pair on both counts and draws one card's power.

what it costs

The box draws 620 W at the wall with both containers loaded and serving — 247 W on one TP2 card, 328 W on the other, 45 W on the single-card 3090 sitting at P8 with its weights resident. Those are the monitor's readings from the day the containers went up, not a measurement taken during these runs; the governor from the meters post is power-capping it, holding 79°C.

At the API's list price — $3.00 per million output tokens, DeepInfra's rate for Qwen3.8-27B as listed on llm-stats — one million generated tokens takes about 6,667 seconds at 150 tokens/sec. At 620 W that is 1.15 kWh. On the Bulgarian dual-rate tariff that is 0.34 BGN by day (0.30 BGN/kWh) or 0.21 BGN by night (0.18 BGN/kWh) — roughly $0.20 or $0.12. The API price is 15x to 25x the electricity, depending on when the box runs.

I am not going to pretend that makes the API a bad deal. The API does not cost you 620 W of heat in your room, it does not need a 60 GB machine, it does not need you to keep a W4A16 quant on a 24 GB card, and its $3.00 buys you their datacenter's power, cooling and idle capacity, not yours. But for a box that is already on and already paying its standby tax, every token it serves is almost pure margin. The electricity argument for local inference is not "it is cheaper than the API" in any total-cost sense — it is "the marginal cost of a token on hardware you already run is an order of magnitude below the list price."

the verdict

Qwen3.8-27B is the first 27B-class dense model I have run that I would put in front of real work on this box, and the numbers say why:

It fits. W4A16 on one 3090, 22.6 GB, with room for a 128k context window. The TP2 variant doubles the context to 262k by splitting the weights.
It is fast enough to feel like an API. 125–190 tokens/sec decode is faster than most hosted endpoints feel in a chat UI. The part you notice is a cold long prompt: several seconds on either lane.
The two-card setup is not 2x. It is roughly +20% decode and +14% cold prefill, and 2–3x on warm follow-ups because the pair keeps a bigger cache. Buy the second card for the context window and the cache, not for raw speed.
The marginal cost is 0.21–0.34 BGN, about $0.12–0.20, per million tokens against a $3.00 list price, on hardware that is already drawing its standby power.
Benchmark with the cache in mind. If a prefill number looks like 15,000 tokens/sec on consumer cards, it is the cache. Salt every request, or turn prefix caching off, and report which one you did.

The script, the raw JSON and the charts are in the repo: qwen38-bench. If you run a different quant or a different card, I want your numbers — the interesting part of this post is the shape of the curve, and the shape changes with the hardware.

vader: 3x RTX 3090 (two used here), dual Xeon E5-2660 v4, 60 GB RAM, Ubuntu 26.04, vLLM, W4A16-AutoRound-fast quantization.