The electricity cost of local inference is three numbers multiplied together, and two of them you have to measure yourself. This page gives you the formula, works it through with every assumption labelled, and tells you which figure to substitute. No number below is a measurement of any particular machine.
The three inputs, and where each comes from
- Power draw under generation, in watts. Not the board’s rated power — NVIDIA publishes 450 W for the RTX 4090, for instance, and that is a design ceiling rather than what generation draws. Measure yours with
nvidia-smi --query-gpu=power.draw --format=csv -l 1during a long generation and take the median. Add the rest of the system: CPU, RAM, drives, fans and PSU losses, which for a desktop is commonly another 60–120 W and is best taken from a wall meter if you have one. - Your electricity rate, per kWh. This is on your bill, it is the only figure you can look up exactly, and it varies by more than a factor of three across countries. The worked examples below assume 0.18 per kWh; the US Energy Information Administration put the 2026 US residential average near that in its Short-Term Energy Outlook, and the EIA publishes updated averages monthly.
- Your generation rate, in tokens per second. This is the one nobody can give you. It depends on the model, the quantization, the card, the context length and the concurrency, and it is not published for any specific combination. Measure it and substitute your own.
The 0.18 per kWh figure is an assumption stated for the arithmetic, and electricity tariffs move constantly — in some markets, hourly. Substitute the rate on your own most recent bill before you rely on any conclusion below.
Cost per hour of generation
cost_per_hour = (P_watts / 1000) x rate
Assuming P = 300 W GPU + 100 W rest-of-system = 400 W,
and rate = 0.18 per kWh:
0.400 kW x 0.18 = 0.072 per hour
Over a working day of continuous generation (8 h): 0.576
Over a month of continuous generation (730 h): 5.26
That is the whole hourly picture, and the striking thing about it is how small it is. Continuous generation on a desktop-class card costs a few cents an hour. The intuition that a GPU is expensive to run comes from the purchase price, not the meter.
Two adjustments matter, and the first one usually dominates. Almost nobody generates continuously. A machine that is powered on all day and generating for two hours of it is paying an idle draw for the other twenty-two, so the honest daily figure is a duty-cycle weighting rather than the generation cost alone:
cost_per_day = ((P_gen / 1000) x h_gen + (P_idle / 1000) x (24 - h_gen)) x rate
With P_gen = 400 W, P_idle = 90 W (whole machine, model resident),
h_gen = 2 hours, rate = 0.18:
generating: 0.400 x 2 = 0.800 kWh
idling: 0.090 x 22 = 1.980 kWh
total 2.78 kWh x 0.18 = 0.50 per day
Note which term is larger. At a 2-hour duty cycle the idle time costs
more than the work does, and no amount of tuning the model changes that.
That inversion is the single most useful thing on this page. Below roughly a 20% duty cycle the machine’s standing draw is the dominant cost, and the lever that matters is whether the machine is on at all — the idle case has its own arithmetic and its own set of controls. Above roughly 50%, generation dominates and the levers below start to matter.
The second adjustment is cooling. If the room is air conditioned, every watt of GPU is a watt of heat that has to be removed; at a coefficient of performance of 3 on the air conditioning, that adds a third to every figure above. In a cold climate during heating season the sign flips and the GPU is displacing heating you were paying for anyway, which is a real effect and a seasonal one.
Cost per million tokens
Per-token is the form that compares to anything else, and it comes from the same three inputs plus a division:
seconds_per_million = 1e6 / tokens_per_second
kWh_per_million = (P_watts / 1000) x (seconds_per_million / 3600)
cost_per_million = kWh_per_million x rate
Worked, with P = 400 W, rate = 0.18, and R = 40 tok/s
(R is a placeholder — substitute your own measured rate):
seconds = 1e6 / 40 = 25,000 s = 6.944 hours
kWh = 0.400 x 6.944 = 2.778 kWh
cost = 2.778 x 0.18 = 0.50 per million output tokens
Same machine, same tariff, at R = 15 tok/s:
1e6 / 15 = 66,667 s = 18.52 h; 0.400 x 18.52 = 7.41 kWh; = 1.33
The sensitivity is worth reading off. Cost per token is inversely proportional to tokens per second and directly proportional to watts, so a configuration that is 2.5x slower costs 2.5x more per token at the same power. That is why quantization decisions have an electricity consequence that is easy to miss: a quant that fits entirely in VRAM and one that spills a few layers to CPU differ in generation rate by far more than they differ in watts, so the spilled configuration can cost several times as much per token while drawing roughly the same power. Fitting the model is a cost optimisation as well as a speed one.
Why concurrency changes the answer
Single-request generation does not saturate a GPU. Each token requires reading the entire set of active weights from memory to produce one token’s worth of arithmetic, so the card is bandwidth-bound and its arithmetic units are largely idle — while still drawing power for the memory traffic, the clocks and the fans.
Serving several requests at once reuses each weight read across all of them. The power draw rises, but nothing like proportionally to the throughput, so cost per token falls substantially. Run the same formula with a concurrency-8 aggregate rate:
Same 400 W machine, but aggregate R = 200 tok/s across 8 concurrent slots
(again a placeholder — measure yours):
1e6 / 200 = 5,000 s = 1.389 h
0.400 kW x 1.389 h = 0.556 kWh
0.556 x 0.18 = 0.10 per million tokens
The power went up somewhat and the cost per token fell about fivefold.
The mechanism is why hosted inference is cheap per token in a way a single desktop cannot be: a provider runs at high batch sizes continuously. If your local workload is genuinely serial — one person typing at one chat window — you are on the expensive end of that curve by construction, and no amount of tuning moves you off it. Configuring slots is how you exploit concurrency locally when you have it.
What this number leaves out
Electricity is the marginal cost and it is the smallest term. The purchase price of the card amortised over its life is normally larger per token than the power it uses, and so is the time you spend maintaining the thing. A card bought for 1,500 and used for three years is 1.37 per day of ownership before a single token is generated — against 0.072 for an hour of generation.
It also excludes the failure modes: the full cost of local inference covers the ones that are not on any meter. And it excludes carbon, which does not track cost at all — grid intensity varies by region and by hour of day, so the same kWh can differ by an order of magnitude in emissions. Inference carbon figures are a separate calculation from this one.
The comparison this arithmetic invites — local watts against a hosted per-token price — usually resolves as “both” rather than either: the local model for high-volume, low-stakes or private work and a hosted one for what it cannot do. That means one call site talking to two backends with different auth, different streaming shapes and different context limits. Multigrid exists to make that one API call with a routing rule, so the cost decision stays a configuration change rather than a rewrite.
Top comments (0)