Which hardware should run this model? Most teams answer that question with habit instead of arithmetic. The model card says GPU. The cluster has a GPU quota. The last project used a GPU, so this one will too. Then the invoice arrives, and the model that should cost pennies is costing dollars, because nobody asked the only question that matters: how many tokens per second does this workload actually need?
That number exists before you buy anything. A background job that summarizes a document overnight needs a handful of tokens per second. A chatbot in front of a user needs dozens. A batch re-ranker that processes a queue while nobody watches needs whatever keeps the queue from growing faster than it drains. Those are three different hardware answers, and two of them do not involve a GPU at all. This article shows how to compute your ceiling before you spend, how to measure the machine you already have, and how to turn the result into a decision you can defend in a review meeting.
The Two Phases Have Different Bottlenecks
Every LLM generation run has two phases, and they are not the same kind of work. Prefill processes the whole prompt at once: every token attends to every other token, which is a wall of matrix multiplications. That phase is compute-bound. More FLOPs per second wins, and GPUs win that game by an order of magnitude.
Decode is the second phase. The model emits one token, feeds it back, emits the next. Each step moves the entire weight matrix from memory into the compute units, does a comparatively small amount of arithmetic on it, and writes one token out. The arithmetic per token is tiny. What dominates is the memory traffic: the weights have to cross the memory bus once per token, every token, until generation stops.
That is why decode speed on a given machine is set by memory bandwidth, not by FLOPs. The compute units sit idle waiting for weights to arrive. A GPU still wins decode because its memory subsystem is wider and faster, but the win is a bandwidth win, and bandwidth is a property a CPU can also have. The question is whether the CPU has enough of it for your workload.
The Decode Ceiling Is a Bandwidth Calculation
The ceiling is arithmetic, and it takes one line. A model reads its weights once per token, so the maximum tokens per second is the memory bandwidth divided by the size of the weights in bytes.
def decode_ceiling(bandwidth_gb_s: float, weights_gb: float) -> float:
# pure arithmetic: bytes per second divided by bytes per token
return bandwidth_gb_s / weights_gb
Plug in typical numbers to see the shape. A 7-billion-parameter model stored at 4 bits per parameter is roughly 3.5 GB on disk and in memory. A laptop with 40 GB/s of effective memory bandwidth computes a ceiling near 11 tokens per second. A machine with 100 GB/s of bandwidth doubles that. A GPU with 500 GB/s or more pushes the same model past 100. None of these are measurements of any particular product. They are the theoretical ceiling the machine cannot beat, and they come straight from the specification sheet.
Two consequences fall out immediately. First, the model size in the denominator matters as much as the hardware in the numerator: a 1-billion-parameter model at 4 bits is about 0.5 GB, which gives a CPU a ceiling of dozens of tokens per second. Small models on CPUs are not a compromise; they are the point where the arithmetic stops favoring the GPU. Second, a workload that needs 8 tokens per second has no business paying for a machine whose ceiling is 200. The GPU is overkill by a factor of twenty-five, and you pay for the whole factor.
Where the GPU Stops Being Worth It
Three workload shapes routinely fall under the CPU ceiling. The first is throughput-insensitive work: nightly summarization, document classification, log triage, anything that runs in the background and has no human waiting. Nobody notices whether a batch job finishes in four minutes or forty, as long as it finishes before the morning. The second is low-concurrency work. A single user or a single queue means one stream of generation at a time. GPUs earn their price when many streams share the hardware; a lone stream uses a fraction of it. The third is long-context re-reading. A job that feeds a large document to a model spends most of its time in prefill, and prefill on a CPU is slow enough to matter — but if the same document is processed every night on a schedule, slow is still fast enough.
Each of these maps onto the ceiling formula. Write down the required tokens per second. Multiply by the margin you want for spikes and retries. If the product stays under the measured ceiling of the CPU you already own, the purchase decision is already made and the answer is to buy nothing.
A Harness That Measures Your Machine
Spec sheets give ceilings. Your machine gives reality, and reality is what decides. The harness below measures decode throughput directly: load a GGUF model with llama-cpp-python, generate a fixed number of tokens, and divide the count by the wall time. No external server, no cloud account, no vendor dashboard.
import time
from llama_cpp import Llama
llm = Llama(model_path="model.gguf", n_ctx=8192, n_threads=8, verbose=False)
prompt = "Explain why a GPU is not always the right answer for inference."
start = time.perf_counter()
out = llm(prompt, max_tokens=512)
elapsed = time.perf_counter() - start
tokens = out["usage"]["completion_tokens"]
print(f"prefill+decode wall time : {elapsed:.2f}s")
print(f"decode throughput : {tokens / elapsed:.1f} tokens/s")
Run the same script on the GPU box and on the CPU box, with the same model file and the same prompt length. The ratio between the two is the real answer to the purchase question, and it is usually smaller than marketing suggests, because decode is bandwidth-bound on both machines. Do not trust the number printed by a benchmark suite you did not write; trust the number you produced with your own workload shape.
Reading the Numbers
The measured throughput is a fact, not a verdict. Turning it into a decision requires the requirement side of the equation. A chatbot needs its first token fast and its subsequent tokens steady; the decode rate matters on every interaction. A batch worker needs the average rate to exceed the arrival rate of the queue; short bursts do not matter, sustained rate does. A report generator that runs once a day needs the total wall time to fit inside a maintenance window.
Compare the requirement against the measurement with an explicit margin. If the requirement is 8 tokens per second and the CPU measured 14, the decision is easy. If the CPU measured 10 and the requirement is 9, the margin is too thin for comfort and the GPU wins on headroom, not on peak speed. The margin is a policy choice, not a physics constant, which is exactly why it belongs in code.
The Decision Procedure
The whole process compresses into a small function that takes the requirement and the measurement and returns a verdict. Putting it in code forces the assumptions out of the conversation and into a file where they can be reviewed.
def choose_hardware(required_tokens_s: float, measured_cpu: float, margin: float = 1.5) -> str:
needed = required_tokens_s * margin
if measured_cpu >= needed:
return "cpu"
if measured_cpu >= required_tokens_s:
return "cpu, thin margin — recheck after quantization"
return "gpu"
The verdicts read as plain language on purpose. The first branch is the whole article in one line: measured throughput beats the requirement with margin, so the GPU is overkill. The second branch says the arithmetic is close enough to test one more variable before spending. The third branch is the only one that actually buys hardware.
What Quantization Changes
The denominator of the ceiling formula is bytes per parameter, and quantization is how you shrink it. A model at 8 bits per parameter takes twice the memory of the same model at 4 bits, which halves the CPU decode ceiling for the same bandwidth. Dropping from 8 bits to 4 bits doubles the ceiling and can move a machine from the third branch to the first.
The catch is that quantization trades bytes for accuracy, and the trade is task-dependent. A model that classifies boilerplate may survive 4-bit without a visible difference. The same model doing precise extraction from legal text may not. The correct test is cheap and mechanical: run your own evaluation set through both quantizations, compare the outputs, and let the comparison decide. Never assume the ceiling math is the only math in the room.
Guardrails and Regression Harness
The measurement is only trustworthy while the machine and the software stay the same. A dependency upgrade can silently change the runtime, a shared machine can lose bandwidth to a noisy neighbor, and a model file swap can change the effective size. The cheap protection is a regression harness that re-measures and asserts, run after every change that touches the pipeline.
import json
from pathlib import Path
def assert_decode_ceiling(path: str, floor: float) -> None:
result = json.loads(Path(path).read_text())
measured = result["decode_tokens_per_second"]
assert measured >= floor, f"decode fell below {floor}: {measured:.1f}"
Wire the harness into CI next to the unit tests. It will not catch every regression, but it catches the expensive kind: the one where the model still answers correctly and the bill simply grows.
The conclusion is not a recommendation to abandon GPUs. It is a recommendation to stop guessing. The requirement is a number you can write down. The ceiling is a number you can compute from a spec sheet. The measured throughput is a number you can produce in five minutes on the machine you already own. When all three numbers exist, the hardware decision stops being a belief and becomes a calculation — and a surprising fraction of the time, the calculation ends in buying nothing.
Originally published on Dispatch.
Top comments (1)
Measurement-first is the right instinct here. GPU feels like the default answer because it is visible and exciting, but latency targets, batch size, model shape, and utilization often decide the architecture before hardware does.