Every capacity plan I have seen for an inference service in the last two years has been built on the wrong number. Teams size their clusters in FLOPs, negotiate discounts per GPU-hour, and then watch the cost per token refuse to fall the way the roadmap promised. The reason is not the model. It is that the model has become a memory problem wearing a compute costume.
Between 2020 and 2026, peak dense floating-point throughput per accelerator grew by roughly twenty times. Memory bandwidth per accelerator grew by about three and a half. That is not a rounding difference. It is a five-fold widening of a gap that was already the binding constraint, and every economic decision downstream inherits it.
The arithmetic intensity trap
A decoder doing batch-one inference on a long context does almost no arithmetic per byte it touches. To emit one token it must read the entire weight matrix and the whole key-value cache. The weights are fixed cost per token; the cache is not. It grows with context length, and at 128k tokens it dwarfs everything else in the memory hierarchy.
This is why the same model on the same hardware can cost ten times more per token at long context than at short context, and why vendors quote per-token prices that quietly assume a context length you will never stay under.
The chart above is the whole story in one frame. The blue bars are what the marketing deck shows you. The orange line is what the memory controller can actually feed. Everything between them is a queue, and queues cost money.
Where the dollar actually goes
I modelled a dense decoder at batch one across four context lengths and split serving cost into two buckets: moving bytes and doing arithmetic. At 4k tokens the split is roughly even. By 128k, three quarters of the cost is movement. At 512k it is more than eighty percent.
The practical consequence is that optimisation effort is misallocated almost everywhere. Teams spend quarters on kernel fusion to shave arithmetic, when the arithmetic is already the cheap part. The wins that actually move the bill are the unglamorous ones: quantising the KV cache, evicting it aggressively, and refusing to re-read weights you could have kept resident.
Why the gap keeps widening
The gap is not an engineering failure. It is a consequence of how the two technologies scale. Transistor density gives you more multiply-accumulate units almost for free, because they are regular and local. Bandwidth is bounded by the number of pins on the package and the signal integrity of the traces that reach them. You can double the math by spending area. You cannot double the pins.
High-bandwidth memory helped, and stacked memory helped more, but each generation buys a step, not a multiple. Meanwhile the model side keeps finding new ways to touch more bytes per useful operation: mixture-of-experts routing touches many experts per token, long context touches the whole cache, and retrieval touches an index that lives in the same memory.
What this does to product decisions
Three decisions follow directly, and they are not the ones most teams make.
First, context length is a pricing decision, not a feature decision. If your product lets users paste a hundred-page document, you have committed to a cost curve that your per-seat price probably does not cover. The right move is to charge for it explicitly, or to compress the document before it enters the model, not to hope the next hardware generation closes the gap.
Second, batch size is the only real lever on cost per token. Larger batches amortise the weight read across more tokens, which is exactly the resource that is scarce. A service that runs at batch one is paying for the full weight read on every request and passing none of it to a neighbour.
Third, memory is now a capacity-planning input in its own right. If you plan capacity in GPU-hours, you will systematically under-provision memory and over-provision compute, and your utilisation numbers will look fine while your latency tail falls apart.
The market arithmetic
If the current trajectory holds, bandwidth per accelerator roughly doubles every three years while compute per accelerator doubles every eighteen months. That means the gap widens until something breaks the trend: either memory moves onto the package in a way that scales pins, or the dominant workload shifts to something with higher arithmetic intensity.
My estimate is that the second happens first, and it is already visible. Inference workloads are migrating toward small, specialised models with short context and heavy retrieval, not because they are better in the abstract, but because they are the only shape that fits the economics. A 7B model with a good retriever costs a fraction of a 70B model with a long context and beats it on most narrow tasks.
On the supply side, this is bullish for anything that increases bytes per joule at the package level: stacked DRAM, silicon photonics for die-to-die links, and near-memory compute. It is bearish for the assumption that compute is the scarce input. The companies that will win the next cycle are the ones that price in bytes, not the ones that price in FLOPs.
For a consumer, the visible effect is a plateau in the price of AI features. Assistants will stop getting cheaper per query and start getting metered, and the marketing will call it "fair use". The underlying cause will be a pin count.
The counter-argument worth taking seriously
The obvious objection is that the gap closes itself, because model architectures are already adapting to it. Mixture-of-experts routing touches fewer weights per token. Speculative decoding amortises the weight read across several candidate tokens. Quantisation shrinks the bytes each weight occupies. All three are real, and all three are already in production.
None of them changes the direction of the argument, for a simple reason: they reduce bytes per token, but they do not change the ratio between bytes and arithmetic that the hardware offers. A four-bit weight still has to be fetched from somewhere, and a speculative draft still has to be verified against the full model.
What they do change is the shape of the optimum. With quantisation, the memory footprint falls faster than the compute requirement, so the balance shifts back toward arithmetic. That is why the teams reporting the best cost curves are the ones that quantise aggressively and then re-tune their batch sizes, rather than the ones that chase the largest model they can afford.
My reading is that architecture work buys roughly two years of headroom against the widening gap, and then the gap reasserts itself because the next model is larger and the context is longer. The trend is the trend. Optimisation moves you along the curve; it does not change its slope.
What to do on Monday
Instrument your service for bytes moved per token, not just tokens per second. You will almost certainly find that one endpoint, usually the one with a document upload box, is responsible for most of your memory traffic and most of your bill.
Then decide, explicitly, whether you are selling that endpoint at a profit or subsidising it. Most teams have never asked the question, because the dashboard was measuring the wrong thing.
If this saved you time, a tip is appreciated.
HCqPmPJM3NVtS6YrmVaTZYnuP8XZ2PFgQZKT94VK1LkS
Top comments (0)