Every accelerator launch follows the same script. More HBM stacks, more bandwidth, a bigger number on the slide. HBM3E gave us roughly 1.2 TB/s per stack. HBM4 promises twice that. And yet the teams running inference at scale keep reporting the same thing: the model is not compute-bound, it is traffic-bound, and buying bandwidth has stopped buying throughput at the rate it used to.
The reason is structural. Decode is a bandwidth problem, and bandwidth does not scale the way capacity does. Adding stacks adds both, but the energy per bit moved across a package edge is the part that refuses to improve, and at some point the power budget, not the pin count, is what limits you.
Part One: Why the wall is a wall
A 70B model in FP8 is 70 GB of weights. Generating one token requires reading all of them once. At 3 TB/s of aggregate bandwidth, that is 23 milliseconds per token as a hard floor, before any batching, before any kernel inefficiency. Batch thirty-two requests and you amortise the read, which is why serving systems batch aggressively. But batching does not reduce the bytes per token, it reduces the bytes per user-token.
Now add attention. Each decoded token attends to the whole key-value cache of its own conversation. At 100,000 tokens of context, a 70B model reads tens of gigabytes of cache per step. The weights are read once per step, the cache is read once per step per sequence. At long context the cache dominates, and the cache lives in the same memory you were already saturating.
So the arithmetic is unforgiving: the number of bytes you must move grows with both model size and context length, while the bytes per second you can move grows with process node and packaging, roughly 20-30% per generation. The gap between the two curves is the memory wall, and it is widening, not closing.
Part Two: The three conventional answers
The industry has three ways to attack this without changing the memory itself.
Widen the bus. More stacks, more channels, wider PHYs. This is what HBM4 does, and it works, at the cost of package area, power and yield. Every additional stack is another set of known-good-die requirements, and the package becomes the yield limiter rather than the logic die.
Compress the traffic. FP8, FP4, sparsity, KV cache quantisation. This is the highest-leverage intervention available today, because it reduces bytes directly rather than trying to move them faster. It also has a floor: at 4 bits per weight, quality loss becomes workload-dependent and you cannot promise it universally.
Restructure the computation. Flash attention, grouped-query attention, speculative decoding. These reduce the number of times you touch memory rather than the size of what you touch. Speculative decoding in particular trades compute for traffic: a small draft model proposes tokens, the large model verifies them in one pass, and you amortise one weight read across several accepted tokens.
All three are incremental. None of them changes the fact that the multiply sits on one side of a wire and the data on the other.
Part Three: What PIM actually changes
Processing in memory puts a small amount of arithmetic inside the memory die or the memory stack's logic layer. For inference, the operation that matters is the matrix-vector product in decode: for each output element you need a dot product between a weight row and an activation vector. If the weight never leaves the DRAM stack, you eliminate the off-chip read entirely.
The published HBM-PIM work reports roughly two times the system throughput on bandwidth-bound workloads, and a little over two times lower energy per operation, using the same DRAM process and the same package. That is a large number for a memory generation that usually delivers 20%.
The catch is that PIM only helps the operations that are memory-bound and regular. Prefill is compute-bound and wants big matrix-matrix multiplies on the accelerator. Attention is irregular. So a PIM system ends up heterogeneous: prefill on the main die, decode partly in the memory stack, a scheduler deciding which runs where. That heterogeneity is where the engineering cost lives, and it is why PIM has been "two years away" for a decade.
Part Four: The economics, not the physics
Here is the part that decides adoption. A PIM stack costs more per gigabyte than a standard stack, because the logic layer consumes die area that would otherwise hold cells. If the price premium is 20% and the effective bandwidth per watt improves by 2.5 times, the payback depends entirely on your duty cycle.
For a service running decode at high utilisation, the bandwidth saving shows up in the power bill and in the number of accelerators required. My model, assuming constant accelerator price and a 4x reduction in off-chip reads for the attention path, puts the cost per million output tokens at roughly 30% below a standard HBM3E system by 2027. That is the difference between a viable product and a marginal one in a price-sensitive API market.
For a team running fine-tuning, PIM is close to useless. Fine-tuning is compute-bound, and the memory traffic is a small fraction of the total. Paying a premium for bandwidth you do not saturate is how hardware roadmaps get cancelled.
The honest framing is that PIM is a decode technology, and decode is where the money is. Training clusters will keep buying bandwidth. Inference fleets are the ones that will pay for locality.
Part Five: What builders should do now
Three practical positions.
First, treat bandwidth per token as the metric you optimise, not bandwidth per second. A system that reads 4 bits per weight at 2 TB/s beats one that reads 8 bits at 3 TB/s, and the first is available today.
Second, measure your prefill-to-decode ratio before you buy anything exotic. If 80% of your GPU time is prefill, PIM will not help you, and neither will any memory-side accelerator. Fix the batching first.
Third, assume heterogeneity. The next three years of accelerator design are about moving work to wherever it is cheapest to execute, which means your serving stack will have to schedule across different kinds of compute. Teams that assume a single uniform device will find themselves unable to use the hardware they bought.
The transferable lesson
Every generation of computing has one resource that is cheap at the margin and expensive in aggregate. In the 1990s it was disk seeks. In the 2000s it was network round trips. In the 2010s it was memory bandwidth for analytics. Now it is bytes moved per generated token.
The pattern is always the same: the industry attacks it with a bigger pipe, the pipe helps for one generation, and then the physics reasserts itself and the only remaining move is to stop moving the data. PIM is not a faster pipe. It is the decision to stop using the pipe, and that is why it will eventually win despite the packaging cost.
SOL: HCqPmPJM3NVtS6YrmVaTZYnuP8XZ2PFgQZKT94VK1LkS
Creating this takes real effort — thank you for the support.
Top comments (0)