Every capacity conversation in AI infrastructure for the last five years has been a conversation about high-bandwidth memory. Stacks of it, sitting on an interposer, feeding an accelerator through a bus so wide that the package looks like a small circuit board. That is where the bandwidth records are set, and it is where almost all the engineering attention has gone.
It is also, increasingly, the wrong place to look. The number of inference operations that happen inside a rack is growing, but the number that happen inside a phone, a laptop, a car and a camera is growing faster, and those devices do not have an interposer, a cold plate, or a kilowatt to spend. Inside a twenty-watt envelope, the question is not how many gigabytes per second you can buy. It is how many you can buy per joule, and the answer changes which memory wins.
Two ways to buy bandwidth
HBM buys bandwidth with width and proximity. A stack sits millimetres from the die, connected through thousands of parallel wires, and the energy per bit is low precisely because the distance is short. What it costs is area, packaging complexity, and yield: an interposer is expensive, the stack is expensive, and the whole assembly has to be tested as a unit.
LPDDR buys bandwidth with efficiency per pin and a mature supply chain. It runs at lower signalling rates over longer traces, uses a narrower bus, and costs a fraction per gigabyte. What it gives up is peak: a mobile-class package tops out at a small fraction of what a single HBM stack delivers.
The trade looks obvious until you put both inside a power budget. A datacentre accelerator can spend two hundred watts on memory movement because it has a kilowatt of headroom and a cold plate. A handset has three to five watts for the entire memory subsystem, and at that level the efficiency of the interface, not its peak width, sets the ceiling.
Why on-device inference is a different workload
The workloads that matter on a device are not scaled-down datacentre workloads. They have small batch sizes by construction, because there is one user. They have short contexts, because a phone conversation is not a 128k-token document. And they increasingly route through mixture-of-experts architectures, which touch a small fraction of the weights per token but must hold all of them in memory.
That combination produces a very specific profile: capacity-bound rather than bandwidth-bound for the weights, bandwidth-bound for the key-value cache, and brutally sensitive to the energy cost of every byte moved. A model that needs twelve gigabytes of weights does not care whether the interface can do three terabytes a second. It cares whether twelve gigabytes fit, and whether reading them costs more than the thermal budget allows.
The decisive metric on a device is not bandwidth per second but bandwidth per joule, and that single change in units reorders the entire memory hierarchy.
Once you plot the two families on energy per bit rather than peak rate, the gap narrows dramatically and the packaging cost becomes the tiebreaker. LPDDR does not win because it is faster. It wins because at the power levels a battery can supply, it is the only one whose peak is reachable at all.
The cost structure nobody puts on the datasheet
There is a second asymmetry, and it is financial rather than electrical. HBM capacity is priced by a supply chain with three credible suppliers, long qualification cycles, and allocation that follows accelerator orders. LPDDR capacity is priced by a much broader market that also serves phones, cars and consumer electronics, and it is available in volumes that a device maker can actually plan against.
That matters more than it sounds. A product team choosing memory in 2027 is not choosing a peak number, it is choosing a supply position: whether the bill of materials can survive a demand spike, whether a second source exists, and whether the package can be assembled by more than one factory. HBM wins every benchmark and loses several of those questions.
The mechanism behind the forecast is straightforward. As on-device models move from one to eight billion parameters and then toward sparse models with far more parameters but similar active compute, the binding constraint shifts from compute to capacity, and capacity is the axis where LPDDR has both a cost and a supply advantage. The forecast that follows is not that LPDDR replaces HBM, but that the two split along a power line, with HBM above roughly 150 watts per package and LPDDR below it.
What this does to products
If that split holds, the practical consequences arrive in a predictable order. First, phones and laptops get usable local assistants without a network round trip, because twelve gigabytes of weights at four gigabytes per second is enough to generate text faster than a person reads. Second, the marginal cost of an inference falls toward the cost of the electricity, which is where the business model changes: a feature that costs a fraction of a cent per use can be given away, and one that costs a cent cannot.
Third, and least discussed, the datacentre stops being the only place where model quality is decided. A vendor that ships a smaller model tuned for a device budget can beat a larger one that has to be reached over a network, and the competitive pressure moves from parameter count to efficiency per joule.
The plausible range is wide, and worth stating with the reasoning attached. On-device inference silicon is currently a few billion dollars a year; if the power-line split holds and the average handset carries eight to sixteen gigabytes of dedicated model memory by 2029, the addressable memory revenue for this segment lands somewhere between fifteen and forty billion dollars annually, with the low end assuming that most inference stays in the cloud and the high end assuming that a third of consumer AI traffic moves local. The uncertainty is not in the direction, it is in how quickly the software stack learns to exploit a small, fast, local memory pool.
What is actually transferable
The transferable idea is that every performance argument is an argument about units. HBM is optimised for bandwidth per millimetre, because its buyer has millimetres and watts to spare. LPDDR is optimised for bandwidth per joule, because its buyer has neither. Neither is better in the abstract; each is correct inside a different constraint.
The same reasoning explains why flash displaced disk in laptops long before it displaced it in archives, why SRAM persists inside chips where DRAM would be cheaper, and why edge accelerators keep choosing smaller, cheaper memory over the fastest available. The winning memory is the one whose cost lands on the axis the product is actually constrained by.
For a decade the memory industry sold peak numbers because peak numbers were the constraint. At the edge the constraint is a battery, and batteries do not read datasheets.
If this saved you time, a tip is appreciated: HCqPmPJM3NVtS6YrmVaTZYnuP8XZ2PFgQZKT94VK1LkS
Top comments (0)