The New Memory Hierarchy.
If you run AI models in production, the bill has a shape you’ve probably noticed: a surprising amount of it traces back to memory. Every token a model generates has to be served out of fast memory sitting right next to the GPU, and that memory — High Bandwidth Memory, or HBM — is scarce, pricey, and effectively sold out for 2026. It’s the same crunch that pushed DDR5 prices three to four times higher over the past year: chipmakers aimed their best wafers at HBM because that’s where the margins are. For anyone serving long-context models or agents, memory has become the single largest cost driver in inference.
High Bandwidth Flash — HBF — is built to attack exactly that cost. The pitch is much of HBM’s speed at a fraction of the price per gigabyte, by taking the cheap, dense flash from your SSDs and rebuilding it to keep pace with a GPU. If it delivers, it could meaningfully lower what it costs to run large models. So it’s worth understanding three things: what HBF actually is, how it would cut that cost, and how seriously to take the claim — because it isn’t shipping yet, and the savings so far live in simulations, not invoices.
Why HBF exists at all
Start with the problem, because HBF was built to solve a specific one. Modern AI inference — the part where a trained model answers your prompt — is hungry for two things at once: enormous memory capacity and enormous memory bandwidth. Capacity, because the models are huge and because long conversations build up a large running cache of context (the “KV cache”) that has to stay in fast memory. A single 128K-token conversation on a 70-billion-parameter model can consume roughly 40GB of HBM per concurrent user. Multiply that across users and you run out of room fast. Bandwidth, because the accelerator has to stream all that data in and out continuously or it sits idle.
Here’s the bind. The memory that delivers the bandwidth — HBM — is expensive and physically limited in how much it can hold. The memory that delivers the capacity cheaply — NAND flash, the stuff in SSDs — is far too slow to feed an accelerator directly. Engineers call the gap between what the processor wants and what memory can deliver the “memory wall,” and for inference it’s now the single largest cost driver. You can’t fix it by buying more HBM, because there isn’t enough capacity per stack and you’d be stacking GPUs just to hold data that’s mostly sitting still.
HBF is one attempt to climb that wall: take cheap, high-capacity flash and make it fast enough to sit much closer to the accelerator than an SSD ever could.
What HBF actually is
HBF is NAND flash — the same base technology as the SSD in your laptop — rebuilt to behave much more like HBM. It stacks flash dies vertically and wires them together with the same dense, high-speed packaging HBM uses, so a single stack can move data at around 1.6 TB/s, close to HBM’s range. The payoff is capacity. Because flash packs far more bits per layer than DRAM, an HBF stack is projected to hold up to 512GB, against roughly 48–64GB for an HBM4 stack — on the order of ten times the data, at lower cost per gigabyte.
The standardization piece is real and worth one sentence: SanDisk and SK hynix are jointly defining a common HBF spec under the Open Compute Project, which matters because it means HBF is heading toward an industry standard rather than a single vendor’s experiment, and Samsung is reportedly exploring the same direction.
Now the part that the “rivalry” framing gets wrong. HBF is not built to replace HBM. It’s built to sit beside it, as a new tier between HBM and the SSD. The reason comes down to one idea that’s worth slowing down for.
The split that makes HBF make sense
Think about what a model actually touches while it works. A large share of that data never changes during the job: the model’s weights are frozen, and in long-context systems there’s a big precomputed cache that gets read constantly but rewritten rarely. Then there’s a smaller pool of fast-changing data the model updates with every token it generates.
Those two jobs want two different kinds of memory.
The fast-changing churn wants HBM. DRAM can be rewritten endlessly and responds in nanoseconds, which is exactly right for data that updates every token. The giant frozen pile — weights and the read-mostly cache — is a poor match for HBM for one expensive reason: HBM doesn’t have the capacity, so you end up chaining together GPUs just to store data that barely moves.
That frozen pile is what HBF is for. Flash has two well-known weaknesses: it wears out after a limited number of writes, and it’s slower to reach than DRAM, microseconds versus nanoseconds. Neither hurts much if you’re mostly reading. You don’t rewrite frozen weights, so write endurance barely matters, and because inference reads data in a predictable order, the system can fetch the next chunk before it’s needed and hide most of the delay. SK hynix calls this division of labor H³ — read-only data parked in HBF, everything else in HBM, both feeding the same GPU.
The honest one-liner: HBM and HBF aren’t fighting over the same job. One holds the data that moves; the other holds the data that mostly sits.
HBF is a near-memory tier: it sits on the GPU package alongside HBM, not out past the system bus like RAM and SSDs. Closer to the GPU means faster but costlier per gigabyte — which is why each kind of data lands where it does. (Specs approximate; HBF is projected for 2026–27.)
Will it actually work — and when?
Here’s the reality check the headlines tend to skip. HBF is not a product you can buy. It’s at the prototype-and-simulation stage. SanDisk expects first samples in the second half of 2026, with the first inference systems using it sampling in early 2027. There is no shipping silicon today, and the performance numbers in circulation come from the vendors’ own models, not independent benchmarks.
With that caveat stated plainly, the early numbers are encouraging — and they point straight at cost. In SK hynix’s H³ simulation — eight HBM3E stacks and eight HBF stacks on an Nvidia Blackwell B200 — the hybrid reportedly delivered up to 2.69x higher throughput per watt than an HBM-only system, and on very long contexts, 1.25x the tokens per second at one million tokens, rising to 6.14x at ten million. Both numbers map onto an inference bill. More throughput per watt means less power for the same work; more tokens per second from the same GPUs means the giant model weights and KV cache get spread across more requests, so the fixed cost of holding them in memory is shared more widely. That is the mechanism behind “cheaper” — not a discount on the parts, but more useful work squeezed out of each GPU. And the longer the context, the more it pays off, which happens to be the workload growing fastest.
There’s also real engineering risk between here and production. Stacking sixteen thin, fragile NAND dies and bonding them without cracking the wafers is hard, and flash’s microsecond latency only stays hidden if the prefetching works as modeled under messy real-world loads. None of that is settled yet. Treat HBF as a credible, well-backed bet — not a done deal.
HBF is not the only answer to the memory wall
This is the part most HBF coverage leaves out, and it’s the part that actually informs a buying decision. HBF is one approach among several that the industry is pursuing at the same time, and most of them are complementary rather than winner-take-all. A few worth knowing:
CXL memory expansion. CXL is an interconnect standard that lets a system attach large pools of ordinary DRAM that the accelerator can reach coherently. Models that don’t fit in local HBM can spill into a slower but much larger CXL-attached tier instead of forcing you to add GPUs for capacity alone. SK hynix is building CXL Memory Modules for exactly this. CXL is a DRAM-based capacity tier; HBF is a flash-based one. They can coexist.
Processing-in-memory (PIM). Instead of moving data to the processor, PIM puts small compute units inside the memory chip so it can do some of the math — especially the attention operations at the heart of LLMs — and send back only the results. Industry estimates put more than 80% of the time and energy in some AI workflows into data movement, which is what PIM attacks. SK hynix has already shown its AiM accelerator, including a GDDR6-based card demonstrated at CES 2026 alongside Nvidia H100 GPUs, and Samsung has its own PIM line.
LPDDR server modules (SOCAMM). This one is already shipping. SOCAMM is an Nvidia-designed modular form factor built on low-power LPDDR5X. Micron is sampling a 256GB SOCAMM2 module, SK hynix has begun mass production of a 192GB version for Nvidia’s Vera Rubin platform, and the modules claim more than double the bandwidth and about 75% better power efficiency than conventional RDIMM server memory. It’s a near-term capacity-and-efficiency play for CPU-attached memory, available now rather than in 2027.
HBM itself isn’t standing still. HBM4 is reaching production in 2026 with over 2 TB/s per stack and capacity pushing toward 64GB, plus customizable base dies. The fast tier keeps getting faster and a bit roomier.
Read together, these don’t describe a single winner. They describe a stack of tiers — HBM (and HBM4) for speed, HBF and CXL for cheap capacity, PIM for cutting data movement, LPDDR modules for efficient bulk memory — that future systems will mix and match. HBF’s specific niche is the high-capacity, read-heavy flash tier. That’s a real and useful slot, but it’s one slot among several.
What this means if you buy and sell hardware
For your day-to-day right now, very little changes, and that’s the honest takeaway. HBF won’t appear in a system you can purchase until 2027, and it won’t reach the secondary market for years after that. It doesn’t make your current accelerators, your DDR5 RAM, or your SSDs obsolete. The technology to watch isn’t a new sticker price — it’s whether the read-heavy flash tier proves out in real systems.
The price story does have a real kernel, just not the dramatic one. HBF is made from NAND, the same flash that goes into SSDs, and the economics that pulled DRAM toward HBM apply to flash too: if a fab earns more turning NAND into a premium AI part than a consumer drive, the incentive points the same way. But HBF isn’t in volume production, so it is not the reason flash prices have been climbing in 2026 — that’s broad AI-and-datacenter demand straining every memory type. HBF is better read as a signal than a cause: the same force that reshaped DRAM is now reaching flash. Whether HBF specifically ever consumes enough NAND to dent SSD supply is an open question that won’t be answerable until real volume exists in 2027 and beyond.
One practical point cuts the other way. Tight supply props up the value of working hardware. Used and surplus memory and flash hold their worth better when new supply is scarce, which is part of why decommissioned components have stayed in demand through this stretch. If you’re sitting on pulled memory or drives from a refresh, that gear is worth more in a tight market than a loose one.
The thing to keep an eye on isn’t a price tag, it’s a milestone: the first 2027 systems that put HBF in front of a real workload. That’s when you’ll learn whether it lives up to the simulations — and whether the flash tier becomes a fixture of AI memory or a footnote.
I originally published this piece of work at BuySellRam.com, a platform for people to sell used parts, e.g., sell memory, sell used SSD, sell GPU, etc.
Top comments (0)