DEV Community

RuChating
RuChating

Posted on

The gigabyte is the new flop: sizing AI infrastructure after the memory repricing

For most of the last three years, capacity planning for LLM serving started with a flop count and ended with a GPU count. That arithmetic no longer closes. In the fourth quarter of 2026 the cost of a gigabyte of memory is rising faster than the cost of a flop is falling, and the tooling has started to assume it. If you own an inference deployment, the decisions that matter this quarter are memory decisions.

Three things happened in the same seven days to make that concrete: a major inference engine changed what "default" means, a memory price forecast landed at roughly twenty percent quarter-over-quarter, and a vendor shipped an accelerator with 432 GB of HBM4 in it. None of them is a benchmark record. Together they redraw the sizing sheet.

What vLLM 0.34 actually changed

vLLM 0.34.0 shipped on 16 November: 1,061 commits, 412 contributors, 120 of them first-timers. Release notes are worth reading in full, but one item deserves attention before you touch a production cluster.

Extensible KV cache is now the default, and gpu_memory_utilization now defaults to 1.0 instead of 0.9. The engine no longer sizes the KV cache from an estimate of what fraction of the device it may occupy. It sizes it from what is actually free after CUDA graph capture and warmup have taken their share. Practically, the engine measures first and allocates second.

The upstream example: MiniMax-M3 on an H200 goes from 433k cached tokens to 672k. That is a 55% increase in resident context, and it arrives without a code change. More cache means higher concurrency before preemption, and lower time-to-first-token under load, because fewer requests get queued behind a full block table.

The same release notes state the cost plainly, which is rarer than it should be: per-token decode latency can increase by 6–18% on some H200 configurations at concurrency 128. If you sell a latency SLO, and you sized your cluster around gpu_memory_utilization: 0.9, this default is not a free upgrade. It is a different operating point. You can pin the old behaviour with --gpu-memory-utilization 0.9 or --no-enable-extensible-kv-cache, but you have to know to do it before the rollout, not after.

Two more defaults moved in the same release. The block size default went from 16 to 64, which changes both fragmentation behaviour and memory accounting on some models. And Model Runner V1 is gone: setting VLLM_USE_V2_MODEL_RUNNER=0 is now a startup failure rather than a fallback, so any Helm chart carrying that env var from an older migration is a landmine. ROCm 7.2 artifacts are no longer published, huggingface_hub 2.0 is required, and the Marlin W4A8 path was replaced by --quantization humming.

The throughput numbers are real, and they are the reason people will upgrade anyway: up to 1.95× decode on Step5 at GB200 scale, 1.6× throughput on GLM-5.3-Flash, and 1.24–1.40× faster TTFT on long prompts with GLM-5.3. Speculative decoding also moved onto the new runner, with suffix decoding no longer requiring Arctic Inference, a plugin API for custom speculators, and CUDA graphs for draft models.

Here is the migration checklist I would actually run, in order. Grep every deployment manifest for VLLM_USE_V2_MODEL_RUNNER and block-size. Pin gpu-memory-utilization explicitly rather than inheriting the new default. Re-run your own latency benchmark at your real concurrency, not at concurrency 1, because the regression shows up under load. Then check whether the extra cache lets you drop a replica — the 55% context gain sometimes pays for the TPOT cost outright.

Why the memory math got harder

TrendForce put fourth-quarter DRAM contract prices at roughly 20% quarter-over-quarter in a 20 October note, with conventional DRAM at 10–15% and NAND at 15–20% in the 30 September forecast. The trajectory across the year is the interesting part: about 80% in Q2, 40% in Q3, 20% in Q4. Growth is decelerating. It is also the fourth consecutive quarter of increases, which makes it a cost plateau, not a spike.

HBM is the sharper edge. Analyst estimates put a HBM3 stack near $200, HBM3E near $300, and HBM4 near $550. Treat those as estimates rather than contract prices, but the ratio is the point: moving an accelerator to HBM4 nearly doubles the memory line in its bill of materials. That is why the current generation is expensive to build, and it is why NVIDIA's $130 billion quarterly guide reflects supply chain throughput rather than demand.

There is a second-order effect that catches people who do not follow memory markets. Micron has described roughly a 3-to-1 wafer conversion ratio between HBM and DDR5. Every wafer that becomes HBM stops being DDR5. So the AI buildout does not merely consume memory, it removes general-purpose memory from the market, which is why your ordinary server refresh is also more expensive this year.

TrendForce expects HBM to stay supply-constrained through 2027, and notes that 8-Hi stacks will carry a 10–20% per-gigabit premium over 12-Hi because the base-die cost spreads across less capacity. The engineering translation: stack height is now a purchasing decision, not a datasheet detail.

The hardware is already answering

Look at what AMD announced on 12 November and the memory-first logic is explicit. The Instinct MI430X carries 432 GB of HBM4 at 23.3 TB/s, alongside 288 TFLOPS of hardware FP64 for simulation workloads. The 432 GB number matters more than the flops for a specific reason: models that previously required two accelerators to hold their weights and KV cache now fit in one. That halves the interconnect traffic, halves the failure domain, and removes a tensor-parallel shard.

AMD also introduced an MI450X variant whose distinguishing feature is a different memory capacity for a different workload profile. A vendor shipping a second SKU defined by memory rather than by clock is telling you where the constraint sits.

Treat the vendor's own FP64 comparison against NVIDIA's Vera Rubin carefully — AMD's 8.7× figure is a July 2026 engineering projection from its own performance labs, not a measurement on shipped parts. Use it as an upper bound on expectations, nothing more.

At the other end of the scale, NVIDIA used SC26 to introduce a four-node DGX Spark kit: four GB10 Grace Blackwell systems in a ring, each wired directly to two neighbours over ConnectX-7 with no switch, 512 GB of combined memory, 16 TB of NVMe, and up to 700 billion parameters in NVFP4 across the cluster. The pitch is not throughput. It is that debugging a multi-node MPI job stops requiring a queue.

That is the same principle at desk scale: 512 GB of cheap-to-schedule memory beats a faster chip you can only reach on Tuesdays.

What to change in your sizing sheet

Sizing from flops made sense when compute was the binding constraint and memory was a rounding error. Reverse the two and three practical rules fall out.

Size from GB per replica, not TFLOPS per replica. Compute your weights, your KV cache at target concurrency and context length, and your activation peak, and let that number pick the SKU. With 432 GB parts available, the question "can this fit on one device" has a new answer more often than it did last quarter.

Price the memory refresh separately from the compute refresh. Your accelerator line and your DRAM line now inflate on different schedules, and a five-year amortization model that assumes both fall in price is wrong in one of the two columns.

Model the queue, not the rack. New AI halls are being built at 120–180 kW per rack, and the constraint on a new cluster is increasingly a grid interconnection date rather than an allocation letter for GPUs. If your capacity plan has a launch date but no energization date, it has a hole in it.

The uncomfortable conclusion

The interesting thing about this week is not that any single product got faster. It is that the software and the silicon converged on the same assumption within days of each other: memory is the scarce resource, so measure it, allocate it precisely, and let everything else follow.

vLLM now measures free memory after warmup instead of trusting a configured fraction. AMD now sells memory configurations instead of a single flagship. NVIDIA now sells a 512 GB desk cluster as a development tool. Each is a bet that the gigabyte, not the flop, is what engineers will be optimizing against for the next two years.

If your runbooks, your capacity sheets and your rollout checklists still start with a flop count, this is the quarter they stop being right.

Top comments (0)