DEV Community

Cover image for The Hopper Trap: When the Optimal Tile Size Wastes Half Your Register File
Dikhyant Krishna Dalai
Dikhyant Krishna Dalai

Posted on Originally published at Medium

The Hopper Trap: When the Optimal Tile Size Wastes Half Your Register File

TL;DR

On NVIDIA Hopper, a larger tile can create enough register pressure to make a "smaller" configuration ~30% faster. In a December 2023 Colfax Research case study implementing FlashAttention-2, the supposedly optimal 128×128 tile hit 208.7 TFLOPs/s, while 128×64 hit 295.7 TFLOPs/s. The fix was counterintuitive: deliberately leave register capacity unused. This post walks through why.

The performance paradox

When optimizing GPU kernels, bigger tiles seem like the obvious choice: more data per tile, better reuse, fewer iterations, higher throughput.

A December 2023 Colfax Research case study implementing FlashAttention-2 on Hopper benchmarked different tile configurations. At head dimension 128:

Tile configuration Performance
128 × 128 208.7 TFLOPs/s
128 × 64 295.7 TFLOPs/s
64 × 128 289.3 TFLOPs/s

The "optimal" 128×128 configuration was ~30% slower than the smaller alternatives.

Why: register pressure

The second GEMM needed its operands and accumulators resident in registers simultaneously. The larger tile created a resource bottleneck that affected execution — the tile that minimized memory traffic maximized register pressure.

This is the trap: reuse is not free. You pay for it in registers. When simultaneously-live values exceed the register file, the compiler spills to local memory, reloads sit on the critical path of the inner loop, and occupancy drops.

Register pressure: a large tile exceeds the register file and spills to local memory

The result is a cliff, not a slope: performance is fine right up to the register limit, then falls off sharply.

The counterintuitive workaround

Instead of adding more parallelism, the researchers used one warpgroup (128 threads) per CTA.

This deliberately left the register capacity that another warpgroup could have consumed unused. Using fewer resources produced better performance.

A second warpgroup would have split the same register file between twice as many threads, cutting each thread's allocation and pushing the kernel back over the spill threshold — the same trap, just one level up.

More Hopper pitfalls

The problem doesn't end with tile sizing. Hopper's setmaxnreg instruction introduces additional considerations:

  • Compiler behavior around __launch_bounds__.
  • Silently dropped register-allocation requests.
  • Verifying USETMAXREG in generated SASS.
  • Register-pool allocation and potential deadlocks in warp-specialized kernels.

A dropped setmaxnreg request is invisible at the source level — the compiler accepts it, but the generated SASS doesn't contain the instruction. The practical check is to dump the SASS and look for USETMAXREG before trusting your occupancy numbers; if it's missing, your register budget is not what you think it is.

Takeaway

Hopper's performance model isn't a simple extension of Ampere's. Tensor-core throughput is only part of the problem; register allocation, warp specialization, and compiler behavior can fundamentally change the outcome.

Before you trust a tile-size heuristic, look at what the compiler actually emitted. The optimal tile size for your model is not necessarily the optimal tile size for your hardware.


Originally published on Medium. If you've hit this on a different architecture, I'd like to hear about it.

References

  1. Bikshandi, G. & Shah, J. (2023). A Case Study in CUDA Kernel Fusion: Implementing FlashAttention-2 on NVIDIA Hopper Architecture using the CUTLASS Library.
  2. NVIDIA PTX ISA Documentation.
  3. NVIDIA CUTLASS.

Top comments (0)