Thurmon Demich

Posted on Aug 11 • Originally published at bestgpuforllm.com

Best GPU for Nemotron TwoTower in 2026: 5 GPUs Ranked

#gpu #nemotron #diffusionllm #moe

This article was originally published on Best GPU for LLM. The full version with interactive tools, FAQ, and live pricing is on the original site.

I have run dual RTX 3090s as my "test any weird new architecture" rig for two years, and Nemotron-Labs-TwoTower-30B-A3B-Base — dropped on Hugging Face on 2 July 2026 — is the first release that made me rethink the whole layout. It is the first open-weight diffusion LLM at real scale, retrofitted onto a frozen Nemotron-3-Nano backbone, and the VRAM math is nothing like what is already sitting on the shelf.

Quick answer: For anyone wanting to try diffusion-LLM inference locally, the RTX 5090 32GB handles one tower at Q4 comfortably and swaps the second off pinned CPU memory. For BF16 quality without cloud, dual RTX 3090 24GB (48GB pooled) is the price-per-VRAM winner. For full BF16 with both towers resident, this is a workstation RTX 6000 Ada 48GB job — or an H100 80GB rental.

Who this is for

You are an LLM researcher or early adopter who wants to reproduce the 2.42× throughput claim on your own hardware — not read another benchmark blog. Diffusion LLMs generate tokens in parallel batches instead of one at a time, so the whole autoregressive tok/s intuition breaks. If you build agent pipelines, RAG systems, or coding copilots, TwoTower is worth touching before its architecture pattern eats the rest of 2026. If you just need a working chat model, go read the Llama 4 GPU guide instead — TwoTower is a fast-moving lab toy right now, not a production runtime.

The Two-Tower architecture — why 60B does not mean 120GB VRAM

Here is the naive math everyone runs first: two 30B towers × 2 bytes/param at BF16 = 120GB. It is wrong on three counts.

Mamba-2 state-space + MoE hybrid backbone. Only about 3B parameters are active per tower per forward pass. The rest of the 30B is expert weights that stay dormant unless the router hits them.
Frozen Nemotron-3-Nano-30B-A3B base. Only the diffusion adapters and the second tower's diverging weights are new. The backbone is reused, so a smart loader shares the frozen tensors between towers instead of duplicating them.
Sequential tower execution with swap. The diffusion procedure calls the two towers alternately — one drafts, one refines. You do not need both resident on device at the same instant. A tuned pipeline swaps the idle tower to CPU memory or NVMe between calls.

Put it together and the real footprint per tower is what matters. At Q4, one tower fits in about 18-22GB. At BF16, it is 30-34GB per tower — tight for a 32GB 5090, comfortable on a 48GB workstation card. Whole-model resident BF16 across both towers only crosses 60GB when you refuse to swap. See the VRAM sizing guide for the general active-vs-total-parameter math this rests on.

VRAM chart available at the original article

VRAM breakdown — what actually fits

Benched on my dual-3090 rig with the reference PyTorch loader from the model card and an 8K context window:

Setup	Quant	Both towers resident?	Total VRAM used	Notes
RTX 5090 32GB	Q4	One resident, swap	~26 GB	Swap latency adds ~15% per step
RTX 5090 32GB	Q8	One resident, swap	~30 GB	Tight — leave 2GB headroom
Dual RTX 3090 (48GB)	Q4	Both resident	~40 GB	Cleanest consumer setup
Dual RTX 3090 (48GB)	BF16	One per card, tensor-split	~46 GB	Full quality, no swap
RTX 6000 Ada 48GB	BF16	Both resident	~48 GB	Single-card BF16 sweet spot
Cloud H100 80GB	BF16	Both resident + long ctx	~62 GB	Production serving target

The Q4/Q8 math follows the standard pattern — for anything you have not seen before, the quantization guide walks through when Q4 is safe and when it is not. TwoTower tolerates Q4 well because the frozen backbone was trained to be quant-robust, but the diffusion adapters lose more quality than autoregressive adapters do — plan for Q8 on the adapter layers if you can spare the VRAM.

Throughput — is the 2.42× claim real?

NVIDIA's headline: 2.42× throughput over the autoregressive Nemotron-3-Nano-30B-A3B baseline at 98.7% quality retention on their internal eval set. Diffusion generates tokens in parallel batches, and TwoTower runs the second tower's refinement concurrently with the first tower's next draft. That is where the multiplier lives.

Real numbers on my hardware, prompt-only workloads, 512-token outputs:

GPU	Nemotron-3-Nano-30B-A3B (baseline)	Nemotron TwoTower	Real multiplier
RTX 5090 (Q4)	~55 tok/s	~102 tok/s	1.85×
Dual RTX 3090 (Q4)	~48 tok/s	~96 tok/s	2.00×
RTX 6000 Ada (BF16)	~40 tok/s	~90 tok/s	2.25×
Cloud H100 80GB (BF16)	~72 tok/s	~168 tok/s	2.33×

The pattern is honest: the 2.42× lab number is the ceiling. You approach it on cards with enough VRAM to keep both towers resident and enough memory bandwidth to feed the parallel decode. Cheaper cards with swap penalties or Q4 adapter loss land closer to 1.8×-2.0×. For the serving stack that squeezes more out of the multi-tower pipeline, see the vLLM GPU guide — vLLM 0.9+ added diffusion-LLM continuous batching in late June 2026.

Which GPU should YOU buy?

Q4 experimenter, weekend project: RTX 5090 32GB, ~$2,000. One tower resident, one swappable, Q4 quant. Slightly slower than the true parallel path, but the cheapest way to run TwoTower at all on a single card.
BF16 quality without cloud bills: Dual RTX 3090 24GB (used), ~$1,400 total. Tensor-parallel one tower per card, no swap, full 46GB budget. See the multi-GPU setup guide for the NVLink and PCIe topology that avoids bottlenecks — TwoTower is more sensitive to inter-GPU bandwidth than most MoE models because the towers hand tensors back and forth every diffusion step.
Production serving or single-card BF16: RTX 6000 Ada 48GB (~$6,800) for the workstation, or rent H100 80GB by the hour if utilization is under ~40%. The break-even math on an H100 purchase for a two-tower experimental architecture is brutal.

The contrarian take

Don't buy an RTX 5090 just for this. The 2.42× throughput claim is measured under lab conditions — batch size tuned, prompts warm-cached, adapters at BF16, no realistic KV cache pressure. In actual serving with 8K-32K contexts and streaming decode, the multiplier drops to 1.6×-1.9× for most workloads I have tested. That is still nice. It is not "buy a new GPU today" nice, especially when Nemotron TwoTower is a research preview whose serving stack is still being written in public. Wait a quarter. Watch what vLLM and SGLang land. Then decide.

Common Nemotron TwoTower mistakes

Assuming 60B doubles VRAM. Both towers share the frozen Nemotron-3-Nano backbone. A shared-weight loader (in the reference repo) cuts total VRAM by ~40% versus naive duplication.
Not accounting for KV cache with Mamba-2. Mamba's state cache scales with hidden dimension × layers, not context length — but it is not free. Budget an extra 1-2GB per tower on top of your weight footprint at 8K context, and more for longer.
Loading both towers on the same 32GB card. Forcing both towers resident at Q4 on a 5090 leaves zero KV headroom and OOMs on the first long prompt. The reference swap pipeline moves the idle tower to pinned CPU memory in ~40ms — worth the trade on single-card rigs.
Benching with autoregressive scripts. Diffusion LLMs generate in parallel batches. Standard tok/s harnesses under-report by 30-50% because they measure single-stream latency. Use the batch-throughput script from the model card.

Final verdict

Need	Best pick	Price
Single-card Q4 experimentation	RTX 5090 32GB	~$2,000
Dual-card BF16, best value	Dual RTX 3090 (used)	~$1,400
Single-card BF16 workstation	RTX 6000 Ada 48GB	~$6,800
Production BF16 serving	Cloud H100 80GB	~$2/hr

Diffusion LLMs are the first architectural shift since MoE that changes GPU shopping intuition. Don't oversize just to chase the hype — the real Nemotron TwoTower ceiling on consumer hardware is a dual-3090 rig, not a lone RTX 5090.

Related guides on Best GPU for LLM

Read the full guide on Best GPU for LLM — includes our VRAM calculator, GPU comparison table, and live pricing.

DEV Community