Thurmon Demich

Posted on Jul 10 • Originally published at bestgpuforllm.com

Best Multi-GPU Setup for Local LLM in 2026 (Dual)

#multigpu #dualgpu #nvlink #tensorsplitting

This article was originally published on Best GPU for LLM. The full version with interactive tools, FAQ, and live pricing is on the original site.

Quick answer: Dual RTX 3090s (~$1,700 used) give you 48GB combined VRAM for running 70B models locally. Tensor splitting in llama.cpp works over PCIe without NVLink. For new hardware, dual RTX 5080s (32GB combined) or a single RTX 5090 (32GB) are cleaner options.

Why go multi-GPU?

Single consumer GPUs top out at 32GB VRAM (RTX 5090). To run 70B+ models at usable quantization, or 34B models at high quality, you need more memory. Multi-GPU setups combine VRAM from two or more cards using tensor splitting — the model is sliced across GPUs, and each card processes its portion in parallel.

VRAM chart available at the original article

How tensor splitting works

Tensor splitting (also called tensor parallelism) divides model layers across GPUs. When llama.cpp or Ollama runs a model:

Each GPU holds a portion of the model weights
During inference, GPUs compute their layers and pass activations to the next card
Communication happens over PCIe (or NVLink if available)

The key insight: you do not need NVLink. PCIe 4.0 x16 provides ~32 GB/s per direction, which is sufficient for inference. NVLink helps but is not required.

Best multi-GPU configurations

Setup	Total VRAM	Max model	Cost	Best for
2x RTX 3090 (used)	48GB	70B Q4_K_M	~$1,700	Best value
2x RTX 5080	32GB	34B Q5_K_M	~$2,000	Modern + fast
2x RTX 4090	48GB	70B Q5_K_M	~$3,200	Maximum speed
2x RTX 5090	64GB	70B Q8_0	~$4,000	Endgame
RTX 5090 + RTX 3090	56GB	70B Q5_K_M	~$2,850	Mixed budget

Dual RTX 3090 — best value multi-GPU

Two used RTX 3090s is the most popular multi-GPU LLM setup for good reason — we wrote a full step-by-step dual RTX 3090 setup guide covering hardware, software, and troubleshooting:

48GB combined VRAM fits 70B models at Q4_K_M (~40GB)
~$1,700 total — less than a single RTX 5090
Proven community setup with thousands of builds documented
Tensor splitting in llama.cpp is well-tested on this configuration

Performance with 70B Q4_K_M: expect ~8-12 tok/s depending on PCIe bandwidth and model. That is usable for interactive chat, though not blazing fast. For a detailed look at exactly how much VRAM each quantization level of a 70B model requires, see how much VRAM for a 70B model.

What you need for dual 3090s

Component	Requirement
Motherboard	ATX with 2x PCIe x16 slots (at least x8 electrical each)
CPU	Any modern CPU with enough PCIe lanes (AMD Ryzen 7/9, Intel i7/i9)
PSU	1000W+ (two 3090s draw ~700W combined under load)
Case	Full tower with good airflow — these cards are thick and hot
RAM	64GB DDR4/DDR5 (model loading requires system RAM)
Slot spacing	Minimum 3-slot gap between cards for thermal headroom

NVLink: do you need it?

NVLink provides a high-speed direct connection between GPUs (up to 112 GB/s on RTX 3090 NVLink bridges). Here is the honest assessment:

For inference: NVLink helps but is not critical. PCIe x16 is the bottleneck only on very large models with many cross-GPU transfers. Typical speedup with NVLink: 10-20% for 70B inference.
For training/fine-tuning: NVLink matters significantly. Gradient synchronization is bandwidth-intensive.
Availability: RTX 3090 supports NVLink bridges (~$80-100 used). RTX 4090 and RTX 5090 do not support consumer NVLink.

If you are only doing inference, skip NVLink and save the money. If you plan to fine-tune on dual 3090s, the NVLink bridge is worth the $80 — and our LLM fine-tuning GPU guide covers the full VRAM math for LoRA and full fine-tuning on multi-GPU setups.

Setting up tensor splitting

llama.cpp / Ollama

In llama.cpp, specify GPU split with the --tensor-split flag:

# Split evenly between two GPUs
./llama-cli -m model.gguf --tensor-split 0.5,0.5 -ngl 99

# Split by VRAM ratio (e.g., 5090 + 3090)
./llama-cli -m model.gguf --tensor-split 0.57,0.43 -ngl 99

Ollama handles splitting automatically when multiple GPUs are detected. No configuration needed.

Mixed GPU setups

You can mix different NVIDIA GPUs for tensor splitting. Common combinations:

RTX 5090 + RTX 3090 (56GB): Uneven split, weight the 5090 heavier for speed
RTX 4090 + RTX 3090 (48GB): Both 24GB, even split works well
RTX 4090 + RTX 4060 Ti 16GB (40GB): Budget expansion of existing 4090

The faster GPU should handle more layers. llama.cpp's --tensor-split ratio lets you tune this. Mixed setups work well for inference but are suboptimal for training.

Multi-GPU vs single large GPU

Factor	Multi-GPU (2x 3090)	Single GPU (RTX 5090)
Total VRAM	48GB	32GB
Cost	~$1,700	~$2,000
Power draw	~700W	~575W
Complexity	Higher	Plug and play
Inference speed (34B)	~20 tok/s	~40 tok/s
Max model	70B Q4	34B Q5 / 70B Q3

For 34B models, a single RTX 5090 is faster and simpler. Multi-GPU only makes sense when you need more VRAM than any single card provides, or when building on a budget with used cards.

Which multi-GPU setup should you buy?

Want 70B models at the lowest cost? Get dual RTX 3090s used ($1,700). The 48GB combined VRAM fits Llama 2 70B at Q4_K_M, and no other setup under $2,000 can do that.
Already own an RTX 4090 and want 70B access? Add a used RTX 3090 as a second card ($850). You get 48GB total for under $1,000 extra investment.
Want maximum speed on 70B? Get dual RTX 4090s ($3,200). The doubled bandwidth over dual 3090s gives you 15-20 tok/s on 70B Q4 versus 8-12 tok/s.
Models fit in 32GB but you want headroom? Skip multi-GPU and get a single RTX 5090. Simpler, less power, faster inference on models that fit.

Common mistakes to avoid

Buying an NVLink bridge for inference-only workloads. NVLink gives only 10-20% speedup for inference. Save the $80-100 unless you plan to fine-tune.
Using a motherboard with x4 electrical on the second PCIe slot. Many consumer boards only provide x4 bandwidth to the second GPU slot, cutting inter-GPU transfer speed by 75%. Verify x8 minimum per slot — our best motherboard for dual-GPU LLM guide lists boards with confirmed x8/x8 bifurcation.
Running dual GPUs on a 750W PSU. Two RTX 3090s draw ~700W under load, leaving zero headroom for CPU, RAM, and fans. A 1000W PSU is the minimum, and 1200W gives you safe margin — see our PSU guide for dual-GPU LLM builds for specific unit recommendations.
Mixing NVIDIA and AMD GPUs. Tensor splitting requires both cards on the same driver stack. Cross-vendor multi-GPU does not work for LLM inference. Multi-GPU setups also behave differently on Windows versus Linux due to PCIe bandwidth and driver handling — our Windows vs Linux for local LLM guide covers what to expect on each platform.

Our recommendation

For most users wanting to run 70B models locally, dual RTX 3090s are the best value in 2026. At ~$1,700, you get 48GB of VRAM, proven software support, and enough speed for interactive inference. Just make sure your PSU and case can handle the heat.

If you want a simpler build and your models fit in 32GB, a single RTX 5090 is the cleaner choice — and within that price tier our best GPU for LLM under $2,000 guide compares the 5090 against alternatives. If you already own an RTX 4090 and want to expand, adding a used RTX 3090 as a second card gives you 48GB total for under $1,000 extra.

Related guides on Best GPU for LLM

Continue on Best GPU for LLM for the complete guide with interactive calculators and current GPU prices.

DEV Community