DEV Community: Thurmon Demich

Best GPU for AI Training at Home in 2026 (Ranked)

Thurmon Demich — Sat, 08 Aug 2026 01:13:24 +0000

From the Best GPU for AI archive. The canonical version has interactive calculators, an up-to-date GPU comparison table, and live pricing.

Training AI at home sounds intimidating, but most practical home workflows are actually fine-tuning rather than training from scratch. LoRA and QLoRA have made it possible to customize foundation models on a single consumer GPU. The question is which GPU fits your use case — and your power bill.

Quick answer: The RTX 4090 (24GB) is the best GPU for AI training at home. It runs LoRA and QLoRA fine-tuning on 7B-13B models comfortably, handles full fine-tuning on smaller models, and has enough VRAM headroom for experimenting. For tighter budgets, the RTX 4060 Ti 16GB is a capable LoRA training card.

Training vs fine-tuning: what are you actually doing?

Most people who say they want to "train AI at home" mean one of these:

Full training from scratch — building a model from random weights on your own dataset. This requires massive compute and is not realistic on consumer hardware for anything beyond toy models.

Full fine-tuning — updating all weights of an existing pretrained model. Requires roughly 2x the model's VRAM in float16. A 7B model needs ~14GB minimum. Possible on 24GB cards, impractical on 16GB.

LoRA fine-tuning — training low-rank adapter layers that attach to a frozen model. Memory-efficient, trains only a small fraction of parameters. A 7B model LoRA trains comfortably on 12-16GB VRAM.

QLoRA fine-tuning — LoRA on a 4-bit quantized model. Enables fine-tuning of 13B-34B models on a single 24GB card. This is the practical sweet spot for home training.

For the full fine-tuning guide, see best GPU for fine-tuning.

VRAM requirements for home training

Task	Minimum VRAM	Recommended
SD LoRA (Kohya, Dreambooth)	8GB	12GB+
7B LLM LoRA (float16)	12GB	16GB
7B LLM QLoRA (4-bit)	8GB	12GB
13B LLM LoRA	24GB	32GB
13B LLM QLoRA (4-bit)	12GB	16GB
34B LLM QLoRA (4-bit)	20GB	24GB
Full fine-tune 7B	16GB	24GB

VRAM chart available at the original article

Best overall: RTX 4090 (24GB)

The RTX 4090's 24GB VRAM is what makes it the home training champion. With QLoRA you can fine-tune models up to 34B parameters on a single card — that covers the vast majority of useful home fine-tuning scenarios. With full LoRA on float16, 13B models fit with a reasonable batch size.

Beyond VRAM: the 4090's 1008 AI TOPS (tensor core throughput) cuts training time significantly versus older Ampere cards. A LoRA run that takes 8 hours on an RTX 3090 may complete in 4-5 hours on the 4090.

For LoRA training specifically, the 4090 enables higher batch sizes and larger training images compared to 16GB cards, which matters for final model quality.

Value option: RTX 4060 Ti 16GB

At $400, the RTX 4060 Ti 16GB is a capable home training card for realistic workloads:

SD LoRA training (Kohya SS, Dreambooth): comfortable
7B LLM LoRA (float16): fits, though batch sizes are small
7B LLM QLoRA: plenty of headroom
13B QLoRA: works with careful configuration

The 16GB version of the 4060 Ti is specifically what you want. The 8GB version is not a training card — it is fine for inference but too constrained for most fine-tuning setups.

Serious setup: RTX 5090 (32GB)

The RTX 5090 opens the door to training scenarios that the 4090 cannot handle:

34B models at QLoRA with comfortable headroom
Full fine-tuning of 13B models
Larger batch sizes on 7B LoRA for higher quality
Multi-modal model fine-tuning (vision + language)

At 1800 AI TOPS (Blackwell), training is also meaningfully faster. If you plan to train models professionally or as a serious project — not just experimenting — the 5090's extra 8GB VRAM and faster compute make a real difference.

Power, cooling, and PSU: what nobody tells you

This is often the part that catches first-time home training setups off guard:

RTX 4090: 450W TDP. Needs a 750W+ PSU (850W recommended). Dual 8-pin connectors or a 16-pin adapter. Generates significant heat in a case — good airflow is essential.

RTX 5090: 575W TDP. Needs a 1000W+ PSU. Even more demanding on cooling.

RTX 4060 Ti: 165W TDP. A 550W PSU is fine. Runs cool, no exotic cooling needed.

Training runs your GPU at maximum utilization for extended periods — unlike gaming, which has variable load. Your cooling setup matters: open-air cases run better than tight ITX builds for long training jobs. Monitor GPU temperatures with nvidia-smi and ensure you stay below 80°C under sustained load.

For multi-GPU setups, see the deep learning GPU guide.

LoRA training workflow: quick setup

Install Kohya SS for Stable Diffusion LoRA, or axolotl for LLM fine-tuning
Prepare your dataset (20-100 images for SD LoRA, 100-1000 examples for LLM LoRA)
Configure training parameters — learning rate, batch size, steps
Monitor VRAM usage during the first few minutes: watch -n 1 nvidia-smi
Adjust batch size down if you get OOM errors

Start with existing guides for your specific model and tool — the parameters tuned for a 7B LoRA are very different from a Stable Diffusion LoRA.

Which GPU should YOU buy?

You want to do SD/SDXL LoRA (Dreambooth, Kohya): RTX 4060 Ti 16GB is plenty. The 4090 is faster but not necessary.
You want to fine-tune 7B LLMs with LoRA: RTX 4060 Ti 16GB handles this well. The 4090 gives you larger batch sizes and faster training.
You want to fine-tune 13B+ models: RTX 4090 (24GB). Non-negotiable for comfortable QLoRA at this size.
You want 34B model fine-tuning on a single card: RTX 4090 for QLoRA, RTX 5090 for more headroom.
You want to train but hate waiting: RTX 5090. Blackwell's training throughput is substantially faster than Ada Lovelace.
You want to experiment before committing to hardware: Use cloud GPU time first (RunPod, Vast.ai) to validate your setup and dataset quality. If you are weighing cloud versus local long-term, our cloud GPU vs home GPU for AI guide breaks down when renting makes sense and when buying wins.

Common mistakes to avoid

Starting with too large a model. Fine-tune a 7B model first. Learn the tooling, validate your dataset quality, and only scale up once you know what you are doing. A 34B model with bad training data is worse than a well-tuned 7B.
Ignoring PSU requirements. An RTX 4090 on a 650W PSU will throttle or crash mid-training. Check your total system power draw before buying a high-end card.
Choosing batch size 1 to fit VRAM. Gradient accumulation lets you simulate larger batch sizes while keeping VRAM use constant. Use it — batch size 1 often leads to unstable training.
Not validating the dataset before a long run. Run a quick 100-step test first. A 10-hour training run with corrupted or poorly formatted data is wasted time.
Underestimating cooling. Training loads are more sustained than gaming. Check temperatures, ensure case airflow, and consider a case with good GPU clearance.

Final verdict

Scenario	Best GPU
SD/SDXL LoRA, 7B LLM LoRA	RTX 4060 Ti 16GB
7B full LoRA, 13B QLoRA	RTX 4090
34B QLoRA, fastest training	RTX 5090
Experimenting before buying	Cloud GPU (RunPod/Vast.ai)

Home AI training is more accessible than it has ever been. QLoRA has democratized fine-tuning to a point where an RTX 4090 can do in a weekend what required a multi-GPU server a few years ago. Start small, validate your dataset, and scale the hardware to match the ambition.

Frequently Asked Questions

Can you train AI models on a consumer GPU?

Yes. With LoRA and QLoRA fine-tuning techniques, you can customize 7B–34B parameter models on a single consumer GPU. Full training from scratch is not realistic on consumer hardware, but fine-tuning — which adapts a pretrained model to your data — works well on cards like the RTX 4090 (24GB) and even the RTX 4060 Ti (16GB) for smaller models. The key is choosing the right fine-tuning method for your VRAM budget.

How much VRAM do you need for fine-tuning?

It depends on the method and model size. QLoRA fine-tuning of a 7B model needs just 8–12GB VRAM. Full LoRA on a 7B model requires 12–16GB. For 13B models, QLoRA needs 12–16GB and full LoRA needs 24GB. Full fine-tuning (updating all weights) of a 7B model requires 16–24GB VRAM. The general rule is that QLoRA roughly halves the VRAM requirement compared to full LoRA by quantizing the base model to 4-bit.

Is LoRA training possible on 8GB VRAM?

Yes, but only for smaller models and with constraints. QLoRA on a 7B model fits on 8GB VRAM with careful configuration, and SD/SDXL LoRA training with Kohya SS is workable at 8GB. However, batch sizes will be very small (often batch size 1), which can lead to less stable training and lower quality results. 12–16GB is strongly recommended for comfortable LoRA training with reasonable batch sizes.

How long does fine-tuning take on a home GPU?

A typical LoRA fine-tune of a 7B LLM on 1,000 examples takes roughly 2–4 hours on an RTX 4090 and 6–10 hours on an RTX 4060 Ti. Stable Diffusion LoRA training on 50–100 images usually completes in 1–3 hours on a 4090. Training time scales roughly linearly with dataset size and inversely with GPU compute speed. Always run a short 100-step test first to validate your setup before committing to a full training run.

Related guides on Best GPU for AI

The full version lives on Best GPU for AI — VRAM calculator, GPU comparison table, and live Amazon pricing.

How to Run a 70B LLM on a Single GPU in 2026 (Q3-Q4)

Thurmon Demich — Fri, 07 Aug 2026 01:13:27 +0000

This article was originally published on Best GPU for LLM. The full version with interactive tools, FAQ, and live pricing is on the original site.

Yes -- you can run a 70B model on a single GPU, but only with aggressive quantization. At Q2_K, Llama 3 70B compresses to roughly 25GB, which fits on the RTX 5090's 32GB with room for a short context window. The trade-off is measurable quality loss compared to Q4_K_M. Whether that trade-off is acceptable depends entirely on your use case.

Who this is for

You want to run 70B-parameter models like Llama 3 70B, Qwen 2.5 72B, or DeepSeek-R1 70B on a single consumer GPU. You are willing to accept quantization trade-offs to avoid multi-GPU setups or cloud costs.

The VRAM math for 70B models

Quantization	Model Size	KV Cache (4K ctx)	Total VRAM	Fits on RTX 5090 (32GB)?
Q2_K	~25GB	~1.5GB	~27GB	Yes
Q3_K_S	~29GB	~1.5GB	~31GB	Barely (short context)
Q3_K_M	~32GB	~1.5GB	~34GB	No
Q4_K_M	~40GB	~1.5GB	~42GB	No
Q8_0	~70GB	~1.5GB	~72GB	No
FP16	~140GB	~1.5GB	~142GB	No

The only consumer GPU that fits a 70B model at any quantization is the RTX 5090 with 32GB. At Q2_K, you get a tight but workable fit. At Q3_K_S, you are at the absolute limit with minimal context window.

VRAM chart available at the original article

Play with the quant setting below and watch which cards drop out:

Quality impact of aggressive quantization

This is the part most guides skip. Dropping from Q4_K_M to Q2_K is not free:

Perplexity increases 5-15% depending on the model. You will notice more factual errors and less coherent long-form output.
Reasoning quality drops -- multi-step logic tasks suffer the most. Simple Q&A and summarization hold up better.
Code generation degrades -- syntax accuracy drops noticeably at Q2. Use a 32B model at Q4 instead if code is your primary use case.
Creative writing is less affected -- style and fluency remain mostly intact at Q2_K.

For tasks where 70B quality matters (complex reasoning, nuanced analysis), the quality loss at Q2_K may defeat the purpose of running a 70B model in the first place. A 32B model at Q4_K_M often produces better results than a 70B at Q2_K.

GPU options for single-card 70B

GPU	VRAM	Best 70B Quantization	Speed	Verdict
RTX 5090	32GB	Q2_K (~25GB)	~12 tok/s	Only single-GPU option
RTX 4090	24GB	Won't fit	--	Too little VRAM
RTX 3090	24GB	Won't fit	--	Too little VRAM

The RTX 5090 is the only consumer card that can run 70B models without offloading. Every other card requires either multi-GPU or CPU offloading (which tanks speed to 2-5 tok/s).

Partial offloading: the middle ground

If you own an RTX 4090 (24GB) and want to try 70B models, partial CPU offloading is possible through llama.cpp:

Load 60-70% of layers on GPU, remaining on CPU RAM
Expect 5-8 tok/s with a fast CPU and DDR5 RAM
Requires 32GB+ system RAM alongside GPU VRAM
Usable for batch processing, painfully slow for interactive chat

This is a compromise, not a solution. If you need 70B regularly, either get an RTX 5090 or use dual GPUs.

The dual-GPU alternative

Two RTX 4090s (48GB combined) run 70B at Q4_K_M with room for 8K context. This costs $3,200 but gives you significantly better quality than a single RTX 5090 at Q2_K. If 70B model quality is your priority, dual cards are the better path.

Which approach should you take?

If you want 70B on a single card and accept Q2_K quality, the RTX 5090 ($2,000) is your only option. If you want 70B at Q4_K_M quality, you need dual RTX 4090s ($3,200) or cloud GPUs. If you want the best single-GPU experience without quality compromise, run a 32B model at Q4_K_M on an RTX 4090 ($1,600) instead -- it often beats a degraded 70B in practical tasks.

Common mistakes to avoid

Assuming Q2_K quality is "good enough" without testing. Run your specific tasks at Q2_K and Q4_K_M side-by-side before committing. The quality gap is task-dependent.
Using CPU offloading for interactive chat. At 5-8 tok/s, you will wait 10+ seconds for short responses. Offloading is only viable for batch workloads.
Buying an RTX 5090 specifically for 70B when 32B covers your needs. The 32B class of models (DeepSeek-R1 32B, Qwen 2.5 32B) handles most tasks well at Q4_K_M on an RTX 4090 for $400 less.
Forgetting context length limits. At Q2_K on a 32GB card, you have roughly 5GB for KV cache. That limits you to 2K-4K context, which may not be enough for complex multi-turn conversations.

Our recommendation

Running 70B on a single GPU is technically possible but comes with real trade-offs. For most users, a 32B model at Q4_K_M on an RTX 4090 delivers better practical results than a 70B at Q2_K on an RTX 5090. If you specifically need 70B quality and want to stay on one card, the RTX 5090 is the only path -- just go in with realistic expectations about quantization impact.

A 32B model at good quantization almost always beats a 70B model at bad quantization. Size is not everything.

For dedicated 70B hardware planning, see our best GPU for Llama 70B guide. For the full RTX 5090 vs 4090 breakdown, check our 5090 vs 4090 comparison.

Frequently asked questions

Is a 32GB GPU enough to run a 70B LLM without quantization?

No. Without quantization, a 70B model in FP16 needs roughly 140GB of VRAM, and even near-lossless Q8 still needs around 70GB. A 32GB card like the RTX 5090 only fits a 70B model at aggressive Q2_K or Q3_K_S quantization, which comes with measurable quality loss. Running 70B unquantized requires datacenter-class hardware, not a consumer GPU.

How do I run a 70B LLM locally?

You have three realistic paths: a single RTX 5090 running Q2_K quantization with a short context window, dual GPUs such as two RTX 4090s (48GB combined) for Q4_K_M quality, or partial CPU offloading on a 24GB card, which drops speed to roughly 5-8 tok/s and only suits batch work. Tools like llama.cpp handle the layer splitting and offloading.

How much VRAM does a 70B model need at Q4 quantization?

At Q4_K_M, the model weights alone are roughly 40GB, and with KV cache overhead you need about 42GB of total VRAM — more than any single consumer card offers. That is why Q4-quality 70B setups use dual GPUs (two RTX 4090s give 48GB combined) or cloud instances. On a single 32GB card you are limited to Q2_K or Q3_K_S.

Is Q2_K quality good enough for a 70B model?

It depends on the task. Perplexity rises roughly 5-15% versus Q4_K_M, and multi-step reasoning and code generation suffer the most, while creative writing and simple Q&A hold up better. For complex reasoning work, a 32B model at Q4_K_M often produces better results than a 70B at Q2_K, so test your own workload side-by-side before committing.

Related guides on Best GPU for LLM

Continue on Best GPU for LLM for the complete guide with interactive calculators and current GPU prices.

ROCm vs CUDA for AI in 2026: Should AMD Be on Shortlist?

Thurmon Demich — Thu, 06 Aug 2026 01:13:43 +0000

From the Best GPU for AI archive. The canonical version has interactive calculators, an up-to-date GPU comparison table, and live pricing.

Most ROCm vs CUDA comparisons on the internet are written for data center buyers weighing MI300X against H100. This article is about something different: whether the RX 7900 XTX, RX 7800 XT, or RX 7700 XT belong on a consumer AI build shortlist in 2026.

The honest answer is nuanced. For specific workflows on Linux, AMD consumer GPUs are genuinely viable. For Windows users, training-heavy workflows, or anyone who needs TensorRT, NVIDIA is still the safer choice — and increasingly the clearer one.

Quick answer: AMD consumer GPUs work for Stable Diffusion and Ollama inference on Linux. They lag NVIDIA by a community-reported 15–25% at equivalent price points for most AI tasks, with meaningful gaps in specialized library support. For Stable Diffusion on a budget with a Linux system, the RX 7900 XTX (24GB) is worth considering. For everything else, CUDA is still the pragmatic default.

Why this article isn't about the MI300X

Enterprise ROCm content focuses on AMD's MI300X accelerator because that's where the interesting competitive story is — 192GB HBM3 at a fraction of H100 pricing for large-scale inference. That context doesn't help if you're buying a desktop GPU for local AI work.

Consumer AMD GPUs use the same ROCm software stack, but they're a different target: RX 7000-series discrete cards with 12–24GB GDDR6, running on a desktop machine next to your keyboard. The questions are different: Does PyTorch install cleanly? Does ComfyUI work? Can I run Ollama? Does it require Linux or does Windows work?

These are the questions this article addresses.

Current state of consumer ROCm in 2026

ROCm has improved substantially since its early consumer-hostile period. Key milestones that matter for desktop AI use:

PyTorch ROCm has been stable since PyTorch 2.0. Installation via pip with the ROCm wheel is straightforward on Linux. Training and inference both work for most standard use cases.
RX 7000 series (RDNA 3) is officially supported. RX 7900 XTX, RX 7900 XT, RX 7800 XT, and RX 7700 XT are all in ROCm's supported device list.
RX 9000 series (RDNA 4) support is emerging. Expect RX 9070 XT coverage to mature through 2026.

What hasn't fully caught up: specialized libraries that NVIDIA has had years to optimize for CUDA — TensorRT, flash attention (AMD has its own implementation but with inconsistent tool support), some quantization libraries, and triton kernels. These gaps matter more for training than inference.

The bigger limiting factor for most consumer users isn't ROCm itself — it's Windows.

Windows vs Linux: matters doubly for AMD users

On NVIDIA hardware, the Windows/Linux performance gap for AI workloads is modest. PyTorch and CUDA work well on both platforms with similar results.

On AMD, this gap is significant:

Linux + ROCm: Full PyTorch support, Ollama native, ComfyUI via ROCm, llama.cpp with HIP support. This is the mature, recommended path.
Windows: No native ROCm for consumer GPUs. Options are DirectML (via ONNX Runtime or DirectML-patched ComfyUI) and llama.cpp Vulkan/DirectML backends. Performance is noticeably lower than Linux ROCm, and not all tools support it.

If you're building an AMD AI rig and you plan to run Windows, you're accepting a significant capability and performance penalty versus the same GPU on Linux. This isn't a theoretical concern — community comparisons consistently show AMD Windows DirectML setups running 20–40% slower than Linux ROCm for comparable workloads. For a broader look at how the operating system choice affects AI performance across both NVIDIA and AMD hardware, see our Windows vs Linux for AI guide.

For comparison, an NVIDIA card runs CUDA on both platforms with equivalent performance.

Tool-by-tool reality check

Stable Diffusion (ComfyUI, A1111):
Works on Linux via ROCm. ComfyUI with ROCm is the recommended path — community reports show it functional with SDXL, Flux, and ControlNet workflows. A1111 ROCm support exists but is less maintained. On Windows, DirectML-patched ComfyUI works at reduced performance. See best GPU for stable diffusion for full VRAM requirements.

Ollama:
Works natively on Linux with ROCm. RX 7000 series is recognized automatically in recent Ollama versions. Performance is solid for inference. Windows support via Ollama is more limited for AMD — ROCm isn't supported on Windows via Ollama as of early 2026.

llama.cpp:
Best AMD support of any major framework. The HIP backend (for Linux ROCm) and Vulkan backend (cross-platform including Windows) both work well. llama.cpp is the recommended inference backend for AMD Windows users.

Kohya_ss (LoRA/DreamBooth training):
Spotty. ROCm PyTorch training works in principle, but Kohya's xformers dependency and some attention implementations have known AMD compatibility issues. Functional with workarounds on Linux; more painful on Windows. Not recommended as a primary AMD use case.

vLLM:
Linux only for AMD, and requires more manual setup than the NVIDIA path. If vLLM is a core part of your workflow, NVIDIA is significantly smoother.

TensorRT:
NVIDIA-exclusive. Any workflow that depends on TensorRT for deployment optimization is incompatible with AMD consumer hardware.

Performance gap: honest numbers

Community benchmarks across Reddit, GitHub issues, and comparative posts consistently put RX 7000-series consumer AMD cards 15–25% behind NVIDIA at equivalent price points for AI inference workloads. The gap varies by task:

Inference (Ollama, llama.cpp): Closer to 10–15% gap. AMD holds up reasonably well here.
Stable Diffusion generation: Closer to 15–20% gap at equivalent VRAM capacity, partly due to bandwidth differences.
Training (PyTorch): Gap widens to 20–30%+ for many training workloads. CUDA's mature ecosystem of optimized kernels (flash attention, fused ops, cuDNN) accumulates advantage.

The gap is narrowing with each ROCm release, but it hasn't closed. AMD's trajectory is positive; the question is whether it's closed enough today for your specific use case.

The RX 7900 XTX's 24GB VRAM at its current street price is the most compelling AMD value argument — it offers 24GB for less than an RTX 4090, and for inference-heavy use cases on Linux, the VRAM advantage can outweigh the compute gap. For VRAM-per-dollar comparisons, see nvidia vs amd for ai.

When AMD consumer makes sense

Linux-first users: If you run Linux as your primary OS for AI work, AMD's ROCm path is viable and the friction is manageable. PyTorch installs cleanly, Ollama works, ComfyUI runs.

Stable Diffusion focus: SD workflows on Linux are the best-supported AMD AI use case. If Stable Diffusion generation is your primary workload, AMD is a legitimate option — the gap to NVIDIA narrows considerably for inference compared to training.

Cost-sensitive builds with VRAM priority: The RX 7900 XTX at 24GB is the strongest consumer AMD value argument. If 24GB VRAM matters more to your workflow than raw compute speed, and you're on Linux, this card is worth evaluating seriously.

llama.cpp inference on any platform: llama.cpp's HIP and Vulkan backends give AMD the widest cross-platform coverage of any framework. If llama.cpp is your inference runtime, AMD's cross-platform story is better than anywhere else.

When NVIDIA is still the safe pick

Windows users: CUDA works natively, DirectML is the only AMD option and it's slower and less capable. NVIDIA on Windows requires no special setup.
Training-heavy workflows (LoRA, fine-tuning, DreamBooth): CUDA's mature ecosystem of optimized training kernels gives NVIDIA a 20–30% advantage on many training tasks. The gap is largest here.
TensorRT requirements: TensorRT is NVIDIA-exclusive. If your deployment pipeline uses TensorRT, AMD is off the table.
vLLM deployments: vLLM's NVIDIA path is more mature, better documented, and easier to set up. AMD support exists but requires more work.
Cutting-edge model support: New model architectures and quantization methods frequently land on CUDA first, with AMD support following weeks or months later.
Workflow certainty: If you're not sure exactly what AI tools you'll run, CUDA is the safe default. Every tool works.

Specific card picks for AMD AI builds

RX 7900 XTX (24GB GDDR6): The headline AMD consumer AI card. 24GB VRAM at less than RTX 4090 pricing is a real value proposition for VRAM-hungry workloads — large model inference, Flux Dev with full pipelines, multi-LoRA Stable Diffusion. Linux required for full ROCm support. For inference workloads where VRAM is the bottleneck, this card can outperform lower-VRAM NVIDIA options despite the compute gap.

RX 7800 XT (16GB GDDR6): The budget AMD AI option. 16GB at competitive pricing for Stable Diffusion and Ollama inference on Linux. Not a strong training card, but solid for the inference use cases AMD handles well. Worth considering if 16GB is your target budget and you're on Linux. Our RX 7800 XT AI compatibility deep-dive covers exactly which tools work and where ROCm still trips.

RX 7700 XT (12GB GDDR6): Functional but marginal. 12GB covers SD 1.5 and basic SDXL, but you'll hit walls on Flux Dev and complex ComfyUI workflows. At this price point, comparing against RTX 3060 12GB or RTX 4060 on the NVIDIA side is worthwhile — the CUDA ecosystem advantage matters more at the budget tier.

The trajectory matters

ROCm is meaningfully better in 2026 than it was in 2023. The trend is positive. AMD is investing in consumer ROCm support, and major frameworks are improving AMD compatibility. The RX 9000 series (RDNA 4) launching in 2026 will have better native support by the time most buyers are purchasing.

The honest assessment is that AMD consumer AI in 2026 is viable for specific use cases on Linux, not yet recommended as a general-purpose CUDA replacement for mixed workloads. If your use case is on the supported list and you're on Linux, AMD deserves a slot on your shortlist. If you're Windows-first or running mixed training/inference pipelines, NVIDIA remains the pragmatic default.

Related guides on Best GPU for AI

Continue on Best GPU for AI for the complete guide with interactive calculators and current GPU prices.

Local LLM VRAM 2026: The 12GB Trap Most Buyers Hit

Thurmon Demich — Wed, 05 Aug 2026 01:13:30 +0000

Cross-posted from Best GPU for LLM — visit the original for our VRAM calculator, GPU comparison table, and current Amazon pricing.

Quick answer: For most local LLM users, 16GB VRAM is the sweet spot. It handles all 7B models and most 13B models quantized. If you want to run 34B+ models, you need 24GB or more.

VRAM chart available at the original article

The VRAM rule of thumb

A rough formula for estimating VRAM needs:

VRAM needed = (Parameters in billions x Bytes per parameter) + 2GB overhead

At FP16 (2 bytes per param): a 7B model needs ~16GB. But with quantization, you can cut that dramatically.

You can sanity-check any model-and-quant combo right here:

VRAM requirements by model size

7B parameter models (Llama 3 8B, Mistral 7B, Gemma 7B)

Quantization	Model Size	VRAM Needed	Minimum GPU
FP16	~14GB	16GB	RTX 4060 Ti 16GB
Q8	~7.5GB	10GB	RTX 3060 12GB
Q6_K	~5.5GB	8GB	RTX 4060 8GB
Q4_K_M	~4.5GB	6GB	RTX 3060 12GB
Q3_K_M	~3.5GB	6GB	Any 6GB GPU

Verdict: Any modern GPU with 8GB+ VRAM handles 7B models comfortably. 16GB gives you room for full precision and longer contexts. For Llama 3 8B specifically, see how much VRAM Llama 3 8B actually uses across all quantization levels.

13B parameter models (Llama 2 13B, CodeLlama 13B)

Quantization	Model Size	VRAM Needed	Minimum GPU
FP16	~26GB	28GB	RTX 5090 32GB
Q8	~13GB	16GB	RTX 4060 Ti 16GB
Q6_K	~10GB	12GB	RTX 3060 12GB
Q4_K_M	~7.5GB	10GB	RTX 3060 12GB
Q3_K_M	~6GB	8GB	RTX 4060 8GB

Verdict: 12-16GB VRAM is ideal for 13B models. The RTX 4060 Ti 16GB or RTX 4070 handles these well.

34B parameter models (CodeLlama 34B, Yi 34B)

Quantization	Model Size	VRAM Needed	Minimum GPU
FP16	~68GB	70GB+	Multi-GPU required
Q8	~34GB	36GB+	A6000 48GB
Q6_K	~26GB	28GB	RTX 5090 32GB
Q4_K_M	~20GB	22GB	RTX 4090 24GB
Q3_K_M	~15GB	18GB	RTX 4070 Ti Super 16GB (tight)

Verdict: 24GB is the minimum for comfortable 34B use. The RTX 4090 is the go-to card for this tier. For a detailed test of whether the 12GB RTX 5070 can handle 34B at all, see can the RTX 5070 run 34B?

70B parameter models (Llama 3 70B, Qwen 72B)

Quantization	Model Size	VRAM Needed	Minimum GPU
Q8	~70GB	72GB+	Multi-GPU required
Q6_K	~54GB	56GB+	Multi-GPU required
Q4_K_M	~40GB	42GB+	2x RTX 4090 or A6000
Q3_K_M	~32GB	34GB	RTX 5090 (tight)
Q2_K	~25GB	28GB	RTX 5090 32GB

Verdict: 70B models are not practical on a single consumer GPU at good quality. Plan for dual GPUs, a workstation card, or cloud. For a specific look at what a 16GB mid-range card can do, see can the RTX 4060 Ti run Llama 70B? — and for the complete VRAM breakdown at every quantization level, see how much VRAM for a 70B model. For 1M-context MoE models like MiniMax M3 specifically, KV cache dominates the VRAM math — see our best GPU for MiniMax M3 breakdown.

What is quantization?

Quantization reduces the precision of model weights to use less memory. The trade-off is quality:

Level	Bits	Quality	Use Case
FP16	16-bit	Best	If VRAM allows
Q8	8-bit	Near-perfect	Default if you have room
Q6_K	6-bit	Excellent	Best quality/size balance
Q5_K_M	5-bit	Very good	Slight degradation
Q4_K_M	4-bit	Good	Most popular for daily use
Q3_K_M	3-bit	Acceptable	Noticeable on complex reasoning
Q2_K	2-bit	Poor	Emergency only

The sweet spot is Q4_K_M to Q6_K. Below Q4, quality degrades noticeably. Above Q6, the quality gains are minimal for most use cases. For a deeper dive into how quantization choices affect real-world output, see our best quantization for local LLM guide.

Context length eats VRAM too

VRAM usage isn't just the model. The KV cache for context grows with conversation length:

Context Length	Additional VRAM (7B)	Additional VRAM (13B)
2,048 tokens	~0.5GB	~1GB
4,096 tokens	~1GB	~2GB
8,192 tokens	~2GB	~4GB
16,384 tokens	~4GB	~8GB
32,768 tokens	~8GB	~16GB

This is why a model that "fits" in VRAM can crash during a long conversation. Always leave 2-4GB of headroom beyond the base model size.

GPU recommendations by budget

VRAM Target	Best GPU	Price	Handles
8GB	RTX 4060	~$300	7B quantized
12GB	RTX 3060 12GB (used)	~$250	7B full, 13B quantized
16GB	RTX 4060 Ti 16GB	~$400	7B-13B comfortably
24GB	RTX 4090	~$1,600	Up to 34B quantized
32GB	RTX 5090	~$2,000+	34B comfortable, 70B tight
48GB	2x RTX 4090 or A6000	~$3,200+	70B at good quality

Which GPU should YOU buy?

Running 7B models only? Get an RTX 3060 12GB ($250 used) or RTX 4060 Ti 16GB ($400) for more headroom.
Running 13B-34B models? Get an RTX 4090 ($1,600). Nothing else in consumer space gives you 34B at usable quantization on a single card.
Future-proofing for 34B+ and experimental 70B? Get an RTX 5090 ($2,000). The only single consumer GPU that handles it.
Need 70B at good quality? Get 2x RTX 4090 ($3,200) or try cloud GPUs for occasional use.

Common mistakes

"I'll just use CPU offloading" — Offloading layers to RAM works, but inference speed drops 5-10x for offloaded layers. A model that's 50% on GPU and 50% offloaded is painfully slow. If your primary use case is local LLM summarization, long-document context is your binding constraint — plan VRAM for the KV cache first, not just the model weights.

"8GB is enough for everything" — It's enough for 7B quantized. That's it. If you plan to explore different models, get at least 12-16GB.

"I'll buy the cheapest card with enough VRAM" — Memory bandwidth matters too. An old card with 16GB VRAM but low bandwidth will have much slower inference than a newer card with the same VRAM.

"More VRAM is always better" — True, but only up to what you need. Don't buy a $3,500 A6000 if you only run 7B models. Wondering if a Mac Mini's unified memory counts the same way? See our can the Mac Mini run LLMs guide for how Apple's shared memory pool changes the VRAM math. For model-family specific breakdowns, see how much VRAM Qwen 3 needs if Qwen is your target, or how much VRAM Gemma 4 needs for Google's latest.

Final verdict

You want to run...	Get this GPU
7B models (Mistral, Llama 8B)	RTX 4060 Ti 16GB ($400)
13B models	RTX 4070 Ti Super 16GB ($700)
34B models	RTX 4090 24GB ($1,600)
70B models	2x RTX 4090 or cloud GPU

16GB is the new minimum for anyone serious about local LLM inference. The landscape has shifted — 8GB cards are increasingly limiting as models grow.

Buy for the model size you want to run, not the model size you're running today. VRAM is the one thing you can't upgrade later.

Frequently Asked Questions

What LLMs can I run with 8GB VRAM?

With 8GB VRAM, you can run all 7B parameter models (Llama 3 8B, Mistral 7B, Gemma 7B) at Q4_K_M to Q6_K quantization. You can also fit some 7B models at Q8 with tight headroom. 13B models will not fit at any usable quantization level. Budget 2GB for context window overhead, so plan for models under 6GB in size.

What LLMs can I run with 16GB VRAM?

16GB VRAM handles all 7B models at full FP16 precision and all 13B models at Q4_K_M to Q8 quantization. Popular models like CodeLlama 13B, Qwen 14B, and Phi-3 Medium 14B all fit comfortably. You can also run 34B models at Q3_K_M in a pinch, though quality degrades noticeably at that quantization level. 16GB is the sweet spot for most local LLM users in 2026.

How much VRAM does Llama 4 need?

Llama 4 Scout (17B active parameters, 109B total in a mixture-of-experts architecture) is still early in community testing, but early reports suggest you should expect to need roughly 12-20GB VRAM at Q4 quantization depending on context length and runtime. Llama 4 Maverick (17B active, 400B total) requires significantly more — likely in the 40-50GB range at Q4, meaning dual RTX 4090s or a workstation-class GPU. These figures may shift as quantization support matures.

Does quantization reduce VRAM usage?

Yes, dramatically. Quantization compresses model weights from 16-bit (FP16) down to 4-bit or lower, reducing VRAM usage by 2-4x. A 7B model uses about 14GB at FP16 but only 4.5GB at Q4_K_M. The quality trade-off is minimal between Q4_K_M and Q8 for most use cases — below Q4, quality degrades noticeably on complex reasoning tasks.

Related guides on Best GPU for LLM

Read the full guide on Best GPU for LLM — includes our VRAM calculator, GPU comparison table, and live pricing.

Best GPU for PyTorch in 2026: RTX 4090 + 6 Backups

Thurmon Demich — Tue, 04 Aug 2026 01:13:35 +0000

From the Best GPU for AI archive. The canonical version has interactive calculators, an up-to-date GPU comparison table, and live pricing.

Quick answer: The RTX 4090 is the best GPU for PyTorch in 2026 for most users. It has 24GB VRAM, excellent CUDA support (compute capability 8.9), and handles training, fine-tuning, and inference at consumer-grade prices.

Why GPU choice matters for PyTorch

PyTorch runs on CUDA, and not all CUDA GPUs are equal. Three things determine your PyTorch experience:

VRAM — limits model size, batch size, and dataset loading
CUDA compute capability — determines which features and optimizations are available
Tensor core generation — newer tensor cores support FP8, BF16, and mixed precision more efficiently

For a spec-level look at the two flagship training cards, run the quick comparison below:

CUDA compute capability by GPU

Compute capability determines which PyTorch features you can use. Higher is better.

GPU	Compute Capability	Architecture	Key PyTorch Features
RTX 5090	10.0	Blackwell	FP8, FP16, BF16, INT8, flash attention
RTX 5080/5070 Ti/5070	10.0	Blackwell	FP8, FP16, BF16, INT8, flash attention
RTX 4090	8.9	Ada Lovelace	FP8, FP16, BF16, INT8, flash attention
RTX 4070 Ti Super	8.9	Ada Lovelace	FP8, FP16, BF16, INT8, flash attention
RTX 4060 Ti 16GB	8.9	Ada Lovelace	FP8, FP16, BF16, INT8, flash attention
RTX 3090	8.6	Ampere	FP16, BF16, INT8, TF32
RTX 3060 12GB	8.6	Ampere	FP16, BF16, INT8, TF32
RX 7800 XT	N/A (ROCm)	RDNA 3	Limited — ROCm only

Minimum for modern PyTorch: Compute capability 7.0+ (Volta or newer). Anything below 7.0 lacks tensor cores and mixed precision support.

Best GPUs for PyTorch ranked

GPU	VRAM	Training Speed (ResNet-50)	PyTorch Fine-tune (7B LoRA)	Price
RTX 5090	32GB	~1,850 img/s	~45 tokens/s	~$2,000+
RTX 4090	24GB	~1,400 img/s	~32 tokens/s	~$1,600
RTX 5080	16GB	~950 img/s	~24 tokens/s	~$1,000
RTX 4070 Ti Super	16GB	~750 img/s	~18 tokens/s	~$700
RTX 4060 Ti 16GB	16GB	~450 img/s	~11 tokens/s	~$400
RTX 3090	24GB	~1,050 img/s	~22 tokens/s	~$700 used

RTX 4090 — best overall for PyTorch

The RTX 4090 remains the best single GPU for PyTorch development:

24GB VRAM fits most research-scale models and datasets
Compute capability 8.9 supports every current PyTorch optimization
torch.compile() runs fastest on Ada Lovelace architecture
Excellent mixed precision training with BF16 and FP8
Massive CUDA core count (16,384) accelerates both training and inference
Widely used in research — most PyTorch tutorials and benchmarks target this card

RTX 5090 — for large-scale training

If your PyTorch workloads consistently hit 24GB VRAM limits:

32GB VRAM enables larger batch sizes and model sizes
Blackwell architecture brings improved FP8 training support
~30-40% faster training than the RTX 4090
Worth the premium if you're training (not just fine-tuning)

Budget picks for PyTorch

RTX 4060 Ti 16GB (~$400): The cheapest way to get 16GB VRAM with full CUDA 8.9 support. Handles LoRA fine-tuning and small model training. Slower than flagships but functional for learning and prototyping.

RTX 3090 used (~$700): 24GB VRAM at half the price of a new RTX 4090. Older compute capability (8.6) but still fully supported by PyTorch. Best value for VRAM-heavy workloads. See our used GPU buying guide for tips, and our RTX 3090 vs 4090 for AI head-to-head if you're weighing the upgrade.

PyTorch-specific optimizations

Maximize your GPU regardless of which card you buy:

Use torch.compile() — automatic kernel fusion, 20-50% speedup on Ada/Blackwell
Enable BF16 mixed precision — torch.autocast('cuda', dtype=torch.bfloat16)
Use Flash Attention — built into PyTorch 2.x via F.scaled_dot_product_attention
Set pin_memory=True in DataLoaders for faster CPU-to-GPU transfer
Use gradient checkpointing to trade compute for VRAM on large models

What about AMD for PyTorch?

PyTorch supports AMD GPUs through ROCm, but the experience is rougher:

Fewer pre-built wheels — you may need to compile from source
Some operations fall back to CPU (slower)
Community support and debugging resources are NVIDIA-focused
ROCm version compatibility can be finicky

If PyTorch is your primary framework, stick with NVIDIA. Apple Silicon's MPS backend works for inference and small experiments, but PyTorch training on macOS still trails CUDA significantly — see our Mac vs NVIDIA for AI breakdown if you're considering an M-series Mac as a PyTorch box. Your OS also matters — PyTorch on Linux generally outperforms Windows for training workloads, and the gap widens on AMD hardware. See our Windows vs Linux for AI guide for the full breakdown. For users working with the Hugging Face ecosystem — Transformers, Diffusers, and PEFT — see our best GPU for Hugging Face guide for library-specific recommendations.

Which GPU should you buy?

You are learning PyTorch or running tutorials: The RTX 4060 Ti 16GB at $400 handles every beginner workload with full CUDA 8.9 support. Do not overspend while learning.
You train custom models or fine-tune 7B+ LLMs: The RTX 4090 at 24GB is the default choice. Enough VRAM for serious work, fast enough to keep training loops productive.
You need maximum VRAM for large models or big batch sizes: The RTX 5090 at 32GB is the only consumer option. Worth it if you consistently OOM on 24GB.
You mostly run inference and prototyping, not heavy training: A 16GB card like the RTX 5070 Ti gives you enough VRAM without the flagship price. If you specifically run a local LLM as an always-on assistant, our best GPU for AI assistant guide covers inference-tuned picks.

Common mistakes to avoid

Choosing AMD GPUs to save money on PyTorch workloads. ROCm support is improving but still lags CUDA in stability, pre-built wheels, and community resources. The debugging time you spend is not worth the hardware savings.
Not using torch.compile() on Ada Lovelace or Blackwell GPUs. This single line can give you a 20-50% speedup through automatic kernel fusion. It is free performance you are leaving on the table.
Running FP32 training by default. Always enable BF16 mixed precision with torch.autocast. It halves VRAM usage and often trains faster with no accuracy loss on modern architectures.
Ignoring pin_memory=True in DataLoaders. This small setting speeds up CPU-to-GPU data transfer significantly, especially with large datasets. It costs nothing and helps every training run.

Our recommendation

GPU tier list available at the original article

For most PyTorch users: buy the RTX 4090. It gives you enough VRAM for serious work, top-tier compute capability, and the best ecosystem support. If budget is tight, the RTX 4060 Ti 16GB gets you into real PyTorch work for a quarter of the price.

Check our best GPU for AI guide for a broader comparison across all AI workloads. If your primary framework is TensorFlow rather than PyTorch, see our best GPU for TensorFlow guide for XLA and production-deployment considerations.

The best GPU for PyTorch is the one with enough VRAM for your model and enough compute to keep your training loop from becoming a waiting game.

Frequently Asked Questions

What CUDA version should I use with PyTorch in 2026?

For RTX 30-series (Ampere) GPUs, use CUDA 12.4 or later with PyTorch 2.5+. For RTX 40-series (Ada Lovelace) and RTX 50-series (Blackwell), CUDA 12.4+ is recommended to unlock FP8 and BF16 tensor core optimizations. Always install PyTorch via the official pip command from pytorch.org, which bundles the correct CUDA runtime — manually installing CUDA separately is unnecessary and a common source of version mismatch issues.

Does the RTX 3090 support bfloat16 in PyTorch?

Yes. The RTX 3090 (Ampere, compute capability 8.6) supports BF16 through its tensor cores. You can use torch.autocast('cuda', dtype=torch.bfloat16) for mixed-precision training. However, BF16 throughput on Ampere is lower than on Ada Lovelace (RTX 40-series) or Blackwell (RTX 50-series), which have more optimized BF16 tensor core paths. For most training workloads, the 3090's BF16 performance is still a significant improvement over FP32.

What is the minimum CUDA compute capability for PyTorch?

PyTorch 2.5+ requires CUDA compute capability 7.0 or higher (Volta architecture and newer). This means GTX 10-series (Pascal, compute capability 6.1) and older GPUs are no longer officially supported. Any RTX card — from the RTX 2060 onward — meets this requirement. For practical deep learning work, compute capability 8.0+ (Ampere or newer) is recommended to access BF16, TF32, and efficient tensor core operations.

Is an AMD GPU good for PyTorch?

PyTorch supports AMD GPUs through ROCm, but the experience is rougher than CUDA. Fewer pre-built wheels are available, some operations fall back to CPU, and community debugging resources overwhelmingly target NVIDIA. ROCm version compatibility can also be finicky, requiring specific driver and kernel combinations. If PyTorch is your primary framework, NVIDIA GPUs provide a significantly smoother experience.

Related guides on Best GPU for AI

The full version lives on Best GPU for AI — VRAM calculator, GPU comparison table, and live Amazon pricing.

Rent a GPU for LLM Fine-Tuning: The $30 Weekend Project

Thurmon Demich — Mon, 03 Aug 2026 01:13:38 +0000

From the Best GPU for LLM archive. The canonical version has interactive calculators, an up-to-date GPU comparison table, and live pricing.

You've got a dataset of 10,000 support tickets, a free weekend, and a plan: fine-tune a 7B model to answer like your best support engineer. Then you check GPU prices and the plan stalls. An RTX 4090 is ~$1,600. A used 3090 is ~$800. For one training run?

Stop. This is a rental job.

A QLoRA fine-tune on a 7B-13B model takes somewhere between 3 and 12 hours of GPU time for a dataset that size. Rented by the hour, your entire weekend project costs less than a nice dinner. The real decision isn't "which GPU should I buy" — it's "am I buying hours or buying hardware."

What your fine-tune actually needs

The method determines the hardware, and the hardware determines the bill. QLoRA quantizes the base model to 4-bit and trains only adapter layers, which is why a 7B run needs roughly 8GB of VRAM and a 13B run roughly 14GB — both fit a rentable 24GB RTX 4090 at roughly $0.35-0.50/hr as of mid-2026.

Step up to QLoRA on 70B, or full LoRA on a 13B, and you want an A100 80GB at roughly $0.75-1.50/hr. Full-parameter fine-tunes of anything mid-size push you into multi-GPU H100 territory at roughly $2.00-3.00/hr per card — rarely worth it when adapter methods get you 95% of the quality.

What you're training	Rent this	Approx $/hr (mid-2026)	Typical weekend cost
QLoRA 7B-13B	RTX 4090 (24GB)	$0.35-0.50	~$10-25
QLoRA 34B-70B / full LoRA 13B	A100 80GB	$0.75-1.50	~$20-60
Full fine-tune 7B-13B	2-4x H100 80GB	$2.00-3.00 each	~$100-400

That first row is the whole story for most weekend projects. Ten dollars.

The break-even math

Say you fine-tune once a quarter, 10 GPU-hours per run. On a rented A100 that's roughly $40-60 a year. The RTX 4090 you were about to buy costs $1,600 — a 25-40 year payback, ignoring electricity. For occasional fine-tuning, renting wins by an order of magnitude and it isn't close. Our cloud vs self-hosted TCO breakdown runs the full numbers.

The math flips if you're iterating constantly. Ten runs a month — sweeping hyperparameters, testing dataset variants, retraining weekly — burns $75-150/month in rentals, and now a used RTX 3090 pays for itself inside a year. (It also doubles as your inference box, which the rental never does.) That's when our buying guide for fine-tuning GPUs becomes the relevant article, not this one.

For the heavy-iteration case, the 24GB card is the one to own:
See the recommended pick on the original guide
handles QLoRA up to 34B locally, and every experiment after purchase is free.

Renting without wasting money

A few habits separate a $15 weekend from a $70 one.

Checkpoint to persistent storage. Spot instances get reclaimed with minutes of warning. Save checkpoints every 30-60 minutes to a network volume or push adapters to Hugging Face — losing hour nine of a ten-hour run is the classic rental disaster.
Use spot pricing. Interruptible instances run 50-80% off on-demand rates on RunPod and Vast.ai. With checkpointing in place, interruptions cost you minutes, not money.
Watch egress fees. Many providers charge $0.05-0.12/GB to move data out. A LoRA adapter is a few hundred MB — trivial. A full merged 13B model is 25GB+, so merge locally instead of downloading merged weights.
Shut it down. An idle A100 bills exactly like a busy one. Terminate the instance when training ends; don't leave it running "in case" you want to poke at the results Monday.

Common mistakes

Renting an H100 for a QLoRA job. A 7B QLoRA run doesn't saturate an H100; you're paying 5x the rate for maybe 2x the speed. Rent the cheapest GPU that fits your VRAM requirement.
Uploading the dataset over and over. Prep your data locally, upload once to persistent storage, and mount it across runs. Re-uploading 10GB on every instance start wastes both time and, on some providers, money.
Skipping a dry run. Launch with 1% of your dataset first. Catching a broken tokenizer config after ten minutes costs pennies; catching it after ten hours costs the whole weekend budget.

Renting for fine-tuning: quick answers

How much does it cost to rent a GPU for fine-tuning?

A rented RTX 4090 runs roughly $0.35-0.50/hr as of mid-2026, and an A100 80GB roughly $0.75-1.50/hr. A weekend QLoRA run on a 7B-13B model typically lands around $10-40 total including setup and some idle time. Spot pricing can cut that by half or more, provided you checkpoint regularly in case the instance gets reclaimed.

Can I fine-tune a 7B model on a rented RTX 4090?

Yes, comfortably. QLoRA on a 7B model needs roughly 8GB of VRAM, so the 4090's 24GB handles it with room to raise batch size or sequence length. A 13B QLoRA run at roughly 14GB also fits without trouble. Full LoRA on 13B is where you outgrow 24GB and should step up to a rented A100 80GB instead.

Do I need an H100 to fine-tune an LLM?

Almost never for adapter-based methods. QLoRA on models up to 34B fits a single 24GB card, and an A100 80GB covers QLoRA on 70B. H100s earn their roughly $2.00-3.00/hr rate on full-parameter fine-tunes and multi-GPU training runs where raw throughput matters more than cost — not on a weekend adapter project.

Verdict

Fine-tuning occasionally? Rent. A weekend QLoRA project on a 7B-13B model costs roughly $10-40 on a rented 4090 or A100 — the same GPU hours you'd get from a $1,600 purchase, minus the purchase. Iterating daily? Buy the 24GB card and rent bigger cloud GPUs only for the runs that don't fit.

If you can count your fine-tuning runs per year on one hand, rent the GPU — the hardware you were about to buy takes decades to pay for itself.

Related guides on Best GPU for LLM

The full version lives on Best GPU for LLM — VRAM calculator, GPU comparison table, and live Amazon pricing.

Best GPU for Deep Learning in 2026 (5 Picks Ranked)

Thurmon Demich — Sun, 02 Aug 2026 01:13:39 +0000

From the Best GPU for AI archive. The canonical version has interactive calculators, an up-to-date GPU comparison table, and live pricing.

Quick answer: The RTX 4090 (24GB) is the best value GPU for deep learning in 2026. The RTX 5090 (32GB) is the top performer for large-scale training, and the RTX 4060 Ti 16GB is the budget entry point for students and beginners.

Why the GPU is everything for deep learning

Deep learning frameworks like PyTorch and TensorFlow offload almost all computation to the GPU. Training a neural network is fundamentally a series of matrix multiplications — exactly what GPU tensor cores are designed for. The two metrics that matter most:

VRAM — determines maximum model size, batch size, and whether training is even possible
Compute throughput — determines how fast each training step runs (measured in TFLOPS for FP16/BF16)

Miss on VRAM and training simply fails. Miss on compute and training works but takes far longer. Both matter, but VRAM is the hard constraint.

VRAM: the non-negotiable requirement

Understanding how much VRAM a model needs requires knowing where the memory goes during training:

Component	Memory consumed
Model weights (FP16)	~2 bytes × parameter count
Optimizer states (Adam)	~8 bytes × parameter count
Gradients	~2–4 bytes × parameter count
Activations (batch-dependent)	Variable
Total training footprint	~12–16 bytes × parameter count

Practical VRAM requirements for fine-tuning and training common model sizes:

Model size	Inference only	LoRA fine-tune	Full fine-tune
1B parameters	2–3GB	6–8GB	14–18GB
3B parameters	6–8GB	10–14GB	OOM on 24GB
7B parameters	14–16GB	16–20GB	Needs A100
13B parameters	26–28GB	24GB (4-bit)	Needs multi-GPU

This is why a 7B model in FP16 needs roughly 14GB just to load weights — and doubles to 28GB when you add optimizer states for training. 24GB is the practical ceiling for consumer fine-tuning; 16GB works with 4-bit quantization (QLoRA).

Training speed benchmarks

Relative throughput on mixed workloads — ResNet-50 classification, BERT fine-tuning, and ViT training (normalized to RTX 5090):

GPU	VRAM	FP16 TFLOPS	Relative speed	Price	Value tier
RTX 5090	32GB GDDR7	~220	1.00x	~$2,000	Research
RTX 4090	24GB GDDR6X	~165	0.75x	~$1,600	Best value
RTX 5080	16GB GDDR7	~137	0.62x	~$1,000	Mid-range
RTX 5070 Ti	16GB GDDR7	~105	0.48x	~$750	Mid-range
RTX 4070 Ti Super	16GB GDDR6X	~88	0.40x	~$700	Budget pro
RTX 3090 (used)	24GB GDDR6X	~121	0.55x	~$800	VRAM value
RTX 4060 Ti 16GB	16GB GDDR6	~62	0.28x	~$400	Entry level

The RTX 5090's 220 TFLOPS in FP16 is a significant leap over the 4090, but at $2,000 the price jump is steep. The 4090 at $1,600 with 165 TFLOPS and 24GB remains the most compelling overall package unless you regularly work with models that need 32GB.

Batch size vs VRAM: practical impact

Larger batch sizes improve training stability and convergence. More VRAM = larger viable batch sizes:

GPU (VRAM)	Typical max batch (ResNet-50)	BERT batch (seq 128)	Notes
RTX 4060 Ti (16GB)	64	16	Limited but functional
RTX 4070 Ti Super (16GB)	64	16	Same VRAM, faster compute
RTX 4090 (24GB)	128	32	Comfortable for most research
RTX 5090 (32GB)	256	64	Large model experimentation
RTX 3090 (24GB)	128	32	Same as 4090 but slower

For practical training, the difference between 16GB and 24GB often means the difference between batch size 8 and batch size 24 on 7B model fine-tuning — which affects both training speed and gradient quality.

PyTorch vs TensorFlow GPU considerations

Both major frameworks support all NVIDIA cards on this list, but there are nuances:

PyTorch (dominant for research):

Best-in-class CUDA integration
torch.compile() in PyTorch 2.x significantly improves performance on Ampere and newer
Excellent support for BF16 on RTX 30-series and newer (better than FP16 for training stability)
Flash Attention 2 works natively on CUDA

TensorFlow (production and enterprise):

Strong XLA compilation support
Multi-GPU via MirroredStrategy works well on consumer cards
Some newer optimizations arrive later than PyTorch

Both frameworks support FP16 and BF16 mixed precision on Ampere (RTX 30xx) and newer. The RTX 50-series Blackwell architecture adds improved FP8 and FP4 support for inference quantization. For pure research and experimentation, PyTorch is the standard. For production systems already on TensorFlow, it doesn't matter — both support the hardware equally well. If you do most of your work through the Hugging Face ecosystem, see our best GPU for Hugging Face guide for PEFT and Transformers-specific setup.

FP16 and BF16 mixed precision: free performance

Mixed precision training is one of the most impactful optimizations available — and it's free:

FP16 — cuts VRAM by ~50% versus FP32, faster on tensor cores. Can have numerical stability issues with very small gradients.
BF16 — same memory savings as FP16, better numerical range for training stability. Supported on Ampere (RTX 30xx) and newer.
Practical impact: Doubles your effective batch size, cuts training time by 20–40%, often with no accuracy loss.

Always enable mixed precision in PyTorch:

from torch.cuda.amp import autocast, GradScaler
scaler = GradScaler()
with autocast(dtype=torch.bfloat16):  # or float16
    output = model(input)

Multi-GPU scaling for deep learning

Consumer multi-GPU for deep learning (via DataParallel or DistributedDataParallel) works, but has limitations. Before committing to a multi-GPU build, see our how many GPUs you actually need for AI training guide — for most home setups, one large card beats two smaller ones:

PCIe bandwidth between consumer cards is the bottleneck — professional NVLink is 10–20x faster for inter-GPU communication
PyTorch DDP scales reasonably on 2–4 GPUs even over PCIe for data parallel training
Two RTX 4090s gives 48GB aggregate VRAM and roughly 1.5x effective throughput (not 2x due to communication overhead)
Model parallelism (splitting a large model across GPUs) is painful on consumer hardware without NVLink

For most home deep learning setups, one large GPU with more VRAM beats two smaller GPUs — simpler, less overhead, better actual throughput on single large models.

RTX 5090 — the research choice

At 32GB GDDR7 and ~220 FP16 TFLOPS, the RTX 5090 is the only consumer GPU that handles 13B+ model fine-tuning at full precision without resorting to 4-bit quantization:

Fine-tune 7B models at FP16 with comfortable batch sizes
Run 13B models with QLoRA (4-bit base + FP16 adapter layers)
Inference on 30B+ models with 4-bit quantization
Train small models from scratch at speeds that actually make sense

The RTX 5090 makes sense for researchers who need maximum headroom, graduate students doing serious ML work, or professionals building models for production. For learning and hobby projects, it's hard to justify the cost. For a dedicated overview of hardware for academic and applied ML research, see our best GPU for AI research guide.

RTX 4060 Ti 16GB — best for beginners

At ~$400, the RTX 4060 Ti 16GB is the right starting point for anyone learning deep learning:

Runs every PyTorch and TensorFlow tutorial without issues
16GB handles QLoRA fine-tuning of 7B models
Slow for serious training, but fine for understanding concepts and running experiments
Low power draw (~165W) makes it easy to add to any existing PC

The RTX 4060 Ti's 128-bit memory bus limits bandwidth compared to higher-end cards, which matters for training throughput but not for learning. Start here and upgrade when you consistently need more.

Not ready for dedicated hardware? Cloud GPU is the answer

Before spending $1,600+ on a GPU, consider renting compute for specific experiments. RunPod and Vast.ai offer A100 80GB and RTX 4090 instances for $0.40–$1.50/hr — more cost-effective than buying a card if you're training sporadically.

For a broader overview of picking AI hardware, see our Best GPU for AI guide.

GPU tier list available at the original article

Which GPU should YOU buy for deep learning?

Student or beginner learning the fundamentals: RTX 4060 Ti 16GB at $400. Runs every tutorial, handles 7B QLoRA fine-tuning, easy to start with. Upgrade when you outgrow it.
Serious hobbyist fine-tuning models up to 7B at FP16: RTX 4090 at 24GB is the sweet spot. Best VRAM-per-dollar on the market, fast enough for meaningful research. Used RTX 3090 at ~$800 if budget is tight.
Researcher or professional training 13B+ models: RTX 5090 at 32GB is the only consumer card with enough VRAM for 7B FP16 training at useful batch sizes without resorting to quantization tricks.
Running inference more than training: 16GB is enough. RTX 5070 Ti or RTX 4070 Ti Super save significant money with no inference penalty.
Training models professionally on a schedule: Rent cloud GPUs (RunPod/Vast.ai) for large runs, use a local 4090 for experimentation. If most of your training happens on a desktop in your study, our best GPU for AI training at home guide weighs power, noise, and thermals for that exact build.
Training code models specifically: Code generation models (CodeLlama, DeepSeek Coder, Qwen Coder) have their own VRAM patterns — see our best GPU for codegen AI guide.
Working with audio (Whisper, MusicGen): Audio workloads are surprisingly light — our best GPU for Whisper guide ranks the budget options.

Common mistakes to avoid

Buying based on CUDA cores instead of VRAM. More cores speed up training, but running out of VRAM stops training entirely. Prioritize memory capacity first, then compute.
Choosing AMD to save money. ROCm support in PyTorch and TensorFlow still lags CUDA significantly — especially for newer optimization techniques. The debugging time erases the hardware savings.
Skipping mixed precision training. BF16/FP16 cuts VRAM usage by ~50% with minimal accuracy loss. Always enable it. It's free performance that should be default in every training script.
Expecting multi-GPU to scale linearly. Two consumer GPUs won't give you 2x speed. PCIe communication overhead and lack of NVLink means you'll see 1.3–1.6x at best on typical deep learning tasks.
Not using gradient checkpointing on large models. Reduces peak VRAM by 30–40% at the cost of ~20% longer training time. Enable it before buying a bigger GPU.

Final verdict

The RTX 4090 is the best deep learning GPU for most people in 2026. If 24GB isn't enough for your specific workloads, the RTX 5090 at 32GB is the upgrade. If $1,600 is too much, a used RTX 3090 gives you the same 24GB at half the price with slower compute.

For students, the RTX 4060 Ti 16GB at $400 is genuinely all you need to learn. For the NVIDIA vs AMD comparison in more depth, we cover ROCm's current limitations in detail. If you primarily use TensorFlow over PyTorch, see our best GPU for TensorFlow guide for XLA-optimized setups.

The best GPU for deep learning is the one with the most VRAM you can afford — training speed matters, but running out of memory stops you completely.

Related guides on Best GPU for AI

The full version lives on Best GPU for AI — VRAM calculator, GPU comparison table, and live Amazon pricing.

Best GPU for DeepSeek V4: The Honest VRAM Math (81GB Minimum)

Thurmon Demich — Sat, 01 Aug 2026 12:56:52 +0000

Cross-posted from Best GPU for LLM — visit the original for our VRAM calculator, GPU comparison table, and current Amazon pricing.

No single consumer GPU runs DeepSeek V4. The smallest working quant of V4-Flash weighs roughly 81GB as of July 2026, so before you spend anything, here is what every hardware tier actually gets you.

Hardware tier	Usable memory	What fits	Expected experience
Single 24GB (RTX 3090/4090)	24GB	Nothing from V4 — run DeepSeek-R1 32B instead	Great, just not V4
Dual 24GB (2x 3090/4090)	48GB	Still ~33GB short of V4-Flash's smallest quant	Dead end for V4
Quad 24GB (4x RTX 3090)	96GB	V4-Flash at aggressive ~2-bit quants (~81GB weights)	Works, tight on context
96GB workstation (2x RTX 6000 Ada)	96GB	Same fit, two slots instead of four	Cleaner, pricier
128GB Mac Studio (unified)	~115GB usable	V4-Flash at low-to-mid quants	Usable, slow prompt processing
Cloud 2x H200 (rented)	282GB	V4-Flash Q4_K_M (~158GB) + ~10GB KV cache	Comfortable, full context
Anything you own	—	V4-Pro: not happening (~800GB at Q4)	API only

The pattern I keep seeing in V4 discussion threads: people anchor on "13B active parameters" and assume 13B-class hardware. That is the most expensive misconception in local AI right now, and this guide is the math that corrects it. If you want a card for the older, smaller DeepSeek models, our DeepSeek GPU guide covers R1 and Coder — those still fit on one 24GB card.

VRAM chart available at the original article

V4-Pro is not a consumer decision

Quick reality check so we can move on. DeepSeek V4-Pro is a 1.6-trillion-parameter MoE with 49B active parameters per token. Even at Q4, the weights alone land around 800GB — ten H100s before you allocate a single byte of KV cache. That is a datacenter deployment, and no quality-preserving quant changes it in 2026.

So for local purposes, "best GPU for DeepSeek V4" really means "best hardware for V4-Flash." Everything below is about Flash.

V4-Flash: the quant math that matters

V4-Flash is 284B total parameters with 13B active. The community converted it fast — GGUF, AWQ, and GPTQ builds all appeared within days of the release, and llama.cpp and LM Studio both run the GGUFs without drama. What the quant sizes actually look like:

Quant level	Approx. weights	Fits on	Quality verdict
Smallest working (~2-bit)	~81GB	96GB rig (4x 24GB)	Noticeably degraded, still coherent
Low-to-mid (~3-bit)	~105-115GB	128GB Mac Studio	The practical unified-memory sweet spot
Q4_K_M (community default)	~158GB	2x H200-class	Where Flash stops feeling quantized
Q8 and above	280GB+	Datacenter	Diminishing returns

Then add the part almost everyone forgets: KV cache. At long context, budget roughly 10GB on top of the weights. That is why a comfortable Q4_K_M deployment totals about 170GB — and why a 96GB rig running the ~81GB quant has only ~15GB left for context and overhead. It runs. It just isn't roomy, and you'll feel it the first time you paste a large codebase into the prompt.

My honest read on the aggressive quants: 2-bit Flash still beats any 70B dense model I can run in the same footprint, because those 284B total parameters retain a lot of knowledge even when brutally compressed. But if your work is code generation or anything where subtle correctness matters, the gap between ~2-bit and Q4_K_M is real.

Mac unified memory vs. multi-GPU

The 128GB Mac Studio is the sleeper option here, and it's a genuinely different tradeoff rather than a worse one.

The Mac case: one silent box, no 1600W power supply, and about 115GB actually allocatable to the model. Because Flash only activates 13B parameters per token, generation speed on Apple Silicon is respectable — MoE models are far kinder to unified memory bandwidth than dense ones. The pain is prompt processing: feeding in a long document takes several times longer than on NVIDIA hardware. We covered the general pattern in our Mac hardware for LLMs breakdown.

The multi-GPU case: 4x RTX 3090 gives you 96GB with vastly faster prefill and the full CUDA ecosystem. The cost is complexity — a server board with enough PCIe lanes, a big PSU, heat, noise, and llama.cpp layer-splitting configuration. Our multi-GPU setup guide walks through the actual build; read it before ordering parts, because the motherboard is the part everyone gets wrong.

Speed favors NVIDIA. Sanity favors the Mac. Budget is roughly a wash — four used 3090s plus a platform to host them lands near a 128GB Studio.

Which setup should you build?

You own a single 24GB card: don't build anything for V4. Run DeepSeek-R1 32B or a 70B at low quant — our 70B on a single GPU guide covers that path — and hit the API when you genuinely need V4.
You want the cheapest real V4-Flash box: 4x used RTX 3090 (~$2,800-3,600 in cards) for 96GB. Accept the ~2-bit quant and short-ish context.
You want V4-Flash without a server in your office: 128GB Mac Studio. Mid quants, quiet, one power cable.
You want Q4_K_M quality: rent. 2x H200 on-demand covers the ~170GB footprint for a few dollars an hour, which buys a lot of sessions before matching workstation money.

Common mistakes to avoid

Buying 2x 4090s expecting V4-Flash comfort. 48GB is not close. The smallest working quant is ~81GB — a dual-card build strands you $3,000+ deep with no path to running the model at all.
Confusing 13B active with 13B VRAM. MoE routing reduces compute per token, not memory. All 284B parameters must sit in memory because any expert can be selected at any step.
Ignoring KV cache in the budget. Weights-only math says 81GB fits in 96GB "with room to spare." Long-context KV eats ~10GB, and suddenly your 32K session is crashing.
Provisioning for V4-Pro because it benchmarks higher. Pro is ~800GB at Q4. If someone quotes you hardware for it, they're quoting a datacenter.

Our verdict

Your situation	Best move	Approx. cost
Cheapest working V4-Flash	4x used RTX 3090 (96GB)	~$3,500 in cards
Cleanest 96GB, two slots	2x RTX 6000 Ada 48GB	~$12,000
Quiet single box	Mac Studio 128GB	~$4,800
Best quality per dollar	Rent 2x H200, run Q4_K_M	~$7/hr
Already own one 24GB GPU	Run R1 32B locally, API for V4	$0

DeepSeek V4-Flash draws a hard line at roughly 81GB, and no amount of optimism moves it. Build to 96GB, buy 128GB unified, or rent 282GB — those are the three doors, and pretending a 48GB rig is a fourth one is how people waste real money.

DeepSeek V4 VRAM questions

How much VRAM does DeepSeek V4-Flash need?

As of July 2026, the smallest working quant of V4-Flash occupies roughly 81GB, so about 96GB of total VRAM is the realistic entry point once you account for KV cache and overhead. The community-default Q4_K_M build is much larger — roughly 158GB of weights plus around 10GB of KV cache at long context, or about 170GB total.

Can DeepSeek V4 run on a single RTX 5090?

No. The RTX 5090's 32GB is far below the roughly 81GB that V4-Flash's smallest usable quant requires, and no foreseeable quantization closes that gap. A single 5090 is still an excellent card for DeepSeek-R1 32B and other dense models in the 30-70B range — just not for anything in the V4 family.

Is a Mac Studio good for DeepSeek V4?

A 128GB Mac Studio is a legitimate single-box option for V4-Flash at low-to-mid quants, with roughly 115GB allocatable to the model. Because Flash activates only 13B parameters per token, generation speed is reasonable on Apple Silicon. The tradeoff is prompt processing, which runs several times slower than on comparable NVIDIA multi-GPU setups.

What is the hardware difference between V4-Pro and V4-Flash?

V4-Pro is 1.6T total parameters and needs roughly 800GB even at Q4 — strictly datacenter or API territory. V4-Flash is 284B total and spans roughly 81GB at its smallest quant to about 170GB at Q4_K_M with long-context KV cache, which puts it within reach of 96GB multi-GPU rigs, 128GB Macs, and rented H200 pairs.

Related guides on Best GPU for LLM

The full version lives on Best GPU for LLM — VRAM calculator, GPU comparison table, and live Amazon pricing.

Best GPU for Z-Image Turbo in 2026: Budget Picks From $180

Thurmon Demich — Fri, 31 Jul 2026 01:14:22 +0000

Cross-posted from Best GPU for AI — visit the original for our VRAM calculator, GPU comparison table, and current Amazon pricing.

Alibaba's Z-Image Turbo is the most budget-friendly serious text-to-image model I have covered this year. It is a 6B-parameter open-weight model that Alibaba positions as FLUX.1-class quality at a fraction of the compute, and — as of late July 2026 — the community has already pushed it down to cards that Flux users would not even consider. The full BF16 build wants roughly 14-16GB of VRAM, the FP8 build fits in about 8GB, and community GGUF quants run on 6GB cards. That spread is the entire buying question, so this guide is organized as a tier ladder instead of a leaderboard.

Quick answer: The RTX 5060 Ti 16GB (~$430) is the best-value GPU for Z-Image Turbo. It runs the full BF16 build in ComfyUI with headroom to spare, and 16GB-class consumer cards are turning out images in roughly 2-3 seconds each — fast enough that iteration stops feeling like waiting.

The 6GB-to-16GB tier ladder

Every tier below runs Z-Image Turbo today. What changes is which build you load and what you give up. Prices are typical street prices as of late July 2026.

VRAM tier	Z-Image build	GPU pick	Street price	What you sacrifice
6GB	Community GGUF quant	RTX 3050 6GB (or used RTX 2060)	~$180 (~$110 used)	Noticeable quality loss from aggressive quantization, slow generation, no LoRA headroom
8GB	FP8	RTX 4060 8GB	~$290	Minor quality delta vs BF16; tight once you add upscaling or ControlNet
12GB	FP8 with real headroom	RTX 3060 12GB	~$250	Older Ampere architecture, slower per-step speed than the 8GB tier's newer silicon
16GB	Full BF16	RTX 5060 Ti 16GB (alt: RTX 4060 Ti 16GB, ~$450)	~$430	Nothing, at this model size

The odd wrinkle is the middle of the ladder. The RTX 3060 12GB is cheaper than the RTX 4060 8GB despite having 4GB more VRAM, because it is a two-generation-old card. For Z-Image Turbo specifically, that trade is genuinely interesting: the 3060 loads FP8 with room for extras, but the 4060's newer tensor cores chew through FP8 faster per step. More on that decision below.

GPU tier list available at the original article

Who Z-Image Turbo is for

Z-Image Turbo exists for exactly one kind of buyer: someone who wants modern prompt adherence and clean text rendering without paying modern-flagship prices. If you have been eyeing Flux.1 and wincing at the 24GB-class VRAM demands, this is the model that makes your existing card — or a $300 card — relevant again.

The "Turbo" part matters too. This is a distilled, few-step model, which is why 16GB consumer cards are producing images in roughly 2-3 seconds rather than the 20-30 seconds older diffusion workflows trained us to tolerate. At that speed you iterate differently. You stop crafting one perfect prompt and start generating in bursts, keeping the best of eight instead of the best of one.

It runs in ComfyUI, and ComfyUI is where the community quant work is happening (the GGUF loaders that make the 6GB tier possible are custom nodes, not official Alibaba releases). If you have never touched a node graph, our ComfyUI GPU guide covers what the frontend itself wants from your hardware — the short version is that ComfyUI adds almost no overhead of its own.

Z-Image Turbo vs the Flux ecosystem

The honest comparison is not "which model is better" — it is "what does each one cost you in hardware."

Flux.1 Dev at full precision is a 24GB conversation, and even its quantized builds are happiest on 16GB. Krea 2 plays in similar territory. Alibaba's own Qwen Image, a 7B model, needs about 14GB at FP16 — so 16GB is its floor for comfortable use, not its ceiling.

Z-Image Turbo undercuts all of them. At 6B parameters with an efficient distilled architecture, its BF16 build lands in the 14-16GB range as of late July 2026, its FP8 build fits ~8GB, and GGUF quants go lower still. The claim Alibaba makes — FLUX.1-class output at much lower compute — held up well enough in community testing that the model became ComfyUI's budget default almost overnight.

My take after watching this segment for two years: parameter count is quietly becoming the most important spec in image generation, because it decides which humans get to run the model at all. A 6B model that trades a few percentage points of quality for an $800 lower hardware bill is not a compromise. It is the correct engineering target.

Which GPU should YOU buy?

Buy the RTX 5060 Ti 16GB (~$430) if:

You want the full BF16 build with zero babysitting
You plan to add LoRAs, upscaling, or ControlNet later
You might also run Flux quants or Qwen Image on the same card

Buy the RTX 4060 8GB (~$290) if:

FP8 quality is good enough (for most people, it is)
You want the fastest card under $300 for few-step models
You will not stack extras on top of base generation

Buy the RTX 3060 12GB (~$250) if:

You want FP8 plus headroom for upscale passes at the lowest price
You accept older, slower silicon in exchange for VRAM
You are buying used and can find one near $200

Buy the RTX 3050 6GB (~$180) if:

The budget is genuinely fixed and GGUF quality is acceptable
Z-Image Turbo is an experiment, not a workflow

Skip:

Anything above 16GB for this model alone. A 4090-class card makes sense for Flux or video work, but Z-Image Turbo cannot use the extra VRAM.

Common mistakes to avoid

Buying 24GB for a 6B model. Z-Image Turbo's entire reason to exist is that it does not need flagship VRAM. If this model is your workload, the money above ~$450 buys you nothing.
Assuming the 6GB GGUF tier feels like the demos. It runs, which is remarkable, but aggressive quantization visibly softens fine detail and text rendering. Treat 6GB as a trial tier, not a destination.
Ignoring the used market at the 12GB tier. A used RTX 3060 12GB near $200 is arguably the best price-per-usable-gigabyte in this entire ladder, and Ampere is still well supported in ComfyUI.
Grabbing the first "Z-Image" checkpoint you see. The ecosystem moved fast and mislabeled quants are common. Match the build to your VRAM deliberately — BF16 for 16GB, FP8 for 8-12GB, GGUF below that — instead of letting a random download decide.

Final verdict

Budget	GPU	Z-Image Turbo experience
~$180	RTX 3050 6GB	GGUF quant only; proof it works, not a daily driver
~$250	RTX 3060 12GB	FP8 with headroom; best used-market value
~$290	RTX 4060 8GB	FP8, fastest sub-$300 option
~$430	RTX 5060 Ti 16GB	Full BF16, ~2-3s per image, no compromises
~$450	RTX 4060 Ti 16GB	Full BF16 on Ada; fine alternative if the 5060 Ti is out of stock

Buy the RTX 5060 Ti 16GB if you can stretch to it. Buy the RTX 4060 or a used RTX 3060 12GB if you cannot — both run FP8 well, and FP8 is closer to full quality than the tier label suggests.

Z-Image Turbo inverts the usual GPU advice: the question is not how much VRAM you can afford, but how little you can get away with. For most people, the honest answer is 16GB at ~$430 — and not a dollar more.

Z-Image Turbo hardware FAQ

What is the minimum VRAM for Z-Image Turbo?

As of late July 2026, community GGUF quants run on 6GB cards, making that the practical floor. The FP8 build fits in roughly 8GB, and the full BF16 build wants around 14-16GB. For comfortable daily use with LoRAs or upscaling in the workflow, a 16GB card is the sensible recommendation rather than the bare minimum.

Can Z-Image Turbo run on an 8GB card like the RTX 4060?

Yes. The FP8 build of Z-Image Turbo fits within roughly 8GB of VRAM, and the RTX 4060's tensor cores handle FP8 efficiently, so base generation works well. The constraint shows up when you stack extras — upscaling passes or ControlNet can push past 8GB, which is where the 12GB and 16GB tiers earn their price.

Is Z-Image Turbo better than Flux for low-VRAM GPUs?

For low-VRAM hardware, yes. Z-Image Turbo is a 6B model positioned as FLUX.1-class quality at much lower compute, and its BF16 build fits in the 14-16GB range where Flux.1 Dev at full precision needs 24GB-class cards. On a budget card, Z-Image Turbo runs natively where Flux requires heavy quantization or offloading.

Does Z-Image Turbo work in ComfyUI?

Yes, ComfyUI is the primary way to run it. The official builds load through standard diffusion workflows, and the community GGUF quants that enable 6GB cards are distributed as ComfyUI custom nodes. Because it is a distilled few-step model, generation is quick — 16GB-class consumer cards produce images in roughly 2-3 seconds each.

Related guides on Best GPU for AI

Continue on Best GPU for AI for the complete guide with interactive calculators and current GPU prices.

Can You Run Kimi K3 Locally? No — Here's the Exact Math

Thurmon Demich — Thu, 30 Jul 2026 01:14:39 +0000

From the Best GPU for LLM archive. The canonical version has interactive calculators, an up-to-date GPU comparison table, and live pricing.

The weights dropped on July 27. Moonshot put Kimi K3 on Hugging Face under a Modified MIT license, the download button works, and somewhere right now a person with an RTX 4090 is watching a 594GB MXFP4 checkpoint crawl toward a 2TB SSD, convinced that llama.cpp will sort out the rest. It won't. I want to kill this dream cleanly, with numbers, before you waste a weekend on it.

Short answer: no. Not on a 4090, not on a 5090, not on a Mac Studio, not on the 8x H100 box your startup rents. As of late July 2026, Kimi K3 is a multi-node deployment or it is an API call. There is no third option yet.

The memory math, tier by tier

Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model with 896 experts. The native MXFP4 release is already aggressively compressed — about 594GB just to download — and the total weight footprint lands around 1.4TB once scales and runtime overhead are counted. Here's what that means against real hardware:

Tier	Memory you have	Kimi K3 needs	Coverage
RTX 4090	24GB	~594GB (MXFP4 weights alone)	~4% of the download
RTX 5090	32GB	~594GB	~5%
4x RTX 5090 rig	128GB	~594GB	~22%
Mac Studio 512GB	512GB unified	~594GB	short before overhead
8x H100 80GB (one node)	640GB	~1.4TB total footprint	not enough
Multi-node H100/H200 cluster	1.5TB+	~1.4TB	the actual floor

Sit with those middle rows for a second. A full DGX-class node — eight H100s, the machine most people picture when they say "serious AI hardware" — cannot hold the full weights. The realistic floor is multiple interconnected nodes of H100s or H200s. That's not a homelab. That's a rack with a networking budget.

"But it's MoE — only 16 experts are active!"

This is the myth doing the heavy lifting in every optimistic Reddit thread. Yes, K3 routes each token through just 16 of its 896 experts. And yes, that makes inference compute dramatically cheaper than a dense 2.8T model would be. But the router picks different experts every single token, which means every expert has to be sitting in memory, ready to fire. MoE sparsity saves FLOPs, not bytes. All 896 experts are resident, all the time.

Smaller MoE models like Kimi K2 get away with local deployment because their total quantized footprint squeezes into prosumer territory. K3's footprint is roughly 20x too large for that trick, and its native format is already 4-bit — there's no magic quant waiting to shrink it another 95%. Days after release, no full-fidelity GGUF port has been confirmed either, so the "wait for community quants" plan currently has nothing on the calendar that changes the math.

What you can actually do (three lanes)

Lane 1 — hosted API. Moonshot's own API, OpenRouter, Together, and Fireworks all serve K3, and day-one vLLM support meant hosted capacity spun up fast. If you want K3's quality, this is the honest path for anyone without a cluster line item.

Lane 2 — rent the cluster. If you genuinely need K3 under your own control — fine-tuning experiments, private data, weight surgery — rent multi-GPU capacity by the hour instead of pretending a workstation will do it. Our cloud GPU for LLM guide walks the provider math; for scale context on why datacenter cards are a different species, see RTX 5090 vs H100.

Lane 3 — self-host the K2 class instead. This is what your money actually buys locally. A 24GB card runs Kimi K2 at Q4 with real agent-grade reliability, and Kimi K2.7 Code is the current lighter self-host path in the same family. The best GPU for Kimi K2 guide covers the exact builds.
See the recommended pick on the original guide
The same logic applies to any big-model itch: running 70B on a single GPU is a solved problem; running 2.8T is not.

Common mistakes I'm already seeing

Buying hardware for a model you'll run twice. Even if a heroic community effort eventually squeezes some K3 variant onto a 512GB unified-memory machine, ask how often you'd load it versus a fast K2.7 you use daily. Novelty inference is a terrible reason to spend five figures.
Confusing "open weights" with "runnable weights." Modified MIT means you may run it anywhere. The 1.4TB footprint decides whether you can.
Extrapolating from K2's local success. K2 fitting on a 4090 at Q4 was a happy accident of architecture and size. K3 is 2.8T with a 4-bit native format — the compression lever is already pulled.

Verdict

As of late July 2026: use K3 through Moonshot, OpenRouter, Together, or Fireworks; rent a multi-node cluster if you truly need custody of the weights; and put your local budget into the K2 class, which is the best Moonshot model that actually fits inside a house.

Kimi K3 being open-weight is genuinely great for the ecosystem. It is not an invitation to your GPU. Run K2.7 at home, rent H100s when you need the weights, and let the API handle the rest.

Kimi K3 hardware: quick answers

How much VRAM does Kimi K3 need?

Far more than any single machine offers. The native MXFP4 download alone is roughly 594GB, and the total weight footprint with scales and overhead lands around 1.4TB. Even an 8x H100 80GB node (640GB) can't hold the full weights, so realistic deployment means multiple interconnected H100 or H200 nodes — datacenter territory, not a workstation.

Can a quantized Kimi K3 run on consumer GPUs?

No. The release format is already 4-bit MXFP4, so the usual "wait for Q4 quants" play doesn't apply — that compression is baked in. As of late July 2026 there are no confirmed full-fidelity GGUF ports, and even hypothetical aggressive quants would still measure in hundreds of gigabytes, more than ten times what a 32GB flagship card offers.

What's the closest model you can actually run at home?

Kimi K2.7 Code is the lighter self-host path in the same Moonshot family, and the broader Kimi K2 class runs well at Q4 on 24GB cards like the RTX 4090 or a used RTX 3090. You keep the agentic strengths and the permissive license, at a hardware cost measured in hundreds or low thousands of dollars instead of a cluster.

Related guides on Best GPU for LLM

The full version lives on Best GPU for LLM — VRAM calculator, GPU comparison table, and live Amazon pricing.

Best GPU for FLUX 3: What to Buy Days After Launch

Thurmon Demich — Wed, 29 Jul 2026 01:14:21 +0000

From the Best GPU for AI archive. The canonical version has interactive calculators, an up-to-date GPU comparison table, and live pricing.

Black Forest Labs shipped FLUX 3 on July 23, 2026 — a unified model that generates image, video, and audio from one set of open weights. So which GPU actually runs it? Here's the uncomfortable truth: two days after release, nobody outside BFL and their launch partners can answer that with hard numbers. What I can do is separate what's confirmed from what the Flux family's track record strongly implies, and tell you which cards are safe to buy right now without regretting it once real benchmarks land.

What's confirmed vs. what we expect

Confirmed, as of late July 2026:

FLUX 3 released July 23, 2026, with open weights.
It's genuinely multimodal — image, video, and audio generation in a single unified model, not three bolted-together checkpoints.

That's it. No official VRAM requirements table, no ComfyUI reference workflow with published memory footprints, no third-party benchmark sweep. Any article claiming exact FLUX 3 generation times this week is making them up.

Everything below is inference from how this family has behaved before — and Flux has been unusually consistent:

Flux 2 Dev wanted roughly 24GB in FP16 and around 12GB in FP8. A unified multimodal model almost certainly sits above that, because video generation carries temporal layers and much larger latent buffers than still images.
Flux 2 Klein squeezed a 4B distilled variant into about 10GB. BFL has shipped a small distilled sibling for every major release, and I'd be surprised if FLUX 3 broke the pattern.
NVIDIA shipped NVFP4 and FP8 Flux 2 variants within weeks of launch, with roughly 2.5-3x speedups and a 55-60% VRAM cut — but the NVFP4 gains were Blackwell-only.

My read: expect image-only FLUX 3 inference to land near Flux 2's footprint, and expect the video and audio paths to push well past 24GB in full precision until quantized checkpoints arrive. If you ran the numbers for Flux 2 hardware, treat those as the floor, not the target.

VRAM chart available at the original article

Expected hardware by workload

Labeled clearly as projection — these bands come from Flux 2's measured behavior plus the overhead multimodal architectures typically add:

FLUX 3 workload	Expected VRAM band	GPU pick today	Price
Image-only, quantized (FP8/NVFP4)	~12-16GB	RTX 5070 Ti 16GB	~$750
Image-only, full precision	~24-28GB	RTX 4090 24GB	~$1,600
Image + video pipeline, quantized	~20-28GB	RTX 4090 24GB	~$1,600
Image + video + audio, full pipeline	~28-40GB+	RTX 5090 32GB	~$2,000
Distilled "Klein-class" variant (if it comes)	~10-12GB	RTX 4070 Ti Super 16GB	~$700

The pattern worth internalizing: image generation is the cheap part. The moment FLUX 3's video path enters your workflow, VRAM demand jumps a tier, because you're holding dozens of frames of latents plus temporal attention state instead of one. Audio is comparatively light, but it stacks on top of everything else in a unified pipeline.

Why Blackwell NVFP4 matters more this time

When NVIDIA optimized Flux 2, the FP8 path helped every card from Ada up — but the NVFP4 checkpoints, with their 2.5-3x throughput gains and roughly 55-60% VRAM reduction, only ran on Blackwell tensor cores. If NVIDIA repeats that playbook for FLUX 3 (and a flagship open-weights multimodal release is exactly the kind of model they optimize first), the gap between RTX 40-series and 50-series widens from "somewhat faster" to "runs workloads the older card physically can't fit."

That's the strongest argument for buying Blackwell today even without FLUX 3 benchmarks. A 16GB RTX 5070 Ti with a future NVFP4 checkpoint could plausibly fit a video pipeline that a 16GB RTX 4070 Ti Super never will. I covered how this dynamic played out between the flagships in RTX 5090 vs 4090 for Flux 2 — the short version is that architecture generation started mattering as much as raw VRAM, and FLUX 3 will likely push that further.

Which GPU should you buy today?

You want zero regret and you'll use the full multimodal pipeline: RTX 5090, ~$2,000. 32GB covers the worst-case VRAM projection and Blackwell catches every NVIDIA optimization. This is the only unconditional recommendation I can make this week.
You mainly generate images and video is a curiosity: RTX 4090, ~$1,600. 24GB runs any plausible quantized FLUX 3 image workload and probably a quantized video path. You give up future NVFP4 checkpoints — go in with eyes open.
You're budget-limited but want to stay Blackwell: RTX 5070 Ti, ~$750. 16GB is my floor for FLUX 3; below that you're betting everything on a distilled variant that doesn't exist yet.
You find a discounted RTX 4070 Ti Super: Fine for image-only work at ~$700, and it runs Flux 2 beautifully today. Just accept it's probably locked out of FLUX 3's video path long-term.
You're still happily on Flux 1: No urgency. My original Flux buyer's guide picks all still hold for that model, and FLUX 3's requirements will be far clearer in a month.

There's a fifth option I'd genuinely consider right now: rent. An hour of cloud RTX 5090 or A100 time costs a few dollars, which is a cheap way to test FLUX 3's actual behavior on your workflows before committing $2,000 to a guess. Our GPU rental guide covers the cost math.

Common mistakes in week one

Buying an 8-12GB card for a multimodal model. FLUX 3's video path will not fit, quantized or not, on 8GB — and 12GB looks marginal even for images once you add control modules. 16GB is the realistic floor.
Assuming your Flux 2 workflow transfers. Every major Flux release changed the text encoder pairing, sampler recommendations, or node graph. Expect the same, and wait for official example workflows instead of debugging a Franken-graph.
Waiting indefinitely for perfect information. If your current card can't run Flux 2 comfortably, it can't run FLUX 3 either — the upgrade decision doesn't actually depend on benchmarks that haven't shipped.
Panic-selling a 4090. It's still a 24GB card that runs everything released before last week. Wait for real FLUX 3 numbers before eating a used-market loss.

Final verdict

Budget	GPU	FLUX 3 outlook (projected)
~$700	RTX 4070 Ti Super 16GB	Image-only, quantized; likely no video path
~$750	RTX 5070 Ti 16GB	Image now; NVFP4 could unlock quantized video
~$1,600	RTX 4090 24GB	Full-precision images, quantized video plausible
~$2,000	RTX 5090 32GB	Full multimodal pipeline, catches every optimization

FLUX 3 is two days old and the honest hardware advice is conditional: buy 32GB if you refuse to gamble, 16GB Blackwell if you're betting on quantization, and rent by the hour if you'd rather let the benchmarks settle first.

Frequently asked questions

How much VRAM will FLUX 3 need?

Nobody has published official requirements as of late July 2026. Based on Flux 2's measured footprint and typical multimodal overhead, expect roughly 12-16GB for quantized image-only work, somewhere past 24GB for video pipelines in higher precision, and comfortable headroom only at 32GB. Treat these as projections until real benchmarks land.

Can FLUX 3 run on an RTX 4090?

Very likely yes for image generation, and plausibly for quantized video work, given the 4090 handled Flux 2 in FP16 with room to spare. The open question is optimization: if NVIDIA ships Blackwell-only NVFP4 checkpoints as they did for Flux 2, the 4090 keeps working but misses the largest speed and VRAM gains.

Should I wait for FLUX 3 benchmarks before buying a GPU?

If your current card already struggles with Flux 2, no — anything that fails today fails harder on a larger multimodal model, so the upgrade logic is already settled. If you're choosing between two capable cards, waiting a few weeks costs little, and renting cloud GPU time in the meantime lets you test FLUX 3 on real workloads first.

Will FLUX 3 get an FP8 or NVFP4 version?

There's no announcement yet, but the precedent is strong: NVIDIA published optimized Flux 2 variants within weeks of that launch, with big throughput gains and VRAM cuts of over half on Blackwell hardware. A flagship open-weights release is exactly the model class they prioritize, so quantized checkpoints seem probable rather than guaranteed.

Related guides on Best GPU for AI

Continue on Best GPU for AI for the complete guide with interactive calculators and current GPU prices.

RTX 5090 vs H100 for LLM in 2026 ($2K vs $30K Debate)

Thurmon Demich — Tue, 28 Jul 2026 01:14:14 +0000

Cross-posted from Best GPU for LLM — visit the original for our VRAM calculator, GPU comparison table, and current Amazon pricing.

Quick answer: If you're a hobbyist reading this, the answer is RTX 5090. Move on. Don't overthink it. The H100 SXM only earns its 15× price tag in three narrow scenarios — FP8 training, serving 50+ concurrent users, or 256K+ context windows. For everything else, a $2,000 5090 does the same job at consumer wattage.

Who this comparison is actually for

I keep seeing the same question in Reddit threads and Discord servers: "Should I stretch my budget to an H100?" The person asking is almost always a solo developer running Llama 3 8B in Ollama. So let me split the audience up front, because "5090 vs H100" is really two very different questions.

You're in the wrong debate if you're:

A hobbyist running 7B–34B models at home
A solo indie dev serving one client through an API
Anyone whose "server" is a workstation under their desk

You're in the right debate if you're:

A small lab or startup sizing inference infrastructure
Someone planning FP8 training runs on 70B+ models
A developer serving batched requests to dozens of concurrent users
Anyone who thinks "TCO" and "SLA" without having to look them up

The first group buys the 5090. The second group might not. That's the whole article, really — but the details are where the money hides.

Specs side-by-side

Numbers first, opinions second. Here's the raw hardware, no marketing fluff.

Spec	RTX 5090	H100 SXM
VRAM	32GB GDDR7	80GB HBM3
Memory bandwidth	1,792 GB/s	3,350 GB/s
Architecture	Blackwell	Hopper
TGP / power	575W	700W
FP8 support	Native	Native
FP16 TFLOPS	~105	~1,979 (sparse)
NVLink / interconnect	PCIe Gen5 x16	NVLink 900 GB/s
Form factor	Consumer PCIe	SXM5 module
Price (2026)	~$2,000	~$30,000+

Two numbers matter most on that table. Bandwidth (3,350 vs 1,792 GB/s) governs token throughput on large models. VRAM (80 vs 32) governs which models fit at all. The H100 wins both — but by roughly 2× on each axis, at 15× the price. That's the whole cost-per-perf story in one sentence.

VRAM chart available at the original article

Real workload comparison

Specs are a starting point. Real inference numbers are what you actually feel. Here's what I've measured (or pulled from the vLLM and llama.cpp community benchmarks) at the workloads people actually run:

Workload	RTX 5090	H100 SXM
Llama 3 70B Q4_K_M (single request)	~30–40 tok/s	~55–70 tok/s
Llama 3 70B FP8 (batched, 16 users)	~180 tok/s aggregate	~800 tok/s aggregate
Llama 3.1 405B Q4	Needs 4× 5090 (multi-GPU)	Fits single card
1M context KV cache	Caps around ~64K in practice	Handles 256K+ comfortably
FP8 training (7B LoRA)	Works but slow	Native, ~4× faster

Single-user 70B inference on the 5090 is perfectly usable — 30–40 tok/s is faster than most people read. The H100 pulls ahead once you batch requests: the extra bandwidth and Tensor cores let it serve many concurrent users at once instead of one very fast one. That's the split most benchmarks bury.

The 405B row is where the 5090 story gets messy. You physically cannot load Llama 3.1 405B on a single 5090 at Q4 — you're looking at four cards and tensor parallelism through PCIe Gen5. The H100 loads it and moves on. If your product depends on frontier-class dense models, the H100 stops being a luxury and starts being the only path.

Here's the batched-serving math that most single-user benchmarks skip. Under vLLM with continuous batching, the H100's 3,350 GB/s bandwidth and Hopper Tensor cores keep dozens of KV caches hot in parallel. The 5090 has the compute to serve batches too, but 32GB fills fast: at Llama 70B FP8, weights alone eat ~70GB — you can't fit that on a 5090 at all, so you drop to Q4 and lose quality per batched user. The H100 keeps you at FP8 and stacks users. That is the actual moat, not raw tok/s.

Which should YOU buy?

I'll give you the decision tree I actually use when friends ask.

You should buy the RTX 5090 if:

You're a single user running local LLMs for coding, chat, or hobby projects — see our best GPU for LLM server guide for the small-scale serving story
Your workloads top out around 70B Q4 and 32K context, which is where our best GPU for Llama 70B article lives
You want to fine-tune small models (7B–13B LoRAs) without paying cloud rates
You're comparing consumer flagships and want the honest sibling story — read RTX 5090 vs 4090 for LLM next

You should look at the H100 (buy or rent) if:

You need to serve 50+ concurrent users through vLLM — see our best GPU for vLLM writeup for the throughput math
You want native FP8 training runs on 70B+ models without gradient checkpointing acrobatics
Your product hinges on 256K–1M context windows for long documents or codebases
You already run a rack: PSU, cooling, and 208V power are solved problems, not new problems

There's a third path that gets ignored too often: rent the H100. If you only need Hopper class hardware for a specific training run or a launch spike, renting on RunPod costs $2–4/hour and takes exactly zero rack space in your apartment.

Common mistakes I keep seeing

Buying an H100 for solo hobbyist use. I've watched people drop $30K on an H100 to run Qwen 3 32B in Ollama for personal coding assistance. That workload doesn't even saturate a 4070. The 80GB sits mostly empty, the 700W blows a home breaker, and the SXM form factor won't fit any consumer motherboard without a $5K server chassis. Please do not do this.

Underestimating total cost of ownership. The H100 sticker price is $30K, but that's the beginning, not the end. Add a Supermicro or Dell chassis ($4K–8K), redundant 2000W+ PSUs, a proper rack (or rented colocation), 208V circuits if you're serious, and cooling that doesn't sound like a jet engine in your bedroom. Real all-in cost for a single-H100 workstation runs $40K–50K. Compare that to a 5090 in a $1,500 PC — under $4,000 total. That's your actual cost multiplier: closer to 12–13× than 15×.

Ignoring what "context" costs in VRAM. The 5090's 32GB is generous for weights but tight for KV cache. A 70B model at Q4 uses ~35GB with weights alone at long context. The how much VRAM for local LLM guide breaks this down — if you need 128K+ context on a big model regularly, the 5090 will disappoint you and the H100 won't.

Assuming FP8 is a 5090 story. Both cards support FP8 natively. Only the H100 has the tensor throughput and HBM3 bandwidth to make training runs feel snappy — expect roughly 4× the throughput on FP8 gradient passes at 70B. For pure inference, though, 5090 FP8 is genuinely good and gets you most of the way there.

Forgetting about depreciation. A $30K H100 is worth $30K today and quite possibly $12K in eighteen months when the next Hopper successor hits general availability. A $2K 5090 is a $2K 5090 that sells used for $1,400 in the same window. Absolute dollar loss on the H100 dwarfs the entire cost of the 5090. If you're not going to fully amortize the H100 through revenue-generating workloads, you're just holding a very expensive space heater.

Final verdict

Your situation	Best pick	Why
Solo hobbyist, any model up to 70B Q4	RTX 5090	32GB and 1,792 GB/s at consumer price
Small team, occasional big model	5090 + rent H100 on demand	Best cost per launch
Serving 50+ concurrent users	H100 SXM	Batched throughput crushes the 5090
FP8 training on 70B+	H100 SXM	Tensor throughput is the moat
256K+ context production workloads	H100 SXM	KV cache doesn't fit anywhere else
Anyone unsure which bucket they're in	RTX 5090	If you have to ask, it's the 5090

One-sentence verdict: Buy the RTX 5090 unless you're serving batched production traffic or doing FP8 training on 70B+ models — and if you're not sure which of those you are, you're not, so buy the 5090.

Related guides on Best GPU for LLM

Continue on Best GPU for LLM for the complete guide with interactive calculators and current GPU prices.