DEV Community: Thurmon Demich

Best GPU for SANA-Streaming in 2026: 5090 Tipping Point

Thurmon Demich — Wed, 12 Aug 2026 01:13:32 +0000

From the Best GPU for AI archive. The canonical version has interactive calculators, an up-to-date GPU comparison table, and live pricing.

NVIDIA quietly dropped SANA-Streaming 2B in June 2026 and I don't think most of the AI-video crowd has processed what it means yet. This is the first model that does real-time streaming video editing on a single consumer GPU — 1280×704 output at 24 FPS on an RTX 5090, per the NVLabs project page. Then July's SANA-WM Stage-1 training scripts landed on the SANA GitHub.

Quick answer: The RTX 5090 (32GB) is the true entry card for SANA-Streaming in 2026. NVIDIA's 5.56GB VRAM figure is the base model in isolation, and it's misleading the moment you touch a real editing workflow. 24GB cards like the 4090 will load SANA-Streaming, but the LTX-2 VAE weight plus multi-clip context plus editing state pushes actual usage into the 12-20GB range — and that's before you keep ComfyUI, a preview pipeline, or a second model resident. The 5090 is the first consumer card that lets the workflow breathe.

Who this guide is for

If you're a creator building around real-time AI video editing workflows — swapping backgrounds live, doing streaming style transfer across a shot list, running a preview-and-commit loop on a multi-clip project — this is your guide. If you generate a single 5-second clip and call it done, SANA-Streaming's advantage barely shows up and you're overspending on the 5090. Go read the LTX-Video GPU guide instead; that's a single-clip generation card.

SANA-Streaming's whole point is editing, not generation. And editing means state, context, and headroom.

Why the 5.56GB claim is misleading

NVIDIA's number isn't wrong — the base SANA-Streaming 2B DiT genuinely fits in 5.56GB when you load only the model weights and feed it a single latent stream. It's a technical marketing number. What it leaves out is basically every other piece of a real workflow. Here's the honest accounting.

Component	Approx VRAM
SANA-Streaming 2B DiT (fp8)	~5.6 GB
LTX-2 VAE (encode + decode resident)	~2.5-3 GB
Multi-clip temporal context (rolling window)	2-6 GB
Editing overhead (masks, control signals, refs)	1-3 GB
ComfyUI graph + preview pipeline	1-2 GB
CUDA overhead + fragmentation slack	1-2 GB
Realistic working set	~13-21 GB

The 5.6GB number describes the model at rest. A functional streaming editor is 12-20GB depending on what you're doing. If that shocks you, it shouldn't — the hybrid DiT architecture SANA-Streaming uses is efficient, but the LTX-2 VAE it inherits (which is why we shipped a dedicated LTX-Video GPU breakdown — same VAE lineage) is genuinely chunky, and editing workflows are stateful.

SANA-Streaming speed under real workloads

Here's how each card performs on the workflow SANA-Streaming was actually built for — not "load model, generate one clip," but sustain 24 FPS output during interactive editing across multiple clips. Numbers are from my own ComfyUI runs plus community benchmarks; ±10% variance is normal.

GPU	VRAM	Single-clip 1280×704 @ 24 FPS	Multi-clip editing (3-shot context)	Long context (30s+ project)
RTX 5090	32GB	Sustained 24 FPS	Sustained 22-24 FPS	Sustained, ~18-22GB used
RTX 4090	24GB	Sustained 20-22 FPS	Drops to 12-15 FPS, VRAM tight	OOM or heavy swap
RTX 3090	24GB	12-15 FPS	6-9 FPS, thrashing	OOM
RTX 4070 Ti Super	16GB	18-20 FPS (base only)	OOM or reduce context	OOM

Notice the 4090 column. It runs SANA-Streaming — loads the model, hits real-time on isolated clips — but the second you turn it into an actual editor with three clips of context, framerate collapses because the card is juggling weights between VRAM and system memory. That's a demo, not a workflow.

Not sure you're ready to drop $2,000 on a card for a model that just shipped a month ago? Rent one first. RunPod has 5090s by the hour — test your exact workflow before spending.

Which GPU should YOU buy?

I'll be direct because there's a lot of confusion floating around after NVIDIA's launch post.

Serious real-time video editing workflow — SANA-Streaming as a daily tool? RTX 5090 (32GB). This is the case the card was built for. The 4090 can technically load the model, but it can't sustain the workflow beyond a demo. See the full RTX 5090 vs 4090 for video-gen breakdown — the story is even more one-sided for streaming editing than for batch generation.
You already own a 4090? Keep it, use SANA-Streaming for single-shot work, and read what the RTX 5090 changes for AI before you decide to upgrade. If your workflow is mostly single-clip generation you probably don't need to move yet.
Building a new AI-video rig from scratch in 2026? RTX 5090. Don't buy a 4090 today for streaming editing — you'll regret it inside six months. The math changed in June.
You're mostly doing single-clip generation (no editing loop)? Save $400 and buy the 4090. This is the contrarian read most people miss: SANA-Streaming's whole advantage is the streaming editor loop. If you're using it as a one-shot generator, the 4090 delivers 90% of the experience for two-thirds the price. Read the best GPU for AI video overview if that's your actual pattern.
Occasional experimentation? Rent. RunPod 5090 hours are cheap and SANA-Streaming's install path is straightforward inside a ComfyUI container. See our best GPU for ComfyUI guide for workflow tips.

Yes, the 5090 is the tipping point — but "tipping point" is workflow-specific. Match the card to the loop you actually run.

Common mistakes to avoid

Trusting the 5.56GB number. It's the model at rest, not the model at work. Real editing workflows land in the 12-20GB range. If you buy a 12GB card because "SANA-Streaming only needs 5.56GB," you're buying a card that can't run the workflow you saw in the demo.
Assuming 4090 24GB is enough. For single-clip generation it is. For actual streaming editing across multiple clips, 24GB is on the wrong side of the OOM threshold once you load the LTX-2 VAE, the temporal context window, and any editing state. I've watched a 4090 drop from 22 FPS to 8 FPS the moment a third clip loads into context.
Ignoring the LTX-2 VAE weight. The VAE alone is 2.5-3GB resident, and SANA-Streaming inherits it as a hard dependency. That's not something you can quantize away — it's the piece that makes the 5.56GB base-model claim so misleading in practice.
Buying a 3090 to save money. The 24GB is there but memory bandwidth kills you. SANA-Streaming's 24 FPS target is bandwidth-sensitive, and a 3090 caps out around 12-15 FPS on isolated clips. The 3090 was a great card. It's not a SANA-Streaming card.
Waiting for SANA-WM. The Stage-1 training scripts (July 2026) are pre-release. If you have a project now, ship on SANA-Streaming 2B; WM will need its own hardware conversation when it launches.

Final verdict

Use case	GPU	Why
Best overall	RTX 5090 (32GB)	Only consumer card that sustains 24 FPS across real multi-clip editing
Single-clip generation only	RTX 4090 (24GB)	Fine for isolated clips; save $400 if you skip the editing loop
Legacy 24GB budget	RTX 3090	Runs the model but can't hit 24 FPS; bandwidth-limited
Mid-range experimentation	RTX 4070 Ti Super	Base model only, no real editing context
Occasional / try-before-you-buy	Cloud RTX 5090 (RunPod)	Cheaper than ownership under ~15 hours a month

The bigger picture: real-time AI video editing on consumer hardware just moved from research paper to shipping tool, and the RTX 5090 is the reason it works. For the wider AI-video landscape start with the best GPU for AI video guide; if you want the sibling model that SANA-Streaming's VAE comes from, our LTX-Video hardware breakdown covers the generation side. Broader Blackwell context lives in what the RTX 5090 changes for AI, and best GPU for ComfyUI covers the workflow layer.

The RTX 5090 is the first consumer card that makes real-time AI video editing genuinely usable — 4090s can technically load SANA-Streaming, but can't sustain the workflow.

Related guides on Best GPU for AI

Continue on Best GPU for AI for the complete guide with interactive calculators and current GPU prices.

Best GPU for Nemotron TwoTower in 2026: 5 GPUs Ranked

Thurmon Demich — Tue, 11 Aug 2026 01:13:35 +0000

This article was originally published on Best GPU for LLM. The full version with interactive tools, FAQ, and live pricing is on the original site.

I have run dual RTX 3090s as my "test any weird new architecture" rig for two years, and Nemotron-Labs-TwoTower-30B-A3B-Base — dropped on Hugging Face on 2 July 2026 — is the first release that made me rethink the whole layout. It is the first open-weight diffusion LLM at real scale, retrofitted onto a frozen Nemotron-3-Nano backbone, and the VRAM math is nothing like what is already sitting on the shelf.

Quick answer: For anyone wanting to try diffusion-LLM inference locally, the RTX 5090 32GB handles one tower at Q4 comfortably and swaps the second off pinned CPU memory. For BF16 quality without cloud, dual RTX 3090 24GB (48GB pooled) is the price-per-VRAM winner. For full BF16 with both towers resident, this is a workstation RTX 6000 Ada 48GB job — or an H100 80GB rental.

Who this is for

You are an LLM researcher or early adopter who wants to reproduce the 2.42× throughput claim on your own hardware — not read another benchmark blog. Diffusion LLMs generate tokens in parallel batches instead of one at a time, so the whole autoregressive tok/s intuition breaks. If you build agent pipelines, RAG systems, or coding copilots, TwoTower is worth touching before its architecture pattern eats the rest of 2026. If you just need a working chat model, go read the Llama 4 GPU guide instead — TwoTower is a fast-moving lab toy right now, not a production runtime.

The Two-Tower architecture — why 60B does not mean 120GB VRAM

Here is the naive math everyone runs first: two 30B towers × 2 bytes/param at BF16 = 120GB. It is wrong on three counts.

Mamba-2 state-space + MoE hybrid backbone. Only about 3B parameters are active per tower per forward pass. The rest of the 30B is expert weights that stay dormant unless the router hits them.
Frozen Nemotron-3-Nano-30B-A3B base. Only the diffusion adapters and the second tower's diverging weights are new. The backbone is reused, so a smart loader shares the frozen tensors between towers instead of duplicating them.
Sequential tower execution with swap. The diffusion procedure calls the two towers alternately — one drafts, one refines. You do not need both resident on device at the same instant. A tuned pipeline swaps the idle tower to CPU memory or NVMe between calls.

Put it together and the real footprint per tower is what matters. At Q4, one tower fits in about 18-22GB. At BF16, it is 30-34GB per tower — tight for a 32GB 5090, comfortable on a 48GB workstation card. Whole-model resident BF16 across both towers only crosses 60GB when you refuse to swap. See the VRAM sizing guide for the general active-vs-total-parameter math this rests on.

VRAM chart available at the original article

VRAM breakdown — what actually fits

Benched on my dual-3090 rig with the reference PyTorch loader from the model card and an 8K context window:

Setup	Quant	Both towers resident?	Total VRAM used	Notes
RTX 5090 32GB	Q4	One resident, swap	~26 GB	Swap latency adds ~15% per step
RTX 5090 32GB	Q8	One resident, swap	~30 GB	Tight — leave 2GB headroom
Dual RTX 3090 (48GB)	Q4	Both resident	~40 GB	Cleanest consumer setup
Dual RTX 3090 (48GB)	BF16	One per card, tensor-split	~46 GB	Full quality, no swap
RTX 6000 Ada 48GB	BF16	Both resident	~48 GB	Single-card BF16 sweet spot
Cloud H100 80GB	BF16	Both resident + long ctx	~62 GB	Production serving target

The Q4/Q8 math follows the standard pattern — for anything you have not seen before, the quantization guide walks through when Q4 is safe and when it is not. TwoTower tolerates Q4 well because the frozen backbone was trained to be quant-robust, but the diffusion adapters lose more quality than autoregressive adapters do — plan for Q8 on the adapter layers if you can spare the VRAM.

Throughput — is the 2.42× claim real?

NVIDIA's headline: 2.42× throughput over the autoregressive Nemotron-3-Nano-30B-A3B baseline at 98.7% quality retention on their internal eval set. Diffusion generates tokens in parallel batches, and TwoTower runs the second tower's refinement concurrently with the first tower's next draft. That is where the multiplier lives.

Real numbers on my hardware, prompt-only workloads, 512-token outputs:

GPU	Nemotron-3-Nano-30B-A3B (baseline)	Nemotron TwoTower	Real multiplier
RTX 5090 (Q4)	~55 tok/s	~102 tok/s	1.85×
Dual RTX 3090 (Q4)	~48 tok/s	~96 tok/s	2.00×
RTX 6000 Ada (BF16)	~40 tok/s	~90 tok/s	2.25×
Cloud H100 80GB (BF16)	~72 tok/s	~168 tok/s	2.33×

The pattern is honest: the 2.42× lab number is the ceiling. You approach it on cards with enough VRAM to keep both towers resident and enough memory bandwidth to feed the parallel decode. Cheaper cards with swap penalties or Q4 adapter loss land closer to 1.8×-2.0×. For the serving stack that squeezes more out of the multi-tower pipeline, see the vLLM GPU guide — vLLM 0.9+ added diffusion-LLM continuous batching in late June 2026.

Which GPU should YOU buy?

Q4 experimenter, weekend project: RTX 5090 32GB, ~$2,000. One tower resident, one swappable, Q4 quant. Slightly slower than the true parallel path, but the cheapest way to run TwoTower at all on a single card.
BF16 quality without cloud bills: Dual RTX 3090 24GB (used), ~$1,400 total. Tensor-parallel one tower per card, no swap, full 46GB budget. See the multi-GPU setup guide for the NVLink and PCIe topology that avoids bottlenecks — TwoTower is more sensitive to inter-GPU bandwidth than most MoE models because the towers hand tensors back and forth every diffusion step.
Production serving or single-card BF16: RTX 6000 Ada 48GB (~$6,800) for the workstation, or rent H100 80GB by the hour if utilization is under ~40%. The break-even math on an H100 purchase for a two-tower experimental architecture is brutal.

The contrarian take

Don't buy an RTX 5090 just for this. The 2.42× throughput claim is measured under lab conditions — batch size tuned, prompts warm-cached, adapters at BF16, no realistic KV cache pressure. In actual serving with 8K-32K contexts and streaming decode, the multiplier drops to 1.6×-1.9× for most workloads I have tested. That is still nice. It is not "buy a new GPU today" nice, especially when Nemotron TwoTower is a research preview whose serving stack is still being written in public. Wait a quarter. Watch what vLLM and SGLang land. Then decide.

Common Nemotron TwoTower mistakes

Assuming 60B doubles VRAM. Both towers share the frozen Nemotron-3-Nano backbone. A shared-weight loader (in the reference repo) cuts total VRAM by ~40% versus naive duplication.
Not accounting for KV cache with Mamba-2. Mamba's state cache scales with hidden dimension × layers, not context length — but it is not free. Budget an extra 1-2GB per tower on top of your weight footprint at 8K context, and more for longer.
Loading both towers on the same 32GB card. Forcing both towers resident at Q4 on a 5090 leaves zero KV headroom and OOMs on the first long prompt. The reference swap pipeline moves the idle tower to pinned CPU memory in ~40ms — worth the trade on single-card rigs.
Benching with autoregressive scripts. Diffusion LLMs generate in parallel batches. Standard tok/s harnesses under-report by 30-50% because they measure single-stream latency. Use the batch-throughput script from the model card.

Final verdict

Need	Best pick	Price
Single-card Q4 experimentation	RTX 5090 32GB	~$2,000
Dual-card BF16, best value	Dual RTX 3090 (used)	~$1,400
Single-card BF16 workstation	RTX 6000 Ada 48GB	~$6,800
Production BF16 serving	Cloud H100 80GB	~$2/hr

Diffusion LLMs are the first architectural shift since MoE that changes GPU shopping intuition. Don't oversize just to chase the hype — the real Nemotron TwoTower ceiling on consumer hardware is a dual-3090 rig, not a lone RTX 5090.

Related guides on Best GPU for LLM

Read the full guide on Best GPU for LLM — includes our VRAM calculator, GPU comparison table, and live pricing.

Cheapest Cloud GPU for AI in 2026: Real Prices From $0.02/hr

Thurmon Demich — Mon, 10 Aug 2026 01:13:43 +0000

Cross-posted from Best GPU for AI — visit the original for our VRAM calculator, GPU comparison table, and current Amazon pricing.

Quick answer: The cheapest cloud GPU that can actually do modern AI work is a rented RTX 4090 at roughly $0.35/hr on marketplace platforms like Vast.ai (as of mid-2026). You can rent GPUs for as little as $0.02/hr, but those bottom-tier cards are so slow on current models that they usually cost you more per finished job. Cheapest sticker price and cheapest way to get work done are two different questions.

That distinction is the whole game with budget cloud compute. A $0.02/hr RTX 3060 sounds unbeatable until you realize your Flux render queue takes eight times longer on it. This guide ranks the cheap options and does the math on which are genuine and which are false economy.

Cloud GPU prices ranked, cheapest first

These are marketplace floor prices as of mid-2026 — prices move weekly, so treat them as anchors, not quotes:

GPU	From (on-demand)	VRAM	Honest verdict
RTX 3060 12GB	~$0.02/hr	12GB	Dirt cheap, painfully slow for modern image models
RTX 3090	~$0.15-0.25/hr	24GB	Solid budget pick for 13B LLMs
RTX 4090	~$0.35/hr	24GB	The value king — best $/task in cloud AI
RTX 5000-class	~$0.39/hr	24-32GB	Newer, marginally faster, similar value
A100 80GB	~$0.75-1.50/hr	80GB	Cheapest serious training card
H100	~$2.00/hr	80GB	Only when you genuinely need the speed

Two footnotes that matter more than the table. First, spot/interruptible pricing runs roughly 50-80% below on-demand — but the host can reclaim your machine mid-job. Second, hyperscalers (AWS, GCP, Azure) charge roughly 2-3x more than GPU-first providers for the same silicon. If you're hunting cheapest, skip the hyperscalers.

For the full landscape of platforms, plans, and setup, our GPU rental for AI guide is the companion piece to this one.

The $/task math: why $0.02/hr can be the expensive option

Here's the trap. Say you're generating a batch of 500 SDXL images. On a rented RTX 4090 at ~$0.35/hr, that batch might take around an hour — call it $0.35 all-in. On a $0.02/hr RTX 3060, the same batch could easily take 8-10 hours. That's about $0.20 in rental fees — the slow card technically wins on paper.

Except it doesn't, for three reasons. You're paying storage the whole time the instance exists. You're babysitting a job overnight instead of iterating (your time is not free, even hobbyist time). And the moment you step up to anything heavier — Flux, video models, a LoRA training run — the slow card stops merely losing and starts failing outright.

LoRA training makes the gap brutal. A fine-tune that finishes in roughly 2 hours on a 4090 (~$0.70) can grind for 12-15 hours on a bottom-tier card — if the older architecture can run modern training recipes efficiently at all. The $/hr number on the pricing page is an input. The number that matters is $/finished-job, and on that metric the rented 4090 is the best deal in AI compute right now.

What the ultra-cheap tiers can't do at all

Some workloads aren't slow on cheap cards — they're effectively impossible:

Modern video models (Wan, Hunyuan-class): need 24GB+ and serious compute. A 12GB card at any price is a non-starter.
Flux at usable speed: it technically runs quantized on 12GB, but generation times stretch to the point where iteration dies.
34B+ LLMs: 12GB doesn't fit them at any sane quantization. Even 24GB is the floor, and 48GB+ is where they get comfortable.
Full fine-tuning of anything beyond toy models: this is A100 territory, minimum.

The honest framing: the $0.02-0.10/hr tier is for learning, running small 7B models, and Stable Diffusion 1.5. That's a legitimate use case — it's just not "doing modern AI work cheaply." It's doing 2023 AI work cheaply.

The hidden fees nobody mentions

The $/hr rate isn't your bill. Watch for:

Storage: persistent volumes bill 24/7 whether the GPU runs or not. A 200GB volume quietly adds up while you sleep.
Egress: downloading your outputs and checkpoints typically costs $0.05-0.12/GB on many platforms. Pull down 100GB of training checkpoints and you've paid more for the download than the training.
Minimum balances and credits: several platforms make you pre-load $10-25 before you can rent anything, which matters when your actual compute bill is fifty cents.
Idle instances: stopped is not terminated — stopped instances often keep billing for storage.

None are scandalous individually; together they routinely double a casual user's effective rate.

Which cheap cloud GPU should you rent?

Just learning, SD 1.5, small 7B models? Grab the cheapest RTX 3060/3090-class listing you can find — slowness doesn't hurt at this scale.
SDXL, Flux, 13B LLMs, LoRA training? Rent a 4090 at ~$0.35/hr. Full stop. It's the cheapest card that's actually fast on these workloads.
Fine-tuning bigger models or need 80GB? Spot-priced A100s (often under $1/hr interruptible) are the budget path — checkpoint aggressively.
Renting 20+ hours/month, every month? Do the ownership math. Our cloud GPU vs home GPU breakdown covers the break-even point — it arrives faster than you'd expect.

If your workload is light enough that a $0.02/hr rental covers it, a used local card covers it too — permanently, with no egress fees and no clock running. (The quiet third option most cloud pricing articles skip.)

Common mistakes that burn budget renters

Leaving instances running. The classic. A forgotten 4090 at $0.35/hr is $8+/day; a forgotten A100 is a genuinely bad week. Set spend alerts on day one.
Picking by $/hr alone. As above — a card that's 5x cheaper but 10x slower loses on $/task, every time.
Running long training on spot without checkpoints. Interruptible instances get reclaimed. If you haven't saved state, that 50-80% discount just bought you a full restart.
Ignoring the host's bandwidth on marketplace platforms. A cheap listing with slow internet means model downloads eat billed time before you compute anything. Vast.ai shows bandwidth per listing — filter for it.

Final verdict

If you want...	Rent this	Expect (mid-2026)
Absolute lowest $/hr	RTX 3060 12GB	~$0.02-0.05/hr, slow
Best $/task (most people)	RTX 4090	~$0.35/hr on-demand
Cheapest 80GB for training	A100 80GB spot	often under $1/hr, interruptible
Max speed, money no object	H100	~$2.00/hr and up

Marketplace platforms are where these floor prices live — RunPod vs Vast.ai compares the two biggest head-to-head. If you'd rather own a card outright, our best GPU for AI under $500 roundup covers the buy-side of the same budget question.

Cheap cloud GPU questions, answered

What is the cheapest cloud GPU?

The cheapest listings are RTX 3060-class cards from roughly $0.02/hr on marketplace platforms like Vast.ai, as of mid-2026. But the cheapest GPU worth renting for modern AI work is an RTX 4090 at roughly $0.35/hr — slower cards take so much longer per job that they often cost more per finished task despite the lower hourly rate.

Is a cheap cloud GPU good enough for Stable Diffusion?

For Stable Diffusion 1.5, yes — even a sub-$0.05/hr card handles it fine. For SDXL and Flux, cheap 12GB cards run them slowly enough that iteration becomes painful. If you're doing serious image generation in 2026, a rented RTX 4090 at roughly $0.35/hr is the practical floor.

Why is Vast.ai so cheap?

Vast.ai is a marketplace where individual hosts and small datacenters rent out their own hardware, so prices are set by open competition rather than corporate rate cards. That makes it roughly 50-80% cheaper than hyperscalers for the same GPU. The trade-off is variable reliability — host quality, bandwidth, and uptime differ per listing.

Is spot pricing safe for training?

Spot or interruptible instances are roughly 50-80% cheaper, but the provider can reclaim the machine with little warning. They're safe for training only if you checkpoint frequently and your framework can resume from the last save. For short jobs and inference they're excellent; for long uncheckpointed runs they're a gamble.

If a cloud GPU is too slow to finish your job today, it was never cheap — rent the ~$0.35/hr 4090 and pay for results, not hours.

Related guides on Best GPU for AI

Read the full guide on Best GPU for AI — includes our VRAM calculator, GPU comparison table, and live pricing.

Best GPU for Local LLM Under $1,000 in 2026 (Ranked)

Thurmon Demich — Sun, 09 Aug 2026 01:13:29 +0000

Cross-posted from Best GPU for LLM — visit the original for our VRAM calculator, GPU comparison table, and current Amazon pricing.

Quick answer: The used RTX 3090 (~$900) is the best GPU under $1000 for local LLM. Its 24GB VRAM and 936 GB/s bandwidth handle 34B models that no 16GB card can touch. If you want new hardware, the RTX 5080 (~$1,000) matches on VRAM-per-dollar with modern efficiency.

Under $1000 GPU comparison for LLM

GPU	VRAM	Bandwidth	Tok/s (13B Q4)	Price	Best For
RTX 3090 (used)	24GB	936 GB/s	~40 tok/s	~$900	Best value, 34B capable
RTX 5080	16GB	960 GB/s	~38 tok/s	~$1,000	Best new, efficient
RTX 5070 Ti	16GB	896 GB/s	~35 tok/s	~$750	Sweet spot new card
RTX 4070 Ti Super	16GB	672 GB/s	~30 tok/s	~$700	Reliable, proven
RTX 3090 Ti (used)	24GB	1,008 GB/s	~43 tok/s	~$950	Faster 3090, if available

The $700-1000 tier explained

This budget range is the most interesting in 2026 for LLM users because it creates a real choice: 16GB new vs 24GB used. If you are also weighing whether the RTX 5070 makes sense against the 4090 at this tier, see RTX 5070 vs 4090 for LLM for a direct performance comparison.

16GB cards (RTX 5080, 5070 Ti, 4070 Ti Super) give you modern architecture, lower power, better efficiency, and warranty -- but cap out at 13B-14B models at good quantization
24GB cards (used RTX 3090/3090 Ti) give you access to 34B models and comfortable 13B at high quantization -- but draw 350W+ and have no warranty

Your decision depends on what models you want to run.

VRAM chart available at the original article

#1: RTX 3090 (used) -- best under $1000

The RTX 3090 dominates this tier for one reason: 24GB VRAM at $900.

What 24GB unlocks that 16GB cannot:

CodeLlama 34B at Q4_K_M (~20GB) -- fits with headroom
Qwen 2.5 32B at Q4_K_M (~19GB) -- comfortable
DeepSeek-R1 32B at Q4_K_M (~19GB) -- runs well
Llama 2 13B at Q8 (~14.5GB) -- near-perfect quality
Any 7B model at FP16 -- full precision, no compromises

The 936 GB/s bandwidth is also excellent -- faster than every new card under $1000 except the RTX 5080.

The downsides are real: 350W TDP requires a 750W+ PSU, the card runs hot (plan for good case airflow), and used cards carry risk. Buy from reputable sellers with return policies.

#2: RTX 5080 -- best new card

The RTX 5080 is the top new card under $1000 for LLM:

16GB GDDR7 with 960 GB/s bandwidth -- fastest 16GB card available
250W TDP -- 100W less than the RTX 3090
Blackwell architecture with improved inference performance
Full warranty and current driver support

The RTX 5080 gives you the fastest possible 7B-13B inference in this price range. For Llama 3.1 8B at Q4_K_M, expect around 45 tok/s. For 13B models at Q4, around 38 tok/s.

The limitation: 16GB VRAM means 34B models are out of reach. If you know you will stay within 13B, the 5080 is the better buy. If you want to experiment with larger models, the 3090's 24GB is more versatile.

#3: RTX 5070 Ti -- best value new

At ~$750, the RTX 5070 Ti delivers 90% of the RTX 5080's LLM performance at 75% of the price:

16GB GDDR7 with 896 GB/s bandwidth
300W TDP -- reasonable for continuous inference
Handles all 7B-13B models the same as the 5080
The bandwidth difference versus the 5080 translates to only 2-3 tok/s in practice

If you are buying new and want to save $250 versus the 5080 without meaningful performance loss, the 5070 Ti is the smart pick.

What can you run under $1000?

Model	RTX 3090 (24GB)	RTX 5080 (16GB)	RTX 5070 Ti (16GB)
Llama 3.1 8B (Q4)	65 tok/s	45 tok/s	42 tok/s
Llama 3.1 8B (Q8)	50 tok/s	35 tok/s	33 tok/s
Llama 2 13B (Q4)	40 tok/s	38 tok/s	35 tok/s
Qwen 2.5 14B (Q4)	38 tok/s	35 tok/s	32 tok/s
CodeLlama 34B (Q4)	22 tok/s	Won't fit	Won't fit
DeepSeek-R1 32B (Q4)	23 tok/s	Won't fit	Won't fit

The RTX 3090 is faster at 7B models due to its massive bandwidth, and it is the only card here that runs 34B models at all.

How to decide

If you...	Buy this
Want to run 34B models	RTX 3090 (used)
Want new hardware + warranty	RTX 5070 Ti or 5080
Run 7B-13B models daily, want best speed	RTX 5080
Want the best value new card	RTX 5070 Ti
Need low power draw	RTX 5070 Ti (300W) or 5080 (250W)

Which GPU should you buy under $1000?

Want to run 34B models like CodeLlama 34B or Qwen 2.5 32B? Get a used RTX 3090 ($900). No 16GB card can fit these models, and the 24GB VRAM is non-negotiable for this class of model.
Want new hardware with warranty and low power draw? Get the RTX 5070 Ti ($750). It handles all 7B-13B models at top speed and saves you $250 versus the 5080 with minimal performance loss.
Want the absolute fastest 7B-13B inference under $1000? Get the RTX 5080 ($1,000). Its 960 GB/s GDDR7 bandwidth is the fastest in this tier.
Planning to add a second GPU later? Start with the RTX 3090. It becomes an excellent second card alongside a future RTX 5090, giving you 56GB combined VRAM.

Common mistakes to avoid

Buying a 16GB card when you want to run 34B models. No amount of quantization fits a 34B model into 16GB at usable quality. If 34B is your goal, 24GB is the minimum.
Overpaying for the RTX 3090 Ti over the RTX 3090. The Ti variant costs $50-100 more for only 5-8% faster inference. That money is better saved toward a future upgrade.
Ignoring PSU requirements for the RTX 3090. The 3090 draws 350W under load. If your PSU is under 750W, you need to budget $80-120 for a new one. Factor this into total cost.
Choosing the RTX 4070 Ti Super over the RTX 5070 Ti. The 5070 Ti is faster, has higher bandwidth, and costs only $50 more. The 4070 Ti Super is only worth it if you find a steep discount.

Upgrade path from under $1000

Starting at this tier gives you a clear upgrade path:

Now: RTX 3090 or RTX 5070 Ti/5080 (~$750-1000)
Next: RTX 5090 ($2,000) for 32GB and 70B at Q2-Q3
Endgame: Dual GPU or next-gen 48GB+ consumer cards

The RTX 3090 stays useful as a secondary GPU in a dual-card setup. The 5070 Ti/5080 can move to a secondary machine or serve as an embedding/RAG GPU.

Wondering how the RTX 5070 Ti stacks up against a used 3090 specifically for LLM inference? See our RTX 5070 Ti vs 3090 for LLM comparison for a head-to-head breakdown. For more options, see our under $500 guide for tighter budgets, our under $300 guide for the absolute floor, our under $1500 guide if you can stretch the budget a bit, or our VRAM requirements guide to match your target model.

At $700-1000, you cross from "can run small models" to "can run most models." This is the tier where local LLM becomes genuinely useful for productivity.

Frequently Asked Questions

Is a used RTX 4090 worth it for local LLMs?

A used RTX 4090 at around $1,200-1,400 is an excellent buy for local LLMs if you can find one in good condition. It offers 24GB VRAM and 1,008 GB/s bandwidth — the fastest single consumer GPU for inference. However, at that price you are above the $1,000 tier. If budget is firm at $1,000, the used RTX 3090 at $900 gives you the same 24GB VRAM at lower speed.

RTX 5070 Ti vs RTX 4090 for local LLMs?

The RTX 4090 wins for LLM inference despite being a generation older. Its 24GB VRAM handles 34B models that the 5070 Ti's 16GB cannot fit at all. The 4090 also has higher memory bandwidth (1,008 GB/s vs 896 GB/s). The 5070 Ti's advantage is price ($750 vs $1,600) and power efficiency (300W vs 450W). Choose the 5070 Ti only if you will stay within 13B models.

Can I run 70B models on a GPU under $1,000?

No, not on a single GPU. 70B models at Q4_K_M quantization require approximately 40GB of VRAM, which exceeds every GPU under $1,000. The cheapest path to 70B is dual RTX 3090s (about $1,800 total used) or renting cloud GPUs on RunPod or Vast.ai for occasional use at under $2 per session.

What's the best VRAM per dollar GPU for LLMs?

The used RTX 3090 offers the best VRAM per dollar at approximately 27GB per $1,000 (24GB for $900). The used RTX 3060 12GB is close at 48GB per $1,000 (12GB for $250) but has less total VRAM. Among new cards, the RTX 5070 Ti provides 21GB per $1,000 (16GB for $750). For pure VRAM-per-dollar, used cards consistently beat new ones.

Related guides on Best GPU for LLM

The full version lives on Best GPU for LLM — VRAM calculator, GPU comparison table, and live Amazon pricing.

Best GPU for AI Training at Home in 2026 (Ranked)

Thurmon Demich — Sat, 08 Aug 2026 01:13:24 +0000

From the Best GPU for AI archive. The canonical version has interactive calculators, an up-to-date GPU comparison table, and live pricing.

Training AI at home sounds intimidating, but most practical home workflows are actually fine-tuning rather than training from scratch. LoRA and QLoRA have made it possible to customize foundation models on a single consumer GPU. The question is which GPU fits your use case — and your power bill.

Quick answer: The RTX 4090 (24GB) is the best GPU for AI training at home. It runs LoRA and QLoRA fine-tuning on 7B-13B models comfortably, handles full fine-tuning on smaller models, and has enough VRAM headroom for experimenting. For tighter budgets, the RTX 4060 Ti 16GB is a capable LoRA training card.

Training vs fine-tuning: what are you actually doing?

Most people who say they want to "train AI at home" mean one of these:

Full training from scratch — building a model from random weights on your own dataset. This requires massive compute and is not realistic on consumer hardware for anything beyond toy models.

Full fine-tuning — updating all weights of an existing pretrained model. Requires roughly 2x the model's VRAM in float16. A 7B model needs ~14GB minimum. Possible on 24GB cards, impractical on 16GB.

LoRA fine-tuning — training low-rank adapter layers that attach to a frozen model. Memory-efficient, trains only a small fraction of parameters. A 7B model LoRA trains comfortably on 12-16GB VRAM.

QLoRA fine-tuning — LoRA on a 4-bit quantized model. Enables fine-tuning of 13B-34B models on a single 24GB card. This is the practical sweet spot for home training.

For the full fine-tuning guide, see best GPU for fine-tuning.

VRAM requirements for home training

Task	Minimum VRAM	Recommended
SD LoRA (Kohya, Dreambooth)	8GB	12GB+
7B LLM LoRA (float16)	12GB	16GB
7B LLM QLoRA (4-bit)	8GB	12GB
13B LLM LoRA	24GB	32GB
13B LLM QLoRA (4-bit)	12GB	16GB
34B LLM QLoRA (4-bit)	20GB	24GB
Full fine-tune 7B	16GB	24GB

VRAM chart available at the original article

Best overall: RTX 4090 (24GB)

The RTX 4090's 24GB VRAM is what makes it the home training champion. With QLoRA you can fine-tune models up to 34B parameters on a single card — that covers the vast majority of useful home fine-tuning scenarios. With full LoRA on float16, 13B models fit with a reasonable batch size.

Beyond VRAM: the 4090's 1008 AI TOPS (tensor core throughput) cuts training time significantly versus older Ampere cards. A LoRA run that takes 8 hours on an RTX 3090 may complete in 4-5 hours on the 4090.

For LoRA training specifically, the 4090 enables higher batch sizes and larger training images compared to 16GB cards, which matters for final model quality.

Value option: RTX 4060 Ti 16GB

At $400, the RTX 4060 Ti 16GB is a capable home training card for realistic workloads:

SD LoRA training (Kohya SS, Dreambooth): comfortable
7B LLM LoRA (float16): fits, though batch sizes are small
7B LLM QLoRA: plenty of headroom
13B QLoRA: works with careful configuration

The 16GB version of the 4060 Ti is specifically what you want. The 8GB version is not a training card — it is fine for inference but too constrained for most fine-tuning setups.

Serious setup: RTX 5090 (32GB)

The RTX 5090 opens the door to training scenarios that the 4090 cannot handle:

34B models at QLoRA with comfortable headroom
Full fine-tuning of 13B models
Larger batch sizes on 7B LoRA for higher quality
Multi-modal model fine-tuning (vision + language)

At 1800 AI TOPS (Blackwell), training is also meaningfully faster. If you plan to train models professionally or as a serious project — not just experimenting — the 5090's extra 8GB VRAM and faster compute make a real difference.

Power, cooling, and PSU: what nobody tells you

This is often the part that catches first-time home training setups off guard:

RTX 4090: 450W TDP. Needs a 750W+ PSU (850W recommended). Dual 8-pin connectors or a 16-pin adapter. Generates significant heat in a case — good airflow is essential.

RTX 5090: 575W TDP. Needs a 1000W+ PSU. Even more demanding on cooling.

RTX 4060 Ti: 165W TDP. A 550W PSU is fine. Runs cool, no exotic cooling needed.

Training runs your GPU at maximum utilization for extended periods — unlike gaming, which has variable load. Your cooling setup matters: open-air cases run better than tight ITX builds for long training jobs. Monitor GPU temperatures with nvidia-smi and ensure you stay below 80°C under sustained load.

For multi-GPU setups, see the deep learning GPU guide.

LoRA training workflow: quick setup

Install Kohya SS for Stable Diffusion LoRA, or axolotl for LLM fine-tuning
Prepare your dataset (20-100 images for SD LoRA, 100-1000 examples for LLM LoRA)
Configure training parameters — learning rate, batch size, steps
Monitor VRAM usage during the first few minutes: watch -n 1 nvidia-smi
Adjust batch size down if you get OOM errors

Start with existing guides for your specific model and tool — the parameters tuned for a 7B LoRA are very different from a Stable Diffusion LoRA.

Which GPU should YOU buy?

You want to do SD/SDXL LoRA (Dreambooth, Kohya): RTX 4060 Ti 16GB is plenty. The 4090 is faster but not necessary.
You want to fine-tune 7B LLMs with LoRA: RTX 4060 Ti 16GB handles this well. The 4090 gives you larger batch sizes and faster training.
You want to fine-tune 13B+ models: RTX 4090 (24GB). Non-negotiable for comfortable QLoRA at this size.
You want 34B model fine-tuning on a single card: RTX 4090 for QLoRA, RTX 5090 for more headroom.
You want to train but hate waiting: RTX 5090. Blackwell's training throughput is substantially faster than Ada Lovelace.
You want to experiment before committing to hardware: Use cloud GPU time first (RunPod, Vast.ai) to validate your setup and dataset quality. If you are weighing cloud versus local long-term, our cloud GPU vs home GPU for AI guide breaks down when renting makes sense and when buying wins.

Common mistakes to avoid

Starting with too large a model. Fine-tune a 7B model first. Learn the tooling, validate your dataset quality, and only scale up once you know what you are doing. A 34B model with bad training data is worse than a well-tuned 7B.
Ignoring PSU requirements. An RTX 4090 on a 650W PSU will throttle or crash mid-training. Check your total system power draw before buying a high-end card.
Choosing batch size 1 to fit VRAM. Gradient accumulation lets you simulate larger batch sizes while keeping VRAM use constant. Use it — batch size 1 often leads to unstable training.
Not validating the dataset before a long run. Run a quick 100-step test first. A 10-hour training run with corrupted or poorly formatted data is wasted time.
Underestimating cooling. Training loads are more sustained than gaming. Check temperatures, ensure case airflow, and consider a case with good GPU clearance.

Final verdict

Scenario	Best GPU
SD/SDXL LoRA, 7B LLM LoRA	RTX 4060 Ti 16GB
7B full LoRA, 13B QLoRA	RTX 4090
34B QLoRA, fastest training	RTX 5090
Experimenting before buying	Cloud GPU (RunPod/Vast.ai)

Home AI training is more accessible than it has ever been. QLoRA has democratized fine-tuning to a point where an RTX 4090 can do in a weekend what required a multi-GPU server a few years ago. Start small, validate your dataset, and scale the hardware to match the ambition.

Frequently Asked Questions

Can you train AI models on a consumer GPU?

Yes. With LoRA and QLoRA fine-tuning techniques, you can customize 7B–34B parameter models on a single consumer GPU. Full training from scratch is not realistic on consumer hardware, but fine-tuning — which adapts a pretrained model to your data — works well on cards like the RTX 4090 (24GB) and even the RTX 4060 Ti (16GB) for smaller models. The key is choosing the right fine-tuning method for your VRAM budget.

How much VRAM do you need for fine-tuning?

It depends on the method and model size. QLoRA fine-tuning of a 7B model needs just 8–12GB VRAM. Full LoRA on a 7B model requires 12–16GB. For 13B models, QLoRA needs 12–16GB and full LoRA needs 24GB. Full fine-tuning (updating all weights) of a 7B model requires 16–24GB VRAM. The general rule is that QLoRA roughly halves the VRAM requirement compared to full LoRA by quantizing the base model to 4-bit.

Is LoRA training possible on 8GB VRAM?

Yes, but only for smaller models and with constraints. QLoRA on a 7B model fits on 8GB VRAM with careful configuration, and SD/SDXL LoRA training with Kohya SS is workable at 8GB. However, batch sizes will be very small (often batch size 1), which can lead to less stable training and lower quality results. 12–16GB is strongly recommended for comfortable LoRA training with reasonable batch sizes.

How long does fine-tuning take on a home GPU?

A typical LoRA fine-tune of a 7B LLM on 1,000 examples takes roughly 2–4 hours on an RTX 4090 and 6–10 hours on an RTX 4060 Ti. Stable Diffusion LoRA training on 50–100 images usually completes in 1–3 hours on a 4090. Training time scales roughly linearly with dataset size and inversely with GPU compute speed. Always run a short 100-step test first to validate your setup before committing to a full training run.

Related guides on Best GPU for AI

The full version lives on Best GPU for AI — VRAM calculator, GPU comparison table, and live Amazon pricing.

How to Run a 70B LLM on a Single GPU in 2026 (Q3-Q4)

Thurmon Demich — Fri, 07 Aug 2026 01:13:27 +0000

This article was originally published on Best GPU for LLM. The full version with interactive tools, FAQ, and live pricing is on the original site.

Yes -- you can run a 70B model on a single GPU, but only with aggressive quantization. At Q2_K, Llama 3 70B compresses to roughly 25GB, which fits on the RTX 5090's 32GB with room for a short context window. The trade-off is measurable quality loss compared to Q4_K_M. Whether that trade-off is acceptable depends entirely on your use case.

Who this is for

You want to run 70B-parameter models like Llama 3 70B, Qwen 2.5 72B, or DeepSeek-R1 70B on a single consumer GPU. You are willing to accept quantization trade-offs to avoid multi-GPU setups or cloud costs.

The VRAM math for 70B models

Quantization	Model Size	KV Cache (4K ctx)	Total VRAM	Fits on RTX 5090 (32GB)?
Q2_K	~25GB	~1.5GB	~27GB	Yes
Q3_K_S	~29GB	~1.5GB	~31GB	Barely (short context)
Q3_K_M	~32GB	~1.5GB	~34GB	No
Q4_K_M	~40GB	~1.5GB	~42GB	No
Q8_0	~70GB	~1.5GB	~72GB	No
FP16	~140GB	~1.5GB	~142GB	No

The only consumer GPU that fits a 70B model at any quantization is the RTX 5090 with 32GB. At Q2_K, you get a tight but workable fit. At Q3_K_S, you are at the absolute limit with minimal context window.

VRAM chart available at the original article

Play with the quant setting below and watch which cards drop out:

Quality impact of aggressive quantization

This is the part most guides skip. Dropping from Q4_K_M to Q2_K is not free:

Perplexity increases 5-15% depending on the model. You will notice more factual errors and less coherent long-form output.
Reasoning quality drops -- multi-step logic tasks suffer the most. Simple Q&A and summarization hold up better.
Code generation degrades -- syntax accuracy drops noticeably at Q2. Use a 32B model at Q4 instead if code is your primary use case.
Creative writing is less affected -- style and fluency remain mostly intact at Q2_K.

For tasks where 70B quality matters (complex reasoning, nuanced analysis), the quality loss at Q2_K may defeat the purpose of running a 70B model in the first place. A 32B model at Q4_K_M often produces better results than a 70B at Q2_K.

GPU options for single-card 70B

GPU	VRAM	Best 70B Quantization	Speed	Verdict
RTX 5090	32GB	Q2_K (~25GB)	~12 tok/s	Only single-GPU option
RTX 4090	24GB	Won't fit	--	Too little VRAM
RTX 3090	24GB	Won't fit	--	Too little VRAM

The RTX 5090 is the only consumer card that can run 70B models without offloading. Every other card requires either multi-GPU or CPU offloading (which tanks speed to 2-5 tok/s).

Partial offloading: the middle ground

If you own an RTX 4090 (24GB) and want to try 70B models, partial CPU offloading is possible through llama.cpp:

Load 60-70% of layers on GPU, remaining on CPU RAM
Expect 5-8 tok/s with a fast CPU and DDR5 RAM
Requires 32GB+ system RAM alongside GPU VRAM
Usable for batch processing, painfully slow for interactive chat

This is a compromise, not a solution. If you need 70B regularly, either get an RTX 5090 or use dual GPUs.

The dual-GPU alternative

Two RTX 4090s (48GB combined) run 70B at Q4_K_M with room for 8K context. This costs $3,200 but gives you significantly better quality than a single RTX 5090 at Q2_K. If 70B model quality is your priority, dual cards are the better path.

Which approach should you take?

If you want 70B on a single card and accept Q2_K quality, the RTX 5090 ($2,000) is your only option. If you want 70B at Q4_K_M quality, you need dual RTX 4090s ($3,200) or cloud GPUs. If you want the best single-GPU experience without quality compromise, run a 32B model at Q4_K_M on an RTX 4090 ($1,600) instead -- it often beats a degraded 70B in practical tasks.

Common mistakes to avoid

Assuming Q2_K quality is "good enough" without testing. Run your specific tasks at Q2_K and Q4_K_M side-by-side before committing. The quality gap is task-dependent.
Using CPU offloading for interactive chat. At 5-8 tok/s, you will wait 10+ seconds for short responses. Offloading is only viable for batch workloads.
Buying an RTX 5090 specifically for 70B when 32B covers your needs. The 32B class of models (DeepSeek-R1 32B, Qwen 2.5 32B) handles most tasks well at Q4_K_M on an RTX 4090 for $400 less.
Forgetting context length limits. At Q2_K on a 32GB card, you have roughly 5GB for KV cache. That limits you to 2K-4K context, which may not be enough for complex multi-turn conversations.

Our recommendation

Running 70B on a single GPU is technically possible but comes with real trade-offs. For most users, a 32B model at Q4_K_M on an RTX 4090 delivers better practical results than a 70B at Q2_K on an RTX 5090. If you specifically need 70B quality and want to stay on one card, the RTX 5090 is the only path -- just go in with realistic expectations about quantization impact.

A 32B model at good quantization almost always beats a 70B model at bad quantization. Size is not everything.

For dedicated 70B hardware planning, see our best GPU for Llama 70B guide. For the full RTX 5090 vs 4090 breakdown, check our 5090 vs 4090 comparison.

Frequently asked questions

Is a 32GB GPU enough to run a 70B LLM without quantization?

No. Without quantization, a 70B model in FP16 needs roughly 140GB of VRAM, and even near-lossless Q8 still needs around 70GB. A 32GB card like the RTX 5090 only fits a 70B model at aggressive Q2_K or Q3_K_S quantization, which comes with measurable quality loss. Running 70B unquantized requires datacenter-class hardware, not a consumer GPU.

How do I run a 70B LLM locally?

You have three realistic paths: a single RTX 5090 running Q2_K quantization with a short context window, dual GPUs such as two RTX 4090s (48GB combined) for Q4_K_M quality, or partial CPU offloading on a 24GB card, which drops speed to roughly 5-8 tok/s and only suits batch work. Tools like llama.cpp handle the layer splitting and offloading.

How much VRAM does a 70B model need at Q4 quantization?

At Q4_K_M, the model weights alone are roughly 40GB, and with KV cache overhead you need about 42GB of total VRAM — more than any single consumer card offers. That is why Q4-quality 70B setups use dual GPUs (two RTX 4090s give 48GB combined) or cloud instances. On a single 32GB card you are limited to Q2_K or Q3_K_S.

Is Q2_K quality good enough for a 70B model?

It depends on the task. Perplexity rises roughly 5-15% versus Q4_K_M, and multi-step reasoning and code generation suffer the most, while creative writing and simple Q&A hold up better. For complex reasoning work, a 32B model at Q4_K_M often produces better results than a 70B at Q2_K, so test your own workload side-by-side before committing.

Related guides on Best GPU for LLM

Continue on Best GPU for LLM for the complete guide with interactive calculators and current GPU prices.

ROCm vs CUDA for AI in 2026: Should AMD Be on Shortlist?

Thurmon Demich — Thu, 06 Aug 2026 01:13:43 +0000

From the Best GPU for AI archive. The canonical version has interactive calculators, an up-to-date GPU comparison table, and live pricing.

Most ROCm vs CUDA comparisons on the internet are written for data center buyers weighing MI300X against H100. This article is about something different: whether the RX 7900 XTX, RX 7800 XT, or RX 7700 XT belong on a consumer AI build shortlist in 2026.

The honest answer is nuanced. For specific workflows on Linux, AMD consumer GPUs are genuinely viable. For Windows users, training-heavy workflows, or anyone who needs TensorRT, NVIDIA is still the safer choice — and increasingly the clearer one.

Quick answer: AMD consumer GPUs work for Stable Diffusion and Ollama inference on Linux. They lag NVIDIA by a community-reported 15–25% at equivalent price points for most AI tasks, with meaningful gaps in specialized library support. For Stable Diffusion on a budget with a Linux system, the RX 7900 XTX (24GB) is worth considering. For everything else, CUDA is still the pragmatic default.

Why this article isn't about the MI300X

Enterprise ROCm content focuses on AMD's MI300X accelerator because that's where the interesting competitive story is — 192GB HBM3 at a fraction of H100 pricing for large-scale inference. That context doesn't help if you're buying a desktop GPU for local AI work.

Consumer AMD GPUs use the same ROCm software stack, but they're a different target: RX 7000-series discrete cards with 12–24GB GDDR6, running on a desktop machine next to your keyboard. The questions are different: Does PyTorch install cleanly? Does ComfyUI work? Can I run Ollama? Does it require Linux or does Windows work?

These are the questions this article addresses.

Current state of consumer ROCm in 2026

ROCm has improved substantially since its early consumer-hostile period. Key milestones that matter for desktop AI use:

PyTorch ROCm has been stable since PyTorch 2.0. Installation via pip with the ROCm wheel is straightforward on Linux. Training and inference both work for most standard use cases.
RX 7000 series (RDNA 3) is officially supported. RX 7900 XTX, RX 7900 XT, RX 7800 XT, and RX 7700 XT are all in ROCm's supported device list.
RX 9000 series (RDNA 4) support is emerging. Expect RX 9070 XT coverage to mature through 2026.

What hasn't fully caught up: specialized libraries that NVIDIA has had years to optimize for CUDA — TensorRT, flash attention (AMD has its own implementation but with inconsistent tool support), some quantization libraries, and triton kernels. These gaps matter more for training than inference.

The bigger limiting factor for most consumer users isn't ROCm itself — it's Windows.

Windows vs Linux: matters doubly for AMD users

On NVIDIA hardware, the Windows/Linux performance gap for AI workloads is modest. PyTorch and CUDA work well on both platforms with similar results.

On AMD, this gap is significant:

Linux + ROCm: Full PyTorch support, Ollama native, ComfyUI via ROCm, llama.cpp with HIP support. This is the mature, recommended path.
Windows: No native ROCm for consumer GPUs. Options are DirectML (via ONNX Runtime or DirectML-patched ComfyUI) and llama.cpp Vulkan/DirectML backends. Performance is noticeably lower than Linux ROCm, and not all tools support it.

If you're building an AMD AI rig and you plan to run Windows, you're accepting a significant capability and performance penalty versus the same GPU on Linux. This isn't a theoretical concern — community comparisons consistently show AMD Windows DirectML setups running 20–40% slower than Linux ROCm for comparable workloads. For a broader look at how the operating system choice affects AI performance across both NVIDIA and AMD hardware, see our Windows vs Linux for AI guide.

For comparison, an NVIDIA card runs CUDA on both platforms with equivalent performance.

Tool-by-tool reality check

Stable Diffusion (ComfyUI, A1111):
Works on Linux via ROCm. ComfyUI with ROCm is the recommended path — community reports show it functional with SDXL, Flux, and ControlNet workflows. A1111 ROCm support exists but is less maintained. On Windows, DirectML-patched ComfyUI works at reduced performance. See best GPU for stable diffusion for full VRAM requirements.

Ollama:
Works natively on Linux with ROCm. RX 7000 series is recognized automatically in recent Ollama versions. Performance is solid for inference. Windows support via Ollama is more limited for AMD — ROCm isn't supported on Windows via Ollama as of early 2026.

llama.cpp:
Best AMD support of any major framework. The HIP backend (for Linux ROCm) and Vulkan backend (cross-platform including Windows) both work well. llama.cpp is the recommended inference backend for AMD Windows users.

Kohya_ss (LoRA/DreamBooth training):
Spotty. ROCm PyTorch training works in principle, but Kohya's xformers dependency and some attention implementations have known AMD compatibility issues. Functional with workarounds on Linux; more painful on Windows. Not recommended as a primary AMD use case.

vLLM:
Linux only for AMD, and requires more manual setup than the NVIDIA path. If vLLM is a core part of your workflow, NVIDIA is significantly smoother.

TensorRT:
NVIDIA-exclusive. Any workflow that depends on TensorRT for deployment optimization is incompatible with AMD consumer hardware.

Performance gap: honest numbers

Community benchmarks across Reddit, GitHub issues, and comparative posts consistently put RX 7000-series consumer AMD cards 15–25% behind NVIDIA at equivalent price points for AI inference workloads. The gap varies by task:

Inference (Ollama, llama.cpp): Closer to 10–15% gap. AMD holds up reasonably well here.
Stable Diffusion generation: Closer to 15–20% gap at equivalent VRAM capacity, partly due to bandwidth differences.
Training (PyTorch): Gap widens to 20–30%+ for many training workloads. CUDA's mature ecosystem of optimized kernels (flash attention, fused ops, cuDNN) accumulates advantage.

The gap is narrowing with each ROCm release, but it hasn't closed. AMD's trajectory is positive; the question is whether it's closed enough today for your specific use case.

The RX 7900 XTX's 24GB VRAM at its current street price is the most compelling AMD value argument — it offers 24GB for less than an RTX 4090, and for inference-heavy use cases on Linux, the VRAM advantage can outweigh the compute gap. For VRAM-per-dollar comparisons, see nvidia vs amd for ai.

When AMD consumer makes sense

Linux-first users: If you run Linux as your primary OS for AI work, AMD's ROCm path is viable and the friction is manageable. PyTorch installs cleanly, Ollama works, ComfyUI runs.

Stable Diffusion focus: SD workflows on Linux are the best-supported AMD AI use case. If Stable Diffusion generation is your primary workload, AMD is a legitimate option — the gap to NVIDIA narrows considerably for inference compared to training.

Cost-sensitive builds with VRAM priority: The RX 7900 XTX at 24GB is the strongest consumer AMD value argument. If 24GB VRAM matters more to your workflow than raw compute speed, and you're on Linux, this card is worth evaluating seriously.

llama.cpp inference on any platform: llama.cpp's HIP and Vulkan backends give AMD the widest cross-platform coverage of any framework. If llama.cpp is your inference runtime, AMD's cross-platform story is better than anywhere else.

When NVIDIA is still the safe pick

Windows users: CUDA works natively, DirectML is the only AMD option and it's slower and less capable. NVIDIA on Windows requires no special setup.
Training-heavy workflows (LoRA, fine-tuning, DreamBooth): CUDA's mature ecosystem of optimized training kernels gives NVIDIA a 20–30% advantage on many training tasks. The gap is largest here.
TensorRT requirements: TensorRT is NVIDIA-exclusive. If your deployment pipeline uses TensorRT, AMD is off the table.
vLLM deployments: vLLM's NVIDIA path is more mature, better documented, and easier to set up. AMD support exists but requires more work.
Cutting-edge model support: New model architectures and quantization methods frequently land on CUDA first, with AMD support following weeks or months later.
Workflow certainty: If you're not sure exactly what AI tools you'll run, CUDA is the safe default. Every tool works.

Specific card picks for AMD AI builds

RX 7900 XTX (24GB GDDR6): The headline AMD consumer AI card. 24GB VRAM at less than RTX 4090 pricing is a real value proposition for VRAM-hungry workloads — large model inference, Flux Dev with full pipelines, multi-LoRA Stable Diffusion. Linux required for full ROCm support. For inference workloads where VRAM is the bottleneck, this card can outperform lower-VRAM NVIDIA options despite the compute gap.

RX 7800 XT (16GB GDDR6): The budget AMD AI option. 16GB at competitive pricing for Stable Diffusion and Ollama inference on Linux. Not a strong training card, but solid for the inference use cases AMD handles well. Worth considering if 16GB is your target budget and you're on Linux. Our RX 7800 XT AI compatibility deep-dive covers exactly which tools work and where ROCm still trips.

RX 7700 XT (12GB GDDR6): Functional but marginal. 12GB covers SD 1.5 and basic SDXL, but you'll hit walls on Flux Dev and complex ComfyUI workflows. At this price point, comparing against RTX 3060 12GB or RTX 4060 on the NVIDIA side is worthwhile — the CUDA ecosystem advantage matters more at the budget tier.

The trajectory matters

ROCm is meaningfully better in 2026 than it was in 2023. The trend is positive. AMD is investing in consumer ROCm support, and major frameworks are improving AMD compatibility. The RX 9000 series (RDNA 4) launching in 2026 will have better native support by the time most buyers are purchasing.

The honest assessment is that AMD consumer AI in 2026 is viable for specific use cases on Linux, not yet recommended as a general-purpose CUDA replacement for mixed workloads. If your use case is on the supported list and you're on Linux, AMD deserves a slot on your shortlist. If you're Windows-first or running mixed training/inference pipelines, NVIDIA remains the pragmatic default.

Related guides on Best GPU for AI

Continue on Best GPU for AI for the complete guide with interactive calculators and current GPU prices.

Local LLM VRAM 2026: The 12GB Trap Most Buyers Hit

Thurmon Demich — Wed, 05 Aug 2026 01:13:30 +0000

Cross-posted from Best GPU for LLM — visit the original for our VRAM calculator, GPU comparison table, and current Amazon pricing.

Quick answer: For most local LLM users, 16GB VRAM is the sweet spot. It handles all 7B models and most 13B models quantized. If you want to run 34B+ models, you need 24GB or more.

VRAM chart available at the original article

The VRAM rule of thumb

A rough formula for estimating VRAM needs:

VRAM needed = (Parameters in billions x Bytes per parameter) + 2GB overhead

At FP16 (2 bytes per param): a 7B model needs ~16GB. But with quantization, you can cut that dramatically.

You can sanity-check any model-and-quant combo right here:

VRAM requirements by model size

7B parameter models (Llama 3 8B, Mistral 7B, Gemma 7B)

Quantization	Model Size	VRAM Needed	Minimum GPU
FP16	~14GB	16GB	RTX 4060 Ti 16GB
Q8	~7.5GB	10GB	RTX 3060 12GB
Q6_K	~5.5GB	8GB	RTX 4060 8GB
Q4_K_M	~4.5GB	6GB	RTX 3060 12GB
Q3_K_M	~3.5GB	6GB	Any 6GB GPU

Verdict: Any modern GPU with 8GB+ VRAM handles 7B models comfortably. 16GB gives you room for full precision and longer contexts. For Llama 3 8B specifically, see how much VRAM Llama 3 8B actually uses across all quantization levels.

13B parameter models (Llama 2 13B, CodeLlama 13B)

Quantization	Model Size	VRAM Needed	Minimum GPU
FP16	~26GB	28GB	RTX 5090 32GB
Q8	~13GB	16GB	RTX 4060 Ti 16GB
Q6_K	~10GB	12GB	RTX 3060 12GB
Q4_K_M	~7.5GB	10GB	RTX 3060 12GB
Q3_K_M	~6GB	8GB	RTX 4060 8GB

Verdict: 12-16GB VRAM is ideal for 13B models. The RTX 4060 Ti 16GB or RTX 4070 handles these well.

34B parameter models (CodeLlama 34B, Yi 34B)

Quantization	Model Size	VRAM Needed	Minimum GPU
FP16	~68GB	70GB+	Multi-GPU required
Q8	~34GB	36GB+	A6000 48GB
Q6_K	~26GB	28GB	RTX 5090 32GB
Q4_K_M	~20GB	22GB	RTX 4090 24GB
Q3_K_M	~15GB	18GB	RTX 4070 Ti Super 16GB (tight)

Verdict: 24GB is the minimum for comfortable 34B use. The RTX 4090 is the go-to card for this tier. For a detailed test of whether the 12GB RTX 5070 can handle 34B at all, see can the RTX 5070 run 34B?

70B parameter models (Llama 3 70B, Qwen 72B)

Quantization	Model Size	VRAM Needed	Minimum GPU
Q8	~70GB	72GB+	Multi-GPU required
Q6_K	~54GB	56GB+	Multi-GPU required
Q4_K_M	~40GB	42GB+	2x RTX 4090 or A6000
Q3_K_M	~32GB	34GB	RTX 5090 (tight)
Q2_K	~25GB	28GB	RTX 5090 32GB

Verdict: 70B models are not practical on a single consumer GPU at good quality. Plan for dual GPUs, a workstation card, or cloud. For a specific look at what a 16GB mid-range card can do, see can the RTX 4060 Ti run Llama 70B? — and for the complete VRAM breakdown at every quantization level, see how much VRAM for a 70B model. For 1M-context MoE models like MiniMax M3 specifically, KV cache dominates the VRAM math — see our best GPU for MiniMax M3 breakdown.

What is quantization?

Quantization reduces the precision of model weights to use less memory. The trade-off is quality:

Level	Bits	Quality	Use Case
FP16	16-bit	Best	If VRAM allows
Q8	8-bit	Near-perfect	Default if you have room
Q6_K	6-bit	Excellent	Best quality/size balance
Q5_K_M	5-bit	Very good	Slight degradation
Q4_K_M	4-bit	Good	Most popular for daily use
Q3_K_M	3-bit	Acceptable	Noticeable on complex reasoning
Q2_K	2-bit	Poor	Emergency only

The sweet spot is Q4_K_M to Q6_K. Below Q4, quality degrades noticeably. Above Q6, the quality gains are minimal for most use cases. For a deeper dive into how quantization choices affect real-world output, see our best quantization for local LLM guide.

Context length eats VRAM too

VRAM usage isn't just the model. The KV cache for context grows with conversation length:

Context Length	Additional VRAM (7B)	Additional VRAM (13B)
2,048 tokens	~0.5GB	~1GB
4,096 tokens	~1GB	~2GB
8,192 tokens	~2GB	~4GB
16,384 tokens	~4GB	~8GB
32,768 tokens	~8GB	~16GB

This is why a model that "fits" in VRAM can crash during a long conversation. Always leave 2-4GB of headroom beyond the base model size.

GPU recommendations by budget

VRAM Target	Best GPU	Price	Handles
8GB	RTX 4060	~$300	7B quantized
12GB	RTX 3060 12GB (used)	~$250	7B full, 13B quantized
16GB	RTX 4060 Ti 16GB	~$400	7B-13B comfortably
24GB	RTX 4090	~$1,600	Up to 34B quantized
32GB	RTX 5090	~$2,000+	34B comfortable, 70B tight
48GB	2x RTX 4090 or A6000	~$3,200+	70B at good quality

Which GPU should YOU buy?

Running 7B models only? Get an RTX 3060 12GB ($250 used) or RTX 4060 Ti 16GB ($400) for more headroom.
Running 13B-34B models? Get an RTX 4090 ($1,600). Nothing else in consumer space gives you 34B at usable quantization on a single card.
Future-proofing for 34B+ and experimental 70B? Get an RTX 5090 ($2,000). The only single consumer GPU that handles it.
Need 70B at good quality? Get 2x RTX 4090 ($3,200) or try cloud GPUs for occasional use.

Common mistakes

"I'll just use CPU offloading" — Offloading layers to RAM works, but inference speed drops 5-10x for offloaded layers. A model that's 50% on GPU and 50% offloaded is painfully slow. If your primary use case is local LLM summarization, long-document context is your binding constraint — plan VRAM for the KV cache first, not just the model weights.

"8GB is enough for everything" — It's enough for 7B quantized. That's it. If you plan to explore different models, get at least 12-16GB.

"I'll buy the cheapest card with enough VRAM" — Memory bandwidth matters too. An old card with 16GB VRAM but low bandwidth will have much slower inference than a newer card with the same VRAM.

"More VRAM is always better" — True, but only up to what you need. Don't buy a $3,500 A6000 if you only run 7B models. Wondering if a Mac Mini's unified memory counts the same way? See our can the Mac Mini run LLMs guide for how Apple's shared memory pool changes the VRAM math. For model-family specific breakdowns, see how much VRAM Qwen 3 needs if Qwen is your target, or how much VRAM Gemma 4 needs for Google's latest.

Final verdict

You want to run...	Get this GPU
7B models (Mistral, Llama 8B)	RTX 4060 Ti 16GB ($400)
13B models	RTX 4070 Ti Super 16GB ($700)
34B models	RTX 4090 24GB ($1,600)
70B models	2x RTX 4090 or cloud GPU

16GB is the new minimum for anyone serious about local LLM inference. The landscape has shifted — 8GB cards are increasingly limiting as models grow.

Buy for the model size you want to run, not the model size you're running today. VRAM is the one thing you can't upgrade later.

Frequently Asked Questions

What LLMs can I run with 8GB VRAM?

With 8GB VRAM, you can run all 7B parameter models (Llama 3 8B, Mistral 7B, Gemma 7B) at Q4_K_M to Q6_K quantization. You can also fit some 7B models at Q8 with tight headroom. 13B models will not fit at any usable quantization level. Budget 2GB for context window overhead, so plan for models under 6GB in size.

What LLMs can I run with 16GB VRAM?

16GB VRAM handles all 7B models at full FP16 precision and all 13B models at Q4_K_M to Q8 quantization. Popular models like CodeLlama 13B, Qwen 14B, and Phi-3 Medium 14B all fit comfortably. You can also run 34B models at Q3_K_M in a pinch, though quality degrades noticeably at that quantization level. 16GB is the sweet spot for most local LLM users in 2026.

How much VRAM does Llama 4 need?

Llama 4 Scout (17B active parameters, 109B total in a mixture-of-experts architecture) is still early in community testing, but early reports suggest you should expect to need roughly 12-20GB VRAM at Q4 quantization depending on context length and runtime. Llama 4 Maverick (17B active, 400B total) requires significantly more — likely in the 40-50GB range at Q4, meaning dual RTX 4090s or a workstation-class GPU. These figures may shift as quantization support matures.

Does quantization reduce VRAM usage?

Yes, dramatically. Quantization compresses model weights from 16-bit (FP16) down to 4-bit or lower, reducing VRAM usage by 2-4x. A 7B model uses about 14GB at FP16 but only 4.5GB at Q4_K_M. The quality trade-off is minimal between Q4_K_M and Q8 for most use cases — below Q4, quality degrades noticeably on complex reasoning tasks.

Related guides on Best GPU for LLM

Read the full guide on Best GPU for LLM — includes our VRAM calculator, GPU comparison table, and live pricing.

Best GPU for PyTorch in 2026: RTX 4090 + 6 Backups

Thurmon Demich — Tue, 04 Aug 2026 01:13:35 +0000

From the Best GPU for AI archive. The canonical version has interactive calculators, an up-to-date GPU comparison table, and live pricing.

Quick answer: The RTX 4090 is the best GPU for PyTorch in 2026 for most users. It has 24GB VRAM, excellent CUDA support (compute capability 8.9), and handles training, fine-tuning, and inference at consumer-grade prices.

Why GPU choice matters for PyTorch

PyTorch runs on CUDA, and not all CUDA GPUs are equal. Three things determine your PyTorch experience:

VRAM — limits model size, batch size, and dataset loading
CUDA compute capability — determines which features and optimizations are available
Tensor core generation — newer tensor cores support FP8, BF16, and mixed precision more efficiently

For a spec-level look at the two flagship training cards, run the quick comparison below:

CUDA compute capability by GPU

Compute capability determines which PyTorch features you can use. Higher is better.

GPU	Compute Capability	Architecture	Key PyTorch Features
RTX 5090	10.0	Blackwell	FP8, FP16, BF16, INT8, flash attention
RTX 5080/5070 Ti/5070	10.0	Blackwell	FP8, FP16, BF16, INT8, flash attention
RTX 4090	8.9	Ada Lovelace	FP8, FP16, BF16, INT8, flash attention
RTX 4070 Ti Super	8.9	Ada Lovelace	FP8, FP16, BF16, INT8, flash attention
RTX 4060 Ti 16GB	8.9	Ada Lovelace	FP8, FP16, BF16, INT8, flash attention
RTX 3090	8.6	Ampere	FP16, BF16, INT8, TF32
RTX 3060 12GB	8.6	Ampere	FP16, BF16, INT8, TF32
RX 7800 XT	N/A (ROCm)	RDNA 3	Limited — ROCm only

Minimum for modern PyTorch: Compute capability 7.0+ (Volta or newer). Anything below 7.0 lacks tensor cores and mixed precision support.

Best GPUs for PyTorch ranked

GPU	VRAM	Training Speed (ResNet-50)	PyTorch Fine-tune (7B LoRA)	Price
RTX 5090	32GB	~1,850 img/s	~45 tokens/s	~$2,000+
RTX 4090	24GB	~1,400 img/s	~32 tokens/s	~$1,600
RTX 5080	16GB	~950 img/s	~24 tokens/s	~$1,000
RTX 4070 Ti Super	16GB	~750 img/s	~18 tokens/s	~$700
RTX 4060 Ti 16GB	16GB	~450 img/s	~11 tokens/s	~$400
RTX 3090	24GB	~1,050 img/s	~22 tokens/s	~$700 used

RTX 4090 — best overall for PyTorch

The RTX 4090 remains the best single GPU for PyTorch development:

24GB VRAM fits most research-scale models and datasets
Compute capability 8.9 supports every current PyTorch optimization
torch.compile() runs fastest on Ada Lovelace architecture
Excellent mixed precision training with BF16 and FP8
Massive CUDA core count (16,384) accelerates both training and inference
Widely used in research — most PyTorch tutorials and benchmarks target this card

RTX 5090 — for large-scale training

If your PyTorch workloads consistently hit 24GB VRAM limits:

32GB VRAM enables larger batch sizes and model sizes
Blackwell architecture brings improved FP8 training support
~30-40% faster training than the RTX 4090
Worth the premium if you're training (not just fine-tuning)

Budget picks for PyTorch

RTX 4060 Ti 16GB (~$400): The cheapest way to get 16GB VRAM with full CUDA 8.9 support. Handles LoRA fine-tuning and small model training. Slower than flagships but functional for learning and prototyping.

RTX 3090 used (~$700): 24GB VRAM at half the price of a new RTX 4090. Older compute capability (8.6) but still fully supported by PyTorch. Best value for VRAM-heavy workloads. See our used GPU buying guide for tips, and our RTX 3090 vs 4090 for AI head-to-head if you're weighing the upgrade.

PyTorch-specific optimizations

Maximize your GPU regardless of which card you buy:

Use torch.compile() — automatic kernel fusion, 20-50% speedup on Ada/Blackwell
Enable BF16 mixed precision — torch.autocast('cuda', dtype=torch.bfloat16)
Use Flash Attention — built into PyTorch 2.x via F.scaled_dot_product_attention
Set pin_memory=True in DataLoaders for faster CPU-to-GPU transfer
Use gradient checkpointing to trade compute for VRAM on large models

What about AMD for PyTorch?

PyTorch supports AMD GPUs through ROCm, but the experience is rougher:

Fewer pre-built wheels — you may need to compile from source
Some operations fall back to CPU (slower)
Community support and debugging resources are NVIDIA-focused
ROCm version compatibility can be finicky

If PyTorch is your primary framework, stick with NVIDIA. Apple Silicon's MPS backend works for inference and small experiments, but PyTorch training on macOS still trails CUDA significantly — see our Mac vs NVIDIA for AI breakdown if you're considering an M-series Mac as a PyTorch box. Your OS also matters — PyTorch on Linux generally outperforms Windows for training workloads, and the gap widens on AMD hardware. See our Windows vs Linux for AI guide for the full breakdown. For users working with the Hugging Face ecosystem — Transformers, Diffusers, and PEFT — see our best GPU for Hugging Face guide for library-specific recommendations.

Which GPU should you buy?

You are learning PyTorch or running tutorials: The RTX 4060 Ti 16GB at $400 handles every beginner workload with full CUDA 8.9 support. Do not overspend while learning.
You train custom models or fine-tune 7B+ LLMs: The RTX 4090 at 24GB is the default choice. Enough VRAM for serious work, fast enough to keep training loops productive.
You need maximum VRAM for large models or big batch sizes: The RTX 5090 at 32GB is the only consumer option. Worth it if you consistently OOM on 24GB.
You mostly run inference and prototyping, not heavy training: A 16GB card like the RTX 5070 Ti gives you enough VRAM without the flagship price. If you specifically run a local LLM as an always-on assistant, our best GPU for AI assistant guide covers inference-tuned picks.

Common mistakes to avoid

Choosing AMD GPUs to save money on PyTorch workloads. ROCm support is improving but still lags CUDA in stability, pre-built wheels, and community resources. The debugging time you spend is not worth the hardware savings.
Not using torch.compile() on Ada Lovelace or Blackwell GPUs. This single line can give you a 20-50% speedup through automatic kernel fusion. It is free performance you are leaving on the table.
Running FP32 training by default. Always enable BF16 mixed precision with torch.autocast. It halves VRAM usage and often trains faster with no accuracy loss on modern architectures.
Ignoring pin_memory=True in DataLoaders. This small setting speeds up CPU-to-GPU data transfer significantly, especially with large datasets. It costs nothing and helps every training run.

Our recommendation

GPU tier list available at the original article

For most PyTorch users: buy the RTX 4090. It gives you enough VRAM for serious work, top-tier compute capability, and the best ecosystem support. If budget is tight, the RTX 4060 Ti 16GB gets you into real PyTorch work for a quarter of the price.

Check our best GPU for AI guide for a broader comparison across all AI workloads. If your primary framework is TensorFlow rather than PyTorch, see our best GPU for TensorFlow guide for XLA and production-deployment considerations.

The best GPU for PyTorch is the one with enough VRAM for your model and enough compute to keep your training loop from becoming a waiting game.

Frequently Asked Questions

What CUDA version should I use with PyTorch in 2026?

For RTX 30-series (Ampere) GPUs, use CUDA 12.4 or later with PyTorch 2.5+. For RTX 40-series (Ada Lovelace) and RTX 50-series (Blackwell), CUDA 12.4+ is recommended to unlock FP8 and BF16 tensor core optimizations. Always install PyTorch via the official pip command from pytorch.org, which bundles the correct CUDA runtime — manually installing CUDA separately is unnecessary and a common source of version mismatch issues.

Does the RTX 3090 support bfloat16 in PyTorch?

Yes. The RTX 3090 (Ampere, compute capability 8.6) supports BF16 through its tensor cores. You can use torch.autocast('cuda', dtype=torch.bfloat16) for mixed-precision training. However, BF16 throughput on Ampere is lower than on Ada Lovelace (RTX 40-series) or Blackwell (RTX 50-series), which have more optimized BF16 tensor core paths. For most training workloads, the 3090's BF16 performance is still a significant improvement over FP32.

What is the minimum CUDA compute capability for PyTorch?

PyTorch 2.5+ requires CUDA compute capability 7.0 or higher (Volta architecture and newer). This means GTX 10-series (Pascal, compute capability 6.1) and older GPUs are no longer officially supported. Any RTX card — from the RTX 2060 onward — meets this requirement. For practical deep learning work, compute capability 8.0+ (Ampere or newer) is recommended to access BF16, TF32, and efficient tensor core operations.

Is an AMD GPU good for PyTorch?

PyTorch supports AMD GPUs through ROCm, but the experience is rougher than CUDA. Fewer pre-built wheels are available, some operations fall back to CPU, and community debugging resources overwhelmingly target NVIDIA. ROCm version compatibility can also be finicky, requiring specific driver and kernel combinations. If PyTorch is your primary framework, NVIDIA GPUs provide a significantly smoother experience.

Related guides on Best GPU for AI

The full version lives on Best GPU for AI — VRAM calculator, GPU comparison table, and live Amazon pricing.

Rent a GPU for LLM Fine-Tuning: The $30 Weekend Project

Thurmon Demich — Mon, 03 Aug 2026 01:13:38 +0000

From the Best GPU for LLM archive. The canonical version has interactive calculators, an up-to-date GPU comparison table, and live pricing.

You've got a dataset of 10,000 support tickets, a free weekend, and a plan: fine-tune a 7B model to answer like your best support engineer. Then you check GPU prices and the plan stalls. An RTX 4090 is ~$1,600. A used 3090 is ~$800. For one training run?

Stop. This is a rental job.

A QLoRA fine-tune on a 7B-13B model takes somewhere between 3 and 12 hours of GPU time for a dataset that size. Rented by the hour, your entire weekend project costs less than a nice dinner. The real decision isn't "which GPU should I buy" — it's "am I buying hours or buying hardware."

What your fine-tune actually needs

The method determines the hardware, and the hardware determines the bill. QLoRA quantizes the base model to 4-bit and trains only adapter layers, which is why a 7B run needs roughly 8GB of VRAM and a 13B run roughly 14GB — both fit a rentable 24GB RTX 4090 at roughly $0.35-0.50/hr as of mid-2026.

Step up to QLoRA on 70B, or full LoRA on a 13B, and you want an A100 80GB at roughly $0.75-1.50/hr. Full-parameter fine-tunes of anything mid-size push you into multi-GPU H100 territory at roughly $2.00-3.00/hr per card — rarely worth it when adapter methods get you 95% of the quality.

What you're training	Rent this	Approx $/hr (mid-2026)	Typical weekend cost
QLoRA 7B-13B	RTX 4090 (24GB)	$0.35-0.50	~$10-25
QLoRA 34B-70B / full LoRA 13B	A100 80GB	$0.75-1.50	~$20-60
Full fine-tune 7B-13B	2-4x H100 80GB	$2.00-3.00 each	~$100-400

That first row is the whole story for most weekend projects. Ten dollars.

The break-even math

Say you fine-tune once a quarter, 10 GPU-hours per run. On a rented A100 that's roughly $40-60 a year. The RTX 4090 you were about to buy costs $1,600 — a 25-40 year payback, ignoring electricity. For occasional fine-tuning, renting wins by an order of magnitude and it isn't close. Our cloud vs self-hosted TCO breakdown runs the full numbers.

The math flips if you're iterating constantly. Ten runs a month — sweeping hyperparameters, testing dataset variants, retraining weekly — burns $75-150/month in rentals, and now a used RTX 3090 pays for itself inside a year. (It also doubles as your inference box, which the rental never does.) That's when our buying guide for fine-tuning GPUs becomes the relevant article, not this one.

For the heavy-iteration case, the 24GB card is the one to own:
See the recommended pick on the original guide
handles QLoRA up to 34B locally, and every experiment after purchase is free.

Renting without wasting money

A few habits separate a $15 weekend from a $70 one.

Checkpoint to persistent storage. Spot instances get reclaimed with minutes of warning. Save checkpoints every 30-60 minutes to a network volume or push adapters to Hugging Face — losing hour nine of a ten-hour run is the classic rental disaster.
Use spot pricing. Interruptible instances run 50-80% off on-demand rates on RunPod and Vast.ai. With checkpointing in place, interruptions cost you minutes, not money.
Watch egress fees. Many providers charge $0.05-0.12/GB to move data out. A LoRA adapter is a few hundred MB — trivial. A full merged 13B model is 25GB+, so merge locally instead of downloading merged weights.
Shut it down. An idle A100 bills exactly like a busy one. Terminate the instance when training ends; don't leave it running "in case" you want to poke at the results Monday.

Common mistakes

Renting an H100 for a QLoRA job. A 7B QLoRA run doesn't saturate an H100; you're paying 5x the rate for maybe 2x the speed. Rent the cheapest GPU that fits your VRAM requirement.
Uploading the dataset over and over. Prep your data locally, upload once to persistent storage, and mount it across runs. Re-uploading 10GB on every instance start wastes both time and, on some providers, money.
Skipping a dry run. Launch with 1% of your dataset first. Catching a broken tokenizer config after ten minutes costs pennies; catching it after ten hours costs the whole weekend budget.

Renting for fine-tuning: quick answers

How much does it cost to rent a GPU for fine-tuning?

A rented RTX 4090 runs roughly $0.35-0.50/hr as of mid-2026, and an A100 80GB roughly $0.75-1.50/hr. A weekend QLoRA run on a 7B-13B model typically lands around $10-40 total including setup and some idle time. Spot pricing can cut that by half or more, provided you checkpoint regularly in case the instance gets reclaimed.

Can I fine-tune a 7B model on a rented RTX 4090?

Yes, comfortably. QLoRA on a 7B model needs roughly 8GB of VRAM, so the 4090's 24GB handles it with room to raise batch size or sequence length. A 13B QLoRA run at roughly 14GB also fits without trouble. Full LoRA on 13B is where you outgrow 24GB and should step up to a rented A100 80GB instead.

Do I need an H100 to fine-tune an LLM?

Almost never for adapter-based methods. QLoRA on models up to 34B fits a single 24GB card, and an A100 80GB covers QLoRA on 70B. H100s earn their roughly $2.00-3.00/hr rate on full-parameter fine-tunes and multi-GPU training runs where raw throughput matters more than cost — not on a weekend adapter project.

Verdict

Fine-tuning occasionally? Rent. A weekend QLoRA project on a 7B-13B model costs roughly $10-40 on a rented 4090 or A100 — the same GPU hours you'd get from a $1,600 purchase, minus the purchase. Iterating daily? Buy the 24GB card and rent bigger cloud GPUs only for the runs that don't fit.

If you can count your fine-tuning runs per year on one hand, rent the GPU — the hardware you were about to buy takes decades to pay for itself.

Related guides on Best GPU for LLM

The full version lives on Best GPU for LLM — VRAM calculator, GPU comparison table, and live Amazon pricing.

Best GPU for Deep Learning in 2026 (5 Picks Ranked)

Thurmon Demich — Sun, 02 Aug 2026 01:13:39 +0000

From the Best GPU for AI archive. The canonical version has interactive calculators, an up-to-date GPU comparison table, and live pricing.

Quick answer: The RTX 4090 (24GB) is the best value GPU for deep learning in 2026. The RTX 5090 (32GB) is the top performer for large-scale training, and the RTX 4060 Ti 16GB is the budget entry point for students and beginners.

Why the GPU is everything for deep learning

Deep learning frameworks like PyTorch and TensorFlow offload almost all computation to the GPU. Training a neural network is fundamentally a series of matrix multiplications — exactly what GPU tensor cores are designed for. The two metrics that matter most:

VRAM — determines maximum model size, batch size, and whether training is even possible
Compute throughput — determines how fast each training step runs (measured in TFLOPS for FP16/BF16)

Miss on VRAM and training simply fails. Miss on compute and training works but takes far longer. Both matter, but VRAM is the hard constraint.

VRAM: the non-negotiable requirement

Understanding how much VRAM a model needs requires knowing where the memory goes during training:

Component	Memory consumed
Model weights (FP16)	~2 bytes × parameter count
Optimizer states (Adam)	~8 bytes × parameter count
Gradients	~2–4 bytes × parameter count
Activations (batch-dependent)	Variable
Total training footprint	~12–16 bytes × parameter count

Practical VRAM requirements for fine-tuning and training common model sizes:

Model size	Inference only	LoRA fine-tune	Full fine-tune
1B parameters	2–3GB	6–8GB	14–18GB
3B parameters	6–8GB	10–14GB	OOM on 24GB
7B parameters	14–16GB	16–20GB	Needs A100
13B parameters	26–28GB	24GB (4-bit)	Needs multi-GPU

This is why a 7B model in FP16 needs roughly 14GB just to load weights — and doubles to 28GB when you add optimizer states for training. 24GB is the practical ceiling for consumer fine-tuning; 16GB works with 4-bit quantization (QLoRA).

Training speed benchmarks

Relative throughput on mixed workloads — ResNet-50 classification, BERT fine-tuning, and ViT training (normalized to RTX 5090):

GPU	VRAM	FP16 TFLOPS	Relative speed	Price	Value tier
RTX 5090	32GB GDDR7	~220	1.00x	~$2,000	Research
RTX 4090	24GB GDDR6X	~165	0.75x	~$1,600	Best value
RTX 5080	16GB GDDR7	~137	0.62x	~$1,000	Mid-range
RTX 5070 Ti	16GB GDDR7	~105	0.48x	~$750	Mid-range
RTX 4070 Ti Super	16GB GDDR6X	~88	0.40x	~$700	Budget pro
RTX 3090 (used)	24GB GDDR6X	~121	0.55x	~$800	VRAM value
RTX 4060 Ti 16GB	16GB GDDR6	~62	0.28x	~$400	Entry level

The RTX 5090's 220 TFLOPS in FP16 is a significant leap over the 4090, but at $2,000 the price jump is steep. The 4090 at $1,600 with 165 TFLOPS and 24GB remains the most compelling overall package unless you regularly work with models that need 32GB.

Batch size vs VRAM: practical impact

Larger batch sizes improve training stability and convergence. More VRAM = larger viable batch sizes:

GPU (VRAM)	Typical max batch (ResNet-50)	BERT batch (seq 128)	Notes
RTX 4060 Ti (16GB)	64	16	Limited but functional
RTX 4070 Ti Super (16GB)	64	16	Same VRAM, faster compute
RTX 4090 (24GB)	128	32	Comfortable for most research
RTX 5090 (32GB)	256	64	Large model experimentation
RTX 3090 (24GB)	128	32	Same as 4090 but slower

For practical training, the difference between 16GB and 24GB often means the difference between batch size 8 and batch size 24 on 7B model fine-tuning — which affects both training speed and gradient quality.

PyTorch vs TensorFlow GPU considerations

Both major frameworks support all NVIDIA cards on this list, but there are nuances:

PyTorch (dominant for research):

Best-in-class CUDA integration
torch.compile() in PyTorch 2.x significantly improves performance on Ampere and newer
Excellent support for BF16 on RTX 30-series and newer (better than FP16 for training stability)
Flash Attention 2 works natively on CUDA

TensorFlow (production and enterprise):

Strong XLA compilation support
Multi-GPU via MirroredStrategy works well on consumer cards
Some newer optimizations arrive later than PyTorch

Both frameworks support FP16 and BF16 mixed precision on Ampere (RTX 30xx) and newer. The RTX 50-series Blackwell architecture adds improved FP8 and FP4 support for inference quantization. For pure research and experimentation, PyTorch is the standard. For production systems already on TensorFlow, it doesn't matter — both support the hardware equally well. If you do most of your work through the Hugging Face ecosystem, see our best GPU for Hugging Face guide for PEFT and Transformers-specific setup.

FP16 and BF16 mixed precision: free performance

Mixed precision training is one of the most impactful optimizations available — and it's free:

FP16 — cuts VRAM by ~50% versus FP32, faster on tensor cores. Can have numerical stability issues with very small gradients.
BF16 — same memory savings as FP16, better numerical range for training stability. Supported on Ampere (RTX 30xx) and newer.
Practical impact: Doubles your effective batch size, cuts training time by 20–40%, often with no accuracy loss.

Always enable mixed precision in PyTorch:

from torch.cuda.amp import autocast, GradScaler
scaler = GradScaler()
with autocast(dtype=torch.bfloat16):  # or float16
    output = model(input)

Multi-GPU scaling for deep learning

Consumer multi-GPU for deep learning (via DataParallel or DistributedDataParallel) works, but has limitations. Before committing to a multi-GPU build, see our how many GPUs you actually need for AI training guide — for most home setups, one large card beats two smaller ones:

PCIe bandwidth between consumer cards is the bottleneck — professional NVLink is 10–20x faster for inter-GPU communication
PyTorch DDP scales reasonably on 2–4 GPUs even over PCIe for data parallel training
Two RTX 4090s gives 48GB aggregate VRAM and roughly 1.5x effective throughput (not 2x due to communication overhead)
Model parallelism (splitting a large model across GPUs) is painful on consumer hardware without NVLink

For most home deep learning setups, one large GPU with more VRAM beats two smaller GPUs — simpler, less overhead, better actual throughput on single large models.

RTX 5090 — the research choice

At 32GB GDDR7 and ~220 FP16 TFLOPS, the RTX 5090 is the only consumer GPU that handles 13B+ model fine-tuning at full precision without resorting to 4-bit quantization:

Fine-tune 7B models at FP16 with comfortable batch sizes
Run 13B models with QLoRA (4-bit base + FP16 adapter layers)
Inference on 30B+ models with 4-bit quantization
Train small models from scratch at speeds that actually make sense

The RTX 5090 makes sense for researchers who need maximum headroom, graduate students doing serious ML work, or professionals building models for production. For learning and hobby projects, it's hard to justify the cost. For a dedicated overview of hardware for academic and applied ML research, see our best GPU for AI research guide.

RTX 4060 Ti 16GB — best for beginners

At ~$400, the RTX 4060 Ti 16GB is the right starting point for anyone learning deep learning:

Runs every PyTorch and TensorFlow tutorial without issues
16GB handles QLoRA fine-tuning of 7B models
Slow for serious training, but fine for understanding concepts and running experiments
Low power draw (~165W) makes it easy to add to any existing PC

The RTX 4060 Ti's 128-bit memory bus limits bandwidth compared to higher-end cards, which matters for training throughput but not for learning. Start here and upgrade when you consistently need more.

Not ready for dedicated hardware? Cloud GPU is the answer

Before spending $1,600+ on a GPU, consider renting compute for specific experiments. RunPod and Vast.ai offer A100 80GB and RTX 4090 instances for $0.40–$1.50/hr — more cost-effective than buying a card if you're training sporadically.

For a broader overview of picking AI hardware, see our Best GPU for AI guide.

GPU tier list available at the original article

Which GPU should YOU buy for deep learning?

Student or beginner learning the fundamentals: RTX 4060 Ti 16GB at $400. Runs every tutorial, handles 7B QLoRA fine-tuning, easy to start with. Upgrade when you outgrow it.
Serious hobbyist fine-tuning models up to 7B at FP16: RTX 4090 at 24GB is the sweet spot. Best VRAM-per-dollar on the market, fast enough for meaningful research. Used RTX 3090 at ~$800 if budget is tight.
Researcher or professional training 13B+ models: RTX 5090 at 32GB is the only consumer card with enough VRAM for 7B FP16 training at useful batch sizes without resorting to quantization tricks.
Running inference more than training: 16GB is enough. RTX 5070 Ti or RTX 4070 Ti Super save significant money with no inference penalty.
Training models professionally on a schedule: Rent cloud GPUs (RunPod/Vast.ai) for large runs, use a local 4090 for experimentation. If most of your training happens on a desktop in your study, our best GPU for AI training at home guide weighs power, noise, and thermals for that exact build.
Training code models specifically: Code generation models (CodeLlama, DeepSeek Coder, Qwen Coder) have their own VRAM patterns — see our best GPU for codegen AI guide.
Working with audio (Whisper, MusicGen): Audio workloads are surprisingly light — our best GPU for Whisper guide ranks the budget options.

Common mistakes to avoid

Buying based on CUDA cores instead of VRAM. More cores speed up training, but running out of VRAM stops training entirely. Prioritize memory capacity first, then compute.
Choosing AMD to save money. ROCm support in PyTorch and TensorFlow still lags CUDA significantly — especially for newer optimization techniques. The debugging time erases the hardware savings.
Skipping mixed precision training. BF16/FP16 cuts VRAM usage by ~50% with minimal accuracy loss. Always enable it. It's free performance that should be default in every training script.
Expecting multi-GPU to scale linearly. Two consumer GPUs won't give you 2x speed. PCIe communication overhead and lack of NVLink means you'll see 1.3–1.6x at best on typical deep learning tasks.
Not using gradient checkpointing on large models. Reduces peak VRAM by 30–40% at the cost of ~20% longer training time. Enable it before buying a bigger GPU.

Final verdict

The RTX 4090 is the best deep learning GPU for most people in 2026. If 24GB isn't enough for your specific workloads, the RTX 5090 at 32GB is the upgrade. If $1,600 is too much, a used RTX 3090 gives you the same 24GB at half the price with slower compute.

For students, the RTX 4060 Ti 16GB at $400 is genuinely all you need to learn. For the NVIDIA vs AMD comparison in more depth, we cover ROCm's current limitations in detail. If you primarily use TensorFlow over PyTorch, see our best GPU for TensorFlow guide for XLA-optimized setups.

The best GPU for deep learning is the one with the most VRAM you can afford — training speed matters, but running out of memory stops you completely.

Related guides on Best GPU for AI

The full version lives on Best GPU for AI — VRAM calculator, GPU comparison table, and live Amazon pricing.

Best GPU for DeepSeek V4: The Honest VRAM Math (81GB Minimum)

Thurmon Demich — Sat, 01 Aug 2026 12:56:52 +0000

Cross-posted from Best GPU for LLM — visit the original for our VRAM calculator, GPU comparison table, and current Amazon pricing.

No single consumer GPU runs DeepSeek V4. The smallest working quant of V4-Flash weighs roughly 81GB as of July 2026, so before you spend anything, here is what every hardware tier actually gets you.

Hardware tier	Usable memory	What fits	Expected experience
Single 24GB (RTX 3090/4090)	24GB	Nothing from V4 — run DeepSeek-R1 32B instead	Great, just not V4
Dual 24GB (2x 3090/4090)	48GB	Still ~33GB short of V4-Flash's smallest quant	Dead end for V4
Quad 24GB (4x RTX 3090)	96GB	V4-Flash at aggressive ~2-bit quants (~81GB weights)	Works, tight on context
96GB workstation (2x RTX 6000 Ada)	96GB	Same fit, two slots instead of four	Cleaner, pricier
128GB Mac Studio (unified)	~115GB usable	V4-Flash at low-to-mid quants	Usable, slow prompt processing
Cloud 2x H200 (rented)	282GB	V4-Flash Q4_K_M (~158GB) + ~10GB KV cache	Comfortable, full context
Anything you own	—	V4-Pro: not happening (~800GB at Q4)	API only

The pattern I keep seeing in V4 discussion threads: people anchor on "13B active parameters" and assume 13B-class hardware. That is the most expensive misconception in local AI right now, and this guide is the math that corrects it. If you want a card for the older, smaller DeepSeek models, our DeepSeek GPU guide covers R1 and Coder — those still fit on one 24GB card.

VRAM chart available at the original article

V4-Pro is not a consumer decision

Quick reality check so we can move on. DeepSeek V4-Pro is a 1.6-trillion-parameter MoE with 49B active parameters per token. Even at Q4, the weights alone land around 800GB — ten H100s before you allocate a single byte of KV cache. That is a datacenter deployment, and no quality-preserving quant changes it in 2026.

So for local purposes, "best GPU for DeepSeek V4" really means "best hardware for V4-Flash." Everything below is about Flash.

V4-Flash: the quant math that matters

V4-Flash is 284B total parameters with 13B active. The community converted it fast — GGUF, AWQ, and GPTQ builds all appeared within days of the release, and llama.cpp and LM Studio both run the GGUFs without drama. What the quant sizes actually look like:

Quant level	Approx. weights	Fits on	Quality verdict
Smallest working (~2-bit)	~81GB	96GB rig (4x 24GB)	Noticeably degraded, still coherent
Low-to-mid (~3-bit)	~105-115GB	128GB Mac Studio	The practical unified-memory sweet spot
Q4_K_M (community default)	~158GB	2x H200-class	Where Flash stops feeling quantized
Q8 and above	280GB+	Datacenter	Diminishing returns

Then add the part almost everyone forgets: KV cache. At long context, budget roughly 10GB on top of the weights. That is why a comfortable Q4_K_M deployment totals about 170GB — and why a 96GB rig running the ~81GB quant has only ~15GB left for context and overhead. It runs. It just isn't roomy, and you'll feel it the first time you paste a large codebase into the prompt.

My honest read on the aggressive quants: 2-bit Flash still beats any 70B dense model I can run in the same footprint, because those 284B total parameters retain a lot of knowledge even when brutally compressed. But if your work is code generation or anything where subtle correctness matters, the gap between ~2-bit and Q4_K_M is real.

Mac unified memory vs. multi-GPU

The 128GB Mac Studio is the sleeper option here, and it's a genuinely different tradeoff rather than a worse one.

The Mac case: one silent box, no 1600W power supply, and about 115GB actually allocatable to the model. Because Flash only activates 13B parameters per token, generation speed on Apple Silicon is respectable — MoE models are far kinder to unified memory bandwidth than dense ones. The pain is prompt processing: feeding in a long document takes several times longer than on NVIDIA hardware. We covered the general pattern in our Mac hardware for LLMs breakdown.

The multi-GPU case: 4x RTX 3090 gives you 96GB with vastly faster prefill and the full CUDA ecosystem. The cost is complexity — a server board with enough PCIe lanes, a big PSU, heat, noise, and llama.cpp layer-splitting configuration. Our multi-GPU setup guide walks through the actual build; read it before ordering parts, because the motherboard is the part everyone gets wrong.

Speed favors NVIDIA. Sanity favors the Mac. Budget is roughly a wash — four used 3090s plus a platform to host them lands near a 128GB Studio.

Which setup should you build?

You own a single 24GB card: don't build anything for V4. Run DeepSeek-R1 32B or a 70B at low quant — our 70B on a single GPU guide covers that path — and hit the API when you genuinely need V4.
You want the cheapest real V4-Flash box: 4x used RTX 3090 (~$2,800-3,600 in cards) for 96GB. Accept the ~2-bit quant and short-ish context.
You want V4-Flash without a server in your office: 128GB Mac Studio. Mid quants, quiet, one power cable.
You want Q4_K_M quality: rent. 2x H200 on-demand covers the ~170GB footprint for a few dollars an hour, which buys a lot of sessions before matching workstation money.

Common mistakes to avoid

Buying 2x 4090s expecting V4-Flash comfort. 48GB is not close. The smallest working quant is ~81GB — a dual-card build strands you $3,000+ deep with no path to running the model at all.
Confusing 13B active with 13B VRAM. MoE routing reduces compute per token, not memory. All 284B parameters must sit in memory because any expert can be selected at any step.
Ignoring KV cache in the budget. Weights-only math says 81GB fits in 96GB "with room to spare." Long-context KV eats ~10GB, and suddenly your 32K session is crashing.
Provisioning for V4-Pro because it benchmarks higher. Pro is ~800GB at Q4. If someone quotes you hardware for it, they're quoting a datacenter.

Our verdict

Your situation	Best move	Approx. cost
Cheapest working V4-Flash	4x used RTX 3090 (96GB)	~$3,500 in cards
Cleanest 96GB, two slots	2x RTX 6000 Ada 48GB	~$12,000
Quiet single box	Mac Studio 128GB	~$4,800
Best quality per dollar	Rent 2x H200, run Q4_K_M	~$7/hr
Already own one 24GB GPU	Run R1 32B locally, API for V4	$0

DeepSeek V4-Flash draws a hard line at roughly 81GB, and no amount of optimism moves it. Build to 96GB, buy 128GB unified, or rent 282GB — those are the three doors, and pretending a 48GB rig is a fourth one is how people waste real money.

DeepSeek V4 VRAM questions

How much VRAM does DeepSeek V4-Flash need?

As of July 2026, the smallest working quant of V4-Flash occupies roughly 81GB, so about 96GB of total VRAM is the realistic entry point once you account for KV cache and overhead. The community-default Q4_K_M build is much larger — roughly 158GB of weights plus around 10GB of KV cache at long context, or about 170GB total.

Can DeepSeek V4 run on a single RTX 5090?

No. The RTX 5090's 32GB is far below the roughly 81GB that V4-Flash's smallest usable quant requires, and no foreseeable quantization closes that gap. A single 5090 is still an excellent card for DeepSeek-R1 32B and other dense models in the 30-70B range — just not for anything in the V4 family.

Is a Mac Studio good for DeepSeek V4?

A 128GB Mac Studio is a legitimate single-box option for V4-Flash at low-to-mid quants, with roughly 115GB allocatable to the model. Because Flash activates only 13B parameters per token, generation speed is reasonable on Apple Silicon. The tradeoff is prompt processing, which runs several times slower than on comparable NVIDIA multi-GPU setups.

What is the hardware difference between V4-Pro and V4-Flash?

V4-Pro is 1.6T total parameters and needs roughly 800GB even at Q4 — strictly datacenter or API territory. V4-Flash is 284B total and spans roughly 81GB at its smallest quant to about 170GB at Q4_K_M with long-context KV cache, which puts it within reach of 96GB multi-GPU rigs, 128GB Macs, and rented H200 pairs.

Related guides on Best GPU for LLM

The full version lives on Best GPU for LLM — VRAM calculator, GPU comparison table, and live Amazon pricing.