From the Best GPU for LLM archive. The canonical version has interactive calculators, an up-to-date GPU comparison table, and live pricing.
Quick answer: The RTX 4090 (24GB) is the best overall GPU for local RAG in 2026. Its 24GB VRAM handles embedding models and LLM inference simultaneously, which is the key bottleneck in RAG pipelines.
See the recommended pick on the original guide
Why RAG needs GPU power
A RAG pipeline has two GPU-intensive stages — and if you are building an autonomous agent setup that chains multiple RAG calls together, the VRAM requirements compound further:
- Embedding -- converting documents into vectors (batch processing, runs once per document)
- LLM inference -- generating answers using retrieved context (runs every query)
The embedding stage is compute-bound and benefits from raw TFLOPS. The inference stage is memory-bandwidth-bound and needs sufficient VRAM to hold the model plus the retrieved context in the prompt.
The critical constraint: your LLM needs enough VRAM for the base model PLUS a large context window. RAG prompts routinely hit 4K-16K tokens with retrieved chunks, which increases KV cache VRAM usage significantly.
VRAM requirements for RAG
| Component | VRAM Usage | Notes |
|---|---|---|
| Embedding model (e5-large, BGE) | 0.5-1.5GB | Small, usually not the bottleneck |
| LLM 7B (Q4_K_M) | ~4.5GB | Fits on most cards |
| LLM 13B (Q4_K_M) | ~7.5GB | Needs 12GB+ GPU |
| LLM 34B (Q4_K_M) | ~20GB | Needs 24GB GPU |
| KV cache (8K context) | 2-4GB | Scales with context length |
| KV cache (16K context) | 4-8GB | RAG often needs long context |
| Total for 13B RAG | ~14-17GB | 16GB minimum |
| Total for 34B RAG | ~26-32GB | 32GB ideal |
The table above is why 16GB cards struggle with serious RAG setups. A 13B model with 16K context and an embedding model can push past 16GB easily.
VRAM chart available at the original article
Best GPUs for RAG ranked
| GPU | VRAM | Bandwidth | RAG 13B | RAG 34B | Price |
|---|---|---|---|---|---|
| RTX 5090 | 32GB | 1,792 GB/s | Excellent | Good | ~$2,000 |
| RTX 4090 | 24GB | 1,008 GB/s | Excellent | Tight | ~$1,600 |
| RTX 5080 | 16GB | 960 GB/s | Good | No | ~$1,000 |
| RTX 5070 Ti | 16GB | 896 GB/s | Good | No | ~$750 |
| RTX 4070 Ti Super | 16GB | 672 GB/s | Acceptable | No | ~$700 |
| RTX 3090 (used) | 24GB | 936 GB/s | Excellent | Tight | ~$900 |
Which GPU should you buy?
If your RAG pipeline uses 7B-13B models with moderate context (up to 8K tokens), a 16GB card like the RTX 5080 ($1,000) handles it well — run the embedding model on CPU to save VRAM for the LLM. If you need 13B with 16K context or want headroom for growth, the RTX 4090 ($1,600) is the sweet spot — 24GB fits the model, embedding model, and long-context KV cache simultaneously. If you are building RAG around 34B models, the RTX 5090 ($2,000) is the only single card with enough VRAM.
Common mistakes to avoid
- Loading the embedding model on GPU alongside the LLM. Embedding models like BGE or e5-large run fast enough on CPU. Keeping them off the GPU frees 1-2GB of VRAM for longer context windows, which matters more for RAG quality.
- Prioritizing quantization quality over context length. In RAG, the retrieved context is what makes the answer good. A 13B Q4 model with 16K context produces better results than a 13B Q6 model limited to 4K context.
- Underestimating KV cache VRAM for long contexts. RAG prompts with retrieved chunks routinely hit 8K-16K tokens. At 16K context, KV cache alone can consume 4-8GB — plan for this on top of model weights.
Our top picks
Best overall: RTX 4090
The RTX 4090 hits the sweet spot for RAG. With 24GB VRAM, you can run a 13B model at Q6_K with a 16K context window and still have room for the embedding model. The 1,008 GB/s bandwidth delivers fast token generation even with large context windows.
For 34B RAG, the 4090 works at Q4 quantization with shorter context windows (4K-8K), but gets tight above that.
Best for 34B+ RAG: RTX 5090
If you are building a RAG system around 34B models like CodeLlama 34B or Yi 34B, the RTX 5090's 32GB VRAM gives you the headroom that the 4090 lacks. The 1,792 GB/s bandwidth is also noticeably faster for long-context generation.
Best value: RTX 3090 (used)
A used RTX 3090 at $900 gives you 24GB VRAM and 936 GB/s bandwidth -- nearly matching the 4090 for RAG capacity at roughly half the price. The trade-off is higher power draw (350W) and older architecture, but for a dedicated RAG server, it is hard to beat.
See the recommended pick on the original guide
See the recommended pick on the original guide
See the recommended pick on the original guide
RAG optimization tips
Run embedding on CPU if VRAM is tight. Modern embedding models like BGE-small or e5-base run fast enough on CPU for most RAG setups. Reserve all your VRAM for the LLM.
Use smaller quantization for the LLM, not shorter context. In RAG, context quality matters more than model precision. A 13B Q4 model with 16K context produces better answers than a 13B Q6 model with 4K context.
Consider splitting stages. Embed documents in batch (overnight if needed), then run inference on a smaller card. The embedding stage is a one-time cost per document.
For more on VRAM planning, see our VRAM requirements guide. If you are on a tighter budget, check our best budget GPU for LLM recommendations. Building a pipeline specifically for document summarization rather than Q&A? Our LLM summarization GPU guide covers the context-length requirements that matter most for that task. If your RAG runs on sensitive corporate or medical data, our best GPU for private AI guide covers the air-gapped deployment angle.
For RAG, buy for VRAM first and bandwidth second. The model plus the context window must fit entirely in GPU memory, or performance falls off a cliff.
Related guides on Best GPU for LLM
- Best GPU for 13B Parameter Models in 2026 (Ranked)
- Best GPU for 34B Models: Yi, CodeLlama & Qwen
- Best GPU for AI Agents in 2026 (5 Picks Ranked)
Continue on Best GPU for LLM for the complete guide with interactive calculators and current GPU prices.
Top comments (0)