Fine-Tuning vs RAG vs Prompt Engineering: When to Use Which
A practical decision framework for choosing the right LLM customization approach.
Introduction
Three main ways to adapt LLMs to your needs. Each has different cost, complexity, and effectiveness. This guide helps you choose.
Quick Decision Matrix
| Factor | Prompt Engineering | RAG | Fine-Tuning |
|---|---|---|---|
| Setup time | Minutes | Hours | Days-Weeks |
| Cost | $0 (API only) | $0-50/mo | $50-5000+ |
| Data needed | None | Documents | Labeled examples |
| Updates | Instant | Re-index | Retrain |
| Hallucination fix | Partial | ✅ Strong | ✅ Strong |
| Domain knowledge | Limited | ✅ Full | ✅ Full |
| Style/format control | ✅ Good | Partial | ✅ Excellent |
| Latency added | None | ~100-500ms | None |
| Expertise needed | Low | Medium | High |
| Reversibility | Trivial | Easy | Hard |
Approach 1: Prompt Engineering
What It Is
Crafting input instructions to guide model behavior without changing the model.
When to Use
- Quick experiments
- General knowledge tasks
- Style/format control
- One-off or varied tasks
- No proprietary data
Techniques
# Zero-shot
"Write a SQL query for..."
# Few-shot
"Examples:\nInput: ...\nOutput: ...\n\nNow: ..."
# Chain-of-thought
"Think step by step: ..."
# Structured output
"Return ONLY valid JSON: {...}"
# Role + Context
"You are a senior engineer. Context: ... Task: ..."
Pros/Cons
| ✅ Pros | ❌ Cons |
|---|---|
| Instant iteration | Limited by context window |
| No training cost | Can't learn new facts |
| Works on any model | Inconsistent on complex tasks |
| Version controllable | Prompt injection risk |
Cost
- API calls only: ~$0.001-0.01 per request
- No infrastructure
Approach 2: RAG (Retrieval-Augmented Generation)
What It Is
Ground model responses in your documents by retrieving relevant chunks at query time.
Architecture
Query → Embed → Vector Search → Top-K Chunks → LLM + Context → Answer
When to Use
- Large document collections (100+ docs)
- Frequently changing data
- Need citations/sources
- Regulatory compliance (audit trail)
- Multiple domains in one system
Implementation Levels
Level 1: Basic (Chroma + OpenAI)
# 50 lines, runs locally
import chromadb
from chromadb.utils import embedding_functions
ef = embedding_functions.SentenceTransformerEmbeddingFunction("all-MiniLM-L6-v2")
client = chromadb.PersistentClient("./db")
col = client.create_collection("docs", embedding_function=ef)
col.add(documents=texts, metadatas=metas, ids=ids)
results = col.query(query_texts=[question], n_results=5)
Level 2: Production (Qdrant + Reranker)
# Hybrid search + cross-encoder reranking
from qdrant_client import QdrantClient
from sentence_transformers import CrossEncoder
client = QdrantClient(url="...", api_key="...")
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
# Retrieve 20, rerank to 5
results = client.query_points(collection, query, limit=20)
pairs = [(q, r.payload["text"]) for r in results]
scores = reranker.predict(pairs)
top5 = sorted(zip(results, scores), key=lambda x: x[1], reverse=True)[:5]
Level 3: Advanced (Agentic RAG)
# Multi-hop reasoning, tool use
# Query → Decompose → Retrieve per subquery → Synthesize
Pros/Cons
| ✅ Pros | ❌ Cons |
|---|---|
| Grounded in your data | Added latency (retrieval) |
| Easy updates (re-index) | Chunking strategy matters |
| Citations built-in | Retrieval can miss context |
| No training needed | Embedding quality critical |
| Works with any LLM | Context window limits |
Cost
| Component | Monthly (1M vectors) |
|---|---|
| Vector DB (Qdrant Cloud) | $0-150 |
| Embeddings (OpenRouter) | $0-20 |
| LLM calls | $10-100 |
| Total | $10-270 |
Approach 3: Fine-Tuning
What It Is
Update model weights on your data to bake in knowledge, style, or behavior.
Types
| Method | Params Trained | VRAM (7B) | Use Case |
|---|---|---|---|
| Full FT | 100% | 80 GB+ | Research, max quality |
| LoRA | 0.1-1% | 16 GB | Most production |
| QLoRA | 0.1-1% | 8 GB | Consumer GPU |
| Prefix/Adapter | <0.1% | 6 GB | Style only |
When to Use
- Consistent style/format requirements
- Domain-specific terminology (legal, medical, code)
- High-volume, repetitive tasks
- Model must work offline
- Regulatory: model = artifact
Data Requirements
// Alpaca format (instruction tuning)
{"instruction": "Write a Python function", "input": "calculate fibonacci", "output": "def fib(n):..."}
// Chat format
{"messages": [
{"role": "system", "content": "You are a Python expert"},
{"role": "user", "content": "Write a fibonacci function"},
{"role": "assistant", "content": "def fib(n):..."}
]}
// Preference pairs (DPO)
{"prompt": "...", "chosen": "better response", "rejected": "worse response"}
Minimums:
- LoRA: 500-1000 quality examples
- Full FT: 10K-100K+
- DPO: 1K-5K pairs
Training Pipeline (QLoRA)
# 1. Prepare data
python prepare_data.py --input raw.jsonl --output train.jsonl --val-size 100
# 2. Train (4-bit, 7B model, 1x A100 40GB)
python train.py \
--model meta-llama/Llama-3.1-8B-Instruct \
--data train.jsonl \
--output ./lora-adapter \
--lora-r 64 --lora-alpha 16 \
--batch-size 2 --grad-accum 4 \
--lr 2e-4 --steps 500 \
--quantization 4bit
# 3. Merge & export
python merge.py --base meta-llama/Llama-3.1-8B-Instruct \
--adapter ./lora-adapter --output ./merged-model
# 4. Convert to GGUF
python convert-hf-to-gguf.py ./merged-model --outfile model.q4_k_m.gguf
Pros/Cons
| ✅ Pros | ❌ Cons |
|---|---|
| Best task performance | High upfront cost |
| No retrieval latency | Hard to update |
| Works offline | Catastrophic forgetting risk |
| Bakes in style/format | Needs ML expertise |
| Single artifact to deploy | GPU required |
Cost
| Scale | Hardware | Time | Est. Cost |
|---|---|---|---|
| 7B QLoRA | 1× A100 40GB | 2-4 hrs | $10-30 |
| 13B QLoRA | 1× A100 80GB | 4-8 hrs | $40-80 |
| 70B QLoRA | 4× A100 80GB | 8-24 hrs | $300-1000 |
| 7B Full FT | 8× A100 80GB | 12-48 hrs | $1000-5000 |
Decision Framework
START: What's your primary need?
│
├─→ "Control output format/style"
│ └─→ **Prompt Engineering** (try first, 15 min)
│
├─→ "Answer from my documents/data"
│ ├─→ Data changes frequently?
│ │ ├─→ Yes → **RAG** (re-index on change)
│ │ └─→ No → Continue
│ ├─→ Need citations?
│ │ └─→ Yes → **RAG**
│ ├─→ < 100 docs?
│ │ └─→ Yes → **RAG** (Chroma local)
│ └─→ > 10K docs, high query volume?
│ └─→ **RAG** (Qdrant/Milvus) or **Fine-tune** if style critical
│
├─→ "Model must know domain deeply"
│ ├─→ Can you get 500+ quality examples?
│ │ ├─→ Yes → **Fine-tune (QLoRA)**
│ │ └─→ No → **RAG + few-shot prompts**
│ ├─→ Need offline/air-gapped?
│ │ └─→ Yes → **Fine-tune** (deploy GGUF)
│ └─→ High volume (>10K req/day)?
│ └─→ **Fine-tune** (cheaper at scale)
│
└─→ "Reduce hallucinations on facts"
├─→ Facts in documents?
│ └─→ Yes → **RAG**
└─→ Facts = training knowledge?
└─→ **Fine-tune** or better prompts
Hybrid Approaches (Real World)
1. RAG + Fine-Tuned Embedder
# Fine-tune embedder on your domain, use with RAG
# Better retrieval → better answers
2. RAG + Fine-Tuned Generator
# Fine-tune small model (7B) for style/format
# Use RAG for knowledge
# Best of both worlds
3. Prompt → RAG → Fine-Tune (Progression)
Week 1: Prompt engineering (validate concept)
Week 2: Add RAG (ground in data)
Week 3: Fine-tune if needed (optimize)
4. Router Pattern
# Route queries to best approach
def route(query):
if is_factual(query): return rag_chain
elif is_creative(query): return prompt_chain
elif is_coding(query): return finetuned_coder
else: return default_chain
Cost Comparison at Scale
| Approach | 1K req/mo | 100K req/mo | 1M req/mo |
|---|---|---|---|
| Prompt (API) | $5 | $500 | $5,000 |
| RAG (managed) | $30 | $300 | $2,000 |
| Fine-tuned (self-host) | $100* | $100* | $100* |
| *Fixed GPU cost | (A100) | (A100) | (4×A100) |
Break-even: Fine-tuning pays off at ~50K req/mo if you own GPUs.
Common Mistakes
| Mistake | Reality |
|---|---|
| "Fine-tune for facts" | Use RAG; FT memorizes poorly |
| "RAG solves everything" | Bad retrieval = bad answers |
| "Prompt engineering is easy" | Requires systematic testing |
| "Bigger model = better" | 7B FT > 70B prompt for specific tasks |
| "One approach fits all" | Hybrid usually wins |
Evaluation Checklist
Before committing:
- [ ] Defined success metrics (accuracy, latency, cost)
- [ ] Created test set (50+ representative queries)
- [ ] Benchmarked current approach
- [ ] Estimated total cost (dev + infra + ongoing)
- [ ] Defined update strategy (how often, how)
- [ ] Considered compliance/privacy needs
My Recommendations by Use Case
| Use Case | Primary | Secondary |
|---|---|---|
| Customer support bot | RAG | Fine-tune style |
| Code assistant | Fine-tune (7B coder) | RAG for docs |
| Legal document analysis | RAG | Fine-tune format |
| Creative writing | Prompt engineering | Fine-tune voice |
| SQL generation | Fine-tune (7B) | RAG for schema |
| Chat with PDFs | RAG | - |
| API documentation bot | RAG | Prompt for format |
| Internal knowledge base | RAG | - |
| Offline mobile app | Fine-tune (GGUF) | - |
| High-volume classification | Fine-tune (small) | - |
Conclusion
Start with prompt engineering — 15 minutes, validates the task.
Add RAG when you have documents and need grounded answers — hours to set up.
Fine-tune only when: you have 500+ examples, need consistent style/format, high volume, or offline deployment.
Most production systems use hybrids: RAG for knowledge + fine-tuned model for behavior.
Part of the LLM Customization series on Dev.to
Top comments (0)