Use retrieval-augmented generation (RAG) when your app needs facts that change or answers you must cite, and fine-tuning when the model must behave differently, such as a strict format or tone. Nodus runs both halves: an OpenAI-compatible inference API with embeddings for the retrieval side, and LoRA fine-tuning jobs where a small 8B run costs an estimated $1.54 of GPU time at list prices. Many production apps end up using both, with a fine-tuned model reading retrieved documents.
What is the actual difference between RAG and fine-tuning?
RAG changes what the model reads; fine-tuning changes the model itself. With RAG, your app searches a document index at request time and pastes the best passages into the prompt. The model's weights never change, so updating its knowledge means updating the index.
Fine-tuning trains the weights on your examples. The model then produces your format or style without being told, but anything it learned is frozen at training time. Changing it means another training run.
The idea of RAG comes from Lewis et al. (2020), who pointed out that knowledge stored in weights is hard to update and gives no provenance. Those are still the two reasons most teams pick RAG today.
Which one teaches the model new facts?
RAG does, and fine-tuning mostly does not. Ovadia et al. (2023) compared unsupervised fine-tuning with RAG on knowledge-intensive tasks. RAG consistently came out ahead, both for facts the model had seen in pretraining and for entirely new ones. They also found that models struggle to learn new facts from fine-tuning unless they see many variations of the same fact.
That matches what we tell people building on Nodus. If the question is "how does the model know our refund policy changed last week", fine-tuning is the wrong tool. Put the policy in an index and retrieve it. A fine-tuned model can also lose general skills it had before, which our catastrophic forgetting post covers.
When is fine-tuning the better choice?
Fine-tuning wins when the problem is behavior, not knowledge. Our rule: if a well-written prompt with a few examples still gets the format or the judgment wrong, collect a few thousand correct examples and fine-tune.
| Your problem | Pick | Why |
|---|---|---|
| Answers depend on documents that change | RAG | Update the index, not the weights |
| Users need a source for each answer | RAG | Retrieved passages are the citation |
| Corpus is far bigger than the context window | RAG | Only the relevant passages go in |
| Output must follow a strict schema or tone | Fine-tuning | The model learns it instead of being told |
| A long instruction prompt repeats on every call | Fine-tuning | Training replaces the prompt tokens |
| A small model must match a bigger one on one task | Fine-tuning | Distill the task into the small model |
| Facts change and the format is strict | Both | Fine-tune the model to use retrieved text |
This table is our decision rule, built from the two papers above and the cost math below. Try prompt engineering first in every row: it costs nothing to change and tells you whether the gap is knowledge or behavior.
How do RAG and fine-tuning change the cost of each request?
RAG adds input tokens to every request, and fine-tuning can remove them. That makes the per-request bill move in opposite directions. Here is a worked example for a support assistant, with token counts we chose as typical round figures:
- Instructions and a few worked examples in the system prompt: 1,500 tokens.
- The user's question: 100 tokens.
- RAG context, 5 retrieved passages of 400 tokens: 2,000 tokens.
- A fine-tuned model replaces the 1,500-token instructions with a 100-token system prompt.
| Setup | Input tokens per request | Input tokens per 1M requests |
|---|---|---|
| Prompt only | 1,600 | 1.6 billion |
| RAG | 3,600 | 3.6 billion |
| Fine-tuned + RAG | 2,200 | 2.2 billion |
| Fine-tuned only | 200 | 0.2 billion |
Retrieval more than doubles the prompt; fine-tuning away the instructions wins back most of it.
Output tokens are the same in every row, so they drop out of the comparison. The useful number is the difference: fine-tuning on top of RAG cuts input by 1,400 tokens per request, 39% of the RAG prompt. At a round $0.50 per million input tokens, the RAG setup costs $1,800 in input tokens per million requests and the fine-tuned + RAG setup costs $1,100. Swap in your own provider's rate; the ratio stays the same.
How much does fine-tuning a small model cost on Nodus?
The GPU time is small; the examples are the real cost. Take Qwen3-8B, which has 8.19 billion parameters per its Hugging Face model files, and train it on 5,000 examples of about 800 tokens for 2 epochs. That is 8 million training tokens.
We estimate it the way our 70B fine-tuning cost post does: about 6 operations per parameter per token for LoRA with gradient checkpointing, at 30% of the GPU's peak.
operations = 6 x 8.19e9 x 8e6 = 3.93e17
GPU-hours = 3.93e17 / (312e12 x 0.30 x 3,600) = 1.17 hours on one A100 80 GB
cost = 1.17 x $1.09 = $1.27, plus 15 min startup ($0.27) = $1.54
The A100 80 GB is one of the GPUs the Nodus sft runtime presets list for QLoRA on Qwen3-8B, and it is listed at $1.09/hr on the Nodus pricing page. Its 312 TFLOPS is NVIDIA's dense BF16 peak, half the sparse figure on the A100 page. Treat $1.54 as a floor, since QLoRA's 4-bit unpacking adds time we have not measured. Prices checked November 5, 2026 on the Nodus pricing page.
So the fine-tune costs about as much as the input tokens of about 860 RAG requests. Writing and checking 5,000 good examples takes far more of your time than the GPU does, and a bad dataset is the usual reason a fine-tune fails. Our LoRA vs QLoRA post covers which method to pick.
When does a fine-tuned model pay for its own GPU?
When you have enough traffic to keep a dedicated GPU busy. A fine-tuned model usually needs its own serving, because shared token-priced APIs serve their own catalog models. That replaces a per-token bill with an hourly one.
One NVIDIA L40S is listed at $0.793/hr on the Nodus pricing page, or $578.89 for a 730-hour month. The RAG setup above costs $0.0018 per request in input tokens at $0.50 per million. Divide one by the other and the dedicated GPU breaks even at 321,606 requests a month, about 7 a minute. For comparison, the RAG input bill is $180 at 100,000 requests, $450 at 250,000, $900 at 500,000 and $1,800 at 1 million.
Below about 320,000 requests a month, the shared API is cheaper even before counting output tokens.
This comparison favors the shared API slightly, since we count only its input tokens while the GPU covers output too. It also assumes one L40S keeps up with your traffic, which you have to measure with your own model and prompt lengths. Below the break-even, keep the instructions in the prompt and fine-tune later.
How do you combine RAG and fine-tuning?
Fine-tune the model on examples that include retrieved passages, so it learns to use them. RAFT (Zhang et al., 2024) trains on questions paired with a mix of useful and distracting documents. The model learns to ignore the distractors and quote the passage that answers the question. RAFT improved in-domain RAG on PubMed, HotpotQA and Gorilla.
On Nodus both halves run on the same account. Embeddings for the index come from the inference API (Beta), copied from the docs:
from openai import OpenAI
client = OpenAI(base_url="https://inference.nodus-compute.ai/v1", api_key="YOUR_NODUS_API_KEY")
vectors = client.embeddings.create(model="nodus/bge-m3", input=["first document", "second document"])
print(vectors.data[0].embedding)
The fine-tune is a TrainingJob on the managed nodus/sft runtime, from the training guide (Beta). Put the question and its retrieved passages in the prompt column and the cited answer in the completion column. The docs example below uses Qwen3-0.6B; swap the model URI for the model you want to train:
apiVersion: nodus.dev/v1beta1
kind: TrainingJob
metadata:
name: support-sft
spec:
runtime: nodus/sft
model:
uri: hf://Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca
data:
volume: support-chats
path: chats.jsonl
format: JSONL
columns: {prompt: question, completion: answer}
parameters:
steps: 2000
lora: {r: 16, alpha: 32}
maxCostUSD: "20.00"
Run nodus apply -f sft.yaml --dry-run -o yaml first. The dry run prices the job before anything starts, and maxCostUSD suspends it at the cap instead of letting it overspend. The trained LoRA adapter comes back as an output under adapter/.
FAQ
How do you keep a RAG index up to date?
Re-embed only the documents that changed and delete the vectors of removed ones. Store a document ID and a content hash on every chunk, so a scheduled job can tell what changed without re-embedding the whole corpus.
Can fine-tuning stop hallucinations?
Not about facts it was never shown. Grounding answers in retrieved passages, and asking the model to cite them, is the more reliable fix for factual errors.
How many examples do I need to fine-tune?
There is no fixed number, so measure it: our cost example uses 5,000. Hold some back as a validation set; on Nodus, data.validationFraction plus parameters.validation splits them and keeps the checkpoint with the lowest validation loss.
Do I need a vector database for RAG?
Not to start. A few thousand passages fit in memory with plain cosine similarity; move to a vector database when the index or the query rate outgrows that.
Should I fine-tune or use a bigger model?
Try the bigger model with a good prompt first, since it costs nothing to test. Fine-tune a smaller model when the bigger one works but is too slow or too expensive at your volume.
Originally published on the Nodus blog.
Top comments (1)
지식의 최신성과 출력 형식 문제를 따로 진단한 접근이 이해하기 쉽네요. 고객지원 정책을 예로 든 부분에서는 정책 수정뿐 아니라 삭제·만료된 문서가 더는 검색되지 않는지도 평가에 넣으면 좋겠어요. 답변의 인용이 실제 최신 문장과 맞는지, 근거가 없을 때 답을 보류하는지를 별도 항목으로 보면 두 방식을 조합한 뒤의 품질도 비교하기 쉬울 것 같습니다.