RAG vs fine-tuning: which does your AI application need?

Learn how document retrieval differs from model training, work through examples, and choose a sensible starting point for your AI application.

Originally published on Stack & Craft.

What you will learn: Explain what changes in a RAG system and in a fine-tuned model, diagnose the kind of problem you have, and design a small experiment before investing in either approach.

Prerequisites: No model-training experience needed. Familiarity with asking a chatbot a question is enough.

One assistant, two different problems

Imagine a support assistant for a fictional product called Atlas. The examples below are teaching examples, not benchmark results.

The answer needs a current fact

A customer asks about the refund window. The policy changed yesterday. The assistant needs access to the current policy and the version that applies to this customer.

Finding and supplying the right policy passage is an information-access problem.

The answer needs a consistent pattern

The assistant has the right policy but repeatedly ignores your required response structure: decision, explanation, and source. You want it to follow the same pattern across many different requests.

Producing the desired structure reliably is a behavior problem. Start by testing clearer instructions and structured output controls where available.

Diagnose before choosing

RAG often helps supply relevant evidence. Fine-tuning can adapt learned task behavior. Neither is a universal fix, and the two needs can occur in the same application.

Understand these terms first

A language model generates text from the input it receives and the patterns learned during training. These terms describe different parts of that process.

Parameters and weights

The learned numerical values used by a model. Training adjusts model parameters so its outputs better match a training objective. Think of them as learned patterns, rather than a searchable filing cabinet.

Inference

Using a model to produce an output. Asking a question and receiving an answer is an inference request. A normal inference request does not itself train the model.

Prompt and context

The prompt is the input given to the model: instructions, the question, and possibly examples or documents. Context is the information available in that request. A context window has a size limit.

Retrieval and index

Retrieval searches a collection for useful information. An index is a representation prepared to make that search possible or faster. Retrieval may use keyword search, semantic search, or both.

Fine-tuning dataset

A collection of examples used for additional training. In supervised fine-tuning, examples demonstrate the inputs and desired outputs for a task. Quality and representative coverage matter.

Evaluation set

Examples reserved for checking whether the application works. Keep them separate from training examples so you are not simply measuring performance on material used to teach the model.

RAG: find evidence when a question arrives

Retrieval-augmented generation means retrieving external information and supplying it to a generation model. A typical application can use an existing model without updating its weights.

Before users ask questions

1. Collect and prepare sources — Parse the policy documents into readable text. Preserve document titles, dates and access permissions.
2. Create searchable passages — Split longer documents into useful pieces and build an index. Embeddings, when used, represent text as numbers for semantic comparison.

At request time

1. Search the permitted sources — Find passages relevant to this customer’s question. Check that the customer is allowed to access them.
2. Build the model request — Supply the question, selected passages, source identifiers and instructions.
3. Generate and check the response — Ask for an answer grounded in the evidence. Validate the cited source and unsupported claims before trusting the result.

Updating a document is not updating the model

When the refund policy changes, update the source and its searchable representation, remove stale versions as appropriate, and verify retrieval. The generation model can remain the same. An outdated index can still return an outdated answer.

Fine-tuning: learn from task examples

Fine-tuning is additional training of a pretrained model. Full fine-tuning updates the base model’s weights. Parameter-efficient approaches such as LoRA can keep those weights frozen while training additional parameters.

Prepare a model for a task

1. Define the target behavior — Specify what a good response should do. For Atlas, this might be consistently returning a decision, a concise explanation and an evidence identifier.
2. Prepare and separate examples — Create representative inputs and desired outputs. Separate training, validation and final evaluation examples. Avoid near-duplicate leakage between them.
3. Run additional training — Train the supported model or adaptation parameters. Save the resulting model or adapter version.
4. Compare on unseen requests — Check task quality, factual support, format adherence and regressions. Deploy only if the improvement justifies the ongoing cost and maintenance.

Training does not automatically provide fresh sources

Fine-tuning can teach patterns and may affect factual recall, but it does not create a reliable live database or an automatic citation system. Do not assume a model can recall every training example exactly, forget a removed document immediately, or authorize access to a fact.

Compare what you will operate

These are practical design considerations. Actual costs and quality depend on the model, collection, traffic and implementation; there is no universal cheaper option.

Question	RAG application	Fine-tuned model
What changes?	The searchable collection and the context supplied per request.	Model parameters or additional adaptation parameters through training.
How do facts change?	Refresh sources and the index; verify stale results are removed.	A new fact is not guaranteed to be learned or recalled. Further training and evaluation may be needed.
How do sources appear?	Preserve source IDs and connect answers to retrieved passages. Check support.	Training alone does not create traceable source references. Supply evidence separately if needed.
What data is needed?	Relevant, readable sources with versions and permissions.	Representative training examples and a separate evaluation set.
What can fail?	Wrong sources, poor parsing, missed passages, stale data or unsupported generation.	Weak examples, overfitting, unwanted behavior changes or poor performance on new cases.
What creates cost?	Ingestion, storage, search, optional reranking and model input/output.	Dataset preparation, training, evaluation and model hosting or usage.
Can they work together?	Retrieve evidence and send it to a fine-tuned generation model.	Adapt task behavior while still using retrieved evidence at inference time.

Choose an experiment, not a label

First write a small set of realistic requests and the evidence or behavior each response needs. Try a clear prompt before adding infrastructure or training.

Start with prompting when the input is already available

If the user provides a short document and asks for a summary, try passing that document directly with clear instructions. An external index may add little value for that single request.

A pasted support email → a summary with three specified fields.

Try retrieval when information is missing

If the answer depends on documents outside the model request, test whether retrieval can find the required evidence. Check the returned passages before evaluating the generated answer.

A current refund policy from a large, regularly updated knowledge base.

Consider fine-tuning for a measured task gap

If prompting and suitable application controls still leave a repeated behavior failure, and you have good training examples, compare a fine-tuned model against the baseline on unseen requests.

Consistent domain-specific classification or response behavior across a large variety of inputs.

Combine approaches only when both are needed

A fine-tuned model can receive retrieved passages just like another model. Compare the combination with simpler alternatives, because it adds both training and retrieval responsibilities.

Fresh policy evidence plus a specialized answer format that improves measurably after training.

Try a small experiment without a paid model

You can practice the key decision on paper or in a text file. No cloud account, API key or training run is required.

Your source collection

Atlas refund policy v2: Annual plans can be refunded within 14 days of purchase. Monthly plans are non-refundable. Requests must include an order ID. This fictional policy supersedes v1, which allowed 30 days for annual plans.

Test request	Expected evidence or behavior	What to inspect
I bought an annual plan 10 days ago. Am I within the window?	The v2 passage says 14 days. The time condition is met; do not invent other eligibility rules.	Does retrieval find v2, and does the answer use 14 rather than the old 30?
I bought a monthly plan yesterday. Can I get a refund?	The v2 passage says monthly plans are non-refundable.	Does the answer distinguish monthly from annual plans?
Can I get a refund after 20 days on an annual plan?	The v2 time window is 14 days.	Does the answer avoid quoting superseded v1?
When will the bank credit my refund?	The supplied policy does not say.	Does the answer acknowledge missing information rather than inventing a number?
Answer using Decision, Reason and Source.	Use the requested format, with v2 as the evidence identifier where relevant.	If the evidence is right but fields are missing, classify that separately as a behavior failure.

Worked diagnosis

Returning 30 days suggests a stale-source or source-selection problem. Returning the correct 14-day fact in the wrong format suggests a format problem. Neither observation alone proves that fine-tuning is necessary. First inspect the source, retrieval result, request and response.

Measure each failure separately

Use the same held-out requests when comparing approaches. Fix the model version, inputs and scoring rules as far as possible, and repeat enough cases to avoid drawing conclusions from one lucky answer.

• Evidence coverage: did the retrieved passages contain the information required to answer?
• Answer support: are the important claims justified by the supplied source?
• Task behavior: were the requested fields or labels produced correctly?
• Missing evidence: did the assistant explain the gap rather than inventing a fact?
• Operations: what were the latency, per-request cost and source-update effort?
• Regressions: did an improvement on one task make other important cases worse?

Two important boundaries

Access checks belong in the application before documents reach the model. Neither fine-tuning nor an instruction to “keep secrets” replaces authorization. Also, a citation is useful only when the cited passage actually supports the answer.

A practical decision sequence

1. Describe one failing user request and what success looks like.
2. Confirm whether the needed information is already in the request.
3. Try clearer instructions or supported output constraints.
4. If information is missing, test retrieval and inspect the passages.
5. If task behavior still fails repeatedly, compare fine-tuning using good examples and held-out evaluation.
6. Choose the simplest version that meets your evidence, quality and operational requirements.

Practice

Use the fictional Atlas policy above to write five expected responses. For each, record the source evidence and required output format separately. Then imagine the policy changes to 7 days: list which source, index, test cases and application checks need updating. Explain why you would not assume that changing a document automatically changes model weights.

Check your understanding

Does uploading a PDF to a RAG application necessarily train the language model?

No. In a typical application, the document is indexed and selected text is supplied at inference time. The generation model can remain unchanged. Some RAG systems also train components; retrieval and training are separate design choices.

Does every fine-tuning method update every base-model weight?

No. Full fine-tuning updates base weights. Methods such as LoRA can freeze base weights and train a smaller set of additional parameters.

Can fine-tuning guarantee that tomorrow’s policy is used?

No. Current policy information must be maintained and supplied or otherwise reliably available. Test freshness and source version selection explicitly.

Can you use RAG and fine-tuning together?

Yes. For example, retrieve current passages and give them to a model fine-tuned for the task. Evaluate whether the combined system improves enough to justify its complexity.

Is a wrong-format answer evidence that you must fine-tune?

No. First test the instructions, examples and application output controls. Consider fine-tuning only if a measured behavior gap remains and representative training data is available.

Sources

• RAG research paper — Lewis et al. (2020): The original research combines retrieval with generation and includes fine-tuning. This is why the guide does not define all RAG systems as training-free.
• LoRA research paper — Hu et al. (2021): Describes adaptation with frozen pretrained weights and trainable additional matrices.
• AWS: Comparing RAG and fine-tuning: Additional implementation trade-offs and combining the approaches. The decision sequence and fictional exercise in this guide are Stack & Craft teaching material.

Continue learning

Read the original guide on Stack & Craft for the illustrated version and related lessons.

The Local RAG Document Assistant includes the companion exercises.