Every company exploring generative AI eventually hits the same fork in the road: should we use retrieval-augmented generation (RAG), fine-tune a model, or build something more custom? The wrong choice usually costs months, not because the technology fails, but because it solves a different problem than the one the business actually has.
Here's a practical way to decide.
Start with the problem, not the technique
Most business use cases fall into one of three buckets:
- The model needs to know things it doesn't know: your product catalogue, policies, contracts, support history.
- The model needs to behave differently: follow a strict format, adopt a house style, classify into your categories, or use domain jargon correctly.
- The model needs to run under constraints: on-premise, offline, at very low cost per request, or with very low latency.
Each bucket points to a different approach.
When RAG is the right answer
If the problem is knowledge, start with RAG. Retrieval-augmented generation fetches relevant documents at query time and gives them to the model as context. It works well when:
- Information changes frequently (prices, policies, inventory)
- Answers must cite sources or be auditable
- Different users should see different documents based on permissions
RAG is usually the fastest path to value. Most of the effort goes into data preparation: cleaning documents, choosing a chunking strategy, adding metadata filters, and measuring retrieval quality separately from answer quality. A weak retriever is the most common reason RAG systems disappoint.
When fine-tuning makes sense
If the problem is behaviour, consider fine-tuning. Fine-tuning adjusts a model's weights using hundreds to thousands of examples of the input-output pairs you want. It's a good fit when:
- Outputs must follow a precise structure every time
- Prompts have grown so long and fragile that they're hard to maintain
- You need a smaller, cheaper model to match the quality of a larger one on a narrow task
Fine-tuning is not a good way to teach a model facts that change. Retraining every time your price list updates is slow and expensive; that's what retrieval is for.
When a custom model is justified
If the problem is constraints, look at smaller open-weight models you can host and adapt yourself. This route makes sense when:
- Data can't leave your infrastructure for regulatory or contractual reasons
- Request volume is high enough that per-token API pricing dominates costs
- You need predictable latency or offline operation
This is where serious LLM development work begins: model selection, quantisation, serving infrastructure, evaluation pipelines and monitoring. It carries the most engineering overhead, so it should be justified by a real constraint rather than a preference.
Most production systems combine all three
In practice, mature systems mix approaches. A common pattern looks like this:
- RAG supplies up-to-date company knowledge
- A lightly fine-tuned or well-prompted model handles formatting and tone
- Guardrails validate outputs against a schema and block unsafe actions
- An evaluation set of real questions runs on every release to catch regressions
The decision isn't "which one?" so much as "what's the minimum combination that meets our quality, cost and compliance targets?"
A quick decision checklist
Answer these five questions before choosing an approach:
- Is the main gap knowledge, behaviour, or constraints?
- How often does the underlying information change?
- Do answers need citations or an audit trail?
- What are the data residency and privacy requirements?
- What volume and latency do you expect in year one?
If you answered "knowledge" and "frequently", start with RAG. If "behaviour" and "stable", test fine-tuning. If residency or cost at scale dominates, evaluate self-hosted models.
The takeaway
The most successful generative AI projects begin with a clear statement of the gap they're closing and a measurable definition of "good enough". Pick the simplest approach that closes that gap, build an evaluation set early, and add complexity only when the numbers demand it. Whether you build in-house or with a partner, that discipline matters more than any specific model, and it's how effective generative AI solutions and LLM development projects reach production.
Top comments (0)