"Should we fine-tune a model on our documents?" is one of the most common questions we hear from teams starting an AI project. Usually the honest answer is no — not because fine-tuning is bad, but because it solves a different problem from the one they have.
This article is the decision framework we use with clients. It is written from production experience: our founder built and runs a multi-tenant retrieval-augmented generation (RAG) pipeline on an AI EdTech platform used across 20+ US schools, with hybrid semantic and keyword search, per-school data isolation on pgvector, and vision-model ingestion of PDFs. The lessons below come from that system and from the client work we do at DablewDevs.
The short answer
RAG changes what the model knows at the moment it answers. Fine-tuning changes how the model behaves. If your problem is "the model doesn't know our policies, products, contracts or customers", you need retrieval. If your problem is "the model knows enough, but answers in the wrong format, tone or structure, or a cheaper model can't follow the task", fine-tuning is a candidate.
Most business knowledge projects are the first kind. Your documents change, different users may see different documents, and people need to check where an answer came from. Retrieval handles all three; fine-tuning handles none of them.
Before either: check whether you need retrieval at all
Anthropic notes that if a knowledge base is under roughly 200,000 tokens (about 500 pages), you can often put the whole thing in the prompt and use prompt caching to keep cost and latency down. For a small, stable document set, that is simpler than building a retrieval pipeline.
Decision table: RAG vs fine-tuning
We score a use case against these questions. If most answers land in the RAG column, start there.
| Question | Points to RAG | Points to fine-tuning |
|---|---|---|
| How often does the source information change? | Weekly, daily or continuously — re-index and the answer changes immediately | Rarely; the behaviour you want is stable |
| Do users need to see where an answer came from? | Yes — citations to the exact passage | No, or the output is not a factual answer |
| Do different users see different data? | Yes — permissions are enforced at retrieval time | No — everything the model learns, every user can extract |
| What is wrong with today’s output? | Missing or wrong facts | Right facts, wrong format, tone, structure or classification |
| How much labelled data do you have? | Documents are enough to start | Hundreds to thousands of high-quality input/output examples |
| Can you delete information on request? | Yes — remove it from the index | Hard — it may be in the weights until you retrain |
| What are you optimising for at scale? | Answer accuracy and freshness | Shorter prompts, lower per-call cost, or a smaller model doing a narrow task |
The last row is where fine-tuning genuinely shines. OpenAI's own optimisation guide frames fine-tuning as a way to supply more examples than fit in a prompt, use shorter prompts at scale, and train a smaller, cheaper, faster model to excel at a particular task. Those are behaviour and cost goals — not "teach the model our handbook".
The misconception: fine-tuning to teach facts
Fine-tuning on your documents feels like it should produce a model that "knows" them. In practice it is a poor way to load facts:
- No citations. The model cannot point to the passage it learned something from, so reviewers have nothing to check against.
- Stale by default. Every document change means another training run, evaluation and deployment.
- No access control. Anything in the training data can potentially surface for any user who asks the right question. In a multi-tenant product, that alone rules it out for customer data.
- Confident blending. A model that has partially absorbed two versions of a policy will answer fluently from both. Retrieval at least lets you show which version was used.
There is also a practical signal in the market: OpenAI's model optimisation guide states that it is winding down its fine-tuning platform and that it is no longer accessible to new users. Fine-tuning remains available elsewhere, but it is not where the major providers are steering knowledge use cases.
When to combine them
The two approaches are not exclusive. The pattern we recommend when both are justified is retrieve the facts, tune the behaviour:
- A support assistant retrieves the relevant help-centre articles (RAG) and uses a tuned or heavily prompted model to answer in your house style and escalation format.
- A document-processing pipeline retrieves the relevant contract clauses, then a small tuned classifier labels them against your risk taxonomy at a fraction of a large model's cost.
- A high-volume workflow starts on a large model with retrieval; once you have thousands of reviewed outputs, those become training data for a smaller model doing the same narrow task.
Notice the order. You almost always build retrieval first, because it produces the reviewed examples you would later need for fine-tuning anyway.
Where RAG actually fails (it is rarely the model)
Choosing RAG is the easy part. When a RAG system gives a wrong answer, the cause is usually that the right passage was never retrieved. These are the failure modes we see most:
1. Ingestion destroys structure
PDFs parsed into a flat stream of text lose their headings, tables lose their column headers, and scanned pages come out empty. On the EdTech platform our founder works on, plain text extraction was not good enough for long course documents, so ingestion uses a vision model to read pages, anchored to the document's table of contents, with deterministic settings and content-addressed deduplication so re-ingesting the same file does not create duplicates.
2. Chunks lose their context
A chunk that says "the fee is waived after 30 days" is useless if it no longer says which product or plan it belongs to. Hierarchical chunking (keeping the section path with each chunk) helps. Anthropic's contextual retrieval technique goes further, prepending a short, model-generated description of where each chunk sits in its document before embedding it.
3. Pure vector search misses exact terms
Embeddings are good at meaning and bad at identifiers: part numbers, error codes, course codes, names. Keyword search (BM25 or PostgreSQL full-text) is the opposite. We run both and merge the results with reciprocal rank fusion (RRF), which scores each document by summing 1 / (k + rank) across the result lists. It needs no training and, in Cormack, Clarke and Büttcher's original SIGIR 2009 paper, outperformed both individual rankers and the Condorcet method. In Anthropic's published experiment, adding contextual BM25 to contextual embeddings cut the top-20 retrieval failure rate by 49%, and adding reranking took the reduction to 67%.
4. Filters applied too late
If you retrieve the top 20 chunks and then remove the ones the user may not see, you can end up with three results, all irrelevant. Tenant and permission filters belong inside the retrieval query. pgvector makes this natural: vectors live in the same PostgreSQL tables as your permission data, so a WHERE clause enforces isolation in the same query that ranks similarity — which is how per-school isolation works in the system described above.
-- Hybrid retrieval, tenant-filtered inside the query (simplified).
WITH semantic AS (
SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> $1) AS rank
FROM chunks
WHERE tenant_id = $2
ORDER BY embedding <=> $1
LIMIT 50
),
keyword AS (
SELECT id, ROW_NUMBER() OVER (ORDER BY ts_rank_cd(tsv, query) DESC) AS rank
FROM chunks, plainto_tsquery('english', $3) AS query
WHERE tenant_id = $2 AND tsv @@ query
ORDER BY ts_rank_cd(tsv, query) DESC
LIMIT 50
)
SELECT id, SUM(1.0 / (60 + rank)) AS rrf_score -- k = 60, as in the RRF paper
FROM (SELECT * FROM semantic UNION ALL SELECT * FROM keyword) AS ranked
GROUP BY id
ORDER BY rrf_score DESC
LIMIT 20;
How to measure which approach is working
Whatever you choose, decide how you will know it is right before you build. We measure two things separately, because they fail for different reasons:
| Metric | Question it answers | How we measure it |
|---|---|---|
| Retrieval recall | Did we fetch the passages that contain the answer? | For each test question, list the source passages a human expects; check whether they appear in the top-k results |
| Answer faithfulness | Did the answer stay true to what was retrieved? | A rubric scored by a person, then by an LLM judge whose verdicts are checked against human labels before we trust it |
| Refusal correctness | Does it say "I don’t know" when nothing relevant was found? | Include out-of-scope questions in the test set |
Start with 50 to 200 real questions from real users, with the documents that should answer them. If retrieval recall is low, no amount of prompt tuning or fine-tuning will fix the answers — improve parsing, chunking or search first. If recall is high but faithfulness is low, then prompt and model changes are worth exploring.
A five-step way to decide
- Write down the failure. Missing facts, wrong permissions, or wrong behaviour? Missing facts point to retrieval; wrong behaviour may point to tuning.
- Measure the size of the knowledge. Under a few hundred pages and stable? Try long-context prompting with caching first.
- Build retrieval with an evaluation set. Hybrid search, metadata filters, citations, and the metrics above.
- Fix the weakest stage. Ingestion, chunking, search or prompt — whichever loses the most answers.
- Consider fine-tuning only for behaviour or cost, once you have enough reviewed examples to train on and a measured reason to expect improvement.
If you already have a RAG prototype whose answers are not good enough to ship, the same framework applies — we start by building the evaluation set and finding where answers are lost. See our RAG development service, or, if the knowledge needs to live inside an existing product, our AI integration work.
FAQ
Is RAG cheaper than fine-tuning?
Usually to start, yes: there is no training run, and updating knowledge means re-indexing documents rather than retraining. At very high volume, a fine-tuned smaller model can be cheaper per call for a narrow task — which is why we sometimes combine the two once a system is proven.
Can fine-tuning reduce hallucinations?
It can make a model follow your format and refuse out-of-scope requests more consistently, but it does not give the model reliable access to facts. For factual answers, grounding the model in retrieved sources and asking it to cite them is the more dependable approach.
Do we need a dedicated vector database for RAG?
Not necessarily. If you already run PostgreSQL, pgvector supports HNSW and IVFFlat indexes and works alongside PostgreSQL full-text search for hybrid retrieval, with permissions enforced in the same query. A separate vector database is worth its operational cost only when scale or specific features demand it.
How long does it take to build a production RAG system?
A focused system over one well-structured document set can be built in a few weeks. Messy documents, many sources, complex permissions and high accuracy requirements extend that. We size it after reviewing a sample of the content.
Sources
- OpenAI — Model optimization guide (fine-tuning use cases; fine-tuning platform wind-down notice), accessed Oct 2026
- Anthropic — Introducing Contextual Retrieval
- pgvector — README (HNSW/IVFFlat indexes, hybrid search with full-text search, RRF)
- Cormack, Clarke & Büttcher (2009) — Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods, SIGIR
Originally published on the DablewDevs blog. DablewDevs builds production AI systems and software — AI agents, RAG, MCP servers and SaaS products. See what we build and what it costs.
Top comments (0)