Choose between RAG and fine-tuning by the failure you can see. Use retrieval when the model is missing a fact, and fine-tuning when the facts are right and a specialist says the output reads wrong. Use agentic search, also called agentic RAG, only when no single query can find the answer.
Prompting comes before all three, because it costs nothing extra to try. Every extra search step is another model call the user waits for, and most production systems end up running several of these layers at once.
| Criterion | Prompting | Retrieval | Fine-tuning | Agentic search |
|---|---|---|---|---|
| Failure you can observe | Output is inconsistent, or wrong in a way you cannot describe | Model does not know a fact, or knows an old one | Facts are right, form is wrong to a specialist | No single query can retrieve the answer |
| What it changes | Instructions, examples, and the output schema | Which passages are fetched for this question | Default form, and judgements a written rule does not settle | Reach, by searching again from what the last search returned |
| What it leaves alone | The weights, and which documents exist | How the model writes once the right passage is present | A fact that changed after the training run | The accuracy of a question one search could have answered |
| What you build | A prompt, examples and an output schema | An index, the jobs that keep it current, and permission filters | A training set, an eval against the base model, and a hosted model | A loop with a stop condition and a record of every step |
| Added latency | None | One search before the model call | None once served | One model call per step, each waiting on the last |
| Cost per call | Tokens | Tokens, the search, and the passages added to the prompt | Can fall, if a small tuned model replaces a large one | Multiplied by the number of steps |
| Ongoing burden | None | Keeping the index current | A model to own and retrain | Showing the loop stopped for a good reason |
| How Bigcircle handles this | Schema-constrained decoding sits in the same request as the other layers on the trademark platform | Retrieval restricted to a curated index of authorities on that platform, live since 2023 | A fine-tuned model in the same request, for structure and register, which retrieval could not fix | On Prospex, a search loop only after hybrid retrieval stopped improving. The insurance voice agent loads its context when the call opens and does not search during a turn |
The standard answer to this question is "RAG for knowledge, fine-tuning for behaviour." That answer is close, and it breaks on the first system that needs both. Our trademark drafting platform has been live since 2023 with a fine-tuned model, retrieval over a curated index of authorities, and schema-constrained decoding in the same request. We did not run all three because we could not choose. Each one fixes a failure the other two cannot reach.
So start from the failure you are looking at, and then pick the least expensive layer that fixes it. Read the columns of the table above from left to right. Prompting is the column to exhaust first, and a later column does not fix a failure that belongs in an earlier one.
When is it just a prompting problem?
When the task is underspecified rather than unknown to the model. Every team believes it has finished prompting, so test it. Take your prompt, hand it to a contractor who knows nothing about your product, and ask them to do the task.
If they come back with questions, the model has those questions too, and it does not ask them. It picks an interpretation, commits to it, and you file the result as a model failure. A support summariser gets a ticket with three separate issues in it, and nothing in the prompt decides whether that is one summary or three. The model picks one reading, and next Tuesday it picks the other. No amount of retrieval touches that, because nothing is missing from what the model knows.
A fix at this layer takes an afternoon and leaves nothing new to host or retrain, which is why it comes first. Before you spend money on another layer, do the following.
Constrain the output mechanically. Structured output with constrained decoding removes invalid tokens from the choices the model has while it writes, so the output always parses. If you are still parsing free text with a regex and a retry loop, the defect is in the prompt, and it will get filed against the model.
Show the output instead of describing it. Four real examples of the exact output do more than two paragraphs of adjectives, because the model copies what it is shown (in-context learning). Before anyone proposes a fine-tune, count the examples in the prompt. If the count is zero, the free version of the same idea has not been tried yet.
Write the refusal condition, one sentence along the lines of "if the retrieved passages do not contain the answer, say so and stop". Without it the model fills a gap with something plausible, and a reader cannot tell which sentences came from the passages and which were a hallucination.
You cannot skip the eval set
Collect twenty to fifty real cases with known-good outputs and score them the same way on every run. Until that set exists you cannot tell a fix from a lucky sample, and you cannot tell whether the next layer helped. Evals are how you read every decision below this point. How to build LLM evals covers building the set, from the first failure note through a model judge.
When do you need RAG?
Use retrieval when the failure is missing or stale knowledge: facts specific to your business, documents written after the model's knowledge cutoff, anything that changes weekly, and anything a user is allowed to see only part of.
Your refund window changed in March and the model still answers with the old one. Fine-tuning will not fix that, and it will not fix it again in June when the window changes back. The mechanics are in why RAG systems fail in production. The question here is whether retrieval alone is enough.
Retrieval quality depends less on the model or the vector database than on four choices: chunking, embeddings, hybrid search so that part numbers and names still match exactly, and reranking so the best passage reaches the context window rather than a related one.
Work on those four before you touch the model, because no model can use a passage that never reached it. In Anthropic's contextual retrieval tests from September 2024, the share of queries whose right chunk missed the top 20 fell from 5.7% to 1.9% once each chunk carried 50 to 100 tokens of generated context, keyword search ran beside embeddings, and a reranker sorted the result. The added context alone cut failures by 35%, keyword search took the cut to 49%, and the reranker to 67%. The model stayed the same throughout, so the whole gain came from those choices.
On Prospex, our own prospect research engine, semantic and keyword search run together, and a claim with no retrieved passage behind it does not appear. On one run that took 420 collected items down to the 6 that answered the question. We estimate it makes 50 to 60 percent fewer external API calls on repeated runs, and that saving comes from caching rather than from retrieval: the runtime is cache-first, so a rerun on an overlapping account reuses work instead of paying for it again.
Teams get two things wrong at this layer. Citations are an architecture decision, and teams keep treating them as formatting. If a regulated user has to trust a number, they need to open the page it came from.
On our ESG filings platform every extracted field carries a link to its source page and a confidence score, and a low-confidence field goes to review instead of into a dashboard. We measured the analysis cycle falling from weeks to hours once the pipeline replaced manual extraction and spreadsheet work. Most of the recovered time came from the confidence flags: analysts who could see which figures were uncertain stopped re-deriving earlier work to be safe.
Permissions belong in the query. A filter applied after retrieval runs once the document is already in the search results, and anything downstream that logs, caches or summarises those results can leak it. Filter by the user's access inside the search itself.
Three ways retrieval turns out not to be enough
Each row below is a symptom you can see in a trace, and the column on the right names where it sends you.
| What you see | What it means | Where to go |
|---|---|---|
| Right passage in context, answer still reads wrong to a specialist | Form problem, not fact problem | Fine-tuning |
| Right passage cannot be found by any single query | Question has to be split into several searches | Agentic search |
| The corpus does not contain the answer | Not a technique problem | Say so and stop |
That third row is worth saying out loud early, because no layer fixes a missing document.
When should you fine-tune instead of using RAG?
When the failure is form, or judgement a written rule does not settle, and never when it is missing knowledge. Fine-tuning does not teach facts reliably. It can be made to, badly, and then you cannot update a fact without retraining. What it moves is default behaviour: output shape, register, domain convention, and the small judgements a specialist makes without thinking.
The usual test is whether the expert can write the rule down. That is close, and the sharper test is whether the written rule decides the output.
Trademark work is the clearest case we have, and the problem is not that practitioners cannot explain anything. They can. The rules are written down, and they are general, and a general rule does not settle a specific draft on its own. Precedent settles it: how comparable cases were decided, and how those decisions have shifted what a defensible specification looks like in practice.
You cannot compress that into prompt instructions, because it is not one book. It is applied interpretation spread across many decisions, where the same rule gives different results depending on the case in front of you. So the layers split cleanly, and each one takes the part it can hold:
| What it is | Where it lives |
|---|---|
| The general rules, written and stable | Prompt instructions |
| The specific authorities and prior marks for this case | Retrieval, restricted to approved sources |
| The pattern of how those rules get applied in practice | The fine-tune |
People try to solve that third row with a longer prompt. A prompt listing the general rules produces a draft that is technically compliant and reads wrong to anyone who has practised. Before training, we normalised years of attorney opinions into one five-section template, tagged by mark type, jurisdiction and NICE class. Without that step the model would have learned a dozen individual writing styles and averaged them. It mattered more than we expected.
The fine-tune also has a failure we designed around. It improves how a citation is phrased, which is the wrong thing to improve when the authority might not exist. The fine-tuned model is never asked to remember which classes exist, which prior marks conflict, or what the registry shows this week. Retrieval over the curated index supplies those, a validator rejects any citation that does not resolve, and an attorney reviews and signs the draft.
What it costs beyond the training run
Budget the training run last. It is the smallest line and often the only one in the estimate, because LoRA and QLoRA shrank the compute. The other lines did not shrink:
| Cost | What it looks like |
|---|---|
| Data | Examples of the right output, written or checked by an expert. On the trademark platform that meant normalising years of opinions into one template before any training |
| Evaluation | Proof the tuned model is better rather than different. The trademark platform logs the base and fine-tuned models on the same inputs, so a regression shows up as a score that moved |
| Permanence | A fork in your stack. Every new base model is a decision about redoing the work, and providers retire the bases you tuned on |
| Operations | Hosting, capacity, version pinning, rollback. A prompt change has none of these |
Synthetic data helps with volume and cannot invent the judgement you are trying to capture. If you generate the examples from your own prompt, you have trained an expensive copy of that prompt. LLM-as-judge scales the evaluation, and the judge is a model with its own biases that needs spot checks against an expert. A tuned model also ages. If nobody reruns the comparison against the current base model, the team keeps paying to host a fork that may now do worse than a prompt on the newer model.
The fine-tune we did not build
A client came to us wanting a fine-tune for cold outreach email, on a small low-cost model, tuned for open rates. It is a reasonable-sounding request, because writing quality is exactly the kind of thing people expect fine-tuning to fix. We did not tune anything, and splitting the job into two calls fixed it.
The first call does a short discovery pass on the prospect, and the second writes the email given that research and a handful of examples of what good looks like in their domain. That is two hops instead of one, with no new weights anywhere.
That is the pattern we keep meeting, where "the output is not good enough" rarely means the model needs different weights. Usually it means the model is being asked to research and write in a single step, with nothing to imitate. Splitting the step and showing examples takes an afternoon, and the fine-tune was going to take a month.
Fine-tune when the behaviour is unwritten, stable enough to outlive a model generation, and measurable. Otherwise try many-shot prompting first and score it on the eval set. Check the provider's retirement dates before you tune, because a provider can stop training new versions of a tune you still depend on: as of 22 October 2026, OpenAI's deprecations page gives 6 January 2027 as the last day existing customers can create fine-tuning jobs. A tune already running on a base with a shutdown date is the case how to replace the model under a production agent walks through.
The other reason to fine-tune
The other reason is cost and latency rather than quality. Distillation moves a large model's behaviour on your one task into a small model you can serve at lower cost or run on a device. Judge that on cost per request, because it is a serving decision and the argument above does not apply.
What is agentic RAG, and when do you need it?
Agentic RAG puts retrieval inside an agent loop. The model decides what to search for, reads what came back, and then decides whether to search again or answer, where ordinary RAG runs one search and answers from whatever it returned. We call it agentic search in this guide, because the loop can call any search tool and a vector index is only one of them.
Single-shot retrieval assumes the question maps to a query, and some questions do not. "Which of our suppliers mentioned tariff exposure in their last two filings?" No query answers that. You need the supplier list, then the filings per supplier, then a read of each, then a decision about when you have enough. An agent that plans a step, acts, reads the result and plans again handles that (plan-and-execute, the ReAct pattern). If one query still answers the question, stay with retrieval. Our guide on agentic workflows vs AI agents separates that case from a scripted workflow and from this loop.
The loop buys reach with latency and cost, so we add one only when a simpler pipeline has visibly fallen short. Two conditions make it worth the price: the corpus changes faster than an index can follow, and exact search tools already exist for the loop to call. Code meets both. Claude Code's team dropped local vector indexes for model-driven file-name and text search (glob and grep), citing stale indexes and permission complexity (Boris Cherny to The Pragmatic Engineer, March 2026). A large document set that changes monthly and has no structured search meets neither, and retrieval still suits it.
Our own systems mostly run retrieval inside a fixed sequence of steps. On Prospex, the reasoning steps (business model, customer fit, strategy, risk, tech stack) run in a set order and each one retrieves for itself, so the evidence for every claim stays attached to that claim and two runs on one company are built to give the same read.
On the executive operations agent, we measured plans completed with no human stepping in rising from roughly 55% to 80% once local retrieval and planner memory grounded the agent in the user's own workspace. The same agent has a critic that chooses continue, retry or escalate after each step, for a different reason: without it, one bad observation carries through every later step and the agent claims success on work it never finished.
When is the loop not worth it?
A loop turns one search into several model calls and searches in sequence, and the user waits for every one of them. Ask three questions before you build it.
Is the question really multi-hop, or is retrieval weak? Many complaints that "the agent needs to search several times" are single-shot retrieval failures nobody diagnosed. Check whether the passage you need ranks near the top for any one query. If it does, fix hybrid search and add reranking, which changes one step of the request, before you add a loop that changes how many steps there are.
There is a real exception, and we hit it on Prospex when retrieval had run out of road. The source data was structured in a way that resisted tuning: the evidence a single question needed was spread across documents that looked nothing like each other, so better chunking and better hybrid search kept arriving at the same ceiling. An agent that searched, read what came back, and searched again got past that. Another month of retrieval work would not have.
So the check is not whether you have tried reranking, but whether you have a reason to believe more retrieval work will move the ceiling. On Prospex we did not, and we could say why.
Is there a latency budget? Rule the loop out for voice before anyone designs it, because the pause a caller expects is short and similar across languages: in a ten-language study, every language's average gap between turns sat within 250 ms of the cross-language mean (Stivers et al., PNAS, 2009). A search loop does not fit in that gap.
Our insurance voice agent came down from 2.5 to 3 seconds to around 900ms end to end in our tests, and it does not search during a turn at all. Policy and claim context loads when the call opens, so no reply waits on it, and every action goes through a typed tool that checks its inputs and permissions.
Can the loop stop, and can you show it did the work? An agent that decides when it has enough evidence also decides when to stop. Solve that before the loop ships, because an agent that stops early still writes a confident answer. Our guide on AI agents that call unfinished work done covers the checks. The loop adds reach, and it makes no single search more accurate, so if one good search can answer the question, that search will be faster and easier to debug.
What does each layer cost to run?
Two cells in the comparison at the top surprise people. Fine-tuning can lower the cost per call, because a small tuned model that matches a large one on your single task costs less to serve and answers faster. When cost matters more than quality, the usual advice inverts.
Retrieval's real running cost is freshness, not the search. Keeping the index correct as documents change, permissions move and pages get deleted is the work, and an index that falls behind raises no error, so nobody notices until an answer quotes a page that no longer exists. The multiplied calls in the agentic column are a separate bill, which our guide on AI agent cost splits into the build, the run, and the work that remains after launch.
Before adding any layer, check prompt caching on a stable system prefix and model routing for the easy requests. Neither one changes what the model does.
What does running all three at once look like?
The trademark path, in production since 2023, runs three layers in one request, and every draft is scored before an attorney sees it.
Remove one and the system fails in a specific way. Without retrieval it invents authorities. Without the fine-tune the drafting reads wrong to a practitioner. Without the schema, the integration breaks the day a model update changes its formatting habits. The evaluation scores each draft from 0 to 10 on six criteria (structure, reasoning depth, citation accuracy, jurisdiction discipline, edge cases and tone), and attorney corrections remain the one signal that automated scoring cannot replace.
What will none of this fix?
These four layers get sold as answers to problems they cannot touch. A missing document stays missing, and if the corpus does not contain the answer, retrieval returns the closest wrong thing and generation makes it sound plausible.
A task nobody has defined stays undefined. If three stakeholders read the spec three ways, a fine-tune learns whichever reading dominated its training examples. Disagreement between experts also stays disagreement. If two specialists mark the same output differently, more training data does not resolve it, and the call belongs to the product owner rather than to the model.
Bad source data stays bad, whether that means duplicated pages, stale exports or a PDF parser that drops table headers, and every layer downstream inherits it.
How do you decide, in order?
Stop at the first question you cannot answer yes to, and fix that before moving on.
- Can a new contractor do the task from your prompt? If not, write the specification, add four real examples, and name the refusal condition. An afternoon.
- Do you have twenty to fifty scored eval cases? If not, build them before changing anything else.
- Is the failure a missing or stale fact? Retrieval: hybrid search, reranking, chunking, permissions. Not a bigger model.
- With the right passage in context, is the answer wrong in form rather than fact? That points to fine-tuning, if the convention is unwritten and stable. Try many-shot prompting first and measure it.
- Does answering require several dependent searches? Agentic search, with a stop condition, a verification step and a latency budget you have checked.
- Are you optimising cost or latency rather than quality? Distillation, model routing, prompt caching, judged on cost per request.
To find where you are, open your last twenty production failures and label each with one word: unspecified, missing, form, or multi-hop. Count the labels. The largest bucket tells you which step to work on, and it is often not the layer the team was planning to build next.
Common questions
Should I use RAG or fine-tuning?
Use retrieval when the model is missing a fact or is still quoting a document that has changed. Fine-tune when the right passage is already in context and a specialist says the draft reads wrong. Finish the specification, the examples, and the refusal line before either of those, because an unspecified task looks like a model failure.
When should you fine-tune instead of using RAG?
Fine-tune for form and for judgement a written rule does not settle, and never to store a fact you will need to update. The authorities and the current documents stay in retrieval. Try many-shot prompting first and score it on your eval set. A tuned model you do not compare with today's base model is a fork you then have to keep.
What is agentic RAG?
Agentic RAG, also called agentic search, is a loop that plans a query, reads what came back, and searches again from that result. Use it when no single query can find the answer, such as a supplier list, then each filing, then a read of each one. Stay with one search when that search can answer, because every step of the loop is another model call the user waits for.
What is the difference between RAG and agentic RAG?
Ordinary RAG runs one search before the model call and answers from what that search returned. Agentic RAG lets the model decide what to search for next, read the result, and search again until it has enough or hits a stop condition. The loop adds reach for questions that need several dependent searches, and it makes no single search more accurate.
Can RAG and fine-tuning run in the same request?
Yes. Bigcircle's trademark drafting platform has done it in production since 2023. It retrieves from a curated index of authorities, drafts with a fine-tuned model, and constrains the output to a five-section schema whose citations must resolve. Drop retrieval and the draft invents authorities, and drop the fine-tune and a practitioner says the form is wrong.
How do I choose between prompting, RAG, and fine-tuning?
Stop at the first step you cannot pass. If a new contractor cannot do the task from the prompt, write the specification and add real examples before you touch a model. If the failure is a missing or stale fact, use retrieval, and try many-shot prompting before a fine-tune when the passage is already in context.
When is an agent the wrong way to search?
Skip the agent when one query can still answer the question. Many complaints that the system must search several times are single-shot retrieval misses, fixed by hybrid search and reranking without adding steps. A voice product answering inside a conversational pause cannot host the loop, so Bigcircle's insurance voice agent loads its context when the call opens and does not search during a turn.
Sources and further reading
- Anthropic, Introducing contextual retrieval, September 2024. Top-20 retrieval failure from 5.7% to 1.9%.
- Gergely Orosz, Building Claude Code with Boris Cherny, The Pragmatic Engineer, March 2026. Why Claude Code dropped vector indexes for glob and grep.
- Stivers et al., Universals and cultural variation in turn-taking in conversation, PNAS, 2009.
- OpenAI, deprecations, checked 22 October 2026. The fine-tuning cut-off date.
- Why RAG systems fail in production. The retrieval layer in depth, failure by failure.
- The practical guide to AI agents. What makes a loop controllable rather than just autonomous.
- Agentic workflows vs AI agents. Whether the task needs this loop, a scripted workflow, or retrieval on its own.
- Why AI agents report unfinished work as done. The verification problem agentic search inherits.
- How LLM inference actually works. Why prefill, decode and caching drive the cost column above.
- Case studies referenced: trademark drafting, prospect intelligence, ESG filings, executive operations, insurance voice.
Originally published on bigcircle.ai. I'm Tirth, founder of BigCircle, where we build production AI systems with governance built in.
Top comments (0)