This article is Part 6 of my RAG in Practice series, where I explain retrieval-augmented generation in practical, production-oriented terms.
In this part, we continue the RAG in Practice series.
TechNova is a fictional company used as a running example throughout this series.
The Question You Should Have Asked Before Building
You built a RAG system in Part 5. It loads documents, chunks them, embeds them, retrieves relevant chunks, and generates answers. It works. But was RAG the right tool for that problem?
Not every knowledge problem needs retrieval. Some problems need behavior change. Some problems are small enough that you can skip retrieval entirely and just put everything in the prompt. Picking the wrong approach does not just waste effort; it solves the wrong problem well.
The mistake is treating these as interchangeable tools. They are not.
Three Approaches, Three Different Questions
RAG, fine-tuning, and long context are not competing solutions to the same problem. Each one answers a different question.
RAG: When the Knowledge Changes
RAG addresses one question. What does the model need to know right now?
When your data changes faster than you can retrain (current pricing, updated policies, today’s inventory), RAG retrieves the current answer at query time. TechNova’s return policy changed from thirty days to fifteen days last quarter. The model does not need to learn the new policy. It needs to find it when asked.
Fine-Tuning: When the Behavior Needs to Change
Fine-tuning addresses one question. How should the model behave?
Fine-tuning means training an existing model further on examples of the behavior you want. The result is a customized version of the model that you call instead of the original. You are changing the model itself, not just the prompt you send at runtime.
Try describing the behavior you want in the prompt first. Fine-tune when prompting alone cannot make the behavior consistent enough.
A quick note, because this is where most developers get tripped up. You may have read that fine-tuning is how you teach a model new facts. That framing is outdated. Common guidance is that fine-tuning teaches behavior: tone, format, reasoning style, output structure. It does not reliably teach new facts.
What “behavior” actually covers is broader than tone. It can mean producing SQL in a specific dialect, following a strict response schema, generating code in your team’s style, or handling a specialized task like medical question answering more reliably. These are patterns in how the model responds, not new facts the model knows. Fine-tuning can also pick up domain patterns and some facts, but it is a poor way to keep facts current. RAG supplies the current facts at answer time.
If TechNova wants the AI assistant to respond in an empathetic support tone, use bullet points for troubleshooting steps, and follow a specific escalation protocol, that is behavior, not knowledge.
Where this goes wrong is predictable. A customer asks TechNova’s fine-tuned assistant about the return policy. It responds warmly, uses bullet points, follows the escalation protocol, and confidently cites the old thirty-day figure. Right tone, wrong facts.
Long Context: When the Data Fits in the Window
Long context addresses one question. Can I just put it all in the prompt?
When your knowledge base fits within the model’s context window, you can skip retrieval entirely. No chunking, no embeddings, no vector database. Just put the documents in the prompt and let the model read them.
TechNova's five support files from Part 5 total only a few thousand tokens, small enough for current long-context models. For those files alone, a retrieval pipeline would be hard to justify. RAG starts to pay off as the library grows, changes faster than you want to resend, or needs traceability.
The 2026 Reality
Context windows have grown dramatically. Major frontier models now support context windows ranging from hundreds of thousands to around a million tokens. For many workloads, “does it fit in the window?” is no longer the hard question. The harder questions are what it costs to send that context, how long it takes to process, and whether the model can reliably use the evidence buried inside it.
You have probably seen the claim that RAG is dead because large context windows make retrieval unnecessary. The argument sounds reasonable until you look at the costs.
Run the math for any current model. A RAG query that sends 1,000 tokens of retrieved context costs a tiny fraction of what a query stuffing 200,000 tokens into the prompt costs, roughly two orders of magnitude per query, before output tokens, embeddings, or infrastructure. Model prices drop over time, but the ratio of input tokens does not. Prompt caching narrows the gap when the same documents go out with every query, because providers may charge less for cached input tokens. It doesn't close the gap, and it doesn't fix the accuracy problem below. For a demo with ten queries a day, it does not matter. For a product handling tens of thousands of daily queries, it is the difference between a manageable API bill and one that makes your finance team ask questions.
Cost is not the only issue. Long context also pays in latency. Unless prompt caching applies, every token has to be processed on every query, even when only a small fraction is relevant. RAG selects first, then sends less.
There is also an accuracy issue, and it is more serious than most practitioners realize. Researchers originally documented what’s called the “lost in the middle” effect: models retrieve information less reliably from the middle of a long context than from its start or end (Liu et al., 2024). More recent evaluations have pushed this further into what practitioners now call “context rot”: the broader finding that model accuracy degrades as input length grows, even when the relevant information is technically present in the prompt.
A 2025 Chroma study using the LongMemEval benchmark found that models answered more accurately when given only the relevant part of a conversation (about 300 tokens) than the full conversation (about 113,000 tokens). Every model family tested did better with the focused input, and the Claude models showed the largest gap, often by declining to answer. The information was in the prompt both times.
The implication is direct. Long context is not a free substitute for retrieval. Putting more tokens in front of the model does not guarantee the model will use them well. RAG gives the model a smaller, more focused context, which makes it more likely the model will use the right evidence.
For many production workloads, that selectivity is the real advantage: lower cost, lower latency, and a better chance the model uses the right evidence.
LaRA, a 2025 benchmark that compared RAG and long context across 11 models, found that neither approach wins everywhere. A practical pattern is to use retrieval to select the right evidence, then use long context to reason over what was selected. Retrieve the three most relevant documents, then let the model read them in full rather than reading your entire corpus every time. That hybrid approach gives you the cost control of RAG with the reasoning depth of long context.
Four Cases, Four Different Answers
The right approach depends on your specific situation. Here are four scenarios that cover the most common patterns.
Case 1: Small stable corpus (internal FAQ, 20 pages, rarely changes). Long context wins. The entire corpus fits easily in a single prompt. No retrieval infrastructure needed. If a fact changes, update the document and the next query sees the change immediately. The simplest path. Start here if your data is small enough.
Case 2: Large dynamic corpus (product documentation, 500+ pages, updated weekly). RAG wins. The corpus may not fit in a single prompt, and even when it does, resending it with every query costs more and gives the model more to wade through. Retrieval selects the relevant documents. Updates to the corpus require re-indexing the changed documents, not retraining the model. This is where Part 5’s pipeline operates.
Case 3: Regulated industry (legal compliance, audit trail required). RAG has an advantage when you need structured per-answer source traceability. When a regulator asks “why did the system give this answer?”, a RAG pipeline that logs each step can answer it. This query retrieved these chunks from these source documents, and the model generated this answer from that context. Long context can still log the full prompt and context you supplied, but it does not give you that same per-answer retrieval record by default. Part 8 covers the version metadata this needs.
Case 4: Rapid prototyping (testing whether AI can solve the problem at all). Long context wins for the prototype. Skip the retrieval infrastructure, put your documents in the prompt, and see if the model can answer your questions well enough to justify building a full system. If the prototype works, migrate to RAG when you need to scale, control costs, or add traceability. Do not build the pipeline before you know the problem is worth solving. One warning, though. Without an evaluation harness in place, you will not know when the prototype’s response quality stops being good enough to keep. Part 7 covers that harness.
The Decision Table
Five variables matter most when choosing an approach: corpus size, update frequency, latency budget, cost at scale, and audit/traceability.
The table shows how the approaches compare. The flowchart shows how to choose a starting point.
The Decision Flowchart
Three branching questions get you to the right starting point.
Does your knowledge change over time? If yes, don't rely on what the model learned in training. The current documents have to reach the model at answer time, through long context if they fit or retrieval if they don't. If no, choose between long context and retrieval based on corpus size and operational needs, and add fine-tuning if you need behavior change.
Does all your data fit in the context window? If yes, long context is the simplest path. But plan for growth: if your corpus is likely to exceed the window, start with RAG now rather than migrating later.
Do you also need behavior change? If yes, add fine-tuning for behavior and keep supplying the facts through RAG or long context. If no, RAG or long context alone is enough to start.
They Are Not Mutually Exclusive
The flowchart gives you a starting point. In practice, many production systems combine approaches.
RAG + fine-tuned model. Fine-tune for behavior and use RAG for knowledge. TechNova fine-tunes the model to respond in their support tone and use bullet-point troubleshooting format. RAG retrieves the current return policy and firmware changelog. The fine-tuned model reasons over the retrieved context in the right style. This combination appears in mature production support systems where teams have invested in both behavior consistency and keeping knowledge current.
RAG + long context. For corpora too large or costly to send whole, retrieve the most relevant documents and place those full documents into the prompt rather than chunking them aggressively. Instead of sending all five TechNova documents every time, retrieve the two most relevant and let the model read them whole. This keeps prompts smaller than full-corpus stuffing and keeps ingestion simpler than fine-grained chunking.
Combinations add complexity. Start with one approach. Add another when evaluation shows a specific gap, not when a blog post says you should.
Choosing Your Starting Point
Pick the approach that answers your actual question.
If the question is “what does the model need to know right now?” supply the current facts through long context when they fit, or RAG when retrieval is useful. If it is “how should the model behave?” consider fine-tuning. If it is “can I just put it all in the prompt?” start with long context, then add RAG when the corpus grows, retrieval improves cost or focus, or you need structured traceability.
Many production systems combine approaches eventually, but every addition should be justified by a measured need.
Next, how do you know whether your RAG system is actually giving good answers? That is Part 7: Your RAG System Is Wrong. Here's How to Find Out Why.
References & artifacts
Sources that ground the specific claims in this article.
- RAG in Practice examples repository
- Anthropic, Prompt caching. First-party documentation on how cached prompt prefixes are reused and billed. Supports the caching point in The 2026 Reality.
- Li et al., LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs (ICML 2025). Peer-reviewed benchmark across 11 LLMs. Core finding: neither RAG nor long context is a silver bullet.
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts (TACL 2024). Original empirical finding that models retrieve less reliably from the middle of long contexts. An early, widely cited finding on long-context retrieval behavior.
- Hong, Troynikov, and Huber, Context Rot: How Increasing Input Tokens Impacts LLM Performance (Chroma Research, 2025). Evaluation of 18 LLMs showing accuracy degrades as input length grows. Primary source for the callout above.
- Wu et al., LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (ICLR 2025). The peer-reviewed benchmark Chroma used. Independent corroboration of the retrieval-first argument.
- Huyen, AI Engineering: Building Applications with Foundation Models (O’Reilly, 2024). Chapter 7 covers fine-tuning in depth. A good next read if you’re seriously considering a fine-tuning project.
Part of the RAG in Practice series, originally published on AI in Practice Hub alongside the full series and companion code.
Top comments (0)