<h2>The Context Problem</h2>
<p>Every production LLM application faces the same fundamental challenge: <strong>how do you give the model access to relevant information that wasn't in its training data?</strong> Your company's documentation, user data, product catalogue, codebase — none of it exists in GPT-4 or Claude's weights. You have two architectural approaches:</p>
<ul>
<li><strong>RAG (Retrieval-Augmented Generation)</strong> — Retrieve relevant chunks from an external datastore and inject them into the prompt</li>
<li><strong>Long Context Windows</strong> — Stuff the entire knowledge base directly into the prompt (e.g., Claude's 200K tokens, Gemini's 1M+ tokens)</li>
</ul>
<p>Neither approach is universally better. The right choice depends on your data volume, query patterns, latency requirements, and budget.</p>
<h2>How RAG Works</h2>
<ol>
<li><strong>Indexing</strong> — Split your documents into chunks (typically 256-1024 tokens), generate embeddings for each chunk, store in a vector database</li>
<li><strong>Retrieval</strong> — When a user query arrives, embed the query, find the top-K most similar chunks via vector similarity search</li>
<li><strong>Generation</strong> — Inject the retrieved chunks into the prompt as context, then generate the answer</li>
</ol>
<pre><code>System: Answer the user's question using ONLY the following context documents.
If the answer isn't in the provided context, say "I don't have information about that."
Context Documents:
{chunk_1}
{chunk_2}
{chunk_3}
User Question: {query}
<h2>How Long Context Works</h2>
<p>With models supporting 100K-1M+ token context windows, you can skip retrieval entirely:</p>
<pre><code>System: You are a documentation expert. Answer questions based on the following
complete documentation set.
{entire_documentation_contents}
User Question: {query}
This is conceptually simpler — no embedding pipeline, no vector database, no chunk management. But it comes with significant trade-offs.
<h2>Decision Matrix</h2>
<table>
<tr><th>Factor</th><th>RAG Wins</th><th>Long Context Wins</th></tr>
<tr><td><strong>Data Volume</strong></td><td>> 200K tokens (millions of docs)</td><td>< 200K tokens total</td></tr>
<tr><td><strong>Cost per Query</strong></td><td>Lower (only relevant chunks sent)</td><td>Higher (entire corpus every query)</td></tr>
<tr><td><strong>Latency</strong></td><td>Lower (smaller prompts = faster generation)</td><td>Higher (processing entire context)</td></tr>
<tr><td><strong>Accuracy</strong></td><td>Can miss relevant context (retrieval failures)</td><td>Model sees everything (no retrieval gap)</td></tr>
<tr><td><strong>Freshness</strong></td><td>Near real-time (re-index changed docs)</td><td>Requires re-sending updated corpus</td></tr>
<tr><td><strong>Infrastructure</strong></td><td>Vector DB + embedding pipeline required</td><td>No additional infrastructure</td></tr>
<tr><td><strong>Multi-hop Reasoning</strong></td><td>Weak (chunks may lack cross-references)</td><td>Strong (model sees full context)</td></tr>
<tr><td><strong>Complexity</strong></td><td>Higher (chunking, embedding, retrieval tuning)</td><td>Lower (just concatenate and send)</td></tr>
</table>
<h2>When to Use RAG</h2>
<ul>
<li><strong>Large knowledge bases</strong> — Customer support with 10,000+ articles, legal document search, enterprise wikis</li>
<li><strong>Cost-sensitive applications</strong> — High query volume where per-token costs matter</li>
<li><strong>Frequently updated data</strong> — Product catalogues, pricing databases, inventory systems</li>
<li><strong>Multi-tenant applications</strong> — Each user has their own data; retrieval naturally scopes to their documents</li>
<li><strong>Latency-critical</strong> — Sub-second response times where processing 200K tokens is too slow</li>
</ul>
<h2>When to Use Long Context</h2>
<ul>
<li><strong>Small, stable corpora</strong> — A single codebase, a company handbook, a product specification</li>
<li><strong>Cross-reference-heavy tasks</strong> — Legal contract analysis, code review across multiple files, research synthesis</li>
<li><strong>Simplicity is paramount</strong> — Prototyping, MVP stage, small teams without ML infrastructure</li>
<li><strong>Summarisation tasks</strong> — The model needs to see the entire document to summarise it properly</li>
<li><strong>One-shot analysis</strong> — Upload a document, ask questions, discard. No need to persist embeddings</li>
</ul>
<h2>The Hybrid Approach</h2>
<p>The best production systems often combine both approaches:</p>
<pre><code>async function answerQuery(query: string): Promise<string> {
// Step 1: RAG retrieval for candidate documents
const candidates = await vectorDB.search(embed(query), { topK: 20 });
// Step 2: Re-rank candidates
const reranked = await reranker.rank(query, candidates);
// Step 3: Take top results and use long context for synthesis
const topDocs = reranked.slice(0, 5).map(d => d.fullContent);
// Step 4: Send full documents (not chunks) to a long-context model
const answer = await longContextModel.generate({
system: "Answer based on these documents. Cite specific sections.",
context: topDocs.join('\n---\n'),
query: query
});
return answer;
}
This gives you RAG's efficiency for retrieval with long-context's accuracy for synthesis. You search broadly (RAG) then reason deeply (long context).
<h2>RAG Pitfalls to Avoid</h2>
<ul>
<li><strong>Chunk sizes too small</strong> — Chunks under 200 tokens often lack sufficient context. Start with 512-1024 tokens</li>
<li><strong>No overlap between chunks</strong> — Use 10-20% overlap so sentences aren't split mid-thought</li>
<li><strong>Ignoring metadata</strong> — Always store and filter by document metadata (source, date, category) alongside embeddings</li>
<li><strong>No evaluation pipeline</strong> — You need to measure retrieval quality (hit rate, MRR) separately from generation quality</li>
<li><strong>Embedding model mismatch</strong> — Use the same embedding model for indexing and querying. Mixing models destroys similarity metrics</li>
</ul>
<h2>How AI Prompt Architect Helps</h2>
<p>Whether you use RAG or long context, the <strong>prompt matters most</strong>. AI Prompt Architect's <strong>Generate</strong> workflow creates structured prompts that work with either architecture — including context injection points, grounding instructions ("only answer from the provided context"), and citation formatting. The <strong>Analyse</strong> workflow can evaluate your RAG prompts for common failure modes like context window overflow and instruction confusion.</p>
<p>Building a RAG backend in Python with Django? Our guide on <a href="/docs/django-rest-framework-prompt">scaffolding Django REST Framework APIs</a> shows how to structure your retrieval endpoints with proper serialization, filtering, and pagination.</p>
This article was originally published with extended interactive STCO schemas on AI Prompt Architect.
Top comments (0)