RAG Architecture: 7 Mistakes That Kill Your Search Quality in Production

I've spent the last few months building RAG (Retrieval-Augmented Generation) systems that handle 100GB+ of private documents. What I learned: most RAG failures aren't about the LLM. They're about the retrieval layer — the part most tutorials gloss over.

Here are the 7 architecture mistakes I see repeatedly in production RAG systems, and how to fix each one.

---

1. The Chunking Trap

The mistake: Blindly splitting documents into fixed-size chunks (512 tokens, 1000 chars) without respecting semantic boundaries.

Why it kills quality: A single paragraph gets fractured across multiple chunks. When retrieval pulls back chunk #3 of a 5-part split, the LLM has no idea what "this approach" or "the second option" refers to. Your context is literally fragmented.

The fix: Parent-Document + Small-to-Large retrieval strategy:

• Chunk by semantic boundaries: paragraphs, sections, code blocks
• Index small child chunks for precise vector matching
• Return the full parent document as context when a child matches
• Add 10-20% overlap between chunks to capture boundary information

# Don't do this
chunks = split_by_token(text, chunk_size=512)

# Do this instead
chunks = split_by_semantic_boundaries(
    text,
    separators=["\n\n", "\n", "。", "！", "？"],
    chunk_size=1000,
    overlap=200
)

2. The 50/50 Weight Trap

The mistake: Setting hybrid search weights to 0.5 vector + 0.5 keyword and calling it done.

Why it kills quality: Different queries need different weight profiles. "What is PostgreSQL?" needs keyword precision. "How do I optimize slow queries?" needs semantic understanding. A fixed 50/50 split is mediocre at both.

The fix: Dynamic weight routing based on query type:

• Factual queries → bias toward BM25 (0.35:0.65)
• Semantic/how-to queries → bias toward vectors (0.75:0.25)
• Default production value → 0.6:0.4 (vector-leaning)
• Run A/B tests to calibrate for your domain

Technical docs with lots of terminology? BM25 matters more. Conversational Q&A? Vectors win.

3. The Embedding Model Default

The mistake: Using whatever embedding model the framework defaults to without testing it on your data.

Why it kills quality: General-purpose embeddings (trained on web text) perform poorly on domain-specific content. A model that excels at news articles will struggle with API documentation, medical records, or legal contracts.

The fix: Benchmark embeddings on your actual workload:

1. Create a test set of 50-100 representative queries with known good answers
2. Test 3-4 embedding models against this set
3. Measure MRR (Mean Reciprocal Rank) — not cosine similarity
4. Pick the winner and re-evaluate quarterly

For code-heavy workloads, models trained on code repositories will dominate general-purpose ones.

4. The "One Vector Store Fits All" Fallacy

The mistake: Choosing a vector database based on hype rather than workload characteristics.

Why it kills quality: Your data shape determines the right tool. 10,000 documents with frequent updates needs a different architecture than 10 million static documents with occasional reads.

The fix: Match the tool to the workload:

• < 100K vectors, frequent updates → In-memory (FAISS, HNSW in-process)
• 100K-10M vectors, managed → Pinecone, Weaviate, Qdrant
• 10M+ vectors, cost-sensitive → pgvector on PostgreSQL, Milvus
• Need hybrid search out of the box → Elasticsearch with dense_vector, Typesense

Don't over-engineer for scale you don't have. Don't under-engineer for scale you'll hit in 6 months.

5. The No-Reranking Shortcut

The mistake: Returning top-K vector search results directly to the LLM without a reranking step.

Why it kills quality: Vector search is great at finding "approximately relevant" documents but terrible at ordering them by actual relevance. Position 3 might be significantly more useful than position 1 — but your LLM gets position 1 first and context-window pressure degrades the rest.

The fix: Add a cross-encoder reranker as a second stage:

• Stage 1: Vector search retrieves top 50-100 candidates (fast, approximate)
• Stage 2: Cross-encoder reranks top 50 down to top 5-10 (slower, precise)
• Pass only the reranked top results to the LLM

This two-stage pattern typically improves answer quality by 20-40% with minimal latency cost. Models like bge-reranker-large or Cohere rerank work well.

6. The Context Window Bloat

The mistake: Stuffing every retrieved chunk into the prompt until you hit the context limit.

Why it kills quality: More context ≠ better answers. Irrelevant context actively degrades response quality (studies show LLMs get confused by noise). And you're burning tokens on content that doesn't contribute to the answer.

The fix:

• Set a context budget (e.g., 8K tokens for retrieval context)
• Select chunks by relevance score, not by count
• Deduplicate overlapping chunks before sending to the LLM
• Include a relevance instruction: "Only use information from the provided context"

Quality of context > quantity of context. Always.

7. The No-Metrics Black Hole

The mistake: Deploying RAG to production without retrieval quality metrics.

Why it kills quality: You can't fix what you don't measure. Without metrics, degradation happens slowly — users get slightly worse answers, they stop using the system, and you never know why.

The fix: Track these three metrics from day one:

1. Retrieval Precision@K: Of the top K retrieved chunks, how many are actually relevant?
2. Answer Faithfulness: Does the generated answer stick to the retrieved context?
3. User Feedback Loop: Thumbs up/down, re-query rate, session abandonment

Set up automated evaluation with a small gold-standard test set that runs nightly. If retrieval precision drops below your threshold, you'll know before your users do.

---

The Bottom Line

RAG is 80% retrieval engineering and 20% prompt engineering. Most teams spend 80% of their time on prompts and wonder why answers are mediocre.

Fix the retrieval layer first. Chunk semantically. Weight dynamically. Rerank aggressively. Measure everything.

The LLM is only as good as what you feed it.

---

What RAG pitfalls have you hit in production? I'd love to hear your war stories in the comments.