Production RAG at Scale: Architecture Patterns for 1M+ Documents

Retrieval-augmented generation (RAG) systems for production require different design than demo-scale systems. This article summarizes documented patterns for systems indexing 1M+ documents, drawing on published engineering reports and community best practices.

Why scale changes everything

At demo scale (<100 docs), basic embedding + cosine similarity works. At production scale (1M+ docs), several patterns emerge:

1. Embedding model selection matters more
2. Retrieval latency grows non-linearly with corpus
3. Hallucination rates grow with irrelevant retrievals
4. Cost per query can become prohibitive

This article focuses on architectures that handle these constraints.

Tiered retrieval architecture

Query
  -> Query rewriting (LLM)
  -> Embedding model
  -> Tier 1: BM25 keyword search (top-100)
  -> Tier 2: Vector similarity (top-100)
  -> Reciprocal rank fusion (RRF) -> top-50
  -> Cross-encoder rerank -> top-10
  -> LLM with context window
  -> Streaming response

Source: Microsoft's RAG research, "Retrieval Augmented Generation for Large Language Models: A Survey" (Gao et al., 2024). Pre-print at https://arxiv.org/abs/2312.10997

Embedding model selection at scale

For 1M+ docs:

• text-embedding-3-large (OpenAI): 1536-3072 dims, $0.13/M tokens. Mature API. Best for general use.
• voyage-large-2 (Voyage AI): Competitive on retrieval benchmarks, especially legal/medical domains.
• cohere-embed-v3 (Cohere): 1024 dims, supports compression. Good for cost-sensitive applications.
• BGE-large-en-v1.5 (open-source): Self-hostable, no per-token API cost. Slightly lower recall but no API dependency.

Benchmark: MTEB Leaderboard (https://huggingface.co/spaces/mteb/leaderboard)

Vector database selection

For 1M+ docs with HNSW index:

• pgvector (Postgres extension): Self-hostable, transactional, good for hybrid workloads.
• Pinecone (managed): Fast at scale ($70/mo for 1M+ vectors), proprietary.
• Weaviate (open-source/managed): Hybrid search built-in, BM25 + vector.
• Qdrant (open-source/managed): Rust-based, low latency.
• Milvus (open-source): Strong at billion-scale.

For multi-tenant systems, consider per-tenant index partitioning.

Hybrid search: BM25 + vectors

For legal/medical text, neither pure keyword nor pure vector retrieval is optimal. Hybrid approach:

1. Generate BM25 scores (Elasticsearch/OpenSearch)
2. Generate vector similarity scores (vector DB)
3. Combine via Reciprocal Rank Fusion (RRF)
4. Optionally rerank with cross-encoder

RRF formula: score(d) = sum(1 / (k + rank_i(d))) for each ranking.

Source: Cormack et al. (2009), "Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods."

Chunking strategy

For 1M+ docs at scale:

• Recursive character chunking: ~95% recall at reasonable compute cost
• Semantic chunking: ~98% recall, but 5-10x index time
• Structural chunking (legal: Article/Section): Best for legal texts

Recommended chunk size: 256-512 tokens, 10-20% overlap.

Source: LangChain text splitters documentation (https://python.langchain.com/docs/modules/data_connection/document_transformers/)

Latency budget

For interactive chat systems:

• Target p95 <2s end-to-end
• Typical breakdown:
  • Query rewrite: 200-400ms
  • Embedding: 100-300ms
  • BM25 + vector search: 100-200ms
  • Rerank: 200-500ms
  • LLM first-token: 500-1500ms
  • Streaming: ongoing

If p95 >3s, user retention drops materially.

Hallucination mitigation

For production legal RAG:

1. Constrain LLM: "Cite only documents provided. Cite as [N] in response."
2. Verify citations post-hoc: Match each cited [N] to actual source.
3. Track hallucination rate weekly: Should be <2% in legal contexts.
4. Use eval framework (ragas, TruLens, DeepEval).
5. Have human review for high-stakes queries.

Source: Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (Magesh et al., 2024). Working paper at https://reglab.stanford.edu/publications/hallucination-free-assessing-reliability-leading-ai-legal-research-tools

Cost optimization

Top 5 levers (estimated monthly savings at 100K queries/mo):

1. Query cache (Redis): 60% hit rate -> -30% LLM cost
2. Mix models (Haiku for simple, Sonnet for complex): -50% LLM cost
3. Reduce retrieval R from 10 to 5: -40% token cost
4. Use smaller embedding model: -25% embedding cost
5. Batch async queries: -10% system cost

Realistic monthly cost at 100K queries: $300-800.

Eval framework

Track weekly:

• Retrieval precision@K (% of retrieved docs that are relevant)
• Retrieval recall@K (% of relevant docs that are retrieved)
• Faithfulness (does answer stick to retrieved context)
• Answer relevance (is answer topical)
• Citation accuracy (% of citations that resolve to real sources)

Tools:

• ragas (Python): https://docs.ragas.io/
• TruLens: https://www.trulens.org/
• DeepEval: https://docs.confident-ai.com/

Operational considerations

• Monitor drift: Re-evaluate weekly to catch regressions
• Version-pin models: Embedding and LLM model versions matter
• Cache invalidation: When corpus changes, invalidate stale cache
• Multi-region: For global latency, deploy in multiple regions
• Cost alerts: Set budget alerts at the LLM provider level

References

1. Gao et al. (2024). Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997
2. Magesh et al. (2024). Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Stanford RegLab.
3. Cormack et al. (2009). Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods.
4. MTEB Leaderboard: https://huggingface.co/spaces/mteb/leaderboard
5. LangChain documentation: https://python.langchain.com/

Acknowledgments

This article summarizes documented engineering practices. For specific implementations, consult official documentation and benchmark on your own data.

Dillon Deutsch has built production RAG systems serving users across all 50 US states. https://courtgpt.ai