Large language models generate fluent text. They fail to meet grounding, traceability, freshness, and access control requirements. Retrieval-Augmented Generation addresses this by forcing models to answer using external evidence.
Early RAG used one simple pipeline. Production systems now use multiple architecture patterns. Each pattern targets a different failure mode. This post explains eight major RAG architectures used in production today.
Project references
NeuronDB site: https://www.neurondb.ai
Source code: https://github.com/neurondb/neurondb
What Is RAG
RAG links three systems: storage, retrieval, and generation. The storage layer holds your documents, chunks, and embeddings. The retrieval layer finds relevant evidence for each query. The generation layer produces answers conditioned on the retrieved context. The pipeline flows from query through evidence retrieval, context building, answer generation, and citation return. You get factual grounding, fresh data usage, private data isolation, and audit trace support. RAG shifted AI engineering from prompt tuning toward data pipeline engineering.
The storage layer supports multiple backends, including vector databases (Pinecone, Weaviate, Milvus), document stores (Elasticsearch, OpenSearch), and hybrid systems. The retrieval layer runs embedding models, keyword search, or graph traversal, depending on the architecture. The generation layer typically uses a large language model with a prompt template. The three layers communicate through a well-defined interface. You swap components without rewriting the full pipeline.
1. Naive RAG
Naive RAG uses direct vector similarity retrieval with no feedback loop. The name comes from the original RAG paper. The architecture remains the baseline for comparison.
Pipeline
Document ingestion loads raw text from files, databases, or APIs. Preprocessing normalizes whitespace, strips markup, and segments by logical boundaries. Text chunking splits documents into fixed-size or variable-size segments. Common choices: 256 tokens, 512 tokens, or sentence-based chunks. Embedding generation converts each chunk into a vector using a pretrained model. Vector storage writes embeddings to a vector database with metadata (source doc, chunk index, timestamp). At query time, the user submits a question. Query embedding converts the question into a vector. Vector search returns the top-k nearest chunks by cosine similarity or Euclidean distance. Context injection concatenates retrieved chunks into a prompt. Response generation passes the prompt to an LLM. Citation return attaches source references to the output.
Strengths
Implementation takes one to two weeks for an experienced engineer. Infrastructure cost stays low: one embedding model, one vector store, one LLM endpoint. The approach works well for static knowledge domains. FAQ corpora, product documentation, and internal wikis fit this pattern. Latency stays under 2 seconds for most deployments. No feedback loops mean deterministic behavior. The same query returns the same retrieval set. Debugging is straightforward.
Weaknesses
No verification loop validates retrieved evidence. Irrelevant chunks slip through when embedding similarity is misleading. Ranking quality depends entirely on embedding similarity. Ambiguous queries return weak results. A query like "how do I fix the error" returns generic troubleshooting content rather than error-specific documentation. Multi-faceted queries suffer. A question about "pricing and integration" retrieves only chunks for one facet. The model hallucinates to fill gaps when retrieval fails.
Chunk Size
Chunk size selection impacts recall quality. Small chunks (128 tokens) give precise matches but miss context. A section on "connection timeout" often fails to identify the cause or solution. Large chunks (512 tokens) capture more context but dilute relevance. The top-k retrieval returns fewer distinct documents. Overlap between chunks (50 tokens) helps preserve context across boundaries. Test multiple chunk sizes (128, 256, 512) against your query set. Measure recall at k=5 and k=10. Choose the size where recall plateaus.
Embedding Models
Embedding model choice impacts semantic coverage. Models trained on general text (OpenAI text-embedding-ada-002, sentence-transformers/all-MiniLM) underperform on domain-specific corpora. Medical, legal, and financial texts use terminology absent from training data. Use domain-tuned embeddings when available. Fine-tune on your corpus with contrastive loss. Or use domain-specific models (e.g., BioBERT for medical applications). Embedding dimension matters. 384-dim models are faster and cheaper. 1536-dim models capture finer distinctions. Benchmark both on your data.
Production Use Cases
FAQ bots with fewer than 10,000 questions. Documentation search for product manuals and API references. Internal knowledge bases where content changes infrequently. POCs and demos where speed of implementation outweighs accuracy.
2. Agentic RAG
Agentic RAG adds planning, tool selection, and iterative reasoning. The agent breaks complex questions into steps, chooses tools for each step, executes them, and synthesizes a final answer. This architecture handles workflows that a single retrieval call cannot support.
Pipeline
Task planning analyzes the user query and produces a step-by-step plan. The planner uses an LLM with few-shot examples or a structured prompt. Plan steps include "retrieve documents about X," "call API Y," and "summarize results." Tool selection maps each step to a tool. Tools include vector search, keyword search, calculator, API calls, and code execution. The agent selects tools based on step descriptions and tool schemas. Multi-step retrieval executes tools in sequence. Outputs from earlier steps feed into later steps. A retrieval about "company revenue" informs a follow-up retrieval about "competitor revenue." Tool execution runs each tool and captures results. Memory update stores tool outputs, intermediate conclusions, and user feedback. Response synthesis generates the final answer from the accumulated context. The agent loops back to planning when synthesis indicates missing information.
Strengths
Handles complex workflows. A query like "compare our Q3 results to our top three competitors and summarize the gap" requires multiple retrievals, API calls, and summarization. You run multiple tools in sequence. Long-running reasoning tasks become feasible. Research assistants draw from papers, patents, and news sources. Competitive intelligence agents aggregate data from multiple sources. Autonomous analytics agents run queries, join data, and produce reports.
Weaknesses
Latency increases with each planning and execution step. A single query often triggers 3 to 10 model calls. End-to-end latency reaches 10 to 30 seconds. Debugging is hard. The agent chooses different tools or paths for similar queries. Reproducing a failure requires logging every decision. Infrastructure cost rises. Each step consumes tokens. State tracking, retry logic, and execution budget control add engineering overhead.
Implementation Guidance
Set a maximum step count. Without limits, agents loop or drift. A typical cap is 5 to 10 steps. Log every tool call and plan step. Store the full execution trace. Reproducibility matters when users report errors. Use deterministic seeds where possible for plan generation. Define tool schemas with clear descriptions. The agent relies on schemas to select tools. Vague descriptions cause wrong tool selection. Implement timeouts per step. A stuck tool blocks the whole pipeline. Add fallback behavior when tools fail. The agent should degrade gracefully.
Production Use Cases
Research automation: literature review, patent analysis, trend summarization. Competitive intelligence: market monitoring, competitor tracking, strategic briefs. Autonomous analytics: ad-hoc reporting, data exploration, dashboard generation.
3. HyDE RAG
HyDE (Hypothetical Document Embeddings) generates synthetic documents to improve retrieval matching. The idea: hypothetical answers are closer in embedding space to real answers than raw queries. This bridges the vocabulary gap between how users ask and how documents are written.
Pipeline
The user submits a query. Hypothetical answer generation produces one or more plausible answers using an LLM. A query like "how do I configure SSL" might generate "To configure SSL, you need to generate a certificate, add the certificate path to the config file, and restart the server." Embedding generation converts the hypothetical answer into a vector. Retrieval uses this vector instead of the query vector to search the corpus. The retrieved chunks are real documents, not hypothetical. Context assembly concatenates retrieved chunks. The final generation produces the actual answer from the retrieved evidence. The model cites real sources rather than hypothetical answers.
Variations
Single HyDE generates one hypothetical answer per query. Multi-HyDE generates 3 to 5 hypothetical answers, embeds each, retrieves the corresponding results for each, and merges the results. Multi-HyDE improves recall, but multiplies cost. HyDE with reranking adds a reranker after retrieval. The reranker scores chunks based on their relevance to the original query. This filters false positives from the expanded retrieval set.
Strengths
Recall quality improves. Benchmarks report 10-30% recall gains over naive retrieval. Vocabulary mismatch between queries and corpus documents drops. Users ask, "Why is my app slow?" while docs say "performance degradation" and "latency issues." Hypothetical answers use doc-like language. Technical search benefits most. Developer questions, error messages, and API usage patterns align better after HyDE.
Weaknesses
Extra inference: the model must generate a hypothetical answer before retrieval. Expect 1.5x to 2x token usage per query. Synthetic bias is a risk. Generated documents sometimes skew retrieval toward certain document types. A model trained on tutorials often generates tutorial-style hypotheticals and over-retrieves tutorials. Production use cases include developer search, technical troubleshooting, and scientific literature retrieval. HyDE works best when combined with reranking models. The reranker filters false positives from the expanded retrieval set.
Implementation Guidance
Use a fast, cheap model for hypothetical generation. You do not need the best model. A 7B parameter model often suffices. Keep hypothetical answers concise. Long hypotheticals add noise. 50 to 100 tokens per hypothetical works well. Consider caching. Repeated queries (e.g., popular FAQs) reuse cached hypotheticals. Cache key: query embedding.
4. Graph RAG
Graph RAG is extracted from entity relationships in knowledge graphs. Documents become nodes and edges. Queries traverse the graph to assemble context. This architecture excels when relationships matter as much as raw text.
Pipeline
Entity extraction identifies named entities in a document, such as people, organizations, products, and concepts. Extraction uses NER models, rule-based patterns, or LLM-based parsing. Entity linking resolves extracted entities to canonical IDs. "Apple Inc" and "Apple" map to the same node. Linking uses knowledge bases (Wikidata, DBpedia) or custom ontologies. Graph construction creates nodes for entities and edges for relationships. Relationships come from co-occurrence, dependency parsing, or relation extraction models. Graph storage writes to a graph database (E.g., Neo4j or Amazon Neptune) or to an in-memory graph. At query time, query understanding identifies entities mentioned in the query. Graph traversal starts from those entities and follows edges. Traversal strategies include k-hop neighborhood, path finding, and community detection. Context assembly pulls text from documents associated with traversed nodes. Generation produces an answer from the assembled context.
Strengths
Multi-hop reasoning becomes tractable. "What drugs interact with the patient's current medication?" requires chaining drug-to-drug relationships across multiple hops. The answer depends on chaining relationships across multiple entities. Explainability is strong. The reasoning path follows explicit graph edges. You show users the path from query entities to answer entities. Relationship-aware retrieval surfaces related concepts naive vector search misses.
Weaknesses
Graph construction is expensive. Entity extraction and linking require trained models or rules. Expect weeks of tuning for a new domain. Schema design is complex. You must decide which relationship types matter for retrieval. Too many relationship types create noise. Too few missed connections. Graph refresh pipelines must align with source data refresh cycles. Stale graphs return stale answers. Production use cases include healthcare decision support, fraud detection, and scientific research, where relationship structure matters as much as raw text.
Implementation Guidance
Start with a minimal schema. Two or three relationship types (e.g., "treats," "interacts with") often suffice. Add more as you validate the need. Use hybrid retrieval. Combine graph traversal with vector search. Graph finds structure. Vectors find semantic similarity. Run incremental updates. Rebuild the full graph only when schema changes. For daily doc updates, add or update affected nodes and edges.
5. Corrective RAG
Corrective RAG adds self-validation and iterative refinement. The system generates an answer, critiques it, and re-retrieves or regenerates when the critique identifies issues. The loop continues until the answer meets a quality threshold. This architecture is well-suited to high-stakes domains where errors are costly.
Pipeline
The initial retrieval fetches the top-k chunks for the query. Initial generation produces a draft answer. Critique evaluates the draft. The critic checks: does the answer cite retrieved evidence? Are claims supported? Are there contradictions? The critic uses an LLM with a structured prompt or a trained classifier. Scoring produces a numeric score (0 to 1) or a pass/fail. Re-query triggers when the critique finds missing evidence or unsupported claims. The re-query reformulates the search or expands k. Re-generation produces a new draft from the expanded context. The loop repeats until the score exceeds a threshold or the maximum iterations (e.g., 3) are reached. Final output returns the best-scoring answer with citations.
Strengths
Factual accuracy improves. Benchmarks show a 15 to 25 percent reduction in hallucination rate. Hallucination rate drops. The critic catches unsupported claims before they reach the user. The architecture suits applications that require robust audit trails. Financial analytics, legal research, and compliance systems need traceable reasoning. Each answer comes with a critique log.
Weaknesses
Higher latency. Most implementations run 2 to 4 generation passes per query. Latency doubles or triples. Token usage rises proportionally. Engineering teams must design scoring functions for the critique stage. The critic must reliably detect factual errors or missing evidence. A weak critic adds cost without benefit. False negatives let errors through. A harsh critic triggers unnecessary re-retrieval. False positives waste tokens and time. Tuning the critic is non-trivial.
Implementation Guidance
Start with a simple criticism: "Does each claim have a citation?" Then add checks for contradiction and hallucination. Use the chain-of-thought for the critic. Ask the critic to explain its reasoning before scoring. This improves reliability. Set a conservative max iteration count. Three passes usually suffice. More passes yield diminishing returns. Log critique scores over time. Track the distribution. Drift indicates a need to retune.
Production Use Cases
Financial analytics: earnings summaries, risk reports, compliance checks. Legal research: case law retrieval, contract analysis, and regulatory lookup. Compliance systems: policy verification, audit support, regulatory reporting.
6. Contextual RAG
Contextual RAG uses conversation state and session memory. Retrieval considers prior turns. Generation maintains continuity. This architecture supports multi-turn dialogues where each question depends on context.
Pipeline
Session storage keeps a log of user messages and assistant responses. Each turn appends to the log. Context summarization runs when the log exceeds a token limit. Summarization compresses old turns into a shorter summary. The summary plus recent turns form the active context. Context-aware retrieval uses the full conversation, not only the latest message. A query "what about the second one?" is retrieved by concatenating "second one" with the prior discussion of a list. Some systems embed the full conversation. Others extract key entities and concepts for retrieval. Response generation receives the retrieved context plus conversation history. The model produces answers referencing prior turns. Memory updates with new information from the current turn for future retrieval.
Strengths
Multi-turn consistency improves. Follow-up questions receive correct answers. "What is the price?" after "Tell me about Product X" returns Product X's price. Personalization based on user history becomes possible. Preferences, prior queries, and corrections influence retrieval and generation. Session continuity supports long interactions. Meeting assistants, customer success tools, and personal knowledge systems rely on this.
Weaknesses
Memory drift is a risk. Stale or irrelevant context accumulates over long sessions. A conversation about "Product A" often drifts to "Product B," but retrieval remains biased toward A. Context contamination occurs when prior turns bias retrieval in unwanted ways. A user correction ("I meant Product B, not A") must override prior context. Implementation is tricky. Memory compaction must run periodically. Without compaction, context windows overflow, and relevance degrades. Summarization loses detail. Aggressive summarization drops information needed for later turns.
Implementation Guidance
Define a context window budget. Reserve tokens for conversation history, retrieval context, and generation. When history exceeds the budget, summarize the oldest turns. Use a sliding window with a summary: keep the last N turns verbatim and summarize the rest. Store user corrections explicitly. "User clarified X" should override prior assumptions. Test with long sessions. Simulate 20-turn conversations. Measure consistency and relevance at turn 5, 10, 15, 20.
Production Use Cases
Meeting assistants: summarization, action items, follow-up questions. Customer success tools: support dialogues, onboarding flows, and feature discovery. Personal knowledge systems: note-taking, research assistants, learning companions.
7. Modular RAG
Modular RAG splits retrieval into independent components. Each component has a single responsibility. You swap, upgrade, or bypass components without rewriting the pipeline. This architecture supports complex enterprise needs where one-size-fits-all fails.
Pipeline
Query rewriting normalizes and expands the user query. Spelling correction, query expansion, and multi-query generation (HyDE-style) run here. Hybrid retrieval runs multiple search strategies in parallel. Vector search, keyword search, and graph traversal execute concurrently. Results feed into a fusion step. Filtering removes irrelevant results. Filters apply metadata constraints (date range, source, access control). Deduplication merges near-duplicate chunks. Reranking scores the filtered set with a cross-encoder or learned ranker. Reranking is expensive, so you run reranking on the top 20 to 50 candidates. Tool routing sends queries to specialized tools. A legal query goes to the legal corpus. A support query goes to the support corpus. Routing uses classifiers or keyword rules. Response synthesis assembles the final answer. Synthesis calls the LLM once or multiple times. Some architectures add a citation verification step.
Strengths
Each module upgrades or replaces independently. Swap the embedding model without touching the retrieval logic. Add a new data source by adding a retrieval module. The architecture supports flexible enterprise workflows. Different departments need different corpora and rules. Modular RAG accommodates this. Adding new data sources or retrieval strategies is straightforward. Implement a new module. Add the module to the pipeline. Configure routing.
Weaknesses
System complexity rises. A full modular pipeline has 6 to 10 components. Each component has its own config, dependencies, and failure modes. Maintenance cost rises. Observability across modules becomes critical. Failures occur at any stage. A bug in query rewriting silently corrupts downstream retrieval. You need per-module metrics and tracing. Latency adds up. Each module adds milliseconds. End-to-end latency requires careful optimization. Production use cases include enterprise AI platforms, large data pipeline systems, and research automation systems where modularity is a core requirement.
Implementation Guidance
Define clear interfaces between modules. Each module accepts a standard input format and produces a standard output format. Use a pipeline framework (e.g., LangChain, LlamaIndex, or a custom DAG) to enforce this. Instrument every module. Log inputs, outputs, and latency. Add tracing IDs to follow a query across modules. Version your pipeline. When you change a module, record the version. A/B test module changes before full rollout. Start minimal. Add modules only when you have a concrete problem. A 3-module pipeline (retrieve, rerank, generate) often suffices for early deployments.
Production Use Cases
Enterprise AI platforms: multi-tenant, multi-corpus, role-based access. Large data pipeline systems: billions of documents, multiple retrieval backends. Research automation: federated search, specialized tools, reproducibility.
8. Hybrid RAG
Hybrid RAG combines keyword retrieval and semantic retrieval. Keyword search finds exact and lexical matches. Semantic search finds conceptual matches. Together, they cover cases where either alone fails.
Pipeline
Query parsing extracts keywords and optionally generates a semantic query. Keyword search runs on an inverted index (e.g., BM25 or Elasticsearch). Semantic search runs against a vector index. Both return ranked lists. Rank fusion merges the two lists. Reciprocal Rank Fusion (RRF) is the common baseline: score = sum(1/(k + rank)) across lists. k is typically 60. Other methods include weighted linear combination and learned fusion. Optionally, reranking scores the fused list. Reranking uses a cross-encoder or a learned model. Generation receives the top chunks and produces the answer.
Keyword vs Semantic
Keyword search excels at exact matches. Product IDs, error codes, and proper nouns. "ERR_SSL_PROTOCOL_ERROR" retrieves the right doc. Semantic search fails here if the embedding does not capture the token. Semantic search excels at paraphrasing and conceptual queries. "How do I fix connection problems" matches "troubleshooting network connectivity." Keyword search misses this. Hybrid covers both. A query about "Q3 revenue" gets keyword hits on "Q3" and "revenue" plus semantic hits on earnings reports and financial summaries.
Strengths
Precision comes from keyword matching. Recall comes from semantic search. Structured and unstructured data both work. Keyword search handles tables, metadata, and structured fields. Semantic search handles free text. Production use cases include legal search, compliance audits, and enterprise search platforms, where both precision and recall matter.
Weaknesses
Ranking tuning is complex. Rank fusion models require continuous optimization. You must balance keyword and semantic signals. RRF assumes equal contribution. Your data often needs different weights. Learned fusion models often outperform RRF but need training data. You need labeled query-document pairs. Tuning is iterative. Add keyword weight when users complain about missed exact matches. Assign semantic weight when users report missed conceptual matches.
Implementation Guidance
Start with RRF. No training required. Tune k (typically 40-80) on a small validation set. Add metadata filters. Both keyword and semantic results benefit from source, date, and access filters. Consider query-type routing. Short queries (1 to 3 words) often need more keyword weight. Long, conceptual queries need more semantic weight. Implement both paths in parallel. Parallel execution keeps latency low. Fusion adds minimal overhead.
Production Use Cases
Legal search: case law, contracts, regulations. Compliance audit: policy lookup, regulatory check. Enterprise search: intranet, document management, knowledge base.
Cross-Architecture Comparison
Naive RAG: low complexity, medium accuracy, low cost, low latency. Implementation in days. Best for static, narrow corpora.
Agentic RAG and Modular RAG: high complexity, high accuracy, high cost, higher latency. Implementation in weeks or months. Best for complex workflows and enterprise needs.
Corrective RAG: high accuracy, high latency, high token usage. Best for high-stakes domains where verification matters.
HyDE, Contextual, and Hybrid RAG: medium complexity, cost, and latency with accuracy gains over Naive RAG. Implementation in one to two weeks. Best for technical search, multi-turn dialogue, or mixed precision-recall needs.
Choose an architecture by failure mode. Naive RAG solves simplicity. Agentic RAG solves autonomy. HyDE solves vocabulary mismatch. Graph RAG solves relationship reasoning. Corrective RAG solves verification. Contextual RAG solves memory. Modular RAG solves enterprise workflow composition. Hybrid RAG solves the balance between precision and semantic coverage.
Decision Framework
Ask: Does your corpus change frequently? Yes favors Naive or Modular. Does your domain have rich entity relationships? Yes, favors Graph. Do users ask multi-turn questions? Ye,s favors Contextual. Do you need high factual accuracy and audit trails? Yes, favors Corrective. Do users and docs use different terminology? Yes, favors HyDE or Hybrid. Do you need multiple tools and complex workflows? Yes favors Agentic or Modular.
Conclusion
RAG is no longer a single architecture. Each pattern solves a specific problem. Production success depends on pipeline design, data quality, and evaluation discipline. The strongest systems combine multiple RAG patterns into a single, orchestrated platform. A single system might use Hybrid retrieval, Corrective verification, and Contextual memory. Future RAG systems will look less like search pipelines and more like distributed data operating systems.
Top comments (0)