Ama Senevirathne

Posted on Jul 26

RAG Architecture in Production: The Decisions That Actually Determine Quality

#ai #machinelearning #architecture #programming

RAG Architecture in Production: The Decisions That Actually Determine Quality

By Ama Senevirathne | Full-Stack Engineer | .NET / Angular / Agentic AI

Tags: rag, ai, software-architecture, llm, dotnet, vector-databases, information-retrieval

Retrieval-Augmented Generation (RAG) is the architecture that lets you ask an LLM questions about your own data — your internal documentation, your product knowledge base, your codebase, your customer records — without fine-tuning the model and without stuffing your entire corpus into the context window.

The concept is simple: before generating a response, retrieve the relevant documents; include them in the context; generate against real evidence rather than training-data approximations.

The implementation is not simple. Most teams get RAG to work in a demo relatively quickly. Most teams then discover that "working in a demo" and "reliable at production quality" are separated by a series of non-obvious decisions that the tutorials don't cover, because the tutorials are optimised for getting a working prototype in 20 lines of code, not for the precision, recall, and latency characteristics that determine whether users trust the system.

This article covers those decisions. Everything here is grounded in systems I have built or reviewed, with references to primary sources where the evidence is worth reading directly.

The Architecture at a Glance

A RAG pipeline has three distinct phases:

Ingestion Pipeline (offline)
    │
    ├── Load documents
    ├── Clean and split (chunking)
    ├── Embed each chunk
    └── Store in vector index

Query Pipeline (online, per-request)
    │
    ├── Embed the query
    ├── Retrieve top-K chunks (vector similarity)
    ├── [Optional] Rerank
    ├── [Optional] Hybrid search (vector + BM25)
    ├── Compose context (retrieved chunks + query)
    └── Generate response (LLM)

Each stage has at least one decision that can halve or double your end-quality. Let me go through them in order.

Stage 1: Chunking — The Highest-Leverage Decision in the Entire Pipeline

Chunking is splitting your source documents into the units that will be stored and retrieved. It is also the single most impactful decision in a RAG system, because chunks that are too large drown the relevant signal in noise, and chunks that are too small lose the context that gives the signal meaning.

Fixed-size chunking

Split every N tokens with an overlap of M tokens. Simple to implement and reason about. Works tolerably when documents are homogeneous (dense prose, structured reports). Fails when documents mix content types (a README with code blocks, prose explanation, and a table will be split arbitrarily across all three).

The overlap parameter exists to prevent important content from falling in the gap between two chunks. A reasonable starting point is 20% overlap — for 512-token chunks, that's ~100 tokens of shared context between adjacent chunks.

Structure-aware chunking

Parse the document's structure and chunk at semantic boundaries: paragraphs, sections, functions, classes. For Markdown, split on headers. For code, split on function or class definitions. For HTML, split on <section> or <article> elements.

This is almost always superior to fixed-size chunking. The cost is that it requires format-specific parsers. For a mixed-format corpus, you need multiple strategies and a format-detection step.

The parent-child chunk pattern

Store two representations of each chunk: a small chunk for retrieval precision, and a larger parent chunk (the section the small chunk belongs to) that is sent to the LLM for generation. Retrieve on the small chunk; generate on the parent.

This combination — precise retrieval, rich generation context — resolves the core tension between retrieval quality and generation quality. Small chunks retrieve the right passage; large chunks give the LLM enough context to answer the question coherently.

I use this pattern in production. It costs more index storage (two copies of each chunk), but the quality improvement is worth it on any non-trivial corpus.

Practical defaults:

Retrieval chunk: 256–512 tokens, 10–20% overlap
Parent chunk: the full section (1,000–2,000 tokens)
Always store source document ID, section title, and chunk position as metadata

Stage 2: Embedding Model Selection

The embedding model converts text to vectors. Retrieval works by finding the vectors in the index that are closest to the query vector in the embedding space. If the model's notion of "similarity" doesn't match your domain's notion of relevance, nothing downstream can compensate.

General-purpose vs. domain-specific embeddings

OpenAI's text-embedding-3-large and Cohere's embed-multilingual-v3.0 are the current general-purpose benchmarks for English. They perform well on diverse corpora. For domain-specific corpora — medical, legal, code, financial — domain-trained models consistently outperform general-purpose ones on in-domain retrieval tasks, often by a substantial margin on benchmarks like BEIR.

The recommendation: start with text-embedding-3-large or an open-source equivalent (bge-large-en-v1.5 from Beijing Academy of AI, available on Hugging Face and free to self-host). Establish a retrieval quality baseline. Then evaluate domain-specific embeddings against that baseline on a representative sample of real queries. Only switch if you measure an improvement — not because the domain-specific model sounds more relevant.

Dimensionality

Higher-dimensional embeddings generally encode more information but require more storage and compute. text-embedding-3-large supports 1,536 dimensions by default and up to 3,072. For most production systems, 1,536 dimensions is the right balance. If you are running at large scale with strict cost constraints, 512 dimensions with a good general-purpose model is competitive.

Practical default: text-embedding-3-large at 1,536 dimensions for most English-language corpora. Self-host bge-large-en-v1.5 for cost-sensitive deployments.

Stage 3: Vector Store Selection

The vector store indexes your embeddings and serves approximate nearest-neighbour (ANN) queries. The core performance parameters are:

Recall@K: what fraction of the true top-K documents does the ANN index return? A recall of 0.95 means 5% of the truly relevant documents are missed on every query — that loss compounds across a pipeline.
QPS at p99: how many queries per second can the index serve at a 99th-percentile latency that is acceptable to your application?
Index build time and cost: for large corpora that need frequent re-indexing, this matters.

Option comparison (brief)

pgvector: PostgreSQL extension. If you are already running Postgres, this is the right starting point for corpora up to ~1M vectors. The HNSW index added in pgvector 0.5.0 substantially improved recall and query performance. Operationally, you get one system to manage. The tradeoff: pgvector at high scale requires careful tuning and does not match the raw QPS of purpose-built vector databases.

Qdrant: purpose-built vector database with HNSW indexing, payload filtering, and on-disk indexing support. Good choice for corpora in the 1M–100M vector range where pgvector starts to strain. Rust-based, high QPS, excellent filtering performance. Self-hostable or managed cloud.

Pinecone: managed cloud vector database, easiest operational path if you do not want to self-host. Good for teams where operational simplicity outweighs the cost premium. Not open-source.

Weaviate: supports multimodal embeddings, built-in BM25 hybrid search, and a GraphQL query interface. Good for teams that need hybrid search as a first-class feature.

My recommendation for most .NET projects: start with pgvector (already in your infrastructure), graduate to Qdrant when the corpus exceeds ~500K vectors or when query latency becomes a constraint.

Stage 4: Hybrid Search — Why Vector-Only Retrieval Misses Whole Categories

Vector similarity retrieval is good at semantic matching: "what is the cancellation policy?" retrieves chunks about cancellation even if they don't use the word "policy." It is poor at exact-match retrieval: "what does RFC 2119 say about MUST?" needs BM25 keyword matching, because the term "RFC 2119" is specific enough that semantic proximity to other documents is irrelevant.

Hybrid search combines vector similarity and BM25 keyword retrieval, then fuses the two ranked lists before reranking.

The standard fusion approach is Reciprocal Rank Fusion (RRF):

score(doc) = Σ 1 / (k + rank(doc, result_list))

where k is a constant (typically 60) that dampens the influence of very high ranks. RRF is simple, robust, and does not require tuning a weight between the two result sets — which is its main advantage over a weighted linear combination.

Practical implementation: BM25 via Elasticsearch or a lightweight library (BM25Okapi in Python, or pg_trgm + ts_vector in Postgres for fully integrated deployments). Fuse with RRF. For Weaviate or OpenSearch, hybrid search is a first-class API feature.

Stage 5: Reranking

After retrieval (vector + BM25 + RRF), you have a set of candidate chunks. Reranking runs a cross-encoder model over each (query, chunk) pair and produces a relevance score that is more accurate than the embedding distance or BM25 score, because the cross-encoder sees both texts simultaneously rather than encoding them independently.

Cross-encoders are computationally expensive — too expensive to run over the full index. The standard pattern is bi-encoder retrieval (fast, approximate) followed by cross-encoder reranking (slower, accurate) over the top-50 or top-100 candidates.

Good cross-encoders: Cohere Rerank (managed API), cross-encoder/ms-marco-MiniLM-L-12-v2 (open-source, Hugging Face), Flashrank (lightweight, built for low-latency reranking).

The improvement from adding a reranking step is consistently significant in production systems. On the BEIR benchmark, reranking with a cross-encoder improves nDCG@10 by 5–15 points over dense retrieval alone, depending on the task. The latency cost is real but manageable if you gate reranking to the top-N candidates.

Stage 6: Context Composition and Prompt Architecture

You now have the top-K relevant chunks. How you compose these into the LLM prompt determines generation quality as much as retrieval quality does.

Context window budgeting

Allocate the context window deliberately:

System instructions: ~500 tokens
Retrieved context: 60–70% of remaining budget
Conversation history (for multi-turn): 10–15%
Query + output space: remainder

Do not just concatenate all chunks until you hit the window limit. Sort by relevance score descending. If you have more chunks than fit, prefer the highest-ranked ones.

Grounding instructions

Include explicit instructions in your system prompt that direct the model to answer from the provided context:

Answer the question using ONLY the information provided in the context sections below.
If the context does not contain enough information to answer the question, say
"I don't have enough information to answer this" rather than reasoning from
general knowledge.

This does not fully prevent hallucination — models do not always obey this instruction under adversarial conditions — but it substantially reduces confabulated answers in normal use.

Citation format

For high-stakes applications, instruct the model to cite its sources:

After each claim, cite the source document and section in the format [Source: <title>, <section>].

Include the source metadata (document title, section, URL if available) in the context you provide. The model can only cite sources you gave it.

Stage 7: Abstention and Confidence Thresholds

A RAG system that answers confidently when it doesn't have the relevant information is more dangerous than a system that says "I don't know." Abstention — choosing not to answer — is a correct answer in many cases.

Practical abstention signals:

Retrieval confidence threshold: if the top retrieved chunk has a cosine similarity below a threshold (e.g., 0.7), surface a "low confidence" flag or abstain.
No-context detection: if retrieved chunks don't contain any of the named entities in the query, this is a signal of retrieval failure.
Model-level uncertainty: instruct the model to prefix low-confidence responses with "Based on available information..." and abstain with "I don't have reliable information on this topic" when the context is insufficient.

The threshold values are domain-specific. Tune them against a held-out evaluation set with known-answerable and known-unanswerable questions.

Evaluation: Measuring Whether Your RAG System Actually Works

You cannot improve what you cannot measure. A RAG system without an evaluation harness is a system you are running blind.

The four key metrics (from the RAGAS framework and related work):

Metric	What it measures
Context Precision	Are the retrieved chunks relevant to the question?
Context Recall	Did retrieval find all the relevant information needed?
Faithfulness	Does the generated answer stay within the retrieved context?
Answer Relevance	Does the answer address the question that was asked?

Build a golden set of 50–100 representative question-answer pairs from your corpus. Run your pipeline. Measure these four metrics. Establish a baseline. Every change to chunking, embedding, retrieval parameters, or prompt should be measured against this baseline.

Without this loop, you are making decisions based on vibes about whether your retrieval "seems better" — which is not engineering.

The .NET Integration Pattern

For teams building in .NET, the current best path:

Embedding: call OpenAI's API via the official Azure OpenAI SDK (Azure.AI.OpenAI) or community SDK (OpenAI). For self-hosted models, use the ONNX Runtime with a compatible model file.

Vector store: pgvector via Npgsql.EntityFrameworkCore.PostgreSQL for integrated deployments. Qdrant via its official .NET client (Qdrant.Client) for dedicated vector workloads.

Orchestration: Microsoft Semantic Kernel (Microsoft.SemanticKernel) is the most mature .NET-native RAG orchestration library. It handles embedding, memory (vector store abstraction), and retrieval with built-in support for OpenAI, Azure OpenAI, and Hugging Face models. LangChain4j-equivalent functionality in a .NET-idiomatic API.

Reranking: call Cohere's Rerank API via HttpClient, or run a cross-encoder locally via ONNX Runtime.

Key Takeaways

Chunking is the highest-leverage decision. Use structure-aware chunking + the parent-child pattern.
Evaluate your embedding model against your actual corpus. General-purpose models are good defaults; verify before assuming.
Add hybrid search (vector + BM25 + RRF) before adding a reranker. Hybrid search catches what semantic retrieval misses.
Reranking with a cross-encoder consistently improves quality. Gate to top-50 candidates to keep latency manageable.
Build an evaluation harness before you optimise. Context Precision, Context Recall, Faithfulness, Answer Relevance.
Design explicit abstention paths. A system that says "I don't know" when it doesn't know is more trustworthy than one that confidently confabulates.

Sources

Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS. https://arxiv.org/abs/2005.11401
Thakur et al. (2021). BEIR: A Heterogeneous Benchmark for Zero-Shot Evaluation of Information Retrieval Models. NeurIPS Datasets and Benchmarks. https://arxiv.org/abs/2104.08663
Es et al. (2023). RAGAS: Automated Evaluation of Retrieval Augmented Generation. https://arxiv.org/abs/2309.15217
Microsoft Semantic Kernel documentation. https://learn.microsoft.com/en-us/semantic-kernel/
pgvector HNSW indexing documentation. https://github.com/pgvector/pgvector
Cohere Rerank API documentation. https://docs.cohere.com/reference/rerank

Ama Senevirathne is a full-stack software engineer specialising in .NET, Angular, and agentic AI systems. She builds open-source tools at github.com/amasen02.

Top comments (1)

Alex Shev • Jul 26

The production RAG decisions that matter are usually less glamorous than the model choice: chunk boundaries, metadata, freshness, retrieval filters, citation discipline, and how the system behaves when the answer is not in the corpus.

I would also put evaluation close to ingestion. If source quality changes and nobody notices until users complain, the RAG app is really just a search wrapper with a nicer sentence generator.