A technical guide to chunking, retrieval, re-ranking, and observability for RAG systems at scale
A retrieval-augmented generation (RAG) pipeline is a system that fetches relevant documents from a knowledge base and passes them to a language model as context, enabling the model to answer questions grounded in factual, up-to-date information. The architecture sounds simple: ingest documents, chunk them, embed them, index them, retrieve relevant chunks at query time, optionally re-rank, then pass to an LLM. In practice, each step has enough depth to sink a production system. This guide walks through what works, what fails, and where to spend engineering effort.
Why this matters now
By 2026, RAG has become the default architecture for knowledge-intensive applications. Fine-tuning fell out of favor because it's expensive, stale within weeks, and legally risky for proprietary data. Vector databases matured, retrieval evaluation became a discipline, and the cost of LLM API calls dropped enough that triple-digit latency no longer kills user experience. But maturity also brought scars: teams learned that naive RAG fails catastrophically on edge cases, that chunking decisions made in week two still haunt production in month six, and that retrieval quality isn't a solved problem despite billions in VC funding.
The stakes are high. A RAG system that hallucinates on financial queries or medical advice is worse than honest silence. A system that's slow or unreliable erodes trust faster than a system that doesn't exist. The teams winning are those that treat RAG as an engineering discipline, not a chatbot template, and measure ruthlessly from day one.
Document ingestion and preparation
Photo by Ron Lach on Pexels.
The input layer determines everything downstream. Documents arrive in chaos: PDFs with scanned images, Markdown with front matter, unstructured HTML, tables, nested JSON. Before chunking, they must be parsed, cleaned, and normalized.
Start with a document parser that respects structure. For PDFs, tools like PyPDF2 or pdfplumber extract text, but often mangled. A library like unstructured.io or LlamaIndex's document loaders handle semantic parsing: they recognize headers, lists, tables, and metadata. For web content, HTML parsers like BeautifulSoup or readability libraries strip boilerplate and extract the meat. The goal is structured data: chunks retain metadata (source, date, author, section), and logical boundaries are preserved.
Metadata is not optional. Tag each document with source URI, modification date, version, and domain-specific fields. At retrieval time, you'll filter by date ("only documents from Q3 2025"), by section ("only FAQs, not blog posts"), or by confidence ("only verified sources"). Lose metadata in the ingestion pipeline, and your retriever becomes a blunt instrument.
Normalization matters more than it seems. Standardize encodings (UTF-8), remove zero-width characters, collapse multiple spaces, and strip trailing whitespace. If your corpus mixes PDFs, Notion exports, and web scrapes, small inconsistencies compound at scale. A single unprintable character in a chunk can break downstream NLP pipelines or embedding APIs.
Chunking strategy and size selection
Chunking is the highest-leverage decision in a RAG pipeline, and it gets almost no attention. A bad chunking strategy ruins retrieval quality in ways that no re-ranker can fix.
The goal is to break documents into semantic units that are small enough to fit in an LLM context window, large enough to contain meaning, and aligned with how questions are asked. A 50-token chunk may be retrieved perfectly but contain too little context for the LLM to answer. A 4000-token chunk drowns out the signal and wastes context. The Goldilocks zone is typically 512 to 1024 tokens for most domains.
Start with simple overlap: split on sentence or paragraph boundaries, use fixed-size chunks with overlap (e.g., 768 tokens with 256-token overlap). Overlap prevents answers that split across chunk boundaries. Then measure and iterate. Log which chunks are retrieved for successful queries and which are missed for failures. If failures show you're splitting logic paragraphs, switch to semantic splitting: use a language model or a simple heuristic (e.g., "split on section headers, not paragraphs") to preserve meaning.
Context-aware chunking is worth the effort at scale. If your knowledge base is structured (wikis, API docs), chunk by semantic unit: one chunk per function signature, per FAQ item, per policy section. For unstructured text, a trick that works: extract headers and topic boundaries, then chunk around them. A small hierarchical chunker that understands your domain beats generic token splitting.
One subtle trap: don't chunk before you parse. Chunk after extraction and cleaning. A PDF header that appears in every chunk wastes retrieval signal and context window. Pre-process first, then chunk.
Embedding, indexing, and retrieval
Photo by www.kaboompics.com on Pexels.
Once chunks exist, they must be converted to embeddings (dense vectors) and indexed for fast retrieval. The choice of embedding model is underrated. An older embedding model like text-embedding-ada-002 is cheap and reliable but weak on domain-specific tasks. A newer model like text-embedding-3-large is better but slower and pricier. A small open-source model like BAAI/bge-small-en-v1.5 is fast and free but less accurate on distant domains.
A pragmatic heuristic: choose an embedding model that beats a strong BM25 baseline on your domain. If you don't have a domain benchmark yet, build one. Collect 100 queries (from users, customer support, or synthetic) and annotate which documents are relevant. Then measure. Most teams find that hybrid search (BM25 + semantic) outperforms pure vectors because keywords and meaning are orthogonal signals.
Indexing is straightforward if you choose a mature tool. Vector databases like Pinecone, Weaviate, or Qdrant handle replication and durability. Elasticsearch supports vectors natively. LanceDB and Chroma work for smaller scales. What matters is that you pick something tested and move on. Don't build a vector search engine from scratch unless you have a specific constraint (e.g., everything must be on-premise, or latency is under 50ms with millions of documents).
Retrieval at query time is a two-stage process: (1) fetch top-k candidates (e.g., top 50) using semantic search, (2) optionally re-rank them. The top-k recall is a hard ceiling on end-to-end quality. If the true answer isn't in the top 50, no re-ranker recovers it. A common target is top-100 recall above 85 to 90 percent on a held-out test set. Measure this early and often.
Caching is free wins. Many RAG applications see query repetition: users ask the same question multiple ways, or the same underlying question recurring hourly. Cache embeddings and retrieval results with a simple LRU cache in front of your vector search. This cuts latency by 80 percent for warm queries and reduces API costs. TTL your cache based on how often your knowledge base updates.
Re-ranking and retrieval evaluation
Re-ranking is a controversial step because it adds latency and cost but often improves quality. The idea is simple: retrieve a larger set of candidates using a fast but approximate method (semantic search), then use a more accurate but slower method (cross-encoder or LLM) to re-score the top-k.
A cross-encoder model like ms-marco-MiniLM-L-12-v2 takes a query and a document as input and outputs a relevance score. It's slower than semantic search (50 to 200ms for a batch of 50 documents) but often more accurate. When do you use it? When baseline retrieval recall is >80 percent but precision is low (many irrelevant documents in top-k), or when your latency budget allows. If you're already at 95 percent recall and 90 percent precision at top-5, re-ranking doesn't help.
Measure retrieval quality relentlessly. Build a benchmark: 100 to 500 queries with hand-annotated relevant documents. Compute metrics: Mean Reciprocal Rank (MRR), NDCG@10, hit rate at k. These tell you what's working. MRR@10 of 0.6 means the first relevant document is, on average, at rank 1.67. That's decent but not great. NDCG@10 above 0.7 is a good target for production.
Don't benchmark in a vacuum. Segment your queries by type (factual, comparative, exploratory) and document type (FAQ, policy, code). Retrieval quality often varies wildly by segment. If 80 percent of failures are on comparative queries ("compare X and Y"), your chunking strategy or embedding model may need refinement. If failures cluster on brand-new documents, implement a content freshness signal or boost new items in ranking.
Production monitoring and failure modes
A RAG system that works in a notebook often fails in production because production has different data, different query patterns, and different failure modes. Monitoring is not optional.
Instrument the pipeline at each stage. Log retrieval results: which documents were retrieved, their scores, and how they ranked. Log the final generated answer and any user feedback (thumbs up/down, regeneration request). Compute retrieval hit rate (percentage of queries where at least one relevant document was retrieved) and compare it to your benchmark. If hit rate drops below 80 percent, alert.
Implement a manual audit loop. Sample 1 to 2 percent of queries weekly and have a human review whether the retrieved documents actually answer the question. This catches distribution shifts early. If your knowledge base suddenly includes a lot of outdated or contradictory content, human auditors will see it before metrics do.
Monitor latency at each stage. Embedding takes ~50ms, retrieval ~100ms, re-ranking ~100 to 200ms, LLM generation ~500ms to 5 seconds. If retrieval latency creeps from 100ms to 500ms, investigate. A bloated index or a vector database running out of memory is a slow failure mode. Set latency budgets and track percentiles (p50, p95, p99), not just means.
Track embedding API costs. If your knowledge base grows from 10k to 100k documents and you re-embed everything weekly, costs scale linearly. Consider batch embedding, incremental updates (embed only new documents), or caching embeddings in cold storage.
The most dangerous failure mode is silent hallucination: the LLM generates plausible but false answers that users believe because the source documents don't contradict them. Mitigate by setting a retrieval confidence threshold. If no document scored above 0.5 (using your chosen metric), return "I don't know" rather than generate an answer. Users accept honest uncertainty; they hate confident lies.
When this fails
RAG is not a panacea. Be honest about the limitations.
First, retrieval can fail fundamentally. If a user asks "what is the revenue per customer?" and your knowledge base contains aggregate revenue and customer counts but never computed the ratio, RAG retrieves the raw numbers but the LLM must calculate. That works sometimes and fails silently sometimes. For questions requiring arithmetic, data fusion across documents, or reasoning over temporal data, RAG alone is insufficient. You need either a reasoning layer (chain-of-thought prompting, multi-hop retrieval) or direct integration with data systems (SQL databases, APIs).
Second, chunking decisions have long tails. A chunking strategy optimized for FAQ retrieval may fail on narrative documents. A strategy tuned on Q2 data may break when Q3 introduces new document types. Chunking is not one-time. Plan to revisit it quarterly or when corpus composition changes significantly.
Third, retrieval evaluation doesn't correlate perfectly with end-to-end accuracy. A system with 90 percent MRR@10 and a mediocre LLM can produce worse answers than a system with 70 percent MRR@10 and a strong LLM. Measure end-to-end answer quality, not just retrieval quality. Have humans rate a sample of final answers monthly.
Fourth, latency compounds. If embedding takes 50ms, retrieval 100ms, re-ranking 150ms, and LLM generation 3 seconds, and you have 50 percent cache hit rate, your median latency is around 1.5 seconds. That's acceptable for many applications but not for real-time advisors or chat. Latency tuning requires trade-offs: smaller top-k (worse recall), no re-ranking (worse precision), or faster embeddings (worse quality).
Finally, RAG systems are only as good as their knowledge base. If your documents are contradictory, outdated, or wrong, RAG will retrieve and amplify those problems. Before blaming the pipeline, audit the source data.
The path forward
Building a production RAG pipeline is an iterative discipline. Start with simple choices: a standard embedding model, a vector database, top-50 retrieval, and no re-ranking. Measure baseline retrieval and end-to-end accuracy on a 100-query benchmark. Then optimize: improve chunking, add re-ranking, tune top-k, implement caching. The goal is not perfection but a system that's accurate enough, fast enough, and reliable enough to be useful.
Focus engineering effort on the parts that fail most often in production: evaluation (can you measure quality reliably?), monitoring (can you catch failures?), and chunking (are semantic boundaries preserved?). Those three give you 80 percent of the benefit. Everything else is refinement.
This article was originally published on AI Glimpse.
Top comments (0)