RAG Content Pipeline in Production: 5 Decisions That Separate Working Systems from Demos

Every demo RAG system works. It retrieves something, hands it to an LLM, and produces a plausible answer. Every production RAG system fails — at least once — for reasons that have nothing to do with the model. After spending four months moving a RAG pipeline from a notebook into a system that answers questions for real users, I can tell you exactly where the gap is: the content pipeline. Retrieval is easy. Content is hard.

Decision 1: Chunking strategy is a retrieval decision, not a text-processing one

The first version of our pipeline used fixed 512-token chunks. It was the default in every tutorial, so it felt safe. It was also wrong for about 40% of our documents. We index technical manuals, support threads, and internal specs. Manuals have numbered steps that span multiple chunks; support threads have a question at the top and the accepted answer ten paragraphs later. Fixed-size chunks broke both patterns in the same way: the semantic unit — a complete step, a full Q&A pair — got amputated mid-sentence.

We switched to structure-aware chunking: split on headings for manuals, on conversation boundaries for threads, and on code blocks for anything containing code. The retrieval hit rate on our evaluation set went from 61% to 83%. Not because the embeddings changed. Because the units changed. If your chunks don't align with the atomic units a human would quote when answering the question, no amount of reranking will save you.

Decision 2: Hybrid search beats pure vector search on real queries

Vector search is brilliant at finding "semantically similar" content and terrible at exact terms. Our users search for error codes, version numbers, and function names — strings that embeddings blur. A query like ERR_9001 returned fuzzy matches about timeouts and connection pools, but not the exact error page, because semantically the error code isn't "similar" to anything; it's an identifier.

We added BM25 as a parallel retriever and merged results with a weighted score before reranking. Exact-match queries now surface the right page in the top three almost always. The lesson is boring and important: hybrid search (vector + keyword) is the baseline for production RAG in 2026, not an optimization. If you're shipping pure vector search to users who type product names and error codes, you are shipping a demo.

Decision 3: Freshness beats relevance when they conflict

The hardest failure to catch is the confident stale answer. Our pipeline indexed 12,000 documents; about 8% of them changed monthly. Version two retrieved "whichever chunk was semantically closest" — which, as it turned out, was often last quarter's pricing page. Users didn't complain that the answer was wrong. They silently lost trust and stopped asking.

The fix had three parts:

1. Version stamps on every document — each chunk carries the doc's updated_at timestamp.
2. Recency boost in the scoring function — when two chunks score within 15% of each other, the newer one wins.
3. A periodic sweep — a nightly job re-checks changed sources and re-indexes only the affected chunks (incremental update), instead of rebuilding the entire corpus.

After the fix, stale-answer reports dropped to near zero. If your knowledge base has any source that changes over time — pricing, policy, docs, code — freshness handling is not optional.

Decision 4: Reranking is where the quality budget should go

People ask "which embedding model should I use?" constantly. In our benchmarks, upgrading the embedding model moved retrieval quality by about 4-6 points. Adding a cross-encoder reranker over the top-20 candidates moved it by 12 points — double the gain, for a fraction of the indexing cost (reranking runs at query time, so it only touches the candidates you already fetched).

We use a small cross-encoder that runs in ~30ms per query on CPU. The pipeline fetches 20 candidates from hybrid search, reranks to 5, and feeds those to the LLM. The user-visible quality jump was immediate. Retrieve cheap, rerank precise, generate last is the architecture that made our system feel "smart."

Decision 5: Evaluation is a data problem before it is a metric problem

We built an eval set of 200 real queries with expert-written golden answers. Every change to the pipeline — new chunker, new reranker, new prompt — gets scored against it. This sounds obvious, but it's the single thing that most demo pipelines skip, and it's why they can't improve: without a baseline you cannot tell whether a change helped or hurt.

The eval set is versioned in git alongside the code. When a user reports a bad answer, it becomes a new eval case before we fix anything. That way "fix the bug" and "prevent the regression" are the same task. Our answer-accuracy score went from 74% at launch to 91% now, and the eval set is why we could prove each step contributed.

The pipeline that works

Here is what production looks like for us now:

1. Ingest — structure-aware chunking aligned to document types
2. Index — embeddings + BM25, both written at ingest time
3. Retrieve — hybrid search fetches 20 candidates
4. Rerank — cross-encoder narrows to 5
5. Generate — LLM gets the 5 chunks with source metadata and version stamps
6. Evaluate — every answer logged, bad ones become eval cases

None of these decisions are glamorous. No fancy agent orchestration, no auto-optimizing framework. Just careful choices about what the content pipeline does with your documents before the model ever sees them. That's the difference between a demo that impresses and a system people rely on.

If you're building RAG, start with Decision 1 and 5 — chunking and evaluation. They're the least flashy and the most decisive. The model is rarely your bottleneck. Your content pipeline is.

What failure surprised you most when you moved RAG into production? I'd love to compare notes in the comments.