A logistics company in Rotterdam connected its internal knowledge base to a chatbot last spring. Within three weeks, the bot was confidently citing shipping policies that didn't exist. The problem wasn't the language model. It was everything sitting in front of it: the chunking, the retrieval, the guardrails nobody had built yet.
That gap between "a RAG demo works" and "a RAG application survives real users" is where most teams get stuck. This guide walks through what changes between the two, step by step.
What "Production-Ready" Actually Means
A working prototype answers questions correctly when you test it yourself. A production system answers correctly when a stranger asks something oddly phrased, when the underlying documents change weekly, and when 200 people are hitting it at once. Getting there means treating retrieval as a full pipeline, not a single API call bolted onto a chatbot.
The teams at SolveMotive who build these pipelines for clients tend to break the work into six stages: ingestion, embedding, retrieval, generation, evaluation, and monitoring. Skipping any one of them is usually where things fall apart later.
Step 1: Data Ingestion and Chunking
Raw documents rarely map cleanly onto how people ask questions. A 40-page policy PDF split into one giant chunk will drown the retriever in irrelevant text; split into single sentences, it loses all context.
A few things matter here:
Chunk by semantic boundaries (headings, paragraphs) rather than a fixed character count wherever the source format allows it.
Keep chunks between roughly 200 and 500 tokens, with some overlap between consecutive chunks so context doesn't get cut mid-thought.
Store metadata alongside each chunk: source document, section title, last-updated date, and access permissions. This metadata becomes essential later for filtering and for citing sources back to the user.
Re-run ingestion on a schedule, not once. Documentation changes, and a RAG system built on a six-month-old snapshot will answer questions with outdated information.
Step 2: Choosing an Embedding Model and Vector Store
The embedding model turns each chunk into a vector, and the choice affects both retrieval quality and cost. OpenAI's text-embedding-3 models, Cohere's embed-v3, and open-source options like BGE or E5 each have different strengths depending on domain and language mix.
For the vector store, three factors decide the right pick more than any feature list: expected data volume, whether metadata filtering needs to happen at query time, and how much infrastructure the team wants to manage. Pinecone and Weaviate suit teams that want a managed service; pgvector works well when the data already lives in Postgres and a separate system feels like unnecessary overhead; Qdrant sits somewhere in between with strong open-source performance.
Step 3: Retrieval Strategy
Pure vector similarity search misses a common case: exact keyword matches, like a product code or an error number, which embeddings sometimes fail to rank highly. Combining vector search with traditional keyword search (BM25) and merging the results, known as hybrid search, tends to close that gap.
A second layer worth adding is reranking. After the initial retrieval pulls back, say, the top 20 candidate chunks, a smaller cross-encoder model reorders them by actual relevance to the query before the top 3 to 5 get passed to the language model. Cohere's Rerank API and open-source models like bge-reranker both handle this well, and the latency cost is usually under 200 milliseconds for a meaningful jump in answer accuracy.
Step 4: Prompt Design and Generation
The prompt structure sent to the language model deserves as much attention as the retrieval itself. A prompt that just pastes retrieved chunks above the question invites the model to blend them with its own training knowledge and produce answers that sound right but aren't grounded in the source material.
Explicit instructions help: tell the model to answer only from the provided context, to say when the context doesn't contain an answer, and to cite which chunk supported each claim. Teams building customer-facing tools at SolveMotive have found that adding a short refusal instruction alone (something like "if the context doesn't address the question, say so rather than guessing") cuts hallucinated answers noticeably.
Step 5: Evaluation Before Launch
A RAG system needs a test set before it goes anywhere near real users: 50 to 100 representative questions with known correct answers, drawn from actual support tickets or documentation queries where possible. Run this set through the pipeline and score three things separately: whether retrieval pulled the right chunks, whether the generated answer used them correctly, and whether the answer stayed factually accurate.
Tools like RAGAS and TruLens automate much of this scoring, measuring faithfulness (does the answer match the retrieved context) and answer relevance (does it actually address the question) as distinct metrics. Treating them separately matters, because a system can retrieve perfectly and still generate a poor answer, or the reverse.
Step 6: Monitoring and Observability After Launch
Once live, retrieval quality degrades quietly. Documents get updated without re-indexing, embedding drift creeps in, and user questions start covering topics the original test set never anticipated. Logging every query alongside the chunks retrieved and the final answer, then reviewing a sample weekly, catches most of these issues before users complain.
Latency also needs a budget from day one. A pipeline that takes 8 seconds to answer feels broken even if every answer is correct. Caching common queries, running retrieval and reranking in parallel where possible, and setting a hard timeout with a graceful fallback all keep response times reasonable under load.
Common Failure Points
A handful of mistakes show up across most first attempts at a production RAG system:
Treating the vector store as a single source of truth without access control, so users can retrieve chunks from documents they shouldn't see. Skipping the evaluation step entirely and shipping based on a handful of manual test questions. Using a chunk size that made sense for one document type and applying it uniformly across a mixed corpus of PDFs, wikis, and support tickets. And underestimating how much prompt engineering the generation step still needs, even with a strong retriever in place.
None of these are hard problems on their own. They just tend to get skipped when a demo works well enough to ship, and they surface a few weeks later as a support inbox full of confused users.
Closing Thoughts
Building a RAG application that survives contact with real users comes down to treating each stage of the pipeline as its own engineering problem: solid ingestion, a retrieval strategy that combines multiple signals, a prompt that keeps the model grounded, and an evaluation loop that never really stops. Teams that get this right usually didn't get it right on the first try either. They built it, watched where it broke, and fixed that layer.
If your team is weighing build-versus-buy on a RAG pipeline or stuck on a specific layer of this stack, the engineers at https://www.solvemotive.com/ work through exactly this kind of problem with clients regularly. Let's talk. Your motive, our solution.
Top comments (0)