Building a Retrieval-Augmented Generation (RAG) system is relatively straightforward. Building one that performs well in production is a different challenge altogether.
Once a RAG pipeline starts handling real users and large document collections, common bottlenecks begin to appear: increasing response latency, rising LLM costs, and slower document ingestion.
It's tempting to solve these problems by switching to a larger model or waiting for cheaper APIs. In practice, many of the biggest improvements come from optimizing the architecture around the model rather than changing the model itself.
This article covers several engineering techniques that proved effective while optimizing a production RAG system. None of them are particularly novel on their own, but together they produced meaningful improvements in latency, cost, and overall system efficiency.
1. Route Queries Before Retrieving
One of the easiest ways to waste latency and tokens is to perform semantic retrieval for every user request.
Many queries don't require vector search at all.
Examples include:
- Greetings
- Questions about uploaded documents or metadata
- General conversational interactions
Rather than embedding every query and performing retrieval, a lightweight intent classification stage can determine whether semantic search is actually necessary.
If retrieval isn't required, the request can be answered directly through conversational logic or structured backend data.
Besides reducing unnecessary work, this also frees retrieval resources for queries where semantic search genuinely improves answer quality.
2. Rerank Before Building the Prompt
Most retrieval systems are optimized for recall.
Returning 10–20 potentially relevant chunks is often desirable during retrieval, but sending every retrieved chunk to the LLM usually isn't.
Larger prompts increase:
- Input token usage
- First-token latency
- Overall inference time
- The likelihood of distracting the model with irrelevant context
A lightweight reranking stage can identify the small subset of retrieved passages most likely to answer the user's question.
Only those passages are forwarded to prompt construction.
This keeps prompts focused while significantly reducing unnecessary context.
3. Reduce Context Instead of Reducing Recall
Even after reranking, retrieved chunks frequently contain far more text than necessary.
A document chunk might contain several hundred tokens while only a few sentences are actually relevant.
Instead of embedding smaller chunks—which can hurt retrieval quality—another approach is to trim context after retrieval.
For factual questions, relevant sections can be extracted by locating query-related keywords and selecting a window surrounding those matches.
For summarization tasks, preserving a larger leading section often produces better results than aggressive trimming.
This approach maintains retrieval quality while substantially reducing prompt size.
4. Parallelize Document Processing
Ingestion pipelines are frequently limited by I/O rather than CPU.
Tasks such as:
- PDF extraction
- Office document parsing
- OCR
- Text cleaning
- Chunk preparation
can usually be processed independently.
Instead of processing uploaded files sequentially, these operations can be executed concurrently using a worker pool.
Parallel preprocessing improves indexing throughput and reduces the time required before newly uploaded documents become searchable.
5. Perform OCR Before Retrieval
Supporting scanned PDFs and image-heavy documents is a common production requirement.
One option is to send these documents directly to multimodal vision models.
While effective, this increases both inference cost and latency.
An alternative approach is to perform OCR during preprocessing.
Pages without selectable text are detected automatically and converted into plain text before entering the indexing pipeline.
Performing OCR locally offers several practical advantages:
- Reduces reliance on multimodal inference
- Produces standard text suitable for embedding models
- Lowers preprocessing costs
- Makes downstream retrieval pipelines simpler
To improve retrieval quality, very small or low-confidence OCR outputs should also be discarded before indexing, preventing noisy embeddings from entering the vector store.
6. Measure Every Optimization
Performance improvements should be measured rather than assumed.
Useful metrics include:
- End-to-end latency
- Retrieval latency
- Generation latency
- Prompt token usage
- Completion token usage
- Document ingestion time
- Indexing throughput
Collecting these metrics throughout the pipeline makes it possible to compare changes before and after each optimization.
Without instrumentation, it's difficult to know whether an optimization actually improved the system or simply shifted the bottleneck elsewhere.
Trade-offs
Every optimization introduces trade-offs.
Intent routing adds an additional classification step but avoids unnecessary retrieval.
Reranking introduces another model call but often reduces the total generation cost by producing much smaller prompts.
Local OCR shifts computation from cloud APIs to the application server, reducing inference costs while increasing CPU utilization.
The right choice depends on workload characteristics, latency requirements, infrastructure, and operational costs.
Key Takeaways
One lesson became increasingly clear throughout this work:
Many production RAG bottlenecks are architectural rather than model-related.
Several practical optimizations consistently provided meaningful improvements:
- Route requests intelligently so retrieval only runs when it adds value.
- Rerank retrieved results before prompt construction.
- Trim context after retrieval instead of stuffing entire document chunks into prompts.
- Parallelize document preprocessing and indexing.
- Prefer local OCR when multimodal reasoning is unnecessary.
- Instrument the pipeline so improvements can be measured objectively.
Modern language models continue to improve rapidly, but production performance often depends just as much on retrieval, preprocessing, prompt construction, and observability as it does on the model itself.
I'm currently expanding on these ideas through an open-source diagnostics toolkit for Haystack pipelines. If you're working on production RAG systems or interested in collaborating, I'd be happy to connect and exchange ideas.
Linkedin: https://www.linkedin.com/in/aditya-raut-3b4bba31b/
Haystack Diagnostics: https://github.com/rautaditya2606/haystack-diagnostics
Top comments (1)
Routing greetings and metadata questions around vector search entirely is the cheapest win here and the one people forget, because they embed everything by reflex. I like that you trim context post-retrieval with a keyword window instead of shrinking the chunks, since small chunks wreck recall. Did you measure how often the intent classifier sends a query down the wrong path, and does a misroute fail loudly or just silently answer without the retrieved context?