DEV Community: Aditya Raut

Production RAG Optimization: Practical Techniques That Improved Performance

Aditya Raut — Wed, 01 Jul 2026 15:46:12 +0000

Building a Retrieval-Augmented Generation (RAG) system is relatively straightforward. Building one that performs well in production is a different challenge altogether.

Once a RAG pipeline starts handling real users and large document collections, common bottlenecks begin to appear: increasing response latency, rising LLM costs, and slower document ingestion.

It's tempting to solve these problems by switching to a larger model or waiting for cheaper APIs. In practice, many of the biggest improvements come from optimizing the architecture around the model rather than changing the model itself.

This article covers several engineering techniques that proved effective while optimizing a production RAG system. None of them are particularly novel on their own, but together they produced meaningful improvements in latency, cost, and overall system efficiency.

1. Route Queries Before Retrieving

One of the easiest ways to waste latency and tokens is to perform semantic retrieval for every user request.

Many queries don't require vector search at all.

Examples include:

Greetings
Questions about uploaded documents or metadata
General conversational interactions

Rather than embedding every query and performing retrieval, a lightweight intent classification stage can determine whether semantic search is actually necessary.

If retrieval isn't required, the request can be answered directly through conversational logic or structured backend data.

Besides reducing unnecessary work, this also frees retrieval resources for queries where semantic search genuinely improves answer quality.

2. Rerank Before Building the Prompt

Most retrieval systems are optimized for recall.

Returning 10–20 potentially relevant chunks is often desirable during retrieval, but sending every retrieved chunk to the LLM usually isn't.

Larger prompts increase:

Input token usage
First-token latency
Overall inference time
The likelihood of distracting the model with irrelevant context

A lightweight reranking stage can identify the small subset of retrieved passages most likely to answer the user's question.

Only those passages are forwarded to prompt construction.

This keeps prompts focused while significantly reducing unnecessary context.

3. Reduce Context Instead of Reducing Recall

Even after reranking, retrieved chunks frequently contain far more text than necessary.

A document chunk might contain several hundred tokens while only a few sentences are actually relevant.

Instead of embedding smaller chunks—which can hurt retrieval quality—another approach is to trim context after retrieval.

For factual questions, relevant sections can be extracted by locating query-related keywords and selecting a window surrounding those matches.

For summarization tasks, preserving a larger leading section often produces better results than aggressive trimming.

This approach maintains retrieval quality while substantially reducing prompt size.

4. Parallelize Document Processing

Ingestion pipelines are frequently limited by I/O rather than CPU.

Tasks such as:

PDF extraction
Office document parsing
OCR
Text cleaning
Chunk preparation

can usually be processed independently.

Instead of processing uploaded files sequentially, these operations can be executed concurrently using a worker pool.

Parallel preprocessing improves indexing throughput and reduces the time required before newly uploaded documents become searchable.

5. Perform OCR Before Retrieval

Supporting scanned PDFs and image-heavy documents is a common production requirement.

One option is to send these documents directly to multimodal vision models.

While effective, this increases both inference cost and latency.

An alternative approach is to perform OCR during preprocessing.

Pages without selectable text are detected automatically and converted into plain text before entering the indexing pipeline.

Performing OCR locally offers several practical advantages:

Reduces reliance on multimodal inference
Produces standard text suitable for embedding models
Lowers preprocessing costs
Makes downstream retrieval pipelines simpler

To improve retrieval quality, very small or low-confidence OCR outputs should also be discarded before indexing, preventing noisy embeddings from entering the vector store.

6. Measure Every Optimization

Performance improvements should be measured rather than assumed.

Useful metrics include:

End-to-end latency
Retrieval latency
Generation latency
Prompt token usage
Completion token usage
Document ingestion time
Indexing throughput

Collecting these metrics throughout the pipeline makes it possible to compare changes before and after each optimization.

Without instrumentation, it's difficult to know whether an optimization actually improved the system or simply shifted the bottleneck elsewhere.

Trade-offs

Every optimization introduces trade-offs.

Intent routing adds an additional classification step but avoids unnecessary retrieval.

Reranking introduces another model call but often reduces the total generation cost by producing much smaller prompts.

Local OCR shifts computation from cloud APIs to the application server, reducing inference costs while increasing CPU utilization.

The right choice depends on workload characteristics, latency requirements, infrastructure, and operational costs.

Key Takeaways

One lesson became increasingly clear throughout this work:

Many production RAG bottlenecks are architectural rather than model-related.

Several practical optimizations consistently provided meaningful improvements:

Route requests intelligently so retrieval only runs when it adds value.
Rerank retrieved results before prompt construction.
Trim context after retrieval instead of stuffing entire document chunks into prompts.
Parallelize document preprocessing and indexing.
Prefer local OCR when multimodal reasoning is unnecessary.
Instrument the pipeline so improvements can be measured objectively.

Modern language models continue to improve rapidly, but production performance often depends just as much on retrieval, preprocessing, prompt construction, and observability as it does on the model itself.

I'm currently expanding on these ideas through an open-source diagnostics toolkit for Haystack pipelines. If you're working on production RAG systems or interested in collaborating, I'd be happy to connect and exchange ideas.

Linkedin: https://www.linkedin.com/in/aditya-raut-3b4bba31b/
Haystack Diagnostics: https://github.com/rautaditya2606/haystack-diagnostics

I Got Tired of Debugging Haystack RAG Pipelines Blind, So I Built a Diagnostics Engine

Aditya Raut — Sat, 27 Jun 2026 11:40:13 +0000

RAG pipelines fail in quiet ways.

Retrieval drops. Documents go missing. Metadata gets corrupted somewhere between ingestion and query time. Your generator starts hallucinating and you don't know if it's the retriever, the document store, or something upstream.

The debugging loop is always the same: check traces, grep logs, write a one-off script to inspect the document store, try to diff two runs manually. It works, but it's slow and it doesn't scale.

I hit this enough times while working on a Haystack 2.x pipeline at my internship that I started building something to systematize it.

That became Haystack Diagnostics Engine.

What it actually does

Four things:

Document store validation — checks your vector store for duplicate chunks, missing metadata fields, and short/malformed documents before they silently degrade retrieval quality.

Pipeline introspection — inspects your Haystack pipeline structure, flags misconfigurations, and can visualize the component graph. Useful when you're inheriting a pipeline someone else built.

Retrieval failure classification — when a query returns garbage, this tells you why. Six failure classes: empty results, low-score results, metadata filter mismatch, reranker collapse, score inversion, and retriever timeout. Each has a different fix. Uses Haystack's include_outputs_from for single-pass retriever/reranker diagnostics without re-running the pipeline.

Debug bundle capture and diffing — this is the one I've gotten the most feedback on.

Debug bundles: the part that actually changed my workflow

The typical production debugging scenario: something worked last week, it doesn't work now, and you have no idea what changed.

collect_debug_bundle(pipeline, query, ...) captures the full state of a single query execution as a structured JSON file:

Pipeline graph, Haystack version, component init_parameters
Raw retriever top-k (pre-reranker) — scores, metadata, content previews
Reranked top-k when a reranker is detected
Prompt snapshot and generated answer
Failure classification result
Corpus health checks scoped to only the retrieved document IDs, not the full store

Bundle filenames are {query_slug}_{timestamp}.json — human-readable, sort naturally across runs of the same query. The UUID lives inside the JSON, not in the filename.

diff_debug_bundles(bundle_a, bundle_b) compares two persisted bundles and reports:

Score deltas per document
Docs that appeared or disappeared between runs
Component config changes between the two pipeline states
Character-level answer diff via difflib (no tokenizer dependency)

Config diffs filter out known volatile fields by default — things like InMemoryDocumentStore.index, which regenerates as a random UUID on instantiation and would create false positives on every diff. You can pass ignore_config_paths=set() to disable filtering or extend the defaults with component-specific paths like "retriever.session_id".

There's also a CLI:

python -m diagnostics.debug_bundler diff bundle_a.json bundle_b.json

The workflow this enables: run a query, persist the bundle, deploy a change, run the same query again, diff the two bundles. You get an exact record of what shifted — scores, docs, config, answer — without relying on memory or logs.

What it found on a real deployment

I ran the validator against a live Weaviate-backed RAG instance with 823 chunks and OpenAI embeddings.

195 duplicate chunks (23.7% of the corpus)
14 documents missing required metadata keys
8 anomalous short chunks under the minimum threshold

None of these were obvious from the outside. The pipeline was running, queries were returning results, everything looked fine. The duplicates were inflating retrieval scores for certain topics. The missing metadata was breaking a filter that wasn't catching the error gracefully.

The MCP server benchmarks at ~0.95s for 15 concurrent graph-inspection requests.

Why MCP

I wanted this to be composable, not just another CLI tool you run once and forget.

Wrapping it as an MCP server means you can call validate_document_store, inspect_pipeline, diagnose_retrieval_failure, or collect_debug_bundle directly from Claude Desktop or any MCP-compatible client during a debugging session. The context stays in one place instead of jumping between terminals and notebooks.

Current state

Weaviate support is solid. Qdrant and Pinecone support is in progress. The project has been cloned by 90+ developers since I published it, which was surprising for something this niche.

If you're using a different document store and want to add a backend, contributions are open.

GitHub: https://github.com/rautaditya2606/haystack-diagnostics

Feedback welcome, especially if you hit a retrieval failure mode the engine doesn't classify correctly yet.

Building a Voice-Controlled Local AI Agent on a 4GB GPU

Aditya Raut — Sun, 12 Apr 2026 20:57:55 +0000

What I Built
I built a voice-controlled local AI agent that transcribes
audio, classifies intent, and executes local tools — all
visible through a transparent pipeline trace in a Gradio UI.
The agent supports four intents: create file, write code,
summarize text, and general chat.

Architecture
STT layer: Groq Whisper-large-v3 handles transcription via API.
I chose Groq over local Whisper because my RTX 3050 (4GB VRAM)
cannot run STT and an LLM simultaneously without OOM errors.
Groq's API is actually faster (~300ms) than local whisper-small
would have been.

Intent layer: Ollama serves qwen2.5-coder:1.5b locally. The LLM
returns a structured JSON intent that the tool router uses to
decide which action to take.

Tool layer: Four tools — create_file, write_code, summarize,
general_chat. All file writes are sandboxed to output/.

UI layer: Gradio displays transcription, detected intent, action
taken, and a full pipeline trace with per-stage latency.

Hardware Constraints and Decisions
My machine: Intel i5-12500H, RTX 3050 (4GB VRAM), 15GB RAM.

The core constraint: 4GB VRAM cannot hold both a Whisper model
and an LLM simultaneously.

Decision 1 — STT via Groq API
Running whisper-small locally uses ~1.5GB VRAM. That leaves
only 2.5GB for the LLM, which isn't enough for a useful model.
Offloading STT to Groq frees the entire 4GB for the LLM and
actually improves latency.

Decision 2 — qwen2.5-coder:1.5b via Ollama
A 1.5B model at Q4 quantization fits comfortably in ~1.5GB VRAM.
I initially tried the 7b variant but it exceeded available VRAM
and caused Ollama to offload to RAM, significantly slowing
inference.

Decision 3 — Sequential pipeline
STT completes before Ollama is called. This keeps peak VRAM
usage under 2GB at any given time.

*Challenges I Faced *

VRAM management
Loading two models simultaneously caused OOM errors. Solved
by switching STT to Groq and keeping only the LLM local.
Intent JSON parsing
Ollama sometimes returns malformed JSON or wraps it in
markdown code fences. Solved with a robust parser that
strips fences and falls back to keyword matching if JSON
parsing fails entirely.
Output sandboxing
Naive file creation allowed path traversal (e.g.
../../etc/passwd). Solved with path normalization and
checking that the resolved path starts with the output/
directory.
Gradio mic input format
Gradio returns audio as a tuple (sample_rate, numpy_array)
not a file path. Had to write it to a temp file before
passing to Groq API.

What I'd Do Differently at Scale
For a production version of this system, I would:

Replace Ollama with Triton Inference Server for proper model serving with batching and metrics endpoints.
Add a message queue (Redis or RabbitMQ) between the UI and pipeline so multiple users don't block each other.
Replace the flat logger with structured JSON logs shipped to an observability stack (Grafana + Loki).
Add model versioning — config.yaml currently hardcodes model names. A proper MLOps setup uses a model registry.
Containerize STT locally using a sidecar so the pipeline has no external API dependency in production.

Model Benchmarking

I added a benchmarking tab — set models, prompt, iterations,
get a latency table back.

Model	Avg Latency
qwen2.5-coder:1.5b	~3.2s
qwen2.5-coder:7b	~11.4s

For structured JSON intent extraction, the 1.5b model is
3-4x faster with no meaningful accuracy difference. For a
constrained task like this, bigger isn't better.

Persistent Memory

Every pipeline run is stored in SQLite — transcription,
intent, action, output, and trace. Surfaces in the UI as
a recent runs panel.

This matters for two reasons:

Debugging — if intent classification goes wrong, you can see exactly what transcription and JSON the LLM returned
Auditability — every file written has a corresponding memory entry with the voice command that triggered it

Simple schema, append-only, no ORM:

CREATE TABLE IF NOT EXISTS runs (
    id        INTEGER PRIMARY KEY AUTOINCREMENT,
    timestamp TEXT,
    transcript TEXT,
    intent    TEXT,
    action    TEXT,
    output    TEXT,
    trace     TEXT
)

Links
GitHub: https://github.com/rautaditya2606/Aditya_Raut_Mem0_AI
Demo: https://youtu.be/rhGIQvi4Y74