DEV Community: Aditya Raut

I Got Tired of Debugging Haystack RAG Pipelines Blind, So I Built a Diagnostics Engine

Aditya Raut — Sat, 27 Jun 2026 11:40:13 +0000

RAG pipelines fail in quiet ways.

Retrieval drops. Documents go missing. Metadata gets corrupted somewhere between ingestion and query time. Your generator starts hallucinating and you don't know if it's the retriever, the document store, or something upstream.

The debugging loop is always the same: check traces, grep logs, write a one-off script to inspect the document store, try to diff two runs manually. It works, but it's slow and it doesn't scale.

I hit this enough times while working on a Haystack 2.x pipeline at my internship that I started building something to systematize it.

That became Haystack Diagnostics Engine.

What it actually does

Four things:

Document store validation — checks your vector store for duplicate chunks, missing metadata fields, and short/malformed documents before they silently degrade retrieval quality.

Pipeline introspection — inspects your Haystack pipeline structure, flags misconfigurations, and can visualize the component graph. Useful when you're inheriting a pipeline someone else built.

Retrieval failure classification — when a query returns garbage, this tells you why. Six failure classes: empty results, low-score results, metadata filter mismatch, reranker collapse, score inversion, and retriever timeout. Each has a different fix. Uses Haystack's include_outputs_from for single-pass retriever/reranker diagnostics without re-running the pipeline.

Debug bundle capture and diffing — this is the one I've gotten the most feedback on.

Debug bundles: the part that actually changed my workflow

The typical production debugging scenario: something worked last week, it doesn't work now, and you have no idea what changed.

collect_debug_bundle(pipeline, query, ...) captures the full state of a single query execution as a structured JSON file:

Pipeline graph, Haystack version, component init_parameters
Raw retriever top-k (pre-reranker) — scores, metadata, content previews
Reranked top-k when a reranker is detected
Prompt snapshot and generated answer
Failure classification result
Corpus health checks scoped to only the retrieved document IDs, not the full store

Bundle filenames are {query_slug}_{timestamp}.json — human-readable, sort naturally across runs of the same query. The UUID lives inside the JSON, not in the filename.

diff_debug_bundles(bundle_a, bundle_b) compares two persisted bundles and reports:

Score deltas per document
Docs that appeared or disappeared between runs
Component config changes between the two pipeline states
Character-level answer diff via difflib (no tokenizer dependency)

Config diffs filter out known volatile fields by default — things like InMemoryDocumentStore.index, which regenerates as a random UUID on instantiation and would create false positives on every diff. You can pass ignore_config_paths=set() to disable filtering or extend the defaults with component-specific paths like "retriever.session_id".

There's also a CLI:

python -m diagnostics.debug_bundler diff bundle_a.json bundle_b.json

The workflow this enables: run a query, persist the bundle, deploy a change, run the same query again, diff the two bundles. You get an exact record of what shifted — scores, docs, config, answer — without relying on memory or logs.

What it found on a real deployment

I ran the validator against a live Weaviate-backed RAG instance with 823 chunks and OpenAI embeddings.

195 duplicate chunks (23.7% of the corpus)
14 documents missing required metadata keys
8 anomalous short chunks under the minimum threshold

None of these were obvious from the outside. The pipeline was running, queries were returning results, everything looked fine. The duplicates were inflating retrieval scores for certain topics. The missing metadata was breaking a filter that wasn't catching the error gracefully.

The MCP server benchmarks at ~0.95s for 15 concurrent graph-inspection requests.

Why MCP

I wanted this to be composable, not just another CLI tool you run once and forget.

Wrapping it as an MCP server means you can call validate_document_store, inspect_pipeline, diagnose_retrieval_failure, or collect_debug_bundle directly from Claude Desktop or any MCP-compatible client during a debugging session. The context stays in one place instead of jumping between terminals and notebooks.

Current state

Weaviate support is solid. Qdrant and Pinecone support is in progress. The project has been cloned by 90+ developers since I published it, which was surprising for something this niche.

If you're using a different document store and want to add a backend, contributions are open.

GitHub: https://github.com/rautaditya2606/haystack-diagnostics

Feedback welcome, especially if you hit a retrieval failure mode the engine doesn't classify correctly yet.

Building a Voice-Controlled Local AI Agent on a 4GB GPU

Aditya Raut — Sun, 12 Apr 2026 20:57:55 +0000

What I Built
I built a voice-controlled local AI agent that transcribes
audio, classifies intent, and executes local tools — all
visible through a transparent pipeline trace in a Gradio UI.
The agent supports four intents: create file, write code,
summarize text, and general chat.

Architecture
STT layer: Groq Whisper-large-v3 handles transcription via API.
I chose Groq over local Whisper because my RTX 3050 (4GB VRAM)
cannot run STT and an LLM simultaneously without OOM errors.
Groq's API is actually faster (~300ms) than local whisper-small
would have been.

Intent layer: Ollama serves qwen2.5-coder:1.5b locally. The LLM
returns a structured JSON intent that the tool router uses to
decide which action to take.

Tool layer: Four tools — create_file, write_code, summarize,
general_chat. All file writes are sandboxed to output/.

UI layer: Gradio displays transcription, detected intent, action
taken, and a full pipeline trace with per-stage latency.

Hardware Constraints and Decisions
My machine: Intel i5-12500H, RTX 3050 (4GB VRAM), 15GB RAM.

The core constraint: 4GB VRAM cannot hold both a Whisper model
and an LLM simultaneously.

Decision 1 — STT via Groq API
Running whisper-small locally uses ~1.5GB VRAM. That leaves
only 2.5GB for the LLM, which isn't enough for a useful model.
Offloading STT to Groq frees the entire 4GB for the LLM and
actually improves latency.

Decision 2 — qwen2.5-coder:1.5b via Ollama
A 1.5B model at Q4 quantization fits comfortably in ~1.5GB VRAM.
I initially tried the 7b variant but it exceeded available VRAM
and caused Ollama to offload to RAM, significantly slowing
inference.

Decision 3 — Sequential pipeline
STT completes before Ollama is called. This keeps peak VRAM
usage under 2GB at any given time.

*Challenges I Faced *

VRAM management
Loading two models simultaneously caused OOM errors. Solved
by switching STT to Groq and keeping only the LLM local.
Intent JSON parsing
Ollama sometimes returns malformed JSON or wraps it in
markdown code fences. Solved with a robust parser that
strips fences and falls back to keyword matching if JSON
parsing fails entirely.
Output sandboxing
Naive file creation allowed path traversal (e.g.
../../etc/passwd). Solved with path normalization and
checking that the resolved path starts with the output/
directory.
Gradio mic input format
Gradio returns audio as a tuple (sample_rate, numpy_array)
not a file path. Had to write it to a temp file before
passing to Groq API.

What I'd Do Differently at Scale
For a production version of this system, I would:

Replace Ollama with Triton Inference Server for proper model serving with batching and metrics endpoints.
Add a message queue (Redis or RabbitMQ) between the UI and pipeline so multiple users don't block each other.
Replace the flat logger with structured JSON logs shipped to an observability stack (Grafana + Loki).
Add model versioning — config.yaml currently hardcodes model names. A proper MLOps setup uses a model registry.
Containerize STT locally using a sidecar so the pipeline has no external API dependency in production.

Model Benchmarking

I added a benchmarking tab — set models, prompt, iterations,
get a latency table back.

Model	Avg Latency
qwen2.5-coder:1.5b	~3.2s
qwen2.5-coder:7b	~11.4s

For structured JSON intent extraction, the 1.5b model is
3-4x faster with no meaningful accuracy difference. For a
constrained task like this, bigger isn't better.

Persistent Memory

Every pipeline run is stored in SQLite — transcription,
intent, action, output, and trace. Surfaces in the UI as
a recent runs panel.

This matters for two reasons:

Debugging — if intent classification goes wrong, you can see exactly what transcription and JSON the LLM returned
Auditability — every file written has a corresponding memory entry with the voice command that triggered it

Simple schema, append-only, no ORM:

CREATE TABLE IF NOT EXISTS runs (
    id        INTEGER PRIMARY KEY AUTOINCREMENT,
    timestamp TEXT,
    transcript TEXT,
    intent    TEXT,
    action    TEXT,
    output    TEXT,
    trace     TEXT
)

Links
GitHub: https://github.com/rautaditya2606/Aditya_Raut_Mem0_AI
Demo: https://youtu.be/rhGIQvi4Y74