DEV Community: Aditya Raut

I Got 28 TPS Out of Free Kaggle GPUs. Here's What It Took.

Aditya Raut — Sun, 23 Aug 2026 12:20:19 +0000

I want to be upfront about something: this whole project runs on free Kaggle T4 notebooks, an AWS EC2 t3.micro relay that costs almost nothing, and public internet. No A100s. No private datacenter network. No budget.

And yet, ShardFlow v2.1 hits 28.10 TPS peak on Qwen2.5-7B across two separate cloud regions over WAN.

This is the story of how that happened, and specifically the one fix in v2.1 that I did not see coming.

The Problem: Running a 7B Model When You Have No Money

A 7B parameter model in FP16 needs roughly 15 GB of VRAM. A single Kaggle T4 has 16 GB. Technically it fits, barely, with nothing left over for a KV cache.

The solution is tensor parallelism: split the model across two machines. Node 0 (Iowa) handles layers 0 to 14. Node 1 (Oregon) handles layers 14 to 28, plus the LM head and final verification. They talk to each other through a TCP relay running on an EC2 t3.micro in Ohio.

The baseline throughput with this setup and no tricks: 4.92 TPS. Usable, but not fast.

Speculative Decoding: The Idea

LLM inference is slow because it's sequential. You generate one token, wait, generate another, wait. Each round trip across WAN costs you ~86ms RTT. At 1 token per round trip, you're fighting the network the whole time.

Speculative decoding flips this. Instead of sending one token at a time, you run a tiny draft model locally to guess the next K tokens ahead. Then you send all K guesses to the verifier in one shot. If the big model agrees with M of them, you've committed M tokens in a single round trip instead of one.

ShardFlow uses Qwen2.5-0.5B as the draft model, running on cuda:1 of Node 0 while the 7B target slice runs on cuda:0. Zero VRAM contention. The drafter proposes 8 candidates, Node 1 verifies them all in parallel, and you get an average of 4.07 tokens per round trip instead of 1.

With speculative decoding in eager mode: 14.3 TPS peak. 3x better.

The Wall I Hit

I thought 14.3 was the ceiling. The network was the obvious bottleneck: two Kaggle instances in different states, an EC2 relay in between, public internet routing. What else could you do?

Then I looked more carefully at what the draft model was actually doing.

Every round, generating 8 candidate tokens meant running 8 separate forward passes through the 0.5B model. Each forward pass launched roughly 1,500 CUDA kernels, one by one, from a Python loop.

Here's the problem: each CUDA kernel executes in 2 to 5 microseconds on the GPU. But Python needs 8 to 10 microseconds just to issue the launch call. The GPU was sitting idle for more time than it was actually computing. Draft generation per round: 112ms. The GPU idle rate: 65%.

Python was quietly murdering GPU utilization and I had no idea.

CUDA Graphs: What They Are and Why They Helped

A CUDA Graph is a way to capture a sequence of GPU operations once and replay them as a single driver call.

Normally, every time your model does a forward pass, Python issues hundreds or thousands of individual kernel launches. Each one is a separate call to the CUDA driver. That overhead adds up fast, especially when you're doing it in a loop.

With CUDA Graphs, you capture the entire forward pass of the 0.5B draft model: all 24 transformer layers, the LM head, the argmax for the next token. You do this once. After that, replaying the whole thing costs one driver call. No Python in the hot path at all.

Draft generation: 112ms to 25ms. 4.5x faster.

Why It Kept Breaking (and How I Fixed It)

Every time I tried CUDA Graphs, the model started looping: "the the the the the". Clearly something was wrong.

CUDA Graphs capture exact GPU memory addresses at record time. If any tensor gets reallocated during replay, the graph reads from a stale address and you get garbage output.

HuggingFace's default KV cache (DynamicCache) calls torch.cat every single token step. That allocates a new buffer every time. The graph had captured the old address. Replay read from it. Output: garbage.

Four changes fixed this:

1. StaticCache instead of DynamicCache. StaticCache pre-allocates fixed-size buffers for the KV cache. No reallocations during generation. The addresses the graph captured stay valid.

2. In-place tensor mutation. Instead of creating new tensors for intermediate values, everything gets written in-place. Same memory, same address, graph stays happy.

3. Explicit position_ids updates. The graph needs to know which position each token is at. With dynamic allocation, this was implicit. With a static graph, you have to update position_ids manually before each replay.

4. In-place KV rewind. When the speculative verifier rejects some draft tokens, the KV cache needs to roll back to the last accepted position. This rewind has to happen in-place, not by creating a new cache object.

Once all four were in place: no more loops. Clean output. 25ms draft generation.

The Numbers

On Qwen2.5-7B across 2 Kaggle T4s over WAN:

Version	TPS
v1.0 (REST relay)	2.27
v2.0 baseline	4.92
v2.0 + neural drafter (eager)	14.3 peak
v2.1 + CUDA graphs	28.10 peak / 20.31 avg

Also tested on Qwen2.5-14B with 4-bit NF4 quantization, same two T4s: 14.43 TPS average over WAN. A 14.7B parameter model. Free GPUs.

What I Actually Learned

The network was not the bottleneck. I spent a lot of time assuming the WAN latency was the hard ceiling, that there was nothing left to squeeze. The real bottleneck was Python kernel launch overhead, and it was invisible until I looked at GPU idle time.

Profiling matters more than intuition. "The network is slow" is an easy assumption to make. "Python is launching 1,500 kernels from a loop and the GPU is idle 65% of the time" requires actually measuring.

CUDA Graphs are not magic. They are very specific. Captured addresses must stay valid. Any dynamic allocation breaks them. The StaticCache + in-place mutation combination is what makes them work for autoregressive generation.

Try It Yourself

ShardFlow is open source and designed to reproduce on free Kaggle notebooks. You need two Kaggle accounts and an EC2 t3.micro (or any machine with a public IP).

github.com/rautaditya2606/Shardflow

The README has step-by-step instructions for reproducing the exact benchmark. 583 people have already cloned it. I'd love to know if you get different numbers on different hardware.

v3 is whenever someone sponsors me actual GPUs. Until then, free T4s and Ohio relays it is.

Production RAG Optimization: Practical Techniques That Improved Performance

Aditya Raut — Wed, 01 Jul 2026 15:46:12 +0000

Building a Retrieval-Augmented Generation (RAG) system is relatively straightforward. Building one that performs well in production is a different challenge altogether.

Once a RAG pipeline starts handling real users and large document collections, common bottlenecks begin to appear: increasing response latency, rising LLM costs, and slower document ingestion.

It's tempting to solve these problems by switching to a larger model or waiting for cheaper APIs. In practice, many of the biggest improvements come from optimizing the architecture around the model rather than changing the model itself.

This article covers several engineering techniques that proved effective while optimizing a production RAG system. None of them are particularly novel on their own, but together they produced meaningful improvements in latency, cost, and overall system efficiency.

1. Route Queries Before Retrieving

One of the easiest ways to waste latency and tokens is to perform semantic retrieval for every user request.

Many queries don't require vector search at all.

Examples include:

Greetings
Questions about uploaded documents or metadata
General conversational interactions

Rather than embedding every query and performing retrieval, a lightweight intent classification stage can determine whether semantic search is actually necessary.

If retrieval isn't required, the request can be answered directly through conversational logic or structured backend data.

Besides reducing unnecessary work, this also frees retrieval resources for queries where semantic search genuinely improves answer quality.

2. Rerank Before Building the Prompt

Most retrieval systems are optimized for recall.

Returning 10–20 potentially relevant chunks is often desirable during retrieval, but sending every retrieved chunk to the LLM usually isn't.

Larger prompts increase:

Input token usage
First-token latency
Overall inference time
The likelihood of distracting the model with irrelevant context

A lightweight reranking stage can identify the small subset of retrieved passages most likely to answer the user's question.

Only those passages are forwarded to prompt construction.

This keeps prompts focused while significantly reducing unnecessary context.

3. Reduce Context Instead of Reducing Recall

Even after reranking, retrieved chunks frequently contain far more text than necessary.

A document chunk might contain several hundred tokens while only a few sentences are actually relevant.

Instead of embedding smaller chunks—which can hurt retrieval quality—another approach is to trim context after retrieval.

For factual questions, relevant sections can be extracted by locating query-related keywords and selecting a window surrounding those matches.

For summarization tasks, preserving a larger leading section often produces better results than aggressive trimming.

This approach maintains retrieval quality while substantially reducing prompt size.

4. Parallelize Document Processing

Ingestion pipelines are frequently limited by I/O rather than CPU.

Tasks such as:

PDF extraction
Office document parsing
OCR
Text cleaning
Chunk preparation

can usually be processed independently.

Instead of processing uploaded files sequentially, these operations can be executed concurrently using a worker pool.

Parallel preprocessing improves indexing throughput and reduces the time required before newly uploaded documents become searchable.

5. Perform OCR Before Retrieval

Supporting scanned PDFs and image-heavy documents is a common production requirement.

One option is to send these documents directly to multimodal vision models.

While effective, this increases both inference cost and latency.

An alternative approach is to perform OCR during preprocessing.

Pages without selectable text are detected automatically and converted into plain text before entering the indexing pipeline.

Performing OCR locally offers several practical advantages:

Reduces reliance on multimodal inference
Produces standard text suitable for embedding models
Lowers preprocessing costs
Makes downstream retrieval pipelines simpler

To improve retrieval quality, very small or low-confidence OCR outputs should also be discarded before indexing, preventing noisy embeddings from entering the vector store.

6. Measure Every Optimization

Performance improvements should be measured rather than assumed.

Useful metrics include:

End-to-end latency
Retrieval latency
Generation latency
Prompt token usage
Completion token usage
Document ingestion time
Indexing throughput

Collecting these metrics throughout the pipeline makes it possible to compare changes before and after each optimization.

Without instrumentation, it's difficult to know whether an optimization actually improved the system or simply shifted the bottleneck elsewhere.

Trade-offs

Every optimization introduces trade-offs.

Intent routing adds an additional classification step but avoids unnecessary retrieval.

Reranking introduces another model call but often reduces the total generation cost by producing much smaller prompts.

Local OCR shifts computation from cloud APIs to the application server, reducing inference costs while increasing CPU utilization.

The right choice depends on workload characteristics, latency requirements, infrastructure, and operational costs.

Key Takeaways

One lesson became increasingly clear throughout this work:

Many production RAG bottlenecks are architectural rather than model-related.

Several practical optimizations consistently provided meaningful improvements:

Route requests intelligently so retrieval only runs when it adds value.
Rerank retrieved results before prompt construction.
Trim context after retrieval instead of stuffing entire document chunks into prompts.
Parallelize document preprocessing and indexing.
Prefer local OCR when multimodal reasoning is unnecessary.
Instrument the pipeline so improvements can be measured objectively.

Modern language models continue to improve rapidly, but production performance often depends just as much on retrieval, preprocessing, prompt construction, and observability as it does on the model itself.

I'm currently expanding on these ideas through an open-source diagnostics toolkit for Haystack pipelines. If you're working on production RAG systems or interested in collaborating, I'd be happy to connect and exchange ideas.

Linkedin: https://www.linkedin.com/in/aditya-raut-3b4bba31b/
Haystack Diagnostics: https://github.com/rautaditya2606/haystack-diagnostics

I Got Tired of Debugging Haystack RAG Pipelines Blind, So I Built a Diagnostics Engine

Aditya Raut — Sat, 27 Jun 2026 11:40:13 +0000

RAG pipelines fail in quiet ways.

Retrieval drops. Documents go missing. Metadata gets corrupted somewhere between ingestion and query time. Your generator starts hallucinating and you don't know if it's the retriever, the document store, or something upstream.

The debugging loop is always the same: check traces, grep logs, write a one-off script to inspect the document store, try to diff two runs manually. It works, but it's slow and it doesn't scale.

I hit this enough times while working on a Haystack 2.x pipeline at my internship that I started building something to systematize it.

That became Haystack Diagnostics Engine.

What it actually does

Four things:

Document store validation — checks your vector store for duplicate chunks, missing metadata fields, and short/malformed documents before they silently degrade retrieval quality.

Pipeline introspection — inspects your Haystack pipeline structure, flags misconfigurations, and can visualize the component graph. Useful when you're inheriting a pipeline someone else built.

Retrieval failure classification — when a query returns garbage, this tells you why. Six failure classes: empty results, low-score results, metadata filter mismatch, reranker collapse, score inversion, and retriever timeout. Each has a different fix. Uses Haystack's include_outputs_from for single-pass retriever/reranker diagnostics without re-running the pipeline.

Debug bundle capture and diffing — this is the one I've gotten the most feedback on.

Debug bundles: the part that actually changed my workflow

The typical production debugging scenario: something worked last week, it doesn't work now, and you have no idea what changed.

collect_debug_bundle(pipeline, query, ...) captures the full state of a single query execution as a structured JSON file:

Pipeline graph, Haystack version, component init_parameters
Raw retriever top-k (pre-reranker) — scores, metadata, content previews
Reranked top-k when a reranker is detected
Prompt snapshot and generated answer
Failure classification result
Corpus health checks scoped to only the retrieved document IDs, not the full store

Bundle filenames are {query_slug}_{timestamp}.json — human-readable, sort naturally across runs of the same query. The UUID lives inside the JSON, not in the filename.

diff_debug_bundles(bundle_a, bundle_b) compares two persisted bundles and reports:

Score deltas per document
Docs that appeared or disappeared between runs
Component config changes between the two pipeline states
Character-level answer diff via difflib (no tokenizer dependency)

Config diffs filter out known volatile fields by default — things like InMemoryDocumentStore.index, which regenerates as a random UUID on instantiation and would create false positives on every diff. You can pass ignore_config_paths=set() to disable filtering or extend the defaults with component-specific paths like "retriever.session_id".

There's also a CLI:

python -m diagnostics.debug_bundler diff bundle_a.json bundle_b.json

The workflow this enables: run a query, persist the bundle, deploy a change, run the same query again, diff the two bundles. You get an exact record of what shifted — scores, docs, config, answer — without relying on memory or logs.

What it found on a real deployment

I ran the validator against a live Weaviate-backed RAG instance with 823 chunks and OpenAI embeddings.

195 duplicate chunks (23.7% of the corpus)
14 documents missing required metadata keys
8 anomalous short chunks under the minimum threshold

None of these were obvious from the outside. The pipeline was running, queries were returning results, everything looked fine. The duplicates were inflating retrieval scores for certain topics. The missing metadata was breaking a filter that wasn't catching the error gracefully.

The MCP server benchmarks at ~0.95s for 15 concurrent graph-inspection requests.

Why MCP

I wanted this to be composable, not just another CLI tool you run once and forget.

Wrapping it as an MCP server means you can call validate_document_store, inspect_pipeline, diagnose_retrieval_failure, or collect_debug_bundle directly from Claude Desktop or any MCP-compatible client during a debugging session. The context stays in one place instead of jumping between terminals and notebooks.

Current state

Weaviate support is solid. Qdrant and Pinecone support is in progress. The project has been cloned by 90+ developers since I published it, which was surprising for something this niche.

If you're using a different document store and want to add a backend, contributions are open.

GitHub: https://github.com/rautaditya2606/haystack-diagnostics

Feedback welcome, especially if you hit a retrieval failure mode the engine doesn't classify correctly yet.

Building a Voice-Controlled Local AI Agent on a 4GB GPU

Aditya Raut — Sun, 12 Apr 2026 20:57:55 +0000

What I Built
I built a voice-controlled local AI agent that transcribes
audio, classifies intent, and executes local tools — all
visible through a transparent pipeline trace in a Gradio UI.
The agent supports four intents: create file, write code,
summarize text, and general chat.

Architecture
STT layer: Groq Whisper-large-v3 handles transcription via API.
I chose Groq over local Whisper because my RTX 3050 (4GB VRAM)
cannot run STT and an LLM simultaneously without OOM errors.
Groq's API is actually faster (~300ms) than local whisper-small
would have been.

Intent layer: Ollama serves qwen2.5-coder:1.5b locally. The LLM
returns a structured JSON intent that the tool router uses to
decide which action to take.

Tool layer: Four tools — create_file, write_code, summarize,
general_chat. All file writes are sandboxed to output/.

UI layer: Gradio displays transcription, detected intent, action
taken, and a full pipeline trace with per-stage latency.

Hardware Constraints and Decisions
My machine: Intel i5-12500H, RTX 3050 (4GB VRAM), 15GB RAM.

The core constraint: 4GB VRAM cannot hold both a Whisper model
and an LLM simultaneously.

Decision 1 — STT via Groq API
Running whisper-small locally uses ~1.5GB VRAM. That leaves
only 2.5GB for the LLM, which isn't enough for a useful model.
Offloading STT to Groq frees the entire 4GB for the LLM and
actually improves latency.

Decision 2 — qwen2.5-coder:1.5b via Ollama
A 1.5B model at Q4 quantization fits comfortably in ~1.5GB VRAM.
I initially tried the 7b variant but it exceeded available VRAM
and caused Ollama to offload to RAM, significantly slowing
inference.

Decision 3 — Sequential pipeline
STT completes before Ollama is called. This keeps peak VRAM
usage under 2GB at any given time.

*Challenges I Faced *

VRAM management
Loading two models simultaneously caused OOM errors. Solved
by switching STT to Groq and keeping only the LLM local.
Intent JSON parsing
Ollama sometimes returns malformed JSON or wraps it in
markdown code fences. Solved with a robust parser that
strips fences and falls back to keyword matching if JSON
parsing fails entirely.
Output sandboxing
Naive file creation allowed path traversal (e.g.
../../etc/passwd). Solved with path normalization and
checking that the resolved path starts with the output/
directory.
Gradio mic input format
Gradio returns audio as a tuple (sample_rate, numpy_array)
not a file path. Had to write it to a temp file before
passing to Groq API.

What I'd Do Differently at Scale
For a production version of this system, I would:

Replace Ollama with Triton Inference Server for proper model serving with batching and metrics endpoints.
Add a message queue (Redis or RabbitMQ) between the UI and pipeline so multiple users don't block each other.
Replace the flat logger with structured JSON logs shipped to an observability stack (Grafana + Loki).
Add model versioning — config.yaml currently hardcodes model names. A proper MLOps setup uses a model registry.
Containerize STT locally using a sidecar so the pipeline has no external API dependency in production.

Model Benchmarking

I added a benchmarking tab — set models, prompt, iterations,
get a latency table back.

Model	Avg Latency
qwen2.5-coder:1.5b	~3.2s
qwen2.5-coder:7b	~11.4s

For structured JSON intent extraction, the 1.5b model is
3-4x faster with no meaningful accuracy difference. For a
constrained task like this, bigger isn't better.

Persistent Memory

Every pipeline run is stored in SQLite — transcription,
intent, action, output, and trace. Surfaces in the UI as
a recent runs panel.

This matters for two reasons:

Debugging — if intent classification goes wrong, you can see exactly what transcription and JSON the LLM returned
Auditability — every file written has a corresponding memory entry with the voice command that triggered it

Simple schema, append-only, no ORM:

CREATE TABLE IF NOT EXISTS runs (
    id        INTEGER PRIMARY KEY AUTOINCREMENT,
    timestamp TEXT,
    transcript TEXT,
    intent    TEXT,
    action    TEXT,
    output    TEXT,
    trace     TEXT
)

Links
GitHub: https://github.com/rautaditya2606/Aditya_Raut_Mem0_AI
Demo: https://youtu.be/rhGIQvi4Y74