Shourya Parashar

Posted on Sep 5 Originally published at shouryaparashar.in

Building a Voice RAG System Under a 200ms Latency Budget

#rag #ai #python

What happens when you combine Voice AI + RAG + multilingual retrieval, but give yourself a strict 200ms latency budget?

That was the challenge behind my HH Goa 2026 Task 2 project.

I built a voice-driven RAG system on the AI4Bharat MSMARCO-XI dataset, designed to retrieve relevant information, generate grounded answers, and gracefully refuse when the evidence isn't strong enough.

The Pipeline

Voice
  ↓
Speech-to-Text
  ↓
Query Guard
  ↓
Hybrid Retrieval
  ↓
RRF + MMR
  ↓
Grounded Answer
  ↓
Verification

What made it interesting?

Instead of relying only on vector search, I combined:

Dense retrieval
BM25+ sparse retrieval
Reciprocal Rank Fusion
MMR reranking
Multiple chunking strategies
Retrieval confidence checks
Prompt-injection protection
Grounding verification
Extractive fallback answers

The goal wasn't just to make the system answer.

It was to make it answer only when it has enough evidence.

The part I cared about most: latency

I instrumented every stage and benchmarked the complete pipeline.

The final results:

Metric	Result
P50	7.87 ms
P70	9.62 ms
P100	40.71 ms
Benchmark runs	927
Within 200ms	100%

I also tested the actual HTTP service separately, instead of relying only on in-process benchmarks.

A few lessons

The biggest takeaway wasn't a particular model or framework.

It was that good AI systems are often about making the right engineering trade-offs.

A bigger model isn't always better.

More retrieval isn't always better.

And sometimes the best answer is simply:

"I don't have enough evidence to answer that."

This project pushed me to think about RAG not just as retrieval + generation, but as a complete system with latency, reliability, safety, and verification built into the architecture.

Built with

Python · FastAPI · BM25+ · Dense Retrieval · RRF · MMR · Sarvam · Claude · Docker

If you're interested in the implementation, the complete project is here:

GitHub: https://github.com/im-shourya/HHGOA-TASK2

Built for HH Goa 2026 — Shortlisting Task 2.

AI #RAG #Python #LLM #MachineLearning #VoiceAI #InformationRetrieval

DEV Community