DEV Community: Pavan Indupudi

Inside VOICE-RAG: From Speech to Grounded Answers

Pavan Indupudi — Tue, 20 Oct 2026 19:50:59 +0000

Pavan Ganesh Indupudi (Infernix) · Originally published at the-infernix.github.io

Ask most RAG demos "Who is the Prime Minister of India?" and they will happily hallucinate an answer from whatever passage happens to look semantically close. VOICE-RAG — built for HH Goa 2026 — answers that question with a refusal and a number:

Refused: Top retrieval score 0.779 below relevance floor 0.840

That sentence is the whole thesis. The system answers spoken questions in English, Hindi, Gujarati, and Telugu — strictly from a curated 40,000-passage corpus — and refuses when the corpus does not contain the answer. This article covers why that turned out to be the hard part, and how the 56 ms core around it was built.

The problem: fluent answers with no evidence

Retrieval-augmented generation fails in two directions. Too loose, and the generator papers over missing evidence with fluent nonsense. Too strict, and the system refuses questions it could have answered. Most demos only tune for the first failure — making answers sound grounded. We optimized for the second failure being visible: every refusal states its score and its floor, and the UI shows the entire evidence chain either way.

Constraints

Voice is the primary input. Hold-to-talk capture, 16 kHz mono WAV, Sarvam saaras:v3 speech-to-text. Voice adds roughly 1 s of STT; everything after it had to be fast enough to disappear.
Four languages, one index. English, Hindi, Gujarati, Telugu over the MSMARCO-XI corpus — 10k passages per language, 384-dim multilingual E5 embeddings.
Corpus-bounded by design. World-knowledge questions with zero supporting passages must be refused, not answered from parametric memory.
No bundler. The frontend is a vanilla-JS SPA served directly by FastAPI. Zero node_modules, zero build step.

Architecture: three guards around a 56 ms core

The pipeline is a strict sequence, and every stage reports its latency and its decision into a trace the UI renders as a "why this answer" chain:

The evidence drawer and latency trace — every guard decision is user-visible, not server-side secret sauce.

Input guard → embed → vector search → relevance guard → generate → grounding guard. Measured medians (n = 2,000 benchmark queries):

Stage	P50 latency	What happens
Input guard	0.02 ms	Injection / unsafe content / length checks
Embed query	23.67 ms	E5 `query:` prefix → 384-dim vector
Vector search	30.74 ms	Cosine over 40k passages + language rerank
Relevance guard	0.01 ms	Top score vs calibrated floor
Answer generation	0.09 ms	Best-sentence extraction from top passage
Grounding guard	0.17 ms	Token overlap, embedding fallback if needed
Total core	55.75 ms	Target: < 200 ms

Design decision
Default to extractive answering — pick the best-supported sentence from the top passage — and make LLM generation strictly opt-in. Extractive answers are 0.09 ms and grounded by construction; generative answers take multiple seconds through OpenRouter and need a citation contract to stay honest. Latency budget first, generative flair when asked for.

The refusal engine: floors calibrated from measurements

The relevance guard is one comparison — top-1 cosine score against a per-language floor — but the floors are not guesses. They were calibrated by measuring score distributions: known-relevant queries clustered around P25–P50 ≈ 0.84–0.86, off-corpus junk around ≈ 0.79–0.85, and the floors were set at the separation point:

Language	Floor	Low-confidence band
English	0.840	+0.03
Hindi	0.850	+0.03
Gujarati	0.840	+0.03
Telugu	0.840	+0.03

The query's language selects the floor — not the top chunk's — so cross-language junk cannot sneak through under a borrowed threshold. Language detection is Unicode-script based (Telugu block → te, Gujarati → gu, Devanagari → hi, else en), and auto-detected language applies a soft +0.03 rerank bonus toward same-language passages over a 3× candidate pool.

Q: Who is the Prime Minister of India?   → REFUSED  (0.779 < 0.840)
Q: Who won the FIFA World Cup 2022?      → REFUSED  (0.796 < 0.840)
Q: How much does an Xbox 360 cost?       → answered ($80–$250, conf 0.933)

The grounding guard: answers must be traceable

Passing the relevance gate is not enough. The grounding guard verifies that the answer is actually supported by retrieved text: token overlap ≥ 0.231 passes instantly; below that, an embedding-similarity check (≥ 0.794) runs as fallback. Ungrounded generations are replaced with the extractive answer or refused outright.

When generative mode is enabled, the LLM must return strict JSON — {answer, citations[], confidence} — citing only supplied passages. Malformed output gets exactly one retry. Invented or out-of-range citations are dropped, and if none survive, the pipeline falls back to extractive. An ungrounded generation can never reach the user.

Experiments: 2,000 queries, and the hybrid detour

The benchmark suite runs 2,000 corpus queries (500 per language, balanced, seed-fixed) against the prebuilt index:

Metric	All	English	Hindi	Gujarati	Telugu
P50	55.8 ms	54.9	56.0	56.7	55.3
P90	69.4 ms	—	—	—	—
Within 200 ms	99.95%	100%	99.8%	100%	100%

Guard activity in that run: 1,893 answered · 107 refused — weak-match queries correctly declined rather than answered with junk. The single P100 outlier (273 ms, Gujarati) was one OS-scheduling event; the next-worst query across all languages was 178 ms.

We also built and measured a hybrid BM25 + dense leg (bm25s, reciprocal rank fusion), evaluated against MSMARCO's own gold labels:

Metric	Dense	Hybrid (BM25 + RRF)
HitRate@10	80.31%	80.69%
MRR@10	0.4664	0.4669
Search P50	16.2 ms	16.9 ms

Trade-off
Hybrid bought +0.4 pp hit rate for ~1 ms and a large maintenance surface. Measured parity means dense-only stays the default; the sparse leg ships flag-ready behind retrieval.hybrid.enabled and off by default. RRF is used for candidate-pool expansion only — final ordering is always cosine, which keeps retrieval scores calibrated with the refusal floors. A hybrid ordering would have silently invalidated every floor we measured.

Every search also runs against a 200 ms deadline; skipped legs surface as explicit degradation strings (sparse_leg_skipped_deadline) in the stage trace, never as silent timeouts.

Limitations

Corpus-bounded knowledge. "What is the capital of India?" has zero supporting passages and is correctly refused — a feature, but it surprises users expecting a chatbot.
The borderline zone is real. A band around 0.84–0.86 overlaps between weak-relevant and strong-junk retrievals. Per-language floors trade a little recall for a lot of precision; some answerable questions get refused.
Voice adds ~1 s. Sarvam STT dominates end-to-end time; the RAG core's 56 ms is invisible under it.
Generative mode is opt-in precisely because it breaks the latency budget.

Lesson
Calibrate thresholds from measured score distributions, not intuition. The floors took an afternoon of histogram work and saved the product: without them, the system would have been confidently wrong in exactly the situations where being wrong matters most. A refusal with a stated score is also infinitely more debuggable than a silent abstention.

Reproducibility

The prebuilt index (40k × 384-dim vectors) ships with the repo, so first run needs no indexing step. Raw per-query benchmark data is committed at bench/results/results.json.

git clone https://github.com/The-Infernix/VOICE-RAG.git
cd VOICE-RAG
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
uvicorn api.main:app --port 7860   # SPA at http://localhost:7860

pytest tests/ -q                   # 178 passing
python bench/run_bench.py --limit 2000      # latency
python bench/eval_retrieval.py               # HitRate@10 / MRR@10 vs gold
python bench/run_bench.py --limit 500 --hybrid --soft-lang \
  --out bench/results/ab_hybrid              # the hybrid A/B above

Takeaways

Refusal is a product feature, not an error path. Make it measurable, calibrated, and visible.
Guard every stage independently — input, relevance, grounding — and surface every decision in the UI trace.
When a measured improvement is inside the noise, ship the simpler system and keep the complex one behind a flag.
Keep retrieval ordering aligned with whatever your thresholds were calibrated against.

References

VOICE-RAG repository — bench results, 178-test suite, config
bench/results/summary.json — the latency numbers quoted above
bench/eval_retrieval.py — gold-label evaluation against MSMARCO is_selected labels
guardrails/relevance_guard.py — the calibrated per-language floors
Built for HH Goa 2026, Task 2

Pavan Ganesh Indupudi (Infernix) — CS student at Andhra University building privacy-preserving AI. Full article and more at the-infernix.github.io · GitHub · LinkedIn

Building PRIVYSE: Designing a Privacy-First Browser Agent

Pavan Indupudi — Tue, 20 Oct 2026 19:24:27 +0000

Pavan Ganesh Indupudi (Infernix) · Originally published at the-infernix.github.io

The pitch for browser agents is seductive: give a vision-language model access to your tabs, and it can fill forms, file reports, and navigate tools you never wrote an integration for. The implementation detail that pitch skips is brutal — to see the page, the model sees everything on the page. Your email. Your PAN. The card number half-typed into a KYC form. Your face in a video call thumbnail.

PRIVYSE — built for SIH track 26171, On-device Visual Perception for Lightweight Browser Agents — starts from a different question: how do we let the model control the browser without giving it the user's private screen? The answer is not a privacy policy bolted onto a normal agent. It is an inverted pipeline where privacy is the architecture.

The problem: agents that trust the model with the screen

A conventional computer-use agent does one thing every step: capture the viewport, ship the pixels, ask the model what to do. That single habit can expose emails, phone numbers, government IDs, passwords, OTPs, API keys, faces, and sensitive documents — all in one JPEG. Prompting the model to "ignore" the PII does nothing; the data has already left the device.

Sanitizing after the model has seen the screen is not sanitization. The only place privacy can be enforced is before the privacy boundary — between the browser and the model.

Constraints that shaped the design

The boundary must be demonstrable. Not "we promise not to log it" — a gate a user can watch, failing closed, on every step.
The DOM is not enough. Modern pages hide values in <canvas>, SVG, images, video, and tiny rendered text. A DOM-only sanitizer has holes you cannot see.
Redaction cannot destroy context. If every email becomes a black box, the model cannot complete tasks that need to reference it. The rewrite has to preserve referential structure.
The agent must stay useful. A gate that blocks every image-heavy page forever is a denial-of-service, not a privacy system — degradation has to be explicit, not silent.

Architecture: capture → sanitize → gate → reason → act

Every agent step runs the same visible pipeline. The most important component in it is not the model — it is the gate.

One agent step end-to-end. The gate (04) is the only path across the privacy boundary.

Capture — captureVisibleTab grabs the viewport; a DOM walker serializes interactive elements (id, label, role, value, bbox) capped at 120 elements, and hashes pixels for the perception cache.
Perceive — MobileViT-Small (quantized ONNX, 6.3 MB), BlazeFace, and region-restricted Tesseract.js OCR run locally, in parallel.
Sanitize — three tiers of redaction (below), then Set-of-Marks paints numbered [N] tags so the model targets integers, never coordinates.
Gate — two independent checks, detailed in the next section. Any hit blocks the step.
Reason — only the redacted JPEG plus numbered DOM reach the local FastAPI server, which routes to a Qwen2.5-VL 3B or 7B and gets exactly one JSON action back.
Act — a guarded executor resolves element ids and drives pointer/keyboard events through scheme allow-lists and a DO-NOT-MODIFY veto.

Three tiers of redaction — and why tokens beat black boxes

Not all sensitive data deserves the same protection. PRIVYSE classifies into three tiers:

Tier	Information	Protection
Tier A	Passwords, OTPs, CVVs, API keys	Solid black box — no text, no token
Tier B	Email, phone, PAN, Aadhaar, cards, names	Stable privacy token
Tier C	Faces, canvas, video, non-DOM graphics	Blur / region taint

Tier B is the interesting one. Destroying a value with a black box removes context the model needs; leaving it is a leak. The compromise is stable tokenization — the same raw value always maps to the same token for the run, so the model keeps continuity without ever seeing the secret:

john@example.com           4111 1111 1111 1111          9876543210@ybl
       ↓                           ↓                          ↓
   [EMAIL_1]                   [CARD_1]                   [UPI_1]

Cross-source continuity is deliberate: a value discovered by OCR in an image receives the same token when it later appears in the DOM. Indian identifiers — UPI IDs, voter ID, driving licence, passport, PAN, Aadhaar, IFSC — have dedicated detection patterns, measured at 100% precision/recall on the GT benchmark pages.

Design decision
Tokens must be stable across the whole run, not per-screenshot. Per-frame tokens would make the model re-learn the mapping every step and break any task that references a value found earlier. Global stability is also what lets the gate recognize a token as already-sanitized rather than as a leak.

The zero-leak gate: two checks, one fail-closed rule

Before anything crosses the boundary, the outbound payload is verified twice:

DOM verification — a pure regex scan (assertNoLeaks) over the outbound DOM JSON. Always runs. Cannot be skipped.
Pixel verification — the actual sanitized screenshot bytes are OCR-scanned again. This is the check that catches PII the DOM never carried.

Stated exception — read this before trusting the boundary
The two layers do not fail identically. A DOM-layer failure can never send. A pixel-layer check that runs and finds PII can never send. But if the on-device vision host is down, PII that exists only in pixels is not OCR-verified on that step. Blocking every image-heavy page indefinitely would make the agent unusable, so the degradation is deliberate — the side panel reports gate: skip — and all nine failure paths are walked in docs/perception-failure-path-audit.md. It is a real, narrow gap, not a claim of airtightness.

The exact bytes that cross the privacy boundary — reproduced by the benchmark suite's demo-image runner.

Perception-driven model routing

On-device perception pays a second dividend: the MobileViT tile map tells the server how much reasoning the step needs before the model is called. Interactive steps (form fill, navigation, typing) route to the 3B; read/report/captcha steps route to the 7B. Measured on the routing benchmark:

Routing config	Visual-context tasks	Steps
qwen2.5vl:3b only	2 / 3	4.7
qwen2.5vl:7b only	3 / 3	1.3
Routed 3B / 7B	2 / 3	5.0

The routed read/report failure is documented as a property of the harder prose-heavy test page, not of the routing logic — interactive accuracy is 3/3 either way. The small model is never trusted blindly: streamed actions are validated, coerced, and guarded before execution.

Experiments: what the benchmarks actually say

All numbers come from the repo's own suite (benchmarks/results/dashboard.json plus the suite runners), against ground-truth-labeled pages:

Metric	Result
PII detection F1 (pixels)	0.984
Precision / Recall	0.968 / 1.000
Zero-leak	PASS
Adversarial privacy suite	30 / 30 blocked
Executor guard tests	27 / 27 passed
Live /act spot-check (forged hostile responses)	10 / 10 refused
Raw data off-device	0 MB
Routed end-to-end latency	≈ 12.7–13 s / step

The adversarial suite is the number I trust most, because it is the least flattering: canvas PII, SVG PII, tiny text, obfuscated values, visual and text prompt injection, exfiltration via data:/blob:/javascript: form actions, forged /act responses, and protected-field overwrite attempts. Thirty attempts, thirty blocks. The executor's DO-NOT-MODIFY veto is deterministic, which means it defends against a hostile server response, not just an honest model making a mistake.

Known flake — honest reporting
The zero-leak OCR gate is occasionally nondeterministic — roughly 1 failure in 4 full runs, a blur-sample variance on one image page. The critical property: it has never failed open. When OCR cannot confirm, the step is blocked. A flaky gate that fails closed is a latency problem; a reliable gate that fails open is a privacy incident.

Limitations: the two defects we document instead of bury

Our own audit of the human-in-the-loop path found two high-severity defects, now tracked publicly in the README:

D2 (open, queued first) — history[].result and the user's task string are attached to the outbound body untokenized, and the DOM gate only walks body.dom[].text/value. PII typed into a human-in-the-loop answer could leave in plaintext with the gate blind to it. Until D2 is closed, the honest statement of the boundary is: DOM-sourced PII is provably blocked; PII introduced through the agent's own conversation with the user is not yet covered.
D1 (open) — the risky-action confirm detects Cancel via startsWith("cancel"), so typing "no" executes the action. It needs an explicit allow-list.

Listing these is not humility theater. A privacy claim a reader cannot check is marketing; a privacy claim with a public defect table is engineering.

Lesson
Build the audit harness before the feature that needs auditing. D2 was found by walking the human-in-the-loop paths with a test matrix, not by staring at the sanitizer. The same discipline that produced 30/30 on the adversarial suite produced the defect table — it is one muscle.

Reproducibility

Everything above is reproducible from the public repo. The short version:

cd server
.venv\Scripts\activate
pip install -r requirements.txt
uvicorn app:app --reload          # http://127.0.0.1:8000/health

# optional perception routing
$env:PERCEPTION_ROUTING = "1"
$env:VLM_MODEL_SMALL = "qwen2.5vl:3b"
$env:VLM_MODEL_BIG   = "qwen2.5vl:7b"

cd ../benchmarks
npm run build
node run.mjs                      # sanitizer P/R + zero-leak + latency
node aggregate.mjs                # merge into results/dashboard.json

The latency budget model, the perception failure-path audit, and the HITL test matrix live in docs/. If a number in this article disagrees with the repo, the repo wins.

Takeaways

Privacy in an agent system is an ordering constraint, not a feature. Sanitize before the model is allowed to look.
A gate users can watch — flash PASS/BLOCKED every step — changes what "trust" means. Trust becomes verification.
On-device perception is not just a privacy cost center; the same inference that protects the screen also routes the model and caches the page.
Publish your defects. The defect table is the most credible paragraph on the repo.

References

PRIVYSE repository — benchmarks, docs, defect table
SIH 26171 track brief — On-device Visual Perception for Lightweight Browser Agents
docs/perception-failure-path-audit.md — all nine gate failure paths
docs/hitl-test-matrix.md — the audit that found D1 and D2
docs/latency-budget-judge-gpu.md — the p50 < 5 s/step target model

Pavan Ganesh Indupudi (Infernix) — CS student at Andhra University building privacy-preserving AI. Full article and more at the-infernix.github.io · GitHub · LinkedIn