Let’s be brutally honest: most AI hackathon projects are just thin UI wrappers around an OpenAI API key. They look great in a demo, but they instantly break under real-world enterprise constraints.
Living around Kondapur, I face a specific friction point almost daily: The Metropolitan Migration Gap. I give delivery instructions in English or Hindi, but the driver natively speaks rapid Telugu over a compressed, noisy cellular line. Communication breaks down, food gets cold, and orders get canceled.
When I entered the Sarvam AI "Build In' Hours" Hackathon, I didn't want to build another consumer chatbot. In my day-to-day work scaling systems as an SDE-II, I know that enterprises like Swiggy or Urban Company will never throw away their existing tech stacks to adopt a weekend hack.
They need middleware.
Out of 6,000+ builders, our project (Patha-Darshak) made it to the Top 13 Finalists. Here is the exact technical blueprint, the architecture, and the 3 hard engineering lessons I learned building a real-time, 22-language translation proxy.
🏗️ The "Drop-In" Architecture
The core philosophy was simple: Do not force the client to rewrite their code.
Instead of building a new voice app, I built Setu—a Python/FastAPI engine that acts as a transparent proxy between legacy enterprise telecom systems and Sarvam AI’s native Indic models.
Architecture Flow
🔥 Lesson 1: Cloud VAD Will Kill Your Latency (Run It Locally)
If you stream continuous raw audio (including silence and background street noise) to a cloud Speech-to-Text API, two things happen:
- You burn through API credits instantly.
- The model hallucinates trying to transcribe traffic noise.
The Fix: I ran Silero VAD (Voice Activity Detection) as a local ONNX model directly inside my FastAPI server. It calculates the exact millisecond the user stops speaking and only flushes valid speech chunks to the Sarvam WebSocket.
import numpy as np
import onnxruntime as ort
class LocalVADProcessor:
def __init__(self, model_path: str = "silero_vad.onnx"):
self.session = ort.InferenceSession(model_path)
self.reset_states()
def is_speech(self, pcm_16k_chunk: bytes, threshold: float = 0.5) -> bool:
# Convert raw PCM bytes to normalized float32 tensor
audio_int16 = np.frombuffer(pcm_16k_chunk, dtype=np.int16)
audio_float32 = audio_int16.astype(np.float32) / 32768.0
inputs = {
'input': np.expand_dims(audio_float32, axis=0),
'sr': np.array(16000, dtype=np.int64),
'h': self._h,
'c': self._c
}
# Run local inference in <10ms
out, self._h, self._c = self.session.run(None, inputs)
return out[0][0] > threshold
Result: Shaved ~600ms off the end-to-end latency and eliminated background noise hallucinations.
đź“ž Lesson 2: Telecom Audio is Garbage (Resample on the Fly)
Modern STT models expect pristine 16kHz studio audio. But if you intercept a live phone call via Exotel or Twilio, you receive 8kHz, 16-bit PCM payloads. If you feed 8kHz audio into a 16kHz model, the AI hears a slow-motion, distorted mess.
The Fix: You must upsample the incoming stream and downsample the outgoing AI voice on the fly using Python's native audioop library.
import audioop
def process_telecom_stream(pcm_8k_bytes: bytes) -> bytes:
# 1. Upsample incoming 8kHz telecom audio to 16kHz for the AI
ai_ready_pcm, _ = audioop.ratecv(pcm_8k_bytes, 2, 1, 8000, 16000, None)
# ... [Pass to Sarvam STT -> Translate -> Sarvam TTS] ...
# 2. Downsample the generated 16kHz AI voice back to 8kHz for the phone call
phone_ready_pcm, _ = audioop.ratecv(generated_16k_pcm, 2, 1, 16000, 8000, None)
return phone_ready_pcm
🎠Lesson 3: The "Wire-Compatible" Proxy Hack
I wanted companies to use Sarvam's Indic models for their chatbots without changing their code.
Western tokenizers (like GPT-4o) fragment Indian scripts terribly, costing 3x-5x more per word. Sarvam's models cut token bloat by 80%. To make the switch frictionless, I built a FastAPI route that perfectly mimics OpenAI's /v1/chat/completions endpoint.
A developer only has to change one line of code in their existing app:
from openai import OpenAI
client = OpenAI(
# Just swap the base URL to our proxy!
base_url="https://api.patha-darshak.com/v1",
api_key="pd-tenant-123"
)
# The proxy intercepts this, formats it for Sarvam, and returns an OpenAI-shaped response
response = client.chat.completions.create(
model="sarvam-105b",
messages=[{"role": "user", "content": "Bhaiya order late kyun hai?"}]
)
🏆 The Results
By focusing on deep infrastructure constraints rather than just a flashy UI, we survived the cull from 6,000+ builders down to the final 13.
- Latency: Consistently achieved sub-2-second speech-to-speech translation.
- Economics: Demonstrated an ~80% reduction in token costs for Indic languages.
- Open Source Mentality: The core engine acts as a foundation for any developer wanting to build real-time voice bridges.
A massive shoutout to Sarvam AI and HackCulture for hosting a truly grueling, high-caliber event.
If you are building in the real-time audio space, wrestling with WebRTC, or trying to scale vernacular AI, drop a comment below. Let’s talk architecture! ⚡👇
Top comments (0)