Most AI demos you see online are either simple text chatbots or basic wrappers around an API.
I wanted to build something that felt like a real product solving a real problem. So I built Maya—a real-time voice AI receptionist for a fictional clinic in Bengaluru called SwasthyaCare Clinic.
Instead of typing into a chat box, you just talk to your microphone like you're on a real phone call. Maya answers your questions, checks doctor schedules, books appointments, reschedules or cancels existing visits, and gives clinic
information—all in real-time with sub-second voice latency.
Here is a breakdown of how the architecture works, the real issues I faced during development (rate limits, background noise, Docker container paths), and how I solved them.
1. What Maya Does
When someone calls a clinic, they don't want to navigate a robotic IVR menu ("Press 1 for appointments..."). They want to talk to a human receptionist.
Maya handles that conversational flow:
• Checks live doctor availability: Understands dates and doctor specialties (General Physician vs. Dentist).
• Gathers details naturally: Asks for missing information (name, 10-digit mobile number, preferred slot) across multiple turns.
• Explicit confirmation before booking: Summarizes the appointment details and only commits the booking once the user says "Yes".
• Enforces business rules: Only books up to 14 days in advance, checks for slot collisions, and enforces a strict 2-hour cancellation/rescheduling policy.
• Medical triage guardrails: If someone mentions severe symptoms like acute chest pain or breathing issues, Maya immediately instructs them to dial 112 or visit an emergency room rather than booking a routine outpatient slot.
2. The Architecture & Tech Stack
To make voice feel natural, latency has to be as close to human conversational speed as possible. A typical turn looks like this:
User speaks into Browser Mic
↓ (WebRTC Audio Stream)
LiveKit Cloud (ap-south / Mumbai)
↓
Silero VAD (Voice Activity Detection on-device)
↓
Groq Whisper (Speech-to-Text)
↓
Groq LLM (Reasoning + Function/Tool Calling)
↓
SQLite Database (Appointment State Engine)
↓
Cartesia TTS (Streaming Indian English Voice)
↓ (WebRTC Audio)
Browser Speakers (User hears response)
The Stack Breakdown:
• Audio Transport: LiveKit (WebRTC). Handles real-time, low-latency audio streaming between the browser and backend worker.
• VAD (Voice Activity Detection): Silero VAD. Runs locally inside the worker to detect when the user starts and stops speaking.
• STT (Speech-to-Text): Groq Whisper (whisper-large-v3-turbo). Transcribes speech into text in ~100–200ms.
• LLM Reasoning & Tool Calling: Groq (openai/gpt-oss-120b). Fast reasoning and function calling.
• TTS (Text-to-Speech): Cartesia (sonic-turbo). Uses the "Priya" voice profile—a natural, clear Indian English tone suited for a Bengaluru clinic.
• State & Database: SQLite + Python. Manages appointments, availability checks, and audit trails.
• Frontend: React + Vite with @livekit/components-react and Tailwind CSS.
3. The Real-World Engineering Problems I Hit
Building the basic happy path is easy; making real-time voice work reliably is where the real learning happened. Here are 4 specific problems I ran into:
Bug 1: The 429 Rate Limit Trap on Groq
When I first ran voice tests, everything would work for 2 or 3 turns, and then suddenly crash with an HTTP 429 Too Many Requests (Rate Limit Exceeded).
Why it happened:
On Groq's free tier, there is an 8,000 Tokens Per Minute (TPM) limit. My initial system prompt combined with the JSON schemas for 5 function tools was taking ~1,900 tokens per single LLM call. If the user spoke 4 times in a minute,
that was 1,900 * 4 = 7,600+ tokens, immediately blowing through the 8,000 token limit.
How I fixed it:
I completely compressed the system prompt and tool docstrings. I removed repetitive instructions, used dense bullet points, and kept the tool parameters minimal while preserving all clinical guardrails. This brought the request size
down from ~1,900 tokens to ~500 tokens (a 73% reduction). Suddenly, I could have 15+ turns a minute without hitting rate limits.
Bug 2: Ambient Background Noise Eating API Quota
While testing with my laptop mic, I noticed the agent would randomly trigger and start speaking even when I hadn't said anything.
Why it happened:
Default VAD sensitivity was picking up subtle background sounds—fan noise, keyboard typing, and breathing. Each micro-sound triggered LiveKit's turn detector, which immediately dispatched an STT call, an LLM call, and a TTS synthesis
call. This was burning API quota and interrupting the conversation.
How I fixed it:
I tuned the Silero VAD parameters and turn handling in LiveKit:
vad_instance = silero.VAD.load(
min_speech_duration=0.25, # Ignore clicks and breath sounds under 250ms
min_silence_duration=0.65, # Wait for a clean pause before marking end-of-turn
prefix_padding_duration=0.3,
activation_threshold=0.6, # Require clearer vocal energy over ambient noise
)
session = AgentSession(
turn_handling={
"endpointing": {"min_delay": 0.6, "max_delay": 3.0},
"preemptive_generation": {"enabled": False}, # Only generate audio when speech is complete
"interruption": {"enabled": True, "min_duration": 0.5},
}
)
Disabling preemptive generation and requiring at least 250ms of vocal energy eliminated the false triggers completely.
Bug 3: Finding the Right Voice (US Accent vs. Indian Accent)
Initially, I used Groq's built-in TTS. While it worked, it only had US and Arabic voice profiles, and the free-tier rate limits were strict for voice generation.
For a clinic located in HSR Layout, Bengaluru, a North American voice felt out of place. I integrated Cartesia's sonic-turbo model with their Priya voice (an Indian English female voice profile).
The difference was night and day:
• Latency dropped below 100ms.
• The cadence and pronunciation of Indian names sounded authentic.
Bug 4: Deploying Voice AI is Not Like Deploying a Chatbot
When I tried deploying the Python backend, I quickly realized you can't just throw a voice agent onto serverless platforms like Vercel or AWS Lambda.
A text chatbot handles a quick HTTP request and terminates in 1 second. A voice agent, on the other hand, is a persistent WebRTC worker daemon. It maintains an active bidirectional audio socket to LiveKit Cloud 24/7.
The Solution:
• React Frontend: Deployed on Vercel with a serverless token endpoint (/api/token) that generates short-lived LiveKit JWT access tokens without exposing LIVEKIT_API_SECRET to the browser.
• Python Agent Worker: Containerized via Docker and deployed to LiveKit Cloud Agent Hosting in the ap-south (Mumbai) region for ultra-low ping.
When deploying the Docker container, I ran into a ModuleNotFoundError: No module named 'agent.prompts'. The container entrypoint was running python agent/agent.py start, which put /app/agent into Python's sys.path instead of the root
/app. I fixed this by adding ENV PYTHONPATH="/app" in the Dockerfile and adding a defensive path resolution in Python.
4. Key Takeaways from Building This
- Latency is king in voice: A 1.5-second pause in a text chat is fine. In a phone call, a 1.5-second pause feels awkward and broken. Streaming audio chunks and optimizing model TTFT (Time-to-First-Token) is critical.
- Prompt token economy matters: In voice agents, every single spoken phrase sends the entire context back to the LLM. Keeping prompts concise directly prevents rate limit crashes and saves money.
- Noise suppression isn't optional: Real users don't sit in soundproof recording studios. Tuning VAD thresholds is just as important as choosing the LLM.
5. What's Next & Looking for Feedback
This project gave me a massive appreciation for what it takes to build reliable real-time AI systems.
I'd love to connect with other engineers and builders working in Voice AI, WebRTC, and LLM tool calling:
• How are you handling ambient noise and interruption handling in your voice agents?
• What TTS providers have you found best for regional accents?
Check out the code and feel free to share your thoughts or suggestions!
🔗 GitHub Repo: https://github.com/CosmosTechy/maya-ai-voice-receptionist
🌐 Live Demo: https://maya-ai-voice-receptionist.vercel.app/
Top comments (0)