How I Built Maya: A Real-Time Voice AI Clinic Receptionist

CosmosTechy — Fri, 04 Sep 2026 08:12:17 +0000

Most AI demos you see online are either simple text chatbots or basic wrappers around an API.

I wanted to build something that felt like a real product solving a real problem. So I built Maya—a real-time voice AI receptionist for a fictional clinic in Bengaluru called SwasthyaCare Clinic.

Instead of typing into a chat box, you just talk to your microphone like you're on a real phone call. Maya answers your questions, checks doctor schedules, books appointments, reschedules or cancels existing visits, and gives clinic
information—all in real-time with sub-second voice latency.

Here is a breakdown of how the architecture works, the real issues I faced during development (rate limits, background noise, Docker container paths), and how I solved them.

1. What Maya Does

When someone calls a clinic, they don't want to navigate a robotic IVR menu ("Press 1 for appointments..."). They want to talk to a human receptionist.
Maya handles that conversational flow:

• Checks live doctor availability: Understands dates and doctor specialties (General Physician vs. Dentist).
• Gathers details naturally: Asks for missing information (name, 10-digit mobile number, preferred slot) across multiple turns.
• Explicit confirmation before booking: Summarizes the appointment details and only commits the booking once the user says "Yes".
• Enforces business rules: Only books up to 14 days in advance, checks for slot collisions, and enforces a strict 2-hour cancellation/rescheduling policy.
• Medical triage guardrails: If someone mentions severe symptoms like acute chest pain or breathing issues, Maya immediately instructs them to dial 112 or visit an emergency room rather than booking a routine outpatient slot.

2. The Architecture & Tech Stack

To make voice feel natural, latency has to be as close to human conversational speed as possible. A typical turn looks like this:

User speaks into Browser Mic
       ↓ (WebRTC Audio Stream)
LiveKit Cloud (ap-south / Mumbai)
       ↓
Silero VAD (Voice Activity Detection on-device)
       ↓
Groq Whisper (Speech-to-Text)
       ↓
Groq LLM (Reasoning + Function/Tool Calling)
       ↓
SQLite Database (Appointment State Engine)
       ↓
Cartesia TTS (Streaming Indian English Voice)
       ↓ (WebRTC Audio)
Browser Speakers (User hears response)

The Stack Breakdown:

• Audio Transport: LiveKit (WebRTC). Handles real-time, low-latency audio streaming between the browser and backend worker.
• VAD (Voice Activity Detection): Silero VAD. Runs locally inside the worker to detect when the user starts and stops speaking.
• STT (Speech-to-Text): Groq Whisper (whisper-large-v3-turbo). Transcribes speech into text in ~100–200ms.
• LLM Reasoning & Tool Calling: Groq (openai/gpt-oss-120b). Fast reasoning and function calling.
• TTS (Text-to-Speech): Cartesia (sonic-turbo). Uses the "Priya" voice profile—a natural, clear Indian English tone suited for a Bengaluru clinic.
• State & Database: SQLite + Python. Manages appointments, availability checks, and audit trails.
• Frontend: React + Vite with @livekit/components-react and Tailwind CSS.

3. The Real-World Engineering Problems I Hit

Building the basic happy path is easy; making real-time voice work reliably is where the real learning happened. Here are 4 specific problems I ran into:

Bug 1: The 429 Rate Limit Trap on Groq

When I first ran voice tests, everything would work for 2 or 3 turns, and then suddenly crash with an HTTP 429 Too Many Requests (Rate Limit Exceeded).

Why it happened:
On Groq's free tier, there is an 8,000 Tokens Per Minute (TPM) limit. My initial system prompt combined with the JSON schemas for 5 function tools was taking ~1,900 tokens per single LLM call. If the user spoke 4 times in a minute,
that was 1,900 * 4 = 7,600+ tokens, immediately blowing through the 8,000 token limit.

How I fixed it:
I completely compressed the system prompt and tool docstrings. I removed repetitive instructions, used dense bullet points, and kept the tool parameters minimal while preserving all clinical guardrails. This brought the request size
down from ~1,900 tokens to ~500 tokens (a 73% reduction). Suddenly, I could have 15+ turns a minute without hitting rate limits.

Bug 2: Ambient Background Noise Eating API Quota

While testing with my laptop mic, I noticed the agent would randomly trigger and start speaking even when I hadn't said anything.

Why it happened:
Default VAD sensitivity was picking up subtle background sounds—fan noise, keyboard typing, and breathing. Each micro-sound triggered LiveKit's turn detector, which immediately dispatched an STT call, an LLM call, and a TTS synthesis
call. This was burning API quota and interrupting the conversation.

How I fixed it:
I tuned the Silero VAD parameters and turn handling in LiveKit:

vad_instance = silero.VAD.load(
    min_speech_duration=0.25,   # Ignore clicks and breath sounds under 250ms
    min_silence_duration=0.65,  # Wait for a clean pause before marking end-of-turn
    prefix_padding_duration=0.3,
    activation_threshold=0.6,   # Require clearer vocal energy over ambient noise
)

session = AgentSession(
    turn_handling={
        "endpointing": {"min_delay": 0.6, "max_delay": 3.0},
        "preemptive_generation": {"enabled": False},  # Only generate audio when speech is complete
        "interruption": {"enabled": True, "min_duration": 0.5},
    }
)

Disabling preemptive generation and requiring at least 250ms of vocal energy eliminated the false triggers completely.

Bug 3: Finding the Right Voice (US Accent vs. Indian Accent)

Initially, I used Groq's built-in TTS. While it worked, it only had US and Arabic voice profiles, and the free-tier rate limits were strict for voice generation.

For a clinic located in HSR Layout, Bengaluru, a North American voice felt out of place. I integrated Cartesia's sonic-turbo model with their Priya voice (an Indian English female voice profile).

The difference was night and day:

• Latency dropped below 100ms.
• The cadence and pronunciation of Indian names sounded authentic.

Bug 4: Deploying Voice AI is Not Like Deploying a Chatbot

When I tried deploying the Python backend, I quickly realized you can't just throw a voice agent onto serverless platforms like Vercel or AWS Lambda.

A text chatbot handles a quick HTTP request and terminates in 1 second. A voice agent, on the other hand, is a persistent WebRTC worker daemon. It maintains an active bidirectional audio socket to LiveKit Cloud 24/7.

The Solution:

• React Frontend: Deployed on Vercel with a serverless token endpoint (/api/token) that generates short-lived LiveKit JWT access tokens without exposing LIVEKIT_API_SECRET to the browser.
• Python Agent Worker: Containerized via Docker and deployed to LiveKit Cloud Agent Hosting in the ap-south (Mumbai) region for ultra-low ping.

When deploying the Docker container, I ran into a ModuleNotFoundError: No module named 'agent.prompts'. The container entrypoint was running python agent/agent.py start, which put /app/agent into Python's sys.path instead of the root
/app. I fixed this by adding ENV PYTHONPATH="/app" in the Dockerfile and adding a defensive path resolution in Python.

4. Key Takeaways from Building This

Latency is king in voice: A 1.5-second pause in a text chat is fine. In a phone call, a 1.5-second pause feels awkward and broken. Streaming audio chunks and optimizing model TTFT (Time-to-First-Token) is critical.
Prompt token economy matters: In voice agents, every single spoken phrase sends the entire context back to the LLM. Keeping prompts concise directly prevents rate limit crashes and saves money.
Noise suppression isn't optional: Real users don't sit in soundproof recording studios. Tuning VAD thresholds is just as important as choosing the LLM.

5. What's Next & Looking for Feedback

This project gave me a massive appreciation for what it takes to build reliable real-time AI systems.

I'd love to connect with other engineers and builders working in Voice AI, WebRTC, and LLM tool calling:

• How are you handling ambient noise and interruption handling in your voice agents?
• What TTS providers have you found best for regional accents?

Check out the code and feel free to share your thoughts or suggestions!

🔗 GitHub Repo: https://github.com/CosmosTechy/maya-ai-voice-receptionist
🌐 Live Demo: https://maya-ai-voice-receptionist.vercel.app/

DEV Community: CosmosTechy