DEV Community: DanitZ

Building Vidya: An Ultra-Fast Bilingual Voice AI Tutor with Murf Falcon & LiveKit (10 Days of Voice Agents)

DanitZ — Sat, 15 Aug 2026 10:04:02 +0000

Building Vidya: An Ultra-Fast Bilingual Voice AI Tutor.

Over the past 10 days, as part of the #VoiceForBharat 10 Days of Voice Agents Challenge, I built Vidya — a real-time, bilingual (English/Hindi) AI Voice Tutor designed to make learning interactive, accessible, and human-like for students across India.

In this post, I’ll share the story of how Vidya came to life, dive into the architecture behind ultra-low latency voice agents, highlight the key features built over the 10 days, discuss the toughest challenges faced, and walk you through building your own production-ready voice agent using Murf Falcon TTS, LiveKit Agents, Deepgram STT, and Google Gemini.

1. Why Voice? Meet Vidya 🇮🇳

In India, text-based educational platforms often face a steep digital literacy and language barrier. Millions of learners feel intimidated by typing long queries or struggling through English-only user interfaces. Voice unlocks immediate, natural, and hands-free learning — allowing students to speak naturally in English, Hindi, or code-mixed Hinglish.

Vidya serves as a personal AI learning companion:

Problem Solved: Overcomes literacy and language barriers by providing empathetic, spoken literacy and science lessons.
Track & Target Audience: Education & Accessibility for students, young learners, and non-native English speakers across India.
Why Voice is Essential: Voice removes UI friction. Learners can simply talk to Vidya, attempt spoken exercises, ask questions, and receive instant, warm, spoken feedback in a natural Indian accent.

2. Key Features Built Across the 10-Day Journey

Over 10 intensive days, Vidya grew from a basic echo bot into a multi-agent system equipped with tools, memory, telephony, analytics, and safety guardrails:

🎙️ 1. Ultra-Low Latency Indian Voice Powered by Murf Falcon

Using Murf Falcon TTS (livekit-murf), Vidya speaks with a natural, conversational Indian voice (Anisha). Streaming TTS with sentence tokenization (min_sentence_len=2) and text pacing delivers speech chunks with sub-second latency, giving the agent a human-like flow.

🇮🇳 2. Code-Mixed (Hinglish) & Bilingual Support

Powered by Deepgram Nova-3 STT (language="multi") and Google Gemini, Vidya fluently handles English, Hindi in Devanagari script, and code-mixed Hinglish phrases (e.g., "Namaste! Aaj hum beginner reading practice karenge.").

🧠 3. Personalized Memory for Returning Users

Vidya remembers returning students! Using a persistent profile store (user_store.py), Vidya recalls the user's name, preferred language, current learning level, and last interaction date, greeting them warmly:

"Namaste Aarav, welcome back! You were working on beginner exercises. Last seen on August 14."

🛠️ 4. Interactive Learning Tools & Scraper

Vidya is equipped with specialized function tools:

fetch_next_exercise: Retrieves level-appropriate practice prompts tagged with data freshness timestamps (last_updated).
score_spoken_answer: Evaluates spoken pronunciations and answers on a 0–100 scale.
award_learning_star: Awards virtual gold stars (🌟) to keep learners motivated.
scrape_website: Fetches live web pages in real-time (web_scraper.py) for live context extraction.

📞 5. Outbound Telephony (SIP Calls)

Integrated with LiveKit's SIP Trunking (telephony/outbound/dial.py), Vidya can initiate active outbound phone calls to learners' mobile phones for daily study check-ins and practice sessions.

🛡️ 6. Human Escalation Engine with Explicit Consent

If a learner is stuck, frustrated, or requests a human teacher, create_escalation logs an escalation ticket (ESC-12345) and alerts support staff. Vidya follows strict privacy guardrails — asking for explicit permission before saving any personal details or submitting tickets.

📊 7. Call Analytics Dashboard

Session outcomes are tracked in call_store.py — logging call duration, agent type (browser vs. telephony), completion status (successful / failed), and success reasons (exercise_scored, star_awarded, escalated_to_human).

🔄 8. Dynamic Multi-Agent Handoffs

When a student asks a physics question (e.g., "Why does an apple fall from a tree?"), Vidya seamlessly hands off the conversation to Dr. Homi (Physics Specialist) using LiveKit's context.session.update_agent(). When physics practice ends, Dr. Homi hands the student back to Vidya for reading practice!

3. The Tough Parts: Lessons Learned

Building a real-time voice agent isn't just about linking APIs together. Here are three major hurdles faced and solved:

Challenge #1: Eliminating Awkward Conversational Latency

The Problem: Combining STT, LLM, and TTS in sequence created a 2–3 second delay before the agent responded, ruining natural turn-taking.
The Solution: We enabled streaming at every tier — Deepgram Nova-3 streaming STT, Gemini Flash Lite for fast first-token generation, LiveKit's preemptive generation, and Murf Falcon's low-latency streaming TTS with sentence boundary tokenization (min_sentence_len=2).

Challenge #2: Explicit Consent & Privacy in Spoken AI

The Problem: Function tools automatically firing to record user profiles or create tickets without the user realizing.
The Solution: We designed prompt-level guardrails requiring Vidya to explicitly ask out loud: "I can remember that for next time — may I save this?" or "May I share your details with a real teacher?" before invoking mutation tools.

Challenge #3: Preserving Session State During Agent Handoffs

The Problem: Switching from Vidya to Dr. Homi mid-call could cause audio glitches or lose participant state.
The Solution: Leveraged LiveKit Agents SDK's native session.update_agent() combined with WebRTC data channel events (agent_handoff) to update the Next.js frontend UI live without dropping the WebRTC room session.

4. Architecture Overview & Code Highlights

+------------------+      WebRTC Audio Stream     +---------------------+
|                  |  ------------------------->  |  Deepgram Nova-3    |
|   Learner / UI   |                              |  Streaming STT      |
|  (Next.js App)   |  <-------------------------  +----------+----------+
+--------+---------+      Real-time Audio Out                |
         ^                                                   v
         | RTC Data Channel                       +---------------------+
         | (State & Handoffs)                     |  Google Gemini LLM  |
         |                                        | (Flash Lite Model)  |
         +--------------------------------------  +----------+----------+
                                                             |
                                                             v
                                                  +---------------------+
                                                  |  Murf Falcon TTS    |
                                                  | (Streaming Indian)  |
                                                  +---------------------+

Code Snippet: LiveKit Pipeline with Murf Falcon TTS (`agent.py`)

from livekit.agents import AgentSession, AgentServer, room_io
from livekit.plugins import deepgram, google, murf, silero, noise_cancellation

session = AgentSession(
    stt=deepgram.STT(model="nova-3", language="multi"),
    llm=google.LLM(model="gemini-3.5-flash-lite"),
    tts=murf.TTS(
        voice="Anisha",          # Murf Falcon Indian accent voice
        style="Conversation",
        tokenizer=tokenize.basic.SentenceTokenizer(min_sentence_len=2),
        text_pacing=True,
    ),
    turn_detection=MultilingualModel(),
    vad=ctx.proc.userdata["vad"],
    preemptive_generation=True,
)

Code Snippet: Dynamic Multi-Agent Handoff (`agent.py`)

@function_tool
async def transfer_to_physics_specialist(self, context: RunContext, reason: str) -> str:
    """Hand off the conversation to Dr. Homi when user asks physics questions."""
    logger.info("Handing off conversation to PhysicsSpecialist. Reason: %s", reason)
    specialist = PhysicsSpecialist()
    context.session.update_agent(specialist)

    # Notify Next.js frontend UI via WebRTC data channel
    payload = json.dumps({
        "type": "agent_handoff",
        "from_agent": "Vidya (Literacy Tutor)",
        "to_agent": "Dr. Homi (Physics Specialist)",
        "message": "🔄 Switched conversation to Physics Specialist (Dr. Homi)"
    })
    await context.room.local_participant.publish_data(payload=payload.encode("utf-8"))

    return "I will connect you to our physics specialist."

5. How to Build Your Own Voice Agent (Quickstart)

Want to build your own voice AI agent? You can clone and run our open-source repository in minutes!

Step 1: Clone the Repository

git clone https://github.com/hotokeAtlast/murf-livekit-starter.git
cd murf-livekit-starter

Step 2: Set Up Backend Environment Keys

Copy backend/.env.example to backend/.env.local and fill in your keys:

LIVEKIT_URL=wss://your-livekit-project.livekit.cloud
LIVEKIT_API_KEY=your_key
LIVEKIT_API_SECRET=your_secret
MURF_API_KEY=your_murf_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_google_gemini_api_key

Step 3: Run the Python Backend

cd backend
uv sync
uv run python src/agent.py dev

Step 4: Run the Next.js Frontend

In a new terminal:

cd frontend
pnpm install
pnpm dev

Open http://localhost:3000, click Connect, and start talking to your voice agent!

6. Links & Repository

🐙 GitHub Repository: murf-livekit-starter on GitHub
🔊 TTS Engine: Powered by Murf Falcon Streaming API
⚡ Voice Framework: Built with LiveKit Agents SDK

🌀 Vortex: From a Firebase Toy to a Real Multi-Vendor Marketplace (Finished, Finally)

DanitZ — Fri, 05 Jun 2026 11:48:29 +0000

Post body

GitHub "Finish-Up-A-Thon" Challenge Submission

This is a submission for the GitHub Finish-Up-A-Thon Challenge.

What I Built

Vortex is a full-stack, multi-vendor e-commerce platform — a "mystical archive" of products from independent sellers, with an admin moderation layer, role-based access, and a real Razorpay checkout.

It's a React 19 + Vite 8 + Tailwind 4 frontend talking to Firebase (Auth + Firestore) for data and a small Express 5 backend for payments. It started life as a sleepy single-admin Firebase CRUD app and grew into something I'd actually feel comfortable showing a recruiter.

🔗 Live: https://vortex-n60k.onrender.com/
🔗 Repository: https://github.com/hotokeAtlast/Vortex
🔗 Backend health check: GET /api/health → { "status": "active", "message": "The Vortex is awake." }

What's actually in the box

Three-tier role system — Buyer, Seller, Admin — with a real users/{uid} profile document, an isAdmin() claim derived from email, and an isActiveSeller() lookup helper used in Firestore rules.
Seller Portal — full product CRUD, stock tracking, image upload to Cloudinary, gated by an "active" sellerStatus so pending sellers can't dump products into the marketplace.
Admin Moderation Panel — a single dashboard with stats (pending sellers, review queue, active listings), approve/reject flows for both sellers and products, and content deletion.
Razorpay checkout with server-side verification — /api/create-order issues the order, /api/verify-payment re-derives the HMAC-SHA256 signature on the Express side and only marks the Firestore order as Paid when the signatures match.
114 lines of Firestore security rules — seller-owned product enforcement, admin moderation authority, public read for approved products only, user privacy, and order history isolation.
UI/UX polish — en-IN currency formatting (₹1,23,456), proper empty states with CTAs, sticky order summary, dark/light theme, mobile-first responsive grid, scroll-to-top, toast notifications.
Single-service deploy — the same Express process serves the Vite-built dist/ and the /api/* routes, so one Render Web Service runs the whole thing.

Stack at a glance

Layer	Tech
Frontend	React 19, Vite 8, Tailwind 4, React Router 7, FontAwesome
Data	Firebase Auth, Cloud Firestore, Cloudinary (image upload)
Payments	Razorpay (orders + HMAC signature verification)
Backend	Node 18+, Express 5, CORS, dotenv
Tooling	ESLint 9, Vite Rolldown, concurrently
Deploy	Render (single Web Service hosts the whole monorepo)

Demo

🔗 Live: https://vortex-n60k.onrender.com/
🔗 Repo: https://github.com/hotokeAtlast/Vortex

Try the three roles in order:

Buyer — sign up with a normal email, browse, add to cart, hit the Razorpay test checkout (use the test card 4111 1111 1111 1111).
Seller — sign up, pick "Seller" during registration, submit a couple of products. They show up in your SellerPortal immediately as pending.
Admin — sign in with the admin email configured in firestore.rules (hotoke.atlast@gmail.com) and use the moderation panel to approve your own products.

Repo quick links

server/server.cjs — Express API + static frontend host
firestore.rules — role-based access enforcement
src/pages/SellerPortal.jsx — seller product management
src/pages/Admin.jsx — moderation panel
src/pages/Checkout.jsx — Razorpay integration
render.yaml — single-service deploy config

The Comeback Story

Where it started

A few months ago, Vortex was a Vite + Firebase project with a single hardcoded admin who could add, edit, and delete products. That was the whole feature set. One user role ("user or the one specific admin email"), a half-finished cart, a checkout that didn't actually charge anyone, no seller concept, no moderation, and a basic Tailwind UI I'd gotten tired of looking at.

I left it because the obvious next step — letting other people post products — opened a can of worms. Sellers need their own role, their own dashboard, their own approval flow, and the database needs rules that keep one seller from editing another seller's product. Admin needs a moderation queue. Payments need a server. Deploy needs to actually work. None of it was a Saturday afternoon.

So the project sat. Hosted, but rotting.

What I changed

Area	Before	After
Roles	`user` vs `admin` (one hardcoded email)	`buyer` / `seller` / `admin` with `sellerStatus: pending\
Marketplace	Admin posts products directly	Sellers apply → admin approves → sellers post → admin approves → buyers see it
Auth flow	Sign up → you're in	Sign up → pick a role → sellers land in {% raw %}`pending` state, gated by rules
Checkout	"Place order" button that did nothing	Razorpay test mode + `/api/create-order` + HMAC-SHA256 verification on `/api/verify-payment`
Data integrity	No server, all rules imagined	114 lines of `firestore.rules` with helper functions, role checks, and ownership
UI	Light theme, no real empty states, broken currency	Dark-first theme toggle, ₹ formatting with `en-IN` locale, proper loading + empty states
Deploy	Frontend on Vercel, backend half-deployed to Render	Single Render Web Service: Vite build → Express serves both `dist/` and `/api/*`

The deploy line is the one I'm most embarrassed about, because the original render.yaml was technically there but completely broken — it had startCommand: npm start && npm run dev and there was no start script in the root. The build never even reached the server startup phase. I just kept ignoring the failed deploy logs.

The finishing touches (this session)

When I finally decided to "finish" Vortex for this challenge, I expected to spend an evening polishing the UI. Instead I spent half of it chasing a deployment that wouldn't boot:

render.yaml had a typo that crashed every deploy — startCommand: npm start && npm run dev because I never wrote a start script. Fixed with npm start + a real start entry that resolves the right entry point.
ESM-vs-CommonJS collision — the root package.json was "type": "module", but the server file used require(). Renamed server/server.js → server/server.cjs so it stays CommonJS regardless of the parent's type, and updated the root start command to point at it.
Build was missing devDependencies — Vite and @vitejs/plugin-react lived in devDependencies, but Render's NODE_ENV=production made npm install skip them. Added --include=dev to the install command and NPM_CONFIG_PRODUCTION=false to the env vars as a belt-and-braces fix.
Razorpay would crash the process if env vars were missing — moved instantiation to lazy, added a startup warning and a 503 response when keys aren't configured. Now a fresh Render deploy without Razorpay secrets still boots cleanly.
/api/health was the only thing that worked — refactored the Express server to also serve the built Vite dist/ folder and act as an SPA catch-all (regex route that returns index.html for any non-/api/* path) so a single Render service hosts the whole thing.
Frontend checkout was hardcoded to someone else's Render URL — https://vortex-api-6wk1.onrender.com/api/... from an earlier experiment. Replaced with a VITE_API_BASE_URL env var, empty by default, so same-origin requests just work.
The page wouldn't scroll — leftover Vite starter CSS had #root { width: 1126px; min-height: 100svh; display: flex; flex-direction: column; } which broke the scroll chain. Replaced it with a 4-line reset (html, body, #root { height: 100%; } body { margin: 0; }) and let Tailwind handle the rest.

After that, npm run build && npm start boots in 3 seconds, the health check pings green, and the Vite bundle is served from the same Express process. The site finally works in production — which is, in retrospect, the most basic requirement of "finished."

My Experience with GitHub Copilot

Copilot was a co-author on basically every part of this project — but the way I used it changed shape across the two phases.

Phase 1: shipping features (the original sessions)

I used Copilot heavily in agent mode to design the role-based auth flow. The thing I kept getting wrong on my own was the invariants of the role transition: a new user should land as buyer, a buyer should be able to apply to become a seller (which sets role: 'seller' and sellerStatus: 'pending' simultaneously), but a pending seller should not be able to flip themselves to active and start posting. That's a tiny state machine with a security guarantee attached to it, and I was over-engineering it.

What worked was describing the state machine to Copilot in plain English and asking it to write the Firestore rule helpers first, then the React context code second. The helpers — isAdmin(), isSelf(), isActiveSeller(), userRequestsSellerRoleChange() — came out almost unchanged from its first draft, and they made the rest of the system obviously correct. I wasn't learning "how to use Firestore rules" — I was learning "the shape of a security-relevant state machine" by watching one get drawn.

Same story on the seller portal UI. I had a vague "sellers should be able to manage their own products" idea. Copilot turned that into a <SellerPortal /> page with a form, an inventory list, an image upload to Cloudinary, and a "pending review" badge — all in a single iteration. I edited maybe 20% of it (mostly the date/time formatting and the stock number input), but the shape was right on the first pass.

On the Razorpay side, the lesson was: the integration is short, but the trust boundary is the part you have to be careful about. The whole "create order" endpoint is 20 lines. The verify endpoint is 30. The reason I shipped it confidently is that Copilot and I went over the signature verification together: it wrote the crypto.createHmac('sha256', key).update(sign).digest('hex') snippet, I asked "what if the order id is missing," and it added the input check. That's not autocomplete — that's a code review partner.

Phase 2: finishing the deploy (this week)

The deployment debugging was where Copilot's value flipped. Most of this work was not writing new code — it was reading logs, forming hypotheses, and making tiny targeted changes. Copilot was great at the "here's a thing I broke, explain why" loop:

"Why would npm start && npm run dev fail on Render?" → It explained the && short-circuit and the missing start script in one sentence, then suggested the right start script body. I would've eventually figured this out from the deploy log, but Copilot got me there in seconds.
"What does Error [ERR_MODULE_NOT_FOUND]: Cannot find package '@vitejs/plugin-react' mean in this context?" → Instantly: "Vite is in devDependencies and Render's NODE_ENV=production is hiding them. Use --include=dev or set NPM_CONFIG_PRODUCTION=false." That was the exact fix I needed.
"Show me an Express middleware that serves dist/ for a Vite SPA and falls back to index.html for any non-/api/* route." → Drop-in snippet, I pasted it, it worked, I moved on.

The thing I appreciated most was the discipline to not over-use it. When the page wouldn't scroll and I suspected the leftover #root { width: 1126px; } from the Vite starter, I could've asked Copilot to rewrite the whole stylesheet. Instead I asked "is this specific CSS rule the problem?" — Copilot said yes, I deleted 100 lines and wrote 4, and the scrollbar came back. Sometimes the right Copilot prompt is the short one.

What I still pushed back on

Copilot's first draft of the Firestore create rule on users/{userId} was permissive about role changes. I tightened it: only admins can set role: 'admin', and the seller-role upgrade has to come with sellerStatus: 'pending'. Verified by writing a failing test case (a non-admin trying to set role: 'admin') and watching the rule correctly reject it.
The first version of the Express static-file handler was app.get('*', ...) which would have intercepted /api/* requests too. I caught it and switched to the ^(?!api/).* regex.
I didn't let Copilot touch firestore.rules "for me" — security rules are the one place I want my name on every line. It helped me think about the rule shape, but I wrote and tested them by hand against the Firebase emulator.

What's next

Vortex is in a state I'm actually happy with. Things I might still do, in priority order:

Replace the hardcoded admin email in isAdmin() with a real Firebase Auth custom claim, so admin status is provable and revocable from the Firebase console.
Add a "withdraw product" flow so sellers can delist items without admin intervention.
Wire up a proper email receipt via the existing SMTP path.
Add Playwright e2e tests for the three role flows (buyer checkout, seller CRUD, admin moderation).

None of those are blockers for calling this finished. The thing this challenge gave me was the deadline pressure to actually ship what I had, instead of letting the next-feature itch keep the project in permanent limbo.

Thanks for reading — and if you've got an abandoned side project, this is your sign: open the deploy logs, fix the typo, push the button, and call it done.