DEV Community

Suyash Sahu
Suyash Sahu

Posted on

Sydney: Building a Hinglish AI/ML Mentor That Actually Talks — 10 Days of Voice Agents

#ai #llm #agents #machinelearning

One day seating in the on the

TL;DR

Over 10 days, as part of Murf AI's 10 Days of Voice Agents — Voice for Bharat Challenge 2026, I built Sydney — a Hinglish-speaking AI/ML learning companion that teaches concepts like RAG, backpropagation, embeddings, and agent architectures entirely through voice conversation. She remembers returning learners, fetches practice exercises, makes outbound calls, escalates to a human when a learner is genuinely struggling, tracks her own success rate on a live dashboard, and hands off to specialist agents when a question needs deeper expertise.

This post covers what she does, the features that mattered most, the bugs that actually broke things (not the polished version — the real ones), and enough practical detail for you to build your own.

Repo: github.com/suyashsahu00/10-days-of-AI-voice-agents-voice-for-bharat-challenge-2026

Built with: LiveKit Agents, Murf Falcon TTS, Deepgram STT, Google Gemini, and a Next.js frontend.

1. The problem and the users

Most AI/ML learning material is written for people who already read fluently in technical English. But a huge number of learners in India think and explain concepts to themselves in Hinglish — a natural mix of Hindi and English — even when the underlying material (RAG, backprop, vector databases) is inherently English-first, jargon-heavy, and unforgiving to skim.

Text-based tutorials also have a structural problem for concept-learning specifically: they let you feel like you understood something without ever making you explain it back. A voice conversation doesn't have that escape hatch — if you can't explain backprop out loud in your own words, the gap in understanding shows up immediately, in real time, to both of you.

Sydney is built for that gap: learners — students, early-career developers, self-taught engineers — who want to talk through ML/AI concepts the way they'd talk to a patient senior engineer, in whatever mix of Hindi and English feels natural, without switching mental gears to formal technical writing.

2. What the voice agent does

Sydney is a voice-only AI/ML mentor with:

  • A live wave-visualizer frontend (Next.js + custom HTML5 canvas) that shows connection state — ready, connecting, listening, speaking — so the interaction feels present, not like talking into a void
  • Persistent memory of returning learners — name, language preference, topics covered, common mistakes
  • A practice exercise tool that fetches level-appropriate questions across RAG, backprop, embeddings, LangGraph, and chunking
  • Outbound calling — Sydney can call a learner for a scheduled daily practice session over a real phone line
  • Human escalation — if a learner is genuinely distressed or explicitly asks for a human teacher, Sydney asks permission, then creates a real, tracked request for a mentor
  • A call analytics dashboard tracking total, successful, and failed sessions, using actual interaction signal rather than a timer
  • Specialist handoff — a RAG Deep-Dive Specialist and an Exam & Interview Prep Specialist that Sydney transfers learners to when a question goes beyond her general-mentor scope

3. How the system works

At the core, every voice agent — Sydney included — is four pieces wired together in a loop:

Your voice → STT (speech-to-text) → LLM (reasoning) → TTS (text-to-speech) → Your ears
                                          ↓
                                    Tools / DB / APIs
Enter fullscreen mode Exit fullscreen mode

Speech-to-text (STT): Deepgram's nova-3 model, running in multi-language mode, so it picks up Hindi and English in the same sentence without needing to be told which language is coming.

LLM: Google's gemini-3.5-flash-lite — fast enough to keep the conversation from feeling laggy, which matters more in voice than in chat, since silence reads as the agent being broken rather than "still typing."

Text-to-speech (TTS): Murf Falcon, using the "Anisha" voice in Conversation style. Falcon is genuinely fast — that speed is what makes a Hinglish tutoring session feel like a conversation instead of a slow request-response loop.

Real-time transport: LiveKit Agents handles the WebRTC session, turn detection (MultilingualModel, since fixed-language turn detectors misfire on code-switched speech), and voice activity detection (Silero VAD).

Tools: Python function tools the LLM calls itself — memory lookup/save, exercise fetch, escalation creation, specialist handoff — each with a careful docstring, since the model decides when to call a tool based on that description alone.

4. The most important features — and why they exist

Memory that actually persists (not just a database that works)

Adding a SQLite table for caller memory was the easy part. The hard part, which I didn't see coming, was identity — specifically, how does the backend know two separate calls are the same person?

Over voice, there's no login. I generate a UUID in the browser, persist it in localStorage, and pass it through the token request so it becomes the LiveKit participant identity the backend reads. That's the mechanism. Section 5 covers what actually went wrong building this.

Exercises with a real failure path

The exercise-fetch tool pulls from a hand-curated local dataset (no public API exists for "next AI/ML exercise by learner level," and the README says so honestly). The part that mattered wasn't the JSON file — it was designing what Sydney says when the file is missing or corrupted. A voice agent going silent, or inventing a fake question, is worse than a real API failure, because there's no error banner to blame — just a person talking to nothing.

Escalation gated at two layers, not one

When a learner is distressed or asks for a human, Sydney doesn't just decide to escalate — she states what she wants to share, asks for explicit permission, and only then calls create_escalation, which posts a structured, PII-scrubbed summary to a Discord webhook with a timestamped reference ID. The consent check lives in both the system prompt and is required before the tool does anything useful — because a prompt instruction alone is something a model can be talked around, especially under the exact emotional pressure that triggers escalation in the first place.

A dashboard that measures the right thing

"Successful call" for Sydney means the learner actually engaged with a practice exercise — not that a session lasted longer than some arbitrary number of seconds. Getting this definition right took an actual failed attempt first (see Section 5).

Specialist handoff with real context carry-over

When a learner wants to go deep on RAG architecture (chunking trade-offs, vector DB choice, hybrid search) or wants mock interview-style quizzing instead of teaching, Sydney hands off to a dedicated specialist agent — with the prior conversation passed along via LiveKit's chat_ctx.copy(exclude_instructions=True), so the learner never has to re-explain what they already asked.

5. The difficult parts — real bugs, not the highlight reel

Bug 1: The identity mismatch that made memory pointless

I built the SQLite memory layer, the lookup/save tools, the consent-gated prompt — all correct in isolation. Then I tested it: call once, hang up, call again. Sydney treated me as a stranger every time.

The cause, once I actually traced it end to end: the frontend was sending participant_identity (snake_case) in the token request body, and my backend endpoint was reading body.participantIdentity (camelCase). Two different field names, silently mismatched — no error, just a random fallback identity generated on every single call. The database was working perfectly. The identity feeding into it never was.

The fix was one line. Finding it required logging the actual request body and reading it character by character, not assuming the code "should" be working because it compiled and ran without errors.

Lesson: a feature that runs without throwing an exception is not the same as a feature that does what you designed it to do. Trace the actual data end to end, especially across a frontend/backend boundary.

Bug 2: A fallback string is a suggestion, not a rule

For the exercise tool's failure path, I wrote a clear instruction: if the data file is missing, say so plainly and don't invent a question. I tested it by deleting the file mid-conversation. Sydney kept asking questions anyway — just invented from her own knowledge instead of pulled from the dataset.

Functionally, the conversation didn't break. But it wasn't the guardrail I built. A string returned from a tool, described in the system prompt as "say this instead," is advisory — the model can route around it. If a failure path genuinely needs to be enforced, it has to be checked in code before generation, not just described in text.

Bug 3: A 10-second timer isn't a success metric

My first pass at "successful call" for the Day 8 dashboard used call duration as a proxy — over 10 seconds counted as success. It technically produced numbers. It also marked a call successful when a learner heard the greeting and immediately hung up, because the greeting alone took past 10 seconds.

I rebuilt it around actual behavior: success only counts when the exercise-fetch tool fired and the learner responded afterward. Not perfect — someone could still say "I don't know" and hang up — but it measures something closer to what the task doc actually asked for: did the learner engage.

6. Build and run it yourself

Prerequisites

  • Python 3.10+ with uv for dependency management
  • Node.js 18+ for the frontend
  • Accounts: LiveKit Cloud, Murf AI (Falcon TTS), Deepgram, Google AI Studio (Gemini)
  • Optional: Twilio account for outbound calling, a Discord server for escalation webhooks

Setup 

git clone https://github.com/suyashsahu00/10-days-of-AI-voice-agents-voice-for-bharat-challenge-2026
cd 10-days-of-AI-voice-agents-voice-for-bharat-challenge-2026

# Backend
cd backend
uv sync
cp .env.example .env.local   # add your API keys here — never commit this file
uv run python src/agent.py dev

# Frontend (separate terminal)
cd ../frontend
npm install
npm run dev
Enter fullscreen mode Exit fullscreen mode

Where API keys go

All secrets — LIVEKIT_API_KEY, LIVEKIT_API_SECRET, Murf, Deepgram, Google, Twilio, Discord webhook — live in backend/.env.local, which is git-ignored and never committed. The frontend never touches raw API keys directly; it talks to a Next.js token endpoint (/api/token) that mints short-lived LiveKit access tokens server-side.

Testing a conversation

Open the frontend at localhost:3000, tap the mic, and talk. To test memory persistence, have a short conversation, hang up, and call again from the same browser — Sydney should greet you by name and reference what you discussed.

7. What I'd improve next

  • Enforce the exercise-failure fallback in code, not just in the prompt — check the tool's return state before letting the LLM generate freely, closing the gap from Bug 2.
  • Give specialists access to escalation. Right now, if a learner gets distressed while talking to a specialist agent, Day 7's safety net doesn't reach that far — a real gap for anything beyond a demo.
  • Add hand-back to the main agent so a learner can return to general mentoring without ending the call and reconnecting.
  • Wire real LiveKit room events into the frontend state, replacing a placeholder timer-based UI state simulation with the agent's actual speaking/listening state.

8. Links, code, and demos

Built as part of Murf AI's 10 Days of Voice Agents — Voice for Bharat Challenge 2026, using Murf Falcon — the fastest TTS API — for every word Sydney speaks.

#VoiceForBharat #10DaysOfVoiceAgents #AIVoiceAgent #BuildInPublic #EdTech #GenerativeAI #VoiceAI

Top comments (0)