DEV Community

Cover image for Building DeutschMate: A Voice-Powered German Tutor & Interview Coach for Bharat 10 Days of Voice Agents
Ritesh Patil
Ritesh Patil

Posted on

Building DeutschMate: A Voice-Powered German Tutor & Interview Coach for Bharat 10 Days of Voice Agents

#ai #productivity #opensource #learning

My journey through 10 Days of Voice Agents — VoiceForBharat Edition, building a real-time healthcare voice agent with Murf Falcon, LiveKit, memory, tools, telephony, human escalation, analytics, and multi-agent handoffs.

Introduction

What if learning German could be as simple as speaking naturally?

Instead of opening an app, navigating through lessons, and typing answers, imagine simply saying:

“Mujhe German practice karni hai.”
“Give me a German exercise.”
“I want to practice for a German job interview.”

That idea became DeutschMate — a voice-first German learning companion I built during 10 Days of Voice Agents — VoiceForBharat Edition.

DeutschMate is designed for Bharat-based learners who want to improve their German through real conversations, whether they're learning for education, work, relocation, or simply to become more confident speaking German.

GitHub: DeutschMate - GitHub

The Problem

Learning a language is not just about memorizing vocabulary or completing written exercises. At some point, you have to speak.

Traditional language-learning interfaces often make learners move through menus, read questions, type answers, and practice in a way that doesn't always feel like a real conversation.

For Indian learners, conversations can also naturally move between languages:

“Mujhe German mein introductions practice karna hai.”
“Can you give me an A1 exercise?”
“Mujhe German job interview ke liye prepare karna hai.”

DeutschMate is built to make that interaction feel more natural.

Instead of treating voice as just another input method, I wanted voice to be the primary interface for learning and practicing German.

What DeutschMate Does

DeutschMate combines real-time voice interaction with learning tools, memory, phone calls, analytics, and specialist agents.

🎙️ Real-time voice conversation
🇮🇳 Indian-English and code-mixed conversations, with voice powered by Murf Falcon
📚 German exercises for vocabulary, grammar, and conversational practice
🧠 Learner memory for returning users, including learning goals and topics covered
📞 Outbound German practice calls
🧑‍🏫 Human escalation when the learner needs help beyond what the agent should handle
📊 Call analytics dashboard showing total, successful, and failed calls
💼 German Job Interview Coach for specialized interview practice
🤖 Agent handoffs so the main tutor can delegate specialized conversations
🛡️ Safety and behavioral guardrails to keep the agent within its intended role

The goal isn't simply to make an AI that can talk.

The goal is to build a voice learning system that can understand what the learner is trying to accomplish and take the appropriate action.

How the System Works

##How the System Works

The Stack

Layer Technology
Voice transport LiveKit
Speech-to-text Deepgram
LLM / Reasoning Google Gemini
Text-to-speech Murf Falcon
Backend Python + LiveKit Agents
Frontend Next.js / React
Database PostgreSQL
Learner memory PostgreSQL
German exercises Local JSON dataset + Agent Tools
Outbound telephony LiveKit SIP
Call analytics PostgreSQL + Analytics Dashboard
Specialist agent German Job Interview Coach
Human escalation Email / SMTP
Real-time communication LiveKit

The Voice Layer

Murf Falcon handles the text-to-speech layer of DeutschMate, turning the agent's responses into natural spoken conversation.

The voice experience is designed around the way Indian learners actually communicate - conversations can naturally move between English, Hindi, Hinglish, and German depending on the learner's comfort level.

German Learning Tools

DeutschMate isn't just a conversational chatbot.

It has tools specifically designed around German learning.

get_german_practice
Enter fullscreen mode Exit fullscreen mode

The exercise system loads German exercises from:

backend/data/german_exercises.json

Each exercise contains information such as:

{
  "id": "...",
  "level": "beginner",
  "topic": "...",
  "type": "...",
  "question": "...",
  "answer": "..."
}
Enter fullscreen mode Exit fullscreen mode

The exercise loader can filter exercises by learner level, topic, and practice type before selecting an exercise.

For example:

Learner: "Give me a German exercise."


                ↓


get_german_practice()


                ↓


Select exercise
        ↓
Level / topic / type
        ↓
German question
        ↓
Learner answers verbally
        ↓
Gemini evaluates the response
Enter fullscreen mode Exit fullscreen mode

One important design decision here was keeping the learning experience conversational.

The learner doesn't have to click through a traditional quiz interface.

They can simply speak their answer.

Exercise Completion and Call Success

For Day 8, I needed a concrete definition of what a successful call means.

For DeutschMate:

A successful call means the learner completes the intended German learning task during the session.

Because the original exercise system didn't have a programmatic completion state, I added an explicit completion mechanism.

The agent can mark an exercise as completed after determining that the learner answered correctly:

Learner
   ↓
German exercise
   ↓
Learner attempts answer
   ↓
Gemini evaluates response
   ↓
Correct?
   ├── Yes → mark_exercise_complete()
   │              ↓
   │          Success flag
   │
   └── No  → Continue practice
Enter fullscreen mode Exit fullscreen mode

When the session ends, that state is used to record the call outcome.

This gave the analytics system an actual definition of success vs. failure, instead of simply counting every conversation as successful.

Memory for Returning Learners

One of the most useful features of DeutschMate is persistent learner memory.

A returning learner shouldn't have to start from zero every time.

With consent, DeutschMate can save useful learning information such as:

  • learner name
  • German level
  • language preference
  • learning goal
  • topics covered
  • common mistakes

The memory system explicitly checks consent before saving learner information.

For example:

First session:

Learner → "My name is Ritesh."
        → "I'm a beginner in German."
        → "I want German for interviews."


                ↓


        Memory consent


                ↓


        PostgreSQL
Enter fullscreen mode Exit fullscreen mode

On a later session:

Returning learner
        ↓
PostgreSQL lookup
        ↓
Previous learning context
        ↓
DeutschMate
        ↓
Personalized conversation
Enter fullscreen mode Exit fullscreen mode

The agent logs also confirm that learner memory is loaded from PostgreSQL when a returning learner starts a session.

The principle here is simple:

Don't store everything. Store useful learning context, and only when the learner has consented.

Outbound Voice Calls

The project also moved beyond the normal:

Learner → Agent

model.

DeutschMate can initiate an outbound German practice call.

The outbound session reuses the same learner memory and practice capabilities as the normal conversation flow.

There is also an important transparency rule for outbound calls.

The agent introduces itself first:

"Hallo! This is DeutschMate, your AI German tutor.
I'm calling for your daily German practice session.
You can hang up anytime if you'd like to stop.
Are you ready to practice?"
Enter fullscreen mode Exit fullscreen mode

The system explicitly avoids pretending to be a human caller and ends the call if the learner asks it to stop.

Human Escalation

AI shouldn't always be the final destination.

Sometimes a learner simply needs a human teacher.

DeutschMate supports escalation when the learner needs human assistance, but the important part is that consent is explicit.

The flow is:

Learner
   ↓
Needs human help
   ↓
Explain why escalation is needed
   ↓
Explain what information will be shared
   ↓
Ask for consent
   ↓
Clear confirmation?
   ├── No → Continue normally
   │
   └── Yes
        ↓
   create_escalation()
        ↓
   Generate reference ID
        ↓
   Save escalation request
        ↓
   Attempt human notification
Enter fullscreen mode Exit fullscreen mode

For example:

Learner: "I'm feeling overwhelmed and I need a teacher."


DeutschMate:
"I can send a short summary to a human teacher
so they can help you. Would you like me to send
that request?"
Enter fullscreen mode Exit fullscreen mode

Only after a clear confirmation does the escalation tool execute.

The actual logs show the agent creating escalation requests after explicit consent and generating a reference ID for the request.

This became an important lesson:

An AI tutor should know when it has reached the edge of what it should handle itself.

Call Analytics Dashboard

Once a voice agent starts handling real conversations, another question becomes important:

How well is it actually performing?

For Day 8, I added a call_analytics system backed by PostgreSQL.

The dashboard tracks:

  • Total calls
  • Successful calls
  • Failed calls

The data comes from actual agent sessions rather than hardcoded dashboard values.

The basic flow is:

Voice Session
     ↓
Learner completes / does not complete task
     ↓
Session ends
     ↓
Analytics writer
     ↓
PostgreSQL
     ↓
Dashboard
Enter fullscreen mode Exit fullscreen mode

A successful German learning call is one where the intended exercise is completed.

A failed call does not necessarily mean that the system crashed.

It simply means the defined learning objective wasn't completed.

This distinction matters because:

System failure and user-outcome failure are not the same thing.

During development, the analytics writer also exposed a PostgreSQL connection-lifecycle issue during LiveKit shutdown. The logs initially showed:

cannot perform operation: another operation is in progress
Enter fullscreen mode Exit fullscreen mode

and:

ConnectionDoesNotExistError:
connection was closed in the middle of operation
Enter fullscreen mode Exit fullscreen mode

That led to an analytics fix using an isolated database connection and retry handling rather than relying directly on a shared pool operation.

Multi-Agent Handoff

One of the most interesting parts of DeutschMate is that the main agent doesn't have to do everything itself.

The main agent handles general German learning.

When the learner asks for interview-specific help, the conversation can be handed to a specialist:

German Job Interview Coach

For example:

Learner:

"I have a German job interview tomorrow."


        ↓


Main DeutschMate Agent
        ↓
Recognizes interview intent
        ↓
handoff_to_job_interview_coach()
        ↓
German Job Interview Coach
Enter fullscreen mode Exit fullscreen mode

This isn't theoretical — the actual agent logs show the main DeutschMate agent detecting:

"Have interview tomorrow."
Enter fullscreen mode Exit fullscreen mode

and executing:

handoff_to_job_interview_coach

with the interview date passed into the handoff context.

The idea is simple:

One agent coordinates. Specialists handle specialized tasks.

This is much easier to extend than trying to put every possible capability into one enormous system prompt.

Context Travels With the Handoff

A handoff is only useful if the specialist understands why the learner was transferred.

For example:

Main Agent
↓
Intent: German job interview
↓
Interview date: tomorrow
↓
Target role: Data Science
↓
Job Interview Coach

The specialist can then continue the conversation instead of asking the learner to repeat everything.

At the same time, sensitive information shouldn't unnecessarily travel between agents.

The goal is to pass task-relevant context, not the entire conversation.

Privacy Was a Design Requirement, Not an Afterthought

DeutschMate deals primarily with learning information rather than medical or financial information, but privacy still matters.

The system follows several boundaries:

**1. Learner memory requires consent

  1. Only useful learning context is stored
  2. Sensitive credentials are never part of learner memory
  3. API keys belong in environment variables
  4. Full private conversations should not be exposed through public analytics
  5. Specialist handoffs should carry relevant context rather than unnecessary private information**

The memory implementation explicitly blocks saving when consent is not enabled.

The principle is:

Personalization should not require storing everything about the learner.

Project Structure

The project is organized into separate backend and frontend applications:

murf-livekit-starter/
│
├── backend/
│   ├── .github/
│   ├── data/
│   │   └── german_exercises.json
│   │
│   ├── src/
│   │   ├── agent.py
│   │   ├── analytics.py
│   │   ├── dashboard.py
│   │   ├── migrate.py
│   │   └── ...
│   │
│   ├── tests/
│   │   ├── test_agent.py
│   │   ├── test_practice.py
│   │   ├── test_analytics.py
│   │   └── test_day9_handoff.py
│   │
│   ├── .env.example
│   ├── Dockerfile
│   ├── pyproject.toml
│   ├── railway.toml
│   └── README.md
│
├── frontend/
│   ├── app/
│   ├── components/
│   ├── fonts/
│   ├── hooks/
│   ├── lib/
│   ├── public/
│   ├── styles/
│   ├── .env.example
│   ├── next.config...
│   └── package.json
│
└── README.md
Enter fullscreen mode Exit fullscreen mode

How to Run the Project

Clone the repository:

git clone https://github.com/riteshpatil9686-lgtm/murf-livekit-starter.git
cd murf-livekit-starter
Enter fullscreen mode Exit fullscreen mode

Backend

cd backend
uv sync
Enter fullscreen mode Exit fullscreen mode

Frontend

cd frontend
pnpm install
Enter fullscreen mode Exit fullscreen mode

Environment variables

Create your local environment files and never commit them.

For example:

LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret


MURF_API_KEY=your_murf_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_google_api_key


DATABASE_URL=your_postgresql_database_url
Enter fullscreen mode Exit fullscreen mode

If outbound calling is enabled, additional SIP configuration is required.

Never publish:

  • API keys
  • API secrets
  • Database credentials
  • SMTP credentials
  • SIP credentials
  • Private learner information 
#Run the backend
cd backend
uv run python src/agent.py dev

#Run the frontend
cd frontend
pnpm dev
Enter fullscreen mode Exit fullscreen mode

Then open:

http://localhost:3000

Allow microphone access and start talking.

Running the Analytics Dashboard

The Day 8 dashboard runs separately:

cd backend
python src/dashboard.py
Enter fullscreen mode Exit fullscreen mode

Then open:

http://localhost:8888

The dashboard reads its numbers from PostgreSQL rather than hardcoding them.

Testing the Agent

I tested the system across several different paths instead of testing only the happy path.

  1. German practice 
"Give me a German exercise."
Enter fullscreen mode Exit fullscreen mode
  1. Returning learner 
"Do you remember what we practiced last time?"
Enter fullscreen mode Exit fullscreen mode
  1. Interview handoff 
"I have a German job interview tomorrow."
Enter fullscreen mode Exit fullscreen mode
  1. Human escalation 
"I need help from a human teacher."
Enter fullscreen mode Exit fullscreen mode
  1. Code-mixed conversation 
"Mujhe German mein practice karni hai."
Enter fullscreen mode Exit fullscreen mode
  1. Outbound practice

Trigger the outbound practice flow and verify that DeutschMate introduces itself before beginning the lesson.

The project also includes automated tests covering the agent and the newer analytics and handoff functionality.

The latest backend test run completed with:

============================= test session starts =============================


collected 14 items


tests/test_agent.py ...
tests/test_analytics.py .
tests/test_day9_handoff.py ...
tests/test_escalation.py .......


============================== 14 passed ==============================


14 passed, 7 warnings
Enter fullscreen mode Exit fullscreen mode

The warnings were dependency deprecation warnings rather than test failures.

The Hardest Part: Making Everything Work Together

The hardest part of this project wasn't getting an LLM to answer a question.

It was getting all the moving pieces to work together.

The system combines:

Voice
  +
Real-Time Transport
  +
Speech-to-Text
  +
LLM
  +
Text-to-Speech
  +
Tools
  +
PostgreSQL
  +
Memory
  +
SIP
  +
Analytics
  +
Multi-Agent Handoff
Enter fullscreen mode Exit fullscreen mode

And each layer can introduce a completely different kind of problem.

For example, the outbound calling implementation required dealing with LiveKit dispatch, SIP participant creation, SIP URI handling, learner metadata, memory loading, and the outbound-specific conversation flow.

Another challenge was analytics during asynchronous session shutdown.

The first implementation could encounter a PostgreSQL connection error while LiveKit was shutting down. Fixing that required understanding that the database operation needed its own connection lifecycle rather than simply changing the SQL query.

That was probably one of my biggest lessons:

Voice AI isn't only an AI problem.

It's real-time systems, networking, databases, asynchronous Python, telephony, frontend state, and AI orchestration — all working together.

What I Learned

Before this challenge, I thought about a voice agent roughly like this:

Speech → AI → Speech

After ten days, the architecture looks much more like:

Voice
  +
Real-Time Transport
  +
Speech-to-Text
  +
Reasoning
  +
Tools
  +
Memory
  +
Telephony
  +
Analytics
  +
Human Escalation
  +
Specialist Agents
Enter fullscreen mode Exit fullscreen mode

And the most important lesson wasn't about making an AI sound human.

It was about building a system that knows:

when to speak, when to use a tool, when to remember, when to transfer, when to ask for permission, and when to let another agent handle the task.

What I'd Build Next

DeutschMate is still a starting point.

The next things I'd like to improve are:

1. More German exercises **— expand the exercise library across A1–C1 levels
**2. Smarter exercise evaluation — move beyond purely LLM-based evaluation toward more structured scoring
3. Better learner progress tracking — track improvement across grammar, vocabulary, speaking, and interview skills
4. More Indian languages — make code-mixed interaction even more natural
5. Production telephony — expand beyond development SIP testing
6. Deeper analytics — latency, tool success rate, handoff success, completion rate, and failure reasons
7. Richer interview coaching — role-specific interview questions, feedback, and structured mock interviews
8. Stronger privacy architecture — authentication, access controls, retention policies, and better separation of learner data

Final Thoughts

Ten days ago, the goal was simple:

Build a voice agent.

Today, DeutschMate can listen, understand, practice German with the learner, remember consented learning context, use tools, make outbound practice calls, track call outcomes, escalate to human help, and hand conversations to a specialized German Job Interview Coach.

The biggest shift in my thinking was realizing that a voice agent isn't just:

"ChatGPT, but with a voice."

It is a complete system.

The voice is only the interface.

Behind it are:

real-time communication, reasoning, tools, memory, databases, telephony, safety boundaries, analytics, and specialized agents.

And for language learning, voice makes the experience particularly interesting because the learner isn't just asking an AI questions.

They are actually speaking the language they are trying to learn.

That's what I wanted DeutschMate to become:

A German tutor you can simply talk to.

Project & Resources

Source code:
DeutschMate — GitHub

Challenge:
10 Days of Voice Agents — VoiceForBharat Edition

Built with:
Murf Falcon + LiveKit + Deepgram + Google Gemini + PostgreSQL

Top comments (0)