How I created "Beacon"

# Building Beacon AI: A 10-Day Journey in Voice AI Engineering — LiveKit + Murf Falcon

Over the past 10 days, I took part in the Voice for Bharat Challenge 2026, where I built Beacon AI, a multi-agent educational voice assistant powered by real-time voice AI.

This journey gave me hands-on experience with speech pipelines, streaming audio, latency optimization, agentic workflows, and multi-agent routing.

In this post, I’ll walk through what I built, the architecture behind it, the major technical challenges I faced, and how I solved them.

🎙️ Introducing Beacon AI

Beacon AI is an interactive voice tutor designed to make learning general knowledge, space trivia, language translation, and mental math accessible through natural voice conversations.

Why voice?

Voice interfaces can remove a major barrier to interaction. For learners who struggle with typing or simply prefer conversational learning, a voice-based assistant can make education feel more natural and accessible.

Beacon AI also supports code-mixed Hindi/English conversations, helping bridge the gap between traditional text-based interfaces and real-world conversations.

⚙️ Core Architecture

Beacon AI runs as a real-time voice pipeline consisting of four major components:

User
  ↓
LiveKit WebRTC
  ↓
Deepgram STT
  ↓
Google Gemini
  ↓
Agent Tools / Specialist Agents
  ↓
Murf Falcon TTS
  ↓
User

1. Transport Layer — LiveKit

LiveKit handles the real-time WebRTC connection between the user and the voice agent.

It manages the streaming audio session and provides the infrastructure required for low-latency voice interactions.

2. Speech-to-Text — Deepgram

Deepgram converts the user's speech into text in real time.

This transcription is then passed to the language model for reasoning and intent detection.

3. Language Model — Google Gemini

Google Gemini acts as the reasoning layer.

It processes the transcription, determines what the user wants, calls tools when necessary, and generates the response.

4. Text-to-Speech — Murf Falcon

Finally, Murf Falcon converts the generated response back into natural-sounding speech.

One of the biggest focuses of this project was keeping the voice interaction fast and responsive. Murf Falcon's extremely low latency makes it particularly useful for real-time conversational agents.

✨ What I Built Over 10 Days

Beacon AI evolved from a basic voice assistant into a small agentic ecosystem.

🧠 Dynamic User Memory

Beacon AI can look up users from a local database using their name.

This allows the assistant to recognize returning users and remember things such as:

• Previously covered topics
• Past mistakes
• Learning progress
• Conversation context

🤖 Specialist Micro-Agents

Instead of forcing one agent to handle every type of question, Beacon AI uses specialist agents.

The main agent can hand conversations over to:

• Beacon Maths — arithmetic drills and mental math
• Beacon Cosmos — science, space, and gravity trivia
• Beacon Bhasha — grammar and language practice

This keeps the individual agent prompts focused and makes the overall system easier to extend.

📰 Live News Integration

Beacon AI can retrieve current headlines using RSS feeds from BBC News.

Users can ask for:

• General news
• Science news
• Technology news

The agent retrieves the relevant feed and presents the information conversationally.

🧑‍🏫 Human Escalation

Sometimes the best response from an AI system is knowing when to involve a human.

If a user becomes frustrated or explicitly asks for a human tutor, Beacon AI creates an escalation ticket and sends a webhook notification to a dedicated Discord tutor channel.

📊 Call Analytics Dashboard

I also built a Next.js analytics dashboard to monitor the voice agent.

It tracks metrics such as:

• Call duration
• User turns
• Call success rate
• Active escalation tickets

This gives a clearer picture of how the system behaves beyond the voice interaction itself.

---

🚧 The Hardest Technical Challenges

Building a real-time voice agent isn't just about connecting an LLM to a TTS API.

Timing, concurrency, streaming, and unpredictable LLM behavior can create some interesting problems.

Two challenges stood out during development.

1. The Handoff Transition Race Condition

The problem

Initially, when the main agent handed a conversation to a specialist, I used background tasks and arbitrary asyncio.sleep() timers to trigger the specialist's response.

That seemed to work during simple tests.

But on slower networks, the main agent could still be draining its TTS stream when the transition task fired.

This caused conflicts in the session and, in some cases, left the user with silence.

The solution

Instead of trying to guess when the previous agent would finish, I moved the logic to LiveKit's native on_enter() lifecycle hook.

The specialist now waits until it has actually taken control before generating its response:

async def on_enter(self) -> None:
    logger.info("Specialist entered. Triggering reply.")
    self.session.generate_reply(
        chat_ctx=self.session.history
    )

This eliminated the timing race condition and made the handoff much more reliable.

---

2. LLMs Skipping Tool Calls

This was another interesting problem.

Initially, I instructed the main agent to do something like:

"I will connect you to our Cosmos specialist."

and then call the handoff_to_cosmos tool.

The problem was that smaller/faster models could sometimes generate the conversational sentence but skip the tool call.

That meant the user would be told they were being transferred, but the transfer wouldn't actually happen.

The solution

I separated the conversational response from the tool invocation.

Instead of asking the LLM to both speak and call the tool, I instructed it to immediately execute the handoff tool.

The transition message itself is spoken inside the tool:

@function_tool
async def handoff_to_cosmos(
    self,
    context: RunContext
) -> Agent:

    await context.session.say(
        "I will connect you to our Cosmos specialist.",
        allow_interruptions=True
    )

    return specialist

This separation made the routing significantly more reliable.

The LLM handles decision-making, while the tool handles the actual transition and announcement.

---

🌐 Language & Script Handling

Since Beacon AI supports multilingual and code-mixed conversations, I also added explicit language and script instructions to the system prompt.

LANGUAGE & SCRIPT

Always write every language in its own native script.

Hindi → Devanagari (नमस्ते)
Never use romanized Hindi (never "namaste").

Follow the same rule for all non-English languages.

This helps prevent the model from responding with romanized versions of languages that should be rendered in their native scripts.

---

🚀 Running Beacon AI Locally

If you want to experiment with the project yourself, you'll need:

• Python 3.10+
• uv
• Node.js
• pnpm
• A LiveKit Cloud account

Environment Variables

Create a .env.local file in both the backend and frontend where required:

LIVEKIT_URL=wss://your-livekit-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_key
LIVEKIT_API_SECRET=your_livekit_secret

MURF_API_KEY=your_murf_falcon_key
DEEPGRAM_API_KEY=your_deepgram_key
GOOGLE_API_KEY=your_gemini_key

Start the Backend

cd backend
uv sync
uv run python src/agent.py dev

Start the Frontend

cd frontend
pnpm install
pnpm dev

Then open:

http://localhost:3000

Click "Ignite Conversation" and start talking to Beacon AI.

---

🏗️ What I Learned

The biggest lesson from this challenge was that building a voice agent is very different from building a traditional chatbot.

A good voice experience requires thinking about:

Latency → Streaming → Interruptions → State → Routing → Tool execution → TTS timing

Even a small delay or incorrectly timed handoff can make a conversation feel broken.

I also learned that multi-agent architectures aren't just about adding more agents. The routing logic between them is just as important as the agents themselves.

---

🔗 Source Code

The complete project, including the Next.js analytics dashboard and multi-agent handoff implementation, is open source:

GitHub: asn-bharadwaj/beacon-ai-agent

---

🎯 Final Thoughts

Building Beacon AI over these 10 days has been an incredible introduction to real-time voice AI engineering.

From basic speech pipelines to specialist-agent handoffs, memory, live news, human escalation, and analytics, the project grew into something much more advanced than what I initially planned.

A huge part of the experience was learning how seemingly small engineering decisions — especially around latency, concurrency, and agent handoffs — can have a huge impact on the user experience.

And most importantly, I got to build a voice agent using Murf Falcon, one of the fastest TTS APIs, while exploring what is possible with modern real-time AI systems.

Thanks to the Voice for Bharat Challenge 2026 for the opportunity to build, experiment, break things, and learn along the way. 🚀

If you're building with LiveKit, Murf Falcon, Gemini, or multi-agent voice systems, I'd love to hear what you're working on!

Official Resources

Murf Falcon 2 Documentation:
https://murf.ai/api/docs/text-to-speech-models/falcon-2

Murf LiveKit Starter Template:
https://github.com/murf-ai/murf-livekit-starter

LiveKit Voice AI Quickstart:
https://docs.livekit.io/agents/start/voice-ai/

Day 10 Challenge Task:
https://github.com/murf-ai/voice-for-bharat-challenge-2026/blob/main/challenges/Day%2010%20Task.md

---

Tags

#AI #VoiceAI #GenerativeAI #Murf #LiveKit #Python #Gemini #Deepgram #MultiAgentAI #BuildInPublic