Riya Tewari

Posted on Aug 15

Building Aira: From a Voice Assistant to Learning What I'm Capable Of

#ai #voiceai #python #nextjs

When I signed up for VoiceForBharat, I honestly had no idea what I was getting myself into.

I remember looking at the project and thinking:

"What have I done? Am I even going to be able to build this?"

Voice agents, AI models, APIs, databases, telephony, memory, backend systems...

There were so many things I didn't know.

And if I'm being completely honest, in the beginning, it felt *really difficult.

I didn't know everything.

I didn't even know if I was capable of building something like this.

But I decided to start anyway.

And somewhere between the first error and the hundredth fix, I realized something:

I wasn't just building a project. I was learning how to build one.

That project slowly became Aira.

🤍 It Started With Doubt

When you look at a finished project, it is very easy to forget what it looked like at the beginning.

You see the interface.

The voice assistant.

The memory system.

The phone calls.

The analytics.

The backend.

And it looks like everything was planned from day one.

It wasn't.

There were so many moments where I had absolutely no idea what I was doing.

I had to learn things while building them.

I had to break things.

Fix them.

Break them again.

Search for answers.

Read documentation.

Understand errors.

Try something.

Realize it was wrong.

Try again.

And slowly, things started making sense.

That's when I understood something I think is important for anyone who is learning technology:

You don't always need to know how to build something before you start. Sometimes you learn how to build it by actually trying to build it.

Taking the Risk

One thing this project taught me is that you can spend a lot of time asking yourself:

"Am I ready?"

But sometimes, you'll never feel ready.

If I had waited until I knew enough about AI, voice systems, backend development, APIs, telephony, and everything else involved in this project, I probably wouldn't have started.

And I definitely wouldn't have learned what I know now.

There is something about taking a risk and saying:

"I'll figure it out."

That changes everything.

Because until you actually try something difficult, you don't really know what you're capable of.

You might think you have a certain limit.

You might think you're "not good enough" at something.

You might think someone else is naturally better at it.

But sometimes the only thing separating you from that ability is the fact that you haven't given yourself the chance to learn it yet.

You have to push yourself far enough to discover your own limits.

And sometimes, you'll realize you don't have as many limits as you thought.

So, What Exactly Is Aira?

Aira is a multilingual voice assistant designed to communicate naturally in Hindi, English, and Hinglish.

But I didn't want it to be just:

User speaks → AI responds.

I wanted to build something that could actually interact with people and handle different parts of a conversation.

Aira can:

Have real-time voice conversations
Remember relevant user information and preferences
Use tools such as currency exchange
Make outbound phone calls
Escalate conversations when human help is needed
Support specialist/human handoff
Track call outcomes
Store conversations and memories
Keep different users' data separated
Provide a web interface to manage and interact with the system

And building each of those features taught me something different.

Giving Aira Memory

One of the first things I wanted was for Aira to remember users.

A typical chatbot can answer a question, but the next conversation often starts from zero.

I wanted Aira to provide a more continuous experience.

User information, preferences, facts, and conversations can be stored so Aira can use relevant context in future interactions.

I built the memory layer using SQLite, with a backend API handling access to that information.

The frontend also has a dedicated Your Space section where users can view:

Saved information
Language preferences
Stored facts
Previous conversations
Conversation summaries

Users can also choose to Forget Me, which removes the stored memory associated with their account.

But building memory taught me something beyond databases.

It made me think about how important context is.

A system can be powerful, but if it doesn't understand the right context, it can still behave incorrectly.

And that became even more obvious when I started testing multiple users.

One User Is Not Every User

This became one of the most important debugging challenges.

At one point, logging in with a different account could still result in Aira using information from the previous user.

A new user shouldn't see someone else's memories.

Aira shouldn't call a new user by someone else's name.

So I had to trace the entire flow:

Signup → Login → User Identity → Backend User ID → Memory → Conversations

Each account has its own user_id, and frontend requests use that identity when fetching memory and conversation data.

The dashboard and Your Space pages now load information specifically for the currently logged-in user.

It was frustrating to debug.

But it taught me one of the most important lessons of this project:

Getting a feature to work is one thing. Getting it to work correctly is another.

Turning Aira Into an Agent

Aira isn't limited to generating conversational responses.

I also added tools that allow the assistant to perform actual tasks.

One example is the currency exchange tool.

Instead of simply guessing an exchange rate, Aira can call an external exchange-rate API and return the result conversationally.

The basic flow is:

User asks → Aira determines whether a tool is needed → Tool runs → Aira explains the result

This was one of the first moments where Aira started feeling less like a chatbot and more like an agent.

Making Aira Call People

Then came one of the biggest steps:

Outbound calling.

I connected Aira with LiveKit's telephony infrastructure so the assistant could participate in actual phone conversations.

And honestly, this was one of those moments where the project suddenly felt very different.

A voice assistant inside a browser is one thing.

An AI assistant that can actually participate in a phone call feels much closer to something that could exist in the real world.

But with that came a whole new set of challenges:

Call connection
Agent dispatch
Telephony configuration
Voice latency
Conversation flow
Call completion
Failure handling

And once Aira could make calls, I had another question:

How do I know whether those calls actually worked?

Making Calls Measurablev

So I added call outcome tracking.

Calls can now be categorized as:

Successful
Failed

The backend stores these outcomes, while the dashboard retrieves the data for analytics.

The dashboard can show:

Total Calls

The number of calls Aira has handled.

Successful Calls

Calls that reached a successful outcome.

Failed Calls

Calls that didn't complete successfully.

The analytics also refresh periodically so the dashboard can reflect updated call data without requiring a manual refresh.

This was an important shift.

I wasn't just building something that works.

I was building something that could be observed and measured.

Knowing When AI Should Step Aside

Another thing I learned while building Aira is that making AI do everything isn't necessarily the goal.

Sometimes, the best thing an AI can do is recognize:

"This needs a human."

So I added an escalation system.

When a conversation requires human help, Aira can generate structured information such as:

User
Reason
Summary
Urgency
Language
Preferred follow-up
Status
Timestamp

These escalation requests can then be reviewed through the Aira Support dashboard.

Instead of simply saying:

"Please contact support."

the system creates structured information that a human can actually understand and act on.

Specialist Handoff

The escalation system also made me think beyond simply sending everything to "support."

Different problems require different kinds of expertise.

So I started working on specialist handoff — allowing Aira to recognize when a conversation is better handled by someone with specific expertise.

This is one of the ideas I find most interesting about agentic systems.

The goal isn't to make AI do absolutely everything.

It's to make AI understand:

"I can handle this."

"This would be better handled by someone else."

Knowing when to act is important.

Knowing when to step aside is equally important.

Building the Interface

All of these systems needed somewhere to live.

So I built a web interface using Next.js.

The application includes:

Login

Users can sign into their Aira account.

Signup

New users can create an account.

Dashboard

The main place to view Aira analytics and access different features.

Your Space

A personal area containing memories, preferences, and previous conversations.

Support

A place where escalation requests can be reviewed.

Voice Interface

The main interaction area where users can actually talk to Aira.

I also wanted the interface to reflect Aira's personality — soft, warm, and minimal, rather than looking like a generic developer dashboard.

What's Under the Hood?

The current architecture looks roughly like this:

Frontend

→ Next.js
→ React
→ Tailwind CSS

Backend

→ Python
→ FastAPI
→ SQLite

Voice Pipeline

→ Deepgram for speech recognition
→ Gemini for reasoning and conversation
→ Murf Falcon for voice generation
→ LiveKit for real-time voice and telephony

Additional Systems

→ Memory
→ Conversations
→ Tool calling
→ Outbound calls
→ Escalations
→ Call analytics
→ Specialist handoff

At its core, the voice flow looks like:

** User speaks → Deepgram STT → Gemini → Murf Falcon TTS → LiveKit → User hears the response**

So Aira isn't just one model sitting behind a microphone.

It's a collection of systems working together.

And that is probably one of the biggest technical lessons this project gave me:

AI isn't just the model.

It's everything you build around it.

The Things That Almost Worked

Some of the most valuable parts of this project weren't the flashy features.

They were the things that almost worked.

A new user could log in successfully...

but Aira might still remember the previous user's name.

The support page could exist...

but the support button could overlap with another UI element.

The call could happen...

but I still needed a reliable way to know whether it actually succeeded.

The backend could store information...

but the frontend needed to retrieve only the current user's data.

These problems were frustrating.

But every one of them taught me something.

I learned that building an AI product isn't simply:

"Add AI → Done."

It's also:

Identity + State + APIs + Databases + UI + Error Handling + Observability + User Experience.

Learning to Build Responsibly

While building the project, I also became much more conscious about handling credentials properly.

Services like:

LiveKit
Murf
Deepgram
Gemini

all require credentials, and those keys shouldn't be hard-coded into the application or committed to GitHub.

Aira uses environment variables for these secrets.

The repository contains .env.example files that show the expected variable names without exposing actual secrets.

Never publish API keys, phone numbers, caller data, or other private information.

These may seem like small implementation details, but they matter when moving from experimenting with code to building actual applications.

Want to Build Your Own Voice Agent?

If you want to build something similar to Aira, you don't need to start completely from scratch.

I built Aira on top of the Murf LiveKit Starter, which provides the basic real-time voice pipeline and can then be extended with your own personality, tools, memory, UI, and agent logic.

Explore the Code

The complete project is publicly available on GitHub:

Aira — GitHub Repository

You can inspect the code, experiment with the architecture, and build your own use case on top of it.

1. Prerequisites

You'll need:

Python 3.10+
Node.js 18+
uv for Python package management
pnpm for frontend dependencies
A LiveKit Cloud project
API credentials for Murf, Deepgram, and your chosen LLM

2. Clone the Repository

git clone https://github.com/tewaririya117-ops/murf-livekit-starter.git
cd murf-livekit-starter

3. Configure Environment Variables

The project keeps credentials in environment files rather than hard-coding them into the application.

The important variables include:

LIVEKIT_URL
LIVEKIT_API_KEY
LIVEKIT_API_SECRET
MURF_API_KEY
DEEPGRAM_API_KEY
GOOGLE_API_KEY

Depending on the LLM configuration, an OPENAI_API_KEY can also be used.

Create your environment files from the provided .env.example templates in the backend and frontend directories.

Never publish your actual API keys, phone numbers, caller information, or other private data to GitHub or your blog.

4. Install Backend Dependencies

From the backend directory:

cd backend
uv sync
uv run python src/agent.py download-files

The backend contains the main voice agent and connects the speech-to-text, LLM, and text-to-speech components.

5. Install Frontend Dependencies

From the frontend directory:

cd frontend
pnpm install

The frontend is built with Next.js and provides the interface through which users connect to the voice agent.

6. Run the Project

You can either use the provided startup script or run the services separately.

On macOS/Linux:

chmod +x start_app.sh
./start_app.sh

Or run the components separately:

livekit-server --dev

Then, in another terminal:

cd backend
uv run python src/agent.py dev

And in another terminal:

cd frontend
pnpm dev

Once everything is running, open:

http://localhost:3000

7. Start a Conversation

Open the frontend in your browser and click Start talking.

Allow microphone access when your browser asks for permission and start speaking.

The basic voice pipeline works like this:

** User speaks → Deepgram STT → LLM → Murf Falcon TTS → LiveKit → User hears the response**

LiveKit handles the real-time audio transport while the backend agent coordinates the voice pipeline.

8. Customize the Agent

Once the starter project is running, the interesting part begins.

The system prompt can be changed in:

backend/src/agent.py

This allows you to turn the starter into different types of agents, such as:

Customer support
Language tutor
Receptionist
Personal assistant
Domain-specific voice agent

For Aira, I went much further than changing the prompt. I added memory, tools, outbound calling, user-specific data handling, escalation, analytics, and specialist handoff around the core voice pipeline.

That's the part that turned the starter into Aira.

9. Understand the Core Architecture

At its simplest, a real-time voice agent needs four major pieces:

Speech-to-Text (STT)
Converts the user's speech into text.

LLM
Understands the request, reasons about the conversation, and generates a response.

Text-to-Speech (TTS)
Converts the response back into natural-sounding speech.

Real-Time Transport
Moves audio between the user and the agent with low latency.

In Aira, these pieces come together through:

Deepgram + Gemini + Murf Falcon + LiveKit

And then the rest of the application is built around that core.

And Then I Realized Something About Learning

The biggest thing Aira gave me wasn't just a project.

It was proof that I can learn things I initially thought were beyond me.

When I started, I didn't know everything I needed to know.

And honestly, I still don't.

But I know much more now because I was willing to sit with the things I didn't understand.

That's something I want to remember beyond this project.

You don't become capable by waiting until you feel capable.

You become capable by doing difficult things while you're still unsure of yourself.

Take the risk.

Start the project.

Apply even when you don't feel completely ready.

Try the thing that looks too difficult.

Because sometimes you don't need confidence before taking the first step.

Sometimes confidence is what you gain after taking it.

Stop Letting Other People's Opinions Define Your Limits

There is another lesson I've been thinking about throughout this journey.

Stop believing every opinion people have about you.

People will have opinions.

They'll decide what you're good at.

What you're not good at.

What you can do.

What you probably can't do.

How far they think you can go.

And sometimes, without even realizing it, we start accepting those opinions as facts.

But someone else's opinion is not a measurement of your potential.

People can put a limit on you because that's how far they can see.

That doesn't mean that's how far you can go.

So listen to advice.

Learn from people.

Accept genuine criticism.

But don't hand someone else the authority to decide what you're capable of.

You are not limited to someone else's imagination of you.

Your Mind Is a Lot Like an AI

One of the things I find interesting about building AI is how much it makes me think about the human mind.

An AI model learns from what it is given.

The data.

The examples.

The patterns.

The feedback.

The training.

And in a way, our minds work similarly.

What we repeatedly feed ourselves matters.

The thoughts we keep repeating.

The people we constantly listen to.

The things we tell ourselves.

The challenges we expose ourselves to.

The knowledge we choose to consume.

They all shape the way we think.

Humans are obviously far more complex than AI models, but the analogy still reminds me of something important:

Be careful about what you repeatedly feed your mind.

Train it to question.

Train it to learn.

Train it to try again.

Train it to see possibilities instead of automatically accepting limitations.

Because what you repeatedly put into a system can influence what eventually comes out.

What Aira Has Become

When I look at the project now, it's very different from where it started.

It began with:

"Let's make a voice assistant."

Now it's closer to:

"Let's build an AI system that can interact, remember, act, communicate through phone calls, measure its performance, and know when to involve a human."

But somewhere along the way, it also became:

"Let's see what I am actually capable of building."

And that's probably my biggest takeaway.

The interesting part of AI isn't always the model itself.

It's everything you build around the model.

The memory.

The tools.

The APIs.

The user identity.

The telephony.

The analytics.

The escalation logic.

The interface.

And the person behind the screen who is willing to keep learning when things don't work.

🤍 What's Next?

Aira is still a work in progress.

There are many things I want to improve next:

Better conversation summaries
More useful tools
More robust specialist routing
Stronger authentication
Improved analytics
Better error handling
More natural multilingual conversations
A more polished mobile experience

There is still so much I don't know.

And strangely, I'm not as scared of that anymore.

Because now I know that not knowing something doesn't mean I can't learn it.

That's probably the biggest difference between the person who started this project and the person writing this blog.

I started by asking:

"Can I actually do this?"

Now I'm asking:

"What should I build next?"

And I think that's progress.

From a basic voice agent to:

Memory + Tools + Telephony + Analytics + Escalation + Human Handoff + A Complete Interface

But beyond all of that, Aira gave me something I didn't expect.

A little more belief in myself.

So if you're standing at the beginning of something and thinking:

"I don't know enough."
"This looks too difficult."
"What if I fail?"
"Maybe I'm not capable of doing this."

Start anyway.

You don't have to know everything.

You just have to be willing to learn the next thing.