DEV Community: Mayank Sharma

Uttar: grounded reply drafts for small sellers, on an open-weight model

Mayank Sharma — Mon, 19 Oct 2026 04:56:20 +0000

Uttar (उत्तर — "answer") drafts replies for small Instagram/WhatsApp sellers who answer the same ten questions all day: price? COD? Jorhat kitne din? size M hai? Paste the customer message, get 2–3 grounded drafts in the seller's own language and voice. The seller taps copy and sends it themselves.

The core loop:

Seller pastes a customer message: "bhaiya ye muga wala dupatta kitne ka hai? delivery ghar tak hoga?"
Uttar fuzzy-matches 1–3 products from the seller's own catalog (so the model never sees the whole price list — a small model stays accurate when you only show it what matters).
Ollama runs gemma3:4b with the matched facts, the shop's delivery/payment/return rules, and 20 of the seller's real past replies as few-shot examples.
Output: "Namaste bhaiya! Muga wala dupatta ₹2,400 ka hai. Jorhat tak delivery hoti hai, 3-4 din lagte hain ₹80 charge se…" — price from the catalog, delivery days from the rules, voice from the seller's own history.
Never auto-sends. The seller reviews, edits if needed, copies, sends from WhatsApp/Instagram themselves.

If the answer isn't in the catalog, it says NEEDS_HUMAN and flags the message instead of inventing. "Border pe custom design bana sakte ho?" → refused, because the seller never told Uttar anything about customization. That refusal is the trust feature: replies can only come from facts the seller entered.

Why the open pieces are what make it work

The cost math only closes with a self-hosted model. The target price is ₹299–499/month per shop. Each draft request costs ~2,500–3,000 input tokens — not because the app is bloated, but because grounding is what makes a 4B model accurate: matched catalog slice + rules + 20 voice examples. At 3,000 messages/month (a busy shop, ~100/day):

GPT-4o-class pricing: ~9M input + 0.9M output ≈ $30+/month — the model alone costs ~10× the plan price.
Cheap mini-class pricing: ~$1.90/month — sounds fine until you add hosting for the app, and the margin on ₹299 evaporates while you still don't control Hinglish quality.
gemma3:4b on a small server: ₹0 marginal. The few-shot grounding that makes replies accurate is free when tokens cost nothing. Per-token pricing punishes exactly the technique that keeps a small model honest.

Customer data never leaves the machine. These are screenshots of people's order conversations — names, phone numbers, addresses. In Uttar, the catalog, the past replies, and every customer message go to localhost Ollama and nowhere else. There's no third-party API in the loop at all.

Swappable and tunable — which I needed within one day. The contest's "pick whichever handles Hinglish best in your tests" is not a slogan here:

Model	Accuracy (10 fixed Hinglish/Assamese msgs)	Warm latency	Verdict
gemma3:4b	9/10	~11–12s	Default — same accuracy, ~3× faster
qwen2.5:7b	9/10	~30–35s	Better Assamese voice ("Dikhim kya chahiye?"), too slow for paste-and-reply

Two weeks after launch I could swap the model behind a dropdown, fine-tune few-shots per region, or change refusal behavior — without asking anyone's API team. Try telling a closed API to change its refusal style for your users.

Where the open approach bit me — and what I changed

I'm keeping the failures in, because they're the interesting part.

The model kept inventing policies. First prompt: ask "border customize kar sakte hain?" and gemma would cheerfully reply "sorry, hum customize nahi karte" — except the seller never said that. A "no" not in your facts is just as fabricated as a fake price. Fixes, in order:

Hard rule: "answering 'we don't do customization' when no policy is listed is WRONG, because that 'no' is also an invented fact. Silence = NEEDS_HUMAN" — with negative examples in the prompt.
Deterministic guard (lib/guard.js): regex layer for known-unknown intents (customize, repair, wholesale…) that runs before the model. Small open models are inconsistent at refusal; a rules layer is not. Honest architecture: model for language, code for guarantees.
Rule-category matching so pure policy questions ("COD hai kya?" — no product tokens at all) reach the rules instead of dying at the fuzzy-match gate.

Measured results (20-message test set, gemma3:4b, scripts/realtest.js):

19/20 correct behavior — 15/15 in-catalog questions drafted, 4/5 refusals correct
9/9 price/fact checks passed — every number in every draft came from the catalog
Known failure: "dam koto muga orutia?" — "orutia" isn't in the catalog, and the model offers alternatives (muga scarf, dupatta) instead of a clean refusal. Arguably friendly, but it's the model guessing at product knowledge it doesn't have. It's also where Assamese product names in Roman script are weakest.
One draft needed an edit: a returns answer appended "₹50 COD fee return ke time lagta hai" — stitched from two unrelated facts. Great demo of why a human approves every reply.

Honest caveat: it does not run on the seller's phone

Most sellers live on phones, and running a 4B model on a phone isn't realistic yet. The setup is your server running Ollama (a $5–10 VPS or your laptop for one shop). That still gives: zero marginal cost, no vendor lock-in, tuning freedom, and data staying off somebody else's server. What it does not give: fully offline-on-device. Saying so plainly seemed more useful than a demo nobody's hardware can run.

There's also no WhatsApp/Instagram API integration, deliberately. Official APIs are heavy, and scraping gets shops banned. Copy-paste is the MVP: the seller keeps their fingers on the send button, so nothing about this trips automation rules.

Run it

git clone https://github.com/mayankinkk/utar.git && cd utar
brew install ollama && brew services start ollama
ollama pull gemma3:4b
npm install && npm run dev   # http://localhost:3000

First run seeds a demo shop (Muga Ghar, Guwahati): 8 products, delivery/payment/return rules, 12 voice examples. The Setup tab is where a real seller enters their own list. It's a PWA — install it on a phone and it opens like an app.

Stack: Next.js (JavaScript), Tailwind, better-sqlite3, Ollama. Fuzzy catalog matching is token scoring, no vector DB — for a 200-product shop, embeddings are a dependency you don't need.

First user

Pending — to be filled with a real seller's ratings: "X of 20 send-as-is / edit / useless".

Built for Hacktoberfest Open-Source AI Challenge Week 2 (Side Hustle). Agent session replay: [paste DevRelay link here]

I built an AI that gives you one outdoor mission, then shuts up (Gemma + Ollama, runs locally)

Mayank Sharma — Mon, 12 Oct 2026 05:26:20 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Offscreen is a tiny app with one rule: the screen should be the shortest part of the experience.

You tell it how much time you have (15 to 60 minutes). It checks your local weather and sunset time, and a local open model (Gemma via Ollama) writes one small, specific outdoor mission. Not "go for a walk". Something like a shadow hunt: find five interesting shadows and photograph them. Then you tap "Speak & go", it reads the mission out loud, and you close the laptop.

It's for anyone who opens their phone to "take a break" and ends up in a scroll hole. Generic advice like "go touch grass" never works for me. A concrete 15-minute task does.

Demo

Code

mayankinkk / offscreen

offscreen — Hacktoberfest 2026 Week 1: Touch Grass

One screen, 30 seconds, then go outside.

A local-first offscreen agent. You tell it how much time you have (15–60 min), it checks weather + sunset and a local open model (Gemma via Ollama) generates one small, specific outdoor mission instead of “go for a walk” — then goes quiet. The screen is the shortest part.

Challenge prompt: Build something with open-source AI at its core that gets people off the screen and into the world.

Entry: offscreen/ — npm run dev at http://localhost:5173/
Challenge hub: https://dev.to/hacktoberfest
Dates: Oct 05–11, 2026

Why this one wins the prompt

Not crowded: The prompt lists bird ID, garden planner, route builder — those will be flooded. A mission generator with a short-screen-time philosophy is a different angle.
Real “why open” story: Your location, schedule and habits never leave your laptop, it works offline once the…

View on GitHub

How I Built It

Gemma 2B via Ollama runs on my laptop and generates the mission. The prompt is time-aware: a 15-minute mission has to stay within a 5-minute walk, a 60-minute one can go further. The model returns JSON, which I parse and de-duplicate against my last 8 missions so it doesn't repeat itself.
Open-Meteo for weather and sunrise/sunset. No API key needed.
Reverse geocoding (BigDataCloud with a Nominatim fallback) so the app says "your city" instead of raw coordinates.
ElevenLabs reads the mission aloud, with the browser's Web Speech API as a free fallback. This is the "eyes off the screen" part.
React + Vite for the UI, and a localStorage mission log where I add photos and notes after each mission, with JSON/Markdown export.
Fallback generator: a hand-written pool of 16+ missions across time, weather and time-of-day, so the app still works if Ollama isn't running.

Field Test (the part that matters)

[FILL THIS IN FROM YOUR field-log.md. Be specific and honest:]

Which missions did you actually do, and where?
One mission that was great (what happened?)
One mission the model got wrong or that felt cringe (be honest, this is what readers and judges respond to)
Did the "speak and close the laptop" flow actually get you out the door?

Why Does Open Innovation Matter?

Offscreen needs your location, your free time, and your daily routine. That's exactly the data I don't want sitting on someone else's server.

Privacy: the model runs on localhost. The mission is generated on my machine. The only network calls are weather and place-name lookups.
Cost: $0 to run, no per-call billing, no rate limits.
Control: I can swap models and tune the prompt however I want. When Gemma gave bland missions, I changed the prompt, not a vendor's settings.
Honest limitation: mission generation works with no internet once the model is pulled, but the weather and place lookups need a connection. Without one, the app falls back to my built-in mission pool. So it's local-first, not fully offline.

Prize Categories

Best Use of Gemma
Best Use of ElevenLabs

I built a private WhatsApp digest for my friend who mutes every college group

Mayank Sharma — Mon, 05 Oct 2026 05:12:29 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

My friend Mimi mutes every college WhatsApp group. It's not that she doesn't care, it's that 400 messages a day is unreadable. The result: missed deadlines, missed notices, and the classic "wait, the form was due yesterday?"

So I built WhatsApp Group Digest, a small tool that takes an exported WhatsApp chat and turns it into a short, readable digest:

Deadlines and assignments with dates resolved properly (not just "tomorrow")
Exams and action items that actually need a response
Important updates and a TL;DR so the rest of the chat can stay muted
"What did I miss?", a weekly catch-up with key announcements, missed deadlines, events, and the occasional drama

It runs fully on your own machine. The chat never leaves the laptop.

Demo

To try it: export a group chat from WhatsApp (group name → Export chat → Without media), upload the .txt file on the web page, and get your digest.

Code

mayankinkk / whatsapp-digest

WhatsApp Group Digest

For the friend who mutes every college group. Upload a WhatsApp export and get a digest of deadlines, assignments, exams, and action items — powered entirely by open-source AI running locally via Ollama.

Why open?

Private: Your chat never leaves your machine. No cloud, no server, no data collection.
Offline: Works with no internet connection once the model is pulled.
Free: Zero cost to run — no API keys, no usage limits.
Swappable: Run Gemma, Qwen, Llama — any open-weight model you choose.

Quick start

# 1. Install Ollama (if you haven't)
curl -fsSL https://ollama.com/install.sh | sh

# 2. Pull a model
ollama pull gemma3:4b

# 3. Start Ollama
ollama serve &

# 4. Install and run the app
npm install
npm start

Open http://localhost:3000.

Export a chat from WhatsApp

Open the group → tap the group name
Export chat →…

View on GitHub

How I Built It

The whole thing is deliberately small:

Model: Gemma 3 (4B), an open-weight model, running locally through Ollama. The model is configurable with one environment variable, so swapping in Qwen or Llama is a one-line change (OLLAMA_MODEL=qwen2.5:7b npm start).
Backend: Node.js + Express. It parses the WhatsApp .txt export, extracts messages with their timestamps and senders, and sends them to Ollama's local API with a prompt asking for a structured digest.
Frontend: Vanilla JS + HTML/CSS. No framework, just an upload box and the results. The page also shows basic stats: message count, senders, and date range.

The date problem. Chat messages are full of relative dates: "submit by tomorrow", "exam on Friday", "form closes tonight". Without knowing when the message was sent, those mean nothing, and a small model will happily guess. So the backend passes each message's timestamp along with its text, and the prompt asks the model to resolve relative dates against it. That way "tomorrow" becomes an actual calendar date in the digest instead of a vague word.

Why Does Open Innovation Matter?

Group chats are some of the most private text people have: names, plans, and personal conversations mixed in with the notices. Sending that to a closed API just to get a summary didn't feel right, and it isn't my call to make on behalf of everyone else in those groups.

Running an open-weight model locally gave me things a closed API couldn't:

Private: The chat never leaves the machine. No cloud, no server, no data collection.
Offline: Once the model is pulled, it works with no internet connection.
Free: No API keys, no usage limits, no subscription for a college student to worry about.
Swappable: Gemma, Qwen, Llama, any open-weight model works. If a better one comes out next month, I change one variable.

A closed API would have been quicker to set up, but it couldn't make the privacy promise, and for this project that promise is the whole point.

Prize Categories

Best Use of Gemma: the digest is generated by Gemma 3 running locally through Ollama.

Building RupeeGPT: A Multilingual Voice AI Financial Assistant for Bharat

Mayank Sharma — Sat, 15 Aug 2026 16:51:27 +0000

How I Built RupeeGPT: A Voice-First AI Financial Assistant for India in 10 Days

#VoiceForBharat | Built with the fastest TTS API — Murf Falcon | 10 Days of Voice Agents

Ten days ago, I started with a blank repo and a challenge: build a production-ready voice AI agent for Indian users — one that could speak naturally in English, Hindi, and Hinglish; remember returning callers; escalate to humans when things got serious; and hand off conversations to specialist agents without ever making the caller repeat themselves.

What came out the other side is RupeeGPT — a conversational AI financial assistant that helps any Indian user navigate banking, UPI, government welfare schemes, loans, and financial safety. Here's everything I built, what broke, how I fixed it, and how you can build your own.

The Problem: Finance Advice Is Inaccessible to Most Indians

India has over 500 million smartphone users, but financial literacy remains a barrier for hundreds of millions of people — especially in tier-2 and tier-3 cities and rural areas. The information exists: government scheme portals, RBI guidelines, banking apps. But it's buried in bureaucratic language, English-only interfaces, and long PDF documents.

A voice agent changes that. You don't need to read anything. You don't need to know the right portal URL. You just talk.

Who it's for: First-generation bank account holders, rural farmers checking PM Kisan eligibility, street vendors exploring PM SVANidhi loans, anyone who's ever been told to "read the fine print" and couldn't.

Why voice: Voice meets people where they are. It removes the literacy barrier, it's faster than navigating apps, and for many rural users, calling is the most intuitive interface they know.

The Architecture

🎙️ User speaks
    → Deepgram STT (nova-3, multilingual)
    → Gemini LLM (gemini-3.5-flash-lite via LiveKit Inference)
    → Murf Falcon TTS (Anisha — Indian English, en-IN)
    → LiveKit real-time transport
    → 🔊 User hears

The stack:

Backend: Python 3.12, LiveKit Agents SDK, uv for dependency management
Frontend: Next.js 14 (App Router), TypeScript, Tailwind
Memory: MongoDB Atlas (persistent caller profiles)
Transport: LiveKit (WebRTC)
TTS: Murf Falcon — more on why this matters below

Feature 1: An Indian Voice That Actually Sounds Indian

The default for most TTS-backed voice agents is a US English voice. For an Indian user asking about PM Kisan Samman Nidhi, hearing a generic American accent reading scheme names in English phonetics feels jarring and impersonal.

Murf Falcon's Anisha voice — Indian English, en-IN, Conversation style — changes this completely. But there was a subtlety: even with an Indian voice, scheme names like "PM Kisan Samman Nidhi" or "Pradhan Mantri Jan Dhan Yojana" are read with English phonetics when spelled in Roman script.

My fix: a TTS pronunciation layer (tts_hindi.py)

_ENGLISH_TO_HINDI: tuple[tuple[str, str], ...] = (
    ("pm kisan samman nidhi",           "पीएम किसान सम्मान निधि"),
    ("pm jan dhan yojana",              "पीएम जन धन योजना"),
    ("pradhan mantri jan dhan yojana",  "प्रधानमंत्री जन धन योजना"),
    ("pm svanidhi",                     "पीएम स्वनिधि"),
    ("aadhaar",                         "आधार"),
    ("yojana",                          "योजना"),
    # ... more
)

Before any text reaches Murf Falcon, it passes through this whitelist rewriter. Known Hindi/Indian terms are converted to Devanagari, so the voice says "पीएम किसान सम्मान निधि" — exactly as a native speaker would say it on TV — instead of "P M Kisan Samman Nidhi" with English stress patterns.

The rewriter is safe to apply for every language mode: a pure-English sentence with none of these terms passes through byte-for-byte unchanged. I also built a detect_language() function that classifies each user utterance as english, hindi, or hinglish using Devanagari character detection and a curated Hinglish marker word list:

HINGLISH_MARKERS = ("mujhe", "kaise", "kya", "chahiye", "baat", "namaste",
                    "yojana", "sarkari", "paise", "rupaye", "bharat", ...)

The agent mirrors the caller's language — answers in Hindi if they speak Hindi, Hinglish if they code-switch — without ever asking them to repeat.

Feature 2: Personality, Objectives, and Safety Guardrails

The system prompt defines the entire character of RupeeGPT: what it will help with, what it refuses, and how it escalates.

Key guardrails baked into the system prompt:

Never ask for OTPs, PINs, passwords, or Aadhaar/PAN numbers — ever, for any reason
Never guarantee loan approval, scheme eligibility, or returns
Never impersonate bank officials or government employees
Two mandatory escalation triggers: suspected fraud/unauthorized transactions, and official decision overrides (e.g., custom loan limit requests)

For the two escalation scenarios, the agent must:

Stop assisting and explain the situation
Name exactly what information it will share
Get explicit spoken consent before proceeding
Call create_escalation() only after consent

This pattern — ask before acting, require a clear YES — became a design principle throughout the whole project.

Feature 3: Multilingual TTS — English, Hindi, and Hinglish

The TTS node hooks into the LiveKit Agents pipeline using Agent.default.tts_node:

async def tts_node(self, text, model_settings):
    language = self._tts_language()

    async def _tracked():
        async for part in tts_hindi.stream_for_tts(text, language=language):
            yield part

    async for frame in Agent.default.tts_node(self, _tracked(), model_settings):
        yield frame

The stream_for_tts function accumulates the LLM's streaming text output into complete sentences before passing each sentence through the Devanagari rewriter. This is important: if a scheme name like "PM Kisan Samman Nidhi" were split across two streamed chunks, the phrase-level rewriter would miss it.

Feature 4: Persistent Memory for Returning Callers

Every caller gets a persistent browser ID (stored in localStorage and passed as a LiveKit participant attribute). The agent reads this at the start of every session and calls lookup_user() to fetch any saved profile from MongoDB.

But here's the part that took the most iteration: consent architecture.

The agent is not allowed to save any personal fact without:

The caller explicitly sharing the fact
The agent asking whether to remember it (naming the exact fact)
The caller saying a clear YES
The agent calling grant_user_memory_consent() with that exact value
Only then calling save_user_memory() with the same value

# Tools must be called in sequence, only after explicit spoken consent:
# 1. grant_user_memory_consent(name="Rahul", ...)
# 2. save_user_memory(name="Rahul", ...)

The save_user_memory tool actively checks the in-session consent store and blocks saves for anything that wasn't consented to in the current call. Returning callers are greeted naturally: "Namaste Rahul, welcome back. Would you like to continue from PM Jan Dhan Yojana?"

The MongoDB document looks like this:

{
  "user_id": "abc123",
  "name": "Rahul",
  "language_preference": "Hinglish",
  "facts": {
    "schemes_checked": ["PM Jan Dhan Yojana"],
    "eligibility_answers": { "income_bracket": "below 3 lakh", "farmer": true }
  },
  "last_interaction": "2026-08-14T10:30:00Z"
}

Feature 5: Tools That Fetch Real Data

Three function-calling tools give the agent live (or near-live) data:

find_eligible_schemes — Matches the caller's profile (age, state, income, occupation, caste, residence, disability, BPL status) against a local dataset of Indian government welfare schemes. Returns preliminary matches with names, benefits, documents required, and official portal URLs.

get_usd_inr_rate — Live USD/INR exchange rate.

get_lending_rates — Current base lending rates and MCLR data.

The scheme-matching tool uses a careful LLM prompt to avoid hallucination:

It never invents schemes or eligibility criteria
If the result set is empty, it says so clearly
If the tool errors, it says only that it cannot check right now — never speculates

Feature 6: Outbound Phone Calls

Using LiveKit's SIP integration, the agent can place outbound calls to real phone numbers. The session pipeline automatically detects SIP participants and switches the noise cancellation model:

noise_cancellation=lambda params: (
    noise_cancellation.BVCTelephony()
    if params.participant.kind == rtc.ParticipantKind.PARTICIPANT_KIND_SIP
    else noise_cancellation.BVC()
)

BVCTelephony is optimized for the narrowband audio characteristics of SIP/PSTN calls. The call analytics dashboard logs whether each session was a web or sip call.

Feature 7: Human Escalation with a Live Dashboard

When a caller reports suspected fraud or requests a decision override, the agent collects their name, contact number, issue summary, and urgency level — all with explicit consent — then calls create_escalation(). This writes to escalations.json (read by the Next.js frontend) and POSTs to a webhook endpoint.

The Escalation Desk (/demo route) shows open escalations in real time:

Caller name, contact, issue summary, urgency badge
Status: Open / In Progress / Resolved
Reference ID (e.g., ESC-492716) that the agent reads back to the caller

Feature 8: Call Analytics Dashboard

Every call session — web or SIP — is logged on close. The _on_close handler fires when LiveKit closes the room:

def _on_close(ev) -> None:
    call_record = {
        "id": ctx.room.name,
        "created_at": start_time,
        "ended_at": datetime.now(timezone.utc).isoformat(),
        "duration_seconds": round(duration, 2),
        "success": userdata.get("success", False),
        "success_reason": userdata.get("success_reason", ""),
        "call_type": call_type,   # 'web' or 'sip'
        "user_id": userdata.get("user_id", "")
    }
    # Write to calls.json + POST to Next.js API

A call is marked successful when the caller either checks their government scheme eligibility or creates a human escalation.

The /dashboard page shows:

Total calls, success rate, successful calls, failed calls (live-updating every 3s)
Filterable call log table with duration, type badge, timestamp, and outcome

Feature 9: Agent Handoff to a Specialist

This was Day 9, and probably the most elegant feature technically. When a caller needs deep, focused help with government schemes — step-by-step application guidance, documents checklist, portal navigation — the main assistant hands off to a dedicated GovernmentSchemeSpecialist agent:

async def transfer_to_scheme_specialist(self, context: RunContext, ...) -> str:
    specialist = GovernmentSchemeSpecialist(
        chat_ctx=context.session.history.copy()  # full conversation history
    )
    context.session.update_agent(specialist)     # live transition, no interruption
    return "Handoff complete."

The specialist receives the full chat_ctx, so the caller never has to repeat themselves. The specialist introduces itself once, then continues the conversation in-context. It's focused: it only handles government scheme questions, and explicitly declines general banking/UPI questions.

The Hard Parts

1. Streaming text rewriting without splitting phrases

When the LLM streams its reply in chunks, a phrase like "PM Kisan Samman Nidhi" might arrive as "PM Kisan" in one chunk and " Samman Nidhi" in the next. My first implementation fed each chunk directly through the regex rewriter — which meant phrase-boundary splits caused silent failures where terms stayed in Roman script.

Fix: Buffer streamed chunks and only emit text at sentence boundaries (".", "!", "?", "\n"). Since scheme names never cross sentence boundaries, the rewriter always sees the full phrase. Added a safety flush at 512 characters for run-on sentences.

2. The LLM saving memory without consent

Early versions of the memory tools had a subtle problem: the LLM would call save_user_memory in the same turn the caller first mentioned a fact, before any consent was sought. I fixed this with two layers:

Prompt-level: Explicit multi-step instructions in the system prompt
Tool-level: save_user_memory checks an in-session consent dict before writing anything; if the consent key isn't there, it returns a detailed refusal explaining exactly what's missing

This meant the enforcement was in the code, not just in the LLM's instruction-following.

3. Deprecation of `gemini-2.5-flash` mid-challenge

Around Day 7, calls started returning 404 errors. The model gemini-2.5-flash had been deprecated. Migrating to gemini-3.5-flash-lite via the LiveKit Inference plugin fixed it — but it required updating both agent.py and the test harness configuration. Always pin your model versions.

4. Getting consistent Hinglish detection

My first Hinglish marker list was too broad — common words like "hai" appeared in some proper nouns — and too narrow — it missed many common code-switch patterns. I iterated through actual test conversations, adding and removing markers until detect_language() was reliably stable across English, Hindi, and code-switched Hinglish inputs.

How to Build Your Own

The Four Core Components

Component	What it does	Used in this project
STT	Turns speech to text (the ears)	Deepgram nova-3, multilingual
LLM	Generates responses (the brain)	Gemini 3.5 Flash Lite
TTS	Turns text to speech (the voice)	Murf Falcon, Anisha (en-IN)
Transport	Real-time audio	LiveKit (WebRTC)

The key insight: these four components are independent and swappable. You can use any STT, any LLM, any TTS — as long as they're wired through a common agent runtime (LiveKit Agents in this case).

Why Murf Falcon for TTS?

55ms model latency — the agent sounds instant, not laggy
130ms time-to-first-audio — faster than any alternative I benchmarked
150+ voices across 35+ languages including Indian English and Hindi
$0.01/1000 characters — dramatically cheaper than comparable options
Native Indian English voices that sound genuinely natural for Indian users

Quickstart

git clone https://github.com/murf-ai/murf-livekit-starter.git
cd murf-livekit-starter

Set up API keys — never commit these to git:

Create backend/.env.local (copy from backend/.env.example):

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_key
LIVEKIT_API_SECRET=your_secret
MURF_API_KEY=your_murf_key          # murf.ai/api/dashboard
DEEPGRAM_API_KEY=your_deepgram_key  # deepgram.com
GOOGLE_API_KEY=your_google_key      # aistudio.google.com

Create frontend/.env.local (copy from frontend/.env.example):

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_key
LIVEKIT_API_SECRET=your_secret

Install and run:

# Backend (Python)
cd backend
uv sync
uv run python src/agent.py download-files

# Frontend (Node)
cd ../frontend
pnpm install

# Run everything from repo root
chmod +x start_app.sh && ./start_app.sh

Open http://localhost:3000, click Start talking, allow microphone access, and speak.

How to customise

The entire personality lives in one constant at the top of backend/src/agent.py:

SYSTEM_PROMPT = """You are RupeeGPT, a personal AI assistant for Indian users.
...
"""

Change that string and you have a completely different agent. Change the voice:

tts=murf.TTS(voice="Anisha", locale="en-IN", style="Conversation")
# Browse all voices: murf.ai/api/docs/voices-styles/voice-library

Tracing a conversation turn

After ./start_app.sh, check the backend terminal for:

[STT] user said: <transcript> — what Deepgram heard
[LLM] metrics model=gemini-3.5-flash-lite ttft=0.42s — LLM latency
[TTS] metrics ttfb=0.13s — Murf Falcon time-to-first-byte
[CALL LOG] Saved to file ... — session logged to dashboard

Architecture Overview

Browser (Next.js)
    ↕ WebRTC (audio)
  LiveKit Server
    ↕ WebRTC
Python Agent Worker
  ├─ Deepgram STT    (nova-3, multilingual)
  ├─ Gemini LLM      (gemini-3.5-flash-lite)
  ├─ Murf Falcon TTS (Anisha, en-IN)
  ├─ tts_hindi.py    (Devanagari pronunciation rewriter)
  ├─ memory.py       (MongoDB caller profiles)
  ├─ schemes.py      (government scheme matching)
  ├─ telephony/      (SIP outbound calls)
  └─ Function tools:
       lookup_user()
       save_user_memory()
       grant_user_memory_consent()
       find_eligible_schemes()
       get_usd_inr_rate()
       get_lending_rates()
       create_escalation()
       transfer_to_scheme_specialist()

Next.js frontend routes:
  /          Voice agent UI
  /demo      Escalation Desk dashboard
  /dashboard Call Analytics dashboard

What I'd Do Differently

Design strict tool schemas from Day 1. The OpenAI strict schema validator requires every object to declare additionalProperties: false. Retrofitting this was painful. The _pick_arg() helper pattern I built to handle both LLM invocations and test-harness direct calls is something I'd design in from the beginning.
Pin model versions immediately. gemini-2.5-flash deprecating mid-challenge cost me debugging time.
Integration test with real audio early. Unit tests caught logic errors; only real voice sessions caught the chunk-splitting bug in the TTS rewriter.

Repository

🔗 GitHub: github.com/murf-ai/murf-livekit-starter

⚠️ Never publish API keys, phone numbers, caller data, or any private information. Use .env.local (gitignored) for all secrets.

Closing Thoughts

Ten days. One voice agent. Nine features that went from zero to production-ready code:

✅ Indian voice with Murf Falcon (Anisha, en-IN)
✅ Personality, guardrails, and safety rules
✅ English / Hindi / Hinglish support with Devanagari TTS rewriting
✅ Frontend showing agent state
✅ Persistent memory with consent enforcement
✅ Real data tools (government schemes, exchange rates, lending rates)
✅ Outbound phone calls via SIP
✅ Human escalation with a live dashboard
✅ Specialist agent handoff

The most important lesson: voice agents are not just chatbots with audio bolted on. The interaction model is fundamentally different — no markdown, no bullet points, short turns, immediate feedback. You have to design for listening, not reading.

And for Indian users specifically, language flexibility and natural pronunciation are the difference between a tool that feels foreign and one that feels like talking to someone who genuinely gets it.

If you're building in this space, I hope this gives you a useful foundation. The code is open, the architecture is documented, and the patterns — consent-gated memory, language-aware TTS rewriting, live handoffs — are all reusable.

Build something for Bharat. Ship it.

Built during **10 Days of Voice Agents — VoiceForBharat Edition, powered by the fastest TTS API: **Murf Falcon.
Tag @MurfAI | #VoiceForBharat

DEV Community: Mayank Sharma

Uttar: grounded reply drafts for small sellers, on an open-weight model

Why the open pieces are what make it work

Where the open approach bit me — and what I changed

Honest caveat: it does not run on the seller's phone

Run it

First user

I built an AI that gives you one outdoor mission, then shuts up (Gemma + Ollama, runs locally)

What I Built

Demo

Code

mayankinkk / offscreen

offscreen — Hacktoberfest 2026 Week 1: Touch Grass

Why this one wins the prompt

How I Built It

Field Test (the part that matters)

Why Does Open Innovation Matter?

Prize Categories

I built a private WhatsApp digest for my friend who mutes every college group

What I Built

Demo

Code

mayankinkk / whatsapp-digest

WhatsApp Group Digest

Why open?

Quick start

Export a chat from WhatsApp

How I Built It

Why Does Open Innovation Matter?

Prize Categories

Building RupeeGPT: A Multilingual Voice AI Financial Assistant for Bharat

How I Built RupeeGPT: A Voice-First AI Financial Assistant for India in 10 Days

The Problem: Finance Advice Is Inaccessible to Most Indians

The Architecture

Feature 1: An Indian Voice That Actually Sounds Indian

Feature 2: Personality, Objectives, and Safety Guardrails

Feature 3: Multilingual TTS — English, Hindi, and Hinglish

Feature 4: Persistent Memory for Returning Callers

Feature 5: Tools That Fetch Real Data

Feature 6: Outbound Phone Calls

Feature 7: Human Escalation with a Live Dashboard

Feature 8: Call Analytics Dashboard

Feature 9: Agent Handoff to a Specialist

The Hard Parts

1. Streaming text rewriting without splitting phrases

2. The LLM saving memory without consent

3. Deprecation of gemini-2.5-flash mid-challenge

4. Getting consistent Hinglish detection

How to Build Your Own

The Four Core Components

Quickstart

How to customise

Tracing a conversation turn

Architecture Overview

What I'd Do Differently

Repository

Closing Thoughts

3. Deprecation of `gemini-2.5-flash` mid-challenge