DEV Community

omkar
omkar

Posted on

Building NagarDrishti: A Live AI Agent for Road Safety in India

#blog

This content was created for the Gemini Live Agent Challenge hackathon. #GeminiLiveAgentChallenge

The Problem: Monsoon Potholes Kill

Pune, India — 7 million people, 2,273 kilometers of roads. During monsoon season, potholes and waterlogging cause hundreds of accidents every year. In October 2025, the Bombay High Court mandated that every pothole must be repaired within 48 hours after multiple fatalities, with personal liability on senior officials for delays.

Yet citizens have no efficient way to report hazards, and municipal officers lack real-time situational awareness.

The Solution: NagarDrishti (City Vision)

We built NagarDrishti — an AI-powered platform where citizens report road hazards through video or natural voice conversation, and municipal officers manage them through an intelligent dashboard with AI chat capabilities.

The Live Agent: Commute Mode

The heart of NagarDrishti is Commute Mode — a hands-free, always-listening voice agent powered by Gemini Live API.

How It Works

  1. Tap once to activate Commute Mode while driving
  2. Speak naturally — "There's a big pothole ahead"
  3. AI responds instantly via audio — "Got it. Pothole logged. Stay safe."
  4. Report created with GPS, severity, and jurisdiction — all without touching your phone

What Makes It "Live"

This isn't a chatbot. It's a real-time AI agent with:

  • Bidirectional audio streaming via WebSocket (not turn-based text)
  • Voice Activity Detection tuned for noisy traffic (800ms silence threshold)
  • Function calling during live audio — creates database records while you speak
  • Interruptible — you can speak mid-response, agent handles gracefully
  • Context-aware — knows your GPS, nearby hazards, previous reports this session
  • Multilingual — English, Hindi, Marathi system prompts

Response latency: 2–4 seconds from speech to AI audio reply.

The AI Pipeline: 5 Steps in <15 Seconds

When a citizen submits a video report, the AI pipeline processes it automatically:

  1. Classification — Gemini Vision analyzes video frames → hazard type + severity (1–5)
  2. Jurisdiction Routing — GPS-based routing to PMC/PCMC/PMRDA/Cantonment
  3. Deduplication — BigQuery spatial query (20m radius) + Gemini LLM similarity
  4. Weather Intelligence — Open-Meteo 48h rain forecast → severity multiplier (1.0×–2.0×)
  5. Dimension Estimation — Gemini Vision estimates length/width/depth + repair cost

Result: Verified report in BigQuery, ready for officer action.

The Dashboard: AI Chat for Officers

Municipal officers use the PMC Dashboard — a Next.js web app with:

  • Live heatmap with severity-colored dots (Mapbox)
  • AI Chat powered by Gemini with 11 BigQuery function-calling tools
  • Agent-as-tool pattern — generate_ward_report is a sub-agent that queries BigQuery + Gemini to produce structured ward reports on demand

Example Queries

Officer: "Which areas have the most critical hazards?"

AI: (queries BigQuery) "Dapodi ward has 12 severity-4 potholes, Baner has 8..."

Officer: "Generate report for ward 15"

AI: (invokes sub-agent) "Ward 15 Report: 23 active hazards, 18 overdue, top type: waterlogging..."

Real Data, Real Roads

Every report in the system is from actual road hazards in Pune — no synthetic or mock data.

Our team physically traveled across Pune on bikes to collect 24+ real field reports from:

  • PCMC: Dapodi, Sangvi, Pimple Saudagar, Chinchwad
  • PMC: Aundh, Baner, Kothrud, Pashan

We drove through these areas and recorded hazards using both video capture and Commute Mode voice reporting.

Technical Highlights

Google Cloud Stack

  • Cloud Run — FastAPI backend + Next.js frontend (asia-south1)
  • BigQuery — Hazard reports, users, verifications (45 reports currently)
  • Cloud Storage — Video evidence, thumbnails (nagardrishti-media bucket)
  • Terraform — Infrastructure-as-Code for all GCP resources

Gemini Models

  • Gemini 3 Flash (gemini-3-flash-preview) — Vision + Text classification
  • Gemini 2.5 Flash Native Audio (gemini-2.5-flash-native-audio-preview-12-2025) — Live API audio streaming

Mobile App

  • Flutter (Android) with Riverpod state management
  • Features: Video capture (10s auto-stop), Commute Mode, heatmap, leaderboard, proximity alerts
  • Gamification: Points system (+10 report, +5 verify, +2 endorse) with tier progression (Bronze → Diamond)

Challenges We Solved

1. Gemini Live API Latency

Problem: Initial voice response took 10–15 seconds.

Solution: Tuned VAD parameters:

  • silence_duration: 1800ms → 800ms
  • prefix_padding: 300ms
  • Disabled dynamic thinking in Live API config

Result: 2–4 second response latency.

2. Duplicate Detection at Scale

Problem: Simple distance-based matching caused false positives.

Solution: Combined BigQuery spatial query (20m radius) + Gemini LLM comparison of hazard descriptions + hazard type matching.

Result: Precision deduplication with community confirmation boost.

3. Jurisdiction Routing Accuracy

Problem: GPS coordinates near municipal boundaries gave incorrect assignments.

Solution: Deterministic polygon-based routing instead of AI classification.

Result: 100% accurate jurisdiction assignment.

What We Learned

  • Gemini Live API's native audio capabilities are remarkably good at understanding Indian English accents and Hindi/Marathi mixed speech
  • Voice Activity Detection tuning is critical for real-world driving conditions — default settings are designed for quiet environments
  • BigQuery's serverless architecture handles the read-heavy pattern of a dashboard perfectly — no infrastructure management needed
  • Real field data collection is irreplaceable — AI models behave very differently with actual monsoon road conditions vs. clean test videos

Try It Yourself

Demo Credentials

PMC Dashboard (Web):

  • Email: admin@pmc.gov.in
  • Password: pmc2026

Mobile App:

  • Email: test@email.com
  • Password: test123

What's Next

  • FCM push notifications when report status changes
  • Multi-city expansion beyond Pune (Mumbai, Nagpur)
  • Integration with PMC's existing grievance portal (SARATHI)
  • Community verification rewards with blockchain-anchored proof

Built With

Python • FastAPI • Flutter • Next.js • TypeScript • TailwindCSS • Google Gemini API • Gemini Live API • Google GenAI SDK • Google Cloud Run • Google BigQuery • Google Cloud Storage • Terraform • Docker • Mapbox

This project was built for the Gemini Live Agent Challenge hackathon. #GeminiLiveAgentChallenge

Project Category: Live Agents 🗣️

Submission Date: March 2026

Top comments (0)