DEV Community

Cover image for Sutham : Building a Real-Time Multimodal AI Agent with Gemini Live & Gemma 4
Ankit Kumar Surana
Ankit Kumar Surana

Posted on

Sutham : Building a Real-Time Multimodal AI Agent with Gemini Live & Gemma 4

#devchallenge #weekendchallenge #ai #showdev

DEV Weekend Challenge: Passion Edition Submission

This is a submission for Weekend Challenge: Passion Edition

What I Built

Waste sorting is confusing, especially in India where rules are hyper-local (like Bengaluru's strict 6-bin system) and multi-layered packaging is common. I built Sutham (meaning "clean" in Tamil), an adaptive, continuous multimodal AI agent that acts as your personal waste-sorting buddy.

Sutham isn't a simple "snap a photo and get a label" app. It runs a full autonomous loop (sense -> decide -> act -> check), asks clarifying questions when it's unsure, and literally learns from its mistakes on your device so it never fails the same way twice.

It features an Adaptive Dual-Engine:

  1. Fully Offline (Local-First): When you have no network, Sutham runs entirely on-device using a local vision model. It can classify waste, ask follow-up questions, and map to local rules without ever hitting a server.
  2. Live Multimodal Mode: When online, it opens a continuous audio/video WebSocket to the cloud. You can hold your phone, point it at a pile of waste, and just have a normal conversation. It speaks natively in English, Hindi, Kannada, Tamil, and Telugu. It can even proactively warn you—if you pan the camera across a battery in silence, Sutham will interrupt to say, "Whoa, I see a battery. Don't put that in the bin, I've mapped a drop-off kiosk for you."

Demo

Code

GitHub logo AnkitSurana / sutham-app

Sutham helps Indian households sort waste correctly, in any language, with or without internet. It speaks to you live through the camera and warns you about e-waste and hazardous items before you ask. Learns from your corrections.

Sutham

An on-device, agentic waste-sorting assistant for everyone.

Point your phone at any waste item and Sutham identifies it, reasons about its material, and tells you the correct BBMP bin (Wet / Dry / Reject / Sanitary / E-waste / Hazardous) and how to prepare it. It runs the full sense -> decide -> act -> check loop: it asks when it is unsure, defers e-waste and hazardous items to a human drop-off, and learns your corrections so the next scan of the same item is instant - all on-device, no signal required.

Built for the Google DeepMind Bangalore Hackathon:

  • Best Use of Gemma 4 (local-first agents) - the offline agent loop on on-device Gemma 4 E2B (vision).
  • PS1 Real-Time Multimodal - a live voice-to-voice mode on the Gemini Live API where you talk to Sutham and show it your bin.

Adaptive engine

The app picks its brain by connectivity…

View on GitHub

How I Built It

I built the mobile application using Flutter (Dart) to ensure a smooth, native-feeling camera and audio experience across devices.

Here are the most interesting technical decisions I made:

  • Custom PCM Audio & WebSockets: For the Live Mode, I couldn't rely on basic turn-based REST APIs. I built a custom live_controller.dart that manages a continuous WebSocket connection. It captures 16kHz audio from the mic, sends it via base64 chunks alongside ~1FPS silent video frames, and decodes incoming 24kHz PCM audio chunks to play the AI's voice natively.
  • Barge-In State Management: True conversational AI needs to handle interruptions. If Sutham is speaking and the user talks over it ("Wait, what about the cap?"), the app instantly drops the current playback buffer and pivots context so it doesn't sound robotic or echo itself.
  • Proactive Vision Ticks: Most AI only speaks when spoken to. I implemented a _proactiveTick timer. Even if the user is silent, the app nudges the vision model every 5 seconds with the current camera frame. This allows Sutham to act as a true ambient buddy, catching hazards (like e-waste) completely unprompted.
  • On-Device Learning Memory: Instead of forcing complex RAG, I built a local learning_store.dart. When Sutham proposes a bin and the user corrects it (e.g., "No, this specific cup is Reject waste"), it saves that specific signature locally. The next time the user scans that exact item, the response is instant and corrected.

Prize Categories

Best Use of Google AI
I heavily utilized the absolute bleeding-edge of the Google AI stack to make Sutham adaptive:

  • Gemma 4 E2B (On-Device Vision): Powers the offline local-first agent loop via flutter_gemma.
  • Gemini 3.1 Flash Live: Powers the continuous, sub-second multimodal (voice-to-voice + video) streaming engine.
  • Gemini 3.5 Flash: Used as a fallback for high-fidelity photo reasoning when full Live streaming isn't needed.

Top comments (0)