The latest articles on DEV Community by Achintya Sharma (@sharmaachintya). https://dev.to/sharmaachintya https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3822740%2F8814dcb3-3d0c-4c0f-9025-4495adfb0636.png https://dev.to/sharmaachintya en Achintya Sharma Fri, 13 Mar 2026 17:14:35 +0000 https://dev.to/sharmaachintya/building-eyeguide-a-real-time-ai-visual-companion-for-the-blind-with-gemini-live-api-4ab7 https://dev.to/sharmaachintya/building-eyeguide-a-real-time-ai-visual-companion-for-the-blind-with-gemini-live-api-4ab7 <p><em>This blog post was created for the purposes of entering the Gemini Live Agent Challenge hackathon. #GeminiLiveAgentChallenge</em></p> <h2> The Problem </h2> <p>285 million people worldwide are visually impaired. While screen readers help with digital interfaces, nothing helps them "see" the physical world around them. I wanted to change that.</p> <h2> The Solution: EyeGuide </h2> <p>EyeGuide is a real-time AI companion that sees through a phone camera and talks naturally with the user. It describes surroundings, reads text, warns about hazards, and can be interrupted mid-sentence — just like talking to a friend.</p> <h2> Tech Stack </h2> <ul> <li> <strong>Gemini 2.5 Flash Native Audio</strong> — for real-time bidirectional audio + vision</li> <li> <strong>Google ADK (Agent Development Kit)</strong> — bidi-streaming runtime with LiveRequestQueue</li> <li> <strong>FastAPI + WebSocket</strong> — real-time server communication</li> <li> <strong>Google Cloud Run</strong> — serverless deployment</li> <li> <strong>Google Cloud Firestore</strong> — user preferences</li> </ul> <h2> How I Built It </h2> <p>The core challenge was connecting the browser's camera and microphone to Google's Gemini Live API in real-time. Here's the flow:</p> <ol> <li>Browser captures audio (16kHz PCM) and camera frames (1 FPS JPEG)</li> <li>WebSocket sends both to a FastAPI backend on Cloud Run</li> <li>ADK's <code>LiveRequestQueue</code> feeds data to Gemini's Live API</li> <li>Gemini processes audio + video simultaneously and responds with voice</li> <li>Audio response streams back through WebSocket to the browser</li> </ol> <p>The most impressive part? <strong>Barge-in support</strong> — users can interrupt the AI mid-sentence, and Gemini's built-in Voice Activity Detection handles it seamlessly.</p> <h2> Key Learnings </h2> <ol> <li>Only <code>gemini-2.5-flash-native-audio-latest</code> supports <code>bidiGenerateContent</code> on the Google AI API</li> <li>ADK's <code>RunConfig</code> with <code>StreamingMode.BIDI</code> is the correct way to configure live streaming</li> <li>1 FPS video is surprisingly sufficient for scene understanding</li> <li>System prompt engineering makes or breaks a voice agent's persona</li> </ol> <h2> Try It </h2> <ul> <li> <strong>Live App</strong>: <a href="https://eyeguide-966189115030.us-central1.run.app/" rel="noopener noreferrer">https://eyeguide-966189115030.us-central1.run.app/</a> </li> <li> <strong>GitHub</strong>: <a href="https://github.com/sharmaachintya/EyeGuide" rel="noopener noreferrer">https://github.com/sharmaachintya/EyeGuide</a> </li> </ul> <p><em>Created for the Gemini Live Agent Challenge hackathon. #GeminiLiveAgentChallenge</em></p> a11y devchallenge gemini showdev