Modern digital infrastructure is fundamentally broken for millions of people. Despite strict global regulations, 95% to 98% of the world's top websites still fail basic web accessibility standards. For users with visual, motor, or cognitive disabilities, everyday web navigation is full of insurmountable barriers: missing descriptive alt tags for screen readers, broken keyboard navigation layouts, and dense multi-step checkouts that induce heavy cognitive and physical fatigue. In the spirit of generosity and inclusion for International Day of Charity, I built Prapatti AI: AI Navigation Tool for the Disabled.Prapatti AI is an intelligent desktop intermediary that restores digital autonomy to physically challenged individuals. Instead of forcing users to adapt to rigid screen readers or brittle keyboard shortcuts, Prapatti AI uses multimodal vision-language models to "see" and interpret desktop interface layouts directly. By connecting custom physical hardware sensors (like switch controls, eye-tracking systems, or adaptive joysticks) to Gemini's visual reasoning engine, Prapatti AI converts simple sensor gestures into complete, automated desktop interactions—bypassing non-compliant HTML, removing popups, and filling multi-step forms automatically.
demo.comScreenshot Preview:Prapatti AI parsing screen coordinates in real time to aid user navigation.
How I Built ItPrapatti AI operates on a local Sense-Reason-Act loop running directly on the user's desktop:Sense: Custom hardware sensor telemetry (e.g., switch triggers or eye-gaze holds) is ingested via lightweight Python serial listeners.Reason (Google AI Studio & Gemini): When a user initiates a goal, the client captures the desktop state and sends the raw screen image along with the user's intent to Gemini via Google AI Studio API calls.Act: Gemini analyzes the visual layout—identifying UI elements, buttons, and form fields purely by visual inspection—and returns structured coordinate commands. The Python execution handler (Playwright / PyAutoGUI) then translates these coordinates into desktop mouse clicks, typing, or scrolling actions.Architecture HighlightsVisual UI Layout Parsing: By leveraging visual capabilities instead of DOM parsing, Prapatti AI works on inaccessible websites, desktop applications, and legacy software that lack proper ARIA accessibility tags.Autonomous Friction Reduction: The AI engine proactively detects modal popups, cookie consent banners, and multi-step inputs, resolving them autonomously so the user experiences minimal cognitive drain.Deterministic Coordinate Outputs: Structured JSON schemas and zero-temperature prompt configs ensure pinpoint accuracy when mapping click targets on dynamic screen resolutions.JSON{
"action": "click",
"coordinates": { "x": 1650, "y": 820 },
"confidence_score": 0.98,
"reason": "Dismissing cookie consent banner to clear navigation path."
}
Prize CategoriesBest Use of Google AIPrapatti AI relies entirely on Google AI Studio and Gemini as its core brain:Multimodal Visual Reasoning: We prototyped and optimized the agent system instructions in Google AI Studio using the gemini-3.8-flash model. Gemini's high-speed visual token processing enables real-time analysis of full-desktop screenshots.Computer Use Integration: By incorporating Gemini's spatial coordinate estimation and Computer Use framework, Prapatti AI accurately locates small buttons, forms, and navigation controls across varying display resolutions without requiring pre-labeled training data.
Top comments (0)