Vaachak - AI companion for Elderly people with Eyesight issues to read anything

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Built Vaachak - An AI companion app for Elderly people who has blurry visions and eyesight issues for the love of reading

Target Audience :

1. People who loves reading, but cant read, because of eyesight issues.
2. People who are elder 60+ years age with eyesight issues, this app helps them read any paper be it Bill, Letter, NewsPaper etc.
3. People who are suffering from blindeness
4. People who dont want to read english, and want to understand a article to be read in their own regional language.
5. People who want to convert a pic to a readable memory.

The Mobile App:

• It all starts with a Eye Test to understand user's eyesight scores.

Web App for Admin:

This is a FastAPI based server

Admin panel to test the apis for Sarvam Models, Gemma Models, BackBoard for Memory, and local Ollama serve apis

Using Gemma Model https://ai.google.dev/gemma/docs/core/gemma_on_gemini_api

What does it do ?

Presentation -
https://docs.google.com/presentation/d/11WM5la02aXDbEHukz5thxk6YvKBBeb10_8SE1YJAPOo/edit?usp=sharing

A smart, voice-controlled Android app that helps visually impaired people navigate their surroundings safely. It acts as an intelligent assistant that can "see" objects, warn about obstacles, and guide the user

Offline first -> Web App (built using Nextjs + App router) + Android App (Kotlin + Android Studio) that uses open source gemma models (gemma-26b) locally in case, if the user does not have internet, he can run the web app and the server using Docker locally

🌟 What It Can Do

🧭 Easy Navigation

• 🎧 3D Sound Feedback: The app plays sounds to let you know exactly where objects are around you (left, right, near, or far).
• 📍 Object Memory: The app remembers where things are. If you walk past a cup, you can ask "Where is my cup?" and it will tell you.
• 🔍 Find Mode: Ask the app to "Find a door" or "Find a chair," and it will guide you to it using sound and vibrations.

🧠 Smart Vision

• 📷 Instant Object Detection: Uses the phone's camera to instantly spot obstacles and things in your path, without needing the internet.
• 🖼️ Scene Description: Ask "Describe what you see," and the app will tell you exactly what the room looks like.
• 📖 Read Text (OCR): Point the camera at a sign or book, and it will read the text out loud.
• 😊 Facial Recognition: Recognizes people you know when they are nearby and tells you their names.

🛡️ Safety First

• 🎙️ Voice Control: Control everything using just your voice. No need to look at the screen.
• 🚨 Fall SOS: If the phone detects you have fallen, it automatically sends an emergency SOS message to your contacts.
• 🔦 Manual & Auto-Flashlight: You can turn the flashlight on and off yourself with a voice command, or let the app do it automatically in the dark so the camera can keep seeing clearly.

🗣️ Voice Commands to Try

• 🗣️ "Describe what you see"
• 🗣️ "Find a chair"
• 🗣️ "Where is my cup?"
• 🗣️ "Read"
• 🗣️ "Turn off flashlight"
• 🗣️ "Switch to spatial" (for 3D sound)
• 🗣️ "Start eye test" (for visual acuity calibration)

🏗️ How It Works (Diagrams)

1. The Big Picture

graph TD
    A[Camera Feed] --> B(Vision Processor)
    B --> C{Offline AI}
    C -->|Fast Detection| D[3D Sound]
    C -->|Guidance| E[Vibration & Navigation]
    C -->|Remember Objects| F[(Memory Database)]

    A -->|Ask a Question| G{Cloud AI}
    G -->|Detailed Answer| H[Voice Response]

    I[Phone Sensors] --> J[Fall Detection]
    I --> F

2. How 3D Sound Works

graph LR
    A[See an Object] --> B(Get its Location)
    B -->|Left/Right| C(Sound Direction)
    B -->|Size| D(Sound Volume)
    B -->|Up/Down| E(Sound Pitch)
    C --> F((Your Earphones))
    D --> F
    E --> F

3. Remembering Objects

sequenceDiagram
    participant Camera
    participant AI
    participant Compass
    participant Database
    participant User

    Camera->>AI: Sees a 'Cup'
    Compass->>Database: Saves direction (e.g. East)
    AI->>Database: Saves 'Cup' at East
    Note over Camera,User: User turns around
    User->>Camera: "Where is my cup?"
    Camera->>Database: Look for 'Cup'
    Database-->>Camera: It's at East
    Compass-->>Camera: You are facing West
    Camera->>User: "The cup is directly behind you."

4. Emergency Fall Detection

graph TD
    A[Phone Moves Fast] --> B{Did it drop?}
    B -- Yes --> C[Fall Detected!]
    C --> D{Did it hit the ground?}
    D -- Yes --> E[Start Emergency]
    E --> F[Say: 'Fall Detected']
    E --> G[Send SOS Message]
    G --> H((Emergency Contacts))

Demo

Youtube - https://youtu.be/aJADQ3tX5s4

Web App - https://vaachak-ebon.vercel.app/
Backend API - https://vaachakfastapibackend.onrender.com/

🛠️ Technology Used

Part	Tech
Language	Kotlin
UI	Jetpack Compose
Offline AI	TensorFlow Lite, Google ML Kit
Online AI	Vaachak Render API (Gemma, Sarvam, Backboard, Supabase)
Database	Room (SQLite)

Code

sujaykundu777 / Vaachak

An offline-first reading companion for elders with weak eyesight

Vaachak — An Assistive App for the Visually Impaired

As part of Hacktoberfest 2026 x Kraftcrowd Hackathon we are building a Offline first Web App + Android App that uses open source gemma models locally

PRESENTATION

https://docs.google.com/presentation/d/11WM5la02aXDbEHukz5thxk6YvKBBeb10_8SE1YJAPOo/edit?usp=sharing

DEMO VIDEO

Youtube - https://youtu.be/aJADQ3tX5s4 https://youtu.be/onqhgiilUOk

DEMO URL

Web app Hosted :https://vaachak-ebon.vercel.app/

API endpoint https://vaachakfastapibackend.onrender.com/

A smart, voice-controlled Android app that helps visually impaired people navigate their surroundings safely. It acts as an intelligent assistant that can "see" objects, warn about obstacles, and guide the user.

---

🌟 What It Can Do

🧭 Easy Navigation

• 🎧 3D Sound Feedback: The app plays sounds to let you know exactly where objects are around you (left, right, near, or far).
• 📍 Object Memory: The app remembers where things are. If you walk past a cup, you can ask "Where is my cup?" and it will tell you.
• 🔍 Find Mode: Ask the app to "Find…

View on GitHub

How I Built It

• OpenCode -> For Coding
• OpenDesign -> For Design Generation
• Ollama API -> For using local Gemma Models locally models

Models used while building this:

• Deepseek V3
• Qwen 3.8
• Gemma 4

MCP Servers Used:

https://devrelay.com
https://github.com/github/github-mcp-server

Why Does Open Innovation Matter?

My Agent Session

My First User

Prize Categories

🏆 Hackathon Tracks & Gemma Integration

As part of Hacktoberfest 2026 x Kraftcrowd Hackathon, Vaachak is built as an offline-first assistive Android application leveraging open-source Gemma & NVIDIA AI models.

Best Use of Gemma Model

• G2. Voice-First AI Interaction: Integrate voice-based intelligence that lets visually impaired users interact naturally through speech, evaluate acuity, and receive spoken audio responses.

• G3. Multimodal AI Intelligence: Combines real-time camera imagery, optical character recognition (OCR), spatial 3D audio, and spoken voice inputs to understand surroundings and guide users.

• G5. Open Innovation with Gemma 4: Powered by NVIDIA AI / Gemma vision & language models for clinical visual acuity scoring and real-time scene understanding.

Best Use of Backboard

Using Backboard for storing the conversations happening

Resources

• Gemma 4 Resources & Overview
• Gemma Quickstart
• Gemma API Documentation
• Gemma Beginner Guide
• Using Gemma Model with Gemini