How I Built an Edge-to-Cloud AI Assistant for the Blind (Kotlin + Vertex AI)

#showdev #android #ai #python

Navigating complex physical environments is a daily challenge for the visually impaired. While standard GPS apps help with macro-navigation, they fail to detect immediate physical obstacles, read environmental text, or recognize faces in real-time.

To bridge this gap, I engineered binAI—a cutting-edge, voice-controlled Android application that acts as a real-time mobility instructor and visual assistant for the blind.

By combining lightning-fast on-device sensors with powerful Google Vertex AI cloud models, binAI keeps users safe and aware of their surroundings with zero latency.

(Note: You can read the original architectural breakdown on my Software Engineering Portfolio).

🛠️ The Tech Stack: Android, FastAPI, and Vertex AI

Building a real-time mobility radar requires a highly optimized, concurrent architecture. I designed binAI using a hybrid Edge-to-Cloud approach to ensure maximum speed and reliability.


| Layer               | Technology Used               | Purpose                                               |
| :------------------ | :---------------------------- | :---------------------------------------------------- |
| **Frontend (Edge)** | Kotlin, Jetpack Compose       | Native Android UI and hardware integration            |
| **On-Device AI**    | Google ML Kit, CameraX        | Zero-latency object, face, and text detection         |
| **Backend (Cloud)** | Python 3, FastAPI             | High-performance API routing and data processing      |
| **Cloud AI**        | Google Vertex AI (Gemini 2.5) | Spatial reasoning and complex scene analysis          |
| **Infrastructure**  | Docker, Google Cloud Run      | Auto-scaling (scales to 0) with CI/CD via Cloud Build |

⚡ The Hybrid Edge-to-Cloud Pipeline

To provide the fastest and safest experience, an AI assistant for the blind cannot rely solely on the cloud (which introduces latency) or solely on the phone (which lacks deep reasoning). Here is how the hybrid architecture solves this:

1. The Edge (Phone): The Android app uses Kotlin Coroutines to process camera frames locally at high speeds, detecting bounding boxes, calculating obstacle distances in feet, and finding faces.

2. The Cloud (Server): The app injects this mathematical sensor data into a prompt alongside a compressed image and sends it to the FastAPI backend.

3. The Brain (Vertex AI): The Gemini model reads the image and the exact sensor data to generate highly accurate, spatial, and urgent instructions (e.g., "STOP! Stairs going down").

💻 Code Spotlight: Solving Edge-Case Complexities

1. Fixing Mobile Camera Rotation for Face Recognition

A major issue with Android cameras is that they often send frames rotated by 90 degrees, causing standard face recognition libraries to fail on the backend. I engineered a fallback mechanism using NumPy to rotate the image matrix and fix the underlying C++ memory contiguity before re-scanning.


# 1. Look for faces in the standard orientation
face_locations = face_recognition.face_locations(img)

# 2. THE ROTATION FIX (With C++ Memory Fix)
if not face_locations:
    # Rotate -90 degrees and fix memory layout for the C++ backend
    img_rotated = np.ascontiguousarray(np.rot90(img, k=-1))
    face_locations = face_recognition.face_locations(img_rotated)

    if face_locations:
        img = img_rotated
    else:
        # Fallback: Rotate +90 degrees
        img_rotated = np.ascontiguousarray(np.rot90(img, k=1))
        face_locations = face_recognition.face_locations(img_rotated)
        if face_locations:
            img = img_rotated

2. Sensor Data Injection into Vertex AI

To prevent the LLM from hallucinating distances, the backend dynamically injects the exact mathematical data calculated by the phone's edge sensors directly into the system prompt.


@app.post("/navigate")
async def navigate(image: UploadFile = File(...), on_device_data: Optional[str] = Form(None)):

    # Injecting Edge ML data into the Cloud LLM prompt
    sensor_injection = f"ON-DEVICE SENSOR DATA: {on_device_data}. Use these exact distances for your evasion commands." if on_device_data else ""

    prompt = f"""
    You are a real-time mobility radar for a blind person walking forward.
    {sensor_injection}

    CRITICAL CONSTRAINTS:
    - Do NOT use full sentences.
    - Do NOT be polite.
    - Prioritize distance (feet/steps) and directional commands (left/right).
    """

    result = call_vision_model(prompt, await image.read())
    return {"status": "success", "script": result}

💡 The Impact of AI for the Visually Impaired

binAI demonstrates how modern AI infrastructure—when combined with thoughtful, accessibility-first mobile engineering—can directly improve human lives. By leveraging Google Cloud Run and Vertex AI, the application remains highly scalable, cost-effective, and incredibly fast.

I'd love to hear your thoughts on Edge-to-Cloud architectures! Have you worked with CameraX or Vertex AI recently? Let me know in the comments. 👇

👨‍💻 About the Developer

I am Muhammad Zubair, a Software, DevOps, and Platform Engineer from Pakistan. I specialize in architecting scalable AI infrastructure, cloud platforms (AWS/Kubernetes), and full-stack systems.

My passion lies in solving real-world problems through code, whether that's building real-time mobility radars for the visually impaired or preserving regional languages through AI.

🔗 Let's Connect:

Portfolio & Case Studies: mzubair.online
GitHub (76+ Repos): Muhammad-Zubair796
LinkedIn: Muhammad Zubair

DEV Community