How I Built an Edge-to-Cloud AI Assistant for the Blind (Kotlin + Vertex AI)

Muhammad Zubair — Thu, 10 Sep 2026 11:50:35 +0000

Navigating complex physical environments is a daily challenge for the visually impaired. While standard GPS apps help with macro-navigation, they fail to detect immediate physical obstacles, read environmental text, or recognize faces in real-time.

To bridge this gap, I engineered binAI—a cutting-edge, voice-controlled Android application that acts as a real-time mobility instructor and visual assistant for the blind.

By combining lightning-fast on-device sensors with powerful Google Vertex AI cloud models, binAI keeps users safe and aware of their surroundings with zero latency.

(Note: You can read the original architectural breakdown on my Software Engineering Portfolio).

🛠️ The Tech Stack: Android, FastAPI, and Vertex AI

Building a real-time mobility radar requires a highly optimized, concurrent architecture. I designed binAI using a hybrid Edge-to-Cloud approach to ensure maximum speed and reliability.


| Layer               | Technology Used               | Purpose                                               |
| :------------------ | :---------------------------- | :---------------------------------------------------- |
| **Frontend (Edge)** | Kotlin, Jetpack Compose       | Native Android UI and hardware integration            |
| **On-Device AI**    | Google ML Kit, CameraX        | Zero-latency object, face, and text detection         |
| **Backend (Cloud)** | Python 3, FastAPI             | High-performance API routing and data processing      |
| **Cloud AI**        | Google Vertex AI (Gemini 2.5) | Spatial reasoning and complex scene analysis          |
| **Infrastructure**  | Docker, Google Cloud Run      | Auto-scaling (scales to 0) with CI/CD via Cloud Build |

⚡ The Hybrid Edge-to-Cloud Pipeline

To provide the fastest and safest experience, an AI assistant for the blind cannot rely solely on the cloud (which introduces latency) or solely on the phone (which lacks deep reasoning). Here is how the hybrid architecture solves this:

1. The Edge (Phone): The Android app uses Kotlin Coroutines to process camera frames locally at high speeds, detecting bounding boxes, calculating obstacle distances in feet, and finding faces.

2. The Cloud (Server): The app injects this mathematical sensor data into a prompt alongside a compressed image and sends it to the FastAPI backend.

3. The Brain (Vertex AI): The Gemini model reads the image and the exact sensor data to generate highly accurate, spatial, and urgent instructions (e.g., "STOP! Stairs going down").

💻 Code Spotlight: Solving Edge-Case Complexities

1. Fixing Mobile Camera Rotation for Face Recognition

A major issue with Android cameras is that they often send frames rotated by 90 degrees, causing standard face recognition libraries to fail on the backend. I engineered a fallback mechanism using NumPy to rotate the image matrix and fix the underlying C++ memory contiguity before re-scanning.


# 1. Look for faces in the standard orientation
face_locations = face_recognition.face_locations(img)

# 2. THE ROTATION FIX (With C++ Memory Fix)
if not face_locations:
    # Rotate -90 degrees and fix memory layout for the C++ backend
    img_rotated = np.ascontiguousarray(np.rot90(img, k=-1))
    face_locations = face_recognition.face_locations(img_rotated)

    if face_locations:
        img = img_rotated
    else:
        # Fallback: Rotate +90 degrees
        img_rotated = np.ascontiguousarray(np.rot90(img, k=1))
        face_locations = face_recognition.face_locations(img_rotated)
        if face_locations:
            img = img_rotated

2. Sensor Data Injection into Vertex AI

To prevent the LLM from hallucinating distances, the backend dynamically injects the exact mathematical data calculated by the phone's edge sensors directly into the system prompt.


@app.post("/navigate")
async def navigate(image: UploadFile = File(...), on_device_data: Optional[str] = Form(None)):

    # Injecting Edge ML data into the Cloud LLM prompt
    sensor_injection = f"ON-DEVICE SENSOR DATA: {on_device_data}. Use these exact distances for your evasion commands." if on_device_data else ""

    prompt = f"""
    You are a real-time mobility radar for a blind person walking forward.
    {sensor_injection}

    CRITICAL CONSTRAINTS:
    - Do NOT use full sentences.
    - Do NOT be polite.
    - Prioritize distance (feet/steps) and directional commands (left/right).
    """

    result = call_vision_model(prompt, await image.read())
    return {"status": "success", "script": result}

💡 The Impact of AI for the Visually Impaired

binAI demonstrates how modern AI infrastructure—when combined with thoughtful, accessibility-first mobile engineering—can directly improve human lives. By leveraging Google Cloud Run and Vertex AI, the application remains highly scalable, cost-effective, and incredibly fast.

I'd love to hear your thoughts on Edge-to-Cloud architectures! Have you worked with CameraX or Vertex AI recently? Let me know in the comments. 👇

👨‍💻 About the Developer

I am Muhammad Zubair, a Software, DevOps, and Platform Engineer from Pakistan. I specialize in architecting scalable AI infrastructure, cloud platforms (AWS/Kubernetes), and full-stack systems.

My passion lies in solving real-world problems through code, whether that's building real-time mobility radars for the visually impaired or preserving regional languages through AI.

🔗 Let's Connect:

Portfolio & Case Studies: mzubair.online
GitHub (76+ Repos): Muhammad-Zubair796
LinkedIn: Muhammad Zubair

How I Engineered the World's First Khattak Pashto LLM (Qwen2 + LoRA)

Muhammad Zubair — Thu, 10 Sep 2026 11:12:58 +0000

Standard Large Language Models (LLMs) are incredible, but they have a massive blind spot: rural and regional dialects.

While standard Pashto is supported by some models, they completely fail to capture the rich, localized grammar of the Khattak dialect spoken in Karak, Nowshera, and Kohat (Pakistan).

To solve this, I built Khatta-ka-LLM—the world's first AI language model fine-tuned specifically for the Khattak (Khatak) dialect.

(Note: You can read the original architectural breakdown on my Software Engineering Portfolio or test the model live below).

Watch the Live Testing Video:

🛠️ Technical Architecture

As a Software Engineer specializing in AI Infrastructure, my focus was on creating a clean, scalable, and reproducible pipeline for low-resource languages. Here is how I built it:

1. The Custom Dataset

Because no dataset existed for this dialect, I curated a custom dataset of 2,000+ English-to-Khattak pairs. This represents the first digitized collection of Khattak linguistic markers, focusing heavily on the unique grammar used in the Karak and Kohat regions.

2. The Fine-Tuning Pipeline

To achieve high performance on consumer-grade hardware, I utilized the following stack:

Base Model: Qwen2 Architecture
Framework: Unsloth (for highly optimized, faster training)
Technique: LoRA (Low-Rank Adaptation)
Performance: Through careful hyperparameter tuning, the training loss dropped significantly from 3.44 to 0.22.

📘 The Linguistic Engineering (Rulebook)

To ensure the AI didn't just speak "standard Pashto with a Khattak accent," I had to engineer specific linguistic rules into the model's training weights.

Here are a few of the core rules the model learned:

1. Pronouns & Possession

My / Mine: The model uses مو والا (Mo wala) instead of the standard Zama.
To me: Uses موته (Mota) instead of standard Mala.
We: Uses موخ (Moxh) instead of standard Mung.

2. The "Noon Ghunna" (ں) Rule

In the Khattak dialect, first-person verbs end in a nasal Noon Ghunna (ں) rather than the standard "M".

Standard: Za Kar Kawom (I do work)
Khattak AI: زه چار کاوں (Za char kaon)

3. Phonetic "O" Shifts

The model successfully converts standard "A" sounds to the deep Khattak "O":

Asmaan ➡️ Asmon (اسمون)
Baraan ➡️ Baron (بارون)
Bazaar ➡️ Bazor (بازور)

💡 Why This Matters

By focusing on high-quality, localized data, Khatta-ka-LLM successfully translates English into pure Khattak Pashto. As a Software Engineer from Pakistan, my goal is to bridge the gap between modern AI infrastructure and regional linguistic heritage. We shouldn't have to lose our dialects to use modern technology.

I would love to hear your thoughts on fine-tuning models for low-resource languages! Have you worked with Unsloth or LoRA recently? Let me know in the comments. 👇

👨‍💻 About the Developer

My passion lies in solving real-world problems through code, whether that's building real-time mobility radars for the visually impaired or preserving regional languages through AI.

🔗 Let's Connect:

Portfolio & Case Studies: mzubair.online
GitHub (76+ Repos): Muhammad-Zubair796
LinkedIn: Muhammad Zubair

DEV Community: Muhammad Zubair

How I Built an Edge-to-Cloud AI Assistant for the Blind (Kotlin + Vertex AI)

🛠️ The Tech Stack: Android, FastAPI, and Vertex AI

⚡ The Hybrid Edge-to-Cloud Pipeline

💻 Code Spotlight: Solving Edge-Case Complexities

1. Fixing Mobile Camera Rotation for Face Recognition

2. Sensor Data Injection into Vertex AI

💡 The Impact of AI for the Visually Impaired

👨‍💻 About the Developer

How I Engineered the World's First Khattak Pashto LLM (Qwen2 + LoRA)

🛠️ Technical Architecture

1. The Custom Dataset

2. The Fine-Tuning Pipeline

📘 The Linguistic Engineering (Rulebook)

1. Pronouns & Possession

2. The "Noon Ghunna" (ں) Rule

3. Phonetic "O" Shifts

💡 Why This Matters

👨‍💻 About the Developer