Echoes - A Screen-Free, Voice-First AI Walking Guide

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

Screen time is the enemy of a good walk. Most outdoor AI apps force you to download massive models over LTE or stare at your screen while walking. I wanted a tool that would connect people to their surroundings without demanding their visual attention.
That is why I built Echoes: an open-weight, voice-first walking companion. You tap "Start Walk," put your phone in your pocket, and let the AI guide you while you take in the world.

What I Built

Echoes transforms an ordinary outdoor walk into a hyper-local, audio-guided experience that requires zero screen time once started.

• Story Mode (Hyper-Local Grounded Audio): As you walk, the app fetches real Wikipedia history and live nearby parks and landmark data via SerpApi within a 1.5km radius. A hosted Gemma model acts as your narrator, weaving these facts into short, contextual audio snippets.
• Notice Mode (Sensory Fallback): If you are in a quiet residential area with no historical facts, the app doesn't hallucinate history. It deterministically switches to 'Notice Mode'—delivering open-ended sensory mindfulness prompts based on the local time of day (e.g., "Notice the sound underneath the loudest one").
• Push-To-Talk (PTT): You can hold the mic button to ask questions about nearby landmarks. The app uses Groq Whisper for millisecond transcription and Gemma to provide grounded answers.

The Moat: Code-Enforceable Grounding
We don't just ask the AI to tell the truth; we enforce it in code. The backend forces Gemma to output inline citations for every factual claim. A deterministic Python validator parses the stream sentence-by-sentence and automatically drops any AI claim that lacks a valid source attribution before it reaches the text-to-speech engine.

Demo

• Live Web App: https://echoes-ai-nine.vercel.app/
• GitHub Repository: https://github.com/owaismohammed79/echoes-ai

How I Built It

• Frontend (React/Vite/PWA): Built for the mobile web[cite: 1]. Implemented an iOS-safe synchronous AudioContext unlocking trick inside the "Start Walk" tap handler, allowing uninterrupted background playback while the phone is locked. It also uses the Media Session API so you can pause/play directly from your lock screen.

• Backend (Python/FastAPI): Hosted on Render, this orchestrates the pipeline. It makes concurrent calls to Wikipedia GeoSearch and SerpApi (using Google Maps engine coordinates), filtering out infrastructure noise using regex deny-lists.

• The AI Loop: Gemma streams the narrative tokens, the backend validates and strips the citation tags, and the clean text is instantly streamed to ElevenLabs TTS to minimize time-to-first-audio-byte. Push-to-Talk is powered by Groq's Whisper API (whisper-large-v3).

Things That Broke (And How I Fixed Them)

Building a real-time, hands-free conversational loop outdoors is messy. Here is what failed during development and how it was engineered around:

1. SerpApi Coordinate & Distance Sorting: Google Maps search via SerpApi returned relevant parks, but ordered them by popularity rather than proximity (e.g., a park 9km away appearing before a park 1.4km away). I wrote a custom parser in app/services/parks.py to extract exact GPS coordinates, compute Haversine distance, cap results at 5, and sort nearest-first before feeding them into Gemma.
2. Key Leakage in URL Logs: Both SerpApi and Wikipedia pass query parameters in HTTP GET requests. During error logging, httpx dumped full URLs—including raw API keys—into stdout. I built app/services/redact.py to intercept and redact API keys across all outbound requests before writing to console logs.
3. Strict Context Validation: The Regex validator aggressively dropped sentences without citation tags. While crucial for preventing hallucinated history, it was deleting Gemma's atmospheric transition lines (e.g., weather and mood setup). I updated the validator logic to permit bounded "framing sentences" while strictly requiring citations on claims derived from SerpApi and Wikipedia data.

Why Does Open Innovation Matter?

Building with open-weight models like Gemma and Whisper is the only way this architecture works. Open innovation allows us to treat the LLM not as a black-box oracle, but as a swappable reasoning engine within a strict, deterministic pipeline.

Instead of relying on fragile system prompts to prevent hallucinations, utilizing open models allowed me to build a custom validation layer that parses the model's output stream, verifies its citations against fetched API data, and intercepts fabrications before the user ever hears them. Open AI makes it possible to build systems that respect the user's environment, ensuring that when you step outside to touch grass, the technology enhances reality rather than replacing it.

My Agent Session

• Agent Session Logs (Gist): https://gist.github.com/owaismohammed79/59754d551eb1041a0caa423e7c3262a8

Prize Categories

• Best Use of Gemma: Core streaming intelligence (LLM orchestration) used for narrating grounded local walks, handling mood framing, and processing Push-To-Talk Q&A.
• Best Use of ElevenLabs: Low-latency text-to-speech streaming, delivering the narrative voice seamlessly to the locked mobile device.
• Best Use of SerpApi: Live Google Maps engine search integration (Parks nearby with lat/lon parameters) to fetch real-world parks, local landmarks, and place details within the user's 1.5km radius.
• Best Use of Backboard: Used Backboard's R-CLI terminal agent to autonomously build, refactor, and debug the React state machine, FastAPI services, and test suites.
• Best Use of Entire: Complete agent session logs exported and shared via GitHub Gist (echoes-backboard-session.md), documenting the step-by-step terminal agent interactions that built the application.
• Best Use of Render: Hosts the Python FastAPI backend, real-time citation validation pipeline, and low-latency audio streaming proxy configured via render.yaml.