DEV Community

Cover image for TrailWhisper: A Screen-Zero, Pocket-First AI Audio Companion for Hiking
DracFiendMG
DracFiendMG

Posted on

TrailWhisper: A Screen-Zero, Pocket-First AI Audio Companion for Hiking

#devchallenge #hf26challenge

Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass Submission 🌿

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Most modern outdoor applications—from trail navigators to bird identification apps—suffer from an ironic paradox: they are built to connect people with nature, yet they demand that you stare at a glass screen the entire time you're outside. You stop on the trail, shield your phone from the sun, squint through menus, and read Wikipedia-style descriptions while the forest passes by unnoticed.

I built TrailWhisper, a screen-zero, pocket-first AI audio companion designed to get hikers out into the world and keep their eyes on the trail.

Here is how the experience works:

  1. Phone in Pocket, Earbuds In: Before stepping onto the trail, you slip your phone into your pocket or pack and activate Pocket Mode.
  2. Predicting Biodiversity with Tabular AI: As you walk, TrailWhisper evaluates your real-time spatial and environmental telemetry (GPS coordinates, elevation, time of day, month, canopy density, and temperature). Instead of relying on static database queries, it uses Prior Labs' TabPFN tabular foundation model to compute zero-shot sighting probabilities for local flora and fauna based on curated eBird and GBIF trail observation datasets.
  3. Audio-First Sensory Guidance with Google Gemma: When a species is likely nearby (e.g., a Steller's Jay, Douglas Squirrel, or Pacific Trillium), Google's open-weight Gemma synthesizes a concise, 3-to-4 sentence spoken field note. It is prompt-engineered strictly for the ear: zero markdown, natural pauses, and vivid directional cues ("Tilt your head up toward the lower hemlock branch... hear that sharp, metallic rattle? That's a Steller's Jay caching cones...").
  4. Natural Spoken Voice: ElevenLabs streams low-latency narration directly into your earbuds.
  5. Lock-Screen Media Controls: Through the HTML5 MediaSession API, hikers can play, pause, or skip to the next waypoint directly from their headphone buttons or lock screen without unlocking the phone.
  6. Alpine Trail Simulator: To ensure anyone can test the system anywhere (even from a desk before heading to the trailhead), TrailWhisper includes an interactive Olympic National Park trail simulator that steps through valley floors, hemlock groves, old-growth ridges, and alpine meadows.

Demo

💡 Tip for Testing: Connect your earbuds or headphones, toggle "Pocket Mode Active", and use the Trail Simulator buttons ("Next Waypoint") to step through Olympic Trail coordinates and hear the live nature whispers stream to your headphones!

Key Screens & Architecture:

  • Pocket Mode HUD: Forest emerald dark mode with ambient audio pulse waves and zero screen clutter.
  • Sightings Radar: Visualizes TabPFN probability distributions across birds, mammals, plants, and amphibians.
  • Media Session Integration: Full notification tray and lock-screen metadata display (Whisper: Steller's Jay nearby).
  • Telemetry Drawer: Real-time inspection of TabPFN model confidence, Gemma narration scripts, and Sentry trace IDs.

Code

GitHub logo DracFiendMG / trailwhisper

A screen-zero, pocket-first audio companion that gets users outdoors

🌲 TrailWhisper

Screen-Zero, Pocket-First AI Audio Companion for Nature Trails
Predicts nearby flora and fauna sightings with tabular foundation models and whispers contextual nature stories directly through your earbuds as you hike.

License: MIT Orchestrator: Mastra Tabular Model: TabPFN LLM: Google Gemma Voice: ElevenLabs Observability: Sentry Deployment: Render

🧭 The Screen-Zero Experience

Most outdoor and nature guide apps force hikers to keep their eyes glued to phone screens while walking through forests and trails. TrailWhisper inverts the interaction paradigm:

  1. Keep Your Phone in Your Pocket: Audio streams directly to your headphones or earbuds as you walk.
  2. Tabular Foundation Model Predictions: Prior Labs' TabPFN evaluates hyper-localized ecological factors (GPS coordinates, elevation, hour, month, canopy density, and temperature) against curated trail biodiversity data to estimate species sighting probabilities in real time.
  3. Conversational Audio Whispers: Google's open-weight Gemma (google/gemma-4-26b-a4b-it:free via the OpenRouter SDK) synthesizes concise, immersive 3-to-4 sentence spoken field notes directing your senses outward into the canopy or trail edges ("Take a…
View on GitHub

The codebase is organized as an open-source monorepo:

trailwhisper/
├── apps/
│   ├── agent-server/              # Mastra orchestrator + Sentry tracing (Node.js)
│   │   ├── src/
│   │   │   ├── mastra/
│   │   │   │   ├── agents/        # NatureGuideAgent definition & prompts
│   │   │   │   ├── tools/         # TabPFN, Gemma (OpenRouter), ElevenLabs tools
│   │   │   │   └── index.ts       # Mastra core & LibSQL storage exports
│   │   │   ├── sentry.ts          # Sentry OpenTelemetry spans & profiling
│   │   │   └── server.ts          # Express API server (/api/trail/step)
│   │   ├── Dockerfile
│   │   └── package.json
│   └── web/                       # Screen-Zero Pocket Client (Vite + React)
│       ├── src/
│       │   ├── components/        # AudioPlayer, SightingsRadar, TelemetryDrawer
│       │   ├── utils/             # Olympic Trail Simulator & waypoints
│       │   ├── App.tsx            # Pocket Mode HUD & MediaSession controls
│       │   └── index.css          # Forest dark mode & glassmorphism
│       └── package.json
├── services/
│   └── tabpfn-service/            # Python FastAPI microservice for TabPFN
│       ├── app.py                 # /predict and /health endpoints
│       ├── data/                  # Sample trail observation CSV (eBird/GBIF)
│       ├── requirements.txt
│       └── Dockerfile
├── render.yaml                    # Multi-service Render deployment Blueprint
├── package.json                   # Root monorepo workspace configuration
└── README.md
Enter fullscreen mode Exit fullscreen mode

How I Built It

TrailWhisper couples open-weight foundation models with modern agent orchestration and low-latency audio:

1. Tabular Foundation Models: Prior Labs TabPFN

Species presence depends heavily on tabular features: GPS coordinates, elevation, time of day, seasonal month, temperature, and tree canopy density. Instead of training and tuning a separate model for every park, I used TabPFN (tabpfn-v2).

  • Running in a standalone Python 3.11 FastAPI microservice (services/tabpfn-service).
  • Ingests tabular observations from Pacific Northwest biodiversity records.
  • Evaluates zero-shot probabilities for candidate species in milliseconds.
  • Includes a resilient Scikit-Learn ensemble fallback to guarantee uninterrupted field predictions in CPU-constrained environments.

2. Spoken Script Synthesis: Google Gemma

To turn tabular probabilities into an immersive wilderness experience, I used Google's open-weight Gemma (google/gemma-4-26b-a4b-it:free) via the official @openrouter/sdk:

  • Prompted strictly as an intimate audio companion whispering in the hiker's ear.
  • Strict formatting guardrails: no markdown formatting, no bullet points, no emojis, and an exact length limit of 3 to 4 sentences (45–60 words).
  • Directs attention outward into physical space: describing bark textures, lighting angles, mossy edges, and bird calls.

3. Agent & State Orchestration: Mastra & LibSQL

The agent pipeline is orchestrated using Mastra (@mastra/core):

  • Coordinates the multi-tool workflow: predictSightings (TabPFN) ➔ generateFieldGuide (Gemma) ➔ synthesizeAudio (ElevenLabs).
  • Uses Mastra's official persistent storage adapter (@mastra/libsql) with an embedded LibSQL/SQLite database (file:trailwhisper.db), ensuring conversation state, memory, and trail history persist across server restarts.

4. Low-Latency Voice: ElevenLabs

  • Spoken scripts stream directly via the ElevenLabs Turbo v2.5 engine with the warm storyteller voice (JBFqnCBsd6RMkjVDRZzb - George).
  • Graceful client-side Web Speech API fallback if an API key is unconfigured, ensuring the screen-zero experience works out of the box.

5. Observability: Sentry Agent Tracing

  • Full distributed tracing using @sentry/node with OpenTelemetry instrumentation.
  • Every trail step creates an ai.agent.trail_step transaction with sub-spans for ai.tool.tabpfn.predict, ai.tool.gemma.generate, and ai.tool.elevenlabs.tts, capturing duration, token counts, and model metadata.

6. Cloud Deployment: Render Blueprint

  • A multi-service declarative render.yaml deploys the Python ML microservice, Node.js agent server, and React static PWA in a single click.

Why Does Open Innovation Matter?

Open innovation is what made TrailWhisper technically and philosophically possible:

  1. Open-Weight Models Enable Specialized Physical Interfaces: Closed commercial APIs are predominantly optimized for chat interfaces, code generation, and markdown-heavy text boxes. With open-weight models like Google Gemma, developers have the freedom to steer the model towards non-traditional interfaces—in this case, audio-first, screen-zero cadence designed for the ear rather than the eye.
  2. Tabular Foundation Models Democratize Domain AI: Prior Labs' TabPFN represents a major breakthrough: applying transformer-based foundation models to tabular data. Historically, predicting ecological events required training bespoke machine learning pipelines on massive labeled datasets. TabPFN's zero-shot capabilities allow hyper-localized predictions out of the box with zero fine-tuning overhead.
  3. Extensibility & Privacy for the Outdoors: Outdoor tech needs to function everywhere, eventually transitioning to edge devices and offline trail kits. Open-source frameworks like Mastra paired with local storage like LibSQL mean that the orchestrator isn't locked behind a proprietary cloud walled garden.
  4. Community Biodiversity Data: TrailWhisper relies on open science data from eBird and GBIF. Combining open datasets with open AI creates software that respects users' attention while fostering a deeper connection with the physical planet.

My Agent Session

TrailWhisper was architected, scaffolded, and debugged in pair-programming sessions with Google Antigravity IDE, orchestrating multi-service Docker builds, TabPFN ensemble fallbacks, OpenRouter Gemma toolchains, and Mastra LibSQL persistent storage integrations.

Prize Categories

I am entering TrailWhisper into the following partner prize categories:

  • Best Use of TabPFN: Used Prior Labs' TabPFN tabular foundation model inside a FastAPI microservice to perform real-time, zero-shot biodiversity sighting probability predictions on hiking trail features (GPS, elevation, canopy density, hour, temperature).
  • Best Use of Gemma: Powered by Google's open-weight Gemma (google/gemma-4-26b-a4b-it:free along with a fallback google/gemma-3-12b-it and google/gemma-3-4b-it via OpenRouter SDK), tailored with custom prompts to produce markdown-free, audio-first field notes directing hikers' senses into the wild.
  • Best Use of Mastra: Agent orchestration, tool pipeline dispatch (predictSightings, generateFieldGuide, synthesizeAudio), and persistent state storage using @mastra/core and @mastra/libsql.
  • Best Use of ElevenLabs: Low-latency voice streaming using ElevenLabs Turbo v2.5 to deliver hands-free audio narration directly to the hiker's headphones.
  • Best Use of Render: Complete multi-service production deployment managed declaratively with a render.yaml Blueprint covering the Python ML worker, Node.js agent orchestrator, and Vite static PWA.
  • Best Use of Sentry: Integrated @sentry/node agent tracing with OpenTelemetry spans tracking end-to-end execution, tool latencies, and model tokens across the multi-model pipeline.

Top comments (1)

Collapse
 
suppdevbot profile image
DEV SUPPORTS • 

You need to verify your account.
Enter fullscreen mode Exit fullscreen mode

tr.ee/dev-to