Field Companion: Your Voice-First Nature Guide That Works Deep in the Wild With Zero Signal

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

---

What I Built

Most modern AI apps are built around screens, dashboards, and constant cloud connectivity. But when you hike into the deep woods, mountain passes, or national parks, cellular signal drops to zero. That is usually the exact moment you encounter an unfamiliar bird singing in the canopy or spot an unknown plant on the trail.

I built Field Companion (Offline Nature Scanner) — an open-source, voice-first mobile companion engineered to get people off their screens and out into nature.

Instead of typing queries into a chatbot or squinting at a screen under the midday sun:

1. You listen and speak: Tap the mic or let it passively record ambient audio.
2. On-device Bioacoustics: It identifies bird vocalizations and wildlife songs in real time.
3. On-device Open-Weight Intelligence: Powered by quantized Gemma weights running locally on your phone, it answers questions like "What bird is making that flute-like call right now?" and guides your eyes back to the trees.
4. Offline Checklist Grounding: It cross-references observations against a local SQLite database of 731 regionally verified species to prevent hallucination without requiring any internet connection.
5. Field Journaling: Automatically creates rich journal entries with timestamps, audio clips, and sightings that sync only when you decide to return home.

The goal is simple: make screen time the shortest part of the hike so you can look up, listen, and touch grass.

---

Demo

• Live Web App & Download Hub: https://offline-nature-scanner-git-main-hartej46s-projects.vercel.app
• Live Production Backend on Render: https://offline-nature-scanner-1.onrender.com
• Standalone Android APK Direct Download: Download FieldCompanion.apk (69.4 MB)
• Expo Preview / Instant Test: Available via Expo Go

Live Feature Tour:

• Audio Identification: Analyzes bird songs with acoustic feature extraction and confidence scoring.
• Visual Plant & Wildlife Identification: Client-side camera capture with offline heuristic checklists.
• Model Weight Manager: Transparent 1-click model weight installer that pulls weights directly onto device storage with zero recurring API costs.
• Offline PWA Landing Page: Even the web landing page uses service workers (sw.js) so hikers can review the sideloading guide without an active data connection.

---

Code

The entire codebase is open-source under the MIT License on GitHub:

hartej46 / Offline-Nature-scanner

Offline-Nature-scanner

View on GitHub

• Repository: https://github.com/hartej46/Offline-Nature-scanner
• Backend: FastAPI containerized in Docker, with SQLite grounding and acoustic pipelines.
• Mobile Client: React Native + Expo configured for standalone offline Android APK builds.
• Web Landing: React 19 + Vite PWA deployed to Vercel and configured for Render static site mirroring.

---

How I Built It

The system is designed around a three-tier offline-first architecture:

                            +-------------------------------------------------------+
                            |                  RENDER CLOUD                         |
                            |                                                       |
   [ Mobile Device ]        |   +-----------------------------------------------+   |
           |                |   |  FastAPI Backend (Docker on Render)           |   |
           |-- Download APK-|-->|  - GET /download/app (69.4 MB Standalone APK) |   |
           |                |   |  - Regional Checklist & Journal (SQLite)      |   |
           |-- Model Weights|-->|  - GET /model/download/{id} (Redirect)        |   |
           |   Redirect     |   +-----------------------+-----------------------+   |
           v                |                           |                           |
   +---------------+        +---------------------------|---------------------------+
   | ON-DEVICE AI  |                                    |
   | (100% Offline)|                                    v
   | - Gemma 2B    |                    +-------------------------------+
   | - BirdNET     |                    | Hugging Face / Model Weights  |
   | - SQLite 731  |                    | (One-time download to phone)  |
   +---------------+                    +-------------------------------+

1. Open-Weight Foundation: Google Gemma

To achieve realistic reasoning on mobile hardware without cloud latency or fees, Field Companion utilizes Gemma (gemma-2-2b-it-Q4_K_M.gguf / gemma-4:e2b).

• On mobile devices, the model weights live directly inside device document storage.
• The model is prompted with structured JSON grounding schemas constrained by our local checklist. When an observer hears a warbler, Gemma doesn't fabricate global species — it cross-checks the regional bounding box and returns actionable identification advice.

2. Acoustic Bioacoustics Pipeline

Audio recordings captured through the phone's microphone are analyzed through an acoustic feature extraction pipeline:

• Peak frequency detection and harmonic spectral ratio calculation.
• Frequency centroid and temporal syllable counting to match bird calls against acoustic patterns (e.g., flute-like whistles vs. rapid trills).

3. Production Deployment with Render & Docker

The backend companion service is hosted on Render as a Docker Web Service:

• Built on Python 3.11-slim with system audio libraries (libsndfile1, ffmpeg, sqlite3).
• Handles model redirection, APK distribution, and automated /health checks.
• Backed by an automated render.yaml Blueprint specification for reproducible deployments.

---

Why Does Open Innovation Matter?

If this project had been built using closed proprietary APIs (like ChatGPT or proprietary cloud vision services), it simply would not work in the real world:

1. The Woods Have No Wi-Fi: Closed APIs require an internet ping on every question. The moment you cross the trailhead and step under a thick forest canopy, closed AI turns into a useless loading spinner. Open-weight models like Gemma allow the intelligence to sit directly in your pocket.
2. True Location Privacy: Nature enthusiasts often encounter endangered, vulnerable, or protected species (such as nesting raptors or rare orchids). Sending exact GPS coordinates and timestamps to third-party commercial cloud APIs risks location leaks and tracking. With local models and local SQLite storage, your field observations never leave your device unless you choose to export them.
3. Zero Financial Friction ($0/month): Closed APIs charge per token. A student or park ranger making 50 audio and visual queries during an afternoon hike would run up a bill. Open innovation means the model is downloaded once over home Wi-Fi and costs exactly $0 to run indefinitely.
4. Community Longevity: Open-weight weights cannot be deprecated, paywalled, or discontinued by a sudden API shutdown. A hiker can pull out this phone five years from now and it will identify a chickadee just as reliably as it does today.

---

Prize Categories

• Best Use of Render — Deployed the companion backend Docker Web Service with automatic container healthchecks, persistent regional storage, and dynamic APK streaming on Render.
• Best Use of Gemma — Integrated Google's open-weight Gemma architecture for local offline nature reasoning, acoustic observation verification, and on-device field guidance.