Field Guide: A Nature Companion That Teaches You to Look, Built on Open-Weight AI
This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass
What I Built
Most bird apps identify the bird for you. Field Guide teaches you to look.
Field Guide is a screen-minimizing, offline-first nature companion designed to get you off your screen and out into the real living world.
When you spot an unfamiliar bird perched on a branch or a flowering tree in a neighborhood park, you don't scroll through endless image search results. Instead, you tap a single button and describe what you see in your own words (or speak your field notes). A locally running open-weight model suggests what regional species it could be, then asks you a sharp diagnostic question about the real organism right in front of you. With the optional voice layer enabled, you hear the naturalist mentor directly through your earbuds:
"Look up at its head right now: Does it have a spiky crest like a little crown, or a smooth rounded head?"
Your eyes go back up to the bird. You observe it. You answer. The detective engine narrows it down with a second check on its behavior:
"Where is it active right now: foraging on the ground, or perched high in the leafy canopy?"
You answer. The species is confirmed. A collectible card drops into your offline Field Journal with its vernacular Tamil name, regional frequency, and local ecological fact.
The entire loop takes about 35 to 45 seconds — with most of that time spent looking at nature with your own eyes, not staring at glass.
Who Is It For?
Walkers, school students, urban birders, and curious residents in Coimbatore, Tamil Nadu (Western Ghats foothills) — and anyone who wants to rediscover the living world outside their window.
The app is grounded in 105 verified regional species (80 local birds and 25 native trees) harvested from GBIF occurrence records within the Coimbatore coordinate bounding box, cross-referenced with Wikidata vernacular Tamil names (தமிழ்) and Wikipedia natural history records, served entirely from a lightweight, self-contained local SQLite database.
Core Features
- Diagnostic Detective Mode: Rather than handing you a passive photo match, the app prompts you to inspect diagnostic field marks on the real bird or tree (beak curvature, tail fork, leaf arrangement, eye patch, foraging posture).
-
Open-Weight AI Dialogue: Google's Gemma 2 9B (
gemma2:9b), running locally via Ollama, acts as an educational field naturalist that reasons over grounded regional candidate sets. - Offline Field Journal: Every verified sighting is recorded into a collectible deck with vernacular Tamil names (e.g., மாம்பழச்சிட்டு for Common Iora, அரச மரம் for Sacred Fig) and diagnostic field facts.
- Photo Assurance: Optional camera capture to photograph real specimens on the trail for verified badge proof and +25 bonus Nature XP.
- Touch Grass Pocket Mode (Trail Walk): An eye-rest mode where you tuck your phone into your pocket during walks. A gentle outdoor audio radar periodically delivers listening cues ("How many distinct bird calls can you isolate right now?") and counts your outdoor trail time, with accidental pocket-touch protection.
- Explorer Progression & Daily Quests: Gamified nature connection with daily objectives ("Touch Grass Today", "Perched Watcher", "Vibrant Markings") and naturalist ranks from Field Scout to Master Naturalist.
- Voice-First Audio Guide (Optional): Powered by ElevenLabs Scribe v1 (STT) and Turbo v2.5 (TTS), enabling hands-free, eyes-up nature walks.
Demo
The core application runs locally with open-weight Gemma 2 9B and an offline SQLite pack, requiring zero cloud dependencies or paid APIs to identify species and maintain your journal.
The Full Field Identification Loop:
- Notice Wildlife Outdoors: Spot a bird or tree during your walk.
- Describe or Speak: Speak your observation naturally ("small bright green bird with yellow forehead" or "black bird with forked tail on a wire").
- Diagnostic Multi-Turn Inquiry: Gemma 2 9B isolates the top regional suspects and asks you to verify an observable field mark through audio or text.
- Field Corroboration: You confirm or refute. On confirmation, it acknowledges and verifies its perching behavior or habitat before unlocking the card.
- Collection Reveal: The verified species drops into your Field Journal with +50 Nature XP, native Tamil nomenclature, and its real bird call audio clip.
- Pocket Mode: Tap Walk to put your device in your pocket for a mindful 15-minute nature walk with audio listening prompts.
Watch the Field Walkthrough Demo:
(Direct link: YouTube Video Demo)
Code
🌿 Field Guide — Touch Grass AI Nature Companion
-orange.svg){kind=icon}
A voice-first, screen-minimizing field guide built entirely on Open-Weight AI (Google Gemma 2 9B) and Open-Source foundations. Built to help you put the phone down, observe wildlife with your own eyes, and build an offline collectible field journal.
🏛️ Built on Open Weights & Open Source
This project is built from the ground up around open-weight and open-source models, eliminating closed proprietary cloud LLM lock-in and ensuring complete privacy, zero tracking, and true offline capability for hikers and naturalists:
-
Primary Open-Weight Intelligence: Google Gemma 2 9B (
gemma2:9b)- High-capacity 9-billion parameter open-weight reasoning model by Google DeepMind.
- Handles multi-turn biological inquiry, morphological reasoning (beak structures, leaf margins, bark textures), and cross-lingual English/Tamil vernacular matching.
- Runs locally and privately through Ollama with zero data leaving your machine.
-
Edge Tier Support: Google Gemma 2 2B (
gemma2:2b)- Seamlessly falls back…
Architecture & Key Files at a Glance:
| File | Purpose |
|---|---|
app/matcher.py |
Local Gemma 2 9B inference via Ollama, grounded candidate pre-filtering, and hallucination rejection |
app/encounter.py |
Multi-turn Field Detective engine generating diagnostic field mark and behavior questions |
app/voice.py |
Voice layer: ElevenLabs Scribe v1 STT + Turbo v2.5 TTS (with Web Speech API fallback) |
app/pack_builder.py |
Generates grounded regional species packs for any global coordinates using GBIF & Wikidata |
app/main.py |
FastAPI application handling offline collection persistence, quests, and encounter turns |
data/field_guide.db |
Self-contained SQLite database with 105 Coimbatore species, Tamil names, traits, and audio links |
app/static/js/app.js |
Mobile-first frontend with audio player, pocket mode timer, and photo verification |
How I Built It
The Open-Weight AI Core
The core intelligence of Field Guide is powered by Google Gemma 2 9B (gemma2:9b), running locally on-device through Ollama. No closed proprietary API is used for species matching or field mark extraction.
# Pull and run the open-weight reasoning model locally
ollama run gemma2:9b
Why Gemma 2 9B?
Distinguishing between subtle local species (such as the House Crow vs. Large-billed Crow, or the Purple-rumped Sunbird vs. Loten's Sunbird) requires contextual natural-history reasoning that small 1B–2B models often oversimplify or confuse.
Gemma 2's sliding window attention and knowledge density make it capable of fine-grained morphological reasoning over multi-trait descriptions. It comfortably runs on a standard laptop with 8GB–16GB RAM using 4-bit quantization, making open local inference practical without cloud compute costs.
The Grounded Ranker Architecture
Gemma 2 9B does not guess species out of thin air. To completely eliminate AI hallucinations, the pipeline uses a strict two-stage grounded ranker:
[User Observation (Text or Speech)]
│
▼
┌──────────────────────────────────────────────┐
│ Stage 1: Deterministic Python Pre-Filter │
│ Filters 105 local species to Top 5–8 using │
│ colors, size, beak shape & habitat tokens │
└──────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────┐
│ Stage 2: Gemma 2 9B Local Naturalist Ranker │
│ Reasons strictly over the 5–8 candidates: │
│ matches morphology & extracts missing traits │
└──────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────┐
│ Stage 3: Strict Schema & ID Validation │
│ Rejects any hallucinated species not in DB │
└──────────────────────────────────────────────┘
# app/matcher.py: Gemma reasons strictly over verified local candidates
prompt = build_matching_prompt(prefiltered_candidates, observation)
raw_response = call_local_ollama(prompt, model="gemma2:9b")
validated_result = validate_and_ground_candidates(raw_response, verified_species_pool)
Because Gemma receives only verified candidates from the local database, it cannot invent a fictitious bird. If the user's description is vague, the engine reports low confidence and triggers a diagnostic question instead of guessing.
Multi-Turn Detective Dialogue
Rather than spitting out an answer immediately, encounter.py formulates an interactive two-stage inquiry:
- Turn 1 (Diagnostic Field Mark): Inspects the top suspect and prompts the user to check an unmistakable diagnostic marker (e.g., spiky crest, yellow eye patch, scissor-forked tail, needle-curved bill, feathery leaflets).
- Turn 2 (Behavior & Habitat Verification): If confirmed, asks a verification question about whether it is foraging on the ground, high in the tree canopy, or swooping from a wire. If refuted, the engine smoothly pivots to the regional runner-up suspect.
- Turn 3 (Verified Registration): Confirms the specimen, writes it to SQLite, awards XP, and plays its authentic bird call audio clip.
Voice Layer (ElevenLabs Integration)
To keep hands free on muddy trails, Field Guide includes an optional voice layer:
- Speech-to-Text (ElevenLabs Scribe v1): Transcribes natural spoken field observations directly into the matching pipeline.
- Naturalist Speech (ElevenLabs Turbo v2.5): Synthesizes conversational, spoken feedback so you hear prompts in your earbuds without looking down.
-
Offline Fallback: If internet is disconnected or an API key is absent, the app falls back gracefully to the browser's built-in Web Speech API (
SpeechSynthesis), ensuring zero interruptions on remote trails.
Full Technology Stack
| Layer | Component | Notes |
|---|---|---|
| Local LLM | Google Gemma 2 9B (gemma2:9b) via Ollama |
Open-weight, runs on CPU/GPU locally |
| Edge Fallback | Google Gemma 2 2B (gemma2:2b) |
Auto-selected on memory-constrained devices |
| Backend | FastAPI (Python 3.10+) & Uvicorn | Async REST endpoints with CORS support |
| Database | SQLite3 (data/field_guide.db) |
105 species, traits, Tamil names, audio URLs |
| Frontend | Vanilla HTML5, Modern CSS, ES6 JavaScript | Mobile-first PWA, dark navy theme |
| Voice STT | ElevenLabs Scribe v1 | High-accuracy field note speech transcription |
| Voice TTS | ElevenLabs Turbo v2.5 + Web Speech Fallback | Low-latency naturalist narration |
| Data Pipeline | GBIF API + Wikidata + Wikipedia API | Automated regional pack builder for any lat/lon |
Field Testing & Observations
During field tests in parks and residential avenues in Coimbatore:
- What Worked Remarkably Well: The two-stage diagnostic dialogue excelled with easily confused species. When presented with "black bird with a long tail on a wire", the engine correctly isolated the Black Drongo from the Asian Koel by asking about the scissor-forked tail, followed by asking about its swooping insect-catching behavior.
- Tamil Language Integration: Having native vernacular names (செம்மீசைச் சின்னான் for Red-whiskered Bulbul, வேப்ப மரம் for Neem Tree) made the app immediately engaging for local students and elders who know local birds by regional names rather than English ornithological terms.
- Safety First: Botanical guidelines explicitly warn users never to taste leaves, seeds, or berries to identify trees — relying strictly on visual traits (leaf shape, bark texture, canopy spread) and fallen leaf fragrance.
Why Does Open Innovation Matter?
Building Field Guide on open-weight AI and open data fundamentally changed what was possible:
The Core Works with Zero Cell Signal:
Real nature observation happens in river valleys, forested hills, and rural trails where internet connectivity drops to zero. Because Gemma 2 9B runs locally through Ollama and the species database is a local SQLite file, you can identify wildlife and log sightings deep in the Western Ghats foothills in airplane mode. A closed, API-only model would fail the moment you actually stepped into the wild.Absolute Privacy of Field Observations:
Where you walk, what wildlife you find, and your voice notes never need to leave your device. Open local inference keeps personal outdoor exploration private.Auditable & Community-Correctable:
Closed vision models often misclassify regional Indian species as North American lookalikes. Because our dataset and prompts are completely open, every regional field mark, Tamil translation, and trait rule can be inspected, vetted by local birdwatchers, and corrected directly in the repository.Forkable for Any City in the World:
pack_builder.pytakes any latitude and longitude and automatically builds a new regional species pack using open GBIF and Wikidata sources. A naturalist in Bengaluru, Nairobi, or São Paulo can generate their own local pack in 5 minutes and run the exact same app with their local biodiversity.
Prize Categories
- Best Use of Gemma: Google Gemma 2 9B (running locally via Ollama) serves as the core reasoning engine, performing grounded regional species ranking and dynamic diagnostic question generation without hallucinations.
- Best Use of ElevenLabs: Integration of Scribe v1 for trail speech transcription and Turbo v2.5 for low-latency audio guide prompts, enabling an eyes-up, hands-free nature walk experience.
Top comments (1)
Official Platform Update
Security protocols have been updated for all developer accounts.