Nook

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

Nook: I Built an AI Field Guide That Sends You Back Outside

What I Built

Nook is a nature fieldwork companion that turns your phone into a naturalist's notebook, and then invites you to keep exploring your environment.

You snap a photo of something you found outside: a leaf between two colors, a bark carved by insects, an odd shadow, a pebble that looks like it's been through something. Gemma 4, running as the core of the backend, looks at it with a field naturalist's eye.

The app is for anyone who's walked past interesting things a thousand times and never stopped to actually look. Kids, curious adults, people who want to spend more time outside but don't know where to start.

Demo

https://nook-i5bl.onrender.com

I recorded the demo with my phone to be able to take photos. And also, because it's the main use finality of this project:

Demo video

Welcome tab: the landing screen

• on a smartphone: )

Discovery tab: The result of gemma's analysis

• on a smartphone:
  

• on a computer:
  

Poster tab: The journal of all of your discoveries

• on a smartphone:
  

• on a computer:
  

The full flow in three steps:

1. Observe — snap or upload a photo from your phone (mobile-first, works with capture="environment")

2. Investigate — Gemma 4 returns a structured Field Card:

• What it sees (actual, specific observations from your photo — not generic filler)
• Possible identification + alternatives
• Two hypotheses on why it looks the way it does
• What the photo can't tell (honestly stated)
• An investigation question to sharpen your attention
• A safe outdoor mission to go do immediately
• An Atelier — a concrete creative task you can make from what you found (a leaf rubbing, a bark sketch, a pine cone decoration, a pressed flower)

3. Investigate + Preserve — talk to the ElevenLabs voice agent when you return, then save the session to your local field journal

Code

https://github/RoxyMaxy/Nook

Stack: Flask · Vanilla JS · Gemma 4 via Google AI Studio · ElevenLabs TTS + Agent · Render · Service Worker PWA

How I Built It

The AI layer is two services. ai_service.py sends the image to gemma-4-31b-it via the google-genai SDK with a strict JSON schema prompt — the model has to return eight structured fields, each capped at 25 words, in a concrete naturalist tone. No markdown, no filler, just what it actually sees. parser_helper.py sanitizes the response in case Gemma sneaks in backticks.

The narration layer (narration_service.py) takes the Field Card text and sends it to ElevenLabs TTS (eleven_turbo_v2_5, Rachel voice) as an audio stream. If there's no connection, it falls back to a procedurally generated E-major chime — an actual chord, not silence, so there's always audio feedback.

The voice agent is an ElevenLabs conversational widget embedded in the page. It loads after the Field Card appears, so the user reads what Gemma found, then taps to debrief out loud. The agent knows it's a field guide — it asks follow-up questions, sends you back outside if needed, never claims certainty.

The journal is pure localStorage. Base64 photo, full Field Card JSON, atelier completion status, your notes. It loads on startup so past entries are there when you return. Nothing goes to a server. Nothing is deleted unless you delete it.

The offline layer is a Service Worker that pre-caches the shell. If you analyze a photo while offline, it queues and syncs automatically when connection returns. The app works in a forest with no signal — the UI loads, past entries load, and the analyze call will retry when you're back.

The backend is Flask + Gunicorn, deployed on Render's free tier via render.yaml. Three environment variables: GOOGLE_API_KEY, ELEVENLABS_API_KEY, ELEVENLABS_AGENT_ID.

Why Does Open Innovation Matter?

I built this from Antananarivo, Madagascar.

Closed AI APIs aren't just expensive here — they're often inaccessible. Most require a credit card from a supported country. Payoneer as a transit layer, SWIFT, international transfer fees — all of that just to run POST /analyze. And if the API goes down or the pricing changes, your app breaks.

Gemma running through Google AI Studio changed the calculation. One Google account, one API key, no card required to get started. And because the model is open-weight, the honest path is clear: if Google AI Studio ever becomes a problem, you point the same google-genai client at a local Ollama instance and nothing else changes. The model is swappable. The app doesn't have to die with a vendor decision.

The offline design matters for the same reason. When I take this outside in a park with no signal — which is most of the time — the UI still loads, the journal still loads, the Service Worker still serves the shell. The analysis queues and fires when I'm back on Wi-Fi. The screen is the smallest part of the experience, which is exactly what the prompt asked for.

Open innovation didn't make this easier. It made it possible.

My Agent Session

Dut to some problem in my laptop, I couldn't use DevRelay. But here is my prompt: You are a senior full-stack developer.
Design and generate the complete code for a fieldwork web application called "Nook" (with the tagline: Observe. Investigate. Create.).
This project is an ecological application designed with the open-weight model Gemma 4 at its core framework.

However, the generated code must be complete, production-grade, and immediately functional, before tokens run out.

FRAMEWORK, RENDERING, AND PORT CONSTRAINTS:

• The backend application must strictly use FLASK
• The application must run locally on its native default port: PORT 5000 (localhost:5000).
• The frontend must be served directly by Flask via its static and template folders, using only native HTML5 / CSS3 / Vanilla JavaScript (no separate Node.js frontend server).
• Provide a configuration file for Render for deployment on their free tier.

AI ENGINE (GEMMA 4 ):

• The application's core logic depends on API execution or inference of the "gemma-4-31b-it" or "gemma-4-26b-a4b-it" multimodal model.
• You must use the official new google-genai Python SDK to handle all backend AI operations.
• Configure the default model parameter to use "gemma-4-31b-it", butmake sure it reads from an environment variable GEMMA_MODEL_ID (so the user can easily swap it to "gemma-4-26b-a4b-it" via the .env file).
• The backend must feature full validation and standard JSON parsing structures processing raw output string responses from Gemma safely.

USER STORY:
The goal of Nook is to encourage the user to leave their screen and explore nature.
This is the code pipeline:

1. OBSERVE: A paccessible mobile-first interface (capture="environment") with dark mode posibility). The user snaps or uploads a photo from their phone.
2. ANALYZE: The Flask receives the image, validates the type (jpeg, png, webp, gif), encodes it to base64 bytes, and directly queries the l Gemma 4 model using the google-genai client. The prompt sent to Gemma MUST demand a JSON object . The returned JSON must contain (written in a cautious, naturalist tone, max 25 words per string):
   • "seen": 3 to 6 direct visual observations such as (color, shape, texture).
   • "possible_identification": The possiblr identification might be still wrong, even if just rarely. It is better to state "possible", not confirming if not sure.
   • "alternatives": 1 or 2 other plausible options of why something had became what it was when taken photos
   • "hypotheses": 2 possible explanations regarding the state or origin of the found object
   • "uncertain": What the photo cannot tell due to its limitations. The AI Asks further questions in the investigation_question
   • "investigation_question": The AI asks further questions about the found objects for the AI to confirm hypothesis or suggestions, and helps people develop their observation and attention skills.
   • "observation_mission": An action to do outside immediately without touching or disturbing nature, like following ants' paths to their anthill, or give a rabit a leaf, or caressing barks to feel the relief, etc...
   • "atelier": A creative crafting task that should be a concrete, safe action to do , based with the found objects (those can be a herbier with a clover, a drawing of a bark's patterns, a flower crown, a petal coloration, squirrel's best ohotoshoot, a christmas decoration made from pine cones, leaves, etc...).
3. NARRATE: A Flask endpoint POST /narrate receives the card text and uses the ElevenLabs TTS API (Model: eleven_turbo_v2_5, default Voice ID) to return a binar audio/mpeg stream. The JavaScript interface displays a "Field Card " button to fetch and play this audio stream directly in the browser.
4. INVESTIGATE The interface integrates the official ElevenLabs conversational voice widget (<elevenlabs-convai> loaded via its script) by dynamically injecting the Agent ID. The user speaks to this voice guide in real-time to report their findings after returning from their outdoor mission.
5. PRESERVE: A field journal saves the session locally (base64 photo, Field Card JSON, atelier completion status, and user text notes) into the browser's localStorage. A history of past journal entries must be displayed upon loading the application, fot the hapiness to find memories back and remember what was learned through the little travel outside;.

SECURITY AND ROBUSTNESS:

• Strict exception handling with clean Python logging
• Robust parser helper to sanitize Gemma's JSON string in case it contains accidental markdown backticks ( json ... ).
• Full isolation of secrets using an environment file (.env) containing: GOOGLE_API_KEY, GEMMA_MODEL_ID, ELEVENLABS_API_KEY, ELEVENLABS_AGENT_ID, and optional ElevenLabs voice variables.

STYLE & LAYOUT:
Make the web app present the theme as following:

• Artisanal Minimalism: Clean design without clutter.
  • Smart-Casual Tone: Low pressure, high intention. Cool attitude withpit forgettong the serious execution.
  • Background: #0F0F11
  • Primary Text & Headers: #F5F5F7
  • Secondary Text & Metadata: #86868B
  • Borders & Dividers: #2C2C2E
  • Interactiveity and Hover State: #FFFFFF Those are by default. But people should be able to choose among lots of color themes (blue, orange, purple, green, etc...) and light or dark theme.

USER PROFILE:
Save everything (the user's color themes, options, and nature journal entries) directly inside the browser's memory, never on a distant database. This keeps their data safe and ensures the app works 100% offline.

ONLINE/OFFLINE FUNCTIONALITY:

• The app must work even with zero internet (like deep in the forest).
• Create a 'sw.js' file (Service Worker) so the app's pages and designs load offline.
• If the user takes a photo or finishes a task while offline, save it inside the browser's local memory immediately.
• As soon as the phone gets internet back, the app must automatically send those saved photos and notes to the Flask backend without making the user refresh the page.

RESULTING JOURNAL:
The result of PRESERVE user story: a personalized poster, or a little digital journal, according to the user's choices, contains all the result of the functionalities: the taken photo, the description, the possible causes of their state, cool artistic or useful things to craft from the shot objects.

Generate all the necessary files for this application now, ready to be executed on port 5000.

Prize Categories

• Best Use of Gemma — Gemma 4 (gemma-4-31b-it) is the entire analysis engine. It receives the raw image bytes, returns a structured eight-field naturalist Field Card, and powers every section the user sees. Swappable to gemma-4-26b-a4b-it via one environment variable.
• Best Use of ElevenLabs — Two integrations: TTS narration of the Field Card via eleven_turbo_v2_5 (so the user can listen and go back outside), and the conversational Agent widget for real-time voice debrief when they return.
• Best Use of Render — Flask + Gunicorn deployed on Render's free tier. The render.yaml defines the full service with environment variables. Auto-deploys from GitHub on every push.