I Built NatureSnap + Go: Gemma Turns a Nature Photo into an Outdoor Mission 🌿

Umesh Pal — Mon, 12 Oct 2026 06:59:13 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

What if AI didn't give you another reason to stare at your screen, but a reason to put your phone away?

That's the idea behind NatureSnap + Go: a small experiment in using multimodal AI to turn a photo of something in nature into a reason to step outside.

Upload a photo of a leaf, flower, bird, insect, or tree. Gemma 4 helps identify what you're looking at, points out a detail worth noticing, and suggests one outdoor mission you can complete in 5, 15, or 30 minutes.

We can spend a surprising amount of time looking at nature through screens: searching for the name of a plant, wondering what kind of bird we just saw, or scrolling through pictures instead of exploring what's around us.

I wanted to build something that uses AI to make that first step into the real world a little easier.

NatureSnap + Go turns a photo of something in nature into a small invitation to explore.

Here's how it works:

Snap: Upload a photo of a plant, flower, tree, bird, insect, or something else from nature. You can also choose an example photo or add a short note about where you found it.
Discover: Gemma 4 examines the image and provides a likely identification, a short explanation, and a confidence label: High, Medium, or Low.
Look closer: Instead of giving you a long article to read, it points out one specific detail worth noticing.
Go: It suggests one outdoor mission with a time estimate. You can choose whether you have 5, 15, or 30 minutes, and the mission is tailored to the time you've available.

The goal isn't to turn nature into another endless AI conversation. It's to give you enough curiosity and direction to step outside, look more carefully, and discover something for yourself.

The name is only there to make the mission specific. The mission is the product, and the screen is meant to be the shortest part of the experience.

Here is a real answer from my first test with a photo:

Example: Identifying a hibiscus flower

Likely Hibiscus flower (Flower)
This is a vibrant red hibiscus flower with large, delicate petals. It features a long, prominent central stamen topped with yellow pollen.
Look closer: Look at the very tip of the long pink tube to see the small, rounded stigma
Mission: Walk around your garden or a nearby park to find a plant with a different color of flower. Observe how its shape compares to this hibiscus.

I kept it small on purpose: no accounts, no history, no map, no chat window. You can add a short note ("Found near my society, in a park…") and a time, and that's all the setup there is. I built it with everyday nature in Indian neighbourhoods in mind, which is why the prompt's own example is a Peepal tree.

Demo

🌿 Try NatureSnap + Go:
nature-snap-go.vercel.app (no sign-up; tap one of the four example photos if you don't have one handy)

Code

GitHub repository: https://github.com/72umesh/NatureSnap-Go

The repository includes the frontend, API handlers, Gemma integration, prompts, safety checks, Sentry instrumentation, and automated tests.

MIT licensed. It has automated tests for the reply parser, the safety check, the API endpoints and the Sentry trace link.

How I Built It

This was my first project integrating an AI API into a complete application, and I wanted to learn more than simply sending a prompt and displaying the response.

The main stack is:

React + Vite for the interface
Tailwind CSS for styling
Gemma 4 (gemma-4-26b-a4b-it) through the Gemini API for image understanding and mission generation
Node.js for server-side validation, model requests, response parsing, and safety checks
Sentry for tracing, error monitoring, and connecting user reports to the relevant AI request
Vercel for deployment

The flow

The browser shrinks the photo to a 1024px JPEG before sending it, which keeps uploads fast and well under Vercel's 4.5 MB request limit. A small Node endpoint validates the upload, rate-limits requests, keeps the API key off the browser, and returns clean JSON to a React + Vite + Tailwind front end. It's deployed on Vercel. There's no database.

One prompt, three jobs

The photo and the note go to gemma-4-26b-a4b-it through the Gemini API in a single multimodal request. One system prompt gives the model three connected jobs: understand the photo, help the user observe it, and get them outside. It replies in a fixed labelled format:

TITLE: [short name, e.g. "Likely Peepal tree"]
CATEGORY: [Plant, Tree, Flower, Bird, Insect, Fungus or Other]
CONFIDENCE: [exactly one word: High, Medium or Low]
WHAT I SEE: [2-3 short sentences]
LOOK CLOSER: [one thing to observe, 1 sentence]
MISSION: [exactly ONE activity, 1-3 sentences]
TIME: [rough estimate, e.g. "~5 minutes"]

I used labelled lines instead of relying on a structured-output mode. A tolerant parser reads them, copes with stray markdown, and fills sensible defaults if a field is missing, so a slightly messy reply doesn't break the screen.

The prompt choices that matter most:

Honest uncertainty. If Gemma isn't sure it must start the title with "Likely" and say so in the confidence label. A confident wrong name is worse than a hedged right one.
One mission, not a list. Lists get skimmed. One instruction gets done.
Fit the time. If you tapped "15 min", the mission has to fit 15 minutes.
The note is context only. What you type is wrapped as a note and the model is told never to follow instructions inside it.
A non-nature photo gets a kind answer with Low confidence and a mission to go and find something to photograph.

Safety, enforced twice

The prompt tells Gemma never to suggest touching, eating, picking, collecting or approaching anything unfamiliar. But prompts can fail, and I'm sending people outdoors, so a second check runs on every reply on the server. If the mission still suggests touching or picking something, it's swapped for a safe one and logged:

const RISKY = /\b(touch(?:ing)?|pluck(?:ing)?|taste|eat(?:ing)?|squeeze|handle|collect(?:ing)?|catch(?:ing)?|feed(?:ing)?|pick(?:ing)?\s+(?:up|it|them|some|any|fallen|a\s+(?:leaf|flower|berry|fruit|twig|stone)|leaves|flowers?|berries))\b/gi;
const NEGATION = /\b(don'?t|do not|never|avoid|without|not|no need to|instead of)\b/i;
// "Don't touch it" passes. "Pick some leaves and taste one" does not.

My first version flagged "Pick a quiet spot and watch the bird" because it saw the word pick. I narrowed the rule to picking up or collecting specific things and added tests for both the harmful and the harmless sentences.

Every screen also carries a small reminder that identification is a best guess, not an expert's answer.

Why I added Sentry

I didn't want AI failures to disappear behind a generic error message. I wanted to understand what happened when a model request failed, how it performed, and what to investigate when a user reported a bad answer.

I integrated Sentry tracing around the API and model call. The traces include information such as the model, request status, retry count, and token usage when the provider returns it. Estimated cost can also be recorded when pricing configuration is supplied.

I also added a Report this answer feature.

A user can report an answer for reasons such as an incorrect identification, an unsafe suggestion, or a mission that doesn't really get them outdoors. The report carries the relevant answer metadata and Sentry trace context, allowing the report to be associated with the original request rather than becoming an isolated complaint.

The report doesn't need to attach the original photo or the full conversation to provide that connection.

For me, this was an important lesson: building with AI isn't only about generating a good answer. It's also about making failures observable and giving yourself a way to investigate them.

I added automated tests for API validation, response parsing, safety handling, report processing, and Sentry trace correlation. I then ran the test suite and tested the deployed site.

The second-photo follow-up experience is still unfinished, so it isn't part of the completed core flow I'm presenting here. I'd like to return to it later.

Sentry in action: User reports for an inaccurate identification and a mission that didn't encourage outdoor exploration appear alongside an application error. This gives me a starting point for investigating issues instead of relying only on user feedback.

What Went Wrong

Three tests failed because I missed an outdated handler. After merging a larger update, three tests failed because handler.js hadn't been updated with the latest changes. The model name, safety check, and Sentry trace link were missing from the old file. The tests helped me catch the inconsistency before shipping the wrong version.
I didn't notice a lost connection. During development, the connection dropped in between, but I didn't realise it at the time. The same thing happened while pushing changes to Git. It taught me not to assume that a command or update has completed successfully just because I started it.
My deployed app didn't work because I missed an environment variable. The live Vercel deployment initially failed because I hadn't configured the required environment variable there. I added it and redeployed the app, which fixed the issue.
My safety check was too broad at first. The initial rule flagged “Pick a quiet spot and watch the bird” as unsafe because it matched the word pick. I narrowed the rule to focus on potentially harmful actions and added tests for both unsafe and harmless sentences.

Why Does Open Innovation Matter?

I want to be precise, because "open" gets claimed loosely.

What open weights gave me:

A swappable model. One environment variable (GEMINI_MODEL) moves the app from gemma-4-26b-a4b-it to gemma-4-31b-it. The prompt, parser, safety check and UI don't change.
Guardrails that don't depend on the model. Because I can swap models, I can't assume any one of them behaves. The safety check and the confidence handling live in my code, so they stay if the model changes.
Behaviour I control. What the app does lives in a prompt in my repo, not in someone else's app settings. When a mission was too vague or too risky, I changed the prompt and the check myself.
A project anyone can run. The repo is MIT licensed. Bring your own API key, run npm install and npm run dev. What it is not: this version calls the hosted Gemma through the Gemini API, so it needs internet, and your photo goes to Google's servers. That's a real trade-off for something you use on a walk, and a truly offline version would be better for the theme. My laptop has 8 GB of RAM and already struggles with VS Code and a browser open together, so running a vision model locally wasn't realistic for me. The repo includes an optional local provider (LLM_PROVIDER=ollama) that I have not run on real hardware, so I'm not claiming an offline version.

I haven't benchmarked Gemma against a closed model, so I can't say it was better on accuracy. What I can say is that building on an open model let me swap it, wrap it in my own guardrails, and open-source the whole thing. I built and tested the app using a free-tier Gemini API key.

Prize Categories

Best Use of Gemma: Gemma 4 does the multimodal work, reading the photo and the note together in one request through the Gemini API (Google's hosted serving). The model is swappable with one setting, and my own safety check sits around it.
Best Use of Sentry Agent Tracing: every Gemma call is traced with its model, input and output tokens and duration. Errors are captured, and a user can report a bad answer in one tap and have it joined to the exact trace. This is a single-call app rather than a full agent, and I say so above.

Try it

Open nature-snap-go.vercel.app on your phone, snap the first plant, bug or bird you walk past, and then actually go and do the mission.

I'd love to hear what you photographed, and whether the confidence label was right. Tell me in the comments what you'd want it to say.

DEV Community: Umesh Pal