DEV Community: Kavya Goel

FieldEval: Testing Gemma 4 on Real Outdoor Fieldwork Photos

Kavya Goel — Sun, 11 Oct 2026 12:47:42 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

When photographed flat on a table, Gemma 4 31B counts four leaves on a specimen correctly. When the phone tilts forty-five degrees and sunlight casts diagonal shadows across the stem, the count flips from four to five. On an outdoor surveyor tag, bright direct glare obscures digits that the human eye reads in half a second.

These failure modes do not appear on synthetic benchmark tables. Standard vision evaluations test models on clean, centered indoor datasets under flat lighting. FieldEval is a mobile web app that takes model testing into the field.

What I Built

FieldEval is an offline-first Progressive Web App (PWA) for testing open-weight vision models on outdoor fieldwork photos. It is built for field researchers, surveyors, and gardeners who need to know if a model can handle wet leaves, high-contrast shadows, and steep camera angles before trusting it outdoors.

The app uses two core protocols:

Commit-then-reveal: You walk outside and pull a scene from the 6-scene specimen deck. You photograph the specimen, enter your answer, and tap "Lock my answer". Your ground truth is committed to local storage with a timestamp before the model is called. The model answers blind. Only after the model responds does the screen slap a diagonal flagging-tape banner across the photo: Flag Lime for a match, Flag Pink for a miss.
Paired shots: You photograph each scene twice—variant A (flat lay or diffused light) and variant B (inclined angle or direct sunlight). FieldEval compares the two to compute the flip rate: how often changing the camera angle or lighting changes the model's answer, even when the physical specimen did not change.

Demo

Live Web App: [TODO: Paste your live Vercel URL here, e.g. https://fieldeval.vercel.app]
Video Walkthrough: [TODO: Paste your 60-90 second demo video link here, e.g. YouTube or Loom]

The 6-Scene Results Wall

At the end of a walk, FieldEval displays The Wall: a visual matrix of every scene tile showing matched, missed, and excluded shots, alongside exact Wilson 95% confidence intervals and paired flip rates.

Judges can also tap "Sample data from the author's field session" directly on the live site to load real fieldwork observations and inspect the complete scoring breakdown without consuming API credits.

Code

CoderKavyaG / FieldEval

FieldEval 🌲📱

A camera-first outdoor vision model evaluation Progressive Web App (PWA) testing open-weight models (Gemma 4 31B) against human ground truth on outdoor fieldwork photos.

Built for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass.

What It Is

Most vision benchmarks evaluate multimodal models on curated, high-resolution indoor datasets under flat studio lighting. Real fieldwork is different: harsh direct sunlight causes high-dynamic-range glare, foliage overlaps in dense 3D planes, and steep perspective angles compress object geometry.

FieldEval is an offline-first mobile web application that takes model evaluation outside. A field surveyor walks through a 6-scene evaluation deck, captures photographs of real physical specimens, locks their human ground-truth answer before any model is invoked (commit-then-reveal), and then queries an open-weight vision model (Gemma 4 31B) to measure exact-match accuracy, paired flip rates across lighting/angles, and failure boundaries.

Screenshots

1. Field Deck & Ground Truth	2. Commit-Then-Reveal

…

View on GitHub

Repository: https://github.com/CoderKavyaG/FieldEval
License: MIT License (Application), Apache 2.0 (Gemma 4 model weights)

How I Built It

FieldEval is built around Gemma 4 31B (gemma-4-31b-it), an open-weight vision model released by Google DeepMind under the Apache 2.0 license.

Architecture

Framework: Next.js App Router (Node.js runtime) with TypeScript strict mode.
Client Storage: idb (IndexedDB). All full-size photos, session metadata, and human answers stay on the phone. The camera uses native <input capture="environment">, avoiding WebRTC permission prompts.
Image Normalization: Client-side canvas scales images to a 1024px maximum edge and re-encodes to JPEG at quality 0.82. This drops all EXIF metadata and GPS coordinates before transmission.
Server Route: POST /api/infer proxies requests to the Google Gemini API using @google/genai. The server constructs prompts from fixed task definitions; client-supplied arbitrary prompts are rejected.
Observability: @sentry/nextjs traces the pipeline with four discrete spans: validate, provider_call, parse, and retry. Strict privacy rules strip API keys, request bodies, task prompts, and raw image bytes.

Why Does Open Innovation Matter?

Open innovation is what makes FieldEval viable:

Swapping Model Sizes with One Config Change: Because Gemma is open-weight, we can run Gemma 4 31B in cloud inference, swap to Gemma 4 9B with a single environment variable (MODEL_ID), or point the client to a local quantized model running via Ollama without changing the application logic or data schema.
API Keys Never Touch the Phone: The client never handles provider credentials. A single serverless route proxies inference with strict rate limiting.
Photos Stay on the Device: In field biology, location and specimen data can be sensitive. Raw photos and GPS coordinates are never uploaded to an image hosting service or cloud database.
Zero Infrastructure Cost: The entire system runs within free-tier serverless limits with no database fees.

What 20 to 30 Photos Cannot Prove

A walk of 20 to 30 photos taken by one person on one afternoon cannot prove general model capabilities across global biodiversity or varied geographic biomes.

What it does prove is where specific visual boundaries collapse: specular highlights on wet surfaces, shadow-induced edge hallucination, and perspective foreshortening. By measuring paired invariance, FieldEval tests real field stability rather than synthetic benchmark claims.

Prize Categories

FieldEval qualifies for two partner prize categories based on technology used in the repository:

Best Use of Gemma: We use Google DeepMind's open-weight vision model Gemma 4 (gemma-4-31b-it) via @google/genai across the entire evaluation pipeline to score real physical field specimens.
Best Use of Sentry Agent Tracing: We instrumented the inference pipeline using @sentry/nextjs with a root trace (FieldEval Inference Pipeline) and four spans (validate, provider_call, parse, retry), capturing latency, token counts, and status while enforcing strict zero-PII redaction.

Author: @ccoderkavyag

I built OrderMind so my friend Ishan stops rebuilding orders from WhatsApp chats

Kavya Goel — Sun, 04 Oct 2026 11:24:21 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

I built OrderMind for my friend Ishan Kumar, who runs InTheBox, a packaging company that provides consultation, design and manufacturing.

Ishan's customers don't fill in structured order forms. They message him on WhatsApp and Instagram: a text, then a voice note, then a photo of a box they like, then "make it a little taller", then "actually 200 extra", then "same material as last time".

The real order lives across messages and media files, and Ishan has to reconstruct it himself.

In Ishan's words:

“A lot of order details are scattered across WhatsApp and Instagram, so I often have to go back through old conversations to figure out what the customer actually confirmed.”

OrderMind takes that mess and turns it into one order you can trust.

Dump it in. Import WhatsApp conversations, voice notes, photos and PDFs.
See what the customer actually asked for. Every extracted field keeps its source evidence and is marked CONFIRMED, INFERRED, MISSING or CONFLICTING.
Catch problems early. "Make it 1 cm taller" becomes an inferred change that Ishan can confirm. "Same material as last time" is checked against the previous order, and a mismatch is surfaced as a conflict instead of silently guessed.
Follow the real business workflow. Orders are handled according to InTheBox's service structure across consultation, design and manufacturing.
Respect business boundaries. Requests outside the business scope, such as logo creation or marketing copy, are flagged rather than being treated as packaging orders.

The core principle behind OrderMind is simple:

AI must never silently turn a guess into business truth.

Demo

Watch the walkthrough

Code

CoderKavyaG / OrderMind

OrderMind is an AI-powered order intelligence platform that turns messy WhatsApp and Instagram conversations into structured, reliable, production-ready orders. It detects changes, missing details, and conflicts while preserving the context behind every order.

OrderMind — Precision Packaging Intelligence from Unstructured Customer Chats

OrderMind bridges the costly disconnect between commercial sales communication (WhatsApp threads, raw voice memos, and mockup sketches) and precision manufacturing execution for packaging converters and box manufacturers.

The Problem: Why We Built OrderMind

Packaging manufacturing is an unforgiving custom-manufacturing industry with razor-thin margins and massive financial risk:

Unstructured Communication Chaos: B2B packaging clients don't submit structured ERP orders. They communicate via fragmented WhatsApp chats, audio messages on the go ("make it 20mm taller and use the same gold foil as last month"), and rough photos of competitor boxes.
The "Game of Telephone": Customer service reps manually summarize chats into emails for estimators. Estimators pass notes to prepress CAD technicians. Subtle change requests ("actually change 100 to 500 pcs", or "same GSM as the festival run") are missed, resulting in ₹1,00,000+ substrate waste on the…

View on GitHub

How I Built It

Open-source AI at the core: Gemma

OrderMind uses Gemma as the core model for understanding customer conversations and extracting structured order information.

The model sits behind a small LLMProvider interface, which keeps the AI layer replaceable without coupling the rest of the application to one model provider.

For the deployed demo, Gemma is accessed through OpenRouter using google/gemma-3-27b-it. During development, the same interface can also work with local Ollama inference.

Extraction takes roughly 0.8 seconds per message in the current implementation.

The pipeline

The system is deliberately split into separate stages rather than relying on one giant prompt.

Normalize every source

WhatsApp exports, voice notes, images and PDFs are converted into a common internal message format.

Transcribe voice notes

Voice input is converted into text before entering the same extraction pipeline as normal messages.

Extract typed claims

Gemma extracts structured claims from each message and resolves references such as "same as last time".

Validate evidence

Every extracted claim must include an exact quote from the source message. If the evidence does not exist, the claim is rejected.

Build immutable order events

Claims become order events. The model does not directly write the current order state.

Reconstruct deterministic state

A deterministic reducer replays the events to calculate the current order state.

Detect changes and conflicts

Deterministic logic identifies changed values, missing fields, conflicting information and out-of-scope requests. Gemma can help phrase clarification questions, but it does not become the source of truth.

Human confirmation

When information is uncertain or conflicting, the user reviews and confirms it before it becomes part of the canonical order.

Production handoff

Once the required information and business conditions are satisfied, OrderMind can generate a structured production brief.

The important architectural decision

The most important design choice was keeping AI output separate from business state.

Instead of:


text
Customer message
→ LLM
→ update database