This is a submission for Weekend Challenge: Dog Days Edition.
What I Built
I have a dog named Chan back home.
I currently live in Canada, far from him, and I often miss the ordinary moments we shared. Our walks were never simply about getting from one place to another. A route I might have crossed without thinking could become a complete investigation for Chan. He would pause at the base of a tree, study the edge of a path, notice movement across the grass, and find something interesting in places I would have overlooked.
Chan the original field researcher behind SNIFF.
While thinking about what to build for the Dog Days challenge, I kept returning to those walks. That became the starting point for SNIFF. I wanted to create an experience that encourages people to slow down and look at a familiar environment with the curiosity of a dog.
SNIFF is a multimodal web experience that turns an everyday photograph into an interactive, dog-oriented visual field report.
Users can upload a JPEG, PNG, or WebP image, capture a scene with their browser camera, or explore a clearly labeled pre-analyzed sample. SNIFF uses Google Gemini to examine visible surfaces, boundaries, pathways, vegetation, lighting, movement, and spatial cues. It then presents the results as numbered markers connected to a discovery dossier.
Each generated Field Report includes:
- A scene classification and short summary
- Four to six visible discoveries
- A relative SNIFF score for each discovery
- A grounded explanation and model confidence
- Normalized coordinates that position discoveries on the image
- A safe, scene-specific observational SNIFF Quest
- An optional Dog View inspired by canine dichromatic vision
The goal was never to claim that software can smell a photograph or read a dog's mind. It was to create a playful reason to inspect ordinary environments from a lower, more dog-oriented perspective while staying honest about what the image actually shows.
SNIFF is my way of carrying a little of Chan's curiosity with me while I am far from home. I dedicate it to him and to everyone who has ever followed a dog on a walk and wondered what made that one particular tree so important.
Demo
Try SNIFF
Watch the walkthrough
The video covers the pre-analyzed sample, interactive discovery markers, Original and Dog View, a live photograph upload, Gemini processing, and the generated Field Report.
Code
The complete project is open source:
SNIFF
The world is different down here.
Live Experience · Video Demo · Source Code
SNIFF is a multimodal web experience that uses Google Gemini to examine an environment from a dog-oriented exploratory perspective.
Users can upload a photograph or capture a scene with their camera. SNIFF analyzes visible features such as surfaces, vegetation, movement, people, pathways, lighting, and environmental structure, then transforms the result into an interactive field report.
The experience is designed as an editorial field guide rather than a traditional AI chatbot or dashboard.
Demo
The demo covers the pre-analyzed sample experience, interactive discovery markers, Original and Dog View, a live photograph upload, Gemini processing, and the generated Field Report.
Overview
SNIFF explores a simple question:
What might stand out if we looked at an everyday environment from a lower, dog-oriented perspective?
For each analyzed scene, SNIFF generates:
- A concise scene classification and summary
- Four to six…
View the SNIFF repository on GitHub
How I Built It
From photograph to Field Report
The application uses React 19, TypeScript, Vite, Tailwind CSS, Motion, the Google GenAI SDK, Gemini, and Vercel Functions.
The analysis pipeline is:
Photograph or camera capture
↓
Client-side image normalization
↓
Vercel /api/sniff function
↓
Gemini multimodal analysis
↓
Schema-constrained JSON
↓
Server-side validation
↓
Client-side validation
↓
Interactive Field Report
Before upload, the browser resizes large photographs to a maximum dimension of 1920 pixels and converts them to JPEG. This reduces transfer size and keeps requests predictable without forcing the user to prepare an image manually.
The server sends the normalized photograph and a grounding-focused instruction to Gemini. Instead of asking for free-form prose, I request structured JSON that can drive the interface directly.
A discovery follows this shape:
{
"label": "TREE BASE",
"category": "exploration",
"interestScore": 92,
"explanation": "The rough tree base creates a distinct natural landmark with a different surface and texture from the surrounding grass.",
"confidence": 0.95,
"location": {
"x": 0.32,
"y": 0.55
}
}
The normalized x and y coordinates keep each marker attached to the same visible feature as the image changes size. Selecting a marker updates the matching dossier entry, and selecting a dossier entry updates the marker.
Using Google AI as part of the interaction
Gemini is not used only to write a paragraph. Its multimodal output controls the structure of the experience:
- Scene classification determines the report context
- Discovery labels and categories populate the dossier
- Coordinates place interactive markers
- Interest scores establish visual hierarchy
- Confidence values communicate model certainty
- Explanations connect every result to visible evidence
- The generated quest gives the user a safe observation prompt
This made structured output essential. The response is validated on the server and again in the browser before it reaches the UI. Coordinates, confidence values, discovery counts, categories, and required text fields all have defined limits.
Grounding the interpretation
A photograph cannot reveal scent molecules, unsupported sounds, hidden objects, or an animal's thoughts. I did not want SNIFF to pretend it could smell a photograph or understand what a dog was thinking. That limitation shaped the prompt, schema, validation, and interface copy.
The Gemini instruction focuses on visible evidence such as:
- Surfaces and terrain transitions
- Vegetation and objects
- People and animals
- Paths, edges, and boundaries
- Lighting and shadows
- Visibly supported movement
- Spatial relationships and texture
It prevents claims about detected smells, chemical traces, unseen objects, emotions, preferences, or predicted behavior. The smell category means a visible feature may have scent-related relevance during exploration. It does not mean an odor was detected.
Dog View follows the same principle. It is a simplified visual approximation inspired by canine dichromatic vision, not a scientific recreation of an individual dog's complete sensory experience.
Designing it as a field guide
I wanted SNIFF to feel less like a dashboard and more like an editorial field guide. The interface uses warm paper surfaces, environmental photography, forest and earth tones, numbered observations, serif display typography, and monospaced report metadata.
Pre-analyzed samples are labeled clearly so they cannot be confused with live Gemini results. Accessibility work includes keyboard-operable controls, visible focus states, reduced-motion support, modal focus management, descriptive marker labels, live loading announcements, and user-controlled sound.
Handling the less glamorous parts
Multimodal requests and browser cameras introduce plenty of failure modes. SNIFF handles unsupported files, oversized uploads, camera permission failures, concurrent analyses, stale camera requests, invalid responses, provider errors, and a 45-second timeout.
Each analysis receives a request ID and an abort controller. If the user changes scenes while a request is active, the older response cannot overwrite the new state.
I also learned that camera streams are tiny chaos machines. They work beautifully until an asynchronous permission request resolves after the modal has closed. Request generation guards are not as photogenic as Dog View, but they help the camera light turn off when it should.
Prize Categories
I am submitting SNIFF for Best Use of Google AI.
Google Gemini provides the multimodal scene understanding behind every live Field Report. Its schema-constrained response does more than generate text: it determines the discoveries, marker positions, scores, confidence values, explanations, and observational quest that power the interface.
What I Learned
The biggest lesson was that structured multimodal output can make a model part of an application's interaction architecture. Gemini does not simply provide copy for SNIFF. Its validated output becomes navigable interface state.
I also learned that grounding cannot live only in a prompt. It needs reinforcement in the schema, runtime validation, UI labels, sample disclosure, and product language.
Most importantly, building SNIFF brought me back to my walks with Chan and something he continues to teach me: curiosity changes a place. A park is no longer just a park when you pay attention to its edges, textures, movement, shade, pathways, and suspiciously interesting tree bases.
This project is for Chan, for the dogs who turn routine walks into daily expeditions, and for the people who happily wait while every leaf receives a thorough inspection.
If you try SNIFF, I hope it makes you pause over one detail you would normally walk past. Chan would probably have found it first.
Live app: sniff-omega.vercel.app
Source code: github.com/SushyamNagallapati/sniff
Video demo: youtu.be/3nAHfvdLjY8
Top comments (0)