PawSpective: See the World Closer to How Your Dog Sees It

This is a submission for Weekend Challenge: Dog Days Edition.

Have you ever bought a bright red toy that looked impossible to miss only to watch your dog struggle to find it on green grass?

Humans and dogs do not experience color in the same way. That inspired me to build PawSpective, an application that combines a canine-vision approximation, AI-assisted scene understanding, deterministic visibility calculations, and playful narrated videos.

The central idea is one connected experience:

Capture a scene → explore a canine-vision approximation → discover which objects remain visible → turn the moment into a fictional story.

I also wanted PawSpective to be entertaining without pretending it can read a dog’s mind. It does not claim to determine exact canine vision, gaze, thoughts, feelings, smell, or intent.

What I Built

PawSpective helps dog owners explore how objects in an everyday scene may appear under a canine-inspired color transformation.

A user can:

1. Create a lightweight dog profile.
2. Open the Live Dog Lens.
3. compare Human View with a canine-vision approximation.
4. Record or upload a short video.
5. Let Google Gemini identify visible objects and scene events.
6. Review, rename, or remove incorrect AI detections.
7. Calculate the relative dog-visible contrast of an object.
8. Explore possible attention cues through the Curiosity Map.
9. Compare alternative toy colors in the Toy Color Lab.
10. Generate and download a fictional narrated Story Reel.

Live Dog Lens

The Live Dog Lens uses a browser-based WebGL transformation to provide an immediate Human/Dog Vision comparison.

The transformation preserves more blue and yellow differentiation while reducing red and green differentiation. A comparison slider makes the effect easy to understand, and a dog-height guide helps users frame the scene from a lower point of view.

This is deliberately described as a canine-vision approximation, not an exact reconstruction of what a particular dog sees.

AI scene analysis

After a user uploads a 5–15 second video, PawSpective sends a normalized, silent version of the clip to Google Gemini.

Gemini returns structured scene evidence, including:

• Visible object labels
• Approximate bounding boxes
• Timestamps
• Object categories
• Confidence values
• Visible evidence
• Coarse motion levels

The response must pass a strict Pydantic and JSON Schema contract before the application accepts it.

Users can then remove incorrect events, rename objects, and select which object should be analyzed. These corrections become the source of truth for visibility calculations and Story Reel generation.

Visibility Lab

The Visibility Lab measures how strongly a selected object differs from its nearby background.

The backend:

• Seeks to the event timestamp.
• Samples pixels inside the object’s corrected bounding box.
• Samples a surrounding background region.
• Applies the same canine color transformation used by the frontend.
• Converts colors into CIE Lab.
• Measures foreground/background color and luminance separation.
• Produces a relative dog-visible contrast score.

The result is presented as a product score and qualitative visibility band—not as a scientific probability.

For example:

Dog-visible contrast: High

The blue toy remains visually distinct from the surrounding grass after the canine-vision transformation.

Curiosity Map

The Curiosity Map highlights objects that may visually stand out in the scene.

It combines:

• AI-inferred motion
• Measured dog-visible contrast
• Apparent object size
• A small optional profile-relevance bonus

Every result explains why it appeared. PawSpective calls these possible attention cues, because the application does not track a dog’s gaze or know what the dog is actually paying attention to.

Toy Color Lab

Toy Color Lab compares six screen colors against the measured background around a selected object.

It keeps the surrounding background unchanged, simulates alternative colors for the object, applies the canine transformation, and ranks the colors by approximate dog-visible contrast.

The preview is illustrative—it does not claim that a physical toy will look exactly the same or guarantee a dog’s response—but it makes the red-versus-blue-on-grass problem immediately understandable.

Story Reel

The final step turns the reviewed scene into an 8–10 second vertical video.

Google Gemini uses only the user-reviewed scene timeline to generate grounded story and animation direction. ElevenLabs creates the fictional dog narration, and FFmpeg combines the video, narration, captions, overlays, music, watermark, and disclaimer into a downloadable 9:16 MP4.

The result remains clearly labeled Just for fun.

Demo

The application is available at:

👉 https://pawspective.onrender.com/

Demo note: [My Gemini API daily usage limit ran out, so I couldn’t publish the complete demo.]

The interface and deployed application can still be explored through the link above. The complete demonstration will be added to the repository once the provider usage limit is available again.

Code

The complete source code is available on GitHub:

kernelKain / pawSpective

See the world closer to how your dog sees it. PawSpective combines canine-vision simulation, AI-powered object visibility analysis, curiosity mapping, and playful narrated story reels.

PawSpective

See the world closer to how your dog sees it. PawSpective combines a canine-vision approximation, reviewed AI scene analysis, deterministic visibility scoring, color comparison, curiosity mapping, and downloadable fictional Story Reels.

What it does

• Opens a live Human/Dog Vision comparison or accepts a 5-15 second video.
• Detects visible objects with Gemini and lets the user correct the result.
• Measures foreground/background contrast with OpenCV and CIE Lab color.
• Shows possible attention cues in a timestamp-aligned Curiosity Map.
• Compares six screen colors in Toy Color Lab.
• Creates an 8-10 second animated dog-height POV reel with fictional narration.
• Supports a SHA-256-bound controlled demo for offline rehearsals.

PawSpective labels deterministic calculations as Research-grounded model interpretation as AI-inferred, and fictional output as Just for fun. It does not claim exact canine vision, gaze, thoughts, emotions, smell, intent, or behavioral diagnosis. See the AI disclosure for data flow, limitations, and provider details.

…

View on GitHub

Repository: https://github.com/kernelKain/pawSpective

The repository also contains the product contract, AI disclosure, deployment guide, exported schemas, automated tests, Docker configuration, and release checklist.

How I Built It

PawSpective uses a Next.js frontend and a Python media-processing backend.

Frontend

The frontend uses:

• Next.js 16
• React 19
• TypeScript
• WebGL for the live color transformation
• Canvas and video APIs for previews and overlays
• Vitest and Testing Library
• Playwright for end-to-end smoke testing

The browser handles the live Dog Lens, comparison slider, dog profile, recording and upload flow, object-correction interface, visibility results, Toy Color Lab, Curiosity Map, job polling, and Story Reel download.

Backend

The backend uses:

• FastAPI
• Pydantic
• Google Gemini
• OpenCV
• NumPy
• ElevenLabs
• FFmpeg and FFprobe
• SQLite

FastAPI validates uploaded media, normalizes videos, coordinates AI requests, performs deterministic calculations, and manages background Story Reel jobs.

SQLite stores job metadata and progress, while generated media is stored on a writable backend volume. Story rendering runs through a bounded background worker so the frontend can submit a job, poll its progress, and download the completed reel.

Google Gemini

Gemini is used for three related tasks:

1. Scene analysis
   
   Gemini examines the short clip and returns visible objects, approximate bounding boxes, timestamps, motion levels, and supporting visual evidence.

2. Grounded story generation
   
   Gemini receives the reviewed event timeline instead of being allowed to invent details directly from the raw video.

3. Animation direction
   
   Depending on the configured model and access, Gemini Omni or Veo can create a canine-vision-inspired artistic edit using the source clip or reference frames.

One of my most important engineering decisions was requiring structured model output. Gemini responses are validated against strict schemas, and malformed coordinates, unsupported fields, impossible timestamps, or ungrounded results are rejected.

ElevenLabs

ElevenLabs generates the complete fictional dog narration for the Story Reel.

Only the finalized narration text is sent to ElevenLabs—the original video is not. The selected voice is intentionally described as a fictional dog voice, rather than an attempt to recreate a real animal’s voice or internal thoughts.

The application also supports narration failure handling and cached demo output so a temporary voice-service problem does not break the entire user journey.

Deterministic computer vision

I did not want an AI confidence value to become a fake scientific visibility score.

Instead, the Visibility Lab and Toy Color Lab use deterministic OpenCV and NumPy calculations. They sample pixels from the video, apply the documented color transformation, convert the colors to CIE Lab, and measure relative foreground/background contrast.

This creates a clear boundary between what the model inferred and what the application calculated.

Video composition

FFmpeg performs the final media work:

• Video normalization
• Portrait 9:16 composition
• Captions
• Narration synchronization
• Curiosity overlays
• Visibility result cards
• Music
• Watermarking
• Final MP4 encoding

Using a fixed template made the rendering pipeline more reliable and kept the project focused on one polished experience instead of becoming a general-purpose video editor.

Reliability and responsible AI

PawSpective separates its output into three categories:

• Research-grounded: deterministic transformations and calculations
• AI-inferred: visible-object and motion interpretation from Gemini
• Just for fun: fictional narration and story framing

The application includes recovery paths for camera denial, unsupported browsers, malformed AI output, dark footage, empty detections, narration failures, rendering errors, portrait and landscape uploads, and missing profile photos.

It also supports a controlled rehearsal mode. Cached analysis and media are bound to one exact video using a SHA-256 fingerprint, preventing saved bounding boxes from being incorrectly applied to an unrelated upload.

This separation became one of the most important lessons from the project: an AI product can be playful while still explaining where every result came from.

Prize Categories

Best Use of Google AI

Google Gemini powers PawSpective’s structured scene analysis, grounded story generation, and optional video-animation workflow.

Gemini is not used as an unrestricted narrator. Its output is schema-validated, reviewed by the user, and checked against visible scene evidence before it enters later stages of the application.

Best Use of ElevenLabs

ElevenLabs converts the grounded fictional script into narration for the downloadable Story Reel.

Voice generation is integrated into a complete media pipeline rather than presented as an isolated text-to-speech example. The narration is synchronized with captions, scene events, overlays, music, and the final vertical video.

What I Learned

The hardest part was not adding more AI. It was deciding which parts should not depend on AI.

Gemini is useful for interpreting a scene, but object visibility is better represented by deterministic calculations over measured pixels. Storytelling can be imaginative, but it must remain grounded in events the user has reviewed.

That led to the design principle behind PawSpective:

Use AI for interpretation and creativity, deterministic code for measurable calculations, and clear labels wherever uncertainty remains.

PawSpective started as a fun question about how dogs experience color. It became an experiment in building an AI-powered product that combines delight, practical insight, transparency, and responsible storytelling.