From a 15-Second Walk to a Digital Memory: Building a Canine Cognitive Twin

This is a submission for Weekend Challenge: Dog Days Edition

What I Built

Canine Cognitive Twin — a living digital twin for a dog.

A phone captures fifteen seconds of a morning walk. Gemini analyzes the video and returns structured observations plus first-person narration. ElevenLabs turns that narration into speech. Snowflake Dynamic Tables build a rolling baseline, Cortex flags potential gait drift, and Solana records each chapter under the owner's own keys.

One capture becomes one chapter of a memoir the dog "writes" in its own voice — while also creating a longitudinal record that remains under the owner's control.

Not a veterinary device. Not diagnostic. Consult a veterinarian.

Demo Status

There is currently no live demo video or deployed demo available.

Rather than presenting a simulated walkthrough as a live demo, I want to be transparent about what has been implemented and what is currently synthetic.

The project is structured so the complete pipeline can be demonstrated end-to-end once the live demo environment is ready.

Implemented Pipeline

Step	Technology	Purpose
Capture 15s video + audio	Expo	Records the dog's walk
Video analysis	Gemini 3 Pro	Extracts behavior tags, gait asymmetry, emotion, and narration
Text-to-speech	ElevenLabs Flash v2.5	Converts narration into the dog's voice
Data processing	Snowflake Dynamic Tables	Builds rolling summaries and baselines
Anomaly detection	Snowflake Cortex	Detects potential gait drift from historical data
Immutable record	Solana Devnet	Stores the chapter index, content hash, and anomaly flag
Dashboard	Streamlit in Snowflake	Displays the dog's memoir and health radar

Synthetic Data Disclosure

Because anomaly detection requires historical data, the project includes generated history for demonstration and development purposes.

Data	Status	Reason
90 days of gait/vitals history	Synthetic	A single capture is not enough to establish a meaningful baseline
Injected gait drift around day 60	Synthetic	Used to demonstrate anomaly detection
Voice Design voice	Pre-configured	Created once and reused for the dog's persona
Sound effects	Pre-rendered	Sniffing, panting, collar jingles, and environmental sounds are prepared ahead of time
Gemini fallback response	Development fallback	Prevents a temporary API/rate-limit failure from blocking the pipeline

Every dashboard panel that uses generated history is explicitly marked with a SYNTHETIC HISTORY badge.

There is no intentional blending of synthetic history with live data without disclosure.

Architecture

The end-to-end data flow is:

Expo Mobile App
      │
      │ 15s video + audio
      ▼
Cloudflare Worker
      │
      ├──► Cloudflare R2
      │       └── Media storage
      │
      ├──► SHA-256
      │
      └──► Gemini 3 Pro
              │
              ├── behavior_tags
              ├── gait_asymmetry_score
              ├── emotion
              └── narration
                    │
                    ▼
             ElevenLabs
                    │
                    ▼
              Phone Playback

Gemini Result
      │
      ▼
Snowflake RAW_EVENTS
      │
      ▼
Dynamic Tables
      │
      ├──► CLEANED_EVENTS
      │
      └──► DAILY_SUMMARY
                │
                ▼
      Snowflake ML Anomaly Detection
                │
                ▼
          GAIT_ANOMALIES
                │
                ▼
      Streamlit-in-Snowflake
         ├── Memoir
         └── Health Radar
                │
                ▼
          Oracle Relay
                │
                ▼
          Solana Devnet
         record_memory

Why I Made These Architecture Decisions

Oracle Relay Instead of Direct Snowflake Signing

Snowflake Dynamic Tables cannot directly sign Solana transactions or call arbitrary external APIs.

Instead, a lightweight oracle-relay polls Snowflake's V_ORACLE_PAYLOAD, signs the transaction, and submits it to Solana Devnet.

This keeps blockchain operations outside the request path, so a slow Solana transaction doesn't block the more time-sensitive Gemini + TTS pipeline.

The Anchor program still verifies the oracle public key on-chain.

Separate Snowflake Clients

The ingestion worker runs inside the Cloudflare Workers runtime, while the oracle relay runs in Node.js with tsx.

Although both use keypair-based authentication, their runtime crypto APIs are different. I intentionally kept the implementations separate rather than introducing an abstraction that would only serve two different runtime environments.

Fallback Instead of Silent Failure

The live Gemini API can fail because of rate limits or temporary availability issues.

Instead of pretending everything is live, the worker can load a pre-recorded response. The API response contains:

{
  "source": "fallback"
}

The dashboard can then communicate that state instead of hiding it.

Prize Technology

Technology	Used For	Why It Matters
Google AI / Gemini	Multimodal video analysis and structured JSON responses	Provides the actual observation layer
Snowflake	Dynamic Tables, historical baselines, anomaly detection, Streamlit	Turns individual observations into longitudinal data
ElevenLabs	Voice Design and Flash v2.5 narration	Gives the digital twin its personality and voice
Solana	Dog identity and immutable memory records	Keeps the ownership/proof layer outside a centralized database

These aren't decorative integrations. Each technology has a specific role in the architecture.

Code

PrinceXDev / dog-day-edition

canine-cognitive-twin/
├── snowflake/          Dynamic Tables, Cortex anomaly model, Streamlit app
├── gemini-pipeline/    Schema-constrained Gemini analysis
├── elevenlabs-service/ Voice Design, narration, SFX
├── ingestion-worker/   Cloudflare Worker: R2 → Gemini → Snowflake → TTS
├── oracle-relay/       Snowflake → Solana transaction relay
├── solana-program/     Anchor program for dog identity and memories
├── mobile-capture/     Expo capture and upload application
└── demo-assets/        Fallback response and demo assets

What's Next

The next milestone is completing and publishing the live end-to-end demo.

After that, I would like to build:

• Owner-facing export of the memoir and on-chain proof
• Multi-dog households with one Solana wallet and a PDA per dog
• Per-dog voice calibration
• Longer-term behavioral and mobility trends
• Better visualization of changes over time

Prize Categories

I'm submitting this project for:

• Best Use of Snowflake
• Best Use of Solana
• Best Use of ElevenLabs
• Best Use of Google AI

The goal wasn't simply to combine four technologies. Each one solves a different part of the problem: observe → understand → remember → prove.

Final Note

There isn't a live demo to link yet, and I don't want to present a pre-recorded or synthetic walkthrough as if it were live.

The implementation, architecture, generated-data boundaries, and fallback behavior are documented in the repository so the project can be evaluated honestly.

The idea is simple:

A phone, fifteen seconds, and a digital memory that belongs to the dog — and ultimately, its owner.

Thanks for reading. 🐶