This is a submission for Weekend Challenge: Dog Days Edition
What I Built
Canine Cognitive Twin — a living digital twin for a dog.
A dog owner uploads a short morning-walk video directly from the browser. Gemini analyses the video and returns structured observations plus first-person narration. ElevenLabs turns that narration into the dog's voice. Snowflake Dynamic Tables transform individual observations into a longitudinal baseline, while Cortex identifies potential gait drift that may be difficult to notice manually. Solana records each chapter with a content hash and anomaly flag.
The result is more than a video analyser:
One walk becomes one chapter in a digital memoir — with a longitudinal behavioural record and an ownership/proof layer.
Not a veterinary device. Not diagnostic. Consult a veterinarian.
Demo
🎥 Live Demo
💻 Source Code
View the Canine Cognitive Twin repository on GitHub
Demo Status
The demo demonstrates the implemented end-to-end architecture while clearly distinguishing live processing from synthetic historical data used for anomaly detection.
The browser upload, Gemini analysis, ElevenLabs narration, Snowflake pipeline, anomaly detection, dashboard, and Solana recording are implemented as separate services connected through the ingestion pipeline.
Historical gait data used to establish the anomaly-detection baseline is intentionally synthetic and is explicitly labelled as SYNTHETIC HISTORY in the dashboard.
Implemented Pipeline
| Step | Technology | Purpose |
|---|---|---|
| Browser video upload | React + Vite | Allows the owner to upload a dog-walk video directly from the browser |
| Video analysis | Gemini 3 Pro | Extracts behaviour tags, gait asymmetry, emotion, and first-person narration |
| Structured output | Gemini responseSchema
|
Keeps model output predictable and machine-readable |
| Text-to-speech | ElevenLabs Flash v2.5 | Converts the dog's narration into speech |
| Data processing | Snowflake Dynamic Tables | Transforms raw observations into cleaned events and longitudinal summaries |
| Anomaly detection | Snowflake Cortex | Detects potential gait drift against historical patterns |
| Dashboard | Streamlit in Snowflake | Presents the dog's memoir and health-radar style insights |
| Immutable record | Solana Devnet | Records chapter index, content hash, and anomaly flag |
| Ownership layer | Solana PDA | Gives each dog a persistent on-chain identity |
Synthetic Data Disclosure
Anomaly detection requires a historical baseline. A single video capture cannot establish meaningful longitudinal behaviour, so the repository includes generated history for demonstration and development.
| Data | Status | Reason |
|---|---|---|
| 90 days of gait/vitals history | Synthetic | Required to establish a baseline for anomaly detection |
| Injected gait drift around day 60 | Synthetic | Demonstrates that the anomaly pipeline can identify a change from baseline |
| Voice Design voice | Pre-configured | Designed once and reused as the dog's consistent voice persona |
| Sound effects | Pre-rendered | Sniffing, panting, collar jingles, and environmental sounds are prepared ahead of time |
| Gemini fallback response | Development fallback | Prevents a temporary API/rate-limit failure from stopping the demo |
The synthetic history is generated by snowflake/generate_synthetic_history.py.
Every dashboard panel relying on generated historical data is explicitly marked with a SYNTHETIC HISTORY badge.
There is no intentional blending of synthetic historical data with live observations without disclosure.
Architecture
The end-to-end data flow is:
React Web App
│
│ Browser video upload
▼
Cloudflare Worker
│
├──► Supabase Storage
│ └── Media storage
│
├──► SHA-256
│
└──► Gemini 3 Pro
│
├── behavior_tags
├── gait_asymmetry_score
├── emotion
└── narration
│
▼
ElevenLabs
│
▼
Web App Playback
Gemini Result
│
▼
Snowflake RAW_EVENTS
│
▼
Dynamic Tables
│
├──► CLEANED_EVENTS
│
└──► DAILY_SUMMARY
│
▼
Snowflake ML Anomaly Detection
│
▼
GAIT_ANOMALIES
│
▼
Streamlit-in-Snowflake
├── Memoir
└── Health Radar
│
▼
Oracle Relay
│
▼
Solana Devnet
record_memory
Why I Made These Architecture Decisions
Browser Upload Instead of a Mobile App
The final implementation uses a browser-based upload flow instead of requiring a separate mobile application.
This makes the experience immediately accessible:
Open the web app → upload a dog-walk video → watch the cognitive twin process it.
The browser sends the video to the Cloudflare Worker, which becomes the entry point for the processing pipeline.
Oracle Relay Instead of Direct Snowflake Signing
Snowflake Dynamic Tables cannot directly sign Solana transactions or freely call arbitrary external APIs.
Instead, a lightweight oracle-relay polls Snowflake's V_ORACLE_PAYLOAD, signs the transaction, and submits it to Solana Devnet.
This keeps blockchain operations outside the main Gemini + TTS request path, so a slow blockchain transaction does not block the user-facing processing pipeline.
The Anchor program still verifies the oracle public key on-chain.
Separate Runtime Implementations
The ingestion worker runs inside the Cloudflare Workers runtime, while the oracle relay runs in Node.js with tsx.
Although both components use keypair-based authentication, their runtime crypto APIs and execution environments differ. Keeping them separate avoids introducing an abstraction purely to unify two different runtime environments.
Fallback Instead of Silent Failure
External AI APIs can fail because of rate limits or temporary availability issues.
Instead of silently pretending the response was generated live, the worker can load a pre-recorded Gemini response when the live request fails.
The response explicitly contains:
{
"source": "fallback"
}
This allows the dashboard and pipeline to communicate the state rather than hiding it.
Prize Technology
| Technology | Used For | Why It Matters |
|---|---|---|
| Google AI / Gemini | Multimodal video analysis, schema-constrained JSON, and contextual reasoning | Provides the actual observation and interpretation layer |
| Snowflake | Dynamic Tables, historical baselines, Cortex anomaly detection, and Streamlit | Turns individual observations into longitudinal intelligence |
| ElevenLabs | Voice Design, Flash v2.5 narration, and sound design | Gives the digital twin a consistent personality and voice |
| Solana | Dog identity and immutable memory records | Adds an ownership and proof layer outside a centralised application database |
These technologies are not decorative integrations.
Each one has a specific role in the product:
Gemini → Observe & Understand
Snowflake → Remember & Detect Change
ElevenLabs → Give the Twin a Voice
Solana → Own & Prove
Code
Canine Cognitive Twin
A living digital twin for a dog. An owner uploads a clip of a morning walk from the browser; Gemini reads the video and returns structured observations plus a first-person narration; ElevenLabs speaks it; Snowflake Dynamic Tables fold it into a rolling baseline; Cortex flags gait drift the eye would miss; Solana records the chapter under the owner's own keys.
Not a veterinary device. Not diagnostic. Consult a veterinarian.
New here and just want to get something running? Jump to Quick start.
What is live vs. what is synthetic
This table is the first thing in the repo on purpose. Everything below the line is generated, and it is labeled generated on screen too.
Live — clicked during the demo, nothing faked
| Step | Tech | Latency |
|---|---|---|
| Owner uploads a video of a walk from the browser, straight to the Worker | React + Vite web app | ~5s |
| Video |
canine-cognitive-twin/
├── snowflake/ Dynamic Tables, Cortex anomaly model, Streamlit app
├── gemini-pipeline/ Schema-constrained Gemini analysis
├── elevenlabs-service/ Voice Design, narration, SFX
├── ingestion-worker/ Cloudflare Worker: Supabase → Gemini → Snowflake → TTS
├── oracle-relay/ Snowflake → Solana transaction relay
├── solana-program/ Anchor program for dog identity and memories
├── web-app/ React + Vite browser application
└── demo-assets/ Fallback response and demo assets
What's Next
The current implementation establishes the core pipeline from video → observation → narration → longitudinal analysis → immutable memory.
The next milestones would be:
- Owner-facing export of the memoir and on-chain proof
- Multi-dog households with one Solana wallet and a PDA per dog
- Per-dog voice calibration
- Longer-term behavioural and mobility trends
- More advanced gait and movement visualisations
- Additional behavioural signals beyond gait
- Stronger owner-facing explanations around detected changes
- Continuous longitudinal learning from real owner-approved captures
Prize Categories
I'm submitting this project for:
- Best Use of Snowflake
- Best Use of Solana
- Best Use of ElevenLabs
- Best Use of Google AI
The goal wasn't simply to combine four technologies.
Each technology solves a different part of the same problem:
Observe → Understand → Remember → Give a Voice → Prove
Final Note
I wanted the project to demonstrate something more meaningful than a simple AI video analyser.
A single video can describe what a dog is doing right now.
A cognitive twin should be able to understand what is changing over time.
That's why the architecture combines multimodal AI, longitudinal data processing, anomaly detection, voice, and an ownership layer.
The final experience is intentionally simple:
Upload a walk. Let the twin understand it. Turn it into a memory. Track what changes over time. Keep the record under the owner's control.
Thanks for reading. 🐶
Top comments (0)