DEV Community: Pacy

"I Built an Operating System for Dogs (with On-Chain Noseprints, 60 FPS Protanopia Vision, and Acoustic Anxiety AI)"

Pacy — Mon, 17 Aug 2026 08:08:12 +0000

🐶 What If Your Dog Had Their Own Operating System?

Think about how fragmented our pet care ecosystem is today:

Lost Pet Identification is Broken: If your dog slips their collar, you rely on a 15-digit microchip implanted under the skin that requires a specialized RFID wand and a database that isn't connected to municipal shelters in real-time.
Behavioral Training is Clunky: You juggle a plastic mechanical clicker, a phone stopwatch, a treat bag, and a YouTube tutorial, missing the sub-millisecond marking window required for operant conditioning.
Sensory Misunderstanding: Humans buy bright red dog toys because they look vibrant in the pet store, unaware that dogs are dichromats and red literally blends into green grass like camouflage.
Emergency Sitter Handoffs: When leaving town, you handwrite a messy sticky note on the fridge with feeding gram portions, medication times, and emergency vet numbers that get lost.
Toxicology Guesswork: If your dog grabs dark chocolate or a baked treat with xylitol off the counter, panic sets in while you search random forum threads trying to calculate toxic milligrams per kilogram.

Over the weekend, we built KRYPTO: a full-spectrum, multi-modal Canine Operating System designed to bridge veterinary science, sensory neuroscience, biometric blockchain identity, positive-reinforcement training, and social pack connection into a single web application.

Here is the deep dive into what we built, the science behind it, and why this represents a modern paradigm shift for pet parenting.

🏛️ Master Architecture & System Topology

graph TD
    subgraph "CLIENT INGESTION & SENSORY LAYER"
        A[Camera & Mic Feed] --> B[Biometric Scanner Lab]
        A --> C[CanineSense Vision 60 FPS WebGL Shader]
        A --> D[Meyda Acoustic Feature Extractor]
        E[Handler Inputs] --> F[Woof Woof Pack Lounge]
        E --> G[AKC Training Academy]
        E --> H[Krypto AI Assistant Copilot]
    end

    subgraph "AI & MULTI-MODAL INTELLIGENCE"
        B -->|Rhinarium Ingestion| I[Google GenAI: Gemini 3.6 Flash]
        B -->|Rx Pill Bottle OCR| I
        B -->|Morphology & Genetic Risk| I
        H -->|Streaming Canine Knowledge| I
        G -->|Clicker Acoustic Sync| J[Web Audio API 3.2kHz Marker]
        D -->|Spike Detection >2.5kHz| K[432Hz Bio-Acoustic Calming Engine]
    end

    subgraph "DECENTRALIZED & CLOUD INFRASTRUCTURE"
        I -->|Biometric Hash & Memo| L[Solana Devnet: Immutable Passports]
        I -->|1-of-1 Art Keepsake Synthesis| M[Keepsake Studio 4K Engine]
        D -->|Acoustic Events Stream| N[Snowflake Data Cloud: REST SQL API]
        F -->|Pack Threads & Posts| O[Firebase Cloud Firestore / RTDB]
        O -.->|Offline Sync Fallback| P[Client IndexedDB: idb-keyval]
    end

    subgraph "OUTPUT ARTIFACTS"
        L --> Q[Verifiable Solana Explorer Record]
        M --> R[Downloadable 1024x1024 Pure Art & Passports]
        O --> S[Print-Ready 8.5x11 Sitter Dossier PDF]
        M -->|Default Verified PFP| F
    end

🔬 The 12 Core Innovations in KRYPTO

1. 👃 Anatomical Rhinarium Biometrics: The Snout Fingerprint

Canine nose prints (rhinaria) are dermatoglyphically unique. The polygonal micro-ridges, dual alar nostril lobes, and central philtrum groove form a lifelong invariant physical identifier.

How It Works: Our optical HUD captures high-contrast snout topography, computes localized Shannon entropy ($E = -\sum p \log_2 p$), and derives a 64-bit cryptographic hash (e.g., f4e2-a9b1-08dc-37aa).
Multi-Modal Verification: Ingested via the Google GenAI Interactions API (gemini-3.6-flash) with structured landmark point extraction.
On-Chain Solana Sealing: The verified hash is minted directly onto Solana Devnet as an unforgeable, decentralized pet passport with live Solana Explorer transaction verification.

2. 👁️ CanineSense: 60 FPS Protanopia Vision Shader & Salience Checker

Dogs lack red-green retinal cone photoreceptors. They perceive the world through a dual-wavelength spectral blend of shortwave S-cones ($429\text{ nm}$, Blue) and longwave M/L-cones ($555\text{ nm}$, Yellow).

We implemented the biological Brettel-Viénot-Mollon Protanopia Projection directly in a 60 FPS WebGL/Canvas shader:

$$\begin{bmatrix} L' \ M' \ S' \end{bmatrix} = \begin{bmatrix} 0.0 & 1.05118 & -0.05116 \ 0.0 & 1.0 & 0.0 \ 0.0 & 0.0 & 1.0 \end{bmatrix} \begin{bmatrix} L \ M \ S \end{bmatrix}$$

Visual Salience Checker: Features a real-time side-by-side CIE Lab $\Delta E$ contrast comparator that reveals why a red ball ($L^=53, a^=80, b^*=67$) completely disappears against grass, while blue and yellow objects pop with vibrant contrast.

3. 🎙️ Acoustic Separation Anxiety Monitor & 432Hz Autonomous Calming

Using the Web Audio API and Meyda audio feature extraction, KRYPTO processes ambient audio through 512-sample FFT windows:

Territorial Barks ($500\text{--}1800\text{ Hz}$): Normal alert activity.
Distress & Separation Anxiety Whines ($>2500\text{ Hz}$): High-pitch acoustic spikes.
The Math: $$\text{Distress Score} = w_1 \cdot \min\left(100, \frac{f_c - 1000}{30}\right) + w_2 \cdot (A_{\text{RMS}} \times 100)$$
Autonomous Calming Engine: When the anxiety score crosses $70\%$, KRYPTO automatically cues a 432Hz bio-acoustic pink noise stream to decompress and soothe your pup.
Snowflake Ingestion: All acoustic distress events stream in real-time into the Snowflake Data Cloud via SQL REST API v2 (KRYPTO_DB.CANINE_TELEMETRY.SENSORY_EVENTS).

4. 🎓 AKC CanineTraining Studio & Arousal Stopwatch

Built on positive reinforcement (LIMA) and operant conditioning principles:

6-Skill AKC CGC Curriculum: Step-by-step masteries for Sit, Down, Stay, Emergency Recall, Paw / High-Five, and Loose Leash / Heel.
Sub-Millisecond Acoustic Marker: Injects a pure $3.2\text{ kHz}$ acoustic clicker tone at precise reward moments.
Variable Ultrasonic Whistle: Generates high-frequency tones from $12\text{ kHz}$ to $22\text{ kHz}$ for long-distance recall.
Yerkes-Dodson Arousal Curve: Tracks dog arousal states (Bored $\rightarrow$ Optimal $\rightarrow$ Overstimulated/Frustrated) with repetition counters ($1/5$ reps) and confetti rewards.

5. 🎨 1-of-1 NFT Keepsake Studio (4 Fine-Art Aesthetics & Dual Downloads)

Every companion deserves museum-quality fine art:

4 Coherent Masterwork Styles: 3D Pixar Animation, Cyber-Tactical Sci-Fi HUD, Royal Victorian Heritage Oil, and Neo-Tokyo Anime Hero.
Phenotypic Likeness Extraction: Gemini analyzes custom uploaded dogs to synthesize tailored 1024x1024 portraits matching exact breed markings, ear structure, and coat patterns.
Dual One-Click Downloads:
1. 🖼️ Full Passport Dossier (800x800 PNG): Bordered collector card with Solana seal and anatomical nose stamp.
2. 🎨 Pure Fine-Art Portrait (1024x1024 PNG): Clean, unbordered masterwork for phone wallpapers and canvas prints.

6. 🐾 "Woof Woof" Community Pack Lounge (Keepsake PFP Utility)

A 2026 decentralized pack commons:

Keepsake Verified Avatar: The dog's generated 1-of-1 NFT Keepsake automatically becomes their verified community avatar with glowing PFP badges.
Multi-Channel Feeds: Training & Agility, Health & Nutrition, Show & Brag, Ask Vet / Behaviorist, and Lost Dog Radar.
Interactive Reactions: 🐾 Give Paws (+1 spark upvote) and 🍖 Give Treats.
Dual-Mode Cloud Sync: Synchronizes with Google Firebase Cloud Firestore & Realtime Database with local IndexedDB offline fallback.

7. 🤖 "Krypto AI": Omniscient Canine Veterinary & OS Copilot

A floating assistant orb powered by the modern Google GenAI Interactions API (@google/genai with dynamic models gemini-3.6-flash, gemini-3.5-flash-lite, gemini-3.1-pro-preview):

Context-Aware: Ingests real-time active dog data (name, breed, weight, medical conditions, active prescriptions, emergency contacts).
Instant Expert Guidance:
- Precision toxicology calculations ($mg/kg$ hazard formulas).
- AKC positive-reinforcement cue progression.
- Instant app navigation assistance.

8. 💊 Multi-Modal Prescription Label OCR & Contraindication Radar

Ingests veterinary medication pill bottles via camera.
Extracts drug name, dosage ($mg$), administration frequency, and instructions.
Automatically cross-references contraindications with the dog's breed and age profile.

9. ⚠️ Deterministic $mg/kg$ Clinical Toxicology Calculator

Calculates exact chemical hazards based on patient body weight ($34.5\text{ kg}$):

Theobromine (Chocolate): Analyzes White, Milk, Dark 70%, Dark 85%, and Baker's cocoa.
Xylitol / Birch Bark Sweetener: Evaluates lethal hypoglycemic risk ($>0.1\text{ g/kg}$).
Grapes / Raisins, NSAIDs / Ibuprofen, Acetaminophen, and Garlic.
Displays color-coded risk gauges (SAFE, MILD, MODERATE, SEVERE, LETHAL) with direct 24/7 ER vet clinic speed-dial.

10. 📄 Emergency Sitter Passkey (`/pass/[id]`) & Printable 8.5x11" PDF Dossier

Web Passkey Route: Standalone PWA route accessible without login. Sitters can access feeding times, allergies, and vet contacts on their phones.
Print-Ready 8.5x11" Sitter Sheet: Generates a physical PDF with print media queries containing QR passkey tokens, daily feeding portions ($340\text{g}$), and trauma center addresses.

11. 🚨 Solana Lost Dog Emergency Beacon

One-click decentralized emergency broadcast that toggles on-chain status (isLostBeaconActive: true) on Solana Devnet.
Generates an emergency printable flyer with microchip ID, owner contact, and reward.

12. 🎙️ ElevenLabs First-Person Dog Voice Persona Narrator

Generates first-person spoken audio persona narration using eleven_turbo_v2_5.
4 vocal profiles (Confident Hero, Playful Energetic, Gentle Soul, Wise Veteran).
Seamlessly falls back to the browser's native Web Speech API if offline or without an API key.

🛠️ Complete Tech Stack Matrix

Category	Technology	Purpose
Framework	Next.js 15.2.0 (App Router, React 19)	High-performance SSR, Edge APIs, PWA routes
AI / Multi-Modal	Google GenAI Interactions API (`@google/genai`)	Gemini 3.6 Flash, 3.5 Flash-Lite, 3.1 Pro
Blockchain	Solana Devnet (`@solana/web3.js`)	Immutable on-chain pet passports & Lost Dog Beacon
Data Cloud	Snowflake SQL REST API v2	Acoustic anxiety & sensor telemetry warehouse
Cloud Database	Google Firebase (Firestore & RTDB)	Real-time Woof Woof community feeds
Audio Processing	Meyda + Web Audio API	Real-time spectral centroid & 3.2kHz acoustic clicker
Voice Synthesis	ElevenLabs (`eleven_turbo_v2_5`)	First-person dog persona voice narration
Offline Cache	IndexedDB (`idb-keyval`)	Offline-first data ledger
Styling	Tailwind CSS & Framer Motion	Dark-glass Cyber-Bio HUD & micro-interactions

🛡️ Offline-First & Real-World Resilience

Dog emergencies often happen in remote areas with zero cell reception:

No Internet in the Park? All passkey data, veterinary contacts, and the training stopwatch operate 100% offline via local IndexedDB.
AI Quota Reached? The system automatically switches to deterministic clinical algorithms and native browser speech synthesis with zero audible or visual degradation.
Permission Denials? Missing camera or mic permissions immediately offer synthetic audio streams and drag-and-drop file uploads.

🚀 Try It Yourself!

We built KRYPTO to bring real science, joy, and peace of mind to dog handlers everywhere.

🌐 Live Demo: https://krypto-canine-os.vercel.app/
💻 GitHub Repository: https://github.com/pacylabs/krypto
📄 Full Technical Whitepaper: Check out CANINE_OS_SPECIFICATION.md in the repo for the complete math formulas and clinical toxicity tables.

Drop a comment below with your dog's breed and your favorite feature! 🐾❤️

Introducing TCG, The Cruise god - the world's first multiplayer voice AI concierge, game master, and local guide.

Pacy — Fri, 27 Mar 2026 13:10:33 +0000

We Built a Multiplayer Voice AI That Actually Runs the Party
Live demo: thecruisegod.vercel.app

Six friends. One city. Zero agreement on what to do next.

Here's the thing nobody talks about: every AI voice assistant ever built assumes you're alone. You whisper into it. It whispers back. Clean, personal, useless for a group.

Real life is messier. Three problems hit at once:

Where are we going? Someone's Googling. Someone's scrolling TikTok for vibes. A third texted "idk, you pick." Nobody picks.
Who do we call? The group needs a DJ for Saturday or a barber open right now. Someone texts their cousin. The cousin doesn't respond. The moment dies.
What are we doing right now? You're already together but the energy's flat. The night either catches fire or it doesn't — and it usually doesn't.
No AI handles any of this. And none of them can do it for a whole room at once.

So we built TCG — The Cruise God.

What Is TCG?
TCG is the world's first multiplayer voice AI concierge, game master, and local guide. It's a live, conversational AI that physically drives its own React UI while it talks — serving an entire group simultaneously.

You tap the character. TCG wakes up. You tell it where you are and what the vibe is. It runs the night — out loud, with personality, without anyone touching a single button.

Three modes. Each purpose-built for a real group scenario.

🟢 Locator Mode
"Find us somewhere to go."

Tell TCG the vibe — rooftop bar, late-night spot, live music — and it:

Dynamically builds a contextual search query from your location and energy
Live-scrapes the web via Firecrawl
Reads back real venue recommendations while the UI snaps into a card layout
It's not a Google search. It knows your city, your crew size, and what kind of night you're trying to have.

🔌 Plug Mode
"Find me someone who can handle this."

Need a DJ for next weekend? A same-day mechanic? A barber open right now?

Plug Mode rewrites queries with urgency context — "available now open today same-day" — runs a live search, and voices the results. It's the "who do you know?" answer, surfaced by AI.

🎮 Game Master Mode
"Let's play something."

This is where TCG earns its name.

Say "teach us a drinking game for 6 people, chaotic energy" and TCG doesn't just name a game. It scrapes the full rule set in Markdown via Firecrawl and reads the rules aloud, step by step, to the room.

The UI opens a live Game Session dashboard — scoreboard, current turn, player list, rules summary — that TCG updates as the game progresses.

Plus 9 built-in party tools available on command:

Tool Details
Spin the Bottle Synced with the room's live guest list
Truth or Dare 3 intensity levels
Charades 4 categories
Coin Flip / Dice Roll Voice-triggered
Randomizer Splits guests into named groups
Timer Countdown, voice-controlled
Scoreboard Live tracked
Bill Splitter Camera → Gemini Vision → split result read aloud

The Tech Stack

ElevenLabs — The Heartbeat
We didn't use ElevenLabs just for TTS. We architected 10 custom Client Tools that let the agent autonomously control the React frontend in real-time:

switchMode → Snaps the UI between Locator / Plug / Game Master / Tools
openTool → Opens any party tool on command
showQR → Displays the CruiseHQ join QR code
randomizeGroups → Splits guests into named groups
setGroupLeader → Elects and announces a group captain
updateGameState → Syncs the live scoreboard
displayResults → Populates venue/service result cards
analyzeImage → Opens camera → Gemini Vision
createMemory → Screenshot → Supabase Storage → Trophy + viral caption
stopListening → Cuts the mic at hardware level
TCG says "let me pull that up" — and the UI snaps. The conversation never breaks.

Firecrawl — The Search Pipeline
A production-grade 3-tier pipeline:

Supabase (7-day cache) → Upstash Redis (15-min hot cache) → Firecrawl live scrape
Queries are dynamically rewritten for context. "Chaotic game for 6" becomes:

"for 6 players wild hilarious high-energy party game rules how to play"

Game searches extract full Markdown rule sets. TCG teaches the room, rule by rule, out loud.

Supabase Realtime — The Multiplayer Backbone
Friends scan a QR code from their phones and land in a full CruiseHQ interface. They're not just watching — they're in the room.

Supabase Realtime handles:

Presence tracking — live roster of who's in the room
Broadcast channels — dares, group chats, poll votes, co-host voice transcripts
Sub-second latency across every connected device
When a guest sends TCG a dare from their phone, it's injected as a live user message into the active ElevenLabs voice session. TCG hears it. TCG acts on it. In front of the whole room.

Gemini 2.5 Flash Vision — Eyes for the Room
Vision isn't just about reading text—it's TCG's way of physically "seeing" the party. It integrates directly with the live ElevenLabs voice session as natural context, so TCG actually sees the room and talks about it.

Act as a Referee: Show TCG a chaotic game board, a Charades drawing, or a physical challenge, and its Game Vision acts as an impartial, live judge.
Split the Check: Point the camera at a receipt → TCG reads the total, splits it per person, auto-detects dynamically the currency from your GPS location, and reads the result aloud to the group.
Verify Items: Scan a barcode → instantly validated against the Open Food Facts API to tell you exactly what you're holding.

What Makes It Actually Different
We didn't add "multiplayer" as a feature. We rebuilt the entire architecture around the assumption that the AI is serving a room, not a person.

CruiseHQ submissions: A word submitted from a guest's phone can be silently inserted into the agent's context — invisible to the host's transcript, visible only to TCG. The AI knows the word. The host doesn't. The game works.IT can also be public, leaving the power in your hands. This lets you post dares, suggestions- anything that requires a submission, CruiseHQ is equipped to process.

The Groups trick: When the host says "randomize the groups," the randomizeGroups Client Tool runs, CruiseHQ auto-creates group tabs, every guest's interface updates, and TCG announces the result out loud with personality — all in one voice command.

Wingman Protocols: Users can set secret instructions ("always suggest dive bars", "roast me constantly") that are injected into the ElevenLabs system prompt on every session start. TCG already knows you before you say a word.

Trophy Room: When a moment lands, TCG proactively captures it — screenshot via html2canvas, uploaded to Supabase Storage, saved as a Trophy with a generated viral caption, shared from /trophy-room.

Try It
👉 thecruisegod.vercel.app

Open it on one device. Share the QR code with your friends. Tell TCG where you are.

Let it run the night.

From Clinic to Code: Engineering GozAI, a Clinically- grounded AI Copilot for Low Vision & Visually impaired

Pacy — Mon, 16 Mar 2026 14:33:49 +0000

For the 2.2 billion people experiencing vision impairment globally, standard assistive technology often feels like a burden rather than a bridge.
The seed for my latest project, GozAI, was planted back in 2022 during my 500-level Optometry externship at the University of Ilorin Teaching Hospital (UITH) in Nigeria. Working under the guidance of a low-vision and pediatric specialist, I spent my days with countless patients navigating life with severe visual impairments.
Day after day, I witnessed the same frustrating reality: traditional low-vision aids are incredibly fragmented and prohibitively expensive. Patients were being asked to carry and pay a premium for clunky, single-purpose digital magnifiers.
Even then, it raised a glaring question: We are in the digital age. Why hasn't this been solved using the devices already in our pockets? At the time, I had the clinical domain expertise to know exactly what the solution should look like, but I lacked the software engineering skills to actually build it. The idea stayed on the shelf.

The Catalyst: Agentic Coding and Clinical Reality

Fast forward to 2026. Recent clinical research has completely validated what I saw in the clinic years ago, showing a staggering abandonment rate of up to 50% for specialized assistive gadgets. The primary culprits? Cost, severe battery drain, and the social stigma of carrying them. People don’t want another heavy device to manage. They just want the smartphone they already own to act as their eyes.
At the same time, we entered the era of agentic coding. With recent leaps in AI development tools, the technical barriers to entry that once held me back vanished. By building entirely with Google's Antigravity, I was finally able to take the clinical blueprints from my externship experience and code them into reality.

Building GozAI: A Multimodal Accessibility Copilot

When I entered the Gemini Live Agent Challenge, I set out with an uncompromising goal: build an accessibility copilot that is battery-optimized, emotionally reassuring, and rooted entirely in clinical reality.
GozAI is a real-time voice and vision assistant powered by the Gemini 2.0 Flash Multimodal Live API and hosted on Google Cloud Run. Instead of just building a "talking camera" that overwhelms the user, I engineered it to act as an intelligent, empathetic filter.

Bridging the Gap: Clinical Need & Technical Execution

Verified Clinical Grounding: To ensure GozAI wasn't just another tech novelty, I didn't want it just guessing. I engineered the backend RAG architecture (built using the Google ADK) to strictly rely on real, verified optometry data and textbooks. It operates under strict clinical guardrails, ensuring the advice and spatial orientation it provides are safe and accurate.
Semantic Vibro-Acoustics: Continuous audio feedback causes massive cognitive fatigue for low-vision users. GozAI captures frames at a battery-saving 1 FPS and uses synchronized haptics for navigation. It only delivers audio warnings for immediate hazards, keeping the user's mental bandwidth clear.
The Glaucoma Protocol: Progressive vision loss carries a high risk of depression and anxiety. GozAI’s persona is strictly constrained to be an anchor of psychological safety—calm, steady, and reassuring. It doesn't just see for the user; it acts as a supportive companion.
The UI Navigator: For digital environments, GozAI acts as a bridge where standard screen readers fail. Using Gemini’s multimodal capabilities, it can interpret unlabelled buttons on a screen, read medical labels offline (via Google ML Kit), and even inject synthetic screen taps for the user. ## The Technical Backbone To make this vision a reality, we leveraged a robust Google-native stack:

Flutter PWA / Mobile
└── Gemini Multimodal Live API (Bidirectional WebSocket)
└── Google ADK Agent (Cloud Run)
├── Tool: Optometry Guidelines (Verified RAG)
├── Tool: Medication Lookup (OpenFDA)
├── Tool: SOS → Firestore
└── Tool: Clinical Stats
└── Firebase / Firestore (Companion Memory)
The mobile client connects directly to the Gemini Multimodal Live API via a bidirectional WebSocket, streaming real-time audio and camera frames simultaneously — ensuring near-zero latency for those critical safety moments.

Closing the Gap
GozAI proves that when you combine deep clinical empathy with bleeding-edge tools like the Gemini Live API and Google's Antigravity, you can build solutions that actually restore independence, rather than just adding another gadget to a patient's bag.

Everyone deserves to navigate the world with confidence.

This piece was created for the purposes of entering the Gemini Live Agent Challenge. 🚀

Check out the code: https://github.com/Kryptopacy/GozAI
🌐 Try the Demo: gozai-app.web.app
🚀 Built for: Gemini Live Agent Challenge | Tracks: Live Agents + UI Navigator