We've all been there. Your dog is acting weird. Is she anxious? Bored? Sick? Or just being a dog?
I spent three months building an AI system that tries to answer that question — and along the way I learned some wild things about multimodal models, behavior classification, and why pet data is surprisingly hard to get right.
Here's what I built, what broke, and the actual code.
The Problem
Pets can't talk. Owners miss behavioral signals all the time — not because they don't care, but because subtle changes in posture, vocalization patterns, or eating habits are easy to overlook.
Vets see animals for maybe 20 minutes a year. That gap? It's enormous.
The idea: use a combination of image classification + behavioral pattern analysis to give pet owners an early-warning system.
The Stack
- Python 3.11 — backbone
- OpenAI Vision API — image-based posture analysis
- scikit-learn — behavioral pattern classification
- FastAPI — REST interface
- PostgreSQL + pgvector — storing and querying behavioral embeddings
Step 1: Posture Analysis with Vision Models
The first signal: body language. A dog with a tucked tail, flat ears, and lowered head is communicating something — and modern vision models are surprisingly good at picking this up.
import base64
import httpx
from pathlib import Path
def encode_image(image_path: str) -> str:
with open(image_path, "rb") as f:
return base64.b64encode(f.read()).decode("utf-8")
def analyze_pet_posture(image_path: str, species: str = "dog") -> dict:
"""Analyze pet body language via OpenAI Vision."""
client = httpx.Client()
image_data = encode_image(image_path)
prompt = f"""
You are an expert animal behaviorist. Analyze this {species} image.
Return a JSON object with:
- mood_estimate: one of [relaxed, anxious, playful, alert, stressed, neutral]
- confidence: 0.0 to 1.0
- signals: list of observed behavioral cues (tail position, ear position, etc.)
- recommended_action: brief owner guidance
Be conservative — only flag stress if clear signals exist.
"""
response = client.post(
"https://api.openai.com/v1/chat/completions",
headers={"Authorization": f"Bearer {OPENAI_API_KEY}"},
json={
"model": "gpt-4o",
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": prompt},
{"type": "image_url", "image_url": {
"url": f"data:image/jpeg;base64,{image_data}"
}}
]
}],
"response_format": {"type": "json_object"}
}
)
return response.json()["choices"][0]["message"]["content"]
Step 2: Behavioral Pattern Tracking
Single snapshots are noisy. The real signal is in the pattern over time.
We track daily inputs from owners: eating, sleep, activity, social behavior — then run a lightweight anomaly detector.
import numpy as np
from sklearn.ensemble import IsolationForest
from datetime import datetime, timedelta
from typing import List
class PetBehaviorTracker:
def __init__(self, contamination: float = 0.05):
self.model = IsolationForest(
contamination=contamination,
random_state=42
)
self.is_fitted = False
def encode_daily_log(self, log: dict) -> List[float]:
"""Convert daily behavior log to feature vector."""
return [
log.get("appetite_score", 3) / 5.0, # 0-1
log.get("activity_minutes", 30) / 120.0, # normalized
log.get("sleep_hours", 10) / 16.0, # normalized
log.get("social_score", 3) / 5.0, # 0-1
log.get("vocalization_score", 2) / 5.0, # 0-1
float(log.get("vomiting", False)),
float(log.get("hiding", False)),
]
def fit(self, historical_logs: List[dict]):
"""Train on at least 30 days of baseline behavior."""
if len(historical_logs) < 14:
raise ValueError("Need at least 14 days of data for baseline.")
X = np.array([self.encode_daily_log(log) for log in historical_logs])
self.model.fit(X)
self.is_fitted = True
def score_today(self, today_log: dict) -> dict:
"""Score today's behavior against learned baseline."""
if not self.is_fitted:
raise RuntimeError("Tracker not fitted yet.")
x = np.array([self.encode_daily_log(today_log)])
score = self.model.score_samples(x)[0] # more negative = more anomalous
is_anomaly = self.model.predict(x)[0] == -1
return {
"anomaly_detected": bool(is_anomaly),
"anomaly_score": float(score),
"severity": "high" if score < -0.3 else "medium" if score < -0.1 else "low"
}
Step 3: Combining Signals
The magic is in the fusion layer — combining image posture analysis with behavioral anomaly scores to produce a single wellness alert.
def compute_wellness_alert(posture_result: dict, behavior_result: dict) -> dict:
"""
Fuse posture and behavioral signals into a wellness score.
Returns: alert_level, explanation, recommended_action
"""
posture_stress = posture_result.get("mood_estimate") in ["anxious", "stressed"]
posture_confidence = posture_result.get("confidence", 0)
behavior_anomaly = behavior_result.get("anomaly_detected", False)
severity = behavior_result.get("severity", "low")
# Both signals agree → high confidence alert
if posture_stress and behavior_anomaly:
return {
"alert_level": "high",
"explanation": "Behavioral pattern anomaly combined with stress body language detected.",
"action": "Consider contacting your vet within 24 hours."
}
# One signal → medium alert
if posture_stress and posture_confidence > 0.7:
return {
"alert_level": "medium",
"explanation": "Body language suggests possible stress or discomfort.",
"action": "Monitor closely for the next 24 hours."
}
if behavior_anomaly and severity == "high":
return {
"alert_level": "medium",
"explanation": "Today's routine deviated significantly from baseline.",
"action": "Note any environmental changes. Monitor tonight."
}
return {
"alert_level": "none",
"explanation": "No concerning signals detected.",
"action": "All good — keep it up!"
}
What I Got Wrong (A Lot)
1. Assuming pet owners log consistently. They don't. The model falls apart with gaps. I had to build imputation logic and handle streaks of missing data gracefully.
2. Overconfident vision outputs. GPT-4o is too good at filling in gaps. It'll confidently tell you a fluffy Samoyed looks "relaxed" when it's clearly a stuffed animal. Prompt engineering matters enormously here.
3. Breed variation is huge. A Shar Pei always looks grumpy. A Husky always looks dramatic. Posture baselines need to be breed-adjusted — I had to build a breed profile layer.
4. The feedback loop is slow. Unlike spam detection (instant feedback), pet health outcomes take weeks to validate. Evaluation is hard.
What Actually Works
The anomaly detector is genuinely useful once there's 3+ weeks of baseline. Owners consistently report "I didn't notice but now I see it" moments — subtle drops in appetite or activity that preceded a vet visit.
The posture model is better as a second opinion than a primary signal. High confidence + behavioral anomaly = something worth paying attention to.
Open Questions
- How do you handle multi-pet households (shared behavior patterns)?
- What's the right threshold for alerting vs. false-positive fatigue?
- Can audio (barking/meowing patterns) add a useful third modality?
If you've worked on animal behavior ML or multimodal health monitoring, I'd genuinely love to hear what you've found.
This work is part of a larger project around pet wellness — if you're curious about the live application, MyPetTherapist is the production version of these ideas.
Drop your questions in the comments — especially if you've hit any of these problems in adjacent domains.
Top comments (0)