Introduction
"Claude can read code, analyze images — but couldn't watch a video. Until this."
This is article #154 in the "One Open Source Project a Day" series. Today's project is claude-video — a Claude skill plugin by Brad Bonanno (Solaris Automation) built around the /watch command, giving Claude the ability to actually watch videos and answer questions about their content.
/watch https://youtube.com/watch?v=xxx What's the core argument in this talk?
/watch bug-repro.mov What error is happening in this recording?
/watch product-demo.mp4 Extract all features into a list
15,200 Stars. MIT license.
What You'll Learn
- The seven-step video analysis pipeline and its technical design
- Subtitle-first strategy: why this design decision cuts costs dramatically
- The MAD dedup algorithm: how 16×16 thumbnails catch slow fades
- Automatic token budget management: dynamic frame scaling by video duration
- Four detail modes and available parameters
- Installation on Claude Code and other agent hosts
Prerequisites
- Experience with Claude Code or a similar AI coding agent
- Basic video concepts (frames, transcription)
- No Python or ffmpeg knowledge needed
Background: The Problem
Claude supports image input but not video natively. Video files are too large to inject directly into context, and even if they weren't, a pure frame sequence without transcript text loses most of the information.
Existing "make AI watch video" approaches typically do: full download → fixed-fps uniform sampling → all frames to the model. Three problems:
- Wasted tokens: static scenes produce many near-identical frames, each consuming tokens
- Missing audio: no transcription means speech, narration, and dialogue are lost
- Unpredictable costs: a 50-minute video with fixed-fps sampling might produce hundreds of images
claude-video optimizes at every stage.
Seven-Step Pipeline
Video URL / local path
↓ [1] yt-dlp attempts subtitle retrieval
↓ [2] Subtitles found? Use them, skip download
↓ [3] No subtitles? Download video, extract frames via ffmpeg
↓ [4] MAD dedup algorithm, scene-change aware
↓ [5] Whisper transcription (Groq preferred)
↓ [6] Frames + transcript sent to Claude context
↓ [7] Claude answers, temp files cleaned up
Key Optimization 1: Subtitle-First, Zero Download
Many video platforms (YouTube especially) have native or CC subtitles. yt-dlp checks for subtitles before downloading the video.
If subtitles are found:
- No video download, no frame extraction, no Whisper call
- Parse the VTT subtitle file, use it directly as the transcript
- Total time drops from ~37 seconds (downloading a 76MB video) to ~4.5 seconds
A 49-minute YouTube lecture in transcript mode: 4.5 seconds, near-zero token cost.
Key Optimization 2: Scene-Aware Frame Extraction
Instead of uniform fps sampling, the tool detects scene changes — frames are only captured when the visual content actually changes.
How it works:
- ffmpeg outputs a raw frame sequence
- Each frame shrinks to a 16×16 grayscale thumbnail (standard library only, no Pillow needed)
- Compute the Mean Absolute Difference (MAD, 0–255) against the last kept frame
- MAD ≤ 2.0: near-duplicate, discard
- Frame budget cap applies after deduplication
One design detail worth noting: comparing against the last kept frame, not the previous frame.
This handles slow fades:
Frame A → Frame B → Frame C → Frame D (full scene cut)
A vs B: MAD = 0.5 (similar, B discarded)
A vs C: MAD = 1.0 (still similar, C discarded)
A vs D: MAD = 8.0 (significant change, D kept)
If you compared against the previous frame, each step's delta would be small even through a slow fade — only the final frame would cross the threshold. Comparing against the last kept frame accumulates the difference over time and catches the transition correctly.
Automatic Token Budget Management
Frame count isn't fixed — it scales with video duration:
| Video Duration | Default Frame Budget |
|---|---|
| ≤ 30 seconds | ~30 frames |
| 30s – 1 minute | ~40 frames |
| 1 – 3 minutes | ~60 frames |
| 3 – 10 minutes | ~80 frames |
| > 10 minutes | 100 frames (cap mode) |
Token estimate per frame: (width × height) / 750. At default 512px width, each frame is roughly 197 tokens. 100 frames ≈ ~20,000 visual tokens — manageable when combined with the transcript.
Videos over 10 minutes trigger a "sparse scan" warning, suggesting the user use --start/--end to focus on the section they care about.
Four Detail Modes
Controlled via the --detail parameter:
| Mode | Frame Cap | Speed | Best For |
|---|---|---|---|
transcript |
0 frames | ~4.5s | Talks, podcasts, audio-only content |
efficient |
50 frames | ~0.5s | Quick preview, rough overview |
balanced (default) |
100 frames | ~21s | Everyday use |
token-burner |
No limit | ~21s | Full coverage, cost not a concern |
# Summarize a lecture (subtitles only, no frames)
/watch talk.mp4 --detail transcript Summarize the key arguments
# Quick scan of a demo
/watch demo.mp4 --detail efficient What features are shown?
# Analyze a bug recording with full frame coverage
/watch bug.mov --detail token-burner What error occurred and when?
Focus Mode
For long videos, analyze a specific time range — higher frame density, lower cost:
# Focus on 5:30 to 8:00 only
/watch lecture.mp4 --start 5:30 --end 8:00 What's covered in this segment?
# Capture specific timestamps
/watch product.mp4 --timestamps 1:20,3:45,6:10 What's shown at each moment?
# Higher resolution (needed for reading on-screen text)
/watch screen-recording.mov --resolution 1024 Read the code on screen
Supported Video Sources
Via yt-dlp: 50+ platforms including YouTube, Loom, TikTok, X (Twitter), Instagram, Bilibili, Vimeo, and more.
Local files: .mp4, .mov, .mkv, .webm.
Installation
Claude Code (simplest):
/plugin marketplace add bradautomates/claude-video
/plugin install watch@claude-video
Codex / Cursor / GitHub Copilot / Gemini CLI / others:
npx skills add bradautomates/claude-video -g
claude.ai web:
Download the watch.skill file, upload it in Settings → Features → Skills.
First run auto-detects and installs system dependencies:
- macOS: auto-installs yt-dlp and ffmpeg via
brew - Linux: prints the
apt/dnfcommands to run - Windows: prints
wingetcommands
API Key Requirements
| Capability | Requirement | Cost |
|---|---|---|
| Download + native subtitles | No API key | Free |
| Whisper transcription (preferred) | Groq API Key | Very cheap, very fast |
| Whisper transcription (fallback) | OpenAI API Key | Standard pricing |
| Disable transcription |
--no-whisper flag |
Free, frames only |
Without a Groq or OpenAI key, --no-whisper still lets you analyze visual content — you just won't get audio transcription.
Common Use Cases
Competitor analysis:
/watch competitor-demo.mp4 List all features and UI design patterns
Bug diagnosis:
/watch bug-repro.mov Describe the full sequence of events and what the UI shows
Video notes:
/watch training-video.mp4 --detail balanced Extract key points with timestamps
Cut through marketing fluff:
/watch product-ad.mp4 --detail transcript Strip the marketing language, what does it actually do?
Project Links
- GitHub: bradautomates/claude-video
- Author: Brad Bonanno, @bradbonanno
- Company: Solaris Automation
Summary
claude-video's engineering reflects a clear judgment: find the lowest-cost path at each stage rather than brute-forcing the problem.
Subtitle-first reduces the most common case (YouTube videos with subtitles) to near-zero cost. MAD frame deduplication eliminates redundant tokens from static scenes. Dynamic frame budget scaling gives short videos dense coverage without blowing the budget on long ones. These three optimizations stack to make actual token consumption an order of magnitude lower than fixed-fps sampling.
15,200 Stars tells you "let AI watch video" is a real need, not a demo project. The 50+ agent host support shows the author cared about ecosystem compatibility rather than just Claude Code users.
Explore PrimeSkills — A marketplace for handpicked AI Agents and skills. Each is validated in real enterprise workflows, stripping away hype and keeping only what truly works.
Welcome to my Homepage for more useful insights and interesting products.
Top comments (0)