Video to shorts, done by your coding agent: the Video to Shorts Skill is a free, MIT-licensed agent skill for Claude Code, Codex, Cursor and OpenClaw that takes a long recording, finds the moments worth clipping, renders them as 1080x1920 vertical clips with word-by-word captions, and posts them to TikTok, Instagram Reels, YouTube Shorts and Facebook Reels once you say yes. The heavy lifting is three Python scripts, ffmpeg and Pillow, all running on your machine.
Repo: github.com/social-media-skills/video-to-shorts-skill
Frames from the repo's worked example, cut from a public-domain NASA interview (jsc2020m000132). Footage courtesy of NASA; not endorsed by NASA.
Why a skill and not a single script
Cutting a clip is the easy part. The judgment calls are where clips fail: which 40 seconds out of an hour, where the first word lands, whether a stranger can follow it, whether the crop still has the speaker in it. Those calls need something that can read a transcript and look at a frame, which is what an agent already does. So the skill splits the work: deterministic scripts for transcription and rendering, and a SKILL.md that tells the agent how to choose, check and confirm.
The pipeline
video.mp4
-> check_setup.py what this machine can do, with install commands
-> transcribe.py work/transcript.txt + work/transcript.json (word timings)
-> agent reads transcript, scores candidates with the rubric, you pick
-> clips.json start, end, hook, title, caption, framing per clip
-> make_shorts.py --preview three frames per clip with the planned crop
-> make_shorts.py shorts/clip-01.mp4 ..., contact-sheet.jpg, manifest.json
-> WoopSocial MCP upload, validate, confirm, post, read results
Install
npx skills add social-media-skills/video-to-shorts-skill -g -a claude-code -y
Other routes: openclaw skills install @social-media-skills/video-to-shorts-skill (ClawHub), gh skill install social-media-skills/video-to-shorts-skill video-to-shorts-skill (GitHub CLI), or gemini extensions install https://github.com/social-media-skills/video-to-shorts-skill (Gemini CLI). For Codex or Cursor, run npx skills add social-media-skills/video-to-shorts-skill and pick the agent.
Dependencies are deliberately small: Python 3.8+, ffmpeg and Pillow.
| OS | ffmpeg |
|---|---|
| macOS | brew install ffmpeg |
| Windows | winget install ffmpeg |
| Debian, Ubuntu | sudo apt install ffmpeg |
Then pip install pillow. The skill needs an agent that can run ffmpeg locally; it won't render real video inside a browser chat sandbox.
Step 1: check_setup.py
python3 scripts/check_setup.py
It prints ok, missing or optional for ffmpeg, Pillow, each transcription route and OpenCV, plus the install command for the OS it's running on. The agent runs it first so it never tries to render on a machine that can't.
Step 2: transcribe.py and the four routes
python3 scripts/transcribe.py VIDEO --out work/ # best route available
python3 scripts/transcribe.py VIDEO --engine local --model small
python3 scripts/transcribe.py VIDEO --from captions.srt # existing SRT or VTT
| Route | Trigger | Trade-off |
|---|---|---|
| Existing transcript | --from file.srt |
Free and instant; word timing estimated from segments |
| Groq | GROQ_API_KEY |
About $0.04 per audio hour, a minute or two per hour; audio leaves your machine |
| OpenAI | OPENAI_API_KEY |
Uses whisper-1, which OpenAI plans to retire in February 2027 |
| Local | pip install faster-whisper |
Free and private; roughly 0.5x to 1x the video's length on a laptop with small
|
Auto mode tries them in that order. Outside ffmpeg, the script uses only the standard library.
The output is built for an agent's context window. transcript.txt has one timestamped line per sentence, compact enough to read an hour in one go. transcript.json holds every word with start and end in seconds, which make_shorts.py uses for captions and for snapping cuts to word edges.
[03:33.0-03:46.6] I think another piece of it is kind of coming up with a concept of how for a highly automated vehicle, you don't lose sight of kind of the need for the crew to always have a safe path home.
Step 3: scoring candidates with the clip rubric
The agent reads the whole transcript and lists 8 to 12 candidates. Each passes four hard gates or is dropped: the first sentence needs no context, it holds one complete idea, it contains no intro, outro, sponsor read or crosstalk, and it has no music or footage the user doesn't own. Survivors are scored out of 100:
| Criterion | Points |
|---|---|
| Hook in the first 3 seconds | 25 |
| Stands alone | 20 |
| Payoff or turn | 20 |
| Emotion and energy | 15 |
| Specifics | 10 |
| Ending | 10 |
70+ ships, 55 to 69 is a maybe, the rest goes. On the NASA example, a moment about training for zero surprises scored 88, "save the crew, not the mission" 82, a line about future crews cursing the designers' names 76, and an abstract stretch opening with "And so" scored 48 and was dropped. The agent shows the table and the user picks.
Default length is 25 to 45 seconds, 60 at most, 90 only for a story scoring 85+. Clips carrying any third-party audio or footage stay at 60 seconds or less, since YouTube blocks longer Shorts with a Content ID claim.
Step 4: clips.json
{
"video": "talk.mp4",
"style": {"accent": "#FFD43B", "words_per_line": 3, "uppercase": false, "hook": true},
"clips": [
{"id": "01", "start": 754.2, "end": 801.9,
"hook": "Text shown on screen for the first seconds",
"title": "Title for YouTube Shorts", "caption": "Caption for TikTok and Reels",
"reframe": "auto", "focus_x": 0.5, "tighten": true}
]
}
reframe takes auto (face crop when OpenCV finds one speaker, otherwise fit), face, center or fit. focus_x from 0 to 1 places the crop by hand. tighten removes pauses over 0.45 seconds.
Step 5: preview, then render
python3 scripts/make_shorts.py clips.json --transcript work/transcript.json --out shorts/ --preview
python3 scripts/make_shorts.py clips.json --transcript work/transcript.json --out shorts/
The preview writes a preview-XX.jpg per clip with three frames and the planned crop box. The agent opens each one; if the box misses the speaker it sets focus_x, and for two people far apart or a screen share it switches to fit, which scales the whole frame onto a blurred, darkened copy of itself.
The render produces 1080x1920 at the source frame rate, H.264 High with AAC 48 kHz, moov at the front. Audio gets an 80 Hz high-pass, loudness normalisation to -14 LUFS with a -1 dB true-peak ceiling, and 10 ms fades at every cut so removed pauses don't click. Pauses the transcriber hides inside a stretched word are removed too, along with stutters, "um", "uh" and "you know,", and every second segment after a cut punches in slightly so jump cuts look intended. Rendering takes about as long as the clips themselves; --best is slower and slightly smaller, and --only 02 re-renders one clip.
Why captions are drawn with Pillow
Burning text with ffmpeg usually means the subtitles or drawtext filter, and not every ffmpeg package ships with the libraries those need. Drawing the caption frames with Pillow and overlaying them keeps the requirement at "any ffmpeg build", which is what makes a one-line install hold on Mac, Windows and Linux. A GitHub Actions matrix renders a crop clip and a fit clip on all three and asserts the output is 1080x1920.
The captions themselves: Poppins ExtraBold (bundled, SIL Open Font License) at a fixed 96 px, white with an 8 px black outline, up to three words at a time on one line, the spoken word in the accent colour; groups break at punctuation and pauses and never end on a small word like "the". Everything is centred on the frame and kept inside x 180 to 900 and y 290 to 1250, the area that TikTok, Reels and Shorts leave free of buttons, captions and tabs. The hook shows for the first 3.5 seconds, black on a white rounded box, and returns for 2 seconds halfway through clips over 40 seconds.
Step 6: the review loop
After rendering, the agent watches the first 5 seconds of every clip and opens contact-sheet.jpg. It re-renders when a clip opens mid-thought, a caption covers the face, the hook and the first spoken words repeat each other, or the ending trails off. manifest.json lists byte sizes, durations, texts and warnings; every warning gets resolved.
Step 7: posting through MCP
Posting runs through WoopSocial, a social media API and MCP server for AI agents. Disclosure: the skill is maintained by Frank Heijdenrijk and the team behind WoopSocial. The free plan covers 2 social accounts with everything included; connection steps per agent are at woopsocial.com/agents.
The flow per clip: media_uploads_create_session with the exact byte size from the manifest, scripts/upload_parts.py to PUT each part to its presigned URL, media_uploads_complete_session for the media ID. Then one post per clip per platform (TikTok VIDEO, Instagram REEL, YouTube with title and privacy, Facebook REEL), each run through posts_validate. The agent shows one table of clip, platform, account, text and time with timezone, and calls posts_create only after a clear yes. With several accounts it varies clips or at least hooks and captions per account and staggers the times.
WoopSocial doesn't write captions or crop media; the skill writes the text and renders the vertical file before anything is uploaded. Skip this step entirely and you still get finished MP4s plus titles and captions in the manifest.
Reading results back
Three to seven days later, the agent calls social_account_posts_list and logs views, likes, comments, shares and saves (where a platform reports them, with the collection date) in clips-log.md next to the source timestamps and hook. Missing fields are recorded as not reported, never as zero. The next batch is pulled from the parts of the video that beat the account's usual numbers.
Guardrails in SKILL.md
- Only footage the user owns or has permission to clip; otherwise the agent stops.
- No identical clip posted at the same time to many accounts.
- Hooks, titles and captions say only what the clip says.
- Nothing posts without a yes.
Try it
npx skills add social-media-skills/video-to-shorts-skill -g -a claude-code -y
Then ask your agent: "Turn talk.mp4 into three shorts for TikTok and YouTube." The worked example with transcript, clips.json, contact sheet and manifest is in examples/nasa-interview/. Issues and pull requests are welcome on GitHub, and more free skills are at social-media-skills.com.
Top comments (0)