DEV Community

Kelvaro
Kelvaro

Posted on Edited on

Build a local Python pipeline that turns a long talk into six 9:16 Shorts

If you cut Shorts from podcasts or interviews, most of the clock time is not creative. It is the same checklist: find a 30-second stretch that stands on its own, drop the dead air, reframe to 9:16, put the face in the middle, burn in captions, add a hook title, maybe a zoom and a sound on the punch word. That checklist is regular enough to automate on your own machine.

This article walks through a local Python pipeline that does that. No cloud editor, no monthly seat. The blocks are ordinary: faster-whisper for a word-timed transcript, a small scorer (or an LLM that only sees text) to pick clips, OpenCV to track a face, and FFmpeg to render. I will be honest about what this replaces and what it does not.

What the pipeline should emit

A useful default is six clips, each 20–60 seconds (30–45 is the sweet spot), already in 9:16, with silence trimmed, a vertical frame, word-by-word captions, a hook title in the first couple of seconds, light zooms, optional B-roll and sound effects, a progress bar, and a clean end that can loop.

Support two inputs, because the media is different:

  • talk — one long file (podcast, interview, vlog). Find the best moments and cut each as its own Short.
  • lifestyle — a folder of raw clips with little speech. Edit the folder to a beat: speed ramps, flash or whip transitions, kinetic type, a lower third, a simple grade.

The rest of this article is talk mode. Lifestyle is a second renderer on the same command, not a second architecture.

A hard rule: only process video you or your client have the rights to. yt-dlp can fetch a URL, but downloading someone else's upload and reposting it breaks copyright and platform rules, automation or not.

The command shape

Keep one entry point:

# heuristic selection, then render six Shorts
python clip.py podcast.mp4 --lang en

# German audio, more accurate captions (slower)
python clip.py interview.mp4 --lang de --model medium

# export a transcript for an LLM to pick clips, then render that file
python clip.py podcast.mp4 --analyze
python clip.py podcast.mp4 --clips work/podcast/clips.json --only 2 5
Enter fullscreen mode Exit fullscreen mode

Useful flags: --fast (FFmpeg veryfast), --no-sfx, --no-broll, --music default for a quiet bed with ducking, and --crop for a full-bleed vertical crop. Cropping is fashionable; it also chops hands and slides. A safer default is to fit the whole landscape frame onto 9:16 and use a blurred copy of the same frame as background.

You need Python 3.12, FFmpeg and ffprobe on PATH, faster-whisper, OpenCV, NumPy, Pillow, SciPy, soundfile, and librosa. A GPU helps Whisper, but a CPU-only run with --model base and --fast is enough for a first pass.

Step 1: a transcript with word timestamps

Captions and clip boundaries both need word times, not paragraph times. faster-whisper (the CTranslate2 port of Whisper) writes {word, start, end} locally, including German with --lang de. From that list you can group sentences on punctuation, treat long gaps as silence, and later highlight the active caption word.

Write work files into work/<slug>/. A replayable transcript lets you re-render clip 3 without transcribing again.

Step 2: find stretches that work as Shorts

A Short has to survive without the rest of the episode. Score seven checks from 0–10 and skip anything with a mean under 6:

  1. Hook in the first two seconds. A question, a hard claim, a number, a contradiction. Not "so, yeah…"
  2. Self-contained. No dangling "he" or "as I said."
  3. One thought. Setup, example, payoff.
  4. Payoff at the end, and a cut on the payoff, not after it trails off.
  5. Energy. A laugh, a pace spike, a personal story.
  6. Something to comment on. A take, a myth, a surprise.
  7. Length. Prefer 30–45 s; allow 20–60 s. Short and sharp beats long and soft.

Start on the first word of the hook sentence. End on the last word of the payoff (keep the laugh if it comes right after). Do not let clips overlap.

A first version does not need an LLM. Slide a sentence window across the 20–60 s range. Boost hook words in the first sentence, questions, and emotion or controversy terms (English and German). Penalize openings that start mid-thought (And, So, Und, Also). Prefer a clean sentence end and a duration near 38 s. Measure RMS energy with FFmpeg (f32le mono at 8 kHz) and reward variance, plus a mild words-per-second bonus. Sort by score and greedily keep non-overlapping winners.

That already beats "cut every 45 seconds from the start." It still misses quiet jokes and can over-rank a segment that merely contains the word "secret."

The better, still cheap path is --analyze: export a timestamped sentence list plus a short briefing, and let Claude or ChatGPT apply the seven checks and write a clips.json (start, end, a hook title of at most six words, keywords to highlight, optional speaker name, optional B-roll search terms). The model never sees the pixels, only text. A 20-minute episode is roughly 5,000–8,000 tokens. Then render exactly that file.

Step 3: reframe to 9:16 without a jumpy crop

For each output frame, detect the face (OpenCV's YuNet model is small and MIT-licensed), aim a 9:16 window at it, and smooth the window over time. Without a low-pass, every head nod becomes a camera bump.

Two framing policies:

  • fit (default): letterbox the full frame onto 9:16, blur and scale the same frame as background. Nothing important is shaved off.
  • crop: fill the frame and follow the face. Better for a tight talking-head; worse when two people share the shot or the speaker holds up a notebook.

If two faces are in frame, prefer a window where one person is clearly speaking, or start from a file that is already vertical.

Step 4: render in one FFmpeg pass

Once you have (start, end) and the word list, the renderer is bookkeeping: cut the source, drop pauses above a threshold (around 0.35 s works), overlay word-by-word captions with the current word emphasized, show a hook title for about 2.5 seconds, punch-in zoom on three to eight keywords, optionally lay B-roll on those hits, add light SFX, duck a music bed, and draw a progress bar with a last-frame hold so the clip can loop.

For B-roll, a local folder named by keyword is the cleanest option. If you fall back to a stock API, write a credits file and put required CC BY attribution in the description.

Generating ASS or title/drawtext filters from the word timestamps is more reliable than typography frame-by-frame in Python, except for motion-graphics layers you pre-render with Pillow.

Watch every file before you post. The pipeline will caption a mis-heard name, start a cut on a breath, or zoom on the wrong word. A two-minute review saves a bad upload.

Lifestyle mode, briefly: skip Whisper, detect the beat with librosa, lay clips on the downbeats, and add ramps plus flash, glitch, zoom, or whip-pan transitions. This will not beat a careful CapCut or After Effects pass on tracking and music feel. Treat it as a rough cut.

Rights, cost, and an honest ceiling

Use client-cleared or your own recordings. Running locally means no subscription for the core path. Optional API keys (Pexels, Pixabay, or an LLM) are extras. Budget about fifteen minutes on Windows for Python, FFmpeg, a venv, and pip install. Linux and macOS should work with Python 3.12 and FFmpeg; I have only treated Windows as the tested baseline.

A pipeline like this can take over something like 70–80% of the repetitive cutting. It does not replace a good editor, and it will not invent a hook that is not in the tape. The win is that you spend your time on selection and a last pass, not on clicking the same five effects in a timeline.

I packaged this exact flow as a paid kit (the Auto-Clipping-System): the Python source, a German step-by-step guide, a Windows installer script, the clip-scoring rules, prompts for Claude, bundled CC0 sounds, OFL fonts, and demo clips. It is 29 € once, no subscription. If you want the boxed version instead of wiring the stages yourself, it is here: https://whop.com/kelvaro-creator-kits/auto-clipping-system

Disclosure: that kit is my own paid product. No guarantee of views or revenue. Please preview every clip before you publish it.


Free Clip Checklist (seven checks plus copy-paste FFmpeg commands) as a PDF:
https://whop.com/kelvaro-creator-kits/gratis-clip-checkliste/

If you want the full local Auto-Clipping-System: launch offer, code LAUNCH40 = 40% off, first 10 buyers only, until 16 November 2026.
https://whop.com/kelvaro-creator-kits/auto-clipping-system

Disclosure: the paid kit is my own product. The checklist is free. No guarantee of views or revenue.

Top comments (1)

Collapse
 
fredabila profile image
Fred Abila •

Keeping the LLM to text-only clip selection and leaving the timing to the word transcript is what keeps this cheap and predictable. One rule I'd add to the scorer: reject candidates that start mid-sentence or open on a pronoun with nothing before it ("he said that..."), since those are the Shorts that don't stand on their own even when the score is high. Disclosure: I build Vidmoat, a hosted AI video editor, and for people who'd rather not run the Python and OpenCV side, that's the part it handles: you ask in chat for the vertical cuts, captions and trimmed silences, and each lands on a real timeline you can still adjust before rendering the MP4. Your local pipeline is the better fit when footage can't leave the machine or you want full control of the scorer.