DEV Community

Syncoders
Syncoders

Posted on Edited on

Building an AI Shorts pipeline with ffmpeg: 9:16, Ken Burns zoom, captions that match the voice

#ffmpeg #ai #node #video

TL;DR

  • Ask an LLM for a scene plan in JSON, and check every field before you use it.
  • Crop each image to 9:16 first, then add a slow zoom with ffmpeg's zoompan filter.
  • Make each scene as long as its voice-over (measured with ffprobe), not as long as the LLM guessed.
  • Split the narration into short caption chunks, time them by length, and burn them into the video.
  • Encode every scene the same way, then join them with no re-encoding.
  • Retry rate limits and server errors. Fail fast on bad requests and bad keys.

Calling an image API and a text-to-speech API is the easy part of an "AI video generator". Turning their output into a video people will actually watch is the work. This post builds that pipeline with Node and plain ffmpeg. Each step is one ffmpeg command you can paste into a terminal first, then wrap in code.

A few words first

  • LLM: a large language model, the kind of AI that writes text (and JSON).
  • TTS (text-to-speech): a service or program that turns text into a voice recording.
  • ffmpeg / ffprobe: free command-line tools. ffmpeg converts and edits video and audio. ffprobe reads facts about a file, such as its length.
  • 16:9 and 9:16: width-to-height ratios. 16:9 is a landscape YouTube video. 9:16 is a vertical Short, Reel or TikTok.
  • Filter: an ffmpeg step that changes the picture or sound, like scale (resize) or crop.

The whole pipeline

Pipeline in three rows. Plan: brief, LLM writes a scene plan in JSON, validate it. Assets, per scene, in parallel: image from an image API, and a voice-over from text-to-speech that gets measured with ffprobe. Render: ffmpeg per scene crops to 9:16, zooms and adds captions, then ffmpeg concat joins the clips into final.mp4.

One brief, like "3 tips for better sleep", goes in. The LLM breaks it into scenes. Each scene gets an image and a voice-over. ffmpeg turns each scene into a short clip, then joins the clips into one MP4.

Step 1: get a scene plan you can trust

What goes wrong: the LLM's answer crashes your code, or produces a video that's far too long.

Everyday example: you ask for 6 scenes in JSON. The model wraps the JSON in a code fence, adds "Here's your plan!" before it, returns 14 scenes, and writes "durationSec": "3s" as text instead of a number.

Why it happens. An LLM predicts likely text. It usually follows your format, but nothing forces it to.

The fix: ask for JSON, then check every field before you use it. On OpenAI-compatible APIs, response_format: { type: "json_object" } asks the model for JSON only. The plan looks like this:

{
  "title": "3 tips for better sleep",
  "scenes": [
    { "imagePrompt": "dim bedroom, warm lamp, cinematic", "narration": "Keep the same wake-up time, even on weekends.", "durationSec": 3 },
    { "imagePrompt": "phone face down on a nightstand", "narration": "Put screens away thirty minutes before bed.", "durationSec": 3 }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Then, in your code:

  1. Strip any code fences and text around the JSON.
  2. JSON.parse it.
  3. Check every field has the right type.
  4. Clamp the numbers: for example 2 to 12 seconds per scene, and a maximum scene count.
  5. If anything is still wrong, throw an error your job queue will retry.

Step 2: turn a still image into a moving 9:16 clip

What goes wrong: the Short looks stretched, or looks frozen.

Everyday example: your image API returns a 16:9 landscape picture. You resize it to 720×1280, and the round sun becomes a tall oval. Or you keep it sharp and still, and viewers swipe away because nothing moves.

Why it happens. Resizing a landscape image to a portrait size stretches it, because the shapes don't match. The right move is to crop: keep a 9:16 slice from the middle and drop the sides.

Three panels. A 16:9 source image with a green box showing the 9:16 slice a crop keeps. Wrong: the same image squashed to 9:16, with the round sun stretched into an oval. Right: a real ffmpeg frame, cropped to 9:16 first, then zoomed.

For motion, the cheapest effect that looks deliberate is the Ken Burns effect: a slow zoom into a still photo. It's named after the documentary maker who used it heavily. ffmpeg's zoompan filter does it. Each frame shows a slightly smaller, centred part of the image, stretched to fill the screen, so the picture slowly pushes in.

Left: the cropped image with three boxes showing the visible area at frame 1 (zoom 1.00), frame 85 (zoom 1.07, the end of a 3.4 second scene) and the 1.15 cap. Middle and right: the real first and last frames of the clip, showing the slow push-in.

The fix: scale up, crop to 9:16, then zoom. Here's the whole command for one scene:

W=720; H=1280; FPS=25; DUR=3.4
FRAMES=85   # DUR × FPS

ffmpeg -y -loop 1 -i scene.png -i voice.mp3 -filter_complex "\
[0:v]scale=900:1600:force_original_aspect_ratio=increase,crop=900:1600,\
zoompan=z='min(zoom+0.0008,1.15)':x='iw/2-(iw/zoom/2)':y='ih/2-(ih/zoom/2)':d=$FRAMES:s=${W}x${H}:fps=$FPS,\
format=yuv420p[v]" \
  -map "[v]" -map 1:a -af apad -t $DUR \
  -c:v libx264 -preset veryfast -c:a aac -ar 44100 -ac 2 segment.mp4
Enter fullscreen mode Exit fullscreen mode

What each part does:

  • -loop 1 -i scene.png repeats the still image as a video stream. -i voice.mp3 adds the voice-over.
  • scale=900:1600:force_original_aspect_ratio=increase,crop=900:1600 grows the image until it covers 900×1600, then cuts away what sticks out. That gives a 9:16 picture whatever shape the source was. It's 1.25× bigger than the output on purpose: zoompan jitters when it zooms into a small image.
  • z='min(zoom+0.0008,1.15)' zooms in 0.08% per frame, which is 2% per second at 25 fps, and stops at 15%. The x and y parts keep the zoom centred.
  • d=$FRAMES:s=${W}x${H}:fps=$FPS sets the clip's frame count, output size and frame rate.
  • format=yuv420p uses the pixel format phones and browsers can play.
  • -af apad pads the voice with silence, and -t $DUR cuts the clip to the exact length.

For 16:9, swap the numbers: 1600:900 and s=1280x720.

Step 3: make each scene as long as its voice

What goes wrong: the narration gets cut off mid-word at the end of a scene.

Everyday example: the LLM planned 3 seconds for "Keep the same wake-up time, even on weekends." The voice takes 3.11 seconds to say it, so "weekends" gets clipped.

Why it happens. The LLM has no idea how fast your TTS voice speaks. Its durationSec is a guess.

Timeline bars. Voice-over: 3.11 seconds. Planned scene: 3.0 seconds, so the last 0.11 seconds of the voice is cut off. Fixed scene: max(3.0, 3.11 + 0.25) = 3.36 seconds, which includes a 0.25 second breath after the voice ends.

The fix: generate the voice first, measure it with ffprobe, then set the scene length:

ffprobe -v error -show_entries format=duration -of csv=p=0 voice.mp3
# 3.108571
Enter fullscreen mode Exit fullscreen mode 
const speech = await probeDuration(audioPath);            // runs the ffprobe command above
const dur = Math.max(scene.durationSec, speech + 0.25);   // never shorter than the voice, plus a breath
Enter fullscreen mode Exit fullscreen mode

That one line is the difference between a clip that sounds edited and one that sounds broken.

Step 4: burn in captions that follow the voice

What goes wrong: viewers with the sound off don't know what the video says.

Everyday example: someone scrolls past your Short on a train, with their phone on mute. There's no text on screen, so they never hear your three tips.

Why it happens. Many people watch Shorts on mute, and a voice-over alone tells them nothing.

The fix: split the narration into chunks of 3 or 4 words, give each chunk a start and end time, and draw them onto the video. The timed chunks are saved as an SRT file, a plain-text subtitle format. ffmpeg's subtitles filter draws ("burns") them into the picture. It uses libass, a subtitle library that standard ffmpeg builds include.

Most TTS services don't tell you when each word is spoken. Giving each chunk time in proportion to its length is close enough to read along:

Timeline of one 3.36 second scene. Three caption cues sit above the real voice waveform: 

function captionCues(text: string, duration: number, maxWords = 4) {
  const words = text.trim().split(/\s+/);
  const chunks: string[] = [];
  for (let i = 0; i < words.length; ) {
    let n = Math.min(maxWords, words.length - i);
    for (let k = 0; k < n - 1; k++) if (/[.,!?;:]$/.test(words[i + k])) { n = k + 1; break; } // break at punctuation
    chunks.push(words.slice(i, i + n).join(" "));
    i += n;
  }
  const total = chunks.reduce((s, c) => s + c.length + 2, 0);
  let t = 0;
  return chunks.map((c) => { const d = (duration * (c.length + 2)) / total; const cue = { start: t, end: t + d, text: c }; t += d; return cue; });
}

const ts = (s: number) => new Date(s * 1000).toISOString().slice(11, 23).replace(".", ",");
const toSrt = (cues: { start: number; end: number; text: string }[]) =>
  cues.map((c, i) => `${i + 1}\n${ts(c.start)} --> ${ts(c.end)}\n${c.text}\n`).join("\n");
Enter fullscreen mode Exit fullscreen mode

For the example line, this writes:

1
00:00:00,000 --> 00:00:01,576
Keep the same wake-up

2
00:00:01,576 --> 00:00:02,056
time,

3
00:00:02,056 --> 00:00:03,358
even on weekends.
Enter fullscreen mode Exit fullscreen mode

Then add the subtitles filter to the Step 2 command, just before format=yuv420p:

,subtitles=scene-0.srt:force_style='FontName=DejaVu Sans,FontSize=10,Bold=1,Outline=1.6,Shadow=0.8,Alignment=2,MarginV=64'
Enter fullscreen mode Exit fullscreen mode

Two things that cost me time:

  1. FontSize is not in pixels. For SRT files, libass lays text out on a canvas 288 units tall, then scales it to the video. So FontSize=10 on a 1280-pixel-tall video is about 44 px. For text about 6% of the short side, use FontSize = 0.06 × min(W, H) × 288 / H.
  2. File paths inside a filter string are painful. Colons, backslashes and quotes all need escaping, and Windows drive letters make it worse. Run ffmpeg from the folder that holds the SRT and pass just the file name. Never put user text directly into a filter string; write it to a file first.

Step 5: a free voice for drafts

What goes wrong: every test render costs money.

Everyday example: you're tuning caption sizes and render the same video 30 times. With a paid voice API, you pay for the same three sentences 30 times.

Why it happens. Paid TTS bills per character, even for drafts nobody will see.

The fix: use an offline voice for drafts. If your ffmpeg build includes flite (a small offline speech engine), ffmpeg can speak on its own:

ffmpeg -y -f lavfi -i "flite=textfile=line.txt:voice=slt" -af loudnorm=I=-16 -ar 44100 voice.mp3
Enter fullscreen mode Exit fullscreen mode

On macOS, say -o voice.aiff -f line.txt works too. On Linux and Windows, espeak-ng -w voice.wav -f line.txt does. These voices sound robotic, so keep them for drafts. Put them behind the same interface as your paid voice, so switching is one environment variable:

interface VoiceProvider { name: string; speak(text: string, outPath: string): Promise<void> }
Enter fullscreen mode Exit fullscreen mode

Step 6: join the scenes without re-encoding

What goes wrong: joining the clips is slow and loses quality, or the audio drifts out of sync.

Everyday example: joining a 40-scene video crawls along because every frame is compressed again, or halfway through, the voice starts to lag behind the picture.

Why it happens. Re-encoding means decoding the video and compressing it again, which is slow and loses a bit of quality each time. ffmpeg's concat demuxer can join clips without re-encoding, but only if every clip was encoded the same way.

The fix: encode every scene with identical settings (resolution, frame rate, pixel format, codec, audio sample rate and channel count), then join them:

printf "file '%s'\n" scene-0.mp4 scene-1.mp4 scene-2.mp4 > list.txt
ffmpeg -y -f concat -safe 0 -i list.txt -c copy -movflags +faststart final.mp4
Enter fullscreen mode Exit fullscreen mode
  • -c copy copies the video and audio as they are, so it's almost instant.
  • +faststart moves the file's index to the front, so the video starts playing in a browser before it has fully downloaded.

If concat complains or the audio drifts, one clip was encoded differently. Usually it's a mono voice file that skipped -ac 2.

Step 7: survive real APIs

What goes wrong: renders that work on your laptop fail in production.

Everyday example: the image API answers "429 Too Many Requests" on scene 12 of 40, and the whole job dies. You retry it, and it pays for a new LLM plan before failing again.

Why it happens. Real APIs have rate limits and bad moments. Some errors fix themselves if you wait. Others never will, like a wrong API key.

Flowchart: an API call fails. On 429 (rate limit) or 5xx (server error), wait and retry at 1 s, 2 s, 4 s and so on, plus random jitter; if it works, carry on with the saved plan. On 400, 401 or 403 (bad request or bad key), or if retries keep failing, fail the job with a clear message.

The fix: three rules.

  • Retry only errors that can fix themselves. Retry 429 and 5xx with exponential backoff (wait 1 s, then 2 s, then 4 s) plus jitter (a small random extra wait, so many jobs don't retry at the same instant). Don't retry 400, 401 or 403. Fail the job at once with a clear message instead of burning credits.
  • Save finished steps on the job. Store the validated plan on the job record. When the job retries after an image failure, it skips the LLM call it already paid for.
  • Limit how much runs at once. Render, say, 3 scenes at a time and 2 jobs at a time, and make both configurable. Firing 40 requests at once with Promise.all is how you get rate-limited.

For one machine, a simple file-based job store is enough: one job.json per job, written atomically, plus a worker loop that picks up queued jobs. On startup, put any job still marked running back in the queue, so a crash doesn't strand it. When you outgrow one machine, put Redis/BullMQ or Postgres SKIP LOCKED behind the same interface.

The checklist

  • Validate the LLM's JSON. Clamp scene count and durations.
  • Scale up, crop to the target shape, then zoompan.
  • Measure the voice with ffprobe. Scene length = max(planned, speech + 0.25 s).
  • SRT chunks of 3 to 4 words, burned in with subtitles. Font size is in 288-unit canvas terms, not pixels.
  • Run ffmpeg in the asset folder. Text goes in files.
  • Identical encode settings, then concat -c copy with +faststart.
  • Retry only what's retryable, keep the plan on the job, limit concurrency.

Questions about any of the ffmpeg filters above? Ask in the comments and I'll answer.

Want this wired together? The AI Video Pipeline Kit ($49) is a Next.js + TypeScript + ffmpeg pipeline with a parallel job queue and 37 tests, and it runs with zero API keys (mock art and an offline voice).

Top comments (0)