takahiro hashito

Posted on Sep 2

Stitching scene-based explainer videos with ffmpeg: audio first, one encode profile

#node #automation #showdev

Background

I generate short explainer videos for my sites out of HTML/CSS screen recordings, free text-to-speech, and ffmpeg. No paid narration service, no paid image generation — a local, free stack end to end.

This post skips the architecture tour and focuses on the ffmpeg side: getting audio first, recording scenes to match, and joining them without the result falling apart.

How it works

Four steps per video:

Turn each scene's script into audio with free TTS (VOICEVOX if it is running, otherwise edge-tts, otherwise macOS say).
Measure that audio's duration first, then record the screen for exactly that long (Playwright + Chromium, producing webm).
Mux video and audio into one mp4 per scene.
Concatenate every scene into the final file.

The TTS engine is pluggable so the pipeline runs on whatever the machine happens to have, with all engines normalized to 24 kHz mono wav.

Implementation

Build the audio first so the duration is fixed

Recording video first and fitting narration to it guarantees drift. Instead, synthesize the audio, measure it with ffprobe, and record for exactly that many seconds.

import subprocess, json
def probe_dur(path):
    o = subprocess.run(
        ["ffprobe", "-v", "quiet", "-show_entries", "format=duration",
         "-of", "csv=p=0", path],
        capture_output=True, text=True)
    return float(o.stdout.strip())

total = probe_dur("scene_audio.wav")   # pass this to the recorder as its length

Playwright's recordVideo writes the webm and gets stopped at total seconds, so the two tracks come out the same length by construction.

Mux, with an encode profile chosen for compatibility

The silent webm and the wav become one mp4. libx264 plus aac, with -pix_fmt yuv420p, plays reliably in social feeds and in every player I tested. -shortest trims to the audio.

ffmpeg -y -i scene_video.webm -i scene_audio.wav \
  -t "$total" -r 30 \
  -c:v libx264 -preset veryfast -pix_fmt yuv420p \
  -c:a aac -b:a 128k -shortest scene_01.mp4

Every scene uses this exact profile. That is not tidiness — the next step depends on it.

Concatenate

With no transition needed, the concat demuxer joins scenes with no re-encoding at all, which is by far the fastest path:

# list.txt contains: file 'scene_01.mp4' / file 'scene_02.mp4' / ...
ffmpeg -y -f concat -safe 0 -i list.txt -c copy out.mp4

For a crossfade between scenes, xfade re-encodes instead. That lifts the matching-codec requirement but costs real processing time:

# joining two scenes with a fade; offset is (previous scene length - fade duration)
ffmpeg -i scene_01.mp4 -i scene_02.mp4 -filter_complex \
  "[0:v][1:v]xfade=transition=fade:duration=0.35:offset=OFF[v]" \
  -map "[v]" -c:v libx264 -preset veryfast -pix_fmt yuv420p out.mp4

For background music, mix it in after concatenation with amix, then run loudnorm at the end so per-scene level differences get evened out.

Gotchas

-c copy in concat assumes identical codecs. Re-encode-free joining requires the codec, resolution, frame rate and pix_fmt to match across every input. One mismatch and the output tears or the audio drifts. This is exactly why the scene encode profile lives in one place and never varies.
-pix_fmt yuv420p is not optional. Leave it out and some players — mobile and in-feed players especially — render a black rectangle.
Record slightly longer than the narration. If the recording is shorter than the audio, the tail gets cut. Add a small margin to total when recording, then trim back with -t $total and -shortest during the mux.
Give silent scenes a real silent track. A scene with no narration needs an anullsrc audio stream of the right length, or the later concat and amix stages break on the missing stream.

The through-line in all four: the concat step is the fragile one, and everything upstream is arranged so that it can stay a stream copy. Uniform encode settings, matched durations, and a real audio stream on every scene are all just preconditions for -c copy to work.

The result

One of the sites these generated videos promote: https://manga.autoarticles.net

Wrap-up

The pipeline in one line: free TTS first, duration fixed by ffprobe, one mp4 per scene under a single encode profile, then concat demuxer (or xfade when you want transitions).

Pin the encode settings in exactly one place and a local, entirely free stack turns out to be perfectly capable of assembling explainer videos on a schedule. Almost every problem I hit came from letting a scene deviate from the shared profile — so the profile is the thing worth guarding.

This article is about my own side project. It was written with AI assistance.

DEV Community