Concatenating scene-split videos with ffmpeg: the offset that only breaks at scale

What I built

I run a handful of small content sites, and I wanted short product-comparison videos generated from their article data automatically. The pipeline takes a script as JSON, and for each scene it synthesizes speech, records an HTML/CSS stage with headless Chromium, muxes the two with ffmpeg, and finally joins every scene into one file.

The part that ate the most of my time was the final join, so that is all this post is about. If your clips have a fixed duration, a single concat demuxer call is the whole story. But the duration of synthesized speech is not known ahead of time. What happens when you cross-fade clips whose length is variable is the real subject here.

For context, the TTS layer tries VOICEVOX, then edge-tts, then macOS say, in that order. Which engine ends up being used changes the duration for the same script. Variable length is not an accident; it is baked into the design.

The shape of the pipeline

The order of operations below is the answer to the problem.

script JSON
  |- per scene (in parallel)
  |    1. synthesize a wav per line with TTS, measure it with ffprobe
  |    2. accumulate line start times to fix the scene duration `total`
  |    3. pass that `total` to the recorder and capture the HTML stage
  |    4. trim the video to `total` and mux the audio -> scene_NN.mp4
  |- join every scene with xfade / acrossfade -> final.mp4

The obvious approach is "record the video first, fit the audio afterwards". Do that and the speech gets cut off when it is longer than the clip, or the clip sits in silence when it is shorter. Deciding that audio owns the duration, and moving that decision to the top of the pipeline, was the fix.

The interesting parts

1. Fix the duration from audio, then record

Synthesize a wav per line, accumulate start times with a GAP of 0.25 seconds between lines. The sum is the scene duration.

lines = scene.get("lines", []); timed = []; wavs = []; t = 0.0; GAP = 0.25
for i, ln in enumerate(lines):
    w = os.path.join(sdir, f"l{i}.wav")
    r = sh(["python3", TTS, "--text", ln["text"],
            "--voice", ln.get("voice", "metan"), "--out", w, "--engine", engine])
    d = json.loads(r.stdout.strip().splitlines()[-1])["duration"]
    timed.append({"text": ln["text"], "start": round(t, 3), "dur": round(d, 3)})
    wavs.append((w, d)); t += d + GAP
total = max(t, 2.0)

Having the TTS script report its own duration is what makes this work: the length is settled the moment the audio exists. timed then goes straight to the recorder as subtitle timings. The max(t, 2.0) is a floor so that a scene with zero lines (a title card, say) does not come out at zero seconds.

For joining the lines, instead of inserting silence between them with aevalsrc, I concatenate first and stretch the whole thing to total with apad.

filt = "".join(f"[{i}:a]" for i in range(len(wavs))) \
     + f"concat=n={len(wavs)}:v=0:a=1[c];[c]apad=whole_dur={total}[a]"

You can build a filtergraph that pads between every line, but it grows with the line count and becomes unreadable. Fitting the duration once at the end turned out to survive script edits much better.

The mux is just trimming the video to total and attaching the audio.

ffmpeg -y -i scene.webm -i scene.wav -c:v libx264 -pix_fmt yuv420p \
       -c:a aac -b:a 128k -shortest scene_01.mp4

2. The xfade offset is a position on the output timeline

This is the heart of it. The offset of xfade is not where the second input starts. It is the time at which the cross-fade begins within the output you have assembled so far. Because each join overlaps by xf seconds, the output gets shorter than the sum of the inputs, so adding raw clip durations does not line up.

fc = []; vlab = "0:v"; alab = "0:a"; off = durs[0] - xf
for i in range(1, n):
    nv = f"v{i}"; na = f"a{i}"
    fc.append(f"[{vlab}][{i}:v]xfade=transition=fade:duration={xf}:offset={off:.3f}[{nv}]")
    fc.append(f"[{alab}][{i}:a]acrossfade=d={xf}[{na}]")
    vlab = nv; alab = na; off += durs[i] - xf
total = sum(durs) - (n - 1) * xf

Drop the - xf in off += durs[i] - xf and everything drifts further back with every scene you add. With three scenes it just feels "slightly off" and you move on; it only becomes unmistakable around ten. You have to raise the scene count to see it.

Concretely, with 10-second scenes and xf = 0.5:

joining clip	correct offset	offset without - xf
2nd	9.5	9.5
3rd	19.0	20.0
4th	28.5	30.0
10th	85.5	90.0

The first join matches exactly. A minimal two-clip test can never catch this.

Note that acrossfade takes no offset at all. It simply overlaps d seconds with whatever came before, so unlike the video side you do not carry an accumulator. The two filters look symmetric and are not; worth re-reading the docs every time you write this.

3. A short scene makes cross-fading impossible

Since durations are variable, a scene shorter than xf will eventually show up. Feeding that to xfade produces broken output, so detect it and fall back to a plain concatenation.

if n < 2 or xf <= 0 or any(d <= xf + 0.2 for d in durs):
    lst = os.path.join(wd, "concat.txt")
    open(lst, "w").write("".join(f"file '{s['mp4']}'\n" for s in scenes_out))
    sh(["ffmpeg", "-y", "-f", "concat", "-safe", "0", "-i", lst,
        "-af", LOUDNORM, "-c:v", "libx264", "-preset", "veryfast",
        "-pix_fmt", "yuv420p", "-c:a", "aac", final])
    return final

The + 0.2 margin exists because the duration ffprobe reports and the real container duration differ slightly, and I did not want clips sitting exactly on the boundary to blow up.

4. Background music and final loudness

The music is not length-matched at all. It is looped in with -stream_loop -1 and cut to the program by amix with duration=first. loudnorm runs exactly once, at the end.

LOUDNORM = "loudnorm=I=-16:TP=-1.5:LRA=11"
fc.append(f"[{bi}:a]volume=0.10,apad[bg]")
fc.append(f"[{alab}][bg]amix=inputs=2:duration=first[amx]")
fc.append(f"[amx]{LOUDNORM}[aout]")

The important part is not applying loudnorm per scene. Normalizing scene by scene lifts the quiet scenes and flattens the dynamics of the whole video. Normalize the final output once.

Things that bit me

1. Forgetting to accumulate the offset (above). It does not reproduce with two scenes, so write the test with at least four.
2. Choosing between the concat demuxer and filter_complex. When there is no music and a plain join will do, -c copy is dramatically faster, but it assumes every scene shares a codec, resolution and frame rate. If the recorder does not pin -r 30, -c copy desynchronizes the audio immediately.
3. Forgetting -shortest. Even with the video trimmed to total, some containers leave a few frames at the tail. With the flag it reliably stops at the shorter stream.
4. Parallelism. Scenes run TTS, recording and mux through a ThreadPoolExecutor, but the default is three workers. Raise it and headless Chromium eats memory until a recording dies mid-way. Even on a fast machine, not being greedy finishes sooner.

The result

One of the sites where these videos are published: https://manga.autoarticles.net

Wrapping up

When your source material has a variable duration, do not try to reconcile it at the join. Put the stage that decides the duration (here, speech synthesis) at the top of the pipeline and make the video, the subtitles and the lip sync all follow that number. Join exactly once, at the end.

Carrying the xfade offset as an accumulator is the same principle. Map every clip onto one shared output timeline instead of tracking each clip's local time, and the arithmetic holds no matter how many scenes you add. Skip that, and you get the worst kind of bug: one that only appears once the input grows.

---

This article is about my own side project. It was written with AI assistance.