How We Stitch Parallel AI Video Segments With Frame Accuracy

james will — Mon, 09 Nov 2026 13:24:18 +0000

AI video models have a length limit, and the one we use renders at most 15 seconds per clip. That is a problem when a template has several shots and each cut has to land exactly where it does in the reference video.

So at IAmTheMeme we render multi-shot templates one shot at a time, in parallel, and stitch the results together afterwards. The rendering is the easy part to describe. The stitching is where a single wrong frame breaks everything after it.

Here is how that stitching works, including the ffmpeg flags that turned out to matter.

The pipeline

reference video, split at its hard cuts
        |
        v
shot 1   shot 2   shot 3 ...      rendered in parallel
        |
        v
trim each shot to an exact frame count
        |
        v
concat into one silent video
        |
        v
optional upscale
        |
        v
add the original audio, deliver

Audio goes on last because the upscaler works on the silent base video. That also means audio can never drift during the stitch, because it isn't there yet.

Count frames, not seconds

Our model only accepts whole seconds. A shot that lasts 65 frames at 30 fps is about 2.17 seconds, so we ask for 3 seconds and get roughly 90 frames back. We need the first 65.

Seconds are the wrong unit for that cut. -t 2.17 depends on rounding and on the timestamps in the file, while the template's timing is really a list of frame counts. So each template stores its frame rate and one frame count per shot, and that list is the source of truth for the entire edit.

An earlier version used a filter chain with fps and trim. On a real template it came out one frame short, which is the kind of bug you only notice as a slightly late cut. We replaced it with -frames:v on the output:

ffmpeg -v error -y -i shot.mp4 \
  -vf "fps=30,scale=480:854,setsar=1,format=yuv420p" \
  -frames:v 65 \
  -an -c:v libx264 -preset veryfast -crf 16 -bf 0 \
  shot-trimmed.mp4

Each flag has a job:

fps, scale, setsar, format make every shot identical in frame rate, size, pixel aspect ratio and pixel format. The concat step depends on this.
-frames:v 65 keeps exactly the first 65 frames.
-crf 16 is close to lossless, because this file still goes to the upscaler. Quality lost here can't be recovered later.
-bf 0 turns off B-frames, so each part's first frame starts at timestamp zero and the seams don't drift.

Yes, this re-encodes every shot. The cut points rarely fall on keyframes, and stream copy can only cut cleanly on a keyframe, so you can end up with too many or too few frames. It costs some CPU, and it is the only way we found to get exact cuts.

Concat can be a stream copy

Re-encoding pays off in the next step. All the parts now share the same parameters, so the concat demuxer can join them without encoding anything again:

# list.txt
# file 'part-0.mp4'
# file 'part-1.mp4'
# file 'part-2.mp4'

ffmpeg -v error -y -f concat -safe 0 -i list.txt \
  -c copy -movflags +faststart stitched.mp4

Verify before you deliver

An exit code of 0 from ffmpeg doesn't mean you got the frames you asked for. If a model returns a shot shorter than expected, -frames:v simply stops early and everything after it lands late.

So we check the count after every part and again after the concat:

ffprobe -v error -select_streams v:0 \
  -show_entries stream=width,height,nb_frames \
  -of json stitched.mp4

In MP4, nb_frames comes from the container's sample table, so this is cheap and doesn't decode anything. The check in the service looks like this (simplified):

const probe = parseProbedVideo(await run('ffprobe', PROBE_VIDEO_ARGS(partPath)));
if (probe.frames !== frames[index]) {
  return sendJson(res, 422, { ok: false, error: 'part_too_short' });
}

// ...after the concat, the same check against the total
const total = frames.reduce((sum, value) => sum + value, 0);
if (result.frames !== total) {
  return sendJson(res, 422, { ok: false, error: 'frame_count_mismatch' });
}

The status code carries meaning here. A 4xx says the input is wrong and retrying won't help. A 5xx says the container failed and a retry might work. The job's state machine uses that distinction to decide what to do next.

Running ffmpeg next to Cloudflare Workers

The app runs on Cloudflare Workers, and Workers can't execute native binaries. ffmpeg therefore runs in Cloudflare Containers, behind a small Worker that is only reachable through a service binding from the main app.

Three decisions shaped that service:

The container has no credentials and no network access. The Worker reads the parts from R2, posts them to the container as multipart form data, and writes the result back. ffmpeg parsing untrusted media is a real attack surface, and a compromised container here has no route to the bucket.

Output keys are deterministic. The stitched file's key is derived from the job's segments. If that object already exists, the Worker returns it without waking a container, so a retry after a timeout costs nothing.

The container's arguments are pure functions. Each ffmpeg command is built by a plain function that unit tests can assert on directly. When ffmpeg is available locally, an end-to-end test stitches solid-color clips at 24 and 30 fps, then checks that each cut lands on the expected frame and that a too-short part gets rejected. Using a different color per shot makes a misplaced cut easy to detect.

Container size mattered too. Scaling an upscaled 720p video takes around 27 CPU seconds, which gets too close to the call timeout on a quarter vCPU, so we run on half a vCPU.

Charging for a job that can fail halfway

Parallel shots add a billing question: what happens if shot 4 of 5 fails?

We reserve the credits when the order is placed, before any rendering starts. If the job ends without a finished video, the reservation is released and the credits go back to the user. Every step is a conditional update on the job's state, so a retry or a concurrent worker can't render or charge twice.

The short version

Treat frame counts as the source of truth and derive durations from them, never the other way round.
Normalize every part before concat, even when the parts all come from the same model.
Verify output with ffprobe, because a successful exit code proves very little.
Keep credentials out of whatever process touches untrusted media.

If you want to see the output, our Tiny Desk AI video generator is built from five stitched shots at 24 fps.

How are you handling frame-accurate cuts in your own pipelines? We'd especially like to hear from anyone who has found a reliable way to do it without re-encoding.

DEV Community: james will