How to Add Subtitles to a Video With FFmpeg (and Time Them From the Audio)

There are two ways to add subtitles to a video with FFmpeg:

• Soft subtitles: a separate text track the viewer can turn on or off. For MP4 that means converting the .srt to mov_text: ffmpeg -i in.mp4 -i subs.srt -map 0 -map 1 -c:v copy -c:a copy -c:s mov_text -metadata:s:s:0 language=eng out.mp4
• Burned-in subtitles: the text is drawn into the pixels, so it shows everywhere, including players that ignore subtitle tracks: ffmpeg -i in.mp4 -vf "subtitles=subs.srt" -c:a copy out.mp4

Both commands are short. What goes wrong is everything around them: cue timings that drift from the speech, an .srt that MP4 refuses to carry, and an FFmpeg build that has no subtitles filter at all. This post builds the whole pipeline in Python, times the cues from the audio itself, and tests every step with pytest. All measurements are from FFmpeg 9.0.1 on macOS.

The test material

The audio is a real text-to-speech take: a three-step cooking lesson generated with Gemini TTS and published as an example on Voor's Gemini TTS page (lesson-flash.mp3, 16.36 s, 24 kHz mono MP3). Because it is TTS, the script is known exactly:

Step one: heat the oven to two hundred degrees.
Step two: while it warms, cut the vegetables into even pieces, so everything roasts at the same speed.
Step three: a little oil, a pinch of salt, and into the oven they go.

That is the ideal case for subtitles: you don't need speech recognition, only timing. The second fixture is a French museum announcement from the same page, used to test two subtitle languages in one file.

Step 1: Find the speech with silencedetect

silencedetect prints where silence starts and ends. Everything between is speech:

def speech_segments(audio, noise_db=-35, min_silence=0.35):
    log = run([FFMPEG, "-hide_banner", "-nostats", "-i", str(audio),
               "-af", f"silencedetect=noise={noise_db}dB:d={min_silence}", "-f", "null", "-"]).stderr
    starts = [float(x) for x in re.findall(r"silence_start: ([\d.]+)", log)]
    ends = [float(x) for x in re.findall(r"silence_end: ([\d.]+)", log)]
    total = duration(audio)
    segs, cursor = [], 0.0
    for s, e in zip(starts, ends + [total] * (len(starts) - len(ends))):
        if s - cursor > 0.05:
            segs.append((round(cursor, 3), round(s, 3)))
        cursor = e
    if total - cursor > 0.05:
        segs.append((round(cursor, 3), round(total, 3)))
    return segs

On the lesson this returns four segments for three sentences:

Segment	Start (s)	End (s)	Speech
1	0.000	3.239	Step one …
2	4.109	11.093	Step two …
3	11.595	12.198	"Step three:"
4	12.602	15.999	… into the oven they go.

The voice pauses for 0.40 s after "Step three:". Map segments to lines one-to-one and every cue after that point is wrong.

Step 2: Merge the shortest pauses until the counts match

Instead of tuning the silence threshold until it happens to work, merge across the shortest gaps until there is exactly one segment per subtitle line. If there are fewer segments than lines, fail loudly:

def fit_segments(segs, n):
    if len(segs) < n:
        raise ValueError(f"{len(segs)} speech segments for {n} lines")
    segs = list(segs)
    while len(segs) > n:
        gaps = [segs[i + 1][0] - segs[i][1] for i in range(len(segs) - 1)]
        i = gaps.index(min(gaps))
        segs[i:i + 2] = [(segs[i][0], segs[i + 1][1])]
    return segs

Then turn segments into cues with a small lead-in (80 ms) and tail (250 ms), clamped so a cue never runs past the audio or into the next cue:

def cues(lines, segs, total, lead=0.08, tail=0.25):
    segs = fit_segments(segs, len(lines))
    out = []
    for k, (text, (s, e)) in enumerate(zip(lines, segs)):
        start = max(0.0, s - lead)
        end = min(total, e + tail)
        if k + 1 < len(segs):
            end = min(end, segs[k + 1][0] - lead - 0.001)
        out.append((round(start, 3), round(end, 3), text))
    return out

The resulting SRT (note the comma before the milliseconds; SRT uses 00:00:03,489, WebVTT uses a dot):

1
00:00:00,000 --> 00:00:03,489
Step one: heat the oven to two hundred degrees.

2
00:00:04,029 --> 00:00:11,343
Step two: while it warms, cut the vegetables into even pieces, so everything roasts at the same speed.

3
00:00:11,515 --> 00:00:16,249
Step three: a little oil, a pinch of salt, and into the oven they go.

Step 3: Make the video, with an exact length

For the test, the picture is a plain 1280×720 background. The obvious command uses -shortest to stop when the audio ends. In my runs it didn't: the output had 18.43 s and 18.67 s of video over 16.36 s of audio. Passing the probed audio length with -t gave 16.367 s every time:

def base_video(audio, out, size="1280x720", color="0x1d2330", exact=True):
    length = ["-t", f"{duration(audio):.3f}"] if exact else ["-shortest"]
    run([FFMPEG, "-y", "-f", "lavfi", "-i", f"color=c={color}:s={size}:r=30",
         "-i", str(audio), *length, "-c:v", "libx264", "-pix_fmt", "yuv420p",
         "-c:a", "aac", "-b:a", "128k", "-movflags", "+faststart", str(out)])

If your subtitles end exactly when the speech ends, two extra seconds of silent picture is the first thing a viewer notices.

Step 4: Soft subtitles in MP4 need mov_text

The tempting command copies everything:

ffmpeg -i lesson.mp4 -i lesson.en.srt -map 0 -map 1 -c copy out.mp4

It fails, with exit code 234:

Could not find tag for codec subrip in stream #2, codec not currently supported in container
Could not write header (incorrect codec parameters ?): Invalid argument

MP4 can't store SubRip. Copy the audio and video, and convert only the subtitle stream to mov_text. Set a language tag on each track and mark the default one:

cmd += ["-c:v", "copy", "-c:a", "copy", "-c:s", "mov_text"]
for i, (_, lang, title) in enumerate(tracks):
    cmd += [f"-metadata:s:s:{i}", f"language={lang}", f"-metadata:s:s:{i}", f"title={title}"]
cmd += ["-disposition:s:0", "default", str(out)]

With the French museum take and an English translation timed to the same cues, ffprobe shows exactly what a player will offer:

stream|index=2|codec_name=mov_text|disposition:default=1|tag:language=fre
stream|index=3|codec_name=mov_text|disposition:default=0|tag:language=eng

(If you need SRT to survive untouched, use MKV instead: -c:s copy works there.)

Step 5: Burned-in subtitles need libass

Homebrew's default ffmpeg 9.0.1 bottle on this machine was built without libass, so it has no subtitles filter. The error message depends on how you write the filter, and one version is misleading:

$ ffmpeg -i lesson.mp4 -vf "subtitles=lesson.en.srt" ...
No option name near 'lesson.en.srt'
$ ffmpeg -i lesson.mp4 -vf "subtitles=filename=lesson.en.srt" ...
No such filter: 'subtitles'

The first looks like a quoting problem. It isn't. Check the build before you debug the command:

def has_filter(name, ffmpeg=FFMPEG):
    listing = run([ffmpeg, "-hide_banner", "-filters"]).stdout
    return re.search(rf"^\s*\S+\s+{re.escape(name)}\s", listing, re.M) is not None

The ffmpeg-full formula includes libass. The burn step refuses to run without it, and escapes the path so colons and quotes in file names don't break the filter graph:

def burn(video, srt, out, ffmpeg=FFMPEG, style="FontName=Arial,FontSize=26,Outline=2,MarginV=40"):
    if not has_filter("subtitles", ffmpeg):
        raise RuntimeError(f"{ffmpeg} has no 'subtitles' filter (built without libass)")
    path = str(srt).replace("\\", "/").replace(":", "\\:").replace("'", "\\'")
    run([ffmpeg, "-y", "-i", str(video), "-vf", f"subtitles='{path}':force_style='{style}'",
         "-c:v", "libx264", "-crf", "18", "-pix_fmt", "yuv420p", "-c:a", "copy", str(out)])

Step 6: Prove the text is there, and only when it should be

A burned subtitle is just pixels, so test pixels. Decode one frame to grayscale and average the lower band where subtitles sit:

def bottom_band_luma(path, t, band=(0.78, 0.95)):
    raw = subprocess.run([FFMPEG, "-v", "error", "-ss", str(t), "-i", str(path), "-frames:v", "1",
                          "-f", "rawvideo", "-pix_fmt", "gray", "-"], capture_output=True).stdout
    ...  # mean of rows y0..y1

Time	Plain video	Burned
1.5 s (cue 1)	35.0	41.86
3.6 s (pause between cues)	35.0	35.0
7.5 s (cue 2)	35.0	45.44
14.0 s (cue 3)	35.0	42.84

Text appears during each cue and the band is identical to the plain video in the pause. That second check catches cues that overlap or run long.

The tests

Ten pytest checks run against the real TTS audio in about two seconds:

def test_pause_inside_a_sentence_makes_an_extra_segment():
    segs = S.speech_segments(LESSON)
    assert len(segs) == 4
    merged = S.fit_segments(segs, 3)
    assert merged[2][0] == segs[2][0] and merged[2][1] == segs[3][1]

def test_srt_cannot_be_stream_copied_into_mp4(work):
    r = S.run([S.FFMPEG, "-y", "-loglevel", "error", "-i", str(work["base"]), "-i", str(work["srt"]),
               "-map", "0", "-map", "1", "-c", "copy", str(work["d"] / "bad.mp4")])
    assert r.returncode != 0 and "subrip" in r.stderr

def test_burned_text_appears_only_during_cues(work):
    out = S.burn(work["base"], work["srt"], work["d"] / "burned.mp4", ffmpeg=FULL)
    gap = (work["cues"][0][1] + work["cues"][1][0]) / 2
    plain = S.bottom_band_luma(work["base"], 1.5)
    assert S.bottom_band_luma(out, 1.5) > plain + 3
    assert abs(S.bottom_band_luma(out, gap) - plain) < 1
    assert S.subtitle_streams(out) == []

The others cover cue order and overlap, the SRT timestamp format, the -shortest overshoot, the mov_text track and its language tag, two language tracks with the right default, and the clear error when libass is missing.

test_subtitles.py::test_pause_inside_a_sentence_makes_an_extra_segment PASSED
test_subtitles.py::test_too_few_segments_is_an_error PASSED
test_subtitles.py::test_cues_are_ordered_and_inside_the_audio PASSED
test_subtitles.py::test_srt_uses_comma_milliseconds PASSED
test_subtitles.py::test_shortest_can_overshoot_but_exact_length_matches PASSED
test_subtitles.py::test_srt_cannot_be_stream_copied_into_mp4 PASSED
test_subtitles.py::test_soft_track_is_mov_text_with_language PASSED
test_subtitles.py::test_two_language_tracks_first_is_default PASSED
test_subtitles.py::test_burn_needs_libass PASSED
test_subtitles.py::test_burned_text_appears_only_during_cues PASSED
10 passed in 2.14s

Limits

• Timing from silence only works when you already have the exact words and the speaker pauses between lines. For recorded speech with no script you need speech recognition first.
• fit_segments merges the shortest pauses. If the speaker pauses longer inside a sentence than between two sentences, it will merge the wrong pair; the cue-order test won't catch that, so check one or two cues by ear.
• The -35 dB / 0.35 s thresholds suit clean TTS. Noisy recordings need different values.
• Long lines (like step two) wrap to three lines when burned in. Split them in the script if that matters.

Where the script comes from

This approach works best when you write the narration first and generate the voice from it, because then the subtitle text is exact by construction. The lesson audio here was made that way: on Voor's Gemini TTS generator, the "Step-by-step lesson" script shape with voice Kore, quoted at 4 credits for this script (credits count the spoken text; style instructions are free). Generate the take, keep the script, and the code above does the rest.