Add Sound Effects to Video With Python and ffmpeg: Measuring the Alignment Error

AI video generators are very good at motion and often silent. If you generate a clip of a ball bouncing, a door closing, or a glass being set on a table, you usually get pixels and no audio. Adding the right "thud" by hand in an editor is fine for one clip. It doesn't scale to a batch.

This post builds a small Python + ffmpeg pipeline that does three things:

1. Finds the frames where an impact happens.
2. Mixes a short sound effect at exactly those times.
3. Measures how far the audio actually lands from the true impact time.

That third step is the one most tutorials skip. "It sounds about right" isn't a number. I wanted one.

All code below ran on a Linux box with Python 3.13, NumPy and ffmpeg 7.1. Everything is in three files you can copy.

The test clip is synthetic, on purpose

To measure alignment error you need ground truth: the exact moment each impact really happens. Real AI video doesn't give you that. So the test clip is synthetic: a white ball bouncing on a grey floor, rendered frame by frame with NumPy, piped into ffmpeg.

The bounce period is 1.27 s. At 30 fps that's 38.1 frames, so impacts deliberately fall between frames. A period like 1.3 s (exactly 39 frames) would land every impact on a frame and make any detector look perfect. I tried that first and got 0.0 ms error everywhere, which tells you nothing.

"""Render a synthetic silent clip: a ball bouncing on a floor. Ground-truth impact times are known exactly."""
import json, subprocess, numpy as np
W, H, FPS, DUR = 640, 360, 30, 8.0
FLOOR, R, PERIOD = 300, 18, 1.27         # one bounce every 1.27 s (not frame-aligned on purpose)
N = int(FPS * DUR)
def y_at(t):
    ph = (t % PERIOD) / PERIOD             # 0 -> 1 within a bounce
    return FLOOR - R - 220 * 4 * ph * (1 - ph)   # parabola, touches floor at ph=0
impacts = [round(k * PERIOD, 4) for k in range(1, int(DUR / PERIOD) + 1) if k * PERIOD < DUR]
ff = subprocess.Popen(["ffmpeg", "-y", "-loglevel", "error", "-f", "rawvideo", "-pix_fmt", "gray",
    "-s", f"{W}x{H}", "-r", str(FPS), "-i", "-", "-c:v", "libx264", "-pix_fmt", "yuv420p", "silent.mp4"],
    stdin=subprocess.PIPE)
yy, xx = np.mgrid[0:H, 0:W]
for i in range(N):
    t = i / FPS
    f = np.full((H, W), 30, np.uint8); f[FLOOR:, :] = 90
    cx, cy = 80 + i * 2, y_at(t)
    f[(xx - cx) ** 2 + (yy - cy) ** 2 <= R * R] = 230
    ff.stdin.write(f.tobytes())
ff.stdin.close(); ff.wait()
json.dump({"fps": FPS, "impacts_s": impacts}, open("ground_truth.json", "w"), indent=1)
print("impacts:", impacts)

Output:

impacts: [1.27, 2.54, 3.81, 5.08, 6.35, 7.62]

Step 1: detect impacts from motion

For this clip, detection is simple: threshold the bright ball, take its vertical centroid per frame, and call an impact wherever downward motion flips to upward. ffmpeg decodes to raw grayscale and Python reads it from a pipe, so there are no temporary image files.

The key detail is sub-frame refinement. The detected frame is only accurate to ±half a frame (±16.7 ms at 30 fps). Fitting a parabola through the three samples around the lowest point gives a fractional frame index. It's a cheap trick, and the measurements below show whether it pays off.

Step 2: mix the effect with adelay + amix

The sound effect here is also synthetic: a 120 ms decaying 70 Hz sine with a short click on top, generated with NumPy and written as a WAV. In a real pipeline you'd swap in a recorded or generated effect.

Each impact gets its own adelay branch (delay in milliseconds), then amix sums them. normalize=0 matters: without it, amix scales each input down by the number of inputs, so six thuds come out much quieter than one. The ffmpeg filters documentation covers both filters.

"""Detect impact frames in a silent video and mix a short 'thud' at each one with ffmpeg."""
import json, subprocess, sys, wave, numpy as np
SRC, OUT = sys.argv[1], sys.argv[2]
W, H = 640, 360
def probe_fps(path):
    r = subprocess.run(["ffprobe", "-v", "error", "-select_streams", "v:0", "-show_entries",
        "stream=r_frame_rate", "-of", "csv=p=0", path], capture_output=True, text=True).stdout.strip()
    a, b = r.split("/"); return int(a) / int(b)
def frames(path):
    p = subprocess.Popen(["ffmpeg", "-loglevel", "error", "-i", path, "-f", "rawvideo",
        "-pix_fmt", "gray", "-"], stdout=subprocess.PIPE)
    while (buf := p.stdout.read(W * H)) and len(buf) == W * H:
        yield np.frombuffer(buf, np.uint8).reshape(H, W)
fps = probe_fps(SRC)
# 1) track the brightest blob's vertical centroid per frame
ys = []
for f in frames(SRC):
    m = f > 180
    ys.append(np.nonzero(m)[0].mean() if m.any() else np.nan)
ys = np.array(ys)
# 2) an impact = local maximum of y (lowest on screen) where motion reverses from down to up
v = np.diff(ys)
hits = [i for i in range(1, len(v)) if v[i - 1] > 0 and v[i] <= 0]
# refine to sub-frame with a parabola through the 3 samples around the peak
def refine(i):
    a, b, c = ys[i - 1], ys[i], ys[i + 1]
    d = a - 2 * b + c
    return i + (0.5 * (a - c) / d if d else 0.0)
REFINE = "--no-refine" not in sys.argv
times = [(refine(i) if REFINE else i) / fps for i in hits if 0 < i < len(ys) - 1]
# 3) synthesize a 120 ms thud (decaying 70 Hz sine + click) as the effect
SR = 48000
t = np.arange(int(0.12 * SR)) / SR
thud = (np.sin(2 * np.pi * 70 * t) * np.exp(-t * 35) + 0.3 * np.exp(-t * 900)) * 0.8
with wave.open("thud.wav", "wb") as w:
    w.setnchannels(1); w.setsampwidth(2); w.setframerate(SR)
    w.writeframes((thud * 32767).astype(np.int16).tobytes())
# 4) one adelay branch per hit, then amix, then mux onto the original video stream
dur = len(ys) / fps
inputs = ["-i", SRC] + sum([["-i", "thud.wav"] for _ in times], [])
graph = ";".join(f"[{k+1}:a]adelay={int(round(tt*1000))}:all=1[a{k}]" for k, tt in enumerate(times))
graph += ";" + "".join(f"[a{k}]" for k in range(len(times))) + \
    f"amix=inputs={len(times)}:normalize=0,apad,atrim=0:{dur:.3f}[mix]"
subprocess.run(["ffmpeg", "-y", "-loglevel", "error", *inputs, "-filter_complex", graph,
    "-map", "0:v", "-map", "[mix]", "-c:v", "copy", "-c:a", "aac", "-b:a", "128k", OUT], check=True)
json.dump({"fps": fps, "detected_s": [round(x, 4) for x in times]}, open("detected.json", "w"), indent=1)
print(f"fps={fps} hits={len(times)}", [round(x, 3) for x in times])

Note -c:v copy: the video stream isn't re-encoded, so frames stay identical to the input.

Step 3: measure where the audio really lands

Here's the part I cared about. measure.py decodes the final MP4's audio track (after AAC encoding, not the intermediate WAV), finds each thud's onset with a simple amplitude threshold, and compares it with the ground truth.

"""Decode the output audio and measure where each thud actually starts vs ground truth."""
import json, subprocess, sys, numpy as np
OUT = sys.argv[1]; SR = 48000
raw = subprocess.run(["ffmpeg", "-loglevel", "error", "-i", OUT, "-f", "f32le", "-ac", "1",
    "-ar", str(SR), "-"], capture_output=True).stdout
x = np.abs(np.frombuffer(raw, np.float32))
thr = 0.1 * x.max(); onsets = []; i = 0
while i < len(x):
    if x[i] > thr: onsets.append(i / SR); i += int(0.3 * SR)
    else: i += 1
gt = json.load(open("ground_truth.json"))["impacts_s"]
det = json.load(open("detected.json"))["detected_s"]
rows = []
for g in gt:
    o = min(onsets, key=lambda t: abs(t - g)) if onsets else float("nan")
    d = min(det, key=lambda t: abs(t - g)) if det else float("nan")
    rows.append((g, d, o, (d - g) * 1000, (o - g) * 1000))
print("truth_s  detected_s  audio_onset_s  detect_err_ms  audio_err_ms")
for r in rows: print(f"{r[0]:7.3f}  {r[1]:10.3f}  {r[2]:13.3f}  {r[3]:13.1f}  {r[4]:12.1f}")
e = np.array([r[4] for r in rows])
print(f"onsets found={len(onsets)} truth={len(gt)} | audio err mean={e.mean():.1f} ms, "
      f"max |err|={np.abs(e).max():.1f} ms, frame={1000/json.load(open('ground_truth.json'))['fps']:.1f} ms")
json.dump({"rows": rows, "onsets": onsets}, open("measure.json", "w"))

Results

I ran the pipeline twice on the same clip: once snapping to the nearest frame (--no-refine), once with sub-frame refinement. These are the actual outputs:

== mode --no-refine
truth_s  detected_s  audio_onset_s  detect_err_ms  audio_err_ms
  1.270       1.267          1.267           -3.3          -3.0
  2.540       2.533          2.533           -6.7          -7.0
  3.810       3.800          3.800          -10.0         -10.0
  5.080       5.067          5.067          -13.3         -13.0
  6.350       6.333          6.333          -16.7         -17.0
  7.620       7.633          7.633           13.3          13.0
onsets found=6 truth=6 | audio err mean=-6.2 ms, max |err|=17.0 ms, frame=33.3 ms
== mode refine
truth_s  detected_s  audio_onset_s  detect_err_ms  audio_err_ms
  1.270       1.268          1.268           -1.5          -2.0
  2.540       2.537          2.537           -2.6          -3.0
  3.810       3.807          3.807           -2.9          -3.0
  5.080       5.078          5.078           -2.2          -2.0
  6.350       6.350          6.350            0.0           0.0
  7.620       7.622          7.622            2.2           2.0
onsets found=6 truth=6 | audio err mean=-1.3 ms, max |err|=3.0 ms, frame=33.3 ms

What the numbers say:

• Nearest-frame snapping: mean error −6.2 ms, worst case 17.0 ms. The error drifts because the 1.27 s period slides against the 33.3 ms frame grid, then wraps when the nearest frame flips to the other side.
• Sub-frame parabola: mean error −1.3 ms, worst case 3.0 ms. Roughly a 5–6× smaller worst case for a three-line function.
• The audio errors are whole milliseconds because adelay takes integer ms here. Detection error and final audio error match to within 1 ms, so the AAC encode and mux added no measurable offset in this run.

Both modes stay under one frame (33.3 ms) here, but the refined version leaves far more headroom for the extra error a real detector will add.

What this doesn't tell you (caveats)

I want to be clear about the limits, because the clean numbers above come from a clean test:

• The clip is synthetic. One bright object, a static background, no motion blur, no compression artifacts beyond one H.264 encode. Real AI video is messier, and a centroid tracker will fail on it. You'd replace step 1 with optical flow, an object detector, or a model that labels events.
• Six impacts is a small sample. The numbers are exact for this run, not a statistical benchmark.
• The onset detector is naive. A 10% amplitude threshold works for a sharp synthetic thud. Soft sounds (rain, ambience) don't have a clean onset to measure.
• Physical timing isn't perceptual timing. Some sounds feel better slightly early. The measurement tells you where the audio is, not where it should be for a given viewer.

When to build it, when to use a tool

This pipeline is useful when you control the footage, the events are visually obvious, and you want reproducible timing across a batch, such as product demos with taps and clicks, or procedurally generated clips.

When the "event" is semantic, like a pour, a crowd, or a city street, writing the detector is the hard part, and matching the kind of sound to the scene is harder still. That's where a hosted tool that reads the clip and generates context-matched audio makes more sense. If that's your case, iMideo has a page for when you want to add sound effects to video from a text prompt (it currently lists clips up to 8 seconds for audio generation). Disclosure: I work on iMideo.

Either way, measure. Even a rough measurement turns "it sounds about right" into a number you can track when you change the detector.

If you run this on real footage, I'd be curious what your worst-case error looks like. Drop it in the comments.