DEV Community

Mason K
Mason K

Posted on

FFmpeg 9.0 upgrade check: a 60-line harness that diffs your recipes across two builds

#ffmpeg #video #python #devops

📦 Code: github.com/USER/ffmpeg-upgrade-check (replace before publishing)

TL;DR

FFmpeg 9.0 "Lei" (released 2026-08-04) bumped every core library and removed some old support. Exit code 0 does not mean "same output". We'll build a ~60 line Python script that runs your real command lines with the old and new ffmpeg, then diffs ffprobe fields and SSIM. Python 3.10+, any ffmpeg.

🎯 What we're building

A script that takes two ffmpeg binaries, one source clip, and a JSON file of your recipes, and prints OK, DIFF, or FAIL for each one. Think of it as a snapshot test where the snapshot is "whatever the old build did".

What changed in 9.0, per the release coverage I read (news write-ups, not the changelog itself, so skim the official notes for your own recipes):

Area Change
Libraries Every core library got a major bump; libavcodec and libavformat are in the 63 series
Removed Standalone CELT decoding and Ogg/CELT parsing (Opus is not affected)
Removed NVIDIA Video Codec SDK versions before 11.1, plus previously deprecated NVENC options
Added ONNX Runtime DNN backend on GPU, v360_vulkan, transpose_cuda, native animated WebP decode, LCEVC track muxing in MP4

⚠️ Note: I could not find the names of the removed NVENC options in any source I read, so I won't guess them. The harness below is how you find out which of your flags are affected.

1. Pick the recipes that matter 🛠️

Not every ffmpeg command in your repo. The five to fifteen that produce things customers see. Put them in a JSON file as argument lists (no shell strings, so quoting bugs can't bite).

// recipes.json
{
  "ladder_360p": {
    "args": ["-vf","scale=-2:360","-c:v","libx264","-crf","23","-preset","medium",
             "-g","60","-c:a","aac","-b:a","128k"]
  },
  "audio_only_aac": {
    "ext": "m4a",
    "compare_video": false,
    "args": ["-vn","-c:a","aac","-b:a","96k"]
  },
  "faststart_copy": {
    "args": ["-c","copy","-movflags","+faststart"]
  }
}
Enter fullscreen mode Exit fullscreen mode

Optional per-recipe keys: ext (output extension), compare_video (skip SSIM for audio-only), and min_ssim (default 0.99).

2. Write the harness 

#!/usr/bin/env python3
# upgrade_check.py
"""Run the same ffmpeg recipes with two builds and diff the results."""
import json, re, subprocess, sys, tempfile
from pathlib import Path

OLD, NEW = sys.argv[1], sys.argv[2]
SRC = sys.argv[3]
RECIPES = json.loads(Path(sys.argv[4]).read_text())

def run(ffmpeg, args, out):
    cmd = [ffmpeg, "-hide_banner", "-v", "error", "-y", "-i", SRC, *args, out]
    p = subprocess.run(cmd, capture_output=True, text=True)
    return p.returncode, p.stderr.strip()

def probe(path):
    p = subprocess.run(
        ["ffprobe", "-v", "error", "-show_entries",
         "stream=codec_name,profile,pix_fmt,width,height,r_frame_rate,nb_frames,channels,sample_rate",
         "-show_entries", "format=duration", "-of", "json", path],
        capture_output=True, text=True)
    return json.loads(p.stdout)

def ssim(ffmpeg, a, b):
    p = subprocess.run([ffmpeg, "-hide_banner", "-i", a, "-i", b,
                        "-lavfi", "[0:v][1:v]ssim", "-f", "null", "-"],
                       capture_output=True, text=True)
    m = re.search(r"All:([0-9.]+)", p.stderr)
    return float(m.group(1)) if m else None

failures = 0
with tempfile.TemporaryDirectory() as d:
    for name, r in RECIPES.items():
        ext = r.get("ext", "mp4")
        a, b = f"{d}/{name}.old.{ext}", f"{d}/{name}.new.{ext}"
        rc_a, err_a = run(OLD, r["args"], a)
        rc_b, err_b = run(NEW, r["args"], b)
        if rc_a != 0 or rc_b != 0:
            print(f"FAIL  {name}: old rc={rc_a} new rc={rc_b}\n      new stderr: {err_b[:200]}")
            failures += 1
            continue
        pa, pb = probe(a), probe(b)
        diffs = []
        for sa, sb in zip(pa["streams"], pb["streams"]):
            for k in sa:
                if sa.get(k) != sb.get(k):
                    diffs.append(f"{k}: {sa.get(k)} -> {sb.get(k)}")
        if len(pa["streams"]) != len(pb["streams"]):
            diffs.append("stream count differs")
        q = ssim(NEW, a, b) if r.get("compare_video", True) else None
        if q is not None and q < r.get("min_ssim", 0.99):
            diffs.append(f"ssim {q:.4f} < {r.get('min_ssim', 0.99)}")
        if diffs:
            failures += 1
            print(f"DIFF  {name}: " + "; ".join(diffs))
        else:
            print(f"OK    {name} (ssim={q})")
sys.exit(1 if failures else 0)
Enter fullscreen mode Exit fullscreen mode

Three design choices worth knowing:

  • Probe fields, not just success. profile, pix_fmt, nb_frames, and sample_rate are where silent drift tends to show up.
  • SSIM between old and new outputs, not against the source. We aren't grading quality; we're asking "did the upgrade change the picture?"
  • Exit code mirrors failures, so you can drop it straight into CI.

3. Run it 🚀

Grab a short representative clip (a few seconds with motion and audio is plenty) and run:

# ffmpeg-old and ffmpeg-new are the two builds you are comparing
python3 upgrade_check.py /opt/ffmpeg-8.1/bin/ffmpeg /opt/ffmpeg-9.0/bin/ffmpeg sample.mp4 recipes.json
echo "exit=$?"
Enter fullscreen mode Exit fullscreen mode

I only had FFmpeg 6.1.1 in my sandbox, so I could not produce a real 8.1-vs-9.0 diff. To show what the output looks like and to prove the harness works, I ran 6.1.1 against itself with one deliberately broken recipe added:

$ python3 upgrade_check.py ffmpeg ffmpeg in.mp4 recipes.json
OK    ladder_360p (ssim=1.0)
OK    audio_only_aac (ssim=None)
OK    faststart_copy (ssim=1.0)
FAIL  broken_recipe: old rc=8 new rc=8
      new stderr: Unrecognized option 'no_such_option'.
Error splitting the argument list: Option not found
exit=1
Enter fullscreen mode Exit fullscreen mode

That FAIL is what a removed option will look like on 9.0. A DIFF line is the other thing you are hunting for, for example profile: High -> Main or nb_frames: 120 -> 119.

💡 Tip: x264 is deterministic for the same build and input, so a clean self-run gives ssim=1.0. Across versions, a score a hair below 1.0 may be a legitimate encoder improvement. The point is that a human decides, instead of nobody noticing.

4. Triage what comes out

My suggested order for a 9.0 upgrade, driven by what the release removed:

  1. NVENC recipes. Check your driver/SDK image first (SDK older than 11.1 is no longer supported), then run your hardware recipes.
  2. Odd audio. Run a folder of weird uploads through, not just clean fixtures. The CELT removal is narrow, but legacy Ogg files are the sort of thing that lives in a big library.
  3. Stream-copy recipes. Muxers moved. Compare durations, stream counts, and seek behavior.
  4. New features last. Upgrade first, adopt the GPU ONNX backend later, so a regression has one suspect.

5. Wire it into CI and pin your build 🔒 

# .github/workflows/ffmpeg-upgrade.yml (sketch)
name: ffmpeg-upgrade-check
on: [pull_request]
jobs:
  diff-builds:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ./scripts/fetch_ffmpeg.sh old new   # your script; downloads two pinned builds
      - run: python3 upgrade_check.py ./bin/old/ffmpeg ./bin/new/ffmpeg fixtures/sample.mp4 recipes.json
Enter fullscreen mode Exit fullscreen mode

Also record the exact build string (ffmpeg -version | head -1) in your output metadata. When support asks "when did this change?", you can answer in one query.

What's next

  • Read the official 9.0 release notes against your own flags; this post leans on news coverage.
  • Add a min_ssim per recipe if you deliberately want to accept small encoder changes.
  • Pair this with a VMAF check if you already track perceptual quality. Our harness only asks whether the output moved, not whether it got better.

If you run it on a real 8.x to 9.0 diff, I'd love to hear what showed up. #ffmpeg #video #devops

Top comments (0)