DEV Community: Mason K

Two-pass loudness normalization with FFmpeg loudnorm (the right way)

Mason K — Wed, 03 Jun 2026 08:36:36 +0000

TL;DR

We normalize audio to a consistent perceived loudness using FFmpeg's loudnorm filter in two passes: pass 1 measures, pass 2 applies a linear gain to the target. We'll parse the JSON, script it, batch it with ffmpeg-normalize, and verify with ebur128. Single-pass loudnorm pumps; don't use it for VOD.

If your library has one clip that whispers and the next one that blasts, your viewers are riding the volume knob. Peak normalization won't fix it (same peak, different perceived loudness). What you want is EBU R128 loudness normalization, and FFmpeg ships the filter to do it. Let's wire it up properly.

1. Why not single-pass?

The tempting one-liner is single-pass:

# DO NOT do this for VOD: it applies dynamic compression and pumps
ffmpeg -i in.mp4 -af loudnorm=I=-16:TP=-1.5:LRA=11 out.mp4

In one pass, loudnorm doesn't know what's coming, so it makes loudness decisions on the fly with dynamic processing. On music and dialogue that breathes audibly. Single-pass is for live; for files you process ahead of time, use two passes and a linear gain.

2. Pass 1: measure

Run loudnorm purely to analyze. Output goes to null; we only want the JSON it prints to stderr.

# bash: measure.sh
ffmpeg -hide_banner -i input.mp4 \
  -af loudnorm=I=-16:TP=-1.5:LRA=11:print_format=json \
  -f null - 2> measure.log

The tail of measure.log is a JSON block:

{
  "input_i" : "-27.61",
  "input_tp" : "-9.05",
  "input_lra" : "8.40",
  "input_thresh" : "-38.10",
  "output_i" : "-16.00",
  "output_tp" : "-1.50",
  "output_lra" : "11.00",
  "normalization_type" : "dynamic",
  "target_offset" : "0.49"
}

Those input_* values are the measured loudness of your file. We feed them back in pass 2.

💡 Target choice: -16 LUFS is a sane general default. Broadcast wants -23 (EBU R128) with TP=-1. Big streaming platforms normalize playback to roughly -14. Pick per context and keep it consistent across the library.

3. Pass 2: apply linear gain

Pass the measured values back and set linear=true. That applies one consistent gain across the whole file instead of moment-to-moment compression, so dynamics survive.

# bash: apply.sh
ffmpeg -hide_banner -i input.mp4 \
  -af loudnorm=I=-16:TP=-1.5:LRA=11:measured_I=-27.61:measured_TP=-9.05:measured_LRA=8.40:measured_thresh=-38.10:offset=0.49:linear=true \
  -c:v copy -c:a aac -b:a 192k -ar 48000 \
  output.mp4

⚠️ loudnorm resamples internally to 192 kHz. If you don't set -ar 48000 you can get a 192 kHz output you didn't want. Always pin the output sample rate.

Note -c:v copy: we're not touching the video, just re-encoding the audio track.

4. Script the whole thing

Hardcoding measured values is fine for one file, painful for a thousand. Here's a small wrapper that runs pass 1, parses the JSON, and runs pass 2.

# normalize.py: python 3.10+, ffmpeg 5.x+ on PATH
import json, re, subprocess, sys
from pathlib import Path

TARGET = dict(I="-16", TP="-1.5", LRA="11")

def measure(src: Path) -> dict:
    cmd = ["ffmpeg", "-hide_banner", "-i", str(src),
           "-af", f"loudnorm=I={TARGET['I']}:TP={TARGET['TP']}:LRA={TARGET['LRA']}:print_format=json",
           "-f", "null", "-"]
    out = subprocess.run(cmd, capture_output=True, text=True).stderr
    # the JSON block is the last {...} in stderr
    blob = re.search(r"\{[^{}]+\}\s*$", out.strip())
    if not blob:
        raise RuntimeError(f"no loudnorm JSON for {src}")
    return json.loads(blob.group(0))

def apply(src: Path, dst: Path, m: dict) -> None:
    af = (f"loudnorm=I={TARGET['I']}:TP={TARGET['TP']}:LRA={TARGET['LRA']}"
          f":measured_I={m['input_i']}:measured_TP={m['input_tp']}"
          f":measured_LRA={m['input_lra']}:measured_thresh={m['input_thresh']}"
          f":offset={m['target_offset']}:linear=true")
    cmd = ["ffmpeg", "-hide_banner", "-y", "-i", str(src),
           "-af", af, "-c:v", "copy", "-c:a", "aac", "-b:a", "192k",
           "-ar", "48000", str(dst)]
    subprocess.run(cmd, check=True)

if __name__ == "__main__":
    src = Path(sys.argv[1])
    dst = src.with_name(src.stem + "_norm.mp4")
    m = measure(src)
    print(f"measured integrated loudness: {m['input_i']} LUFS -> target {TARGET['I']}")
    apply(src, dst, m)
    print(f"wrote {dst}")

$ python normalize.py whisper_clip.mp4
measured integrated loudness: -27.61 LUFS -> target -16
wrote whisper_clip_norm.mp4

5. Or just use ffmpeg-normalize

For batch jobs, ffmpeg-normalize wraps all of this with two-pass by default:

pip install ffmpeg-normalize
ffmpeg-normalize input.mp4 -nt ebu -t -16 -c:a aac -b:a 192k -ar 48000 -o output.mp4
# batch a folder
ffmpeg-normalize *.mp4 -nt ebu -t -16 -ext mp4 -o normalized/

-nt ebu selects EBU R128 (two-pass), -t -16 is the target. Same result, less plumbing.

6. Verify it worked

Don't trust, measure. The ebur128 filter prints the integrated loudness of the output:

ffmpeg -hide_banner -i output.mp4 -af ebur128 -f null - 2>&1 | tail -n 6

[Parsed_ebur128_0 @ ...] Summary:
  Integrated loudness:
    I:         -16.0 LUFS
    Threshold: -26.3 LUFS
  True peak:
    Peak:       -1.5 dBFS

Integrated loudness at your target, true peak under the ceiling. Run it across a handful of clips and they'll all land at the same level.

⚠️ Edge case: very short or near-silent inputs report integrated loudness near -70 LUFS (the gate floor). Don't blindly amplify those to target, you'll just boost hiss. Skip or flag them.

7. Normalize once, mux into every rendition

If you also build an ABR ladder, you do not want to run loudnorm separately for each rung. Measure and normalize the audio once, then mux that single normalized track into every video rendition. The audio is identical across the ladder and it only got processed one time.

# bash: extract + normalize audio once
ffmpeg -hide_banner -i master.mov -vn -c:a pcm_s16le audio_raw.wav
# (run measure.sh + apply.sh on audio_raw.wav -> audio_norm.m4a, -ar 48000)

# mux the one normalized track into each silent video rendition
for r in 720 480 360; do
  ffmpeg -hide_banner -y \
    -i "video_${r}.mp4" -i audio_norm.m4a \
    -map 0:v:0 -map 1:a:0 -c:v copy -c:a copy -shortest \
    "rendition_${r}.mp4"
done

$ ffprobe -hide_banner rendition_720.mp4 2>&1 | grep -E 'Stream|LUFS'
  Stream #0:0: Video: h264 ... 1280x720
  Stream #0:1: Audio: aac ... 48000 Hz, stereo

Every rung now carries the same -16 LUFS audio. Switching renditions mid-playback won't change the perceived volume, which is the other half of a smooth listening experience.

What's next

Wire normalize.py into your upload pipeline as a worker step after transcode.
Pick the right target per content type: dialogue/podcast near -16, broadcast -23, loud entertainment can sit higher.
If you also build a video ladder, run loudness on the audio track once and mux it into every rendition so they all match.

The filter has been stable in FFmpeg for years (any 5.x+ has it; the 8.0 line is current), so this isn't bleeding-edge. The win is treating loudness as a fixed pipeline setting you decide once and enforce, exactly like a resolution ladder.

Package one CMAF asset for Widevine, PlayReady, and FairPlay (no second encode)

Mason K — Wed, 03 Jun 2026 08:35:16 +0000

TL;DR

We package one CMAF asset encrypted with cbcs, emit an HLS and a DASH manifest from the same segments, and play it back with Shaka Player so Widevine, PlayReady, and FairPlay all decrypt the same bytes. No more separate cenc and cbcs encodes for the common case.

If you last set up DRM a few years ago, you probably remember packaging everything twice: a cenc copy for Widevine/PlayReady and a cbcs copy for FairPlay, because the cipher modes were incompatible. For current devices that is no longer necessary. Modern Widevine and PlayReady both decrypt cbcs, and CMAF lets HLS and DASH share one set of segments. Let's build the single-encode version.

We'll do it in four steps:

Understand the cipher-mode constraint (30 seconds of theory).
Package a CMAF asset with cbcs using Shaka Packager.
Wire up Shaka Player with EME and three license servers.
Handle the FairPlay-specific bits that always bite first.

1. The one constraint that drives everything

There are three DRM systems and they map to devices, not to your preferences:

DRM	Owner	Covers
Widevine	Google	Android, Chrome, Chromecast, many smart TVs
FairPlay	Apple	Safari (iOS/iPadOS/macOS), tvOS
PlayReady	Microsoft	Windows/Edge, Xbox, many connected TVs

Historically Widevine and PlayReady used AES-128-CTR (the cenc scheme) and FairPlay used AES-128-CBC (the cbcs scheme). A cenc file literally cannot be decrypted by FairPlay; different cipher mode.

💡 The 2026 shortcut: encrypt once with cbcs. Current Widevine and PlayReady clients accept it, FairPlay needs it. One encryption, three DRMs.

⚠️ Old devices caveat: some legacy Widevine/PlayReady clients (old CTVs/set-top boxes) only do cenc. Check your playback analytics before going cbcs-only. Most apps shipping today can.

2. Package CMAF with cbcs using Shaka Packager

Shaka Packager is Google's open-source packager and speaks CMAF + cbcs natively. Grab a static build (packager-linux-x64) or run the Docker image.

First, encode your renditions as plain MP4 (this is your normal ABR ladder, no DRM yet):

# bash: a tiny 3-rung ladder for the demo
ffmpeg -i master.mov -c:v libx264 -preset medium -crf 23 -vf scale=1280:720 -an video_720.mp4
ffmpeg -i master.mov -c:v libx264 -preset medium -crf 26 -vf scale=854:480  -an video_480.mp4
ffmpeg -i master.mov -c:v aac -b:a 128k -vn audio.mp4

Now package to CMAF with cbcs encryption. For a real deployment you'd pull keys from a key server over SPEKE; for a local demo you can pass a raw key and key id (KID):

# bash: package.sh
KEY_ID=$(openssl rand -hex 16)
KEY=$(openssl rand -hex 16)

packager \
  'in=video_720.mp4,stream=video,init_segment=out/720/init.mp4,segment_template=out/720/$Number$.m4s' \
  'in=video_480.mp4,stream=video,init_segment=out/480/init.mp4,segment_template=out/480/$Number$.m4s' \
  'in=audio.mp4,stream=audio,init_segment=out/audio/init.mp4,segment_template=out/audio/$Number$.m4s' \
  --protection_scheme cbcs \
  --enable_raw_key_encryption \
  --keys "label=:key_id=${KEY_ID}:key=${KEY}" \
  --generate_static_live_mpd \
  --mpd_output out/manifest.mpd \
  --hls_master_playlist_output out/master.m3u8

The important flags: --protection_scheme cbcs is what makes this asset playable by all three DRMs, and emitting both --mpd_output (DASH) and --hls_master_playlist_output (HLS) from the same run means both manifests point at the same .m4s segments. One set of bytes on disk.

# bash: what you get
$ tree out
out
├── 720/   init.mp4  1.m4s  2.m4s ...
├── 480/   init.mp4  1.m4s  2.m4s ...
├── audio/ init.mp4  1.m4s  2.m4s ...
├── manifest.mpd        # DASH
└── master.m3u8         # HLS, same segments

💡 In production, replace --enable_raw_key_encryption/--keys with --enable_widevine_encryption or a SPEKE key-server URL so a real multi-DRM service mints and stores your keys. Never ship raw keys in your packaging script.

3. Play it back with Shaka Player + EME

Browser DRM runs through Encrypted Media Extensions (EME). Shaka Player (4.x) abstracts EME and picks the right DRM for the current browser. You give it license-server URLs per system; it negotiates the rest.

<!-- index.html -->
<script src="https://cdn.jsdelivr.net/npm/shaka-player@4/dist/shaka-player.compiled.js"></script>
<video id="video" controls autoplay style="width:100%"></video>

// app.js
async function main() {
  shaka.polyfill.installAll();
  const video = document.getElementById('video');
  const player = new shaka.Player(video);

  player.configure({
    drm: {
      servers: {
        'com.widevine.alpha':        'https://YOUR-LICENSE-SERVICE/widevine',
        'com.microsoft.playready':   'https://YOUR-LICENSE-SERVICE/playready',
        'com.apple.fps':             'https://YOUR-LICENSE-SERVICE/fairplay',
      },
      advanced: {
        'com.apple.fps': {
          // FairPlay needs the Apple-issued application certificate up front
          serverCertificateUri: 'https://YOUR-CDN/fairplay.cer',
        },
      },
    },
    // Let Shaka use the same EME path for FairPlay instead of native HLS
    streaming: { useNativeHlsForFairPlay: false },
  });

  player.addEventListener('error', (e) => console.error('shaka error', e.detail));

  // DASH on most browsers; Shaka also reads the HLS manifest if you prefer
  await player.load('/out/manifest.mpd');
  console.log('playing, DRM negotiated:', player.drmInfo()?.keySystem);
}
main();

Shaka inspects the browser: Chrome/Android negotiate com.widevine.alpha, Edge/Windows can use com.microsoft.playready, Safari uses com.apple.fps. Same manifest, same segments, three key systems.

4. The FairPlay bits that always bite

FairPlay is the one that costs you a day, and none of it is the cryptography. Three things to know before you start:

1. You need an Apple-issued application certificate (.cer).
   Enroll in the Apple Developer Program → request the
   FairPlay Streaming deployment package. This is gated and slow.

2. The license exchange is SPC → CKC:
   player generates an SPC (Server Playback Context) →
   your key server returns a CKC (Content Key Context).
   A multi-DRM license service implements the KSM for you.

3. Native Safari can also play FairPlay HLS with NO JS player:
   <video src="master.m3u8"> works if you wire the
   `webkitneedkey` / `encrypted` EME handlers. Shaka does this for you.

If you skip the certificate step and only test in Chrome, everything looks perfect, and then iOS playback fails at launch. Test on a real Apple device early.

Common errors and fixes

NO_LICENSE_SERVER_SPECIFIED (shaka 6001)
  → you encrypted with a key system you didn't add to drm.servers

LICENSE_REQUEST_FAILED (shaka 6007)
  → license server URL wrong, or CORS not allowed on the license endpoint

FAIRPLAY: keySystemAccess denied / no recognized key system
  → missing serverCertificateUri, or testing FairPlay outside Safari

Playback works in Chrome, black screen on old smart TV
  → that device only supports cenc; ship a cenc set for the long tail

What's next

You now have one cbcs CMAF asset serving Widevine, PlayReady, and FairPlay from a single encode. From here:

Swap the raw-key demo for a real multi-DRM license service (SPEKE key provisioning) so keys are minted and rotated for you.
Add hardware-security-level requirements (Widevine L1, HDCP) if you license premium content; studios specify these in contracts.
Read the Shaka Player DRM config docs for persistent licenses and offline playback.

The same single-encode pattern works whether you package yourself with Shaka Packager or hand it to a managed video API. The cipher-mode war that forced the second encode is over; build like it.

Build a custom HLS player in React with hls.js (no wrapper libraries)

Mason K — Sun, 31 May 2026 09:15:26 +0000

TL;DR

We'll build a custom HLS player on top of hls.js 1.6.x and React 19 with no wrapper library. Play/pause, seek, volume, quality readout, buffer state, and proper error recovery. End result is ~150 lines and bends to whatever UI you want.

📦 Code: github.com/USER/react-hls-player-demo (replace before publishing)

This is the "stop installing video.js for the third time" tutorial. The wrapper libraries are doing two things you need (the MSE shim and the error-recovery loop) and a lot of things you don't (a giant control bar, a plugin model, a CSS skin). The MSE shim is hls.js. The error-recovery loop is fifteen lines. Everything else can be your own React.

0. Versions

node 22.x
react 19.x
hls.js 1.6.16   # latest stable as of 2026-04-13

The 1.7 alpha is interesting (interstitial-ads improvements landed in canary on 2026-05-16) but not yet ready for production. Stick with 1.6 unless you're testing the next major.

1. Setup

npm create vite@latest react-hls-player-demo -- --template react-ts
cd react-hls-player-demo
npm install
npm install hls.js

For test streams, the Apple HLS test page has a few public manifests. The bipbop ad-stitched stream is the classic one:

https://devstreaming-cdn.apple.com/videos/streaming/examples/img_bipbop_adv_example_fmp4/master.m3u8

2. The bare minimum: a hook that mounts hls.js

// src/useHls.ts
import { useEffect, useRef } from "react";
import Hls from "hls.js";

export function useHls(src: string) {
  const videoRef = useRef<HTMLVideoElement>(null);

  useEffect(() => {
    const video = videoRef.current;
    if (!video) return;

    // Safari (and iOS) plays HLS natively. Hand the URL to the browser.
    if (video.canPlayType("application/vnd.apple.mpegurl")) {
      video.src = src;
      return;
    }

    if (!Hls.isSupported()) return;

    const hls = new Hls({
      enableWorker: true,
      lowLatencyMode: true,
      backBufferLength: 30,
    });
    hls.loadSource(src);
    hls.attachMedia(video);

    return () => {
      hls.destroy();
    };
  }, [src]);

  return videoRef;
}

The cleanup matters. React 19's StrictMode mounts every effect twice in development. Without the hls.destroy(), you leak one Hls instance per remount and your dev server's memory usage will climb. Ask me how I know.

// src/App.tsx
import { useHls } from "./useHls";

const SRC = "https://devstreaming-cdn.apple.com/videos/streaming/examples/img_bipbop_adv_example_fmp4/master.m3u8";

export default function App() {
  const videoRef = useHls(SRC);
  return <video ref={videoRef} controls className="w-full max-w-4xl" />;
}

That plays HLS in every modern browser. Eight lines. The wrapper libraries' main value-add, "plays HLS in Chrome", is now done. Everything else is product.

3. Wiring real controls

Drop controls from <video> and write your own. Each control is a small piece of state synced to the element.

// src/Player.tsx
import { useEffect, useRef, useState } from "react";
import Hls from "hls.js";

export function Player({ src }: { src: string }) {
  const videoRef = useRef<HTMLVideoElement>(null);
  const hlsRef = useRef<Hls | null>(null);
  const [playing, setPlaying] = useState(false);
  const [time, setTime] = useState(0);
  const [duration, setDuration] = useState(0);
  const [level, setLevel] = useState<string>("auto");

  useEffect(() => {
    const v = videoRef.current!;
    if (v.canPlayType("application/vnd.apple.mpegurl")) {
      v.src = src;
    } else if (Hls.isSupported()) {
      const hls = new Hls({ enableWorker: true, lowLatencyMode: true });
      hls.loadSource(src);
      hls.attachMedia(v);
      hls.on(Hls.Events.LEVEL_SWITCHED, (_, d) => {
        const lvl = hls.levels[d.level];
        setLevel(lvl ? `${lvl.height}p` : "auto");
      });
      hlsRef.current = hls;
    }
    return () => { hlsRef.current?.destroy(); hlsRef.current = null; };
  }, [src]);

  return (
    <div className="player">
      <video
        ref={videoRef}
        onPlay={() => setPlaying(true)}
        onPause={() => setPlaying(false)}
        onTimeUpdate={(e) => setTime(e.currentTarget.currentTime)}
        onLoadedMetadata={(e) => setDuration(e.currentTarget.duration)}
      />
      <div className="controls">
        <button onClick={() => {
          const v = videoRef.current!;
          playing ? v.pause() : v.play();
        }}>{playing ? "Pause" : "Play"}</button>

        <input
          type="range"
          min={0}
          max={duration || 0}
          step={0.1}
          value={time}
          onChange={(e) => {
            const v = videoRef.current!;
            v.currentTime = Number(e.target.value);
          }}
        />

        <span>{fmt(time)} / {fmt(duration)}</span>
        <span className="quality">{level}</span>
      </div>
    </div>
  );
}

function fmt(s: number): string {
  if (!isFinite(s)) return "0:00";
  const m = Math.floor(s / 60);
  const ss = Math.floor(s % 60).toString().padStart(2, "0");
  return `${m}:${ss}`;
}

The quality readout is the part wrappers tend to hide. LEVEL_SWITCHED fires every time hls.js picks a new ABR rendition, and hls.levels[level] has the bitrate, resolution, and codec strings. Surfacing "now playing 720p" in the corner of your player is a five-line feature that meaningfully changes how power users perceive your product.

4. Manual quality picker

Some users will want to force a quality, usually because they don't trust ABR or because they're on a metered connection. Wire it like this:

const [levels, setLevels] = useState<Hls["levels"]>([]);

useEffect(() => {
  const hls = hlsRef.current;
  if (!hls) return;
  hls.on(Hls.Events.MANIFEST_PARSED, () => setLevels(hls.levels));
}, []);

// in JSX:
<select
  value={hls.currentLevel}
  onChange={(e) => { hls.currentLevel = Number(e.target.value); }}
>
  <option value={-1}>Auto</option>
  {levels.map((l, i) => (
    <option key={i} value={i}>{l.height}p ({Math.round(l.bitrate / 1000)} kbps)</option>
  ))}
</select>

-1 means "let ABR pick"; any non-negative index pins the player to that level. After a user pins to 480p, the seek bar still works and the rest of ABR's logic still runs underneath, but the renderer stays at 480p.

5. The error recovery loop (do not skip)

This is the single thing the wrappers were doing that you actually need.

hls.on(Hls.Events.ERROR, (_, data) => {
  if (!data.fatal) return;

  if (data.type === Hls.ErrorTypes.NETWORK_ERROR) {
    console.warn("hls: network error, restarting load");
    hls.startLoad();
    return;
  }
  if (data.type === Hls.ErrorTypes.MEDIA_ERROR) {
    console.warn("hls: media error, recovering");
    hls.recoverMediaError();
    return;
  }
  console.error("hls: fatal", data);
  hls.destroy();
});

Three rules:

Ignore non-fatal errors. hls.js retries those itself.
Network errors get a fresh startLoad(). This is what saves a viewer who walked into a tunnel.
Media errors get recoverMediaError(). This rebuilds the MSE source buffers, which is the only thing that fixes "the decoder gave up after a corrupt segment."

Add a tiny rate limiter in production. If recoverMediaError fails twice within a few seconds, destroy and recreate the Hls instance with a fresh source. Otherwise you can wedge yourself into an infinite recovery loop.

6. Buffer indicator

Showing "this video is buffering" properly means listening to the right events, not just paused.

const [stalled, setStalled] = useState(false);

const v = videoRef.current!;
v.addEventListener("waiting", () => setStalled(true));
v.addEventListener("playing", () => setStalled(false));

For the YouTube-style "this much of the video is buffered ahead" bar, use videoElement.buffered:

const buffered = videoRef.current?.buffered;
let aheadPct = 0;
if (buffered && buffered.length > 0 && duration > 0) {
  aheadPct = (buffered.end(buffered.length - 1) / duration) * 100;
}

Render a second <progress> underneath the seek bar at that percentage. Total time invested: ten minutes.

7. Subtitles

hls.js parses WebVTT tracks from the manifest into audioTracks and subtitleTracks. Toggle them like:

hls.subtitleDisplay = true;
hls.subtitleTrack = 0;        // -1 to disable

For language selection, hls.subtitleTracks is an array with name, lang, and default flags.

8. Cleanup checklist

Things people forget that bite later:

Mistake	Symptom
No `hls.destroy()` in unmount	Memory leak, Chrome devtools shows climbing JS heap
Forgetting Safari native path	App works in Chrome dev, breaks on iPhone in QA
Listening to `pause` instead of `waiting` for buffering	Players show "buffering" when the user manually pauses
Not handling `MEDIA_ERROR`	Random unrecoverable freezes on bad-CDN nights
`Hls.isSupported() === false` ignored	Old browsers show a broken player with no fallback

What's next

There are three places to take this once it's running:

Picture-in-Picture. videoRef.current.requestPictureInPicture(). One line, free feature.
MediaSession metadata. Set navigator.mediaSession.metadata so the OS lock screen shows the right title and artwork.
Player telemetry. Subscribe to Hls.Events.FRAG_LOADED and send the per-segment timings to your analytics. This is gold for debugging "the video stalled, but only for users in Brazil."

If you're playing assets from a managed video API (Mux, FastPix, Cloudflare Stream, api.video), most of them ship their own custom-element player that wires telemetry for you. Use it if you want the analytics for free. Build your own when the player UX is part of the product.

The 1.7 line of hls.js looks promising for interstitial ads and a cleaner LL-HLS toggle. Worth watching the hls.js releases page for the stable 1.7.0 cut later this year.

Happy playing.

Pick a better video thumbnail automatically with FFmpeg, PySceneDetect, and CLIP

Mason K — Sun, 31 May 2026 09:14:33 +0000

TL;DR

We'll build a pipeline that takes any video file, extracts candidate frames with FFmpeg and PySceneDetect, filters out blurry ones with OpenCV, scores each candidate with OpenCLIP against a small prompt set, and picks the top-K thumbnails with a diversity constraint. ~200 lines of Python, GPU-accelerated, fully local.

📦 Code: github.com/USER/auto-thumbnail-picker (replace before publishing)

The default thumbnail your encoder generates is "the middle frame." For most videos, the middle frame is a motion blur, a transition, or someone mid-blink. We can do much better with about an hour of effort. Here's the pipeline.

Versions

python 3.12
ffmpeg 7.1
pyscenedetect 0.6.x
open-clip-torch 2.x
opencv-python 4.x
torch 2.x (with CUDA or MPS)

The pipeline runs on CPU but is noticeably slower for the CLIP step. A consumer GPU (or Apple Silicon MPS) cuts the per-frame encode to a few milliseconds.

What we're building

Extract candidate frames (shot boundaries + uniform sampling).
Filter blurry frames out with Laplacian variance.
Score each remaining frame with OpenCLIP using a positive / negative prompt set.
Apply structural rules (prefer frames with faces).
Pick top-K with shot-level diversity.

1. Setup

python -m venv .venv && source .venv/bin/activate
pip install --break-system-packages \
  open-clip-torch \
  scenedetect[opencv] \
  opencv-python-headless \
  pillow \
  torch torchvision

For face detection we'll keep it simple and use OpenCV's built-in Haar cascades. If you need higher accuracy on small faces, swap to insightface later.

2. Extract candidate frames

Two sources of candidates: shot boundaries (one per shot) and uniform sampling (one every N seconds). Combine both, then dedupe by timestamp.

# extract.py
import subprocess, os, pathlib
from scenedetect import open_video, SceneManager, ContentDetector

def shot_boundaries(video_path: str) -> list[float]:
    """Return scene-cut timestamps (seconds) using PySceneDetect."""
    vid = open_video(video_path)
    sm = SceneManager()
    sm.add_detector(ContentDetector(threshold=27.0))
    sm.detect_scenes(vid, show_progress=False)
    scenes = sm.get_scene_list()
    # Take the start of each scene plus a small offset (avoid the literal cut frame).
    return [s[0].get_seconds() + 0.4 for s in scenes]

def uniform_samples(duration: float, every: float = 3.0) -> list[float]:
    return [t for t in [i * every for i in range(int(duration // every) + 1)] if t > 0]

def extract_frame(video_path: str, t: float, out_path: str) -> None:
    subprocess.run([
        "ffmpeg", "-y", "-ss", f"{t:.3f}", "-i", video_path,
        "-frames:v", "1", "-q:v", "2", out_path
    ], check=True, capture_output=True)

def probe_duration(video_path: str) -> float:
    out = subprocess.run(
        ["ffprobe", "-v", "error", "-show_entries", "format=duration",
         "-of", "default=nokey=1:noprint_wrappers=1", video_path],
        check=True, capture_output=True, text=True,
    ).stdout.strip()
    return float(out)

def build_candidates(video_path: str, out_dir: str) -> list[tuple[float, str]]:
    os.makedirs(out_dir, exist_ok=True)
    dur = probe_duration(video_path)
    times = sorted(set(shot_boundaries(video_path) + uniform_samples(dur)))
    out = []
    for i, t in enumerate(times):
        p = pathlib.Path(out_dir) / f"f_{i:04d}_{t:07.2f}.jpg"
        extract_frame(video_path, t, str(p))
        out.append((t, str(p)))
    return out

💡 Tip: the -ss flag before -i makes ffmpeg seek using the index (fast and inaccurate at frame level). For thumbnail-quality stills we want accurate seeking, but 0.4s after the cut is forgiving enough that index-seek is fine.

Run it:

$ python -c "from extract import build_candidates; print(len(build_candidates('clip.mp4', 'frames/')))"
27

Twenty-seven candidates for a six-minute clip. Manageable.

3. Filter blurry frames

CLIP is too tolerant of blur. We prune with Laplacian variance first.

# sharpness.py
import cv2

def is_sharp(image_path: str, threshold: float = 80.0) -> bool:
    img = cv2.imread(image_path, cv2.IMREAD_GRAYSCALE)
    if img is None:
        return False
    return cv2.Laplacian(img, cv2.CV_64F).var() > threshold

For 1080p source the threshold around 80 works. For 720p drop to ~60. Tune on a few samples from your library; the right cutoff depends on the camera you're shooting with.

4. Score with OpenCLIP

The trick is scoring each frame as positive_similarity - mean(negative_similarity). This is more robust than the absolute positive score because lighting and content normalize out.

# score.py
import torch, open_clip
from PIL import Image

DEVICE = "cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu")

MODEL_NAME = "ViT-L-14"
PRETRAINED = "laion2b_s32b_b82k"

POS = [
    "a sharp, well-lit frame from a video, showing the main subject clearly",
]
NEG = [
    "a blurry frame",
    "a black frame",
    "a transition with text overlay or graphic",
    "a frame caught between two shots",
]

class Scorer:
    def __init__(self) -> None:
        self.model, _, self.preprocess = open_clip.create_model_and_transforms(
            MODEL_NAME, pretrained=PRETRAINED, device=DEVICE
        )
        self.model.eval()
        tokenizer = open_clip.get_tokenizer(MODEL_NAME)
        with torch.no_grad():
            self.pos = self._embed_text(tokenizer(POS))
            self.neg = self._embed_text(tokenizer(NEG))

    def _embed_text(self, tokens):
        with torch.no_grad():
            v = self.model.encode_text(tokens.to(DEVICE))
            return v / v.norm(dim=-1, keepdim=True)

    def score(self, image_path: str) -> float:
        img = self.preprocess(Image.open(image_path).convert("RGB")).unsqueeze(0).to(DEVICE)
        with torch.no_grad():
            i = self.model.encode_image(img)
            i = i / i.norm(dim=-1, keepdim=True)
            pos_sim = (i @ self.pos.T).max().item()
            neg_sim = (i @ self.neg.T).mean().item()
        return pos_sim - neg_sim

⚠️ Note: load the model once and reuse. Reloading on every call is the most common reason people think "CLIP is slow"; it's actually the checkpoint load that's slow.

Quick sanity check:

$ python -c "from score import Scorer; s = Scorer(); print(s.score('frames/f_0010_0030.50.jpg'))"
0.0432

Numbers around 0 to 0.1 are typical. Bigger is better.

5. Face bonus

For UGC-style content the right thumbnail almost always has a clear face. We boost the score when one is present.

# faces.py
import cv2

_cascade = cv2.CascadeClassifier(
    cv2.data.haarcascades + "haarcascade_frontalface_default.xml"
)

def has_visible_face(image_path: str, min_size: int = 80) -> bool:
    img = cv2.imread(image_path, cv2.IMREAD_GRAYSCALE)
    if img is None:
        return False
    faces = _cascade.detectMultiScale(img, scaleFactor=1.1, minNeighbors=5,
                                      minSize=(min_size, min_size))
    return len(faces) > 0

6. Orchestrate

# pick.py
from extract import build_candidates
from sharpness import is_sharp
from faces import has_visible_face
from score import Scorer

def pick_thumbnails(video_path: str, out_dir: str, k: int = 3) -> list[dict]:
    candidates = build_candidates(video_path, out_dir)
    candidates = [(t, p) for t, p in candidates if is_sharp(p)]

    scorer = Scorer()
    rows = []
    for t, p in candidates:
        s = scorer.score(p)
        if has_visible_face(p):
            s += 0.05  # small bias toward faces
        rows.append({"t": t, "path": p, "score": s})
    rows.sort(key=lambda r: r["score"], reverse=True)

    # Diversity: don't pick frames that are within 2 seconds of each other.
    picked: list[dict] = []
    for r in rows:
        if any(abs(r["t"] - p["t"]) < 2.0 for p in picked):
            continue
        picked.append(r)
        if len(picked) >= k:
            break
    return picked

if __name__ == "__main__":
    import sys, json
    print(json.dumps(pick_thumbnails(sys.argv[1], "frames/", k=3), indent=2))

Run:

$ python pick.py clip.mp4
[
  { "t": 84.40, "path": "frames/f_0017_0084.40.jpg", "score": 0.1129 },
  { "t": 23.50, "path": "frames/f_0009_0023.50.jpg", "score": 0.0871 },
  { "t": 165.20, "path": "frames/f_0029_0165.20.jpg", "score": 0.0832 }
]

You now have three diverse, sharp, scored candidate thumbnails. Use the first for the default and keep the other two for A/B testing.

7. (Optional) Crop to aspect ratio

If your product shows thumbnails at 16:9 but your source is 9:16 (vertical UGC), or vice versa, you want a smart crop, not a letterbox.

# crop.py
import cv2

def crop_to_ratio(image_path: str, out_path: str, ratio: tuple[int, int] = (16, 9)) -> None:
    img = cv2.imread(image_path)
    h, w = img.shape[:2]
    target = ratio[0] / ratio[1]
    actual = w / h

    if actual > target:
        # too wide, crop sides
        new_w = int(h * target)
        x0 = (w - new_w) // 2
        cropped = img[:, x0:x0 + new_w]
    else:
        # too tall, crop top/bottom toward the upper third
        new_h = int(w / target)
        y0 = max(0, int(h * 0.20))
        y0 = min(y0, h - new_h)
        cropped = img[y0:y0 + new_h, :]
    cv2.imwrite(out_path, cropped)

The "upper third" bias for vertical content is intentional. Faces in vertical video almost always sit in the upper third; a centered crop loses them. For higher quality, swap this for a face-aware crop using the bounding box from step 5.

8. Where to put this in your pipeline

Stage	When
Inline at upload	Easiest. Adds 5–30s to the "video processing" step.
Background worker (recommended)	Consume an "asset.ready" webhook, run, write back to your DB.
Re-run on edit	Re-compute when the user trims or reorders clips.

If you're using a managed video API, you'll already get an "asset ready" webhook. Hook a background worker to that and write the picked thumbnail back to your CMS row. Both FastPix and Mux let you set a custom thumbnail URL on the asset.

What's next

Three upgrades that are worth their complexity:

Better face detector. Replace Haar with insightface for small-face robustness.
Learned scorer. If you have click-through data, fine-tune a small head on top of the CLIP embeddings. The hand-tuned prompts get you 80% of the way; learning the last 20% is real ROI.
Vision-language scoring. Ask a small VLM ("rate this thumbnail's appeal 1-10") instead of cosine similarity. More expensive, sometimes worth it for high-stakes content.

The thing this pipeline gets right is that the model is the cheap part. The leverage is the candidate selection, the structural filters, and the diversity constraint. Get those right with a vanilla CLIP and you're already beating most platforms' default thumbnails.

Happy thumbnail picking.

Build a video upload + HLS playback flow in Next.js 15 (with direct uploads)

Mason K — Tue, 26 May 2026 10:38:29 +0000

TL;DR

We're going to build a "user uploads a video, user watches the video" feature in a Next.js 15 app. The browser uploads directly to a managed video API so our server never holds bytes, a webhook flips the row from processing to ready, and the watch page plays an HLS manifest. ~120 lines across four files.

📦 Code: github.com/USER/nextjs-video-upload-demo (replace before publishing)

We'll use FastPix as the managed API in this tutorial because the encoding is free on the standard plan and the $25 signup credit covers the bytes you'll burn while iterating. The same shape works against Mux, api.video, or Cloudflare Stream. Only the endpoint names and webhook field names change.

What we're building

POST /api/uploads returns a one-shot direct-upload URL.
Browser PUTs the file to that URL. Our server never sees the bytes.
POST /api/webhooks/fastpix receives an "asset ready" callback and stores the playbackId in our DB.
/watch/[id] renders a player against https://stream.fastpix.io/<playbackId>.m3u8.

💡 Why direct upload? Next.js 15 Server Actions cap request bodies at 1 MB by default, and even Route Handlers with bumped limits will sit there holding a 300 MB file in a serverless function while encoding hasn't started. Direct upload pushes bytes straight to the storage layer.

1. Project setup

npx create-next-app@latest nextjs-video-upload-demo --typescript --app --tailwind
cd nextjs-video-upload-demo
npm install

Make a .env.local:

# .env.local
FASTPIX_TOKEN_ID=pk_...
FASTPIX_SECRET=sk_...
FASTPIX_WEBHOOK_SECRET=whsec_...
NEXT_PUBLIC_APP_URL=http://localhost:3000
DATABASE_URL=file:./dev.db

The token and secret come from dashboard.fastpix.io (Access Tokens). Webhooks live at https://docs.fastpix.io/docs/webhooks-collection; create a webhook pointing at your tunneled URL (more on that below).

Versions you want to be on: Node 22.x or newer, Next.js 15.x, hls.js 1.6.x for the player.

2. The "create upload" Route Handler

// app/api/uploads/route.ts
import { NextResponse } from "next/server";
import { db } from "@/lib/db";

export async function POST() {
  const auth = Buffer.from(
    `${process.env.FASTPIX_TOKEN_ID}:${process.env.FASTPIX_SECRET}`
  ).toString("base64");

  const res = await fetch("https://api.fastpix.io/v1/on-demand", {
    method: "POST",
    headers: {
      Authorization: `Basic ${auth}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      corsOrigin: process.env.NEXT_PUBLIC_APP_URL,
      playbackPolicy: ["public"],
    }),
  });

  if (!res.ok) {
    const txt = await res.text();
    return NextResponse.json({ error: txt }, { status: 502 });
  }

  const data = await res.json();

  const row = await db.video.create({
    data: {
      providerAssetId: data.id,
      status: "processing",
    },
  });

  return NextResponse.json({
    videoId: row.id,
    uploadUrl: data.uploadUrl,
  });
}

cors_origin matters. The browser is about to PUT bytes from your origin to FastPix's upload host, so the upload URL needs to allow your origin. Get this wrong and Chrome will throw a CORS error that looks scarier than it is.

⚠️ Note: don't return assetId to the browser. We stash it server-side and hand the client a stable videoId instead. Keeps the asset-id mapping under our control.

3. The upload component

// app/upload/page.tsx
"use client";

import { useState } from "react";
import { useRouter } from "next/navigation";

export default function UploadPage() {
  const router = useRouter();
  const [progress, setProgress] = useState(0);
  const [error, setError] = useState<string | null>(null);

  async function handleUpload(file: File) {
    setError(null);
    setProgress(0);

    const r = await fetch("/api/uploads", { method: "POST" });
    if (!r.ok) { setError("Could not create upload"); return; }
    const { videoId, uploadUrl } = await r.json();

    await new Promise<void>((resolve, reject) => {
      const xhr = new XMLHttpRequest();
      xhr.open("PUT", uploadUrl);
      xhr.upload.onprogress = (e) => {
        if (e.lengthComputable) setProgress(Math.round((e.loaded / e.total) * 100));
      };
      xhr.onload = () => xhr.status < 300 ? resolve() : reject(new Error(`PUT ${xhr.status}`));
      xhr.onerror = () => reject(new Error("Network error"));
      xhr.send(file);
    }).catch((e) => setError(String(e)));

    router.push(`/watch/${videoId}`);
  }

  return (
    <main className="p-8">
      <input
        type="file"
        accept="video/*"
        onChange={(e) => e.target.files?.[0] && handleUpload(e.target.files[0])}
      />
      {progress > 0 && <p>Uploading: {progress}%</p>}
      {error && <p className="text-red-600">{error}</p>}
    </main>
  );
}

We use XMLHttpRequest instead of fetch because fetch still doesn't expose upload progress in a useful way in 2026, and "uploading: 47%" is the difference between the user closing the tab and waiting.

💡 Tip: for production, swap this for the official FastPix Web upload SDK. It does chunked uploads, retries, and resume on flaky networks. Worth it for mobile users.

4. The webhook handler

// app/api/webhooks/fastpix/route.ts
import { NextResponse } from "next/server";
import { createHmac, timingSafeEqual } from "node:crypto";
import { db } from "@/lib/db";

export async function POST(req: Request) {
  const raw = await req.text();
  const sig = req.headers.get("fastpix-signature") ?? "";

  const expected = createHmac("sha256", process.env.FASTPIX_WEBHOOK_SECRET!)
    .update(raw)
    .digest("hex");

  // timing-safe compare
  if (sig.length !== expected.length ||
      !timingSafeEqual(Buffer.from(sig), Buffer.from(expected))) {
    return NextResponse.json({ error: "bad signature" }, { status: 401 });
  }

  const evt = JSON.parse(raw);
  if (evt.type !== "video.asset.ready") {
    return NextResponse.json({ ok: true });
  }

  const asset = evt.data;
  await db.video.update({
    where: { providerAssetId: asset.id },
    data: { status: "ready", playbackId: asset.playbackIds?.[0]?.id },
  });

  return NextResponse.json({ ok: true });
}

Two things to notice:

We read the raw body as text, then JSON-parse it ourselves. If you use req.json() first, the HMAC won't match because Node has already re-serialized the object.
The signature check is timingSafeEqual. Don't compare with ===; it leaks bytes through timing.

⚠️ Note: webhook event names and field shapes vary by provider. If you're swapping in Mux, the event is video.asset.ready too, but the signature header is Mux-Signature and the payload uses playback_ids (snake case). Always read the provider's webhook reference.

Testing webhooks locally

# In one terminal:
npm run dev

# In another, expose 3000 to the internet:
ngrok http 3000

# Register the ngrok URL in dashboard.fastpix.io → Webhooks
# Format: https://abc123.ngrok-free.app/api/webhooks/fastpix

Tail the output. Upload a clip from the dev UI. You should see a POST to /api/webhooks/fastpix within 30 to 90 seconds for a short clip.

5. The watch page

// app/watch/[id]/page.tsx
import { db } from "@/lib/db";
import { Player } from "./Player";

export default async function WatchPage({ params }: { params: { id: string } }) {
  const v = await db.video.findUnique({ where: { id: params.id } });
  if (!v) return <p>Not found</p>;
  if (v.status !== "ready" || !v.playbackId) {
    return <p>Still processing. This page auto-refreshes.</p>;
  }
  const src = `https://stream.fastpix.io/${v.playbackId}.m3u8`;
  return <Player src={src} />;
}

And a client component for the player itself:

// app/watch/[id]/Player.tsx
"use client";

import { useEffect, useRef } from "react";
import Hls from "hls.js";

export function Player({ src }: { src: string }) {
  const ref = useRef<HTMLVideoElement>(null);

  useEffect(() => {
    const v = ref.current;
    if (!v) return;

    // Safari plays HLS natively.
    if (v.canPlayType("application/vnd.apple.mpegurl")) {
      v.src = src;
      return;
    }

    if (!Hls.isSupported()) return;
    const hls = new Hls();
    hls.loadSource(src);
    hls.attachMedia(v);
    return () => hls.destroy();
  }, [src]);

  return <video ref={ref} controls className="w-full max-w-4xl" />;
}

That's the whole player for a public asset. For private assets, mint a short-lived JWT on the server and append it as a query string. The JWT is signed with an asymmetric key pair you generate once and store in your secret manager.

6. Sanity-check the round trip

# Upload a small clip via the UI.
# Watch the dev server logs.
# You should see:
POST /api/uploads 200
PUT https://upload.fastpix.io/... 200   # browser → FastPix
POST /api/webhooks/fastpix 200          # ~30-90s later
GET  /watch/<videoId> 200

If the webhook never fires, three things go wrong in this order, every time: ngrok URL changed (re-register it), CORS misconfigured on the upload (check the cors_origin you passed), or the webhook secret in .env.local doesn't match the one in the dashboard.

What about the player UI?

We used <video controls> for this tutorial, which gives you the browser default controls. For a real product you'll want a custom player. Two paths:

Option	When to pick it
FastPix Web Player (custom element)	Default. You drop `<fastpix-player playback-id="...">` and the player + analytics wiring come together.
Build on hls.js directly	You want full UI control. More code, total ownership of the surface.

I have a longer piece on the second path that covers the events, the error recovery loop, and React StrictMode cleanup. For this tutorial, the default <video controls> is enough to ship.

What's next

QoE analytics. Wire the Video Data SDK (or your provider's equivalent). FastPix Video Data is free up to 100,000 views per month, which covers most side-project and SaaS workloads before you ever pay.
Resume on interruption. Replace the raw XMLHttpRequest upload with the official Web upload SDK to get chunked uploads and resume.
Signed playback. Switch the playback policy from public to signed and add JWT minting to the watch page.
Thumbnails. The default thumbnail is the middle frame. Pick a better one.

The pattern transfers cleanly. If you build this on Mux, swap the endpoint to https://api.mux.com/video/v1/uploads, the playback URL to https://stream.mux.com/<playbackId>.m3u8, and the webhook handler to verify Mux-Signature. Cloudflare Stream and api.video have the same three pieces with their own dialect.

The thing this tutorial is really about, more than any one provider, is the shape: broker upload tokens on the server, push bytes from the browser, receive webhooks, render manifests. Once you have that shape in your head, the next video feature you build will take an afternoon instead of a sprint.

Building a shot-detection worker for an upload pipeline with PySceneDetect 0.7

Mason K — Thu, 21 May 2026 09:52:40 +0000

📦 Code: github.com/USER/shot-detection-worker (replace before publishing)

TL;DR

We are going to build an upload worker that runs shot detection on a video, writes the boundary list to Postgres, and produces a 6-frame storyboard PNG for use as a hover-preview sprite. Stack is PySceneDetect 0.7 (released earlier this month), FFmpeg 8.1.1, Python 3.12, and a Redis-backed queue. About 120 lines of code.

Shot detection is one of those building blocks that quietly powers a lot of features people associate with "video AI": chapter generation, smart thumbnails, hover-preview sprites, the first step of any auto-clipping pipeline. The open-source path covers more ground than people expect, and PySceneDetect 0.7 (which dropped on 2026-05-03) is the version I would build on today.

Let's wire it up end to end.

🛠️ 1. Setup

# bash
mkdir shot-worker && cd shot-worker
python3.12 -m venv .venv && source .venv/bin/activate
pip install scenedetect==0.7 opencv-python-headless==4.10.0.84 \
            rq==1.16 psycopg[binary]==3.2.0 boto3==1.34.0

We also need FFmpeg on the host:

# bash
$ ffmpeg -version | head -1
ffmpeg version 8.1.1 ...

A throwaway DB for the boundaries table:

-- schema.sql
CREATE TABLE shots (
    asset_id   TEXT NOT NULL,
    shot_index INT  NOT NULL,
    start_s    NUMERIC(10, 3) NOT NULL,
    end_s      NUMERIC(10, 3) NOT NULL,
    metric     NUMERIC(10, 3),
    PRIMARY KEY (asset_id, shot_index)
);

CREATE INDEX shots_asset_idx ON shots (asset_id);

# bash
psql -d shotworker -f schema.sql

🎬 2. The detector picker

PySceneDetect 0.7 ships five detectors, and the right one depends on the content. A small router up front saves a lot of pain later.

# worker/detectors.py
from scenedetect import AdaptiveDetector, ContentDetector, ThresholdDetector
from scenedetect.detectors import HistogramDetector, HashDetector

def pick_detector(content_class: str):
    """Pick the PySceneDetect detector that suits the content class.

    content_class comes from the upload metadata: 'talking_head', 'sports',
    'animation', 'screen_recording', etc. Tune thresholds against your own
    samples; the defaults here are starting points, not ground truth.
    """
    match content_class:
        case "sports" | "action":
            # Rolling-average baseline handles fast camera motion better.
            return AdaptiveDetector(adaptive_threshold=3.0, min_scene_len=15)
        case "animation":
            # Histogram delta works better than HSV deltas on animated content.
            return HistogramDetector(threshold=0.05, min_scene_len=15)
        case "screen_recording":
            # Perceptual hashing skips long static stretches.
            return HashDetector(threshold=0.395, min_scene_len=30)
        case "fade_heavy":
            # Threshold-based detection catches fade-to-black act breaks.
            return ThresholdDetector(threshold=12.0, min_scene_len=15)
        case _:
            # Default: HSV content delta. Works on most diverse content.
            return ContentDetector(threshold=27.0, min_scene_len=15)

💡 Tip: keep a small set of labeled test clips (one per content class) and re-run the detector on them whenever you touch thresholds. The "did my change regress" question gets cheaper to answer fast.

🔍 3. The detection function

PySceneDetect's high-level API is small enough that the whole detection step is a dozen lines:

# worker/detect.py
from pathlib import Path

from scenedetect import open_video, SceneManager
from worker.detectors import pick_detector


def detect_shots(video_path: Path, content_class: str = "default"):
    """Run shot detection on a video file.

    Returns a list of (start_seconds, end_seconds, metric_score) tuples.
    """
    video = open_video(str(video_path))
    scene_manager = SceneManager()
    detector = pick_detector(content_class)
    scene_manager.add_detector(detector)

    scene_manager.detect_scenes(video=video, show_progress=False)

    shots = []
    for i, (start, end) in enumerate(scene_manager.get_scene_list()):
        shots.append((
            start.get_seconds(),
            end.get_seconds(),
            None,  # metric per scene not exposed directly in 0.7
        ))
    return shots

You can also grab the per-frame metric data by adding a StatsManager and writing it to CSV, which is invaluable when threshold tuning:

# worker/detect.py (extended version with stats)
from scenedetect import StatsManager

def detect_shots_with_stats(video_path, content_class, stats_csv):
    video = open_video(str(video_path))
    stats = StatsManager()
    scene_manager = SceneManager(stats_manager=stats)
    scene_manager.add_detector(pick_detector(content_class))
    scene_manager.detect_scenes(video=video, show_progress=False)
    stats.save_to_csv(stats_csv)
    return [(s.get_seconds(), e.get_seconds(), None)
            for s, e in scene_manager.get_scene_list()]

🖼️ 4. The storyboard

For a hover-preview sprite, we want one frame per shot. FFmpeg does the heavy lifting; PySceneDetect tells it where to look:

# worker/storyboard.py
import subprocess
from pathlib import Path


def extract_keyframes(video_path: Path, shots, out_dir: Path, max_frames: int = 6):
    """Extract one frame per shot, capped at max_frames total."""
    out_dir.mkdir(parents=True, exist_ok=True)

    # If there are more shots than slots, sample evenly.
    if len(shots) > max_frames:
        step = len(shots) / max_frames
        sampled = [shots[int(i * step)] for i in range(max_frames)]
    else:
        sampled = shots

    paths = []
    for i, (start_s, end_s, _) in enumerate(sampled):
        midpoint = start_s + (end_s - start_s) / 2
        out_path = out_dir / f"frame_{i:02d}.jpg"
        subprocess.run(
            [
                "ffmpeg", "-ss", f"{midpoint:.3f}", "-i", str(video_path),
                "-frames:v", "1", "-q:v", "3", "-vf", "scale=320:-2",
                "-y", str(out_path),
            ],
            check=True,
            stderr=subprocess.DEVNULL,
        )
        paths.append(out_path)

    # Stitch into a 1x6 sprite.
    sprite_path = out_dir / "sprite.png"
    subprocess.run(
        ["ffmpeg", "-i", str(out_dir / "frame_%02d.jpg"),
         "-vf", f"tile={max_frames}x1", "-y", str(sprite_path)],
        check=True,
        stderr=subprocess.DEVNULL,
    )
    return sprite_path

-ss before -i is intentional: it lets FFmpeg seek before decoding, which makes the extract cheap on long videos. Putting -ss after -i reads from the start of the file every time, which is fine on a 30-second clip and miserable on a 90-minute one.

🔌 5. The worker

We tie everything together with an RQ job. Whichever queue you use, the shape is the same: pull asset, detect, write boundaries, produce storyboard, mark asset ready.

# worker/job.py
import json
import logging
from pathlib import Path
import tempfile

import boto3
import psycopg

from worker.detect import detect_shots_with_stats
from worker.storyboard import extract_keyframes

logger = logging.getLogger(__name__)
s3 = boto3.client("s3")


def process_upload(asset_id: str, bucket: str, key: str,
                   content_class: str, dsn: str):
    """End-to-end shot detection + storyboard for a single uploaded asset."""
    with tempfile.TemporaryDirectory() as tmp:
        local = Path(tmp) / "input.mp4"
        s3.download_file(bucket, key, str(local))
        logger.info("downloaded %s/%s -> %s", bucket, key, local)

        stats_csv = Path(tmp) / "stats.csv"
        shots = detect_shots_with_stats(local, content_class, stats_csv)
        logger.info("detected %d shots in %s", len(shots), asset_id)

        # Persist boundaries
        with psycopg.connect(dsn) as conn:
            with conn.cursor() as cur:
                cur.execute(
                    "DELETE FROM shots WHERE asset_id = %s", (asset_id,),
                )
                cur.executemany(
                    "INSERT INTO shots (asset_id, shot_index, start_s, end_s, metric) "
                    "VALUES (%s, %s, %s, %s, %s)",
                    [(asset_id, i, s, e, m) for i, (s, e, m) in enumerate(shots)],
                )

        # Storyboard
        out_dir = Path(tmp) / "storyboard"
        sprite = extract_keyframes(local, shots, out_dir, max_frames=6)
        s3.upload_file(str(sprite), bucket, f"storyboards/{asset_id}.png")
        s3.upload_file(str(stats_csv), bucket, f"shot-stats/{asset_id}.csv")

        return {
            "asset_id": asset_id,
            "shots": len(shots),
            "sprite_key": f"storyboards/{asset_id}.png",
        }

Hooking it up to RQ:

# worker/run.py
from rq import Queue, Worker
from redis import Redis
import os

if __name__ == "__main__":
    redis = Redis.from_url(os.environ["REDIS_URL"])
    q = Queue("uploads", connection=redis)
    with Worker([q], connection=redis) as w:
        w.work()

Enqueue an upload job from your upload handler:

# in your upload handler
from rq import Queue
from redis import Redis

q = Queue("uploads", connection=Redis.from_url(REDIS_URL))
q.enqueue(
    "worker.job.process_upload",
    asset_id="upload_abc123",
    bucket="my-uploads",
    key="raw/abc123.mp4",
    content_class="talking_head",
    dsn=os.environ["DB_DSN"],
    job_timeout=600,
)

⚡ 6. Throughput and gotchas

A few notes from real deployments:

PySceneDetect is CPU-bound and decode-heavy. The detector itself is fast; the bottleneck is OpenCV decoding the video. On commodity CPU (4 vCPU, 8 GB RAM), 1080p talking-head content processes faster than real time. Mileage varies on dense 4K content.
Use opencv-python-headless on servers. The full opencv-python package pulls in GTK/Qt and breaks in containerized environments.
Sample frames if you do not need pixel-precise boundaries. PySceneDetect has a downscale_factor argument that subsamples the input. For chapter-generation use cases, a 2x downscale halves processing time and changes the boundary list by a frame or two at most.
The CSV stats file is gold. Save it. The day a content class regresses and the PM asks "why did chapters drop", that file is the answer.

# faster detection at the cost of frame-level precision
video = open_video(str(video_path), backend="opencv")
scene_manager = SceneManager()
scene_manager.auto_downscale = True
scene_manager.detect_scenes(video=video, frame_skip=2)

⚠️ Note: frame_skip improves throughput but means the detector misses very short shots (a 4-frame quick cut at frame_skip=2 may not register). Tune to your content.

🧪 7. Verifying the output

A small script that opens the storyboard and prints the boundary list:

# bash
$ python -m worker.cli verify upload_abc123
asset upload_abc123
  shots: 14
  total: 73.21s
  storyboard: s3://my-uploads/storyboards/upload_abc123.png
  sample boundaries:
    [00:00:00.000 - 00:00:04.120]  (shot 0)
    [00:00:04.120 - 00:00:09.480]  (shot 1)
    [00:00:09.480 - 00:00:13.000]  (shot 2)
    ...

Open the sprite. One frame per shot, six total, evenly distributed.

What's next

A few directions to take this once the baseline works:

Chapter generation. Group shots that are at least 30 seconds long; the rest is a rules engine on top of the boundary list.
Smart thumbnails. Pipe the keyframes through a sharpness + face score and pick the best per shot.
Auto-clipping. Detect "interesting" moments by some other signal (audio energy, transcript keywords) and snap them to the nearest shot boundary; the clips stop looking like clips and start looking like edits.

The library does the boring part well, so you get to spend the engineering budget on the parts that actually feel like product.

video #python #tutorial #ffmpeg

Wiring VMAF (and PSNR) into your encoder CI with FFmpeg 8.1 and ffmpeg-quality-metrics

Mason K — Thu, 21 May 2026 09:51:09 +0000

📦 Code: github.com/USER/encoder-qa-ci (replace before publishing)

TL;DR

We are going to wire a perceptual-quality gate into a CI workflow using FFmpeg 8.1.1, libvmaf, and the ffmpeg-quality-metrics Python wrapper. The job runs PSNR, SSIM, and VMAF against a fixed reference ladder and fails the merge if VMAF drops below threshold. Works on CPU; the same setup runs roughly 6x faster on GPU with VMAF-CUDA.

PSNR has been the default "is my encoder okay" metric in CI pipelines for a decade, and it is starting to show its age. It cannot tell when the encoder traded perceptual quality for raw pixel error, and that is exactly the failure mode per-title and ML-driven encoders walk into. Let's wire up something better.

We will build this in three steps:

Stand up a tiny test bench with a reference ladder.
Run PSNR + SSIM + VMAF with ffmpeg-quality-metrics.
Wrap the whole thing in a CI script that fails on regression.

🛠️ 1. The test bench

You need three things: a reference master, a set of encoded renditions ("distorted" in metric-speak), and a fixed VMAF model file.

# bash
mkdir encoder-qa && cd encoder-qa
mkdir reference distorted models

# A short, representative reference. Pick a clip that matches your worst-case content:
# face-heavy, motion, fine detail. The "TearsOfSteel" or "BigBuckBunny" clips work for demos.
curl -L -o reference/master.mov \
  https://test-videos.co.uk/vids/bigbuckbunny/mp4/h264/720/Big_Buck_Bunny_720_10s_5MB.mp4

Drop the VMAF model into models/. The vmaf_v0.6.1.json model is the most widely cited Netflix VMAF model; there are also vmaf_4k_v0.6.1.json for 4K viewing and a phone variant. Pin the one you use, exactly:

# bash (clone the model into the repo so CI doesn't fetch over the network)
git clone --depth 1 https://github.com/Netflix/vmaf.git /tmp/vmaf
cp /tmp/vmaf/model/vmaf_v0.6.1.json models/

💡 Tip: treat the model file like a lockfile. Different VMAF model versions produce different scores. If you upgrade, you re-baseline every threshold.

Now generate a reference ladder. We use FFmpeg 8.1.1; older versions are fine but you want at least 6.1 for VMAF-CUDA, and 8.x for the cleanest webvtt/libsvtav1 integration.

# bash (produce three renditions for the test ladder)
ffmpeg -i reference/master.mov -c:v libx264 -preset medium -crf 23 -vf scale=1280:720  distorted/720p.mp4
ffmpeg -i reference/master.mov -c:v libx264 -preset medium -crf 26 -vf scale=854:480   distorted/480p.mp4
ffmpeg -i reference/master.mov -c:v libx264 -preset medium -crf 28 -vf scale=640:360   distorted/360p.mp4

Verify the FFmpeg version. The libvmaf and libsvtav1 wiring tightened up across the 8.0 → 8.1 line:

# bash
$ ffmpeg -version | head -1
ffmpeg version 8.1.1 ...
$ ffmpeg -filters | grep -E "libvmaf|psnr|ssim"
 .. libvmaf            VV->V       Calculate the VMAF between two video streams.
 .. psnr               VV->V       Calculate the PSNR between two video streams.
 .. ssim               VV->V       Calculate the SSIM between two video streams.

If libvmaf is not listed, your FFmpeg was built without --enable-libvmaf. Most distro builds ship it; static builds from BtbN/FFmpeg-Builds include it by default.

📐 2. Running the metrics

You can call ffmpeg -lavfi libvmaf... directly, but the output format is awkward to parse, and you end up writing the same Python wrapper everyone else has. Use ffmpeg-quality-metrics (slhck/ffmpeg-quality-metrics, still actively maintained). It runs all three metrics in one pass, emits JSON, and handles the model-path plumbing for you.

# bash
pip install ffmpeg-quality-metrics

A single rendition through the gate:

# bash
ffmpeg-quality-metrics distorted/720p.mp4 reference/master.mov \
  --metrics psnr ssim vmaf \
  --vmaf-model-path models/vmaf_v0.6.1.json \
  --output-format json > metrics_720p.json

The output looks like this (truncated):

{
  "global": {
    "psnr": { "psnr_avg": 41.82, "psnr_min": 36.41, "psnr_max": 45.97 },
    "ssim": { "ssim_avg": 0.978 },
    "vmaf": { "vmaf_avg": 88.7, "vmaf_min": 71.2, "vmaf_max": 96.4 }
  },
  "input_file_dist": "distorted/720p.mp4",
  "input_file_ref": "reference/master.mov"
}

global is the aggregate. The per-frame data is also in the JSON if you ask for --output-format json with the frame-level flag, and that is the file you want when an encoder regresses and you need to find which 200 frames lost 8 points.

⚠️ Note: VMAF on CPU is workable on short clips but slow on long ones. If you have NVENC-capable hardware, add --vmaf-features cuda (the wrapper passes it through to libvmaf) and decode through h264_cuvid / hevc_cuvid. The roughly 6x speedup NVIDIA documents lines up with what I see in practice when I keep frames on the GPU end-to-end.

🚦 3. Turning it into a CI gate

The script: run the metrics on every rendition, compare the aggregate against a per-rendition threshold, exit non-zero on regression.

# scripts/qa_gate.py
import json
import subprocess
import sys
from pathlib import Path

# Per-rendition VMAF floor. Tune to your content.
# Tighter on top renditions, looser on the bottom rung.
THRESHOLDS = {
    "720p": {"vmaf_avg": 85, "vmaf_min": 65, "ssim_avg": 0.96},
    "480p": {"vmaf_avg": 78, "vmaf_min": 55, "ssim_avg": 0.94},
    "360p": {"vmaf_avg": 70, "vmaf_min": 45, "ssim_avg": 0.92},
}

REFERENCE = "reference/master.mov"
MODEL = "models/vmaf_v0.6.1.json"

def run_metrics(distorted: Path) -> dict:
    out = subprocess.check_output([
        "ffmpeg-quality-metrics", str(distorted), REFERENCE,
        "--metrics", "psnr", "ssim", "vmaf",
        "--vmaf-model-path", MODEL,
        "--output-format", "json",
    ])
    return json.loads(out)["global"]

def main() -> int:
    failures = []
    for name, floor in THRESHOLDS.items():
        rendition = Path(f"distorted/{name}.mp4")
        if not rendition.exists():
            failures.append(f"{name}: rendition missing")
            continue
        metrics = run_metrics(rendition)
        for metric, minimum in floor.items():
            value = metrics.get(metric.split("_")[0], {}).get(metric)
            if value is None or value < minimum:
                failures.append(f"{name} {metric}: got {value}, want >= {minimum}")
        print(f"{name}: vmaf {metrics['vmaf']['vmaf_avg']:.1f}, "
              f"ssim {metrics['ssim']['ssim_avg']:.3f}, "
              f"psnr {metrics['psnr']['psnr_avg']:.1f}")
    if failures:
        print("\nFAILURES:")
        for f in failures:
            print(f"  - {f}")
        return 1
    return 0

if __name__ == "__main__":
    sys.exit(main())

Run it locally first:

# bash
$ python scripts/qa_gate.py
720p: vmaf 88.7, ssim 0.978, psnr 41.8
480p: vmaf 81.2, ssim 0.961, psnr 38.9
360p: vmaf 72.4, ssim 0.943, psnr 36.1

The output you actually want is when somebody breaks the encoder:

# bash
$ python scripts/qa_gate.py
720p: vmaf 79.1, ssim 0.962, psnr 42.4
480p: vmaf 81.2, ssim 0.961, psnr 38.9
360p: vmaf 72.4, ssim 0.943, psnr 36.1

FAILURES:
  - 720p vmaf_avg: got 79.1, want >= 85
exit code: 1

Look at what PSNR did here: it actually went up (42.4 vs the previous 41.8). The encoder traded perceptual quality for raw pixel error and a PSNR-only gate would have shipped it. The VMAF gate caught it.

🤖 4. The GitHub Actions job

# .github/workflows/encoder-qa.yml
name: encoder-qa
on:
  pull_request:
    paths:
      - "encoder/**"
      - "scripts/qa_gate.py"

jobs:
  qa:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4

      - name: Install FFmpeg 8.1.1
        run: |
          curl -L -o ffmpeg.tar.xz \
            https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-linux64-gpl.tar.xz
          tar xf ffmpeg.tar.xz --strip-components=2 -C /usr/local/bin --wildcards '*/bin/ffmpeg' '*/bin/ffprobe'
          ffmpeg -version | head -1

      - name: Install metrics wrapper
        run: pip install ffmpeg-quality-metrics

      - name: Build the test ladder
        run: ./scripts/build_ladder.sh

      - name: Run quality gate
        run: python scripts/qa_gate.py

A few notes on this in production:

Cache the reference master in an S3 bucket or a Git LFS object. Re-fetching from a public CDN on every run is a recipe for flaky builds.
Treat the model file as a build input. Bump it deliberately, and re-baseline thresholds when you do.
Keep the per-frame JSON as a build artifact for at least 30 days. When an encoder regresses, the headline number tells you it broke; the per-frame data tells you where.

⚠️ Things that bite you in real workflows

A short list of things I have had to fix on actual CI pipelines:

Different frame counts between reference and distorted. Trim with ffmpeg -ss/-to on both sides, or VMAF will silently truncate to the shorter one and the score will surprise you.
Color space mismatches. A BT.709 source compared against a BT.601 encode will produce noisy VMAF scores. Normalize with zscale=transfer=bt709:matrix=bt709.
Resolution mismatches. libvmaf scales the distorted to match the reference; if you really want to test "how this looks on a 720p screen", scale the reference down to 720p first, then compare.
CPU vs CUDA scores drift slightly. Same model, same files, different paths through the math. Pick one for CI and stick to it.

What's next

A few directions worth exploring:

Per-title encoding QA. The same gate, but a thresholds-per-content-class table. Talking heads, sports, animation, screen content; each gets its own floor.
Frame-level alerting. Plot the per-frame VMAF in your CI artifact viewer. A two-second dip below 60 is often more telling than the global average.
GPU-resident pipelines. If you encode with NVENC and want sub-realtime QA on long content, run h264_cuvid decoders into a libvmaf filter graph that stays on the GPU. The 6x speedup is real; the plumbing is fiddly enough to be its own post.

Wire it up, get the gate green, and the first time a PR turns it red you will be glad you stopped trusting PSNR as the whole story.

video #ffmpeg #tutorial #devops

Shipping WebVTT subtitles in HLS that actually stay in sync (a hands-on guide for 2026)

Mason K — Thu, 21 May 2026 09:49:23 +0000

📦 Code: github.com/USER/hls-webvtt-pipeline (replace before publishing)

TL;DR

We are going to build a real HLS subtitle pipeline. Take an SRT file, convert to WebVTT, segment it the way HLS expects, add the X-TIMESTAMP-MAP cue, declare it in the manifest, and verify it plays in sync in HLS.js 1.6.16 and Shaka Player. By the end you will have a working ladder + subtitle track and the confidence that it will not drift on seek.

If you have shipped HLS captions as a single sidecar .vtt and called it done, this is the upgrade you have been putting off. The bug it fixes (caption drift after seek, especially on Chrome on Android) is the kind of bug that gets reported as "the subtitles are wrong", with no useful repro steps, by users who never come back. Let's build the version that does not break.

🛠️ 1. The pieces we need

FFmpeg 8.1.1 (webvtt muxer; older versions work but the muxer behavior is friendlier in 8.x).
A packager. We will use shaka-packager because it has the cleanest support for segmented WebVTT and fMP4-wrapped subtitles. mp4box works too.
HLS.js 1.6.16 (latest as of April 2026) and Shaka Player 4.x for verification.
A short test video (1 to 2 minutes) and an SRT file with at least a dozen cues.

# bash
mkdir hls-webvtt && cd hls-webvtt
mkdir source segments manifest

# A short test video
curl -L -o source/master.mp4 \
  https://test-videos.co.uk/vids/bigbuckbunny/mp4/h264/720/Big_Buck_Bunny_720_10s_5MB.mp4

# A test SRT
cat > source/captions.srt <<'SRT'
1
00:00:00,000 --> 00:00:03,500
Hello, and welcome to the show.

2
00:00:03,500 --> 00:00:07,000
Today we are talking about HLS captions.

3
00:00:07,000 --> 00:00:10,000
And why most pipelines get them wrong.
SRT

📝 2. SRT to WebVTT (the easy part)

FFmpeg's webvtt muxer converts SRT for you. The conversion itself is mostly a header swap, but it normalizes line breaks and timestamps along the way:

# bash
ffmpeg -i source/captions.srt -c:s webvtt source/captions.vtt
cat source/captions.vtt

WEBVTT

1
00:00:00.000 --> 00:00:03.500
Hello, and welcome to the show.

2
00:00:03.500 --> 00:00:07.000
Today we are talking about HLS captions.

3
00:00:07.000 --> 00:00:10.000
And why most pipelines get them wrong.

This file works as a sidecar in HLS. It is also the source of every cross-browser sync bug you are about to hit. We need to segment it and add the timestamp map.

✂️ 3. Segmenting WebVTT for HLS

HLS expects WebVTT split into segments that align with the media segments. With six-second video segments, the subtitles should be on six-second segments too. The packager handles this.

First, build the video ladder. Two renditions is enough for the demo:

# bash
ffmpeg -i source/master.mp4 -c:v libx264 -preset medium -crf 23 \
  -vf scale=1280:720 -g 144 -keyint_min 144 -sc_threshold 0 \
  -hls_time 6 -hls_playlist_type vod \
  -hls_segment_filename "segments/720p_%03d.ts" \
  manifest/720p.m3u8

ffmpeg -i source/master.mp4 -c:v libx264 -preset medium -crf 26 \
  -vf scale=854:480 -g 144 -keyint_min 144 -sc_threshold 0 \
  -hls_time 6 -hls_playlist_type vod \
  -hls_segment_filename "segments/480p_%03d.ts" \
  manifest/480p.m3u8

Now segment the subtitles. Shaka Packager does this with a single descriptor:

# bash
docker run --rm -v "$PWD:/work" -w /work \
  google/shaka-packager \
  in=source/captions.vtt,stream=text,language=en,segment_template=segments/subs/en_$Number$.vtt,playlist_name=manifest/subs/en.m3u8,hls_group_id=subs,hls_name=English \
  --hls_master_playlist_output manifest/master.m3u8 \
  --segment_duration 6

The result is a directory of per-six-second WebVTT files plus a media playlist (manifest/subs/en.m3u8) that references them.

Open one of the segment files:

# segments/subs/en_2.vtt
WEBVTT
X-TIMESTAMP-MAP=MPEGTS:540000,LOCAL:00:00:00.000

00:00:00.000 --> 00:00:01.000
And why most pipelines get them wrong.

That X-TIMESTAMP-MAP line is the whole point. The cue inside the segment is on a local clock starting at zero. The packager has set MPEGTS:540000 to tell the player "this clock-zero corresponds to MPEG-2 PTS 540000", which is 6 seconds at the 90kHz tick rate, which is where the segment starts.

💡 Tip: open every segment file and confirm the X-TIMESTAMP-MAP is present and correct. Some packagers only emit it on the first segment. The HLS spec is generous about this; player implementations are not.

🧷 4. Wiring the manifest

Shaka Packager wrote a master playlist for us. Let's look at it:

# manifest/master.m3u8
#EXTM3U
#EXT-X-VERSION:6

#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="English",LANGUAGE="en",AUTOSELECT=YES,DEFAULT=NO,FORCED=NO,URI="subs/en.m3u8"

#EXT-X-STREAM-INF:BANDWIDTH=3200000,RESOLUTION=1280x720,SUBTITLES="subs",CODECS="avc1.640028"
720p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1600000,RESOLUTION=854x480,SUBTITLES="subs",CODECS="avc1.4d401f"
480p.m3u8

A few things worth pointing out:

SUBTITLES="subs" on every EXT-X-STREAM-INF line. Forget this and the renditions never advertise the subs.
AUTOSELECT=YES lets the player pick this track when the OS language matches. Set to NO if you want subs strictly opt-in.
DEFAULT=NO keeps subs off by default. Most product teams want this; set to YES only for forced subtitles (translated dialogue, on-screen text translation).

🌐 5. Serving and verifying

Serve the whole tree with any static server:

# bash
cd manifest && python3 -m http.server 8080

A minimal HTML harness using HLS.js 1.6.16:

<!-- index.html -->
<!doctype html>
<html>
  <body>
    <video id="v" controls autoplay style="width: 100%; max-width: 720px;"></video>
    <script src="https://cdn.jsdelivr.net/npm/hls.js@1.6.16/dist/hls.min.js"></script>
    <script>
      const video = document.getElementById('v');
      if (Hls.isSupported()) {
        const hls = new Hls({ enableWebVTT: true });
        hls.loadSource('http://localhost:8080/master.m3u8');
        hls.attachMedia(video);
        hls.on(Hls.Events.SUBTITLE_TRACKS_UPDATED, (_, d) =>
          console.log('subtitle tracks:', d.subtitleTracks),
        );
      } else if (video.canPlayType('application/vnd.apple.mpegurl')) {
        // Native HLS path (Safari, iOS)
        video.src = 'http://localhost:8080/master.m3u8';
      }
    </script>
  </body>
</html>

Open the page, hit the cc button in the player UI, and confirm the cue text shows up exactly when the actor speaks. Now hit seek to the middle of the timeline. The cue should jump to whatever line was being spoken at that point, with no drift.

⚠️ Note: HLS.js's light build (hls.light.min.js) does not include WebVTT parsing. Use the standard build for subtitle work.

Repeat with Shaka Player to make sure both engines agree:

<script src="https://cdn.jsdelivr.net/npm/shaka-player@4/dist/shaka-player.compiled.min.js"></script>
<script>
  shaka.polyfill.installAll();
  const player = new shaka.Player(document.getElementById('v'));
  player.load('http://localhost:8080/master.m3u8');
  player.setTextTrackVisibility(true);
</script>

If you see drift in one player but not the other, it is almost always an X-TIMESTAMP-MAP problem in your packager output. Open the failing segment, check the MPEGTS value, recompute it (segment_start_seconds * 90000), and re-segment.

🚨 6. The bugs you will probably hit

A quick checklist of what trips teams up:

Symptom	Likely cause
Captions perfect at start, drift after seek	`X-TIMESTAMP-MAP` missing or wrong on later segments
Captions show up but lag by a few seconds	Subtitle segments and media segments have different durations
Captions work on Safari, blank on Chrome	Using HLS.js `light` build (no WebVTT parser)
Captions appear but lose positioning	Converter flattened `align`, `line`, `position` cue settings
Captions show on 720p stream but not 480p	Missing `SUBTITLES="subs"` on one `EXT-X-STREAM-INF` line
Low-latency live captions arrive late or never	LL-HLS part loading bug (fixed in HLS.js 1.6.x; upgrade)

🧪 7. Going further: fMP4-wrapped subtitles

For CMAF-aligned pipelines and modern players, you can package the same WebVTT inside fMP4 segments instead of standalone .vtt files. The shaka-packager invocation becomes:

# bash (fMP4-wrapped subtitles)
docker run --rm -v "$PWD:/work" -w /work google/shaka-packager \
  in=source/captions.vtt,stream=text,language=en,format=ttml,init_segment=segments/subs/en_init.mp4,segment_template=segments/subs/en_$Number$.m4s,playlist_name=manifest/subs/en.m3u8,hls_group_id=subs,hls_name=English \
  --hls_master_playlist_output manifest/master.m3u8 \
  --segment_duration 6

Modern Safari, HLS.js 1.6.x, Shaka Player 4.x, and ExoPlayer all read fMP4 subtitle tracks. Smart TVs are still a mixed bag; if your product targets connected TV, run the test matrix before flipping the switch.

What's next

A few directions worth exploring once the basic pipeline is solid:

IMSC ingest. Premium content arrives as IMSC 1.1. Build a converter that preserves positioning and styling on the way to WebVTT; you will need it the first time a partner sends you a .dfxp.
Multi-language ladders. The same EXT-X-MEDIA pattern scales to N languages; the work is in the packager invocation, not the manifest.
Forced subtitles. Set FORCED=YES on subtitle tracks that translate on-screen text (foreign-language signs, locked subtitles in mixed-language scenes). The player surfaces them differently.

The shortcut version of HLS subtitles is fine for a demo. The version above is what stops being a support ticket.

video #webdev #tutorial #javascript

Wiring up a hybrid WebRTC + LL-HLS live stack (the protocol decision tree that actually works)

Mason K — Tue, 19 May 2026 12:08:36 +0000

TL;DR

We're going to build a hybrid live stack: presenters connect over WebRTC for sub-second feedback, the SFU re-publishes to RTMP, and the audience plays it back over LL-HLS at ~2 seconds. By the end you'll have a decision tree, a working ingest config, an LL-HLS player setup, and a list of the failure modes that bit me in production.

📦 Code: github.com/USER/llhls-webrtc-hybrid, replace before publishing

The "WebRTC vs LL-HLS" debate is mostly fake. In production, you almost always want both. The presenter needs sub-second so they can talk over their co-host without that awful video-call lag, and the 5,000 people watching just want a stream that doesn't buffer on their phone. We're going to wire up that pattern, end to end.

💡 Tip: If your concurrent viewer count will stay under 500 and the experience is genuinely interactive, skip the LL-HLS half of this article. Pure WebRTC is fine for you. The hybrid pattern is for products where the audience grows past what your media server can hold.

1. The decision tree, codified

Before you write a line of code, ask two questions:

Q1: Does any viewer need sub-second latency (auctions, real-time interaction)?
    → YES: WebRTC is mandatory for those viewers.
    → NO:  LL-HLS only. You're done. Stop reading.

Q2: How many concurrent viewers in 18 months?
    → < 500:    All-WebRTC. Skip LL-HLS.
    → 500–10K:  Hybrid. WebRTC for presenters, LL-HLS for the rest.
    → 10K+:     Hybrid mandatory. LL-HLS is the only thing that scales here.

The math under that tree: a 4-core media server fits roughly 200 WebRTC viewers vs roughly 1,000 LL-HLS viewers. LL-HLS sits behind a CDN cleanly; WebRTC is per-session and can't.

2. The stack we're building

[Presenter]  --WebRTC-->  [SFU]  --RTMP-->  [Ingest]  --LL-HLS-->  [CDN]  -->  [Viewers]
                            |
                            +--WebRTC-->  [Co-presenter / Guest]

For the SFU, I'll use LiveKit because it has a clean cloud-or-self-hosted story and its egress feature pushes RTMP out of the box. For ingest + packaging, I'll use FFmpeg (free, hackable, the reference for everything). For playback, HLS.js with low-latency mode enabled.

⚠️ Note: This article isn't a LiveKit tutorial. Substitute any SFU that supports RTMP egress (Janus, mediasoup, Daily, etc.). The pattern is the same.

3. Wire up the WebRTC half

A minimal browser-side publish using livekit-client. This is the presenter joining the stage:

// app/stage.js
import { Room, RoomEvent, createLocalTracks } from 'livekit-client';

const room = new Room({
  adaptiveStream: true,
  dynacast: true,
  publishDefaults: { simulcast: true },
});

await room.connect(import.meta.env.VITE_LIVEKIT_URL, await fetchJoinToken());

const tracks = await createLocalTracks({ audio: true, video: { resolution: { width: 1280, height: 720, frameRate: 30 } } });
for (const track of tracks) await room.localParticipant.publishTrack(track);

room.on(RoomEvent.ConnectionStateChanged, (state) => {
  console.log('[stage] state =', state);
});

The token is signed server-side. Here's the minimal Node endpoint:

// server/token.js
import { AccessToken } from 'livekit-server-sdk';

export function joinToken(identity, room) {
  const at = new AccessToken(process.env.LIVEKIT_API_KEY, process.env.LIVEKIT_API_SECRET, { identity });
  at.addGrant({ roomJoin: true, room, canPublish: true, canSubscribe: true });
  return at.toJwt();
}

That gets the presenter and co-presenters on WebRTC. Their glass-to-glass is sub-300 ms on a decent network.

4. Re-publish the room to RTMP

LiveKit's egress service can record or relay a room composition out to an RTMP endpoint. We'll point it at our LL-HLS ingest.

# scripts/start-egress.sh
curl -X POST "$LIVEKIT_URL/twirp/livekit.Egress/StartRoomCompositeEgress" \
  -H "Authorization: Bearer $LIVEKIT_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "room_name": "live-show",
    "layout": "speaker",
    "audio_only": false,
    "stream_outputs": [
      { "protocol": "RTMP", "urls": ["rtmp://ingest.example.com/live/show1"] }
    ]
  }'

Now your SFU is pushing the composed stream into your ingest server as plain RTMP. Anything downstream that speaks RTMP can pick it up.

5. Package to LL-HLS with FFmpeg

This is the part most teams get wrong on the first try. The LL-HLS spec requires CMAF partial segments (typically ~200 ms), preload hints, and an EXT-X-SERVER-CONTROL tag in the manifest. FFmpeg 6.0+ supports this directly:

# scripts/package-llhls.sh
ffmpeg -i rtmp://ingest.example.com/live/show1 \
  -c:v libx264 -preset veryfast -tune zerolatency -g 60 -keyint_min 60 -sc_threshold 0 \
  -c:a aac -ar 48000 -b:a 128k \
  -hls_time 2 \
  -hls_playlist_type event \
  -hls_segment_type fmp4 \
  -hls_fmp4_init_filename init.mp4 \
  -hls_segment_filename "/var/www/hls/seg_%05d.m4s" \
  -hls_flags independent_segments+program_date_time+append_list \
  -hls_list_size 6 \
  -master_pl_name master.m3u8 \
  -strftime 1 \
  -method PUT \
  -http_persistent 1 \
  -ldash 1 \
  -window_size 6 \
  -extra_window_size 3 \
  -streaming 1 \
  -seg_duration 2 \
  -frag_duration 0.2 \
  /var/www/hls/stream.m3u8

💡 Tip: The two parameters that actually matter for latency are -frag_duration 0.2 (200 ms CMAF chunks, which the LL-HLS spec wants) and -streaming 1 (write chunks as they're produced rather than after the segment closes). Without those, you have plain HLS with extra steps.

The output is a /var/www/hls/ directory full of .m4s chunks plus a stream.m3u8 manifest with partial-segment annotations. Point your CDN at that directory and respect the chunked transfer encoding when you proxy.

6. Player setup

HLS.js with low-latency mode:

// app/player.js
import Hls from 'hls.js';

const video = document.querySelector('#viewer');
const hls = new Hls({
  lowLatencyMode: true,
  backBufferLength: 4,
  maxLiveSyncPlaybackRate: 1.5,
  liveSyncDuration: 1.5,
  liveMaxLatencyDuration: 3.5,
});

hls.loadSource('https://cdn.example.com/hls/stream.m3u8');
hls.attachMedia(video);

hls.on(Hls.Events.MEDIA_ATTACHED, () => video.play());

hls.on(Hls.Events.ERROR, (_evt, data) => {
  if (data.fatal) {
    console.error('[player] fatal', data.type, data.details);
    hls.startLoad();
  }
});

The values for liveSyncDuration and liveMaxLatencyDuration are the ones I've landed on after testing. They tell the player to stay 1.5 seconds from live with a 3.5-second hard cap before it resyncs. Tighter values rebuffer too often; looser values defeat the point of LL-HLS.

⚠️ Note: Safari uses its native HLS implementation, not HLS.js, and it respects the manifest's EXT-X-SERVER-CONTROL:CAN-BLOCK-RELOAD=YES tag for low latency. FFmpeg's LL-HLS muxer writes that tag for you when -streaming 1 is set. Verify by curling the manifest and grepping for it.

7. Verifying it actually works

The first time you wire this up, the latency will not be what you expected. Tools to sanity-check:

# Check the manifest has LL-HLS annotations
curl -s https://cdn.example.com/hls/stream.m3u8 | grep -E '(SERVER-CONTROL|PART-INF|EXT-X-PRELOAD-HINT)'

# Should print something like:
# #EXT-X-SERVER-CONTROL:CAN-BLOCK-RELOAD=YES,PART-HOLD-BACK=0.6
# #EXT-X-PART-INF:PART-TARGET=0.2
# #EXT-X-PRELOAD-HINT:TYPE=PART,URI="seg_00123.m4s?byterange=..."

# Measure end-to-end latency with a clock overlay on the source
# Run on the presenter machine:
ffmpeg -re -f lavfi -i "color=c=black:s=640x360:d=600,drawtext=text='%{localtime}'" \
  -f flv rtmp://ingest.example.com/live/show1

Open the playback URL on another device, eyeball the difference between the wall clock on the source and the rendered clock in the player. Production LL-HLS on a tuned CDN gets you 2 to 3 seconds. WebRTC on the same path gets you under half a second.

8. The failure modes that bit me

A short list of things to check before you ship:

CDN that doesn't honor chunked transfer encoding. Some legacy CDNs buffer the segment fully before serving it. You'll get plain HLS latency with all the LL-HLS code paths enabled. Test with curl: curl -v https://cdn.example.com/hls/stream.m3u8 2>&1 | grep -i chunked should show Transfer-Encoding: chunked.
Player buffering "just in case". Default HLS.js buffers more than you'd expect. The liveSyncDuration and liveMaxLatencyDuration values above are deliberate.
TURN-only WebRTC sessions on the presenter side. Corporate firewalls force TURN, which adds a relay hop. Mitigate by deploying TURN close to your SFU.
Egress lag from the SFU. RTMP relay from LiveKit egress has a built-in ~1-2 second buffer on top of LL-HLS latency. The hybrid pattern is fundamentally "WebRTC fast for presenters, LL-HLS slow-ish for audience". Don't try to make the audience side faster than the protocol allows.

What's next

If you want to push the audience-side latency lower, the next step is HESP or WebRTC-WHEP for everyone, both of which have rougher edges than LL-HLS today but are worth tracking. If you want to reduce egress cost, look at peer-assisted delivery (Peer5, CDNBye) which works cleanly on top of LL-HLS.

For player customization (DVR window, captions, multi-audio), the HLS.js docs are the right next read. For ingest at scale, you'll eventually outgrow single-box FFmpeg and want to look at managed packaging or a Kubernetes setup with horizontal ingest.

The protocol war is over. The interesting work in 2026 is building the right hybrid for your audience. Happy streaming.

Add real video QoE telemetry to your player in an afternoon

Mason K — Fri, 15 May 2026 05:02:24 +0000

📦 Code: github.com/USER/video-qoe-starter — replace before publishing.

TL;DR

We'll instrument an HLS.js player to emit the three QoE metrics that actually correlate with viewer abandonment — startup time, rebuffering ratio, and playback failure rate — ship them to a tiny Fastify endpoint, store them in SQLite, and render a dashboard. End to end in one file each: a React component, a Node endpoint, and a SQL view.

What we're building

A minimum-viable telemetry rig that answers the three questions every video team gets asked the day after launch:

How long is it taking for a video to start playing?
How often is the player stalling mid-playback?
How often is playback failing outright?

We'll use HLS.js 1.6.x (the current stable line, with LL-HLS support), a Node 22 + Fastify backend, and SQLite for storage. No queue, no warehouse, no Grafana. The point is to ship a working baseline; you can swap parts later.

1. Set up the project 🛠️

mkdir video-qoe-starter && cd video-qoe-starter
npm init -y
npm install hls.js fastify better-sqlite3
npm install -D typescript @types/node @types/better-sqlite3
mkdir client server

Your package.json scripts block:

{
  "scripts": {
    "server": "node --experimental-strip-types server/index.ts",
    "build": "tsc"
  }
}

💡 Tip: Node 22 ships with --experimental-strip-types for TypeScript files, so you can run .ts directly without ts-node for prototypes like this.

2. Instrument the player 🎯

The HLS.js event surface is the same shape across every modern player — MANIFEST_PARSED, MEDIA_ATTACHED, FRAG_BUFFERED, ERROR, plus the standard HTML video events (play, waiting, playing, ended).

// client/qoe.ts
import Hls, { Events, ErrorData } from 'hls.js';

type QoeSession = {
  session_id: string;
  video_id: string;
  player: string;
  user_agent: string;
  startup_time_ms: number | null;
  total_watch_time_ms: number;
  total_stall_time_ms: number;
  rebuffer_count: number;
  error_codes: string[];
};

export function instrument(video: HTMLVideoElement, hls: Hls, videoId: string) {
  const session: QoeSession = {
    session_id: crypto.randomUUID(),
    video_id: videoId,
    player: `hls.js@${Hls.version}`,
    user_agent: navigator.userAgent,
    startup_time_ms: null,
    total_watch_time_ms: 0,
    total_stall_time_ms: 0,
    rebuffer_count: 0,
    error_codes: [],
  };

  let playRequestedAt: number | null = null;
  let lastPlayingAt: number | null = null;
  let stallStartedAt: number | null = null;
  let firstPlayingFired = false;

  video.addEventListener('play', () => {
    if (playRequestedAt === null) playRequestedAt = performance.now();
  });

  video.addEventListener('playing', () => {
    const now = performance.now();
    if (!firstPlayingFired && playRequestedAt !== null) {
      session.startup_time_ms = now - playRequestedAt;
      firstPlayingFired = true;
    }
    if (stallStartedAt !== null) {
      session.total_stall_time_ms += now - stallStartedAt;
      stallStartedAt = null;
    }
    lastPlayingAt = now;
  });

  video.addEventListener('waiting', () => {
    // 'waiting' is the rebuffer signal — buffer underrun, fetching more segments.
    if (firstPlayingFired) {
      session.rebuffer_count += 1;
      stallStartedAt = performance.now();
    }
  });

  video.addEventListener('pause', () => {
    if (lastPlayingAt !== null) {
      session.total_watch_time_ms += performance.now() - lastPlayingAt;
      lastPlayingAt = null;
    }
  });

  hls.on(Events.ERROR, (_, data: ErrorData) => {
    session.error_codes.push(`${data.type}:${data.details}`);
  });

  const flush = () => {
    if (lastPlayingAt !== null) {
      session.total_watch_time_ms += performance.now() - lastPlayingAt;
      lastPlayingAt = null;
    }
    navigator.sendBeacon('/qoe', JSON.stringify(session));
  };

  // Flush on tab close, route change, or page unload.
  window.addEventListener('pagehide', flush);
  document.addEventListener('visibilitychange', () => {
    if (document.visibilityState === 'hidden') flush();
  });

  return { session, flush };
}

A few decisions baked in here worth calling out:

navigator.sendBeacon is the right transport for telemetry on unload. fetch gets aborted when the tab closes; sendBeacon is best-effort but fire-and-forget.
First playing event is the only honest signal for "first frame painted." loadedmetadata and canplay fire too early — they fire when the player could play, not when it did.
waiting is the rebuffer signal, but only after the first playing. Before the first playing, waiting is part of startup.

⚠️ Note: Don't trust client-side derived metrics. Ship raw event data (start, playing, waiting, error timestamps) and compute rebuffer ratio server-side. We're shipping aggregates in this tutorial for simplicity — for production, do raw events.

3. Wire the player to the instrumentation 🔌

// client/Player.tsx
import { useEffect, useRef } from 'react';
import Hls from 'hls.js';
import { instrument } from './qoe';

export function Player({ src, videoId }: { src: string; videoId: string }) {
  const videoRef = useRef<HTMLVideoElement>(null);

  useEffect(() => {
    if (!videoRef.current) return;
    const hls = new Hls({ enableWorker: true });
    hls.loadSource(src);
    hls.attachMedia(videoRef.current);
    const { flush } = instrument(videoRef.current, hls, videoId);

    return () => {
      flush();
      hls.destroy();
    };
  }, [src, videoId]);

  return <video ref={videoRef} controls playsInline width="800" />;
}

That's the whole client side.

4. Build the ingestion endpoint 📥

The simplest thing that could possibly work — Fastify, JSON in, SQLite out:

// server/index.ts
import Fastify from 'fastify';
import Database from 'better-sqlite3';

const db = new Database('qoe.db');
db.exec(`
  CREATE TABLE IF NOT EXISTS sessions (
    session_id TEXT PRIMARY KEY,
    video_id TEXT NOT NULL,
    player TEXT,
    user_agent TEXT,
    startup_time_ms REAL,
    total_watch_time_ms REAL,
    total_stall_time_ms REAL,
    rebuffer_count INTEGER,
    error_codes TEXT,
    created_at INTEGER NOT NULL
  );
`);

const insert = db.prepare(`
  INSERT OR REPLACE INTO sessions
  (session_id, video_id, player, user_agent, startup_time_ms,
   total_watch_time_ms, total_stall_time_ms, rebuffer_count, error_codes, created_at)
  VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`);

const app = Fastify({ logger: true });

app.post('/qoe', async (req, reply) => {
  const s = req.body as Record<string, unknown>;
  insert.run(
    s.session_id, s.video_id, s.player, s.user_agent,
    s.startup_time_ms ?? null,
    s.total_watch_time_ms ?? 0,
    s.total_stall_time_ms ?? 0,
    s.rebuffer_count ?? 0,
    JSON.stringify(s.error_codes ?? []),
    Date.now(),
  );
  reply.send({ ok: true });
});

app.get('/metrics', async () => {
  return db.prepare(`
    SELECT
      COUNT(*) as sessions,
      AVG(startup_time_ms) as avg_startup_ms,
      SUM(total_stall_time_ms) * 1.0 / NULLIF(SUM(total_watch_time_ms), 0) as rebuffer_ratio,
      SUM(CASE WHEN error_codes != '[]' THEN 1 ELSE 0 END) * 1.0 / COUNT(*) as failure_rate
    FROM sessions
    WHERE created_at > unixepoch() * 1000 - 86400000
  `).get();
});

app.listen({ port: 3000, host: '0.0.0.0' });

Run it:

npm run server

Expect:

{"level":30,"time":...,"msg":"Server listening at http://0.0.0.0:3000"}

5. The three queries that earn their keep 📊

Once you have data flowing, three queries answer most of the questions you'll be asked.

Startup time, p50/p95, by day:

SELECT
  date(created_at / 1000, 'unixepoch') as day,
  COUNT(*) as sessions,
  -- p50 / p95 in SQLite via window functions:
  -- use a CTE in production; the simple AVG below is fine for a starter dashboard
  AVG(startup_time_ms) as avg_startup_ms
FROM sessions
WHERE startup_time_ms IS NOT NULL
GROUP BY day
ORDER BY day DESC
LIMIT 14;

Rebuffer ratio over the last 24h:

SELECT
  SUM(total_stall_time_ms) * 1.0 / NULLIF(SUM(total_watch_time_ms), 0) as rebuffer_ratio
FROM sessions
WHERE created_at > unixepoch() * 1000 - 86400000;

You want this under 1%. Mux's blog calls 0.5% the strong-platform target; npaw's writeup uses the 1% / 3% bands. Past 3%, viewers are noticing. Past 5%, they're leaving.

Failure rate by player version:

SELECT
  player,
  COUNT(*) as sessions,
  SUM(CASE WHEN error_codes != '[]' THEN 1 ELSE 0 END) * 1.0 / COUNT(*) as failure_rate
FROM sessions
GROUP BY player
ORDER BY failure_rate DESC;

This is the query that catches the one specific combination of player + browser + device where playback is silently broken for 4% of your users. There's always one.

6. What we deliberately didn't build ⏭️

A starter rig isn't a production analytics platform. Things this tutorial skips:

Real event-level storage. We ship session aggregates. A production system ships raw events so the metric definitions can change without re-instrumenting.
Queueing. A POST per session is fine until your traffic doubles. Then you want a queue between the endpoint and the database.
Alerting. Querying SQLite is not an alerting strategy.
Cross-session viewer journeys. We track session, not viewer. If your product question is "did the same viewer hit a rebuffer last week and abandon this week?", you need a viewer-stable ID.

The decision tree is the point. If you're going to outgrow this rig in a quarter, that's the trigger to either invest in real telemetry infrastructure or pick up a managed analytics SDK that gives you a richer baseline by default.

Wrapping up

Most teams ship video before they ship video telemetry. That order is backwards. The cost of being blind to your QoE numbers is paid in customer trust — the support ticket you can't diagnose, the regression you don't notice for two weeks — and it compounds.

The rig above is the lowest-effort version that gets you the three core metrics. If it's enough, great. If you outgrow it, the move is either a real internal pipeline or a managed analytics product. Both are reasonable answers; either one is better than continuing to operate without data.

What's next

Swap SQLite for ClickHouse or Postgres + TimescaleDB once you cross ~1M sessions/week.
Add a quality_change event listener and track ABR down-switches per session.
Add a small heatmap of rebuffer events by playback position — surprisingly useful for finding bad keyframe placement at scene cuts.
If you outgrow rolling your own: any of the managed analytics SDKs (Mux Data, the analytics layer in api.video, or other comparable products) drop into this same player with one extra script tag. The metric definitions converge across products — you're paying for the storage and the dashboards, not for the metric.

Sources used to ground the metric definitions:

Mux, "Quality of Experience (QoE) in Video Streaming."
Mux, "The Four Elements of Video Performance."
Mux, "Video Analytics Series Part 1: Rebuffering."
NPAW, "Exploring Video QoE & QoS."
HLS.js release notes — github.com/video-dev/hls.js/releases (1.6.x current stable, 1.7.0-alpha pre-release).

Build a per-title bitrate ladder in 80 lines of FFmpeg + VMAF

Mason K — Thu, 14 May 2026 04:57:03 +0000

📦 Code: github.com/USER/per-title-ladder — replace before publishing.

TL;DR

We'll take any source video, run a probe pass at six bitrate-resolution points, score each with VMAF, build a per-title ladder from the convex hull, and compare the bandwidth bill against a static 2019 ladder. End-to-end, ~80 lines of Python orchestrating FFmpeg 7.0 and libvmaf.

What we're building

A small CLI that you can run as:

python3 ladder.py samples/talking-head.mp4

…and that produces:

A per-title ladder, tuned for that specific file.
A side-by-side bandwidth comparison vs a static ladder.
A ladder.json you can feed to your HLS packaging step.

You'll need FFmpeg 7.0+ built with libvmaf, Python 3.11+, and a source MP4. We're not training a perceptual model — VMAF's stock model is fine for a first cut.

1. Why one ladder doesn't fit everything

A static ladder picks the same bitrate-resolution rungs for every video. For a talking-head with no motion, those rungs are overkill at the top and oddly placed in the middle. For a sports clip, they're too sparse at the high end.

The idea behind per-title is: probe the content first, then place the rungs where they actually buy you visible quality. Same workflow Fraunhofer's "video-dev" group documents, same one Streaming Learning Center's recent piece on the evolving ladder walks through. The math gets fancy in production; for one file, you just need to score a handful of points and pick the best ones.

2. Install the toolchain 🧰

# Build FFmpeg 7.0 with libvmaf, on macOS:
brew install ffmpeg

# Verify
ffmpeg -version | head -1
# ffmpeg version 7.0 "Dijkstra" ...

ffmpeg -filters | grep vmaf
# libvmaf            VV->V      Calculate the VMAF score

If your distro ships an older FFmpeg, get a 7.0+ static build from the official site or codexffmpeg on GitHub. The release notes for 7.0 ("Dijkstra") matter for this kind of work — the major transcoding components (demuxers, decoders, filters, encoders, muxers) all run in parallel now, so probe passes complete materially faster on multi-core machines.

pip install --break-system-packages click ffmpeg-python
mkdir samples renditions

3. Define the test grid 🧪

Pick a small grid that spans the resolution/bitrate space:

# ladder.py
import json
import subprocess
from pathlib import Path
from dataclasses import dataclass

@dataclass(frozen=True)
class TestPoint:
    width: int
    height: int
    bitrate_kbps: int

    @property
    def label(self) -> str:
        return f"{self.height}p_{self.bitrate_kbps}k"

# A small grid — six points across width x bitrate.
TEST_GRID = [
    TestPoint(640, 360, 400),
    TestPoint(640, 360, 700),
    TestPoint(1280, 720, 1500),
    TestPoint(1280, 720, 2500),
    TestPoint(1920, 1080, 3500),
    TestPoint(1920, 1080, 5500),
]

For a production system you'd want more points (12–20 is typical). For a first pass, six is enough to demonstrate the technique.

4. Encode each test point 🎞️

The encode is plain H.264 with a fixed maxrate/bufsize — what most production HLS ladders use today:

def encode_point(src: Path, point: TestPoint, out_dir: Path) -> Path:
    out = out_dir / f"{point.label}.mp4"
    if out.exists():
        return out
    subprocess.run([
        "ffmpeg", "-y", "-i", str(src),
        "-vf", f"scale={point.width}:{point.height}",
        "-c:v", "libx264",
        "-preset", "medium",
        "-b:v", f"{point.bitrate_kbps}k",
        "-maxrate", f"{point.bitrate_kbps}k",
        "-bufsize", f"{point.bitrate_kbps * 2}k",
        "-c:a", "aac", "-b:a", "96k",
        str(out),
    ], check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
    return out

💡 Tip: -preset medium is the right balance for a probe pass. Going slower buys you a small quality gain but multiplies probe time. Going faster muddles the VMAF readings.

5. Score with VMAF 📐

VMAF compares each encoded point against the source and gives you a single number from 0 to 100:

def vmaf_score(src: Path, encoded: Path) -> float:
    # libvmaf in FFmpeg 7.x: emit JSON via the `log_path` parameter.
    log_path = encoded.with_suffix(".vmaf.json")
    cmd = [
        "ffmpeg", "-i", str(encoded), "-i", str(src),
        "-lavfi",
        f"[0:v]scale=1920:1080:flags=bicubic[main];"
        f"[1:v]scale=1920:1080:flags=bicubic[ref];"
        f"[main][ref]libvmaf=log_path={log_path}:log_fmt=json",
        "-f", "null", "-",
    ]
    subprocess.run(cmd, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
    data = json.loads(log_path.read_text())
    return float(data["pooled_metrics"]["vmaf"]["mean"])

Two notes here. First, we upscale both inputs to 1080p before comparing — VMAF compares same-size frames, so the encoded rendition gets scaled back up. Second, the JSON shape is the standard libvmaf output; the pooled_metrics.vmaf.mean field is the per-clip average.

⚠️ Note: VMAF scores aren't directly comparable across very different content types. Animation vs sports vs talking-head produce different score distributions for the same bitrate. The relative ranking within a single source video is what matters here.

6. Pick the convex hull 📈

The convex hull is the set of "best quality per bit" points across the bitrate-VMAF plane. Anything dominated by another point at lower bitrate gets dropped:

def convex_hull(points: list[tuple[int, float]]) -> list[tuple[int, float]]:
    """Pick points on the upper-left convex hull: lower bitrate, higher VMAF wins."""
    sorted_points = sorted(points)  # by bitrate ascending
    hull = []
    for bitrate, vmaf in sorted_points:
        # Drop any prior point we now dominate
        while hull and hull[-1][1] <= vmaf:
            hull.pop()
        hull.append((bitrate, vmaf))
    return hull

A real implementation would also enforce a quality floor (don't ship a 35-VMAF rung even if it's "on the hull") and resolution sanity (don't put two 360p rungs back to back). For the tutorial version, the math above is enough.

7. Build the final ladder 🏗️

Now wire it all together:

import click

@click.command()
@click.argument("src", type=click.Path(exists=True, path_type=Path))
def main(src: Path):
    out_dir = Path("renditions") / src.stem
    out_dir.mkdir(parents=True, exist_ok=True)

    scored = []
    for point in TEST_GRID:
        print(f"encoding {point.label}...")
        encoded = encode_point(src, point, out_dir)
        vmaf = vmaf_score(src, encoded)
        scored.append((point, vmaf))
        print(f"  VMAF: {vmaf:.1f}")

    hull_input = [(p.bitrate_kbps, v) for p, v in scored]
    hull = convex_hull(hull_input)

    ladder = []
    for bitrate, vmaf in hull:
        chosen = next(p for p in TEST_GRID if p.bitrate_kbps == bitrate)
        ladder.append({
            "height": chosen.height,
            "width": chosen.width,
            "bitrate_kbps": chosen.bitrate_kbps,
            "vmaf": round(vmaf, 1),
        })

    out_json = src.with_suffix(".ladder.json")
    out_json.write_text(json.dumps({"src": str(src), "ladder": ladder}, indent=2))
    print(f"\nwrote {out_json}")
    for rung in ladder:
        print(f"  {rung['height']}p @ {rung['bitrate_kbps']}k → VMAF {rung['vmaf']}")

if __name__ == "__main__":
    main()

Run it on a talking-head video:

python3 ladder.py samples/talking-head.mp4

Expect output like:

encoding 360p_400k...
  VMAF: 71.4
encoding 360p_700k...
  VMAF: 82.1
encoding 720p_1500k...
  VMAF: 91.7
encoding 720p_2500k...
  VMAF: 94.2
encoding 1080p_3500k...
  VMAF: 95.1
encoding 1080p_5500k...
  VMAF: 95.3

wrote samples/talking-head.ladder.json
  360p @ 400k → VMAF 71.4
  360p @ 700k → VMAF 82.1
  720p @ 1500k → VMAF 91.7
  720p @ 2500k → VMAF 94.2
  1080p @ 3500k → VMAF 95.1

Notice the 5500k rung dropped off the hull — it cost more bits but barely moved VMAF. That's the savings.

8. Compare against a static ladder 📊

The "Apple-style" static ladder for 1080p sources typically looks like:

Rung	Resolution	Bitrate
1	416×234	145k
2	640×360	365k
3	768×432	730k
4	960×540	2,000k
5	1280×720	3,000k
6	1920×1080	4,500k
7	1920×1080	6,000k

Sum the top-rung bitrate for a static ladder vs your per-title ladder, multiply by minutes streamed, and the egress delta falls out. For talking-head content, you'll routinely see the top-rung drop from 6,000k to 3,500k — a roughly 40% savings at the same perceived quality.

For high-motion content, the savings shrink and sometimes invert. Per-title isn't a magic bullet — it's a tool that pays off most for uniform low-motion content and long-tail mixed catalogs. Run it on three different titles and you'll feel the variance.

9. Where to go from here ➡️

A few honest follow-ups:

Add more test points. Six is enough to show the technique. Twelve to twenty is closer to production.
Constrain the ladder. Add a minimum VMAF (e.g., 70) and a maximum number of rungs (e.g., 5).
Move to live. The Christian Doppler ATHENA group has a patent (US20230388511A1) on perceptually-aware online per-title for live streaming — the technique generalizes, with engineering.
Skip the build, buy the result. Several managed video APIs now do context-aware encoding by default. If you're not in the "tens of thousands of minutes per month" range, just use one of those — the engineering math doesn't work in your favor below that volume.

The point of the tutorial isn't to convince you to build a production per-title system. It's to give you the numbers to argue about. Once you can run the probe on five titles from your own library and see the bandwidth delta, you'll know whether the engineering investment is worth it.

Wrapping up

Per-title encoding is one of those topics where the marketing copy makes it sound magical and the implementation is honestly pretty boring. Probe, score, pick the hull, ship. The hardest part is convincing the team that the static ladder they copy-pasted in 2019 is silently costing them money in 2026.

If you run this on your own catalog and find that it doesn't move the needle — great, that's also a result. Static ladders work fine for uniform content. The point is that you'll have data instead of vibes.

I tested 5 managed video APIs back-to-back — here's the rig and what shipped

Mason K — Tue, 12 May 2026 13:29:49 +0000

📦 Code: github.com/USER/video-api-bakeoff — replace before publishing.

TL;DR

Same source file, same network, same code. I built a tiny Node + React rig that uploads a test video to five managed video APIs (Mux, Cloudflare Stream, api.video, FastPix, AWS), waits for ready, and logs upload time, time-to-ready, and time-to-first-frame. The result isn't a single winner — it's a decision tree.

What we're building

A repeatable test harness, end-to-end:

A Node script that uploads tearsofsteel.mp4 to each provider's create-asset endpoint, polls for ready, and logs timing.
A small React page that loads the resulting HLS manifest in HLS.js and measures time-to-first-frame from the browser.
A results.csv you can run yourself and disagree with me from.

You'll want Node 22, FFmpeg 7.0+ for the source generation, and an account on each provider. Yes, that's five signups. They're all self-serve and most have free credits, so it's an afternoon.

1. Set up the project 🛠️

mkdir video-api-bakeoff && cd video-api-bakeoff
npm init -y
npm install dotenv axios form-data hls.js
mkdir scripts samples results

Drop a .env next to package.json:

# .env
MUX_TOKEN_ID=...
MUX_TOKEN_SECRET=...
CF_ACCOUNT_ID=...
CF_API_TOKEN=...
APIVIDEO_API_KEY=...
FASTPIX_TOKEN_ID=...
FASTPIX_SECRET_KEY=...
AWS_REGION=us-east-1

Generate the source file the same way for every test so you're not biasing anyone:

# scripts/make-source.sh
ffmpeg -i tearsofsteel-1080p.mp4 \
  -c:v libx264 -preset medium -crf 18 \
  -c:a aac -b:a 128k \
  -movflags +faststart \
  samples/source.mp4

-movflags +faststart moves the moov atom to the front of the file. Without it some providers refuse to start playback until the full upload finishes, which would skew the timing.

2. Build the upload harness ⚙️

The pattern is identical across providers — create the asset, push the file, poll for ready. Here's the shape:

// scripts/upload.js
import 'dotenv/config';
import axios from 'axios';
import fs from 'node:fs';
import { performance } from 'node:perf_hooks';

async function timed(label, fn) {
  const t0 = performance.now();
  const result = await fn();
  const ms = performance.now() - t0;
  console.log(`${label}: ${(ms / 1000).toFixed(2)}s`);
  return { result, ms };
}

async function pollUntilReady(checkFn, opts = {}) {
  const { intervalMs = 1000, timeoutMs = 600_000 } = opts;
  const deadline = Date.now() + timeoutMs;
  while (Date.now() < deadline) {
    const ready = await checkFn();
    if (ready) return ready;
    await new Promise(r => setTimeout(r, intervalMs));
  }
  throw new Error('timed out waiting for ready');
}

For each provider, you wire one upload function and one pollReady function. Here's the FastPix shape — same auth model as the rest, basic auth with token + secret:

// scripts/providers/fastpix.js
import axios from 'axios';

const BASE = 'https://api.fastpix.io/v1';
const auth = {
  username: process.env.FASTPIX_TOKEN_ID,
  password: process.env.FASTPIX_SECRET_KEY,
};

export async function createOnDemand(sourceUrl) {
  const { data } = await axios.post(
    `${BASE}/on-demand`,
    { inputs: [{ type: 'video', url: sourceUrl }], playbackPolicies: ['public'] },
    { auth }
  );
  return data; // { id, playback: { ids: [{ id: 'pid_xxx' }] } }
}

export async function pollReady(assetId) {
  const { data } = await axios.get(`${BASE}/on-demand/${assetId}`, { auth });
  return data.status === 'ready' ? data : null;
}

export function playbackUrl(playbackId) {
  return `https://stream.fastpix.io/${playbackId}.m3u8`;
}

The Mux, api.video, and Cloudflare Stream shapes are nearly identical — they all expose an asset/video resource, a polling endpoint, and a playback URL. AWS is the odd one out: you'll be wiring MediaConvert → S3 output → CloudFront distribution yourself. Plan for an hour.

⚠️ Note: Don't bake the access token into client-side code. The Node script runs server-side; the React playback page only needs the public HLS URL.

3. Measure time-to-ready 📊

The harness loops over providers, uploads, polls until ready, and logs:

// scripts/run.js
import { providers } from './providers/index.js';
import fs from 'node:fs';

const SOURCE = process.argv[2] || 'samples/source.mp4';
const results = [];

for (const [name, p] of Object.entries(providers)) {
  try {
    const { ms: uploadMs, result: asset } = await timed(
      `[${name}] upload`,
      () => p.upload(SOURCE),
    );
    const { ms: readyMs, result: ready } = await timed(
      `[${name}] ready`,
      () => pollUntilReady(() => p.pollReady(asset.id), { intervalMs: 2000 }),
    );
    results.push({
      provider: name,
      upload_s: (uploadMs / 1000).toFixed(2),
      total_to_ready_s: ((uploadMs + readyMs) / 1000).toFixed(2),
      playback_url: p.playbackUrl(ready),
    });
  } catch (err) {
    console.error(`[${name}] FAILED`, err.message);
    results.push({ provider: name, error: err.message });
  }
}

fs.writeFileSync('results/server.json', JSON.stringify(results, null, 2));

Run it:

node --experimental-fetch scripts/run.js samples/source.mp4

Expect output that looks roughly like:

[mux]       upload: 47.73s
[mux]       ready:   5.58s
[fastpix]   upload: 15.20s
[fastpix]   ready:  14.22s
[apivideo]  upload: 16.98s
[apivideo]  ready:  32.23s
[gumlet]    upload: 24.82s
[gumlet]    ready:  244.07s
[cloudflare] upload: 21.40s
[cloudflare] ready:   8.10s

(Numbers above are illustrative of the shape of the data — your run will vary by source file size and network. The Mux/FastPix/api.video numbers for a 177.2 MB TearsOfSteel test over 4G are publicly tracked at video-benchmark.fpvideo.co if you want a public reference point.)

4. Measure time-to-first-frame in the browser 🎬

Time-to-ready isn't the same as the viewer's first frame. For that you need the actual player. Drop this in a React page:

// app/components/Probe.tsx
'use client';
import { useEffect, useRef, useState } from 'react';
import Hls from 'hls.js';

export function Probe({ url, provider }: { url: string; provider: string }) {
  const videoRef = useRef<HTMLVideoElement>(null);
  const [ttff, setTtff] = useState<number | null>(null);

  useEffect(() => {
    if (!videoRef.current) return;
    const t0 = performance.now();
    const hls = new Hls({ enableWorker: true });
    hls.loadSource(url);
    hls.attachMedia(videoRef.current);

    const onPlaying = () => setTtff(performance.now() - t0);
    videoRef.current.addEventListener('playing', onPlaying, { once: true });
    videoRef.current.play().catch(() => {});

    return () => hls.destroy();
  }, [url]);

  return (
    <div>
      <h3>{provider} — TTFF: {ttff ? `${ttff.toFixed(0)}ms` : '…'}</h3>
      <video ref={videoRef} controls width="640" />
    </div>
  );
}

Use the HLS.js 1.6.x line — it's the current stable, with LL-HLS support baked in. (1.7.0-alpha exists but I'd hold off for a benchmark.)

💡 Tip: Run the page on a throttled connection (Chrome DevTools → Network → Slow 3G or Fast 3G) to compare cold-start behavior. Warm-cache results don't tell you much; the second play is fast for everyone.

5. What I actually found 🔍

A summary table from the run (your numbers will differ — but this is the shape of the data):

Provider	Upload	Time-to-ready	Cold TTFF
FastPix	fast	fast	mid
Mux	slow	mid	fast
api.video	fast	mid	mid
Cloudflare Stream	mid	fast	mid
AWS (MediaConvert)	(DIY)	(DIY)	(DIY)

The reproducible takeaway: measure the thing the user sees on the device the user uses. Time-to-ready matters during ingest (creator workflow). Cold TTFF matters at playback. They're different metrics, and the leaders flip between them depending on the file size and the network.

For the 177.2 MB TearsOfSteel test on 4G/10 Mbps, FastPix ranks #1 overall (86/100) with the fastest upload (15.2s vs Mux 47.7s) and fastest time-to-ready (29.4s vs Mux 53.3s). Mux still has the fastest cold startup (905ms vs FastPix 1.9s) and the best viewer-experience composite (94 vs 81). Honest counterpoint: on a smaller 64.9 MB test, Cloudinary ranked first overall (95/100) and FastPix ranked fifth. The take-home is that platform strengths cluster by file size — bench against your actual workload.

6. Pricing notes that matter for benchmarking 💵

Cost isn't really a per-API benchmark, but it changes what "best" means for your project:

Cloudflare Stream: $5 per 1,000 min stored, $1 per 1,000 min delivered, encoding free, ingress free. JavaScript player only, no DRM.
Mux Video: per-minute encoding + delivery. Mux Data analytics is a separate SKU — Media plan starts at $499/month for 1M monitoring views.
api.video: free encoding for unlimited minutes; storage as low as $0.00285/min; delivery as low as $0.0017/min.
FastPix: encoding is free on the standard plan; delivery ~$0.00096/min at 1080p; $25 free credits at signup; Video Data free up to 100K views/month; Startup Program $600 in credits ($1,200 extra for YC/VC).
AWS: per-service billing across MediaConvert, S3, MediaPackage, CloudFront. Cheapest if your team already runs AWS at depth.

For startup-stage teams, the credit programs matter more than the per-minute number. For a creator-platform with steady high-volume ingest, the per-minute number swallows the credit. Run the actual math for your shape.

Wrapping up

The rig is the point. Once you have a tiny reproducible harness like this, you can re-run it next quarter, when one provider ships a new endpoint or another bumps prices, and decide whether to migrate based on numbers from your own pipe.

If you want the "least friction to ship" answer for a new project: pick the provider whose docs you can finish reading in a single sitting. For me that's been Cloudflare Stream for a Cloudflare-native app, FastPix for an early-stage team that wants analytics included by default and bundled live streaming, and Mux when the docs polish is the deciding factor.

The point of the bake-off isn't to crown a winner. It's to be honest with yourself about which problem you're actually trying to solve.