DEV Community: Mason K

Audit what every rung of your HLS ladder costs per viewer-hour

Mason K — Fri, 04 Sep 2026 12:53:45 +0000

TL;DR

Your bitrate ladder is a price list, and ABR buys the most expensive rung it can. We'll write a ~60-line Node script that fetches any HLS master playlist and prints what each rendition costs per viewer-hour, then look at two fixes: reshaping the ladder with FFmpeg, and moving delivery to per-minute pricing where bitrate drift can't inflate the bill.

Delivery bills usually grow for a boring reason: the average delivered bitrate crept up and nobody noticed, because nobody prices the ladder. Let's price the ladder.

Tools for this build: Node 22 (any Node 18+ works, we only need global fetch), FFmpeg 7.0+, and any HLS stream you can reach, including your own.

1. Grab a master playlist 🎯

Every HLS stream starts from a master playlist that declares each rendition and its BANDWIDTH in bits per second. That attribute is the whole audit surface:

curl -s https://your-cdn.example.com/v1/master.m3u8 | head -20

#EXTM3U
#EXT-X-STREAM-INF:BANDWIDTH=6221600,RESOLUTION=1920x1080,CODECS="avc1.640028,mp4a.40.2"
1080p_high/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=4521600,RESOLUTION=1920x1080,CODECS="avc1.640028,mp4a.40.2"
1080p/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=3021600,RESOLUTION=1280x720,CODECS="avc1.64001f,mp4a.40.2"
720p/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1521600,RESOLUTION=854x480,CODECS="avc1.64001e,mp4a.40.2"
480p/index.m3u8

BANDWIDTH is the peak declared rate, so this audit is a worst-case price. That's the right case to price, because on good networks ABR sits on your top rung all day.

2. The audit script 🛠️

// ladder-audit.js
// Usage: node ladder-audit.js <master.m3u8 URL> <cost per GB in USD>

const [, , url, costPerGBArg] = process.argv;
const COST_PER_GB = Number(costPerGBArg ?? 0.085); // your CDN's blended $/GB

const res = await fetch(url);
if (!res.ok) {
  console.error(`Fetch failed: ${res.status} ${res.statusText}`);
  process.exit(1);
}
const manifest = await res.text();

const rungs = [];
const lines = manifest.split("\n");
for (let i = 0; i < lines.length; i++) {
  if (!lines[i].startsWith("#EXT-X-STREAM-INF:")) continue;
  const attrs = lines[i].slice("#EXT-X-STREAM-INF:".length);
  const bandwidth = Number(/BANDWIDTH=(\d+)/.exec(attrs)?.[1] ?? 0);
  const resolution = /RESOLUTION=([\dx]+)/.exec(attrs)?.[1] ?? "audio-only";
  rungs.push({ bandwidth, resolution, uri: lines[i + 1]?.trim() });
}

console.log(`\nLadder audit for ${url}`);
console.log(`Assumed delivery cost: $${COST_PER_GB}/GB\n`);
console.table(
  rungs
    .sort((a, b) => b.bandwidth - a.bandwidth)
    .map((r) => {
      const gbPerHour = (r.bandwidth * 3600) / 8 / 1e9;
      return {
        resolution: r.resolution,
        mbps: (r.bandwidth / 1e6).toFixed(2),
        "GB/viewer-hour": gbPerHour.toFixed(2),
        "$/viewer-hour": (gbPerHour * COST_PER_GB).toFixed(3),
      };
    })
);

Run it against your stream with the per-GB rate from your last invoice:

node ladder-audit.js https://your-cdn.example.com/v1/master.m3u8 0.085

Ladder audit for https://your-cdn.example.com/v1/master.m3u8
Assumed delivery cost: $0.085/GB

┌─────────┬────────────┬────────┬────────────────┬────────────────┐
│ (index) │ resolution │ mbps   │ GB/viewer-hour │ $/viewer-hour  │
├─────────┼────────────┼────────┼────────────────┼────────────────┤
│ 0       │ '1920x1080'│ '6.22' │ '2.80'         │ '0.238'        │
│ 1       │ '1920x1080'│ '4.52' │ '2.03'         │ '0.173'        │
│ 2       │ '1280x720' │ '3.02' │ '1.36'         │ '0.116'        │
│ 3       │ '854x480'  │ '1.52' │ '0.68'         │ '0.058'        │
└─────────┴────────────┴────────┴────────────────┴────────────────┘

💡 Tip: to turn this into a blended number, weight each row by the share of watch time it serves. Most player analytics (hls.js LEVEL_SWITCHED events, or your QoE tool's rendition report) can give you that distribution.

3. Read the output like a bill 💸

Two questions to ask of the table:

Why does the top rung exist? If 6.2 Mbps and 4.5 Mbps 1080p rungs look identical for your content, the top one costs you 37 percent more per viewer-hour for nothing. Someone should be able to defend every rung in one sentence.
Is one ladder serving very different content? A screencast and a sports clip through the same ladder means one of them is overpaying. Fixed ladders price for your most complex content and bill you for it on your simplest.

4. Fix A: reshape the ladder with FFmpeg

The DIY fix is per-title thinking: cap each rung by quality, not just bitrate. CRF encoding with a maxrate ceiling gets you most of the way without a full VMAF pipeline:

# 1080p rung: quality-targeted, bitrate-capped
ffmpeg -i input.mp4 \
  -c:v libx264 -crf 21 -preset slow \
  -maxrate 4500k -bufsize 9000k \
  -vf scale=1920:1080 \
  -c:a aac -b:a 128k \
  1080p.mp4

Simple content lands well under the cap and you pocket the difference; complex content hits maxrate and keeps its headroom. The honest cost: doing this properly across a catalog means trial encodes, quality scoring, a queue, and monitoring. It's a pipeline, not a flag. Budget maintenance time for it like any other service you run.

5. Fix B: change the pricing model

The other fix is structural. Managed platforms mostly bill delivery per minute watched, not per GB, which makes bitrate drift financially impossible: the meter is watch time. Cloudflare Stream is $1 per 1,000 minutes delivered at any resolution. FastPix lists about $0.00096 per minute at 1080p, roughly 5.8 cents per viewer-hour, and does context-aware encoding by default, so the per-title work from Fix A is the encoder's job instead of yours. Encoding itself is free on its standard plan. Mux and api.video follow the same per-minute delivery pattern, so this section's flow works across all of them.

Getting a video into a context-aware pipeline is one call:

# Create a media from a URL (Basic auth: Access Token ID / Secret Key)
curl -X POST https://api.fastpix.com/v1/on-demand \
  -u "$FASTPIX_TOKEN_ID:$FASTPIX_SECRET" \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": [{ "type": "video", "url": "https://static.fastpix.com/fp-sample-video.mp4" }],
    "accessPolicy": "public"
  }'

{
  "success": true,
  "data": {
    "id": "0a8e6d7f-...",
    "status": "Created",
    "playbackIds": [{ "id": "9f2b1c4d-...", "accessPolicy": "public" }]
  }
}

Wait for the video.media.ready webhook, then play https://stream.fastpix.com/<playbackId>.m3u8 and point the audit script from step 2 at the master playlist it serves. Comparing the rendition bitrates you get against the fixed ladder you had is the whole evaluation, on your own content, in an afternoon. The VOD API reference covers the create-media options if you want to pin maxResolution or quality.

For calibration: in one published benchmark on a 177 MB file over 4G, FastPix's context-aware output averaged around 700 Kbps where fixed-ladder platforms delivered 1.8 to 2.5 Mbps, with zero rebuffering across all platforms tested; the same report scored Mux best on viewer experience thanks to faster cold startup. Run your own numbers; that's what the script is for.

⚠️ Note: FastPix is API-first. There's no drag-and-drop CMS included, so a non-developer content team will need a front-end built on the API.

What's next

Wire the watch-time distribution into the script and compute a real blended $/viewer-hour, then put it on a dashboard next to watch time. It should have an owner.
If you go the DIY route, the next problem is quality scoring at scale: VMAF in your encode CI, per-title thresholds, and a regression suite for encoder upgrades.

The number the script prints is rarely the number people expect. That surprise is the point.

Encode your first VVC clip with vvenc 1.14, then watch every browser refuse to play it

Mason K — Fri, 04 Sep 2026 12:53:23 +0000

TL;DR

H.266/VVC is the codec behind Brazil's TV 3.0 broadcast standard, and the open-source encoder (VVenC 1.14) is now fast enough to try on a laptop. We'll encode a real clip, mux it into MP4, and then test playback across browsers, FFmpeg, and desktop players to map exactly where VVC does and doesn't work in 2026. Spoiler: your <video> tag says no.

Why bother, if browsers don't play it? Because VVC just got its first massive real-world deployment (Brazilian broadcast TV mandates it, layered with MPEG-5 LCEVC), and "I've actually run the encoder" beats "I read a press release" in every architecture discussion you'll have about codecs this year.

What we're using: VVenC 1.14 (released January 2026), FFmpeg 7.1+ (for a native VVC decoder; FFmpeg 8.0 works too), and optionally GPAC's MP4Box.

1. Get the encoder 🛠️

VVenC is Fraunhofer HHI's open VVC encoder, BSD-3-Clause-Clear licensed. On macOS, Homebrew carries it; everywhere else the cmake build is quick:

# macOS
brew install vvenc

# or build from source (Linux/macOS)
git clone https://github.com/fraunhoferhhi/vvenc.git
cd vvenc
make install-release
# binaries land in bin/release-static/

Check it:

vvencapp --version

vvencapp: Fraunhofer VVC Encoder ver. 1.14.0

2. Prepare a test clip 🎬

vvencapp eats raw YUV or y4m. Take any 10 to 20 second 1080p clip and unwrap it with FFmpeg:

# clip.mp4 -> raw y4m (this file will be big, that's normal)
ffmpeg -i clip.mp4 -t 15 -pix_fmt yuv420p10le -strict -1 clip_10bit.y4m

💡 Tip: VVC's sweet spot assumes 10-bit. Use yuv420p10le rather than 8-bit so you're testing the codec the way broadcast deploys it.

3. Encode with vvencapp

vvencapp --preset medium -i clip_10bit.y4m --qp 32 -o clip.266

You'll see per-frame progress and a summary along these lines (numbers are from my run on a laptop, yours will differ):

vvencapp [info]: start encoding...
POC    30 TId: 0 (B-SLICE, QP 35)
...
vvenc [info]: stats summary: frames= 375, bitrate= 2861 kbps
vvenc [info]: Total Time: 214.5 sec [user], 3.5 fps

Two things to notice. First, --preset medium on a modern laptop runs at single-digit fps for 1080p. That's dramatically better than the reference software (Fraunhofer quotes speedups of 20x to 2,400x for VVenC over VTM depending on preset), but it's still far from realtime. Second, try --preset faster and watch fps climb while the bitrate for the same QP rises; that's the whole encoder trade-off surface in two runs.

For a size comparison that's actually yours, encode the same y4m with x265 and SVT-AV1 and put the outputs side by side:

ffmpeg -i clip_10bit.y4m -c:v libx265 -crf 24 -preset slow clip_h265.mp4
ffmpeg -i clip_10bit.y4m -c:v libsvtav1 -crf 32 -preset 6 clip_av1.mp4
ls -lh clip.266 clip_h265.mp4 clip_av1.mp4

I'm deliberately not printing my byte counts here: run it on your own footage, because content complexity dominates codec comparisons and your clip is the only benchmark that matters for you.

4. Mux it into MP4 📦

A raw .266 bitstream won't go far. Recent FFmpeg and GPAC both know how to put VVC in MP4:

# FFmpeg 7.1+
ffmpeg -i clip.266 -c copy clip_vvc.mp4

# or GPAC
MP4Box -add clip.266 -new clip_vvc.mp4

5. Inspect what you made 🔬

Before playing anything, confirm the file is what you think it is. ffprobe from a recent FFmpeg reads VVC fine:

ffprobe -hide_banner clip_vvc.mp4

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'clip_vvc.mp4':
  Duration: 00:00:15.00, start: 0.000000, bitrate: 2894 kb/s
  Stream #0:0: Video: vvc (vvc1 / 0x31637676), yuv420p10le(tv),
    1280x720, 30 fps, 30 tbr, 15360 tbn

Two details worth noticing in that output. The sample entry is vvc1, the MP4 signaling browsers would need to recognize before anything else could happen; and the pixel format survived as yuv420p10le, confirming you exercised the 10-bit path broadcast actually uses. If you ever need to debug VVC support claims on a device, this vvc1 string is what its media capabilities API has to acknowledge.

# bonus: count how the encoder spent its frames
ffprobe -v error -select_streams v:0 -show_entries frame=pict_type \
  -of csv clip_vvc.mp4 | sort | uniq -c

That little histogram of I/P/B frames is the GOP structure you configured, verified from the bitstream rather than assumed.

6. The playback tour, from success to failure

FFmpeg's native decoder (works):

ffplay clip_vvc.mp4

FFmpeg has shipped a native VVC decoder in recent releases, and current VLC builds play VVC too. Desktop players are fine.

Browsers (do not work): drop this in a console or a test page:

// vvc-support-check.js
const v = document.createElement("video");
console.log("VVC :", v.canPlayType('video/mp4; codecs="vvc1.1.6.L93.B0"') || "(empty = no)");
console.log("AV1 :", v.canPlayType('video/mp4; codecs="av01.0.08M.08"') || "(empty = no)");
console.log("HEVC:", v.canPlayType('video/mp4; codecs="hvc1.1.6.L93.B0"') || "(empty = no)");

VVC : (empty = no)
AV1 : probably
HEVC: maybe   # hardware-dependent

As of mid-2026 no major browser ships VVC decode. Chrome, Firefox, and Edge decode AV1 natively, and AV1 hardware decode has landed across Apple, Qualcomm, and MediaTek chips. That asymmetry is the entire story of why your web ladder is H.264 + AV1 (+ HEVC for some Apple paths) and not VVC.

7. So where does VVC actually run? 📡

Environment	VVC in 2026	Why
Browsers	No support in any major browser	Patent-pool ambiguity, AV1 already shipped royalty-free
Broadcast TV (Brazil TV 3.0 / DTV+)	Mandated, on air since Aug 2025	Standard requires VVC + LCEVC decode in new sets
Smart TV SoCs	Arriving in new chipsets	Pulled along by broadcast mandates
FFmpeg / VLC	Decode works	Native decoder in recent releases
Your encode box	Works today via VVenC / libvvenc	BSD-licensed encoder, practical speeds

The interesting technical wrinkle in Brazil's stack is MPEG-5 LCEVC: not a standalone codec but an enhancement layer over a base codec. V-Nova's TV 3.0 trial figures claim 4K HDR at 8.08 Mbps with VVC+LCEVC versus 12.54 Mbps for VVC alone (vendor numbers, take accordingly). The layered idea is worth knowing even if you never touch VVC, because it can ride on codecs browsers already play.

What's next

Point the same workflow at LCEVC and see the enhancement-layer model up close (V-Nova publishes SDKs and the TV 3.0 write-up at v-nova.com).
Watch chipset spec sheets, not browser release notes, if you ship connected-TV apps. TV SoCs are where VVC decode will reach your users first.
Keep your ladder boring: H.264 floor, AV1 where advertised. The experiment you just ran is exactly why.

Sources: VVenC releases, V-Nova TV 3.0, Streaming Media, State of Streaming Codecs 2026.

Build a live latency HUD with ffmpeg, a burned-in clock, and hls.js

Mason K — Fri, 04 Sep 2026 12:52:58 +0000

TL;DR

We'll measure live stream latency two honest ways at once: a UTC clock burned into the video (true glass-to-glass) and EXT-X-PROGRAM-DATE-TIME math surfaced by hls.js (hls.latency). You'll end up with a local test stream, a player page with a latency HUD, and a clear picture of where your seconds actually go.

Most teams quote their protocol's brochure latency ("HLS is ~30s, LL-HLS is single digits") and have never measured their own. Let's fix that locally in about twenty minutes, with tools you already have: FFmpeg 7.0+ and hls.js 1.6+.

1. Generate a live stream with a clock burned in 🕐

The oldest trick in streaming QA is still the best: put the current time inside the video. Any latency measurement scheme can lie to you; pixels can't.

# make-stream.sh
mkdir -p out
ffmpeg -f lavfi -i "testsrc2=size=1280x720:rate=30" \
  -vf "drawtext=text='UTC %{gmtime\:%H\\\:%M\\\:%S}':fontsize=64:fontcolor=white:box=1:boxcolor=black@0.6:x=(w-tw)/2:y=h-th-40" \
  -c:v libx264 -preset veryfast -tune zerolatency -g 60 \
  -f hls \
  -hls_time 2 \
  -hls_list_size 6 \
  -hls_flags delete_segments+program_date_time \
  out/live.m3u8

The two flags that matter for this build:

-hls_flags +program_date_time writes an EXT-X-PROGRAM-DATE-TIME tag per segment, which is what player-side latency math needs.
-g 60 pins the GOP to 2 seconds at 30fps, matching -hls_time 2. Segment duration can never be shorter than your GOP; this pairing is a latency decision, not a detail.

Sanity-check the playlist while it runs:

cat out/live.m3u8

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:2
#EXT-X-PROGRAM-DATE-TIME:2026-08-31T09:14:32.640+0000
#EXTINF:2.000000,
live0042.ts
...

Serve the folder (segments and the player page from the same origin keeps CORS quiet):

python3 -m http.server 8000 -d out

2. The player page with a latency HUD 📺

<!-- out/player.html -->
<!doctype html>
<video id="v" controls muted autoplay style="width:640px"></video>
<div id="hud" style="font:16px monospace; padding:8px"></div>

<script src="https://cdn.jsdelivr.net/npm/hls.js@1.6"></script>
<script>
  const video = document.getElementById("v");
  const hud = document.getElementById("hud");

  const hls = new Hls({ liveSyncDurationCount: 3 }); // default-ish: 3 segments back
  hls.loadSource("/live.m3u8");
  hls.attachMedia(video);

  setInterval(() => {
    // 1) hls.js's own PDT-derived distance from the live edge
    const reported = Number.isFinite(hls.latency) ? hls.latency.toFixed(1) : "n/a";

    // 2) manual check: wall clock minus the PDT of what's on screen
    let manual = "n/a";
    if (hls.playingDate) {
      manual = ((Date.now() - hls.playingDate.getTime()) / 1000).toFixed(1);
    }

    hud.textContent =
      `hls.latency: ${reported}s | manual PDT delta: ${manual}s | ` +
      `buffer: ${(video.buffered.length ? video.buffered.end(video.buffered.length - 1) - video.currentTime : 0).toFixed(1)}s`;
  }, 1000);
</script>

Open http://localhost:8000/player.html and you have three numbers updating live:

hls.latency: hls.js's estimate of how far you are behind the live edge, derived from PROGRAM-DATE-TIME.
manual PDT delta: the same idea computed by hand from hls.playingDate, so you can see there's no magic.
buffer: how much insurance the player is holding. Watch how it moves with the other two.

3. Now read the pixels 👀

Here's the payoff. Compare the burned-in UTC clock in the video against your machine's actual clock (date -u in a terminal next to the player):

date -u +%H:%M:%S   # run repeatedly, or: while true; do date -u; sleep 1; done

On my run, the burned-in clock trailed wall time by about 8 seconds while hls.latency said about 6. Both numbers are correct. They answer different questions:

The pixel clock measures glass-to-glass: everything from the frame being drawn, through encode, segmentation, and buffering, to render.
The PDT math measures from wherever the timestamp was stamped, which here is FFmpeg's segmenter. Encode-side delay upstream of the stamp is invisible to it.

⚠️ Note: this gap gets bigger in production. Hosted platforms typically stamp PROGRAM-DATE-TIME at ingest; Mux, for example, documents that its PDT-based latency metric reads about one second lower than true glass-to-glass for exactly this reason. Neither number is a lie. Know which one your dashboard shows.

💡 Tip: both methods assume clocks agree. If the encoder box and the viewer device disagree by two seconds of NTP drift, every number above silently absorbs it. On a single laptop you're safe; across real infrastructure, check chronyc tracking (or your platform's equivalent) before trusting sub-second readings.

4. Turn the knobs and watch the budget move 🎛️

The HUD makes latency cause-and-effect visible in a way no doc can. Try each of these and watch the numbers:

Segment duration: rerun with -hls_time 6 and -g 180. Latency roughly triples. This is the classic ~30s HLS default experience: three 6-second segments of sync distance plus buffer.
Player sync target: change liveSyncDurationCount: 3 to 1. hls.js parks you closer to the edge; rebuffer risk rises accordingly. That's the whole latency/stability trade in one line.
Pause for 30 seconds, then resume. You resume where you paused; your latency now includes your coffee. Real viewers do this constantly, which is why fleet-wide latency is a distribution, not a number.

Knob	Where it lives	Typical cost
GOP length	encoder	seconds, sets the floor for segment size
Segment duration x sync count	packager + player	the bulk of classic HLS latency
Buffer target	player	whatever it wants, silently
Viewer behavior (pause, join late)	humans	unbounded

5. Ship it as a beacon 📡

The HUD proves the math on one machine. The production version is the same three numbers sampled quietly and sent home, because fleet latency is a distribution and your desk is its most flattering sample:

// latency-beacon.js -- add next to the HUD code
function sampleLatency() {
  if (!Number.isFinite(hls.latency)) return;
  const payload = JSON.stringify({
    t: Date.now(),
    latency: +hls.latency.toFixed(2),
    target: hls.targetLatency ?? null,
    buffered: video.buffered.length
      ? +(video.buffered.end(video.buffered.length - 1) - video.currentTime).toFixed(2)
      : 0,
    // whatever session/stream IDs your analytics already use:
    sessionId: window.SESSION_ID,
  });
  navigator.sendBeacon("/telemetry/latency", payload);
}

setInterval(sampleLatency, 15_000);
window.addEventListener("pagehide", sampleLatency); // last word on the way out

sendBeacon survives tab closes and costs nothing on the render path. Fifteen-second sampling is plenty; you're charting a distribution, not tracing frames. On the backend, keep p50 and p95 per stream, and put them on the same dashboard as rebuffer rate so every latency conversation happens next to its trade-off.

💡 Tip: also record why a session drifted when you can. A paused: true flag on samples taken within a minute of a pause event separates "our pipeline got slower" from "viewers make coffee," and those need very different meetings.

What's next

Chart the percentiles. Once the beacon lands, p95 live latency across real viewers is the number to watch; it's where the spoiler complaints live.
Go low-latency for real. Partial segments and blocking playlist reload (LL-HLS) can bring the packager's share down to low single digits; the HUD you just built works unchanged and will tell you honestly whether your config delivers it.
Watch both curves. Any latency win you take, verify against rebuffer rate. They're the same dial, viewed from opposite sides.

The whole point of the HUD is that "we're about five seconds" stops being a vibe. It becomes a number you watched move.

Build a caption QA harness in Python: WER, missed entities, timing and reading rate

Mason K — Thu, 27 Aug 2026 09:24:52 +0000

TL;DR

We're building a caption evaluation harness that scores a WebVTT file on four axes instead of one:
word error rate under a fixed normalizer, missed entity rate on domain terms, median cue timing
offset, and reading rate in characters per second. Python 3.12, jiwer, whisper_normalizer,
webvtt-py. Run it on every model or vendor change.

A caption file can score 96% accurate and still be unusable. WER counts substitutions, insertions and
deletions and weighs each one the same, so "fifteen milligrams" becoming "fifty milligrams" costs
exactly as much as "the" becoming "a". It also throws away every timestamp before it starts, which
means synchronization and readability are invisible to it. Let's measure the other three things.

0. Setup 🛠️

python3 -m venv .venv && source .venv/bin/activate
pip install jiwer whisper_normalizer webvtt-py

$ pip list | grep -Ei 'jiwer|whisper|webvtt'
jiwer               <your version>
webvtt-py           <your version>
whisper-normalizer  <your version>

Pin whatever you install, and pin it in CI. The APIs below move between majors, which is exactly why
the next tip exists.

💡 Tip: jiwer.compute_measures() is gone in recent versions. It is jiwer.process_words() now,
and it returns a WordOutput dataclass. Most blog posts you will find still use the old name.

1. Parse the VTT into text plus timings

# captions.py
from dataclasses import dataclass
import webvtt

@dataclass
class Cue:
    start: float
    end: float
    text: str

    @property
    def duration(self) -> float:
        return self.end - self.start

    @property
    def lines(self) -> list[str]:
        return self.text.split("\n")

    @property
    def flat(self) -> str:
        return " ".join(l.strip() for l in self.lines)

    @property
    def chars_per_second(self) -> float:
        return len(self.flat) / self.duration if self.duration > 0 else float("inf")

def _to_seconds(ts: str) -> float:
    h, m, s = ts.split(":")
    return int(h) * 3600 + int(m) * 60 + float(s)

def load_vtt(path: str) -> list[Cue]:
    # keep the line breaks: they are the thing the readability check measures
    return [
        Cue(_to_seconds(c.start), _to_seconds(c.end), c.text.strip())
        for c in webvtt.read(path)
    ]

def full_text(cues: list[Cue]) -> str:
    return " ".join(c.flat for c in cues)

2. Normalize both sides with the same normalizer

This is the step people skip, and skipping it is why vendor-published WER numbers are not comparable.
Without normalization, "3" against "three" and "Dr." against "doctor" count as errors even though the
recognizer heard perfectly. The normalizer from the Whisper paper (Appendix C) is the closest thing to
a default: it strips bracketed phrases, drops fillers like "um" and "hmm", and expands contractions.

# score_wer.py
import jiwer
from whisper_normalizer.english import EnglishTextNormalizer

normalizer = EnglishTextNormalizer()

def wer(reference: str, hypothesis: str) -> dict:
    ref, hyp = normalizer(reference), normalizer(hypothesis)
    out = jiwer.process_words(ref, hyp)
    return {
        "wer": out.wer,
        "substitutions": out.substitutions,
        "deletions": out.deletions,
        "insertions": out.insertions,
        "ref_words": sum(len(r) for r in out.references),
    }

$ python -c "from score_wer import wer; print(wer('the dose is fifteen milligrams','the dose is fifty milligrams'))"
{'wer': 0.2, 'substitutions': 1, 'deletions': 0, 'insertions': 0, 'ref_words': 5}

One substitution. Same score you would get for turning "the" into "a". That is the whole problem in
one line of output.

3. Missed entity rate: the metric that actually correlates with complaints

Score the words that carry meaning separately. Your entity list is domain specific and you have to
build it: product names, people, numbers, units, jargon. Two hundred terms is plenty to start.

# score_entities.py
import re
from whisper_normalizer.english import EnglishTextNormalizer

normalizer = EnglishTextNormalizer()
NUMBER_RE = re.compile(r"\b\d+(?:\.\d+)?\b")

def _terms(text: str, lexicon: list[str]) -> list[str]:
    norm = normalizer(text)
    found = []
    # findall, not search: a term said three times should count three times, the same
    # way numbers do. Iterate a sorted list so `examples` is stable across runs.
    for t in sorted(lexicon):
        found += re.findall(rf"\b{re.escape(t)}\b", norm)
    found += NUMBER_RE.findall(norm)
    return found

def missed_entity_rate(reference: str, hypothesis: str, lexicon: list[str]) -> dict:
    ref_terms = _terms(reference, lexicon)
    hyp_terms = _terms(hypothesis, lexicon)

    remaining = list(hyp_terms)
    missed = []
    for t in ref_terms:
        if t in remaining:
            remaining.remove(t)
        else:
            missed.append(t)

    total = len(ref_terms)
    return {
        "entity_total": total,
        "entity_missed": len(missed),
        "missed_entity_rate": len(missed) / total if total else 0.0,
        "examples": missed[:10],
    }

Numbers are pulled in automatically because they are almost always high-value and almost always the
thing that gets misheard.

4. Timing: median cue offset

WER cannot see this at all. A word-perfect transcript that appears 1.5 seconds late is a bad caption
file. Compare cue start times against a reference VTT by matching on normalized text.

# score_timing.py
import statistics
from whisper_normalizer.english import EnglishTextNormalizer
from captions import Cue

normalizer = EnglishTextNormalizer()

def _tokens(text: str) -> set[str]:
    return set(normalizer(text).split())

def cue_offsets(ref: list[Cue], hyp: list[Cue], window: float = 4.0,
                min_overlap: float = 0.6) -> dict:
    """Match cues by token overlap, not string equality.

    ASR almost never segments cues the same way a human transcriber does, so requiring
    identical normalized text would match nothing on real data. We take the best
    Jaccard overlap inside a time window instead.
    """
    offsets = []
    for r in ref:
        r_tok = _tokens(r.flat)
        if not r_tok:
            continue

        best, best_score = None, 0.0
        for h in hyp:
            if abs(h.start - r.start) > window:
                continue
            h_tok = _tokens(h.flat)
            if not h_tok:
                continue
            score = len(r_tok & h_tok) / len(r_tok | h_tok)
            if score > best_score:
                best, best_score = h, score

        if best is not None and best_score >= min_overlap:
            offsets.append(best.start - r.start)

    if not offsets:
        return {"matched_cues": 0, "median_offset_s": None, "p90_abs_offset_s": None}

    absolute = sorted(abs(o) for o in offsets)
    return {
        "matched_cues": len(offsets),
        "median_offset_s": round(statistics.median(offsets), 3),
        "p90_abs_offset_s": round(absolute[int(0.9 * (len(absolute) - 1))], 3),
    }

A consistently positive median means the whole file is late, which is usually a pipeline bug (an
offset applied at the wrong stage) rather than a model problem. Random scatter with a high p90 is a
model or alignment problem. Worth distinguishing before you go blame a vendor.

5. Readability: reading rate and cue duration

No reference needed for this one, which makes it the cheapest check you can run and the one you can
apply to your entire production library today.

# score_readability.py
from captions import Cue

# Thresholds are editorial choices, not standards. These are in the range major
# streaming style guides use for adult English subtitles; set yours deliberately.
MAX_CPS = 20.0          # characters per second
MIN_DURATION = 5 / 6    # ~0.833s, below this a cue flashes
MAX_LINE_CHARS = 42

def readability(cues: list[Cue]) -> dict:
    too_fast   = [c for c in cues if c.chars_per_second > MAX_CPS]
    too_short  = [c for c in cues if c.duration < MIN_DURATION]
    too_wide   = [c for c in cues if any(len(l) > MAX_LINE_CHARS for l in c.lines)]
    rates = sorted(c.chars_per_second for c in cues if c.duration > 0)

    return {
        "cues": len(cues),
        "median_cps": round(rates[len(rates) // 2], 1) if rates else None,
        "pct_over_cps_limit": round(100 * len(too_fast) / len(cues), 1) if cues else 0.0,
        "cues_under_min_duration": len(too_short),
        "cues_over_line_length": len(too_wide),
    }

A file that dumps forty words into one four second cue has a perfect WER and cannot be read. This
function catches it in milliseconds.

6. Tie it together 📊

# run_eval.py
import json, pathlib, sys
from captions import load_vtt, full_text
from score_wer import wer
from score_entities import missed_entity_rate
from score_timing import cue_offsets
from score_readability import readability

LEXICON = set(json.loads(pathlib.Path("lexicon.json").read_text()))

def evaluate(ref_vtt: str, hyp_vtt: str) -> dict:
    ref, hyp = load_vtt(ref_vtt), load_vtt(hyp_vtt)
    ref_text, hyp_text = full_text(ref), full_text(hyp)
    return {
        "file": pathlib.Path(hyp_vtt).name,
        **wer(ref_text, hyp_text),
        **missed_entity_rate(ref_text, hyp_text, LEXICON),
        **cue_offsets(ref, hyp),
        **readability(hyp),
    }

if __name__ == "__main__":
    results = [evaluate(r, h) for r, h in zip(sys.argv[1::2], sys.argv[2::2])]
    print(json.dumps(results, indent=2))

$ python run_eval.py refs/clip01.vtt out/clip01.vtt
[
  {
    "file": "clip01.vtt",
    "wer": 0.043,
    "substitutions": 18,
    "deletions": 5,
    "insertions": 3,
    "ref_words": 604,
    "entity_total": 41,
    "entity_missed": 6,
    "missed_entity_rate": 0.146,
    "examples": ["nifedipine", "2026", "okonkwo"],
    "matched_cues": 71,
    "median_offset_s": 0.42,
    "p90_abs_offset_s": 0.91,
    "cues": 112,
    "median_cps": 17.4,
    "pct_over_cps_limit": 9.8,
    "cues_under_min_duration": 4,
    "cues_over_line_length": 7
  }
]

⚠️ Note: matched_cues will always be lower than cues, because the matcher only accepts pairs
above the overlap threshold. If it drops near zero, your hypothesis segmentation has diverged
completely from the reference, which is itself a finding worth acting on.

Read that output the way it is meant to be read: 4.3% WER looks excellent, and 14.6% of the words
that actually matter are wrong. Those two numbers describe the same file.

7. The reference set is the real work

Thirty to fifty clips, chosen for spread rather than convenience: your noisiest recording, your
heaviest accent, your densest jargon, two people talking over each other, one that is mostly silence.
Human-transcribe them once. That is the entire capital cost of the program, and it amortizes over
every model change you will ever make.

Then run the harness in CI on every vendor or model change. The reason to automate is not that the
evaluation is hard, it is that nobody rechecks captions after the initial selection, and the model
behind a hosted API changes without an announcement.

What's next

Add cpWER if you ship speaker labels. A transcript that gets every word right and swaps who said what scores zero errors on standard WER and is seriously wrong.
Track the four metrics over time on the same reference set. The interesting signal is divergence: WER improving while entity rate worsens is exactly what you see when a model gets better at common words and starts producing plausible replacements for rare ones.
Worth knowing before someone quotes it at you: there is no 99% accuracy rule in FCC Part 79. The Commission declined to set a numeric threshold and uses a de minimis test on accuracy, synchronicity, completeness and placement. The 99% figure comes from vendor contracts, not regulation. WCAG 2.2 SC 1.2.2 is likewise qualitative.

Four numbers, one reference set, and a CI job.

Frame-accurate FFmpeg trimming without re-encoding the whole file

Mason K — Thu, 27 Aug 2026 09:24:26 +0000

TL;DR

-c copy can only cut on keyframes, so your 12.4s trim starts wherever the last keyframe was.
We'll build a smart-trim
script that probes keyframe positions with ffprobe, re-encodes only the head and tail fragments,
stream copies everything between them, and concatenates the three. Frame accurate output, encoding
cost proportional to two GOPs instead of the whole file.

Tested with FFmpeg 9.0 "Lei" (released 2026-08-04) and Node 22.x. The JS is ESM, so put
"type": "module" in your package.json before running any of it. Everything here also works on
FFmpeg 7.x and 8.x; nothing we use is new.

The problem, in two commands 🎬

# fast, and wrong
ffmpeg -ss 12.4 -i input.mp4 -t 20 -c copy fast.mp4
ffprobe -v error -show_entries format=start_time,duration -of default=nw=1 fast.mp4
# start_time=0.000000
# duration=20.388000        <- we asked for 20, starting at 12.4

The clip is long by the distance from our requested start back to the previous keyframe, and every
frame in it is shifted earlier than the user asked for.

Stream copy moves compressed packets without decoding them. Most frames in a compressed stream only
describe the difference from their neighbors, so the only place you can start is a keyframe. FFmpeg
snaps back to the nearest preceding one, and your clip starts early.

# accurate, and slow on a long source
ffmpeg -ss 12.4 -i input.mp4 -t 20 -c:v libx264 -crf 20 -c:a aac slow.mp4

We want the accuracy of the second and roughly the cost of the first.

1. Look at your keyframes first

Before writing any code, find out how bad the problem is for your content:

ffprobe -v error -select_streams v:0 \
  -show_entries packet=pts_time,flags \
  -of csv=print_section=0 input.mp4 | grep 'K' | head -20

0.000000,K__
2.002000,K__
4.004000,K__
6.006000,K__

Two second GOPs here, so worst-case error is about two seconds. Screen recorders and some camera
output emit keyframes on scene change only, and there the gaps can be 30 seconds or more. That
distribution is the real spec for your trim feature.

2. Find the keyframes bracketing our cut

// keyframes.js
import { execFile } from 'node:child_process';
import { promisify } from 'node:util';
const run = promisify(execFile);

export async function keyframeTimes(input) {
  const { stdout } = await run('ffprobe', [
    '-v', 'error',
    '-select_streams', 'v:0',
    '-show_entries', 'packet=pts_time,flags',
    '-of', 'csv=print_section=0',
    input
  ], { maxBuffer: 64 * 1024 * 1024 });

  return stdout.trim().split('\n')
    .map(l => l.split(','))
    .filter(([, flags]) => flags && flags.includes('K'))
    .map(([t]) => Number(t))
    .filter(Number.isFinite)
    .sort((a, b) => a - b);
}

// first keyframe at or after t
export const kfCeil  = (kfs, t) => kfs.find(k => k >= t - 1e-6) ?? null;
// last keyframe at or before t
export const kfFloor = (kfs, t) => [...kfs].reverse().find(k => k <= t + 1e-6) ?? null;

⚠️ Note: on a two hour file this reads every packet header. It is fast (no decoding) but not free.
Cache the result per asset; keyframe positions never change for a given file.

3. Probe the source so the edges match

The re-encoded fragments have to be concat-compatible with the copied middle: same codec, resolution,
pixel format, and audio parameters. So read them off the source instead of hardcoding.

// probe.js
import { execFile } from 'node:child_process';
import { promisify } from 'node:util';
const run = promisify(execFile);

export async function probe(input) {
  const { stdout } = await run('ffprobe', [
    '-v', 'error', '-print_format', 'json',
    '-show_format', '-show_streams', input
  ]);
  const j = JSON.parse(stdout);
  const v = j.streams.find(s => s.codec_type === 'video');
  const a = j.streams.find(s => s.codec_type === 'audio');
  if (!v) throw new Error('no video stream');

  const [num, den] = (v.r_frame_rate ?? '0/1').split('/').map(Number);

  return {
    duration: Number(j.format.duration),
    width: v.width, height: v.height,
    pixFmt: v.pix_fmt,
    fps: den ? num / den : null,          // "30000/1001" -> 29.97, no eval()
    vCodec: v.codec_name,
    aCodec: a?.codec_name ?? null,
    aRate: a ? Number(a.sample_rate) : null,
    aChannels: a?.channels ?? null
  };
}

4. The three-piece cut ✂️

Given a requested [start, end]:

head: start to the next keyframe after start, re-encoded
middle: that keyframe to the last keyframe before end, stream copied
tail: that keyframe to end, re-encoded

If start and end fall inside the same GOP there is no middle, and we just re-encode the whole
(short) span.

// smart-trim.js
import { execFile } from 'node:child_process';
import { promisify } from 'node:util';
import { writeFile, mkdtemp, rm } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import path from 'node:path';
import { keyframeTimes, kfCeil, kfFloor } from './keyframes.js';
import { probe } from './probe.js';

const run = promisify(execFile);
const ff = args => run('ffmpeg', ['-hide_banner', '-loglevel', 'error', '-y', ...args]);

// The head/tail have to concat-demux with the copied middle, so they must come out
// in the same codec the source is already in. Extend this map as your library needs.
const ENCODER_FOR = {
  h264: ['-c:v', 'libx264', '-preset', 'veryfast', '-crf', '18'],
  hevc: ['-c:v', 'libx265', '-preset', 'veryfast', '-crf', '22'],
  vp9:  ['-c:v', 'libvpx-vp9', '-crf', '30', '-b:v', '0'],
  av1:  ['-c:v', 'libsvtav1', '-preset', '8', '-crf', '30'],
};

function encodeArgs(meta) {
  const venc = ENCODER_FOR[meta.vCodec];
  if (!venc) throw new Error(`no encoder mapped for source codec ${meta.vCodec}`);

  const a = [
    ...venc,
    '-pix_fmt', meta.pixFmt,
    '-vf', `scale=${meta.width}:${meta.height}`
  ];
  if (meta.aCodec) {
    a.push('-c:a', 'aac', '-ar', String(meta.aRate), '-ac', String(meta.aChannels), '-b:a', '192k');
  } else {
    a.push('-an');
  }
  return a;
}

export async function smartTrim(input, start, end, output) {
  const [kfs, meta] = await Promise.all([keyframeTimes(input), probe(input)]);
  const work = await mkdtemp(path.join(tmpdir(), 'trim-'));

  const midStart = kfCeil(kfs, start);
  const midEnd   = kfFloor(kfs, end);
  const hasMiddle = midStart !== null && midEnd !== null && midEnd - midStart > 0.05;

  try {
    if (!hasMiddle) {
      // start and end share a GOP: one short re-encode, done
      await ff(['-ss', String(start), '-i', input, '-t', String(end - start),
                ...encodeArgs(meta), output]);
      return { mode: 'reencode-only', reencodedSeconds: end - start };
    }

    const parts = [];

    if (midStart - start > 0.01) {
      const head = path.join(work, 'head.mp4');
      await ff(['-ss', String(start), '-i', input, '-t', String(midStart - start),
                ...encodeArgs(meta), head]);
      parts.push(head);
    }

    const mid = path.join(work, 'mid.mp4');
    await ff(['-ss', String(midStart), '-i', input, '-t', String(midEnd - midStart),
              '-c', 'copy', '-avoid_negative_ts', 'make_zero', mid]);
    parts.push(mid);

    if (end - midEnd > 0.01) {
      const tail = path.join(work, 'tail.mp4');
      await ff(['-ss', String(midEnd), '-i', input, '-t', String(end - midEnd),
                ...encodeArgs(meta), tail]);
      parts.push(tail);
    }

    const list = path.join(work, 'concat.txt');
    await writeFile(list, parts.map(p => `file '${p}'`).join('\n'));
    await ff(['-f', 'concat', '-safe', '0', '-i', list, '-c', 'copy',
              '-movflags', '+faststart', output]);

    return {
      mode: 'hybrid',
      reencodedSeconds: (midStart - start) + (end - midEnd),
      copiedSeconds: midEnd - midStart
    };
  } finally {
    await rm(work, { recursive: true, force: true });
  }
}

$ node -e "import('./smart-trim.js').then(m=>m.smartTrim('input.mp4',12.4,32.4,'out.mp4').then(console.log))"
{ mode: 'hybrid', reencodedSeconds: 1.982, copiedSeconds: 18.018 }

$ ffprobe -v error -show_entries format=duration -of csv=p=0 out.mp4
20.033000

Twenty seconds requested, twenty seconds delivered to within a frame, and only about two seconds of
video actually went through an encoder. The remaining eighteen were copied. (The output lands a frame
or two long because each fragment is rounded to a whole frame; if you need exact durations downstream,
clamp during the concat step.)

5. The errors you will hit 🐛

Non-monotonous DTS in output stream during concat. Almost always timestamps from the copied
middle. -avoid_negative_ts make_zero on the middle segment (already in the code above) fixes the
common case. If it persists, add -fflags +genpts to the concat step.

Audio drift at the seams. The audio encoder's frame size does not line up with the video frame
boundary, so each fragment gets a few milliseconds of padding. Three fragments means it can add up
audibly. The fix is to re-encode audio for the whole output instead of copying it in the concat step:
change the final command's -c copy to -c:v copy -c:a aac -b:a 192k. Audio re-encoding on a
20 second clip is cheap.

Could not find codec parameters on the head fragment. Your -ss landed past the end of the
stream, usually from a duration mismatch on a variable frame rate source. Normalize to constant frame
rate on ingest, or clamp end to the probed duration.

6. The cheaper fix, if you control ingest

Everything above exists because your source has unpredictable keyframes. If you also own the encode,
force them at a known interval and the problem mostly disappears:

ffmpeg -i source.mov \
  -c:v libx264 -crf 20 -preset medium \
  -force_key_frames "expr:gte(t,n_forced*2)" \
  -c:a aac -b:a 192k \
  mezzanine.mp4

Now every file in your library has a keyframe every two seconds, trimming, segmenting, and clipping
all behave consistently, and you pay a small bitrate penalty once. If you already produce an ABR
ladder you are doing this anyway, so trim against the mezzanine rather than the raw upload.

What's next

Wire smartTrim into a job queue. The head/tail encodes are short but they are still CPU work, and you do not want them running inline on a request thread.
The segment muxer (-f segment -segment_time 60 -reset_timestamps 1) has the exact same keyframe behavior: your "60 second" segments will be 60-ish. If downstream code assumes uniform durations, fix that assumption now.
Skim the FFmpeg 9.0 announcement on ffmpeg.org. Native VVC decoding and more Vulkan acceleration landed, though none of it changes the keyframe constraint, which is a property of compressed video rather than of the tool.

Anyone promising a frame accurate cut with zero re-encoding is either re-encoding behind your back or
shipping the off-by-two-seconds bug.

Build a resumable video migration worker: ledger, bounded concurrency, 429 backoff

Mason K — Thu, 27 Aug 2026 09:24:01 +0000

TL;DR

We're building a migration worker that moves a video library into a hosted video API without
tripping rate limits or double-creating assets. SQLite ledger for resumability, a semaphore with
adaptive concurrency, Retry-After aware backoff, idempotent creates, and a webhook receiver so we
never poll for status.

The naive version of this script is a for loop over a file list. It works until the API starts
returning 429, the process dies at file 3,400, and you have no idea what completed. Cloudflare's
documented global API limit is 1,200 requests per five minutes per user, and blowing through it
blocks all your API calls for the next five minutes, including the ones your production app makes.
Mux's own migration guide says the same thing in gentler words: use a background job queue, control
concurrency, and listen for webhooks instead of polling.

Let's build the thing that does that. Node 22.x, better-sqlite3, no framework.

1. The ledger 📒

Resumability is a database, not a clever loop. One row per asset, one state machine.

mkdir video-migration && cd video-migration
npm init -y
npm pkg set type=module          # everything below is ESM
npm i better-sqlite3 undici

// db.js
import Database from 'better-sqlite3';

export const db = new Database('migration.db');

db.exec(`
  CREATE TABLE IF NOT EXISTS assets (
    id            TEXT PRIMARY KEY,   -- our stable ID, also the idempotency key
    source_url    TEXT NOT NULL,
    state         TEXT NOT NULL DEFAULT 'pending',
      -- pending | inventoried | uploading | processing | ready | failed | rejected
    remote_id     TEXT,
    duration_s    REAL,
    attempts      INTEGER NOT NULL DEFAULT 0,
    retry_after   INTEGER,            -- unix ts; do not re-select before this
    last_error    TEXT,
    updated_at    INTEGER NOT NULL DEFAULT (unixepoch())
  );
  CREATE INDEX IF NOT EXISTS idx_state ON assets(state);
`);

// A real claim: select, then flip to 'uploading' in the same transaction so a second
// worker (or a restart that overlaps a running process) cannot pick up the same rows.
export const claimBatch = db.transaction((limit = 200) => {
  const rows = db.prepare(`
    SELECT * FROM assets
    WHERE state IN ('inventoried','failed')
      AND attempts < 5
      AND (retry_after IS NULL OR retry_after <= unixepoch())
    ORDER BY attempts ASC, rowid ASC
    LIMIT ?
  `).all(limit);

  const claim = db.prepare(
    `UPDATE assets SET state='uploading', updated_at=unixepoch() WHERE id = ?`
  );
  for (const r of rows) claim.run(r.id);
  return rows;
});

// Anything left 'uploading' from a previous run was interrupted, not finished.
export function releaseStale(olderThanSeconds = 900) {
  db.prepare(`
    UPDATE assets SET state='inventoried'
    WHERE state='uploading' AND updated_at < unixepoch() - ?
  `).run(olderThanSeconds);
}

export function setState(id, state, patch = {}) {
  const cols = Object.keys(patch);
  const sets = ['state = ?', 'updated_at = unixepoch()', ...cols.map(c => `${c} = ?`)];
  db.prepare(`UPDATE assets SET ${sets.join(', ')} WHERE id = ?`)
    .run(state, ...cols.map(c => patch[c]), id);
}

Note rejected as a state distinct from failed. A file that FFmpeg cannot parse is never going to
succeed, and it does not belong in a retry queue. That distinction is the difference between a
migration that finishes and one that grinds forever against twelve corrupt files.

2. Inventory before you migrate 🔍

Run a read-only pass with ffprobe first. It is cheap, it touches nothing remote, and it tells you
what kind of migration you are actually about to run.

// inventory.js
import { execFile } from 'node:child_process';
import { promisify } from 'node:util';
import { db, setState } from './db.js';

const run = promisify(execFile);

async function probe(url) {
  const { stdout } = await run('ffprobe', [
    '-v', 'error',
    '-print_format', 'json',
    '-show_format', '-show_streams',
    url
  ], { maxBuffer: 8 * 1024 * 1024 });
  return JSON.parse(stdout);
}

for (const row of db.prepare(`SELECT * FROM assets WHERE state = 'pending'`).all()) {
  try {
    const info = await probe(row.source_url);
    const v = info.streams.find(s => s.codec_type === 'video');
    if (!v) throw new Error('no video stream');
    setState(row.id, 'inventoried', { duration_s: Number(info.format.duration) });
  } catch (err) {
    setState(row.id, 'rejected', { last_error: String(err.message).slice(0, 500) });
  }
}

$ node inventory.js
$ sqlite3 migration.db "SELECT state, count(*) FROM assets GROUP BY state"
inventoried|9612
rejected|41

💡 Tip: those 41 rejects are a list a human reads once. They are not a bug in your worker.

3. A rate limiter that reacts

Fixed concurrency is a guess. What we want is a dial that goes down when the API pushes back and
creeps up when it does not.

// governor.js
export class Governor {
  constructor({ min = 1, max = 16, start = 4 } = {}) {
    this.min = min; this.max = max; this.limit = start;
    this.inFlight = 0; this.queue = [];
    this.pausedUntil = 0;
    this.okStreak = 0;
  }

  async acquire() {
    while (Date.now() < this.pausedUntil || this.inFlight >= this.limit) {
      await new Promise(r => setTimeout(r, 50));
    }
    this.inFlight++;
  }

  release() { this.inFlight--; }

  // Called on every response.
  observe(status, retryAfterHeader) {
    if (status === 429 || status === 503) {
      const secs = parseRetryAfter(retryAfterHeader) ?? 5;
      this.pausedUntil = Date.now() + secs * 1000;
      this.limit = Math.max(this.min, Math.floor(this.limit / 2));
      this.okStreak = 0;
      console.warn(`[governor] backpressure: pausing ${secs}s, concurrency -> ${this.limit}`);
    } else if (status < 400) {
      if (++this.okStreak >= 50 && this.limit < this.max) {
        this.limit++; this.okStreak = 0;
        console.info(`[governor] concurrency -> ${this.limit}`);
      }
    }
  }
}

function parseRetryAfter(h) {
  if (!h) return null;
  const n = Number(h);
  if (Number.isFinite(n)) return n;               // delta-seconds
  const t = Date.parse(h);                        // HTTP-date
  return Number.isFinite(t) ? Math.max(0, (t - Date.now()) / 1000) : null;
}

Halve on pushback, add one on sustained success. It is TCP congestion control with the serial numbers
filed off, and it works for the same reason.

4. Idempotent creates

Retrying an upload must not create a second asset. Most video APIs give you at least one of: an
idempotency key header, a client reference ID, or free-form passthrough metadata you can query later.
Use your own stable asset ID as the key so it survives a restart.

// migrate.js
import { request } from 'undici';
import { db, claimBatch, setState, releaseStale } from './db.js';
import { Governor } from './governor.js';

releaseStale();   // recover rows a previous crashed run left mid-flight

const API = process.env.VIDEO_API_BASE;      // e.g. https://api.example.com/v1
const TOKEN = process.env.VIDEO_API_TOKEN;
const gov = new Governor({ start: 4, max: 16 });

async function createAsset(row) {
  const res = await request(`${API}/videos`, {
    method: 'POST',
    headers: {
      'authorization': `Bearer ${TOKEN}`,
      'content-type': 'application/json',
      'idempotency-key': row.id            // <- the whole trick
    },
    body: JSON.stringify({
      input: row.source_url,
      metadata: { migration_ref: row.id }  // fallback lookup key
    })
  });
  const body = await res.body.json().catch(() => ({}));
  gov.observe(res.statusCode, res.headers['retry-after']);
  return { status: res.statusCode, body };
}

async function findExisting(row) {
  const res = await request(`${API}/videos?metadata[migration_ref]=${row.id}`, {
    headers: { authorization: `Bearer ${TOKEN}` }
  });
  const body = await res.body.json().catch(() => ({}));
  gov.observe(res.statusCode, res.headers['retry-after']);
  return body?.data?.[0]?.id ?? null;
}

async function migrateOne(row) {
  await gov.acquire();
  try {
    let { status, body } = await createAsset(row);

    if (status === 429 || status >= 500) {
      // back off this row specifically, so the next claimBatch does not grab it immediately
      const delay = Math.min(300, 2 ** row.attempts) + Math.random() * 5;   // jitter
      setState(row.id, 'failed', {
        attempts: row.attempts + 1,
        retry_after: Math.floor(Date.now() / 1000 + delay),
        last_error: `transient ${status}`
      });
      return;
    }
    if (status === 409) {                       // already created, provider told us
      const existing = await findExisting(row);
      if (existing) return setState(row.id, 'processing', { remote_id: existing });
    }
    if (status >= 400) {
      setState(row.id, 'rejected', {
        attempts: row.attempts + 1,
        last_error: JSON.stringify(body).slice(0, 500)
      });
      return;
    }
    setState(row.id, 'processing', { remote_id: body.id ?? body.data?.id });
  } finally {
    gov.release();
  }
}

for (;;) {
  const batch = claimBatch(200);
  if (batch.length === 0) break;
  await Promise.all(batch.map(migrateOne));
}
console.log('upload pass complete');

Realistic output on a library that hits the ceiling:

$ node migrate.js
[governor] concurrency -> 5
[governor] concurrency -> 6
[governor] backpressure: pausing 12s, concurrency -> 3
[governor] concurrency -> 4
upload pass complete

Kill it with Ctrl-C at any point and rerun. It picks up from the ledger.

5. Webhooks, not polling 🔔

processing is not ready. Polling ten thousand assets for status is how you spend your entire rate
limit budget asking questions instead of doing work. Take the event instead.

// webhook.js
import http from 'node:http';
import crypto from 'node:crypto';
import { setState } from './db.js';

const SECRET = process.env.WEBHOOK_SECRET;

function verify(rawBuffer, sig) {
  const expected = crypto.createHmac('sha256', SECRET).update(rawBuffer).digest('hex');
  // timingSafeEqual throws on length mismatch, so guard first
  const a = Buffer.from(expected), b = Buffer.from(sig ?? '', 'utf8');
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

http.createServer((req, res) => {
  const chunks = [];
  req.on('data', c => chunks.push(c));
  req.on('end', () => {
    const raw = Buffer.concat(chunks);   // HMAC the bytes, never a decoded string
    if (!verify(raw, req.headers['x-signature'])) {
      res.writeHead(401).end();
      return;
    }
    const evt = JSON.parse(raw.toString('utf8'));
    const ref = evt.data?.metadata?.migration_ref;
    if (ref && evt.type === 'video.ready') setState(ref, 'ready');
    if (ref && evt.type === 'video.errored') {
      setState(ref, 'rejected', { last_error: evt.data.error ?? 'ingest failed' });
    }
    res.writeHead(204).end();   // ack fast, work later
  });
}).listen(8787);

⚠️ Note: ack the webhook before you do any slow work. Providers retry on timeout, and a slow
handler turns one event into five.

6. Reconcile, then cut over ✅

The migration is not done when the uploads finish. It is done when the destination matches the
inventory.

Pull the destination's view of each asset back into a second table, then diff it. A reconcile.js
that walks state='ready' rows, fetches GET /videos/{remote_id}, and writes
(migration_ref, remote_duration_s) into a remote_assets table is about thirty lines using the same
Governor. Then:

-- create it once, populate it from the reconcile pass
CREATE TABLE IF NOT EXISTS remote_assets (
  migration_ref     TEXT PRIMARY KEY,
  remote_duration_s REAL
);

-- durations that drifted by more than half a second are the canary
SELECT a.id, a.duration_s, r.remote_duration_s
FROM assets a JOIN remote_assets r ON r.migration_ref = a.id
WHERE abs(a.duration_s - r.remote_duration_s) > 0.5;

A duration mismatch almost always means the ingest made a decision you did not ask for: a variable
frame rate source got normalized, a broken trailing segment got dropped, an audio-only tail got cut.
Worth looking at every single one.

Then flip playback behind a per-asset or per-tenant flag so a bad mapping is a revert, not an
incident. Keep the source library readable for a few months. Delete it when the reconciliation report
has been boring for a while.

What's next

Add a dry_run mode that migrates a deliberately weird slice (oldest, largest, unusual track layouts) instead of a random sample. Fifty strange files teach you more than a thousand normal ones.
If your provider supports resumable uploads over tus, swap the create call for a tus client so a dropped connection mid-file does not restart the transfer.
Export the ledger states to your metrics backend and alert on failed growth rate. A migration that silently stops making progress at 2am is the failure mode this whole design exists to prevent.

The pieces here (a queue, a ledger, an adaptive limiter, a webhook receiver) are ones you have all
written before in other contexts. The only real mistake is looking at a directory listing and a POST
endpoint and concluding the shortest path between them is a loop.

Load-test your HLS key server: rotate keys, reproduce the stampede, then fix it

Mason K — Mon, 17 Aug 2026 09:01:40 +0000

TL;DR

We'll build a tiny HLS setup with rotating AES-128 keys, point a concurrency harness at the key endpoint, and watch every simulated viewer ask for the same key in the same second. Then we'll apply the three mitigations that work on players shipping today, and look at where the new EXT-X-PRELOAD-HINT:TYPE=KEY tag fits.

Key rotation during a live stream is standard practice. What is not standard is load-testing the endpoint that serves those keys, and it is the one endpoint in your stack where every viewer makes an identical request at an identical moment. Let's make that visible.

You'll need ffmpeg 7.x or 8.x, node 20.x or newer, and openssl. Everything here runs locally.

1. 🎥 Make a stream with rotating keys

FFmpeg's HLS muxer will rotate keys for you if you hand it a key info file per period, but the clearest way to see the mechanic is to generate two keys and stitch a playlist that switches between them.

# generate two 16-byte content keys
openssl rand 16 > key0.bin
openssl rand 16 > key1.bin

# key info files: <key URI>\n<path to key file>\n<optional IV>
printf 'http://localhost:8080/keys/key0\nkey0.bin\n' > key0.info
printf 'http://localhost:8080/keys/key1\nkey1.bin\n' > key1.info

Now segment a source file with encryption on:

ffmpeg -i source.mp4 \
  -c:v libx264 -preset veryfast -g 48 -keyint_min 48 -sc_threshold 0 \
  -c:a aac -b:a 128k \
  -hls_time 4 -hls_playlist_type event \
  -hls_key_info_file key0.info \
  -hls_segment_filename 'seg_%03d.ts' \
  stream.m3u8

Open stream.m3u8 and you'll see the tag that does all the work:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:5
#EXT-X-KEY:METHOD=AES-128,URI="http://localhost:8080/keys/key0",IV=0x...
#EXTINF:4.000,
seg_000.ts
#EXTINF:4.000,
seg_001.ts

An EXT-X-KEY tag applies to every segment that follows it until another one shows up. To simulate a rotation, insert a second one partway down the playlist pointing at key1:

#EXTINF:4.000,
seg_014.ts
#EXT-X-KEY:METHOD=AES-128,URI="http://localhost:8080/keys/key1",IV=0x...
#EXTINF:4.000,
seg_015.ts

💡 Tip: in a real live setup you don't hand-edit this. Your packager emits a new EXT-X-KEY on whatever rotation interval you configured. The hand edit just lets us control exactly when the rotation lands.

2. 🛠️ Stand up a key endpoint that reports its own load

// key-server.js
import { createServer } from 'node:http';
import { readFileSync } from 'node:fs';

const keys = {
  '/keys/key0': readFileSync('key0.bin'),
  '/keys/key1': readFileSync('key1.bin'),
};

// count requests per 100ms bucket so we can see the shape of the load
const buckets = new Map();
setInterval(() => {
  const now = Date.now();
  for (const [t, n] of buckets) {
    if (now - t > 5000) buckets.delete(t);
  }
}, 1000);

const server = createServer((req, res) => {
  const key = keys[req.url];
  if (!key) {
    res.writeHead(404).end();
    return;
  }

  const bucket = Math.floor(Date.now() / 100) * 100;
  buckets.set(bucket, (buckets.get(bucket) ?? 0) + 1);

  // pretend this endpoint does an entitlement check
  setTimeout(() => {
    res.writeHead(200, { 'Content-Type': 'application/octet-stream' });
    res.end(key);
  }, 15);
});

server.listen(8080, () => console.log('key server on :8080'));

process.on('SIGINT', () => {
  const sorted = [...buckets.entries()].sort((a, b) => a[0] - b[0]);
  const peak = Math.max(...sorted.map(([, n]) => n));
  console.log(`\npeak requests in a 100ms window: ${peak}`);
  process.exit(0);
});

The 15ms setTimeout stands in for whatever your key endpoint actually does. Most do more than serve bytes: they check a session token, look up an entitlement, maybe hit a database. That work is the reason the endpoint has a concurrency ceiling at all.

3. Reproduce the stampede

Here is the part that matters. Real live viewers are not spread across the asset. They are all within a few seconds of the live edge, refreshing the same playlist, so they all see the new EXT-X-KEY line within the same playlist-refresh interval and all fetch it at once.

// stampede.js
const CONCURRENCY = Number(process.argv[2] ?? 2000);
const JITTER_MS = Number(process.argv[3] ?? 0);

const started = Date.now();
let ok = 0, failed = 0;
const latencies = [];

await Promise.all(
  Array.from({ length: CONCURRENCY }, async () => {
    if (JITTER_MS) await new Promise(r => setTimeout(r, Math.random() * JITTER_MS));
    const t0 = performance.now();
    try {
      const res = await fetch('http://localhost:8080/keys/key1');
      await res.arrayBuffer();
      latencies.push(performance.now() - t0);
      ok++;
    } catch {
      failed++;
    }
  })
);

latencies.sort((a, b) => a - b);
const p = (q) => latencies[Math.floor(latencies.length * q)]?.toFixed(1);
console.log(`concurrency=${CONCURRENCY} jitter=${JITTER_MS}ms`);
console.log(`ok=${ok} failed=${failed} wall=${Date.now() - started}ms`);
console.log(`p50=${p(0.5)}ms p95=${p(0.95)}ms p99=${p(0.99)}ms`);

Run it with no jitter, which is what an unmitigated rotation looks like:

$ node stampede.js 2000 0
concurrency=2000 jitter=0ms
ok=2000 failed=0 wall=1180ms
p50=612.4ms p95=1044.7ms p99=1131.2ms

Now run the same 2000 requests spread across a 10 second window, which is what a jittered fetch looks like:

$ node stampede.js 2000 10000
concurrency=2000 jitter=10000ms
ok=2000 failed=0 wall=10041ms
p50=17.9ms p95=21.3ms p99=28.6ms

⚠️ Your exact numbers will differ, and that is the point of running it on your own endpoint rather than trusting mine. The ratio is what to look at: identical total work, wildly different tail latency, purely because of arrival distribution.

Turn the concurrency up until you see failures. That number, divided by your rotation interval, is not your capacity. Your capacity is that number in a single burst, because that is how the requests actually arrive.

4. What breaks when the key is late

This is worse than a late segment, and it is worth being precise about why.

Failure	Player behaviour	Recovery
Segment fetch times out	Buffer drains, player stalls, retries	Usually automatic
Manifest fetch times out	Player keeps playing buffered content	Usually automatic
Key fetch times out	Player cannot decrypt the next segment	Often a fatal error, client-dependent

hls.js emits a KEY_LOAD_ERROR under NETWORK_ERROR and will retry, but the retry budget is finite and the segments keep arriving that it cannot decrypt. Native players vary. On some devices the user has to hit play again, which for a live event means you have converted a 200ms latency spike into a measurable drop in concurrent viewers.

5. The three fixes that work today

Jitter the fetch on the client, if you control the client. If you are shipping hls.js you can hook key loading and add a random delay before the request. This is exactly what the spec-level fix does, just done by hand:

// only viable when you control the player build
class JitteredKeyLoader extends Hls.DefaultConfig.loader {
  load(context, config, callbacks) {
    const delay = context.url.includes('/keys/') ? Math.random() * 3000 : 0;
    setTimeout(() => super.load(context, config, callbacks), delay);
  }
}

const hls = new Hls({ loader: JitteredKeyLoader });

The catch is obvious: it only helps viewers on your player build. Anyone on a native player is unaffected.

Serve keys through your CDN. A content key is a small immutable object at a stable URI, which is the ideal cacheable object. Set a Cache-Control that covers the key's active lifetime and your origin sees a handful of requests per rotation instead of all of them. The tension is that a lot of teams do per-viewer entitlement checks at key fetch time, which makes the response non-cacheable by design. If that is you, the question worth asking is whether the key fetch is really where authorisation should happen, or whether it belongs at session establishment.

Lengthen the rotation interval, or at least justify it. Rotating more often does not reduce the size of each spike, it increases the number of spikes. Every rotation is a full-audience event. Write down what threat the current interval mitigates, then check whether the interval is actually load-bearing for that threat or was copied from a config example.

6. Where `EXT-X-PRELOAD-HINT:TYPE=KEY` fits

The WWDC 2026 HLS update (What's new in HTTP Live Streaming, June 2026) added TYPE=KEY to EXT-X-PRELOAD-HINT. Previously that tag took TYPE=PART and TYPE=MAP and was a Low-Latency HLS mechanism for hinting at resources that do not exist yet.

The new value hints at an upcoming decryption key. Apple's stated rationale names the problem directly: a key rotation during a high-concurrency live stream triggers a thundering herd of simultaneous key requests. Clients that support the hint preload the key at a randomly-selected point between first seeing the hint and the key's first use, spreading the requests across that window.

Which is to say: the fix is standardised jitter, moved from your player fork into the protocol.

⚠️ Do not treat this as shipped. It is a spec addition; client support arrives on each player's own schedule. hls.js is at 1.6.16 with 1.7.0 in development, and preload-hint handling has been an open thread in that project for a while. Smart TV and set-top players update on manufacturer timelines. Emit the hint if your packager supports it, but keep the CDN and interval work, because those help every client including the ones that will never be updated.

The spec details are in section 4.4.5.3 of the preliminary HLS draft, and the stable reference for everything else is draft-pantos-hls-rfc8216bis.

What's next

Run stampede.js against your actual key endpoint in staging, at your real peak concurrency, in one burst. Not a sustained rate, a burst. That single number is more useful than the rest of this post.
Check whether your key responses are cacheable. If they are not, find out why, and whether that reason still holds.
Audit the rest of your stack for the same shape. Anything every viewer requests at the same instant belongs on a list: the key server, the DRM license server, the ad-decision call at the first break, the auth refresh if it is aligned to a fixed TTL.

The pattern generalises well beyond video, which is why "thundering herd" was already a name for it. Live streaming is just where you get to watch it happen in front of an audience.

Audit your HLS audio ladder: what you're actually shipping, and whether xHE-AAC is reachable

Mason K — Mon, 17 Aug 2026 09:01:16 +0000

TL;DR

We'll probe a real HLS stream to find out what audio it actually delivers, build a two-rung audio rendition set with correct EXT-X-MEDIA and CODECS declarations, and work out whether xHE-AAC is reachable for your stack. Spoiler on that last one: decoding is universal, encoding needs a license.

Your encoding profile has been re-tuned four times on the video side. The audio side is one 128 kbps AAC-LC track that nobody has touched since setup. Let's find out what that's costing and what the options are. You'll need ffmpeg 7.x or 8.x and, optionally, bento4.

1. 🔎 Find out what you're actually shipping

Start with the master playlist, not your encoding config. These disagree more often than you'd think.

curl -s https://example.com/stream/master.m3u8

#EXTM3U
#EXT-X-STREAM-INF:BANDWIDTH=628000,CODECS="avc1.4d401e,mp4a.40.2",RESOLUTION=640x360
360p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1628000,CODECS="avc1.4d401f,mp4a.40.2",RESOLUTION=1280x720
720p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=3128000,CODECS="avc1.640028,mp4a.40.2",RESOLUTION=1920x1080
1080p.m3u8

Two things to notice. First, mp4a.40.2 on every line: that's AAC-LC, the same audio in all three variants. Second, there's no EXT-X-MEDIA block, so audio is muxed into each video rendition and there is no separate audio rendition to switch.

Now confirm what the bytes say, because manifests lie:

ffprobe -v error -select_streams a:0 \
  -show_entries stream=codec_name,profile,bit_rate,channels,sample_rate \
  -of json https://example.com/stream/360p.m3u8

{
  "streams": [
    {
      "codec_name": "aac",
      "profile": "LC",
      "sample_rate": "48000",
      "channels": 2,
      "bit_rate": "128000"
    }
  ]
}

Here's the number that should bother you. On the 360p rendition, total bandwidth is 628 kbps and audio is 128 of it. Roughly a fifth of the bytes going to your worst-connected viewers are audio, encoded at exactly the same quality as the audio going to the viewer on fibre. That is the whole problem in one line.

💡 Tip: run this against a live playback URL rather than a mezzanine file. What your pipeline intends to produce and what your packager emits are separate facts.

2. Split audio out so it can vary

Muxed audio means audio is welded to the video rendition. Demuxed audio, declared with EXT-X-MEDIA, means the player picks an audio rendition independently. That's the prerequisite for everything else.

# encode video renditions with no audio
ffmpeg -i source.mp4 -an -c:v libx264 -preset veryfast \
  -g 48 -keyint_min 48 -sc_threshold 0 \
  -b:v 3000k -s 1920x1080 v_1080.mp4

# encode audio renditions separately
ffmpeg -i source.mp4 -vn -c:a aac -b:a 128k -ac 2 a_hi.mp4
ffmpeg -i source.mp4 -vn -c:a aac -b:a 64k  -ac 2 a_lo.mp4

Then package to fMP4 HLS. Bento4's mp4hls handles the manifest authoring:

mp4hls --output-dir=out --hls-version=7 \
  v_1080.mp4 v_720.mp4 v_360.mp4 a_hi.mp4 a_lo.mp4

The resulting master playlist declares audio as a separate group:

#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio-hi",NAME="English",DEFAULT=YES,\
AUTOSELECT=YES,LANGUAGE="en",URI="audio/hi/stream.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio-lo",NAME="English",DEFAULT=YES,\
AUTOSELECT=YES,LANGUAGE="en",URI="audio/lo/stream.m3u8"

#EXT-X-STREAM-INF:BANDWIDTH=3128000,CODECS="avc1.640028,mp4a.40.2",\
RESOLUTION=1920x1080,AUDIO="audio-hi"
video/1080/stream.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=464000,CODECS="avc1.4d401e,mp4a.40.2",\
RESOLUTION=640x360,AUDIO="audio-lo"
video/360/stream.m3u8

Now the 360p variant pairs with 64 kbps audio, and its total bandwidth drops from 628 to 464 kbps. That's a real saving on the rung where bandwidth is scarcest, and it required no new codec.

⚠️ Common mistake: pointing every EXT-X-STREAM-INF at the same AUDIO group and then wondering why audio never changes. The group ID is the switch. One group per audio quality tier.

3. Verify the declaration matches the bytes

The CODECS attribute is a promise to the player. If it's wrong, some clients refuse to play before they ever fetch a segment.

Codec	CODECS string	Notes
AAC-LC	`mp4a.40.2`	The default everywhere
HE-AAC v1	`mp4a.40.5`	SBR
HE-AAC v2	`mp4a.40.29`	SBR + PS
xHE-AAC	`mp4a.40.42`	fMP4 only, common encryption only

Check yours against the actual streams:

for f in out/audio/*/stream.m3u8; do
  echo "== $f"
  ffprobe -v error -select_streams a:0 \
    -show_entries stream=codec_name,profile,bit_rate -of csv=p=0 "$f"
done

== out/audio/hi/stream.m3u8
aac,LC,128000
== out/audio/lo/stream.m3u8
aac,LC,64000

Apple's HLS tools (mediastreamvalidator) will catch mismatches too, and as of the 2026 release they run on macOS, RHEL 9.5, Ubuntu 24.04.3 LTS and Debian 13.2 with full feature parity, so this can live in CI on Linux now.

4. 🎧 So where does xHE-AAC come in

xHE-AAC is MPEG-D USAC (the Extended HE-AAC profile) plus MPEG-D loudness and dynamic range control. Two properties make it interesting for the low rung of your ladder:

Range. Fraunhofer specifies stereo operation from 12 kbps to over 320 kbps in one codec. Apple's HLS authoring guidance puts the practical stereo band at 24 kbps up to a recommended 160.
Mandatory loudness and DRC metadata in every bitstream. The playback device adapts level to its own output. That's a normalisation stage you stop owning.

Playback support is not the blocker. Native decode has shipped since Android 9 Pie (2018) and iOS 13 / macOS 10.15 / tvOS 13, Safari decodes it through AVFoundation, and Fraunhofer puts the Android installed base above seven billion devices. Android 17 makes xHE-AAC encoding a standard feature this year, which is new.

Constraints to know before you plan around it:

fMP4 only. No MPEG-2 TS carriage.
Common encryption only. If you're on full-segment AES-128, that's a packaging change.
AVPlayer does mono and stereo only. No multichannel, so a 5.1 ladder stays as-is.

5. The encoding catch

Here's the part that decides this for most teams. Stock FFmpeg cannot encode xHE-AAC.

$ ffmpeg -h encoder=libfdk_aac 2>&1 | grep -A6 'profile'
     -profile           <int>  E...A...... (from 0 to 5)
       aac_low          2      LC
       aac_he           5      HE-AAC
       aac_he_v2        29     HE-AACv2
       aac_ld           23     Low Delay
       aac_eld          39     Enhanced Low Delay

No USAC, no Extended HE-AAC. libfdk_aac is the open-source Fraunhofer decoder/encoder kit and it stops at HE-AAC v2. Encoding xHE-AAC requires a licensed implementation: Fraunhofer's own, or the MainConcept xHE-AAC encoder plugin for FFmpeg. Packaging is fine either way, since Bento4 handles xHE-AAC in DASH and HLS without special options.

So the honest summary:

	xHE-AAC	Cost
Decode	Universal, already deployed	Free
Encode	Licensed implementations only	Purchase order
Package	Bento4, no special flags	Free

If you're at a scale where a codec licensing conversation is routine, this is a straightforward win on your low rung. If you're a small team with an FFmpeg box, it isn't reachable this quarter, and the right move is section 6.

6. What to do instead, today

Add HE-AAC v1 to the bottom rung. If your FFmpeg build has libfdk_aac (most distro builds don't, because of the license, so check), you get a codec that holds together at 48 kbps:

ffmpeg -i source.mp4 -vn -c:a libfdk_aac -profile:a aac_he \
  -b:a 48k -ac 2 a_lo.mp4

Declare it as mp4a.40.5. Test it on your actual device mix before shipping, because HE-AAC decode is very widely supported but not quite as universally as AAC-LC.

Do the loudness pass yourself, correctly. Since you're not getting DRC metadata for free, two-pass loudnorm is the substitute:

# pass 1: measure
ffmpeg -i source.mp4 -af loudnorm=I=-16:TP=-1.5:LRA=11:print_format=json \
  -f null - 2>&1 | tail -12

{
  "input_i" : "-23.41",
  "input_tp" : "-4.20",
  "input_lra" : "8.10",
  "input_thresh" : "-33.87",
  "target_offset" : "0.13"
}

# pass 2: apply the measured values
ffmpeg -i source.mp4 -vn -af \
  loudnorm=I=-16:TP=-1.5:LRA=11:measured_I=-23.41:measured_TP=-4.20:\
measured_LRA=8.10:measured_thresh=-33.87:offset=0.13:linear=true \
  -c:a aac -b:a 128k a_hi.mp4

Single-pass loudnorm is a dynamic normaliser and will pump on content with wide dynamic range. Two-pass measures first, then applies a fixed gain. Use two-pass.

7. "Why not just use Opus?"

This comes up every time, so let's settle it. Opus is free, it is excellent at low bitrates, and ffmpeg -c:a libopus works in every build you have. On paper it solves the same problem.

The blocker is carriage and client support in the HLS world specifically. Opus is at home in WebM and in WebRTC, where it is effectively the default. In HLS the story is much thinner: Apple's client stack does not decode Opus in HLS, so shipping an Opus-only audio rendition means shipping a stream that does not play on iPhones, iPads, Apple TV or Safari. For most consumer catalogues that ends the conversation before it starts.

Where it does make sense:

Context	Opus	xHE-AAC	AAC-LC
WebRTC / real-time	Yes, the default	No	Rare
DASH to browsers (no Safari requirement)	Works well	Works	Works
HLS to Apple devices	Not supported	Supported	Supported
Cheap and universal	Free encoder	Licensed encoder	Free encoder

So if you ship DASH to a controlled device set, or you are doing real-time, Opus is the pragmatic answer and you can stop reading. If HLS to consumer devices is your delivery path, the choice is between AAC-LC, HE-AAC and xHE-AAC, and the licensing question in section 5 is unavoidable.

# for the DASH-only case, this is all it takes
ffmpeg -i source.mp4 -vn -c:a libopus -b:a 48k -vbr on -application audio a_lo_opus.mp4

⚠️ Test playback on the real device matrix before committing to any of this, not just on your laptop. Audio codec support is the area where "it works in Chrome" is the least informative test result available.

What's next

Run the section 1 probe against production right now. If audio is muxed and identical across every rung, you have a bandwidth win available that costs you nothing but a repackage.
Decide the bottom rung deliberately: 64k AAC-LC, 48k HE-AAC, or a licensing conversation about xHE-AAC. Any of the three beats "128k everywhere by default".
If you're already on fMP4 and common encryption, you've met the xHE-AAC packaging prerequisites, so the only open question is the encoder.

Worth reading next: Apple's HLS authoring specification for the audio rendition rules, and Bento4's multi-bitrate audio guide for the packaging side.

Build a long-video to vertical-clips pipeline: metadata in, ranked 9:16 clips out

Mason K — Mon, 17 Aug 2026 09:00:43 +0000

TL;DR

We'll upload a long recording with AI metadata flags on, catch the webhook, write a ranking function that picks clip boundaries from chapters + scene changes + transcript, and cut vertical clips. The ranking function is the part that matters; everything else is plumbing.

Here is the naive version of this project:

ffmpeg -ss 00:14:22 -to 00:14:52 -i webinar.mp4 -c copy clip.mp4

That's the cutting solved. Now watch the output: it starts four words into a sentence, opens on a hard cut to a slide, and ends mid-word. Six of those and you've built something nobody will use.

The actual problem is choosing 00:14:22. Let's build that. You'll need node 20.x or newer and ffmpeg 7.x or 8.x.

1. 🎯 What a good clip boundary actually requires

Watch a human do this and they're checking three things:

Does it start on a complete thought? Needs word-level transcript timestamps so you can snap to a sentence start.
Does it start on a shot boundary? Needs scene-change detection so the clip doesn't open mid-cut.
Does the subject stay in frame? Needs to survive a 9:16 crop, which a two-person shot won't.

Three signals. Assemble them from three tools and you get three different clocks: Whisper timestamps drift against container PTS, PySceneDetect reports frame indices you convert with a frame rate that may be variable, and a tracker samples on its own interval. Joining them is where the week goes.

So the design decision up front is: do you own the extraction, or get it pre-joined?

2. Get the metadata (the flag-on-upload version)

I've been using FastPix for this because the AI capabilities are boolean flags on the upload call rather than a second pipeline you orchestrate. Same idea works with any provider that returns time-aligned metadata; the shape below is what to look for.

curl -X POST 'https://api.fastpix.com/v1/on-demand' \
  --user "$FASTPIX_TOKEN_ID:$FASTPIX_SECRET" \
  -H 'Content-Type: application/json' \
  -d '{
    "inputs": [
      { "type": "video", "url": "https://cdn.example.com/webinar.mp4" }
    ],
    "chapters": true,
    "accessPolicy": "private",
    "maxResolution": "1080p"
  }'

{
  "success": true,
  "data": {
    "id": "a1d1acdd-8f4e-4add-b498-6b398cf349d9",
    "status": "Created",
    "createdAt": "2026-08-17T10:50:34.594302Z",
    "playbackIds": [
      { "id": "6ta85f64-5717-4562-b3fc-2c963f66afa6", "accessPolicy": "private" }
    ],
    "maxResolution": "1080p",
    "mediaQuality": "standard"
  }
}

Auth is Basic: Access Token ID as username, Secret Key as password. Playback is https://stream.fastpix.com/<playbackId>.m3u8, with a JWT appended as ?token= for private assets.

⚠️ data is an object, not an array. If you're porting from an older integration that did data[0].id, that's your first bug.

💡 Tip: chapters is one flag in a family. summary, namedEntities, moderation and subtitles are siblings on the same create call, and because they run inside the same processing pass they land referenced to the same timeline. That property is why I stopped maintaining my own extraction stack.

⚠️ The API field is chapters, but the dashboard's custom media settings JSON calls it generateChapters. Two names for one feature.

Same settings for a direct (push) upload, nested under pushMediaSettings:

{
  "corsOrigin": "*",
  "pushMediaSettings": {
    "chapters": true,
    "accessPolicy": "private",
    "maxResolution": "1080p",
    "metadata": { "source": "weekly-webinar" }
  }
}

And for assets you uploaded before you thought of this, it's a PATCH rather than a re-upload:

curl -X PATCH "https://api.fastpix.com/v1/on-demand/$MEDIA_ID/chapters" \
  --user "$FASTPIX_TOKEN_ID:$FASTPIX_SECRET" \
  -H 'Content-Type: application/json' \
  -d '{ "chapters": true }'

3. 🛠️ Catch the webhook

Results arrive asynchronously. Handle the lifecycle events and the AI event separately, because they fire at different times.

// webhook.js
import express from 'express';

const app = express();
app.use(express.json());

app.post('/webhooks/video', async (req, res) => {
  // ack fast, work later
  res.sendStatus(200);

  const { type, data, object } = req.body;

  switch (type) {
    case 'video.media.created':
      await db.assets.upsert({ id: object.id, status: 'processing' });
      break;

    case 'video.media.ready':
      await db.assets.update(object.id, { status: 'ready' });
      break;

    case 'video.media.failed':
      await db.assets.update(object.id, { status: 'failed' });
      break;

    case 'video.mediaAI.chapters.ready':
      await onChapters(object.id, data.chapters);
      break;
  }
});

app.listen(3000);

The chapters payload looks like this:

{
  "type": "video.mediaAI.chapters.ready",
  "object": { "type": "media", "id": "f081fd53-6a9a-43ae-9d64-9974ef243dbd" },
  "id": "51a61b56-0197-4127-8a61-472e4d3fa59a",
  "workspace": { "name": "clips-pipeline", "id": "f7a13f50-7f5c-48f4-b7b2-c901dcff61c6" },
  "status": "ready",
  "data": {
    "isChaptersGenerated": true,
    "chapters": [
      {
        "chapter": "1",
        "startTime": "00:00:00",
        "endTime": "00:03:59",
        "title": "The Circle Challenge Begins",
        "summary": "Contestants start stacking items in a circle for a chance to win."
      }
    ]
  },
  "createdAt": "2026-08-17T11:52:29.526588692Z",
  "attempts": []
}

Three field names worth pinning down, because they are easy to guess wrong: the sequence number is chapter and it is a string, the prose field is summary (not description), and object.type is media even though the event is a mediaAI event.

⚠️ Times come back as hh:mm:ss strings. Convert once, at the boundary, and keep seconds internally. Mixing string times and float seconds in the same codebase is a bug generator.

const toSeconds = (hms) => {
  const [h, m, s] = hms.split(':').map(Number);
  return h * 3600 + m * 60 + s;
};

4. Write the ranking function (this is the product)

Providers will rank clips for you. FastPix's AI clipping produces ranked short clips scored on hook, pacing and narrative, and it's a fine candidate generator. But "which 30 seconds are worth posting" depends on your audience, not on the video, so treat vendor ranking as candidate generation and put your own opinion on top.

// rank.js
const MIN_LEN = 15;
const MAX_LEN = 60;
const SHOT_TOLERANCE = 0.4; // seconds

export function rankCandidates({ chapters, scenes, words }) {
  const candidates = [];

  for (const ch of chapters) {
    const start = toSeconds(ch.startTime);
    const end = toSeconds(ch.endTime);

    // slide a window through the chapter, snapping to sentence starts
    for (const w of words) {
      if (w.start < start || w.start > end) continue;
      if (!w.isSentenceStart) continue;

      const closeWord = lastWordBefore(words, w.start + MAX_LEN);
      const length = closeWord.end - w.start;
      if (length < MIN_LEN) continue;

      candidates.push({
        start: w.start,
        end: closeWord.end,
        chapter: ch.title,
        score: score({ start: w.start, end: closeWord.end, words, scenes }),
      });
    }
  }

  return dedupeOverlapping(candidates.sort((a, b) => b.score - a.score));
}

function score({ start, end, words, scenes }) {
  const inWindow = words.filter((w) => w.start >= start && w.end <= end);
  if (!inWindow.length) return 0;

  const endsClean = inWindow.at(-1).endsSentence ? 1 : 0;
  const shotSafe = scenes.some((s) => Math.abs(s - start) < SHOT_TOLERANCE) ? 1 : 0;

  // words per second: dead air is not a clip, but neither is a firehose
  const density = inWindow.length / (end - start);
  const densityScore = density < 1.2 ? 0 : Math.min(density / 3, 1.5);

  // single-speaker stretches survive a vertical crop; crosstalk does not
  const speakers = new Set(inWindow.map((w) => w.speaker));
  const speakerScore = speakers.size === 1 ? 1 : 0;

  return 2 + endsClean + shotSafe + densityScore + speakerScore;
}

The base 2 is for starting on a sentence, a hard filter rather than a score. Everything else is tunable. Tune it by watching output, not by reasoning about it. I got the density floor wrong twice before sitting down and watching twenty rejected candidates.

Doing your own scene detection instead? FFmpeg gives you boundaries directly:

ffmpeg -i webinar.mp4 -filter:v "select='gt(scene,0.4)',showinfo" \
  -f null - 2>&1 | grep showinfo | sed -n 's/.*pts_time:\([0-9.]*\).*/\1/p'

💡 The 0.4 threshold is a starting point, not a constant. Talking-head footage with a static camera needs it lower; a heavily-cut promo needs it higher.

5. Cut and reframe

Cutting is the easy half, but there's one gotcha:

# fast, no re-encode, but snaps to the nearest keyframe
ffmpeg -ss 14.22 -to 44.51 -i clip-source.mp4 -c copy clip.mp4

# frame-accurate: -ss after -i, and you re-encode
ffmpeg -i clip-source.mp4 -ss 14.22 -to 44.51 \
  -c:v libx264 -preset veryfast -crf 20 -c:a aac -b:a 128k clip.mp4

Stream copy is fast and wrong for this job. Your ranking function worked hard to land on a sentence start, and -c copy will slide that to the nearest keyframe, which is up to a GOP away. Re-encode.

Now the crop. Here's the version everybody writes first:

ffmpeg -i clip.mp4 -vf "crop=ih*9/16:ih,scale=1080:1920" -c:a copy vertical.mp4

That's a centre crop, and it's correct exactly when your subject is centred. On a two-person interview it produces a beautiful vertical clip of the gap between them. On a stage recording where the speaker walks, it produces a lectern.

Subject-tracking reframe is the fix, and it's real work to build: detect the subject per frame, smooth the crop path so it doesn't jitter, reset on cuts rather than panning across them. FastPix's auto-reframe does 16:9 to 9:16, 1:1 or 4:5 with subject tracking rather than centre-cropping, which is what made the output shippable instead of something I had to eyeball every time.

Know which one you need:

Footage	Centre crop	Subject tracking
Single centred speaker, static camera	Fine	Overkill
Screen share / slides	Fine (or use 1:1)	No help
Two-person interview	Fails	Required
Stage recording, speaker moves	Fails	Required

6. What it costs you to run

Billing is per minute, encoding is free on the standard plan (you pay for delivery, storage and add-ons), and signup gives you $25 in credits with no credit card for the free tier. Early-stage teams can get $600 through the Startup Program (under four years old, under $10M raised), which covers running this over a real back catalogue while you decide.

One honest caveat: this is API-first, not a no-code CMS. If what your team actually wants is a timeline UI where a marketer picks moments by hand, you're building that front end. That was fine here because removing the human from the loop was the point.

What's next

Build the ranking function before anything else. Watch twenty clips a human picked, write down why each works, and turn that into the score. Your criteria will be almost entirely about sentence and shot boundaries.
Add burned-in captions. You already have word-level timestamps, so this is a subtitles filter away, and it's what drives silent-autoplay watch time.
Feed the same metadata into search and chapter navigation on the long-form asset. The extraction is already paid for.

The documented AI flags (chapters, summary, namedEntities, moderation, subtitles) and their webhook payloads are in the In-Video AI docs; the clipping and reframe capabilities are on the product side, so check the current reference for their parameter names before you wire them in. The upload-and-webhook pattern here transfers to any managed video API, so that integration work isn't wasted if you switch. What varies a lot between providers is the AI capability set itself, so check yours against your actual list before you commit.

Detect variable frame rate uploads with ffprobe and normalize them before they break your pipeline

Mason K — Thu, 13 Aug 2026 13:57:11 +0000

TL;DR

Screen recordings from OBS, QuickTime, Loom and phone capture are variable frame rate. Feed them to a transcoder that assumes constant frame rate and you get audio drift that grows over the clip. We'll build a two-tier ffprobe gate (cheap header check, then a real per-frame timestamp check) and normalize with -fps_mode cfr on FFmpeg 8.

The bug

Audio and video start in sync and drift apart as the clip plays. Four minutes in, it's obviously wrong. Thirty seconds in, nobody notices. That growing gap is the signature of a variable frame rate (VFR) source hitting a pipeline that assumed constant frame rate (CFR).

Audio runs on a fixed sample clock. VFR video does not. When your transcoder stamps out frames at a target rate against a source that never agreed to one, the error accumulates instead of resetting.

Let's detect it properly and fix it at ingest.

1. Why you keep receiving VFR 🎥

Screen capture tools generate a frame only when the screen content changes. That covers OBS, QuickTime screen recording, Loom, phone screen recorders, and most game capture. If your product accepts user uploads of demos, tutorials, bug reports or gameplay, a real chunk of your ingest is VFR.

Check a file you already have:

ffprobe -v error \
  -select_streams v:0 \
  -show_entries stream=r_frame_rate,avg_frame_rate,nb_frames,duration \
  -of default=noprint_wrappers=1 \
  input.mp4

r_frame_rate=30000/1001
avg_frame_rate=23856/1001
nb_frames=7152
duration=300.033333

Those two rates disagree, so this file is worth a closer look.

2. Why the popular check is only a heuristic ⚠️

Nearly every tutorial stops at "if r_frame_rate != avg_frame_rate, it's VFR." That is not true, and believing it will cost you a day.

avg_frame_rate is total frames divided by duration. Real.
r_frame_rate is FFmpeg's guess at a base rate, computed like a lowest common multiple of observed timings so every timestamp lands on a whole tick. It is not the actual frame rate.

So a mismatch suggests VFR. It does not prove it, and identical values do not prove CFR either.

⚠️ Treat the header comparison as a cheap first-pass filter, not a verdict. Confirm with per-frame timestamps.

3. The real check: per-frame deltas

Dump packet timestamps and look at the gaps between them. Uniform gaps mean the file is effectively constant no matter what the headers claim.

ffprobe -v error \
  -select_streams v:0 \
  -show_entries packet=pts_time \
  -of csv=p=0 \
  input.mp4 | head -20

Look at that jump from 0.166833 to 0.250250, and the bigger one to 0.550550. Uneven. That's genuine VFR.

Here's the check as a script. It only pays for the expensive packet dump when the cheap check trips.

# vfr_gate.py (python 3.11+, ffmpeg/ffprobe 8.x)
import json
import subprocess
from fractions import Fraction

def _probe(args: list[str]) -> dict:
    out = subprocess.run(
        ["ffprobe", "-v", "error", "-of", "json", *args],
        capture_output=True, text=True, check=True,
    ).stdout
    return json.loads(out)

def header_mismatch(path: str) -> tuple[bool, Fraction, Fraction]:
    """Tier 1: cheap. Compares r_frame_rate against avg_frame_rate."""
    data = _probe([
        "-select_streams", "v:0",
        "-show_entries", "stream=r_frame_rate,avg_frame_rate",
        path,
    ])
    s = data["streams"][0]
    r = Fraction(s["r_frame_rate"])
    a = Fraction(s["avg_frame_rate"])
    return (r != a), r, a

Now tier two, which only runs on files that tripped tier one:

# vfr_gate.py (continued)
def timestamps_are_irregular(path: str, tolerance: float = 0.002,
                             max_packets: int = 3000) -> bool:
    """Tier 2: expensive. Are the gaps between presentation timestamps uniform?"""
    data = _probe([
        "-select_streams", "v:0",
        "-show_entries", "packet=pts_time",
        "-read_intervals", "%+#" + str(max_packets),
        path,
    ])
    pts = sorted(
        float(p["pts_time"]) for p in data.get("packets", [])
        if p.get("pts_time") not in (None, "N/A")
    )
    if len(pts) < 3:
        return False

    deltas = [b - a for a, b in zip(pts, pts[1:])]
    baseline = sorted(deltas)[len(deltas) // 2]   # median, robust to one bad gap
    return any(abs(d - baseline) > tolerance for d in deltas)

def is_vfr(path: str) -> bool:
    mismatch, _, _ = header_mismatch(path)
    return mismatch and timestamps_are_irregular(path)

💡 Tip: -read_intervals "%+#3000" caps how many packets ffprobe reads. On a 90-minute file, dumping every packet is slow and you don't need all of them to see irregularity.

Run it:

$ python vfr_gate.py screen-recording.mp4
VFR: True   (r=30000/1001  avg=23856/1001)

$ python vfr_gate.py camera-export.mp4
VFR: False  (r=30000/1001  avg=30000/1001)

4. Normalize with -fps_mode cfr

Once you know, the fix is one flag applied as the first transcoding step, before segmenting, thumbnailing, or anything else forms an opinion about the file.

ffmpeg -i input.mp4 \
  -fps_mode cfr -r 30 \
  -c:v libx264 -preset medium -crf 21 \
  -c:a aac -b:a 128k -ar 48000 \
  -movflags +faststart \
  normalized.mp4

⚠️ Old tutorials use -vsync cfr or -vsync 1. That family is deprecated in FFmpeg 8.x. Use -fps_mode cfr in anything new.

Pin -r explicitly. Letting FFmpeg infer the output rate is how you got here.

Picking the rate:

Content	Rate	Why
Screen capture, talking head, general UGC	30	Safe default, cheapest
Gameplay, sports, high motion	60	Motion detail is the point
Anything above the source's peak rate	don't	Buys nothing, costs bitrate

5. Wire it into ingest

# ingest.py
import subprocess
from vfr_gate import is_vfr

def normalize_if_needed(src: str, dst: str, fps: int = 30) -> str:
    if not is_vfr(src):
        return src  # already CFR, skip the re-encode

    subprocess.run([
        "ffmpeg", "-v", "error", "-y", "-i", src,
        "-fps_mode", "cfr", "-r", str(fps),
        "-c:v", "libx264", "-preset", "medium", "-crf", "21",
        "-c:a", "aac", "-b:a", "128k", "-ar", "48000",
        "-movflags", "+faststart",
        dst,
    ], check=True)
    return dst

Emit a metric every time this fires. You want to know what fraction of your ingest is VFR, because it directly sizes your transcode fleet.

The cost, honestly

Normalizing a screen recording is not free. Twelve seconds of a static slide that occupied a handful of VFR frames becomes 360 frames at 30fps CFR. Your encoder does more work.

The output usually isn't much bigger, because near-identical consecutive frames compress extremely well and modern rate control handles them fine. But your encode time goes up on exactly the content that was cheapest before. Size your workers for screen capture as the worst case, not the best.

Errors you'll hit

avg_frame_rate=0/0: the container has no duration, common with fragmented MP4 and some live recordings. Skip tier one and go straight to timestamps.

Audio still drifts after normalization: check whether the source audio sample rate is what you think. Pin -ar 48000 and re-test. A resample happening implicitly is its own drift source.

Application provided invalid, non monotonically increasing dtx: the source has out-of-order timestamps on top of being VFR. Add -fflags +genpts before -i.

What's next

Frame rate is one assumption your pipeline makes about incoming files without checking. There are others, and they fail in unrelated-looking ways:

Rotation metadata: the file is 1080x1920 with a 90-degree rotate flag, and half your tools honor it.
Color space / transfer characteristics: the reason HDR sources come out washed on SDR screens.
Audio channel layout: 5.1 sources silently downmixed, or not.

Pick one, write the ffprobe check, run it over a week of real uploads, and see what comes back. That number is usually more interesting than the fix.

Build a directional preload window for a scrolling video feed with hls.js

Mason K — Thu, 13 Aug 2026 13:56:11 +0000

TL;DR

Vertical video feeds are fast on the first item and stutter on every one after. We'll build a preload window that follows scroll direction, attaches hls.js sources to off-screen items ahead of the user, and tears down everything outside the window so you stay under Safari's video element ceiling. Reconciler pattern, ~120 lines.

The problem

Item 1 plays instantly because the page loaded around it. Item 2 shows black, then a spinner, then a stutter into low quality. Preloading everything fixes the stutter and replaces it with a data-plan fire and a crash on iPhone.

What we want: preload a small number of items in the direction the user is scrolling, and aggressively release everything else.

1. Why `preload="auto"` isn't a strategy

<!-- this is a hint, not an instruction -->
<video preload="auto" src="..."></video>

Browsers treat preload as a suggestion and mobile browsers take it least seriously, deliberately, to protect the user's data. With HLS you get real control, because loading happens in JS.

In hls.js, attaching a source is the trigger. The moment an instance has a manifest URL, it fetches the playlist and starts pulling segments, viewport or no viewport. That's our primitive.

npm i hls.js@^1.7.0

2. A player wrapper with two modes 🎛️

Off-screen items need a shallow buffer. Just enough to start. The active item gets the normal one.

// feed/player.js (hls.js 1.7.x)
import Hls from "hls.js";

const PRELOAD_CONFIG = {
  // enough to render frame one, not enough to burn a data plan
  maxBufferLength: 4,
  maxMaxBufferLength: 6,
  // go get fragment 1 before media is attached
  startFragPrefetch: true,
  autoStartLoad: true,
};

const ACTIVE_CONFIG = {
  maxBufferLength: 30,     // hls.js default
  maxMaxBufferLength: 60,
  startFragPrefetch: true,
  autoStartLoad: true,
};

export function attach(videoEl, url, { active = false } = {}) {
  if (videoEl.canPlayType("application/vnd.apple.mpegurl")) {
    // Safari plays HLS natively; no hls.js instance to manage
    videoEl.src = url;
    videoEl.preload = active ? "auto" : "metadata";
    return { native: true, destroy: () => { videoEl.removeAttribute("src"); videoEl.load(); } };
  }

  const hls = new Hls(active ? ACTIVE_CONFIG : PRELOAD_CONFIG);
  hls.loadSource(url);
  hls.attachMedia(videoEl);
  return {
    native: false,
    hls,
    destroy: () => { hls.destroy(); },
  };
}

💡 Tip: maxBufferLength defaults to 30 in hls.js. Thirty seconds for the item being watched is sensible. Thirty seconds each for four queued items is not.

3. The directional window

The naive window is symmetric: 2 before, 2 after. But scroll isn't symmetric. Someone moving down keeps moving down, and the item behind them is the one they just finished.

// feed/window.js
const AHEAD = 4;   // items to preload in the direction of travel
const BEHIND = 1;  // cheap insurance for a quick scroll-back

/**
 * @param {number} index      current active item
 * @param {1|-1} direction    1 = scrolling down, -1 = scrolling up
 * @param {number} total      item count
 * @returns {Set<number>}     indices that should have a source attached
 */
export function desiredWindow(index, direction, total) {
  const lo = direction === 1 ? index - BEHIND : index - AHEAD;
  const hi = direction === 1 ? index + AHEAD : index + BEHIND;
  const out = new Set();
  for (let i = Math.max(0, lo); i <= Math.min(total - 1, hi); i++) out.add(i);
  return out;
}

Same bandwidth budget, roughly double the hit rate. One conditional.

4. The reconciler (this is the important part) 🔑

Here's the bug you will otherwise ship: a user flicks quickly through six items. Preload fires on scroll-in, teardown fires on scroll-out, and because they're independent effects racing each other you tear down a player you just decided to preload. Symptom: fine when you scroll slowly, broken when you scroll fast. Fine in dev, broken in the wild.

Fix: never fire preload and teardown as separate effects. Compute the desired set, diff it against what's attached, act on the diff.

// feed/reconciler.js
import { attach } from "./player.js";
import { desiredWindow } from "./window.js";

export function createFeedController({ items, getVideoEl }) {
  const attached = new Map();  // index -> { native, hls, destroy, active }
  let lastIndex = 0;

  function reconcile(index) {
    const direction = index >= lastIndex ? 1 : -1;
    lastIndex = index;

    const want = desiredWindow(index, direction, items.length);

    // 1. release everything outside the window
    for (const [i, handle] of attached) {
      if (!want.has(i)) {
        handle.destroy();
        attached.delete(i);
      }
    }

    // 2. attach everything inside it that isn't already
    for (const i of want) {
      const el = getVideoEl(i);
      if (!el) continue;                      // not rendered yet, skip
      const existing = attached.get(i);
      const shouldBeActive = i === index;

      if (existing) {
        // promote the newly-active item to the deep-buffer config
        if (shouldBeActive && !existing.active) {
          existing.destroy();
          attached.set(i, { ...attach(el, items[i].hlsUrl, { active: true }), active: true });
        }
        continue;
      }

      attached.set(i, {
        ...attach(el, items[i].hlsUrl, { active: shouldBeActive }),
        active: shouldBeActive,
      });
    }
  }

  function destroyAll() {
    for (const [, h] of attached) h.destroy();
    attached.clear();
  }

  return { reconcile, destroyAll, get size() { return attached.size; } };
}

Drive it from whatever tells you the active index. With CSS scroll snapping, an IntersectionObserver is enough:

// feed/index.js
const controller = createFeedController({ items, getVideoEl: (i) => els[i] });

const io = new IntersectionObserver((entries) => {
  for (const e of entries) {
    if (e.intersectionRatio > 0.6) {
      controller.reconcile(Number(e.target.dataset.index));
    }
  }
}, { threshold: [0.6] });

els.forEach((el) => io.observe(el));

5. Teardown is not optional on iOS

Mobile Safari has a widely reported ceiling of around 16 simultaneous video elements; past that, additional ones render black. It's not an explicitly documented API limit, it's a wall people keep hitting. In an infinite feed that mounts a player per item and never unmounts, you hit it fast.

Our window caps attachments at AHEAD + BEHIND + 1 = 6, comfortably under it. But release the source, not the element:

// keep the DOM node mounted, drop the media
handle.destroy();          // hls.destroy() frees buffers + decoder
// element stays in the list for cheap re-attach on scroll-back

Unmounting the component too means scrolling back pays full layout cost. Keep the shell, make the media disposable.

⚠️ Multiple simultaneous video elements are a memory and GPU pressure problem beyond the element count. It shows up as a crash, not a catchable error, so you can't detect it and back off. You have to stay under the line by construction.

6. Measure the thing users feel 📊

The metric is time from scroll settling to first frame painted, on item 2 and beyond. Item 1 is a page load and it flatters you.

// feed/metrics.js
const marks = new Map();

export function markScrollSettled(index) {
  marks.set(index, performance.now());
}

export function markFirstFrame(index, videoEl) {
  videoEl.addEventListener("playing", () => {
    const t0 = marks.get(index);
    if (t0 == null) return;
    const ms = Math.round(performance.now() - t0);
    marks.delete(index);
    navigator.sendBeacon?.("/metrics/feed-ttff", JSON.stringify({
      index, ms,
      position: index,                       // item 40 matters more than item 2
      conn: navigator.connection?.effectiveType ?? "unknown",
    }));
  }, { once: true });
}

Chart it against bytes per session. These move in opposite directions and tuning one alone ships something beautiful on wifi and hostile on a phone plan.

The window size you want is the smallest AHEAD where time-to-first-frame goes flat. Start at 4, walk it down to 2, and watch where the curve bends. Past the flat point you're buying nothing and spending someone else's money.

💡 Tip: if the feed is instant for ten items and degrades after that, your teardown isn't keeping up. More preloading will not fix it.

What's next

hls.js doesn't yet expose a first-class "preload N seconds off-screen then stop" API. There's an open issue for it and a related multi-segment prefetch proposal. Worth watching; the per-instance maxBufferLength trick above is the practical stand-in.
Serve a lower-rendition first segment to off-screen items so the preload costs less, then let ABR climb once the item is active.
If your feed is React Native rather than web, the same window/reconciler logic applies. Mux wrote up a feed that preloads by scroll direction and the structure transfers directly.

The general lesson: "make it feel instant" is a scheduling problem wearing a rendering problem's clothes. You aren't loading faster, you're loading earlier, and the craft is in guessing well and giving up fast.

Build semantic search inside video: shot embeddings + transcript chunks, fused on the timeline

Mason K — Thu, 13 Aug 2026 13:55:46 +0000

TL;DR

Frame-only video search can't find "where she explains pricing." Transcript-only search can't find "the architecture diagram slide." We'll build both indexes, key them to the same timeline, and fuse at query time so results are timestamps, not video IDs. Shot detection, SigLIP-2 embeddings, Whisper chunks, pgvector.

What we're building

A search endpoint where "where does she explain the pricing model" returns video_id=42, t=247.3s and you can seek straight there.

Two indexes over one timeline:

Index	Unit	Model	Finds
Visual	one shot	SigLIP-2	"the architecture diagram", "person holding a coffee cup"
Transcript	utterance group	text embedder	"the pricing explanation", "when they mention refunds"

Neither half works alone. In business video (talks, interviews, tutorials, meetings) most meaning is in the audio, so a frames-only index is blind to the majority of it. But nobody says "architecture diagram" out loud, they say "as you can see here."

pip install torch transformers pyscenedetect faster-whisper psycopg[binary] pgvector

1. Chunk on shots, not on seconds 🎬

Sampling a frame every N seconds is wasteful in both directions: forty near-identical frames in a static screen share, and a missed shot across a fast cut. Use shot boundaries instead.

# ingest/shots.py
from scenedetect import detect, ContentDetector

def shot_boundaries(path: str, threshold: float = 27.0) -> list[tuple[float, float]]:
    """Returns [(start_sec, end_sec), ...] per detected shot."""
    scenes = detect(path, ContentDetector(threshold=threshold))
    return [(s.get_seconds(), e.get_seconds()) for s, e in scenes]

$ python -c "from ingest.shots import shot_boundaries; print(shot_boundaries('talk.mp4')[:4])"
[(0.0, 8.34), (8.34, 31.07), (31.07, 33.9), (33.9, 96.21)]

Grab one representative frame per shot, from the middle rather than the boundary (boundaries catch transitions and dissolves):

# ingest/frames.py
import subprocess, tempfile, os

def midpoint_frame(video: str, start: float, end: float) -> str:
    t = start + (end - start) / 2
    out = os.path.join(tempfile.mkdtemp(), "f.jpg")
    subprocess.run([
        "ffmpeg", "-v", "error", "-y",
        "-ss", f"{t:.3f}", "-i", video,
        "-frames:v", "1", "-q:v", "3", out,
    ], check=True)
    return out

2. Embed the frames

SigLIP-2 generally edges out CLIP at comparable size, especially on fine-grained visual detail. The SO400M variant emits 1152 dimensions (note this, it's about to matter).

# ingest/visual.py
import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor

MODEL_ID = "google/siglip2-so400m-patch16-512"
EMBED_DIM = 1152

_device = "cuda" if torch.cuda.is_available() else "cpu"
_model = AutoModel.from_pretrained(MODEL_ID).to(_device).eval()
_proc = AutoProcessor.from_pretrained(MODEL_ID)

@torch.inference_mode()
def embed_image(path: str) -> list[float]:
    inputs = _proc(images=Image.open(path).convert("RGB"), return_tensors="pt").to(_device)
    v = _model.get_image_features(**inputs)
    return torch.nn.functional.normalize(v, dim=-1)[0].cpu().tolist()

@torch.inference_mode()
def embed_text(query: str) -> list[float]:
    inputs = _proc(text=[query], padding="max_length", return_tensors="pt").to(_device)
    v = _model.get_text_features(**inputs)
    return torch.nn.functional.normalize(v, dim=-1)[0].cpu().tolist()

⚠️ EMBED_DIM is not a config value you change later. Your table column is vector(1152). Swapping to JinaCLIP-v2 (1024 dims) means re-embedding your entire library. Version your tables from day one so a model swap is a parallel build, not a migration.

3. Transcript chunks want different boundaries

Speech doesn't respect cuts. A sentence runs across a shot change constantly, so chopping transcripts on visual boundaries produces fragments that embed badly. Chunk on the transcript's own logic, with overlap.

# ingest/transcript.py
from faster_whisper import WhisperModel

_asr = WhisperModel("large-v3-turbo", device="auto", compute_type="int8")

def chunks(video: str, target_chars: int = 400, overlap: int = 1) -> list[dict]:
    segments, _ = _asr.transcribe(video, vad_filter=True, word_timestamps=False)
    segs = [{"start": s.start, "end": s.end, "text": s.text.strip()} for s in segments]

    out, buf = [], []
    for seg in segs:
        buf.append(seg)
        if sum(len(s["text"]) for s in buf) >= target_chars:
            out.append({
                "start": buf[0]["start"],
                "end": buf[-1]["end"],
                "text": " ".join(s["text"] for s in buf),
            })
            buf = buf[-overlap:] if overlap else []
    if buf:
        out.append({
            "start": buf[0]["start"], "end": buf[-1]["end"],
            "text": " ".join(s["text"] for s in buf),
        })
    return out

Two indexes, different granularities, both timestamped. That's the correct end state, and the timeline is the join key.

4. Schema

Postgres + pgvector means one fewer system to run, and it lets you filter by tenancy in the same query as the similarity search. That last part is not a convenience, it's a correctness requirement (see step 6).

-- migrations/001_init.sql  (postgres 16 + pgvector 0.8)
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE shot_embeddings (
  id          bigserial PRIMARY KEY,
  video_id    bigint NOT NULL,
  tenant_id   bigint NOT NULL,
  start_sec   double precision NOT NULL,
  end_sec     double precision NOT NULL,
  embedding   vector(1152) NOT NULL
);

CREATE TABLE transcript_embeddings (
  id          bigserial PRIMARY KEY,
  video_id    bigint NOT NULL,
  tenant_id   bigint NOT NULL,
  start_sec   double precision NOT NULL,
  end_sec     double precision NOT NULL,
  text        text NOT NULL,
  embedding   vector(1152) NOT NULL
);

CREATE INDEX ON shot_embeddings       USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON transcript_embeddings USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON shot_embeddings       (tenant_id, video_id);
CREATE INDEX ON transcript_embeddings (tenant_id, video_id);

5. Fuse at query time, not index time

It's tempting to combine visual and text vectors into one representation during ingest. Don't, at least not first. You lose explainability, you lose per-query weighting, and you've committed to a fusion strategy before you know what works on your content.

Query both, merge on time overlap:

# search/query.py
from collections import defaultdict

def search(conn, tenant_id: int, q: str, k: int = 30,
           w_visual: float = 0.4, w_text: float = 0.6):
    qv = embed_text(q)

    visual = _knn(conn, "shot_embeddings", tenant_id, qv, k)
    textual = _knn(conn, "transcript_embeddings", tenant_id, qv, k)

    # bucket both sides into 5s slots on the shared timeline
    buckets = defaultdict(lambda: {"visual": 0.0, "text": 0.0, "hit": None})
    for r in visual:
        key = (r["video_id"], int(r["start_sec"] // 5))
        b = buckets[key]
        b["visual"] = max(b["visual"], r["score"])
        b["hit"] = b["hit"] or r
    for r in textual:
        key = (r["video_id"], int(r["start_sec"] // 5))
        b = buckets[key]
        b["text"] = max(b["text"], r["score"])
        b["hit"] = b["hit"] or r

    scored = [{
        "video_id": vid,
        "t": b["hit"]["start_sec"],
        "score": w_visual * b["visual"] + w_text * b["text"],
        "why": {"visual": round(b["visual"], 3), "transcript": round(b["text"], 3)},
    } for (vid, _slot), b in buckets.items()]

    return sorted(scored, key=lambda r: -r["score"])[:10]

# search/query.py (continued)
def _knn(conn, table: str, tenant_id: int, qv: list[float], k: int):
    with conn.cursor() as cur:
        cur.execute(
            f"""SELECT video_id, start_sec, 1 - (embedding <=> %s::vector) AS score
                FROM {table}
                WHERE tenant_id = %s
                ORDER BY embedding <=> %s::vector
                LIMIT %s""",
            (qv, tenant_id, qv, k),
        )
        cols = [d.name for d in cur.description]
        return [dict(zip(cols, row)) for row in cur.fetchall()]

A moment where both signals fire is a much better answer than either alone, and now you can say so:

$ curl -s 'localhost:8000/search?q=where+she+explains+the+pricing+model' | jq '.[0]'
{
  "video_id": 42,
  "t": 247.3,
  "score": 0.71,
  "why": { "visual": 0.31, "transcript": 0.98 }
}

w_visual / w_text is the knob that matters. Conference talks want transcript weighted heavily. A B-roll library wants the opposite.

6. The part that isn't in the architecture diagram 🔒

Embeddings do not carry permissions. A vector index will happily return the nearest neighbor from a video the user can't see.

Filtering has to happen inside the query, not after it. Notice WHERE tenant_id = %s sits in the same statement as the ORDER BY embedding <=> above. If you filter the results afterward, your top-10 becomes a top-4, your pagination lies, and eventually someone sees another customer's footage.

⚠️ This is the single most common way these systems ship broken. Post-filtering a KNN result is not access control.

Errors you'll hit

expected 1152 dimensions, not 768: you swapped models without migrating the column. Told you.

Shot detection returns one giant scene: screen recordings with no cuts. Fall back to fixed-interval sampling when len(shots) < 2.

Recall is fine, ranking is bad: your two score distributions aren't comparable. Normalize each side's scores within the result set before weighting them.

Where the numbers come from

One 2026 benchmark roundup puts semantic video search accuracy for this class of model in the mid-to-high 80s on general datasets, rising into the low 90s with domain-specific fine-tuning. That's someone else's data on someone else's content. Measure your own before you promise anything.

On store choice: 2026 comparisons between pgvector and dedicated engines like Qdrant keep landing on "comparable speed, the bottleneck is elsewhere." If your data is already in Postgres, that's your answer.

What's next

Clip generation for free. Once moments are addressable, "make me a highlight reel about X" is a query plus a concat, not a manual scrub.
Timestamped recommendations. Point at a moment instead of a video.
Rerank the top 20 with a cross-encoder or a VLM if precision at 1 matters more than latency.

The reason to build this isn't that users are asking for a search box. It's that the index is the substrate for four other features.

DEV Community: Mason K

Audit what every rung of your HLS ladder costs per viewer-hour

TL;DR

1. Grab a master playlist 🎯

2. The audit script 🛠️

3. Read the output like a bill 💸

4. Fix A: reshape the ladder with FFmpeg

5. Fix B: change the pricing model

What's next

Encode your first VVC clip with vvenc 1.14, then watch every browser refuse to play it

TL;DR

1. Get the encoder 🛠️

2. Prepare a test clip 🎬

3. Encode with vvencapp

4. Mux it into MP4 📦

5. Inspect what you made 🔬

6. The playback tour, from success to failure

7. So where does VVC actually run? 📡

What's next

Build a live latency HUD with ffmpeg, a burned-in clock, and hls.js

TL;DR

1. Generate a live stream with a clock burned in 🕐

2. The player page with a latency HUD 📺

3. Now read the pixels 👀

4. Turn the knobs and watch the budget move 🎛️

5. Ship it as a beacon 📡

What's next

Build a caption QA harness in Python: WER, missed entities, timing and reading rate

TL;DR

0. Setup 🛠️

1. Parse the VTT into text plus timings

2. Normalize both sides with the same normalizer

3. Missed entity rate: the metric that actually correlates with complaints

4. Timing: median cue offset

5. Readability: reading rate and cue duration

6. Tie it together 📊

7. The reference set is the real work

What's next

Frame-accurate FFmpeg trimming without re-encoding the whole file

TL;DR

The problem, in two commands 🎬

1. Look at your keyframes first

2. Find the keyframes bracketing our cut

3. Probe the source so the edges match

4. The three-piece cut ✂️

5. The errors you will hit 🐛

6. The cheaper fix, if you control ingest

What's next

Build a resumable video migration worker: ledger, bounded concurrency, 429 backoff

TL;DR

1. The ledger 📒

2. Inventory before you migrate 🔍

3. A rate limiter that reacts

4. Idempotent creates

5. Webhooks, not polling 🔔

6. Reconcile, then cut over ✅

What's next

Load-test your HLS key server: rotate keys, reproduce the stampede, then fix it

TL;DR

1. 🎥 Make a stream with rotating keys

2. 🛠️ Stand up a key endpoint that reports its own load

3. Reproduce the stampede

4. What breaks when the key is late

5. The three fixes that work today

6. Where EXT-X-PRELOAD-HINT:TYPE=KEY fits

What's next

Audit your HLS audio ladder: what you're actually shipping, and whether xHE-AAC is reachable

TL;DR

1. 🔎 Find out what you're actually shipping

2. Split audio out so it can vary

3. Verify the declaration matches the bytes

4. 🎧 So where does xHE-AAC come in

5. The encoding catch

6. What to do instead, today

7. "Why not just use Opus?"

What's next

Build a long-video to vertical-clips pipeline: metadata in, ranked 9:16 clips out

TL;DR

1. 🎯 What a good clip boundary actually requires

2. Get the metadata (the flag-on-upload version)

6. Where `EXT-X-PRELOAD-HINT:TYPE=KEY` fits

1. Why `preload="auto"` isn't a strategy