DEV Community: Mason K

Turn on CMCD: make your CDN logs explain rebuffering

Mason K — Mon, 15 Jun 2026 09:32:34 +0000

TL;DR

CMCD (Common Media Client Data, CTA-5004) lets your player attach its state (buffer length, bitrate, "I just rebuffered") to every segment request, so your CDN's own logs can explain QoE. We'll enable it in hls.js and Shaka, choose header vs query mode, and write a tiny Node parser that turns the payload into a per-session story.

What we're building

CMCD turned on in the player (hls.js + Shaka).
A decision on transmission mode (header vs query) and why it matters.
A server-side parser that reads the CMCD payload and groups by session.

Versions: hls.js 1.x, shaka-player 4.x, node 20.x.

The keys you'll actually use

CMCD sends short keys on each request. The important ones:

Key	Meaning	Why you care
`sid`	session id	the join key between player and CDN
`cid`	content id	group requests by asset
`bl`	buffer length (ms)	how close to starving
`bs`	buffer starvation flag	true = a rebuffer just happened
`su`	startup flag	request is part of initial buffering
`mtp`	measured throughput (kbps)	client's bandwidth estimate
`br`	requested encoded bitrate	which rendition
`ot`	object type (v/a/m/i)	video, audio, manifest, init

💡 Boolean keys like bs and su are sent as just the key name when true (no =value). Your parser has to handle that.

1. Enable CMCD in hls.js

Enabling the cmcd config object turns it on. Start with header mode:

// app/playerHls.js
import Hls from "hls.js";

const hls = new Hls({
  cmcd: {
    sessionId: crypto.randomUUID(),   // your sid
    contentId: "asset-12345",         // your cid
    useHeaders: true,                 // header mode (vs query string)
  },
});

hls.loadSource("https://cdn.example.com/video/master.m3u8");
hls.attachMedia(document.querySelector("video"));

With useHeaders: false, hls.js appends a CMCD query argument instead. Same data, different transport (see the trade-off below).

2. Enable CMCD in Shaka

Shaka's CmcdManager serializes per CTA-5004 in header or query mode:

// app/playerShaka.js
import shaka from "shaka-player";

const player = new shaka.Player(document.querySelector("video"));

player.configure({
  cmcd: {
    enabled: true,
    sessionId: crypto.randomUUID(),
    contentId: "asset-12345",
    useHeaders: true,
  },
});

await player.load("https://cdn.example.com/video/master.mpd");

3. The transmission-mode decision

This is the part that actually needs thought.

Mode	Pros	Cons
Header (`CMCD-Request: ...`)	clean URLs, cache key untouched	needs CORS `Access-Control-Allow-Headers` for `CMCD-*`; CDN must log custom headers
Query (`?CMCD=...`)	works everywhere, easy to log	changes the URL = changes the cache key unless the CDN ignores the `CMCD` arg

⚠️ Query mode can wreck your cache hit ratio: every unique CMCD payload looks like a different object. Configure the CDN to exclude the CMCD query arg from the cache key before you ship query mode.

For cross-origin segment hosts in header mode, you need this on the segment server:

Access-Control-Allow-Headers: CMCD-Request, CMCD-Object, CMCD-Session, CMCD-Status

4. Parse it server-side

CMCD values are a comma-separated key=value list, with quoted strings and bare keys for booleans. Here's a minimal parser:

// server/parseCmcd.js - node 20+
export function parseCmcd(raw) {
  // raw: e.g.  bl=21300,br=2500,bs,cid="asset-12345",mtp=48200,ot=v,sid="abc-123",su
  const out = {};
  if (!raw) return out;

  // split on commas that aren't inside quotes
  const parts = raw.match(/(?:[^,"]+|"[^"]*")+/g) || [];
  for (const part of parts) {
    const eq = part.indexOf("=");
    if (eq === -1) {
      out[part.trim()] = true;             // bare key = boolean true (e.g. bs, su)
      continue;
    }
    const key = part.slice(0, eq).trim();
    let val = part.slice(eq + 1).trim();
    if (val.startsWith('"') && val.endsWith('"')) {
      val = val.slice(1, -1);              // quoted string
    } else if (!Number.isNaN(Number(val))) {
      val = Number(val);                    // numeric
    }
    out[key] = val;
  }
  return out;
}

Read it from whichever transport you chose:

// server/ingest.js (Express)
import { parseCmcd } from "./parseCmcd.js";

app.use((req, _res, next) => {
  const fromHeader = req.headers["cmcd-request"];        // header mode
  const fromQuery = req.query.CMCD;                       // query mode
  req.cmcd = parseCmcd(fromHeader || fromQuery);
  next();
});

5. Turn it into a QoE signal

Now the payoff. Group your request logs by sid and the starvation flags light up the exact requests around a stall:

// server/sessionReport.js (sketch)
function sessionReport(logLines) {
  const bySession = {};
  for (const line of logLines) {
    const c = line.cmcd;
    if (!c?.sid) continue;
    (bySession[c.sid] ??= []).push({
      url: line.url,
      edgeCache: line.cacheStatus,   // HIT / MISS from your CDN log
      ttfb: line.ttfbMs,             // edge latency from your CDN log
      bufferMs: c.bl,
      starved: c.bs === true,        // the gold signal
      startup: c.su === true,
      bitrate: c.br,
    });
  }
  return bySession;
}

A starved: true row sitting next to a MISS with a high ttfb is a root cause you can act on, not a guess. That's the whole point: the player labeled the bad request, in the CDN's own log.

6. Confirm it in the browser before you touch the CDN

Before you go configuring edge logging, verify the player is actually emitting CMCD. Open DevTools, go to the Network tab, filter to your .ts or .m4s segment requests, and look at one.

In header mode, the request headers include a CMCD-Request (and possibly CMCD-Object, CMCD-Session, CMCD-Status) entry:

CMCD-Request: bl=21300,dl=21300,mtp=48200,su
CMCD-Object: br=2500,d=6000,ot=v,tb=4500
CMCD-Session: cid="asset-12345",sid="abc-123"

In query mode, the same data is URL-encoded on the segment URL:

GET /video/seg-042.m4s?CMCD=bl%3D21300%2Cbr%3D2500%2Ccid%3D%22asset-12345%22%2Cot%3Dv%2Csid%3D%22abc-123%22

Two things to sanity-check right here, before any server work:

[ ] sid is stable across every segment in one playback (same value), and different across page reloads. If it's changing mid-session, your join key is broken.
[ ] bs appears on requests right after a forced stall. Throttle your network to "Slow 3G" in DevTools, let it rebuffer, and watch for bs showing up. If it never appears, the player isn't tracking starvation the way you think it is.

💡 Catch these in the browser and you save yourself an afternoon of staring at CDN logs wondering why nothing groups correctly. The bug is almost always client-side config, not the log pipeline.

Version note

CMCD v1 is stable and supported across hls.js, Shaka, dash.js, ExoPlayer/Media3, and JWPlayer. CTA-5004 v2 (2026) adds new keys, an event-reporting mode, and stricter value encoding; as of early 2026 hls.js was adding optional v2 support. Pin a version so a v2 client doesn't send keys your v1 parser rejects.

What's next

Configure your CDN to log the CMCD header (or parse the query arg) and exclude CMCD from the cache key.
Start with three keys: sid, bl, bs. Trace one rebuffer end to end before adding the rest.
Add a CMCD validator in CI (the video-dev community maintains one) so malformed payloads fail fast.

Build scrub-bar thumbnail previews with FFmpeg and a WebVTT sprite

Mason K — Mon, 15 Jun 2026 09:32:01 +0000

TL;DR

We're going to add hover-preview thumbnails (the little image that follows your cursor on a video scrub bar) to a player. Backend is one FFmpeg command that builds a sprite sheet, plus a tiny script that writes a WebVTT index. Frontend points the player at the VTT. Total: well under 100 lines.

What we're building

Two static artifacts and a few lines of player config:

storyboard.jpg - a sprite sheet: many small frames tiled into one image.
storyboard.vtt - a WebVTT file mapping each timeline range to a rectangle in the sprite via a #xywh fragment.
Player wiring (Video.js shown, hls.js note at the end).

Versions used: ffmpeg 8.0.2 (anything 7.1+ works), node 20.x.

1. Generate the sprite with FFmpeg

The core command. One frame every 10 seconds, each scaled to 160px wide, tiled into a 10x10 grid:

ffmpeg -i input.mp4 \
  -vf "fps=1/10,scale=160:-1,tile=10x10" \
  -frames:v 1 storyboard.jpg

What each filter does:

fps=1/10 - sample one frame per 10 seconds (not every frame).
scale=160:-1 - 160px wide, height auto from aspect ratio.
tile=10x10 - pack frames into a 10-column, 10-row grid (up to 100 tiles).

# terminal output you'll see
frame=    1 fps=0.0 q=24.0 Lsize=N/A time=N/A bitrate=N/A
video:118kB audio:0kB subtitle:0kB ...

⚠️ A 10x10 grid covers 1000 seconds at a 10s interval. Longer videos overflow one sheet, and a single image can also hit the browser's max-canvas limit (~16k–32k px per side). For anything long, generate multiple sprites. We handle that in the generator below.

2. Know your numbers with ffprobe

Before writing the VTT, get the real duration so the last (partial) row is handled correctly:

ffprobe -v error -show_entries format=duration \
  -of csv=p=0 input.mp4
# 642.40

3. Generate the WebVTT index

This is the part that has to be exact. Each cue's time range must match the FFmpeg interval, or previews drift the deeper you scrub. Generate it from the same interval value, never by hand.

// scripts/makeVtt.js - node 20+
import { writeFileSync } from "node:fs";

const INTERVAL = 10;      // seconds, MUST match fps=1/INTERVAL
const TILE_W = 160;
const TILE_H = 90;        // know your source aspect; 16:9 -> 90
const COLS = 10;
const ROWS = 10;
const PER_SHEET = COLS * ROWS;

function ts(sec) {
  const h = String(Math.floor(sec / 3600)).padStart(2, "0");
  const m = String(Math.floor((sec % 3600) / 60)).padStart(2, "0");
  const s = String(Math.floor(sec % 60)).padStart(2, "0");
  return `${h}:${m}:${s}.000`;
}

export function makeVtt(durationSec, sheetName = "storyboard") {
  const count = Math.ceil(durationSec / INTERVAL);
  let out = "WEBVTT\n\n";

  for (let i = 0; i < count; i++) {
    const start = i * INTERVAL;
    const end = Math.min(start + INTERVAL, durationSec);

    const indexInSheet = i % PER_SHEET;
    const sheet = Math.floor(i / PER_SHEET);     // 0, 1, 2...
    const col = indexInSheet % COLS;
    const row = Math.floor(indexInSheet / COLS);
    const x = col * TILE_W;
    const y = row * TILE_H;

    const img = `${sheetName}-${sheet}.jpg`;     // matches multi-sheet output
    out += `${ts(start)} --> ${ts(end)}\n`;
    out += `${img}#xywh=${x},${y},${TILE_W},${TILE_H}\n\n`;
  }
  return out;
}

const duration = Number(process.argv[2] || 642.4);
writeFileSync("storyboard.vtt", makeVtt(duration));
console.log("wrote storyboard.vtt");

A few cues from the output:

WEBVTT

00:00:00.000 --> 00:00:10.000
storyboard-0.jpg#xywh=0,0,160,90

00:00:10.000 --> 00:00:20.000
storyboard-0.jpg#xywh=160,0,160,90

00:00:20.000 --> 00:00:30.000
storyboard-0.jpg#xywh=320,0,160,90

4. Generate multiple sheets for long video

To keep each sprite under the browser's image cap, split the extraction by segment. Loop over 1000-second windows and tile each into its own storyboard-N.jpg:

# scripts/make_sheets.sh
DUR=$(ffprobe -v error -show_entries format=duration -of csv=p=0 input.mp4)
WINDOW=1000           # 100 tiles * 10s
i=0
start=0
while [ "$(echo "$start < $DUR" | bc)" -eq 1 ]; do
  ffmpeg -ss "$start" -t "$WINDOW" -i input.mp4 \
    -vf "fps=1/10,scale=160:-1,tile=10x10" \
    -frames:v 1 "storyboard-${i}.jpg"
  i=$((i+1))
  start=$((start+WINDOW))
done

💡 Put -ss before -i for a fast keyframe-accurate seek so you're not decoding from the start of the file on every window.

5. Wire it into the player

Video.js with the thumbnails plugin reads the VTT directly:

<!-- index.html -->
<link href="https://vjs.zencdn.net/8.10.0/video-js.css" rel="stylesheet" />
<video id="player" class="video-js" controls preload="auto" width="800">
  <source src="https://cdn.example.com/video/master.m3u8" type="application/x-mpegURL" />
</video>

// app/player.js
import videojs from "video.js";
import "videojs-vtt-thumbnails";

const player = videojs("player");

player.vttThumbnails({
  src: "https://cdn.example.com/video/storyboard.vtt",
});

That's the whole frontend. The plugin parses the #xywh fragments and crops the sprite as you hover.

Rolling your own on a custom seek bar is just as small: parse the VTT once, then on mousemove over the bar, find the cue whose range contains the hovered time and set the preview element's background-image + background-position from the fragment.

// app/customSeekPreview.js (sketch)
function showPreview(hoverTimeSec, cues, el) {
  const cue = cues.find(c => hoverTimeSec >= c.start && hoverTimeSec < c.end);
  if (!cue) return;
  const { img, x, y, w, h } = cue;            // parsed from #xywh
  el.style.width = `${w}px`;
  el.style.height = `${h}px`;
  el.style.backgroundImage = `url(${img})`;
  el.style.backgroundPosition = `-${x}px -${y}px`;
}

6. Verify alignment before you ship

The single most common bug here is drift: the previews look right at the start and wander off near the end. It's almost always an interval mismatch between FFmpeg and the VTT. Catch it in ten seconds with a spot check instead of discovering it in QA.

Extract the frame your VTT claims is at a known timestamp, and eyeball it against the sprite tile:

# what does the real video look like at 5:00 (300s)?
ffmpeg -ss 300 -i input.mp4 -frames:v 1 -q:v 2 check-300.jpg

Then find the cue covering 300s in storyboard.vtt:

00:05:00.000 --> 00:05:10.000
storyboard-0.jpg#xywh=0,360,160,90

Crop exactly that rectangle out of the sprite and compare:

ffmpeg -i storyboard-0.jpg -vf "crop=160:90:0:360" tile-check.jpg

check-300.jpg and tile-check.jpg should show the same shot. If they don't, your INTERVAL, TILE_H, or grid math is off by something. Fix it now, because a confidently wrong preview is worse than no preview at all.

💡 Wire this into CI: render the cropped tile and the real frame for three timestamps, and fail the build if they diverge past a similarity threshold. Cheap insurance against a regression in your generator.

Gotchas checklist

[ ] VTT interval must equal the FFmpeg fps=1/N. Mismatch = drift.
[ ] Watch the browser image-size cap; split into multiple sheets for long video.
[ ] Set TILE_H to your real aspect ratio, or the crop rectangles are wrong.
[ ] Cache sprite + VTT on your CDN; they're static and shared across all viewers.
[ ] Drop JPEG quality a notch (-q:v 5 or so) - nobody pixel-peeps a hover preview.

What's next

Hang sprite generation off the same worker that runs your transcode, so every upload gets previews automatically.
Try a WebP sprite for smaller files at the same visual quality.
The same sprite + VTT pattern powers chapter markers and share-card previews, so the worker pays for itself more than once.

Replacing a 5-service AWS video pipeline with one API call

Mason K — Mon, 15 Jun 2026 09:30:22 +0000

TL;DR

We had a video feature built on S3 + MediaConvert + CloudFront + a hand-rolled player + a half-finished CloudWatch dashboard. It worked, but every change touched five services. This post shows the glue code we actually ran, then the same flow as one upload call against a managed video API (FastPix here, but the pattern is identical for Mux / Cloudflare Stream / api.video). You'll see exactly what code disappears.

The setup we were maintaining

Here's the honest shape of the DIY pipeline. Upload to S3, trigger MediaConvert on the object-created event, write renditions back, serve through CloudFront. The Lambda that kicks off encoding looked roughly like this:

// lambda/triggerEncode.js - fires on S3 ObjectCreated
import { MediaConvertClient, CreateJobCommand } from "@aws-sdk/client-mediaconvert";

const mc = new MediaConvertClient({ region: "us-east-1" });

export async function handler(event) {
  const key = event.Records[0].s3.object.key;
  const input = `s3://uploads-bucket/${key}`;

  await mc.send(new CreateJobCommand({
    Role: process.env.MC_ROLE_ARN,
    Settings: {
      Inputs: [{ FileInput: input }],
      OutputGroups: [{
        Name: "Apple HLS",
        OutputGroupSettings: {
          Type: "HLS_GROUP_SETTINGS",
          HlsGroupSettings: {
            Destination: `s3://renditions-bucket/${key}/`,
            SegmentLength: 6,
            MinSegmentLength: 0,
          },
        },
        // ...one Output block PER rendition: 1080p, 720p, 480p, 360p...
        Outputs: [ /* 40+ lines of codec settings per rendition */ ],
      }],
    },
  }));
}

That Outputs array is where the weekends go. Every rendition is a block of codec settings. Every tweak to the ladder is a redeploy. And this Lambda is only the encode trigger. You still need:

An IAM role with the right MediaConvert + S3 permissions.
A second event path to know when the job finished (EventBridge → another Lambda).
CloudFront in front of the renditions bucket, with an OAC so the bucket isn't public.
A player that knows the manifest URL.
Something that records whether playback actually worked.

⚠️ None of these is hard. The problem is that there are five of them, and they fail independently.

The cost shape (why tuning didn't save us)

The bill wasn't one number, it was several that move on different axes:

Line item	Service	Scales with
Encoding	MediaConvert	output minutes (renditions × source)
Origin storage	S3	originals + renditions retained
Delivery / egress	CloudFront	GB delivered (i.e. your success)
Glue compute	Lambda	events
QoE analytics	CloudWatch + custom	engineering time

MediaConvert is billed per output minute, and CloudFront egress is billed per GB, so a popular video is a bigger bill on two axes at once.¹ We dropped a rendition and moved to a cheaper CloudFront price class. It trimmed the edges. It did not change the shape.

The replacement: one upload call

Here's the same upload-encode-store-deliver flow as a single request. Auth is Basic with an Access Token ID + Secret Key.

// server/createAsset.js - node 20+
const res = await fetch("https://api.fastpix.io/v1/on-demand", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    Authorization: "Basic " + Buffer
      .from(`${process.env.FASTPIX_TOKEN_ID}:${process.env.FASTPIX_SECRET}`)
      .toString("base64"),
  },
  body: JSON.stringify({
    inputs: [{ type: "video", url: "https://your-bucket/source.mp4" }],
    // ABR ladder is generated for you; no per-rendition codec blocks
    metadata: { project: "demo" },
  }),
});

const { data } = await res.json();
console.log("playbackId:", data.playbackId);

Playback is then just an HLS URL you can hand to any player:

// app/player.js - works with hls.js, Video.js, Shaka, or the platform player
const src = `https://stream.fastpix.io/${playbackId}.m3u8`;

The 40-line-per-rendition Outputs array is gone, because the ABR ladder is generated server-side. The "did it finish" EventBridge path becomes a single webhook you subscribe to once. And the analytics project I never finished on CloudWatch ships as Video Data, free up to 100K views a month, where Mux's comparable Media plan starts at $499/month.²

Want resumable uploads from the browser instead of importing from a URL? That's the upload SDK, not a fresh S3 multipart implementation:

npm install @fastpix/upload

// app/upload.js
import { uploadFile } from "@fastpix/upload";

await uploadFile({
  endpoint: signedUploadUrl,   // created server-side via the same API
  file,                        // a File from an <input type="file">
  onProgress: (p) => setProgress(p),
});

Before / after, honestly

	DIY AWS stack	Managed video API
Encode trigger	Lambda + MediaConvert job JSON	one POST
Ladder config	per-rendition codec blocks	generated
Job-finished signal	EventBridge → Lambda	one webhook
Delivery	CloudFront + OAC setup	playback URL
Player	hand-wrapped	included or BYO
QoE analytics	build it yourself	included (free up to 100K views/mo)
Bill	5 line items, 5 axes	usage-based, one model

On total cost, FastPix publishes a "~70% cheaper than AWS" figure, but it's tied to a specific scenario (1-minute 1080p video streamed to 500K viewers), so treat it as scenario-specific, not a blanket promise.³ The win I'd actually stand behind is predictability: one usage-based bill instead of five that move independently.

💡 Trade-off worth naming: this is an API-first approach, not a no-code video CMS. If your team wants a drag-and-drop back office with zero engineering, you'll still build a front-end on top of the API.⁴

What's next

Subscribe to processing webhooks so your UI flips from "processing" to "ready" without polling.
Add signed playback (JWT) if your content is gated.
If you're migrating an existing library, look for a batch import tool rather than scripting S3 copies by hand.
Compare the numbers for your delivery-to-encode ratio: FastPix pricing next to AWS MediaConvert pricing. The right answer depends on your workload, not on a blog post.

If you're at extreme sustained scale, keep self-hosting; the math flips in your favor there. For a video feature inside a product that does something else, one API call beats five services.

MediaConvert per-output-minute and CloudFront per-GB egress: aws.amazon.com/mediaconvert/pricing (verify current). ↩
FastPix Video Data free up to 100K views/month; Mux Data Media plan $499/month (verify Mux pricing). ↩
"~70% cheaper than AWS" is FastPix's published, methodology-specific comparison (1-min 1080p, 500K viewers). ↩
Approved FastPix trade-off: API-first, not a no-code CMS. ↩

Build a video webhook handler that survives duplicates (Express + Postgres)

Mason K — Mon, 15 Jun 2026 09:29:44 +0000

📦 Code: github.com/USER/video-webhook-receiver (replace before publishing)

TL;DR

We build a production-grade webhook receiver for async video events (asset.ready and friends): verify HMAC-SHA256 on the raw body, dedupe on the event id with a Postgres unique index, persist + ack with 200 fast, then process from a queue. Webhooks are at-least-once, so duplicates are guaranteed; we design for them.

Managed video APIs encode asynchronously. You upload, you get status: preparing, and minutes later a webhook fires asset.ready. If that handler is wrong, you hit the worst bug shape there is: upload succeeded, encode succeeded, and the video never shows up. No error anywhere. Let's build the handler so that never happens.

Four steps:

The raw-body trap (this breaks signature checks for almost everyone).
Verify the HMAC signature in constant time.
Dedupe with a unique index so duplicates are no-ops.
Ack fast, process from a queue.

1. The raw-body trap 🪤

Providers sign the exact bytes of the request body. If you reparse JSON and re-serialize before checking, the bytes differ and the signature never matches. The classic cause is a global express.json():

// app.js: THE BUG
const express = require('express');
const app = express();
app.use(express.json());          // ❌ consumes + reparses the body globally

app.post('/webhooks/video', (req, res) => {
  // req.body is parsed; the raw bytes are gone; signature check will always fail
});

The fix: mount a raw body parser on the webhook route only, and keep JSON parsing for the rest of your app.

// app.js: THE FIX
const express = require('express');
const app = express();

// raw buffer ONLY for the webhook route, registered before any json() middleware
app.post('/webhooks/video',
  express.raw({ type: 'application/json' }),
  handleVideoWebhook
);

app.use(express.json()); // normal parsing for everything else

Now req.body inside handleVideoWebhook is a Buffer of the exact bytes the provider signed.

2. Verify the signature (constant time)

Most providers send HMAC-SHA256, often with a timestamp to block replays (Stripe-style t=...,v1=...; Mux uses a similar scheme). Recompute and compare with crypto.timingSafeEqual so you don't leak the secret through timing.

// verify.js
const crypto = require('crypto');

function verifySignature(rawBody, header, secret) {
  // header like: "t=1735689600,v1=4f2c...". Adapt to your provider's format.
  const parts = Object.fromEntries(header.split(',').map(kv => kv.split('=')));
  const timestamp = parts.t;
  const sent = parts.v1;
  if (!timestamp || !sent) return false;

  // reject old timestamps (replay protection): 5 min tolerance
  const ageSec = Math.abs(Date.now() / 1000 - Number(timestamp));
  if (ageSec > 300) return false;

  const signedPayload = `${timestamp}.${rawBody.toString('utf8')}`;
  const expected = crypto
    .createHmac('sha256', secret)
    .update(signedPayload)
    .digest('hex');

  const a = Buffer.from(sent, 'hex');
  const b = Buffer.from(expected, 'hex');
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

module.exports = { verifySignature };

// handleVideoWebhook.js (top half)
const { verifySignature } = require('./verify');
const SECRET = process.env.WEBHOOK_SECRET;

async function handleVideoWebhook(req, res) {
  const sigHeader = req.get('webhook-signature') || '';
  if (!verifySignature(req.body, sigHeader, SECRET)) {
    return res.status(401).send('bad signature');   // reject, don't process
  }
  const event = JSON.parse(req.body.toString('utf8'));
  // ... step 3 below
}

3. Dedupe: the load-bearing wall

Webhook delivery is at-least-once. You WILL get the same event twice (the provider retries when unsure). Use the event id as an idempotency key and a unique index to make a duplicate a no-op.

-- schema.sql
CREATE TABLE webhook_events (
  event_id    text PRIMARY KEY,        -- provider's unique event id
  type        text NOT NULL,
  raw         jsonb NOT NULL,          -- persist the payload for replay/debug
  received_at timestamptz NOT NULL DEFAULT now(),
  processed   boolean NOT NULL DEFAULT false
);

// handleVideoWebhook.js (continued)
const { pool } = require('./db');
const { enqueue } = require('./queue');

  // inside handleVideoWebhook, after JSON.parse:
  const eventId = event.id;
  const client = await pool.connect();
  try {
    await client.query('BEGIN');

    // INSERT ... ON CONFLICT DO NOTHING returns 0 rows if we've seen it
    const ins = await client.query(
      `INSERT INTO webhook_events (event_id, type, raw)
       VALUES ($1, $2, $3) ON CONFLICT (event_id) DO NOTHING`,
      [eventId, event.type, event]
    );

    if (ins.rowCount === 0) {
      await client.query('COMMIT');
      return res.status(200).send('duplicate ignored');   // idempotent no-op
    }

    await client.query('COMMIT');
  } catch (err) {
    await client.query('ROLLBACK');
    throw err;
  } finally {
    client.release();
  }

💡 If you do real DB work inline instead of queuing, put the dedupe insert and that work in the same transaction. Otherwise a crash between them leaves you fulfilled-but-not-recorded, and the retry double-fulfills.

4. Ack fast, process async ⏱️

The most dangerous failure isn't an error, it's being slow. If your handler finishes the work after the provider's timeout (often 5-15s), the provider assumes failure and retries, running your logic twice. So do the minimum in the request: verify, dedupe, persist, enqueue, return 200. A worker does the real work.

// handleVideoWebhook.js (end)
  await enqueue({ eventId, type: event.type });   // durable queue (SQS/BullMQ/etc.)
  return res.status(202).send('accepted');
}

// worker.js
const { dequeue } = require('./queue');
const { pool } = require('./db');

async function processEvent({ eventId }) {
  const { rows } = await pool.query(
    'SELECT raw, processed FROM webhook_events WHERE event_id = $1', [eventId]
  );
  if (!rows.length || rows[0].processed) return;     // gone or already done

  const event = rows[0].raw;
  switch (event.type) {
    case 'video.asset.ready':
      await markVideoReady(event.data.asset_id, event.data.playback_id);
      break;
    case 'video.asset.errored':
      await markVideoFailed(event.data.asset_id, event.data.error);
      break;
    // ordering is NOT guaranteed: reconcile by asset_id, don't assume sequence
  }

  await pool.query(
    'UPDATE webhook_events SET processed = true WHERE event_id = $1', [eventId]
  );
}

dequeue(processEvent);

Two mistakes that survive code review

❌ Returning 500 on a business error
   A 500 tells the provider to RETRY. If the event itself is bad
   (malformed asset, constraint violation), you've made a poison
   message that retries forever and floods your logs.
   ✅ Ack 200 once persisted; handle business failures in the worker
      with your own retries + a dead-letter queue.

❌ Treating webhooks as the only source of truth
   If your endpoint is down for an hour, those events can be lost
   for good. ✅ Run a periodic job that lists assets from the
   provider and reconciles against your DB.

Quick test

# replay a captured event twice; second should be a no-op
curl -X POST localhost:3000/webhooks/video \
  -H 'content-type: application/json' \
  -H "webhook-signature: t=$(date +%s),v1=$(...)" \
  --data-binary @event.json
# → 202 accepted
# run it again → 200 duplicate ignored

What's next

Swap the in-house queue for SQS/BullMQ and add a dead-letter queue.
Add a /webhooks/replay admin endpoint that re-enqueues a stored event_id.
Add the reconciliation cron that lists provider assets and backfills anything missed.

This is the same at-least-once reliability shape you'd build for any messaging system. Video just makes the stakes obvious: get it wrong and the upload succeeds, the encode succeeds, and the video never appears.

Two-pass loudness normalization with FFmpeg loudnorm (the right way)

Mason K — Wed, 03 Jun 2026 08:36:36 +0000

TL;DR

We normalize audio to a consistent perceived loudness using FFmpeg's loudnorm filter in two passes: pass 1 measures, pass 2 applies a linear gain to the target. We'll parse the JSON, script it, batch it with ffmpeg-normalize, and verify with ebur128. Single-pass loudnorm pumps; don't use it for VOD.

If your library has one clip that whispers and the next one that blasts, your viewers are riding the volume knob. Peak normalization won't fix it (same peak, different perceived loudness). What you want is EBU R128 loudness normalization, and FFmpeg ships the filter to do it. Let's wire it up properly.

1. Why not single-pass?

The tempting one-liner is single-pass:

# DO NOT do this for VOD: it applies dynamic compression and pumps
ffmpeg -i in.mp4 -af loudnorm=I=-16:TP=-1.5:LRA=11 out.mp4

In one pass, loudnorm doesn't know what's coming, so it makes loudness decisions on the fly with dynamic processing. On music and dialogue that breathes audibly. Single-pass is for live; for files you process ahead of time, use two passes and a linear gain.

2. Pass 1: measure

Run loudnorm purely to analyze. Output goes to null; we only want the JSON it prints to stderr.

# bash: measure.sh
ffmpeg -hide_banner -i input.mp4 \
  -af loudnorm=I=-16:TP=-1.5:LRA=11:print_format=json \
  -f null - 2> measure.log

The tail of measure.log is a JSON block:

{
  "input_i" : "-27.61",
  "input_tp" : "-9.05",
  "input_lra" : "8.40",
  "input_thresh" : "-38.10",
  "output_i" : "-16.00",
  "output_tp" : "-1.50",
  "output_lra" : "11.00",
  "normalization_type" : "dynamic",
  "target_offset" : "0.49"
}

Those input_* values are the measured loudness of your file. We feed them back in pass 2.

💡 Target choice: -16 LUFS is a sane general default. Broadcast wants -23 (EBU R128) with TP=-1. Big streaming platforms normalize playback to roughly -14. Pick per context and keep it consistent across the library.

3. Pass 2: apply linear gain

Pass the measured values back and set linear=true. That applies one consistent gain across the whole file instead of moment-to-moment compression, so dynamics survive.

# bash: apply.sh
ffmpeg -hide_banner -i input.mp4 \
  -af loudnorm=I=-16:TP=-1.5:LRA=11:measured_I=-27.61:measured_TP=-9.05:measured_LRA=8.40:measured_thresh=-38.10:offset=0.49:linear=true \
  -c:v copy -c:a aac -b:a 192k -ar 48000 \
  output.mp4

⚠️ loudnorm resamples internally to 192 kHz. If you don't set -ar 48000 you can get a 192 kHz output you didn't want. Always pin the output sample rate.

Note -c:v copy: we're not touching the video, just re-encoding the audio track.

4. Script the whole thing

Hardcoding measured values is fine for one file, painful for a thousand. Here's a small wrapper that runs pass 1, parses the JSON, and runs pass 2.

# normalize.py: python 3.10+, ffmpeg 5.x+ on PATH
import json, re, subprocess, sys
from pathlib import Path

TARGET = dict(I="-16", TP="-1.5", LRA="11")

def measure(src: Path) -> dict:
    cmd = ["ffmpeg", "-hide_banner", "-i", str(src),
           "-af", f"loudnorm=I={TARGET['I']}:TP={TARGET['TP']}:LRA={TARGET['LRA']}:print_format=json",
           "-f", "null", "-"]
    out = subprocess.run(cmd, capture_output=True, text=True).stderr
    # the JSON block is the last {...} in stderr
    blob = re.search(r"\{[^{}]+\}\s*$", out.strip())
    if not blob:
        raise RuntimeError(f"no loudnorm JSON for {src}")
    return json.loads(blob.group(0))

def apply(src: Path, dst: Path, m: dict) -> None:
    af = (f"loudnorm=I={TARGET['I']}:TP={TARGET['TP']}:LRA={TARGET['LRA']}"
          f":measured_I={m['input_i']}:measured_TP={m['input_tp']}"
          f":measured_LRA={m['input_lra']}:measured_thresh={m['input_thresh']}"
          f":offset={m['target_offset']}:linear=true")
    cmd = ["ffmpeg", "-hide_banner", "-y", "-i", str(src),
           "-af", af, "-c:v", "copy", "-c:a", "aac", "-b:a", "192k",
           "-ar", "48000", str(dst)]
    subprocess.run(cmd, check=True)

if __name__ == "__main__":
    src = Path(sys.argv[1])
    dst = src.with_name(src.stem + "_norm.mp4")
    m = measure(src)
    print(f"measured integrated loudness: {m['input_i']} LUFS -> target {TARGET['I']}")
    apply(src, dst, m)
    print(f"wrote {dst}")

$ python normalize.py whisper_clip.mp4
measured integrated loudness: -27.61 LUFS -> target -16
wrote whisper_clip_norm.mp4

5. Or just use ffmpeg-normalize

For batch jobs, ffmpeg-normalize wraps all of this with two-pass by default:

pip install ffmpeg-normalize
ffmpeg-normalize input.mp4 -nt ebu -t -16 -c:a aac -b:a 192k -ar 48000 -o output.mp4
# batch a folder
ffmpeg-normalize *.mp4 -nt ebu -t -16 -ext mp4 -o normalized/

-nt ebu selects EBU R128 (two-pass), -t -16 is the target. Same result, less plumbing.

6. Verify it worked

Don't trust, measure. The ebur128 filter prints the integrated loudness of the output:

ffmpeg -hide_banner -i output.mp4 -af ebur128 -f null - 2>&1 | tail -n 6

[Parsed_ebur128_0 @ ...] Summary:
  Integrated loudness:
    I:         -16.0 LUFS
    Threshold: -26.3 LUFS
  True peak:
    Peak:       -1.5 dBFS

Integrated loudness at your target, true peak under the ceiling. Run it across a handful of clips and they'll all land at the same level.

⚠️ Edge case: very short or near-silent inputs report integrated loudness near -70 LUFS (the gate floor). Don't blindly amplify those to target, you'll just boost hiss. Skip or flag them.

7. Normalize once, mux into every rendition

If you also build an ABR ladder, you do not want to run loudnorm separately for each rung. Measure and normalize the audio once, then mux that single normalized track into every video rendition. The audio is identical across the ladder and it only got processed one time.

# bash: extract + normalize audio once
ffmpeg -hide_banner -i master.mov -vn -c:a pcm_s16le audio_raw.wav
# (run measure.sh + apply.sh on audio_raw.wav -> audio_norm.m4a, -ar 48000)

# mux the one normalized track into each silent video rendition
for r in 720 480 360; do
  ffmpeg -hide_banner -y \
    -i "video_${r}.mp4" -i audio_norm.m4a \
    -map 0:v:0 -map 1:a:0 -c:v copy -c:a copy -shortest \
    "rendition_${r}.mp4"
done

$ ffprobe -hide_banner rendition_720.mp4 2>&1 | grep -E 'Stream|LUFS'
  Stream #0:0: Video: h264 ... 1280x720
  Stream #0:1: Audio: aac ... 48000 Hz, stereo

Every rung now carries the same -16 LUFS audio. Switching renditions mid-playback won't change the perceived volume, which is the other half of a smooth listening experience.

What's next

Wire normalize.py into your upload pipeline as a worker step after transcode.
Pick the right target per content type: dialogue/podcast near -16, broadcast -23, loud entertainment can sit higher.
If you also build a video ladder, run loudness on the audio track once and mux it into every rendition so they all match.

The filter has been stable in FFmpeg for years (any 5.x+ has it; the 8.0 line is current), so this isn't bleeding-edge. The win is treating loudness as a fixed pipeline setting you decide once and enforce, exactly like a resolution ladder.

Package one CMAF asset for Widevine, PlayReady, and FairPlay (no second encode)

Mason K — Wed, 03 Jun 2026 08:35:16 +0000

TL;DR

We package one CMAF asset encrypted with cbcs, emit an HLS and a DASH manifest from the same segments, and play it back with Shaka Player so Widevine, PlayReady, and FairPlay all decrypt the same bytes. No more separate cenc and cbcs encodes for the common case.

If you last set up DRM a few years ago, you probably remember packaging everything twice: a cenc copy for Widevine/PlayReady and a cbcs copy for FairPlay, because the cipher modes were incompatible. For current devices that is no longer necessary. Modern Widevine and PlayReady both decrypt cbcs, and CMAF lets HLS and DASH share one set of segments. Let's build the single-encode version.

We'll do it in four steps:

Understand the cipher-mode constraint (30 seconds of theory).
Package a CMAF asset with cbcs using Shaka Packager.
Wire up Shaka Player with EME and three license servers.
Handle the FairPlay-specific bits that always bite first.

1. The one constraint that drives everything

There are three DRM systems and they map to devices, not to your preferences:

DRM	Owner	Covers
Widevine	Google	Android, Chrome, Chromecast, many smart TVs
FairPlay	Apple	Safari (iOS/iPadOS/macOS), tvOS
PlayReady	Microsoft	Windows/Edge, Xbox, many connected TVs

Historically Widevine and PlayReady used AES-128-CTR (the cenc scheme) and FairPlay used AES-128-CBC (the cbcs scheme). A cenc file literally cannot be decrypted by FairPlay; different cipher mode.

💡 The 2026 shortcut: encrypt once with cbcs. Current Widevine and PlayReady clients accept it, FairPlay needs it. One encryption, three DRMs.

⚠️ Old devices caveat: some legacy Widevine/PlayReady clients (old CTVs/set-top boxes) only do cenc. Check your playback analytics before going cbcs-only. Most apps shipping today can.

2. Package CMAF with cbcs using Shaka Packager

Shaka Packager is Google's open-source packager and speaks CMAF + cbcs natively. Grab a static build (packager-linux-x64) or run the Docker image.

First, encode your renditions as plain MP4 (this is your normal ABR ladder, no DRM yet):

# bash: a tiny 3-rung ladder for the demo
ffmpeg -i master.mov -c:v libx264 -preset medium -crf 23 -vf scale=1280:720 -an video_720.mp4
ffmpeg -i master.mov -c:v libx264 -preset medium -crf 26 -vf scale=854:480  -an video_480.mp4
ffmpeg -i master.mov -c:v aac -b:a 128k -vn audio.mp4

Now package to CMAF with cbcs encryption. For a real deployment you'd pull keys from a key server over SPEKE; for a local demo you can pass a raw key and key id (KID):

# bash: package.sh
KEY_ID=$(openssl rand -hex 16)
KEY=$(openssl rand -hex 16)

packager \
  'in=video_720.mp4,stream=video,init_segment=out/720/init.mp4,segment_template=out/720/$Number$.m4s' \
  'in=video_480.mp4,stream=video,init_segment=out/480/init.mp4,segment_template=out/480/$Number$.m4s' \
  'in=audio.mp4,stream=audio,init_segment=out/audio/init.mp4,segment_template=out/audio/$Number$.m4s' \
  --protection_scheme cbcs \
  --enable_raw_key_encryption \
  --keys "label=:key_id=${KEY_ID}:key=${KEY}" \
  --generate_static_live_mpd \
  --mpd_output out/manifest.mpd \
  --hls_master_playlist_output out/master.m3u8

The important flags: --protection_scheme cbcs is what makes this asset playable by all three DRMs, and emitting both --mpd_output (DASH) and --hls_master_playlist_output (HLS) from the same run means both manifests point at the same .m4s segments. One set of bytes on disk.

# bash: what you get
$ tree out
out
├── 720/   init.mp4  1.m4s  2.m4s ...
├── 480/   init.mp4  1.m4s  2.m4s ...
├── audio/ init.mp4  1.m4s  2.m4s ...
├── manifest.mpd        # DASH
└── master.m3u8         # HLS, same segments

💡 In production, replace --enable_raw_key_encryption/--keys with --enable_widevine_encryption or a SPEKE key-server URL so a real multi-DRM service mints and stores your keys. Never ship raw keys in your packaging script.

3. Play it back with Shaka Player + EME

Browser DRM runs through Encrypted Media Extensions (EME). Shaka Player (4.x) abstracts EME and picks the right DRM for the current browser. You give it license-server URLs per system; it negotiates the rest.

<!-- index.html -->
<script src="https://cdn.jsdelivr.net/npm/shaka-player@4/dist/shaka-player.compiled.js"></script>
<video id="video" controls autoplay style="width:100%"></video>

// app.js
async function main() {
  shaka.polyfill.installAll();
  const video = document.getElementById('video');
  const player = new shaka.Player(video);

  player.configure({
    drm: {
      servers: {
        'com.widevine.alpha':        'https://YOUR-LICENSE-SERVICE/widevine',
        'com.microsoft.playready':   'https://YOUR-LICENSE-SERVICE/playready',
        'com.apple.fps':             'https://YOUR-LICENSE-SERVICE/fairplay',
      },
      advanced: {
        'com.apple.fps': {
          // FairPlay needs the Apple-issued application certificate up front
          serverCertificateUri: 'https://YOUR-CDN/fairplay.cer',
        },
      },
    },
    // Let Shaka use the same EME path for FairPlay instead of native HLS
    streaming: { useNativeHlsForFairPlay: false },
  });

  player.addEventListener('error', (e) => console.error('shaka error', e.detail));

  // DASH on most browsers; Shaka also reads the HLS manifest if you prefer
  await player.load('/out/manifest.mpd');
  console.log('playing, DRM negotiated:', player.drmInfo()?.keySystem);
}
main();

Shaka inspects the browser: Chrome/Android negotiate com.widevine.alpha, Edge/Windows can use com.microsoft.playready, Safari uses com.apple.fps. Same manifest, same segments, three key systems.

4. The FairPlay bits that always bite

FairPlay is the one that costs you a day, and none of it is the cryptography. Three things to know before you start:

1. You need an Apple-issued application certificate (.cer).
   Enroll in the Apple Developer Program → request the
   FairPlay Streaming deployment package. This is gated and slow.

2. The license exchange is SPC → CKC:
   player generates an SPC (Server Playback Context) →
   your key server returns a CKC (Content Key Context).
   A multi-DRM license service implements the KSM for you.

3. Native Safari can also play FairPlay HLS with NO JS player:
   <video src="master.m3u8"> works if you wire the
   `webkitneedkey` / `encrypted` EME handlers. Shaka does this for you.

If you skip the certificate step and only test in Chrome, everything looks perfect, and then iOS playback fails at launch. Test on a real Apple device early.

Common errors and fixes

NO_LICENSE_SERVER_SPECIFIED (shaka 6001)
  → you encrypted with a key system you didn't add to drm.servers

LICENSE_REQUEST_FAILED (shaka 6007)
  → license server URL wrong, or CORS not allowed on the license endpoint

FAIRPLAY: keySystemAccess denied / no recognized key system
  → missing serverCertificateUri, or testing FairPlay outside Safari

Playback works in Chrome, black screen on old smart TV
  → that device only supports cenc; ship a cenc set for the long tail

What's next

You now have one cbcs CMAF asset serving Widevine, PlayReady, and FairPlay from a single encode. From here:

Swap the raw-key demo for a real multi-DRM license service (SPEKE key provisioning) so keys are minted and rotated for you.
Add hardware-security-level requirements (Widevine L1, HDCP) if you license premium content; studios specify these in contracts.
Read the Shaka Player DRM config docs for persistent licenses and offline playback.

The same single-encode pattern works whether you package yourself with Shaka Packager or hand it to a managed video API. The cipher-mode war that forced the second encode is over; build like it.

Build a custom HLS player in React with hls.js (no wrapper libraries)

Mason K — Sun, 31 May 2026 09:15:26 +0000

TL;DR

We'll build a custom HLS player on top of hls.js 1.6.x and React 19 with no wrapper library. Play/pause, seek, volume, quality readout, buffer state, and proper error recovery. End result is ~150 lines and bends to whatever UI you want.

📦 Code: github.com/USER/react-hls-player-demo (replace before publishing)

This is the "stop installing video.js for the third time" tutorial. The wrapper libraries are doing two things you need (the MSE shim and the error-recovery loop) and a lot of things you don't (a giant control bar, a plugin model, a CSS skin). The MSE shim is hls.js. The error-recovery loop is fifteen lines. Everything else can be your own React.

0. Versions

node 22.x
react 19.x
hls.js 1.6.16   # latest stable as of 2026-04-13

The 1.7 alpha is interesting (interstitial-ads improvements landed in canary on 2026-05-16) but not yet ready for production. Stick with 1.6 unless you're testing the next major.

1. Setup

npm create vite@latest react-hls-player-demo -- --template react-ts
cd react-hls-player-demo
npm install
npm install hls.js

For test streams, the Apple HLS test page has a few public manifests. The bipbop ad-stitched stream is the classic one:

https://devstreaming-cdn.apple.com/videos/streaming/examples/img_bipbop_adv_example_fmp4/master.m3u8

2. The bare minimum: a hook that mounts hls.js

// src/useHls.ts
import { useEffect, useRef } from "react";
import Hls from "hls.js";

export function useHls(src: string) {
  const videoRef = useRef<HTMLVideoElement>(null);

  useEffect(() => {
    const video = videoRef.current;
    if (!video) return;

    // Safari (and iOS) plays HLS natively. Hand the URL to the browser.
    if (video.canPlayType("application/vnd.apple.mpegurl")) {
      video.src = src;
      return;
    }

    if (!Hls.isSupported()) return;

    const hls = new Hls({
      enableWorker: true,
      lowLatencyMode: true,
      backBufferLength: 30,
    });
    hls.loadSource(src);
    hls.attachMedia(video);

    return () => {
      hls.destroy();
    };
  }, [src]);

  return videoRef;
}

The cleanup matters. React 19's StrictMode mounts every effect twice in development. Without the hls.destroy(), you leak one Hls instance per remount and your dev server's memory usage will climb. Ask me how I know.

// src/App.tsx
import { useHls } from "./useHls";

const SRC = "https://devstreaming-cdn.apple.com/videos/streaming/examples/img_bipbop_adv_example_fmp4/master.m3u8";

export default function App() {
  const videoRef = useHls(SRC);
  return <video ref={videoRef} controls className="w-full max-w-4xl" />;
}

That plays HLS in every modern browser. Eight lines. The wrapper libraries' main value-add, "plays HLS in Chrome", is now done. Everything else is product.

3. Wiring real controls

Drop controls from <video> and write your own. Each control is a small piece of state synced to the element.

// src/Player.tsx
import { useEffect, useRef, useState } from "react";
import Hls from "hls.js";

export function Player({ src }: { src: string }) {
  const videoRef = useRef<HTMLVideoElement>(null);
  const hlsRef = useRef<Hls | null>(null);
  const [playing, setPlaying] = useState(false);
  const [time, setTime] = useState(0);
  const [duration, setDuration] = useState(0);
  const [level, setLevel] = useState<string>("auto");

  useEffect(() => {
    const v = videoRef.current!;
    if (v.canPlayType("application/vnd.apple.mpegurl")) {
      v.src = src;
    } else if (Hls.isSupported()) {
      const hls = new Hls({ enableWorker: true, lowLatencyMode: true });
      hls.loadSource(src);
      hls.attachMedia(v);
      hls.on(Hls.Events.LEVEL_SWITCHED, (_, d) => {
        const lvl = hls.levels[d.level];
        setLevel(lvl ? `${lvl.height}p` : "auto");
      });
      hlsRef.current = hls;
    }
    return () => { hlsRef.current?.destroy(); hlsRef.current = null; };
  }, [src]);

  return (
    <div className="player">
      <video
        ref={videoRef}
        onPlay={() => setPlaying(true)}
        onPause={() => setPlaying(false)}
        onTimeUpdate={(e) => setTime(e.currentTarget.currentTime)}
        onLoadedMetadata={(e) => setDuration(e.currentTarget.duration)}
      />
      <div className="controls">
        <button onClick={() => {
          const v = videoRef.current!;
          playing ? v.pause() : v.play();
        }}>{playing ? "Pause" : "Play"}</button>

        <input
          type="range"
          min={0}
          max={duration || 0}
          step={0.1}
          value={time}
          onChange={(e) => {
            const v = videoRef.current!;
            v.currentTime = Number(e.target.value);
          }}
        />

        <span>{fmt(time)} / {fmt(duration)}</span>
        <span className="quality">{level}</span>
      </div>
    </div>
  );
}

function fmt(s: number): string {
  if (!isFinite(s)) return "0:00";
  const m = Math.floor(s / 60);
  const ss = Math.floor(s % 60).toString().padStart(2, "0");
  return `${m}:${ss}`;
}

The quality readout is the part wrappers tend to hide. LEVEL_SWITCHED fires every time hls.js picks a new ABR rendition, and hls.levels[level] has the bitrate, resolution, and codec strings. Surfacing "now playing 720p" in the corner of your player is a five-line feature that meaningfully changes how power users perceive your product.

4. Manual quality picker

Some users will want to force a quality, usually because they don't trust ABR or because they're on a metered connection. Wire it like this:

const [levels, setLevels] = useState<Hls["levels"]>([]);

useEffect(() => {
  const hls = hlsRef.current;
  if (!hls) return;
  hls.on(Hls.Events.MANIFEST_PARSED, () => setLevels(hls.levels));
}, []);

// in JSX:
<select
  value={hls.currentLevel}
  onChange={(e) => { hls.currentLevel = Number(e.target.value); }}
>
  <option value={-1}>Auto</option>
  {levels.map((l, i) => (
    <option key={i} value={i}>{l.height}p ({Math.round(l.bitrate / 1000)} kbps)</option>
  ))}
</select>

-1 means "let ABR pick"; any non-negative index pins the player to that level. After a user pins to 480p, the seek bar still works and the rest of ABR's logic still runs underneath, but the renderer stays at 480p.

5. The error recovery loop (do not skip)

This is the single thing the wrappers were doing that you actually need.

hls.on(Hls.Events.ERROR, (_, data) => {
  if (!data.fatal) return;

  if (data.type === Hls.ErrorTypes.NETWORK_ERROR) {
    console.warn("hls: network error, restarting load");
    hls.startLoad();
    return;
  }
  if (data.type === Hls.ErrorTypes.MEDIA_ERROR) {
    console.warn("hls: media error, recovering");
    hls.recoverMediaError();
    return;
  }
  console.error("hls: fatal", data);
  hls.destroy();
});

Three rules:

Ignore non-fatal errors. hls.js retries those itself.
Network errors get a fresh startLoad(). This is what saves a viewer who walked into a tunnel.
Media errors get recoverMediaError(). This rebuilds the MSE source buffers, which is the only thing that fixes "the decoder gave up after a corrupt segment."

Add a tiny rate limiter in production. If recoverMediaError fails twice within a few seconds, destroy and recreate the Hls instance with a fresh source. Otherwise you can wedge yourself into an infinite recovery loop.

6. Buffer indicator

Showing "this video is buffering" properly means listening to the right events, not just paused.

const [stalled, setStalled] = useState(false);

const v = videoRef.current!;
v.addEventListener("waiting", () => setStalled(true));
v.addEventListener("playing", () => setStalled(false));

For the YouTube-style "this much of the video is buffered ahead" bar, use videoElement.buffered:

const buffered = videoRef.current?.buffered;
let aheadPct = 0;
if (buffered && buffered.length > 0 && duration > 0) {
  aheadPct = (buffered.end(buffered.length - 1) / duration) * 100;
}

Render a second <progress> underneath the seek bar at that percentage. Total time invested: ten minutes.

7. Subtitles

hls.js parses WebVTT tracks from the manifest into audioTracks and subtitleTracks. Toggle them like:

hls.subtitleDisplay = true;
hls.subtitleTrack = 0;        // -1 to disable

For language selection, hls.subtitleTracks is an array with name, lang, and default flags.

8. Cleanup checklist

Things people forget that bite later:

Mistake	Symptom
No `hls.destroy()` in unmount	Memory leak, Chrome devtools shows climbing JS heap
Forgetting Safari native path	App works in Chrome dev, breaks on iPhone in QA
Listening to `pause` instead of `waiting` for buffering	Players show "buffering" when the user manually pauses
Not handling `MEDIA_ERROR`	Random unrecoverable freezes on bad-CDN nights
`Hls.isSupported() === false` ignored	Old browsers show a broken player with no fallback

What's next

There are three places to take this once it's running:

Picture-in-Picture. videoRef.current.requestPictureInPicture(). One line, free feature.
MediaSession metadata. Set navigator.mediaSession.metadata so the OS lock screen shows the right title and artwork.
Player telemetry. Subscribe to Hls.Events.FRAG_LOADED and send the per-segment timings to your analytics. This is gold for debugging "the video stalled, but only for users in Brazil."

If you're playing assets from a managed video API (Mux, FastPix, Cloudflare Stream, api.video), most of them ship their own custom-element player that wires telemetry for you. Use it if you want the analytics for free. Build your own when the player UX is part of the product.

The 1.7 line of hls.js looks promising for interstitial ads and a cleaner LL-HLS toggle. Worth watching the hls.js releases page for the stable 1.7.0 cut later this year.

Happy playing.

Pick a better video thumbnail automatically with FFmpeg, PySceneDetect, and CLIP

Mason K — Sun, 31 May 2026 09:14:33 +0000

TL;DR

We'll build a pipeline that takes any video file, extracts candidate frames with FFmpeg and PySceneDetect, filters out blurry ones with OpenCV, scores each candidate with OpenCLIP against a small prompt set, and picks the top-K thumbnails with a diversity constraint. ~200 lines of Python, GPU-accelerated, fully local.

📦 Code: github.com/USER/auto-thumbnail-picker (replace before publishing)

The default thumbnail your encoder generates is "the middle frame." For most videos, the middle frame is a motion blur, a transition, or someone mid-blink. We can do much better with about an hour of effort. Here's the pipeline.

Versions

python 3.12
ffmpeg 7.1
pyscenedetect 0.6.x
open-clip-torch 2.x
opencv-python 4.x
torch 2.x (with CUDA or MPS)

The pipeline runs on CPU but is noticeably slower for the CLIP step. A consumer GPU (or Apple Silicon MPS) cuts the per-frame encode to a few milliseconds.

What we're building

Extract candidate frames (shot boundaries + uniform sampling).
Filter blurry frames out with Laplacian variance.
Score each remaining frame with OpenCLIP using a positive / negative prompt set.
Apply structural rules (prefer frames with faces).
Pick top-K with shot-level diversity.

1. Setup

python -m venv .venv && source .venv/bin/activate
pip install --break-system-packages \
  open-clip-torch \
  scenedetect[opencv] \
  opencv-python-headless \
  pillow \
  torch torchvision

For face detection we'll keep it simple and use OpenCV's built-in Haar cascades. If you need higher accuracy on small faces, swap to insightface later.

2. Extract candidate frames

Two sources of candidates: shot boundaries (one per shot) and uniform sampling (one every N seconds). Combine both, then dedupe by timestamp.

# extract.py
import subprocess, os, pathlib
from scenedetect import open_video, SceneManager, ContentDetector

def shot_boundaries(video_path: str) -> list[float]:
    """Return scene-cut timestamps (seconds) using PySceneDetect."""
    vid = open_video(video_path)
    sm = SceneManager()
    sm.add_detector(ContentDetector(threshold=27.0))
    sm.detect_scenes(vid, show_progress=False)
    scenes = sm.get_scene_list()
    # Take the start of each scene plus a small offset (avoid the literal cut frame).
    return [s[0].get_seconds() + 0.4 for s in scenes]

def uniform_samples(duration: float, every: float = 3.0) -> list[float]:
    return [t for t in [i * every for i in range(int(duration // every) + 1)] if t > 0]

def extract_frame(video_path: str, t: float, out_path: str) -> None:
    subprocess.run([
        "ffmpeg", "-y", "-ss", f"{t:.3f}", "-i", video_path,
        "-frames:v", "1", "-q:v", "2", out_path
    ], check=True, capture_output=True)

def probe_duration(video_path: str) -> float:
    out = subprocess.run(
        ["ffprobe", "-v", "error", "-show_entries", "format=duration",
         "-of", "default=nokey=1:noprint_wrappers=1", video_path],
        check=True, capture_output=True, text=True,
    ).stdout.strip()
    return float(out)

def build_candidates(video_path: str, out_dir: str) -> list[tuple[float, str]]:
    os.makedirs(out_dir, exist_ok=True)
    dur = probe_duration(video_path)
    times = sorted(set(shot_boundaries(video_path) + uniform_samples(dur)))
    out = []
    for i, t in enumerate(times):
        p = pathlib.Path(out_dir) / f"f_{i:04d}_{t:07.2f}.jpg"
        extract_frame(video_path, t, str(p))
        out.append((t, str(p)))
    return out

💡 Tip: the -ss flag before -i makes ffmpeg seek using the index (fast and inaccurate at frame level). For thumbnail-quality stills we want accurate seeking, but 0.4s after the cut is forgiving enough that index-seek is fine.

Run it:

$ python -c "from extract import build_candidates; print(len(build_candidates('clip.mp4', 'frames/')))"
27

Twenty-seven candidates for a six-minute clip. Manageable.

3. Filter blurry frames

CLIP is too tolerant of blur. We prune with Laplacian variance first.

# sharpness.py
import cv2

def is_sharp(image_path: str, threshold: float = 80.0) -> bool:
    img = cv2.imread(image_path, cv2.IMREAD_GRAYSCALE)
    if img is None:
        return False
    return cv2.Laplacian(img, cv2.CV_64F).var() > threshold

For 1080p source the threshold around 80 works. For 720p drop to ~60. Tune on a few samples from your library; the right cutoff depends on the camera you're shooting with.

4. Score with OpenCLIP

The trick is scoring each frame as positive_similarity - mean(negative_similarity). This is more robust than the absolute positive score because lighting and content normalize out.

# score.py
import torch, open_clip
from PIL import Image

DEVICE = "cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu")

MODEL_NAME = "ViT-L-14"
PRETRAINED = "laion2b_s32b_b82k"

POS = [
    "a sharp, well-lit frame from a video, showing the main subject clearly",
]
NEG = [
    "a blurry frame",
    "a black frame",
    "a transition with text overlay or graphic",
    "a frame caught between two shots",
]

class Scorer:
    def __init__(self) -> None:
        self.model, _, self.preprocess = open_clip.create_model_and_transforms(
            MODEL_NAME, pretrained=PRETRAINED, device=DEVICE
        )
        self.model.eval()
        tokenizer = open_clip.get_tokenizer(MODEL_NAME)
        with torch.no_grad():
            self.pos = self._embed_text(tokenizer(POS))
            self.neg = self._embed_text(tokenizer(NEG))

    def _embed_text(self, tokens):
        with torch.no_grad():
            v = self.model.encode_text(tokens.to(DEVICE))
            return v / v.norm(dim=-1, keepdim=True)

    def score(self, image_path: str) -> float:
        img = self.preprocess(Image.open(image_path).convert("RGB")).unsqueeze(0).to(DEVICE)
        with torch.no_grad():
            i = self.model.encode_image(img)
            i = i / i.norm(dim=-1, keepdim=True)
            pos_sim = (i @ self.pos.T).max().item()
            neg_sim = (i @ self.neg.T).mean().item()
        return pos_sim - neg_sim

⚠️ Note: load the model once and reuse. Reloading on every call is the most common reason people think "CLIP is slow"; it's actually the checkpoint load that's slow.

Quick sanity check:

$ python -c "from score import Scorer; s = Scorer(); print(s.score('frames/f_0010_0030.50.jpg'))"
0.0432

Numbers around 0 to 0.1 are typical. Bigger is better.

5. Face bonus

For UGC-style content the right thumbnail almost always has a clear face. We boost the score when one is present.

# faces.py
import cv2

_cascade = cv2.CascadeClassifier(
    cv2.data.haarcascades + "haarcascade_frontalface_default.xml"
)

def has_visible_face(image_path: str, min_size: int = 80) -> bool:
    img = cv2.imread(image_path, cv2.IMREAD_GRAYSCALE)
    if img is None:
        return False
    faces = _cascade.detectMultiScale(img, scaleFactor=1.1, minNeighbors=5,
                                      minSize=(min_size, min_size))
    return len(faces) > 0

6. Orchestrate

# pick.py
from extract import build_candidates
from sharpness import is_sharp
from faces import has_visible_face
from score import Scorer

def pick_thumbnails(video_path: str, out_dir: str, k: int = 3) -> list[dict]:
    candidates = build_candidates(video_path, out_dir)
    candidates = [(t, p) for t, p in candidates if is_sharp(p)]

    scorer = Scorer()
    rows = []
    for t, p in candidates:
        s = scorer.score(p)
        if has_visible_face(p):
            s += 0.05  # small bias toward faces
        rows.append({"t": t, "path": p, "score": s})
    rows.sort(key=lambda r: r["score"], reverse=True)

    # Diversity: don't pick frames that are within 2 seconds of each other.
    picked: list[dict] = []
    for r in rows:
        if any(abs(r["t"] - p["t"]) < 2.0 for p in picked):
            continue
        picked.append(r)
        if len(picked) >= k:
            break
    return picked

if __name__ == "__main__":
    import sys, json
    print(json.dumps(pick_thumbnails(sys.argv[1], "frames/", k=3), indent=2))

Run:

$ python pick.py clip.mp4
[
  { "t": 84.40, "path": "frames/f_0017_0084.40.jpg", "score": 0.1129 },
  { "t": 23.50, "path": "frames/f_0009_0023.50.jpg", "score": 0.0871 },
  { "t": 165.20, "path": "frames/f_0029_0165.20.jpg", "score": 0.0832 }
]

You now have three diverse, sharp, scored candidate thumbnails. Use the first for the default and keep the other two for A/B testing.

7. (Optional) Crop to aspect ratio

If your product shows thumbnails at 16:9 but your source is 9:16 (vertical UGC), or vice versa, you want a smart crop, not a letterbox.

# crop.py
import cv2

def crop_to_ratio(image_path: str, out_path: str, ratio: tuple[int, int] = (16, 9)) -> None:
    img = cv2.imread(image_path)
    h, w = img.shape[:2]
    target = ratio[0] / ratio[1]
    actual = w / h

    if actual > target:
        # too wide, crop sides
        new_w = int(h * target)
        x0 = (w - new_w) // 2
        cropped = img[:, x0:x0 + new_w]
    else:
        # too tall, crop top/bottom toward the upper third
        new_h = int(w / target)
        y0 = max(0, int(h * 0.20))
        y0 = min(y0, h - new_h)
        cropped = img[y0:y0 + new_h, :]
    cv2.imwrite(out_path, cropped)

The "upper third" bias for vertical content is intentional. Faces in vertical video almost always sit in the upper third; a centered crop loses them. For higher quality, swap this for a face-aware crop using the bounding box from step 5.

8. Where to put this in your pipeline

Stage	When
Inline at upload	Easiest. Adds 5–30s to the "video processing" step.
Background worker (recommended)	Consume an "asset.ready" webhook, run, write back to your DB.
Re-run on edit	Re-compute when the user trims or reorders clips.

If you're using a managed video API, you'll already get an "asset ready" webhook. Hook a background worker to that and write the picked thumbnail back to your CMS row. Both FastPix and Mux let you set a custom thumbnail URL on the asset.

What's next

Three upgrades that are worth their complexity:

Better face detector. Replace Haar with insightface for small-face robustness.
Learned scorer. If you have click-through data, fine-tune a small head on top of the CLIP embeddings. The hand-tuned prompts get you 80% of the way; learning the last 20% is real ROI.
Vision-language scoring. Ask a small VLM ("rate this thumbnail's appeal 1-10") instead of cosine similarity. More expensive, sometimes worth it for high-stakes content.

The thing this pipeline gets right is that the model is the cheap part. The leverage is the candidate selection, the structural filters, and the diversity constraint. Get those right with a vanilla CLIP and you're already beating most platforms' default thumbnails.

Happy thumbnail picking.

Build a video upload + HLS playback flow in Next.js 15 (with direct uploads)

Mason K — Tue, 26 May 2026 10:38:29 +0000

TL;DR

We're going to build a "user uploads a video, user watches the video" feature in a Next.js 15 app. The browser uploads directly to a managed video API so our server never holds bytes, a webhook flips the row from processing to ready, and the watch page plays an HLS manifest. ~120 lines across four files.

📦 Code: github.com/USER/nextjs-video-upload-demo (replace before publishing)

We'll use FastPix as the managed API in this tutorial because the encoding is free on the standard plan and the $25 signup credit covers the bytes you'll burn while iterating. The same shape works against Mux, api.video, or Cloudflare Stream. Only the endpoint names and webhook field names change.

What we're building

POST /api/uploads returns a one-shot direct-upload URL.
Browser PUTs the file to that URL. Our server never sees the bytes.
POST /api/webhooks/fastpix receives an "asset ready" callback and stores the playbackId in our DB.
/watch/[id] renders a player against https://stream.fastpix.io/<playbackId>.m3u8.

💡 Why direct upload? Next.js 15 Server Actions cap request bodies at 1 MB by default, and even Route Handlers with bumped limits will sit there holding a 300 MB file in a serverless function while encoding hasn't started. Direct upload pushes bytes straight to the storage layer.

1. Project setup

npx create-next-app@latest nextjs-video-upload-demo --typescript --app --tailwind
cd nextjs-video-upload-demo
npm install

Make a .env.local:

# .env.local
FASTPIX_TOKEN_ID=pk_...
FASTPIX_SECRET=sk_...
FASTPIX_WEBHOOK_SECRET=whsec_...
NEXT_PUBLIC_APP_URL=http://localhost:3000
DATABASE_URL=file:./dev.db

The token and secret come from dashboard.fastpix.io (Access Tokens). Webhooks live at https://docs.fastpix.io/docs/webhooks-collection; create a webhook pointing at your tunneled URL (more on that below).

Versions you want to be on: Node 22.x or newer, Next.js 15.x, hls.js 1.6.x for the player.

2. The "create upload" Route Handler

// app/api/uploads/route.ts
import { NextResponse } from "next/server";
import { db } from "@/lib/db";

export async function POST() {
  const auth = Buffer.from(
    `${process.env.FASTPIX_TOKEN_ID}:${process.env.FASTPIX_SECRET}`
  ).toString("base64");

  const res = await fetch("https://api.fastpix.io/v1/on-demand", {
    method: "POST",
    headers: {
      Authorization: `Basic ${auth}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      corsOrigin: process.env.NEXT_PUBLIC_APP_URL,
      playbackPolicy: ["public"],
    }),
  });

  if (!res.ok) {
    const txt = await res.text();
    return NextResponse.json({ error: txt }, { status: 502 });
  }

  const data = await res.json();

  const row = await db.video.create({
    data: {
      providerAssetId: data.id,
      status: "processing",
    },
  });

  return NextResponse.json({
    videoId: row.id,
    uploadUrl: data.uploadUrl,
  });
}

cors_origin matters. The browser is about to PUT bytes from your origin to FastPix's upload host, so the upload URL needs to allow your origin. Get this wrong and Chrome will throw a CORS error that looks scarier than it is.

⚠️ Note: don't return assetId to the browser. We stash it server-side and hand the client a stable videoId instead. Keeps the asset-id mapping under our control.

3. The upload component

// app/upload/page.tsx
"use client";

import { useState } from "react";
import { useRouter } from "next/navigation";

export default function UploadPage() {
  const router = useRouter();
  const [progress, setProgress] = useState(0);
  const [error, setError] = useState<string | null>(null);

  async function handleUpload(file: File) {
    setError(null);
    setProgress(0);

    const r = await fetch("/api/uploads", { method: "POST" });
    if (!r.ok) { setError("Could not create upload"); return; }
    const { videoId, uploadUrl } = await r.json();

    await new Promise<void>((resolve, reject) => {
      const xhr = new XMLHttpRequest();
      xhr.open("PUT", uploadUrl);
      xhr.upload.onprogress = (e) => {
        if (e.lengthComputable) setProgress(Math.round((e.loaded / e.total) * 100));
      };
      xhr.onload = () => xhr.status < 300 ? resolve() : reject(new Error(`PUT ${xhr.status}`));
      xhr.onerror = () => reject(new Error("Network error"));
      xhr.send(file);
    }).catch((e) => setError(String(e)));

    router.push(`/watch/${videoId}`);
  }

  return (
    <main className="p-8">
      <input
        type="file"
        accept="video/*"
        onChange={(e) => e.target.files?.[0] && handleUpload(e.target.files[0])}
      />
      {progress > 0 && <p>Uploading: {progress}%</p>}
      {error && <p className="text-red-600">{error}</p>}
    </main>
  );
}

We use XMLHttpRequest instead of fetch because fetch still doesn't expose upload progress in a useful way in 2026, and "uploading: 47%" is the difference between the user closing the tab and waiting.

💡 Tip: for production, swap this for the official FastPix Web upload SDK. It does chunked uploads, retries, and resume on flaky networks. Worth it for mobile users.

4. The webhook handler

// app/api/webhooks/fastpix/route.ts
import { NextResponse } from "next/server";
import { createHmac, timingSafeEqual } from "node:crypto";
import { db } from "@/lib/db";

export async function POST(req: Request) {
  const raw = await req.text();
  const sig = req.headers.get("fastpix-signature") ?? "";

  const expected = createHmac("sha256", process.env.FASTPIX_WEBHOOK_SECRET!)
    .update(raw)
    .digest("hex");

  // timing-safe compare
  if (sig.length !== expected.length ||
      !timingSafeEqual(Buffer.from(sig), Buffer.from(expected))) {
    return NextResponse.json({ error: "bad signature" }, { status: 401 });
  }

  const evt = JSON.parse(raw);
  if (evt.type !== "video.asset.ready") {
    return NextResponse.json({ ok: true });
  }

  const asset = evt.data;
  await db.video.update({
    where: { providerAssetId: asset.id },
    data: { status: "ready", playbackId: asset.playbackIds?.[0]?.id },
  });

  return NextResponse.json({ ok: true });
}

Two things to notice:

We read the raw body as text, then JSON-parse it ourselves. If you use req.json() first, the HMAC won't match because Node has already re-serialized the object.
The signature check is timingSafeEqual. Don't compare with ===; it leaks bytes through timing.

⚠️ Note: webhook event names and field shapes vary by provider. If you're swapping in Mux, the event is video.asset.ready too, but the signature header is Mux-Signature and the payload uses playback_ids (snake case). Always read the provider's webhook reference.

Testing webhooks locally

# In one terminal:
npm run dev

# In another, expose 3000 to the internet:
ngrok http 3000

# Register the ngrok URL in dashboard.fastpix.io → Webhooks
# Format: https://abc123.ngrok-free.app/api/webhooks/fastpix

Tail the output. Upload a clip from the dev UI. You should see a POST to /api/webhooks/fastpix within 30 to 90 seconds for a short clip.

5. The watch page

// app/watch/[id]/page.tsx
import { db } from "@/lib/db";
import { Player } from "./Player";

export default async function WatchPage({ params }: { params: { id: string } }) {
  const v = await db.video.findUnique({ where: { id: params.id } });
  if (!v) return <p>Not found</p>;
  if (v.status !== "ready" || !v.playbackId) {
    return <p>Still processing. This page auto-refreshes.</p>;
  }
  const src = `https://stream.fastpix.io/${v.playbackId}.m3u8`;
  return <Player src={src} />;
}

And a client component for the player itself:

// app/watch/[id]/Player.tsx
"use client";

import { useEffect, useRef } from "react";
import Hls from "hls.js";

export function Player({ src }: { src: string }) {
  const ref = useRef<HTMLVideoElement>(null);

  useEffect(() => {
    const v = ref.current;
    if (!v) return;

    // Safari plays HLS natively.
    if (v.canPlayType("application/vnd.apple.mpegurl")) {
      v.src = src;
      return;
    }

    if (!Hls.isSupported()) return;
    const hls = new Hls();
    hls.loadSource(src);
    hls.attachMedia(v);
    return () => hls.destroy();
  }, [src]);

  return <video ref={ref} controls className="w-full max-w-4xl" />;
}

That's the whole player for a public asset. For private assets, mint a short-lived JWT on the server and append it as a query string. The JWT is signed with an asymmetric key pair you generate once and store in your secret manager.

6. Sanity-check the round trip

# Upload a small clip via the UI.
# Watch the dev server logs.
# You should see:
POST /api/uploads 200
PUT https://upload.fastpix.io/... 200   # browser → FastPix
POST /api/webhooks/fastpix 200          # ~30-90s later
GET  /watch/<videoId> 200

If the webhook never fires, three things go wrong in this order, every time: ngrok URL changed (re-register it), CORS misconfigured on the upload (check the cors_origin you passed), or the webhook secret in .env.local doesn't match the one in the dashboard.

What about the player UI?

We used <video controls> for this tutorial, which gives you the browser default controls. For a real product you'll want a custom player. Two paths:

Option	When to pick it
FastPix Web Player (custom element)	Default. You drop `<fastpix-player playback-id="...">` and the player + analytics wiring come together.
Build on hls.js directly	You want full UI control. More code, total ownership of the surface.

I have a longer piece on the second path that covers the events, the error recovery loop, and React StrictMode cleanup. For this tutorial, the default <video controls> is enough to ship.

What's next

QoE analytics. Wire the Video Data SDK (or your provider's equivalent). FastPix Video Data is free up to 100,000 views per month, which covers most side-project and SaaS workloads before you ever pay.
Resume on interruption. Replace the raw XMLHttpRequest upload with the official Web upload SDK to get chunked uploads and resume.
Signed playback. Switch the playback policy from public to signed and add JWT minting to the watch page.
Thumbnails. The default thumbnail is the middle frame. Pick a better one.

The pattern transfers cleanly. If you build this on Mux, swap the endpoint to https://api.mux.com/video/v1/uploads, the playback URL to https://stream.mux.com/<playbackId>.m3u8, and the webhook handler to verify Mux-Signature. Cloudflare Stream and api.video have the same three pieces with their own dialect.

The thing this tutorial is really about, more than any one provider, is the shape: broker upload tokens on the server, push bytes from the browser, receive webhooks, render manifests. Once you have that shape in your head, the next video feature you build will take an afternoon instead of a sprint.

Building a shot-detection worker for an upload pipeline with PySceneDetect 0.7

Mason K — Thu, 21 May 2026 09:52:40 +0000

📦 Code: github.com/USER/shot-detection-worker (replace before publishing)

TL;DR

We are going to build an upload worker that runs shot detection on a video, writes the boundary list to Postgres, and produces a 6-frame storyboard PNG for use as a hover-preview sprite. Stack is PySceneDetect 0.7 (released earlier this month), FFmpeg 8.1.1, Python 3.12, and a Redis-backed queue. About 120 lines of code.

Shot detection is one of those building blocks that quietly powers a lot of features people associate with "video AI": chapter generation, smart thumbnails, hover-preview sprites, the first step of any auto-clipping pipeline. The open-source path covers more ground than people expect, and PySceneDetect 0.7 (which dropped on 2026-05-03) is the version I would build on today.

Let's wire it up end to end.

🛠️ 1. Setup

# bash
mkdir shot-worker && cd shot-worker
python3.12 -m venv .venv && source .venv/bin/activate
pip install scenedetect==0.7 opencv-python-headless==4.10.0.84 \
            rq==1.16 psycopg[binary]==3.2.0 boto3==1.34.0

We also need FFmpeg on the host:

# bash
$ ffmpeg -version | head -1
ffmpeg version 8.1.1 ...

A throwaway DB for the boundaries table:

-- schema.sql
CREATE TABLE shots (
    asset_id   TEXT NOT NULL,
    shot_index INT  NOT NULL,
    start_s    NUMERIC(10, 3) NOT NULL,
    end_s      NUMERIC(10, 3) NOT NULL,
    metric     NUMERIC(10, 3),
    PRIMARY KEY (asset_id, shot_index)
);

CREATE INDEX shots_asset_idx ON shots (asset_id);

# bash
psql -d shotworker -f schema.sql

🎬 2. The detector picker

PySceneDetect 0.7 ships five detectors, and the right one depends on the content. A small router up front saves a lot of pain later.

# worker/detectors.py
from scenedetect import AdaptiveDetector, ContentDetector, ThresholdDetector
from scenedetect.detectors import HistogramDetector, HashDetector

def pick_detector(content_class: str):
    """Pick the PySceneDetect detector that suits the content class.

    content_class comes from the upload metadata: 'talking_head', 'sports',
    'animation', 'screen_recording', etc. Tune thresholds against your own
    samples; the defaults here are starting points, not ground truth.
    """
    match content_class:
        case "sports" | "action":
            # Rolling-average baseline handles fast camera motion better.
            return AdaptiveDetector(adaptive_threshold=3.0, min_scene_len=15)
        case "animation":
            # Histogram delta works better than HSV deltas on animated content.
            return HistogramDetector(threshold=0.05, min_scene_len=15)
        case "screen_recording":
            # Perceptual hashing skips long static stretches.
            return HashDetector(threshold=0.395, min_scene_len=30)
        case "fade_heavy":
            # Threshold-based detection catches fade-to-black act breaks.
            return ThresholdDetector(threshold=12.0, min_scene_len=15)
        case _:
            # Default: HSV content delta. Works on most diverse content.
            return ContentDetector(threshold=27.0, min_scene_len=15)

💡 Tip: keep a small set of labeled test clips (one per content class) and re-run the detector on them whenever you touch thresholds. The "did my change regress" question gets cheaper to answer fast.

🔍 3. The detection function

PySceneDetect's high-level API is small enough that the whole detection step is a dozen lines:

# worker/detect.py
from pathlib import Path

from scenedetect import open_video, SceneManager
from worker.detectors import pick_detector


def detect_shots(video_path: Path, content_class: str = "default"):
    """Run shot detection on a video file.

    Returns a list of (start_seconds, end_seconds, metric_score) tuples.
    """
    video = open_video(str(video_path))
    scene_manager = SceneManager()
    detector = pick_detector(content_class)
    scene_manager.add_detector(detector)

    scene_manager.detect_scenes(video=video, show_progress=False)

    shots = []
    for i, (start, end) in enumerate(scene_manager.get_scene_list()):
        shots.append((
            start.get_seconds(),
            end.get_seconds(),
            None,  # metric per scene not exposed directly in 0.7
        ))
    return shots

You can also grab the per-frame metric data by adding a StatsManager and writing it to CSV, which is invaluable when threshold tuning:

# worker/detect.py (extended version with stats)
from scenedetect import StatsManager

def detect_shots_with_stats(video_path, content_class, stats_csv):
    video = open_video(str(video_path))
    stats = StatsManager()
    scene_manager = SceneManager(stats_manager=stats)
    scene_manager.add_detector(pick_detector(content_class))
    scene_manager.detect_scenes(video=video, show_progress=False)
    stats.save_to_csv(stats_csv)
    return [(s.get_seconds(), e.get_seconds(), None)
            for s, e in scene_manager.get_scene_list()]

🖼️ 4. The storyboard

For a hover-preview sprite, we want one frame per shot. FFmpeg does the heavy lifting; PySceneDetect tells it where to look:

# worker/storyboard.py
import subprocess
from pathlib import Path


def extract_keyframes(video_path: Path, shots, out_dir: Path, max_frames: int = 6):
    """Extract one frame per shot, capped at max_frames total."""
    out_dir.mkdir(parents=True, exist_ok=True)

    # If there are more shots than slots, sample evenly.
    if len(shots) > max_frames:
        step = len(shots) / max_frames
        sampled = [shots[int(i * step)] for i in range(max_frames)]
    else:
        sampled = shots

    paths = []
    for i, (start_s, end_s, _) in enumerate(sampled):
        midpoint = start_s + (end_s - start_s) / 2
        out_path = out_dir / f"frame_{i:02d}.jpg"
        subprocess.run(
            [
                "ffmpeg", "-ss", f"{midpoint:.3f}", "-i", str(video_path),
                "-frames:v", "1", "-q:v", "3", "-vf", "scale=320:-2",
                "-y", str(out_path),
            ],
            check=True,
            stderr=subprocess.DEVNULL,
        )
        paths.append(out_path)

    # Stitch into a 1x6 sprite.
    sprite_path = out_dir / "sprite.png"
    subprocess.run(
        ["ffmpeg", "-i", str(out_dir / "frame_%02d.jpg"),
         "-vf", f"tile={max_frames}x1", "-y", str(sprite_path)],
        check=True,
        stderr=subprocess.DEVNULL,
    )
    return sprite_path

-ss before -i is intentional: it lets FFmpeg seek before decoding, which makes the extract cheap on long videos. Putting -ss after -i reads from the start of the file every time, which is fine on a 30-second clip and miserable on a 90-minute one.

🔌 5. The worker

We tie everything together with an RQ job. Whichever queue you use, the shape is the same: pull asset, detect, write boundaries, produce storyboard, mark asset ready.

# worker/job.py
import json
import logging
from pathlib import Path
import tempfile

import boto3
import psycopg

from worker.detect import detect_shots_with_stats
from worker.storyboard import extract_keyframes

logger = logging.getLogger(__name__)
s3 = boto3.client("s3")


def process_upload(asset_id: str, bucket: str, key: str,
                   content_class: str, dsn: str):
    """End-to-end shot detection + storyboard for a single uploaded asset."""
    with tempfile.TemporaryDirectory() as tmp:
        local = Path(tmp) / "input.mp4"
        s3.download_file(bucket, key, str(local))
        logger.info("downloaded %s/%s -> %s", bucket, key, local)

        stats_csv = Path(tmp) / "stats.csv"
        shots = detect_shots_with_stats(local, content_class, stats_csv)
        logger.info("detected %d shots in %s", len(shots), asset_id)

        # Persist boundaries
        with psycopg.connect(dsn) as conn:
            with conn.cursor() as cur:
                cur.execute(
                    "DELETE FROM shots WHERE asset_id = %s", (asset_id,),
                )
                cur.executemany(
                    "INSERT INTO shots (asset_id, shot_index, start_s, end_s, metric) "
                    "VALUES (%s, %s, %s, %s, %s)",
                    [(asset_id, i, s, e, m) for i, (s, e, m) in enumerate(shots)],
                )

        # Storyboard
        out_dir = Path(tmp) / "storyboard"
        sprite = extract_keyframes(local, shots, out_dir, max_frames=6)
        s3.upload_file(str(sprite), bucket, f"storyboards/{asset_id}.png")
        s3.upload_file(str(stats_csv), bucket, f"shot-stats/{asset_id}.csv")

        return {
            "asset_id": asset_id,
            "shots": len(shots),
            "sprite_key": f"storyboards/{asset_id}.png",
        }

Hooking it up to RQ:

# worker/run.py
from rq import Queue, Worker
from redis import Redis
import os

if __name__ == "__main__":
    redis = Redis.from_url(os.environ["REDIS_URL"])
    q = Queue("uploads", connection=redis)
    with Worker([q], connection=redis) as w:
        w.work()

Enqueue an upload job from your upload handler:

# in your upload handler
from rq import Queue
from redis import Redis

q = Queue("uploads", connection=Redis.from_url(REDIS_URL))
q.enqueue(
    "worker.job.process_upload",
    asset_id="upload_abc123",
    bucket="my-uploads",
    key="raw/abc123.mp4",
    content_class="talking_head",
    dsn=os.environ["DB_DSN"],
    job_timeout=600,
)

⚡ 6. Throughput and gotchas

A few notes from real deployments:

PySceneDetect is CPU-bound and decode-heavy. The detector itself is fast; the bottleneck is OpenCV decoding the video. On commodity CPU (4 vCPU, 8 GB RAM), 1080p talking-head content processes faster than real time. Mileage varies on dense 4K content.
Use opencv-python-headless on servers. The full opencv-python package pulls in GTK/Qt and breaks in containerized environments.
Sample frames if you do not need pixel-precise boundaries. PySceneDetect has a downscale_factor argument that subsamples the input. For chapter-generation use cases, a 2x downscale halves processing time and changes the boundary list by a frame or two at most.
The CSV stats file is gold. Save it. The day a content class regresses and the PM asks "why did chapters drop", that file is the answer.

# faster detection at the cost of frame-level precision
video = open_video(str(video_path), backend="opencv")
scene_manager = SceneManager()
scene_manager.auto_downscale = True
scene_manager.detect_scenes(video=video, frame_skip=2)

⚠️ Note: frame_skip improves throughput but means the detector misses very short shots (a 4-frame quick cut at frame_skip=2 may not register). Tune to your content.

🧪 7. Verifying the output

A small script that opens the storyboard and prints the boundary list:

# bash
$ python -m worker.cli verify upload_abc123
asset upload_abc123
  shots: 14
  total: 73.21s
  storyboard: s3://my-uploads/storyboards/upload_abc123.png
  sample boundaries:
    [00:00:00.000 - 00:00:04.120]  (shot 0)
    [00:00:04.120 - 00:00:09.480]  (shot 1)
    [00:00:09.480 - 00:00:13.000]  (shot 2)
    ...

Open the sprite. One frame per shot, six total, evenly distributed.

What's next

A few directions to take this once the baseline works:

Chapter generation. Group shots that are at least 30 seconds long; the rest is a rules engine on top of the boundary list.
Smart thumbnails. Pipe the keyframes through a sharpness + face score and pick the best per shot.
Auto-clipping. Detect "interesting" moments by some other signal (audio energy, transcript keywords) and snap them to the nearest shot boundary; the clips stop looking like clips and start looking like edits.

The library does the boring part well, so you get to spend the engineering budget on the parts that actually feel like product.

video #python #tutorial #ffmpeg

Wiring VMAF (and PSNR) into your encoder CI with FFmpeg 8.1 and ffmpeg-quality-metrics

Mason K — Thu, 21 May 2026 09:51:09 +0000

📦 Code: github.com/USER/encoder-qa-ci (replace before publishing)

TL;DR

We are going to wire a perceptual-quality gate into a CI workflow using FFmpeg 8.1.1, libvmaf, and the ffmpeg-quality-metrics Python wrapper. The job runs PSNR, SSIM, and VMAF against a fixed reference ladder and fails the merge if VMAF drops below threshold. Works on CPU; the same setup runs roughly 6x faster on GPU with VMAF-CUDA.

PSNR has been the default "is my encoder okay" metric in CI pipelines for a decade, and it is starting to show its age. It cannot tell when the encoder traded perceptual quality for raw pixel error, and that is exactly the failure mode per-title and ML-driven encoders walk into. Let's wire up something better.

We will build this in three steps:

Stand up a tiny test bench with a reference ladder.
Run PSNR + SSIM + VMAF with ffmpeg-quality-metrics.
Wrap the whole thing in a CI script that fails on regression.

🛠️ 1. The test bench

You need three things: a reference master, a set of encoded renditions ("distorted" in metric-speak), and a fixed VMAF model file.

# bash
mkdir encoder-qa && cd encoder-qa
mkdir reference distorted models

# A short, representative reference. Pick a clip that matches your worst-case content:
# face-heavy, motion, fine detail. The "TearsOfSteel" or "BigBuckBunny" clips work for demos.
curl -L -o reference/master.mov \
  https://test-videos.co.uk/vids/bigbuckbunny/mp4/h264/720/Big_Buck_Bunny_720_10s_5MB.mp4

Drop the VMAF model into models/. The vmaf_v0.6.1.json model is the most widely cited Netflix VMAF model; there are also vmaf_4k_v0.6.1.json for 4K viewing and a phone variant. Pin the one you use, exactly:

# bash (clone the model into the repo so CI doesn't fetch over the network)
git clone --depth 1 https://github.com/Netflix/vmaf.git /tmp/vmaf
cp /tmp/vmaf/model/vmaf_v0.6.1.json models/

💡 Tip: treat the model file like a lockfile. Different VMAF model versions produce different scores. If you upgrade, you re-baseline every threshold.

Now generate a reference ladder. We use FFmpeg 8.1.1; older versions are fine but you want at least 6.1 for VMAF-CUDA, and 8.x for the cleanest webvtt/libsvtav1 integration.

# bash (produce three renditions for the test ladder)
ffmpeg -i reference/master.mov -c:v libx264 -preset medium -crf 23 -vf scale=1280:720  distorted/720p.mp4
ffmpeg -i reference/master.mov -c:v libx264 -preset medium -crf 26 -vf scale=854:480   distorted/480p.mp4
ffmpeg -i reference/master.mov -c:v libx264 -preset medium -crf 28 -vf scale=640:360   distorted/360p.mp4

Verify the FFmpeg version. The libvmaf and libsvtav1 wiring tightened up across the 8.0 → 8.1 line:

# bash
$ ffmpeg -version | head -1
ffmpeg version 8.1.1 ...
$ ffmpeg -filters | grep -E "libvmaf|psnr|ssim"
 .. libvmaf            VV->V       Calculate the VMAF between two video streams.
 .. psnr               VV->V       Calculate the PSNR between two video streams.
 .. ssim               VV->V       Calculate the SSIM between two video streams.

If libvmaf is not listed, your FFmpeg was built without --enable-libvmaf. Most distro builds ship it; static builds from BtbN/FFmpeg-Builds include it by default.

📐 2. Running the metrics

You can call ffmpeg -lavfi libvmaf... directly, but the output format is awkward to parse, and you end up writing the same Python wrapper everyone else has. Use ffmpeg-quality-metrics (slhck/ffmpeg-quality-metrics, still actively maintained). It runs all three metrics in one pass, emits JSON, and handles the model-path plumbing for you.

# bash
pip install ffmpeg-quality-metrics

A single rendition through the gate:

# bash
ffmpeg-quality-metrics distorted/720p.mp4 reference/master.mov \
  --metrics psnr ssim vmaf \
  --vmaf-model-path models/vmaf_v0.6.1.json \
  --output-format json > metrics_720p.json

The output looks like this (truncated):

{
  "global": {
    "psnr": { "psnr_avg": 41.82, "psnr_min": 36.41, "psnr_max": 45.97 },
    "ssim": { "ssim_avg": 0.978 },
    "vmaf": { "vmaf_avg": 88.7, "vmaf_min": 71.2, "vmaf_max": 96.4 }
  },
  "input_file_dist": "distorted/720p.mp4",
  "input_file_ref": "reference/master.mov"
}

global is the aggregate. The per-frame data is also in the JSON if you ask for --output-format json with the frame-level flag, and that is the file you want when an encoder regresses and you need to find which 200 frames lost 8 points.

⚠️ Note: VMAF on CPU is workable on short clips but slow on long ones. If you have NVENC-capable hardware, add --vmaf-features cuda (the wrapper passes it through to libvmaf) and decode through h264_cuvid / hevc_cuvid. The roughly 6x speedup NVIDIA documents lines up with what I see in practice when I keep frames on the GPU end-to-end.

🚦 3. Turning it into a CI gate

The script: run the metrics on every rendition, compare the aggregate against a per-rendition threshold, exit non-zero on regression.

# scripts/qa_gate.py
import json
import subprocess
import sys
from pathlib import Path

# Per-rendition VMAF floor. Tune to your content.
# Tighter on top renditions, looser on the bottom rung.
THRESHOLDS = {
    "720p": {"vmaf_avg": 85, "vmaf_min": 65, "ssim_avg": 0.96},
    "480p": {"vmaf_avg": 78, "vmaf_min": 55, "ssim_avg": 0.94},
    "360p": {"vmaf_avg": 70, "vmaf_min": 45, "ssim_avg": 0.92},
}

REFERENCE = "reference/master.mov"
MODEL = "models/vmaf_v0.6.1.json"

def run_metrics(distorted: Path) -> dict:
    out = subprocess.check_output([
        "ffmpeg-quality-metrics", str(distorted), REFERENCE,
        "--metrics", "psnr", "ssim", "vmaf",
        "--vmaf-model-path", MODEL,
        "--output-format", "json",
    ])
    return json.loads(out)["global"]

def main() -> int:
    failures = []
    for name, floor in THRESHOLDS.items():
        rendition = Path(f"distorted/{name}.mp4")
        if not rendition.exists():
            failures.append(f"{name}: rendition missing")
            continue
        metrics = run_metrics(rendition)
        for metric, minimum in floor.items():
            value = metrics.get(metric.split("_")[0], {}).get(metric)
            if value is None or value < minimum:
                failures.append(f"{name} {metric}: got {value}, want >= {minimum}")
        print(f"{name}: vmaf {metrics['vmaf']['vmaf_avg']:.1f}, "
              f"ssim {metrics['ssim']['ssim_avg']:.3f}, "
              f"psnr {metrics['psnr']['psnr_avg']:.1f}")
    if failures:
        print("\nFAILURES:")
        for f in failures:
            print(f"  - {f}")
        return 1
    return 0

if __name__ == "__main__":
    sys.exit(main())

Run it locally first:

# bash
$ python scripts/qa_gate.py
720p: vmaf 88.7, ssim 0.978, psnr 41.8
480p: vmaf 81.2, ssim 0.961, psnr 38.9
360p: vmaf 72.4, ssim 0.943, psnr 36.1

The output you actually want is when somebody breaks the encoder:

# bash
$ python scripts/qa_gate.py
720p: vmaf 79.1, ssim 0.962, psnr 42.4
480p: vmaf 81.2, ssim 0.961, psnr 38.9
360p: vmaf 72.4, ssim 0.943, psnr 36.1

FAILURES:
  - 720p vmaf_avg: got 79.1, want >= 85
exit code: 1

Look at what PSNR did here: it actually went up (42.4 vs the previous 41.8). The encoder traded perceptual quality for raw pixel error and a PSNR-only gate would have shipped it. The VMAF gate caught it.

🤖 4. The GitHub Actions job

# .github/workflows/encoder-qa.yml
name: encoder-qa
on:
  pull_request:
    paths:
      - "encoder/**"
      - "scripts/qa_gate.py"

jobs:
  qa:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4

      - name: Install FFmpeg 8.1.1
        run: |
          curl -L -o ffmpeg.tar.xz \
            https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-linux64-gpl.tar.xz
          tar xf ffmpeg.tar.xz --strip-components=2 -C /usr/local/bin --wildcards '*/bin/ffmpeg' '*/bin/ffprobe'
          ffmpeg -version | head -1

      - name: Install metrics wrapper
        run: pip install ffmpeg-quality-metrics

      - name: Build the test ladder
        run: ./scripts/build_ladder.sh

      - name: Run quality gate
        run: python scripts/qa_gate.py

A few notes on this in production:

Cache the reference master in an S3 bucket or a Git LFS object. Re-fetching from a public CDN on every run is a recipe for flaky builds.
Treat the model file as a build input. Bump it deliberately, and re-baseline thresholds when you do.
Keep the per-frame JSON as a build artifact for at least 30 days. When an encoder regresses, the headline number tells you it broke; the per-frame data tells you where.

⚠️ Things that bite you in real workflows

A short list of things I have had to fix on actual CI pipelines:

Different frame counts between reference and distorted. Trim with ffmpeg -ss/-to on both sides, or VMAF will silently truncate to the shorter one and the score will surprise you.
Color space mismatches. A BT.709 source compared against a BT.601 encode will produce noisy VMAF scores. Normalize with zscale=transfer=bt709:matrix=bt709.
Resolution mismatches. libvmaf scales the distorted to match the reference; if you really want to test "how this looks on a 720p screen", scale the reference down to 720p first, then compare.
CPU vs CUDA scores drift slightly. Same model, same files, different paths through the math. Pick one for CI and stick to it.

What's next

A few directions worth exploring:

Per-title encoding QA. The same gate, but a thresholds-per-content-class table. Talking heads, sports, animation, screen content; each gets its own floor.
Frame-level alerting. Plot the per-frame VMAF in your CI artifact viewer. A two-second dip below 60 is often more telling than the global average.
GPU-resident pipelines. If you encode with NVENC and want sub-realtime QA on long content, run h264_cuvid decoders into a libvmaf filter graph that stays on the GPU. The 6x speedup is real; the plumbing is fiddly enough to be its own post.

Wire it up, get the gate green, and the first time a PR turns it red you will be glad you stopped trusting PSNR as the whole story.

video #ffmpeg #tutorial #devops

Shipping WebVTT subtitles in HLS that actually stay in sync (a hands-on guide for 2026)

Mason K — Thu, 21 May 2026 09:49:23 +0000

📦 Code: github.com/USER/hls-webvtt-pipeline (replace before publishing)

TL;DR

We are going to build a real HLS subtitle pipeline. Take an SRT file, convert to WebVTT, segment it the way HLS expects, add the X-TIMESTAMP-MAP cue, declare it in the manifest, and verify it plays in sync in HLS.js 1.6.16 and Shaka Player. By the end you will have a working ladder + subtitle track and the confidence that it will not drift on seek.

If you have shipped HLS captions as a single sidecar .vtt and called it done, this is the upgrade you have been putting off. The bug it fixes (caption drift after seek, especially on Chrome on Android) is the kind of bug that gets reported as "the subtitles are wrong", with no useful repro steps, by users who never come back. Let's build the version that does not break.

🛠️ 1. The pieces we need

FFmpeg 8.1.1 (webvtt muxer; older versions work but the muxer behavior is friendlier in 8.x).
A packager. We will use shaka-packager because it has the cleanest support for segmented WebVTT and fMP4-wrapped subtitles. mp4box works too.
HLS.js 1.6.16 (latest as of April 2026) and Shaka Player 4.x for verification.
A short test video (1 to 2 minutes) and an SRT file with at least a dozen cues.

# bash
mkdir hls-webvtt && cd hls-webvtt
mkdir source segments manifest

# A short test video
curl -L -o source/master.mp4 \
  https://test-videos.co.uk/vids/bigbuckbunny/mp4/h264/720/Big_Buck_Bunny_720_10s_5MB.mp4

# A test SRT
cat > source/captions.srt <<'SRT'
1
00:00:00,000 --> 00:00:03,500
Hello, and welcome to the show.

2
00:00:03,500 --> 00:00:07,000
Today we are talking about HLS captions.

3
00:00:07,000 --> 00:00:10,000
And why most pipelines get them wrong.
SRT

📝 2. SRT to WebVTT (the easy part)

FFmpeg's webvtt muxer converts SRT for you. The conversion itself is mostly a header swap, but it normalizes line breaks and timestamps along the way:

# bash
ffmpeg -i source/captions.srt -c:s webvtt source/captions.vtt
cat source/captions.vtt

WEBVTT

1
00:00:00.000 --> 00:00:03.500
Hello, and welcome to the show.

2
00:00:03.500 --> 00:00:07.000
Today we are talking about HLS captions.

3
00:00:07.000 --> 00:00:10.000
And why most pipelines get them wrong.

This file works as a sidecar in HLS. It is also the source of every cross-browser sync bug you are about to hit. We need to segment it and add the timestamp map.

✂️ 3. Segmenting WebVTT for HLS

HLS expects WebVTT split into segments that align with the media segments. With six-second video segments, the subtitles should be on six-second segments too. The packager handles this.

First, build the video ladder. Two renditions is enough for the demo:

# bash
ffmpeg -i source/master.mp4 -c:v libx264 -preset medium -crf 23 \
  -vf scale=1280:720 -g 144 -keyint_min 144 -sc_threshold 0 \
  -hls_time 6 -hls_playlist_type vod \
  -hls_segment_filename "segments/720p_%03d.ts" \
  manifest/720p.m3u8

ffmpeg -i source/master.mp4 -c:v libx264 -preset medium -crf 26 \
  -vf scale=854:480 -g 144 -keyint_min 144 -sc_threshold 0 \
  -hls_time 6 -hls_playlist_type vod \
  -hls_segment_filename "segments/480p_%03d.ts" \
  manifest/480p.m3u8

Now segment the subtitles. Shaka Packager does this with a single descriptor:

# bash
docker run --rm -v "$PWD:/work" -w /work \
  google/shaka-packager \
  in=source/captions.vtt,stream=text,language=en,segment_template=segments/subs/en_$Number$.vtt,playlist_name=manifest/subs/en.m3u8,hls_group_id=subs,hls_name=English \
  --hls_master_playlist_output manifest/master.m3u8 \
  --segment_duration 6

The result is a directory of per-six-second WebVTT files plus a media playlist (manifest/subs/en.m3u8) that references them.

Open one of the segment files:

# segments/subs/en_2.vtt
WEBVTT
X-TIMESTAMP-MAP=MPEGTS:540000,LOCAL:00:00:00.000

00:00:00.000 --> 00:00:01.000
And why most pipelines get them wrong.

That X-TIMESTAMP-MAP line is the whole point. The cue inside the segment is on a local clock starting at zero. The packager has set MPEGTS:540000 to tell the player "this clock-zero corresponds to MPEG-2 PTS 540000", which is 6 seconds at the 90kHz tick rate, which is where the segment starts.

💡 Tip: open every segment file and confirm the X-TIMESTAMP-MAP is present and correct. Some packagers only emit it on the first segment. The HLS spec is generous about this; player implementations are not.

🧷 4. Wiring the manifest

Shaka Packager wrote a master playlist for us. Let's look at it:

# manifest/master.m3u8
#EXTM3U
#EXT-X-VERSION:6

#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="English",LANGUAGE="en",AUTOSELECT=YES,DEFAULT=NO,FORCED=NO,URI="subs/en.m3u8"

#EXT-X-STREAM-INF:BANDWIDTH=3200000,RESOLUTION=1280x720,SUBTITLES="subs",CODECS="avc1.640028"
720p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1600000,RESOLUTION=854x480,SUBTITLES="subs",CODECS="avc1.4d401f"
480p.m3u8

A few things worth pointing out:

SUBTITLES="subs" on every EXT-X-STREAM-INF line. Forget this and the renditions never advertise the subs.
AUTOSELECT=YES lets the player pick this track when the OS language matches. Set to NO if you want subs strictly opt-in.
DEFAULT=NO keeps subs off by default. Most product teams want this; set to YES only for forced subtitles (translated dialogue, on-screen text translation).

🌐 5. Serving and verifying

Serve the whole tree with any static server:

# bash
cd manifest && python3 -m http.server 8080

A minimal HTML harness using HLS.js 1.6.16:

<!-- index.html -->
<!doctype html>
<html>
  <body>
    <video id="v" controls autoplay style="width: 100%; max-width: 720px;"></video>
    <script src="https://cdn.jsdelivr.net/npm/hls.js@1.6.16/dist/hls.min.js"></script>
    <script>
      const video = document.getElementById('v');
      if (Hls.isSupported()) {
        const hls = new Hls({ enableWebVTT: true });
        hls.loadSource('http://localhost:8080/master.m3u8');
        hls.attachMedia(video);
        hls.on(Hls.Events.SUBTITLE_TRACKS_UPDATED, (_, d) =>
          console.log('subtitle tracks:', d.subtitleTracks),
        );
      } else if (video.canPlayType('application/vnd.apple.mpegurl')) {
        // Native HLS path (Safari, iOS)
        video.src = 'http://localhost:8080/master.m3u8';
      }
    </script>
  </body>
</html>

Open the page, hit the cc button in the player UI, and confirm the cue text shows up exactly when the actor speaks. Now hit seek to the middle of the timeline. The cue should jump to whatever line was being spoken at that point, with no drift.

⚠️ Note: HLS.js's light build (hls.light.min.js) does not include WebVTT parsing. Use the standard build for subtitle work.

Repeat with Shaka Player to make sure both engines agree:

<script src="https://cdn.jsdelivr.net/npm/shaka-player@4/dist/shaka-player.compiled.min.js"></script>
<script>
  shaka.polyfill.installAll();
  const player = new shaka.Player(document.getElementById('v'));
  player.load('http://localhost:8080/master.m3u8');
  player.setTextTrackVisibility(true);
</script>

If you see drift in one player but not the other, it is almost always an X-TIMESTAMP-MAP problem in your packager output. Open the failing segment, check the MPEGTS value, recompute it (segment_start_seconds * 90000), and re-segment.

🚨 6. The bugs you will probably hit

A quick checklist of what trips teams up:

Symptom	Likely cause
Captions perfect at start, drift after seek	`X-TIMESTAMP-MAP` missing or wrong on later segments
Captions show up but lag by a few seconds	Subtitle segments and media segments have different durations
Captions work on Safari, blank on Chrome	Using HLS.js `light` build (no WebVTT parser)
Captions appear but lose positioning	Converter flattened `align`, `line`, `position` cue settings
Captions show on 720p stream but not 480p	Missing `SUBTITLES="subs"` on one `EXT-X-STREAM-INF` line
Low-latency live captions arrive late or never	LL-HLS part loading bug (fixed in HLS.js 1.6.x; upgrade)

🧪 7. Going further: fMP4-wrapped subtitles

For CMAF-aligned pipelines and modern players, you can package the same WebVTT inside fMP4 segments instead of standalone .vtt files. The shaka-packager invocation becomes:

# bash (fMP4-wrapped subtitles)
docker run --rm -v "$PWD:/work" -w /work google/shaka-packager \
  in=source/captions.vtt,stream=text,language=en,format=ttml,init_segment=segments/subs/en_init.mp4,segment_template=segments/subs/en_$Number$.m4s,playlist_name=manifest/subs/en.m3u8,hls_group_id=subs,hls_name=English \
  --hls_master_playlist_output manifest/master.m3u8 \
  --segment_duration 6

Modern Safari, HLS.js 1.6.x, Shaka Player 4.x, and ExoPlayer all read fMP4 subtitle tracks. Smart TVs are still a mixed bag; if your product targets connected TV, run the test matrix before flipping the switch.

What's next

A few directions worth exploring once the basic pipeline is solid:

IMSC ingest. Premium content arrives as IMSC 1.1. Build a converter that preserves positioning and styling on the way to WebVTT; you will need it the first time a partner sends you a .dfxp.
Multi-language ladders. The same EXT-X-MEDIA pattern scales to N languages; the work is in the packager invocation, not the manifest.
Forced subtitles. Set FORCED=YES on subtitle tracks that translate on-screen text (foreign-language signs, locked subtitles in mixed-language scenes). The player surfaces them differently.

The shortcut version of HLS subtitles is fine for a demo. The version above is what stops being a support ticket.

video #webdev #tutorial #javascript