DEV Community: Mason K

Detect variable frame rate uploads with ffprobe and normalize them before they break your pipeline

Mason K — Thu, 13 Aug 2026 13:57:11 +0000

TL;DR

Screen recordings from OBS, QuickTime, Loom and phone capture are variable frame rate. Feed them to a transcoder that assumes constant frame rate and you get audio drift that grows over the clip. We'll build a two-tier ffprobe gate (cheap header check, then a real per-frame timestamp check) and normalize with -fps_mode cfr on FFmpeg 8.

The bug

Audio and video start in sync and drift apart as the clip plays. Four minutes in, it's obviously wrong. Thirty seconds in, nobody notices. That growing gap is the signature of a variable frame rate (VFR) source hitting a pipeline that assumed constant frame rate (CFR).

Audio runs on a fixed sample clock. VFR video does not. When your transcoder stamps out frames at a target rate against a source that never agreed to one, the error accumulates instead of resetting.

Let's detect it properly and fix it at ingest.

1. Why you keep receiving VFR 🎥

Screen capture tools generate a frame only when the screen content changes. That covers OBS, QuickTime screen recording, Loom, phone screen recorders, and most game capture. If your product accepts user uploads of demos, tutorials, bug reports or gameplay, a real chunk of your ingest is VFR.

Check a file you already have:

ffprobe -v error \
  -select_streams v:0 \
  -show_entries stream=r_frame_rate,avg_frame_rate,nb_frames,duration \
  -of default=noprint_wrappers=1 \
  input.mp4

r_frame_rate=30000/1001
avg_frame_rate=23856/1001
nb_frames=7152
duration=300.033333

Those two rates disagree, so this file is worth a closer look.

2. Why the popular check is only a heuristic ⚠️

Nearly every tutorial stops at "if r_frame_rate != avg_frame_rate, it's VFR." That is not true, and believing it will cost you a day.

avg_frame_rate is total frames divided by duration. Real.
r_frame_rate is FFmpeg's guess at a base rate, computed like a lowest common multiple of observed timings so every timestamp lands on a whole tick. It is not the actual frame rate.

So a mismatch suggests VFR. It does not prove it, and identical values do not prove CFR either.

⚠️ Treat the header comparison as a cheap first-pass filter, not a verdict. Confirm with per-frame timestamps.

3. The real check: per-frame deltas

Dump packet timestamps and look at the gaps between them. Uniform gaps mean the file is effectively constant no matter what the headers claim.

ffprobe -v error \
  -select_streams v:0 \
  -show_entries packet=pts_time \
  -of csv=p=0 \
  input.mp4 | head -20

Look at that jump from 0.166833 to 0.250250, and the bigger one to 0.550550. Uneven. That's genuine VFR.

Here's the check as a script. It only pays for the expensive packet dump when the cheap check trips.

# vfr_gate.py (python 3.11+, ffmpeg/ffprobe 8.x)
import json
import subprocess
from fractions import Fraction

def _probe(args: list[str]) -> dict:
    out = subprocess.run(
        ["ffprobe", "-v", "error", "-of", "json", *args],
        capture_output=True, text=True, check=True,
    ).stdout
    return json.loads(out)

def header_mismatch(path: str) -> tuple[bool, Fraction, Fraction]:
    """Tier 1: cheap. Compares r_frame_rate against avg_frame_rate."""
    data = _probe([
        "-select_streams", "v:0",
        "-show_entries", "stream=r_frame_rate,avg_frame_rate",
        path,
    ])
    s = data["streams"][0]
    r = Fraction(s["r_frame_rate"])
    a = Fraction(s["avg_frame_rate"])
    return (r != a), r, a

Now tier two, which only runs on files that tripped tier one:

# vfr_gate.py (continued)
def timestamps_are_irregular(path: str, tolerance: float = 0.002,
                             max_packets: int = 3000) -> bool:
    """Tier 2: expensive. Are the gaps between presentation timestamps uniform?"""
    data = _probe([
        "-select_streams", "v:0",
        "-show_entries", "packet=pts_time",
        "-read_intervals", "%+#" + str(max_packets),
        path,
    ])
    pts = sorted(
        float(p["pts_time"]) for p in data.get("packets", [])
        if p.get("pts_time") not in (None, "N/A")
    )
    if len(pts) < 3:
        return False

    deltas = [b - a for a, b in zip(pts, pts[1:])]
    baseline = sorted(deltas)[len(deltas) // 2]   # median, robust to one bad gap
    return any(abs(d - baseline) > tolerance for d in deltas)

def is_vfr(path: str) -> bool:
    mismatch, _, _ = header_mismatch(path)
    return mismatch and timestamps_are_irregular(path)

💡 Tip: -read_intervals "%+#3000" caps how many packets ffprobe reads. On a 90-minute file, dumping every packet is slow and you don't need all of them to see irregularity.

Run it:

$ python vfr_gate.py screen-recording.mp4
VFR: True   (r=30000/1001  avg=23856/1001)

$ python vfr_gate.py camera-export.mp4
VFR: False  (r=30000/1001  avg=30000/1001)

4. Normalize with -fps_mode cfr

Once you know, the fix is one flag applied as the first transcoding step, before segmenting, thumbnailing, or anything else forms an opinion about the file.

ffmpeg -i input.mp4 \
  -fps_mode cfr -r 30 \
  -c:v libx264 -preset medium -crf 21 \
  -c:a aac -b:a 128k -ar 48000 \
  -movflags +faststart \
  normalized.mp4

⚠️ Old tutorials use -vsync cfr or -vsync 1. That family is deprecated in FFmpeg 8.x. Use -fps_mode cfr in anything new.

Pin -r explicitly. Letting FFmpeg infer the output rate is how you got here.

Picking the rate:

Content	Rate	Why
Screen capture, talking head, general UGC	30	Safe default, cheapest
Gameplay, sports, high motion	60	Motion detail is the point
Anything above the source's peak rate	don't	Buys nothing, costs bitrate

5. Wire it into ingest

# ingest.py
import subprocess
from vfr_gate import is_vfr

def normalize_if_needed(src: str, dst: str, fps: int = 30) -> str:
    if not is_vfr(src):
        return src  # already CFR, skip the re-encode

    subprocess.run([
        "ffmpeg", "-v", "error", "-y", "-i", src,
        "-fps_mode", "cfr", "-r", str(fps),
        "-c:v", "libx264", "-preset", "medium", "-crf", "21",
        "-c:a", "aac", "-b:a", "128k", "-ar", "48000",
        "-movflags", "+faststart",
        dst,
    ], check=True)
    return dst

Emit a metric every time this fires. You want to know what fraction of your ingest is VFR, because it directly sizes your transcode fleet.

The cost, honestly

Normalizing a screen recording is not free. Twelve seconds of a static slide that occupied a handful of VFR frames becomes 360 frames at 30fps CFR. Your encoder does more work.

The output usually isn't much bigger, because near-identical consecutive frames compress extremely well and modern rate control handles them fine. But your encode time goes up on exactly the content that was cheapest before. Size your workers for screen capture as the worst case, not the best.

Errors you'll hit

avg_frame_rate=0/0: the container has no duration, common with fragmented MP4 and some live recordings. Skip tier one and go straight to timestamps.

Audio still drifts after normalization: check whether the source audio sample rate is what you think. Pin -ar 48000 and re-test. A resample happening implicitly is its own drift source.

Application provided invalid, non monotonically increasing dtx: the source has out-of-order timestamps on top of being VFR. Add -fflags +genpts before -i.

What's next

Frame rate is one assumption your pipeline makes about incoming files without checking. There are others, and they fail in unrelated-looking ways:

Rotation metadata: the file is 1080x1920 with a 90-degree rotate flag, and half your tools honor it.
Color space / transfer characteristics: the reason HDR sources come out washed on SDR screens.
Audio channel layout: 5.1 sources silently downmixed, or not.

Pick one, write the ffprobe check, run it over a week of real uploads, and see what comes back. That number is usually more interesting than the fix.

Build a directional preload window for a scrolling video feed with hls.js

Mason K — Thu, 13 Aug 2026 13:56:11 +0000

TL;DR

Vertical video feeds are fast on the first item and stutter on every one after. We'll build a preload window that follows scroll direction, attaches hls.js sources to off-screen items ahead of the user, and tears down everything outside the window so you stay under Safari's video element ceiling. Reconciler pattern, ~120 lines.

The problem

Item 1 plays instantly because the page loaded around it. Item 2 shows black, then a spinner, then a stutter into low quality. Preloading everything fixes the stutter and replaces it with a data-plan fire and a crash on iPhone.

What we want: preload a small number of items in the direction the user is scrolling, and aggressively release everything else.

1. Why `preload="auto"` isn't a strategy

<!-- this is a hint, not an instruction -->
<video preload="auto" src="..."></video>

Browsers treat preload as a suggestion and mobile browsers take it least seriously, deliberately, to protect the user's data. With HLS you get real control, because loading happens in JS.

In hls.js, attaching a source is the trigger. The moment an instance has a manifest URL, it fetches the playlist and starts pulling segments, viewport or no viewport. That's our primitive.

npm i hls.js@^1.7.0

2. A player wrapper with two modes 🎛️

Off-screen items need a shallow buffer. Just enough to start. The active item gets the normal one.

// feed/player.js (hls.js 1.7.x)
import Hls from "hls.js";

const PRELOAD_CONFIG = {
  // enough to render frame one, not enough to burn a data plan
  maxBufferLength: 4,
  maxMaxBufferLength: 6,
  // go get fragment 1 before media is attached
  startFragPrefetch: true,
  autoStartLoad: true,
};

const ACTIVE_CONFIG = {
  maxBufferLength: 30,     // hls.js default
  maxMaxBufferLength: 60,
  startFragPrefetch: true,
  autoStartLoad: true,
};

export function attach(videoEl, url, { active = false } = {}) {
  if (videoEl.canPlayType("application/vnd.apple.mpegurl")) {
    // Safari plays HLS natively; no hls.js instance to manage
    videoEl.src = url;
    videoEl.preload = active ? "auto" : "metadata";
    return { native: true, destroy: () => { videoEl.removeAttribute("src"); videoEl.load(); } };
  }

  const hls = new Hls(active ? ACTIVE_CONFIG : PRELOAD_CONFIG);
  hls.loadSource(url);
  hls.attachMedia(videoEl);
  return {
    native: false,
    hls,
    destroy: () => { hls.destroy(); },
  };
}

💡 Tip: maxBufferLength defaults to 30 in hls.js. Thirty seconds for the item being watched is sensible. Thirty seconds each for four queued items is not.

3. The directional window

The naive window is symmetric: 2 before, 2 after. But scroll isn't symmetric. Someone moving down keeps moving down, and the item behind them is the one they just finished.

// feed/window.js
const AHEAD = 4;   // items to preload in the direction of travel
const BEHIND = 1;  // cheap insurance for a quick scroll-back

/**
 * @param {number} index      current active item
 * @param {1|-1} direction    1 = scrolling down, -1 = scrolling up
 * @param {number} total      item count
 * @returns {Set<number>}     indices that should have a source attached
 */
export function desiredWindow(index, direction, total) {
  const lo = direction === 1 ? index - BEHIND : index - AHEAD;
  const hi = direction === 1 ? index + AHEAD : index + BEHIND;
  const out = new Set();
  for (let i = Math.max(0, lo); i <= Math.min(total - 1, hi); i++) out.add(i);
  return out;
}

Same bandwidth budget, roughly double the hit rate. One conditional.

4. The reconciler (this is the important part) 🔑

Here's the bug you will otherwise ship: a user flicks quickly through six items. Preload fires on scroll-in, teardown fires on scroll-out, and because they're independent effects racing each other you tear down a player you just decided to preload. Symptom: fine when you scroll slowly, broken when you scroll fast. Fine in dev, broken in the wild.

Fix: never fire preload and teardown as separate effects. Compute the desired set, diff it against what's attached, act on the diff.

// feed/reconciler.js
import { attach } from "./player.js";
import { desiredWindow } from "./window.js";

export function createFeedController({ items, getVideoEl }) {
  const attached = new Map();  // index -> { native, hls, destroy, active }
  let lastIndex = 0;

  function reconcile(index) {
    const direction = index >= lastIndex ? 1 : -1;
    lastIndex = index;

    const want = desiredWindow(index, direction, items.length);

    // 1. release everything outside the window
    for (const [i, handle] of attached) {
      if (!want.has(i)) {
        handle.destroy();
        attached.delete(i);
      }
    }

    // 2. attach everything inside it that isn't already
    for (const i of want) {
      const el = getVideoEl(i);
      if (!el) continue;                      // not rendered yet, skip
      const existing = attached.get(i);
      const shouldBeActive = i === index;

      if (existing) {
        // promote the newly-active item to the deep-buffer config
        if (shouldBeActive && !existing.active) {
          existing.destroy();
          attached.set(i, { ...attach(el, items[i].hlsUrl, { active: true }), active: true });
        }
        continue;
      }

      attached.set(i, {
        ...attach(el, items[i].hlsUrl, { active: shouldBeActive }),
        active: shouldBeActive,
      });
    }
  }

  function destroyAll() {
    for (const [, h] of attached) h.destroy();
    attached.clear();
  }

  return { reconcile, destroyAll, get size() { return attached.size; } };
}

Drive it from whatever tells you the active index. With CSS scroll snapping, an IntersectionObserver is enough:

// feed/index.js
const controller = createFeedController({ items, getVideoEl: (i) => els[i] });

const io = new IntersectionObserver((entries) => {
  for (const e of entries) {
    if (e.intersectionRatio > 0.6) {
      controller.reconcile(Number(e.target.dataset.index));
    }
  }
}, { threshold: [0.6] });

els.forEach((el) => io.observe(el));

5. Teardown is not optional on iOS

Mobile Safari has a widely reported ceiling of around 16 simultaneous video elements; past that, additional ones render black. It's not an explicitly documented API limit, it's a wall people keep hitting. In an infinite feed that mounts a player per item and never unmounts, you hit it fast.

Our window caps attachments at AHEAD + BEHIND + 1 = 6, comfortably under it. But release the source, not the element:

// keep the DOM node mounted, drop the media
handle.destroy();          // hls.destroy() frees buffers + decoder
// element stays in the list for cheap re-attach on scroll-back

Unmounting the component too means scrolling back pays full layout cost. Keep the shell, make the media disposable.

⚠️ Multiple simultaneous video elements are a memory and GPU pressure problem beyond the element count. It shows up as a crash, not a catchable error, so you can't detect it and back off. You have to stay under the line by construction.

6. Measure the thing users feel 📊

The metric is time from scroll settling to first frame painted, on item 2 and beyond. Item 1 is a page load and it flatters you.

// feed/metrics.js
const marks = new Map();

export function markScrollSettled(index) {
  marks.set(index, performance.now());
}

export function markFirstFrame(index, videoEl) {
  videoEl.addEventListener("playing", () => {
    const t0 = marks.get(index);
    if (t0 == null) return;
    const ms = Math.round(performance.now() - t0);
    marks.delete(index);
    navigator.sendBeacon?.("/metrics/feed-ttff", JSON.stringify({
      index, ms,
      position: index,                       // item 40 matters more than item 2
      conn: navigator.connection?.effectiveType ?? "unknown",
    }));
  }, { once: true });
}

Chart it against bytes per session. These move in opposite directions and tuning one alone ships something beautiful on wifi and hostile on a phone plan.

The window size you want is the smallest AHEAD where time-to-first-frame goes flat. Start at 4, walk it down to 2, and watch where the curve bends. Past the flat point you're buying nothing and spending someone else's money.

💡 Tip: if the feed is instant for ten items and degrades after that, your teardown isn't keeping up. More preloading will not fix it.

What's next

hls.js doesn't yet expose a first-class "preload N seconds off-screen then stop" API. There's an open issue for it and a related multi-segment prefetch proposal. Worth watching; the per-instance maxBufferLength trick above is the practical stand-in.
Serve a lower-rendition first segment to off-screen items so the preload costs less, then let ABR climb once the item is active.
If your feed is React Native rather than web, the same window/reconciler logic applies. Mux wrote up a feed that preloads by scroll direction and the structure transfers directly.

The general lesson: "make it feel instant" is a scheduling problem wearing a rendering problem's clothes. You aren't loading faster, you're loading earlier, and the craft is in guessing well and giving up fast.

Build semantic search inside video: shot embeddings + transcript chunks, fused on the timeline

Mason K — Thu, 13 Aug 2026 13:55:46 +0000

TL;DR

Frame-only video search can't find "where she explains pricing." Transcript-only search can't find "the architecture diagram slide." We'll build both indexes, key them to the same timeline, and fuse at query time so results are timestamps, not video IDs. Shot detection, SigLIP-2 embeddings, Whisper chunks, pgvector.

What we're building

A search endpoint where "where does she explain the pricing model" returns video_id=42, t=247.3s and you can seek straight there.

Two indexes over one timeline:

Index	Unit	Model	Finds
Visual	one shot	SigLIP-2	"the architecture diagram", "person holding a coffee cup"
Transcript	utterance group	text embedder	"the pricing explanation", "when they mention refunds"

Neither half works alone. In business video (talks, interviews, tutorials, meetings) most meaning is in the audio, so a frames-only index is blind to the majority of it. But nobody says "architecture diagram" out loud, they say "as you can see here."

pip install torch transformers pyscenedetect faster-whisper psycopg[binary] pgvector

1. Chunk on shots, not on seconds 🎬

Sampling a frame every N seconds is wasteful in both directions: forty near-identical frames in a static screen share, and a missed shot across a fast cut. Use shot boundaries instead.

# ingest/shots.py
from scenedetect import detect, ContentDetector

def shot_boundaries(path: str, threshold: float = 27.0) -> list[tuple[float, float]]:
    """Returns [(start_sec, end_sec), ...] per detected shot."""
    scenes = detect(path, ContentDetector(threshold=threshold))
    return [(s.get_seconds(), e.get_seconds()) for s, e in scenes]

$ python -c "from ingest.shots import shot_boundaries; print(shot_boundaries('talk.mp4')[:4])"
[(0.0, 8.34), (8.34, 31.07), (31.07, 33.9), (33.9, 96.21)]

Grab one representative frame per shot, from the middle rather than the boundary (boundaries catch transitions and dissolves):

# ingest/frames.py
import subprocess, tempfile, os

def midpoint_frame(video: str, start: float, end: float) -> str:
    t = start + (end - start) / 2
    out = os.path.join(tempfile.mkdtemp(), "f.jpg")
    subprocess.run([
        "ffmpeg", "-v", "error", "-y",
        "-ss", f"{t:.3f}", "-i", video,
        "-frames:v", "1", "-q:v", "3", out,
    ], check=True)
    return out

2. Embed the frames

SigLIP-2 generally edges out CLIP at comparable size, especially on fine-grained visual detail. The SO400M variant emits 1152 dimensions (note this, it's about to matter).

# ingest/visual.py
import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor

MODEL_ID = "google/siglip2-so400m-patch16-512"
EMBED_DIM = 1152

_device = "cuda" if torch.cuda.is_available() else "cpu"
_model = AutoModel.from_pretrained(MODEL_ID).to(_device).eval()
_proc = AutoProcessor.from_pretrained(MODEL_ID)

@torch.inference_mode()
def embed_image(path: str) -> list[float]:
    inputs = _proc(images=Image.open(path).convert("RGB"), return_tensors="pt").to(_device)
    v = _model.get_image_features(**inputs)
    return torch.nn.functional.normalize(v, dim=-1)[0].cpu().tolist()

@torch.inference_mode()
def embed_text(query: str) -> list[float]:
    inputs = _proc(text=[query], padding="max_length", return_tensors="pt").to(_device)
    v = _model.get_text_features(**inputs)
    return torch.nn.functional.normalize(v, dim=-1)[0].cpu().tolist()

⚠️ EMBED_DIM is not a config value you change later. Your table column is vector(1152). Swapping to JinaCLIP-v2 (1024 dims) means re-embedding your entire library. Version your tables from day one so a model swap is a parallel build, not a migration.

3. Transcript chunks want different boundaries

Speech doesn't respect cuts. A sentence runs across a shot change constantly, so chopping transcripts on visual boundaries produces fragments that embed badly. Chunk on the transcript's own logic, with overlap.

# ingest/transcript.py
from faster_whisper import WhisperModel

_asr = WhisperModel("large-v3-turbo", device="auto", compute_type="int8")

def chunks(video: str, target_chars: int = 400, overlap: int = 1) -> list[dict]:
    segments, _ = _asr.transcribe(video, vad_filter=True, word_timestamps=False)
    segs = [{"start": s.start, "end": s.end, "text": s.text.strip()} for s in segments]

    out, buf = [], []
    for seg in segs:
        buf.append(seg)
        if sum(len(s["text"]) for s in buf) >= target_chars:
            out.append({
                "start": buf[0]["start"],
                "end": buf[-1]["end"],
                "text": " ".join(s["text"] for s in buf),
            })
            buf = buf[-overlap:] if overlap else []
    if buf:
        out.append({
            "start": buf[0]["start"], "end": buf[-1]["end"],
            "text": " ".join(s["text"] for s in buf),
        })
    return out

Two indexes, different granularities, both timestamped. That's the correct end state, and the timeline is the join key.

4. Schema

Postgres + pgvector means one fewer system to run, and it lets you filter by tenancy in the same query as the similarity search. That last part is not a convenience, it's a correctness requirement (see step 6).

-- migrations/001_init.sql  (postgres 16 + pgvector 0.8)
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE shot_embeddings (
  id          bigserial PRIMARY KEY,
  video_id    bigint NOT NULL,
  tenant_id   bigint NOT NULL,
  start_sec   double precision NOT NULL,
  end_sec     double precision NOT NULL,
  embedding   vector(1152) NOT NULL
);

CREATE TABLE transcript_embeddings (
  id          bigserial PRIMARY KEY,
  video_id    bigint NOT NULL,
  tenant_id   bigint NOT NULL,
  start_sec   double precision NOT NULL,
  end_sec     double precision NOT NULL,
  text        text NOT NULL,
  embedding   vector(1152) NOT NULL
);

CREATE INDEX ON shot_embeddings       USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON transcript_embeddings USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON shot_embeddings       (tenant_id, video_id);
CREATE INDEX ON transcript_embeddings (tenant_id, video_id);

5. Fuse at query time, not index time

It's tempting to combine visual and text vectors into one representation during ingest. Don't, at least not first. You lose explainability, you lose per-query weighting, and you've committed to a fusion strategy before you know what works on your content.

Query both, merge on time overlap:

# search/query.py
from collections import defaultdict

def search(conn, tenant_id: int, q: str, k: int = 30,
           w_visual: float = 0.4, w_text: float = 0.6):
    qv = embed_text(q)

    visual = _knn(conn, "shot_embeddings", tenant_id, qv, k)
    textual = _knn(conn, "transcript_embeddings", tenant_id, qv, k)

    # bucket both sides into 5s slots on the shared timeline
    buckets = defaultdict(lambda: {"visual": 0.0, "text": 0.0, "hit": None})
    for r in visual:
        key = (r["video_id"], int(r["start_sec"] // 5))
        b = buckets[key]
        b["visual"] = max(b["visual"], r["score"])
        b["hit"] = b["hit"] or r
    for r in textual:
        key = (r["video_id"], int(r["start_sec"] // 5))
        b = buckets[key]
        b["text"] = max(b["text"], r["score"])
        b["hit"] = b["hit"] or r

    scored = [{
        "video_id": vid,
        "t": b["hit"]["start_sec"],
        "score": w_visual * b["visual"] + w_text * b["text"],
        "why": {"visual": round(b["visual"], 3), "transcript": round(b["text"], 3)},
    } for (vid, _slot), b in buckets.items()]

    return sorted(scored, key=lambda r: -r["score"])[:10]

# search/query.py (continued)
def _knn(conn, table: str, tenant_id: int, qv: list[float], k: int):
    with conn.cursor() as cur:
        cur.execute(
            f"""SELECT video_id, start_sec, 1 - (embedding <=> %s::vector) AS score
                FROM {table}
                WHERE tenant_id = %s
                ORDER BY embedding <=> %s::vector
                LIMIT %s""",
            (qv, tenant_id, qv, k),
        )
        cols = [d.name for d in cur.description]
        return [dict(zip(cols, row)) for row in cur.fetchall()]

A moment where both signals fire is a much better answer than either alone, and now you can say so:

$ curl -s 'localhost:8000/search?q=where+she+explains+the+pricing+model' | jq '.[0]'
{
  "video_id": 42,
  "t": 247.3,
  "score": 0.71,
  "why": { "visual": 0.31, "transcript": 0.98 }
}

w_visual / w_text is the knob that matters. Conference talks want transcript weighted heavily. A B-roll library wants the opposite.

6. The part that isn't in the architecture diagram 🔒

Embeddings do not carry permissions. A vector index will happily return the nearest neighbor from a video the user can't see.

Filtering has to happen inside the query, not after it. Notice WHERE tenant_id = %s sits in the same statement as the ORDER BY embedding <=> above. If you filter the results afterward, your top-10 becomes a top-4, your pagination lies, and eventually someone sees another customer's footage.

⚠️ This is the single most common way these systems ship broken. Post-filtering a KNN result is not access control.

Errors you'll hit

expected 1152 dimensions, not 768: you swapped models without migrating the column. Told you.

Shot detection returns one giant scene: screen recordings with no cuts. Fall back to fixed-interval sampling when len(shots) < 2.

Recall is fine, ranking is bad: your two score distributions aren't comparable. Normalize each side's scores within the result set before weighting them.

Where the numbers come from

One 2026 benchmark roundup puts semantic video search accuracy for this class of model in the mid-to-high 80s on general datasets, rising into the low 90s with domain-specific fine-tuning. That's someone else's data on someone else's content. Measure your own before you promise anything.

On store choice: 2026 comparisons between pgvector and dedicated engines like Qdrant keep landing on "comparable speed, the bottleneck is elsewhere." If your data is already in Postgres, that's your answer.

What's next

Clip generation for free. Once moments are addressable, "make me a highlight reel about X" is a query plus a concat, not a manual scrub.
Timestamped recommendations. Point at a moment instead of a video.
Rerank the top 20 with a cross-encoder or a VLM if precision at 1 matters more than latency.

The reason to build this isn't that users are asking for a search box. It's that the index is the substrate for four other features.

Sources

Stop hls.js from flapping between quality levels on cellular (with abrSwitchInterval)

Mason K — Tue, 04 Aug 2026 06:12:58 +0000

TL;DR

ABR "flapping" is when your player hops between quality levels every few seconds on a jittery network, and each hop is a visible lurch. We'll detect it from LEVEL_SWITCHED events, then fix it in layers: widen the bandwidth-estimator memory, make upswitches earn their place, and cap the switch rate with abrSwitchInterval (new in hls.js 1.7). Config + a detection snippet you can paste in today.

📦 Code: github.com/USER/hlsjs-abr-tuning, replace before publishing

The bug nobody reports correctly

Users don't file "my ABR is flapping." They say the video "kept changing" or "couldn't decide." What's happening: on cellular, throughput is spiky, and the player's bandwidth estimator treats every spike as the new truth. One fast segment and it jumps to 1080p, one slow segment and it drops to 240p, over and over. Low rebuffer ratio, good startup time, and still a miserable watch.

Counterintuitively, feeding the player fresher bandwidth data makes this worse, because fresher data is noisier. The fix is a player with a longer memory and slower reflexes. Let's build that.

1. First, detect the flap 📊

Don't tune by vibes. Count level switches per minute of playback. Every switch fires Hls.Events.LEVEL_SWITCHED.

// abr-monitor.js, hls.js 1.7.x, node 20+ tooling / any modern browser
import Hls from "hls.js";

export function attachFlapMonitor(hls) {
  const switches = [];
  hls.on(Hls.Events.LEVEL_SWITCHED, (_evt, data) => {
    const now = performance.now();
    switches.push({ t: now, level: data.level });
    // keep a 60s sliding window
    while (switches.length && now - switches[0].t > 60_000) switches.shift();

    const perMin = switches.length;
    const reversals = countReversals(switches);
    if (perMin >= 6) {
      console.warn(`[abr] flapping: ${perMin} switches/min, ${reversals} reversals`);
    }
  });
}

// a "reversal" = up then down (or down then up), the signature of flapping
function countReversals(s) {
  let r = 0;
  for (let i = 2; i < s.length; i++) {
    const a = Math.sign(s[i - 1].level - s[i - 2].level);
    const b = Math.sign(s[i].level - s[i - 1].level);
    if (a !== 0 && b !== 0 && a !== b) r++;
  }
  return r;
}

# what a flapping session logs on 4G:
[abr] flapping: 11 switches/min, 7 reversals
[abr] flapping: 9 switches/min, 6 reversals

A healthy cellular session still switches. A flapping one switches many times a minute and keeps reversing direction. That reversal count is your target metric.

2. Layer one: give the estimator a longer memory

hls.js estimates bandwidth with an exponentially weighted moving average that has a fast and a slow half-life, and uses the more conservative of the two. Widen both half-lives so short spikes stop moving the estimate.

const hls = new Hls({
  // defaults are ~3.0 / ~9.0 (VoD) and ~3.0 / ~9.0 (live).
  // widen them so the estimate reflects the last several seconds, not the last one.
  abrEwmaFastVoD: 4.0,
  abrEwmaSlowVoD: 15.0,
  abrEwmaFastLive: 4.0,
  abrEwmaSlowLive: 15.0,
});

Bigger numbers mean a slower, calmer estimate. On a jittery connection this alone removes most of the drama.

3. Layer two: make upswitches earn it

Going down to avoid a stall is urgent. Going up is optional. Don't treat them symmetrically. Lower the up-switch factor so the player demands sustained headroom before reaching for a higher rendition.

const hls = new Hls({
  abrEwmaFastVoD: 4.0,
  abrEwmaSlowVoD: 15.0,
  abrBandWidthFactor: 0.95,     // margin applied to down-switch decisions
  abrBandWidthUpFactor: 0.5,    // default ~0.7; lower = more conservative upswitching
});

💡 Tip: abrBandWidthUpFactor is the highest-leverage single knob for "it keeps jumping to a level it can't hold." Drop it before you touch anything else.

4. Layer three: rate-limit the switches with abrSwitchInterval 🚦

The older knobs shape the estimate and hope fewer switches fall out. hls.js 1.7 added a knob that limits the switches themselves: abrSwitchInterval sets a minimum time between ABR level changes.

const hls = new Hls({
  abrEwmaFastVoD: 4.0,
  abrEwmaSlowVoD: 15.0,
  abrBandWidthUpFactor: 0.5,
  // NEW in 1.7: minimum seconds between automatic ABR switches.
  // The estimator still decides WHERE to go; this decides HOW OFTEN it may.
  abrSwitchInterval: 3.0,
});

The 1.7 line also brought CMCD v2, I-frame playlist support, and faster startup via parallel init-segment loading, but abrSwitchInterval is the one that directly kills flapping. Pair it with the EWMA tuning rather than relying on it alone: the estimator picks the target, the interval caps the churn.

⚠️ Don't set the interval so high that a genuine, sustained bandwidth drop can't trigger a timely downswitch. 2 to 4 seconds is a sane starting range. Watch your rebuffer ratio when you raise it.

5. Layer four (last resort): cap levels on known-bad networks

If you have a signal that the viewer is on a constrained connection, cap the ceiling so the top renditions are never even considered. A stable 480p beats a flapping 1080p on a bus.

// use the Network Information API where available as a hint
function applyNetworkProfile(hls) {
  const c = navigator.connection;
  if (c && (c.effectiveType === "2g" || c.effectiveType === "3g" || c.saveData)) {
    // cap to the highest level whose height <= 480
    const cap = hls.levels.reduce((best, lvl, i) => (lvl.height <= 480 ? i : best), 0);
    hls.autoLevelCapping = cap;
  }
}
hls.on(Hls.Events.MANIFEST_PARSED, () => applyNetworkProfile(hls));

6. Putting it together as a profile

Wrap the knobs in a small config so you can A/B it against defaults and watch the flap monitor.

// hls-config.js
export const antiFlapConfig = {
  abrEwmaFastVoD: 4.0,
  abrEwmaSlowVoD: 15.0,
  abrEwmaFastLive: 4.0,
  abrEwmaSlowLive: 15.0,
  abrBandWidthFactor: 0.95,
  abrBandWidthUpFactor: 0.5,
  abrSwitchInterval: 3.0,
};

// usage
import Hls from "hls.js";
import { antiFlapConfig } from "./hls-config.js";
import { attachFlapMonitor } from "./abr-monitor.js";

const hls = new Hls(antiFlapConfig);
attachFlapMonitor(hls);
hls.loadSource("https://example.com/master.m3u8");
hls.attachMedia(video);

Now flip between {} and antiFlapConfig on the same throttled 4G session and compare the monitor output. You want switches-per-minute and reversals to fall while rebuffer ratio stays flat. If rebuffering climbs, you went too conservative: raise abrBandWidthUpFactor back toward 0.7 or lower abrSwitchInterval.

What's next

Once the flap is under control, feed the same LEVEL_SWITCHED and rebuffer data into CMCD v2 so your CDN logs can explain what the client saw, and correlate flap rate against effectiveType buckets to build per-network profiles. The whole fix is one idea: an adaptive player fails in the direction of overconfidence, so slow its reflexes down and make it earn every upswitch.

AV2 shipped its 1.0 encoder. Here's what to actually run in your pipeline instead.

Mason K — Tue, 04 Aug 2026 06:12:49 +0000

TL;DR

AOMedia released the AV2 reference encoder (1.0.0) at the end of May 2026. It's real, it's about 30% more efficient than AV1 on paper, and you cannot ship it yet: the encoder is slow, there's no FFmpeg support, and no hardware decoders in the wild. This post shows what to run today instead: probe your AV1 hardware support, build a clean AV1+H.264 ladder with FFmpeg, and wire a one-line canary so you know the day AV2 becomes worth touching.

📦 Code: github.com/USER/av2-reality-check, replace before publishing

The news, in one paragraph

AOMedia published AV2 reference software 1.0.0 (the AOM Video Model) on May 28, 2026, with the spec announcement in early June. Developer evaluations put it around 28.6% smaller than AV1 by PSNR-YUV and 32.6% by VMAF at equal quality, past 40% on screen content. Royalty-free, same as AV1. Great. Now the catch: the encoder is slow by design (it's a reference implementation, not a production one), there's no FFmpeg integration yet, and no shipping hardware decoders. That combination means AV2 is a "read the release notes" event, not a "change your pipeline" event.

If you've been through the AV1 rollout, this rhymes exactly. So let's spend the effort where it pays off in 2026.

1. Probe what your machines can actually do 🔍

Before touching codecs, find out what your encode boxes support. FFmpeg is the source of truth.

# What AV1 encoders does this build expose?
ffmpeg -hide_banner -encoders | grep -i av1
# Typical output on a 2026 build:
#  V....D libsvtav1            SVT-AV1 (codec av1)
#  V....D libaom-av1           libaom AV1 (codec av1)
#  V....D av1_nvenc            NVIDIA NVENC av1 encoder (codec av1)
#  V....D av1_qsv              AV1 (Intel Quick Sync Video) (codec av1)
#  V....D av1_vaapi            AV1 (VAAPI) (codec av1)

# And confirm AV2 is NOT here yet (it won't be):
ffmpeg -hide_banner -encoders | grep -i av2 || echo "no av2 encoder, expected in 2026"

If you see av1_nvenc, av1_qsv, or av1_vaapi, you have hardware AV1 encoding available, which is the difference between AV1 being practical and AV1 being an overnight batch job. Check for hardware:

# NVIDIA: is the AV1 encode session available?
ffmpeg -hide_banner -f lavfi -i testsrc=size=1280x720:rate=30 -t 2 \
  -c:v av1_nvenc -y /tmp/av1_hw_test.mp4 && echo "AV1 NVENC works"

2. The mistake to avoid: software-decode AV1 on phones

The single most common AV1 own-goal is serving AV1 to every browser that claims support, without checking whether the device decodes it in hardware. Software-decoding AV1 on a phone cooks the battery and throttles. Detect real support on the client instead:

// client: prefer AV1 only when the device is likely to hardware-decode it
async function preferAv1() {
  if (!("mediaCapabilities" in navigator)) return false;
  const cfg = {
    type: "media-source",
    video: {
      contentType: 'video/mp4; codecs="av01.0.05M.08"',
      width: 1920, height: 1080, bitrate: 3_000_000, framerate: 30,
    },
  };
  const info = await navigator.mediaCapabilities.decodingInfo(cfg);
  // "smooth" + "powerEfficient" is your best proxy for hardware decode.
  return info.supported && info.smooth && info.powerEfficient;
}

⚠️ powerEfficient is the key field. A device can report supported: true while decoding in software, which is worse than just serving H.264. Gate AV1 on powerEfficient, not on supported.

3. Build a ladder you can ship today 🎞️

Multi-codec adaptive delivery is the actual 2026 state of the art: AV1 for devices that hardware-decode it, H.264 for everyone else. Here's a two-codec, three-rung ladder with SVT-AV1 (fast, tunable) and libx264 as the universal fallback.

#!/usr/bin/env bash
# build-ladder.sh, AV1 (svt-av1) + H.264 (x264), keyframe-aligned for clean ABR
set -euo pipefail
IN="$1"
GOP=48   # 2s at 24fps; keep identical across every rung so ABR switches cleanly

for rung in "1920x1080:av1:0" "1280x720:av1:0" "854x480:h264:0"; do
  IFS=":" read -r size codec _ <<< "$rung"
  if [ "$codec" = "av1" ]; then
    ffmpeg -y -i "$IN" -vf "scale=$size" \
      -c:v libsvtav1 -preset 7 -crf 30 \
      -g $GOP -keyint_min $GOP -sc_threshold 0 \
      -svtav1-params "film-grain=8" \
      -c:a libopus -b:a 128k "out_${size}_av1.mp4"
  else
    ffmpeg -y -i "$IN" -vf "scale=$size" \
      -c:v libx264 -preset veryfast -crf 21 \
      -g $GOP -keyint_min $GOP -sc_threshold 0 \
      -c:a aac -b:a 128k "out_${size}_h264.mp4"
  fi
done
echo "ladder built. package to HLS/DASH with codec-separated variant streams."

Two details that matter more than the codec choice:

Keyframe alignment. -g and -keyint_min equal, and -sc_threshold 0, on every rung. If GOPs don't align across renditions, ABR switches stutter no matter how good your codec is.
Grain synthesis. film-grain=8 in SVT-AV1 lets the encoder model grain instead of spending bitrate encoding it. On noisy source that's a real saving, available now, no AV2 required.

4. Wire the canary for AV2 🪝

You want to know the moment AV2 becomes worth an experiment. The signal is FFmpeg support landing. Drop this in CI or a weekly cron:

#!/usr/bin/env bash
# av2-watch.sh, pings you when your ffmpeg build grows an av2 encoder
if ffmpeg -hide_banner -encoders 2>/dev/null | grep -qi 'av2\|libaom-av2\|libavm'; then
  echo "AV2 encoder detected in ffmpeg, time to run experiments"
  exit 0
fi
echo "no av2 encoder yet ($(date +%F))"

# terminal output today:
$ ./av2-watch.sh
no av2 encoder yet (2026-08-03)

When that flips, you experiment. You do not put it under production traffic, because a fast software encode still leaves the decode side unsolved. AV2 becomes a real default only when a critical mass of your viewers have hardware decoders, and following AV1's timeline that trails the spec by a couple of years.

5. If you truly need more compression right now

There's exactly one next-gen codec further along than AV2 in 2026, and it's VVC (H.266). It buys AV2-like efficiency over HEVC, but browser support is effectively nonexistent and it is not royalty-free, so its realistic home is smart TVs and set-top boxes where you own the decoder. Same shape of constraint AV2 will have, just earlier. For general web audiences, a well-tuned AV1+H.264 ladder is still the move.

What's next

Run build-ladder.sh on a representative asset, compare the AV1 rung against your current H.264-only output with ffmpeg-quality-metrics (VMAF), and see how much bandwidth a tuned AV1 ladder saves you today. Keep av2-watch.sh running. The ceiling went up this summer; the work that pays off before AV2 is shippable is the ladder you're serving right now.

Benchmark video APIs yourself: upload to playable, measured across five providers

Mason K — Tue, 04 Aug 2026 06:12:40 +0000

TL;DR

Vendor benchmark pages test the file size that flatters them. We're going to build a small Node harness that measures the two numbers that decide your upload UX, time-to-upload and time-to-playable, then look at real results across FastPix, Mux, api.video, Cloudinary, and Gumlet. The punchline: the ranking flips depending on file size, so the only benchmark that matters is the one you run on your own footage.

📦 Code: github.com/USER/video-api-benchmark, replace before publishing

Why roll your own

Every video API has a benchmark page and every one of them wins its own benchmark. That's not fraud, it's selection. Upload performance depends heavily on file size, and providers optimize for different parts of the pipeline. So instead of trusting a leaderboard, let's measure the thing your users actually feel: the "processing..." spinner between hitting upload and the video being playable.

We'll measure two numbers per provider:

Upload time, how long to push the bytes.
Time-to-ready, upload plus processing, until a playable HLS manifest exists.

1. The measurement harness 🛠️

The pattern is the same for every provider: create an asset, upload the file, then poll until the asset reports ready. We time the whole thing. Here's the reusable core.

// bench/measure.mjs, node 20+
import { performance } from "node:perf_hooks";

export async function timed(label, fn) {
  const t0 = performance.now();
  const result = await fn();
  const ms = performance.now() - t0;
  console.log(`${label}: ${(ms / 1000).toFixed(2)}s`);
  return { ms, result };
}

// Poll a status function until it returns "ready" (or throws on error/timeout).
export async function pollUntilReady(getStatus, { intervalMs = 1500, timeoutMs = 600000 } = {}) {
  const start = Date.now();
  for (;;) {
    const status = await getStatus();
    if (status === "ready") return;
    if (status === "errored") throw new Error("asset processing errored");
    if (Date.now() - start > timeoutMs) throw new Error("timed out waiting for ready");
    await new Promise((r) => setTimeout(r, intervalMs));
  }
}

💡 Tip: run every provider from the same machine and the same network, back to back. Cross-machine numbers are noise.

2. Throttle the network so the numbers mean something

Uploads on your office fiber tell you nothing about a user on 4G. Cap the bandwidth. On Linux you can shape the interface with tc; simplest is to run the harness behind a throttled network namespace or a proxy. For a quick and dirty cap that works cross-platform, route your requests through a throttling proxy:

# using @sitespeed.io/throttle to cap the whole machine to ~10 Mbps up/down
npx @sitespeed.io/throttle --up 10000 --down 10000 --rtt 40
# run your benchmark in another shell, then:
npx @sitespeed.io/throttle --stop

The published results below were all captured on a 4G profile capped at 10 Mbps, which is the profile that actually separates these providers.

3. A provider adapter (FastPix as the example)

Each provider needs a tiny adapter with two methods: createAndUpload(file) and status(id). The direct-upload pattern is nearly identical everywhere. Here's one against FastPix using its on-demand endpoint.

// bench/providers/fastpix.mjs
const BASE = "https://api.fastpix.io/v1/on-demand";
const auth =
  "Basic " + Buffer.from(`${process.env.FP_KEY_ID}:${process.env.FP_SECRET}`).toString("base64");

export async function createAndUpload(fileUrl) {
  // Create an asset from a source URL (server-side import).
  const res = await fetch(BASE, {
    method: "POST",
    headers: { Authorization: auth, "Content-Type": "application/json" },
    body: JSON.stringify({ inputs: [{ type: "video", url: fileUrl }] }),
  });
  if (!res.ok) throw new Error(`create failed: ${res.status} ${await res.text()}`);
  const { data } = await res.json();
  return data.id;
}

export async function status(id) {
  const res = await fetch(`${BASE}/${id}`, { headers: { Authorization: auth } });
  const { data } = await res.json();
  return data.status; // "preparing" | "ready" | "errored"
}

The Mux, api.video, and Cloudinary adapters follow the same shape against their own create/status endpoints and their own auth. FastPix, like most of these, authenticates with Basic auth using an access token id and secret key, and playback later comes off a https://stream.fastpix.io/<playbackId>.m3u8 URL. Wire each adapter into the runner:

// bench/run.mjs
import { timed, pollUntilReady } from "./measure.mjs";
import * as fastpix from "./providers/fastpix.mjs";

const SOURCE = process.env.SOURCE_URL; // your representative test file, hosted

async function benchProvider(name, p) {
  const { ms: uploadMs, result: id } = await timed(`${name} upload`, () =>
    p.createAndUpload(SOURCE),
  );
  const { ms: readyMs } = await timed(`${name} time-to-ready`, () =>
    pollUntilReady(() => p.status(id)),
  );
  return { name, uploadSec: uploadMs / 1000, readySec: (uploadMs + readyMs) / 1000 };
}

console.table([await benchProvider("FastPix", fastpix) /*, ...others */]);

4. What the results actually looked like

I ran two files through this harness on the same 4G/10 Mbps profile. Here's the big one first, a 177.2 MB clip, across the four providers that completed it (Cloudinary failed this run):

Provider	Overall	Upload	Time-to-ready	Cold startup
FastPix	86	15.2s	29.4s	1.9s
Mux	83	47.7s	53.3s	905ms
api.video	79	17.0s	49.2s	2.99s
Gumlet	74	24.8s	268.9s	1.25s

FastPix led on getting the file uploaded and playable. But look at the last column: Mux painted the first frame fastest (905ms vs 1.9s). Two different "fast."

Now the same harness on a smaller 64.9 MB file, all five completing:

Provider	Overall	Upload	Time-to-ready
Cloudinary	95	16.6s	20.1s
Mux	83	24.6s	67.8s
api.video	77	15.0s	85.1s
Gumlet	77	17.6s	191.6s
FastPix	73	29.7s	58.9s

The winner flipped. Cloudinary processed the small file in about 2 seconds and ran away with it, while FastPix, first on the big file, landed fifth here. FastPix still processed faster than Mux (27.6s vs 42.4s), but its upload leg on this smaller file was slower, and on a 64.9 MB file the upload is a bigger share of the total.

⚠️ Read a bitrate table carefully. FastPix and api.video use content-aware encoding, so they hit similar visual quality at a lower rendition bitrate (FastPix averaged ~700 Kbps on the big file). Lower bitrate here means less bandwidth billed, not a worse picture. Don't rank providers by bitrate as if higher wins.

5. Turning results into a decision

The pattern held: providers tuned for raw processing speed win on small files, and providers tuned for the upload path win as files grow. That maps to your workload:

Large source files (long-form, screen recordings, camera masters): weight the upload/ingest leg. In separate tests, FastPix processed a 798.5 MB / 28-minute file in 1m52s vs Kaltura's 7m08s, and uploaded a 1.46 GB file in 2m45s vs Mux's 7m40s. The advantage compounds with size.
Short clips where first-frame speed rules: weight cold startup, where Mux was consistently quick.

And pricing sits under all of it, on axes a benchmark won't show:

Provider	Encoding	Delivery	Analytics
FastPix	free (standard plan)	~$0.00096/min @1080p	QoE free up to 100K views/mo
Mux	per-minute	per-minute	Mux Data from $499/mo (1M views)
api.video	free	$0.0017/min	included
Cloudflare Stream	free	$1 / 1,000 min	basic

Prices drift, so check the live pages before committing: FastPix, Mux, api.video, Cloudflare Stream.

💡 One non-benchmark cost: these are API-first platforms, not no-code CMSes. If you need a drag-and-drop dashboard for non-technical uploaders, budget to build that front-end yourself on top of whichever API wins.

What's next

Fork the harness, swap SOURCE_URL for a file that looks like what your product actually ingests, and add adapters for the two or three providers on your shortlist. Run it throttled, three times, and average. The leaderboard you build from your own footage is the only one that stays true. From there, the same measurement loop extends to per-resolution delivery cost and to QoE once you wire up a player and start logging rebuffer events.

The backend was green and the video kept buffering: a QoE debugging playbook

Mason K — Tue, 04 Aug 2026 06:12:26 +0000

TL;DR

Backend metrics can't see rebuffering, slow startups, or failed plays; those happen inside the player. We'll instrument an HTML5/hls.js player to capture real QoE signals in ~40 lines, look at what the numbers mean, and then be honest about where DIY telemetry stops scaling and what the managed options cost.

📦 Code: github.com/USER/player-qoe-starter (replace before publishing)

The incident that teaches everyone this lesson

A user writes in: "the video keeps stopping." You check Grafana. Origin 5xx: flat zero. CDN hit ratio: healthy. Uptime: perfect. You close the ticket as "cannot reproduce," and next week there are four more.

Nothing is lying. Your dashboards describe the server side of the socket, and playback quality is a client-side phenomenon. Here's the mismatch in table form:

Your dashboard sees	The viewer feels
200 OK, 38 ms TTFB	6 seconds of spinner before frame one
Successful segment fetches	Two rebuffers during minute 3
Healthy origin	Player error after a bitrate switch
Requests per second	Nothing: they gave up and left

A segment can download successfully but slower than the play head consumes it. That's a rebuffer, and no access log will ever contain it. So let's capture it where it happens.

1. Instrument the player in ~40 lines

No vendors yet; the browser gives you most of the signal for free.

// qoe.js
export function instrument(video, hls) {
  const session = {
    startupMs: null,
    rebufferCount: 0,
    rebufferMs: 0,
    bitrateSwitches: 0,
    fatalError: null
  };

  let clickedPlayAt = null;
  let stallStartedAt = null;

  video.addEventListener("play", () => {
    if (clickedPlayAt === null) clickedPlayAt = performance.now();
  });

  video.addEventListener("playing", () => {
    if (session.startupMs === null && clickedPlayAt !== null) {
      session.startupMs = Math.round(performance.now() - clickedPlayAt);
    }
    if (stallStartedAt !== null) {
      session.rebufferMs += Math.round(performance.now() - stallStartedAt);
      stallStartedAt = null;
    }
  });

  video.addEventListener("waiting", () => {
    // 'waiting' after playback started = a real stall, not startup
    if (session.startupMs !== null && stallStartedAt === null) {
      session.rebufferCount++;
      stallStartedAt = performance.now();
    }
  });

  video.addEventListener("error", () => {
    session.fatalError = video.error?.code ?? "unknown";
  });

  if (hls) {
    hls.on(Hls.Events.LEVEL_SWITCHED, () => session.bitrateSwitches++);
    hls.on(Hls.Events.ERROR, (_e, data) => {
      if (data.fatal) session.fatalError = data.details;
    });
  }

  return session;
}

Wire it up and dump the session on unload:

// player.js
const video = document.querySelector("video");
const hls = new Hls();
hls.loadSource("/video/lesson1/index.m3u8");
hls.attachMedia(video);

const session = instrument(video, hls);

addEventListener("visibilitychange", () => {
  if (document.visibilityState === "hidden") {
    navigator.sendBeacon("/qoe", JSON.stringify(session));
    console.table(session);
  }
});

Watch a video on throttled DevTools network and you'll see something like:

┌─────────────────┬──────────┐
│ startupMs       │ 4180     │
│ rebufferCount   │ 3        │
│ rebufferMs      │ 9640     │
│ bitrateSwitches │ 4        │
│ fatalError      │ null     │
└─────────────────┴──────────┘

That table is more diagnostic truth about the complaint than your entire backend stack currently has.

💡 Tip: sendBeacon on visibilitychange is the reliable exfil pattern. unload handlers get skipped on mobile and on killed tabs, while a beacon survives the page dying mid-request. Buffer events and flush periodically too, and you'll even catch the sessions that crash.

2. Reading the numbers

Three metrics carry most of the weight:

Startup time (play intent to first playing): viewers abandon fast when this stretches; keep an eye on the p95, not the average.
Rebuffer ratio (rebufferMs / totalWatchMs): the canonical QoE number. Even small single-digit percentages feel awful in a session.
Playback failure rate: sessions ending in a fatal error, sliced by browser/device/network before anything else. Failures cluster; averages hide them.

Startup time also decomposes: manifest fetch, first segment fetch, and decode each contribute. When p95 startup spikes, split it into those pieces before blaming the CDN.

⚠️ Note: the waiting event fires for seeks too. Production-grade trackers separate seek-induced stalls from network stalls (check video.seeking); our snippet keeps it simple.

3. Where DIY hits the ceiling

Our 40 lines capture five signals for one session in one tab. The gap between this and answering "which users had a bad Tuesday and why" is, concretely:

Sessionization: stitching beacons into views, views into viewers, across reloads.
Dimensions: device, OS, browser, network type, geo, CDN, rendition on every event.
Weighted aggregation: rebuffer ratio weighted by watch time, percentile startup by cohort.
Storage and retention: beacons arrive forever; someone owns that pipeline now.
Alerting: "failure rate for Android/Chrome on ISP X doubled" is the actual product.

I've built this in-house before. The v1 was a fun sprint; the maintenance was a permanent tax. Commercial QoE tools exist because items 1 through 5 are a product, not a feature. For scale reference, dedicated platforms track far more than our five signals: Mux Data exposes 30+ metrics, and FastPix's Video Data captures 50+ playback data points per view session.

4. The managed options, honestly

Current public pricing for the two developer-facing routes, plus DIY for contrast:

Option	Free tier	Beyond free	Signals
DIY (above)	your time	your time, forever	~5 until you build more
Mux Data	100K views/mo	$0.60/1K (PAYG); $499/mo Media plan incl. 1M	30+ metrics
FastPix Video Data	100K views/mo	usage-based	50+ per session

Notes from having used both sides of this table:

If your video already runs through Mux, Mux Data is the natural add; same ecosystem.
FastPix bundles Video Data with its video platform (encoding, delivery, player), so if your video is already there, the telemetry is a feature flag away rather than an integration: the player SDKs report into it, there's a real-time dashboard, and the same data is available over the API. It's also independently adoptable; you don't need to move your pipeline to use the analytics.
One caveat on FastPix: the docs are written for developers comfortable living in an API reference. Fine for us; the non-technical folks on your team may want a walkthrough before they can self-serve.
Whatever you pick, keep the DIY instrumentation knowledge. Understanding what waiting vs stalled vs a fatal ERROR means is how you'll interrogate any vendor dashboard instead of nodding at it.

5. The playbook

Next time "the video keeps stopping" lands in your queue:

Don't ask about their wifi yet.
Check whether you have session-level playback data. If not, that's the actual bug; ship the 40 lines above today.
Slice failures and rebuffer ratio by device, browser, and network before anything else. Bad QoE clusters.
Before buying or building a telemetry pipeline, check what your existing video stack already collects. Bundled analytics you're not using is the cheapest observability you'll ever ship.

What's next

Two follow-ups: wire the beacon endpoint into something queryable (a POST /qoe handler plus a table gets you shockingly far for a small app), and if you're on hls.js, look at CMCD, which standardizes shipping these client metrics to your CDN with hls.js 1.7 supporting CMCD v2. The viewer's side of the socket has plenty to say; the only question is whether you're listening.

FFmpeg 8.0 in practice: transcribe with the Whisper filter and probe av1_vulkan

Mason K — Tue, 04 Aug 2026 06:12:15 +0000

TL;DR

FFmpeg 8.0 "Huffman" adds a native whisper audio filter (transcription is now a filter, not a separate service) and av1_vulkan, a cross-vendor GPU AV1 encoder. We'll generate SRT + JSON captions in one command, then write a script that probes for av1_vulkan and falls back to SVT-AV1 when the GPU or driver isn't ready.

📦 Code: github.com/USER/ffmpeg8-pipeline-demo (replace before publishing)

FFmpeg 8.0 shipped in late August 2025. Two features change what your pipeline can offload to FFmpeg itself: speech-to-text and GPU-vendor-agnostic AV1 encoding. Let's actually run both.

1. Confirm you're on 8.0+

ffmpeg -version | head -n 1
# ffmpeg version 8.0.1 Copyright (c) 2000-2025 the FFmpeg developers

⚠️ Note: If your distro still ships 7.x, grab a static build or the updated PPA. The whisper filter and av1_vulkan encoder do not exist before 8.0. 8.0.1 is a safer pin than the initial 8.0 tag.

Check that the new bits are compiled in:

# Is the whisper filter present?
ffmpeg -hide_banner -filters | grep whisper
#  ... whisper           A->A       Transcribe audio using whisper.cpp.

# Is the Vulkan AV1 encoder present?
ffmpeg -hide_banner -encoders | grep av1_vulkan
#  V....D av1_vulkan           AV1 (Vulkan)

If av1_vulkan is missing, your build was compiled without Vulkan encode support. That's fine, we handle it in step 4.

2. Transcribe a video with the whisper filter 🎙️

The filter runs OpenAI's Whisper models through whisper.cpp, inside the filter graph. First grab a model:

# One-time: download a whisper.cpp GGML model
mkdir -p models
curl -L -o models/ggml-base.en.bin \
  https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin

Now generate an SRT in a single command. We drop video (-vn) because the filter only needs audio:

ffmpeg -i input.mp4 -vn -af \
  "whisper=model=models/ggml-base.en.bin:language=en:queue=10:destination=captions.srt:format=srt" \
  -f null -

# terminal output (trimmed)
[Parsed_whisper_0 @ 0x...] Loading model models/ggml-base.en.bin
[Parsed_whisper_0 @ 0x...] whisper backend: CPU
size=N/A time=00:03:41.00 bitrate=N/A speed=2.3x

You now have captions.srt. Want structured output for an index or search instead? Switch the format and send it to stdout:

ffmpeg -loglevel warning -i input.mp4 -vn -af \
  "whisper=model=models/ggml-base.en.bin:language=en:queue=10:destination=-:format=json" \
  -f null - > transcript.json

The one parameter that matters: queue

queue is how many seconds of audio buffer before a transcription pass runs.

queue value	behavior	use it for
small (2-3s)	frequent output, lower accuracy, higher CPU	near-live captioning
large (10-20s)	more context, higher accuracy, less CPU	batch captioning of uploads

💡 Tip: For an upload pipeline, go large. Whisper is more accurate with surrounding context, and you don't need sub-second latency on a file someone uploaded thirty seconds ago.

There's also use_gpu (default true) and language (set it explicitly; autodetect costs you a pass). This one command replaces the classic "extract audio, hand a WAV to a Python Whisper worker, stitch the SRT back" setup. One less service to run.

3. Try the av1_vulkan encoder

av1_vulkan uses Vulkan's video-encode extensions, so it runs across GPU vendors instead of locking you to NVENC (NVIDIA) or QSV (Intel). You need a Vulkan device and a driver that implements the encode extensions:

# Do you have a usable Vulkan device?
ffmpeg -hide_banner -init_hw_device vulkan=vk:0 -f lavfi -i nullsrc -t 0 -f null - 2>&1 | tail -n 3

A real encode uploads frames to the GPU with hwupload:

ffmpeg -i input.mp4 \
  -init_hw_device vulkan=vk:0 -filter_hw_device vk \
  -vf "format=nv12,hwupload" \
  -c:v av1_vulkan -b:v 4M \
  -c:a copy output_av1.mp4

When the hardware isn't ready, you'll see something like:

[av1_vulkan @ 0x...] Encoding of this format is not supported.
Error initializing output stream 0:0 -- Error while opening encoder

That error is the whole reason for the next step.

4. A safe wrapper: probe, then fall back to SVT-AV1 🛠️

Don't put av1_vulkan directly in production. Detect it, verify it actually runs, and fall back to CPU SVT-AV1 (the sober choice for production AV1 today) when it doesn't.

#!/usr/bin/env bash
# encode-av1.sh: prefer GPU av1_vulkan, fall back to SVT-AV1
set -euo pipefail

INPUT="$1"
OUTPUT="$2"
BITRATE="${3:-4M}"

have_vulkan_av1() {
  ffmpeg -hide_banner -encoders 2>/dev/null | grep -q 'av1_vulkan' || return 1
  # Dry-run one frame to confirm the driver really supports encode
  ffmpeg -hide_banner -loglevel error \
    -init_hw_device vulkan=vk:0 -filter_hw_device vk \
    -f lavfi -i testsrc=size=256x256:rate=1 -frames:v 1 \
    -vf "format=nv12,hwupload" -c:v av1_vulkan -f null - 2>/dev/null
}

if have_vulkan_av1; then
  echo "-> using av1_vulkan (GPU)"
  ffmpeg -y -i "$INPUT" \
    -init_hw_device vulkan=vk:0 -filter_hw_device vk \
    -vf "format=nv12,hwupload" \
    -c:v av1_vulkan -b:v "$BITRATE" -c:a copy "$OUTPUT"
else
  echo "-> av1_vulkan unavailable, falling back to SVT-AV1 (CPU)"
  ffmpeg -y -i "$INPUT" \
    -c:v libsvtav1 -preset 6 -crf 30 -c:a copy "$OUTPUT"
fi

chmod +x encode-av1.sh
./encode-av1.sh input.mp4 out.mp4 4M
# -> av1_vulkan unavailable, falling back to SVT-AV1 (CPU)

The dry-run in have_vulkan_av1 matters: the encoder being listed doesn't mean your driver implements the encode queue. Actually encoding one frame is the only honest check.

⚠️ Note: av1_vulkan is early. Even where it runs, don't assume it matches SVT-AV1 or a vendor AV1 encoder on quality-per-bit yet. Prototype and measure; don't rebuild your ladder on it this quarter.

What's next

Wire the whisper JSON output into a search index or a chapter generator (segment timestamps come along in the JSON).
Benchmark av1_vulkan vs libsvtav1 on files that look like your catalog, not a clean test clip, before trusting either.
Look at the other 8.0 additions if they touch you: VVC via VA-API, native APV and ProRes RAW decoders, and AVX-512 CPU optimizations.
Pin 8.0.1 (or newer point release) in your Dockerfile rather than the 8.0 tag.

The theme across both features is the same: work that used to need a second service bolted onto FFmpeg now lives inside FFmpeg. Shorter pipeline, fewer 3 AM pages. That's the upgrade worth doing.

Build multi-CDN failover with HLS content steering, hls.js, and a 40-line server

Mason K — Tue, 04 Aug 2026 06:12:00 +0000

TL;DR

We'll add real multi-CDN failover to an HLS stream without any client-side URL-rewriting hacks. You'll author a multivariant playlist with two pathways, build a ~40-line Express steering server that returns a JSON steering manifest, point hls.js at it, then flip CDN priority live and watch the player reroute with no reload and no lost buffer.

📦 Code: github.com/USER/hls-content-steering-demo (replace before publishing)

The usual multi-CDN "failover" is a catch block that rewrites segment URLs from CDN A to CDN B and reloads the stream. It loses the buffer, resets the ABR decision, and shows a spinner. Content steering moves the routing decision into the manifest layer, where the player already lives. It's standardized in HLS and DASH, and hls.js supports it. Let's build it.

1. Tag your playlist with pathways

A pathway is a named route to your content, usually one per CDN. In the multivariant (master) playlist you point the player at a steering server and give every variant a PATHWAY-ID.

#EXTM3U
#EXT-X-CONTENT-STEERING:SERVER-URI="http://localhost:3000/steering",PATHWAY-ID="cdn-a"

# --- CDN A renditions ---
#EXT-X-STREAM-INF:BANDWIDTH=800000,RESOLUTION=640x360,PATHWAY-ID="cdn-a"
https://cdn-a.example.com/360p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=2400000,RESOLUTION=1280x720,PATHWAY-ID="cdn-a"
https://cdn-a.example.com/720p.m3u8

# --- CDN B renditions (same ladder, different host) ---
#EXT-X-STREAM-INF:BANDWIDTH=800000,RESOLUTION=640x360,PATHWAY-ID="cdn-b"
https://cdn-b.example.com/360p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=2400000,RESOLUTION=1280x720,PATHWAY-ID="cdn-b"
https://cdn-b.example.com/720p.m3u8

Same ladder, listed once per CDN, distinguished by PATHWAY-ID. The SERVER-URI tells the player where to fetch its routing instructions.

2. The steering server

The steering server answers one question: in what order should the player prefer the pathways right now? That's a JSON document.

// server.js
import express from "express";

const app = express();

// In-memory priority. Flip this to simulate a CDN going bad.
let priority = ["cdn-a", "cdn-b"];

app.get("/steering", (req, res) => {
  res.json({
    VERSION: 1,
    TTL: 5, // seconds; keep short for the demo, use ~300 in prod
    "RELOAD-URI": "http://localhost:3000/steering",
    "PATHWAY-PRIORITY": priority,
  });
});

// Ops endpoint: reorder pathways with no deploy, no player reload
app.post("/failover/:winner", (req, res) => {
  const winner = req.params.winner;
  priority = [winner, ...priority.filter((p) => p !== winner)];
  res.json({ ok: true, priority });
});

app.listen(3000, () => console.log("steering server on :3000"));

💡 Tip: TTL controls how often the player re-fetches the manifest. Short values react fast but add requests. In production, something like 300 seconds is normal, with your monitoring flipping priority the instant a CDN degrades.

The fields are the whole spec surface you need:

Field	Meaning
`VERSION`	Steering manifest version (1)
`TTL`	Seconds until the player re-fetches
`RELOAD-URI`	Where to fetch the next manifest (optional)
`PATHWAY-PRIORITY`	Ordered list of pathway IDs, most preferred first

3. Wire up hls.js 🎥

Content steering landed in hls.js 1.5; use a current 1.6.x. The key part is that you write no failover code. hls.js reads the EXT-X-CONTENT-STEERING tag, fetches your manifest, and orders pathways itself.

<!-- index.html -->
<video id="video" controls width="640"></video>
<script src="https://cdn.jsdelivr.net/npm/hls.js@1.6"></script>
<script>
  const video = document.getElementById("video");

  if (Hls.isSupported()) {
    const hls = new Hls({ debug: false });

    // Fires each time the steering manifest is (re)loaded
    hls.on(Hls.Events.STEERING_MANIFEST_LOADED, (_e, data) => {
      console.log("steering priority:", data.steeringManifest["PATHWAY-PRIORITY"]);
    });

    hls.loadSource("http://localhost:8080/master.m3u8");
    hls.attachMedia(video);
  }
</script>

Open the console and you'll see the priority list logged every TTL:

steering priority: ["cdn-a","cdn-b"]
steering priority: ["cdn-a","cdn-b"]

4. Fail over live and watch nothing break

With the stream playing, tell the steering server CDN B wins:

curl -X POST http://localhost:3000/failover/cdn-b
# {"ok":true,"priority":["cdn-b","cdn-a"]}

Within one TTL, the console updates and the network tab shows segment requests move to cdn-b.example.com:

steering priority: ["cdn-b","cdn-a"]

The buffer is intact. The current bitrate is intact. No reload, no spinner. When CDN A recovers, put it back:

curl -X POST http://localhost:3000/failover/cdn-a

Traffic drifts home on the next poll. This is the difference from URL rewriting: you changed a preference, not an address, so the player adjusted future requests instead of tearing down the session.

5. What to harden before production

Don't trust a static list. A global priority order can't know CDN A is degraded in one region only. Feed the server real signals (error rates, latency, per-region health) and reorder from those.
Cache correctly. Serve the steering manifest with a short cache lifetime so a reorder actually reaches players.
Mind CORS. The steering endpoint is fetched by the player; send the right Access-Control-Allow-Origin.
DASH too. dash.js supports the same concept via service locations and CDN priority, so one steering strategy can cover both stacks.

⚠️ Note: The protocol is solved; the decision is not. Recent 2026 research (CADENCE, on dash.js) is all about making steering quality-aware with live per-CDN telemetry instead of static heuristics. Start static to get clean failover, then make it smart.

What's next

Add a third pathway and a region-aware priority function.
Swap the in-memory priority for something driven by your monitoring or a health-check loop.
Read the Apple HLS Content Steering spec and the DASH-IF spec (published as ETSI TS 103 998) for the full attribute set.

You just deleted the failover code you were going to maintain and replaced it with a JSON endpoint. That's a good trade. #webdev

Auto-generate video chapters with Whisper + an LLM (without hallucinated timestamps)

Mason K — Tue, 04 Aug 2026 06:11:49 +0000

TL;DR

We'll auto-generate chapters for a video. The trick: transcribe with timestamps, let the LLM choose chapter boundaries by segment index (never raw time), then map indices back to real timestamps in code. That sidesteps the one thing LLMs are reliably bad at: echoing exact timecodes. Output is a chapters.vtt file plus YouTube-style stamps.

📦 Code: github.com/USER/llm-video-chapters (replace before publishing)

Pasting a transcript into a chat model and asking for "chapters with timestamps" produces great titles and fabricated times. LLMs don't preserve exact timestamps across long inputs, and long-context models also skip the middle. The fix is to stop asking the model to be a clock. Let's build the pipeline that does it right.

1. Transcribe with real timestamps

Use WhisperX (or faster-whisper) for aligned segment timestamps. Each segment has a start, end, and text.

pip install whisperx

# transcribe.py
import whisperx, json

device = "cpu"  # or "cuda"
model = whisperx.load_model("large-v3", device, compute_type="int8")

audio = whisperx.load_audio("talk.mp4")
result = model.transcribe(audio, batch_size=8)

# result["segments"] -> [{"start": 0.0, "end": 4.12, "text": "..."}, ...]
with open("segments.json", "w") as f:
    json.dump(result["segments"], f, indent=2)

print(f"{len(result['segments'])} segments")

python transcribe.py
# 214 segments

💡 Tip: FFmpeg 8.0's native whisper filter can also emit timestamped JSON if you'd rather not add a Python dependency. Either way, the shape you need is a list of segments with start times.

2. Number the segments, keep times in a lookup

The model is about to see indexed text. It is not going to see the raw timestamps. We keep those in a table keyed by index.

# build_index.py
import json

segments = json.load(open("segments.json"))

# Lookup table: index -> start time (seconds)
start_times = {i: seg["start"] for i, seg in enumerate(segments)}

# Numbered transcript the model will read
numbered = "\n".join(f"[{i}] {seg['text'].strip()}" for i, seg in enumerate(segments))

open("numbered.txt", "w").write(numbered)

numbered.txt looks like:

[0] Welcome everyone, thanks for joining.
[1] Today we're going to talk about database indexing.
[2] But first, a quick recap of last week.
...
[24] Okay, let's actually set up the schema.

3. Ask the LLM for boundaries by index

This is where correctness lives. The prompt asks for JSON with a start_segment (an integer) and a title. The model never touches a timestamp.

# chapter.py
import json, os
from openai import OpenAI  # any chat LLM works; this is just an example client

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
numbered = open("numbered.txt").read()
segments = json.load(open("segments.json"))
duration_min = segments[-1]["end"] / 60

# Scale chapter count to length so short videos aren't over-segmented
target = "3-5" if duration_min < 15 else "8-14"

prompt = f"""You are segmenting a video transcript into chapters.
The transcript is a numbered list of segments: [index] text.

Return ONLY JSON: a list of objects with "start_segment" (the index where a
new chapter begins) and "title" (a short, specific 2-6 word label).
Rules:
- The first chapter MUST start at segment 0.
- Choose boundaries where the topic clearly changes.
- Aim for {target} chapters for this {duration_min:.0f}-minute video.
- Do NOT include timestamps. Only segment indices.

Transcript:
{numbered}
"""

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": prompt}],
    response_format={"type": "json_object"},
    temperature=0.2,
)

chapters = json.loads(resp.choices[0].message.content)["chapters"]
json.dump(chapters, open("chapters_raw.json", "w"), indent=2)
print(chapters)

python chapter.py
# [{'start_segment': 0, 'title': 'Introduction'},
#  {'start_segment': 24, 'title': 'Setting up the schema'},
#  {'start_segment': 58, 'title': 'B-tree vs hash indexes'}, ...]

4. Reattach real timestamps in code

The model gave us where (indices). Code supplies when (real times from Whisper). This step cannot hallucinate.

# finalize.py
import json

segments = json.load(open("segments.json"))
chapters = json.load(open("chapters_raw.json"))
start_times = {i: seg["start"] for i, seg in enumerate(segments)}
video_end = segments[-1]["end"]

# Attach start/end times; each chapter ends where the next begins
final = []
for n, ch in enumerate(chapters):
    start = start_times[ch["start_segment"]]
    end = (start_times[chapters[n + 1]["start_segment"]]
           if n + 1 < len(chapters) else video_end)
    final.append({"start": start, "end": end, "title": ch["title"]})

json.dump(final, open("chapters.json", "w"), indent=2)

5. Emit WebVTT + YouTube stamps

Standard, boring output formats. A WebVTT chapters track renders markers on the scrub bar via <track kind="chapters">.

# emit.py
import json

def ts(sec):  # seconds -> HH:MM:SS.mmm
    h, rem = divmod(sec, 3600)
    m, s = divmod(rem, 60)
    return f"{int(h):02d}:{int(m):02d}:{s:06.3f}"

def yt(sec):  # seconds -> mm:ss (or h:mm:ss)
    h, rem = divmod(int(sec), 3600)
    m, s = divmod(rem, 60)
    return f"{h}:{m:02d}:{s:02d}" if h else f"{m:02d}:{s:02d}"

chapters = json.load(open("chapters.json"))

# WebVTT
vtt = ["WEBVTT", ""]
for ch in chapters:
    vtt += [f"{ts(ch['start'])} --> {ts(ch['end'])}", ch["title"], ""]
open("chapters.vtt", "w").write("\n".join(vtt))

# YouTube-style (first stamp MUST be 00:00 or the platform ignores all of them)
print("\n".join(f"{yt(ch['start'])} {ch['title']}" for ch in chapters))

chapters.vtt:

WEBVTT

00:00:00.000 --> 00:01:38.400
Introduction

00:01:38.400 --> 00:04:02.900
Setting up the schema

Attach it to any HTML5 player:

<video controls>
  <source src="talk.mp4" type="video/mp4" />
  <track kind="chapters" src="chapters.vtt" srclang="en" label="Chapters" default />
</video>

Things to still watch 👀

Title style: constrain length in the prompt or you'll get a mix of terse labels and rambling sentences.
Granularity: scaling chapter count to duration (we did this) avoids over-segmenting short clips and under-segmenting long ones, the long-context omission problem.
Review high-stakes videos: timestamps are guaranteed correct; the model's judgment about where a topic turns is not. A quick human pass fixes the rare bad boundary.

What's next

Batch it across your library and store chapters.json next to each asset.
Add on-screen frame captions to the transcript for videos where the audio alone is thin (the Chapter-Llama CVPR 2025 approach blends both).
Feed the same segments into search or a summary endpoint. You already did the expensive part (transcription); reuse it.

The whole idea in one line: let the model decide the seams, let your code keep the time. #tutorial

Lock down HLS playback with JWT signed URLs (Express 5 + hls.js)

Mason K — Tue, 04 Aug 2026 06:11:34 +0000

TL;DR

Anyone who copies your .m3u8 URL can watch your paywalled video from anywhere, forever. We're fixing that with an Express 5 gateway that mints short-lived JWTs, validates them on the manifest AND every segment, and rewrites playlists so the token travels automatically. Plus the bug everyone ships first: tokens that expire mid-movie.

📦 Code: github.com/USER/hls-signed-playback (replace before publishing)

The problem in one curl

Your app has a login wall. Your video does not:

curl -s https://cdn.example.com/course/lesson1/index.m3u8 | head -5
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:6
#EXTINF:6.0,
seg_000.ts

No cookie, no session, full response. Paste that URL into VLC and the whole lesson plays. HLS is a master playlist pointing at segment files, hundreds of separate HTTP requests, and if none of them check authorization, your paywall only protects the HTML around the player.

Let's fix it end to end. You need node 24.x and ffmpeg (any recent build; 7.x or 8.x both fine) to generate test content.

1. Set up the project

mkdir hls-signed-playback && cd hls-signed-playback
npm init -y && npm i express jsonwebtoken
mkdir media public

Generate an HLS rendition of any mp4 you have lying around:

ffmpeg -i sample.mp4 -c:v libx264 -c:a aac \
  -hls_time 6 -hls_playlist_type vod \
  -hls_segment_filename "media/seg_%03d.ts" \
  media/index.m3u8

ls media
index.m3u8  seg_000.ts  seg_001.ts  seg_002.ts  ...

2. Mint short-lived playback tokens

// server.js
import express from "express";
import jwt from "jsonwebtoken";
import fs from "node:fs";
import path from "node:path";

const app = express();
const SECRET = process.env.PLAYBACK_SECRET ?? "dev-only-secret";
const MEDIA_DIR = path.resolve("media");

// In real life this sits behind your session auth.
app.post("/api/playback-token/:videoId", (req, res) => {
  const token = jwt.sign(
    { sub: "user-123", vid: req.params.videoId },
    SECRET,
    { expiresIn: "15m" }
  );
  res.json({ token });
});

The claims matter: vid pins the token to one video, sub ties it to a user (you'll want that later for revocation and concurrency caps), exp comes free from expiresIn.

3. Validate the manifest, and rewrite it

Here's the core trick. The player only requests URLs it was given. So when we serve the manifest, we append the token to every segment URI, and segment auth happens automatically:

// server.js (continued)
function verify(req, res, next) {
  try {
    const payload = jwt.verify(req.query.token, SECRET, { clockTolerance: 60 });
    if (payload.vid !== req.params.videoId) throw new Error("wrong video");
    req.token = req.query.token;
    next();
  } catch {
    res.status(403).json({ error: "invalid or expired token" });
  }
}

app.get("/video/:videoId/index.m3u8", verify, (req, res) => {
  const raw = fs.readFileSync(path.join(MEDIA_DIR, "index.m3u8"), "utf8");
  const signed = raw
    .split("\n")
    .map(line =>
      line && !line.startsWith("#")
        ? `${line}?token=${req.token}`
        : line
    )
    .join("\n");
  res.type("application/vnd.apple.mpegurl").send(signed);
});

app.get("/video/:videoId/:file", verify, (req, res) => {
  res.sendFile(path.join(MEDIA_DIR, path.basename(req.params.file)));
});

app.use(express.static("public"));
app.listen(3000, () => console.log("http://localhost:3000"));

💡 Tip: clockTolerance: 60 forgives a minute of client/server clock skew. Strict exp checks plus one device with a fast clock equals failures you will never reproduce locally.

Non-comment lines in an m3u8 are URIs, so the rewrite is a two-line map. For multi-rendition streams you'd apply the same rewrite to the master playlist's variant URIs too.

4. Wire up the player

<!-- public/index.html -->
<video id="video" controls></video>
<script src="https://cdn.jsdelivr.net/npm/hls.js@1"></script>
<script>
async function play() {
  const { token } = await (await fetch("/api/playback-token/lesson1", {
    method: "POST"
  })).json();

  const video = document.getElementById("video");
  const src = `/video/lesson1/index.m3u8?token=${token}`;

  if (Hls.isSupported()) {
    const hls = new Hls();
    hls.loadSource(src);
    hls.attachMedia(video);
  } else {
    video.src = src; // Safari plays HLS natively
  }
}
play();
</script>

Because the token rides in the URL and we rewrote the playlist, hls.js (v1.7 as of this writing) needs zero special config. If you'd rather use an Authorization header or cookies instead of query strings, hls.js has you covered with xhrSetup:

const hls = new Hls({
  xhrSetup: (xhr) => {
    xhr.withCredentials = true; // cookie mode
    // or: xhr.setRequestHeader("Authorization", `Bearer ${token}`);
  }
});

Run it and confirm both doors are locked:

node server.js
curl -i http://localhost:3000/video/lesson1/index.m3u8
HTTP/1.1 403 Forbidden

curl -i "http://localhost:3000/video/lesson1/seg_000.ts"
HTTP/1.1 403 Forbidden

The copied-link attack is dead: a shared URL stops working when its token expires.

5. The bug you just shipped: expiry at minute 41 ⏰

Set expiresIn: "30s" temporarily and keep watching past the 30 second mark. Around the next segment request:

GET /video/lesson1/seg_007.ts?token=eyJhb... 403

hls.js retries, then surfaces a fatal network error and playback dies. In production this is the two-hour lecture that always fails around minute 41, only for your most engaged viewers, and never in your five-minute tests.

Two sane fixes:

VOD: outlive the content. The token's job is to gate the start of playback and kill shared links, so set the TTL longer than your longest video (say, duration plus an hour). A 15 minute TTL on a 2 hour film adds support tickets, not security.
Live or long sessions: slide the credential. Use cookie mode plus a refresh endpoint the page calls periodically, or re-issue the token when the player refreshes the playlist (live players re-fetch manifests every few seconds anyway).

6. Production notes

Move verification to the edge. The Express gateway is perfect for understanding the pattern, but at scale you don't want segment traffic through your app servers. The same JWT check runs in a CloudFront Function or a Cloudflare Worker, and CloudFront's signed cookies are a managed version of cookie mode.
Mind the CDN cache key. Query-string tokens mean your CDN must either strip the query from the cache key (so all users share cached segments while your edge still validates) or you'll get zero cache hits. Decide which layer validates, then configure caching deliberately.
This is anti-sharing, not DRM. Signed URLs kill link-sharing and hotlinking. They don't stop screen recording. If a content contract requires Widevine/FairPlay, that's a different (and much heavier) project.

7. The hardening checklist

Before this pattern faces the internet:

Query tokens leak into logs. ?token= ends up in CDN access logs, proxy logs, and sometimes Referer headers. Short TTLs bound the damage, cookie mode avoids it entirely, and either way, scrub query strings from logs you retain.
Set the cookie flags. Cookie mode wants HttpOnly, Secure, and SameSite=Lax (or SameSite=None; Secure if your CDN lives on another origin).
HS256 is fine for one service. The moment a CDN worker and your app server both verify tokens, switch to RS256 so the edge only ever holds a public key.
Pin claims deliberately. vid pins content and is cheap. IP pinning sounds tempting but mobile networks rotate addresses; strict IP checks break every commuter on a train.
Rotate the secret. Short-lived tokens make secret rotation nearly painless, which is itself a reason to keep TTLs sane.

What's next

Two natural extensions: cap concurrent sessions per sub claim (you now have the data to do it), and per-user watermarking keyed off the same token. If you're on a managed video platform, check its docs before building any of this; most ship a signed-playback feature that is exactly this pattern with the edge part done for you.

Ship a Go Live button: browser WHIP publishing with MediaMTX, no SDKs

Mason K — Tue, 04 Aug 2026 06:11:17 +0000

TL;DR

We're building a "Go Live" button that broadcasts your camera from a plain browser tab, and a watch page that plays it back with sub-second delay. The stack is WHIP + WHEP against a self-hosted MediaMTX server. Zero vendor SDKs, about 60 lines of JavaScript total.

📦 Code: github.com/USER/whip-go-live (replace before publishing)

What we're building

WHIP (WebRTC-HTTP Ingestion Protocol) became RFC 9725 in March 2025, and it reduces WebRTC broadcast signaling to a single HTTP POST. WHEP is the same idea for playback. Together they mean the browser can publish and play live video with fetch() and RTCPeerConnection, nothing else.

We'll run MediaMTX (a zero-dependency media server that speaks WHIP, WHEP, RTMP, SRT, and LL-HLS) in Docker, write a publisher page, write a viewer page, then point OBS at the same endpoint to prove the "standard" part of the standard.

You need Docker, any local web server, and a webcam. Node isn't required for the pages, but examples assume a current setup (node 24.x if you use one).

1. Start MediaMTX 🐳

docker run --rm -it \
  -p 8889:8889 \
  -p 8189:8189/udp \
  bluenviron/mediamtx

Port 8889 is the WebRTC HTTP endpoint (WHIP and WHEP live here). Port 8189/udp carries the actual media (ICE). You should see startup logs like:

INF MediaMTX vX.Y.Z
INF [WebRTC] listener opened on :8889 (HTTP), :8189 (ICE/UDP)

Every stream path on the server automatically gets two endpoints:

Purpose	URL
Publish (WHIP)	`http://localhost:8889/mystream/whip`
Play (WHEP)	`http://localhost:8889/mystream/whep`

No pre-registration needed; publishing to a path creates it.

2. The publisher: a Go Live button in ~35 lines

<!-- publish.html -->
<video id="preview" autoplay muted playsinline></video>
<button id="go">Go Live</button>

<script>
const WHIP_URL = "http://localhost:8889/mystream/whip";
let pc, sessionUrl;

async function goLive() {
  const stream = await navigator.mediaDevices.getUserMedia({
    video: { width: 1280, height: 720 },
    audio: true
  });
  document.getElementById("preview").srcObject = stream;

  pc = new RTCPeerConnection();
  stream.getTracks().forEach(t => pc.addTrack(t, stream));

  await pc.setLocalDescription(await pc.createOffer());
  await waitForIceGathering(pc);

  const res = await fetch(WHIP_URL, {
    method: "POST",
    headers: { "Content-Type": "application/sdp" },
    body: pc.localDescription.sdp
  });
  if (res.status !== 201) throw new Error(`WHIP publish failed: ${res.status}`);

  sessionUrl = new URL(res.headers.get("Location"), WHIP_URL).href;
  await pc.setRemoteDescription({ type: "answer", sdp: await res.text() });
  console.log("live!");
}

function waitForIceGathering(pc) {
  return new Promise(resolve => {
    if (pc.iceGatheringState === "complete") return resolve();
    pc.addEventListener("icegatheringstatechange", () =>
      pc.iceGatheringState === "complete" && resolve());
  });
}

async function stopLive() {
  if (sessionUrl) await fetch(sessionUrl, { method: "DELETE" });
  pc?.close();
}

document.getElementById("go").onclick = goLive;
addEventListener("beforeunload", stopLive);
</script>

The protocol is all visible right there: POST your SDP offer, get a 201 Created with the answer in the body, and keep the Location header. That URL is your session; DELETE it to hang up cleanly.

💡 Tip: we wait for ICE gathering to finish before POSTing so the offer contains all candidates. WHIP also supports trickle ICE via PATCH, but the wait-then-send version is simpler and fine on a LAN.

Serve it (browsers only allow camera access from secure contexts, and localhost counts):

npx serve .
# then open http://localhost:3000/publish.html

3. The viewer: WHEP in ~25 lines

<!-- watch.html -->
<video id="player" autoplay playsinline controls></video>

<script>
const WHEP_URL = "http://localhost:8889/mystream/whep";

async function watch() {
  const pc = new RTCPeerConnection();
  pc.addTransceiver("video", { direction: "recvonly" });
  pc.addTransceiver("audio", { direction: "recvonly" });

  pc.ontrack = (e) => {
    document.getElementById("player").srcObject = e.streams[0];
  };

  await pc.setLocalDescription(await pc.createOffer());

  const res = await fetch(WHEP_URL, {
    method: "POST",
    headers: { "Content-Type": "application/sdp" },
    body: pc.localDescription.sdp
  });
  await pc.setRemoteDescription({ type: "answer", sdp: await res.text() });
}

watch();
</script>

Same handshake, opposite direction. Open publish.html in one tab, click Go Live, open watch.html in another. Wave at yourself; the delay should be well under a second on localhost.

MediaMTX also ships a built-in test player at http://localhost:8889/mystream if you want to check the stream without your own page.

4. Prove it's a standard: publish from OBS 🎥

OBS has shipped native WHIP output since version 30. In OBS:

Settings → Stream
Service: WHIP
Server: http://localhost:8889/mystream/whip
Bearer Token: leave empty for our unauthenticated dev server
Start Streaming

Your watch.html page now plays the OBS feed instead. Same endpoint, same viewer code, different publisher. FFmpeg can do this too: version 8.0 added a WHIP muxer, so even a headless box can publish sub-second WebRTC (FFmpeg is on 8.1 these days).

That interchangeability is the whole point. The publisher snippet above works unchanged against any WHIP endpoint, including managed ones like Cloudflare Stream or Amazon IVS Real-Time; you swap the URL and add the bearer token they give you.

5. Things that will bite you 🛠️

⚠️ Note: getUserMedia requires HTTPS everywhere except localhost. The moment you demo this to a teammate over LAN IP, the camera prompt silently never appears. Put a TLS proxy (Caddy makes this a one-liner) in front, or tunnel.

UDP reachability. Media flows over 8189/udp. If viewers are remote and behind restrictive NATs or corporate firewalls, you need STUN (MediaMTX configures a default) and eventually a TURN relay. WHIP standardized signaling, not firewall physics.

Auth before you ship. Our dev server accepts any publisher. In mediamtx.yml, give the publish side credentials and pass them from the browser or as the OBS bearer token. Never expose an open WHIP endpoint to the internet unless you enjoy surprise anime streams on your homepage.

WHEP is not an RFC yet. WHIP is a frozen standard. WHEP is still an Internet-Draft (draft-ietf-wish-whep, currently expired awaiting revisions) even though Cloudflare, LiveKit, MediaMTX, OvenMediaEngine, and others already ship it. In practice the flow above is stable across implementations, but pin your server version and skim changelogs when upgrading.

Autoplay policy. Browsers block un-muted autoplay. Our watch page works because you opened it deliberately, but if playback ever starts without a user gesture, add muted to the <video> tag or start paused. Otherwise play() rejects and you'll spend an hour blaming WHEP for what is actually the autoplay police.

Codec negotiation. WHIP doesn't choose codecs; the SDP exchange does. Browsers publish H.264 or VP8 by default and effectively every viewer can decode those. If you experiment with AV1 publishing, verify what your audience's devices can decode before committing a product to it.

Scale is a different article. WebRTC playback holds per-viewer state server-side. For a handful of viewers, one MediaMTX instance is plenty. For thousands, the standard move is hybrid: WHIP in, LL-HLS out for the crowd. MediaMTX already serves LL-HLS of the same stream at http://localhost:8888/mystream with zero extra config, so you can compare both outputs right now:

Output	URL	Rough delay
WHEP (WebRTC)	`:8889/mystream/whep`	sub-second
LL-HLS	`:8888/mystream`	a few seconds

What's next

Two follow-ups worth your time. First, wire real auth: JWT-gated WHIP publish plus tokenized playback, which turns this demo into something you could put behind a product's Go Live button. Second, try the same publisher code against a managed WHIP endpoint and notice how little changes; that's the RFC doing its job.

The era of "install this 20 MB SDK to stream from a browser" is ending. It's fetch() and a peer connection now.

DEV Community: Mason K

Detect variable frame rate uploads with ffprobe and normalize them before they break your pipeline

TL;DR

The bug

1. Why you keep receiving VFR 🎥

2. Why the popular check is only a heuristic ⚠️

3. The real check: per-frame deltas

4. Normalize with -fps_mode cfr

5. Wire it into ingest

The cost, honestly

Errors you'll hit

What's next

Build a directional preload window for a scrolling video feed with hls.js

TL;DR

The problem

1. Why preload="auto" isn't a strategy

2. A player wrapper with two modes 🎛️

3. The directional window

4. The reconciler (this is the important part) 🔑

5. Teardown is not optional on iOS

6. Measure the thing users feel 📊

What's next

Build semantic search inside video: shot embeddings + transcript chunks, fused on the timeline

TL;DR

What we're building

1. Chunk on shots, not on seconds 🎬

2. Embed the frames

3. Transcript chunks want different boundaries

4. Schema

5. Fuse at query time, not index time

6. The part that isn't in the architecture diagram 🔒

Errors you'll hit

Where the numbers come from

What's next

Sources

Stop hls.js from flapping between quality levels on cellular (with abrSwitchInterval)

TL;DR

The bug nobody reports correctly

1. First, detect the flap 📊

2. Layer one: give the estimator a longer memory

3. Layer two: make upswitches earn it

4. Layer three: rate-limit the switches with abrSwitchInterval 🚦

5. Layer four (last resort): cap levels on known-bad networks

6. Putting it together as a profile

What's next

AV2 shipped its 1.0 encoder. Here's what to actually run in your pipeline instead.

TL;DR

The news, in one paragraph

1. Probe what your machines can actually do 🔍

2. The mistake to avoid: software-decode AV1 on phones

3. Build a ladder you can ship today 🎞️

4. Wire the canary for AV2 🪝

5. If you truly need more compression right now

What's next

Benchmark video APIs yourself: upload to playable, measured across five providers

TL;DR

Why roll your own

1. The measurement harness 🛠️

2. Throttle the network so the numbers mean something

3. A provider adapter (FastPix as the example)

4. What the results actually looked like

5. Turning results into a decision

What's next

The backend was green and the video kept buffering: a QoE debugging playbook

TL;DR

The incident that teaches everyone this lesson

1. Instrument the player in ~40 lines

2. Reading the numbers

3. Where DIY hits the ceiling

4. The managed options, honestly

5. The playbook

What's next

FFmpeg 8.0 in practice: transcribe with the Whisper filter and probe av1_vulkan

TL;DR

1. Confirm you're on 8.0+

2. Transcribe a video with the whisper filter 🎙️

The one parameter that matters: queue

3. Try the av1_vulkan encoder

4. A safe wrapper: probe, then fall back to SVT-AV1 🛠️

What's next

1. Why `preload="auto"` isn't a strategy