Build an audio description pipeline that actually fits the gaps

TL;DR

We're building an audio description pipeline in Python: find the silent gaps with FFmpeg, compute a word budget per gap, generate descriptions constrained to that budget, render TTS, reject anything that overruns, and mux the result as a second HLS audio track. The overrun check is the step everyone skips and the one that makes the output usable.

Captions were easy because text sits in its own layer. Audio description has to go into the channel that already has the dialogue in it, so it's a packing problem, not a writing problem. A great 9-second description is a defect if the gap is 2.4 seconds.

That constraint drives the whole design. Let's build it.

What we're building

video.mp4
   ├─ 1. silence map        (ffmpeg silencedetect)  → gaps[]
   ├─ 2. budget per gap     (duration × speech rate) → max_words
   ├─ 3. frame sampling     (ffmpeg, mid-gap frames) → context
   ├─ 4. description        (constrained to max_words)
   ├─ 5. TTS + HARD CHECK   (reject if rendered > gap) → retry shorter
   ├─ 6. mix                (overlay narration on original audio)
   └─ 7. package            (second audio rendition in the HLS ladder)

Requirements: ffmpeg 6.0+ and python 3.11+.

ffmpeg -version | head -1
python3 --version

ffmpeg version 7.1 Copyright (c) 2000-2024 the FFmpeg developers
Python 3.12.3

1. Find the gaps 🔍

silencedetect is the whole first step. It writes to stderr, not stdout, which catches people out:

ffmpeg -i video.mp4 -af silencedetect=noise=-30dB:d=1.2 -f null - 2>&1 | grep silence_

[silencedetect @ 0x14e004c30] silence_start: 3.264
[silencedetect @ 0x14e004c30] silence_end: 6.912 | silence_duration: 3.648
[silencedetect @ 0x14e004c30] silence_start: 19.04
[silencedetect @ 0x14e004c30] silence_end: 21.44 | silence_duration: 2.4

Two parameters to tune per content type. noise=-30dB is the threshold for what counts as silence, and you'll want it lower (more strict) for content with music beds, or you'll "find" gaps that are full of score. d=1.2 is the minimum duration, because a gap shorter than about a second can't hold a useful sentence anyway.

# src/gaps.py
import re
import subprocess
from dataclasses import dataclass

SILENCE_RE = re.compile(
    r"silence_start: (?P<start>[\d.]+)|silence_end: (?P<end>[\d.]+) \| silence_duration: (?P<dur>[\d.]+)"
)

@dataclass
class Gap:
    start: float
    end: float
    duration: float

def find_gaps(path: str, noise_db: int = -30, min_dur: float = 1.2) -> list[Gap]:
    proc = subprocess.run(
        ["ffmpeg", "-i", path, "-af",
         f"silencedetect=noise={noise_db}dB:d={min_dur}", "-f", "null", "-"],
        capture_output=True, text=True,
    )
    gaps, pending_start = [], None
    for m in SILENCE_RE.finditer(proc.stderr):
        if m.group("start"):
            pending_start = float(m.group("start"))
        elif m.group("end") and pending_start is not None:
            end = float(m.group("end"))
            gaps.append(Gap(pending_start, end, float(m.group("dur"))))
            pending_start = None
    return gaps

⚠️ Leave headroom. Starting narration the instant dialogue stops sounds wrong and clips on some decoders. Trim 150ms off each end of the gap before you budget it.

2. Turn duration into a word budget 📏

This is the step that makes the pipeline work, and it's just arithmetic:

# src/budget.py
from src.gaps import Gap

# Tune this for your TTS voice. Measure it, don't trust it:
# render 100 known-length sentences and divide.
WORDS_PER_SECOND = 2.5
EDGE_PADDING = 0.15  # seconds trimmed from each end

def word_budget(gap: Gap) -> int:
    usable = gap.duration - (EDGE_PADDING * 2)
    return max(0, int(usable * WORDS_PER_SECOND))

def usable_gaps(gaps: list[Gap], min_words: int = 4) -> list[tuple[Gap, int]]:
    scored = [(g, word_budget(g)) for g in gaps]
    return [(g, w) for g, w in scored if w >= min_words]

>>> gaps = find_gaps("video.mp4")
>>> len(gaps)
47
>>> len(usable_gaps(gaps))
31

Sixteen gaps were too short to say anything meaningful in. That's normal, and it's information: those are the moments where standard description cannot comply, and where WCAG's extended audio description criterion (1.2.7, Level AAA) exists precisely because the only fix is pausing the video. Log them, don't silently drop them.

💡 Tip: WORDS_PER_SECOND = 2.5 is a starting assumption, not a measured constant. Different voices and languages differ substantially. Measure yours and put the real number here.

3. Sample frames for visual context 🖼️

We need to know what's on screen during the gap, so grab a few frames from inside it:

# src/frames.py
import subprocess
from pathlib import Path
from src.gaps import Gap

def sample_frames(video: str, gap: Gap, out_dir: Path, n: int = 3) -> list[Path]:
    out_dir.mkdir(parents=True, exist_ok=True)
    paths = []
    step = gap.duration / (n + 1)
    for i in range(1, n + 1):
        ts = gap.start + (step * i)
        out = out_dir / f"{gap.start:.2f}_{i}.jpg"
        subprocess.run(
            ["ffmpeg", "-y", "-ss", f"{ts:.3f}", "-i", video,
             "-frames:v", "1", "-q:v", "3", str(out)],
            capture_output=True, check=True,
        )
        paths.append(out)
    return paths

-ss before -i seeks fast by jumping to the nearest keyframe. For a few frames of context that's fine; if you need frame-exact sampling, put -ss after -i and accept it'll be much slower.

4. Generate to the budget, not to a quality bar

Whatever model you use, the budget is the part of the prompt that matters. Put it in hard and put it first:

# src/describe.py
SYSTEM = """You write audio description for blind and low-vision viewers.

HARD CONSTRAINT: your output MUST be {max_words} words or fewer. This is not a
style preference. The narration plays in a {duration:.1f} second gap between
lines of dialogue and will be cut off if it runs long.

Rules:
- Describe only what a sighted viewer gains that the audio does not convey.
- Prioritise: character actions > scene changes > appearance > atmosphere.
- Use present tense. No "we see", no "the camera shows", no interpretation
  of emotion or intent.
- If nothing visually important happens here, output exactly: SKIP
"""

def build_prompt(gap, max_words, prior_descriptions):
    return SYSTEM.format(max_words=max_words, duration=gap.duration), {
        # Prior descriptions let the model stop re-introducing the same character.
        "already_described": prior_descriptions[-5:],
    }

Two details matter more than the model choice. SKIP must be a valid output, because a pipeline that fills all 31 gaps just because it can produces narration that's exhausting to listen to. And pass prior descriptions, or you get "a man in a blue coat" in gap 3 and again in gap 28. Models are weak on narrative continuity; carrying context is the cheap partial fix.

5. Render, then verify the duration 🎙️

This is the section the whole tutorial exists for. Do not trust the word budget. Render the audio and measure the actual file:

# src/render.py
import subprocess
from pathlib import Path

def audio_duration(path: Path) -> float:
    out = subprocess.run(
        ["ffprobe", "-v", "error", "-show_entries", "format=duration",
         "-of", "default=noprint_wrappers=1:nokey=1", str(path)],
        capture_output=True, text=True, check=True,
    )
    return float(out.stdout.strip())

def render_fitting(text: str, gap, out: Path, tts_fn, max_attempts: int = 3):
    """Render TTS and shrink the text until it fits the gap."""
    budget = gap.duration - 0.30  # edge padding
    current = text
    for attempt in range(max_attempts):
        tts_fn(current, out)
        actual = audio_duration(out)
        if actual <= budget:
            return {"ok": True, "text": current, "duration": actual, "attempts": attempt + 1}

        overrun = actual - budget
        keep = int(len(current.split()) * (budget / actual) * 0.9)
        print(f"  overrun {overrun:.2f}s, trimming to ~{keep} words")
        current = " ".join(current.split()[:keep])

    # Speeding up narration to force a fit makes it harder to understand.
    # Flag for human review instead.
    return {"ok": False, "text": current, "duration": audio_duration(out), "reason": "could not fit"}

python3 -m src.pipeline video.mp4

gap 03.26-06.91 (3.65s, budget 8 words)
  overrun 0.41s, trimming to ~7 words
  fit in 2 attempts: "She slides the envelope under the door."
gap 19.04-21.44 (2.40s, budget 5 words)
  fit in 1 attempt: "He notices the broken lock."
gap 44.10-45.80 (1.70s, budget 3 words)
  SKIP (nothing visually significant)
gap 61.22-63.02 (1.80s, budget 4 words)
  overrun 0.62s, trimming to ~3 words
  overrun 0.28s, trimming to ~2 words
  FLAGGED: could not fit -> review queue

24 described, 5 skipped, 2 flagged for review

Those 2 flagged items are the output you want. A pipeline that silently truncates mid-word, or time-compresses narration until it's unintelligible, produces a track that technically exists and nobody can use.

6. Mix and package as a second audio track 🔊

Build a narration bed, then overlay it on the original:

# place each clip at its gap start on a silent bed the length of the video
ffmpeg -f lavfi -t 1800 -i anullsrc=r=48000:cl=stereo \
  -i ad_003.wav -i ad_019.wav \
  -filter_complex "\
    [1:a]adelay=3414|3414[a1]; \
    [2:a]adelay=19190|19190[a2]; \
    [0:a][a1][a2]amix=inputs=3:duration=first:normalize=0[bed]" \
  -map "[bed]" narration_bed.wav

# duck the original under the narration, then mux as a described track
ffmpeg -i video.mp4 -i narration_bed.wav \
  -filter_complex "[0:a][1:a]sidechaincompress=threshold=0.05:ratio=8[ducked]; \
                   [ducked][1:a]amix=inputs=2:duration=first[out]" \
  -map 0:v -map "[out]" -c:v copy -c:a aac -b:a 128k \
  -metadata:s:a:0 title="Audio Description" \
  -metadata:s:a:0 language=eng \
  video_described.mp4

sidechaincompress ducks the original audio whenever narration plays, which is what a professional mix does. Without it the narration competes with the score.

Then it's another rendition in your HLS audio group:

#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aud",NAME="English",LANGUAGE="en",DEFAULT=YES,AUTOSELECT=YES,URI="audio/en.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aud",NAME="English (Described)",LANGUAGE="en",CHARACTERISTICS="public.accessibility.describes-video",AUTOSELECT=NO,URI="audio/en-ad.m3u8"

⚠️ CHARACTERISTICS="public.accessibility.describes-video" is what tells players this is a description track rather than a second language. Without it, players won't surface it in their accessibility menu and iOS won't auto-select it for users who've enabled Audio Descriptions system-wide. Easy to omit, invisible when wrong.

7. Monitor it, or you'll lose it 📡

This is the operational failure nobody plans for. Nobody on your team uses the description track. Your smoke tests check that video plays. A packager change drops the second rendition from the manifest and you find out eighteen months later from a complaint.

# src/monitor.py
import m3u8  # pip install m3u8

DESCRIBES = "public.accessibility.describes-video"

def has_description_track(master_url: str) -> bool:
    playlist = m3u8.load(master_url)
    return any(
        m.type == "AUDIO" and DESCRIBES in (m.characteristics or "")
        for m in playlist.media
    )

def assert_described(master_url: str):
    if not has_description_track(master_url):
        raise AssertionError(f"description track missing from {master_url}")

Wire that into whatever already checks your manifests. An accessibility feature you aren't monitoring is an accessibility feature you used to have.

What's next

• A human review queue is not optional. Automated description is production-viable across many content types with human review, and not reliable enough without it on complex narrative material. Build the queue as a first-class part of the pipeline.
• Extended audio description (WCAG 1.2.7). For gaps that were too short, the compliant answer is a player that pauses, which is a different project. W3C's understanding doc starts there.
• Test with actual screen reader users. Every metric here is a proxy for "is this useful", and proxies drift.

The regulatory deadlines are real (ADA Title II landed April 24, 2026 for larger public entities; the EU Accessibility Act has applied since June 2025), but the better reason to build this is that the pipeline above turns a specialist hour per video into a few minutes of review, and that's the difference between describing your top 50 videos and describing all of them. #a11y #video