DEV Community

Mason K
Mason K

Posted on

FFmpeg 8.0 in practice: transcribe with the Whisper filter and probe av1_vulkan

#ffmpeg #video #tutorial #devops

TL;DR

FFmpeg 8.0 "Huffman" adds a native whisper audio filter (transcription is now a filter, not a separate service) and av1_vulkan, a cross-vendor GPU AV1 encoder. We'll generate SRT + JSON captions in one command, then write a script that probes for av1_vulkan and falls back to SVT-AV1 when the GPU or driver isn't ready.

📦 Code: github.com/USER/ffmpeg8-pipeline-demo (replace before publishing)

FFmpeg 8.0 shipped in late August 2025. Two features change what your pipeline can offload to FFmpeg itself: speech-to-text and GPU-vendor-agnostic AV1 encoding. Let's actually run both.

1. Confirm you're on 8.0+ 

ffmpeg -version | head -n 1
# ffmpeg version 8.0.1 Copyright (c) 2000-2025 the FFmpeg developers
Enter fullscreen mode Exit fullscreen mode

⚠️ Note: If your distro still ships 7.x, grab a static build or the updated PPA. The whisper filter and av1_vulkan encoder do not exist before 8.0. 8.0.1 is a safer pin than the initial 8.0 tag.

Check that the new bits are compiled in:

# Is the whisper filter present?
ffmpeg -hide_banner -filters | grep whisper
#  ... whisper           A->A       Transcribe audio using whisper.cpp.

# Is the Vulkan AV1 encoder present?
ffmpeg -hide_banner -encoders | grep av1_vulkan
#  V....D av1_vulkan           AV1 (Vulkan)
Enter fullscreen mode Exit fullscreen mode

If av1_vulkan is missing, your build was compiled without Vulkan encode support. That's fine, we handle it in step 4.

2. Transcribe a video with the whisper filter 🎙️

The filter runs OpenAI's Whisper models through whisper.cpp, inside the filter graph. First grab a model:

# One-time: download a whisper.cpp GGML model
mkdir -p models
curl -L -o models/ggml-base.en.bin \
  https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin
Enter fullscreen mode Exit fullscreen mode

Now generate an SRT in a single command. We drop video (-vn) because the filter only needs audio:

ffmpeg -i input.mp4 -vn -af \
  "whisper=model=models/ggml-base.en.bin:language=en:queue=10:destination=captions.srt:format=srt" \
  -f null -
Enter fullscreen mode Exit fullscreen mode 
# terminal output (trimmed)
[Parsed_whisper_0 @ 0x...] Loading model models/ggml-base.en.bin
[Parsed_whisper_0 @ 0x...] whisper backend: CPU
size=N/A time=00:03:41.00 bitrate=N/A speed=2.3x
Enter fullscreen mode Exit fullscreen mode

You now have captions.srt. Want structured output for an index or search instead? Switch the format and send it to stdout:

ffmpeg -loglevel warning -i input.mp4 -vn -af \
  "whisper=model=models/ggml-base.en.bin:language=en:queue=10:destination=-:format=json" \
  -f null - > transcript.json
Enter fullscreen mode Exit fullscreen mode

The one parameter that matters: queue

queue is how many seconds of audio buffer before a transcription pass runs.

queue value behavior use it for
small (2-3s) frequent output, lower accuracy, higher CPU near-live captioning
large (10-20s) more context, higher accuracy, less CPU batch captioning of uploads

💡 Tip: For an upload pipeline, go large. Whisper is more accurate with surrounding context, and you don't need sub-second latency on a file someone uploaded thirty seconds ago.

There's also use_gpu (default true) and language (set it explicitly; autodetect costs you a pass). This one command replaces the classic "extract audio, hand a WAV to a Python Whisper worker, stitch the SRT back" setup. One less service to run.

3. Try the av1_vulkan encoder

av1_vulkan uses Vulkan's video-encode extensions, so it runs across GPU vendors instead of locking you to NVENC (NVIDIA) or QSV (Intel). You need a Vulkan device and a driver that implements the encode extensions:

# Do you have a usable Vulkan device?
ffmpeg -hide_banner -init_hw_device vulkan=vk:0 -f lavfi -i nullsrc -t 0 -f null - 2>&1 | tail -n 3
Enter fullscreen mode Exit fullscreen mode

A real encode uploads frames to the GPU with hwupload:

ffmpeg -i input.mp4 \
  -init_hw_device vulkan=vk:0 -filter_hw_device vk \
  -vf "format=nv12,hwupload" \
  -c:v av1_vulkan -b:v 4M \
  -c:a copy output_av1.mp4
Enter fullscreen mode Exit fullscreen mode

When the hardware isn't ready, you'll see something like:

[av1_vulkan @ 0x...] Encoding of this format is not supported.
Error initializing output stream 0:0 -- Error while opening encoder
Enter fullscreen mode Exit fullscreen mode

That error is the whole reason for the next step.

4. A safe wrapper: probe, then fall back to SVT-AV1 🛠️

Don't put av1_vulkan directly in production. Detect it, verify it actually runs, and fall back to CPU SVT-AV1 (the sober choice for production AV1 today) when it doesn't.

#!/usr/bin/env bash
# encode-av1.sh: prefer GPU av1_vulkan, fall back to SVT-AV1
set -euo pipefail

INPUT="$1"
OUTPUT="$2"
BITRATE="${3:-4M}"

have_vulkan_av1() {
  ffmpeg -hide_banner -encoders 2>/dev/null | grep -q 'av1_vulkan' || return 1
  # Dry-run one frame to confirm the driver really supports encode
  ffmpeg -hide_banner -loglevel error \
    -init_hw_device vulkan=vk:0 -filter_hw_device vk \
    -f lavfi -i testsrc=size=256x256:rate=1 -frames:v 1 \
    -vf "format=nv12,hwupload" -c:v av1_vulkan -f null - 2>/dev/null
}

if have_vulkan_av1; then
  echo "-> using av1_vulkan (GPU)"
  ffmpeg -y -i "$INPUT" \
    -init_hw_device vulkan=vk:0 -filter_hw_device vk \
    -vf "format=nv12,hwupload" \
    -c:v av1_vulkan -b:v "$BITRATE" -c:a copy "$OUTPUT"
else
  echo "-> av1_vulkan unavailable, falling back to SVT-AV1 (CPU)"
  ffmpeg -y -i "$INPUT" \
    -c:v libsvtav1 -preset 6 -crf 30 -c:a copy "$OUTPUT"
fi
Enter fullscreen mode Exit fullscreen mode 
chmod +x encode-av1.sh
./encode-av1.sh input.mp4 out.mp4 4M
# -> av1_vulkan unavailable, falling back to SVT-AV1 (CPU)
Enter fullscreen mode Exit fullscreen mode

The dry-run in have_vulkan_av1 matters: the encoder being listed doesn't mean your driver implements the encode queue. Actually encoding one frame is the only honest check.

⚠️ Note: av1_vulkan is early. Even where it runs, don't assume it matches SVT-AV1 or a vendor AV1 encoder on quality-per-bit yet. Prototype and measure; don't rebuild your ladder on it this quarter.

What's next

  • Wire the whisper JSON output into a search index or a chapter generator (segment timestamps come along in the JSON).
  • Benchmark av1_vulkan vs libsvtav1 on files that look like your catalog, not a clean test clip, before trusting either.
  • Look at the other 8.0 additions if they touch you: VVC via VA-API, native APV and ProRes RAW decoders, and AVX-512 CPU optimizations.
  • Pin 8.0.1 (or newer point release) in your Dockerfile rather than the 8.0 tag.

The theme across both features is the same: work that used to need a second service bolted onto FFmpeg now lives inside FFmpeg. Shorter pipeline, fewer 3 AM pages. That's the upgrade worth doing.

Top comments (0)