Feed audio to Whisper in Rust without ever shelling out to ffmpeg

ez-ffmpeg is a Rust crate that runs FFmpeg pipelines inside your process — linked libav libraries behind a high-level API, no subprocess (and no relation to the JavaScript "Ez FFmpeg" project). The narrow claim of this post: its 0.14 SampleExtractor turns a video file into the 16 kHz mono f32 PCM that Whisper-family models consume, in one call, verified end-to-end against whisper-rs 0.16 on a real MP4 — full program, real timings, and the verbatim transcript below. It's on crates.io and GitHub.

Here's the itch. Speech-to-text in Rust is in decent shape — whisper-rs is maintained, candle and ort keep growing — yet if your input is an ordinary MP4 rather than 16 kHz mono PCM, nearly every setup guide in the ML-audio ecosystem starts with the same shell line:

ffmpeg -i in.mp4 -ar 16000 -ac 1 -c:a pcm_s16le out.wav

Step one of your "pure Rust" transcription pipeline is to leave Rust. Even "FFmpeg 8.0 adds Whisper support" making the HN front page doesn't change that for you: that news serves CLI users, while a Rust program wants decoded samples in its own memory, not one more command to fork.

The pipeline everyone actually ships

Whisper's input contract is fixed: 16 kHz, mono, f32 samples normalized to [-1, 1] — that's the &[f32] in whisper-rs's full(). Your MP4 most likely holds 44.1 or 48 kHz stereo AAC. So the standard recipe forks ffmpeg and cleans up afterwards:

// The "leave Rust first" pipeline: subprocess + temp file + wav parsing.
let status = std::process::Command::new("ffmpeg")
    .args(["-i", "in.mp4", "-ar", "16000", "-ac", "1",
           "-c:a", "pcm_s16le", "-y", "out.wav"])
    .status()?; // hope ffmpeg is on PATH, hope the flags are right
// ...then open out.wav, skip the header, convert i16 -> f32 by hand

It works, and it taxes you four ways. Deployment: every target machine and container image needs an ffmpeg executable, and its version is now your problem. Errors: a failed subprocess hands you an exit code and a pile of stderr text — distinguishing "file not found" from "no audio stream" means regexing log lines instead of matching on a typed error. Files: a temporary wav has to be written, named, and cleaned up, and concurrent jobs must not collide. Format: pcm_s16le comes back as i16, so you divide by 32768 yourself. Crates like ffmpeg-sidecar wrap this subprocess dance neatly, but the first three taxes are inherent to the model: there's still an external binary at runtime.

One call, in-process

in.mp4 ─▶ demux ─▶ decode(AAC) ─▶ resample(16 kHz mono f32) ─▶ Vec<f32>
          └────── SampleExtractor::for_whisper(), one call ──────┘
Vec<f32> ─▶ whisper-rs full() ─▶ transcript   (no subprocess, no temp file)

SampleExtractor collapses demux → decode → resample → downmix into a single in-process call — internally it's a normal FFmpeg pipeline with a sink on the end that hands the samples back to you. for_whisper presets exactly the contract above:

let pcm: Vec<f32> = SampleExtractor::for_whisper("in.mp4").collect_samples()?;

Two honest caveats before the code. First, installing and linking FFmpeg is not solved here — the crate sits on the libav libraries, so FFmpeg 7.1–8.x must be present, and that pain is unchanged. What goes away is the subprocess, the temp file, and the stderr scraping — not the FFmpeg dependency itself. Second, the frame/sample export API is marked experimental in the README and may be reshaped in a future minor release.

The complete program, two phases — extract PCM, feed whisper-rs 0.16:

[package]
name = "mp4-to-text"
version = "0.1.0"
edition = "2021"

[dependencies]
ez-ffmpeg = "0.14"    # needs FFmpeg 7.1-8.x installed (links libav)
whisper-rs = "0.16"   # needs cmake + C/C++ toolchain (builds whisper.cpp)

use std::time::Instant;

use ez_ffmpeg::frame_export::SampleExtractor;
use whisper_rs::{FullParams, SamplingStrategy, WhisperContext, WhisperContextParameters};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Phase 1: decode the video's audio track into 16 kHz mono f32 PCM.
    let t_extract = Instant::now();
    let pcm: Vec<f32> = SampleExtractor::for_whisper("jfk.mp4").collect_samples()?;
    let extract_ms = t_extract.elapsed().as_secs_f64() * 1000.0;

    println!("pcm samples : {}", pcm.len());
    println!(
        "pcm duration: {:.3} s (len / 16000.0, mono)",
        pcm.len() as f64 / 16_000.0
    );
    println!("extract time: {extract_ms:.1} ms");

    // Phase 2: transcribe the PCM with whisper.cpp via whisper-rs.
    let t_whisper = Instant::now();
    let ctx = WhisperContext::new_with_params(
        "ggml-tiny.en.bin",
        WhisperContextParameters::default(),
    )?;
    let mut state = ctx.create_state()?;

    let mut params = FullParams::new(SamplingStrategy::Greedy { best_of: 1 });
    params.set_language(Some("en"));
    params.set_print_special(false);
    params.set_print_progress(false);
    params.set_print_realtime(false);
    params.set_print_timestamps(false);

    state.full(params, &pcm)?;

    let mut transcript = String::new();
    let n_segments = state.full_n_segments();
    for i in 0..n_segments {
        if let Some(segment) = state.get_segment(i) {
            transcript.push_str(&segment.to_str_lossy()?);
        }
    }
    let whisper_ms = t_whisper.elapsed().as_secs_f64() * 1000.0;

    println!("whisper time: {whisper_ms:.1} ms");
    println!("transcript  : {}", transcript.trim());
    Ok(())
}

Call by call:

• SampleExtractor::for_whisper("jfk.mp4") — a thin preset, equivalent to SampleExtractor::new(...).sample_rate(16000).channels(Mono). It is worth pointing out that the base builder's defaults preserve the source: source sample rate and channel layout pass through untouched, and only the sample format is pinned to packed f32. Resampling and downmixing are opt-in, so music and analysis users don't get silently converted to 16 kHz; an ASR preset is where that normalization belongs, explicitly.
• .collect_samples()? — runs the pipeline and returns one flat interleaved Vec<f32>. Mono means samples == frames, so len() / 16000.0 is the duration — the program prints exactly that. The whole track sits in memory, so for hours-long input switch the same builder to .samples()?: a streaming iterator that yields chunks as they decode, carries rate/channels/timestamp metadata on each chunk, keeps a small bounded number in flight (default 4), and aborts the run cleanly if you drop it early. Errors have defined landing spots too: option mistakes and "no audio stream" surface when you start the run, mid-decode failures arrive as the iterator's terminal error — typed, not scraped from stderr. start_time_us / duration_us carve out a time window when you chunk long audio for the model.
• WhisperContext::new_with_params + create_state — loads the ggml model from disk and allocates inference state; most of the "whisper time" below lives here and in full.
• FullParams::new(SamplingStrategy::Greedy { best_of: 1 }) — the cheapest decoding strategy; set_language(Some("en")) matches the English-only tiny.en model.
• The five set_print_*(false) calls — whisper.cpp prints transcription progress to stderr by default, so an embedded library behaves like a chatty CLI; these turn that off. Model loading logs another ~30 lines through a separate path; whisper_rs::install_logging_hooks silences those.
• The segment loop — full(), full_n_segments(), get_segment(i), to_str_lossy() — is the whisper-rs 0.16 API. It looks unremarkable, and it is exactly where most published examples stop compiling. That deserves its own section.

The trap nobody documents: whisper-rs 0.16 API drift

The ez-ffmpeg half of this program gave me nothing to write about: the documented one-liner compiled and ran correctly on the first try. Every wall I hit was on the whisper-rs half — starting with my first draft, which followed an older tutorial and failed to compile: one method not found, one ? where ? no longer applies. Checking the current docs turned up three changes that most pre-0.16 blog posts trip over:

1. full() returns Result<(), _> — apply ? and move on; don't expect a number back.
2. full_n_segments() returns a plain i32 — a trailing ? is now a compile error.
3. full_get_segment_text() is gone — use get_segment(i), which returns Option<WhisperSegment>, then to_str() / to_str_lossy() for the text.

The segment loop in the listing above is the current, correct form. The general lesson I took: for a fast-moving crate, trust the method signatures on docs.rs for the exact version in your lockfile, and treat blog posts as intent, not syntax. One more orientation note: whisper-rs's GitHub repository is archived, but the project is not dead — it moved to Codeberg and is maintained there. Most of the tutorials a search engine surfaces predate these changes. (A caveat on my own account: I reconstructed this API history from a single migration, so treat it as one data point, not a changelog.)

Run it

Test input: an 11-second clip of JFK's "ask not" line packed into an MP4 (h264 video + AAC audio track). Model: ggml-tiny.en.bin, 77,704,715 bytes. Two consecutive runs produced identical sample counts and identical transcripts; here is the second run's output, verbatim:

pcm samples : 176128
pcm duration: 11.008 s (len / 16000.0, mono)
extract time: 7.8 ms
whisper time: 635.6 ms
transcript  : And so my fellow Americans ask not what your country can do for you ask what you can do for your country.

The transcript is correct, whole sentence, first pass. The numbers reconcile too: 176,128 ÷ 16,000 = 11.008 s. The source wav was 11.00 s; the extra ≈8 ms is AAC encoder priming/padding introduced when the clip was remuxed — alignment samples the codec pads at the edges of the track, entirely normal. Which is why duration checks against extracted PCM should assert ≈, never == — an exact-equality assertion dies on those 8 ms.

Timings, with the full caveat attached: one machine, an 8-core/16-thread Ryzen 9 5900HX, CPU-only, two consecutive runs. This is a smoke test, not a benchmark.

Phase	Run 1 (cold)	Run 2 (warm)
PCM extraction (SampleExtractor)	16.3 ms	7.8 ms
Whisper phase (incl. loading the 77 MB model from disk)	1463.9 ms	635.6 ms

Two readings. The cold/warm gap in the whisper row is mostly the 77 MB model file being in the OS cache the second time — don't read it as inference getting faster. And for this 11-second clip, extraction is single-digit-to-low-double-digit milliseconds while the model dominates — the PCM step is no longer the thing worth optimizing in this pipeline. Batch throughput and long-file behavior weren't measured here, so no numbers are claimed for them.

When you should not use this

Pure Rust can pull the audio track out of an MP4/MKV and resample it to 16 kHz (symphonia + rubato; symphonium does both in one call; mutter feeds whisper-rs in five lines). But "any video, any codec, one line" hits a codec wall: symphonia still lists HE-AAC and Opus as unfinished and has no AC-3 at all — which is exactly where an FFmpeg backend remains irreplaceable.

So the split is by scenario, and each path has its win. One file, one time: type the ffmpeg command in a terminal; writing a Rust program for a run-once job is a detour. Inputs you control (WAV, FLAC, ordinary AAC) and a zero-C-dependency build as a goal: symphonia + rubato is the right road, and linking FFmpeg buys you nothing. Inputs you don't control — user uploads in whatever container and codec they arrive in — FFmpeg's codec coverage is the point, and that is the case for an FFmpeg-backed crate like ez-ffmpeg. And keep one asterisk in view: even with a pure-Rust audio path, whisper-rs itself compiles whisper.cpp, so "no C toolchain anywhere" was never on the table for this pipeline.

What the first build costs

Honesty about the build, since the Cargo.toml comments above compress it: whisper-rs builds whisper.cpp from source, which needs cmake and a C/C++ toolchain, and the first build takes minutes — it is not hung. On PEP 668 distros (Ubuntu 24.04 and later), pip install --user cmake is refused by the system Python; install cmake inside a venv or just use your distro's cmake package. The ez-ffmpeg side needs FFmpeg 7.1–8.x present for linking, as covered earlier — that prerequisite predates this crate and survives it.

Where this leaves you

Your transcription pipeline now has one subprocess, one temp file, and one hand-rolled i16→f32 conversion less, and its failures arrive as typed Results. From here: switch to samples() plus start_time_us/duration_us to stream and chunk long recordings, or hand the same Vec<f32> to other audio models via candle or ort — 16 kHz mono f32 is the common handshake shape.

The code lives at github.com/YeautyYE/ez-ffmpeg; examples/extract_whisper_pcm in the repository runs as-is.