DEV Community: Yeauty YE

The state of media processing in Rust (2026): what each crate actually covers

Yeauty YE — Tue, 28 Jul 2026 19:54:48 +0000

Every few months someone asks how to decode video or pull audio samples in Rust, and the thread converges on the same advice: just run the ffmpeg binary. This post is a map of why — six paths through the ecosystem as of July 2026, what each one actually covers, how it is maintained, and where each one honestly wins. All numbers are monthly downloads or GitHub stars as verified in July 2026; where I could not verify a number, the table shows a dash instead of a guess.

A note on method: all six paths get the same treatment — where it wins, the honest limitation, a one-line verdict — and the limitation column does not soften for whichever path is newest. One disambiguation up front: the Rust crate ez-ffmpeg discussed below is unrelated to the JavaScript "Ez FFmpeg" that once made the HN front page; same name, different thing.
One more ground rule: there are no benchmarks in this piece — runnable benchmarks later beat adjectives now.

The landscape at a glance

Path	Execution model	Coverage highlights (verified)	Key limitations (verified)	Data point (July 2026)
`std::process::Command` + ffmpeg CLI	subprocess	everything the CLI can do; every recipe on the internet applies as-is	ship the binary; data re-enters your process via stdout/temp files	FFmpeg 8.0 is out; its Whisper support hit the HN front page (1033 points)
ffmpeg-sidecar	subprocess	iterate any video "as if it were an array of raw RGB frames"; no libav linking	still a child process; an ffmpeg executable must be present at runtime	~131K downloads/month
ffmpeg-next	in-process (low-level bindings)	the full libav control plane; the ecosystem's foundation	"maintenance-only mode"; hand-written decode loops (~40 lines before the first frame); third fork in a lineage	1.28M downloads/month
rsmpeg	in-process (safe-wrapper style)	tracks FFmpeg 6/7/8 (0.18.0 added 8.0)	slow cadence: roughly 11 months without major movement	877 stars
video-rs	in-process (high-level)	RGB ndarrays in ten lines; README encodes rainbow.mp4 from ndarray frames; `seek_to_frame`	no audio API; self-described "still a work-in-progress… some parts not flushed out", "Use with caution"; successor rave "not ready for use yet"	~32K downloads/month
symphonia + rubato	in-process (pure Rust)	demux + decode audio, resample to 16 kHz; symphonium does it in one call; mutter feeds whisper-rs in five lines	HE-AAC and Opus listed unfinished; no AC-3; video out of scope	symphonia ~1.07M downloads/month, 0.6.0 (May 2026); whisper-rs ~170K/month (maintained on Codeberg)
ac-ffmpeg	in-process	touches all three domains (video, audio, encoding)	stays at the packet-loop level; no release in 14 months	—
ez-ffmpeg	in-process (runtime)	API and internal scheduling modeled 1:1 on the FFmpeg CLI — command-line knowledge transfers as-is; 0.15 runs pasted ffmpeg commands in-process (`cli` facade, verified subset); FrameExtractor / SampleExtractor / VideoWriter as first-class APIs; color-tag-correct YUV→RGB by default	still links libav — the build pain stays; new APIs marked experimental; early-stage	~340 stars

A dash means I could not verify it, and nothing in this article depends on it.

One boundary note: gstreamer-rs is a real option too — ~564K downloads a month and actively maintained — but it is neither FFmpeg-based nor a few-lines-of-code style of API, so it sits outside this FFmpeg-centered map.

Path 1: `std::process::Command` + the ffmpeg binary

Where it wins. Thirty years of FFmpeg knowledge exists in command-line form, and all of it applies unchanged. The CLI cookbook culture is alive and huge — "FFmpeg by Example" alone scored 920 points on HN. No FFI, no linking, no version matrix. On rustcc, a Chinese Rust forum, a thread on these bindings landed on shelling out as "the most reliable option, with far fewer pitfalls" (translation mine) — not a joke, but the rational endpoint of a lot of scar tissue.

The honest limitation. You ship an ffmpeg binary with your app, and the moment data has to come back into your process — frames, PCM samples — you are parsing stdout or juggling temp files. jaredly's issue from 2015 describes the pattern exactly: "rendering to images then using ffmpeg on the cli". A decade later, plenty of people still work that way. Not because it is good — because the alternatives were never paved.

Verdict: when the output is a file and the recipe exists, still the right answer. When data must land in memory, the tax meter starts.

Path 2: ffmpeg-sidecar — the subprocess, made livable

Where it wins. ~131K downloads a month, and a self-description that gets straight to the point: treat any video "as if it were an array of raw RGB frames". It turns shelling out into a structured API — you write an iterator, not a stdout parser. And because it talks to ffmpeg over stdin/stdout, it never links libav: the entire build/link/platform circus does not apply to you. If you want frames without touching FFI, this is the most comfortable path in the list.

The honest limitation. It is still a child process. An ffmpeg executable has to exist wherever you deploy, and every frame crosses a process boundary. In-process filter graphs and fine-grained control are outside this model. Whether the boundary matters for your workload is a measurement, not an opinion — this survey quotes no numbers for it.

Verdict: if a child process is acceptable, the most comfortable way to get frames.

Path 3: the raw bindings — ffmpeg-next and rsmpeg

Where it wins. When you need the full libav control plane — custom IO, obscure containers, packet-level surgery — nothing else covers that surface. ffmpeg-next sits at 1.28M downloads a month; ez-ffmpeg itself links libav through it. That number is the measure of how much of the ecosystem stands on this layer.

The honest limitation. Start with maintenance: ffmpeg-next is frozen by policy, not abandoned — the maintainer's own words are "maintenance-only mode for the most part… Any PR to improve existing API is unlikely to be merged" — and the crate sits in a fork lineage already three generations long: ffmpeg → ffmpeg-next → ffmpeg-the-third. Then the daily cost: decoding a frame means hand-writing the send_packet/receive_frame ceremony, roughly forty lines before you see pixels, and even "transcode audio and video together" can send you searching — one developer's verdict after going through the examples: "Examples… do not include transcoding of audio+video". When kornel wrote "avoid ffmpeg… all incomplete and poorly maintained", this category's overall state is what he was describing.

One counterintuitive fact about this layer: even a wrapper sold on safety has shipped public APIs annotated "might trigger undefined behavior" (zmwangx/rust-ffmpeg#225). A safe skin over an FFI skeleton is an engineering promise, not a proof.

On rsmpeg specifically: 877 stars, hosted under the larksuite organization, supporting FFmpeg 6, 7 and 8 (0.18.0 added 8.0). The cadence is slow — roughly 11 months without major movement — but calling it dormant overstates it. If you need a low-level wrapper that tracks current FFmpeg majors, it is the fresher one.

Verdict: as a foundation, indispensable; as a daily API, you pay full price.

Path 4: video-rs — high-level video, self-described work in progress

Where it wins. For years this has been the one serious high-level option (~32K downloads/month). PyAV-style decode/encode has been there since 2023: RGB ndarrays within ten lines, a README whose example encodes rainbow.mp4 from ndarray frames, and a seek_to_frame (frame accuracy undeclared). If you want a PyAV-like video experience in Rust today, this is the most convenient existing answer.

The honest limitation. The limitations are written in its own README: "still a work-in-progress… some parts not flushed out", "Use with caution". There is no audio API. The official successor, rave, aims to move off FFmpeg entirely — and is "not ready for use yet".

Verdict: strong on video, and video only; no audio plus self-declared WIP keeps it a specialist.

Path 5: symphonia + rubato — pure Rust until the codec wall

Where it wins. Pure Rust can pull the audio track out of an MP4/MKV and resample it to 16 kHz (symphonia + rubato; symphonium does both in one call; mutter feeds whisper-rs in five lines). The decode-and-resample chain is Rust top to bottom — of the six paths here, the only one that involves no C artifact at any point: no libav to link, no ffmpeg binary to ship. If a clean dependency tree is the goal, this path owns it. One correction to a rumor while I am here: whisper-rs is archived on GitHub but actively maintained on Codeberg, at ~170K downloads a month — do not file it under dead.

The honest limitation. "Any video, any codec, one line" hits a codec wall: symphonia still lists HE-AAC and Opus as unfinished and has no AC-3 at all — which is exactly where an FFmpeg backend remains irreplaceable. And video is out of scope by design.

Verdict: inside its codec table, the cleanest dependency story on this list; outside it, you are back to FFmpeg.

Path 6: ez-ffmpeg — the in-process runtime

The project's stated aim: "the actively-developed high-level in-process FFmpeg runtime for Rust."

The core is not any single feature — it is the ergonomics of moving an FFmpeg command into Rust unchanged. One builder chain reads like one command:

FfmpegContext::builder()
    .input("input.mkv")
    .output(
        Output::from("output.mp4")
            .set_video_codec("libx264")            // -c:v libx264
            .set_video_codec_opt("crf", "23")      // -crf 23
            .set_video_codec_opt("preset", "fast") // -preset fast
            .set_audio_codec("aac"),               // -c:a aac
    )
    .build()?
    .start()?
    .wait()?;

Migration is mechanical, because nothing has to be relearned: seconds become microseconds (-ss 10 → set_start_time_us(10_000_000)); option names work verbatim minus the leading dash (crf, preset, movflags go through set_*_opt straight into FFmpeg's AVOption system — whatever name the CLI accepts, this accepts); the string after -vf drops into filter_desc character for character; stream selection uses the CLI's own specifier syntax (0:a:0). Years of command-line muscle memory stay valid, line for line.

As of 0.15 even the translation step is optional — the command can be pasted in as-is. With the cli feature enabled, from_cli takes a whole ffmpeg command string, parses it, builds the pipeline, and runs it inside your own process — no subprocess:

use ez_ffmpeg::cli::from_cli;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    from_cli("ffmpeg -i input.mp4 -c:v libx264 -crf 28 -preset veryfast -c:a aac -y output.mp4")?
        .start()?
        .wait()?;
    Ok(())
}

The attitude behind it is the part worth noting. Commands sort into three classes: verified shapes may run (six today, each reconciled against the real ffmpeg CLI by a semantic golden suite); unverified ones are translated but never executed; unrecognized ones are rejected on the spot, anchored to the offending token. A sibling function, emit_rust_code, turns the same command into a compile-ready builder program you can extend and maintain. It would rather refuse you clearly than "run, but come out subtly different from ffmpeg."

The isomorphism does not stop at the API surface — it runs down to the source. The internal scheduler is modeled on the ffmpeg CLI's own architecture: fftools' ffmpeg_demux.c / ffmpeg_dec.c / ffmpeg_filter.c / ffmpeg_enc.c / ffmpeg_mux.c / ffmpeg_sched.c / sync_queue.c map file for file to demux_task / dec_task / filter_task / enc_task / mux_task / ffmpeg_scheduler / sync_queue. Two practical consequences: when behavior has to match the CLI, there is a line-by-line reference to diff against — a divergence can be traced to its fftools counterpart; and if you have ever wanted to understand FFmpeg's internals, this codebase reads as a typed, ownership-annotated companion to fftools — a far gentler slope than the C.

On the feature side: history first, then the new parts. Even in 0.13, ez-ffmpeg let you intercept and rewrite every frame via FrameFilter — the repository ships custom tile/volume filter examples. 0.14 adds the dedicated FrameExtractor / SampleExtractor / VideoWriter — turning "possible" into "one line", with color-tag-correct YUV→RGB by default. Concretely: FrameExtractor pulls decoded frames with sampling policies (EveryNth / UniformN / Keyframes); SampleExtractor hands you 16 kHz mono f32 ready for whisper-rs; VideoWriter pushes frames you rendered yourself into an encoding pipeline (write / write_owned), with filter-graph validation; its output surface covers files and RTMP live targets (the live-target claim is API-surface and module-doc backed, not exercised against a live server here). The color detail deserves its own sentence: HD video converts to RGB per BT.709 by default, whereas to_ndarray('rgb24') in older PyAV releases applied BT.601 across the board, which skews saturated colors.

Who should not use ez-ffmpeg

If linking FFmpeg is your blocker, ez-ffmpeg does not remove it. It links libav through ffmpeg-next — the very layer criticized above, whose frozen maintenance and fork-lineage risk it inherits wholesale. The developer who wrote "I've spent the last 12 hours… willing to spend another week" fighting the build would fight the same build here. What it removes is the hundred-line pipeline after linking succeeds — not the linking.
The three new APIs are marked experimental in the README. That word is there on purpose. The cli facade has boundaries of its own: it is an optional feature you must enable, execution currently requires an FFmpeg 7.1 runtime (anything else fails before any I/O), and the subset is deliberately narrow — only those six verified shapes run; two-pass encoding, complex filtergraphs and hardware acceleration are outside it.
It is early-stage: ~340 stars, and effectively one maintainer — bus factor is a real risk. A safe Rust API surface is not the same thing as freedom from C CVEs — libav is still underneath, and the wrapper carries its own unsafe glue: the "safe skin over an FFI skeleton" line above applies to this crate too.
No parity claims with decord's GPU batch decoding or torchaudio's multi-backend DSP.
If a pure-Rust supply chain is the goal, symphonia is your answer — not ez-ffmpeg.

Verdict: if you can already write the ffmpeg command, this is the lowest-migration-cost way to run it inside your own process; when frames and samples must land in memory with FFmpeg-grade format coverage, it is also the shortest path. When linking is the problem, nothing on this layer saves you.

The set-difference that 0.14 targets

Let me state the full picture first, so nothing here reads as a strawman: Rust has never been unable to hand you decoded frames — video-rs iterates RGB ndarrays in ten lines, and ffmpeg-sidecar turns any video into an RGB frame iterator (via subprocess). What was missing is a one-liner with sampling policies and correct color handling inside a single in-process, FFmpeg-grade runtime.

So the set-difference looks like this: in Python that's three libraries — PyAV, decord, torchaudio. In Rust, video-rs skips audio, symphonia skips video, sidecar forks a process. ez-ffmpeg 0.14 is the first to put all three jobs inside one in-process FFmpeg runtime.

And the scope of that claim, stated precisely: as far as I can verify (July 2026), ez-ffmpeg is the only actively-maintained in-process FFmpeg runtime in Rust where decoded-RGB export, 16 kHz f32 audio export, and frame-push encoding are all first-class APIs: video-rs has no audio API at all and self-declares WIP (its successor rave is "not ready for use yet"), while ac-ffmpeg covers all three domains but at packet-loop level, with no release in 14 months. gstreamer-rs's appsink/appsrc can genuinely do all three in-process too — but through a different pipeline model, not an FFmpeg runtime, which is exactly why the boundary note above places it on a different map.

Two honest footnotes. First, PyAV is itself a set of libav bindings — ez-ffmpeg runs on the same engine. So the claim is not "Rust rewrote PyAV"; it is the same engine wearing a native high-level Rust face, without raw FFI and without a child process. Second, why this gap is worth closing now: candle, burn, ort, tract and whisper-rs are all growing, and what they need is frames and audio as data — while step one of feeding them, today, is often a line of shell. For a sense of how absurd the gap can get: someone implemented a decord-style video reader in Rust — video_reader-rs — and ships it only on PyPI. You cannot cargo add it.

When shelling out is the right answer

A selection guide that funnels every case toward one crate is an ad. For these, the plain CLI is the right answer:

The output is a file and the recipe exists. Transcodes, clips, one-off batch jobs — the ML audio world's step one is still ffmpeg -i in.mp4 -ar 16000 -ac 1 -c:a pcm_s16le out.wav. If that line is your whole requirement, do not add a crate.
Your environment ships the ffmpeg binary anyway. The deployment cost is already paid; no need to also pay the linking cost.
You want process isolation. A crashed transcode job that cannot take down your service is a feature of the subprocess model, not a defect (an engineering judgment, not a quote).
Your team knows the CLI, not the libav API, and the task is not on a hot path. The migration cost exceeds the benefit.

The flip side: the moment frames or samples must land in your process memory — feeding a model, real-time processing, logic interleaved with decoding — the CLI starts charging: stdout parsing, temp WAV files, mismatched parameters. jaredly's pattern, running from 2015 to today, is that tax's ten-year invoice.

Picking by task

Output is a file, recipe exists → CLI
Frames without FFI or linking → ffmpeg-sidecar
Full libav control plane → ffmpeg-next (tracking newer FFmpeg majors → rsmpeg)
Video frames only, WIP acceptable → video-rs
Audio only, pure Rust, codecs within the support table → symphonia + rubato
You already write ffmpeg commands and want them in-process at minimal migration cost → ez-ffmpeg (option names, filter strings and stream specifiers are all CLI syntax)
Frames, samples and frame-push all in-process, FFmpeg-grade coverage → ez-ffmpeg (remember both caveats: the linking pain stays, and the new APIs are experimental)

Every path on this map wins somewhere; none of them replaces the others. Pick by task, not by tribe.

The Rust crate ez-ffmpeg lives at github.com/YeautyYE/ez-ffmpeg.

In-process HLS packaging in Rust: fMP4 segments, an ABR ladder, no ffmpeg subprocess

Yeauty YE — Mon, 27 Jul 2026 19:35:53 +0000

ez-ffmpeg is a Rust crate (no relation to the JavaScript "Ez FFmpeg" project) that runs FFmpeg pipelines inside your process — no subprocess, no hand-written decode loop. As of 0.14 its HLS ladder recipe writes fragmented-MP4 segments as well as MPEG-TS, switched by one enum value, and the crate is on crates.io and GitHub. This article packages HLS both ways, verifies every artifact with ffprobe, and walks through two design decisions that only become visible when something fails.

The first decision: fMP4 or TS

If you are packaging HLS yourself, the container question comes before any code. A decision rather than a definition:

Pick fMP4 when you target modern players (hls.js, AVPlayer, ExoPlayer) on a new project. It is the container CMAF standardized on, so HLS and DASH can share one set of segments; header data lives in a single init.mp4 instead of being repeated in every segment; and if you ever head toward LL-HLS, fMP4 is the ground it is built on.
Pick TS when legacy set-top boxes and older smart TVs are a hard requirement. TS also has an operational property fMP4 gives up: every segment is self-contained and independently playable, which makes single-file inspection one step shorter.

Need both? The code below switches with one enum value.

Everything here was compiled and run against FFmpeg 7.1.3 + ez-ffmpeg 0.15; the outputs quoted are real, none are mocked up. What the crate does not fix, stated up front: installing and linking FFmpeg. It sits on the libav libraries, so FFmpeg 7.1–8.x must be present. What it removes is everything after the link succeeds — the hand-written demux/decode/filter/encode/mux loop, or the Command::new("ffmpeg") subprocess and its stderr scraping.

The minimal runnable path: single-bitrate fMP4

[dependencies]
ez-ffmpeg = "0.15"   # Rust >= 1.80, FFmpeg 7.1-8.x

use ez_ffmpeg::{FfmpegContext, Output};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // The hls muxer writes into an existing directory; it does not mkdir by default.
    std::fs::create_dir_all("hls_single")?;

    FfmpegContext::builder()
        .input("input.mp4")
        .output(
            Output::from("hls_single/index.m3u8")
                .set_format("hls")                                // -f hls
                .set_video_codec("libx264")
                .set_audio_codec("aac")
                // Fixed 6 s GOP at 30 fps, no scene-cut keyframes: segment
                // boundaries can only land on keyframes, so pin them down.
                .set_video_codec_opt("g", "180")
                .set_video_codec_opt("sc_threshold", "0")
                .set_format_opt("hls_time", "6")                  // target segment length
                .set_format_opt("hls_playlist_type", "vod")       // full playlist, ENDLIST at the end
                .set_format_opt("hls_segment_type", "fmp4")       // .m4s instead of .ts
                .set_format_opt("hls_fmp4_init_filename", "init.mp4")
                .set_format_opt("hls_segment_filename", "hls_single/seg_%05d.m4s"),
        )
        .build()?
        .start()?
        .wait()?;

    println!("wrote hls_single/index.m3u8");
    Ok(())
}

The input is a 12-second lavfi-generated test clip (640x360, 30 fps, H.264+AAC). Afterwards hls_single/ holds four files — index.m3u8, init.mp4, seg_00000.m4s, seg_00001.m4s — and the playlist reads, verbatim:

#EXTM3U
#EXT-X-VERSION:7
#EXT-X-TARGETDURATION:6
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-MAP:URI="init.mp4"
#EXTINF:6.000000,
seg_00000.m4s
#EXTINF:6.000000,
seg_00001.m4s
#EXT-X-ENDLIST

Twelve seconds split into exactly two 6.000000-second segments, with EXT-X-MAP pointing at the one init segment — the structural signature of fMP4 HLS. The #EXT-X-VERSION:7 is the muxer's doing: EXT-X-MAP requires protocol version 6+, and FFmpeg writes 7 for fMP4 output.

Mapping the options (and the step I got wrong first)

If you know the CLI, the options map one to one — same names, minus the dash. Container options go through set_format_opt, encoder options through set_video_codec_opt; both feed FFmpeg's own AVOption system.

ffmpeg CLI	ez-ffmpeg
`-f hls`	`.set_format("hls")`
`-hls_time 6`	`.set_format_opt("hls_time", "6")`
`-hls_playlist_type vod`	`.set_format_opt("hls_playlist_type", "vod")`
`-hls_segment_type fmp4`	`.set_format_opt("hls_segment_type", "fmp4")`
`-hls_fmp4_init_filename init.mp4`	`.set_format_opt("hls_fmp4_init_filename", "init.mp4")`
`-hls_segment_filename ...`	`.set_format_opt("hls_segment_filename", "...")`
`-g 180 -sc_threshold 0`	`.set_video_codec_opt("g", "180")` etc.

Those two GOP lines are the step I got wrong. My first version left them out, and the segment durations came out as:

#EXTINF:8.333333,
seg_00000.m4s
#EXTINF:3.666667,
seg_00001.m4s

hls_time=6 is a target, not a command. The muxer can only cut at keyframes, and x264's default keyint is 250 — at 30 fps the first cut opportunity is at 8.33 s, so a request for 6 becomes the reality of 8.33 + 3.67. To make the cuts land on 6, make the keyframes land on 6: g = 6 x 30 = 180, plus sc_threshold=0 so scene-cut detection cannot insert extra ones. With those two lines, you get the 6.000000 + 6.000000 playlist above.

On the builder path this trap is yours to find. The ladder recipe in the next section turns it into a validation rule instead: segment_duration must be an integer multiple of the GOP length, and a violation fails before run() does any work — rather than producing a drifted playlist you discover in production.

Probe before you build

FFmpeg builds differ widely — distro packages, vcpkg, and source builds each trim differently. Instead of failing mid-pipeline, ask up front:

use ez_ffmpeg::capabilities;

fn main() {
    // FFmpeg builds differ in what got compiled in. The probe answers for
    // *this* linked build, before any job is wired up.
    if !capabilities::is_muxer_available("hls") {
        eprintln!("linked FFmpeg build has no hls muxer");
        std::process::exit(1);
    }
    // Segments go through the file protocol; probe it the same way. For a
    // stream published elsewhere you would probe "http" or "srt" here.
    if !capabilities::is_output_protocol_available("file") {
        eprintln!("linked FFmpeg build cannot write files");
        std::process::exit(1);
    }
    println!("hls muxer + file output protocol: available");
}

is_muxer_available answers exactly one question — is this muxer compiled into the linked build — and deliberately does not promise that the network or TLS backends a job needs at runtime are also there. One namespace trap worth knowing: muxer names and protocol names are separate worlds. The muxer called srt is the SubRip subtitle format, unrelated to the SRT streaming protocol; protocols go through is_output_protocol_available. Missing encoders are a different path: since 0.14, build() fails with encoder 'libx264' is not available in the linked FFmpeg build — the error names the encoder instead of a generic not-found.

The ABR ladder: one decode, N renditions

A single bitrate is a buffering spinner for anyone on a weak connection. ABR means encoding the same content at several resolution/bitrate pairs and letting the player switch. Done by hand that is: a split filter fan-out, a scale per branch, fixed-GOP keyframe alignment per branch, one hls muxer per branch, and a hand-assembled master playlist. In 0.14 that orchestration is a recipe:

use ez_ffmpeg::recipes::{HlsLadder, HlsSegmentType};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    HlsLadder::new("input.mp4", "hls_out")
        .rendition(854, 480, "1400k")
        .rendition(640, 360, "800k")
        .segment_duration(4.0)
        .segment_type(HlsSegmentType::Fmp4)
        .run()?;
    println!("wrote hls_out/master.m3u8 and per-rendition playlists");
    Ok(())
}

                     ┌─▶ scale 854x480 ─▶ x264 (GOP-aligned) ─▶ hls muxer ─▶ 480p/…
input.mp4 ─▶ decode ─▶ split
                     └─▶ scale 640x360 ─▶ x264 (GOP-aligned) ─▶ hls muxer ─▶ 360p/…
after every rendition succeeds ─▶ write master.m3u8 (BANDWIDTH = video+audio × 1.1)

What it wires for you: the [0:v]split=2[s0][s1];[s0]scale=854:480,... filtergraph; per-rendition g/keyint_min/sc_threshold=0 plus closed-GOP x264 params, so every rendition's keyframe PTS sequence coincides and segments stay switchable across bitrates; and a master playlist whose BANDWIDTH folds video + audio bitrate plus 10% muxing overhead — an engineering estimate from the nominal bitrates (the RFC defines the field as peak segment bit rate) — the number players use to pick a variant, so under-reporting it makes players overestimate their headroom. Swap HlsSegmentType::Fmp4 for MpegTs (or delete the line — TS is the default) and the same code is a TS ladder; the master's EXT-X-VERSION switches between 7 and 3 accordingly.

One wording matter, stated the way the module docs state it: this is fMP4 HLS as a structural claim — init segment, EXT-X-MAP, .m4s segments, probe-able end to end — not a claim of CMAF compliance or Apple validation.

The recipe's boundaries, copied from its docs rather than hidden: input must be constant-frame-rate (probed from the file when possible; callback inputs need an explicit .fps(30, 1)); one video stream plus at most one audio stream; VOD only; no encryption, no audio groups, no live/event playlists; and the master carries no CODECS attribute (strict validators will flag that). And one Apple-specific trap: Apple requires the hvc1 sample-entry tag for HEVC, FFmpeg's MP4/fMP4 muxer writes hev1 by default — the muxer's codec-tag choice, independent of which encoder produced the stream — and this recipe has no per-stream codec-tag override. Prefer H.264 for Apple targets.

Two craftsmanship details (the part that is not in the README)

Directories are created only after a successful build. Before 0.14, the ladder created its output tree first and built the job second — so a configuration rejected at build time (the classic case: a linked FFmpeg without libx264) left a trail of empty rendition directories. The order is now: wire every rendition Output as pure configuration, let build() resolve encoders, and only mkdir once it has accepted the job. File I/O is deferred to start() anyway, so creating directories earlier bought nothing but litter. Verified by asking for an encoder that does not exist:

use ez_ffmpeg::recipes::{HlsLadder, HlsSegmentType};

fn main() {
    // Deliberately request an encoder this FFmpeg build does not have.
    let result = HlsLadder::new("input.mp4", "hls_reject")
        .rendition(640, 360, "800k")
        .segment_type(HlsSegmentType::Fmp4)
        .video_codec("libx264_that_is_not_here")
        .run();

    println!("run() -> {:?}", result.err().map(|e| e.to_string()));
    println!(
        "hls_reject/ exists after the failed build: {}",
        std::path::Path::new("hls_reject").exists()
    );
}

run() -> Some("Open output error: encoder 'libx264_that_is_not_here' is not
available in the linked FFmpeg build — ...")
hls_reject/ exists after the failed build: false

The master playlist follows the same rule: its text is computed up front but written only after the transcode succeeds, so a failed run leaves no dangling master.m3u8. One ordering consequence to know (per the docs and code order; not exercised with a callback input here): a one-shot callback-backed input is consumed by the build step, which happens before directory creation.

Windows paths are parsed twice — and the two parsers disagree. A path handed to FFmpeg is interpreted once by the OS file APIs (which on Windows accept both / and \) and once by the hls muxer's own string splitting, which only knows /. hlsenc.c derives the fMP4 init segment's output directory with strrchr(m3u8_name, '/') — no DOS-path handling on that route, even though the master-URL code elsewhere in the same file does carry a \ fallback. The consequence: pass a \-separated path on Windows and init.mp4 lands silently outside the rendition directory, so the EXT-X-MAP the player fetches is a 404. The 0.14 fix normalizes every path that enters an FFmpeg option string to forward slashes — lossless, since Windows file APIs accept / — with one deliberate exception: verbatim \\?\ paths are prefix-sensitive, rewriting their separators can change which object they name, so they pass through untouched (prefer regular paths for fMP4 output on Windows). The contract is frozen in the test suite: the expected option strings are literal "out/720p/index.m3u8"-style values, forward slashes on every platform.

The run, verbatim

The ladder's output tree (12 s input, 4 s segments, 3 per rendition):

hls_out/
├── master.m3u8
├── 480p/
│   ├── index.m3u8
│   ├── init.mp4          (1.4 KB)
│   └── seg_00000..2.m4s  (3 segments)
└── 360p/
    ├── index.m3u8
    ├── init.mp4
    └── seg_00000..2.m4s

One init.mp4 per rendition is the expected layout — each rendition is its own hls muxer instance, and each media playlist references only its own init segment. The master:

#EXTM3U
#EXT-X-VERSION:7
#EXT-X-STREAM-INF:BANDWIDTH=1020800,RESOLUTION=640x360
360p/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1680800,RESOLUTION=854x480
480p/index.m3u8

ffprobe acceptance. A bare .m4s is not readable on its own — the headers live in the init segment:

$ ffprobe hls_out/480p/seg_00000.m4s
... trun track id unknown, no tfhd was found
... error reading header

Concatenate the init in front and it is a complete stream — which is what "single init segment" means operationally, and a step TS simply does not have:

$ ffprobe -show_entries format=duration \
    "concat:hls_out/480p/init.mp4|hls_out/480p/seg_00000.m4s"
duration=4.023023

Probing hls_out/master.m3u8 directly reads both renditions (h264 854x480 + aac, h264 640x360 + aac), and the two renditions' EXTINF sequences are byte-identical (diff is empty) — the fixed-GOP alignment doing its job.

When not to use this

Large-scale distribution or a live platform: use a dedicated packager (Shaka Packager) or CDN-side just-in-time packaging. Encryption, DRM, audio groups, subtitle renditions, LL-HLS — none of that is here; this recipe is a minimal viable set for VOD ABR.
A one-off conversion: ffmpeg -i in.mp4 -f hls ... on the command line is shorter. Writing a Rust program for a single run is a detour.
Where in-process earns its keep: packaging as part of program logic — packaging user uploads on demand, emitting HLS from programmatically generated content, or running upload/accounting/notification in the same process right after the packaging step, without managing a child process and parsing its stderr.

You now have the minimal fMP4 path, a TS/fMP4-switchable ABR ladder, the probe-before-build pattern, and two design decisions that only show up on the failure path. examples/hls_conversion and examples/hls_abr_ladder in the repository (github.com/YeautyYE/ez-ffmpeg) are the runnable versions.

Pushing your own frames into an FFmpeg pipeline in Rust (headless frame source)

Yeauty YE — Mon, 27 Jul 2026 05:19:31 +0000

ez-ffmpeg is a Rust crate that runs FFmpeg pipelines inside your process — linked libav libraries behind a high-level API, no subprocess (and no relation to the JavaScript "Ez FFmpeg" project). The narrow claim of this post: its VideoWriter (added in 0.14) takes frames you render in Rust — one tightly packed byte buffer per frame — and pushes them through a real filter → encode → mux pipeline into an MP4, shown below as a complete listing that compiles, runs, and is ffprobe-verified verbatim. It's on crates.io and GitHub.

Here's how old this itch is. In 2015, the original rust-ffmpeg binding received its issue number one (meh/rust-ffmpeg#1). The author, jaredly, wanted the most ordinary thing: he had frames drawn in code and wanted them encoded into a video file. He "couldn't find a way to: open a file as writer; encode video frames to it". His workaround: "rendering to a bunch of images and then using ffmpeg on the cli". A decade later, Rust code that renders, simulates, or visualizes still mostly ships that 2015 workaround.

The workaround, costed

Here is the image-sequence detour in its modern form (PPM keeps it dependency-free — a three-line text header over raw RGB is a legal image):

// Step 1: render every frame to its own image file on disk.
let mut rgb = vec![0u8; (WIDTH * HEIGHT * 3) as usize];
for i in 0..120 {
    render_gradient_rgb(&mut rgb, WIDTH, HEIGHT, i as f32 / 30.0);
    let mut f = File::create(format!("frame_{i:04}.ppm"))?;
    write!(f, "P6\n{WIDTH} {HEIGHT}\n255\n")?; // PPM needs no image crate
    f.write_all(&rgb)?;
}

// Step 2: hope the ffmpeg on PATH exists and agrees with your flags.
let status = Command::new("ffmpeg")
    .args(["-y", "-framerate", "30", "-i", "frame_%04d.ppm",
           "-c:v", "mpeg4", "-q:v", "5", "images.mp4"])
    .status()?;

It works — I ran it, 120 frames in, 120 frames out. It also taxes you three ways. Disk: a 640×360, 30 fps, 4-second animation writes 120 PPM files totalling 80 MB of intermediates for a final MP4 of 213 KiB, and the intermediate size scales with resolution and duration. Naming: frame_{i:04} and frame_%04d are the same implicit contract written in two languages; change the zero-padding on one side and ffmpeg silently stops at the first filename that doesn't match. Failure surface: step one fails as a Rust Result, step two fails as an exit code plus stderr text — "encoder not found" and "frame 37 is corrupt" both arrive as log lines for you to fish out.

Fairness demands the third old way on the table too: skip the disk entirely and pipe raw frames into ffmpeg's stdin — ffmpeg -f rawvideo -pix_fmt rgba -s 640x360 -framerate 30 -i - out.mp4. Zero temp files, zero naming contract, and a full pipe blocks the writer, so backpressure comes free (ffmpeg-sidecar wraps exactly this pattern). That kills the disk tax — but the other two stand: failures are still an exit code plus stderr text, the right ffmpeg binary still has to exist at runtime, and a structurally invalid filter graph only blows up mid-run instead of being rejected up front. The API below earns its keep on that second half — typed errors, open()-time validation, no subprocess lifecycle — not on saving temp files.

The in-process version

your render loop              VideoWriter pipeline (same process)
┌────────────┐ write()/      ┌─────────────┐   ┌────────┐   ┌─────────┐
│ render(buf)│─write_owned──▶│bounded queue│──▶│ filters│──▶│ encoder │─▶ gradient.mp4
└────────────┘ blocks if full└─────────────┘   └────────┘   └─────────┘
   a frame-source worker sits where a decoder would; end = explicit in-band EOF

Same animation, pushed straight into the pipeline. One dependency:

[dependencies]
ez-ffmpeg = "0.15"   # needs FFmpeg 7.1-8.x installed (links libav)

The complete listing — copy it whole:

use ez_ffmpeg::{Output, VideoWriter};

const WIDTH: u32 = 640;
const HEIGHT: u32 = 360;
const FPS: i32 = 30;
const SECONDS: i32 = 4;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Name the encoder explicitly. With a bare "gradient.mp4" the linked
    // FFmpeg build picks the container default (H.264 when libx264 is
    // compiled in, otherwise mpeg4 at a low default bitrate).
    let output = Output::from("gradient.mp4")
        .set_video_codec("mpeg4")
        .set_video_qscale(5);

    let mut writer = VideoWriter::builder(WIDTH, HEIGHT)
        .pixel_format("rgba") // the default, spelled out
        .fps(FPS, 1)
        .open(output)?;

    // The writer tells you the exact byte count of one tightly packed frame.
    let mut frame = vec![0u8; writer.frame_size()];

    let total = FPS * SECONDS;
    for i in 0..total {
        render_gradient(&mut frame, WIDTH, HEIGHT, i as f32 / FPS as f32);
        writer.write(&frame)?; // copies the slice; `frame` is reusable
    }

    writer.finish()?; // drains the encoder, writes the trailer, reports errors
    println!("wrote {total} frames to gradient.mp4");
    Ok(())
}

/// Fills `buf` (tightly packed RGBA) with a gradient that drifts right over time.
fn render_gradient(buf: &mut [u8], width: u32, height: u32, t: f32) {
    for y in 0..height {
        let fy = y as f32 / height as f32;
        for x in 0..width {
            let fx = x as f32 / width as f32;
            let idx = ((y * width + x) * 4) as usize;
            buf[idx] = (((fx + t * 0.25) % 1.0) * 255.0) as u8; // red wave, drifting
            buf[idx + 1] = (fy * 255.0) as u8; // green: vertical ramp
            buf[idx + 2] = ((1.0 - fx) * 255.0) as u8; // blue: horizontal ramp
            buf[idx + 3] = 255; // opaque
        }
    }
}

Two honest caveats first. Installing and linking FFmpeg is not solved here — the crate sits on the libav libraries, FFmpeg 7.1–8.x must be present, and that pain is unchanged; what disappears is the subprocess, the 80 MB of intermediates, and the stderr fishing, not the FFmpeg dependency itself. And this API is marked experimental in the docs: new in 0.14, its surface may still be refined in minor releases.

Call by call:

Output::from("gradient.mp4").set_video_codec("mpeg4").set_video_qscale(5) — the destination is the crate's ordinary Output, so encoder, container, bitrate, codec options, and format options (movflags…) are all configured at that layer. mpeg4 keeps this listing runnable on FFmpeg builds without libx264; swap in set_video_codec("libx264") for H.264. Do name an encoder: with a bare path the linked build picks the container default, which on a libx264-less build means mpeg4 at a low default bitrate.
VideoWriter::builder(WIDTH, HEIGHT) — width and height are the only required parameters, positional so they cannot be forgotten.
.pixel_format("rgba") — the default, spelled out. Any non-hardware AVPixelFormat name works (rgb24, gray8, yuv420p, nv12, …); frames are tightly packed, planes concatenated in descriptor order (yuv420p = all of Y, then U, then V). Hardware formats like cuda are rejected with a typed error — a CPU byte buffer cannot fill them.
.fps(FPS, 1) — constant frame rate as a rational; NTSC rates like fps(30000, 1001) work. Every written frame advances exactly den/num seconds.
.open(output)? — validates the configuration (dimensions, fps, pixel format, filter shape, and that the output actually consumes video), builds the pipeline, and starts it without waiting for a first frame. An unknown encoder name fails right here.
writer.frame_size() — the exact byte count one frame must have: av_image_get_buffer_size(pix_fmt, w, h, 1). A wrong-sized buffer gets a typed InvalidSize { expected, got }, not a garbled picture.
writer.write(&frame)? — pushes one frame by copying the borrowed slice, so the same allocation serves the whole loop. Blocks when the internal queue is full; more on that below.
writer.finish()? — closes ingress, drains the encoder, finalizes the container, and returns the pipeline's first error. This is the result path.

How it works: the four contracts

Ownership: write copies, write_owned moves. From the verification build, the contrast in twelve lines:

    // write(): you keep the buffer. The slice is copied into the pipeline,
    // so one allocation serves the whole run.
    let mut frame = vec![0u8; writer.frame_size()];
    for i in 0..2u8 {
        frame.fill(i * 100);
        writer.write(&frame)?;
    }

    // write_owned(): the Vec moves into the pipeline — no borrow-copy on the
    // caller side. Use it when each frame is born as its own Vec anyway.
    for i in 2..4u8 {
        let owned = vec![i * 60; writer.frame_size()];
        writer.write_owned(owned)?;
    }

Both paths still pay one copy at the far end: the frame-source worker fills the bytes plane-by-plane into a pooled, aligned AVFrame — that copy is not skippable. Two corners worth knowing: a Vec handed to write_owned is queued as-is, so spare capacity beyond its length stays allocated while it waits; and write takes &mut self, so the total frame order is fixed at compile time — the writer is Send (move it to a dedicated producer thread) but deliberately not Sync.

There is no demuxer and no decoder behind this. The builder assembles a pipeline whose filtergraph (buffersrc → [your filter chain or null] → buffersink) is fed by a frame-source worker sitting exactly where a decoder would sit. End of stream is an explicit in-band marker, which is why filters that buffer everything (reverse, tpad) still flush correctly at the end.

Backpressure: the queue is bounded, write waits. The ingress queue is bounded by frame count, defaulting to max(1, min(4, 64 MiB / frame_size)) — four frames of 1080p RGBA, two of 4K RGBA. When rendering outruns encoding, write blocks instead of buffering your animation in RAM. A pipeline that dies mid-run does not leave you parked: a blocked write re-checks pipeline status every 100 ms and returns PushError::PipelineClosed. That error is a signpost, not a verdict:

    let mut pushed = 0;
    loop {
        let frame = vec![0u8; writer.frame_size()];
        match writer.write_owned(frame) {
            Ok(()) => pushed += 1,
            // Not an error by itself: the pipeline stopped taking frames.
            // finish() holds the verdict — the real failure, or Ok when the
            // job simply completed (as the frame limit does here).
            Err(PushError::PipelineClosed) => break,
            Err(e) => return Err(e.into()),
        }
    }

Concretely: put set_max_video_frames(3) on the Output and the pipeline completes itself after three frames — subsequent writes report PipelineClosed, finish() returns Ok, and the file holds exactly 3 frames (verified). If instead an encoder genuinely fails mid-stream, the same finish() hands you the real error. The test suite pins this: worker_failure_unblocks_write_and_finish_reports opens an encoder that rejects the frame size only at first-frame time, and asserts that write unblocks and finish reports — the writer never hangs against a dead consumer.

Timing: constant frame rate, no per-frame PTS. That is the explicit v1 scope: no variable frame rate, no audio. What you do get is a correct tail: 120 frames at 30 fps probe as duration 4.000000, not a clipped 119/30 — the explicit EOF marker closes the stream at the accumulated frame-end time, so the last frame keeps its full duration (test: eof_preserves_final_frame_duration, covered for integral and NTSC rates).

Teardown: finish is the verdict, Drop is the safety net, abort is the discard. finish() is the only path that returns the Result. Forgetting it loses no frames: Drop performs the same close-drain-finalize (test: drop_without_finish_keeps_all_frames — push 10, drop the writer, the file probes 10), but an error is only logged, and the drop can block while the encoder drains. When you don't want the file at all, abort() discards the export; the partial output is not guaranteed playable (test: abort_returns_cleanly).

One more gate worth knowing about: filter_desc accepts any single-input single-output video chain ("hue=s=0" verifiably turns solid red frames gray; "pad=ceil(iw/2)*2:ceil(ih/2)*2" is the odd-dimensions remedy), and graphs that cannot work are rejected at open() with typed errors instead of surfacing as a mid-run hang — a stranded pad, a disconnected graph, or an output unreachable from the input each get their own error. The program output for filter_desc("split"), verbatim:

rejected at open(): Video writer error: filter_desc must have exactly one video input pad and one video output pad; found 1 input pad(s) (1 video) and 2 output pad(s) (2 video)

Compositing over generated sources is still allowed when the pushed frames reach the output — "color=c=red:s=64x48[bg];[in][bg]overlay=shortest=1" runs and ends on input EOF, the same rule the CLI applies.

Run it

The listing above, then ffprobe, output verbatim:

$ ffprobe -v error -select_streams v:0 -show_entries \
    stream=codec_name,width,height,r_frame_rate,nb_frames,duration \
    -of default=noprint_wrappers=1 gradient.mp4
codec_name=mpeg4
width=640
height=360
r_frame_rate=30/1
duration=4.000000
nb_frames=120

120 frames in, 120 frames out, duration exact. No intermediate files, no subprocess, typed errors end to end.

What the ecosystem already had

'Rust can't encode your own frames into an MP4' is false — video-rs's README literally encodes rainbow.mp4 from ndarray frames (and video-rs is itself FFmpeg-based, via ffmpeg-next), opencv-rust has VideoWriter, and openh264 + the mp4 crate do it in pure Rust. So the defensible claim is narrower and concrete: what this API adds over those is the contract work — filter chains validated at open() instead of failing mid-run, explicit backpressure and teardown semantics, typed errors end to end — plus an output surface beyond files. Not "nobody else can encode frames."

The filter-graph validation you saw above. On live targets, this post's evidence stops at the API surface: the module documentation lists RTMP targets among supported destinations, and an Output::from("rtmp://…").set_format("flv") writer session compile-checks — but I did not run one against a live server here, so treat that half of the claim as documented-and-type-checked, not exercised.

When you should not use this

The split is by scenario, and each path has its win. The images already exist on disk and you convert once: ffmpeg -framerate 30 -i frame_%04d.png out.mp4 in a terminal wins, and writing a Rust program for it is a detour. You need GStreamer-grade live pipeline orchestration — dynamic topology, multi-branch fan-out, fine-grained clocking: the gstreamer crate's mature pipeline model is the right tool, not this. You need variable frame rate, per-frame PTS, or an audio track muxed in: v1 does not cover any of those (video only, constant rate; stream maps are rejected; and since pushed frames are raw, video stream-copy does not apply — frames are always encoded). And if your frames come out of another video file rather than your own code, that's a transcode — the crate's ordinary Input/Output job covers it without VideoWriter.

What's left — render loops, simulation output, procedural video, anywhere the frames are born in your code and the destination is a file — is exactly what this API is for.

What the first build costs

The prerequisite is the same one the whole crate carries: FFmpeg 7.1–8.x installed where the linker can find it, because ez-ffmpeg links the libav libraries rather than shelling out. That cost predates this crate and survives it; budget your first cargo build accordingly. The writer itself opens no input format context, so there is no version-specific probing behavior across supported FFmpeg releases.

Where this leaves you

Your render loop now emits an MP4 (or MKV, or anything the linked build muxes) directly, with a filter chain if you want one, backpressure managing memory, and failures arriving as typed Results. In the repository, examples/frames_to_video (a plasma animation) and examples/bouncing_balls (a gravity toy with real per-frame state) run as-is: github.com/YeautyYE/ez-ffmpeg.

Extracting video frames in Rust, in-process — no CLI, no hand-written decode loop

Yeauty YE — Sat, 25 Jul 2026 22:52:39 +0000

Here is a task that comes up constantly. You are doing inference in Rust — candle, burn, ort — and you want to feed a video to a model. Step one is always the same: get the frames. And in Rust today, getting frames means one of two things. Either you shell out to Command::new("ffmpeg"), write PNGs to disk or parse -f rawvideo off a pipe, and read them back; or you reach for the low-level bindings and hand-write the send_packet / receive_frame decode loop — forty lines before the first pixel, and you own the EAGAIN handling and the YUV→RGB conversion yourself. The second path is hard enough that an entire cottage industry of "frame grabber" gists grew up around it: search "rust extract frame from video" and the authoritative answer is frequently someone's gist. A decade on, something this basic still has no comfortable standard answer.

This post is the third path: ez-ffmpeg 0.15's FrameExtractor pulls frames in the same process, in one call, and hands you tightly packed RGB bytes ready for an ndarray view or a tensor. You will get runnable code, a sampling-strategy cheat sheet, and two things that actually earn their keep: why your thumbnails may have been subtly wrong on color this whole time, and how fast this path really is — with numbers you can reproduce.

The two old ways: cross a process, or hand-write the decode loop

Show the ugly first. The shell-out path is cheapest when the output is a file, but the instant frames have to come back into memory they cross a process boundary — PNGs to disk and read back, or a raw pipe you slice into frames yourself. The hand-written decode loop keeps everything in-process, but the boilerplate runs like this: open the demuxer, find the video stream, build the decoder, pair up send_packet / receive_frame, service EAGAIN, then run each frame through swscale from YUV to RGB — every step its own opportunity to be wrong. And that last step hides a trap: many hand-rolled paths (and some libraries) apply BT.601 conversion coefficients to HD video too, which skews saturated reds and greens visibly — with no error to tell you. More on that trap below.

Neither path is wrong. When the output is a file and the recipe exists, the CLI is still the right answer; when you need packet-level control, the raw bindings are irreplaceable. They just never made the specific thing — get a batch of frames, in-process, in one line, with the color right — comfortable.

FrameExtractor: a batch of frames in one line

One dependency line (it still links libav through ffmpeg-next, so FFmpeg 7.1–8.x has to be installed):

[dependencies]
ez-ffmpeg = "0.15"   # needs FFmpeg 7.1-8.x installed (links libav)

Pull 32 frames to feed a VLM. Note UniformN — it takes N frames spread uniformly across the whole clip by presentation time, the primitive fixed-budget CLIP/VLM pipelines actually want:

use ez_ffmpeg::frame_export::{FrameExtractor, Sampling};

// 32 frames evenly spread over the full duration, scaled to 224 wide
// (height derived from the aspect ratio).
let frames = FrameExtractor::new("input.mp4")
    .sampling(Sampling::UniformN(32))
    .width(224)
    .collect_frames()?; // Vec<VideoFrame>

for f in &frames {
    // f.as_bytes() is tightly packed, top-down RGB24 with no row padding —
    // width*height*3 bytes, ready for an ndarray view or a tensor.
    let (w, h) = (f.width(), f.height());
    let rgb: &[u8] = f.as_bytes();
    // your_preprocess(rgb, w, h) ...
    println!("frame #{:>2} pts={:?}us {}x{}", f.index(), f.pts_us(), w, h);
}

Three things worth pointing at. new(input) takes a path, a URL, or anything convertible into an Input; the packed layout defaults to Rgb24 and also offers Rgba32 / Gray8. width(224) sets only the width and derives the height from the source aspect ratio (internally scale=224:-2); pin the other side with height, or give both for an exact size. collect_frames() gathers a Vec; swap in frames() for a streaming iterator (Iterator<Item = Result<VideoFrame>>) that decodes and yields as you consume, without holding every frame in memory — reach for it when you export dense frames from a long file.

The shape of this API is not incidental — it deliberately tracks the FFmpeg CLI. The builder chain reads like a command; start_time_us / duration_us map to -ss / -t (seconds become microseconds); the filter graph it assembles internally is the same scale=…,format=rgb24 shape you would type on the command line, plus explicit color-management parameters. That "move the CLI into your Rust process unchanged" throughline is an argument this series' ecosystem survey makes at length; here it is enough to say your command-line muscle memory stays valid.

Sampling: how many frames, and which ones

Half of frame extraction is deciding which frames. Sampling lays the common strategies out:

Strategy	Meaning	Typical use
`All`	every decoded frame (default)	dense export, per-frame analysis
`EveryNth(n)`	one frame in every n	decimate by frame count
`EverySec(k)`	one frame per k seconds (float)	timeline scrubbing, preview strips
`KeyframesOnly`	keyframes only	shot/scene proxies, fast at decode time
`UniformN(n)`	exactly n frames spread by time	VLM / CLIP fixed-budget input

Two deserve a note. KeyframesOnly pins the decoder option skip_frame=nokey, so non-keyframes are skipped during decode — not decoded and then discarded, not decoded at all — which is where its speed comes from. UniformN(n) guarantees exactly n frames: an input too short to hold n distinct frames pads by repeating nearby frames (each repeat keeps its source frame's pts_us), so the "give me exactly 32" a model pipeline was promised is always honored, with no padding logic on the caller's side.

Color: HD converts as BT.709 by default, and that is not a footnote

Back to the color trap. YUV→RGB needs a coefficient set: SD video uses BT.601, HD video uses BT.709, and picking the wrong one skews saturated colors. Many "decode to RGB" shortcuts apply BT.601 to everything — fine for SD, wrong for 1080p, and silent about it; you only see it if you put the result on screen.

FrameExtractor converts per the frame's own color tags by default (ColorPolicy::Tagged): an HD frame tagged BT.709 converts as BT.709 — no guessing, no one-size-fits-all. This is the behavior in the module I am most willing to stand behind. For contrast: to_ndarray('rgb24') in older PyAV releases applied BT.601 across the board (that qualifier matters — newer PyAV changed it; do not quote it as current). For untagged frames there is TaggedOrResolutionGuess (guess by resolution: BT.709 at height ≥ 720), or Force { matrix, range } to pin one interpretation for the whole run. Most people never touch the default tier — but knowing it gets this right for you is worth the paragraph.

Who should not use it

The usual rule: point outward first, not just at my own crate.

The thumbnail is a file on disk, the recipe exists, and you convert once. ffmpeg -i in.mp4 -vf "select=..." -vsync vfr out_%03d.png is a one-liner; pulling in a crate for it is a detour.
You want frames but will not touch FFI or linking, and a subprocess is fine. ffmpeg-sidecar (~131K downloads/month) wraps any video as an RGB frame iterator, links no libav, and is genuinely pleasant — at the cost of an ffmpeg binary at runtime and frames crossing a process boundary.
Video only, WIP acceptable. video-rs iterates RGB ndarrays in ten lines (~32K downloads/month); it has no audio API and self-describes as work-in-progress, but it is strong at its single subject. To be clear: Rust has never been unable to hand you decoded frames — several paths can. FrameExtractor's difference is doing it in-process, with sampling policies and color management, in one line.
You want decord-style GPU batch decoding. That is a different capability tier; no parity is claimed.

And three plain facts to close on. One: the compile-and-link pain, ez-ffmpeg removes none of it — it still links libav, and the linking gauntlet is unchanged. Two: the project is ~340 stars, still early-stage, and a "safe Rust API" is not the same as "no C CVEs" — libav is underneath. Three: the whole frame_export module — FrameExtractor along with its siblings SampleExtractor (audio → 16 kHz PCM) and VideoWriter (frame-push encoding) — is still marked experimental in the docs: 0.15 settled the behavioral semantics (default precision, sampling), but the API shape may still shift.

Closing

Frames never leave the process, one call gets them, the color follows the tags, and the conversion stays on swscale's fast path — the decade-old "render to images, stitch with the CLI" cottage industry can retire. If you are doing ML in Rust, extracting frames should not send you back out to a subprocess.

Runnable examples are all in the repo: examples/uniform_thumbnails (UniformN into a 4×3 contact sheet), examples/extract_rgb_frames (RGB to PPM), examples/keyframe_thumbnails (keyframe proxies), examples/frame_sampling (the sampling strategies side by side). The crate lives at github.com/YeautyYE/ez-ffmpeg.

An open question I would genuinely like answered: in your Rust ML pipeline today, is frame extraction a shell-out, a hand-written decode loop, or something else? And if I've gotten anything above wrong, tell me and I'll fix it.

A CLI-compat layer that refuses half your commands, on purpose

Yeauty YE — Sat, 25 Jul 2026 08:18:42 +0000

Disclosure first: I maintain the Rust crate ez-ffmpeg (no relation to the JavaScript "Ez FFmpeg" that once hit the HN front page). That is why the "who should not use it" section is the honest one.

What you type into a search box is never "how do I initialize an AVFormatContext" — it is "ffmpeg extract audio command." You already know the command; what stops you in a Rust project is the mapping: which API does this command correspond to? For the last year this article would have been a hand-written lookup table — you find your row, you translate -c:v libx264 -crf 23 into builder calls one flag at a time. ez-ffmpeg 0.15's cli feature does something less tedious and more interesting: paste the command string, and it either runs it in-process (from_cli) or translates it into compile-ready Rust (emit_rust_code).

The interesting part is not that it translates — it is the attitude with which it translates. It sorts every command into three classes: verified (safe to run), unverified (generates scaffolding code with a loud warning, and refuses to execute), and unrecognized (rejected on the spot with a token-anchored, typed error). It would rather refuse you clearly than "run, but come out subtly different from ffmpeg" — because that second failure mode is the one that destroys trust. This post is about that contract: how to use it, and why it is built this way.

from_cli: paste the command, run it in-process

Enable the cli feature (libav is still linked underneath, so FFmpeg 7.1–8.x must be installed):

[dependencies]
ez-ffmpeg = { version = "0.15", features = ["cli"] }

A transcode command, pasted in and run — no subprocess:

use ez_ffmpeg::cli::from_cli;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    from_cli("ffmpeg -i input.mp4 -c:v libx264 -crf 28 -preset veryfast -c:a aac -y output.mp4")?
        .start()?
        .wait()?;
    Ok(())
}

I ran exactly this against a real audio+video clip: from_cli parses, classifies, builds the pipeline, and output.mp4 lands — ffprobe confirms H.264 video plus AAC audio, duration intact. The whole chain runs in your own process: no Command::new("ffmpeg"), no stderr scraping. The string form handles quotes and backslash escapes by POSIX rules; to avoid quoting entirely, pass argv directly with from_cli_args(&["-i", "input.mp4", ...]) — zero quoting ambiguity, the shape ffmpeg.wasm also chose.

One prerequisite up front: execution currently requires an FFmpeg 7.1 runtime. Link any other version (including 8.x) and it fails with UnverifiedRuntimeProfile before any I/O — deliberately, for reasons the "gates" section below covers.

emit_rust_code: or translate it to code

Sometimes you do not want it to run the command for you — you want the builder code, to extend and maintain. Same command:

use ez_ffmpeg::cli::emit_rust_code;

fn main() {
    let code = emit_rust_code(
        "ffmpeg -i in.mkv -c:v libx264 -crf 23 -preset fast -c:a aac -y out.mp4"
    ).unwrap();
    println!("{code}");
}

What prints is a complete, compile-ready program (real output — pinned byte-for-byte by the repository's tests):

// Generated from an ffmpeg command by the ez-ffmpeg CLI-compat emitter.
// command: ffmpeg -i in.mkv -c:v libx264 -crf 23 -preset fast -c:a aac -y out.mp4
// dialect: ffmpeg 7.1 command line; manifest: r4; crate: ez-ffmpeg 0.15.0; cargo features: none required
// status: verified shape V1 (H.264/AAC transcode (crf + preset)) — verified by the manifest-driven semantic golden suite (oracle: Transcode) against the ffmpeg CLI; ...

use ez_ffmpeg::{FfmpegContext, Output};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    FfmpegContext::builder()
        .input("in.mkv")
        .output(
            Output::from("out.mp4")
                .set_video_codec("libx264") // -c:v libx264
                .set_audio_codec("aac") // -c:a aac
                .set_video_codec_opt("crf", "23") // -crf 23
                .set_video_codec_opt("preset", "fast") // -preset fast
        )
        .build()?
        .start()?
        .wait()?;
    Ok(())
}

Read the header comments: the original command, the dialect version (ffmpeg 7.1), the manifest revision, the crate version, and the line that matters most — this shape's verification status. This one is labeled verified shape V1: it passed a semantic golden suite that compares stream identity, codecs, dimensions, durations, and playlist topology against the actual ffmpeg CLI. The translation is not "looks about right"; it is "reconciled against the oracle."

The contract: classify everything, approximate nothing

The design premise is stated plainly in the module docs: broad CLI compatibility is a non-goal. The ffmpeg CLI is ~14k lines of option machinery whose semantics shift with every release; chasing it wholesale produces the "runs, but subtly different" failures that destroy trust. So it does the opposite — every argv token must classify against a versioned compatibility manifest, or the entire command is rejected with a token-anchored typed diagnostic. Nothing is dropped, nothing is guessed. Three outcomes:

① Verified shape → runs. V1 above. Six verified shapes today (V1–V6): H.264/AAC transcode, re-encoded clip, audio extract, single-frame thumbnail, scaled transcode, single-rendition VOD HLS — each backed by a semantic golden. Six sounds small; what the number means is that every one of them is reconciled. Widening the subset costs one golden lane per shape, and the manifest grows release by release — breadth is bought with verification, never with approximation.

② Unverified shape → translated, never executed. For example, a transcode that names codecs but omits crf/preset. emit_rust_code still hands you code — but the header flips to a loud warning (real output):

// status: UNVERIFIED SCAFFOLDING — manifest entry U19 (audio+video codec selection only).
// This shape has no semantic golden. The code below compiles against the
// ez-ffmpeg builder API, but its behavior has NOT been checked against the
// ffmpeg CLI and must not be treated as a faithful translation. Review every
// call before use; in-process execution (from_cli / from_cli_args) refuses
// this shape.

And from_cli refuses to run it — the error's first line is command shape is not verified for execution, followed by the parsed options. It gives you scaffolding as a starting point, but says plainly: "I have not reconciled this one — review it yourself."

③ Unrecognized → rejected on the spot, anchored to the token. A command with an option outside the subset — say ffmpeg -i in.mp4 -bogus 1 -y out.mp4:

unsupported option `-bogus` (token #2, output #0)
  this option is not in the CLI-compat subset

Not "ignore that option and run anyway" — the whole command is rejected, and it tells you which token broke it.

A few honest gates

Beyond the three-way sort, a handful of "refuse rather than fudge" decisions say the most about this layer's character:

-y is mandatory. Omit it and you get missing mandatory `-y`. The reasoning is concrete: without -y the ffmpeg CLI prompts before overwriting, and this library always creates/truncates output — it cannot reproduce that prompt, so it requires you to make the overwrite explicit.
No shell emulation. The string form does POSIX word-splitting only. A pipe in the command? A tokenize error anchored to the exact byte: shell operator; pipes, redirects, command lists and subshells are not ffmpeg options — rejected, never emulated. It translates ffmpeg, not your shell.
Strict AVOptions. CLI-initiated pipelines run in strict mode: an option no component consumed fails the whole command (fftools check_avoptions parity), instead of the default builder path's warning. "That parameter silently did nothing" is not something you can miss here.
Runtime-profile gate. Execution additionally requires a verified runtime profile of the linked FFmpeg (currently 7.1 only; 8.1 joins once its version-matched golden lane passes). Anything else fails with UnverifiedRuntimeProfile before any I/O.

The command layout is fixed too: exactly one -i input and one output path, in canonical order; the -/stdin/stdout pseudo-paths are excluded — pipe I/O is process wiring, not part of the in-process subset.

Beyond the subset: the builder is the full surface

from_cli / emit_rust_code are an on-ramp, not the whole road. The subset covers the most common shapes; the full surface is the builder API itself. And migrating from the CLI to the builder needs only three conversion rules:

Seconds become microseconds: -ss 10 → set_start_time_us(10_000_000) (the _us suffix is the reminder).
Option names work verbatim, minus the dash: -crf 23 → set_video_codec_opt("crf", "23"), -movflags +faststart → set_format_opt("movflags", "faststart") — straight into FFmpeg's AVOption system; whatever name the CLI accepts, this accepts.
Filter strings copy over unchanged: the string after -vf drops into .filter_desc(...) character for character.

A watermark, for instance — -filter_complex "overlay=10:10" is multiple .input() calls mapping to [0]/[1], filter string copied as-is:

FfmpegContext::builder()
    .input("input.mp4").input("logo.png")
    .filter_desc("[0:v][1:v]overlay=10:10")
    .output("watermarked.mp4")
    .build()?.start()?.wait()?;

The full mapping of ~50 CLI options and patterns (with each one's implementation and known gaps) lives in the CLI-to-API mapping on docs.rs. Emitted scaffolding plus that table is the complete "any common command → maintainable Rust" path.

Who should not use it

One-off jobs: use the CLI. Transcode one file, try one filter, verify one parameter — a terminal command is the shortest path; a Rust program is the detour. This layer is for commands you run repeatedly in a program, or want to grow into Rust code.
The full power of ffmpeg: from_cli cannot give it. It is a deliberately narrow trusted subset — six verified shapes, a 7.1 runtime. Two-pass encoding, complex -filter_complex, hardware acceleration, device capture are outside the subset and rejected outright. For the full CLI surface, shell out, or write the builder directly.
Three honest facts: the compile/link-libav pain is untouched by this layer; cli is an optional feature you must enable; and the CLI-compat layer is young — the manifest grows release by release.

Closing

Paste the command and it either runs it for you or hands you the code — but it never pretends to have translated something it did not verify. A tool that will say "I have not reconciled this one; review it yourself" is more trustworthy than one that runs anything. Hand-mapping CLI to API is now only needed outside the subset.

The cli feature is exercised in the repo's examples/cli_emitted_* (the pinned translations of all six verified shapes) and the CLI-to-API mapping on docs.rs. The crate lives at github.com/YeautyYE/ez-ffmpeg.

An open question: which ffmpeg command do you most want to paste straight into Rust and run? Drop it in the comments — and if I've gotten anything above wrong, tell me and I'll fix it.

Feed audio to Whisper in Rust without ever shelling out to ffmpeg

Yeauty YE — Fri, 24 Jul 2026 10:39:33 +0000

ez-ffmpeg is a Rust crate that runs FFmpeg pipelines inside your process — linked libav libraries behind a high-level API, no subprocess (and no relation to the JavaScript "Ez FFmpeg" project). The narrow claim of this post: its 0.14 SampleExtractor turns a video file into the 16 kHz mono f32 PCM that Whisper-family models consume, in one call, verified end-to-end against whisper-rs 0.16 on a real MP4 — full program, real timings, and the verbatim transcript below. It's on crates.io and GitHub.

Here's the itch. Speech-to-text in Rust is in decent shape — whisper-rs is maintained, candle and ort keep growing — yet if your input is an ordinary MP4 rather than 16 kHz mono PCM, nearly every setup guide in the ML-audio ecosystem starts with the same shell line:

ffmpeg -i in.mp4 -ar 16000 -ac 1 -c:a pcm_s16le out.wav

Step one of your "pure Rust" transcription pipeline is to leave Rust. Even "FFmpeg 8.0 adds Whisper support" making the HN front page doesn't change that for you: that news serves CLI users, while a Rust program wants decoded samples in its own memory, not one more command to fork.

The pipeline everyone actually ships

Whisper's input contract is fixed: 16 kHz, mono, f32 samples normalized to [-1, 1] — that's the &[f32] in whisper-rs's full(). Your MP4 most likely holds 44.1 or 48 kHz stereo AAC. So the standard recipe forks ffmpeg and cleans up afterwards:

// The "leave Rust first" pipeline: subprocess + temp file + wav parsing.
let status = std::process::Command::new("ffmpeg")
    .args(["-i", "in.mp4", "-ar", "16000", "-ac", "1",
           "-c:a", "pcm_s16le", "-y", "out.wav"])
    .status()?; // hope ffmpeg is on PATH, hope the flags are right
// ...then open out.wav, skip the header, convert i16 -> f32 by hand

It works, and it taxes you four ways. Deployment: every target machine and container image needs an ffmpeg executable, and its version is now your problem. Errors: a failed subprocess hands you an exit code and a pile of stderr text — distinguishing "file not found" from "no audio stream" means regexing log lines instead of matching on a typed error. Files: a temporary wav has to be written, named, and cleaned up, and concurrent jobs must not collide. Format: pcm_s16le comes back as i16, so you divide by 32768 yourself. Crates like ffmpeg-sidecar wrap this subprocess dance neatly, but the first three taxes are inherent to the model: there's still an external binary at runtime.

One call, in-process

in.mp4 ─▶ demux ─▶ decode(AAC) ─▶ resample(16 kHz mono f32) ─▶ Vec<f32>
          └────── SampleExtractor::for_whisper(), one call ──────┘
Vec<f32> ─▶ whisper-rs full() ─▶ transcript   (no subprocess, no temp file)

SampleExtractor collapses demux → decode → resample → downmix into a single in-process call — internally it's a normal FFmpeg pipeline with a sink on the end that hands the samples back to you. for_whisper presets exactly the contract above:

let pcm: Vec<f32> = SampleExtractor::for_whisper("in.mp4").collect_samples()?;

Two honest caveats before the code. First, installing and linking FFmpeg is not solved here — the crate sits on the libav libraries, so FFmpeg 7.1–8.x must be present, and that pain is unchanged. What goes away is the subprocess, the temp file, and the stderr scraping — not the FFmpeg dependency itself. Second, the frame/sample export API is marked experimental in the README and may be reshaped in a future minor release.

The complete program, two phases — extract PCM, feed whisper-rs 0.16:

[package]
name = "mp4-to-text"
version = "0.1.0"
edition = "2021"

[dependencies]
ez-ffmpeg = "0.14"    # needs FFmpeg 7.1-8.x installed (links libav)
whisper-rs = "0.16"   # needs cmake + C/C++ toolchain (builds whisper.cpp)

use std::time::Instant;

use ez_ffmpeg::frame_export::SampleExtractor;
use whisper_rs::{FullParams, SamplingStrategy, WhisperContext, WhisperContextParameters};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Phase 1: decode the video's audio track into 16 kHz mono f32 PCM.
    let t_extract = Instant::now();
    let pcm: Vec<f32> = SampleExtractor::for_whisper("jfk.mp4").collect_samples()?;
    let extract_ms = t_extract.elapsed().as_secs_f64() * 1000.0;

    println!("pcm samples : {}", pcm.len());
    println!(
        "pcm duration: {:.3} s (len / 16000.0, mono)",
        pcm.len() as f64 / 16_000.0
    );
    println!("extract time: {extract_ms:.1} ms");

    // Phase 2: transcribe the PCM with whisper.cpp via whisper-rs.
    let t_whisper = Instant::now();
    let ctx = WhisperContext::new_with_params(
        "ggml-tiny.en.bin",
        WhisperContextParameters::default(),
    )?;
    let mut state = ctx.create_state()?;

    let mut params = FullParams::new(SamplingStrategy::Greedy { best_of: 1 });
    params.set_language(Some("en"));
    params.set_print_special(false);
    params.set_print_progress(false);
    params.set_print_realtime(false);
    params.set_print_timestamps(false);

    state.full(params, &pcm)?;

    let mut transcript = String::new();
    let n_segments = state.full_n_segments();
    for i in 0..n_segments {
        if let Some(segment) = state.get_segment(i) {
            transcript.push_str(&segment.to_str_lossy()?);
        }
    }
    let whisper_ms = t_whisper.elapsed().as_secs_f64() * 1000.0;

    println!("whisper time: {whisper_ms:.1} ms");
    println!("transcript  : {}", transcript.trim());
    Ok(())
}

Call by call:

SampleExtractor::for_whisper("jfk.mp4") — a thin preset, equivalent to SampleExtractor::new(...).sample_rate(16000).channels(Mono). It is worth pointing out that the base builder's defaults preserve the source: source sample rate and channel layout pass through untouched, and only the sample format is pinned to packed f32. Resampling and downmixing are opt-in, so music and analysis users don't get silently converted to 16 kHz; an ASR preset is where that normalization belongs, explicitly.
.collect_samples()? — runs the pipeline and returns one flat interleaved Vec<f32>. Mono means samples == frames, so len() / 16000.0 is the duration — the program prints exactly that. The whole track sits in memory, so for hours-long input switch the same builder to .samples()?: a streaming iterator that yields chunks as they decode, carries rate/channels/timestamp metadata on each chunk, keeps a small bounded number in flight (default 4), and aborts the run cleanly if you drop it early. Errors have defined landing spots too: option mistakes and "no audio stream" surface when you start the run, mid-decode failures arrive as the iterator's terminal error — typed, not scraped from stderr. start_time_us / duration_us carve out a time window when you chunk long audio for the model.
WhisperContext::new_with_params + create_state — loads the ggml model from disk and allocates inference state; most of the "whisper time" below lives here and in full.
FullParams::new(SamplingStrategy::Greedy { best_of: 1 }) — the cheapest decoding strategy; set_language(Some("en")) matches the English-only tiny.en model.
The five set_print_*(false) calls — whisper.cpp prints transcription progress to stderr by default, so an embedded library behaves like a chatty CLI; these turn that off. Model loading logs another ~30 lines through a separate path; whisper_rs::install_logging_hooks silences those.
The segment loop — full(), full_n_segments(), get_segment(i), to_str_lossy() — is the whisper-rs 0.16 API. It looks unremarkable, and it is exactly where most published examples stop compiling. That deserves its own section.

The trap nobody documents: whisper-rs 0.16 API drift

The ez-ffmpeg half of this program gave me nothing to write about: the documented one-liner compiled and ran correctly on the first try. Every wall I hit was on the whisper-rs half — starting with my first draft, which followed an older tutorial and failed to compile: one method not found, one ? where ? no longer applies. Checking the current docs turned up three changes that most pre-0.16 blog posts trip over:

full() returns Result<(), _> — apply ? and move on; don't expect a number back.
full_n_segments() returns a plain i32 — a trailing ? is now a compile error.
full_get_segment_text() is gone — use get_segment(i), which returns Option<WhisperSegment>, then to_str() / to_str_lossy() for the text.

The segment loop in the listing above is the current, correct form. The general lesson I took: for a fast-moving crate, trust the method signatures on docs.rs for the exact version in your lockfile, and treat blog posts as intent, not syntax. One more orientation note: whisper-rs's GitHub repository is archived, but the project is not dead — it moved to Codeberg and is maintained there. Most of the tutorials a search engine surfaces predate these changes. (A caveat on my own account: I reconstructed this API history from a single migration, so treat it as one data point, not a changelog.)

Run it

Test input: an 11-second clip of JFK's "ask not" line packed into an MP4 (h264 video + AAC audio track). Model: ggml-tiny.en.bin, 77,704,715 bytes. Two consecutive runs produced identical sample counts and identical transcripts; here is the second run's output, verbatim:

pcm samples : 176128
pcm duration: 11.008 s (len / 16000.0, mono)
extract time: 7.8 ms
whisper time: 635.6 ms
transcript  : And so my fellow Americans ask not what your country can do for you ask what you can do for your country.

The transcript is correct, whole sentence, first pass. The numbers reconcile too: 176,128 ÷ 16,000 = 11.008 s. The source wav was 11.00 s; the extra ≈8 ms is AAC encoder priming/padding introduced when the clip was remuxed — alignment samples the codec pads at the edges of the track, entirely normal. Which is why duration checks against extracted PCM should assert ≈, never == — an exact-equality assertion dies on those 8 ms.

Timings, with the full caveat attached: one machine, an 8-core/16-thread Ryzen 9 5900HX, CPU-only, two consecutive runs. This is a smoke test, not a benchmark.

Phase	Run 1 (cold)	Run 2 (warm)
PCM extraction (`SampleExtractor`)	16.3 ms	7.8 ms
Whisper phase (incl. loading the 77 MB model from disk)	1463.9 ms	635.6 ms

Two readings. The cold/warm gap in the whisper row is mostly the 77 MB model file being in the OS cache the second time — don't read it as inference getting faster. And for this 11-second clip, extraction is single-digit-to-low-double-digit milliseconds while the model dominates — the PCM step is no longer the thing worth optimizing in this pipeline. Batch throughput and long-file behavior weren't measured here, so no numbers are claimed for them.

When you should not use this

Pure Rust can pull the audio track out of an MP4/MKV and resample it to 16 kHz (symphonia + rubato; symphonium does both in one call; mutter feeds whisper-rs in five lines). But "any video, any codec, one line" hits a codec wall: symphonia still lists HE-AAC and Opus as unfinished and has no AC-3 at all — which is exactly where an FFmpeg backend remains irreplaceable.

So the split is by scenario, and each path has its win. One file, one time: type the ffmpeg command in a terminal; writing a Rust program for a run-once job is a detour. Inputs you control (WAV, FLAC, ordinary AAC) and a zero-C-dependency build as a goal: symphonia + rubato is the right road, and linking FFmpeg buys you nothing. Inputs you don't control — user uploads in whatever container and codec they arrive in — FFmpeg's codec coverage is the point, and that is the case for an FFmpeg-backed crate like ez-ffmpeg. And keep one asterisk in view: even with a pure-Rust audio path, whisper-rs itself compiles whisper.cpp, so "no C toolchain anywhere" was never on the table for this pipeline.

What the first build costs

Honesty about the build, since the Cargo.toml comments above compress it: whisper-rs builds whisper.cpp from source, which needs cmake and a C/C++ toolchain, and the first build takes minutes — it is not hung. On PEP 668 distros (Ubuntu 24.04 and later), pip install --user cmake is refused by the system Python; install cmake inside a venv or just use your distro's cmake package. The ez-ffmpeg side needs FFmpeg 7.1–8.x present for linking, as covered earlier — that prerequisite predates this crate and survives it.

Where this leaves you

Your transcription pipeline now has one subprocess, one temp file, and one hand-rolled i16→f32 conversion less, and its failures arrive as typed Results. From here: switch to samples() plus start_time_us/duration_us to stream and chunk long recordings, or hand the same Vec<f32> to other audio models via candle or ort — 16 kHz mono f32 is the common handshake shape.

The code lives at github.com/YeautyYE/ez-ffmpeg; examples/extract_whisper_pcm in the repository runs as-is.

🚀 Double Your Performance with One Line of Code? The Memory Superpower Every Rust Developer Should Know!

Yeauty YE — Wed, 02 Jul 2025 11:42:11 +0000

In the world of high-performance Rust programming, you often hear about hardcore optimization techniques: asynchronous magic with Tokio, blazing-fast SIMD instructions, lock-free concurrency, and more.

Yet there’s one often-overlooked tool lurking in your stack that can deliver a massive, sometimes game-changing performance boost with minimal effort:

👉 The Memory Allocator.

Yes, really—by simply swapping out Rust’s default allocator, your application can achieve multi-fold throughput improvements and significantly lower latency in high-concurrency or large-data workloads. This isn’t hype—it’s backed by serious benchmarks:

Authoritative Benchmark Highlights

Microsoft mimalloc report: Under heavy multithreaded workloads on Linux, mimalloc delivered 5.3× faster average performance compared to glibc malloc, while cutting RSS memory usage by ~50%. More performance, less cost.

jemalloc research paper: In real-world tests on a 4-core server, glibc malloc achieved only 15% of jemalloc’s throughput—a night-and-day difference for latency-sensitive services.

In this post, you’ll discover why allocators matter so much, what makes modern allocators radically faster, and how you can unlock their benefits with literally one line of code.

🧠 1. Memory Allocators: Your Program’s Invisible Power Broker

Whenever you write something like Vec::with_capacity(100), you’re calling the allocator: the engine that manages heap memory behind the scenes.

Rust’s default allocator is usually the system allocator—e.g., glibc malloc on Linux. It’s reliable and general-purpose, but under concurrency, it can quickly become a bottleneck.

❌ The Problem with Traditional Allocators: Global Lock Contention

Under high concurrency, all threads compete for a single global lock to allocate or free memory. It’s like the entire country trying to buy train tickets from one window during the holidays:

      +-------------------------------------------+
      |      Traditional Allocator (glibc malloc) |
      |          One Big Global Lock 🔒           |
      +-------------------------------------------+
          ^          ^          ^          ^
          |          |          |          |
      [Thread1]  [Thread2]   [Thread3]   [Thread4]
      (waiting)  (waiting)   (holding)   (waiting)

Result: threads pile up, CPUs spend more time context switching than doing real work, and throughput plummets.

✅ Modern Allocators to the Rescue: Thread-Local Caches

Allocators like jemalloc and mimalloc assign each thread its own fast, lock-free cache:

+------------------+ +------------------+ +------------------+
| Thread 1 Cache   | | Thread 2 Cache   | | Thread 3 Cache   |
| (No locks here!) | | (No locks here!) | | (No locks here!) |
+------------------+ +------------------+ +------------------+
        |                  |                  |
        +------------------+------------------+
                           |
                           v
             +-----------------------------+
             |    Global Memory Pool       |
             |   (accessed infrequently)   |
             +-----------------------------+

Most allocations happen without any locking, so threads can operate at full speed. Only when the local cache runs out does it fetch more memory from the global pool.

🔍 2. Why Does Swapping Allocators Make You So Much Faster?

Let’s break down the core reasons:

🔒 2.1 Escaping the Global Lock Hell

This is the biggest win. With thread-local caches, modern allocators eliminate lock contention, unleashing the full power of multi-core CPUs.

In languages like Rust or Go that thrive on concurrency, this difference is especially dramatic.

🧩 2.2 Fighting Memory Fragmentation

Frequent allocations and deallocations can leave memory full of tiny gaps:

Memory State: | Used | Free(1) | Used | Free(2) | Used |
              +------+--------+------+--------+------+

New Request: "I need 3 contiguous blocks..."

Result: Even though total free space >3, allocation fails—classic fragmentation.

Modern allocators prevent this by using binning: pre-categorizing memory into size classes (e.g., 8B, 16B, 32B) and reusing those buckets.

+------------------------------------------------------+
|        Modern Allocator Memory Pools (Size Classes)  |
+------------------------------------------------------+
| [8B Bin][16B Bin][32B Bin][64B Bin][...more bins...] |
+------------------------------------------------------+

This not only reduces fragmentation but also improves cache locality and speeds up memory access.

⚡ 2.3 Optimizing Large Allocations

Traditional allocators often call the kernel (mmap) every time you need a large chunk of memory—an expensive syscall.

Modern allocators instead reserve large arenas up front and manage them in user space, reducing system call overhead.

🔧 3. Rust Made It Simple: One Line to Swap Your Allocator

Using a faster allocator in Rust is ridiculously easy. For example, to enable mimalloc:

Add it to Cargo.toml:

[dependencies]
mimalloc = "0.1"

Declare the global allocator in your main.rs:

#[global_allocator]
static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;

fn main() {
    // No other code changes needed!
}

That’s it. Build and run—your program is now supercharged.

⚠️ Note: You can only have one global allocator per binary. But what if you need to support different allocators per platform without messy #[cfg] flags everywhere? There’s a better way.

✨ 4. The Ultimate Solution: `auto-allocator`—No More Conditional Compilation Hell

auto-allocator is a smart Rust library that auto-detects your target platform and picks the best allocator automatically. Whether you’re building high-concurrency servers, mobile apps, WebAssembly frontends, or embedded devices, it just works.

🌍 Platform-Aware Optimizations:

Linux/Windows/macOS: Enables mimalloc for up to 6× throughput improvements.
iOS: Uses Apple’s optimized libmalloc for stability and performance.
Android: Switches to scudo for efficient and secure allocation.
WebAssembly: Retains the default allocator for maximum compatibility.
Embedded (no_std): Selects embedded-alloc for resource-constrained environments.

How It Works

Auto-Allocator uses a two-stage optimization pipeline:

🛠️ Compile Time                     ⚡ Runtime                    ✅ Final Result
┌────────────────────────┐          ┌───────────────────────┐     ┌─────────────────────────┐
│ Platform Detection.    │          │ CPU Core Analysis     │     │                         │
│                        │─────────▶│                       │     │                         │
│ Feature Analysis       │          │ Memory Detection      │────▶│ Best Allocator Selected │
│                        │─────────▶│                       │     │                         │
│ Compiler Capabilities  │          │ Hardware Optimization │     │                         │
└────────────────────────┘          └───────────────────────┘     └─────────────────────────┘

✅ 90% of decisions happen at compile time—no runtime overhead.

Usage:

Add it to your Cargo.toml:

[dependencies]
auto-allocator = "*"

Import in main.rs:

use auto_allocator;

fn main() {
    let info = auto_allocator::get_allocator_info();
    println!(
        "Using allocator: {:?} | Reason: {}",
        info.allocator_type, info.reason
    );

    let data: Vec<i32> = (0..1_000_000).collect();
    println!(
        "Created Vec with {} elements—allocation optimized automatically!",
        data.len()
    );
}

✨ That’s it. One use statement—done.

For additional security (e.g., guard pages, canary checks), enable the secure feature:

[dependencies]
auto-allocator = { version = "*", features = ["secure"] }

🏁 5. Wrapping Up: Time for a Free Performance Upgrade

By swapping your memory allocator, you can unlock:

🚀 Higher throughput: Handle more traffic on the same hardware.
💰 Lower latency and costs: Faster response times and reduced memory footprint.
🧩 Better stability: Less fragmentation over long-running workloads.

If you care about performance, this is one of the highest ROI optimizations you’ll ever make.

👇 Give it a try—add auto-allocator to your Cargo.toml today and see the results yourself!

👉 GitHub Repository – auto-allocator

Implementing FFmpeg Filters with Rust: A New Approach to Video and Audio Processing

Yeauty YE — Wed, 09 Apr 2025 12:11:51 +0000

Hey there, fellow developers! If you’ve ever wrestled with FFmpeg’s C API in Rust—or shuddered at the thought of memory leaks and unsafe blocks—this one’s for you. Today, we’re diving into ez-ffmpeg, a Rust library that makes implementing custom video and audio filters a breeze. Think safety, simplicity, and yes, even GPU acceleration—all wrapped in Rust’s ergonomic goodness. Ready to level up your multimedia game? Let’s get started.

Why This Matters: The Pain of Traditional FFmpeg

FFmpeg is a powerhouse for multimedia processing—encoding, decoding, filtering, you name it. But using its C API directly in Rust? That’s a different story:

Complexity Overload: Manual memory management can feel like defusing a bomb—one wrong move, and boom, leaks or crashes.
Safety Risks: Foreign Function Interface (FFI) means unsafe blocks, inviting bugs that Rust usually shields us from.
Steep Learning Curve: Custom filters demand deep FFmpeg knowledge—filter graphs, frame handling—not exactly beginner-friendly.

Enter ez-ffmpeg. It’s like a friendly guide that takes you from “FFmpeg newbie” to “filter pro” without the headaches. Whether you’re tweaking live streams, augmenting ML datasets, or spicing up game visuals, this library’s got your back.

What You’ll Learn

We’ll walk through three killer examples:

Brighten Up: A CPU-based brightness filter for YUV420 video.
Go Gray: A GPU-accelerated grayscale filter with OpenGL.
Turn It Down: An audio volume adjustment filter.

Plus, we’ll explore real-world use cases and why ez-ffmpeg is a game-changer. Let’s dive in!

Getting Started: Setup

First, add ez-ffmpeg to your Cargo.toml:

[dependencies]
ez-ffmpeg = "*"

For GPU fun later, enable the opengl feature:

ez-ffmpeg = { version = "*", features = ["opengl"] }

You’ll need FFmpeg 7.0+ and Rust 1.80.0+ installed. Got that? Great—let’s code.

Example 1: Brighten Your Video (YUV420)

Most videos use YUV420, so let’s start with a brightness filter that tweaks the Y (luminance) component.

The Code

Here’s how we define the filter:

use ez_ffmpeg::core::filter::frame_filter::FrameFilter;
use ez_ffmpeg::core::frame::Frame;
use ez_ffmpeg::core::filter::frame_filter::FrameFilterContext;
use ffmpeg_sys_next::{AVMediaType, AVPixelFormat};

struct BrightnessFilter {
    increment: i32,
}

impl FrameFilter for BrightnessFilter {
    fn media_type(&self) -> AVMediaType {
        AVMediaType::AVMEDIA_TYPE_VIDEO
    }

    fn filter_frame(&self, mut frame: Frame, _ctx: &FrameFilterContext) -> Result<Option<Frame>, String> {
        if frame.format() != AVPixelFormat::AV_PIX_FMT_YUV420P {
            return Err("Unsupported pixel format".to_string());
        }
        let y_data = frame.get_data_mut(0).ok_or("Failed to get Y plane")?;
        for y in y_data.iter_mut() {
            let new_y = (*y as i32 + self.increment).clamp(0, 255) as u8;
            *y = new_y;
        }
        Ok(Some(frame))
    }
}

And here’s how to run it:

use ez_ffmpeg::{FfmpegContext, Output, FramePipelineBuilder};
use ffmpeg_sys_next::AVMediaType;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let frame_pipeline_builder: FramePipelineBuilder = AVMediaType::AVMEDIA_TYPE_VIDEO.into();
    let brightness_filter = BrightnessFilter { increment: 20 };
    let frame_pipeline_builder = frame_pipeline_builder.filter("brightness", Box::new(brightness_filter));

    FfmpegContext::builder()
        .input("input.mp4")
        .output(Output::from("output.mp4").add_frame_pipeline(frame_pipeline_builder))
        .build()?
        .start()?
        .wait()?;
    Ok(())
}

What’s Happening?

Filter Logic: We grab the Y plane, bump each value by 20, and clamp it between 0-255.
Pipeline: We plug the filter into a video processing pipeline, transforming input.mp4 into a brighter output.mp4.

Run it, and bam—your video’s glowing!

Example 2: Grayscale with GPU Power (OpenGL)

For real-time magic, let’s use the GPU. This grayscale filter leverages OpenGL for speed.

The Shader

Here’s the fragment shader:

#version 330 core
in vec2 TexCoord;
out vec4 FragColor;
uniform sampler2D texture1;

void main() {
    vec4 color = texture(texture1, TexCoord);
    float gray = 0.299 * color.r + 0.587 * color.g + 0.114 * color.b;
    FragColor = vec4(gray, gray, gray, color.a);
}

The Code

use ez_ffmpeg::{FfmpegContext, Output, FramePipelineBuilder};
use ez_ffmpeg::opengl::OpenGLFrameFilter;
use ffmpeg_sys_next::AVMediaType;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let fragment_shader = r#"
    #version 330 core
    in vec2 TexCoord;
    out vec4 FragColor;
    uniform sampler2D texture1;

    void main() {
        vec4 color = texture(texture1, TexCoord);
        float gray = 0.299 * color.r + 0.587 * color.g + 0.114 * color.b;
        FragColor = vec4(gray, gray, gray, color.a);
    }"#;

    let filter = OpenGLFrameFilter::new_simple(fragment_shader);
    let frame_pipeline_builder: FramePipelineBuilder = AVMediaType::AVMEDIA_TYPE_VIDEO.into();
    let frame_pipeline_builder = frame_pipeline_builder.filter("opengl", Box::new(filter));

    FfmpegContext::builder()
        .input("input.mp4")
        .output(Output::from("output.mp4").add_frame_pipeline(frame_pipeline_builder))
        .build()?
        .start()?
        .wait()?;
    Ok(())
}

Why It Rocks

Speed: GPU acceleration handles big frames fast—perfect for live streams.
Simplicity: No fiddly CPU loops; the shader does the heavy lifting.

Your video’s now a stylish black-and-white masterpiece.

Example 3: Lower the Volume (Audio)

Let’s switch gears to audio with a volume adjustment filter.

The Code

use ez_ffmpeg::core::filter::frame_filter::FrameFilter;
use ez_ffmpeg::core::frame::Frame;
use ez_ffmpeg::core::filter::frame_filter::FrameFilterContext;
use ffmpeg_sys_next::{AVMediaType, AVSampleFormat};

struct VolumeFilter {
    gain: f32,
}

impl FrameFilter for VolumeFilter {
    fn media_type(&self) -> AVMediaType {
        AVMediaType::AVMEDIA_TYPE_AUDIO
    }

    fn filter_frame(&self, mut frame: Frame, _ctx: &FrameFilterContext) -> Result<Option<Frame>, String> {
        if frame.format() != AVSampleFormat::AV_SAMPLE_FMT_S16 {
            return Err("Unsupported sample format".to_string());
        }
        let data = frame.get_data_mut(0).ok_or("Failed to get audio data")?;
        let samples = unsafe { std::slice::from_raw_parts_mut(data.as_mut_ptr() as *mut i16, data.len() / 2) };
        for sample in samples.iter_mut() {
            let new_sample = (*sample as f32 * self.gain).clamp(-32768.0, 32767.0) as i16;
            *sample = new_sample;
        }
        Ok(Some(frame))
    }
}

Run it like this:

use ez_ffmpeg::{FfmpegContext, Output, FramePipelineBuilder};
use ffmpeg_sys_next::AVMediaType;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let frame_pipeline_builder: FramePipelineBuilder = AVMediaType::AVMEDIA_TYPE_AUDIO.into();
    let volume_filter = VolumeFilter { gain: 0.5 };
    let frame_pipeline_builder = frame_pipeline_builder.filter("volume", Box::new(volume_filter));

    FfmpegContext::builder()
        .input("input.mp4")
        .output(Output::from("output.mp4").add_frame_pipeline(frame_pipeline_builder))
        .build()?
        .start()?
        .wait()?;
    Ok(())
}

How It Works

Gain Factor: gain: 0.5 halves the volume (range: -32768 to 32767).
Safety: Rust keeps us crash-free while tweaking samples.

Now your audio’s perfectly chilled.

Where This Shines: Use Cases

Live Streaming: Add effects like brightness or grayscale in real time.
Machine Learning: Transform video frames for data augmentation.
Gaming: Dynamic video effects for immersive experiences.
Audio Tweaks: Fine-tune sound for podcasts or videos.
Surveillance: Process feeds for motion detection.

With ez-ffmpeg, you skip FFmpeg’s filter graph maze and get straight to the fun stuff.

Why `ez-ffmpeg` Wins

Safety First: Rust’s guarantees mean no memory woes.
Ease of Use: Ergonomic APIs lower the entry barrier.
Performance: CPU or GPU, pick your speed.
Flexibility: Video, audio, real-time—handle it all.

What’s Next?

This is just the start. Explore ez-ffmpeg’s hardware encoding, streaming, or metadata tools. Check out the GitHub repo for more examples and docs.

So, what do you think? Ready to ditch the C API and go Rust? Drop a comment—I’d love to hear your ideas or projects!

Happy coding!

How to Easily Implement RTMP Live Streaming in Rust: A Practical Guide

Yeauty YE — Fri, 04 Apr 2025 08:06:00 +0000

Introduction

With the rapid growth of live streaming, RTMP (Real-Time Messaging Protocol) has become a standard choice for broadcasting real-time media content. However, directly working with low-level APIs to implement RTMP streaming can be complex, error-prone, and may introduce memory safety issues, posing significant challenges for developers.

This article explores how you can leverage Rust’s safety, efficiency, and developer-friendly features to simplify RTMP live streaming. We'll discuss common streaming scenarios, technical challenges, and provide practical solutions with ready-to-use examples.

Why Use Rust for RTMP Streaming?

Common pain points when implementing RTMP streaming include:

Complex low-level APIs with a steep learning curve.
Memory management and safety risks when directly interfacing with FFmpeg APIs.
High demands for real-time performance and stability.

Rust, by its very design, addresses these challenges effectively:

Offers zero-cost abstractions and built-in memory safety.
Performance comparable to C/C++ with a significantly improved developer experience.
Rich ecosystem and seamless integration with existing C/C++ libraries.

Thus, Rust is an excellent choice for developing robust and efficient RTMP streaming applications.

Common RTMP Streaming Scenarios

Based on real-world requirements, RTMP streaming usually falls into two scenarios:

1. Streaming to Public Platforms

Use Case: Platforms like Twitch, YouTube, or Facebook Live for large audience broadcasts.
Key Requirements: Stability, low latency, ease of use.

2. Local or Internal Streaming

Use Case: Suitable for internal live streams, testing, development, or LAN video monitoring.
Key Requirements: Easy deployment, flexibility, quick setup, and convenient debugging.

Solution: Rust with ez-ffmpeg

To address these challenges and requirements, we introduce ez-ffmpeg, a Rust library providing a clean abstraction over FFmpeg, significantly simplifying RTMP streaming through user-friendly interfaces and automatic memory management.

Technical Highlights:

FFI Bindings: Safe and efficient interfacing with native FFmpeg APIs.
Automatic Memory Management: Eliminates manual memory handling, enhancing safety and productivity.
Ergonomic Design: Chainable method calls, improving developer experience.

Quick Start Examples

Environment Setup

Install FFmpeg

macOS:

brew install ffmpeg

Windows:

vcpkg install ffmpeg

Rust Dependency

[dependencies]
ez-ffmpeg = { version = "*", features = ["rtmp"] }

Scenario 1: Stream to Public RTMP Servers

The following example shows how to push a local video file to public RTMP platforms (e.g., YouTube, Twitch):

use ez_ffmpeg::{FfmpegContext, Input, Output};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = Input::from("video.mp4");
    let output = Output::from("rtmp://your-platform-address/app/stream_key");

    FfmpegContext::builder()
        .input(input)
        .output(output)
        .build()?
        .start()?
        .wait()?;

    Ok(())
}

Scenario 2: Embedded Local RTMP Server

Ideal for local testing and quick setup:

use ez_ffmpeg::rtmp::embed_rtmp_server::EmbedRtmpServer;
use ez_ffmpeg::{FfmpegContext, Input};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut server = EmbedRtmpServer::new("localhost:1935").start()?;
    let output = server.create_rtmp_input("test-app", "test-stream")?;

    let input = Input::from("video.mp4");

    FfmpegContext::builder()
        .input(input)
        .output(output)
        .build()?
        .start()?
        .wait()?;

    Ok(())
}

Clients can access the stream using the URL:

rtmp://localhost:1935/test-app/test-stream

Conclusion

By leveraging Rust and the ez-ffmpeg library, developers can easily and safely implement RTMP streaming applications. Whether targeting public platforms or setting up local streaming environments, this approach makes development faster and more reliable.

🔗 Project Repository: https://github.com/YeautyYE/ez-ffmpeg

Efficient Video Processing in Rust: Leveraging Hardware Acceleration for High-Resolution Content

Yeauty YE — Wed, 02 Apr 2025 07:03:04 +0000

Introduction

In the world of video processing, the rise of 4K, 8K, and even higher-resolution content has pushed traditional CPU-based approaches to their limits. Whether it’s video editing, live streaming, or format conversion, developers often grapple with excessive CPU usage under heavy workloads. Hardware acceleration steps in as a game-changer, offloading encoding and decoding tasks to GPUs or specialized chips, boosting performance while freeing up CPU resources for a smoother user experience. Rust, with its blend of performance and safety, offers a robust ecosystem for tackling these demands—tools like ez-ffmpeg make hardware-accelerated video processing accessible and efficient.

The Problem: Why Hardware Acceleration Matters

Picture this: you’re building a video transcoding tool to convert a 4K clip into H.264 format. Relying solely on the CPU, the process could take minutes, with usage spiking to 100%, slowing down everything else on the system. Or consider a real-time streaming app—high latency and dropped frames can ruin the viewer’s experience. These challenges highlight a key truth: modern video processing often exceeds what CPUs alone can handle. By tapping into GPUs or dedicated hardware like Intel Quick Sync or Nvidia NVENC, hardware acceleration slashes processing times and eases system strain, delivering results in a fraction of the time.

How It Works: Implementing Hardware Acceleration

At its core, hardware acceleration leverages hardware-specific APIs (think VideoToolbox, CUDA, or Direct3D) and codecs to streamline video workflows. In Rust, FFmpeg serves as a powerful foundation, and higher-level wrappers simplify its use. Here’s a quick example of hardware-accelerated transcoding in Rust:

use ez_ffmpeg::{FfmpegContext, Input, Output};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut input: Input = "test.mp4".into();
    let mut output: Output = "output.mp4".into();

    // Example for macOS: Use VideoToolbox for hardware acceleration
    input = input.set_hwaccel("videotoolbox");
    output = output.set_video_codec("h264_videotoolbox");

    FfmpegContext::builder()
        .input(input)
        .output(output)
        .build()?
        .start()?
        .wait()?;

    Ok(())
}

Breaking It Down

set_hwaccel("videotoolbox"): Activates VideoToolbox on macOS for decoding.
set_video_codec("h264_videotoolbox"): Sets the H.264 codec with VideoToolbox for encoding.
Outcome: The input test.mp4 is transcoded to output.mp4 with hardware acceleration, far outpacing CPU-only methods.

Cross-Platform Flexibility: Adapting to Different Hardware

Hardware acceleration varies across platforms and devices, posing a common hurdle for developers. Fortunately, it’s adaptable to diverse setups:

Windows: Use Direct3D 12 Video Acceleration (d3d12va) for decoding and Media Foundation for encoding:

  input = input.set_hwaccel("d3d12va");
  output = output.set_video_codec("h264_mf");

Nvidia GPUs: Pair CUDA decoding with NVENC encoding:

  input = input.set_hwaccel("cuda").set_video_codec("h264_cuvid");
  output = output.set_video_codec("h264_nvenc");

Heads-Up: Availability depends on your hardware and system. Nvidia GPUs require proper drivers, while VideoToolbox needs Apple hardware. Always verify compatibility before diving in.

Wrapping Up

From speeding up video workflows to cutting resource usage, hardware acceleration is a must-have in today’s development toolkit. Rust’s clean APIs and strong ecosystem make it a breeze to harness this power for high-performance video processing. If you’re on the hunt for a practical solution, check out open-source projects like ez-ffmpeg—it’s a solid launchpad for Rust developers looking to dive in.

🔗 Explore More: ez-ffmpeg

Mastering Video Effects in Rust: Unleashing FFmpeg and OpenGL Like a Pro

Yeauty YE — Mon, 31 Mar 2025 13:51:27 +0000

Introduction: The Video Effects Struggle Is Real

Video effects are everywhere—think short-form video filters, live-stream beauty enhancements, cinematic color grading, or even AI-generated dynamic visuals. They’re the secret sauce that hooks viewers. But let’s be honest: building them as a developer can feel like a nightmare. Sound familiar?

Bulk Processing Woes: Need to slap a blur on hundreds of videos? Manual tools like Premiere or After Effects laugh in your face—way too slow.
FFmpeg Command-Line Hell: Sure, it’s powerful, but those endless cryptic flags? One typo, and you’re debugging for hours.
C API Madness: Calling FFmpeg’s C API directly? Good luck with FFI headaches and memory leaks that haunt your dreams.
Dynamic Effects Bottleneck: Want real-time wave distortions or cyberpunk flickers? CPU-only processing crawls, and GPU acceleration feels like a distant dream.

If you’re a Rust developer, you’ve likely hit these walls. In this post, I’ll show you how to wield Rust, FFmpeg, and OpenGL to craft video effects with elegance and speed—basic filters to hardcore dynamic visuals, all in a few minutes. Let’s dive in.

Why Rust + FFmpeg + OpenGL? The Holy Trinity of Video Magic

Rust brings safety and performance. FFmpeg is the Swiss Army knife of multimedia. OpenGL unlocks GPU power. Together, they’re a dream team for video effects—efficient, scalable, and developer-friendly. Here’s how to make them work for you.

Get Started: Video Effects in 3 Steps

Let’s say you’ve got an input.mp4 and want to add a Gaussian blur, saving it as output.mp4. Here’s the no-nonsense breakdown:

1. Set Up FFmpeg

Not installed yet? It’s quick:

macOS:

  brew install ffmpeg

Windows:

  vcpkg install ffmpeg
  # First-time vcpkg users: set the VCPKG_ROOT env variable

2. Add Rust Dependencies

In your Cargo.toml:

[dependencies]
ez-ffmpeg = "*"

3. Write the Code

Blur that video in a handful of lines:

use ez_ffmpeg::{FfmpegContext, Output};

fn main() {
    FfmpegContext::builder()
        .input("input.mp4")           // Input file
        .filter_desc("boxblur=10:10") // Gaussian blur
        .output("output.mp4")         // Output file
        .build().unwrap()
        .start().unwrap()
        .wait().unwrap();
}

Run it, and boom—output.mp4 has a slick blur. Chained API calls, no FFmpeg flag memorization required. Done.

5 Real-World Use Cases and How to Nail Them

Video effects vary wildly by project. Here are five practical scenarios with code snippets every dev can relate to:

1. Privacy Blur: Gaussian Effect

.filter_desc("boxblur=10:10")

Problem: Live streams or videos need obscured backgrounds or sensitive areas.
Result: Smooth, professional blur—privacy secured, visuals intact.

2. Retro Vibes: Black-and-White Filter

.filter_desc("format=gray")

Problem: Need a nostalgic look for a video reel, but manual grading takes forever.
Result: Instant monochrome magic, old-school feels in one line.

3. Cinematic Polish: Color Grading

.filter_desc("curves=all='0/0 0.5/0.75 1/1'")

Problem: Post-production color tweaks are a slog, especially in bulk.
Result: Film-like brightness and contrast, effortlessly applied.

4. Pixel Art Style: Mosaic

.filter_desc("scale=iw/10:ih/10,scale=iw*10:ih*10")

Problem: Want a stylized look or subtle obfuscation without heavy tools.
Result: Pixelated charm, perfect for art or privacy.

5. Sci-Fi Twist: Negative (Night Vision)

.filter_desc("negate")

Problem: Craving a futuristic vibe but stuck on implementation.
Result: Colors inverted, night-vision style—done in a snap.

These FFmpeg filters are lightweight, fast, and ideal for batch jobs or quick prototypes.

Level Up: Dynamic Effects with OpenGL

Static filters not cutting it? Crank things up with OpenGL and GLSL shaders for real-time, GPU-accelerated effects.

Use Case 1: Wavy Distortion

GLSL shader (fragment.glsl):

#version 330 core
in vec2 TexCoord;
out vec4 FragColor;
uniform sampler2D screenTexture;
uniform float playTime;

void main()
{
    vec2 uv = TexCoord;
    uv.x += sin(playTime + uv.y * 10.0) * 0.02; // Wavy motion
    vec3 texColor = texture(screenTexture, uv).rgb;
    FragColor = vec4(texColor, 1.0);
}

Rust code:

use ez_ffmpeg::opengl::opengl_frame_filter::OpenGLFrameFilter;
use ez_ffmpeg::{FfmpegContext, Output, AVMediaType, filter::frame_pipeline_builder::FramePipelineBuilder};

fn main() {
    let fragment_shader = include_str!("../fragment.glsl");
    let frame_pipeline: FramePipelineBuilder = AVMediaType::AVMEDIA_TYPE_VIDEO.into();
    let filter = OpenGLFrameFilter::new_simple(fragment_shader.to_string()).unwrap();
    let frame_pipeline = frame_pipeline.filter("wave", Box::new(filter));

    FfmpegContext::builder()
        .input("input.mp4")
        .output(Output::from("output.mp4").add_frame_pipeline(frame_pipeline))
        .build().unwrap()
        .start().unwrap()
        .wait().unwrap();
}

Problem: Static effects feel flat; users want motion.
Result: A rippling wave effect—think water or dreamy sequences.

Use Case 2: Cyberpunk Flicker

GLSL shader:

#version 330 core
in vec2 TexCoord;
out vec4 color;
uniform sampler2D screenTexture;
uniform float playTime;

void main()
{
    vec3 texColor = texture(screenTexture, TexCoord).rgb;
    float r_shift = sin(playTime * 1.5) * 0.2 + 0.8;
    float g_shift = cos(playTime * 2.0) * 0.2 + 0.8;
    float b_shift = sin(playTime * 1.8) * 0.2 + 0.8;
    texColor.r *= r_shift;
    texColor.g *= g_shift;
    texColor.b *= b_shift;
    float brightness = sin(playTime * 2.5) * 0.1 + 0.95;
    texColor *= brightness;
    color = vec4(texColor, 1.0);
}

Problem: Short-form content needs eye-catching flair; traditional tools are clunky.
Result: Pulsing colors and flickers—pure cyberpunk energy.

Why This Combo Wins

Scalability: Process hundreds of videos with code, not clicks.
Developer Joy: Chained APIs beat command-line chaos.
Performance: GPU via OpenGL smokes CPU-only workflows.
Safety: Rust handles memory, sparing you C’s nightmares.

Wrap-Up: Your New Rust Superpower

Whether it’s filters for TikTok-style apps, live-stream enhancements, or AI video pipelines, Rust with FFmpeg and OpenGL delivers. Professional effects in a few lines, GPU speed boosts, and no low-level grunt work—every dev’s dream.

Curious to dig deeper? Check out this open-source gem:

🔗 https://github.com/YeautyYE/ez-ffmpeg

Simplify Video Frame Rate Adjustment in Rust: A Practical Guide for Developers

Yeauty YE — Fri, 28 Mar 2025 15:44:09 +0000

Introduction

Adjusting video frame rates (FPS) is a common task in video processing, whether to ensure smooth playback or meet specific platform requirements. For instance, converting a 60 FPS video to 30 FPS might be necessary for compatibility or efficiency. While the FFmpeg command-line tool (ffmpeg -i input.mp4 -r 30 output.mp4) is a popular choice, integrating this functionality directly into Rust projects can offer more control and automation.

In Rust, calling FFmpeg’s C libraries via FFI (Foreign Function Interface) provides access to its full capabilities, though it can be complex and time-consuming. For simpler tasks like frame rate adjustments, ez-ffmpeg offers a more straightforward approach. Built on top of FFI, it provides a user-friendly interface that streamlines common video processing tasks, making it a practical option for developers looking to avoid the intricacies of direct FFI calls.

FFI vs. ez-ffmpeg: Choosing the Right Tool

The Challenges of FFI

FFI allows direct interaction with FFmpeg’s C libraries, offering maximum flexibility:

Full Control: Access every feature of FFmpeg, ideal for advanced use cases.
Complexity: Requires managing structs, pointers, and memory manually, which can lead to verbose and error-prone code.
Steep Learning Curve: Developers need a solid understanding of FFmpeg’s low-level API.

For example, adjusting frame rates via FFI involves configuring AVCodecContext and AVFrame, where mistakes can easily lead to issues.

How ez-ffmpeg Helps

ez-ffmpeg is designed to simplify common tasks without replacing FFI entirely:

Ease of Use: Its API mirrors FFmpeg command-line syntax, making it intuitive for those familiar with the tool.
Safety: Handles memory management automatically, reducing risks.
Focus: Lets developers concentrate on application logic rather than FFmpeg internals.

While FFI is better for intricate, custom workflows, ez-ffmpeg shines in scenarios where simplicity and speed are priorities, such as basic frame rate adjustments.

Get Started: Adjust Video Frame Rates in Rust

Let’s say you need to convert a 60 FPS video to 30 FPS. Here’s how ez-ffmpeg can help.

1. Install FFmpeg

macOS:

brew install ffmpeg

Windows:

vcpkg install ffmpeg
# Set the VCPKG_ROOT environment variable if using vcpkg for the first time

2. Add the Rust Dependency

In your Cargo.toml:

[dependencies]
ez-ffmpeg = "*"

3. Code Examples

Below are two ways to adjust frame rates using ez-ffmpeg:

Method 1: Set Frame Rate via Output

use ez_ffmpeg::{FfmpegContext, Output};
use ffmpeg_sys_next::AVRational;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    FfmpegContext::builder()
        .input("test.mp4") // Input file
        .output(Output::from("output.mp4").set_framerate(AVRational { num: 30, den: 1 })) // Set to 30 FPS
        .build()?
        .start()?
        .wait()?;
    Ok(())
}

Method 2: Use a Filter to Set Frame Rate

use ez_ffmpeg::{FfmpegContext, Output};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    FfmpegContext::builder()
        .input("test.mp4") // Input file
        .filter_desc("fps=30") // Use "fps" filter to set 30 FPS
        .output(Output::from("output.mp4")) // Output file
        .build()?
        .start()?
        .wait()?;
    Ok(())
}

When to Use Each Method

Method 1: Straightforward for basic frame rate changes. Best when that’s the only adjustment needed.
Method 2: More flexible, allowing frame rate changes alongside other filters (e.g., resizing). Ideal for combined operations.

Both methods produce a 30 FPS output.mp4 from test.mp4.

Conclusion

For Rust developers working with video, ez-ffmpeg provides a convenient way to handle tasks like frame rate adjustments. It builds on FFI to offer a simpler, safer interface for common operations, complementing the power of direct FFmpeg calls. While it’s not a solution for every scenario, it can be a valuable tool in your toolkit, especially when ease of use is a priority.

🔗 Learn more: ez-ffmpeg on GitHub

DEV Community: Yeauty YE

The state of media processing in Rust (2026): what each crate actually covers

The landscape at a glance

Path 1: std::process::Command + the ffmpeg binary

Path 2: ffmpeg-sidecar — the subprocess, made livable

Path 3: the raw bindings — ffmpeg-next and rsmpeg

Path 4: video-rs — high-level video, self-described work in progress

Path 5: symphonia + rubato — pure Rust until the codec wall

Path 6: ez-ffmpeg — the in-process runtime

Who should not use ez-ffmpeg

The set-difference that 0.14 targets

When shelling out is the right answer

Picking by task

In-process HLS packaging in Rust: fMP4 segments, an ABR ladder, no ffmpeg subprocess

The first decision: fMP4 or TS

The minimal runnable path: single-bitrate fMP4

Mapping the options (and the step I got wrong first)

Probe before you build

The ABR ladder: one decode, N renditions

Two craftsmanship details (the part that is not in the README)

The run, verbatim

When not to use this

Next

Pushing your own frames into an FFmpeg pipeline in Rust (headless frame source)

The workaround, costed

The in-process version

How it works: the four contracts

Run it

What the ecosystem already had

When you should not use this

What the first build costs

Where this leaves you

Extracting video frames in Rust, in-process — no CLI, no hand-written decode loop

The two old ways: cross a process, or hand-write the decode loop

FrameExtractor: a batch of frames in one line

Sampling: how many frames, and which ones

Color: HD converts as BT.709 by default, and that is not a footnote

Who should not use it

Closing

A CLI-compat layer that refuses half your commands, on purpose

from_cli: paste the command, run it in-process

emit_rust_code: or translate it to code

The contract: classify everything, approximate nothing

A few honest gates

Beyond the subset: the builder is the full surface

Who should not use it

Closing

Feed audio to Whisper in Rust without ever shelling out to ffmpeg

The pipeline everyone actually ships

One call, in-process

The trap nobody documents: whisper-rs 0.16 API drift

Run it

When you should not use this

What the first build costs

Where this leaves you

🚀 Double Your Performance with One Line of Code? The Memory Superpower Every Rust Developer Should Know!

🧠 1. Memory Allocators: Your Program’s Invisible Power Broker

❌ The Problem with Traditional Allocators: Global Lock Contention

✅ Modern Allocators to the Rescue: Thread-Local Caches

🔍 2. Why Does Swapping Allocators Make You So Much Faster?

🔒 2.1 Escaping the Global Lock Hell

🧩 2.2 Fighting Memory Fragmentation

⚡ 2.3 Optimizing Large Allocations

🔧 3. Rust Made It Simple: One Line to Swap Your Allocator

✨ 4. The Ultimate Solution: auto-allocator—No More Conditional Compilation Hell

🏁 5. Wrapping Up: Time for a Free Performance Upgrade

Implementing FFmpeg Filters with Rust: A New Approach to Video and Audio Processing

Why This Matters: The Pain of Traditional FFmpeg

What You’ll Learn

Getting Started: Setup

Example 1: Brighten Your Video (YUV420)

The Code

What’s Happening?

Example 2: Grayscale with GPU Power (OpenGL)

The Shader

The Code

Why It Rocks

Example 3: Lower the Volume (Audio)

The Code

How It Works

Path 1: `std::process::Command` + the ffmpeg binary

✨ 4. The Ultimate Solution: `auto-allocator`—No More Conditional Compilation Hell

Why `ez-ffmpeg` Wins