Blazar Video Encoder for Generative Video

Lately I've been using video diffusion models and learning more about them. As I was digging around the various codebases I noticed that there were a couple of common approaches to actually encoding the frames that are generated by the diffusion models: mainly use PyAV or shell out to FFmpeg. There's nothing especially wrong with either of these approaches but I think it is possible for us to do a little better. So I (along with my robot friends) built Blazar. You can find the code repo for Blazar here.

Today with PyAV the flow is roughly:

1. Convert tensor from BF16 or F32 to u8 and reshape to (H, W, C)
2. D2H transfer
3. RGB to YUV conversion
4. (if NVENC) upload to the GPU
5. Encode

So, I think, this is roughly what we want to do:

1. Be able to take the native output tensor shape of the model as an input. (B, C, T, H, W) seems to be the common tensor shape but a video encoder will want more like (B, T, H, W, C) or (B, T, C, H, W) (or whatever nv12 is with U' and V' interleaved)
2. Do a fused YUV + tensor reshape conversion (does that count as a reshape? maybe not) either on GPU or CPU
3. Encode either using NVENC (or some other HW encoder) or a variety of software encoders
4. Be relatively easily configurable...
5. Probably don't get in the way of the Python GIL

So the robot and I landed on a design where most of the work happens in Rust. Backends for YUV conversion and encoding are swappable between hardware and software so that if you would prefer not to use HW for YUV conversion and encoding you don't have to do that. There is a concept of "Calibrations" that are keyed on model and target quality so you can have multiple calibrations with different latency or quality targets and make them configurable by model. These live in human readable TOML files.

import blazar

cals = blazar.Calibrations.load("calibrations/default")
with blazar.Encoder(calibrations=cals, identity={"family": "wan", "version": "2.1",
                                                 "quant": "bf16", "size": "1.3b"},
                    target="balanced", backend="nvenc", codec="h264",
                    output="out.mp4") as stream:
    stream.submit(frame, width=W, height=H, fps=24.0, layout="thwc", range="unit01")
    # or, with device tensors:
    stream.submit(tensor, fps=24.0, range="unit01")

Here's a pretty simple example of using Blazar to encode frames. You can submit them one at a time or as a chunk of video, in which case they'll be mass converted to YUV and then encoded.

Peformance Comparison

This comparison table was done on a Core i7 14700KF with a 4080 Super in WSL. What we are measuring here is a BF16 (1, 3, 121, 1088, 1920) tensor being passed into the encoder pipelines for these scenarios:

1. Blazar device bf16 (BF16 input tensor, GPU YUV conversion, NVENC)
2. Blazar device u8 -> CPU YUV Conversion -> NVENC
3. PyAV device u8 -> CPU YUV Conversion -> NVENC
4. FFmpeg device u8 -> IPC -> CPU YUV Conversion -> NVENC

FFmpeg CLI is a bit of a strange one but I did see this happening out in some projects so I wanted to include it for that reason.

Path	What it does from the device tensor	720p	1088p
blazar, device tensor	kernel writes NV12 into the surface NVENC reads	1.10 ms (909 fps)	2.20 ms (455 fps)
blazar, host path	quantise → copy to host → CPU convert → upload NV12	3.54 ms (282 fps)	5.37 ms (186 fps)
PyAV	quantise → copy to host → swscale → encode	2.73 ms (367 fps)	8.60 ms (116 fps)
ffmpeg CLI	quantise → copy to host → pipe raw → swscale → encode	not measured	8.24 ms (121 fps)

The optimal path is the device bf16 path. Here we feed the tensor directly into a CUDA kernel that simultaneously converts from RGB to YUV and reformats the buffer into the shape NVENC expects (T,C,H,W). On my 4080 Super we're able to hit 455 fps with 1088p tensors from SGLang. If we copy to host as a u8 that drops to 186 fps. Compared to the PyAV or FFmpeg CLI paths this is a fair improvement still. At smaller resolutions swscale outperforms the Blazar YUV conversion, but Blazar's YUV conversion seems to amortize better over larger frames. I expected the performance in this case would be close to PyAV's performance - but obviously there are YUV conversion implementation differences. Something to look more deeply at.

GIL Impact

Workload	fps beside a Python thread	Python loop keeps
blazar, device tensor	420.1	91%
blazar, host yuv	14.7	93%
PyAV	11.4	92% (90–93%)

Here we're running a tight loop in a second Python thread with a 1088p 121 frame video being encoded on the main thread. Really this test was to try to induce some contention in Python's GIL (Global Interpreter Lock) to test its impact on Blazar and PyAV. Blazar’s device and host paths are both impacted because there is still some work that happens within the scope of the GIL but the device path is significantly less impacted than host and PyAV paths relative to the previous table with no competing threads.

Streams	blazar device	python	blazar host	python	PyAV	python
1	420.1 fps	91%	14.6 fps	93%	11.4 fps	85%
2	822.9 fps	92%	21.1 fps	90%	15.6 fps	79%
4	885.7 fps	86%	41.1 fps	86%	28.2 fps	64%

Here we are looking at how this scales under GIL contention with more threads. Using the zero-copy device path we're limited by NVENC throughput. The encoder unit is sitting at 100% utilization so we're not able to get more out of it past about 2 threads (at least on my machine). However for the host path in Blazar and the Python path we see worsening performance for the tight python loop as the thread count increases, and PyAV is not scaling quite as well as Blazar host path. I think this is probably an area for improvement on the Blazar side too, both the device and host paths reduce the other Python thread's performance to 86% of baseline.

Where we're at today

Today the project has support for:

• Hardware encoders: NVENC (AV1, HEVC, AVC), VideoToolbox (HEVC, AVC)
• Software encoders: x264, x265, vpx (VP9), svt-av1
• Opus audio codec is currently supported
• YUV conversion on GPU or CPU
• fMP4 or Buffer output
• Custom configurations

Where I want to take this

I have ideas for the future of the project. I think if we start to see some adoption there will of course be bugs and feature requests. I think that having model-aware and scene-aware encoding might be an interesting feature. I also want to explore using latents to identify regions of interest or use other signals to inform encoding decisions.

Anyway if you are running generative video models please check this project out, I am happy to take any feedback! If you want to contribute that is even better!