Replacing ffmpeg.wasm with WebCodecs: a 1080p export from 10.3s to 2.6s

I maintain a browser-based video editor. Everything runs client-side — no upload, no server — which for a long time meant ffmpeg compiled to WebAssembly doing the export.

It works. It's just slow. ffmpeg.wasm encodes around 30 frames a second at 1080p, so a ten-second clip takes about ten seconds of watching a progress bar. Not because ffmpeg is slow — because it's a copy of an encoder running in a sandbox, without access to anything the platform already has.

The browser does have a real encoder. VideoEncoder is the same one it uses for WebRTC. Here's what swapping to it involved, and the three bugs that cost me most of the time.

The result first

             ffmpeg.wasm    WebCodecs
  720p  10s      7.7s         2.0s      3.9x
 1080p  10s     10.3s         2.6s      4.0x

Output is pixel-identical to the ffmpeg path — same compositing code, just fed differently.

The shape of it

Four stages, all of them streaming:

mp4box  →  EncodedVideoChunk  →  VideoDecoder  →  VideoFrame
                                                     ↓
                                          composite offscreen
                                                     ↓
                                   VideoEncoder  →  mux

Demuxing. VideoDecoder wants encoded chunks, and a browser won't give you those from a <video> element. mp4box.js parses the container and hands you samples. The part that isn't obvious: the decoder also needs the codec-private data, and that doesn't live in the samples — it's in the sample description box.

function describe(mp4, track) {
  const trak = mp4.getTrackById(track.id);
  for (const entry of trak.mdia.minf.stbl.stsd.entries) {
    const box = entry.avcC || entry.hvcC || entry.av1C || entry.vpcC;
    if (!box) continue;
    const stream = new DataStream(undefined, 0, DataStream.BIG_ENDIAN);
    box.write(stream);
    // first eight bytes are the box header, which the decoder does not want
    return new Uint8Array(stream.buffer, 8);
  }
}

Miss that , 8 and the decoder rejects the config with a message that tells you nothing.

Compositing. I didn't write a second renderer. The editor's preview already draws every clip, title, transition and colour adjustment to a canvas. For export I run that same code offscreen at output resolution, with its frame source swapped from <video> elements to decoded VideoFrames. Which means the export looks like the preview by construction, rather than by me keeping two renderers in sync for ever.

Bug 1: the decoder stops if you hold its output

VideoDecoder applies back-pressure. If you don't close the VideoFrames it gives you, it stops producing. My first feeder queued decoded frames generously — eight in flight, plenty buffered — and the export hung about four seconds in, every time.

The decoder was waiting for me to release frames. I was waiting for the decoder to produce one more. Neither moved.

while (this.at < this.chunks.length
       && this.decoder.decodeQueueSize < 4
       && this.queue.length < 3) {
  this.decoder.decode(this.chunks[this.at++]);
}

Shallow queues. Counter-intuitive if you're used to buffering being free.

Bug 2: setTimeout(0) is not zero

The frame loop yields to let the decoder run. I used setTimeout(0), as you do.

Browsers clamp nested timers. In a real page that yield costs about 7 ms. Over an export that yields tens of thousands of times, that was most of the runtime I'd supposedly saved.

function yieldToTasks() {
  return new Promise((resolve) => {
    const channel = new MessageChannel();
    channel.port1.onmessage = () => resolve();
    channel.port2.postMessage(0);
  });
}

0.04 ms, and not clamped. Same semantics, 175x cheaper.

Bug 3: one character

The loop asks "does the frame I'm holding cover the moment I need?" I wrote:

if (frame.timestamp + frame.duration > want) break;

A frame at 66,666 µs lasting 33,333 µs covers exactly up to 99,999 — which is precisely the moment the next output frame asks for. With a strict >, it could neither advance nor finish. It waited for ever, on some files and not others, depending on where their timestamps started.

>=. One character, an afternoon.

What actually found it was giving up on guessing and making the timeout print its own state:

throw new Error('The decoder stopped producing frames. '
  + `[want=${want} frame=${this.frame?.timestamp} queue=${this.queue.length} `
  + `at=${this.at}/${this.chunks.length} inflight=${this.decoder.decodeQueueSize}]`);

The numbers made it obvious in seconds. I should have done it first.

What it still can't do

• No hardware encoder. VideoEncoder doesn't reach your GPU's dedicated encoder, so desktop ffmpeg is still faster.
• Container support is the browser's. ProRes, some MKVs, HEVC in some browsers — those fall back to the ffmpeg path, which is why I kept it.
• Memory is the tab's. No server to offload to. That's the trade you make for not uploading anything.

Worth it?

For a browser-only editor, yes — four times faster on the operation users actually wait for. And the fallback matters: WebCodecs where it works, ffmpeg.wasm where it doesn't, decided per file.

If you're doing something similar: shallow queues, MessageChannel not setTimeout, and make your timeouts print their own state before you start guessing.

The editor is at whiztools.app/video-editor if you want to throw a file at it.