ffmpeg.wasm 0.12 hung on the first frame — and the real speedup was not the upgrade

I run Utilorax, a set of free browser-based tools. The video tools are ffmpeg.wasm, which means every encode runs on the user's own machine and nothing is ever uploaded. That is the whole point of them — and it is also why they were slow.

This is what a week of trying to fix that actually taught me. The upgrade I was sure would fix it did not, and the thing that did was much less interesting and much more effective.

The starting point: one core

Every encode was pinned to a single thread:

function pinThreads(args) {
  var body = args.slice(0, -1), out = args[args.length - 1];
  return ['-threads', '1'].concat(body, ['-threads', '1', out]);
}

That looks like an oversight. The core is the multi-threaded build, ffmpeg-core.worker.js ships with it, and the page already pays for COOP/COEP so SharedArrayBuffer is available. Letting ffmpeg use every core looks like free speed.

It is not. Take the pin off on ffmpeg.wasm 0.11 and the video filter dies instantly with RuntimeError: function signature mismatch thrown out of worker.js onmessage — a known 0.11 fault where the threaded core trips over its own dynCall table. Nothing renders. 0%.

So the encoder ran on one core on machines that have eight. A 1080p60 clip filtered at about 0.09× real time. 0.12 fixed the threading fault, so 0.12 was obviously the answer.

Upgrading to 0.12: three things nobody mentions

The API changed shape, not just names — run() became exec(), the FS() calls became methods, fetchFile moved package, and everything returns a promise. Rather than rewrite sixty-one call sites twice (once to port, once to revert if it went wrong) I put an adapter in front that presents the 0.11 surface on top of 0.12. That part went fine.

Then the interesting failures started.

1. The UMD bundle is not self-contained

@ffmpeg/ffmpeg's UMD build loads its own worker as a separate webpack chunk, 814.ffmpeg.js, from whatever directory ffmpeg.js came from. It is not in the docs and it is not obvious — you just get a 404 and a load() that never resolves. Ship it alongside ffmpeg.js.

2. classWorkerURL looks like the fix for that. It is a trap.

Passing it sends the UMD build down its ESM branch:

load = ({classWorkerURL: s, ...r} = {}) =>
  this.#e || (this.#e = s
    ? new Worker(new URL(s, "file:///home/jeromewu/…"), {type: "module"})   // ← ESM branch
    : new Worker(new URL(e.p + e.u(814), e.b), {type: undefined}))          // ← classic

A module worker — around a chunk that is a classic script calling importScripts(). It dies on load. Leaving the option out is what makes it work.

3. Workers need the COEP header on themselves

A cross-origin-isolated page will refuse a worker script unless that script's own response carries Cross-Origin-Embedder-Policy: require-corp. Same-origin is not enough. Apache sends nothing on a static file, so the whole chain fails: the class worker, the core it importScripts, and the pthread workers Emscripten spawns.

<If "%{REQUEST_URI} =~ m#/js/vendor/ffmpeg-012/#">
    Header set Cross-Origin-Embedder-Policy "require-corp"
    Header set Cross-Origin-Resource-Policy "same-origin"
</If>

0.11 never needed this because it builds its worker from a blob URL, and a blob inherits the page's policies instead of being checked for headers. That is a difference in how the worker is created, not in the API — you will not find it by reading the diff.

One more: keep your service worker away from those files. It caches whatever headers an entry had when it was written, so one fetch made before the header rule existed gets served back forever, and it surfaces as a COEP error pointing at the worker rather than at the cache.

Then it hung on the first frame

Headers fixed, pin removed, and the filter printed exactly one line:

frame=    1 fps=0.0 q=0.0 size= 0kB time=00:00:00.14 bitrate= 2.6kbits/s speed=2.04x

and then nothing. No error. 0% forever.

The core pre-allocates exactly 32 pthread workers and cannot grow past them here:

getNewWorker: function () {
  if (PThread.unusedWorkers.length == 0) {
    PThread.allocateUnusedWorker();
    PThread.loadWasmModuleToWorker(PThread.unusedWorkers[0]);  // ← async
  }
  …
}

Growing the pool means loading the module into a fresh worker and waiting for it to answer — but ffmpeg runs synchronously inside the very worker that would have to process that answer. The thread it is blocked on can never start. Ask for a 33rd and everything stops, silently.

Left to itself ffmpeg asks for far more than 32. It spends threads in three places at once: one set per input decoder, one for the filter graph, and x264 takes 1.5 per core on top. On anything with nine or more logical cores you are already over the line — which is why this only appears once the pin comes off.

So: give each stage an explicit budget instead of the machine's core count.

var THREADS = Math.max(1, Math.min(4, navigator.hardwareConcurrency || 4));
// worst case ≈ (inputs + 2) × THREADS — a 5-clip merge at 4 is already 28 of the 32
['-threads', n, '-filter_threads', n, '-filter_complex_threads', n]
  .concat(body, ['-threads', n, out]);

-filter_threads matters more than it looks. It is a global option and defaults to one thread per core on its own, which was most of the overspend.

The error that ate its own error message

Two ops aborted instead of hanging, and every one of them reported:

TypeError: Cannot read properties of undefined (reading 'startsWith')

That is not your bug. It is the core's own error handler:

try { Module["_ffmpeg"](args.length, stringsToPtr(args)) }
catch (e) { if (!e.message.startsWith("Aborted")) { throw e } }

e is not always an Error. When it isn't, e.message is undefined, and the handler throws while inspecting the crash — so whatever ffmpeg actually said is lost. If you see that string, the real failure is upstream and you cannot see it.

What I actually shipped

I put the tools back on 0.11.

Not because 0.12 is bad, but because the premise was wrong. 0.12 is worth a few times one core. A 6-minute 1080p60 clip is about 21,000 frames, and software x264 in wasm will not get through that quickly at any thread count. I was optimising the wrong axis.

The real win was not running the encoder at all.

• Container conversion — MKV→MP4 and friends are the same H.264 in a different box. -c copy instead of decode-and-re-encode: seconds, and no second round of lossy compression. Judge the output, not the promise — ffmpeg.wasm resolves even when ffmpeg gave up, so try the copy and fall back if the file is missing or tiny. Trying the copy is the probe, and it is cheaper than the transcode it replaces.
• Rotate — a quarter turn is a flag on the video track, not a change to the pixels. Stream copy plus -metadata:s:v:0 rotate=90.
• Trim — copy the streams instead of rebuilding them. The cut then lands on a keyframe, so it is a real trade rather than a free win: make it a checkbox.

Rotate has a catch worth knowing. The risk is not that the copy fails, it is that it silently does nothing: phone footage often already carries a rotation, and when the streams are copied that existing flag can survive and override the one you set. The file comes out perfectly valid, and unturned. So measure the output rather than trusting it — a <video> applies a container's rotation when it reports videoWidth, so a turn that worked has the sides swapped:

function displayedSize(data, mime) {
  return new Promise(function (resolve) {
    var url = URL.createObjectURL(new Blob([data.buffer], {type: mime}));
    var v = document.createElement('video'), done = false;
    var settle = function (r) { if (!done) { done = true; URL.revokeObjectURL(url); resolve(r); } };
    setTimeout(function () { settle(null); }, 8000);   // an answer that never comes is the freeze you were avoiding
    v.onloadedmetadata = function () { settle({w: v.videoWidth, h: v.videoHeight}); };
    v.onerror = function () { settle(null); };
    v.src = url;
  });
}

No swap, no proof, re-encode.

What I would tell myself a week ago

Threads were the interesting problem. Not encoding was the useful one. The ops that got fast — convert, rotate, trim, mute — got fast because they stopped doing work, not because they did it on more cores.

And when the number of threads is the thing you are tuning, check what the runtime can actually give you before you ask for more. A pool of 32 that cannot grow is a hard ceiling, and the failure mode is a silent hang rather than an error — which is a bad way to find the edge.

For the ops that genuinely must decode every frame — filters, resize, compress — wasm is still wasm, and the next honest step is WebCodecs and the hardware encoder. That is a different post.