DEV Community

Lank_M
Lank_M

Posted on

One runtime, one GPU, two matting models: one matched WASM to 1.3e-6, the other returned an empty mask

The table looked like a clean result for WebGPU. Same model, same frame, same machine: Robust Video Matting with the MobileNetV3 backbone, FP16, 7.5 MB, needed 35.0 ms per steady-state frame on WebGPU and 87.2 ms on 4-thread WASM at 1280×720. Then came the alpha it had produced. By frame 13 the WebGPU mask had 0.00% foreground pixels against 5.03% on WASM, and nothing had complained. Session creation succeeded, every run() resolved, and both backends handed back a Float16Array. Those 35 ms were the time it took to compute a wrong answer.

All of this is one machine: an Apple M4 Mac, desktop Chromium, ONNX Runtime Web 1.27.0. The RVM file and its settings match what the video matting page in ImgIng https://imging.ai/ (a beta feature for desktop browsers) ships in its client script, with full frames going in, downsample_ratio at 512 over the long edge, and recurrent state carried from frame to frame. The benchmark copied those settings from the script so it would match a real deployment. Steady state is the median of frames 2 to 13, and the WebGPU and 4-thread WASM figures are medians of 3 separate browser processes.

The faster backend returned an empty mask

At 512×288, 1280×720 and 1920×1080, WebGPU took 29.4, 35.0 and 43.6 ms per frame, 4-thread WASM 89.4, 87.2 and 91.8 ms, and single-thread WASM 228.3, 251.2 and 279.3 ms (one run). Every WebGPU number there belongs to unusable output. On frame 1 its mask kept 1.85%, 0.54% and 0.37% foreground against about 5.05% on WASM, and after twelve rounds of state feedback it was 0.00% at all three sizes. The ResNet50 variant ran 51, 86 and 95 ms on WebGPU against 307, 504 and 512 ms on WASM, and its WebGPU output was wrong too, just not empty: 1.9 to 3.5% foreground on frame 11 where WASM stayed near 5.0%. A thin mask is harder to spot than a blank one.

The same runtime got another model right

On September 28, same machine, same ORT build, an earlier run had timed ISNet still-image matting at 1024×1024. The INT8 version ran 359 ms on WebGPU and 2133 ms on 4-thread WASM, and the two alphas matched to a maximum per-pixel difference of 1.3×10⁻⁶.

So the device and the runtime stayed put, and only the model changed. A rule like "adapter found, use WebGPU" would have been right for ISNet and silently wrong for RVM on the same machine. The cause hasn't been traced. My guess is how the WebGPU path handles some operators or FP16 precision in this recurrent network, but there was no operator-by-operator comparison, so all I can say is that this model on this machine produced an empty mask on WebGPU. On Windows with a discrete GPU, Safari or Firefox, RVM might come out fine. None of those were tested.

A non-empty check on frame 1 would have passed

At 1280×720 the first WebGPU frame still had 0.54% foreground, so an "is the mask all zeros" test passes. It took a dozen frames of recurrent state to drive it to zero. What caught it was comparing against WASM on the same input: mean absolute difference 0.050 to 0.069, with 5.0 to 6.8% of pixels off by more than 0.5. For a temporal model the reference has to span several frames with state carried over. The speed table also hid a smaller thing: converting the alpha from FP16 to 32-bit floats in JavaScript added roughly 77 to 88 ms per 1080p frame, longer than the 43.6 ms WebGPU inference beside it, which was producing an empty mask anyway.

Pick the backend per model and verify it once

From the outside, ImgIng already splits it this way. Its two RVM tiers are pinned to WASM, and the page says this avoids some WebGPU devices silently producing empty masks. Its heaviest tier, MatAnyone2, tries WebGPU first and switches to WASM with a progress message if that fails. That's a per-model policy, and after these numbers I think it's the right shape.

If I were writing the loader, the device check would stay but stop being the decision. navigator.gpu existing isn't enough, since requestAdapter() can return null, and passing ['webgpu', 'wasm'] falls back silently without telling you which backend ran. I'd create the WebGPU session explicitly, push a bundled 13-frame reference clip through it, compare against stored WASM alphas, and cache the verdict per model hash and adapter. Session creation took about 270 to 290 ms on either backend for the 7.5 MB model, so that's roughly one extra cold start on first use.

The column I now put next to the milliseconds is maximum difference against WASM after state has been carried through. Without it, the RVM rows above read as a 2.1 to 3.0× win for a backend whose mask was empty.

Top comments (0)