What to time when benchmarking in-browser inference, after WebGPU returned empty masks

Does WebGPU beat WASM for in-browser video matting? The first results table in this test said yes, 35.0 ms per 1280×720 frame on WebGPU against 87.2 ms on WASM with 4 threads. The output check said otherwise. By frame 13 the WebGPU alpha mask had 0.00% foreground pixels, while WASM had 5.03%, and nothing threw or warned. The benchmark had timed a backend that quickly computed an empty mask, hence a checklist instead of a speed chart.

The model is the 7.5 MB RVM MobileNetV3 FP16 file that the video matting tool on ImgIng https://imging.ai/ loads for its lightest tier, run through ONNX Runtime Web 1.27.0 in desktop Chromium on an Apple M4 Mac. The input is one synthetic frame (a person from a public-domain VOA green screen clip on a self-made bookshelf background, about 5% of the frame) at 512×288, 1280×720 and 1920×1080, with the recurrent state fed back frame to frame like a video. WebGPU and WASM 4 threads got 3 separate browser processes per size, tables use the median; WASM 1 thread ran once.

Report the stages as separate numbers

A single "ms per frame" hides three costs. Session creation took 270 to 290 ms on both backends. The first inference was 119 to 144 ms on WebGPU and 214 to 269 ms on WASM 4 threads, so cold start came to 406 to 427 ms against 492 to 544 ms. The steady median of frames 2 to 13 was 29.4 / 35.0 / 43.6 ms on WebGPU, 89.4 / 87.2 / 91.8 ms on WASM 4 threads and 228.3 / 251.2 / 279.3 ms on 1 thread. Every WebGPU figure here comes from runs whose mask was empty, so it is the cost of the computation, not of a usable result. Split this way, the cold start gap is only about 100 ms. Resolution barely moved inference either: 14 times the pixels took WASM 4 threads from 89.4 to 91.8 ms, likely because RVM shrinks its backbone input to about a 512 px long side via downsample_ratio (an inference from the architecture, not timed per layer). One more thing showed up only once: a first-ever WebGPU run on this machine can be much slower, so it shouldn't be averaged in quietly.

Stop the clock after the read-back

The timer here stops after the alpha has been read back into JavaScript with getData(), not when run() resolves. Until then nothing can be drawn or checked. This benchmark never timed a run without the read-back, so that slice has no number, but stopping earlier measures something other than what a user waits for.

Check the output before the speed

Both backends returned float16 in a Float16Array, which rules out a read-out mismatch. At 1280×720 the first WebGPU frame kept 0.54% foreground against 5.04% on WASM, and by frame 13 it was 0.00% at all three sizes. The larger RVM ResNet50 also came out wrong on WebGPU, about 1 to 3.5% foreground against about 5% on WASM. The cause wasn't investigated. The guess is operator handling for this FP16 recurrent network; another matting model on the same runtime matched on both backends, so this is "this model, this machine" only. ImgIng had landed in the same place: its two RVM tiers are pinned to WASM, and the page says that avoids some WebGPU devices silently producing empty mattes.

Know which backend actually ran

Listing ['webgpu', 'wasm'] looks safe. Then take a browser where navigator.gpu exists but requestAdapter() returns null. With ['webgpu'] alone, session creation throws after about 111 ms. With both, it quietly runs on WASM and the code gets no hint. WASM threads also need cross-origin isolation (COOP plus COEP), otherwise you get 1 thread, 2.6 to 3.0 times slower than 4 here.

The checklist

1. Report session create, first frame, cold start and steady median separately, and flag a first-ever run.
2. Stop the clock after the output is read back into JS.
3. Decide whether post-processing is inside the timer, and test at your largest resolution. Here a per-frame FP16-to-Float32 conversion in JS added roughly 77 to 88 ms at 1920×1080.
4. Count pixels with alpha above 0.5 on frame 1 and a later frame for each backend before trusting any speed.
5. Create one session per backend, check requestAdapter(), and confirm crossOriginIsolated before calling it "4 threads".

What this test can't close is the empty mask itself. It covers one Mac and Chromium only, and on a Windows box with a discrete GPU the WebGPU column might be fast and correct, which is exactly why item 4 is there.