DEV Community

szp2005
szp2005

Posted on Fully Autonomous

ONNX Runtime Web fixes its thread count once per page, so our timeout retry had to start a new Worker

#webdev #javascript #webassembly #machinelearning

Our background remover runs a BiRefNet model in the browser with ONNX Runtime Web 1.27 on the WASM backend. Multithreaded WASM is the fast path when the page can have it. Sometimes it can't: the worker script gets blocked somewhere on the network, or session creation just hangs. The obvious fix is to catch the timeout and call InferenceSession.create again with one thread.

In ORT-web that retry goes nowhere, and the reason is easy to miss.

Why the same-page retry is a dead end

The WASM backend is a page-level singleton. The thread count has to be settled before the first create in that context, and once the runtime is up, setting ort.env.wasm.numThreads again doesn't give later sessions a different thread pool. So "try multithreaded, fall back to single-threaded" can't happen inside one JavaScript context. You either decide before the first create, or you throw the context away.

We put that at the top of the module that makes the decision, so nobody "fixes" it later (comment translated from Chinese):

/**
 * ORT's WASM backend is a page-level singleton. Thread count must be
 * decided before the first create, so this never returns a
 * "retry single-threaded on the same page" plan.
 */
export function decideWasmSessionInit({ configuredThreads, storedProbeResult }) {
  if (configuredThreads <= 1 || storedProbeResult === "unavailable") {
    return { mode: "single-thread" };
  }
  return { mode: "multi-thread" };
}
Enter fullscreen mode Exit fullscreen mode

Deciding before the first create

The thread count comes from one expression:

const configuredThreads = crossOriginIsolated
  ? Math.min(8, navigator.hardwareConcurrency || 4)
  : 1;
Enter fullscreen mode Exit fullscreen mode

If the page isn't cross-origin isolated (no COOP/COEP headers), you get one thread no matter how many cores the machine has. That's the first thing I'd check when someone says ORT-web feels slow. If it is isolated, we cap at 8 and assume 4 when hardwareConcurrency is missing.

Isolation alone isn't enough, because ORT's threads are workers and a worker script can still fail to load. So before the first create we start a tiny probe worker whose entire source is postMessage("ok") and give it 2.5 seconds. An "ok" means multithreaded. An error or a timeout means single-threaded. Either result goes into sessionStorage, so a tab probes once. If sessionStorage throws, as it can in restricted private modes, we still use this run's probe result; it just isn't remembered.

When create hangs anyway

The probe can pass and session creation can still stall. We race create against a 20-second timer. Normal WASM init takes seconds, so hitting that limit usually means a worker request got stuck in the network layer. ORT has no way to abort a create, so if the abandoned one finishes late, we release that session right away instead of leaking it.

Recovery is where the singleton rule bites. The matting code runs in its own dedicated Worker. On a timeout, that worker reports a WasmSessionInitTimeoutError. The client sets a forceSingleThread flag and terminates the Worker, and its heap and threads go with it. The next request gets a fresh Worker that skips the probe and runs single-threaded. The page doesn't reload, so whatever the user uploaded or edited stays where it was.

What it costs

The request that timed out still fails. The worker client doesn't retry it on its own, because retrying in place would mean reusing the stuck runtime, and the only other option is reloading the page and losing the user's state.

Single-threaded is slower. I haven't measured by how much on real devices, so I'm not going to put a number on it.

One more thing to watch for. The same module still exports a second recovery helper that writes the single-thread flag, reloads the page, and keeps a guard so it only self-heals once. Nothing imports it. Read the module alone and you'd conclude we reload on timeout. We don't. To know which recovery path is live, look at what the caller imports, not at what the helper file offers.

If you run models with ORT-web, this is what I'd take from it:

  • When performance looks wrong, check crossOriginIsolated before anything else.
  • Decide the thread count once, before the first create.
  • After an init timeout, move to a new Worker or a new page, with a guard against loops. Don't rebuild in place.

The cutout tool this comes from: https://koutuxia.com/?utm_source=devto

Disclosure: I build Koutuxia.

Top comments (0)