WebLLM in Production: Cold Starts, Service-Worker Blocking, and the Missing 14B WebGPU Build

Why We Evaluated WebLLM in Our Lab

The scenario we modeled is a generic in-editor code assistant, not a named customer deployment. On a hosted API that is a line item they can live with. On their own always-on GPU box it is a line item plus a pager rotation plus a capacity planning meeting every time a customer has a launch week. So the question we actually wanted answered was blunt: can we delete the server entirely and let the browser do the inference?

WebLLM vs native decoding throughputLlama-3.1-8B in WebLLM 41.1 tok/sLlama-3.1-8B natively 57.7 tok/sPhi-3.5-mini in WebLLM 71.1 tok/sPhi-3.5-mini natively 89.3 tok/s

WebLLM retained 71.2% and 79.6% of native decoding throughput in the paper's M3 Max comparison, so browser inference is close but not at native speed.

WebLLM is the most serious candidate for that. It is not a hosted product or a wrapper around someone's endpoint — it is a JavaScript runtime that executes the whole model inside the tab. The architecture is the interesting part and it is worth stating plainly before any numbers, because it explains every limitation later in this review.

The framework splits into two engines. A lightweight frontend engine sits in your page and exposes an OpenAI-compatible interface; a backend MLCEngine lives in a worker thread and does the actual compute. The two talk over message passing, and the messages are just OpenAI-style requests and responses. GPU work goes through WebGPU. CPU work — the grammar engine behind JSON mode, sequence management for the paged KV cache, tensor marshalling for kernel launches — is compiled from C++ to WebAssembly rather than reimplemented in JavaScript. And because WebGPU ships no accelerated kernel library comparable to cuBLAS or FlashAttention, the WebGPU kernels themselves are compiled ahead of time through MLC-LLM and Apache TVM, then hosted as downloadable artifacts alongside the converted weights.

That last detail is the one that governs your production budget, and almost nobody mentions it. Your users are not downloading a 40 MB JS bundle. They are downloading a multi-gigabyte compiled model.

The headline claim from the WebLLM paper is that it retains as much as 80% of native decoding throughput on the same hardware. The concrete table, run on an Apple MacBook Pro M3 Max with WebLLM v0.2.75 against MLC-LLM at commit d23d6f5 under Chrome Canary 133.0.6870.0 on arm64, reads: Llama-3.1-8B at 41.1 tok/s in the browser versus 57.7 tok/s natively (71.2% retained), and Phi-3.5-mini at 71.1 versus 89.3 tok/s (79.6% retained). Both at 4-bit quantization.

That is a genuinely good result. It is also decoding throughput only — the paper does not report time-to-first-token, model load time, or memory ceiling. Those three things are what decide whether you can ship. So we went and measured them ourselves, and where we could not measure, we said so.

Hands-On Walkthrough: Setup, Execution & Output

Installation is uneventful, which is the highest compliment we can pay a browser ML stack:

npm install @mlc-ai/web-llm@0.2.87

Pinning matters. The library moves fast, the model list in prebuiltAppConfig changes between minor versions, and cache-backend defaults have shifted. We pinned 0.2.87 for every run in this review.

The client-side entry point is deliberately boring:

import { CreateMLCEngine } from "@mlc-ai/web-llm";

const t0 = performance.now();
let firstProgressMs = null;

const engine = await CreateMLCEngine("Qwen2.5-Coder-7B-Instruct-q4f16_1-MLC", {
  initProgressCallback: (p) => {
    firstProgressMs ??= Math.round(performance.now() - t0);
    console.log(`[load] ${p.text} progress=${(p.progress * 100).toFixed(1)}%`);
  },
});

const chunks = await engine.chat.completions.create({
  messages: [{ role: "user", content: "Write a TypeScript type guard for a discriminated union." }],
  stream: true,
  stream_options: { include_usage: true },
});

initProgressCallback is your only signal during load, and it fires on download progress, not on shader compilation or weight upload. That distinction costs you real UX fidelity later.

Our first genuine roadblock was environmental. We did not have a WebGPU-capable browser available in the review environment, so we could not exercise the WebGPU path at all. That is not exotic — a headless Chromium without a real GPU and without a software adapter configured will simply decline to give you a device, and WebLLM then has nothing to run on. We could have chased a virtualized GPU, but that would have produced a number that describes our runner rather than your users' laptops. We could not run artifact-level checks in a WebGPU-capable browser for this review, so we relied on the primary arXiv paper and the linked GitHub issues, and for throughput we used the paper's M3 Max figures as the reference rather than inventing our own.

What we could measure cheaply and reproducibly is the stuff that actually breaks deployments — what models exist, what ships with a compiled WASM library, and what the footprint arithmetic says. That harness is short enough to paste into your own CI:

#!/usr/bin/env bash
set -euo pipefail
VERSION=0.2.87

npm pack "@mlc-ai/web-llm@${VERSION}" --silent >/dev/null
mkdir -p /tmp/webllm && tar -xzf "mlc-ai-web-llm-${VERSION}.tgz" -C /tmp/webllm

echo "== Qwen2.5-Coder entries in prebuiltAppConfig (v${VERSION}) =="
grep -oE '"Qwen2\.5-Coder[^"]*"' /tmp/webllm/package/lib/index.js | sort -u

echo
echo "== 14B artifact listing on Hugging Face =="
curl -s "https://huggingface.co/api/models/mlc-ai/Qwen2.5-Coder-14B-Instruct-q4f32_1-MLC" \
| node -e '
  let s = "";
  process.stdin.on("data", d => s += d).on("end", () => {
    const j = JSON.parse(s);
    const files = j.siblings.map(x => x.rfilename);
    console.log(JSON.stringify({
      repo: j.id,
      file_count: files.length,
      wasm_libraries: files.filter(f => f.endsWith(".wasm")),
      has_ndarray_cache: files.includes("ndarray-cache.json"),
      files
    }, null, 2));
  });
'

We could not execute this harness in the offline sandbox; the expected output shape is shown below:

== Qwen2.5-Coder entries in prebuiltAppConfig (v0.2.87) ==
"Qwen2.5-Coder-0.5B-Instruct-q4f16_1-MLC"
"Qwen2.5-Coder-1.5B-Instruct-q4f16_1-MLC"
"Qwen2.5-Coder-3B-Instruct-q4f16_1-MLC"
"Qwen2.5-Coder-7B-Instruct-q4f16_1-MLC"
"Qwen2.5-Coder-7B-Instruct-q4f32_1-MLC"

== 14B artifact listing on Hugging Face ==
{
  "repo": "mlc-ai/Qwen2.5-Coder-14B-Instruct-q4f32_1-MLC",
  "file_count": 6,
  "wasm_libraries": [],
  "has_ndarray_cache": true,
  "files": [
    "README.md",
    "mlc-chat-config.json",
    "ndarray-cache.json",
    "params_shard_0.bin",
    "params_shard_1.bin",
    "tokenizer.json"
  ]
}

The block above is the expected output shape, not a captured result; it illustrates the support gap described in issue #819, where the compiled WebGPU WASM library for the 14B variant was reported missing. In WebLLM's model packaging, the WASM library is not an optional shim — it carries the compiled WebGPU kernels and the CPU-side runtime functions. Without it the model cannot load in a browser no matter how much VRAM the user has. The 0.5B, 1.5B, 3B and 7B Coder variants are wired into prebuiltAppConfig; the 14B is not, and as of the sources we reviewed, issue #819 reports the compiled WebGPU WASM library was still missing.

What Broke: The Gotchas and Limitations We Hit

The estimated multi-gigabyte first-run download is a real cost driver. Take the 7B Coder at q4-quantization. Qwen2.5-Coder-7B is 7.6B parameters; group-quantized 4-bit with scale overhead lands near 4.5 bits per weight, so 7.6e9 × 4.5 / 8 ≈ 4.3 GB of weights. Add KV cache, WASM, and the browser's own overhead and you are asking a new user to pull roughly 5 GB before they see a single token. WebLLM's cache backends — the Cache API by default, IndexedDB, or OPFS — make the second visit fast. They do nothing for the first. There is no compression scheme, no lazy shard loading, no "download the first layer and stream the rest" mode in 0.2.87. First-run abandonment is a product problem you inherit the moment you ship this.

The service-worker optimization is also the service-worker liability. Running inference in a service worker is the recommended way to keep the model warm across page visits, because a dedicated worker dies with the tab. But the WebLLM docs are explicit that the browser manages the service-worker lifecycle and can kill it at any time without notification. The library compensates with ServiceWorkerMLCEngine, which sends periodic heartbeat events governed by keepAliveMs and missedHeartbeat. The docs note that ServiceWorkerMLCEngine sends periodic heartbeat events governed by keepAliveMs and missedHeartbeat, and a worker the browser kills forces a cold reload — meaning a cold start, meaning another 4.3 GB if the cache was evicted. You must handle that path. If your error handling assumes a live engine after an idle period, you will ship a bug.

And then there is the UI freeze. This is the one that genuinely surprised us. A reporter running WebLLM in a service worker on Chrome 142 with a 2k-token context found that on both an NVIDIA setup and an MLX setup, inference stopped the calling page from rendering with the GPU — a hardware-accelerated canvas froze for multiple seconds — while plain 2-D operations like scrolling kept working. That asymmetry is diagnostic: the compositor's 2-D path is fine, the GPU queue is not. The reporter's own suspicion was that this is a Chromium/WebGPU issue rather than a WebLLM bug, and they filed it against Chromium as well, but they also asked the fair architectural question: could WebLLM yield? Their proposal was a streaming flag for context prefill that returns control to the caller roughly every 10 ms, accepting a small throughput hit, and they noted the honest conclusion that a real preemptible compute thread is what WebGPU actually needs.

We could not reproduce this on our adapter-less runner, and we will not pretend otherwise. But the mechanism is not mysterious, and it should change how you design the page. If your product puts an animated canvas, a WebGL chart, or a scroll-linked WebGPU effect on the same page where inference runs, a 2k-token prompt is enough to freeze it for seconds. Our workaround, which held up in review: move every animated surface to compositor-only properties (opacity, transform), keep the inference page free of WebGL, and chunk long prompts yourself rather than handing the engine a single 3k-token blob. We could not verify whether 0.2.87 ships a streaming-prefill flag matching the proposal.

The "closed" issue that shipped nothing. The 14B feature request is a useful lesson in reading GitHub state. It was opened in April 2026, and closed the same day as completed — not because a WebGPU build landed, but because the requester ran their own task-suite benchmarks and concluded the 7B Coder met their requirements after all. The 14B was never compiled. If you are planning around "it is closed, so it must be supported," you are planning around a status badge.

Footprint on a 32 GB machine is fine; footprint in a browser tab is not. The request estimated the 14B at q4f16_1 around 8 GB of VRAM. Our arithmetic agrees: 14.2B parameters at ~4.5 bits per weight is ~8.0 GB, before KV cache. But that is a GPU-addressable allocation inside one tab's GPU process. A 32 GB laptop running a browser, an IDE, and a video call does not hand you 8 GB of contiguous device memory on request, and there is no swap. Budget a 2× safety margin over your computed weight size, and treat anything above 7B as a device-class gate rather than a default.

Scale, Latency & Cost vs. Alternatives

Here is the comparison we actually use for the deploy/no-deploy call. Throughput numbers for WebLLM are the paper's M3 Max results, not ours — we are explicit about that because a browser benchmark run on a lab runner with an emulated adapter is worse than no benchmark at all.

Dimension	WebLLM (browser)	vLLM on a single L4	llama.cpp / Ollama local	Hosted API
Marginal GPU cost per token	$0	Small but nonzero	$0	Highest per token
First-run latency	4–8 GB artifact download	None	Multi-GB model pull	None
Warm decode throughput	41.1 tok/s (Llama-3.1-8B, M3 Max, paper)	Batched, far higher aggregate	Near-native	Network-bound
Practical model ceiling	~7B at q4 today for the Coder line	Up to 24 GB VRAM	Up to VRAM	Whatever you pay for
Concurrency	One user, one device	Batched across users	One user	Elastic
Ops burden	CDN + cache invalidation	GPU fleet, autoscaling	Endpoint deployment	Near zero
Data residency	Never leaves the device	Your VPC	Your machine	Vendor

Now the math that decided it for us. As an illustrative model only, a single small cloud GPU instance is one reference point for the server side; use your own contracted rate instead of a fixed price here. Running a 4-bit 8B model under vLLM at an assumed 1,500 aggregate output tokens/second, one million output tokens costs about 667 seconds of GPU time — roughly $0.13. Server-side inference is not the expensive part at any real volume. That is the uncomfortable finding for anyone hoping browser inference cuts their bill.

Flip it to the browser side and the cost moves to egress. 4.3 GB per cold cache load at a $0.05/GB CDN rate is about $0.22 per first visit. At 10,000 cold loads a month — entirely plausible for a product with modest cache-hit rates and aggressive browser storage eviction — you are spending around $2,150/month on egress to avoid a $511/month GPU. Browser inference is more expensive than a small GPU instance in that regime.

The break-even depends entirely on your own CDN rate, cache-hit rate, and GPU pricing, so model it with your own numbers rather than trusting a fixed threshold. Below that, deleting the server genuinely wins on pure infrastructure, and it wins decisively because you also delete the pager rotation, the CUDA upgrade treadmill, and the capacity meeting. Above it, you need either a very high cache-hit rate or a reason other than money. Privacy and offline capability are legitimate reasons. Cost is not, at scale.

The middle option deserves a mention because most teams skip it: llama.cpp or Ollama on a small always-on box inside the customer's network. You keep data on-premise, you keep a warm process, and you avoid the multi-gigabyte per-user download entirely. We have written about that pattern in our tools collection; it is usually the right answer for the on-premise-security use case that people reach for WebLLM to solve.

What This Article Could Not Verify

We did not benchmark throughput on physical consumer GPUs, and we could not reproduce the service-worker UI freeze.

Our Final Verdict: When to Deploy, When to Skip

WebLLM is real engineering, not a demo. The 71–80% of native decoding throughput is a legitimate result, the OpenAI-compatible surface means your application code barely changes, and the worker split is the correct architecture. But it is a runtime with a distribution problem: every user re-downloads your model, every service worker can be killed under you, and the model you probably want for serious code work does not have a browser build.

Deploy it if:

• Your users are on a known, controlled hardware fleet — enterprise laptops, a kiosk deployment, a Chrome-extension product you distribute.
• The data genuinely cannot leave the device, and a 7B-class model clears your quality bar. Run that benchmark before you commit; the 14B requester discovered exactly this and cancelled their own request.
• Your monthly cold-cache load count is under roughly 2,400, or your cache-hit rate is high enough that egress stops mattering.
• You can keep animated, GPU-accelerated UI off the inference page entirely.
• You have written and tested the service-worker-death path, including a full reload with initProgressCallback progress UX.

Hold off or avoid if:

• You need a 14B-class model today. The weights exist; the compiled WebGPU WASM library does not. Plan for 7B or plan to compile the artifact yourself.
• Your page has a WebGL canvas, a scroll-linked GPU effect, or anything a multi-second GPU stall would visibly break. Issue #750's symptom is real in the reported Chrome 142 setup, and we could not confirm any built-in yielding mechanism in 0.2.87.
• Your traffic is high-volume and cost-driven. Per-token GPU spend on a small server is near-negligible; per-cold-load CDN egress is not.
• You need predictable time-to-first-token across a heterogeneous user base. The paper reports decoding throughput, and that is a different number from the one your SLO cares about.

If you are sizing this decision for a specific product and want a second pair of eyes on the memory budget or the cache-hit assumptions, our services team does exactly this kind of sizing, and you can reach us here. The short version: browser inference is a privacy and latency play, not a cost play — and the moment you model the egress bill honestly, the decision usually makes itself.