Run an LLM in the browser: WebGPU did 63 tokens/s, WASM did 1.6

Originally published at tensorust.com.

TL;DR: You can run a small language model entirely inside a web page with Transformers.js, a module Web Worker and WebGPU. We do it in BEINGS, a chatroom where each person's AI creatures think on that person's own laptop, so the platform pays $0 for inference. On October 17, 2026 we measured it fresh: the same 360M-parameter model wrote 63 tokens a second on WebGPU and 1.6 on the WASM fallback, a 1B Llama answered in under half a second, and a 0.5B Qwen produced so much gibberish that half its replies at a higher temperature had to be thrown away.

Short answer first, since that's what you searched for. To run an LLM in the browser you need three things: a model already converted to ONNX and quantized (4-bit weights, so a 1B model is about 1.2 GB instead of 5), a runtime that executes it in the page (Transformers.js, which wraps ONNX Runtime Web), and a GPU backend (WebGPU) with CPU WebAssembly as the fallback. Put the model in a Web Worker so the page never freezes. That's about 90 lines of JavaScript, and below is the file we ship, lightly trimmed.

The harder part is everything after "it runs": which model, which sampling settings, what to do when a tiny model starts typing "u.u.u.u.u.", and one bundler trap that silently broke all of it. Real code, real numbers, and the parts that went wrong.

One chat line, start to finish, with numbers from our Oct 17 2026 benchmark (Chrome 155, RTX 4070). No server touches the model.

Why put the model in the user's browser at all?

BEINGS is an old-school chatroom where everyone brings their own AI creatures: a finance bro who thinks pizza is a market signal, a medieval knight who thinks your phone is sorcery, a conspiracy detective. The creatures talk to each other and to the humans, all day, unprompted. That's a lot of model calls for a free toy.

If those calls went through our server to a paid API, every lively room would be a bill. So the architecture bet was the opposite: each creature runs on the device of the person who brought it. Your laptop is literally running your bots. The landing page says it plainly: no API key, no signup, nothing to pay. Cloud brains (OpenAI, Claude, Gemini, OpenRouter) are an optional upgrade where you paste your own key, and that key lives in your browser's localStorage and goes straight to the provider, never through us.

The trade-offs are honest ones. The user downloads the model once (hundreds of MB to over a GB), their GPU does the work, and small models are much dumber than the big hosted ones. For a chaotic chatroom, dumb-but-free is the right deal. For a customer support bot, it probably isn't.

The live BEINGS landing page, captured Oct 17 2026, 2:37 AM ET. The footer: creatures think on your machine using WebGPU, falling back to CPU.

Step 1: the worker that holds the model

Generation is heavy, synchronous number crunching. Run it on the main thread and the page freezes while the model thinks. So the model lives in a module Web Worker: a separate thread that loads Transformers.js, holds one model, and talks to the page with messages. Here's the file, trimmed only of its error branch (public/llm-worker.js):

import {
  pipeline,
  env,
} from "https://cdn.jsdelivr.net/npm/@huggingface/transformers@4.2.0/+esm";

env.allowLocalModels = false;

let pipe = null;
let loadedRepo = "";
let busy = false;

async function load(repo, device) {
  if (pipe && loadedRepo === repo) return;
  pipe = await pipeline("text-generation", repo, {
    device,
    dtype: device === "webgpu" ? "q4f16" : "q4",
    progress_callback: (p) => {
      if (p && p.status === "progress" && typeof p.progress === "number") {
        self.postMessage({ type: "progress", pct: Math.round(p.progress), file: p.file });
      }
    },
  });
  loadedRepo = repo;
}

self.onmessage = async (e) => {
  const msg = e.data;
  if (msg.type === "load") {
    self.postMessage({ type: "status", state: "loading" });
    try {
      await load(msg.repo, msg.device);
    } catch (err) {
      // WebGPU can be present but fail to initialise on some drivers.
      if (msg.device === "webgpu") {
        self.postMessage({ type: "status", state: "falling-back" });
        pipe = null;
        loadedRepo = "";
        await load(msg.repo, "wasm");
      } else {
        throw err;
      }
    }
    self.postMessage({ type: "status", state: "ready" });
    return;
  }
  if (msg.type === "gen") {
    // ... one generation at a time ("busy" guard), then:
    const out = await pipe(msg.messages, {
      max_new_tokens: msg.maxTokens,
      do_sample: true,
      temperature: msg.temperature,
      top_p: 0.9,
      top_k: 40,
      repetition_penalty: 1.12,
      no_repeat_ngram_size: 3,
      return_full_text: false,
    });
    // ... pull the assistant text out and post it back as "gen-done"
  }
};

Four details in there earn their place:

• dtype changes with the device. On WebGPU it loads q4f16 (4-bit weights, 16-bit math); on WASM it loads q4. Those are different files on Hugging Face, and the dtypes guide lists what each model ships. For SmolLM2 the WebGPU file is 272.7 MB and the WASM one is 387.9 MB.
• WebGPU can exist and still fail. "gpu" in navigator being true doesn't mean the driver will cooperate, so a failed WebGPU load retries on WASM instead of showing an error.
• Progress goes back to the page. A 1.2 GB first download with no progress bar looks exactly like a broken site.
• Transformers.js comes from a pinned CDN URL, version 4.2.0 on jsDelivr, not from npm. That one is on purpose, and it's the next section.

The bundler trap: why the worker isn't bundled

The "proper" way to write a worker in a modern framework is a TypeScript file plus new Worker(new URL("./llm.worker.ts", import.meta.url)), and let the bundler handle it. BEINGS is a Next.js 16 app, built with Turbopack. When we did it that way, Turbopack emitted the worker as a raw .ts file in .next/static/media/. No browser can execute TypeScript, so the worker never started, and all local inference broke without a single build error.

The fix is boring and permanent: the worker is hand-written JavaScript in /public, served as-is, loaded with a plain path, and it imports Transformers.js from the CDN so the bundler never touches it. The file's own header comment says so, and the project notes say not to "tidy" it back into the bundle. Here's the page side (lib/brains/client.ts):

// Served from /public as a real ES module; see the comment in that file.
const w = new Worker("/llm-worker.js", { type: "module" });
this.worker = w;

const hasGpu = typeof navigator !== "undefined" && "gpu" in navigator;
w.postMessage({ type: "load", repo, device: hasGpu ? "webgpu" : "wasm" });

If you use Vite or webpack, worker bundling may well work for you. The lesson that travels is smaller: after any build, open the deployed site and check that the worker actually starts. A silently dead worker looks like a slow model.

Step 2: one model, one generation at a time

A user can bring six creatures into one room. If all six called the model at once, a laptop would be loading six copies or fighting over one GPU. So the page side owns exactly one worker and queues every generation behind the last one (lib/brains/client.ts):

/** Serialised: generations run one after another, never concurrently. */
generate(messages: ChatTurn[], maxTokens = 60, temperature = 0.75): Promise<string> {
  const run = () =>
    new Promise<string>((resolve, reject) => {
      if (!this.worker) return reject(new Error("no brain"));
      const id = Math.random().toString(36).slice(2);
      this.pending.set(id, { resolve, reject });
      this.worker.postMessage({ type: "gen", id, messages, maxTokens, temperature });
      setTimeout(() => {
        if (this.pending.has(id)) {
          this.pending.delete(id);
          reject(new Error("timed out"));
        }
      }, 90_000);
    });

  const next = this.queue.then(run, run);
  this.queue = next.catch(() => {});
  return next;
}

this.queue.then(run, run) is the whole trick: each call waits for the previous one to settle, success or failure, so one crashed generation can't jam the line. The 90-second timeout exists because on the WASM fallback a single reply can take most of a minute, as the next section shows.

How fast is an LLM in the browser, really?

The project notes had one uncomfortable line: in-browser WebGPU inference had never been verified on real hardware, because the build session had no browser. So tonight we measured it. Setup: Chrome 155 on Windows 11, an RTX 4070 desktop (WebGPU reports the adapter as nvidia / lovelace), 24 logical CPU cores. A small script opened its own tab on the live site, started the real /llm-worker.js, loaded each model, and asked it 24 real chat turns: the Braxton finance-bro persona from the app, three different room messages, four tries each, at two temperatures. Same 70-token cap the app uses.

Median tokens per second over 12 replies per bar, our own tab on the live site, Chrome 155, RTX 4070, Oct 17 2026, 2:19 to 2:34 AM ET.

The headline is the backend, not the model. SmolLM2 360M wrote 62.9 tokens per second on WebGPU and 1.6 on WASM (medians, temperature 0.75). The median reply took 0.54 seconds on the GPU and 26.9 seconds on the CPU path. In a chatroom, 27 seconds is an eternity: the conversation has moved on before your creature finishes its sentence.

Part of that gap is probably self-inflicted, and it's worth checking on your own site. ONNX Runtime Web only uses multiple CPU threads when the page is cross-origin isolated; otherwise it runs single-threaded (that's straight from its env flags docs). Our page sends Cross-Origin-Opener-Policy: same-origin but no Cross-Origin-Embedder-Policy, and the benchmark tab reported crossOriginIsolated: false with no SharedArrayBuffer. So our WASM fallback ran on one thread of a 24-thread machine. We haven't measured it with isolation turned on yet, so I won't promise a number, but my bet is the fallback gets several times faster. The catch is that COEP blocks cross-origin embeds that don't opt in, and BEINGS embeds YouTube and Wikimedia media. The web.dev guide to COOP and COEP covers the options.

The other surprise: the biggest model was the fastest to reply. Llama 3.2 1B averaged 55 to 64 tokens a second, close to SmolLM2, but its median reply landed in 0.4 to 0.5 seconds because it said what it had to say in about 24 to 30 tokens and stopped. The smaller models rambled to the 70-token cap. Qwen 2.5 0.5B was the slowest of the three at 37 to 40 tokens a second and about 1.8 seconds per reply.

Cold loads downloaded the model files fresh (Cache Storage entries dated Oct 17 2026); warm loads read them from the browser cache a few minutes later.

The download is the real first-run cost

All three models were downloaded fresh during this run (the Cache Storage entries are dated October 17). On this connection, the cold start, download included, was 7.5 seconds for SmolLM2 (273 MB), 13.4 for Qwen (483 MB) and 26.8 for Llama (1,238 MB). Once cached, the same loads took 1.0, 2.2 and 3.1 seconds. Transformers.js stores the files in the browser's Cache Storage, so the second visit is fast without you writing any caching code.

Your users won't all have a fast connection. A 1.2 GB download on a phone over 4G is a "come back later" moment, which is why the app lets you pick a smaller brain, and why the progress callback from Step 1 matters more than it looks.

Sampling settings matter more than model size

Early on, the app sampled with temperature 1.05 and top_p 0.95 for "more chaos". Even Qwen looped into lines like "u.u.u.u.u." and "The The FQ Q M". The fix that stuck is in the worker above: temperature 0.75, top_p 0.9, top_k 40, a repetition penalty of 1.12 and no_repeat_ngram_size: 3, which forbids the model from repeating any three-token sequence. (If those knobs are new to you, the Hugging Face generation docs explain each one.)

Tonight's run tested only the temperature, at 0.75 vs 1.05, with every other setting kept at the safer values. Even that one knob showed up clearly:

• Qwen 0.5B: 2 of 12 replies were thrown out as garbage at 0.75, and 6 of 12 at 1.05. Real examples from the higher temperature: "CODESUPONSEGRAPHTEGOINSTRUMENTPRODUSONGRAPH..." and "What sorceryistransformedintoasolidreturningSorcery."
• SmolLM2 360M: the median reply grew from 15 words to 40.5 at 1.05, and 7 of 12 replies ran into the 70-token cap. The prompt asks for under 20 words; at the higher temperature it stopped listening.
• Llama 3.2 1B: 24 of 24 replies were usable at both temperatures, and stayed in character: "Game-changer alert! WiFi outages when market indices hit bottom is proof!"

If you only change one thing in your own in-browser setup, make it this: keep temperature under 0.8 for anything below 1B parameters, and turn on no_repeat_ngram_size.

Step 3: a garbage filter, because silence beats nonsense

Even with good settings, a 360M model will sometimes produce something no human should see. The rule in BEINGS is "better a silent creature than a broken one", so every reply goes through a cheap, deterministic filter before it's posted. No second model, no API call. It's 64 lines (lib/garbage.ts, trimmed):

export function isGarbage(raw: string): boolean {
  const t = (raw || "").trim();
  if (t.length < 2) return true;

  // markup or template fragments
  if (/[<>]|&[a-z]+;|\{\{|\}\}/i.test(t)) return true;

  // symbol soup: not enough actual letters
  const letters = (t.match(/[a-zA-Z]/g) || []).length;
  if (letters / t.length < 0.55) return true;

  // a short chunk repeated over and over: "u.u.u.u.", "hahahahaha"
  if (/(.{1,4}?)\1{4,}/.test(t)) return true;

  const words = t.toLowerCase().match(/[a-z']+/g) || [];
  if (words.length >= 5) {
    // the same few words cycling
    const uniq = new Set(words);
    if (uniq.size / words.length < 0.5) return true;
    // ... and the same word three times in a row
  }

  // ... scattered one and two letter fragments: "The The FQ Q M"
  // ... invented words: no vowels, or a syllable cycling ("unununu")
  return false;
}

The filter has its own test file with the real bad lines from the bug report as fixtures, plus real good lines that must survive, including weird-but-fine ones like "Penar wingle berry" and "THIS IS BULLISH ACTUALLY, LOAD UP" (scripts/garbage-test.mjs). Tonight it still passed: 14 passed, 0 failed. In the app it sits right before posting, next to a second check for the model parroting its own instructions (lib/engine.ts):

const raw = await think(bot.brain, buildTurns(bot, this.ctx, trigger), 70);
const text = clean(raw, bot.name);
// Better a silent creature than one pasting its briefing or looping "u.u.u.".
if (!text || looksLikeLeak(text) || isGarbage(text)) return;

Every reply from the Oct 17 2026 benchmark, run through the same clean(), leak check and isGarbage() the app uses: 13 of 96 dropped, 0 of them from Llama.

What the filter caught, and what slipped through

Run over all 96 replies from tonight's benchmark, the filter and the leak check dropped 13: 11 as garbage and 2 as leaked instructions (one SmolLM2 reply ended "Speak in character now!"). It caught the obvious failures: the run-together words, the stray --> markup, the Qwen word salads.

It also let things through that a human would bin. Qwen's "Toestrafing GRating" passed, because both words have vowels and neither repeats. So did a SmolLM2 line in all caps about "MULTILATERAL ORTHODOZIC" things. A rule filter catches broken text, not meaningless text. My take: that's fine for a chaos chatroom, where a weird line is half the fun, and nowhere near fine for anything a customer reads. If meaning matters, you need a bigger model, not a smarter regex.

Lines from the benchmark results file (bench.json), Oct 17 2026. Note coi:false: no cross-origin isolation, so the WASM fallback ran single-threaded.

What broke along the way

• The bundled worker (above): Turbopack shipped a .ts file as a static asset. Zero errors, zero inference.
• Chaos settings: temperature 1.05 and top_p 0.95 made small models loop. Lowering them did more for quality than any model swap among the small ones.
• Instruction echo: small models repeat their briefing ("reply in one short line, in character"). The fix was both a leak filter and a prompt change: real chat turns instead of a transcript pasted into one big instruction, because, as a code comment puts it, "small models echo meta-instructions but handle the chat format they were tuned on."
• Six creatures, one laptop: concurrent generation fixed by the queue in Step 2.
• Never tested on real hardware until tonight. It works, and WebGPU is fast. The WASM fallback is the weak spot.

The numbers

Measure	Result
SmolLM2 360M, WebGPU vs WASM	62.9 vs 1.6 tokens/s; 0.54 s vs 26.9 s per reply
Llama 3.2 1B (q4f16), WebGPU	55 to 64 tokens/s, 0.4 to 0.5 s per reply, 24 of 24 replies usable
Qwen 2.5 0.5B, WebGPU	37 to 40 tokens/s, about 1.8 s per reply, 8 of 24 dropped
Download (model file)	273 MB / 483 MB / 1,238 MB
Load, cold vs cached	7.5 / 13.4 / 26.8 s vs 1.0 / 2.2 / 3.1 s
Inference bill for the platform	$0

Source: our own benchmark tab on the live BEINGS site, Chrome 155, RTX 4070, October 17, 2026, 2:19 to 2:35 AM ET. 12 replies per row and temperature: a small sample, good for orders of magnitude, not for second decimals.

Do this in your project

1. Use Transformers.js in a module Web Worker. One worker, one model, one generation at a time, with a queue in front of it.
2. Pick an ONNX model that ships a q4f16 file for WebGPU and a q4 file for the fallback. The onnx-community org on Hugging Face is the place to start.
3. Default to about 1B parameters, not the smallest thing that loads. In our run, Llama 3.2 1B was as fast per reply as SmolLM2 360M and far more coherent than Qwen 2.5 0.5B. Offer the small one as the "slow connection" option.
4. Fall back from WebGPU to WASM inside the worker, and if you care about the fallback, make the page cross-origin isolated so it can use threads. Then test that your embeds still work.
5. Keep temperature at or under 0.75 for small models, with no_repeat_ngram_size: 3 and a mild repetition penalty.
6. Filter before you show. A 60-line rule check with real failure lines as test fixtures beats posting "u.u.u.u.".
7. Show download progress, and tell people the size before they click.
8. Verify on the deployed site, in a real browser, after every build. Our worker broke silently in a way no test would have caught.

Where this goes next

I think the interesting shift isn't that small models are good. They aren't, yet. It's that "the user's device pays" changes what's worth building. A feature that would cost a cent per message on an API is free forever in the browser, so whole categories of silly, chatty, always-on things become possible for a solo developer. Hardware keeps moving the line too: today's 1B model in a tab is roughly what needed a server not long ago. Compare that with the other end of the scale, where an "open" 816B model needs about a terabyte of RAM to run at all.

If your app does call a hosted model, the cheapest token is the one you don't send twice: here's how we stopped paying twice for the same LLM answer. And before you pick between in-browser and an API, price the API side properly. Our $7 LLM API Cost Calculator works out cost per request, per month and per year across 26 models, in Google Sheets or Excel.

Have you shipped an LLM that runs in the user's browser? What did your users' hardware do to your plans: did WebGPU just work, or did the fallback carry you?

Comments

[十方空烬 · Ember ☄️]

In the queue example, what happens if the 90-second timeout fires while WASM is still generating? Does the next request wait for that generation to finish, or can the worker's busy guard reject or drop it? I'd test a forced timeout followed by a second request and check that it completes without reloading the page.

[Matt Busel]

Good catch, you've found a real gap. In the shipped worker there's a separate busy flag, so if the 90-second timeout fires while WASM is still generating, the page-side queue moves on, the next request reaches the worker, and the worker answers it right away with a "busy" error. Nothing hangs and no reload is needed, but that second request is dropped, not delayed. When the stuck generation finally finishes, its late result is ignored (its id was already cleared), busy goes back to false, and the request after that works normally.

That's from reading the code, not from running your forced-timeout test, so take it as a prediction. The fix I'd make: have the timeout actually interrupt the generation instead of just giving up on it (Transformers.js exports an InterruptableStoppingCriteria for this), or have the page queue wait for the worker's real "done" before sending the next one. Adding your test to the list.