Running Whisper in the browser: what broke and how I fixed it

Luiz Mueller — Sun, 25 Oct 2026 13:40:37 +0000

I run a small site of free browser tools, and the one people use most is a transcriber: you drop in an audio or video file and get text or SRT/VTT subtitles back. By default nothing is uploaded. OpenAI's Whisper runs on your own device through transformers.js, and ffmpeg.wasm pulls the audio out of the file first. (There's also an optional Instant mode that uses the browser's Web Speech API, which sends the audio to Google or Microsoft. It's off by default, and this post is about the on-device mode.)

It worked on my laptop on day one. Making it work on other people's devices took a lot longer. These are the problems that showed up in real use, in the order they hurt, and what I changed.

The setup

Audio: ffmpeg.wasm (about 31 MB, downloaded once) converts any file to 16 kHz mono float32 PCM.
Speech: transformers.js 4.3 runs Whisper in a Web Worker. I offer three sizes: tiny ("Fast"), base ("Accurate") and small ("Best").
Backend: WebGPU when it helps, WASM otherwise. The site doesn't send COOP/COEP headers, so WASM runs single-threaded.

Download sizes, measured from the actual .onnx files on the CDN:

Model	WASM (q8)	WebGPU (fp16)
tiny	39 MB	73 MB
base	73 MB	139 MB
small	238 MB	463 MB

1. Big videos ran out of memory before transcription even started

The first version did what most ffmpeg.wasm examples do: read the whole file into JavaScript, then write it into ffmpeg's in-memory filesystem. That's two full copies. A 300 MB phone video went past 1 GB of memory just to extract its audio, and phones gave up.

Fix: mount the file with WORKERFS instead of copying it. ffmpeg then reads straight from the File object, chunk by chunk, so the video's size barely matters. Only the extracted audio lives in memory, about 230 MB per hour of float32 PCM. I've since tested a 2.1 GB file. If the mount fails on an old browser, the code falls back to the copy.

2. Always using WebGPU was the wrong default, twice

On phones. Android Chrome exposes navigator.gpu, and the adapter looks fine. But the worker kept dying in the middle of a transcription, probably because of shared memory or unstable drivers: one user in India hit seven failures in a row on the smallest model. Phones now always use WASM with the q8 weights.

On laptops. I assumed the GPU path was faster everywhere. Then I measured on an ordinary 2021 laptop (Core i5-11300H, Intel Iris Xe graphics) with 60 seconds of speech, timing only the transcription step. This was headless Chrome with the Vulkan backend, one run per model, in minutes per 10 minutes of audio:

Model	WASM (CPU)	WebGPU (Iris Xe)
tiny	2.0	3.4
base	4.0	6.5
small	24.6	86.9

A second run, in headless Chrome with default settings, told the same story for the two smaller models: tiny 2.4 on the CPU against 3.4 on the GPU, base 4.7 against 8.2. I didn't repeat the small model. On that integrated GPU, the processor won every time, by 3.5x on the small model in the first run. Older Intel integrated graphics now go to WASM. The rule checks adapter.info.vendor for intel and an architecture starting with gen-. Newer Intel chips, which I haven't measured, and dedicated GPUs stay on WebGPU.

One detail matters here. transformers.js asks for the high-performance adapter, so the check has to ask for the same one: requestAdapter({ powerPreference: 'high-performance' }). On a laptop with two GPUs, a plain requestAdapter() can return the integrated one while inference runs on the other.

3. The dtype trap

Without an explicit dtype, the WebGPU path downloaded fp32 weights: 923 MB for the small model, without telling anyone. Picking dtypes by hand turned up three things:

q8 on the GPU is slow, probably because int8 isn't accelerated there. In an earlier test on the same laptop, with the same model in both runs, a 35-second clip took about 25 s with fp16 and 196 s with q8.
fp16 on the GPU is twice the download of q8 but much faster, so it's the GPU default.
q4f16 is smaller than q8 for the small model, but it returned unreadable text in my tests. It's out.

On WASM I use q8. Of the formats I tested, it's the smallest that gave readable text. The model repositories also have int8 files that are smaller still, which I haven't tried yet.

4. Failures that should never reach the user

Some errors repeat often enough that showing them is the wrong answer.

Out of memory on the larger models. failed to call OrtRun() with an allocation error means the device can't hold that model. The tool now continues by itself with the smallest model and says so on screen, instead of showing a raw ONNX error.
Model downloads that stall on slow connections. The tool retries once, and files that finished downloading stay in the browser cache. If it stalls again, it steps down to the next smaller model.
A GPU that dies mid-job. The job restarts on the CPU in a fresh worker, and the GPU isn't used again for the rest of that session.

5. Timeouts that fired while work was still happening

In one week, analytics showed four people getting "Reading the audio timed out". The timeout was a fixed 10 minutes, and it started before the 31 MB engine had even downloaded. My best guess is that on a slow connection, the download ate the budget, and files that were still being read failed anyway.

Now the timeout measures lack of progress, not total time. Downloading the engine gets 15 minutes, since it reports no progress. Reading audio only gives up after 3 minutes with no progress event at all.

Phones added one more trap. When you switch apps, the browser freezes the tab, and the timer can fire the moment you come back, before the page has noticed it was hidden. So if the timer fires more than 5 seconds late, or while the page is hidden, it re-arms instead of failing.

6. People leaving halfway

Almost as many people left halfway as hit an error. In one week, 26 people started a transcription: 9 hit an error, and 7 left without any error, before it finished. Several cancelled during the transcription step itself. Small things help:

Screen Wake Lock while processing, so a phone screen doesn't turn off and freeze the tab. It's requested again when the page becomes visible, because the browser releases it when the page is hidden.
Progress in the tab title, like "42% · Transcribing…", and "✓ Transcript ready" if it finishes while you're on another tab.
A beforeunload prompt while a job is running, so an accidental reload doesn't throw away twenty minutes of work.
Honest estimates up front. My page used to promise 5–15 minutes per 10 minutes of audio for the small model. The measurement above says 25. The page now shows the measured numbers.

What's still hard

The small model on phones is slow. Some languages, like Hindi, Tamil and Urdu, need it to come out right.
No speaker labels. Plain Whisper doesn't do diarization.
The first run means a download of tens to hundreds of MB, depending on the model. After that, it starts from the cache.

If you want to try it, the tool is at pixthru.com/video-to-text. It has a 12-second sample if you don't want to use your own file. I'd like to hear from anyone who has measured WebGPU against WASM on other GPUs, especially Apple Silicon and AMD integrated graphics.

DEV Community: Luiz Mueller