I shipped three AI tools that run entirely in the browser — here's everything that broke

Dodly Forbi — Fri, 24 Jul 2026 20:04:12 +0000

I set myself one rule for a side project: every tool has to run entirely in the user's browser. No file ever gets uploaded to a server. If I couldn't do something client-side, it didn't ship.

That rule turned out to be a great filter for what to build, and a brutal filter for how. Here's what actually broke along the way — a broken model file caught by physically-impossible output, an upstream ONNX Runtime bug, and a memory ceiling that only shows up on long files. If you're building AI features that run on-device, some of this will save you a week.

The site is WeConvertIt — image, PDF and audio tools that all run locally. Three of them use real models in the browser: background removal, music/vocal stem separation, and speech-to-text.

1. The model file was broken, and it took a physics violation to prove it

The stem splitter (split a song into vocals and instrumental) uses Meta's HTDemucs, which is MIT-licensed and genuinely state of the art. I grabbed a third-party ONNX export of it, wired it up with onnxruntime-web, and the output was garbage — robotic artifacts, vocals still sitting in the "instrumental" track at nearly full volume.

The obvious suspects were all mine. I'd converted the model to fp16 to halve the download; transformers are famously sensitive to precision loss, so that was suspect number one. I ran the same song through fp16 and fp32 natively: numerically identical to four significant figures. Not it.

Sample rate? Demucs wants 44.1kHz stereo; I logged what actually reached the model. Correct. Not it.

So I did the thing I should have done first: diffed against the reference implementation. I reimplemented HTDemucs's own _spec()/_ispec() (the STFT and inverse STFT) from the PyTorch source, and compared them against the JavaScript versions in my pipeline on identical input. Every probed value matched to eight decimal places, in both directions. My code was innocent.

Then I installed the real PyTorch demucs with the official checkpoint and ran the same full-band song through both. That's where it fell out:

	Reference (PyTorch)	My ONNX model
vocals @ 20–60Hz	4.8% of mix	69.3%
vocals @ 2–6kHz	0.6% of mix	75.4%
bass @ sub-bass	—	145.2%
other @ sub-bass	—	174.7%

Look at the bottom rows. Stems coming out at 145% and 174% of the original mix's own magnitude. That isn't "lower quality" — it's impossible. You cannot extract more energy from a signal than went into it. That single number told me the model file itself was defective, not my pipeline, not the architecture.

The fix was to stop trusting someone else's conversion and export my own ONNX from the official pretrained checkpoint (this export is documented as finicky — it needs custom handling for the STFT ops, which is presumably how the broken one got broken). Then I validated the new export against the reference before shipping it: 0.987/0.999 correlation, no stem exceeding mix magnitude. The tool went from unusable to genuinely good.

Takeaway: when a well-regarded model produces garbage, suspect the file before the architecture — and find a number that's physically impossible if things were working. That's a much faster signal than "sounds bad."

2. An upstream ONNX Runtime bug that only fires on tied embeddings

Speech-to-text uses Whisper via transformers.js. Session creation failed outright:

TransposeDQWeightsForMatMulNBits Missing required scale ... model.decoder.embed_tokens.weight_transposed_DequantizeLinear

My first assumption was that I'd downloaded a 4-bit quantized variant by mistake — MatMulNBits is a 4-bit op. Wrong. The committed file was the standard uint8 build, and the binary contained zero MatMulNBits strings.

The op wasn't in the file. ONNX Runtime's graph optimizer was creating it at session-creation time — a fusion pass that turns DequantizeLinear + MatMul into MatMulNBits, which crashes when a weight initializer is shared by two consumers. Whisper's tied embedding/lm_head is exactly that pattern, so every quantized Whisper decoder trips it. It's a known regression (onnxruntime#28306).

Two things made this worse to diagnose. First, no amount of re-downloading model files would ever have fixed it. Second, @huggingface/transformers statically inlines its onnxruntime-web build at publish time — so bumping onnxruntime-web in your own package.json does nothing to that code path.

The fix was one line: pass session_options: { graphOptimizationLevel: 'basic' } to the pipeline, skipping the buggy aggressive fusion while keeping the cheap optimizations.

Takeaway: if an ONNX error names an operator your model doesn't contain, the optimizer is generating it. Check the graph optimization level before you touch the weights.

3. Long audio doesn't fail loudly, it fails quietly

Two separate traps here, both of which ship silently if you don't test long files.

Whisper's context window is 30 seconds. Hand it a five-minute recording without setting chunk_length_s and it doesn't throw — it transcribes the first 30 seconds and returns a tidy, complete-looking result. I reproduced this: 238 characters returned for a five-minute speech. Imagine a user transcribing an hour-long lecture and never noticing 98% is missing. Now the chunking constants are hardcoded in the worker where the UI can't reach them, plus a sanity check that flags a transcript implausibly short for the audio duration.

Then the memory ceiling. The WASM heap caps around 1GB, and my first architecture handed the entire file's decoded PCM to one transcription call — so memory grew chunk after chunk until it blew, and when it blew it took the whole job with it. That produced the world's most confusing bug report: "transcript is shorter than the audio."

The fix was architectural: process long files in 5-minute segments, decode each segment on demand rather than holding the whole file's PCM, and terminate and recreate the worker between segments so the browser actually reclaims WASM memory. That bounds peak memory to one segment regardless of total length. A failing segment retries once, then gets marked in the transcript — never silently dropped. A 29-minute file now completes in about 7.5 minutes with no crash.

Takeaway: for browser ML, memory is the real duration limit, not the model. Terminating the worker is a blunt but very effective way to force reclamation.

4. Sometimes the ceiling is real

I ran the identical investigation on the background remover, expecting another broken conversion. It wasn't. The model file was byte-identical (MD5-verified) to what the official Python package downloads, and its output matched the reference at 0.9999 correlation. My own post-processing measured 0.00% erosion on three of four deliberately hard test images.

The tool is just… at its ceiling. That model runs at 320×320. Individual hair strands are sub-pixel at that resolution; they physically cannot be resolved. The cloud tools beat it because they run far bigger models on their servers — which is exactly the trade I'm not willing to make.

So the honest answer there was a manual erase/restore brush: let the model do 85% and let the user finish it in ten seconds. Every good cutout tool works this way.

Takeaway: run the same diagnostic even when you expect bad news. "Confirmed ceiling" is a genuinely useful answer — it stops you tuning knobs against a problem that doesn't exist.

What running locally actually buys you

The thing I keep coming back to: for a stem splitter or a transcription tool, the file is someone's unreleased track, or a private interview, or a medical consult. Every mainstream alternative uploads it. Running on-device means the privacy claim is verifiable rather than promised — open the Network tab, run a job, watch nothing leave. When I tested the stem splitter end to end, there were zero requests with a POST body in the entire session. The model comes down; the audio never goes up.

The cost is real: bigger downloads, slower processing, and a hard ceiling on model size. But for this category, I'll take it.

WeConvertIt is a client-side web application built with Next.js, utilizing ffmpeg.wasm for audio, pdf-lib/pdf.js for PDFs, qpdf-wasm for encryption, and Canvas for image processing. The application is live at weconvertit.com. — first real project, built solo, and I'd genuinely value feedback, especially anything that breaks.

DEV Community: Dodly Forbi

I shipped three AI tools that run entirely in the browser — here's everything that broke