What breaks when you ship 21 AI tools that never touch a server

Dazhi Wang — Sun, 06 Sep 2026 02:20:11 +0000

We ship 21 AI-powered tools on FreeToolHub — email and report writers, resume rewriters, background removal, image upscaling, audio transcription, LaTeX OCR, translation, contract analysis. None of them have a backend. No API keys, no upload endpoint, no inference server. The model downloads to your browser and runs there; your text and files never leave the device.

That was a product decision (privacy is the pitch) but it turned into a year of hitting every sharp edge WebML has. This is the honest version — including the tool we had to kill.

The stack, briefly

@huggingface/transformers (transformers.js) 4.2.0 on top of onnxruntime-web
WebGPU where available, WASM as the fallback — the same model can ship two quantizations for the two backends
One TypeScript registry as the single source of truth for all 18 models: HF repo, dtype, backend, download size, min device memory, and which tools use it. If a model ID appears anywhere else, we treat it as a bug.

A taste of what that registry looks like in practice:

{
  id: 'smollm2-1.7b',
  sources: [
    { hfRepo: 'HuggingFaceTB/SmolLM2-1.7B-Instruct', dtype: 'q4f16', backend: 'webgpu', downloadSizeMB: 900 },
    { hfRepo: 'HuggingFaceTB/SmolLM2-1.7B-Instruct', dtype: 'q4',   backend: 'wasm',   downloadSizeMB: 1400 },
  ],
  minMemory: 2048,
  loadTimeoutMs: 600_000,
  usedBy: ['/ai-email-writer', '/ai-resume-rewriter', '/fine-print-decoder', /* +13 more */],
}

Now the part the README doesn't tell you.

1. SharedArrayBuffer, or: how COEP almost ate our ad revenue

Several models need threading, which needs SharedArrayBuffer, which needs the page to be cross-origin isolated. The textbook fix:

Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp

require-corp means every cross-origin subresource must opt in with CORP or CORS headers. Sounds reasonable until you remember what a real page loads: an ad network, a consent-management script, an analytics beacon, image CDNs. None of them send Cross-Origin-Resource-Policy — so require-corp silently breaks all of them. On a free-tool site, that's not a rounding error, that's the business model.

The escape hatch nobody mentions enough:

Cross-Origin-Embedder-Policy: credentialless

credentialless gives you cross-origin isolation (and therefore SharedArrayBuffer) without requiring every third-party resource to opt in — the browser just strips credentials from cross-origin requests instead of blocking them. No-cors subresources still load. Ads still serve. Workers still thread.

Cost to know about: fetch() of cross-origin resources that do need cookies won't send them. Our model downloads come from HuggingFace and a fallback CDN with no cookies involved, so we don't care. If your pipeline needs credentialed cross-origin fetches, this will bite you.

2. The download is the product

The number one UX problem isn't inference speed, it's that a 1.7B LLM is 900 MB at q4f16 and 1.4 GB at q4. Things that actually moved the needle:

Two quantizations per model, chosen by backend: q4f16 for WebGPU (smaller and faster), q4 for WASM where fp16 ops are the bottleneck. Picking one size for both backends is leaving a third of your users behind.
A genuinely small sibling. SmolLM2-360M (200 MB) handles rewrites, captions and short generation fine. For users on a phone over LTE, "good enough in 30 seconds" beats "great in 4 minutes."
Aggressive caching in IndexedDB with versioned keys, so the download happens once per model version — and bumping a model version intentionally invalidates it.
A 10-minute load timeout and honest progress UI. A model download that silently stalls at 87% reads as "this site is broken." A progress bar with the MB count reads as "I'm waiting for a download."

3. onnxruntime-web version hell, concretely

transformers.js pins its own onnxruntime-web. If your app also imports onnxruntime-web directly (we do, for custom ONNX pipelines that don't go through transformers.js), npm will happily give you two different ORT versions in one bundle, and they fight over the WASM glue.

Our package.json ended up looking like this:

"dependencies": { "onnxruntime-web": "1.21.0", "@huggingface/transformers": "^4.2.0" },
"overrides": { "@huggingface/transformers": { "onnxruntime-web": "1.25.1" } }

Yes, that's two ORT versions on purpose. The general lesson: when a JS lib ships WASM artifacts, the binary version and the npm version are only loosely coupled, and every loader that fetches .wasm at runtime is a chance to mix them.

That bit us a second time with @imgly/background-removal, which fetches its WASM and glue .mjs from its own CDN at runtime. A CDN-side update changed the artifact version out from under us and inference just… hung. The fix was to stop trusting the runtime fetch: at build time we copy the exact ort-wasm-simd-threaded*.wasm / .mjs files from our installed node_modules/onnxruntime-web into public/, so the versions can never drift.

4. What we killed: small models and structured output

The most valuable engineering lesson cost us a whole tool. We built an AI presentation generator: describe your deck, get slide JSON, render it. SmolLM2-1.7B is genuinely good at prose. It is not reliably good at emitting valid, well-formed JSON that matches a schema, every time, without a server-side retry loop.

On a server you'd wrap the model with constrained decoding or a validation-repair loop and hide the failures. In the browser, every retry is another user-visible spin of a model that's already been loading for two minutes. Failure rates that would be invisible behind an API became the whole UX.

We pulled it. The current rule: browser-only features must degrade to "the model said something imperfect", not "the product returned garbage." Prose, rewrites, summarization, extraction — fine. Rigid structured output — wait until constrained decoding is realistic in-browser.

5. Honest limits

The 1.7B class is a drafting assistant, not a GPT-class one. We say so on the tool pages.
First-load cost is real. Repeat visits are near-instant (cached), but your landing page has to convince someone to download 200–900 MB of model, so the tools better be the point.
WebGPU support is still uneven; the WASM fallback keeps every tool working, but it's noticeably slower and heavier.

Would we do it again? Yes — zero marginal inference cost, no data leaving the device, and no per-user API bill that scales with success. But budget for the packaging work: headers, caching, quantization matrix, and version pinning are the actual product.

Try the tools without installing anything: freetoolhub.org/ai-workspace — everything runs on your device, and the DevTools network tab is the proof.