Everyone is shipping AI features. Sentiment analysis on user input, speech recognition without sending audio to a server, image classification that never leaves the device. The privacy pitch is real, the latency pitch is real. But nobody's asking the obvious question:
What does running a neural network in the browser actually cost the user?
I decided to find out. I built a benchmark harness, ran four quantized models in Chrome stable, and measured the impact on Core Web Vitals — specifically INP, the metric Google now uses to rank your site.
Here's what I found.
The Setup
The test uses Transformers.js — the library that lets you run Hugging Face models directly in the browser via WebAssembly. All models were loaded in INT8 quantized format (q8) to reflect real production conditions.
Four models, chosen to cover different architectures and modalities:
| Model | Params | Task | Architecture |
|---|---|---|---|
| DistilBERT | 66M | Sentiment analysis | Encoder (6 layers) |
| BERT-base | 110M | Feature extraction | Encoder (12 layers) |
| Whisper Tiny | 39M | Speech recognition | Encoder-Decoder |
| MobileViT-S | 5.7M | Image classification | Vision Transformer |
The benchmark harness is live at benchmark.mspk.me and open source at github.com/srikarphanikumar/cwv-ai-benchmark. Run it yourself.
What Is INP and Why Does It Matter?
INP (Interaction to Next Paint) replaced First Input Delay as Google's interactivity metric in March 2024. It measures how long it takes for the browser to respond to a user interaction — a click, a tap, a keypress — and paint the result.
Google's thresholds:
- ✅ Good: under 200ms
- ⚠️ Needs Improvement: 200–500ms
- ❌ Poor: over 500ms
INP affects your search ranking. More importantly, it affects whether users feel your app is responsive or broken.
When you run neural network inference on the browser's main thread, you're blocking it. That means if a user clicks something while inference is running, their click won't be processed until the model finishes. That delay IS your INP.
The Results
Here's the full table from Chrome stable on an Apple M-series MacBook Pro, 16GB RAM:
| Model | Load Time | Avg Inference | INP | INP Class | Mem Δ | Mem Pressure |
|---|---|---|---|---|---|---|
| DistilBERT | 7.85s | 25.1ms ±0.5 | 27.8ms | ✅ Good | +59.6MB | 2.5% |
| BERT-base | 6.07s | 83.3ms ±1.5 | 85.0ms | ⚠️ Needs Improvement | +65.3MB | 4.1% |
| Whisper Tiny | 6.71s | 496.9ms ±6.2 | 540.3ms | ❌ Poor | +123.9MB | 7.1% |
| MobileViT-S | 1.15s | 66.7ms ±1.0 | 75.6ms | ⚠️ Needs Improvement | +37.0MB | 8.0% |
The Surprising Findings
1. Parameter count doesn't predict INP
Whisper Tiny has only 39M parameters — the fewest of any model tested. It also produces the worst INP at 540.3ms, more than 19x worse than DistilBERT which has 66M parameters.
The culprit is architecture, not size. Whisper is an encoder-decoder model. It doesn't process the full input in a single forward pass — it runs an autoregressive decode loop, generating output tokens one at a time. Each iteration blocks the main thread. The total blocking time accumulates regardless of how aggressively you quantize the weights.
This means no amount of quantization will fix Whisper's INP on the main thread. It's an architectural constraint, not a tuning problem.
2. MobileViT-S loads 6x faster but still misses "Good"
MobileViT-S loads in 1.15s compared to 6–8 seconds for the text models. That's a huge UX win for initial load. But its INP of 75.6ms puts it in "Needs Improvement" territory despite having only 5.7M parameters.
Vision transformer inference carries disproportionate cost relative to parameter count in WASM environments. Something to watch if you're building image classification features.
3. Memory pressure ≠ memory delta
MobileViT-S has the lowest absolute memory consumption (+37MB) but the highest memory pressure at 8.0%. That 37MB represents a larger fraction of the available JS heap than you'd expect — with implications for mid-range Android devices where heap limits are much tighter.
What This Means for Your Architecture
If you're building with encoder-only text models (DistilBERT class):
You're fine on the main thread. 27.8ms INP is negligible. Trigger inference directly on user interactions without worrying about CWV degradation.
If you're using larger encoder models (BERT-base class):
Don't trigger inference synchronously on interactions. At 85ms, stacking this with other main thread work risks crossing 200ms. Move it to a post-interaction background step — run inference after you've already painted the response.
If you're using any encoder-decoder model (Whisper, T5, BART, etc.):
You must offload to a Web Worker. This isn't an optimization — it's a requirement. The main thread will be blocked for hundreds of milliseconds no matter what you do. Transformers.js supports Web Worker execution natively:
import { pipeline } from '@xenova/transformers';
// Run in a Web Worker to avoid blocking main thread
const transcriber = await pipeline('automatic-speech-recognition', 'Xenova/whisper-tiny', {
worker: true
});
If you're using vision transformers:
Test on actual mobile hardware before shipping. The memory pressure numbers on an M-series Mac will look very different on a mid-range Android.
Limitations to Know
TBT couldn't be captured in the deployed environment. The Long Tasks API isn't available in cross-origin deployed contexts — only in locally-served or Chrome DevTools Protocol environments. The INP measurements are real, but the full main thread blocking profile requires a different setup to measure properly.
All numbers are from high-end hardware. An Apple M-series Mac is not the median global web user's device. INP values on mid-range Android will be significantly higher — potentially 3–5x. The relative ordering of models should hold, but don't use these absolute numbers as production thresholds for mobile.
Try It Yourself
The benchmark is live and open source. Run it on your device, your network conditions, your hardware profile. Export the results as JSON or CSV.
- Live benchmark: benchmark.mspk.me
- Source code: github.com/srikarphanikumar/cwv-ai-benchmark
- Full paper: arXiv link coming soon
If you run it on a mid-range Android or a low-end device and want to share the numbers, I'd love to see them — that's exactly the follow-on data this research needs.
TL;DR
- DistilBERT is the only model that stays in Google's "Good" INP range on the main thread
- Whisper Tiny is "Poor" despite being the smallest model — architecture beats quantization
- Encoder-decoder models require Web Worker offloading — no exceptions
- Parameter count is a bad proxy for browser inference cost
- Memory pressure on mobile is a separate concern from memory consumption
The era of client-side AI is here. Now we need to measure what it actually costs.
Top comments (5)
Sound methodology but the "tested only on M-series" caveat is the part most posts skip. We track LCP via Datadog RUM on the public site and
the desktop-vs-Android-low-end spread is large enough that a single-tier headline number is misleading on its own.
Two things worth adding to the harness:
Whisper Tiny at 540ms INP is the spicy result. Any sense of whether moving inference to a Web Worker shifts INP without making TBT worse?
Whisper Tiny's encoder-decoder setup affects INP quite a bit. 540.3ms is a significant impact, especially since it has fewer parameters than DistilBERT. It makes me question how feasible it is to run these complex models on the client side without offloading. If you're dealing with performance issues like this, PracHub's bank is great for system design questions that often explore latency and architecture. It's a good resource if you're getting ready for tech interviews where this might be relevant.
The architecture point here is more important than the raw parameter count result. Whisper Tiny looking "small" on paper but still blowing past the 500ms INP threshold is a good reminder that autoregressive decode loops can be a UX problem even after quantization.
Have you tried moving the inference path fully off the main thread with a Worker and measuring the residual UI cost separately from model latency? It would be interesting to see how much of the pain is compute time versus scheduler contention, because that distinction usually decides whether teams should optimize model choice or just change where the work runs.
Measuring what in-browser models do to Core Web Vitals is exactly the discipline this trend needs, because browser inference is seductive (free, private, no server) and the costs are real but invisible until you measure: a multi-hundred-MB model download wrecks load metrics, and inference on the main thread janks interactivity. The fact that you measured instead of just shipping is the whole point, perceived performance is the actual product, and an AI feature that tanks INP or LCP is a net loss no matter how clever it is. The honest framing your data supports is that on-device inference is a tradeoff, not a free win: you trade server cost and latency-to-first-token for download weight, memory pressure, and main-thread contention, and which side wins depends entirely on the task and the device. The mitigations usually decide it, run inference in a Web Worker so it doesn't block the UI, lazy-load the model so it doesn't hit initial load, and be honest about the low-end-device floor. And the routing instinct applies here too: keep small models local for the privacy-sensitive, latency-tolerant work, fall back to a server for anything that would blow the device budget. Measure the felt cost, then route accordingly. That measure-what-the-user-experiences instinct is core to how I think about Moonshift. Did moving inference to a Web Worker recover the interactivity metrics, or was the model download the dominant hit?
Same experience with the agent orchestration approach! We approached it from took the opposite route — simpler but more manual — how did you decide on your path?
Followed you so I don't miss the next one 👀