Are AI models running directly in the browser finally good enough to use in production? Let's find out! π
It's been a week since my talk at FrontK...
For further actions, you may consider blocking this person and/or reporting abuse
Awww, thanks, Lingfeng! πβ€οΈ I always say that the comments under my articles are better than the articles themselves! π
Love this! Great articles bring even better discussions π I think just quietly reading along is really fun sometimes.
Thanks
Really interesting article! πΈ
I knew
WebGPUexisted, but I honestly wasnβt sure what kind of real-world scenarios it would be most useful for. Running AI models directly in the browser made it click for me, this is exactly the kind of workload where access to the GPU starts to make a lot of sense. πRight now, as you mentioned, browser AI probably works best as an optional enhancement because hardware and browser support still vary quite a bit.
But I can imagine a future where both machines and browsers become powerful enough that using several small, specialized models for different tasks becomes completely normal.
In that world,
WebGPUmight become a much more ordinary part of frontend development than it feels today. πΈExactly! Honestly, not that long ago, running LLMs in the browser felt a bit like forcing a technology into a place where it didn't really belong. But things are moving fast!
One thing I'm really curious about is your point about WebGPU becoming an ordinary part of frontend development. Because, in my opinion, WGSL (the shader language) is... hard! π It's not particularly intuitive for someone who's spent most of their career programming for the CPU.
So I wonder how we'll solve this. Will we get higher-level libraries and abstractions that make GPU programming more accessible? We already have TypeGPU, but personally, I don't think it simplifies things all that much.
Or will we just let LLMs write our shaders? π But that's not a perfect solution either. If we don't understand what's happening under the hood, AI might fix one problem while introducing three new ones.
I'm really curious to see how this evolves!
Great article, and a really interesting topic!
But I canβt help imagining a typical developer setup: multiple Chrome windows, 10+ tabs in each, and now some of those tabs running local AI models. And some customers are still using pretty old hardware, so Chrome could become a real PC performance killer (sometimes it already is, even without AI π). Of course, it depends on the hardware and how the models are used.
Iβm on my phone right now, but I definitely have to try the demos later!
Hmm, if thereβs a background remover, there must be a background adder too, right? So how do we know you didnβt just make up those FrontKon photos? π€£π€£π€£π€£ Just kidding, of course! π€£
Exactly! π These are definitely features that make sense in specific scenarios, and ideally as optional enhancements rather than something every user has to run.
But to be fair, this isn't even the first time we're doing something like this! Chrome is already doing it for us by downloading a ~4 GB Gemini Nano model. π So downloading mysterious little AI models isn't exactly something we're not already doing. π€£
And you're absolutely right about the photos! These days, a photo proves nothing. I could just as easily generate a picture of myself standing in front of the White House and claim I was there too! ππ
Hm.., now that I think about it, smaller models could actually be pretty useful for Chrome extensions or even browser games hm..π
Yeah, I remember seeing quite a few articles about Chrome silently downloading an AI model in the background. A lot of people were angry about it, or maybe just concerned about privacy and security. π
Oh, I'm definitely planning to write about Gemini Nano at some point too, because that's another really interesting one! π
And yes, Chrome extensions are a perfect use case! You don't even have to make users download a model every time they visit a website.
As for games, more and more things are being built with WASM + WebGL/WebGPU! And apparently, someone even managed to get Cyberpunk 2077 (or what is that exact number xD) running in a browser! π We're living in interesting times!
Really interesting experiments, especially the contrast between semantic search and structured ticket generation.
I recently ran a somewhat related experiment on a Raspberry Pi 4, comparing two small local LLMs against deterministic baselines for tasks like order-ID extraction, classification, and document Q&A. One of the more amusing results was that regex extracted 30/30 order IDs correctly, while the two language models managed 28/30 and 8/30. It reinforced something I've been thinking about: before asking where we can run a model, it's worth asking whether the task needs that kind of model in the first place.
Your semantic-search example clearly illustrates the next step. Sometimes a small, specialized model is exactly the right tool, and running it in the browser offers some compelling privacy and deployment advantages. The ticket generator illustrates the other side: even with a model capable of interpreting natural language, producing reliable structured output still requires validation and an explicit review boundary.
I also appreciate that you included hardware failures and download costs, not just the successful demos. Local inference moves computation and some costs onto the user's device; it doesn't make those costs disappear.
One thing I'm curious about: if you were extending these experiments, would you consider benchmarking the model-assisted features against non-AI implementations of the same tasks? I'd be particularly interested in where the added model complexity starts to pay off.
This is such a great comment! I especially love the regex example! π It's a perfect reminder that sometimes the simplest solution is actually the best one, and we really don't need an LLM for everything.
And I love your idea of benchmarking AI-powered solutions against traditional approaches! I'd be particularly interested in comparing semantic search with more traditional search methods. I think the results could be really interesting!
I have to admit, though, I've got quite a lot on my plate lately, so I'm not sure when I'll find the time for another round of experiments. π But if you're interested in exploring this yourself, please go ahead! I'd genuinely love to read about your findings!
And one more thing: I absolutely love how you put it, that running LLMs in the browser changes who pays the costs rather than eliminating them. That's such a good way to describe it! Because, of course, local AI isn't magically free. Someone still has to pay for all that computation. π
Thanks again for such a thoughtful comment!
Fantastic article! This really resonates with me. π I asked the same question at the beginning of the year: Has Anyone Actually Shipped a Free, Offline Mobile LLM?. Back then, there werenβt many answers, but weβve come a long way since!
All my recent hackathon submissions now include an offline AI fallback because itβs cheap, handy, and surprisingly useful. Itβs exciting to see how much more we can do with local AI now! π
Oh, absolutely! π When I was preparing my recent WebMCP talk, I also played around with local models and experimented with Gemini Nano. Really interesting model, and it even supports tool calling! π Things are definitely getting exciting in the local AI space! π
Hello Sylwia, thank you for this wonderful and highly practical article! π»
Your focus on user privacy, offline capabilities, and your honest take on the real-world limits of local models is truly refreshing. The demos are brilliant and inspiring.
I visited the repository β thanks for sharing the link. Starred it βπΈ
Looking forward to your future posts π
Wishing you continued success π
Thank you so much! β€οΈ I'm really happy you found it useful! And thanks for starring the repo! π₯°
The semantic search one surprised me the most. A 23 MB model doing that well is kind of wild. Also loved that you were honest about the ticket creator being harder than expected and the phone timing out, way more believable than the usual βAI can do everythingβ posts π Just followed you on LinkedIn too!
Thanks, Maame! π Hahaha, exactly! Browser AI is still pretty far from being able to do everything! π And thanks for connecting on LinkedIn too! β€οΈ
Thank you for yet another highly interesting and entertaining article! This setup is an intriguing combination of technologies. WebAssembly as a fallback for WebGPU, not exactly the original goal of this technology, but this is how life goes.
Thank you! π Well, you know, AI inference involves a LOT of matrix multiplication, and GPUs are particularly good at that. There's a reason why graphics cards are so ridiculously expensive these days! π€£
Running heavy processes directly in the browser is fascinating. I completely agree with the point in the comments about using Web Workers so the main thread doesn't freeze. I've been focusing heavily on pure Vanilla JS lately to keep real-time text processing blazing fast without relying on bloated frameworks. For these intense in-browser features, do you think sticking to Vanilla JS is necessary for maximum performance, or is the overhead of React negligible compared to the WebGPU load?
I'd say the overhead of React is negligible in most cases. If inference runs on the GPU and React is mainly responsible for the UI, the framework itself usually won't be the bottleneck. Avoiding main thread blocking and unnecessary re-renders matters much more. So I definitely wouldn't give up React just to use WebGPU efficiently!
That makes perfect sense! Separating the GPU inference from the UI rendering definitely puts the React overhead into perspective. I completely agree that managing unnecessary re-renders is the real challenge there. Out of curiosity, when handling those heavy WebGPU tasks alongside React, do you prefer managing the state with native context or a lighter external library to avoid those re-render traps?
Honestly, I don't think you need an external state management library for this! π In one of my demos, for example, I was just passing a tiny uniform to WebGPU. The heavy lifting happens on the GPU anyway, so I prefer keeping the React side as simple as possible. I'd only consider a dedicated state management solution if the application became complex enough to actually need one.
Ah, gotcha. Yeah, adding something like Zustand or context API just for that would definitely be overkill. Keeping the React side simple and letting the GPU do the heavy lifting makes a lot of sense. Appreciate the tip!
Haha, I'm actually working on a talk about WebAssembly, WebGPU, and React right now, and I'm starting to feel like the main takeaway is basically "keep all that stuff as far away from React as possible." π€£
The embedding-first approach makes sense, especially for an offline product. One thing I'd watch closely is how the retrieval pipeline behaves as the local index grows. Even if generating embeddings is cheap, storing them, updating stale vectors, and ranking results can become the real bottlenecks.
For a support-ticket system, I'd also benchmark retrieval quality against actual queries, including cases where users describe the same bug with completely different wording. A lightweight model that consistently retrieves the right ticket can be far more useful than a larger model that looks impressive in a demo.
That's a really interesting point! And I completely agree, there are definitely questions around scalability, incremental updates, retrieval quality, and so on.
Of course, my demo is nowhere near production-ready (otherwise I'd be selling it instead of giving it away as a demo! π). There's still a lot to test and optimize.
But I genuinely think this is a very promising direction worth exploring. Especially because, as you pointed out, the embedding model is so small! The relatively low cost of running it locally makes the whole approach particularly interesting. π
One thing I find interesting here is the offline-first use case. Imagine a field-service app with thousands of repair manuals cached locally. You could generate embeddings on-device and search them without sending queries or documents to a server.
The tricky part is updating that local index when the embedding model changes. You'd need to keep the old index usable while rebuilding the new one, especially on devices with limited storage. That's where the small model size becomes a real architectural advantage, not just a nice demo feature.
Exactly! Offline-first is becoming an increasingly common approach, and I think this is a great example of where local AI could really shine!
As for rebuilding the index when the embedding model changes, that's definitely an interesting challenge. But I think it's worth distinguishing between updating documents and updating the model itself. While document embeddings may need to be refreshed regularly, replacing the entire embedding model would probably happen much less frequently.
In a real-world application, you could stick with a specific model version for quite a long time and only rebuild the entire index when a new model offers a meaningful improvement in retrieval quality.
Still, plenty of interesting architectural challenges to explore here!
That distinction matters. Imagine a field-service app where technicians store repair logs offline, then sync them days later. A model upgrade could leave the local index with vectors generated by the old model while new queries use the new one. Even if the vector dimensions match, the embedding spaces might not, so retrieval quality could quietly degrade.
I'd version the model, tokenizer, and preprocessing pipeline together, then build the new index alongside the old one and run a retrieval evaluation before switching. For example, test whether a query like βhydraulic pressure drops after 20 minutesβ still retrieves tickets describing the same fault as βpump loses pressure once the system heats up.β That kind of semantic mismatch is where a small offline model needs to prove itself beyond the demo.
I have found one more usecase, Sylwia :)
Browser Extension that replaces your New Tab with Dev.to interface and summarizes articles with in Browser LLM :)
this is how I found your article, hehe :) would've miss otherwise.
Wow, this is cool!!!! πππ Did you maybe write a post about it?
Yep :)
But didn't get much attention on Dev unfortunately :(
Details UI polish and calm tab design
I wanted to read more of DEV.to. So I built it into my New Tab
Maybe if you try it out and like it, you can help with exposure :)
If it's something worth for everyone on Dev to use, we could push it so more people see it and evaluate if they like it to install.
For me for couple weeks now it's part of the day. I find articles that wouldn't see otherwise. And it helps Dev.to overall with exposure and DAU/MAU metrics :)
Thanks, Sylwia, for the code and the live demo. I will be testing the demos soon. I would ship the embedding model before the ticket model. Item 4βs embedding model is about 23 MB, nearly zero cost to the user, whereas the ticket model weighs in at 786 MB, with much higher cost.
Mobile data is not free for many users. It costs real money. I havenβt been able to test models in a browser yet, but file size is important for my project (StacksNG), which is completely offline.
Which two of your demos timed out on your phone? Did the semantic search work? Iβll let you know what I find out.
Hey Daniel, thanks for the comment! And you're absolutely right! Mobile data isn't free, and that's a really good point. One possible safeguard would be to only allow model downloads over Wi-Fi.
That's also why I think we're not quite at the point where we should rely on these models for mission-critical tasks.
On my phone, two demos didn't work: the ticket creation demo and the last one using the camera. The others worked, although they were noticeably slower than on desktop.
The exception was the embeddings model, which actually ran almost as fast as on my desktop! Although, to be fair, I had a WebAssembly fallback there. π
Looking forward to hearing how your tests go!
I love seeing more people experimenting with this stuff! I've been playing around with browser-based AI quite a bit myself, mostly with Transformers.js and WebGPU, and it's honestly amazing how much you can do entirely in the browser now.
I've also had my fair share of headaches with WebGPU compatibility, though. Getting something working beautifully on one machine doesn't necessarily mean it'll work on another, which makes the production side of things a little tricky.
I completely agree with your point about smaller, specialized models, too. Not everything needs a big LLM, and sometimes those tiny models are surprisingly good at one particular task. I think that's where browser AI has some really interesting potential, especially when you consider the privacy benefits of keeping everything local.
I haven't experimented with WebMCP yet, but it's been sitting on my ever-growing list of things I want to play with!
Really enjoyed the practical examples here, and especially that you included the limitations and things that didn't work out. That's just as interesting to me as the successful experiments.
Thank you so much! β€οΈ And yes, I've had my fair share of WebGPU compatibility issues too, not just between devices, but even between browsers! π Especially when I was writing shaders myself. Sometimes something would work perfectly in Firefox but fail in Chrome! π Luckily, things have improved quite a bit over the past few months. I really believe WebGPU will eventually be supported everywhere. It's just a matter of time! π
Saved for research future purposes.ποΈ
That's awesome! Hope it comes in handy! πβ€οΈ
The download sizes make the optional-enhancement argument very concrete. For the ticket creator, Iβd want a visible download size and cancel button before loading nearly 800 MB, with the normal ticket form usable throughout. Cold-load UX seems just as important here as inference speed.
That's a fair point! π I think it's especially important on mobile devices, but on desktop I'm not entirely convinced a separate confirmation step is always necessary. I'd definitely show the download size and progress, though, and keep the regular form fully functional while the model loads. And of course, I wouldn't download 800 MB automatically without the user choosing to use the feature! π Cold-load UX absolutely matters, but I think the exact approach depends on the device and use case.
Fair distinction. Choosing the feature can be the consent step itself; an extra modal isn't always helpful. Clear size/progress and a working regular form are the important parts. Thanks for pushing back on that. I'd be glad to stay connected here and compare notes on practical web UX.
The ones that "actually work" always share a trait: they show you something you'd otherwise miss, instead of generating more noise. That's the bar I use for AI features now β does it reveal or does it just produce?
That's a really interesting way to look at it! I especially like the distinction between revealing something useful and just generating more content. We definitely don't need more noise. π
Exactly. The "reveal vs. produce" filter cuts through most AI feature hype. Revealing is also cheaper β you're surfacing what already exists instead of paying to generate more of what nobody asked for. That's honestly the design principle I'd put on a wall for anyone building AI tools right now.
This is a strong production framingβespecially treating local AI as an optional enhancement rather than a critical dependency.
One boundary Iβd add from building an AI browser: local inference changes the data path, but it doesnβt automatically solve the trust problem. Model files and application code still arrive through a web supply chain, page content can be adversarial, and multiple tabs may compete for the same memory/GPU budget. For agentic features, Iβd still keep side-effecting actions behind explicit capability checks and human confirmation, even when the model runs locally.
Iβd also love to see benchmarks include cold-start download time, peak memory, battery impact, and cache eviction/reload frequencyβnot only inference latency. Of your examples, semantic search feels like the best first production feature: small model, clear fallback, measurable quality, and a real privacy win.
Exactly, I agree 100%! And I'd absolutely love to see benchmarks like that. Am I going to run them myself anytime soon? Well, I won't lie, probably not. π€£ So if you feel like exploring this topic, please go ahead!
And yes, I completely agree about human confirmation for agentic features. Actually, as you can see, I'm taking an even more cautious approach by treating local AI as an optional enhancement rather than a critical dependency.
And I agree that semantic search is probably the winner here when it comes to real-world production use cases, although I think the background remover has the biggest "wow" factor.
played golf and i shot 4 hoels and i won the cup of wining
Your architectural split in
practical-browser-ai- especially isolating each pipeline in its own lazy-loaded Web Worker and pairingWorkboxCacheFirstrange-request caching with strict human-in-the-loop review boundaries - nails the exact engineering reality of client-side inference right now. Digging through your repository documentation alongside the article, two low-level decisions stood out immediately: defaultingXenova/all-MiniLM-L6-v2(~22.7M parameters, ~23 MB quantized) to the CPU/WASM execution path to bypass initial WebGPU shader compilation overhead for small batches, and explicitly choosingq4overq4f16foronnx-community/Qwen2.5-0.5B-Instruct. While testing these failure modes locally in anonnxruntimeenvironment, we reproduced that exactfp16intermediate activation overflow (>65,504 in residual/MLP accumulations) where logits collapse intoNaNand anargmaxcomparison loop defaults to token0("!"in the Qwen vocabulary), emitting the dreaded"!!!!!"stream cited inmicrosoft/onnxruntime#26732andhuggingface/transformers.js#1317. Similarly, benchmarking your query ("My edits disappear after I leave the page") against a stemmed lexical baseline vs. L2-normalized 384-d unit-vector dot products (max|cos - dot| = 7.45e-9at ~0.0085 ms for 22 docs and ~1.58 ms for 10,000 docs) shows how lexical matching gets zero-hit amnesia on"Auto-save not persisting form state"while a 23 MB encoder resolves it effortlessly.Building on your struggle with
Qwen2.5-0.5B-Instructand structured output, recent benchmarking on small models (0.6B-4B) under grammar-constrained decoding (OutlinesandXGrammar, Chavan 2026,arXiv:2609.23742) formally confirms what you discovered empirically: while token masking raises structural JSON schema validity from92.9%to100%, a scale-dependent semantic gap persists ("hollow rescue", where required JSON keys are present but populated with degenerate values or mis-ranked priorities). That makes your one-shot prompt + balanced-brace extraction +Zodcoercion pipeline inparseTicket.ts- and your transparent decision to appendFlorence-2-base-ftcontext hints in code rather than hallucinating instruction-following on fixed<DETAILED_CAPTION>task tokens - the most resilient pattern available. This leads me to two questions: (1) WithTransformers.js v4(@huggingface/transformers) piloting the hash-basedCross-Origin Storage API(env.experimental_useCrossOriginStorage = true) alongside browser-managedChrome Built-in AI(Gemini Nano), how do you see the boundary evolving between shipping app-specific ONNX weights in a PWACacheStorageversus delegating general text/VQA tasks to browser-native APIs to avoid per-origin duplicate downloads of 400-800 MB models? (2) As you prepare your upcoming browser-based AI agents post, do you plan to keep tool-call validation purely in post-generationZodrecovery withModelRegistrypre-flight checks, or are you tempted to pair a lightweight grammar bitmask (or a hybrid router whereall-MiniLM-L6-v2pre-filters intent before waking up the0.5Bgenerator) so a mobile GPU like the Galaxy S24 FE doesn't time out mid-thought?At least by offloading inference to the client's
WebGPUdevice, whenQwen2.5-0.5Bhits anfp16NaNexistential crisis and screams"!!!!!"into the void, it's heating up the user's lap for free instead of burning your cloud bill at $0.002 per exclamation mark.Wow, thanks for such a detailed analysis! π And yes, I actually ran into that exact "!!!!!" issue with Qwen myself, so this made me laugh!
As for the browser-native APIs, I think that's definitely where things should be heading. I'd love to avoid shipping hundreds of MB of model weights per app whenever possible, while keeping app-specific models for more specialized tasks.
I'm also very interested in hybrid approaches, especially using smaller models for intent detection before involving a larger one. Still experimenting with that, though!
And haha, at least the GPU is suffering instead of my cloud bill! π
Running AI models directly in the browser could improve privacy, reduce latency, and lower infrastructure costs. The real test is how well they perform across different devices and real-world use cases. Curious to see what you found!
Exactly! π And that's what I wanted to explore with these demos: what actually works today, what doesn't, and where the biggest limitations still are!
The WebGPU adoption number tracks with what I've seen β I ran a quick check last month across 12k visitors on a side project and only ~8% had WebGPU enabled in their browser. Most of those were Chrome on desktop. So yeah, it's fast when it works, but you're shipping a fallback path for 90%+ of users and that's the part nobody talks about. Would love to see the actual production numbers if you've shipped this to real traffic.
Thanks for sharing these numbers! That's a really interesting point, and I absolutely agree that browser support is something we need to consider. But I also think it depends a lot on the project and its target audience. For example, in the applications I work on, most users have WebGPU enabled, simply because they're primarily desktop applications. And in some scenarios, we can even make WebGPU support a requirement.
Also, WebGPU adoption is progressing, and this isn't exactly a new problem in web development. Remember when we had to transpile everything because browsers didn't fully support ES6? π We're seeing similar challenges with newer technologies today. WebMCP is another interesting example: there's a lot of excitement around it, but people sometimes forget that it's still experimental!
That said, your point is absolutely valid. We need to consider actual adoption numbers, especially when building applications for a broad audience. And that's exactly why I keep emphasizing that, at least for now, browser AI shouldn't be relied on for mission-critical functionality without a reliable fallback.
Thanks again for bringing some real-world numbers into the discussion!
Although most of these are pretty cool, I would actually prefer your initial bug report in number three.
It has way more context, way more little side details that may or may not matter, and generally just more. Yeah, 90% of the bugs you submit will be read, immediately recognized, and pushed into a Bug#1234 bucket. But for those last 10%, the details can matter.
I totally get your point! π Unfortunately, you know how it is with support tickets. You have to fill in all those fields and end up spending half an hour figuring out what goes where. π But hey, that stream of consciousness could easily be stored in an additional field too! Anyway, I dream of a feature like this every single time I have to fill out a support ticket! π
the small specialized model beating the general one is the lesson that stuck with me here. i keep hitting the opposite problem β trying to squeeze a big model onto hardware that was never meant for it, and the quality drop shows up fast. the semantic search demo is the one i would actually ship. embedding a support ticket base locally and never sending it to a server is the privacy win that makes the whole local-inference pitch worth it. the webcam pause-and-ask is fun but i agree, not reliable enough for accessibility yet. transformers.js getting to this point in a year is the real story. the gap between "fun curiosity" and "actually useful" closed a lot.
Exactly! I was also genuinely surprised by how well semantic search works with these local models! And it's not just about privacy. We don't always have internet access, after all. Some people spend hours using apps out in the field, often with little or no connectivity. So having these features work entirely offline can be a huge advantage!
Really practical exploration. I especially agree with positioning these as "optional enhancements, not critical dependencies"βthat's exactly where browser-based AI makes sense right now.
We've hit similar tradeoffs at RollTok when experimenting with local inference: model size vs first-load experience. The models you picked (RMBG-1.4 at 44M, MiniLM-L6 at 22M) strike a good balance between parameters and performance.
Quick question on your Semantic Search case: have you run into multilingual scenarios with all-MiniLM-L6-v2? In our tests, it works great for English-heavy docs, but embedding quality drops noticeably with mixed Chinese-English content. If your use case involves multiple languages, you might need to consider variants like multilingual-MiniLM.
Also, the Background Remover case is interestingβ170MB model running in-browser is definitely more private than calling an API, but I'm curious about actual performance. In my tests, WebGPU acceleration makes a huge difference. Did you optimize for fallback when WebGPU isn't available?
Thank you! I haven't tested multilingual semantic search yet, but you're absolutely right about the limitations of all-MiniLM-L6-v2. A multilingual model would definitely be a better choice for that use case!
And yes, I do have a WASM fallback for the Background Remover! I even tested it on mobile. The initial loading time is painfully slow π, but once everything is loaded, it actually works pretty well! Of course, WebGPU makes a huge difference in performance, but I always try to include a fallback whenever possible.
Really interesting article! Running AI directly in the browser has huge potential for privacy and cost savings. Semantic search and background removal are especially impressive. Great insights into both the possibilities and limitations of browser-based AI! π
Thank you so much! π Those two demos are actually my favorites too! A lot of people in the comments have also pointed out that the tiny semantic search model would be perfect for apps that need to work offline. And I totally agree!
how do you handle model size vs init latency on lowβend devices, especially when swapping between onβdevice inference and a remote fallback? also, what's your strategy for keeping the wasm bundle size down without sacrificing quantization accuracy
Good questions! π For these demos, I mostly relied on Transformers.js to handle the inference runtime, so I didn't implement any custom WASM bundle optimizations or local-to-remote fallback logic. My focus was more on exploring what we can realistically achieve with existing tools and smaller models. For low-end devices, I think choosing the smallest model that does the job well is a good starting point. As for the WASM internals and quantization trade-offs, that's probably a question better suited for the Transformers.js or ONNX Runtime maintainers! π
Great read! I was experimenting with EmbeddingGemma 2 and built a small browser game for the ongoing hackathon: doppel-ws59.onrender.com/. I'm using embeddings and cosine similarity to find images that are semantically close even though they're completely different things. Was pretty interesting to see how much I could do with vector similarity right in the browser!
That's so cool! π I really think embeddings have a lot of potential in the browser, especially for offline use cases! π
I feel like the privacy and offline capabilities alone make this worth exploring and I really like your point about treating these as optional enhancements rather than blindly depending on them in production, not every problem needs a massive AI model, sometimes a small specialized one does the job better.
definitely something worth experimenting with!!
Exactly! Maybe one day we'll be able to rely on them for production-critical features, but I don't think we're quite there yet! π
yesss!! waiting for your next article on this now βΊgenuinely loveee your writing πΈ
Awww thanks a million βΊοΈ
Great article and an honest set of demos. One angle I haven't seen in the thread yet: when inference moves into the browser, you lose your server-side eyes. Today, if a backend model starts returning garbage or latency spikes, your dashboards catch it. With in-browser inference, there's no telemetry coming back by default β you're flying blind about which models fail on which GPUs, how often the WASM fallback kicks in, or whether users silently give up during a 786 MB download on a flaky connection. And the browser can evict those cached weights without telling you, so "works offline" quietly becomes "re-downloads everything" for some fraction of users you'll never hear from. To ship this responsibly you'd need a client-side telemetry layer (on-device quality sampling, versioned model manifests, download/abort analytics) built with the same privacy care as the inference itself. That ops work is arguably harder than the demos β and I suspect it's the real gap between "impressive experiment" and "production feature." Would love to see a follow-up on what you'd actually instrument if one of these shipped to real users.
Oh, that's an excellent point! I completely agree. We definitely need client-side analytics and monitoring, just like we do on the backend, especially in larger projects. And things get even more complicated when you have to account for different devices, GPUs, browsers, and fallback mechanisms. π
I also think the privacy aspect is particularly interesting here. We want to move inference to the browser partly to protect user data, so we need to make sure our telemetry doesn't undermine that benefit.
And you're absolutely right, getting a demo to work is one thing, but making it reliable in production is a completely different challenge. This could actually be a great topic for a follow-up article! π
I donβt know why, but the potential of offline AI generation in a browser-based form is still very poorly realized. The technology exists and can be demonstrated in benchmarks, but there are very few solutions that ordinary users can actually use.
Right now, Iβm looking for a browser-based solution that can run a model for converting plain text into ArgDown. The model itself already exists. Specifically, Iβm looking for a browser-based solution that doesnβt need internet access to run or process prompts. And what do I find? Lots of projects that seem to promise this, but canβt run locally from files. So theyβre not really browser-based at all - theyβre self-hosted. In my case, thatβs not a dealbreaker, since Iβm looking for something that works with WebExtensions, and browser extensions donβt have this problem. Still, someone might need a browser-based solution that runs locally without requiring an extension.
And the surprises may not stop there: the code might import scripts from centralized CDNs like jsDelivr, which effectively makes the solution unusable without an internet connection.
So it seems easier to build yet another hack that requires users to set up local inference with LM Studio or something similar than to build a proper extension. But thatβs exactly what Iβd rather avoid.
I totally understand your frustration! π But I think it's worth distinguishing between three things: browser-based inference, fully offline execution, and running an application directly from local files without a server. They're not necessarily the same thing.
For example, you can have an application served over HTTP that performs all inference locally, without sending any user data to a backend. And with Service Workers, you can take it a step further and make the entire application work offline, as long as all the required assets, including model weights, are cached locally.
My demos don't have Service Workers YET π , but the versions I used at conferences did! They worked completely offline. In fact, I didn't even connect my laptop to the internet during the presentations.
That said, I completely agree with your point about hidden dependencies like external CDNs. If your application still needs to fetch scripts or other resources from the internet, it's not truly offline. And making something work in a demo is definitely easier than delivering a reliable, fully offline experience for real users! π
Woow, Pretty cool stuff! I think the semantic search and background remover could be really useful in everyday projects. The best part is that everything runs in the browser without sending your data to a server. The model download sizes and performance on different devices might be a bit of a challenge, though. Definitely interesting to see how this develops.
Keep it up & thanks alotπ
Thank you! π And exactly, performance across different devices is probably the biggest challenge here. That's where semantic search really shines, because the model is so small! π
I actually never knew about running local AI models in the browser. This really looks promising, but as you mentioned, the heavy load they carry in terms of size will be a drawback, joined with the fact that they are not fully functional on mobile devices. It's interesting to see where this goes from here.
Exactly! π Many technologies started with limited support, and now we can't imagine our lives without them. I'm really curious to see how this one evolves!
Really interesting project. Is the repository open source, and are you welcoming contributions as of now?
Thanks! π Yes, the repository is open source, and contributions are definitely welcome! These are primarily educational demos, so I'd love to see improvements, bug fixes, or experiments with different models. Feel free to open an issue or PR!
Thanks for clarifying. I noticed the repository doesn't seem to include a License file, so Iβm curious how contributors can use, modify, and share the code.
RMBG-1.4 at about 170 MB is smaller than I expected for background removal that clean, and running inference in a Web Worker so the UI stays responsive is the detail I'd copy first. I've mostly treated in-browser models as toys, so calling them optional enhancements, like Florence-2 drafting alt text, makes sense to me. How long does that first model download take on a mid-range phone?
On a phone, it actually takes quite a while, around a minute! π The other models are pretty quick, except for the semantic search one. But I expect things to improve, especially now that WebGPU is becoming available in mobile browsers, including on iPhones!
The semantic search demo is the one I'd ship first too. I've found that small tools work best when they solve one clear problem, and keeping the data on-device is a nice bonus. The model size makes this much easier to roll out.
Exactly! I was genuinely surprised by how well it works myself!
Ok... I thought about creating a JavaScript version of my inference engine, but discovered it would be useless in a webbrowser.
That's definitely possible! π You should check out ONNX Runtime Web. It might be exactly what you need for this experiment!
Oh... I didn't know you replied before I edited my comment above. π I'll look-up the ONNX Runtime. Thanks for the tip!
Hahaha, that's because it was such a great comment! π
Solid, practical survey. Treating browser AI as optional enhancements rather than a required backend replacement is the right framing.
The demos that stand out most are the specialized ones β especially semantic search with a small embedding model and background removal. Those show where local models already deliver clear value without needing frontier-scale LLMs. Keeping humans in the loop for structured outputs (ticket fields, alt text) is also the sensible production pattern.
The limitations section is equally useful: WebGPU unevenness, first-load model sizes, and mobile timeouts are exactly the constraints teams need to plan for. Caching helps on return visits, but the initial download still matters.
Nice to see the full code and live demos shared. Makes the βactually worksβ claim easy to verify.
Thanks so much! π I completely agree, smaller, specialized models are where in-browser AI really shines right now. And yes, the initial download and performance across different devices are still major challenges. Hopefully, we'll see some improvements there soon!
This is a refreshingly honest and practical look at browser AI.
Most pieces either hype local models as βthe futureβ or dismiss them as toys. You did the useful middle path: built five real features, measured what actually works, and was transparent about the gaps (model sizes, mobile WebGPU, inconsistent quality on structured output and contextual alt text).
Semantic search with the tiny embedding model and the background remover feel especially production-viable right now as optional enhancements. The Smart Ticket Creator is the one that best illustrates both the promise and the friction β getting reliable structured output from a 0.5B model is still more prompt + validation work than people expect.
Appreciate the clear βoptional enhancement, not critical pathβ stance. Thatβs the right framing for 2026.
Thank you so much, Anne! β€οΈ That's exactly what I wanted to show with this article! I think browser-based AI has reached a point where we can start building genuinely useful features with it, but we still need to be realistic about its limitations. And yes, the Smart Ticket Creator was definitely a good reminder that getting a model to generate something is one thing, but getting reliable, structured output is a whole different challenge! π I'm really excited to see where we'll be a year from now!
The WebGPU to WASM fallback is doing more work than the headline. Most tutorials ship WebGPU only β graceful degradation is what makes this production viable on devices without GPU access yet.
Also the Web Workers architecture is the unglamorous detail nobody demos. Inference on the main thread means your UI freezes halfway through a sentence. Offloading that before you add features rather than retrofitting it later is the call most teams get backwards.
Curious if you hit cold start latency on the first inference call? Loading 44M params even compressed has a perceptible pause on slower connections. Did you add any loading UX or let it be?
Exactly! I keep saying that WebGPU still needs a fallback! And yes, of course I added a loading state so users know the model is being loaded. It's pretty basic, since this is just a demo and not a production app, but informative enough! π And I totally agree about Web Workers. They're an absolute must here!
This looks good. What can I use it for? Is it only for web applications, or is it possible to use it for other things as well?
Unfortunately, this particular implementation is web-only, but the same concept can definitely be applied to native apps too!
Thats not good.
What a fantastic article Sylwiaπ I knew about WebGPU and Transformers.js but I always assumed browser AI was still a fun toy. Your article changed that view completely.
The Smart Ticket Creator demo is what sold me. Watching a 500M parameter model turn messy text into structured JSON, all locally without an API call, feels like a glimpse of where things are going.
Curious: did you find any specific trick that helped with JSON consistency, or was it mostly prompt engineering plus Zod validation?
Thank you so much! β€οΈ That was actually the hardest part! π And yes, it was mostly prompt engineering combined with Zod validation. But I've heard that newer models are getting better at generating structured outputs, so hopefully this will become much easier in the near future! π
I'll be honest β I had never looked into browser-based AI before this, and I didn't know any of it was possible. Reading this was a genuine surprise.
What made it click for me is the semantic search example. A 23 MB model doing something more useful than keyword search is a much lower barrier than I assumed this kind of thing had. It made me think about things I could actually try, not just read about.
My background is mostly backend and ordinary frontend work, so I'm starting from zero on WebGPU and Transformers.js. If someone wanted to learn this properly, what would you point them to first β the Transformers.js docs, or something else you'd recommend before that?
Hey, thanks for your comment! π If you're interested in running AI models in the browser, the Transformers.js docs are definitely enough to get started! If you want to dive deeper into WebGPU itself, that's a whole different story involving shaders and GPU-specific code. But honestly, I don't think you need to go that deep unless you have a specific reason to. It's worth understanding what WebGPU is and how it works, but you don't need to master it to start building cool things with browser-based AI! π
a new food for a thought for me! this is a very interesting read tbh π
Thank you! I'm so glad you enjoyed it! π
Wow A live demo? You got my star!
This is seriously impressive!
That's something I would actually use!
By the way, currently it shows "removed" on the original side and "original" on the "removed" side.
Hahaha, you're right! π I'll have to fix that later! But come on, you have to admit it's ridiculously fun to play with! π€£
congratilations ! π€©
Thanks βΊοΈ
Thanks for sharing this! π
π₯°π₯°
I think a browser can't stream rtsp sources..
You're right, browsers don't support RTSP natively. You'd need something in between to convert the stream to WebRTC or another browser-compatible format. But my demo uses getUserMedia(), so it doesn't need RTSP at all.
cool!
Thanks!
I've tried the background remover, it failed miserably and removed everything π
Hahaha, sorry about that! π Maybe fork the code and try a better prompt? π
Can Do everthing i want.
Ah, clever!!!
I like this.
Thanks π
You can spend quite some time with that background removal, this used to be a painful process in Photoshop, back in the old days.
Hahaha, oh yes, I remember those days! π And the result often looked so obviously cut out, you could spot it immediately! π
I donβt know why, but looking at these images makes me happy π
Aww it's so sweet! Indeed, I was very happy there βΊοΈ
It was an interesting article.
Thanks a lot βΊοΈ
The background remover demo is the one that surprised me most β 44M params for that quality is genuinely impressive, and the βphoto never leaves the deviceβ angle is the right selling point for e-commerce use cases. I've been poking at browser-side segmentation too and built a small pure-frontend background remover at wtoolskit.com/zh/background-remover/ β different model stack, but same bet that on-device inference is finally good enough for this specific task. Your licensing note on RMBG-1.4 is a good reminder that the model layer is still the messy part
The third one surprised me β I assumed it was a gimmick until I used it on a real page. Devtools AI explanations are genuinely handy for unfamiliar CSS.
Hi, I cretaed a libraries. AI cretaed an article about it via md and html version. Can I post here md files?