I Built a RAG AI Assistant That Runs in the Browser with WebGPU

Sébastien Descloux — Sun, 27 Sep 2026 19:28:01 +0000

A website may already contain the answers visitors need.

The real problem is often finding them.

So I built and deployed a first version of a RAG AI assistant for my own website, with one particular constraint:

The response model runs directly in the visitor's browser using WebGPU.

This was not meant to be another chatbot connected to an API.

I wanted to explore what happens when retrieval, local inference, source-based answers, conversation context, and a CMS are designed as one complete workflow.

How the architecture works

The system splits the work between the website server and the browser.

1. The CMS prepares the knowledge base

Selected website content is split into passages and indexed.

The CMS also lets me manage:

which documents are available to the assistant
model profiles
search parameters
assistant instructions
activation and validation

2. The browser understands the question

The browser can use recent conversation context to reformulate follow-up questions.

An embedding model then converts the query into a vector.

3. The server performs vector search

The vector is sent to the website API.

The server searches the document index and returns the most relevant passages together with their sources.

4. WebGPU generates the answer locally

The response model runs on the visitor's GPU.

It receives:

the question
recent conversation context
retrieved passages
CMS instructions

The first result is not displayed immediately.

A second pass checks the draft against the retrieved sources, and the system can perform one corrective search if necessary.

5. The visitor gets an answer with sources

The final response contains links back to the original website content.

If the available information is not sufficient, the assistant is expected to say so instead of inventing an answer.

Why run the model in the browser?

The main experiment was to avoid calling a hosted generation API for every message.

This changes several things.

No per-response generation API call

Once the models are loaded, generation happens on the visitor's device.

That does not make the system free — hosting, downloads, development and maintenance still have costs — but there is no token bill for every generated response in this workflow.

More control over conversation data

The conversation is not sent to an external generation API to produce the answer.

Optional conversation memory is also stored locally in the browser.

The server is still involved in document retrieval, so this is not a fully offline system.

Local memory

I also added optional semantic memory using IndexedDB.

It can keep previous exchanges locally for a limited period and retrieve an older, related question when useful.

An important design decision was to keep three concepts separate:

documents are the factual source
recent conversation maintains context
local memory helps recover previous intent

An old AI-generated answer never becomes a new factual source.

The limitations are just as important

Running AI models in the browser comes with real constraints.

The current response model requires roughly 2 GB of downloads, plus about 279 MB for the embedding model.

Browser support is another factor.

Having navigator.gpu available does not automatically mean that every model and inference engine will work correctly on that device.

Drivers, GPU capabilities, memory limits and browser implementation all matter.

Latency is also very different from a fast cloud API.

With the models already cached, one documented test produced a reviewed answer in:

36.11 seconds on a Samsung Galaxy S25 Ultra
13.06 seconds on a desktop with an NVIDIA Ampere GPU

These are measurements from my own test devices, not performance guarantees.

More complex mobile requests can take over a minute.

Why I still find this architecture interesting

The goal is not to replace every search box with an AI assistant.

The interesting use cases are situations where users have a real question and the website already has reliable information that can answer it.

For example:

product catalogs
technical documentation
training websites
tourism or accommodation
business software
real-estate search

A user can express a need naturally, while structured data and RAG keep the response connected to information that can actually be verified.

The model is only one part of the solution.

The real value comes from combining:

data + retrieval + business rules + UX + performance + validation

What I learned from this first version

Building this project reinforced something I increasingly see in applied AI work:

A better model does not automatically create a better product.

The difficult parts are often around the model:

preparing reliable knowledge
keeping retrieval consistent
managing context
handling model downloads and caching
keeping the interface responsive
validating answers
testing on real devices
deciding when the system should simply say "I don't know"

This first version is now running in production on my own website, and I am continuing to test lighter models, additional browsers, GPUs and possible business applications.

I wrote a more complete article with the architecture, limitations, benchmarks and potential use cases:

👉 RAG and WebGPU: When Your Website Becomes an AI Assistant

There is also a deeper technical case study covering the complete RAG pipeline, CMS integration, embeddings, local memory and model choices.

DEV Community: Sébastien Descloux