Building a 3-tier on-device AI concierge: Gemini Nano -> MiniLM -> keyword, $0/query

Most "AI chat widget" tutorials assume you're calling out to an LLM API. This post is about the version where you don't -- where the model runs in the visitor's own browser, the cost per conversation is exactly zero, and nothing about what someone asks ever leaves their machine.

The problem with the obvious approach

A server-side LLM call is simple to build and expensive/slow/privacy-leaky to run at scale on a marketing site that gets real traffic. For a small business's website, "every visitor's chat questions get logged on someone's server" is also just a worse default than it needs to be.

The 3-tier fallback

1. Chrome's built-in Prompt API (Gemini Nano) -- when the browser supports it, this is a real, conversational model running locally, grounded in a small knowledge base specific to the site.
2. MiniLM via Transformers.js -- when Nano isn't available, fall back to semantic search: embed the query, compare against pre-embedded KB chunks, return the best match. Still 100% on-device.
3. Keyword match -- the final fallback when neither model loads (older browser, disabled flags, whatever). Simple substring/keyword scoring over the same KB.

Every tier is visibly labeled in the UI -- the badge literally says which one answered. Degrading honestly beats pretending.

Nothing loads until you actually open the chat

The whole widget is dead weight until someone clicks the launcher. No KB fetch, no model download, no Transformers.js import -- all of it is gated behind a boot() call that only fires on first open:

launch.addEventListener('click', function () {
  if (panel.hasAttribute('hidden')) {
    panel.removeAttribute('hidden');
    boot(); // KB + tier detection only starts here
  }
});

boot() pulls a /concierge/kb-data.js script tag, then tries tier 1, then tier 2, falling through to tier 3 as a last resort:

function boot() {
  var s = document.createElement('script');
  s.src = '/concierge/kb-data.js';
  s.onload = function () {
    KB = window.LE_KB;
    decodeVecs();
    tryNano().then(function (ok) {
      if (ok) return;
      tryEmbed().then(function (ok2) {
        if (!ok2) { tier = 3; setBadge('keyword match - on-device - private'); }
      });
    });
  };
  document.head.appendChild(s);
}

Tier 1: Gemini Nano, but only if it's actually ready

Chrome's on-device model API exposes an availability() check that can return 'available', 'downloadable', or 'unavailable'. It would be easy to treat 'downloadable' as a yes and kick off the download -- but that's a multi-gigabyte model pull with no progress UI in a 384px corner widget. So the code only takes a model that's already sitting ready:

var avail = await LM.availability();
if (avail !== 'available') return resolve(false); // no silent multi-GB download
nanoSession = await LM.create({ initialPrompts: [{ role: 'system', content: SYSTEM }] });

When Nano answers, it's not just freeform generation -- it's grounded (RAG-style) in whatever the retrieval layer finds first, then asked to answer from that context:

var hits = embedFn ? cosineTop(await embedFn(q)) : liteTop(q);
var ctx = hits.map(h => 'Q: ' + h.e.q + '\nA: ' + h.e.a).join('\n\n');
var prompt = 'Knowledge:\n' + ctx + '\n\nVisitor question: ' + q + '\nAnswer as the concierge...';

Nano's raw output also gets run through a markdown/URL stripper before it ever touches the DOM -- a free-text model will happily emit bullet points, bold, and links that don't belong in a plain chat bubble, and the widget's own rule is that it never pastes a URL out loud in a conversational reply.

Tier 2: real vector search, no server, no vector DB

The knowledge base ships pre-embedded. A build step (this component credits it as "Labs' build-kb.mjs") computes MiniLM embeddings for every KB entry ahead of time and packs them as a base64-encoded Float32Array baked directly into /public/concierge/kb-data.js. At runtime, that's just:

function decodeVecs() {
  var bin = atob(KB.vecsB64), buf = new ArrayBuffer(bin.length), u8 = new Uint8Array(buf);
  for (var i = 0; i < bin.length; i++) u8[i] = bin.charCodeAt(i);
  VECS = new Float32Array(buf);
}

No vector database, no network call for retrieval -- just a flat array and a manual cosine similarity loop over it once the query itself gets embedded live in the browser:

function cosineTop(qv, k) {
  var scores = [];
  for (var i = 0; i < KB.entries.length; i++) {
    var s = 0, off = i * KB.dims;
    for (var d = 0; d < KB.dims; d++) s += qv[d] * VECS[off + d];
    scores.push([s, i]);
  }
  scores.sort((a, b) => b[0] - a[0]);
  return scores.slice(0, k).map(p => ({ score: p[0], e: KB.entries[p[1]] }));
}

Because the embeddings are normalized at encode time, that dot product is the cosine similarity -- no separate normalization step needed at query time.

The tricky part isn't the math, it's version drift: the build-time embedding model and the runtime one have to be the exact same version, or the vector space doesn't line up and every similarity score comes back meaningless. The fix is a version pin baked right into the KB data itself (KB.tfVer), so the runtime import is locked to whatever version actually produced the vectors it's searching:

var mod = await import('https://cdn.jsdelivr.net/npm/@huggingface/transformers@' + KB.tfVer);
var p = await mod.pipeline('feature-extraction', KB.model);

If the cosine match is weak (score under 0.25), the widget doesn't return a low-confidence guess -- it falls through to tier 3 instead.

Tier 3: plain keyword scoring, zero downloads

The floor is a stopword-filtered keyword scorer -- no model, no embeddings, works in any browser:

function liteTop(q, k) {
  var toks = q.toLowerCase().replace(/[^a-z0-9\s]/g, ' ').split(/\s+/).filter(t => t && !STOP[t]);
  var scores = KB.entries.map((e, i) => {
    var hay = (e.q + ' ' + e.a + ' ' + e.tags.join(' ')).toLowerCase();
    var s = 0;
    toks.forEach(t => { if (hay.indexOf(t) !== -1) s += t.length > 5 ? 2 : 1; });
    return [s, i];
  });
  return scores.sort((a, b) => b[0] - a[0]).slice(0, k).filter(p => p[0] > 0);
}

Longer, more specific tokens (>5 characters) score higher than short ones -- a crude but effective way to weight "concierge" over "the."

Failing sideways, not dead-ending

The one thing that can't happen is a visitor hitting a wall. If Nano's session creation succeeds but the actual prompt() call throws mid-conversation (it happens -- Nano is still an experimental API), the widget demotes itself to tier 2 or 3 in the middle of that same request and answers from retrieval instead of surfacing an error:

catch (e) {
  if (tier === 1) {
    tier = embedFn ? 2 : 3;
    nanoSession = null;
    // ...retry via retrieveAnswer() instead of failing the turn
  }
}

Bonus: agents get their own API

Alongside the human-facing chat, every page also registers structured tools via document.modelContext.registerTool() -- get_services, get_pricing, start_free_audit, get_contact -- so a browser AI agent visiting the page programmatically doesn't have to scrape the DOM to find that information. Different surface, same underlying idea: the site should be legible to something other than a human with a mouse.

Why this matters beyond one agency's chat widget

On-device inference is quietly becoming viable for real product surfaces, not just demos. If your product doesn't need a frontier model's full capability -- and most support/FAQ-style chat doesn't -- the on-device path is worth taking seriously before defaulting to a server call. Three tiers sounds like a lot of engineering for a corner-of-the-screen chat widget, but each one is genuinely simple in isolation; the value is entirely in never leaving a visitor stuck when the fanciest tier isn't available.

Live example: https://localenhance.com (bottom-right "Ask AI" widget).