Experiments with On-device AI — What building on Gemini Nano actually teaches you

Chrome ships a real LLM inside the browser now — Gemini Nano, exposed through a handful of built-in JS APIs (LanguageModel, Rewriter, Proofreader, Summarizer, Writer). No API key, no network call, no per-token bill. It runs on-device, which sounds like a free lunch right up until you start building against it and discover that "on-device" changes almost every assumption you've built up shipping against OpenAI or Anthropic's APIs.

I spent the last stretch building a small Chrome extension (Quill — five writing tools, on-device AI only) as a testbed for exactly this. The extension itself isn't really the point here; it's the harness I used to find out what's actually different about building on a model that lives on the user's machine instead of yours. These are the things I'd tell another dev before they start.

There isn't one API — there's a task API and a fallback API, and you need both

Chrome exposes two layers: dedicated task APIs (Rewriter, Proofreader, Summarizer, Writer), each shaped around one job with typed options, and the general-purpose LanguageModel (the "Prompt API"), which you drive with a raw system prompt. They're gated by separate chrome://flags entries and roll out to stable independently — which means on any given real machine, some subset of them is enabled and the rest aren't.

Hard-coding against Rewriter alone means the feature silently stops working on any profile where only the Prompt API is on, or vice versa. The fix is a fallback chain per capability: try the dedicated API, and if it's unavailable, reconstruct the same task as a system prompt for LanguageModel.

async function doRewrite(sharedContext, text, lengthPref, variationHint) {
  const a = await avail('Rewriter');
  if (a !== 'unavailable') {
    const r = await Rewriter.create({ sharedContext, tone: 'as-is', length: lengthPref });
    try { return await r.rewrite(text, { context: variationHint }); }
    finally { r.destroy(); }
  }
  // Rewriter isn't enabled on this machine — do the same job via the Prompt API.
  const sys = `${sharedContext} Preserve the original meaning. Output only the rewritten text.`;
  return promptOnce(variationHint ? `${sys} ${variationHint}` : sys, text);
}

This roughly doubles the code per feature, and it's the difference between "works on my machine" and "works." Treat "the task API is enabled" as a runtime fact you check, not a build-time assumption.

"Unavailable" is a state machine, not a boolean

availability() doesn't return true/false — it returns unavailable, downloadable, downloading, or available. The first time I hit unavailable on a clean profile with the flags apparently on, I assumed it meant the API genuinely couldn't be reached from a content script's isolated world, and started building an offscreen-document relay to work around a permissions boundary that, it turned out, didn't exist. The actual cause: I'd toggled the flag but never restarted the browser, so Chrome was still running with the old flag state. unavailable and "you forgot to restart" produce an identical string.

Once you're actually in the downloadable or downloading states, create() takes a monitor callback that reports download progress — the first call to a newly-enabled API triggers a one-time model pull that can take a while, and if you don't wire this up, your UI just looks hung:

const opts = { sharedContext, tone: 'as-is' };
if (a !== 'available') {
  opts.monitor = (m) => m.addEventListener('downloadprogress', e => {
    showProgress(`Downloading model… ${Math.round(e.loaded * 100)}%`);
  });
}
const r = await Rewriter.create(opts);

Two takeaways: don't build a workaround for "unavailable" until you've confirmed it isn't just an unflipped flag or a pending download, and always assume the first real call from a fresh install is going to be slow.

The model is small enough that instructions aren't guarantees

Task APIs and the Prompt API both run a genuinely small model compared to what you're used to calling over HTTP. Concretely, that means:

• Length and count instructions are approximate. Ask for "exactly one sentence" or "under 200 characters" and you'll get close, not exact, often enough that you can't display raw output where a hard limit matters — measure it in code and treat the model's attempt as a first draft, not ground truth.
• "Give me three different versions" undersells how similar they'll be. A single call asking for 3 variants tends to return 3 variants of the same idea with light word-swaps. Better results come from three separate calls, each with an explicit instruction to lean a different direction (shorter, more formal, more casual) — treat variation as something you engineer via prompts, not something you request as a parameter.
• Output language isn't implicit. Both LanguageModel and the task APIs accept expectedInputs / expectedOutputs with a languages array. Skip it and you get a console warning and (per Chrome's own docs) less reliable output attestation — it's a one-line fix that's easy to miss because nothing breaks without it, it just gets quietly worse.

const opts = {
  initialPrompts: [{ role: 'system', content: systemPrompt }],
  expectedInputs: [{ type: 'text', languages: ['en'] }],
  expectedOutputs: [{ type: 'text', languages: ['en'] }],
};

You can't build a CI pipeline that assumes the model exists

There's no Gemini Nano on a GitHub Actions runner, and probably not on your laptop today either unless you've already downloaded it. If your test suite needs a real model, you don't have a test suite — you have a manual QA checklist. The only way to get real coverage is to mock Rewriter / Summarizer / LanguageModel etc. as jsdom globals and run your actual, unmodified feature code against the mock.

That works, with one sharp edge: your mock has to vary its output the same way the real model does, or it'll hide bugs instead of catching them. I had a tone-variant generator that deduped near-identical outputs (a legit anti-repetition guard), tested against a mock that returned the same canned string regardless of which prompt it was called with. Every "generate 3 variants" test passed — with exactly one result, because the mock made all three collapse to the same dedupe key, and an assertion checking "got at least one result" didn't notice it got the wrong number. The mock's laziness happened to produce output that looked like success. Once the mock was made to vary its response per input (the same way any real generative model does), the test correctly started failing until the dedupe logic was actually right.

General rule: if a feature fans out into N distinct things, your mock has to be capable of returning N distinct things, or you're not testing the fan-out — you're testing that your code doesn't crash.

The browser boundary is real, even if your first guess about it is wrong

A content script — the code that runs on the page you're actually operating on — cannot navigate the tab to chrome:// URLs; that's Chrome policy, not a bug to route around. If part of your UX involves sending someone to chrome://flags to enable a capability, that navigation has to be requested from the extension's background service worker instead, via chrome.tabs.create or similar, triggered by a message from the content script.

// content script: can't navigate to chrome:// itself
chrome.runtime.sendMessage({ type: 'OPEN_FLAGS', flag: 'rewriter-api-for-gemini-nano' });

// background worker: can
chrome.runtime.onMessage.addListener((msg) => {
  if (msg.type === 'OPEN_FLAGS') {
    chrome.tabs.create({ url: `chrome://flags/#${msg.flag}` });
  }
});

This is the boundary I correctly suspected existed — the mistake earlier was assuming the availability check had a similar boundary when it didn't. Worth internalizing both halves: some restrictions on what a page-level script can do are real and permanent, and the fix is to hand the action to a more privileged context. Others are red herrings that look identical from the error message alone. The only way to tell them apart is to isolate the actual cause before you build the workaround.

---

None of this is Quill-specific — it's what building anything on Chrome's on-device AI actually involves, independent of what the feature does.

If you want to see where these came out the other end: Quill is five writing tools (rewrite, proofread, summarize, list/table, freeform compose) running entirely on Gemini Nano, free on the Chrome Web Store.