Chrome ships a real LLM inside the browser now — Gemini Nano, exposed through a handful of built-in JS APIs (LanguageModel, Rewriter, Proofreader, Summarizer, Writer). No API key, no network call, no per-token bill. It runs on-device, which sounds like a free lunch right up until you start building against it and discover that "on-device" changes almost every assumption you've built up shipping against OpenAI or Anthropic's APIs.
I spent the last stretch building a small Chrome extension (Quill — five writing tools, on-device AI only) as a testbed for exactly this. The extension itself isn't really the point here; it's the harness I used to find out what's actually different about building on a model that lives on the user's machine instead of yours. These are the things I'd tell another dev before they start.
There isn't one API — there's a task API and a fallback API, and you need both
Chrome exposes two layers: dedicated task APIs (Rewriter, Proofreader, Summarizer, Writer), each shaped around one job with typed options, and the general-purpose LanguageModel (the "Prompt API"), which you drive with a raw system prompt. They're gated by separate chrome://flags entries and roll out to stable independently — which means on any given real machine, some subset of them is enabled and the rest aren't.
Hard-coding against Rewriter alone means the feature silently stops working on any profile where only the Prompt API is on, or vice versa. The fix is a fallback chain per capability: try the dedicated API, and if it's unavailable, reconstruct the same task as a system prompt for LanguageModel.
async function doRewrite(sharedContext, text, lengthPref, variationHint) {
const a = await avail('Rewriter');
if (a !== 'unavailable') {
const r = await Rewriter.create({ sharedContext, tone: 'as-is', length: lengthPref });
try { return await r.rewrite(text, { context: variationHint }); }
finally { r.destroy(); }
}
// Rewriter isn't enabled on this machine — do the same job via the Prompt API.
const sys = `${sharedContext} Preserve the original meaning. Output only the rewritten text.`;
return promptOnce(variationHint ? `${sys} ${variationHint}` : sys, text);
}
This roughly doubles the code per feature, and it's the difference between "works on my machine" and "works." Treat "the task API is enabled" as a runtime fact you check, not a build-time assumption.
"Unavailable" is a state machine, not a boolean
availability() doesn't return true/false — it returns unavailable, downloadable, downloading, or available. The first time I hit unavailable on a clean profile with the flags apparently on, I assumed it meant the API genuinely couldn't be reached from a content script's isolated world, and started building an offscreen-document relay to work around a permissions boundary that, it turned out, didn't exist. The actual cause: I'd toggled the flag but never restarted the browser, so Chrome was still running with the old flag state. unavailable and "you forgot to restart" produce an identical string.
Once you're actually in the downloadable or downloading states, create() takes a monitor callback that reports download progress — the first call to a newly-enabled API triggers a one-time model pull that can take a while, and if you don't wire this up, your UI just looks hung:
const opts = { sharedContext, tone: 'as-is' };
if (a !== 'available') {
opts.monitor = (m) => m.addEventListener('downloadprogress', e => {
showProgress(`Downloading model… ${Math.round(e.loaded * 100)}%`);
});
}
const r = await Rewriter.create(opts);
Two takeaways: don't build a workaround for "unavailable" until you've confirmed it isn't just an unflipped flag or a pending download, and always assume the first real call from a fresh install is going to be slow.
The model is small enough that instructions aren't guarantees
Task APIs and the Prompt API both run a genuinely small model compared to what you're used to calling over HTTP. Concretely, that means:
- Length and count instructions are approximate. Ask for "exactly one sentence" or "under 200 characters" and you'll get close, not exact, often enough that you can't display raw output where a hard limit matters — measure it in code and treat the model's attempt as a first draft, not ground truth.
- "Give me three different versions" undersells how similar they'll be. A single call asking for 3 variants tends to return 3 variants of the same idea with light word-swaps. Better results come from three separate calls, each with an explicit instruction to lean a different direction (shorter, more formal, more casual) — treat variation as something you engineer via prompts, not something you request as a parameter.
-
Output language isn't implicit. Both
LanguageModeland the task APIs acceptexpectedInputs/expectedOutputswith alanguagesarray. Skip it and you get a console warning and (per Chrome's own docs) less reliable output attestation — it's a one-line fix that's easy to miss because nothing breaks without it, it just gets quietly worse.
const opts = {
initialPrompts: [{ role: 'system', content: systemPrompt }],
expectedInputs: [{ type: 'text', languages: ['en'] }],
expectedOutputs: [{ type: 'text', languages: ['en'] }],
};
You can't build a CI pipeline that assumes the model exists
There's no Gemini Nano on a GitHub Actions runner, and probably not on your laptop today either unless you've already downloaded it. If your test suite needs a real model, you don't have a test suite — you have a manual QA checklist. The only way to get real coverage is to mock Rewriter / Summarizer / LanguageModel etc. as jsdom globals and run your actual, unmodified feature code against the mock.
That works, with one sharp edge: your mock has to vary its output the same way the real model does, or it'll hide bugs instead of catching them. I had a tone-variant generator that deduped near-identical outputs (a legit anti-repetition guard), tested against a mock that returned the same canned string regardless of which prompt it was called with. Every "generate 3 variants" test passed — with exactly one result, because the mock made all three collapse to the same dedupe key, and an assertion checking "got at least one result" didn't notice it got the wrong number. The mock's laziness happened to produce output that looked like success. Once the mock was made to vary its response per input (the same way any real generative model does), the test correctly started failing until the dedupe logic was actually right.
General rule: if a feature fans out into N distinct things, your mock has to be capable of returning N distinct things, or you're not testing the fan-out — you're testing that your code doesn't crash.
The browser boundary is real, even if your first guess about it is wrong
A content script — the code that runs on the page you're actually operating on — cannot navigate the tab to chrome:// URLs; that's Chrome policy, not a bug to route around. If part of your UX involves sending someone to chrome://flags to enable a capability, that navigation has to be requested from the extension's background service worker instead, via chrome.tabs.create or similar, triggered by a message from the content script.
// content script: can't navigate to chrome:// itself
chrome.runtime.sendMessage({ type: 'OPEN_FLAGS', flag: 'rewriter-api-for-gemini-nano' });
// background worker: can
chrome.runtime.onMessage.addListener((msg) => {
if (msg.type === 'OPEN_FLAGS') {
chrome.tabs.create({ url: `chrome://flags/#${msg.flag}` });
}
});
This is the boundary I correctly suspected existed — the mistake earlier was assuming the availability check had a similar boundary when it didn't. Worth internalizing both halves: some restrictions on what a page-level script can do are real and permanent, and the fix is to hand the action to a more privileged context. Others are red herrings that look identical from the error message alone. The only way to tell them apart is to isolate the actual cause before you build the workaround.
None of this is Quill-specific — it's what building anything on Chrome's on-device AI actually involves, independent of what the feature does.
If you want to see where these came out the other end: Quill is five writing tools (rewrite, proofread, summarize, list/table, freeform compose) running entirely on Gemini Nano, free on the Chrome Web Store.
Top comments (10)
Skipping the languages array just making the output quietly worse instead of throwing is the one that would have wrecked a week of my time. It's the nastiest kind of bug, the one where nothing is broken, the model's just a bit dumber than it should be, and there's no error to grep for. Half the post is that same shape actually. "Unavailable" that's really an unflipped flag, three variants that collapse into one, a mock that passes because it's lazy. Feels like the real theme is that on-device pushes a pile of failures out of the exception channel and into the "works but subtly wrong" channel, which is exactly where they hide longest.
That's exactly it, and I think it's a trade nobody advertises when they pitch "runs locally" as a pure win. Server-side you get centralized logging, error tracking, the ability to compare today's output against yesterday's across a fleet of requests — all the infrastructure that turns "quietly a bit worse" into a graph with a slope on it. Local means every one of those signals has to be rebuilt by the developer, by hand, on a device they mostly can't see into, for a model they don't control the version of. You're not just losing the server, you're losing every failure-detection mechanism that was riding along with it for free.
The 'mock the variability, not the output' bit is the underrated gem — hardcoded responses hide real bugs. One to add: that download state means the first user eats a few-hundred-MB pull before anything runs, so pre-warm early or the cliff bites.
Good add — that's exactly right, and it's the sharpest edge in the whole download flow.
availability() returning downloadable means the pull hasn't started yet, if the first thing that triggers create() is a user clicking "rewrite this," they're staring at a progress bar instead of a result.
Triggering that download proactively (on install, or on first extension-icon click) instead of waiting for the first real action turns a several-hundred-MB cliff into something that's already done by the time anyone needs it.
The dedupe-mock story is the sharpest thing here, but it points at a bigger trap specific to on-device: your fallback chain is also untestable in CI, and it's the part most likely to rot. You've got two code paths per feature — the task API and the reconstructed Prompt API version — and only one of them ever runs on a given machine. Whichever path your daily-driver profile happens to have flags for is the one that gets exercised; the other silently drifts until a user on the opposite flag config hits it. That's the worst kind of bug because it's config-dependent, not code-dependent, so it won't reproduce for you.
The thing I'd add to your mock harness: don't just mock the model, mock the availability matrix. Run the full test suite twice — once with
Rewriteret al. present andLanguageModelabsent, once inverted — so both branches of every fallback get real coverage. Otherwise "roughly doubles the code per feature" quietly becomes "half the code per feature is never tested," and the untested half is the sad-path one users are more likely to land on during a staggered rollout.One genuinely-open question I don't have an answer to: since the task APIs and the Prompt API can wrap the same underlying model, do the two paths actually produce comparable output for the same job, or does your
Rewriterfallback subtly change tone/quality vs. the real thing? If they diverge, your two branches aren't just two code paths — they're two products.Same input, two tones, both branches — and it wasn't noise, ran everything twice and got identical output both times. What came back wasn't "same idea, slightly different words" either. One case the Prompt API branch tacked on stuff that wasn't in the original text at all. The other case it was just clunkier, less natural phrasing than the task API version. Two different failure modes, not one consistent quality drop
So yeah, they're not really the same feature wearing two hats, they're just... different, in ways that aren't predictable ahead of time. Kind of changes what I think the fix should be — instead of trying to make the two branches agree, probably just tell the user which one they got, same way a locked Pro tone already tells you it's locked instead of pretending to be the free version. Going to add that. Also going to start mocking the availability split in CI so this kind of drift shows up before a user finds it, not after.
The mock variability point hits home. I’ve seen tests pass because the fake output was too perfect. Real AI needs messy inputs and different outputs to catch the bugs.
Yeah, "too perfect" is exactly the failure mode. A mock that's clean and consistent is a mock that agrees with whatever assumption you built the code under — it can't disagree with you, so it can't catch you being wrong. Real output is inconsistent almost by nature: different phrasing per call, occasionally garbled, sometimes ignoring part of the instruction.
A mock that doesn't reproduce at least the "different every time" part is really just testing that your code runs, not that it holds up against what the model actually does.
the 'model version you cannot pin' point buried here deserves its own section. server side you lock to a specific model version string and your prompt engineering stays stable until you choose to upgrade. on device Chrome controls when Nano gets bumped and you find out at the support ticket, not the deploy. combined with the 'instructions are approximate' observation, that's two independent sources of drift your prompt layer now has to absorb: model quality and flag state, both outside your release cycle.
the CI availability matrix idea from the thread is exactly right as a partial mitigation. what's your approach to prompt stability across Nano version bumps though — regression snapshot tests against recorded outputs, or just accepting the drift as table stakes?
Adding a manual snapshot harness for this — run by hand against real hardware before each release, since CI can't run the model itself. It'll record outputs for a fixed set of tone/action pairs and diff them against the last known-good set, so a Nano bump that changes wording or quality gets caught before it ships on top of a release, not after a support ticket. Won't catch drift the moment Chrome pushes the update — nothing running only in CI can, given the hardware requirement — but it closes the gap between "the model changed" and "I noticed," which right now is entirely support tickets.