DEV Community: Olivier Leplus

Meet Cloudagotchi : Building a virtual pet with a cloud brain (Part 1)

Olivier Leplus — Tue, 21 Jul 2026 09:38:15 +0000

Remember Tamagotchis? Those little egg-shaped keychains from the 90s with a pixelated pet that got hungry, got sad, and, if you were a negligent eight-year-old like me, got dead during math class.

I've been looking for an excuse to play with the Waveshare ESP32-S3 Touch AMOLED 1.8", a $30 board with a gorgeous 368×448 AMOLED touchscreen, an accelerometer, a microphone, and a speaker. And I found one: I'm building a Tamagotchi. But with a twist that makes it worth a four-article series

The pet's body lives on the device. Its brain lives in AWS.

The device renders an adorable pet and captures your taps and shakes. But its hunger, mood, and energy are rows in DynamoDB. It gets hungry overnight because EventBridge Scheduler says so. And, my favorite part (and because it's 2026, we can't not have genAI in a project 🤪), every morning it fetches the AWS news, has Amazon Bedrock rewrite them in its own squeaky personality, and reads them to me out loud through Amazon Polly. A Tamagotchi with a job.

In this series, I will walk you through the whole build:

This article : the architecture, and connecting the ESP32-S3 to AWS IoT Core (the pet's nervous system).
Part 2 : giving it a face: animations and touch with LVGL on the AMOLED.
Part 3 : the serverless brain: Lambda, DynamoDB, and a pet that gets hungry while you sleep.
Part 4 : Bedrock + Polly: my Tamagotchi reads me the AWS news.

⚠️ Reality check before you get too excited (sorry 😅): this is a hobby project, not a product. You'll need the specific Waveshare board (or the patience to adapt the code to yours), an AWS account, and a USB-C cable. The AWS bill for the whole series is well under a dollar a month, but it is not zero, and neither is the time you'll spend staring at idf.py monitor. Worth it, though.

All the code lives in this repository. Each article has its own branch. For this one, git checkout article-1.

Why put a pet's brain in the cloud?

Fair question. The original Tamagotchi ran on a chip with less power than my USB cable. Why does mine need a cloud?

Three reasons, and they're the same reasons real IoT products are built this way:

The pet needs to...	Which means...	AWS gives us...
Get hungry while the device is off	Time must pass server-side	EventBridge Scheduler + Lambda
Remember it was sad yesterday	State must outlive reboots	DynamoDB
Say something new every day	Logic must evolve without reflashing	Bedrock, deployed behind Lambda

That last row is going to save us a lot of time, trust me (been there flashing the device for every little change I did...). When the pet's brain is a Lambda function, giving it a new behavior is just a cdk deploy, not a firmware update. The device stays dumb and stable; the personality iterates daily.

This is, not coincidentally, the exact architecture of every serious connected device you own. Your robot vacuum is a Tamagotchi with a motor. We're just building the fun version.

The architecture

Here's the full picture we're building toward across the series:

Today we're building the arrow in the middle. No pet on screen yet. First, the nervous system, then, the face. (If you came here for cuteness, part 2 will deliver, I promise.)

The pet's vocabulary: MQTT topics

The device and cloud talk over MQTT, the lingua franca of IoT: a lightweight pub/sub protocol where everyone publishes to topics. Designing the topic space first is the IoT equivalent of designing your API routes:

Topic	Direction	Meaning
`cloudagotchi/{thing}/hello`	device → cloud	"I just woke up, what did I miss?"
`cloudagotchi/{thing}/interaction`	device → cloud	"The human fed/petted/played with me"
`cloudagotchi/{thing}/state`	cloud → device	"Here are your current stats, render them"
`cloudagotchi/{thing}/briefing`	cloud → device	"Fresh AWS news, go fetch the audio"

{thing} is the device's unique name (cloudagotchi-01). Baking it into the topic path matters for security, as you're about to see.

Step 1 — The cloud side: an IoT policy with trust issues

A device talking to AWS IoT Core authenticates with an X.509 certificate, a private key that lives on the device and never leaves it. No API keys in the firmware, no secrets in your repo.

But authentication says who you are, not what you may do. That's the IoT policy, and here's the part that trips people up: a lazy policy (iot:* on *, every tutorial's dirty secret) means any compromised device can impersonate the whole fleet. Ours pins every permission to the device's own identity:

// backend/lib/iot-stack.ts — the interesting part.
new iot.CfnPolicy(this, 'CloudagotchiDevicePolicy', {
  policyName: 'CloudagotchiDevicePolicy',
  policyDocument: {
    Version: '2012-10-17',
    Statement: [
      {
        // 1. You may connect ONLY under your own name...
        Effect: 'Allow',
        Action: 'iot:Connect',
        Resource: `arn:aws:iot:${region}:${account}:client/\${iot:Connection.Thing.ThingName}`,
      },
      {
        // 2. ...publish ONLY on your own topics...
        Effect: 'Allow',
        Action: 'iot:Publish',
        Resource: `arn:aws:iot:${region}:${account}:topic/cloudagotchi/\${iot:Connection.Thing.ThingName}/*`,
      },
      {
        // 3. ...and listen ONLY to your own topics.
        Effect: 'Allow',
        Action: ['iot:Subscribe'],
        Resource: `arn:aws:iot:${region}:${account}:topicfilter/cloudagotchi/\${iot:Connection.Thing.ThingName}/*`,
      },
      // (+ iot:Receive, same resource shape — see the repo)
    ],
  },
});

The magic is ${iot:Connection.Thing.ThingName}, an IoT policy variable that resolves at connection time to the name of the thing attached to the certificate. One policy serves the whole fleet, yet cloudagotchi-01 physically cannot publish on cloudagotchi-02's topics. If your pet ever gets kidnapped at a conference (mine will, eventually), the blast radius is one sad pet.

Deploy it:

cd backend && npm install && npx cdk deploy CloudagotchiIotStack

Step 2 — A birth certificate for the pet

Certificates are secrets, so they don't belong in CloudFormation state, we create them per-device with a small script (scripts/provision-device.sh in the repo). The gist:

# Create the thing (the device's identity in AWS).
aws iot create-thing --thing-name cloudagotchi-01

# Create a certificate + private key, saved straight into the firmware tree.
CERT_ARN=$(aws iot create-keys-and-certificate --set-as-active \
  --certificate-pem-outfile firmware/main/certs/device-cert.pem \
  --private-key-outfile   firmware/main/certs/device-key.pem \
  --query certificateArn --output text)

# Tie it all together: this cert may act as this thing, under this policy.
aws iot attach-policy --policy-name CloudagotchiDevicePolicy --target "$CERT_ARN"
aws iot attach-thing-principal --thing-name cloudagotchi-01 --principal "$CERT_ARN"

One command in the repo version:

./scripts/provision-device.sh cloudagotchi-01

💡 The .pem files are gitignored, keep it that way. The private key is the pet's identity. Anyone who has it is your pet, as far as AWS is concerned. The repo ignores certs/*.pem except Amazon's public root CA, and you should never weaken that line.

Step 3 — The firmware: Wi-Fi, then mutual TLS

The firmware is plain ESP-IDF (Espressif's native framework) plus the official Waveshare board support package. Why not Arduino? ESP-IDF's built-in esp-mqtt client does mutual TLS natively, and the BSP hands us the display, touch, and audio drivers we'll need in parts 2 and 4, less glue code overall, which matters when every line has to fit in a blog post.

The build embeds the three credential files into the binary, no filesystem juggling:

# firmware/main/CMakeLists.txt
idf_component_register(
    SRCS "main.c" "app_wifi.c" "app_mqtt.c"
    INCLUDE_DIRS "."
    EMBED_TXTFILES
        "certs/AmazonRootCA1.pem"
        "certs/device-cert.pem"
        "certs/device-key.pem"
)

And the MQTT connection is disarmingly small. This is the entire handshake with AWS:

// firmware/main/app_mqtt.c — the heart of it.
const esp_mqtt_client_config_t cfg = {
    .broker = {
        // 1. AWS IoT speaks MQTT over TLS on port 8883.
        .address.uri = "mqtts://" CONFIG_CLOUDAGOTCHI_IOT_ENDPOINT ":8883",
        // 2. We verify we're talking to the real AWS...
        .verification.certificate = root_ca_pem,
    },
    .credentials = {
        // 3. ...and AWS verifies it's talking to the real us.
        .authentication = {
            .certificate = device_cert_pem,
            .key = device_key_pem,
        },
        // 4. Our policy requires client id == thing name. Not optional!
        .client_id = CONFIG_CLOUDAGOTCHI_THING_NAME,
    },
};

s_client = esp_mqtt_client_init(&cfg);
esp_mqtt_client_start(s_client);

That's mutual TLS: both sides prove who they are before a single byte of MQTT flows. Step 4 is the one people forget, if the client id doesn't match the thing name, our policy's iot:Connect statement rejects the connection, and you get the single most cryptic failure mode in IoT: an endless connect/disconnect loop with no error message. You've been warned. 😅

On connect, the pet introduces itself and subscribes to its own topics:

case MQTT_EVENT_CONNECTED:
    // Listen to everything addressed to *this* pet...
    esp_mqtt_client_subscribe(s_client, "cloudagotchi/" THING "/#", 1);
    // ...and announce we're awake. In part 3, a Lambda answers
    // this with the pet's current state. Today, only silence. 🥲
    app_mqtt_publish("hello", "{\"firmware\":\"article-1\"}");
    break;

The main loop just heartbeats every 30 seconds so we have a pulse to watch.

Step 4 — Proof of life

Configure and flash:

cd firmware
idf.py set-target esp32s3
idf.py menuconfig     # → "Cloudagotchi Configuration": Wi-Fi + IoT endpoint + thing name
idf.py flash monitor

(Your IoT endpoint: aws iot describe-endpoint --endpoint-type iot:Data-ATS.)

⚠️ Region gotcha that will produce the most confusing failure of your build. The device's endpoint and your deployed stacks must be in the same region, the endpoint hostname looks almost identical across regions (xxxx-ats.iot.us-east-1... vs xxxx-ats.iot.us-west-2...), and if they don't match, TLS connects fine and then AWS silently drops you: an endless connect/EOF loop with no error message anywhere. Always take the endpoint from the same CLI region you deployed to (check with aws configure get region), and pass that same value everywhere it's needed.

If the gods of embedded development smile upon you:

I (2841) app_wifi: Wi-Fi connected
I (4127) app_mqtt: Connected to AWS IoT Core

Now the fun part. Open the AWS IoT console → MQTT test client, subscribe to cloudagotchi/#, and watch your pet's heartbeat arrive from your desk:

cloudagotchi/cloudagotchi-01/heartbeat  →  {"alive": true}

And talk back! Publish {"msg":"hello little one"} to cloudagotchi/cloudagotchi-01/greeting and watch it land in your serial monitor:

I (91204) cloudagotchi: Cloud says [cloudagotchi/cloudagotchi-01/greeting]: {"msg":"hello little one"}

A message typed in a browser, through AWS, to a $30 board on your desk, in about 100 ms. It never stops being cool.

Why this matters

Everything we built today (certificate identity, least-privilege topic isolation, a scoped policy shared by a fleet) is the same foundation you'd lay for a real product with a million devices. The difference between a toy IoT tutorial and production IoT isn't the hardware; it's whether ${iot:Connection.Thing.ThingName} shows up in the policy. Now it's in ours, and we never have to think about it again.

Our pet has a nervous system. It can feel (receive) and speak (publish). What it doesn't have is a face, it's currently the philosophical-zombie phase of the project.

Next article: we fix that. LVGL on the AMOLED, a pet with blinking eyes and moods, touch interactions, and the accelerometer, because a pet you can't tickle by shaking the board is just a dashboard.

Try it yourself

The Cloudagotchi repo : git checkout article-1
Waveshare ESP32-S3 Touch AMOLED 1.8 wiki : board docs and schematics
AWS IoT Core policy variables : the ThingName trick, officially
ESP-MQTT docs : the client we used

💡 Disclosure: I'm a Developer Advocate at AWS, that's the cloud my Tamagotchi thinks with. The device-side patterns (mutual TLS, scoped topics) apply to any MQTT broker.

I built a Tamagotchi with its brain in AWS (and it reads me the news)

Olivier Leplus — Mon, 20 Jul 2026 11:30:09 +0000

Remember Tamagotchis? Those little egg-shaped keychains with a pixelated pet that got hungry, got sad, and, if you were a negligent eight-year-old like me, got dead during math class.

I built one. Except mine has a 368×448 AMOLED screen, an accelerometer, a speaker... and its brain runs on AWS.

Meet Cloudagotchi:

What it does

It gets hungry while the device is off : time passes in the cloud, not on the chip. EventBridge Scheduler decides when it's dinner time.
It remembers you ignored it yesterday : its feelings are a DynamoDB row. You can't reset a guilt trip by rebooting.
You feed it by tapping an avocado, pet it by tapping its face, play with it by shaking the board. Every interaction is an MQTT round trip through AWS IoT Core.
It dozes off if you leave it alone, complete with a little zzz.
And my favorite part: every morning at 7:03, it reads me the AWS news out loud : Amazon Bedrock rewrites the headlines in the pet's cute voice, Amazon Polly gives it speech, and the little face moves its mouth while it talks. A Tamagotchi with a job.

How it works (the 30-second version)

The device is the body; AWS is the brain. The ESP32 renders sprites and captures your taps. But every feeling, every stat, every decision comes from the cloud:

The stack, at a glance: ESP-IDF + LVGL on a Waveshare ESP32-S3 Touch AMOLED 1.8", talking mutual-TLS MQTT to AWS IoT Core, with Lambda, DynamoDB and EventBridge Scheduler as the serverless brain, and a Bedrock → Polly → S3 pipeline for the morning briefing. All infrastructure defined with CDK. Total running cost: well under a dollar a month.

💡 Not coincidentally, this is the exact architecture of every serious connected device you own. Your robot vacuum is a Tamagotchi with a motor. We're just building the fun version.

The series

I'm turning the whole build into four articles, each with its own git branch so you can follow along commit by commit:

This is article 0 ^^: It's a teaser so you come back to see the rest of the project 🤫
Meet Cloudagotchi: A virtual pet with a cloud brain : the architecture, and connecting the board to AWS IoT Core with mutual TLS (plus the IoT policy trick that makes a kidnapped pet a non-event).
[Coming July 23rd] Giving it a face : sprite-based LVGL UI, blinking, dozing, shake detection... and the very honest story of how my first "elegant" procedural face became a corrupted mess on real hardware.
[Coming July 27st] It gets hungry while You sleep : the serverless state machine: Lambda, DynamoDB, and the lazy-time-accounting trick that makes a pet age without a single always-on process.
[Coming July 29st] My tamagotchi reads me the AWS News : Bedrock as a squeaky-voiced news anchor, Polly's one-parameter trick that deleted an entire firmware subsystem, and the four embedded-streaming gotchas nobody warns you about.

Fair warning: this series contains real war stories. Panels that wedge themselves, buffers defeated by presigned URLs, a TCP window doing trans-Atlantic math against me. The failures were the best teachers, so they're all in there.

Part 1 drops on Tuesday, July 20th. Follow me to catch it.

The code is open source: github.com/tagazok/cloudagotchi. And, if you build one, please show me your pet. 👻

AI Without a Backend: The Browser's Built-in AI APIs for Web Developers

Olivier Leplus — Wed, 08 Jul 2026 14:12:29 +0000

For years, "adding AI to your web app" meant the same thing: sign up for an API key, wire up a backend proxy so you don't leak that key to the browser, meter your token usage, handle rate limits, and pray your bill doesn't explode when a feature goes viral. Every summary, every classification, every bit of generated text was a round-trip to someone else's data center.

Chrome is quietly changing that. A family of built-in AI APIs now ships a language model — Google's Gemini Nano — inside the browser. Your JavaScript can summarize an article, translate a comment, or generate structured data from an image without a single network call to an inference server. No API key. No backend. No per-token bill. And because the model runs on-device, the user's data never leaves their machine.

In this article, I will walk you through what these APIs are, how to use them safely, and two hands-on examples: summarizing user reviews, and turning a product photo into structured JSON with the Prompt API.

⚠️ Reality check before you get too excited (sorry 😅): these APIs are desktop-only for now (no mobile), require a fairly capable machine, and the model is a ~4 GB download on first use. They're production-viable as progressive enhancements, not as your only code path. We'll cover graceful degradation throughout.

The lineup: task APIs vs. the Prompt API

Chrome's built-in AI comes in two flavors.

Task-specific APIs do one job well, with a clean, purpose-built interface:

API	What it does
Translator	Translates text between languages on-device.
Language Detector	Detects the language of a piece of text.
Summarizer	Condenses long text into key points, a TL;DR, a teaser, or a headline.
Writer	Generates new text from a prompt (e.g. a product description).
Rewriter	Restructures or improves existing text (shorter, more formal, etc.).

The Prompt API (also called the LanguageModel API) is the general-purpose escape hatch. When no task API fits, you talk to Gemini Nano directly with your own prompts — and it's multimodal, accepting text, images, and audio as input.

The mental model: reach for a task API when your need matches one (translation, summarization, rewriting) — it's simpler and more predictable. Reach for the Prompt API when you need something custom, structured, or multimodal.

All of these share the same lifecycle, so once you learn one, you know them all.

The lifecycle every built-in AI API shares

Every API follows the same three-step pattern: feature-detect → check availability → create and use.

1. Feature detection

The interfaces live as globals (Summarizer, LanguageModel, Translator, …). Check before you touch:

if ('Summarizer' in self) {
  // The Summarizer API exists in this browser.
}

2. Availability and the model download

Here's the part that trips people up. The API being present doesn't mean the model is ready. Gemini Nano is downloaded separately the first time an origin uses it — and it's a multi-gigabyte download.

availability() tells you where you stand:

const status = await Summarizer.availability();
// "unavailable"  → not supported on this device/config
// "downloadable" → supported, but the model must download first
// "downloading"  → download in progress
// "available"    → ready to go right now

3. Create the instance (and show download progress)

create() instantiates the API. If the model still needs downloading, create() triggers it — so always give the user feedback, because "downloading 4 GB" is not something to do silently:

const summarizer = await Summarizer.create({
  monitor(m) {
    m.addEventListener('downloadprogress', (e) => {
      console.log(`Model download: ${Math.round(e.loaded * 100)}%`);
    });
  },
});

Two things worth knowing:

The download happens once per device, not once per site. After Gemini Nano is on disk, every origin that uses these APIs shares it. Chrome manages it like a cache — if disk space runs low it may evict the model and re-download later.
create() should be triggered by a user gesture. Kicking off a huge download on page load is hostile. Gate it behind a click, and check navigator.userActivation.isActive if you want to be strict.

Use case #1 — Summarizing user reviews

Picture a product page with dozens of customer reviews. Nobody reads all of them. A classic use case: a "Summarize reviews" button that distills them into a few key points, computed entirely in the browser.

The summarizer's knobs

Summarizer.create() takes options that shape the output:

type — "key-points" (default), "tldr", "teaser", or "headline".
format — "markdown" (default) or "plain-text".
length — "short" (default), "medium", or "long". What that means depends on the type: a short key-points summary is 3 bullets; a short tldr is one sentence.
sharedContext — background info that helps the model do a better job.

Putting it together

async function summarizeReviews(reviewsText) {
  // 1. Feature detection — degrade gracefully if unsupported.
  if (!('Summarizer' in self)) {
    return { supported: false };
  }

  // 2. Availability check.
  const availability = await Summarizer.availability();
  if (availability === 'unavailable') {
    return { supported: false };
  }

  // 3. Create the summarizer, tuned for product reviews.
  const summarizer = await Summarizer.create({
    sharedContext:
      'These are customer product reviews from an e-commerce store. ' +
      'Summarize the overall sentiment and the points customers mention most.',
    type: 'key-points',
    format: 'markdown',
    length: 'medium',
    monitor(m) {
      m.addEventListener('downloadprogress', (e) => {
        updateProgressUI(e.loaded); // your UI hook
      });
    },
  });

  // 4. Summarize.
  const summary = await summarizer.summarize(reviewsText, {
    context: 'Focus on product quality, value for money, and shipping.',
  });

  return { supported: true, summary };
}

Wiring it to a button:

summarizeBtn.addEventListener('click', async () => {
  // innerText, not innerHTML — feed the model clean text, not markup.
  const reviewsText = document.querySelector('#reviews').innerText;

  summarizeBtn.disabled = true;
  const { supported, summary } = await summarizeReviews(reviewsText);

  if (!supported) {
    // Progressive enhancement: fall back to your server, or just hide the feature.
    summaryEl.textContent = 'Review summaries aren’t available on this device.';
  } else {
    summaryEl.textContent = summary;
  }
  summarizeBtn.disabled = false;
});

💡 Feed it clean text. Use innerText, not innerHTML. The model summarizes better when it isn't wading through <div> soup, and you don't waste the context window on markup. If you retrieve the feedback from an API, you can also give it the json directly.

Streaming for long input

For a big block of text, waiting for the whole summary feels sluggish. Stream it instead and render tokens as they arrive:

const stream = summarizer.summarizeStreaming(reviewsText, {
  context: 'Focus on product quality and value for money.',
});

summaryEl.textContent = '';
for await (const chunk of stream) {
  summaryEl.append(chunk); // each chunk is an incremental piece of text
}

A note on languages

The Summarizer currently supports en, ja, es, de, and fr. Declaring the languages up front lets the browser reject an unsupported combination early instead of failing mid-request:

const summarizer = await Summarizer.create({
  type: 'key-points',
  expectedInputLanguages: ['en', 'fr'],
  outputLanguage: 'en',
});

Use case #2 — From a product photo to structured JSON with the Prompt API

The task APIs are great when your need fits one of them. But what about something bespoke — like: "Here's a photo of a product. Generate a draft review and auto-fill the review form."

That's a job for the Prompt API, and it shows off two of its best features: multimodal input (an image) and structured output (a JSON schema).

Step 1 — Create a multimodal session

Because we're sending an image, we declare image (and text) as expected inputs when creating the session:

const session = await LanguageModel.create({
  expectedInputs: [
    { type: 'text', languages: ['en'] },
    { type: 'image' },
  ],
  expectedOutputs: [{ type: 'text', languages: ['en'] }],
  monitor(m) {
    m.addEventListener('downloadprogress', (e) => {
      updateProgressUI(e.loaded);
    });
  },
});

Step 2 — Define the shape you want back

Prompting "reply in JSON" and hoping for the best is a recipe for parse errors. Instead, pass a JSON Schema via responseConstraint, and the model is constrained to produce output matching it:

const reviewSchema = {
  title:       { type: 'string' },
  description: { type: 'string' },
  rating:      { type: 'number', minimum: 1, maximum: 5 },
  pros:        { type: 'array', items: { type: 'string' } },
  cons:        { type: 'array', items: { type: 'string' } },
};

Step 3 — Prompt with text + image, constrained to the schema

The content of a message is an array of typed parts. We combine a text instruction with the image, and pass the schema as a constraint:

async function generateReviewFromPhoto(imageElement) {
  const result = await session.prompt(
    [
      {
        role: 'user',
        content: [
          {
            type: 'text',
            value:
              'You are given a photo of a product. Write a short, honest ' +
              'customer-style review of it. Return the result as JSON.',
          },
          { type: 'image', value: imageElement },
        ],
      },
    ],
    { responseConstraint: reviewSchema },
  );

  // Guaranteed to match the schema — safe to parse.
  return JSON.parse(result);
}

Calling it against an image on the page:

const photo = document.querySelector('#product-photo'); // an <img> element
const review = await generateReviewFromPhoto(photo);

console.log(review);
// {
//   title: "Sleek and sturdy everyday mug",
//   description: "A clean matte-white ceramic mug that looks premium...",
//   rating: 4,
//   pros: ["Solid build", "Comfortable handle"],
//   cons: ["Slightly heavier than expected"]
// }

Because the output is schema-constrained, you skip the usual defensive gymnastics — no regex-scraping a JSON blob out of a chatty response, no try/catch around a hopeful JSON.parse. You get an object shaped exactly like your database row.

💡 Why the schema matters. Without responseConstraint, the model might honor a "reply in JSON" instruction — most of the time. With it, the output is structurally guaranteed. In production, "most of the time" is where the 2 a.m. pages come from. Use the schema!

Reusing the session

A LanguageModel session keeps context across prompts, so you can hold a conversation or run several images through the same instance. When you're done, free the resources:

session.destroy();

Building responsibly: this is progressive enhancement

The single most important design principle for built-in AI: treat it as an enhancement, never a hard dependency. A big chunk of your users won't have it — wrong OS, mobile device, insufficient hardware, or the model simply hasn't downloaded yet.

A robust feature does this:

async function getSummarizer() {
  if (!('Summarizer' in self)) return null;
  const availability = await Summarizer.availability();
  if (availability === 'unavailable') return null;
  return Summarizer.create(/* … */);
}

const summarizer = await getSummarizer();
if (summarizer) {
  // On-device path: instant, private, free.
} else {
  // Fallback: call your server-side summarizer, or hide the feature entirely.
}

Keep these in mind:

Desktop only, for now. The foundation-model APIs (Prompt, Summarizer, Writer, Rewriter) don't run on Android or iOS yet. Translator and Language Detector are desktop-only too.
Real hardware requirements. Roughly: >4 GB of GPU VRAM or a decent CPU with 16 GB RAM, plus ~22 GB free disk for Chrome's profile volume. Not every laptop qualifies.
The first run is a download. Multiple gigabytes. Always show progress, and never block first paint on it.
Privacy is a genuine selling point. Inference is local; user data never hits a server. For a reviews summarizer or a content classifier handling user text, that's a real advantage — say so.

When the browser can't: a cloud fallback

Progressive enhancement is only half a strategy if the "else" branch just hides the feature. For a lot of products you want the capability to work everywhere — on the mobile users, the older laptops, the browsers that haven't shipped these APIs yet — while still giving the on-device path to everyone who can use it.

Think of it as a spectrum rather than a rivalry:

On-device (built-in AI): free, private, offline-capable, zero infra. Ideal for lightweight, high-frequency, privacy-sensitive tasks — summarizing reviews, classifying text, drafting a first pass. The trade-off is a small model with a modest context window, and it's not on every device.
Cloud (a hosted model): handles heavy reasoning, large context windows, and a specific frontier model when you need it — at the cost of latency, per-token pricing, and a backend to secure.

The nice part: your feature-detection branch is already the seam between them. Where the on-device path falls through, call your cloud model instead. Here's the same getSummarizer() pattern with a fallback to Amazon Bedrock, which exposes many foundation models behind one API:

// Client side — same seam as before, now with a real fallback.
const summarizer = await getSummarizer();

let summary;
if (summarizer) {
  // On-device path: instant, private, free.
  summary = await summarizer.summarize(reviewsText);
} else {
  // Cloud path: works on any device, at the cost of a network round-trip.
  const res = await fetch('/api/summarize', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ text: reviewsText }),
  });
  ({ summary } = await res.json());
}

// Server side (Node/Lambda) — a thin proxy to Amazon Bedrock.
import { BedrockRuntimeClient, ConverseCommand } from '@aws-sdk/client-bedrock-runtime';

const client = new BedrockRuntimeClient({ region: 'us-east-1' });

export async function handler(req, res) {
  const { text } = req.body;

  const response = await client.send(new ConverseCommand({
    modelId: 'anthropic.claude-3-5-haiku-20241022-v1:0',
    messages: [{
      role: 'user',
      content: [{ text: `Summarize these product reviews as key points:\n\n${text}` }],
    }],
  }));

  const summary = response.output.message.content[0].text;
  res.json({ summary });
}

Same feature, two engines: the browser serves everyone it can for free, and the cloud quietly covers the rest. Keeping the API key server-side (never in the browser) is exactly the backend concern the on-device path lets you skip for the majority of your users — so you pay for cloud inference only on the requests that genuinely need it.

💡 Disclosure: I'm a Developer Advocate at AWS, so Bedrock is the cloud fallback I reach for. The pattern is vendor-neutral — any hosted model API slots into the same else branch.

Why this matters

Built-in AI shifts a whole class of features from "backend project with ongoing costs" to "a few dozen lines of client-side JavaScript." No key to rotate, no proxy to secure, no bill that scales with usage, and no user data leaving the device.

It won't replace big cloud models for heavy reasoning, and it isn't everywhere yet. But for the everyday tasks that make up so much of real web work — summarize this, classify that, translate the other thing, extract structured data from a mess — the model is right there in the browser, waiting for a function call.

Try it yourself

Chrome Built-in AI docs — start here
Summarizer API doc
Prompt API guide
Structured output with the Prompt API
Tip: install the @types/dom-chromium-ai npm package for TypeScript typings.