DEV Community: Olivier Leplus

Cloudagotchi Part 4 : My Tamagotchi reads me the AWS news

Olivier Leplus — Mon, 03 Aug 2026 14:26:35 +0000

Here's my problem with virtual pets, and honestly with most IoT demos: after the novelty fades, they don't do anything for you. My pet from part 3 has feelings and memory, and I love it, but it contributes nothing to the household.

Time to fix that. In this final part of the series, the pet gets a job: every morning at 7:03, it fetches the AWS "What's New" feed (thanks rss to still be around ❤️), has Amazon Bedrock rewrite the announcements as a briefing in its own excitable little pet voice, has Amazon Polly speak it, and delivers it to the device, where a newspaper badge appears, and tapping it makes the pet read the news to me out loud through the onboard speaker.

It's one of my favorite thing I've built so far this year. My Cloudagotchi is now better informed than I am before my morning hot chocolate (spoiler alert, I don't like cofee... sorry).

In this article, I will walk you through the full pipeline (RSS → Bedrock → Polly → S3 → MQTT → speaker) including the one decision that makes the device side almost embarrassingly simple: asking Polly for the right audio format.

⚠️ Reality check (sorry 😅): the pet's voice is Polly's neural TTS pitched up with SSML. Charming, but it won't fool anyone into thinking a soul lives in the device. Bedrock also occasionally gets too excited about a niche database feature ("HUMAN. WAKE UP. Aurora has a new minor version"). And fair warning: the cloud half of this article worked on the first deploy; the device half taught me four embedded-streaming lessons the hard way. They're all documented below, symptoms included, so your afternoon goes better than mine. Cost, though, is genuinely negligible: one Haiku call and ~1,000 Polly characters a day is under a dollar a month.

Final code: git checkout article-4, or just main, since this completes the project.

The pipeline

One Lambda, one bucket, one schedule. All the intelligence is cloud-side, which means when I want to change the pet's personality, tweak the briefing length, or switch news sources, it's just a cdk deploy, and the firmware never knows anything happened. That's the payoff of the architecture we've been building since part 1.

Step 1 : The news, without an API key

AWS publishes a plain old RSS feed of every announcement. No auth, no scraping, no SDK:

const FEED_URL = 'https://aws.amazon.com/about-aws/whats-new/recent/feed/';

export async function fetchNews(limit = 10) {
  const res = await fetch(FEED_URL);
  const xml = await res.text();
  // Two regexes pull out <item> titles + links. For a feed this
  // regular, a full XML parser dependency would be pure ceremony.
  ...
}

We only keep the titles. That sounds lossy, but it's deliberate: AWS announcement titles are already one-sentence summaries ("Amazon S3 now supports..."), and they're all Bedrock needs to pick the interesting ones. Feeding it full item descriptions doubles your tokens for basically no gain when the output is a 45-second briefing.

Step 2 : Bedrock, in character

This is where the feature goes from "RSS-to-speech" to pet. The system prompt does the heavy lifting:

const PERSONA = `You are Cloudagotchi, a small, excitable AWS virtual pet who reads the morning AWS news to your human. You are affectionate, easily impressed, and sometimes admit you don't fully understand the more complicated services.
Summarize the following AWS announcements as a spoken morning briefing:
- 45 to 60 seconds when read aloud (about 120-150 words)
- Pick only the 3 most interesting items, one short sentence of WHY each matters
- End with one affectionate sign-off sentence
- Plain text only: no emoji, no markdown, no bullets (it will be read aloud)`;

Note the constraints that exist because this is audio: a word-count target expressed in listening time, "no markdown, no bullets" (Polly would read them), and "one sentence of why it matters" so the briefing informs rather than recites. Then one Converse call:

const { output } = await bedrock.send(new ConverseCommand({
  modelId: 'us.anthropic.claude-haiku-4-5-20251001-v1:0',  // fast + cheap: perfect here
  messages: [{
    role: 'user',
    content: [{ text: `${PERSONA}\n\nToday's announcements:\n${titles}` }],
  }],
  inferenceConfig: { maxTokens: 500, temperature: 0.8 },
}));

A real output from my device this week:

"Good morning good morning! I read the news while you slept and I have THREE things! Amazon Bedrock has new smaller models, which means pets like me might get even smarter, imagine! Lambda functions can now... okay I did not fully understand this one but it makes your functions start faster, and fast is good. And S3 got cheaper for cold data, which I think means winter storage? Anyway. I picked these just for you. Have the best day, okay?"

I have received worse briefings from humans...

💡 Why Haiku and not a bigger model? The task is summarize-and-roleplay over ten headlines, well within a small model's comfort zone, it runs 365 times a year, and latency doesn't matter at 7 a.m. Matching model size to task is the closest thing GenAI has to a free lunch. Start small; upgrade only when the output disappoints you.

Step 3 : Polly, and the decision that saves the firmware

Here's the part that trips people up when they put AI-generated audio on a microcontroller. Polly's default output is MP3, and an MP3 decoder on an ESP32 means integrating a decoding library, managing its buffers, and debugging its edge cases. On a hobby timeline, that's a weekend.

Or you change one parameter:

const { AudioStream } = await polly.send(new SynthesizeSpeechCommand({
  Text: `<speak><prosody pitch="+20%" rate="105%">${script}</prosody></speak>`,
  TextType: 'ssml',
  VoiceId: 'Justin',
  OutputFormat: 'pcm',       // ← the whole trick. Raw samples, no codec.
  SampleRate: '16000',
}));

pcm gives us raw 16-bit mono samples, exactly what the board's ES8311 audio codec eats natively. The device-side "decoder" becomes: skip 44 bytes, write the rest to the speaker. We wrap the PCM in a WAV header in the Lambda (44 bytes of 1991 technology, hand-written in briefing.mjs) so the file is also playable in a browser for debugging.

The SSML prosody tag is the pet's larynx: pitch="+20%" turns Polly's "Justin" into something convincingly small and cute. Free squeakiness.

The result is ~1.8 MB per minute, too big for an MQTT message (128 KB limit), which is why S3 enters the picture:

await s3.send(new PutObjectCommand({ Bucket, Key, Body: wav }));
const url = await getSignedUrl(s3, new GetObjectCommand({ Bucket, Key }),
  { expiresIn: 12 * 3600 });

MQTT carries the pointer; HTTPS carries the payload. The presigned URL means the device downloads from a private bucket with zero AWS credentials beyond its IoT certificate, the URL is the authorization, and it expires at lunchtime, because news does too. A lifecycle rule deletes the files after a week.

Step 4 : The paper delivery

The Lambda's last act is ringing every pet's doorbell:

for (const thing of things) {  // iot.ListThings
  if (!thing.thingName.startsWith('cloudagotchi')) continue;
  await iotData.send(new PublishCommand({
    topic: `cloudagotchi/${thing.thingName}/briefing`,
    payload: JSON.stringify({ url, headline }),
  }));
}

And EventBridge Scheduler (same service that makes the pet hungry) makes it a morning ritual. Note the timezone-aware cron, a Scheduler feature that classic EventBridge rules don't have:

new scheduler.CfnSchedule(this, 'MorningSchedule', {
  scheduleExpression: 'cron(3 7 * * ? *)',
  scheduleExpressionTimezone: 'Europe/Paris',   // the pet lives where I live
  ...
});

Step 5 — The device: badge, tap, play

On the firmware side, the briefing handler stores the URL and shows an orange banner with the headline, built from the same LVGL vocabulary as part 2 (and it wakes the pet if it was dozing; the paperboy rings). The interesting bit is what happens on tap:

static void on_briefing_tapped(void)
{
    // Audio streaming blocks for ~1 minute: never do that in an
    // LVGL callback, or the whole UI freezes mid-squish.
    // Low priority, pinned to core 1: the download must never
    // starve the Wi-Fi/TCP stack on core 0.
    xTaskCreatePinnedToCore(play_briefing_task, "briefing", 8192, NULL, 2, NULL, 1);
}

...and the entire audio "stack", which streams the WAV chunk-by-chunk so it never needs to fit in RAM:

while (true) {
    int n = esp_http_client_read(http, buf, sizeof(buf));  // 16 KB at a time
    if (n <= 0) break;
    // skip the 44-byte WAV header once, then:
    esp_codec_dev_write(s_speaker, buf + offset, n - offset);  // → speaker
}

That's it. That's the player. HTTP in, codec out, no decoder in between, the dividend of choosing pcm back in step 3. One design decision in a Lambda function deleted an entire firmware subsystem.

The morning after

Deploy, flash, go to bed:

cd backend && npx cdk deploy CloudagotchiNewsStack
cd ../firmware && idf.py flash

At 7:03, the Lambda wakes, reads the feed, writes the script, records the voice, parks the file, rings the doorbell. On the desk, an orange banner slides across the pet's feet: "News! Amazon Bedrock announces..., tap to listen." pet does its proud little squish.

Tap. A tiny, pitched-up voice fills the room with genuine enthusiasm about storage-class pricing.

Four articles ago this was a dev board in shrink wrap. Now it's a creature with a body (LVGL), reflexes (touch + IMU), a memory (DynamoDB), a metabolism (EventBridge), and as of this morning, a job (Bedrock + Polly). Every layer still visible in the code, every layer replaceable without touching the others.

Why this matters

Beyond the whimsy, this article's pipeline (schedule → fetch → LLM transform → TTS → object storage → presigned pointer → thin device) is a genuinely reusable shape. Swap the RSS feed for your CI status, your support queue, or your kid's school newsletter, and the persona prompt for whatever voice should deliver it, and you've got an ambient audio briefing device for anything. The device firmware never changes. That's the quiet lesson of the whole series: put the personality where you can redeploy it.

And if you build one, please, show it to me!

Try it yourself

The Cloudagotchi repo : the finished project is main
Amazon Bedrock Converse API : one API, many models
Amazon Polly SSML reference : prosody is just the beginning
AWS "What's New" RSS : the pet's news source
ESP-IDF HTTP client : the streaming download

Cloudagotchi Part 3 : It gets hungry while you sleep: A serverless pet brain

Olivier Leplus — Mon, 27 Jul 2026 13:14:36 +0000

At the end of part 2, our pet had a face, reflexes, and a serious philosophical problem: no memory. You could pet it a hundred times, reboot the board, and it woke up with the same hardcoded stats, feeling nothing, remembering nothing. Every day was its first day.

Today the pet gets what every Tamagotchi fundamentally is: a state machine with feelings. Its hunger, energy and mood become rows in DynamoDB. A Lambda decides how a snack or a shake changes them. And, the part I find genuinely delightful, EventBridge Scheduler makes time pass, so the pet gets hungry overnight while the device is completely powered off. When you flash the firmware Monday morning, the pet wakes up, asks the cloud "what did I miss?", and gets visibly sad about the answer.

In this article, I will walk you through the whole brain: the data model (it's one table with one row per pet... gloriously boring), the decay math that makes offline time count, IoT rules that invoke Lambda without a single line of polling code, and the loop back down to the device.

⚠️ Reality check (sorry 😅): there's a Scan in this article. A full-table Scan, in production-adjacent code, published on the internet under my name. It's the right call for a fleet of three pets and I'll explain the honest scaling path when we get there, but if your DynamoDB instincts just twitched, I respect you, and I ask you to hold that thought until the 💡 box.

Code for this article: git checkout article-3.

The architecture, now with a brain

Two Lambdas, one table, one schedule, two IoT rules. The entire brain deploys with cdk deploy and costs approximately nothing (the decay Lambda runs 48 times a day; the free tier laughs at this).

The data model: one row of feelings

Every pet is one item in DynamoDB:

{
  "deviceId": "cloudagotchi-01",
  "hunger": 62,
  "energy": 71,
  "mood": 45,
  "updatedAt": 1752192000000
}

That's it. No GSIs, no sort keys, no single-table-design galaxy brain. The partition key is the thing name from part 1, and updatedAt is the timestamp that makes everything else in this article work.

The CDK for it is appropriately tiny:

const table = new dynamodb.Table(this, 'PetState', {
  tableName: 'cloudagotchi-pets',
  partitionKey: { name: 'deviceId', type: dynamodb.AttributeType.STRING },
  billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
  removalPolicy: cdk.RemovalPolicy.DESTROY,  // it's a pet, not a bank
});

The rules of life: pure functions first

Before any AWS SDK calls, the pet's biology lives in pet-logic.mjs, pure functions, no I/O, trivially testable:

// Stats decay per hour of real time. Tuned so a full day of neglect
// makes the pet visibly sad — without starving it to zero over a weekend.
const DECAY_PER_HOUR = { hunger: 2, energy: 1.5, mood: 1 };

// What each interaction does.
const INTERACTION_EFFECTS = {
  feed: { hunger: +30, energy: +5,  mood: +5 },
  pet:  { hunger: 0,   energy: 0,   mood: +10 },
  play: { hunger: -10, energy: -15, mood: +20 },  // fun is exhausting
};

And here's the single most important function in the whole project. Not a scheduled job, a calculation:

/** The cloud remembers, so the pet ages even while powered off. */
export function applyDecay(pet, now) {
  const hours = (now - pet.updatedAt) / 3_600_000;
  return {
    ...pet,
    hunger: clamp(pet.hunger - DECAY_PER_HOUR.hunger * hours),
    energy: clamp(pet.energy - DECAY_PER_HOUR.energy * hours),
    mood:   clamp(pet.mood   - DECAY_PER_HOUR.mood   * hours),
    updatedAt: now,
  };
}

Here's the part that trips people up about "time passing" in serverless systems: you don't need a process running to make time pass. Time passes for free, you only need to account for it whenever you next look. applyDecay computes elapsed hours since updatedAt and applies them, whether that's 30 minutes (scheduler tick) or a weekend (you went camping, the pet noticed).

Finally, three stats collapse into one face for the device to render:

export function moodOf(pet, hourOfDay) {
  if (hourOfDay >= 22 || hourOfDay < 7) return 'sleeping';
  const worst = Math.min(pet.hunger, pet.energy, pet.mood);
  if (worst > 60) return 'happy';
  if (worst > 30) return 'neutral';
  return 'sad';
}

Math.min : the pet's outlook is determined by its worst stat, not its average. As a design decision this is straight from the Tamagotchi school: you can't cuddle your way out of starvation. 😝

From MQTT to Lambda: IoT rules

How does a {"type":"feed"} on an MQTT topic become a Lambda invocation? IoT rules, SQL statements (yes, SQL) that run against the message stream:

new iot.CfnTopicRule(this, 'InteractionRule', {
  ruleName: 'cloudagotchi_interactions',
  topicRulePayload: {
    sql: `SELECT *, topic(2) AS deviceId, topic(3) AS kind
          FROM 'cloudagotchi/+/interaction'`,
    actions: [{ lambda: { functionArn: onInteraction.functionArn } }],
  },
});

Two tricks earn their keep here:

topic(2) extracts the second topic segment (the device's thing name) and injects it into the event as deviceId. The device never has to say who it is in the payload, which means a buggy (or malicious) device can't claim to be a different pet. Identity comes from the topic, and part 1's policy already guarantees a device can only publish on its own topics. The security work we did in article 1 quietly pays rent here.
The + wildcard means one rule serves every pet, current and future. Provision a hundred devices; deploy nothing.

There's a twin rule on the hello topic, pointing at the same Lambda.

The interaction Lambda: load, decay, apply, save, reply

The handler reads like the pet's inner monologue:

export const handler = async (event) => {
  const { deviceId, kind, type } = event;   // injected by the IoT rule SQL
  const now = Date.now();

  // 1. Load the pet, or meet it for the first time.
  const { Item } = await ddb.send(new GetCommand({ TableName: TABLE, Key: { deviceId } }));
  let pet = Item ?? newbornPet(deviceId, now);

  // 2. Time passed since we last looked. It always does.
  pet = applyDecay(pet, now);

  // 3. A "hello" just wants the current state; an interaction changes it.
  if (kind === 'interaction') {
    pet = applyInteraction(pet, type);
  }

  // 4. Persist, then tell the device what it now feels.
  await ddb.send(new PutCommand({ TableName: TABLE, Item: pet }));
  await publishState(pet);
};

Step 2 is doing quiet heavy lifting: because every code path decays first, the pet's state is always correct-as-of-now, no matter which event arrived or how long it's been. The hello flow (device boots, publishes hello, Lambda answers with fresh state) is the same code path as feeding, minus one if. "What did I miss?" is just "load + decay + reply."

The reply goes back down over MQTT via the IoT data plane:

await iot.send(new PublishCommand({
  topic: `cloudagotchi/${pet.deviceId}/state`,
  qos: 1,
  payload: JSON.stringify({ hunger, energy, mood, face: moodOf(pet, hour) }),
}));

EventBridge Scheduler: the heartbeat of the universe

One question remains: if nobody interacts and the device never says hello, who makes the pet sad? This does:

new scheduler.CfnSchedule(this, 'DecaySchedule', {
  scheduleExpression: 'rate(30 minutes)',
  flexibleTimeWindow: { mode: 'OFF' },
  target: { arn: decay.functionArn, roleArn: schedulerRole.roleArn },
});

Every 30 minutes, the decay Lambda sweeps all pets, ages them, and pushes fresh state to any device that's listening:

export const handler = async () => {
  const { Items = [] } = await ddb.send(new ScanCommand({ TableName: TABLE }));
  const now = Date.now();

  for (const pet of Items) {
    const aged = applyDecay(pet, now);
    await ddb.send(new PutCommand({ TableName: TABLE, Item: aged }));
    await publishState(aged);  // offline device? IoT drops it; the
                               // "hello" on next boot catches it up.
  }
};

💡 About that Scan (deep breath). For a hobby fleet, a 48×/day scan of a table with single-digit rows is free and honest. At real scale you'd flip the model: don't sweep, don't even store decayed values: store only updatedAt plus the stats at last interaction, and compute decay on read (we already do!). Then the scheduled sweep exists solely to push updates to online devices, and you'd drive it from the list of connected things instead of the whole table. The lazy-evaluation trick in applyDecay is the scalable part; the Scan is the shortcut bolted onto it. Know which is which and you can ship both.

Closing the loop on the device

The firmware's on_cloud_message callback from part 1 finally has a job, parse the state and hand it to the face from part 2:

cJSON *json = cJSON_ParseWithLength(payload, len);
pet_ui_set_state(hunger->valueint, energy->valueint,
                 mood->valueint, mood_from_string(face->valuestring));

Fifteen lines of cJSON (in the repo), and the loop is closed: DynamoDB is now the pet's soul, the screen is just its face.

The payoff

Deploy and flash:

cd backend && npx cdk deploy CloudagotchiBrainStack
cd ../firmware && idf.py flash monitor

Now run the experiment that justifies this whole architecture. Tap the pet, the bars nudge up (device → rule → Lambda → DynamoDB → MQTT → face, ~200 ms round trip). Then unplug the board. Go live your life for a day. Plug it back in and watch the boot sequence: hello goes up, and the state that comes back reflects every offline hour. The pet's smile flattens. Its hunger bar sits noticeably lower. It missed you, and it has the DynamoDB row to prove it.

I'm not saying I felt guilty. I'm saying I fed it immediately.

Why this matters

Strip away the pet and what we built is the canonical pattern for cloud-authoritative device state: device sends events, cloud computes truth, truth flows back down, and lazy time-accounting on read replaces any always-running process. That's a smart thermostat's schedule, a subscription's billing period, a game's energy system, an IoT sensor's calibration drift. The Tamagotchi version just has better animations.

Our pet now has a body, reflexes, memory, and consequences. One thing left in the series, and it's the one I've been saving: giving it a job. In part 4, the pet fetches the AWS news every morning, has Amazon Bedrock rewrite them in its own voice, and reads them to me out loud through Amazon Polly. My Cloudagotchi becomes my news anchor. 📰

Try it yourself

The Cloudagotchi repo : git checkout article-3
AWS IoT SQL reference : topic() and friends
EventBridge Scheduler docs : cron for your serverless universe
DynamoDB data modeling : for when your pet fleet IPOs

Cloudagotchi Part 2 : Giving it a face: Sprites and touch on a 1.8" screen

Olivier Leplus — Thu, 23 Jul 2026 15:26:14 +0000

In part 1, we gave our virtual pet a nervous system: the ESP32-S3 now speaks mutual-TLS MQTT with AWS IoT Core, and we can chat with it from the AWS console. Very satisfying. Also very much a black screen with a serial log, a pet only a backend developer could love.

Today we fix that. By the end of this article, a face blinks at you from the screen: two soft eyes and an orange smile that looks suspiciously familiar if you've ever received a package (or used AWS ^^). Tap it and it beams. Shake the board and it goes wide-eyed. Ignore it for two minutes and it dozes off with a little zzz. And every one of those interactions is published to the cloud, ready for the brain we'll build in part 3.

In this article, I will walk you through building a sprite-based pet face with LVGL plus touch input, a shake detector in ~30 lines of IMU code, and the very honest story of why I ended up with sprites after my first approach fell apart on real hardware.

⚠️ Reality check (sorry 😅): my first version of this pet was "elegant" a pet drawn procedurally from LVGL arcs and rounded rectangles, animated at 30 fps. It looked great in my head and became a corrupted mess of color bands on the actual panel. This article teaches the approach that survived contact with the hardware. The detours are documented in the ⚠️ boxes, because they'll save you an afternoon each.

The code for this article is on the article-2 branch:

git clone https://github.com/tagazok/cloudagotchi && cd cloudagotchi
git checkout article-2

The board does the hard part: the BSP

Getting a display, touch controller, and LVGL configured from scratch on embedded hardware is normally an afternoon of datasheets and SPI timing tantrums, even more for me who is a JavaScript developer 😅. The reason I picked ESP-IDF over Arduino in part 1 pays off right here: Waveshare publishes an official Board Support Package (BSP) for this exact board as a managed component. One line in idf_component.yml:

dependencies:
  waveshare/esp32_s3_touch_amoled_1_8: '^2.0.3'

...and one function call:

#include "bsp/esp32_s3_touch_amoled_1_8.h"

bsp_display_start();   // AMOLED on, touch on, LVGL initialized.

Behind that call, the BSP configures the CO5300 display driver over QSPI, the touch controller over I2C, and starts an LVGL task with display flushing hooked up.

Also, know this quirk: flashing leaves the panel dark. The panel keeps running during idf.py flash with nobody driving it. If you also have this problem after flashing, reset the board (Ctrl+T Ctrl+R in the monitor) and it comes up clean.

One more rule to remember, because it will bite you exactly once: LVGL isn't thread-safe. The BSP runs LVGL in its own task, so any other task touching the UI must take the lock:

bsp_display_lock(0);      // 1. take the LVGL mutex (0 = wait forever)
lv_bar_set_value(...);    // 2. mutate the UI
bsp_display_unlock();     // 3. give it back

Forget it and you get the embedded classic: a crash, or worse, a silent no-op.

Why sprites, not shapes: a war story in one paragraph

My first face was procedural: the head an lv_obj with a big corner radius, the mouth an lv_arc, everything animated by LVGL's animation engine: bobbing, blinking, squishing. Beautiful in theory. On hardware, the panel's QSPI transaction queue would overflow under load (Wi-Fi bring-up, TLS downloads) and silently drop color transfers. Animated widgets self-heal, they redraw every frame. But each dropped chunk of a static widget stays on screen as a garbage band forever. The more the UI animated, the more the queue overflowed, the worse the corruption. I tried buffer tuning, cache settings, custom DMA allocations; some attempts made it worse in spectacular new ways.

The approach that survived: make the display's job trivial. Pre-render every expression as a sprite baked into flash. The face is ONE static image widget; changing expression = swapping which sprite is shown = one clean rectangular blit. A blink, a mood change, a talking mouth, all the same cheap operation. Nothing redraws unless an event happens, so nothing can outrun the display.

// The face: ONE static image widget, centered. That's the whole pet.
s_face = lv_image_create(screen);
lv_image_set_src(s_face, &face_neutral);
lv_obj_align(s_face, LV_ALIGN_CENTER, 0, 20);

The sprite pipeline: from PNG to C array

The art is a set of PNGs, one per expression (neutral, happy, sad, sleeping, three talking mouths, plus per-mood blink frames). I generated mine with an AI image generator and composited them onto a common canvas with identical eye positions so frames don't jitter when swapped.

Each PNG becomes a C file via a small Python script (in the repo): pixels converted to RGB565A8, 16-bit color plus an 8-bit alpha plane, LVGL 9's friendliest transparent format, and wrapped in an lv_image_dsc_t:

const lv_image_dsc_t face_happy = {
  .header = { .magic = LV_IMAGE_HEADER_MAGIC, .cf = LV_COLOR_FORMAT_RGB565A8,
              .w = 210, .h = 162, .stride = 420 },
  .data_size = sizeof(happy_map),
  .data = happy_map,   // lives in flash, memory-mapped, zero RAM cost
};

Seven faces at ~100 KB each sounds heavy, but they compile into the 16 MB flash and LVGL blits them straight from there.

Blinking, by image swap

No animation engine needed, a blink is two sprite swaps on a timer:

static void blink_timer_cb(lv_timer_t *t)
{
    if (s_talking || s_dozing || s_mood == PET_MOOD_SLEEPING) return;
    // Dedicated blink frame: closed eyes, SAME mouth as the current mood —
    // a blink should only move the eyes.
    switch (s_mood) {
    case PET_MOOD_HAPPY: set_face(&face_blink_happy);  break;
    case PET_MOOD_SAD:   set_face(&face_blink_sad);    break;
    default:             set_face(&face_blink_neutral); break;
    }
    lv_timer_create(blink_close_cb, 120, NULL);  // reopen after 120 ms
}

The doze: a pet that gets bored

Leave the pet alone for two minutes and it falls asleep (the sleeping sprite, complete with a little zzz). Touch it or shake it and it wakes. One LVGL timer plus a reset-on-activity:

// 2 minutes with no interaction → the pet nods off.
static void idle_timer_cb(lv_timer_t *t)
{
    if (s_talking) return;      // reading aloud is not boredom
    s_dozing = true;
    set_face(&face_sleeping);
}

// Any interaction resets the boredom clock — and wakes the pet.
static void note_activity(void)
{
    if (s_idle_timer) lv_timer_reset(s_idle_timer);
    if (s_dozing) {
        s_dozing = false;
        set_face(mood_sprite(s_mood));
    }
}

A deliberate design choice: cloud updates don't wake it. The stat bars refresh quietly under the napping face; only human interaction wakes the pet. Nobody likes being woken by a cron job.

The HUD: three stats, three icons, one secret button

Across the top: three bars with icons : 🥑 hunger (avocado green), ⚡ energy (teal), ❤️ mood (pink). The icons are tiny 28×28 sprites drawn programmatically by the same PNG→C pipeline.

The avocado is not just a label, it's a button. Tapping it feeds the pet (we wire up what "feed" means in part 3; today it publishes the interaction):

static void food_clicked_cb(lv_event_t *e)
{
    pet_ui_react_happy();                           // nom nom, instantly
    if (s_on_interaction) s_on_interaction(PET_INTERACTION_FEED);
}

Shake detection: the IMU in 30 lines

The board has a QMI8658 6-axis IMU. You might reach for a gesture-recognition library. Don't, a shake is beautifully simple physics: at rest the accelerometer reads ~1 g of gravity (1000 mg); a shake is the magnitude repeatedly leaving that baseline.

// The Waveshare qmi8658 component hands us calibrated milli-g values
// (we asked for mg units once, at init, with qmi8658_set_accel_unit_mg).
float ax, ay, az;
qmi8658_read_accel(&s_imu, &ax, &ay, &az);

// 1. Total acceleration, all axes combined. At rest: ~1000 mg.
float mag = sqrtf(ax*ax + ay*ay + az*az);

// 2. A "jolt" = a reading well above resting gravity.
if (mag > 1800.0f) jolts++;

// 3. Three jolts within 800 ms = the human is shaking us.
if (jolts >= 3 && within_window) {
    on_shake();
    cooldown();   // 4. then ignore everything for 1.5s (debounce!)
}

The full version (in app_imu.c) polls at 50 Hz from its own FreeRTOS task, plenty for human hands.

The three constants are worth tuning to taste:

Constant	Value	Feel
`SHAKE_THRESHOLD_MG`	1800 mg (1.8 g)	Lower = hair-trigger, higher = vigorous shaking required
`SHAKE_JOLTS_NEEDED`	3	Filters out bumping the desk
`SHAKE_COOLDOWN_MS`	1500	One shake = one event, not twelve

💡 Debounce is the whole game. My first version fired 14 "play" events per shake, which in part 3's economy would have turned one wiggle into a pet so overstimulated it refused dinner.

One threading gotcha that bit me: the shake callback runs on the IMU task, not the LVGL task, so the reaction functions take the display lock (it's recursive, so they're safe from touch callbacks too).

Wiring body to nervous system

Everything converges in main.c, and the flow reads like the pet's reflex arc: input, reaction on screen, report to the cloud:

static void on_shake(void)
{
    pet_ui_react_startled();  // reflex: wide eyes, instantly
    app_mqtt_publish("interaction", "{\"type\":\"play\"}");  // then tell the brain
}

void app_main(void)
{
    // Face first: the pet appears before Wi-Fi even starts.
    pet_ui_start(on_interaction);
    pet_ui_set_state(80, 80, 80, PET_MOOD_NEUTRAL);

    app_imu_start(on_shake);
    app_wifi_connect();
    app_mqtt_start(on_cloud_message);
}

That ordering is a deliberate UX decision: react locally first, sync to the cloud second. The happy face flashes the instant you tap, not a network round-trip later. The cloud is the source of truth for state (part 3), but never sits between a touch and its reaction. Pets have reflexes; only their feelings need a brain.

Flash it:

cd firmware && idf.py flash monitor

...and there it is. A face, blinking on that ridiculous little AMOLED, beaming when you tap it, nodding off when you don't. Meanwhile, in the AWS console (MQTT test client, cloudagotchi/#), your affection is now telemetry:

cloudagotchi/cloudagotchi-01/interaction  →  {"type":"pet"}
cloudagotchi/cloudagotchi-01/interaction  →  {"type":"feed"}
cloudagotchi/cloudagotchi-01/interaction  →  {"type":"play"}

Why this matters

Two takeaways travel well beyond virtual pets. First, on constrained display hardware, sprites beat shapes: pre-rendering moves all the expensive work to build time and reduces the runtime to "copy rectangle", the exact operation embedded panels are good at. The corollary: the less your UI redraws, the less can go wrong, and event-driven beats animated. Second, the local-reflex / cloud-state split is the interaction pattern for connected devices. Anything that must feel instant lives on the device; anything that must be remembered lives in the cloud. Get that boundary right and your device feels alive even on flaky hotel Wi-Fi.

Right now, though, our pet publishes its little heart out into the void. Nobody listens. Its stats are hardcoded at 80. It is, developmentally speaking, a very cute reflex with no memory.

Next article: the brain. DynamoDB remembers, Lambda decides, EventBridge Scheduler makes time pass, and the pet finally gets hungry while you sleep.

Try it yourself

The Cloudagotchi repo : git checkout article-2
LVGL images documentation : formats, lv_image_dsc_t, and the online converter
Waveshare BSP on the ESP Component Registry : the one-liner display setup

Meet Cloudagotchi : Building a virtual pet with a cloud brain (Part 1)

Olivier Leplus — Tue, 21 Jul 2026 09:38:15 +0000

Remember Tamagotchis? Those little egg-shaped keychains from the 90s with a pixelated pet that got hungry, got sad, and, if you were a negligent eight-year-old like me, got dead during math class.

I've been looking for an excuse to play with the Waveshare ESP32-S3 Touch AMOLED 1.8", a $30 board with a gorgeous 368×448 AMOLED touchscreen, an accelerometer, a microphone, and a speaker. And I found one: I'm building a Tamagotchi. But with a twist that makes it worth a four-article series

The pet's body lives on the device. Its brain lives in AWS.

The device renders an adorable pet and captures your taps and shakes. But its hunger, mood, and energy are rows in DynamoDB. It gets hungry overnight because EventBridge Scheduler says so. And, my favorite part (and because it's 2026, we can't not have genAI in a project 🤪), every morning it fetches the AWS news, has Amazon Bedrock rewrite them in its own squeaky personality, and reads them to me out loud through Amazon Polly. A Tamagotchi with a job.

In this series, I will walk you through the whole build:

This article : the architecture, and connecting the ESP32-S3 to AWS IoT Core (the pet's nervous system).
Part 2 : giving it a face: animations and touch with LVGL on the AMOLED.
Part 3 : the serverless brain: Lambda, DynamoDB, and a pet that gets hungry while you sleep.
Part 4 : Bedrock + Polly: my Tamagotchi reads me the AWS news.

⚠️ Reality check before you get too excited (sorry 😅): this is a hobby project, not a product. You'll need the specific Waveshare board (or the patience to adapt the code to yours), an AWS account, and a USB-C cable. The AWS bill for the whole series is well under a dollar a month, but it is not zero, and neither is the time you'll spend staring at idf.py monitor. Worth it, though.

All the code lives in this repository. Each article has its own branch. For this one, git checkout article-1.

Why put a pet's brain in the cloud?

Fair question. The original Tamagotchi ran on a chip with less power than my USB cable. Why does mine need a cloud?

Three reasons, and they're the same reasons real IoT products are built this way:

The pet needs to...	Which means...	AWS gives us...
Get hungry while the device is off	Time must pass server-side	EventBridge Scheduler + Lambda
Remember it was sad yesterday	State must outlive reboots	DynamoDB
Say something new every day	Logic must evolve without reflashing	Bedrock, deployed behind Lambda

That last row is going to save us a lot of time, trust me (been there flashing the device for every little change I did...). When the pet's brain is a Lambda function, giving it a new behavior is just a cdk deploy, not a firmware update. The device stays dumb and stable; the personality iterates daily.

This is, not coincidentally, the exact architecture of every serious connected device you own. Your robot vacuum is a Tamagotchi with a motor. We're just building the fun version.

The architecture

Here's the full picture we're building toward across the series:

Today we're building the arrow in the middle. No pet on screen yet. First, the nervous system, then, the face. (If you came here for cuteness, part 2 will deliver, I promise.)

The pet's vocabulary: MQTT topics

The device and cloud talk over MQTT, the lingua franca of IoT: a lightweight pub/sub protocol where everyone publishes to topics. Designing the topic space first is the IoT equivalent of designing your API routes:

Topic	Direction	Meaning
`cloudagotchi/{thing}/hello`	device → cloud	"I just woke up, what did I miss?"
`cloudagotchi/{thing}/interaction`	device → cloud	"The human fed/petted/played with me"
`cloudagotchi/{thing}/state`	cloud → device	"Here are your current stats, render them"
`cloudagotchi/{thing}/briefing`	cloud → device	"Fresh AWS news, go fetch the audio"

{thing} is the device's unique name (cloudagotchi-01). Baking it into the topic path matters for security, as you're about to see.

Step 1 — The cloud side: an IoT policy with trust issues

A device talking to AWS IoT Core authenticates with an X.509 certificate, a private key that lives on the device and never leaves it. No API keys in the firmware, no secrets in your repo.

But authentication says who you are, not what you may do. That's the IoT policy, and here's the part that trips people up: a lazy policy (iot:* on *, every tutorial's dirty secret) means any compromised device can impersonate the whole fleet. Ours pins every permission to the device's own identity:

// backend/lib/iot-stack.ts — the interesting part.
new iot.CfnPolicy(this, 'CloudagotchiDevicePolicy', {
  policyName: 'CloudagotchiDevicePolicy',
  policyDocument: {
    Version: '2012-10-17',
    Statement: [
      {
        // 1. You may connect ONLY under your own name...
        Effect: 'Allow',
        Action: 'iot:Connect',
        Resource: `arn:aws:iot:${region}:${account}:client/\${iot:Connection.Thing.ThingName}`,
      },
      {
        // 2. ...publish ONLY on your own topics...
        Effect: 'Allow',
        Action: 'iot:Publish',
        Resource: `arn:aws:iot:${region}:${account}:topic/cloudagotchi/\${iot:Connection.Thing.ThingName}/*`,
      },
      {
        // 3. ...and listen ONLY to your own topics.
        Effect: 'Allow',
        Action: ['iot:Subscribe'],
        Resource: `arn:aws:iot:${region}:${account}:topicfilter/cloudagotchi/\${iot:Connection.Thing.ThingName}/*`,
      },
      // (+ iot:Receive, same resource shape — see the repo)
    ],
  },
});

The magic is ${iot:Connection.Thing.ThingName}, an IoT policy variable that resolves at connection time to the name of the thing attached to the certificate. One policy serves the whole fleet, yet cloudagotchi-01 physically cannot publish on cloudagotchi-02's topics. If your pet ever gets kidnapped at a conference (mine will, eventually), the blast radius is one sad pet.

Deploy it:

cd backend && npm install && npx cdk deploy CloudagotchiIotStack

Step 2 — A birth certificate for the pet

Certificates are secrets, so they don't belong in CloudFormation state, we create them per-device with a small script (scripts/provision-device.sh in the repo). The gist:

# Create the thing (the device's identity in AWS).
aws iot create-thing --thing-name cloudagotchi-01

# Create a certificate + private key, saved straight into the firmware tree.
CERT_ARN=$(aws iot create-keys-and-certificate --set-as-active \
  --certificate-pem-outfile firmware/main/certs/device-cert.pem \
  --private-key-outfile   firmware/main/certs/device-key.pem \
  --query certificateArn --output text)

# Tie it all together: this cert may act as this thing, under this policy.
aws iot attach-policy --policy-name CloudagotchiDevicePolicy --target "$CERT_ARN"
aws iot attach-thing-principal --thing-name cloudagotchi-01 --principal "$CERT_ARN"

One command in the repo version:

./scripts/provision-device.sh cloudagotchi-01

💡 The .pem files are gitignored, keep it that way. The private key is the pet's identity. Anyone who has it is your pet, as far as AWS is concerned. The repo ignores certs/*.pem except Amazon's public root CA, and you should never weaken that line.

Step 3 — The firmware: Wi-Fi, then mutual TLS

The firmware is plain ESP-IDF (Espressif's native framework) plus the official Waveshare board support package. Why not Arduino? ESP-IDF's built-in esp-mqtt client does mutual TLS natively, and the BSP hands us the display, touch, and audio drivers we'll need in parts 2 and 4, less glue code overall, which matters when every line has to fit in a blog post.

The build embeds the three credential files into the binary, no filesystem juggling:

# firmware/main/CMakeLists.txt
idf_component_register(
    SRCS "main.c" "app_wifi.c" "app_mqtt.c"
    INCLUDE_DIRS "."
    EMBED_TXTFILES
        "certs/AmazonRootCA1.pem"
        "certs/device-cert.pem"
        "certs/device-key.pem"
)

And the MQTT connection is disarmingly small. This is the entire handshake with AWS:

// firmware/main/app_mqtt.c — the heart of it.
const esp_mqtt_client_config_t cfg = {
    .broker = {
        // 1. AWS IoT speaks MQTT over TLS on port 8883.
        .address.uri = "mqtts://" CONFIG_CLOUDAGOTCHI_IOT_ENDPOINT ":8883",
        // 2. We verify we're talking to the real AWS...
        .verification.certificate = root_ca_pem,
    },
    .credentials = {
        // 3. ...and AWS verifies it's talking to the real us.
        .authentication = {
            .certificate = device_cert_pem,
            .key = device_key_pem,
        },
        // 4. Our policy requires client id == thing name. Not optional!
        .client_id = CONFIG_CLOUDAGOTCHI_THING_NAME,
    },
};

s_client = esp_mqtt_client_init(&cfg);
esp_mqtt_client_start(s_client);

That's mutual TLS: both sides prove who they are before a single byte of MQTT flows. Step 4 is the one people forget, if the client id doesn't match the thing name, our policy's iot:Connect statement rejects the connection, and you get the single most cryptic failure mode in IoT: an endless connect/disconnect loop with no error message. You've been warned. 😅

On connect, the pet introduces itself and subscribes to its own topics:

case MQTT_EVENT_CONNECTED:
    // Listen to everything addressed to *this* pet...
    esp_mqtt_client_subscribe(s_client, "cloudagotchi/" THING "/#", 1);
    // ...and announce we're awake. In part 3, a Lambda answers
    // this with the pet's current state. Today, only silence. 🥲
    app_mqtt_publish("hello", "{\"firmware\":\"article-1\"}");
    break;

The main loop just heartbeats every 30 seconds so we have a pulse to watch.

Step 4 — Proof of life

Configure and flash:

cd firmware
idf.py set-target esp32s3
idf.py menuconfig     # → "Cloudagotchi Configuration": Wi-Fi + IoT endpoint + thing name
idf.py flash monitor

(Your IoT endpoint: aws iot describe-endpoint --endpoint-type iot:Data-ATS.)

⚠️ Region gotcha that will produce the most confusing failure of your build. The device's endpoint and your deployed stacks must be in the same region, the endpoint hostname looks almost identical across regions (xxxx-ats.iot.us-east-1... vs xxxx-ats.iot.us-west-2...), and if they don't match, TLS connects fine and then AWS silently drops you: an endless connect/EOF loop with no error message anywhere. Always take the endpoint from the same CLI region you deployed to (check with aws configure get region), and pass that same value everywhere it's needed.

If the gods of embedded development smile upon you:

I (2841) app_wifi: Wi-Fi connected
I (4127) app_mqtt: Connected to AWS IoT Core

Now the fun part. Open the AWS IoT console → MQTT test client, subscribe to cloudagotchi/#, and watch your pet's heartbeat arrive from your desk:

cloudagotchi/cloudagotchi-01/heartbeat  →  {"alive": true}

And talk back! Publish {"msg":"hello little one"} to cloudagotchi/cloudagotchi-01/greeting and watch it land in your serial monitor:

I (91204) cloudagotchi: Cloud says [cloudagotchi/cloudagotchi-01/greeting]: {"msg":"hello little one"}

A message typed in a browser, through AWS, to a $30 board on your desk, in about 100 ms. It never stops being cool.

Why this matters

Everything we built today (certificate identity, least-privilege topic isolation, a scoped policy shared by a fleet) is the same foundation you'd lay for a real product with a million devices. The difference between a toy IoT tutorial and production IoT isn't the hardware; it's whether ${iot:Connection.Thing.ThingName} shows up in the policy. Now it's in ours, and we never have to think about it again.

Our pet has a nervous system. It can feel (receive) and speak (publish). What it doesn't have is a face, it's currently the philosophical-zombie phase of the project.

Next article: we fix that. LVGL on the AMOLED, a pet with blinking eyes and moods, touch interactions, and the accelerometer, because a pet you can't tickle by shaking the board is just a dashboard.

Try it yourself

The Cloudagotchi repo : git checkout article-1
Waveshare ESP32-S3 Touch AMOLED 1.8 wiki : board docs and schematics
AWS IoT Core policy variables : the ThingName trick, officially
ESP-MQTT docs : the client we used

💡 Disclosure: I'm a Developer Advocate at AWS, that's the cloud my Tamagotchi thinks with. The device-side patterns (mutual TLS, scoped topics) apply to any MQTT broker.

I built a Tamagotchi with its brain in AWS (and it reads me the news)

Olivier Leplus — Mon, 20 Jul 2026 11:30:09 +0000

Remember Tamagotchis? Those little egg-shaped keychains with a pixelated pet that got hungry, got sad, and, if you were a negligent eight-year-old like me, got dead during math class.

I built one. Except mine has a 368×448 AMOLED screen, an accelerometer, a speaker... and its brain runs on AWS.

Meet Cloudagotchi:

What it does

It gets hungry while the device is off : time passes in the cloud, not on the chip. EventBridge Scheduler decides when it's dinner time.
It remembers you ignored it yesterday : its feelings are a DynamoDB row. You can't reset a guilt trip by rebooting.
You feed it by tapping an avocado, pet it by tapping its face, play with it by shaking the board. Every interaction is an MQTT round trip through AWS IoT Core.
It dozes off if you leave it alone, complete with a little zzz.
And my favorite part: every morning at 7:03, it reads me the AWS news out loud : Amazon Bedrock rewrites the headlines in the pet's cute voice, Amazon Polly gives it speech, and the little face moves its mouth while it talks. A Tamagotchi with a job.

How it works (the 30-second version)

The device is the body; AWS is the brain. The ESP32 renders sprites and captures your taps. But every feeling, every stat, every decision comes from the cloud:

The stack, at a glance: ESP-IDF + LVGL on a Waveshare ESP32-S3 Touch AMOLED 1.8", talking mutual-TLS MQTT to AWS IoT Core, with Lambda, DynamoDB and EventBridge Scheduler as the serverless brain, and a Bedrock → Polly → S3 pipeline for the morning briefing. All infrastructure defined with CDK. Total running cost: well under a dollar a month.

💡 Not coincidentally, this is the exact architecture of every serious connected device you own. Your robot vacuum is a Tamagotchi with a motor. We're just building the fun version.

The series

I'm turning the whole build into four articles, each with its own git branch so you can follow along commit by commit:

This is article 0 ^^: It's a teaser so you come back to see the rest of the project 🤫
Meet Cloudagotchi: A virtual pet with a cloud brain : the architecture, and connecting the board to AWS IoT Core with mutual TLS (plus the IoT policy trick that makes a kidnapped pet a non-event).
Giving it a face : sprite-based LVGL UI, blinking, dozing, shake detection... and the very honest story of how my first "elegant" procedural face became a corrupted mess on real hardware.
It gets hungry while You sleep : the serverless state machine: Lambda, DynamoDB, and the lazy-time-accounting trick that makes a pet age without a single always-on process.
My tamagotchi reads me the AWS News : Bedrock as a squeaky-voiced news anchor, Polly's one-parameter trick that deleted an entire firmware subsystem, and the four embedded-streaming gotchas nobody warns you about.

Fair warning: this series contains real war stories. Panels that wedge themselves, buffers defeated by presigned URLs, a TCP window doing trans-Atlantic math against me. The failures were the best teachers, so they're all in there.

Part 1 drops on Tuesday, July 20th. Follow me to catch it.

The code is open source: github.com/tagazok/cloudagotchi. And, if you build one, please show me your pet. 👻

AI Without a Backend: The Browser's Built-in AI APIs for Web Developers

Olivier Leplus — Wed, 08 Jul 2026 14:12:29 +0000

For years, "adding AI to your web app" meant the same thing: sign up for an API key, wire up a backend proxy so you don't leak that key to the browser, meter your token usage, handle rate limits, and pray your bill doesn't explode when a feature goes viral. Every summary, every classification, every bit of generated text was a round-trip to someone else's data center.

Chrome is quietly changing that. A family of built-in AI APIs now ships a language model — Google's Gemini Nano — inside the browser. Your JavaScript can summarize an article, translate a comment, or generate structured data from an image without a single network call to an inference server. No API key. No backend. No per-token bill. And because the model runs on-device, the user's data never leaves their machine.

In this article, I will walk you through what these APIs are, how to use them safely, and two hands-on examples: summarizing user reviews, and turning a product photo into structured JSON with the Prompt API.

⚠️ Reality check before you get too excited (sorry 😅): these APIs are desktop-only for now (no mobile), require a fairly capable machine, and the model is a ~4 GB download on first use. They're production-viable as progressive enhancements, not as your only code path. We'll cover graceful degradation throughout.

The lineup: task APIs vs. the Prompt API

Chrome's built-in AI comes in two flavors.

Task-specific APIs do one job well, with a clean, purpose-built interface:

API	What it does
Translator	Translates text between languages on-device.
Language Detector	Detects the language of a piece of text.
Summarizer	Condenses long text into key points, a TL;DR, a teaser, or a headline.
Writer	Generates new text from a prompt (e.g. a product description).
Rewriter	Restructures or improves existing text (shorter, more formal, etc.).

The Prompt API (also called the LanguageModel API) is the general-purpose escape hatch. When no task API fits, you talk to Gemini Nano directly with your own prompts — and it's multimodal, accepting text, images, and audio as input.

The mental model: reach for a task API when your need matches one (translation, summarization, rewriting) — it's simpler and more predictable. Reach for the Prompt API when you need something custom, structured, or multimodal.

All of these share the same lifecycle, so once you learn one, you know them all.

The lifecycle every built-in AI API shares

Every API follows the same three-step pattern: feature-detect → check availability → create and use.

1. Feature detection

The interfaces live as globals (Summarizer, LanguageModel, Translator, …). Check before you touch:

if ('Summarizer' in self) {
  // The Summarizer API exists in this browser.
}

2. Availability and the model download

Here's the part that trips people up. The API being present doesn't mean the model is ready. Gemini Nano is downloaded separately the first time an origin uses it — and it's a multi-gigabyte download.

availability() tells you where you stand:

const status = await Summarizer.availability();
// "unavailable"  → not supported on this device/config
// "downloadable" → supported, but the model must download first
// "downloading"  → download in progress
// "available"    → ready to go right now

3. Create the instance (and show download progress)

create() instantiates the API. If the model still needs downloading, create() triggers it — so always give the user feedback, because "downloading 4 GB" is not something to do silently:

const summarizer = await Summarizer.create({
  monitor(m) {
    m.addEventListener('downloadprogress', (e) => {
      console.log(`Model download: ${Math.round(e.loaded * 100)}%`);
    });
  },
});

Two things worth knowing:

The download happens once per device, not once per site. After Gemini Nano is on disk, every origin that uses these APIs shares it. Chrome manages it like a cache — if disk space runs low it may evict the model and re-download later.
create() should be triggered by a user gesture. Kicking off a huge download on page load is hostile. Gate it behind a click, and check navigator.userActivation.isActive if you want to be strict.

Use case #1 — Summarizing user reviews

Picture a product page with dozens of customer reviews. Nobody reads all of them. A classic use case: a "Summarize reviews" button that distills them into a few key points, computed entirely in the browser.

The summarizer's knobs

Summarizer.create() takes options that shape the output:

type — "key-points" (default), "tldr", "teaser", or "headline".
format — "markdown" (default) or "plain-text".
length — "short" (default), "medium", or "long". What that means depends on the type: a short key-points summary is 3 bullets; a short tldr is one sentence.
sharedContext — background info that helps the model do a better job.

Putting it together

async function summarizeReviews(reviewsText) {
  // 1. Feature detection — degrade gracefully if unsupported.
  if (!('Summarizer' in self)) {
    return { supported: false };
  }

  // 2. Availability check.
  const availability = await Summarizer.availability();
  if (availability === 'unavailable') {
    return { supported: false };
  }

  // 3. Create the summarizer, tuned for product reviews.
  const summarizer = await Summarizer.create({
    sharedContext:
      'These are customer product reviews from an e-commerce store. ' +
      'Summarize the overall sentiment and the points customers mention most.',
    type: 'key-points',
    format: 'markdown',
    length: 'medium',
    monitor(m) {
      m.addEventListener('downloadprogress', (e) => {
        updateProgressUI(e.loaded); // your UI hook
      });
    },
  });

  // 4. Summarize.
  const summary = await summarizer.summarize(reviewsText, {
    context: 'Focus on product quality, value for money, and shipping.',
  });

  return { supported: true, summary };
}

Wiring it to a button:

summarizeBtn.addEventListener('click', async () => {
  // innerText, not innerHTML — feed the model clean text, not markup.
  const reviewsText = document.querySelector('#reviews').innerText;

  summarizeBtn.disabled = true;
  const { supported, summary } = await summarizeReviews(reviewsText);

  if (!supported) {
    // Progressive enhancement: fall back to your server, or just hide the feature.
    summaryEl.textContent = 'Review summaries aren’t available on this device.';
  } else {
    summaryEl.textContent = summary;
  }
  summarizeBtn.disabled = false;
});

💡 Feed it clean text. Use innerText, not innerHTML. The model summarizes better when it isn't wading through <div> soup, and you don't waste the context window on markup. If you retrieve the feedback from an API, you can also give it the json directly.

Streaming for long input

For a big block of text, waiting for the whole summary feels sluggish. Stream it instead and render tokens as they arrive:

const stream = summarizer.summarizeStreaming(reviewsText, {
  context: 'Focus on product quality and value for money.',
});

summaryEl.textContent = '';
for await (const chunk of stream) {
  summaryEl.append(chunk); // each chunk is an incremental piece of text
}

A note on languages

The Summarizer currently supports en, ja, es, de, and fr. Declaring the languages up front lets the browser reject an unsupported combination early instead of failing mid-request:

const summarizer = await Summarizer.create({
  type: 'key-points',
  expectedInputLanguages: ['en', 'fr'],
  outputLanguage: 'en',
});

Use case #2 — From a product photo to structured JSON with the Prompt API

The task APIs are great when your need fits one of them. But what about something bespoke — like: "Here's a photo of a product. Generate a draft review and auto-fill the review form."

That's a job for the Prompt API, and it shows off two of its best features: multimodal input (an image) and structured output (a JSON schema).

Step 1 — Create a multimodal session

Because we're sending an image, we declare image (and text) as expected inputs when creating the session:

const session = await LanguageModel.create({
  expectedInputs: [
    { type: 'text', languages: ['en'] },
    { type: 'image' },
  ],
  expectedOutputs: [{ type: 'text', languages: ['en'] }],
  monitor(m) {
    m.addEventListener('downloadprogress', (e) => {
      updateProgressUI(e.loaded);
    });
  },
});

Step 2 — Define the shape you want back

Prompting "reply in JSON" and hoping for the best is a recipe for parse errors. Instead, pass a JSON Schema via responseConstraint, and the model is constrained to produce output matching it:

const reviewSchema = {
  title:       { type: 'string' },
  description: { type: 'string' },
  rating:      { type: 'number', minimum: 1, maximum: 5 },
  pros:        { type: 'array', items: { type: 'string' } },
  cons:        { type: 'array', items: { type: 'string' } },
};

Step 3 — Prompt with text + image, constrained to the schema

The content of a message is an array of typed parts. We combine a text instruction with the image, and pass the schema as a constraint:

async function generateReviewFromPhoto(imageElement) {
  const result = await session.prompt(
    [
      {
        role: 'user',
        content: [
          {
            type: 'text',
            value:
              'You are given a photo of a product. Write a short, honest ' +
              'customer-style review of it. Return the result as JSON.',
          },
          { type: 'image', value: imageElement },
        ],
      },
    ],
    { responseConstraint: reviewSchema },
  );

  // Guaranteed to match the schema — safe to parse.
  return JSON.parse(result);
}

Calling it against an image on the page:

const photo = document.querySelector('#product-photo'); // an <img> element
const review = await generateReviewFromPhoto(photo);

console.log(review);
// {
//   title: "Sleek and sturdy everyday mug",
//   description: "A clean matte-white ceramic mug that looks premium...",
//   rating: 4,
//   pros: ["Solid build", "Comfortable handle"],
//   cons: ["Slightly heavier than expected"]
// }

Because the output is schema-constrained, you skip the usual defensive gymnastics — no regex-scraping a JSON blob out of a chatty response, no try/catch around a hopeful JSON.parse. You get an object shaped exactly like your database row.

💡 Why the schema matters. Without responseConstraint, the model might honor a "reply in JSON" instruction — most of the time. With it, the output is structurally guaranteed. In production, "most of the time" is where the 2 a.m. pages come from. Use the schema!

Reusing the session

A LanguageModel session keeps context across prompts, so you can hold a conversation or run several images through the same instance. When you're done, free the resources:

session.destroy();

Building responsibly: this is progressive enhancement

The single most important design principle for built-in AI: treat it as an enhancement, never a hard dependency. A big chunk of your users won't have it — wrong OS, mobile device, insufficient hardware, or the model simply hasn't downloaded yet.

A robust feature does this:

async function getSummarizer() {
  if (!('Summarizer' in self)) return null;
  const availability = await Summarizer.availability();
  if (availability === 'unavailable') return null;
  return Summarizer.create(/* … */);
}

const summarizer = await getSummarizer();
if (summarizer) {
  // On-device path: instant, private, free.
} else {
  // Fallback: call your server-side summarizer, or hide the feature entirely.
}

Keep these in mind:

Desktop only, for now. The foundation-model APIs (Prompt, Summarizer, Writer, Rewriter) don't run on Android or iOS yet. Translator and Language Detector are desktop-only too.
Real hardware requirements. Roughly: >4 GB of GPU VRAM or a decent CPU with 16 GB RAM, plus ~22 GB free disk for Chrome's profile volume. Not every laptop qualifies.
The first run is a download. Multiple gigabytes. Always show progress, and never block first paint on it.
Privacy is a genuine selling point. Inference is local; user data never hits a server. For a reviews summarizer or a content classifier handling user text, that's a real advantage — say so.

When the browser can't: a cloud fallback

Progressive enhancement is only half a strategy if the "else" branch just hides the feature. For a lot of products you want the capability to work everywhere — on the mobile users, the older laptops, the browsers that haven't shipped these APIs yet — while still giving the on-device path to everyone who can use it.

Think of it as a spectrum rather than a rivalry:

On-device (built-in AI): free, private, offline-capable, zero infra. Ideal for lightweight, high-frequency, privacy-sensitive tasks — summarizing reviews, classifying text, drafting a first pass. The trade-off is a small model with a modest context window, and it's not on every device.
Cloud (a hosted model): handles heavy reasoning, large context windows, and a specific frontier model when you need it — at the cost of latency, per-token pricing, and a backend to secure.

The nice part: your feature-detection branch is already the seam between them. Where the on-device path falls through, call your cloud model instead. Here's the same getSummarizer() pattern with a fallback to Amazon Bedrock, which exposes many foundation models behind one API:

// Client side — same seam as before, now with a real fallback.
const summarizer = await getSummarizer();

let summary;
if (summarizer) {
  // On-device path: instant, private, free.
  summary = await summarizer.summarize(reviewsText);
} else {
  // Cloud path: works on any device, at the cost of a network round-trip.
  const res = await fetch('/api/summarize', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ text: reviewsText }),
  });
  ({ summary } = await res.json());
}

// Server side (Node/Lambda) — a thin proxy to Amazon Bedrock.
import { BedrockRuntimeClient, ConverseCommand } from '@aws-sdk/client-bedrock-runtime';

const client = new BedrockRuntimeClient({ region: 'us-east-1' });

export async function handler(req, res) {
  const { text } = req.body;

  const response = await client.send(new ConverseCommand({
    modelId: 'anthropic.claude-3-5-haiku-20241022-v1:0',
    messages: [{
      role: 'user',
      content: [{ text: `Summarize these product reviews as key points:\n\n${text}` }],
    }],
  }));

  const summary = response.output.message.content[0].text;
  res.json({ summary });
}

Same feature, two engines: the browser serves everyone it can for free, and the cloud quietly covers the rest. Keeping the API key server-side (never in the browser) is exactly the backend concern the on-device path lets you skip for the majority of your users — so you pay for cloud inference only on the requests that genuinely need it.

💡 Disclosure: I'm a Developer Advocate at AWS, so Bedrock is the cloud fallback I reach for. The pattern is vendor-neutral — any hosted model API slots into the same else branch.

Why this matters

Built-in AI shifts a whole class of features from "backend project with ongoing costs" to "a few dozen lines of client-side JavaScript." No key to rotate, no proxy to secure, no bill that scales with usage, and no user data leaving the device.

It won't replace big cloud models for heavy reasoning, and it isn't everywhere yet. But for the everyday tasks that make up so much of real web work — summarize this, classify that, translate the other thing, extract structured data from a mess — the model is right there in the browser, waiting for a function call.

Try it yourself

Chrome Built-in AI docs — start here
Summarizer API doc
Prompt API guide
Structured output with the Prompt API
Tip: install the @types/dom-chromium-ai npm package for TypeScript typings.