One Day, Three Devices: Building Physical AI with Meta's Muse Gadget SDK

What happens when you take an AI assistant off the screen and put it into objects on your desk. A build log from October 11, 2026.

---

Yesterday I did something I'd been putting off for months: I started building physical devices for my AI setup. Not simulations, not dashboards. Actual hardware with chips and wires and LEDs that I can pick up and hold.

By the end of the day I had three devices in various states of aliveness: an ESP32-S3 board paired with Meta's Muse as a talking gadget with synchronized Christmas lights, a Raspberry Pi Zero 2 W that refused to join Wi-Fi, and an ESP32 camera streaming MJPEG from across the room. I also merged eight pull requests, wired up an AI video platform, and learned an embarrassing lesson about temporal data.

Here's the whole thing.

The starting point: a bench full of parts and zero experience

I want to be honest about where I started. When my assistant asked me about the Raspberry Pi build, I said: "We need to back up a little bit, cause I don't even know what to do with a raspberry pi other than take it out of the package."

That's the actual quote. I had the parts — a Pi Zero 2 W kit, I2S speakers, a PowerBoost charger, lithium batteries, a soldering iron — but no hands-on experience. What I did have was a homelab with real infrastructure and an AI assistant that could read SDK source code instead of guessing.

The plan was to build two things: a desk buddy (Pi Zero 2 W as a battery-powered voice announcer) and a Herald (ESP32-based Muse gadget with speaker and LEDs). The Pi was the ambitious project. The ESP32 was supposed to be simpler.

It went the other way.

Herald: from mystery board to paired gadget in one evening

The ESP32 work started with a basic question: what board do I actually have? I had several — a 3-pack of Lonely Binary ESP32 boards, two Seeed XIAO ESP32-S3 boards (one with camera), and a newly delivered board I'd just opened.

I plugged the new one into my MacBook and it showed up as /dev/cu.usbmodem5C941316811, identified generically as "USB Single Serial." Not helpful. So we probed it directly with esptool:

The silicon came back as ESP32-S3 (QFN56), revision v0.2, with 8 MB embedded PSRAM. That 8 MB of PSRAM matters — Meta's HomeLink tunnel feature requires it, and most cheap ESP32 boards don't have it.

Here's where it got interesting. Meta's muse-gadget-sdk (the facebookincubator/muse-gadget-sdk repo) has board definitions for specific hardware, but nothing for a bare XIAO ESP32-S3. The SDK supports custom board definitions though, so I wrote one:

• esp32/devices/sdkconfig.xiao-s3 — ESP32-S3, 8 MB quad flash, 8 MB octal PSRAM
• BOOT on active-low GPIO0
• No status-light backend (the XIAO's GPIO21 LED is monochrome active-low, not addressable RGB)
• A tools/board.sh profile and a README row

ESP-IDF v6.0.1 installed in the background. The firmware built cleanly — muse-gadget.bin at 0x151000 bytes with 34% free in the smallest app partition.

The SDK token lived in my homelab secrets vault (never printed, never logged — validated by prefix and length only, injected into the build config, firmware rebuilt). Then: copy to MacBook, flash with esptool, verify the hash, hard-reset.

Serial showed healthy heartbeats with 8 MB PSRAM. I completed the pairing flow in the Muse app. The board came up as MuseGadget-D356B4, got on Wi-Fi, and showed setup=done wifi=up ws=up.

I named it Herald. Nickname: Hal.

The firmware that changed everything: remote management

The stock gadget firmware is fine for basic use, but I wanted more. Specifically, I wanted to control everything remotely — lights, sound, Wi-Fi, device management — without ever plugging in USB again.

So I had a firmware prompt written for the Codex session: expose led.set, led.pattern, led.clear, audio.stop, wifi.scan/status/reconnect, device.reboot/info/diagnostics, OTA delivery, and settings management as remote commands.

When Herald re-registered with the new firmware, device.describe showed the full command set. device.info confirmed: firmware 999.0.1, ESP32-S3, ESP-IDF v6.0.1, 16 MB flash, 8 MB PSRAM. Wi-Fi connected at -50 dBm with zero reconnects.

Then the fun part. I synthesized a 39.5-second bell rendition of "Jingle Bells" (public domain, so no copyright issues) and ran it as a Christmas show: audio through the speaker while the two LEDs alternated red and green every 500ms. Both audio.play and led.pattern returned successful completion.

Music plus lights, controlled entirely through remote commands. No USB cable. That's the moment it stopped feeling like a dev board and started feeling like a product.

ElevenLabs: giving Herald a real voice

The next step was voice. Meta's SDK has basic TTS, but I wanted quality — so I wired up ElevenLabs as a voice provider.

The integration: a skill with a Python client, a wrapper script, API key resolved transiently from the vault on my homelab server (never touches local disk). The flow is: pick a voice → render TTS → upload to temporary storage → audio.play on Herald.

Verified end-to-end with "Hello, this is Herald..." in the Eric voice. It worked.

I also discovered that ElevenLabs' Music endpoint returns raw 32kHz PCM without a WAV header (not the 16kHz I requested) — a fun debugging detour that ended with "treat it as 32kHz because the sample count matches the requested duration." Math doesn't lie.

The knowledge graph lesson I won't forget

While the hardware was happening, I was also watching a knowledge graph rebuild. I'd torn down the old graph days earlier (long story, trust issues with the pipeline) and was rebuilding it with a new schema.

My assistant was doing a data-quality check and flagged what looked like errors: my graph showed me as living in Tampa and working at WilmerHale and Ropes & Gray. "That's wrong," the assistant said. "You're conflating travel with residence."

I corrected it: "I did previously work at those places and I did previously live in all those places, just not currently."

The edges had valid_from and valid_until timestamps. The assistant had read temporal edges without their dates.

Then I asked the question that turned it into a design rule: "My problem is, is every agent that uses this platform gonna make the same mistake that you just made and not read any of the temporal data?"

The answer became a spec: the rebuilt KG client's read helpers must default to current-only (valid_until IS NULL OR valid_until > now()), with an explicit parameter required for historical data. Make the right query the default so every consumer doesn't have to remember to filter temporally.

That's now in the kg-client PR. A mistake I caught became a guardrail for every future agent.

Eight PRs merged, one deploy scare

The KG rebuild wasn't just schema — it was 432 old consumers that needed retiring or porting. On Oct 11, eight PRs merged to main:

• kg-client (the temporal-filtering client)
• Retires for Groups B, C, H, I (old extractors, curation, briefings)
• Rewrites for Groups D (identity), E (timeline), F (Atlas), G (search)

Groups E and G had real merge conflicts. Two fresh Codex sessions resolved them, all tests green.

Then the deploy failed. Postgres migration parse error, three services down. The culprit: migration 2210 had already been applied manually during triage, and a backup was probably holding a lock. Retry went through clean. All services up, all eight PRs live, graph steady at 23,224 nodes.

Group J (83 handlers) was still in flight at end of day. The rebuild wasn't done, but the hard part was.

The Pi that wouldn't boot (yet)

Remember the Pi Zero 2 W I knew nothing about? We got Raspberry Pi OS Lite downloaded and checksum-verified. The macOS admin dialog for the SD card flash failed silently (twice), so I flashed it manually with dd.

First boot: solid green LED, but no Wi-Fi, no USB gadget detection. The wpa_supplicant.conf was written, the dwc2 overlay was initially placed under the wrong config section (fixed to [all]). Still no joy at last check.

The Pi is the project that humbled me. The ESP32 — which I expected to be harder — worked first try. The Pi — which should have been "just flash an SD card" — is still sitting there with a green LED and no network.

I'll get it. But there's a lesson in which one fought me.

The camera that just worked

The third device was the easiest. The XIAO ESP32-S3 Sense (the camera variant) got ESPHome firmware with the OV2640 pins verified against three sources. Flashed, joined Wi-Fi at 10.0.0.152, and immediately started serving snapshots and MJPEG.

An MCP server went on the homelab so my AI setup can grab frames programmatically. First frame: a real 800x600 JPEG. It just worked.

What I'd tell someone starting this

1. Read the SDK source, not the marketing. The muse-gadget-sdk README told me the Zero 2 W was supported. The AGENTS.md told me the installer refuses --run-as root and stores tokens 0600. The source is the spec.

2. Probe before you assume. That esptool probe (ESP32-S3, 8MB PSRAM) settled the board question in seconds. Don't guess from packaging.

3. Separate your secrets properly. The SDK token lived in a vault, was validated by prefix only, injected at build time, never printed. This isn't paranoia, it's hygiene.

4. Make the right thing the default. The temporal KG lesson applies everywhere. If every consumer has to remember to do the right thing, they won't. Build it into the API.

5. The "simple" project will humble you. I expected the ESP32 to be hard and the Pi to be easy. Inverse. Budget your humility accordingly.

What's next

Herald needs its reactive LED mode debugged (the audio-reactive firmware hangs — that's a known issue with a fix prompt waiting). The Pi needs to actually boot and join Wi-Fi. The Beambox badge — a little Bluetooth display — is being reverse-engineered to become Herald's face, showing different expressions for idle, listening, and speaking states.

And the KG rebuild needs Group J merged and deployed.

Yesterday was about proving the concept: AI doesn't have to live in a chat window. It can sit on your desk, light up when music plays, and speak in a voice you chose. The infrastructure to make that real — the firmware, the voice pipeline, the device management — is all buildable by one person with a bench full of parts and an AI assistant that reads source code.

Even if you've never touched a Raspberry Pi before.

---

Built with Meta's Muse Gadget SDK, ESP-IDF v6.0.1, ElevenLabs, ESPHome, and a healthy disregard for "I don't know how to do this." All hardware was purchased, not sponsored. The Jingle Bells rendition is public domain.