ESP32-AI: Running a 28.9M-Parameter LLM on an $8 Microcontroller

#esp32 #llm #microcontroller #ai

ESP32-AI is a lightweight local AI project that runs a 28.9M-parameter TinyStories language model on an $8 ESP32-S3 microcontroller, generating text offline with no cloud dependency and about 9.5 tokens per second.

Key Takeaways

A 28.9M-parameter TinyStories LLM runs fully offline on an ESP32-S3 using Per-Layer Embeddings.
Per-layer embedding tables live in flash via execute-in-place memory mapping, leaving dense core weights at about 560KB in SRAM.
Performance lands around 9.5 tokens per second with an OLED UI and no internet after flashing.
The project currently generates short stories, not instruction-following dialogue or factual QA.
ESP32-S3 boards cost roughly $8 to $12, making this one of the lowest-cost generative-AI platforms available.

What slvDev built

Developer slvDev built esp32-ai , a minimal firmware stack that runs a 28.9M-parameter TinyStories model and generated short stories entirely offline on an ESP32-S3 with no internet connection after flashing. The project first appeared on GitHub in late July 2026, then reached the top of Hacker News, where it signaled mainstream curiosity about commodity-edge generative AI. Unlike smartphone-local demos, this implementation fits the full model footprint plus inference runtime under the memory constraints of a production microcontroller. An attached OLED displays generated text word-by-word through a custom interface baked into the firmware. The result is a self-contained narrative machine rather than a chat assistant.

How Per-Layer Embeddings make it possible

Google’s Per-Layer Embeddings technique keeps most model weights in flash memory through memory-mapped execute-in-place execution, so the microcontroller only fetches about 450 bytes per token while preserving roughly 560KB of dense weights in active SRAM. This is the core architectural insight that makes the project viable. In an explainx.ai technical breakdown of the implementation, this pattern shifts the embedding table and output head into flash, leaving only the per-layer attention and MLP matrices as dense runtime memory pressure. Traditional embedding-heavy architectures would immediately overflow the ESP32-S3’s modest SRAM budget, but per-layer decomposition avoids that wall. The tradeoff is heavier flash read traffic, which Atmel’s ESP32 flash controller can service passively without CPU stalls in practice. According to the project README, the firmware places the model into a custom flash partition and stages tied embedding and output heads in PSRAM at boot.

Hardware requirements

The build targets the ESP32-S3 N16R8 variant , which provides 512KB of SRAM and 16MB of onboard flash, paired with a small OLED such as the LILYGO T-Display-S3 for reading generated text. These are commodity development boards priced around $8 to $12 depending on sourcing, and they are easy to assemble on a breadboard. The device uses an attached screen as its human interface because a microcontroller cannot easily stream tokens over a full network stack while the model is running. According to the slvDev GitHub repository, the repo documents exact partition layouts and wiring expectations for compatible OLEDs. XDA Developers noted the feasibility angle for reader replication, framing the project as a practical demo rather than a black-box research artifact. The hardware requirements remain within reach for hobbyists and embedded students, though flashing and partition configuration still demand comfort with the ESP-IDF toolchain.

Why edge-local story generation matters

Running local generative inference on a sub-$10 microcontroller matters most for smart sensors, wearables, and industrial controllers where cloud latency or permanent offline operation rules out standard AI APIs. A self-hosted tiny language model can react to interrupts, log context-rich events, or provide on-device narration without transmitting sensitive sensory data. Reddit’s r/esp32 discussion captured practitioner excitement, with users pointing to the project as proof that token generation itself can be embedded. If we compare this to Bonsai 27B on smartphones, the dividing line is clear: one pushes large-model capability down to a handheld computer at several hundred dollars, while the other pushes it to an eight-dollar chip with no screen beyond what the developer adds manually. Both share the same offline-AI philosophy, but the microcontroller market multiplies the physical deployment surface by orders of magnitude, and that scale raises trust questions about where inference actually executes. Our earlier breakdown of where AI runs and where trust matters showed that on-device execution can preserve privacy when the workload never leaves the chip.

Limitations and realistic expectations

The current esp32-ai implementation generates stories rather than following instructions or answering factual questions, and the firmware remains a prototype without verified production hardening. Token throughput lands around 9.5 tokens per second on current silicon, which feels deliberate rather than conversational. There is no instruction-following layer built on top of TinyStories in this repo, so users who expect ChatGPT-style dialogue will need to adjust expectations. Flash longevity, watchdog behavior, and signed firmware updates have not been validated by the developer at this stage. According to the research brief, the prototype maturity is the most significant risk factor for makers and product teams evaluating the approach. The pattern itself, however, is portable: Per-Layer Embeddings can be applied to other microcontrollers with similar memory hierarchies, which means this is less about one repo and more about a reusable design move.

Trying it yourself

You can flash esp32-ai by cloning slvDev’s repository, configuring the firmware partition for a TinyStories 28.9M PLE model, and attaching a compatible ESP32-S3 board with an OLED display. The project provides an end-to-end firmware walkthrough, though it assumes familiarity with ESP-IDF build and partition manipulation. explainx.ai breaks down the per-layer embedding mechanics and memory mapping for readers who want the supporting context before diving into the code. Generated output quality is best described as atmospheric short fiction rather than factual QA, so test the system against the use case you actually need before embedding it in a product.

If you are deciding between edge-local AI targets for a hardware prototype, microcontroller inference offers the lowest bill of materials and simplest supply chain, even if the present model capabilities are narrow. The broader signal here is that LLM deployment is splitting into phone-class, microcontroller-class, and cloud-class nodes, and each tier will need different quantization, memory, and trust models.

Conclusion

ESP32-AI is a compact proof of concept showing that a 28.9M-parameter LLM can run fully offline on an $8 ESP32-S3 using Per-Layer Embeddings, delivering around 9.5 tokens per second for story generation. The build is prototypical and genre-limited, but it opens a clear path toward sub-$10 embedded generative AI for sensors, wearables, and IoT edge nodes.