When I was a kid, I loved the movie WALL·E. I often wondered if something like that would be possible in the future.
As I grew up, I learned how robots work and how much it costs to build one. I realized that building intelligent machines isn't just about software—it's also about overcoming hardware limitations.
Projects like this remind me that we're slowly bringing intelligence to even the smallest devices.
When people talk about running AI locally, they usually mean a laptop with a decent GPU, a Raspberry Pi, or an NVIDIA Jetson.
Microcontrollers rarely enter this type of conversation.
After all, an ESP32-S3 only has a few megabytes of memory. Running a language model on it sounds impossible.
I saw a post about running an LLM on an ESP32 by slvDev, and I was fascinated by it.
Then I came across another project by Wladimir Avila, which took the idea even further.
Instead of asking:
"How can we fit a bigger model on one ESP32?"
It asks:
"What if multiple ESP32 boards worked together like a tiny AI cluster?"
That's exactly what esp32s3-distributed-ai does.
The Problem
Modern language models are huge.
Even "small" language models often require tens or hundreds of megabytes of memory, which is far beyond what an ESP32-S3 can provide.
Buying more powerful hardware is the obvious solution.
But this project explores another idea:
Split the model across multiple microcontrollers.
Rather than forcing one board to hold the entire model, each board becomes responsible for a different part of the inference pipeline. The boards then communicate wirelessly to generate text together.
The Hardware
The setup is surprisingly simple.
- 3 × ESP32-S3 N16R8 boards
- ESP-NOW for communication
- One board hosts a Wi-Fi access point
- A browser-based interface to interact with the model
No cloud.
No router.
No internet connection after flashing the firmware.
Everything runs locally.
How the Model Is Split
Instead of storing the entire transformer on one board, the project partitions it.
Prompt
│
Board C (Web UI)
│
Board A (Embeddings)
│
Board B (Transformer)
│
Board A (Output Head)
│
Board C (Browser)
Each board has a specific responsibility.
Board A
- Embeddings
- Output head
Board B
- Transformer layers
- KV cache stored in PSRAM
Board C
- Remaining embedding table
- Wi-Fi access point
- Web interface
The boards exchange intermediate activations over ESP-NOW until the next token is generated.
Why ESP-NOW?
One of the most interesting design decisions is the communication layer.
Instead of using MQTT, TCP sockets, or a router, the project relies on ESP-NOW, Espressif's lightweight peer-to-peer wireless protocol.
Advantages include:
- No external infrastructure
- Low communication overhead
- Direct board-to-board messaging
- Fully offline operation
According to the author, the boards communicate through a custom protocol built on top of ESP-NOW, with Board C simultaneously hosting the browser interface.
A 56 Million Parameter Model
The project runs a language model with approximately 56 million parameters.
To make this possible, it uses:
- 4-bit and 8-bit quantization
- Split Per-Layer Embeddings (Split-PLE)
- Flash memory for large embedding tables
- PSRAM for the KV cache
These optimizations allow a model that would never fit on a single board to execute across three inexpensive microcontrollers.
Inspired by Previous Work
This project builds upon another impressive experiment by slvDev, who demonstrated that a 28.9 million-parameter language model could run on a single ESP32-S3.
That work introduced the use of Per-Layer Embeddings (PLE), inspired by Google's Gemma architecture, to dramatically reduce SRAM requirements by storing most embedding parameters in flash memory.
The distributed version extends that idea even further by partitioning the model across multiple devices.
Current Limitations
The project is still experimental.
Some of the current limitations mentioned by the author include:
- Brute-force tokenizer implementation
- Quality loss from aggressive 4-bit quantization
- Generating relatively short responses (around 30 words)
- Focusing on demonstrating distributed inference rather than competing with larger LLMs
Why This Matters
The most exciting part isn't that three ESP32 boards can generate text.
It's the engineering mindset behind it.
Instead of scaling up, this project scales out.
Rather than purchasing more powerful hardware, it distributes computation across multiple low-cost devices.
That idea could inspire future work in:
- Offline robotics
- Smart factories
- Distributed IoT intelligence
- Edge AI
- TinyML research
- Sensor networks
As embedded AI becomes more common, approaches like this could make sophisticated models accessible on hardware that costs only a few dollars.
Final Thoughts
This isn't about replacing ChatGPT.
It's about pushing the boundaries of what's possible on resource-constrained hardware.
Seeing three inexpensive ESP32 boards collaborate to run a 56M-parameter language model is a reminder that innovation often comes from clever system design—not just bigger GPUs.
Projects like this show that the future of AI won't exist only in massive data centers. It may also live on tiny devices working together at the edge.
Project Repository
https://github.com/wladimiravila/esp32s3-distributed-ai
Thanks for reading!
I'm Ganesh, and I'm building MakeSense, an AI tool that turns public GitHub pull requests into concise summaries, prioritized insights, and interactive quizzes. It's free, unlimited, and source-available. If you review open-source code, I'd love for you to give it a try and share your feedback.
MakeSense: https://makesensegithub.com/
Top comments (0)