A Raspberry Pi 5 can now hold a spoken conversation with nothing on the other end of the Wi-Fi. Google's engineers and the Raspberry Pi team spent months tuning LiteRT, Google's on-device inference runtime, so Gemma language models run on the Pi's own CPU and GPU. No API key, no monthly bill, no round trip to a data centre.
What the demo actually does
The showcase build is Reachy Mini, a small desk robot whose only brain is a Pi 5. It hears you, reasons about what you said with Gemma, and answers with speech and movement, entirely offline. For a capstone or thesis team that flips a familiar problem: your robot keeps working when campus Wi-Fi dies during the demo, and none of your lab audio ever leaves the bench.
The numbers behind it
On an 8GB Pi 5 with four CPU threads, LiteRT-LM runs Gemma 4 E2B at 99 tokens/sec prefill and 9 tokens/sec decode, peaking at 1432 MB of RAM. The same model under llama.cpp (Q4_0) manages 24 and 4 tokens/sec while eating 4406 MB. That is roughly 4x the prefill speed in a third of the memory, which is the difference between a robot that answers and one that swaps to disk mid-sentence. Gemma's tokenizer averages about 4.2 characters per token, so the voice demo streams around 27.3 characters per second, near 300 words per minute, or twice normal speaking pace.
Smaller options exist if 1432 MB is too much for your build. Gemma 3 270M suits sentiment analysis or entity extraction, EmbeddingGemma 300M handles on-device semantic search and RAG, and Gemma 3 1B covers multilingual text without the vision layers. The Pi 5's quad-core Cortex-A76 supplies roughly 153.6 GFLOPS before the VideoCore GPU path is even brought in.
Build it yourself
- Raspberry Pi 5, 8GB variant, plus the active cooler. Sustained inference will thermal-throttle a bare board.
- NVMe HAT or a fast microSD. Model files run to a few gigabytes and load time is dominated by storage speed.
- A USB or I2S microphone. I2S mics wire to the Pi's GPIO header on pins 12, 35 and 38, and need
dtoverlaylines inconfig.txtbefore ALSA will see them. - The
litert-lmruntime and a.litertlmmodel file, which you pull from the LiteRT community page on Hugging Face.
Start with Gemma 3 270M to confirm your audio chain works before downloading a multi-gigabyte E2B file. Full benchmark tables and setup steps are on the Raspberry Pi announcement: Mastering edge AI on Raspberry Pi with LiteRT and Gemma.
Originally published on blog.circuit.rocks.
Top comments (0)