Run Gemma Locally on Raspberry Pi 5: LiteRT Hands-On

Running a language model completely offline on a Raspberry Pi 5 is no longer a party trick – what's new is that Raspberry Pi itself now recommends a way to do it. On August 11, 2026, Google AI Edge and Raspberry Pi published a joint announcement: Google's LiteRT runtime is now optimized for the Pi 5, including language models from the Gemma family and experimental GPU acceleration.

I wanted to know what survives contact with reality: how the installation actually goes, how fast Gemma really responds on the Pi, how much RAM the whole thing eats – and whether it's worth switching from Ollama. In this guide you'll set up LiteRT step by step and get your first model answering locally.

What is LiteRT? LiteRT is Google's on-device runtime for AI models and the direct successor to TensorFlow Lite. It runs classic ML models and language models locally on CPU or GPU – officially optimized for the Raspberry Pi 5 since August 2026, including the LLM layer LiteRT-LM. Free and open source.

What Google and Raspberry Pi actually announced

The announcement appeared simultaneously on the Raspberry Pi news blog and Google's developer blog – joint appearances like this are rare. The gist: Google has optimized LiteRT specifically for the Pi 5's hardware, and it delivers three things at once.

First, language models. Through an intermediate layer called LiteRT-LM, models from the Gemma family run directly on the Pi – from Gemma 3 270M for simple tasks up to Gemma 4 E4B for multi-step reasoning. The recommended sweet spot between speed and capability is Gemma 4 E2B.

Second, the GPU. The Pi 5's VideoCore VII can now be used for inference, enabled by experimental WebGPU support in the open Vulkan driver V3DV, which Igalia contributed to Mesa.

Third, a lean command-line tool that downloads, runs, benchmarks and converts models – and only fetches the components you actually use.

The official benchmark numbers

Google published numbers for the Pi 5 with 8 GB of RAM (four threads, 1,024 input / 256 output tokens):

Model	Runtime	Prefill (tok/s)	Decode (tok/s)	Peak RAM
Gemma 4 E2B	LiteRT-LM	99	9	1,432 MB
Gemma 4 E2B	llama.cpp	24	4	4,406 MB

That looks like a landslide – but note that Google compares a QAT-quantized model (quantization-aware training) against a conventionally post-quantized Q4_0 variant in llama.cpp. The advantage is real, but it isn't a clean apples-to-apples comparison. Which is exactly why I ran my own measurements below.

Also remarkable: the bare installation is roughly a 25 MB download versus about 1.44 GB for Ollama. The CLI is a thin Python shell that pulls in runtime components on demand.

If the name LiteRT means nothing to you: it's the renamed and further developed TensorFlow Lite – your old .tflite models still run. What's new above all is large language model support.

What you need

Tested on August 19–20, 2026 with litert-cli-nightly 0.2.0.dev20260819 and litert-lm-nightly 0.17 on Raspberry Pi OS (64-bit) – LiteRT is officially still an early preview, so details may change.

• a Raspberry Pi 5 – Google tests with the 8 GB variant, but my entire test ran on the cheaper Pi 5 with 4 GB, and it turned out to be enough
• Raspberry Pi OS in the 64-bit version – LiteRT only exists for aarch64
• at least 10 GB of free storage – the model brings 2.59 GB, plus ~1 GB of compilation cache on first launch
• active cooling and the official 27 W USB-C power supply – all four cores run continuously while generating
• a bit of patience for the 2.59 GB download (noticeably faster with a Hugging Face token)

Quickly confirm your system meets the requirements:

uname -m           # must return aarch64
cat /etc/os-release
free -h
df -h ~
sudo apt update && sudo apt full-upgrade -y

Installing LiteRT

LiteRT comes as a Python package. Google recommends uv:

curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env

mkdir -p ~/litert-test && cd ~/litert-test
uv venv --clear --python=3.13 --seed
source .venv/bin/activate

uv pip install litert-cli-nightly
litert --help

The most common trap: after every reboot and in every new SSH window you have to activate the environment again (source ~/litert-test/.venv/bin/activate), otherwise the shell won't know the litert command.

A note on packages, because the two official pages contradict each other: PyPI also hosts a stable litert-cli (0.1.1, May 2026). I tried both – both install fine and both know the lm subcommand. Curiously, the stable version auto-installed the LLM components on first use, while the nightly aborted with a misleading error. I'm sticking with the nightly anyway: the official guide uses it and its components are considerably newer (litert-lm 0.17 vs 0.12).

Running Gemma 4 locally

Before the first LLM call, one package is still missing – and this is the nastiest trap of the whole setup. Run litert lm too early and you get:

Error: 'litert-lm' executable not found in PATH.
Please install 'litert-lm-cli' to use this command.

The package this error message names doesn't exist. There is no litert-lm-cli on PyPI. The correct name is:

uv pip install litert-lm-nightly

After that, the actual call works:

litert lm run \
  --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
  gemma-4-E2B-it.litertlm \
  --prompt="What is the capital of France?"

Tip: start the 2.59 GB download inside a tmux session – if your SSH connection drops, the download survives. And set a Hugging Face token first: the repo isn't gated, but without a token the Hub throttles the transfer ("You are sending unauthenticated requests to the HF Hub…"). Not mandatory, but definitely faster.

Leave out --prompt and you land in an interactive chat.

The download quirk nobody documents

Call the same --from-huggingface-repo command a second time and the download starts again at 0 % – every single time. This route never stores the file permanently. Its sibling litert download behaves the other way around: it recognizes an existing file and skips the download. For anyone using the model more than once, this is the better entry point:

litert download litert-community/gemma-4-E2B-it-litert-lm --file "gemma-4-E2B-it.litertlm" --output gemma
litert lm run ./gemma/gemma-4-E2B-it.litertlm --prompt="What is the capital of France?"

Careful: without --file you download the entire repository – Google stores nine builds of the model there (CPU, GPU, browser, Qualcomm/Intel/Tensor NPUs), almost 24 GB in total. For the Pi, only gemma-4-E2B-it.litertlm (2.59 GB) matters.

One more disk-space note: on the very first start, LiteRT compiles the model for your CPU and stores ~1 GB of .xnnpack_cache files next to the model file (2.5 → 3.5 GB). All subsequent starts skip this step.

Does it run on 4 GB of RAM?

Google quotes a peak of 1,432 MB – measured on an 8 GB Pi. None of the official announcements answers whether less is enough. My test device is the Pi 5 with 4 GB, and the answer is a clear yes: while the model was writing a longer answer, used memory sat at 1,572 MB out of 4,049, with swap at zero for the entire generation. Gemma 4 E2B and the 4 GB Pi are a comfortable fit.

Speed

litert lm benchmark ./gemma/gemma-4-E2B-it.litertlm (single run after warm-up):

Value	My measurement (Pi 5, 4 GB)	Google's reference (Pi 5, 8 GB)
Prefill	96.46 tok/s	99 tok/s
Decode	7.74 tok/s	9 tok/s
Init time	0.74 s	–
Time to first token	2.78 s	–

The benchmark defaults to 256 prefill tokens vs Google's 1,024, so it's not exactly the same experiment – but the ballpark matches. The remarkable part: the 4 GB model lands almost level with Google's 8 GB reference. In practice, ~8 tokens per second means the model writes about as fast as you read.

A reality check

I gave the model a typical maker question: "My Raspberry Pi is running too hot. Write me a short bash script that reads the CPU temperature and warns above 70 degrees." The answer looked impressive – cleanly structured, complete with cron job and systemd hint. Except: on a Raspberry Pi, the delivered script doesn't work. It relies on lm-sensors and greps for "Core" entries – desktop-processor territory. On the Pi the sensor is cpu_thermal, so the script would forever hit its own error path. Plus a classic bash blunder: -gt chokes on floats like "45.6".

For completeness, the actual Pi way, no extra packages:

#!/bin/bash
TEMP=$(( $(cat /sys/class/thermal/thermal_zone0/temp) / 1000 ))
if [ "$TEMP" -gt 70 ]; then
  echo "Warning: CPU at ${TEMP} degrees - check your cooling!"
fi

The takeaway: Gemma 4 E2B is a linguistically strong generalist, but no device specialist – expected at this size class. Great for summarizing, translating, explaining; double-check technical specifics.

Not just LLMs: image classification

litert download litert-community/efficientnet_b1 --output efficientnet
litert run efficientnet/efficientnet_b1.tflite --input ~/testimage.jpg

With my test image – a plain coffee mug – EfficientNet-B1 returned coffee mug clearly in front (score 7.84), inference done in 88 ms. The same route gets you object detection (YOLOX), speech recognition (Moonshine) and embedding models.

GPU instead of CPU?

V3D_WEBGPU_OVERRIDE=1 litert run efficientnet/efficientnet_b1.tflite --input ~/testimage.jpg --gpu

Expectation management straight from Raspberry Pi: the GPU is slower than the CPU here – the WebGPU path in the V3DV driver is experimental. My own numbers confirm it: 88 ms on the CPU, 170 ms on the GPU, identical results. The point isn't speed but division of labor: offload continuous vision workloads to the GPU, keep the CPU free for the language model – exactly how Google's Reachy Mini robot demo works. (An Failed to load OpenCL library info line at startup is harmless – LiteRT falls back to Vulkan as intended.)

Troubleshooting

Problem	Solution
'litert-lm' executable not found in PATH	uv pip install litert-lm-nightly – the suggested litert-lm-cli doesn't exist
litert: command not found	venv not active – source ~/litert-test/.venv/bin/activate
uv stuck resolving	export UV_INDEX_URL=https://pypi.org/simple
Unclear error	prefix with LITERT_VERBOSE=1
Failed to build flatbuffer model	usually just a missing/empty file – check path and size
Weird states after aborts	litert clean empties all caches

Also: litert list reports "No managed models found" even when your model sits in an --output folder – it only knows LiteRT's own managed cache. And in general: in this preview state, the error messages love to lead you astray. Read them skeptically and check the simple things first.

LiteRT or Ollama?

I wanted to benchmark both on the same device with the same model and quantization – Ollama now carries gemma4:e2b-it-qat. It didn't happen, and the reason is the actual takeaway:

$ ollama run gemma4:e2b-it-qat "What is the capital of France?"
Error: 500 Internal Server Error: llama-server process has terminated: signal: killed

Ollama can't get the very same model running on the 4 GB Pi. The kernel OOM-kills the loading process – dmesg shows it requested ~4.5 GB (total-vm:4506384kB), almost exactly Google's llama.cpp figure, and more than the device owns. Twice in a row, despite 2 GB of swap.

Criterion (Gemma 4 E2B QAT, Pi 5 with 4 GB)	LiteRT-LM	Ollama
Does the model run?	Yes – 7.74 tok/s output	No – killed by the kernel (OOM)
RAM while running	1,572 MB, swap at 0	requests ~4.5 GB
Model on disk	2.59 GB (+ ~1 GB compile cache)	4.3 GB
Runtime download	~25 MB	~1.44 GB

Staying fair: on an 8 GB Pi the same model runs fine under Ollama, and smaller models like Gemma 3 1B are no problem even on 4 GB. But exactly where things get tight, the architectural difference shows – LiteRT-LM copes far better with E2B's offloaded per-layer embeddings.

In character, the two differ more than in numbers: Ollama is the comfortable everyday route (huge library, background service, Open WebUI). LiteRT is the lean construction kit – for tight memory, for embedding a model into your own application, and for running vision, speech and LLM on one runtime.

What comes next

LiteRT and Gemma are set to support the Hailo accelerators – offloading inference to the Raspberry Pi AI HAT+ 2 with the same commands as above. No date yet; I'll add measurements to the original article as soon as it lands.

Verdict

LiteRT is the first local-LLM runtime that Raspberry Pi itself actively recommends – and it shows. Installation takes ten minutes, the memory appetite is markedly lower than the alternatives, and the same toolbox runs image and speech recognition. It's also explicitly an early preview: misleading error messages, experimental GPU support, modest model selection. If you want a dependable local chatbot today, Ollama remains the more relaxed ride – on an 8 GB Pi. But watching my cheapest Pi 5 run a model that the established incumbent chokes on did leave an impression.

---

This article first appeared on raspberry.tips: Run Gemma Locally on Raspberry Pi 5: LiteRT Hands-On – including all screenshots from the test run and a step-by-step FAQ.