Running Google Gemma Locally on a Raspberry Pi

Artificial intelligence is no longer limited to large cloud servers and expensive GPUs. With the growth of smaller and more efficient Large Language Models (LLMs), it is now possible to run useful AI models locally on relatively low-power hardware such as a Raspberry Pi.

In this article, we will explore how to install and run Google Gemma on a Raspberry Pi and see how well a compact AI model performs on edge hardware.

What is Gemma?

Gemma is a family of lightweight, open AI models developed by Google. It is designed to bring many of the capabilities of larger language models to developers who want to experiment with AI locally or integrate AI into their own applications.

Unlike cloud-based AI services, where your prompts are sent to remote servers for processing, a locally running model can process requests directly on your own hardware.

Gemma models are available in different sizes, making them suitable for a variety of hardware configurations. Smaller models are particularly interesting for devices with limited CPU, memory, and power resources.

Why is Gemma Important?

One of the biggest challenges with modern AI is the amount of computing power required to run large models. Most powerful LLMs depend on data centres equipped with high-end GPUs.

Gemma takes a different approach by providing smaller, efficient models that developers can run in more resource-constrained environments.

Why Run Gemma on a Raspberry Pi?

This is where things get interesting.

A Raspberry Pi is a small, inexpensive, low-power computer that was never designed to compete with dedicated AI hardware. However, its ARM processor, increasing memory capacity, and excellent Linux support make it an interesting platform for experimenting with lightweight AI models.

1. Download LM Studio

Open the official LM Studio download page:
https://lmstudio.ai/download
Download the version appropriate for your Raspberry Pi's operating system and architecture.
Once downloaded, install LM Studio using the provided installation method for your OS.

For this setup, we will use LM Studio's headless daemon 'llmster' to run Gemma on the Raspberry Pi.
Unlike the regular LM Studio desktop application, llmster is designed to run LM Studio's LLM inference capabilities without requiring a graphical interface. This makes it a better fit for a Raspberry Pi running as a server or headless system.

curl -fsSL https://lmstudio.ai/install.sh | bash

After installation, launch LM Studio. You should be presented with the main LM Studio interface, where you can search for and download compatible AI models.

For this experiment, I am using my existing Raspberry Pi 5 with 8 GB of RAM. The Raspberry Pi is already configured as a local AI server and is currently running Ollama and Open WebUI.

After completing the installation, start the LM Studio daemon using the following command:

lms daemon up

This starts the LM Studio headless daemon in the background, allowing us to manage and run local LLMs without launching the graphical LM Studio application.

To explore the available LM Studio commands and their usage, run:

lms --help

This displays the available commands and options provided by the LM Studio CLI, which we can use to manage the daemon, models, and local inference from the terminal.

Since the larger Gemma 4 E2B variants were reported as "Won't Fit" on our Raspberry Pi, we will start with the smaller Gemma 3 1B model.

Use the following command to download the model:

lms get google/gemma-3-1b

LM Studio will prompt you to select an available model variant. For the Raspberry Pi, choose an appropriate quantized variant to keep memory usage low.

The Gemma 3 family is therefore a good fit for experimenting with local AI. Instead of requiring a dedicated GPU server, smaller variants can be deployed on more modest hardware.

For complete technical details and model specifications, refer to the official Google DeepMind Gemma 3 documentation:

Gemma 3 — Google DeepMind

Gemma 3 is a family of lightweight models capable of multimodal understanding, with unparalleled multilingual capabilities for more intelligent applications

deepmind.google

For this Raspberry Pi experiment, we will use the 1B variant and evaluate how it performs on the Raspberry Pi 5 with 8 GB of RAM.

Once the download is complete, the model will be available locally through LM Studio's headless daemon.

After downloading the model, we can use the following command to list all models currently available in LM Studio:

lms ls

This displays the models that have been downloaded and are available locally. It is useful for verifying that the Gemma 3 1B model was downloaded successfully before proceeding with the next step.

Once the model has been downloaded, use the lms load command to load it into memory:

lms load google/gemma-3-1b

This loads the Gemma 3 1B model into the LM Studio inference runtime, making it ready to accept prompts and serve inference requests.

You can check the available options for the command using:

lms load --help

After loading the model, we can use lms ps to check which models are currently loaded in memory:

lms ps

This command displays the models currently loaded by the LM Studio daemon, along with relevant information about their running state and resource usage.

It is a useful way to verify that Gemma 3 1B has been successfully loaded and is ready for inference.

Once the model is loaded, start the LM Studio server on port 5000 using:

lms server start --port 5000

This starts the local inference server and makes the loaded model available through an API on port 5000.

We can now use this endpoint to send requests to Gemma 3 1B from other applications or services on the network.

To verify that the LM Studio server is running and that the model is available through the API, use:

curl http://localhost:5000/v1/models

This sends a request to the OpenAI-compatible /v1/models endpoint. If the server is running correctly, it will return a JSON response containing the models currently available through the LM Studio server.

For example:

{
"object": "list",
"data": [
{
"id": "google/gemma-3-1b"
}
]
}

The exact response may contain additional fields depending on the LM Studio version and configuration.

Now that the API is working, let's send a simple question to Gemma using curl.
curl http://localhost:5000/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"google/gemma-3-1b","messages":[{"role":"user","content":"What is the difference between Docker and Kubernetes?"}]}'
output

LM Studio provides commands to check and control the API server directly from the terminal.

Check Server Status

To check whether the server is currently running:

lms server status
Start the Server

To start the server on port 5000:

lms server start --port 5000
Stop the Server

To stop the running server:

lms server stop

These commands make it easy to manage the LM Studio inference server without requiring the graphical LM Studio application.

By default, the LM Studio server may listen only on the local machine. To make the API accessible from other devices on the same network, start the server with the --host option:

lms server start --port 5000 --bind 0.0.0.0

For example:
curl http://192.168.1.100:5000/v1/models

This is particularly useful when running the Raspberry Pi as a dedicated local AI server, allowing other computers or applications on the LAN to communicate with Gemma.

Since this is a Raspberry Pi server, the cleanest approach is to create a systemd service. That will start llmster and the LM Studio server automatically after reboot.

Create a systemd service:
sudo nano /etc/systemd/system/lmstudio.service

Add the following:
`[Unit]
Description=LM Studio LLM Server
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
User=admin
RemainAfterExit=yes
ExecStart=/usr/bin/lms daemon up
ExecStart=/usr/bin/lms server start --port 5000 --bind 0.0.0.0
ExecStop=/usr/bin/lms server stop

[Install]
WantedBy=multi-user.target`

Note: In this example, admin is the Linux username used on my Raspberry Pi. If your system uses a different username, replace admin with your actual username and update the corresponding home directory path.

Finally, we configured a systemd service so that LM Studio can start automatically when the Raspberry Pi boots.
At this point, the Raspberry Pi is acting as a local AI server, with Gemma 3 1B running through LM Studio and exposing an OpenAI-compatible API over the local network.
In a future article, we will take this a step further and explore how to connect this local Gemma instance with VS Code and use the Raspberry Pi as a local coding assistant.