DEV Community

Cover image for Running an AI Agent Locally: ADK, Gemma 4, and Docker Model Runner
Google Developer Experts profile image Mazlum Tosun
Mazlum Tosun for Google Developer Experts

Posted on Originally published at Medium

Running an AI Agent Locally: ADK, Gemma 4, and Docker Model Runner

#ai #docker #llm #googlecloud

This article was originally published on Medium (Google Cloud Community).

Cloud LLMs are great for production. But during development, every API call has latency, costs money, and requires credentials. What if the LLM ran on your machine, right next to your agent?

In my previous article, I built a football statistics agent using Google ADK, BigQuery MCP via Cloud API Registry, and Gemini 2.5 Flash as the LLM — deployed to Cloud Run and Vertex AI Agent Engine.

In this article, I take the same agent and replace Gemini with Gemma 4 running locally via Docker Model Runner. The agent code barely changes. The BigQuery MCP tools stay the same. But now the LLM runs on my laptop — no cloud inference, no API key, no cost.

The full source code is available on GitHub.

Architecture Diagram

What is Docker Model Runner?

Docker Model Runner is a built-in feature of Docker Desktop that lets you pull and run LLMs locally — just like pulling container images. It exposes an OpenAI-compatible API on your machine, so any tool that speaks the OpenAI protocol can use it.

# Enable the Model Runner with TCP access
docker desktop enable model-runner --tcp 12434

# Pull the model
docker model pull ai/gemma4:E4B

# Verify
docker model ls
Enter fullscreen mode Exit fullscreen mode

Note: On Docker Engine (Linux), TCP access is enabled by default on port 12434 — no extra flag needed. On Docker Desktop (macOS/Windows), TCP must be explicitly enabled with --tcp 12434 to expose the API on localhost. In this article, I'm using Docker Desktop on macOS, which is why this activation step is required.

Once pulled, the model is available at http://localhost:12434/engines/v1 — the same endpoint format as the OpenAI API. You can test it directly:

curl http://localhost:12434/engines/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai/gemma4:E4B",
    "messages": [{"role": "user", "content": "Say hello"}],
    "max_tokens": 20
  }'
Enter fullscreen mode Exit fullscreen mode

No API key needed. No cloud dependency. The model runs entirely on your machine.

What is Gemma 4?

Gemma 4 is Google's latest open-weight model family, available on Docker Hub in several sizes:

Variant On-disk size* Raw params Use case
ai/gemma4:E2B 2.94 GiB 4.65B (2B effective) Fastest, edge-friendly — great for rapid iteration
ai/gemma4:E4B (= latest) 4.74 GiB 7.52B (4B effective) Best trade-off for local agents — current default
ai/gemma4:4B ~6 GB 4B dense (Q4_K_XL quant) Traditional dense variant, quantized
ai/gemma4:26B ~18 GB 26B MoE (~4B active) High quality, sparse mixture-of-experts
ai/gemma4:31B ~20 GB 31B dense Maximum quality, heavy on resources

* Sizes reported by docker model ls after pull. The Docker Hub manifest shows slightly larger figures (e.g. E4B = 6.09 GB on Hub) because the GGUF is repackaged locally.

The E2B and E4B variants use the Per-Layer Embeddings (PLE) architecture — they have more actual parameters than their "effective" size, but run at the compute cost of the smaller effective size. It's the same Matryoshka-style approach introduced with Gemma 3n.

For an agent that makes 3-4 LLM roundtrips per question (reasoning, tool call, error handling, final answer), inference speed matters. The E4B variant gives the best trade-off between quality and speed for local development — it's also now the latest default.

Architecture: what changes, what stays

The beauty of this approach is how little changes. The agent framework (ADK), the tools (BigQuery MCP via Cloud API Registry), and the system instructions are identical to the cloud version. Only the model endpoint changes.

User Question
     │
     ▼
┌──────────────┐     ┌──────────────────────┐
│  Google ADK  │────▶│  Docker Model Runner │
│  LlmAgent    │◀────│  Gemma 4 (local)     │
└──────┬───────┘     └──────────────────────┘
       │
       ▼
┌──────────────────┐     ┌───────────┐
│ Cloud API Registry│────▶│ BigQuery  │
│ BigQuery MCP      │◀────│ (cloud)   │
└──────────────────┘     └───────────┘
Enter fullscreen mode Exit fullscreen mode
Cloud version (previous article) Local version (this article)
LLM Gemini 2.5 Flash (Vertex AI) Gemma 4 E4B (Docker Model Runner)
Inference cost Per-token pricing Free (runs on your CPU/GPU)
Agent code LlmAgent + API Registry MCP Same
Tools BigQuery MCP Same
System instruction Full schema + business rules Same

For the details on the ADK agent setup, BigQuery MCP via Cloud API Registry, the dataset, and the system instruction, refer to my previous article. Here, I'll focus on what's new: the Docker Model Runner integration and the challenges of running Gemma locally with an agent framework.

The agent code: one model string change

Here's the original agent.py from the cloud version:

MODEL = "gemini-2.5-flash"

root_agent = LlmAgent(
    model=MODEL,
    name="football_stats_agent",
    instruction=SYSTEM_INSTRUCTION,
    tools=[toolset],
)
Enter fullscreen mode Exit fullscreen mode

And the local version:

GEMMA_MODEL = os.environ.get("GEMMA_MODEL", "ai/gemma4:E4B")
MODEL = f"openai/{GEMMA_MODEL}"

root_agent = LlmAgent(
    model=MODEL,
    name="football_stats_agent",
    instruction=SYSTEM_INSTRUCTION,
    tools=[toolset],
)
Enter fullscreen mode Exit fullscreen mode

The openai/ prefix tells ADK to route through LiteLLM (included in google-adk[extensions]) to the OpenAI-compatible endpoint — which is Docker Model Runner. The OPENAI_API_BASE environment variable points LiteLLM to the local endpoint:

export OPENAI_API_BASE=http://localhost:12434/engines/v1
export OPENAI_API_KEY=docker-model-runner  # Dummy key, required by LiteLLM
export GEMMA_MODEL=ai/gemma4:E4B
Enter fullscreen mode Exit fullscreen mode

That's it for the model swap. But there's a catch.

The function calling problem

When I first ran the agent with Gemma, it crashed:

Conversation roles must alternate user/assistant/user/assistant/...
Enter fullscreen mode Exit fullscreen mode

The issue: Gemma doesn't have native function calling support. When ADK sends tool-call messages (system → user → tool_result → assistant), Gemma's chat template rejects the non-alternating roles.

Gemini handles this natively — it understands function declarations and returns structured functionCall responses. Gemma needs a different approach: convert tool declarations into text prompts that the model can understand, then extract function calls from the model's text output.

ADK already has this built-in via GemmaFunctionCallingMixin. It also has a Gemma3Ollama class that combines this mixin with the Ollama provider. But Docker Model Runner uses the OpenAI API (not Ollama's /api/chat), so I needed a custom class.

GemmaModelRunner: bridging Gemma and Docker Model Runner

The solution is a custom LLM class that combines:

  • LiteLLM's OpenAI provider — routes requests to Docker Model Runner's /engines/v1/chat/completions
  • ADK's GemmaFunctionCallingMixin — converts tool declarations to text and extracts function calls from responses 
from google.adk.models.gemma_llm import GemmaFunctionCallingMixin
from google.adk.models.lite_llm import LiteLlm
from google.adk.models.registry import LLMRegistry


class GemmaModelRunner(GemmaFunctionCallingMixin, LiteLlm):
    """Gemma model served by Docker Model Runner."""

    @classmethod
    def supported_models(cls) -> list[str]:
        return [r"openai/ai/gemma.*"]

    async def generate_content_async(self, llm_request, stream=False):
        self._move_function_calls_into_system_instruction(llm_request)

        async for response in super().generate_content_async(llm_request, stream):
            self._extract_function_calls_from_response(response)
            yield response


LLMRegistry.register(GemmaModelRunner)
Enter fullscreen mode Exit fullscreen mode

This class:

  1. Registers itself in ADK's LLM registry for model names matching openai/ai/gemma.*
  2. Before each LLM call: moves function declarations from the tool config into the system instruction as text, and converts tool-result messages into user messages
  3. After each LLM call: parses the model's text output to extract any function calls (e.g., when Gemma writes execute_sql(projectId="...", query="...") as text)

The import in agent.py triggers the registration:

import football_stats_agent.gemma_model_runner  # noqa: F401 — registers GemmaModelRunner
Enter fullscreen mode Exit fullscreen mode

Dependencies

The cloud version only needs google-adk. The local version needs google-adk[extensions] to include LiteLLM:

[project]
name = "football-agent-adk-gemma"
version = "0.1.0"
requires-python = ">=3.13"
dependencies = [
    "google-adk[extensions]>=1.27.2",
    "google-cloud-aiplatform>=1.141.0",
]
Enter fullscreen mode Exit fullscreen mode

Running the agent locally

With ADK web UI 

# Pull the model
docker desktop enable model-runner --tcp 12434
docker model pull ai/gemma4:E4B

# Install dependencies
direnv allow
uv sync

# Authenticate with GCP (for BigQuery MCP)
gcloud auth application-default login

# Run
uv run adk web
Enter fullscreen mode Exit fullscreen mode

Open http://localhost:8000, select football_stats_agent, and ask a question.

With Docker Compose 

services:
  adk-agent:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "8080:8080"
    environment:
      - GCP_PROJECT_ID=gb-poc-373711
      - GEMMA_MODEL=ai/gemma4:E4B
      - OPENAI_API_BASE=http://host.docker.internal:12434/engines/v1
      - OPENAI_API_KEY=docker-model-runner
      - GOOGLE_APPLICATION_CREDENTIALS=/tmp/adc.json
    volumes:
      - ${HOME}/.config/gcloud/application_default_credentials.json:/tmp/adc.json:ro
Enter fullscreen mode Exit fullscreen mode

Note host.docker.internal — from inside a container, this is how you reach Docker Model Runner running on the host.

docker compose up --build
Enter fullscreen mode Exit fullscreen mode

The agent in action

I ran three test queries against the E4B agent to see how it handles different difficulty levels:

Simple query — top 3 scorers from France (~22s, one roundtrip)

When you ask "List the top 3 scorers from France", Gemma 4 E4B gets it right on the first try:

  1. Reasons about the question and picks the execute_sql_readonly tool
  2. Generates correct SQL directly using SAFE_CAST (the system instruction warns about string-typed numeric columns)
  3. BigQuery returns results — Kylian Mbappe (8), Olivier Giroud (4), Aurelien Tchouaméni (1)
  4. Formats the answer in natural language

No self-correction needed — E4B follows the system prompt's SAFE_CAST guidance upfront. A smaller model like E2B typically needs a failed attempt before switching to SAFE_CAST.

Complex query — performance rating comparison (~2.5 min, three roundtrips)

When you ask "Compare France and Argentina players by performance rating. Top 3 from each team", things get more interesting:

  1. First attempt — generates a complex window-function SQL, BigQuery rejects it with a syntax error
  2. Second attempt — fixes the syntax but fails on Bad int64 value: "-" (some numeric string columns contain "-")
  3. Third attempt — wraps every cast in SAFE_CAST, query succeeds
  4. Formats the answer with the performance rating formula applied correctly

The self-correction behavior is the same pattern the original Gemini cloud version produced — read the BigQuery error, adjust the SQL, retry. That Gemma 4 E4B handles this loop reliably on a local ~8B-raw-param model is a strong quality signal.

Choosing the right model variant

Hardware used for these benchmarks

All timings below were measured on my personal laptop:

  • MacBook Pro — Apple M1 Max (10 CPU cores: 8 performance + 2 efficiency)
  • 32-core GPU, Metal 3 backend
  • 32 GB unified memory
  • macOS 15.3.1

The M1 Max's unified memory architecture is a big part of why Gemma 4 runs comfortably here — the GPU and CPU share the same 32 GB pool, so the model weights don't need to be copied across a PCIe bus. With 32 GB total, E2B (2.94 GiB) and E4B (4.74 GiB) both leave plenty of headroom for Docker Desktop, the ADK runtime, the browser, and the rest of your dev environment. The larger 26B / 31B variants are technically loadable but you'll feel the squeeze.

Performance will vary significantly on other hardware (base Apple Silicon, Intel Macs, Linux with or without a discrete GPU). I haven't benchmarked those, so treat the timings above as one data point on one specific machine.

Timings

Full agent loop end-to-end including BigQuery roundtrips:

Query type Roundtrips E4B elapsed
Simple (e.g. "top 3 scorers") 1 ~22s
Complex (e.g. "compare two teams by performance rating") 3 (with self-correction) ~2.5 min

Simple queries feel snappy. Complex queries requiring multiple SQL self-corrections take the time you'd expect from a local model making several inference passes.

The E4B variant (now the latest default) is the sweet spot: ~8B raw params running at the compute cost of a 4B model thanks to PLE. It generates correct SQL on the first try for straightforward questions and self-corrects cleanly on complex ones.

If you want maximum speed at the cost of some quality, drop down to E2B — smaller, roughly 2x faster per inference, but more likely to need an extra self-correction round before getting the SQL right.

If you need maximum quality, use 26B / 31B locally (expect multi-minute loops even on a beefy machine) — or switch back to Gemini for production, which is what I recommend:

Environment Model Why
Local dev (default) Gemma 4 E4B (Docker Model Runner) Free, no API key, good quality
Local dev (fastest) Gemma 4 E2B (Docker Model Runner) 2x faster, lower quality
Production Gemini 2.5 Flash (Vertex AI) Higher quality, faster, scalable

The agent code is the same — only the GEMMA_MODEL environment variable (or the model ID for Gemini) changes.

Docker Model Runner version matters

One issue I hit: an older Docker Model Runner version bundled with my Docker Desktop install cannot load Gemma 4 models at all. Every attempt fails with:

unable to load runner: error waiting for runner to be ready:
inference backend took too long to initialize
Enter fullscreen mode Exit fullscreen mode

This looks like a memory or timeout issue, but it's actually an architecture incompatibility. Gemma 4 uses a newer model architecture that older versions of the llama.cpp backend don't support.

The fix: update Docker Desktop to get a recent Model Runner release. You can check your version with:

docker model version
Enter fullscreen mode Exit fullscreen mode

Key takeaways

  1. Docker Model Runner makes local LLMs trivial — docker model pull and you have an OpenAI-compatible API on localhost. No Ollama, no manual setup.

  2. Gemma doesn't do native function calling — You need ADK's GemmaFunctionCallingMixin to convert tool declarations into text prompts. The custom GemmaModelRunner class bridges this gap for Docker Model Runner's OpenAI API.

  3. The agent code barely changes — Same ADK framework, same BigQuery MCP tools, same system instruction. Only the model string and one custom class change.

  4. Pick the right Gemma 4 variant for local agents — 26B / 31B are too slow for multi-roundtrip loops. E4B (the latest default) is the sweet spot: ~22s for simple queries, ~2.5 min for complex ones requiring SQL self-correction. Drop to E2B only if you need maximum speed. Both use PLE so they punch above their effective size.

  5. Keep Docker Desktop updated — Older Model Runner versions can't load newer model architectures. The error message is misleading — it's not a timeout, it's an incompatibility.

  6. Local for dev, cloud for prod — Run Gemma 4 locally during development (free, private, fast feedback). Switch to Gemini on Vertex AI for production (better quality, scalable). The swap is one environment variable.

What's next

In the next article, I'll rebuild the same use case — football statistics agent on the Qatar 2022 World Cup dataset — but replace Google ADK with Docker Agent. Same Gemma 4 via Docker Model Runner, same BigQuery MCP tools, same system instructions — only the agent framework changes. A direct side-by-side comparison of the two approaches.

Try it yourself

The full source code is available on GitHub. Clone it, pull the model, and ask the agent about the 2022 World Cup:

  • "Who scored the most goals?"
  • "Show me the top 5 players by assists"
  • "Which goalkeepers had the highest save percentage?"
  • "Compare France and Argentina players by performance rating"

If you enjoyed this article, follow me for more content on AI agents, Google Cloud, Software, DevOps, Tech and data engineering:

Top comments (0)