# Building Production-Ready Open-Weights AI Agents with Gemma, Google ADK, and Cloud Run GPUs
## The Premise
Most "production-ready AI agent" tutorials are corporate theater. You've got an LLM, some async glue code, and a deployment config file. You call it shipshape. Meanwhile, your agent blows past 8GB RAM on its first real query, deadlocks on a race condition nobody tested, and chokes under anything heavier than ten concurrent users. This guide exists to fix that.
We're covering **Gemma** (Google's open-weights family, the model, not a hardware board), **Google ADK** (Agent Development Kit for structuring agentic workflows), and **Cloud Run with GPU backing**. The goal isn't to impress at demo day. It's to build something that doesn't collapse when a real user touches it.
## Hardware Constraints: Stop Treating 8GB Like a Suggestion
The 8GB RAM ceiling isn't a suggestion. It's a hard boundary imposed by your GPU container spec on Cloud Run. Exceed it and the orchestrator kills your instance with no ceremony.
Here's what actually matters:
- **Quantize Gemma to 4-bit or 8-bit**. The `gguf` format is your friend. A full 7B-parameter float16 Gemma model eats roughly 14GB. A 4-bit quant compresses it to ~4GB, leaving headroom for the runtime, the KV cache, and your agent logic.
- **Stream your context, don't hoard it**. Bounded queues for incoming prompts and outgoing completions keep memory predictable. Anything that grows unbounded will eventually OOM-kill you.
- **Profile before you ship**. Tools like `memory_profiler` and `tracemalloc` aren't optional. Deploying without them means you're guessing.
## Race Conditions: The Silent Instance Killer
Multi-threaded agent loops are where your system quietly corrupts itself. One thread writes state, another reads it, and suddenly your agent generates a response based on a half-updated context window. The fix isn't theoretical. It's mechanical:
python
import threading
import queue
from typing import Optional
Thread-safe bounded queue for conversation history
class BoundedContextQueue:
def init(self, max_tokens: int = 8192):
self._lock = threading.Lock()
self._queue: queue.Queue[str] = queue.Queue(maxsize=50)
self._token_count = 0
self._max_tokens = max_tokens
def add_turn(self, role: str, content: str) -> bool:
"""Add a message turn with token budget enforcement."""
estimated_tokens = len(content) // 4
with self._lock:
if self._token_count + estimated_tokens > self._max_tokens:
# Evict oldest turns to make room (circular buffer strategy)
self._evict_oldest(estimated_tokens)
self._queue.put(f"{role}: {content}")
self._token_count += estimated_tokens
return True
def _evict_oldest(self, needed_tokens: int) -> None:
"""Remove oldest entries until enough space exists."""
while self._token_count > needed_tokens and not self._queue.empty():
old = self._queue.get_nowait()
self._token_count -= len(old) // 4
def get_all(self) -> list[str]:
with self._lock:
items = list(self._queue.queue)
return items
The key principles:
- **Use locks around shared mutable state**. Every time two threads touch the same dictionary or queue, wrap it in `threading.Lock()`. No exceptions.
- **Prefer queues over shared variables**. Python's `queue.Queue` with a `maxsize` limit gives you thread safety and bounded memory in one abstraction.
- **Make critical sections atomic**. If you're reading a config, mutating it, and writing it back, do it in a single locked block. Any gap between read and write is a race condition waiting to fire.
## Development Process: Modular or Regret
A monolithic agent script isn't just bad practice. It's a deployment hazard. Break it down into four isolated modules:
1. **Model loader**: Handles quantization, GPU memory allocation, and warmup. Isolate it so you can swap models without touching the rest of the codebase.
2. **Context manager**: Manages conversation history, bounded by a token limit. This is where your queue lives.
3. **Tool executor**: Each tool (search, API call, code interpreter) runs in its own function with explicit error boundaries.
4. **Orchestrator**: Runs the agent loop. It shouldn't know about quantization, threading, or deployment internals.
python
orchestrator.py
The agent loop. Pure logic, zero infrastructure concerns.
class AgentOrchestrator:
def init(
self,
model: GGMLModel,
context: BoundedContextQueue,
tools: dict[str, Callable],
):
self._model = model
self._context = context
self._tools = tools
def run(self, user_input: str) -> str:
self._context.add_turn("user", user_input)
for step in range(5): # Max 5 reasoning steps
prompt = self._build_prompt()
response = self._model.generate(prompt, max_new_tokens=512)
if tool_call := self._extract_tool_call(response):
result = self._execute_tool(tool_call["name"], tool_call["args"])
self._context.add_turn("assistant", f"[Tool: {result}]")
continue
self._context.add_turn("assistant", response)
return response
return "[MAX STEPS REACHED]"
def _build_prompt(self) -> str:
messages = self._context.get_all()
return "\n".join(messages)
def _execute_tool(self, name: str, args: dict) -> str:
if name not in self._tools:
return f"Unknown tool: {name}"
try:
return self._tools[name](**args)
except Exception as e:
return f"Tool error: {e}"
If your codebase doesn't follow this pattern, you'll spend more time untangling it than shipping.
## Deployment on Cloud Run: What Actually Works
Cloud Run with GPUs is powerful and punishing in equal measure. Cold starts on GPU instances are measured in minutes, not seconds. You need a strategy:
- **Enable minimum instances** (at least 1) to avoid cold starts during traffic. Yes, it costs more. Yes, it's cheaper than a degraded user experience.
- **Set memory to 8GB and CPU to 2-4 cores**. Match your container spec to your quantized model size. Overprovisioning burns money. Underprovisioning triggers OOM kills.
- **Implement health checks that validate GPU memory**. A passing `/healthz` shouldn't just say the server is up. It should confirm the model loaded and GPU memory is available.
- **Cache aggressively**. Your first inference is always slow. Subsequent ones with similar contexts benefit from KV cache reuse.
yaml
cloudrun.yaml
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: gemma-agent
spec:
template:
metadata:
annotations:
autoscaling.knative.dev/minScale: "1"
autoscaling.knative.dev/maxScale: "5"
spec:
timeoutSeconds: 300
serviceAccountName: agent-sa
containers:
- image: gcr.io/project/gemma-agent:latest
resources:
limits:
nvidia.com/gpu: 1
memory: "8Gi"
cpu: "4"
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 120
periodSeconds: 30
readinessProbe:
httpGet:
path: /readyz
port: 8080
initialDelaySeconds: 120
The 120-second initial delay accounts for GPU model loading. Without it, Kubernetes restarts your container before it ever becomes ready, creating an endless crash loop.
## The Bottom Line
Building a production-ready open-weights agent isn't about stacking frameworks. It's about respecting constraints. Memory, concurrency, cold starts, and designing around them. Gemma handles the model. Google ADK structures the agent loop. Cloud Run GPUs provide the compute. Your job is making sure they don't destroy each other.
Stop writing agents that work on your machine and die in production. Ship something that survives.
---
**Open Loop:** When you've got a single GPU instance serving bursty agent traffic, how do you decide between scaling horizontally with more minimal instances versus vertically with a larger, more expensive GPU pod. Where do you draw the line between cost efficiency and latency guarantees?
Top comments (0)