DEV Community: Karl Weinmeister

How to Use the Gemini Deep Research API in Production

Karl Weinmeister — Wed, 04 Mar 2026 16:08:05 +0000

How many of us have gone down the research rabbit hole? Way too many tabs, links, and notes in the pursuit of knowledge? It’s all useful stuff, but time-consuming and distracting.

Since I discovered the Gemini Deep Research Agent, I haven’t turned back. And best of all, it has a powerful and straightforward API to kick off research programmatically. Let’s explore how to use it, and the patterns for including this in a production architecture.

Async changes everything

A single research task can trigger dozens of search queries and take several minutes to complete. The asynchronous Interactions API provides a polling-based interface with a required background=True parameter to check on progress.

If you’ve ever worked with a Pub/Sub pipeline or job queue, this will feel familiar.

Meet the Interactions API

The Interactions API is a newer, unified interface for working with Gemini models and agents. It replaces the older generateContent pattern for scenarios that need state management, tool orchestration, or background execution.

You create an interaction, point it at the deep research agent, and tell it to run in the background:

from google import genai

client = genai.Client(api_key=GEMINI_API_KEY)

# Launch the research agent in the background
interaction = client.interactions.create(
    input="Research the history and future of Solid State Batteries.",
    agent='deep-research-pro-preview-12-2025',
    background=True
)

That call returns immediately with an interaction ID. The agent is now off doing its thing, autonomously planning search queries, reading pages, and iterating on its analysis. Your application is free to do whatever it needs to do in the meantime.

Polling for results

Now you need a way to check whether the agent has finished. The status field tells you everything you need to know:

while True:
    interaction = client.interactions.get(interaction.id)

    if interaction.status == "completed":
        # The full research report is ready
        print(interaction.outputs[-1].text)
        break
    elif interaction.status == "failed":
        print(f"Research failed: {interaction.error}")
        break

    # Still working. Check again in 10 seconds.
    time.sleep(10)

Taking it to production with Cloud Run

In a notebook, a while True loop gets the job done. In production, you want something that scales, recovers from failures, and doesn’t burn compute waiting. Google Cloud offers three Cloud Run compute models that each map to a different integration pattern with the Deep Research agent.

Cloud Run service: webhook-triggered research

A Cloud Run service works when you want to trigger research from an HTTP request. The service accepts the request, kicks off the agent, stores the interaction ID, and returns immediately. A separate mechanism (a Cloud Scheduler cron, a Cloud Workflow, or a callback) handles checking the results later.

from fastapi import FastAPI
from pydantic import BaseModel
from google import genai

app = FastAPI()
client = genai.Client()

class ResearchRequest(BaseModel):
    topic: str

@app.post("/research")
async def start_research(req: ResearchRequest):
    interaction = client.interactions.create(
        input=req.topic,
        agent="deep-research-pro-preview-12-2025",
        background=True,
    )

    # Store the ID for later retrieval (e.g., in Firestore or Cloud SQL)
    save_interaction_id(interaction.id, req.topic)

    return {"interaction_id": interaction.id, "status": "started"}

Cloud Run job: batch research tasks

A Cloud Run job is a natural fit for one-shot or scheduled research. Jobs execute code and stop, which maps cleanly to “launch, poll, write, exit.” If you have a batch of research topics, you can fan them out as parallel job tasks.

from google import genai
from google.cloud import storage

client = genai.Client()

def run_research_job():
    topic = os.environ.get("RESEARCH_TOPIC", "Default research topic")

    interaction = client.interactions.create(
        input=topic,
        agent="deep-research-pro-preview-12-2025",
        background=True,
    )

    # Poll until done
    while True:
        result = client.interactions.get(interaction.id)
        if result.status == "completed":
            # Write the report to Cloud Storage and exit
            bucket = storage.Client().bucket("my-research-reports")
            bucket.blob(f"{interaction.id}.md").upload_from_string(
                result.outputs[-1].text
            )
            return
        elif result.status == "failed":
            raise RuntimeError(f"Research failed: {result.error}")
        time.sleep(10)

run_research_job()

Cloud Run worker pool: continuous research dispatcher

The most interesting option for a production pipeline is a Cloud Run worker pool. Worker pools are designed for continuous, non-HTTP, pull-based background processing. They don’t need a public endpoint, they don’t autoscale by default (you bring your own logic), and they cost up to 40% less than instance-billed services.

If you’re building a system that continuously pulls research requests from a Pub/Sub subscription, dispatches them to the agent, and writes completed reports to storage, a worker pool is purpose-built for that pattern.

from google import genai
from google.cloud import pubsub_v1, storage

client = genai.Client()
subscriber = pubsub_v1.SubscriberClient()
subscription_path = "projects/my-project/subscriptions/research-requests"

def handle_message(message):
    topic = message.data.decode("utf-8")

    interaction = client.interactions.create(
        input=topic,
        agent="deep-research-pro-preview-12-2025",
        background=True,
    )

    # Poll until done, then write results
    while True:
        result = client.interactions.get(interaction.id)
        if result.status == "completed":
            bucket = storage.Client().bucket("my-research-reports")
            bucket.blob(f"{interaction.id}.md").upload_from_string(
                result.outputs[-1].text
            )
            message.ack()
            return
        elif result.status == "failed":
            message.nack() # Retry later
            return
        time.sleep(10)

# Pull messages continuously (worker pool stays alive)
streaming_pull = subscriber.subscribe(subscription_path, callback=handle_message)
streaming_pull.result()

Grounding with your own data

Web research is powerful, but sometimes you need the agent to work with private data or internal documents. The Deep Research agent supports a file search tool for exactly this. Think of it as RAG, but orchestrated automatically by the agent rather than wired up manually.

interaction = client.interactions.create(
    input="Compare our 2025 fiscal year report against current public web news.",
    agent='deep-research-pro-preview-12-2025',
    background=True,
    tools=[{
        "type": "file_search",
        "file_search_store_names": [FILE_SEARCH_STORE_NAME]
    }]
)

This is where the architecture gets interesting for enterprise use cases. The agent can combine internet research with grounded analysis of your internal documents, all within a single research task.

Stateful follow-ups

After a research task completes, you can ask follow-up questions that reference the original research context without re-running the entire workflow:

follow_up = client.interactions.create(
    input="Can you elaborate on the key findings?",
    model="gemini-3.1-pro-preview",
    previous_interaction_id=interaction.id
)

print(follow_up.outputs[-1].text)

Getting started

This Deep Research notebook walks you through the entire flow, from setting up the client to launching research tasks. For pricing details, check the Gemini API pricing page.

Ready to stop Googling and start delegating? Grab the notebook and run your first deep research task. I’d love to hear what you build with it. Come find me on LinkedIn, X, or Bluesky and share what research tasks you’re automating.

Skills Made Easy with Google Antigravity and Gemini CLI

Karl Weinmeister — Thu, 26 Feb 2026 16:52:06 +0000

When you ask an AI assistant a question, you have two choices: hope its training is current, or burn through tokens reading documentation. What if you could give your agent the right answer, right away?

That’s the power of Agent Skills. Skills are reusable packages of knowledge that extend what your agent can do without overwhelming its context window. Defined with a SKILL.md file, they allow you to teach your agent how to accomplish tasks consistently. Instead of forcing an agent to process an entire library’s worth of documentation at once, Skills act as on-demand expertise.

You can learn more about the open standard at Agent Skills and discover community capabilities at skills.sh.

In this post, we’ll explore how to manage these skills in the Gemini CLI, a powerful terminal-native AI assistant, and Antigravity, an advanced agentic coding assistant.

Installing skills

Both the Gemini CLI and Antigravity access skills by reading them from standard directories on your local machine. To add new skills, you can drop them into these locations:

Managing Skills in Gemini CLI

Gemini CLI offers built-in skill management. You can use either interactive slash commands during a session, or terminal commands:

These commands makes it easy to pull in skills from a Git repository or local directory, and manage whether they are active for your current project.

For example, if you want to install a specific skill located inside a subdirectory of a larger repository (like Firebase’s firebase-ai-logic-basics), you can use the --path flag:

gemini skills install https://github.com/firebase/agent-skills.git — path skills/firebase-ai-logic-basics

To audit which skills are currently loaded into your agent’s context, you can run:

gemini skills list

This command provides a clear overview of all discovered skills across your workspace and global environments, showing their descriptions and file locations so you know exactly what expertise your agent has access to.

Unified management with the skills tool

While Gemini CLI has robust built-in tools, what if you want to manage skills across both Gemini CLI and Antigravity simultaneously? Managing them by hand across the different ~/.gemini/skills/ and ~/.gemini/antigravity/skills/ directories can get tedious.

That’s where the open-source CLI tool from vercel-labs/skills shines. It uses a symlink approach to easily install, update, and remove skills centrally, sharing them across multiple agents without duplicating files.

Getting Started with skills

The easiest way to begin with the unified CLI is by using the add command. You can add the -a or --agent parameter for each client you’d like to add the skill to.

For example, suppose you want to equip your agent with deep knowledge of Firebase to help build full-stack apps. You could run:

npx skills add firebase/agent-skills -a gemini-cli -a antigravity

⚠️ Note that the skill will be added to the Gemini CLI even without the -a parameter, as it supports the default ~/.agents/skills global directory. The extra parameter provided here for clarity to show both clients in one command.

This installs the skill and instantly makes it available to both Gemini and Antigravity. By adding firebase/agent-skills, your agents can reliably build and deploy apps with Firebase Auth, Firestore, and more. For more details on how this skill works, read Introducing Agent Skills for Firebase.

If you’re looking for skills related to a specific technology, you can search for them directly from your terminal. For instance, if you’re building a mobile app, you might want to find capabilities related to Flutter. You can use the find command to discover relevant skills:

npx skills find flutter

This command searches the community skills registry and returns a list of matching capabilities, displaying the most popular ones first alongside their installation commands. You can quickly copy those commands to add the expertise directly to your active agents.

Keeping your agent’s context clean

It’s easy to get excited and install dozens of skills. While progressive disclosure means your agent isn’t reading the entire instruction manual for every skill on every prompt, simply loading the names, descriptions, and metadata of 50 different skills can still clutter the initial context window, leading to confusion or degraded performance.

To keep your agents focused and efficient, make sure to keep your essential skills up-to-date with your chosen tool’s update commands. More importantly, if you find you aren’t using a skill anymore, take a moment to disable or remove it (e.g., /skills disable <name> in Gemini CLI or npx skills remove <name>) to free up that precious context space.

By managing skills in Gemini CLI and Antigravity with the skills CLI, you can tailor and organize your environment to your liking. To get more hands-on experience building skills, you can try out the Agent Skills codelab.

Have you built any interesting workflows using Agent Skills? I’d love to hear how you’re extending your agents. Share what you’ve built with me on LinkedIn, X, or Bluesky!

Performance shouldn’t be an afterthought: Hardening the AI-Assisted SDLC

Karl Weinmeister — Mon, 26 Jan 2026 17:31:22 +0000

Performance shouldn’t be an afterthought: Hardening the AI-assisted SDLC

It’s amazing how quickly you can now build a working application with AI assistance. It’s even more amazing how easily you can harden your application for production. But that’s a step that’s often left out of the “vibe coding” software development lifecycle, or SDLC. I hope to change that.

Why does it matter? The impact of high latency is lost users, and the impact of excess memory usage is lost budget.

Study after study shows that your application’s latency directly correlates with user satisfaction, a key ingredient for business success. Meanwhile, your application’s memory usage impacts your Cloud infrastructure cost. For example, Cloud Run offers memory limits at various tiers ranging from 512 MiB to 32 GiB. Not to mention, if you underprovision memory, your application reliability will suffer.

In this post, I’ll walk through steps I recommend that ensure your application is hardened for production. I’ll use Google Antigravity to build an application with sample application code available on GitHub.

Discovery and Tool Selection

If you aren’t an expert in the tooling ecosystem for your application’s language, use AI to bridge the gap. Avoid guessing and ask for industry standards. For example, you can ask:

“I need to profile a Python application for both CPU execution time and memory leaks. What are the most modern, low-overhead tools available? I know about cProfile, but are there better options with visualization (like flame graphs)?”

What modern stack might your AI assistant suggest? scalene is a high-performance profiler whose standout capability is separating time spent in Python versus native code. To dig into memory details, memray can track allocations in native extensions and generate flame graphs that make it easy to spot areas for improvement. Finally, pytest-benchmark is a useful plugin that handles warm-up rounds and statistical analysis automatically.

If you’re writing code in other languages, the same strategy applies. You might discover pprof for Go, clinic.js for Node.js, and other useful tools.

Establish a Baseline

My use case is calculating the perplexity of a given text, which is helpful for AI detection and other use cases. The initial implementation started with a naïve algorithm which processes one token at a time, which isn’t uncommon when you simply ask for a solution.

for i in range(seq_len - 1):
    current_token = input_ids_int64[i]

    # 1. Construct single-token input
    inputs = { "input_ids": np.array([[current_token]]) }
    inputs.update(past_key_values)

    # 2. Run inference for just this token
    outputs = session.run(None, inputs)

Optimize for Speed

While this code works, it’s slow. With our tools selected from the research phase, we can ask our AI agent to benchmark the baseline code.

“Generate a Python script using pytest-benchmark to benchmark my perplexity function against a baseline. Create a mock dataset to simulate load.”

Once we have a benchmark, we can then ask our AI agent to optimize it:

“Profile this baseline code and suggest an optimized routine. Focus on throughput.”

A standard engineering strategy to address loop overhead is vectorization. The revised approach feeds the entire sequence to the model in one go:

def calculate_perplexity_batch(context, text):
    # 1. Encode entire text at once
    input_ids = tokenizer.encode(text)

    # 2. Single inference call for the whole sequence
    outputs = session.run(None, inputs)
    logits = outputs[0] # Shape: [1, SeqLen, Vocab]

    # 3. Vectorized loss calculation (No loops)
    # ... numpy vector operations ...
    return float(np.exp(mean_nll))

In my test environment, this change led to an overall 2.5x speed improvement over the naïve loop.

Optimize Memory Usage

Unfortunately, this speed came at a cost. By loading all logits for the entire sequence into memory at once, I created an unbounded memory situation. Long documents would cause peak memory usage to spike uncontrollably. I had solved for latency, but in doing so, I had broken cost constraints.

How could I prompt Antigravity to help?

“Analyze my optimized perplexity routine. The target environment is Google Cloud Run with a strict 2GB memory limit. Identify the peak memory usage and refactor the code to stay under this limit without reverting to the slow loop.”

The solution balanced speed and memory, processing data in batches large enough to achieve high throughput but small enough to manage peak memory:

chunk_size = 128
logits_list = []

for i in range(len(input_ids) - 1):
    append_tokens(input_ids[i : i + 1])
    logits_list.append(get_logits()[0, 0, :])

    if len(logits_list) >= chunk_size:
        # Process this chunk
        _process_logits_chunk(logits_list, targets_list)

        # Free memory immediately to clip the peak
        logits_list = []

Final Thoughts

Before unleashing this process across your codebase, let’s be clear that performance engineering is a rigorous discipline that goes beyond optimizing functions. Industry veteran Brendan Gregg famously warns against the Streetlight Anti-Method: looking for performance problems where it’s easiest, rather than where the problems actually exist.

Providing your AI assistant the broader context of your application is key, and it’s easy to overlook important details in your prompting. An AI assistant doesn’t know that your production workload is 10 million rows, not the 100 rows in your test script. It can’t see that your database is missing an index or that your network bandwidth is saturated. Most importantly, an AI assistant doesn’t know your intent. If you steer it towards speeding up a query, it will focus on what you asked for, but it likely won’t ask why that data isn’t cached in the first place.

With those considerations in mind, using AI as a final check is a low-risk, high-reward step. It takes minutes and often catches low-hanging fruit that is overlooked. Then, the next step is maintaining your application’s performance. Consider leveraging tools for continuous application monitoring to identify regressions and ensure reliability in a live environment.

I’d love to hear how you’re innovating with your software development lifecycle. Connect with me on LinkedIn, X, or Bluesky!

AI Agent Engineering in Go with the Google ADK

Karl Weinmeister — Tue, 20 Jan 2026 16:41:44 +0000

While Python remains popular for model training and research, the requirements for serving and orchestrating AI agents align closely with Go’s strengths: low latency, high concurrency, and type safety.

Transitioning from a prototype to a production agent introduces engineering challenges that Golang can handle exceptionally well. Go’s static typing eliminates runtime errors when parsing structured LLM outputs. Its lightweight goroutines, which start with just a few kilobytes of stack memory, allow agents to handle thousands of concurrent tool executions without the overhead of heavy thread management.

In recent years, Go’s adoption for cloud-native microservices has surged: it showed the fourth-highest promise for languages, and maintained a 93% satisfaction rate. Google’s Agent Development Kit, or ADK, bridges the gap between these architectural advantages and generative AI.

In this guide, I’ll walk through scaffolding a new project and deploying it as a secure microservice on Google Cloud.

Get Started with the Agent Starter Pack

The good news is you don’t need to start from scratch. The Agent Starter Pack is a CLI tool that scaffolds a production-ready folder structure, including CI/CD pipelines, infrastructure configuration, and boilerplate code.

To get started, just run the create command with uvx:

uvx agent-starter-pack create

The CLI guides you through an interactive setup. For this project, I selected:

Project Name: my-first-go-agent
Template: Option 6 (Go ADK, Simple ReAct agent)
CI/CD: Option 3 (GitHub Actions)
Region: us-central1

Agent Starter Pack CLI

The tool automatically authenticates with Google Cloud, enables the necessary Vertex AI APIs, and configures your local environment. Once you see the green Success! message, you’re good to go.

Web User Interface

One of the most convenient features of the ADK is the ability to visually debug your agent before deploying it. By running the command make install && make playground, you launch a local development server with a built-in UI. Yes, it has a chat window, but it goes way beyond that by tracing events, tool calls, and more.

In the screenshot below, I’m interacting with the newly created agent. The agent is configured with a ReAct (Reasoning and Acting) pattern — a framework introduced by Yao et al. in 2022 that has become foundational in agentic AI. The ReAct pattern’s continuous loop of “Thought,” “Action,” and “Observation” enhances problem-solving and interpretability, making the agent’s decision-making process transparent. It recognized the intent, invoked the get_weather tool, and returned the structured data (“It’s sunny and 72°F”).

Agent Development Kit web user interface

Understanding the Code

Now that we’ve seen the agent in action, let’s look at the Go code that makes this work. The logic lives in agent/agent.go. This file handles tool definitions, model configuration, and initialization.

The ADK uses standard Go structs to define how the Large Language Model (LLM) interacts with your code. For example, to define the input parameters for our weather tool, we simply define a struct with json and jsonschema tags:

type GetWeatherArgs struct {
    City string `json:"city" jsonschema:"City name to get weather for"`
}

GetWeatherResult defines the structure of the data returned to the agent after the tool executes.

type GetWeatherResult struct {
 Weather string `json:"weather"`
}

GetWeather is a standard Golang function that accepts tool.Context and the arguments struct, performing the business logic and returning the result struct.

func GetWeather(_ tool.Context, args GetWeatherArgs) (GetWeatherResult, error) {
 return GetWeatherResult{
  Weather: "It's sunny and 72°F in " + args.City,
 }, nil
}

The NewRootAgent function is responsible for assembling and returning the agent.Agent instance that the application launcher requires. It begins by initializing the model configuration, creating a gemini-2.5-flash model instance backed by genai.BackendVertexAI.

Next, it bridges the gap between Go code and the LLM by wrapping the local GetWeather function into a [functiontool](https://pkg.go.dev/google.golang.org/adk/tool/functiontool). This step registers the tool with the name get\_weather and provides the necessary description for the model’s context. Finally, it constructs the agent using llmagent.New, which combines the initialized Gemini model, the system instructions that define the agent’s behavior, and the slice of available tools into a single unit.

The NewRootAgent looks like this (with some error-handling removed):

func NewRootAgent(ctx context.Context) (agent.Agent, error) {
 model, err := gemini.NewModel(ctx, "gemini-2.5-flash", &genai.ClientConfig{
  Backend: genai.BackendVertexAI,
 })

 weatherTool, err := functiontool.New(functiontool.Config{
  Name: "get_weather",
  Description: "Get the current weather for a city.",
 }, GetWeather)

 rootAgent, err := llmagent.New(llmagent.Config{
  Name: "my-first-go-agent",
  Model: model,
  Description: "A helpful AI assistant.",
  Instruction: "You are a helpful AI assistant designed to provide accurate and useful information.",
  Tools: []tool.Tool{weatherTool},
 })

Testing

The project contains both unit tests for internal logic, and end-to-end tests for server integration.

In agent/agent\_test.go, the GetWeather function is called with a suite of test cases, and verifies that the output string matches its expectations.

func TestGetWeather(t *testing.T) {
 // tests struct initialized with "San Francisco" and "New York"

 for _, tt := range tests {
  t.Run(tt.name, func(t *testing.T) {
   // Pass nil for tool.Context since GetWeather doesn't use it
   result, err := GetWeather(nil, GetWeatherArgs{City: tt.city})
   if err != nil {
    t.Fatalf("GetWeather() error = %v", err)
   }
   if !strings.Contains(result.Weather, tt.wantCity) {
    t.Errorf("GetWeather() = %v, want city %v in response", result.Weather, tt.wantCity)
   }
  })
 }
}

The end-to-end tests verify that the agent works correctly when running as a server, specifically checking that A2A or Agent-to-Agent protocol support is working correctly. The E2E tests start a real instance of the server, sending HTTP requests to it, and check the responses. Here’s a snippet from e2e/integration/server\_e2e\_test.go:

func TestA2AMessageSend(t *testing.T) {
    if testing.Short() { t.Skip("Skipping E2E test in short mode") }

    // Start server (local variable to avoid race conditions)
    t.Log("Starting server process")
    serverProcess := startServer(t)
    defer stopServer(t, serverProcess)

    if !waitForServer(t, 90*time.Second) {
   t.Fatal("Server failed to start")
}
t.Log("Server process started")

You can run all tests with make test:

make test                      
go test -v ./agent/... ./e2e/...
=== RUN TestGetWeather
=== RUN TestGetWeather/San_Francisco
=== RUN TestGetWeather/New_York
--- PASS: TestGetWeather (0.00s)
    --- PASS: TestGetWeather/San_Francisco (0.00s)
    --- PASS: TestGetWeather/New_York (0.00s)
PASS
ok my-first-go-agent/agent 0.218s

Deployment

The make deploy command automatically builds your application from source using Google Cloud Buildpacks, triggered by the --source . flag. It deploys this image to Cloud Run with several production-optimized flags: --memory “4Gi” to provide ample RAM for LLM operations, and --no-cpu-throttling to ensure the CPU remains allocated 24/7. This configuration is particularly valuable for Go applications.

`make deploy` builds the container and deploys to Cloud Run

To ensure your agent runs securely, the command is enabled with a strict configuration. It uses --no-allow-unauthenticated to block all public access by default, requiring Identity and Access Management (IAM) authentication for any requests. It also injects environment variables via --update-env-vars, including the use of Vertex AI GOOGLE\_GENAI\_USE\_VERTEXAI=True. After running the command, I have a service URL!

If you want to view the deployed web UI, I recommend deploying with make deploy IAP=true. This will handle the steps to enable IAP for Cloud Run. You will also need to provide access to users within your organization following the instructions in the documentation.

Adding a principal to IAP with the Google Cloud Console

With IAP enabled, I can now view the web UI or the deployed Agent Card. This card serves as your agent’s standard interface, allowing it to be dynamically discovered by other agents, orchestrators, or human-facing UI:

What’s next?

To continue your journey building production AI agents in Golang:

ADK Documentation: Complete guides on advanced patterns, multi-agent orchestration, and memory systems
Agent Starter Pack: Explore templates, including multi-agent systems and complex architectures
Cloud Run Documentation: Deep dives on performance optimization, scaling strategies, and security best practices
Go Concurrency Patterns: Understanding goroutines and channels will help you build more efficient agent tooling
Vertex AI Agent Engine: For managed agent infrastructure with built-in orchestration and tooling

As you scale from one agent to many, the engineering decisions we’ve discussed here compound in value. Go’s concurrency model and Cloud Run’s autoscaling are both necessary ingredients. Share what you’re building with me on LinkedIn, X, or Bluesky!

The Six Failures of Text-to-SQL (And How to Fix Them with Agents)

Karl Weinmeister — Tue, 11 Nov 2025 14:23:12 +0000

I’ve written countless SQL queries over the years. Unfortunately, like my golf game, I don’t write SQL enough to be a pro at it. Outside of straightforward SELECT statements, I approach SQL queries iteratively. I’ll inspect the tables, draft a query, and hope for the best. If there are any errors, I’ll go through this loop again.

While AI models are much better than me at SQL, they aren’t perfect. And that loop I described is just as important for automated approaches to be effective. Text-to-SQL is a deceptively difficult problem with challenges including linguistic ambiguity and rare SQL operations.

This is where a multi-agent architecture, built with a framework like Google’s Agent Development Kit (ADK), becomes essential. We can build a “virtual data analyst” by composing a team of specialized agents. A SchemaExtractor can find the right tables, a SqlGenerator can write the draft, and a SqlCorrector can critique and fix it. A SequentialAgent acts as the manager, ensuring the process is followed, every single time.

In this guide, we’ll walk through the six most common failure points for Text-to-SQL and show how to solve each one by building out our team of agents, moving from a simple script to a full-fledged agentic system. We’ll use the sample project kweinmeister/text-to-sql-agent to illustrate these solutions.

Problem 1: Agent Order Issues

Here’s the issue with a single LlmAgent that holds all the tools: it decides the order of operations. It might confidently skip fetching the schema and invent a table name. Or it might try to run a query before validating it. A single LLM is deciding what to do next, and it can (and will) make mistakes. That’s not a reliable process.

Solution: SequentialAgent for Order Control

The ADK gives us “Workflow Agents” for this. These specialized agents don’t use an LLM for flow control. They’re deterministic.

The SequentialAgent is the simplest and most powerful one to start with. It runs its sub-agents in the exact order you list them. Using a sequential agent also separates the concerns of “what to do” (our specialized agents) from “the order to do it in” (the workflow agent).

The SequentialAgent also acts as a guardrail. It turns our best practices (“always get the schema first,” “always validate before running”) into enforced infrastructure, not just suggestions in a prompt.

Code Example: Defining the Workflow Manager

Let’s define our root agent. Instead of a single LlmAgent, our root_agent will be a SequentialAgent. We’ll start by defining the specialists as stubs (we’ll build them out in the next sections):

from google.adk.agents import SequentialAgent

from .agents import (
    schema_extractor_agent,
    sql_correction_loop,
    sql_generator_agent,
)
from .callbacks import capture_user_message

root_agent = SequentialAgent(
    name="TextToSqlRootAgent",
    before_agent_callback=capture_user_message,
    sub_agents=[
        schema_extractor_agent,
        sql_generator_agent,
        sql_correction_loop,
    ],
)

Problem 2: LLM Schema Hallucinations

This is the classic failure mode. The LLM just doesn’t know your schema.

A common but flawed fix is to dump the entire database schema into the prompt. This backfires for two reasons. First, huge enterprise schemas won’t even fit in the context window. Second, even if they did, giving the LLM 100 irrelevant tables to find the 2 relevant ones just drowns it in noise and leads to worse results.

Solution: Dedicated Schema-Retrieval Tool

The answer is dynamic retrieval. Don’t give the agent a static block of schema; give it a tool to fetch schema. This lets the LLM reason about what it needs first, and then request only that specific information.

We can build a simple Python function for this. The ADK makes it easy to turn any function into an agent-callable tool with FunctionTool. The agent automatically figures out how to use it from its docstring, a best practice you’ll see in projects like gabrielpreda/adk-sql-agent.

Code Example: The Schema Tool

💡 In the kweinmeister/text-to-sql-agent project, the functions are not wrapped as tools, since they are directly called by a deterministic agent. They are provided centrally in a tools.py file, so that they can be easily leveraged as tools in a future LlmAgent.

import logging
from typing import Any

from .config import DB_URI
from .dialects.dialect import DatabaseDialect

logger = logging.getLogger( __name__ )

def load_schema_into_state(state: dict[str, Any], dialect: DatabaseDialect) -> None:
    """
    Loads the DDL and SQLGlot schema into the state dictionary.
    This function relies on the caching mechanism within the dialect object.
    """
    logger.info(f"Loading schema for dialect: {dialect.name}")

    db_uri = DB_URI
    # Error handling code omitted

    try:
        logger.info(f"Loading schema from database: {db_uri}")
        # The dialect object handles its own caching.
        # The first call to get_ddl will trigger the DB query and cache the DDL.
        logger.info("Calling dialect.get_ddl...")
        state["schema_ddl"] = dialect.get_ddl(db_uri)
        logger.info("DDL loaded successfully")

        # The call to get_sqlglot_schema will use the cached DDL if available,
        # then parse it and cache the result.
        logger.info("Calling dialect.get_sqlglot_schema...")
        state["sqlglot_schema"] = dialect.get_sqlglot_schema(db_uri)
        logger.info("SQLGlot schema loaded successfully")
        logger.info(f"SQLGlot schema keys: {list(state['sqlglot_schema'].keys())}")

    except Exception as e:
        error_msg = f"Error extracting schema: {e}"
        logger.error(error_msg, exc_info=True)
        state["schema_ddl"] = f"Error loading schema: {error_msg}"
        state["sqlglot_schema"] = {"error": error_msg}

Problem 3: Query Logic Errors

Even with the right schema, the LLM can still make logical mistakes with complex joins or aggregations. A human analyst would spot the error, critique it (“That join is wrong, you need to use user_id”), and refine it.

Our SequentialAgent is too simple for this. It’s a waterfall. It can’t go backwards and iterate.

Solution: LoopAgent for Iterative Refinement

The ADK has another workflow agent for this: the LoopAgent. This agent runs its sub-agents iteratively until a condition is met. It’s perfect for a “generate-and-critique” pattern.

We don’t have to replace our SequentialAgent. We can enhance it by nesting workflow agents. We’ll replace the single query generation step inside our SequentialAgent with a new LoopAgent. This loop will contain a team of two specialists:

A Writer Agent: An LlmAgent that writes the SQL draft.
A Critic Agent: A second LlmAgent with a different prompt, whose only job is to correct the writer’s SQL.

This is a powerful way to get LLMs to self-correct, which improves the quality of the final query.

Code Example: Building a “Generate-and-Critique” Loop

sql_generator_agent = Agent(
    name="sql_generator_agent",
    model=MODEL_NAME,
    description="Generates an initial SQL query from a natural language question.",
    instruction=get_generator_instruction,
    output_key="sql_query",
    after_model_callback=clean_sql_query,
)

sql_corrector_agent = Agent(
    name="sql_corrector_agent",
    model=MODEL_NAME,
    description="Corrects a failed SQL query.",
    instruction=get_corrector_instruction,
    output_key="sql_query",
    tools=[],
    after_model_callback=clean_sql_query,
)

sql_correction_loop = LoopAgent(
    name="SQLCorrectionLoop",
    sub_agents=[
        sql_processor_agent,
        sql_corrector_agent,
    ],
    max_iterations=3,
)

Problem 4: Agent Performance and Cost

We’re now using three LLM-powered agents. This is great for quality, but it’s slow and costs money with every API call.

What about simple, deterministic steps? Things like validating SQL syntax, formatting data, or cleaning up LLM output. Using a powerful LLM for these jobs is like using a sledgehammer to hang a picture. It’s slow, expensive, and surprisingly unreliable.

Solution: Custom Agents for Code-Based Logic

The ADK isn’t just for LLMs. You can create a “Custom Agent” by inheriting from BaseAgent and implementing the _run_async_impl method.

This agent has no LLM. It runs pure Python code. It’s fast and 100% deterministic. We’ll create a custom agent for our next problem: validation.

Code Example: Building a Non-LLM ValidationAgent

This agent will use the sqlglot library (which we’ll discuss in detail next) and will be a custom BaseAgent.

class SQLProcessor(BaseAgent):
    """
    Agent that handles the mechanical steps of:
    1. Validating the current SQL.
    2. Executing it ONLY if validation passed.
    3. Escalating to exit the loop on successful execution.
    """

    async def _run_async_impl(self, ctx: InvocationContext) -> AsyncGenerator[Event]:
        logger.info(f"[{self.name}] Starting SQL processing.")
        # ...

Problem 5: Dangerous Query Execution

This is the big one. You can’t execute LLM-generated code directly against your database. Ever. It’s a massive security and stability risk.

We need a fast, reliable check for syntax errors. What if the LLM produces a query that’s syntactically invalid? Or for the wrong SQL dialect?

Solution: Non-Destructive Dry Run with sqlglot

This is where our custom SqlValidationAgent shines. We’ll use the sqlglot library, a pure-Python SQL parser and transpiler.

Why sqlglot? It’s fast and local, building a real Abstract Syntax Tree (AST) which is infinitely more reliable than regex. It’s also dialect-aware, so it can catch syntax errors specific to, say, PostgreSQL.

We can just wrap sqlglot.parse_one(sql) in a try…except block. If it parses, the syntax is valid. If it throws a ParseError, it’s not. This gives us a fast and cheap validation signal.

Code Example: Full ValidationAgent Implementation

Here is the full implementation of the SqlValidationAgent we previewed with sqlglot validation.

from google.adk.agents import BaseAgent
from google.adk.core import InvocationContext, Event
from google.genai.types import Content, Part
import sqlglot
import sqlglot.expressions as exp
import asyncio
from typing import AsyncGenerator

class SQLProcessor(BaseAgent):
    """
    Agent that handles the mechanical steps of:
    1. Validating the current SQL.
    2. Executing it ONLY if validation passed.
    3. Escalating to exit the loop on successful execution.
    """

    async def _run_async_impl(self, ctx: InvocationContext) -> AsyncGenerator[Event]:
        logger.info(f"[{self.name}] Starting SQL processing.")
        state = ctx.session.state
        dialect = get_dialect()

        val_result: dict[str, Any] = run_sql_validation(state, dialect)
        yield Event(
            author=self.name,
            invocation_id=ctx.invocation_id,
            custom_metadata={"validation_result": val_result},
        )

        if val_result.get("status") == "success":
            exec_result: dict[str, Any] = run_sql_execution(state, dialect)

            result_event = Event(
                author=self.name,
                invocation_id=ctx.invocation_id,
                custom_metadata={"execution_result": exec_result},
            )

            # If execution succeeds, this is the final answer.
            # Escalate to exit the loop and provide the final content.
            if exec_result.get("status") == "success":
                logger.info(
                    f"[{self.name}] SQL execution successful. Escalating to exit loop."
                )
                result_event.actions.escalate = True

                final_query: str | None = state.get("sql_query")
                state["final_sql_query"] = final_query

                if final_query:
                    result_event.content = Content(
                        role="model", parts=[Part(text=final_query)]
                    )

            yield result_event
        else:
            logger.info(f"[{self.name}] Skipping execution due to validation failure.")
            state["execution_result"] = {
                "status": "skipped",
                "reason": "validation_failed",
            }

Problem 6: Messy LLM Output

One last thing. LLMs are trained to be helpful conversationalists. So when you ask for a SQL query, you often get this:

“Sure! Here is the SQL query you asked for: SELECT * FROM users;”

That conversational fluff will break our SqlValidationAgent every single time. We need a way to programmatically clean the LLM’s output before it’s passed to the next agent.

Solution: Callbacks for Post-Processing

We could add another CustomAgent just to strip the text, but that feels a bit heavy for such a simple task.

The ADK offers a more elegant solution: Callbacks.

An AfterAgentCallback is a function you attach to an agent that’s guaranteed to run immediately after the agent finishes. It can even modify the agent’s final output.

Code Example: Attaching a Cleanup Callback

import re
from google.adk.core import InvocationContext, Content

def cleanup_sql_output(
    context: InvocationContext,
    agent_output: Content
) -> Content:
    """
    This callback runs *after* the agent and cleans its output.
    """
    raw_text = agent_output.parts.text

    # Simple regex to find content within ```
{% endraw %}
sql...
{% raw %}

match = re.search(r"```

sql\s*(.?)\s

```", raw_text, re.DOTALL | re.IGNORECASE)

cleaned_text = raw_text
if match:
    cleaned_text = match.group(1)
else:
    # Fallback: simple stripping
    cleaned_text = raw_text.strip().strip("`").strip()

# Add a semicolon if it's missing (another common cleanup)
if not cleaned_text.endswith(";"):
    cleaned_text += ";"

# Return a *new* Content object to *replace* the original output
return Content.from_text(cleaned_text)




### Final Architecture

We’ve systematically tackled the six hardest problems in Text-to-SQL, evolving a brittle script into an extensible multi-agent system.

Our final root\_agent is a [SequentialAgent](https://google.github.io/adk-docs/api-reference/python/google.adk.agents.html#google.adk.agents.SequentialAgent) that orchestrates a team of specialists: a schema-fetching agent, a looping agent for iterative query improvement (with its own writer and critic), and a fast, deterministic validation agent using sqlglot.

The point is that modern agent development is about _composition_. You have to choose the right ADK construct for the right task. This table is a cheat sheet for making that decision.

### Agent Design: The “Right Tool for the Job”

![](https://cdn-images-1.medium.com/max/1024/1*OuHQ_Jb7kQ0IaBrlWKVICg.png)

### Conclusion: Building Reliable AI Systems

This pattern of **Specialization** , **Orchestration** , and **Safeguards** is the future of building production-ready AI. It’s not just for SQL, either. You can use this same architecture for autonomous code generation, document analysis, and much more.

So stop trying to build one “super-prompt” and start building teams of specialized agents. Welcome to the world of reliable, agentic systems.

What’s next? Get started in 3 simple steps in the [sample repository](https://github.com/kweinmeister/text-to-sql-agent). If you want a hands-on lab exercise, check out [Build Multi-Agent Systems with ADK](https://codelabs.developers.google.com/codelabs/production-ready-ai-with-gc/3-developing-agents/build-a-multi-agent-system-with-adk?hl=en#0&utm_campaign=CDR_0x2b6f3004_default_b459252462&utm_medium=external&utm_source=blog). To learn about powerful, built-in natural language capabilities in AlloyDB, try out the [AlloyDB AI NL SQL](https://codelabs.developers.google.com/alloydb-ai-nl-sql?hl=en#0&utm_campaign=CDR_0x2b6f3004_default_b459252462&utm_medium=external&utm_source=blog) codelab.

Want to keep the discussion going about multi-agent systems? Connect with me on [LinkedIn](https://www.linkedin.com/in/karlweinmeister/), [X](https://x.com/kweinmeister), or [Bluesky](https://bsky.app/profile/kweinmeister.bsky.social).

* * *

Deploy Faster with Terraform: Your Guide to vLLM on GKE with Infrastructure-as-Code

Karl Weinmeister — Sun, 12 Oct 2025 23:24:44 +0000

Somewhere in your AI journey, you’re going to push the limits of what models can do.

You might need to squeeze out that extra bit of performance, or try to fit a big model right under a GPU’s VRAM limit. All of these situations require tweaking and redeployment. That’s not as simple as it sounds, when the infrastructure includes everything from GPU clusters to storage to networking.

The solution is to treat your infrastructure the same way you treat your application code. It needs to be versioned in Git. It needs to be tested. And it needs to be deployed through an automated pipeline. This practice, known as Infrastructure as Code, or IaC, is the foundation of any serious MLOps strategy.

This article is a practical guide on how to use Terraform for agile ML engineering. I’ll walk through a real-world example of deploying a high-with vLLM on Google Kubernetes Engine. You can follow along with the complete source code on GitHub in the vllm-gke-terraform repository.

We will use the Qwen3–32B model in this article, which can be run on easily accessible NVIDIA L4 GPUs on Google Cloud. The Terraform script has been tested on larger models, such as the Qwen/Qwen3–235B-A22B-Instruct-2507 on a cluster with 8 H100 GPUs.

The scripts currently use GKE standard clusters for maximum flexibility. For production workloads where you want to offload node management and focus purely on the application, it’s recommended to leverage GKE Autopilot capabilities.

Declarative Infrastructure

Terraform uses a declarative language (HCL) where you define the desired end state of your infrastructure. You specify what you need, and Terraform’s engine calculates the necessary API calls to make the real-world infrastructure match that state. Before applying any changes, you can run the terraform plan command to see a detailed preview of what Terraform will create, modify, or destroy.

This allows for a thorough review to ensure the proposed changes align with your intentions, preventing unintended modifications. This declarative model is the key to eliminating configuration drift and ensuring that every environment is provisioned identically, a critical requirement for reproducible experiments.

The Terraform provider for Google Cloud is the interface between Terraform and Google Cloud. For example, the google_container_cluster resource is used to manage a GKE cluster. You can find the full set of GKE resources here.

In our project, the gke.tf file declares the desired state of a GKE cluster with specific node pools:

# gke.tf
resource "google_container_cluster" "qwen_cluster" {
  name = local.cluster_name
  location = var.zone
  project = var.project_id
  # ...
}

resource "google_container_node_pool" "gpu_pools" {
  # ...
  node_config {
    machine_type = each.value.machine_type
    guest_accelerator {
      type = each.value.accelerator_type
      count = each.value.accelerator_count
    }
  }
}

To manage this, Terraform maintains a state file that maps these definitions to their real-world resources. For team collaboration, using a remote state backend like Cloud Storage is recommended. It provides a centralized source of truth and uses locking mechanisms to prevent conflicting changes. Here’s how to instruct Terraform to use GCS as its backend:

# main.tf
terraform {
  backend "gcs" {
    prefix = "terraform/state/vllm-gke"
  }
}

Reusable Modules

Terraform modules are the primary mechanism for abstraction and reuse. MLOps teams can create a library of standardized modules for common components like a GKE cluster or a vector database.

Modules are made reusable through input variables. This allows an engineer to maintain a single, version-controlled set of Terraform files and use variable files (.tfvars) to launch new, isolated deployments.

To test a new model, you could simply create a new variable file like llama3-test.tfvars. By overriding a few default values, you can spin up an entirely new, isolated environment to test Llama-3–8B on L4 GPUs:

# my-experiment.tfvars
project_id = "my-gcp-project"
name_prefix = "my-llama3-deployment"
model_id = "meta-llama/Llama-3-8B-Instruct"
gpu_type = "l4"

Running terraform apply -var-file=llama3-test.tfvars makes spinning up parallel experiments a trivial, declarative operation, dramatically increasing a team’s experimental throughput.

For production systems, this same principle allows for sophisticated, zero-downtime strategies like Blue/Green deployments. A second, parallel “green” version of the entire stack is deployed by instantiating the Terraform configuration with a different set of variables. Once the new environment is fully validated, production traffic can be instantly switched at the load balancer or DNS level. The old “blue” environment can then be decommissioned. By codifying these complex release strategies, the entire deployment process becomes a version-controlled, auditable artifact.

Configuring the vLLM Engine

Provisioning hardware consistently is the first step. Configuring software to utilize that hardware efficiently is next.

The sample project uses the popular vLLM inference engine. Let’s show how to effectively link Terraform variables to configuration parameters in vLLM.

In variables.tf, the high-level knobs for experiments are defined:

# variables.tf
variable "gpu_memory_utilization" {
  description = "GPU memory utilization ratio"
  type = number
  default = 0.9
}

variable "max_model_len" {
  description = "The maximum model length."
  type = number
  default = 8192
}

variable "vllm_max_num_seqs" {
  description = "The maximum number of sequences (requests) to batch together."
  type = number
  default = 64
}

Then, the deployment in kubernetes.tf consumes these variables to construct the vLLM server’s startup arguments:

# kubernetes.tf
...
container {
  name = "vllm-container"
  args = compact([
    # --- Base Model Arguments ---
    "--model",
    var.model_id,
    "--tensor-parallel-size",
    tostring(local.gpu_config.accelerator_count),

    # --- Performance Tuning from Variables ---
    "--gpu-memory-utilization",
    tostring(var.gpu_memory_utilization),
    "--max-model-len",
    tostring(var.max_model_len),
    "--max-num-seqs",
    tostring(var.vllm_max_num_seqs),
  ])
}

Production-Grade Architecture

The sample project showcases a blueprint for a production-grade inference endpoint on GKE designed for both performance and cost-efficiency.

The gke.tf file provisions a GKE cluster with both spot and on-demand GPU node pools, which allows for a flexible and cost-effective approach to managing expensive GPU resources. You can read more here about the strategy to back up spot VMs with an on-demand node pool.

To avoid re-downloading large models on every pod restart, a kubernetes_persistent_volume_claim is created in kubernetes.tf to provide a persistent cache for the Hugging Face models. A Kubernetes Job, defined in kubernetes_jobs.tf, is then used to download the specified model into this persistent volume. This job runs to completion before the main vLLM deployment is scaled up, ensuring the model is ready before the inference server starts.

Automated Workflows

While Terraform itself is a big leap forward from shell scripting, it’s crucial that teams don’t stop there. The next step beyond running manual terraform commands is to embrace an automated, end-to-end CI/CD workflow, often called GitOps. The source control repository becomes the single source of truth for both application code and infrastructure.

The sample project includes a basic GitHub Actions workflow that validates the Terraform code on every push and pull request.

# .github/workflows/terraform-validate.yml
name: 'Terraform Validate'
on: [push, pull_request]

jobs:
  validate:
    name: 'Terraform Validate'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - run: terraform fmt -check -recursive
      - run: terraform init -backend=false
      - run: terraform validate

A complete CI/CD pipeline would extend this by running terraform plan on pull requests to preview changes and automatically running terraform apply on merge to the main branch to deploy them. This creates a flywheel where code is pushed and infrastructure is updated without manual intervention.

Infrastructure-as-Code is a now an AI Competency

The main takeaway is this: mastering Infrastructure as Code isn’t an optional “DevOps” skill. It’s a core competency for the modern ML engineer. For any organization serious about productionizing AI, Terraform on Google Cloud is the a key step toward building a scalable engineering culture.

If you’d like to keep learning more, I recommend the step-by-step guide on using a GKE cluster with Terraform: Quickstart: Deploy a workload with Terraform.

From there, I’d love to hear more about your journey with AI and Cloud infrastructure. Connect on LinkedIn, X, or Bluesky to continue the discussion!

A Developer’s Guide to Model Routing

Karl Weinmeister — Mon, 25 Aug 2025 16:26:04 +0000

Not long ago, building with LLMs meant picking one general-purpose model and sticking with it. Today, the landscape is flooded with thousands of options: large and small, open and closed-source, generalist and specialist, each with unique capabilities and costs.

This explosion of choice has fundamentally changed how we build AI applications. The one-size-fits-all approach is over.

Instead, we architect systems that select the best model for each task. This is the idea behind model routing. This architectural pattern can be implemented today, and has the potential to change the economics of model inference. Let’s get into it!

Understanding Model Routing

As a developer building with LLMs, you’re constantly juggling three competing priorities: performance, cost, and latency.

Performance (Quality): For complex reasoning and creative generation, you might reach for state-of-the-art models like Google’s Gemini 2.5 Pro. These models deliver high-quality, accurate responses.
Cost: While premium models deliver state-of-the-art performance, they represent a significant investment. The key to a sustainable AI strategy is to reserve these powerful models for tasks where their advanced capabilities provide a clear return on investment. For more routine queries, smaller, highly efficient models can deliver excellent results at a fraction of the cost. Recent studies show this approach can yield cost savings without significantly degrading performance.
Latency: In interactive applications like chatbots, a fast response time is critical for a positive user experience. Smaller, specialized models can deliver near-instantaneous responses, making them ideal for real-time, conversational AI. By routing interactive queries to these faster models, you can create a more engaging and responsive application.

Relying on a single model forces an unnecessary compromise. Use a top-tier model for everything, and you pay a premium for power you don’t always need. Use a smaller model for everything, and you sacrifice quality on complex queries. So why are we still forcing ourselves to choose just one?

Model routing is an architectural pattern designed to solve this optimization problem. It involves maintaining a pool of candidate LLMs and routing each incoming prompt to the most suitable model. That’s often the smallest, fastest, and most cost-effective model that can successfully complete the task.

Common Routing Patterns

Implementing a model router involves choosing an architectural pattern that determines how routing decisions are made. These patterns exist on a spectrum of complexity and intelligence, from simple, predefined rules to sophisticated, AI-driven classification. We will focus on dynamic routing patterns that assess the content, intent, and complexity of the prompt to select the optimal model.

Rule-Based Routing

This is the simplest form of dynamic routing. It uses hard-coded logic, typically a series of if/else statements, to make routing decisions based on simple characteristics of the prompt.

The rules are based on easily measurable attributes of the prompt, such as the presence of certain keywords, its overall length, or matches against regular expressions. For instance, a system might check for specific terms to identify a task category or measure the prompt’s length to estimate its complexity.

Pros: This approach is predictable, transparent, and fast to execute. It’s an excellent choice for well-defined, simple workflows where task categories can be reliably distinguished by straightforward heuristics.
Cons: Rule-based systems are brittle and inflexible because they lack a true understanding of language. They can be easily confused by semantic nuance, such as negation or context. The system also becomes difficult to maintain and scale as the number of rules grows.

LLM-Based Routing

This pattern leverages the intelligence of an LLM to perform the routing task itself. A dedicated, often smaller and faster, “router LLM” acts as a classification engine.

The user’s prompt is fed into the router LLM. The router LLM is given a prompt that instructs it to analyze the query and classify it into predefined categories. To ensure the output is machine-readable, the router LLM is instructed to respond in a structured format like JSON. The application then parses this JSON output to determine which model to call next.

Pros: This is a powerful and flexible approach. The router LLM can understand complex, ambiguous, and nuanced language. It can handle multi-intent queries and can be adapted to new routing tasks simply by updating its system prompt.
Cons: The primary drawback is significant overhead. This method introduces an additional, full LLM API call into the critical path of every request. This adds both cost and latency, which can undermine the goals of optimization the router was intended to achieve.

Semantic Routing

Semantic routing offers a powerful compromise, combining the speed of rule-based systems with the intelligence of LLM-based approaches. It operates on the principle of semantic similarity in vector space and is the core mechanism we’ll implement.

The process involves four steps. First, routes are defined, each with a name and a list of representative example phrases, or utterances. Next, a text embedding model converts all of these utterances into high-dimensional numerical vectors that capture their semantic meaning, which are then stored in an efficient index. When a new user query arrives, the same embedding model converts it into a vector. Finally, a vector similarity search is performed between the query’s vector and all the utterance vectors in the index, and the route whose utterances are most similar to the query is selected as the winner.

Pros: This method is fast, with decision times often in the milliseconds, because it relies on optimized vector math rather than a slow, generative LLM call. It’s highly scalable to thousands of potential routes and is more robust than simple keyword matching because it understands meaning and context. Modern libraries often allow this configuration to be externalized into declarative files like YAML, separating the routing logic from the application code for better maintainability.
Cons: The effectiveness of a semantic router is highly dependent on the quality and comprehensiveness of the example utterances provided for each route. It can also struggle with contextual, multi-turn conversational queries where the user’s intent is not explicitly stated in their most recent message.

The choice of routing architecture is governed by the “Router Latency Paradox”: a component designed to reduce overall application latency must itself be exceptionally low-latency. An LLM-based router introduces a full inference step to every request, increasing both latency and cost. For this approach to be a net positive, the downstream savings must consistently outweigh its operational overhead, which is a high bar for most interactive applications. Semantic routing, in contrast, replaces this slow inference with a near-instantaneous vector search. This performance difference establishes semantic routing as the default architectural best practice for dynamic, real-time model routing. LLM-based routing is thus reserved for cases where the routing logic is too complex to be captured by semantic similarity alone and the added latency is an acceptable trade-off.

The Gemini 2.5 Model Family

To build an effective router, you need a solid grasp of the candidate models in your pool. For our implementation, we’ll use Google’s Gemini 2.5 family, a suite of models with a tiered structure of capability and cost that’s perfect for a routing architecture.

A key innovation across the Gemini 2.5 family is their capability as “thinking models.” This means they can be configured to perform internal reasoning steps, akin to a chain of thought, before generating a final response. This feature, controllable via an API parameter known as the “thinking budget,” can significantly improve performance and accuracy on complex tasks. This controllable reasoning becomes another powerful dimension for our routing logic to consider.

Gemini 2.5 Pro

Capabilities: Gemini 2.5 Pro is Google’s flagship model, engineered for maximum performance and state-of-the-art accuracy. It’s optimized for the most complex and demanding tasks, including deep logical reasoning, advanced code generation, and sophisticated multimodal understanding across text, images, audio, and video.
Router Use Case: This is our designated “strong” model. We’ll route only the most challenging queries here: prompts that involve complex problem-solving, novel algorithm design, in-depth analysis of dense technical documents, or multi-step logical puzzles.
Thinking: For this model, the “thinking” capability is on by default, as it’s integral to its high-end performance.

Gemini 2.5 Flash

Capabilities: Gemini 2.5 Flash is designed to be the best model in the family in terms of its price-to-performance ratio. It offers well-rounded, powerful capabilities that approach those of Pro but at a significantly lower operational cost. It also features a controllable thinking budget.
Router Use Case: This is our “default” or “go-to” model. It’s the workhorse that will handle the majority of general-purpose queries. These are tasks that are more complex than simple classification but don’t require the full power (and expense) of Pro. Ideal use cases include general conversation, creative writing, drafting emails, and performing detailed summarizations.

Gemini 2.5 Flash-Lite

Capabilities: As its name suggests, Gemini 2.5 Flash-Lite is the fastest and most cost-efficient model in the 2.5 family. It’s highly optimized for low latency and high-throughput scenarios, making it a cost-effective upgrade from previous generations of Flash models.
Router Use Case: This is our fastest model. We’ll route simple, high-volume, and latency-sensitive tasks here. It’s perfect for text classification (e.g., sentiment analysis), simple data extraction (e.g., pulling names and dates from text), translation, and answering straightforward factual questions.
Thinking: To maximize its speed and cost-efficiency, “thinking” is turned off by default for Flash-Lite. However, it can be optionally enabled, providing granular control for tasks that might need a small boost in reasoning without escalating to the full Flash model.

Implementing a Semantic Router

With the theory covered, let’s get to the code. This section walks through the gemini-model-router project, which builds a semantic router to intelligently distribute queries among the Gemini 2.5 Pro, Flash, and Flash-Lite models. It uses the open-source semantic-router library as its engine and serves it all up with FastAPI.

Embeddings are created upfront for each route, and then matched to queries at runtime

Project Setup

To get started, clone the repository and follow the setup instructions in the README.md file, which covers creating the .env file and installing the required dependencies from requirements.txt.

Centralizing Configuration

A key architectural decision in the gemini-model-router project is the separation of configuration from code. All routing logic, including the routes, their representative utterances, and the specific LLM assigned to each route, is defined in a single router.yaml file. This makes the system highly maintainable and easy to modify without changing the application’s Python code.

The router.yaml file has two main sections:

encoder : Specifies the embedding model to use for converting text to vectors. In this case, it uses Google’s gemini-embedding-001 via the semantic-router’s GoogleEncoder.
routes : A list of route definitions. Each route has:

name: A unique identifier that maps directly to a Gemini model.
description: A human-readable explanation of the route’s purpose.
utterances: A list of example phrases that define the semantic space of the route.
llm: An object specifying the custom class (GoogleLLM), the Python module where it’s defined (main), and the target model ID (e.g., gemini-2.5-pro).

Here is a snippet from the router.yaml file, defining the route for complex queries. A key parameter in the full configuration is the score_threshold. When the router compares a query to its routes, it calculates a similarity score. By setting the threshold to 0.0, we ensure that the router always selects the route with the highest similarity, effectively guaranteeing that a decision is always made.

# router.yaml
encoder_name: gemini-embedding-001
encoder_type: google
routes:
- name: gemini-2.5-pro
  description: For complex, multi-step tasks requiring deep reasoning, code generation, and analysis of large documents.
  utterances:
  - Develop a comprehensive, multi-year business plan for a direct-to-consumer sustainable
    fashion brand, including financial projections and marketing strategies.
  - Write a Python script to perform sentiment analysis on a large CSV of customer
    reviews, generate visualizations, and create a summary report.
  - Compare and contrast the philosophical implications of determinism and free will
    in the context of advanced artificial intelligence, citing relevant academic sources.
  llm:
    module: main
    class: GoogleLLM
    model: gemini-2.5-pro
#... other routes for flash and flash-lite follow...

Routing Logic

The main.py file contains the FastAPI application that serves the router. It includes several key components that work together to bring the YAML configuration to life.

The GoogleLLM Wrapper

The semantic-router library requires a compatible LLM object for each route. To integrate with Google’s GenAI SDK, the project defines a custom GoogleLLM class that inherits from semantic_router.llms.BaseLLM. This class acts as a bridge, translating the semantic-router’s call signature into an asynchronous request to the Vertex AI Gemini API.

# main.py (simplified)
from semantic_router.llms import BaseLLM
from google import genai

class GoogleLLM(BaseLLM):
    _client: ClassVar[Optional[genai.Client]] = None

    @classmethod
    def get_client(cls) -> genai.Client:
        if cls._client is None:
            project_id = os.getenv("GOOGLE_CLOUD_PROJECT")
            cls._client = genai.Client(vertexai=True, project=project_id)
        return cls._client

    async def __acall__ (self, messages: List[Message], **kwargs) -> Optional[str]:
        contents = kwargs.get("multimodal_contents", messages[0].content)
        config = kwargs.get("config", self.kwargs.get("config", {}))

        response = await self.get_client().aio.models.generate_content(
            model=self.name,
            contents=contents,
            **config,
        )
        return response.text if response else ""

The /query Endpoint

The main API endpoint uses a series of helper functions to route and execute the query. The handle_query function orchestrates the process: it extracts text for routing, determines the best route, and executes the LLM call.

# main.py (simplified)
@app.post("/query", response_model=RouterResponse)
async def handle_query(request: QueryRequest, fastapi_request: Request):
    router = fastapi_request.app.state.router
    default_route = fastapi_request.app.state.default_route_name

    # 1. Extract text and determine the route
    text_for_routing = _get_text_for_routing(request.contents)
    route_choice = _determine_route(router, text_for_routing, default_route)
    chosen_route = router.get(route_choice.name)

    # 2. Execute the call using the LLM from the chosen route
    model_response = await _execute_llm_call(
        chosen_route, request.contents, request.config, text_for_routing
    )

    return RouterResponse(
        route_name=chosen_route.name, model_response=model_response
    )

Deploying to Production

While FastAPI’s web server uvicorn is perfect for local development, a production deployment requires a robust, scalable hosting environment. Cloud Run is an ideal choice for this service because it’s a fully managed, serverless platform that takes your containerized application (including the Uvicorn server) and handles all the underlying infrastructure, scaling, and request management.

To deploy the router, you first need to have the Google Cloud SDK installed and configured. Then, you can deploy the service with a single command:

gcloud run deploy gemini-model-router \
  --source . \
  --region us-central1

This command builds a container from your source code, pushes it to the Artifact Registry, and deploys it as a public-facing service. Cloud Run handles all the infrastructure, so you can focus on the application logic.

Production Best Practices

Deploying a model router to production requires building an observable and resilient system. An API management platform like Google Cloud’s Apigee can serve as a unified and secure gateway to your model routing service. It can provide essential capabilities like enforcing security policies, managing traffic with rate limiting and quotas, and offering deep visibility through analytics and monitoring. Let’s review the key principles needed to move beyond a proof-of-concept.

First, treat the router as a mission-critical, standalone service. Because it can be a single point of failure and a performance bottleneck, it must be independently scalable and fault-tolerant. Containerize the router and deploy it on a platform like Cloud Run to ensure high availability, allowing it to scale independently of the applications that consume it.

Second, you cannot optimize what you cannot measure. Implement comprehensive logging and monitoring for every routing decision. For each request, log the chosen route, similarity score, final model, latency, and estimated cost. This data can be fed into Google Cloud’s observability suite to create dashboards for tracking key performance indicators like route distribution, cost per query, and P99 latency. This allows you to set up alerts for anomalies, such as a sudden shift in routing patterns or an increase in fallback rates.

Third, the initial configuration is just a starting point. True optimization requires a data-driven feedback loop. Collect and review production queries to identify misrouted requests, and use this analysis to refine your route utterances. A/B testing frameworks are invaluable for comparing different routing strategies or model configurations in a live environment to validate improvements.

Finally, enterprise-grade reliability requires planning for failure. Implement a chain of fallbacks that goes beyond a simple default route. For instance, if a request to gemini-2.5-pro fails, the system should automatically retry with exponential backoff. If that also fails, it should fall back to the next best model, gemini-2.5-flash.

The Future of Model Routing

There is a broader trend towards more modular and dynamic AI architectures, and model routing is no exception. The future of model routing could include:

Multimodal Routing: The next logical step is routing on more than just text. The current router simplifies the problem by extracting the text from a multimodal prompt, but the concept of vector similarity works for any modality you can embed.
Hierarchical Routing: The concept of system-level model routing is a macro-scale analog of what Mixture-of-Experts or MoE architectures do within a single neural network. In an MoE model, an internal “router” network dynamically selects which “expert” sub-networks should process each token of an input sequence. Our external router does the same thing, but its “experts” are entire, independent LLMs. Future systems may employ hierarchical routing, where a top-level semantic router first selects the best specialized MoE model for a task, which then performs its own fine-grained, internal routing to process the request.

Ultimately, model routing is a foundational building block for the next generation of complex, multi-agent AI systems. As we’ve shown, the combination of a powerful model family like Google’s Gemini 2.5, a serverless platform like Cloud Run, and the open-source gemini-model-router project makes this advanced architecture an achievable engineering task. The tools are here. The patterns are clear.

It’s time to start building. Share what you’ve built with me on LinkedIn, X, or Bluesky!

Mastering Agentic Development with Gemini and Roo Code

Karl Weinmeister — Sun, 20 Jul 2025 04:56:25 +0000

The conversation around AI in software development has matured beyond the “AI as a chatbot” and into sophisticated AI agents. We’re moving toward building a living blueprint that can reason about your code in its entirety and evolve with it over time.

For developers who want a powerful, all-in-one AI experience, Google’s Gemini Code Assist is a fantastic solution that provides a seamless, out-of-the-box experience, bringing the power of Gemini directly into your workflow.

For those who love to assemble best-in-class technologies from the open ecosystem, this article is for you. We will explore a production-ready stack for those who want a customized and self-hosted solution. This stack combines the Roo Code VS Code extension, powered by Google’s underlying Gemini models, and takes it to the next level with a self-hosted Qdrant vector database on Google Kubernetes Engine.

Solution architecture for agentic development with Roo Code, Gemini, and Qdrant

Solution Components

Roo Code is a VS Code extension that can be thought of as an “AI Dev Team” with modes ranging from Architect to Debug. You can give it a high-level task, like “refactor this module to use the new logging service,” and it will create a plan, identify the necessary code changes, and execute them across multiple files. For a deeper dive, check out the Roo Code documentation.

Using Roo Code to update a project README based on the current codebase

You can take full advantage of Roo Code’s capabilities with the massive context window available in Gemini models. This allows Roo Code to hold a vast amount of code in its “short-term memory,” enabling it to understand the intricate relationships between files and modules and to generate code that is consistent with the entire project. You can learn more about the Gemini API in the official documentation.

To make the use of this large context window efficient, Roo Code leverages prompt caching, a feature now available in Gemini models. When Roo Code sends the initial instructions and context to the model, Gemini generates an internal representation and returns a cache reference. On subsequent requests, Roo Code can send this cache reference instead of the full prompt, dramatically reducing token usage and improving latency, which is a key feature for making the system both cost-effective and performant.

For codebase indexing, Roo Code supports Gemini’s gemini-embedding-001 state-of-the-art embedding model. This is crucial for the accuracy of the semantic search, and you can find more information on Gemini’s embedding models here.

Using Gemini Models in Roo Code

The connection between Roo Code and a model is what enables its agentic capabilities: planning, executing commands, and writing code across your entire project. You can connect to Gemini’s models through the Gemini API or through Google Cloud’s Vertex AI.

To use the Gemini API, you simply create an API key in Google AI Studio, then in Roo Code’s settings, select the Google Gemini provider, paste your key, and choose a model. For detailed, step-by-step instructions on this process, refer to the Roo Code documentation for the Gemini provider.

For teams and enterprises using Google Cloud, connecting via Vertex AI provides unified billing, IAM permissions, and more. You will create a service account with the “Vertex AI User” role in the Google Cloud Console and download its JSON key file. Within Roo Code’s settings, select the GCP Vertex AI provider, provide the credentials from your JSON key, and enter your Project ID and Region. The Roo Code documentation for Vertex AI provides a complete walkthrough of this setup.

The Vertex AI LLM provider for Gemini in Roo Code

For both connection methods, we recommend starting with gemini-2.5-pro for the best experience. Its powerful reasoning capabilities and large context window are ideal for complex, multi-step tasks. For faster, more cost-effective use, gemini-2.5-flash is an excellent alternative.

With Roo Code’s reasoning engine now powered by Gemini, the next step is to give it a persistent, long-term memory of your code.

Codebase Indexing

Codebase indexing creates a semantic “long-term memory” of your code that the agent can access at any time. This is a multi-stage process that transforms your source code into a searchable knowledge base.

Intelligent Chunking

First, Roo Code uses Tree-sitter to parse your code into an Abstract Syntax Tree (AST). This gives it a deep, structural understanding of your code, just like a compiler does. Instead of arbitrarily splitting a file every few hundred lines, the AST is used to intelligently chunk the code into complete, semantic blocks.

This “semantic chunking” means the pieces of code being indexed are meaningful and self-contained units, such as:

A complete function or method.
An entire class or struct definition.
A specific configuration block.

This ensures that the context isn’t lost by splitting a function in half. For unsupported languages, Roo Code falls back to line-based chunking.

Generating Embeddings

Once the code is broken down into these intelligent chunks, the next step is to capture their semantic meaning in a way a machine can understand. This is where Gemini’s gemini-embedding-001 model comes in.

Each semantic chunk produced by Tree-sitter is fed into the embedding model, which outputs a high-dimensional numerical vector. This vector is the embedding — a mathematical representation of the code’s meaning. The Gemini embedding model captures fine details with 3072 dimensions in every embedding. For a deeper dive into Matryoshka Representation Learning, a technique used to train the model, see this video:

Storing and Searching Embeddings

With the codebase converted into a collection of semantically-rich embeddings, they need a place to be stored and searched efficiently. Roo Code uses Qdrant, a high-performance vector database, for this purpose.

When you ask a question, Roo Code’s search tool follows this process:

Query: Your natural language query (e.g., “where is our user authentication logic?”) is sent to the Gemini embedding model.
Vectorize: The model converts your query into an embedding vector.
Search: Roo Code performs a vector search in the Qdrant database, looking for the code chunk embeddings that are most similar (i.e., closest in vector space) to your query’s embedding.
Retrieve: The tool then returns the most relevant code snippets, along with their file paths and similarity scores.

Roo Code also provides a user-friendly interface for configuring the codebase indexer. You can easily select your embedding provider, enter your API keys, and specify the Qdrant URL. The advanced configuration options allow you to fine-tune the search behavior by adjusting the Search Score Threshold and Maximum Search Results. You can also specify which files to ignore by adding patterns to a .rooignore file.

Indexing a codebase in Roo Code

From Local to Centralized Indexing

The easiest way to get started is with a local Qdrant instance. As the official Qdrant Quickstart shows, you can be up and running in minutes with a single Docker command:

docker run -p 6333:6333 -v "$(pwd)/qdrant_storage:/qdrant/storage:z" qdrant/qdrant

For an individual developer, this is a fantastic way to get all the benefits of codebase indexing without any external dependencies.

As your team grows, managing dozens of individual Docker instances can become cumbersome. This is where a centralized Qdrant instance provides value — not as a single, conflict-prone shared index, but as a managed, cost-effective platform to host a fleet of personal indexes.

Google Kubernetes Engine, or GKE, is an excellent choice for this, offering high availability and enterprise-grade security. The principle is the same regardless of the platform: provide a robust, central service to host many isolated environments. You can deploy the infrastructure within minutes using the GKE tutorial for deploying Qdrant.

Using the instructions in the tutorial, you can easily access it from your local system using port forwarding:

PROJECT_ID="your-project-id"
REGION="us-central1"

gcloud container clusters get-credentials qdrant-cluster --region "$REGION" --project "$PROJECT_ID"

kubectl port-forward service/qdrant 6333:6333

Roo Code generates a unique Qdrant collection name by hashing the absolute local workspace path. This means that even when using a central Qdrant instance, each developer’s index is completely isolated. To avoid conflicts, each developer needs to ensure they are using a different path:

Developer A: /Users/alice/projects/my-app
Developer B: /Users/bob/projects/my-app

Conclusion

The future of AI-assisted development is about choice. Whether you prefer a powerful, all-in-one solution like Google’s Gemini Code Assist for a seamless, integrated experience, or the composable stack detailed in this article, the goal is the same: to create a truly intelligent development environment.

What will you build with Gemini and Roo Code? Feel free to continue the discussion on LinkedIn, X, and Bluesky.

Getting started with Rust on Google Cloud

Karl Weinmeister — Thu, 27 Mar 2025 04:29:49 +0000

This post will guide you through deploying a simple “Hello, World!” application on Cloud Run. You’ll then extend the application by showing how to integrate with Google Cloud services with experimental Rust client libraries.

I’ll cover the necessary code, Dockerfile configuration, and deployment steps. I’ll also recommend a robust and scalable stack for building web services, especially when combined with Google Cloud’s serverless platform, Cloud Run.

Why Rust and Axum?

Rust has gained significant traction in backend development, earning the title of most-admired language in the StackOverflow 2024 Developer Survey. This popularity stems from its core strengths: performance, memory safety, and reliability. Rust’s low-level control and zero-cost abstractions enable highly performant applications. Its ownership system prevents common programming errors like data races and null pointer dereferences. In addition, Rust’s strong type system and compile-time checks catch errors early in the development process, leading to more reliable software.

The Rust web framework ecosystem is vibrant and evolving. Popular choices include Axum, Rocket, and Actix. In this post, I’ll showcase Axum, but you can apply what you’ve learned here to other Rust web frameworks. Axum’s API is clear and composable, making it easy to build web services. Its modular architecture allows developers to select only the necessary components. Axum is built on Tokio, a popular asynchronous runtime for Rust, which allows it to handle concurrency and I/O operations efficiently.

Hello World Application

Let’s start by exploring a basic “Hello, World!” example from the official Axum repository. In each section of this blog post, you will enhance the example to leverage Google Cloud capabilities. You can access the final code sample in the cloud-rust-example repository.

First, the Cargo.toml manifest file defines the project’s metadata and dependencies:

[package]
name = "example-hello-world"
version = "0.1.0"
edition = "2021"
publish = false

[dependencies]
axum = { path = "../../axum" }
tokio = { version = "1.0", features = ["full"] }

Within this file, you see:

[package]: Contains basic project information like name, version, and the Rust edition. publish = false prevents accidental publication.
[dependencies]: Lists the project’s dependencies — axum for the web framework and tokio for asynchronous capabilities.

Now, let’s examine the core application code, src/main.rs:

use axum::{response::Html, routing::get, Router};

#[tokio::main]
async fn main() {
    // build our application with a route
    let app = Router::new().route("/", get(handler));

    // run it
    let listener = tokio::net::TcpListener::bind("127.0.0.1:3000")
        .await
        .unwrap();
    println!("listening on {}", listener.local_addr().unwrap());
    axum::serve(listener, app).await.unwrap();
}

async fn handler() -> Html<&'static str> {
    Html("<h1>Hello, World!</h1>")
}

This code sets up a minimal web server using Axum and Tokio. The #[tokio::main] macro enables asynchronous execution. The main function creates a Router to handle requests, defines a single route / that responds with “Hello, World!”, binds the server to 127.0.0.1:3000, and starts the server. The handler function generates the HTML response for the root route.

Enhancements for Cloud Run

The basic example above works well for local development, but let’s make some improvements for deploying to Cloud Run. The official example notably does not include a Dockerfile, which is required for Cloud Run.

1. Standalone Deployment: To make the example standalone and deployable, modify the Cargo.toml file. Change the axum dependency from axum = { path = “../../axum” } to axum = “0.8” to use the published version of Axum from crates.io instead of the local path.

2. Dynamic Port Configuration:

Cloud Run dynamically assigns a port to your application, which is provided through the PORT environment variable. The original example hardcodes the port to 3000. To make our application Cloud Run-compatible, modify the main function to read the PORT environment variable and use it if available, falling back to a default port such as 8080 if the variable is not set.

The address should also be changed to 0.0.0.0 to listen on all network interfaces, which is generally preferred for containerized applications.

Here’s the modified main function:

#[tokio::main]
async fn main() {
    // Get the port from the environment, defaulting to 8080
    let port = std::env::var("PORT").unwrap_or_else(|_| "8080".to_string());
    let addr = format!("0.0.0.0:{}", port);

    // build our application with a route
    let app = Router::new().route("/", get(handler));

    // run it
    let listener = tokio::net::TcpListener::bind(addr)
        .await
        .unwrap();
    println!("listening on {}", listener.local_addr().unwrap());
    axum::serve(listener, app).await.unwrap();
}

3. Dockerfile:

To deploy to Cloud Run, you’ll need a Dockerfile. Here’s a simple one that works well for this example:

FROM rust:1.85.1
WORKDIR /app
COPY . .
RUN cargo build --release
EXPOSE 8080
CMD ["./target/release/example-hello-world"]

This Dockerfile uses the official Rust image as a base, copies the project files, builds the application in release mode, exposes port 8080 (the default port), and sets the command to run the compiled executable. You can upgrade to the latest Rust image if you’d like.

4. .gcloudignore file:

You can also add a .gcloudignore file to the project root to exclude unnecessary files (like the target directory containing build artifacts) from the deployment:

.git/
.gitignore
target/

Deploying to Cloud Run

Before deploying, ensure you have the Google Cloud SDK installed and configured, and you have enabled the Cloud Run API in your Google Cloud project. You’ll also need to be in the root directory of your Axum project (where the Cargo.toml file is located).

Before attempting your deployment, you can check the local package and deployment for errors:

cargo check

To deploy directly to Cloud Run from source, use the following command:

gcloud run deploy cloud-rust-example \
    --source . \
    --region us-central1 \
    --allow-unauthenticated

Here’s what each part of the command means:

gcloud run deploy cloud-rust-example: This is the base command to deploy a service to Cloud Run. cloud-rust-example is the name we’re giving to our service. You can choose a different name.
—-source .: This flag tells Cloud Run where to find the source code for your application. The . indicates the current directory. Cloud Run will use the Dockerfile in this directory to build a container image.
—-region us-central1: This specifies the Google Cloud region where your service will be deployed. In this case, we’re using us-central1. You can choose a region closer to your users for lower latency.
—-allow-unauthenticated: This flag makes your deployed service publicly accessible without requiring authentication. This is convenient for initial testing and simple public services. For production applications, you should remove this flag and implement proper authentication and authorization.

Cloud Run will automatically build and deploy your application. You will be provided with a service URL in the output. Accessing this URL in your browser will display the “Hello, World!” message.

Hello world output from / route

Integrating with Google Cloud Services

Let’s now show how to integrate our application with Google Cloud services. I’ve selected a straightforward scenario that doesn’t require any project configuration to work. You’ll add a new application route /project that will display information about your project.

To implement this, you’ll use the google-cloud-rust library to interact with the Resource Manager API and retrieve information about your Google Cloud project.

Note: The google-cloud-rust library is currently experimental. APIs may change, and it’s important to stay updated with the latest releases and documentation.

Add Dependencies

First, add the Resource Manager v3 API and reqwest HTTP client to your Cargo.toml file:

cargo add google-cloud-resourcemanager-v3 reqwest

Implement the handler

There are four key changes we’ll need to make in src/main.rs:

Add /project Route: A new route /project will display project information, implemented by project_handler()

async fn main() {
    // ...
    let app = Router::new()
        .route("/", get(handler))
        .route("/project", get(project_handler))
        .layer(Extension(std::sync::Arc::new(client)));
    // ...
}

project_handler function: The project handler will call get_project() to fetch project details. Finally, it formats the project information into an HTML response. Error handling is included to display any errors that occur during the API call.

async fn project_handler(Extension(client): Extension<Arc<Projects>>) -> Html<String> {
    let project_id = PROJECT_ID.get().expect("Project ID not initialized");
    let project_name = format!("projects/{}", project_id);

    match client.get_project(project_name).send().await {
        Ok(project) => {
            let project_number = project.name.strip_prefix("projects/").unwrap_or("Unknown");

            Html(format!(
                "<h1>Project Info</h1><ul><li>Name: <code>{}</code></li><li>ID: <code>{}</code></li><li>Number: <code>{}</code></li></ul>",
                &project.display_name,
                project_id,
                project_number
            ))
        }
        Err(e) => Html(format!("<h1>Error getting project info: {}</h1>", e)),
    }
}

Share client with handler: For best performance, any one-time configuration should not reside in the handler. The Projects client can be initialized in main() and then shared with the handler with Axum’s Extension.
Add helper function for project metadata : To find out the project ID the container is running in, you’ll need to access the metadata key. That project ID will then be used to call the Resource Manager API to get more information about the project, including its display name and creation time. You can use LazyLock to initialize the project only once.

static PROJECT_ID: OnceLock<String> = OnceLock::new();

async fn main() {
    // ...

    let project_id = get_project_id().await.expect("Failed to get project ID");
    PROJECT_ID.set(project_id).expect("Failed to set PROJECT_ID");

    // ...
}

async fn get_project_id() -> Result<String, String> {
    if let Ok(project_id) = var("GOOGLE_CLOUD_PROJECT") {
        return Ok(project_id);
    }

    let client = reqwest::Client::new();
    let url = "http://metadata.google.internal/computeMetadata/v1/project/project-id";

    let response = client
        .get(url)
        .header("Metadata-Flavor", "Google")
        .send()
        .await;

    match response {
        Ok(res) => {
            if res.status().is_success() {
                Ok(res.text().await.map_err(|e| e.to_string())?)
            } else {
                Err(format!("Metadata server returned error: {}", res.status()))
            }
        }
        Err(e) => Err(format!("Error querying metadata server: {}", e)),
    }
}

Set GOOGLE_CLOUD_PROJECT Environment Variable (Locally)

For local testing, you’ll need to set the GOOGLE_CLOUD_PROJECT environment variable to your Google Cloud project ID. You can do this in your terminal before running the application:

export GOOGLE_CLOUD_PROJECT=your-project-id

Replace your-project-id with your actual project ID. Cloud Run will automatically set this environment variable when deployed.

Enable the Resource Manager API

If you haven’t already, make sure to enable the Resource Manager API within your Google Cloud project.

Provide Resource Manager IAM access

You will need to provide the resourcemanager.projects.get role to the appropriate Cloud Run service account. The instructions here use the Compute Engine default service account. If you are running locally, you’ll also need to provide these permissions to your account.

Redeploy to Cloud Run

Use the same gcloud run deploy command as before to redeploy your updated application:

gcloud run deploy cloud-rust-example \
    --source . \
    --region us-central1 \
    --allow-unauthenticated

Now, when you visit the service URL provided by Cloud Run and navigate to the /project path, you should see information about your Google Cloud project.

Project information output from /project route

Conclusion

This guide demonstrates the process of deploying a Rust Axum application on Cloud Run. I started with a basic “Hello, World!” example from the Axum repository, explained its code, and then showed how to enhance it for Cloud Run compatibility by dynamically configuring the port and creating a Dockerfile. By combining Rust and Axum with Cloud Run’s serverless simplicity, you can efficiently build and deploy robust web services. The sample source code is available in the cloud-rust-example repository.

For more information about Cloud Run, I recommend the quickstart for building and deploying a web application in the documentation. Also, check out this video for a video walkthrough of running Rust on Cloud Run. Feel free to connect on LinkedIn, X, and Bluesky to continue the discussion!

AI Appraiser: Discover the value of your items with Gemini on Google Cloud

Karl Weinmeister — Sun, 09 Mar 2025 04:01:51 +0000

While you were out shopping or cleaning up around the house, have you ever wondered what an item is worth? Estimating the value of items can be tricky, often requiring expert knowledge or time-consuming research. What if you could get a quick, AI-powered appraisal with just a picture?

That’s the idea behind AI Appraiser, a small project I recently built using the Google Gen AI SDK. Whether you’re assessing the value of a beloved collectible, figuring out a fair price for secondhand goods, or simply curious about the worth of everyday objects, AI Appraiser offers a user-friendly way to get AI-powered insights.

Appraising a Google Chromecast remote with AI Appraiser

From curiosity to code

The project started with a question: could I leverage the power of Gemini’s multimodal capabilities and its integrated search to build a practical tool?

I was particularly interested in exploring Gemini’s grounding with Google Search capabilities. This feature seemed perfect for fetching up-to-date pricing information directly from the web, which is crucial for accurate item valuation. By grounding the AI’s analysis in real-time search results, I aimed to build an app that provides more reliable and informed estimates.

To bring AI Appraiser from a concept to reality, I focused on the user experience. Putting myself in the shoes of someone wanting to quickly check the value of an item, I thought, “Okay, what’s the first thing I’d naturally do?” The answer was: show the app the item. That’s why I started by focusing on the image upload feature, really trying to make it feel as smooth and effortless as possible. Drag and drop, click to upload — I wanted it to feel completely natural, like you were just showing the app what you had.

But even with a smooth image uploader, I knew that relying on pictures alone would be limiting, as photos don’t always tell the whole story. Think about those subtle details that can swing an item’s value wildly — the pristine condition of a vintage collectible, the telltale model number on a piece of tech, or those unique markings that authenticate a piece of art. To bridge this gap, I knew I needed to give users a way to add their own insights. That’s why I made sure to include an optional text description box.

Integrating with the Google Gen AI SDK is the heart of the application. All of the key pieces are documented in the Getting Started notebook:

Passing in multimodal image data (the image and optional description)
Enabling Grounding with Search to improve the valuation quality
Enabling controlled generation to ensure the response format is consistent

With Gemini’s grounded search, you can think of the application as a research assistant that can instantly scour countless sources across the web, pulling in pricing data, market trends, and comparable listings.

Beyond that, I didn’t want AI Appraiser to just spit out a valuation and leave you guessing. It wanted it to show you Gemini’s reasoning, laying out the key factors it considered and providing direct links to the sources it used. This way, you’re not just getting a number; you’re getting a glimpse into the AI’s thought process, and you can judge for yourself whether the estimate makes sense.

Architecture: Keeping it Simple and Serverless

For this project, I wanted to keep the architecture straightforward and leverage serverless technologies for ease of deployment and scalability. Here’s the basic setup:

Frontend: Built with HTML, Tailwind CSS, and HTMX. HTMX is a great little library that allows for dynamic UI updates without writing complex JavaScript, perfect for a project like this. Tailwind CSS helped with rapid styling, and plain HTML provided the structure.
Backend: A FastAPI application in Python. FastAPI is excellent for building APIs quickly and efficiently. It handles image uploads, interacts with Google Cloud Storage (optionally), and calls the Gemini API.
Gemini API: The star of the show! Gemini handles the heavy lifting of image analysis, search, and valuation.

Here’s a simplified diagram:

Architecture diagram of AI Appraiser on Google Cloud

Key features in action

Image Upload and Preview

The frontend provides a drag-and-drop interface for image uploads. It uses the Dropzone file input component from the Flowbite CSS library. HTMX handles the asynchronous upload to the backend, and the image is immediately displayed for preview.

<input
   id="image_file"
   name="image_file"
   type="file"
   class="hidden"
   hx-post="/upload-image"
   hx-target="#image-preview"
   hx-encoding="multipart/form-data"
/>

AI-Powered valuation with Gemini

The backend uses the Gemini API, specifically leveraging its grounding with Google Search capability. As highlighted in the documentation, this feature allows Gemini to augment its responses with real-time information from Google Search. The prompt instructs Gemini to act as a professional appraiser, using search to find comparable items and provide a reasoned valuation.

valuation_prompt = f"""You are a professional appraiser, adept at determining the value of items based on their description and market data.
Here is additional information provided by the user: {description}.
Your task is to estimate the item's fair market value
... prompt continues ..."
"""
google_search_tool = Tool(google_search=GoogleSearch())
config_with_search = GenerateContentConfig(tools=[google_search_tool])

response_with_search = client.models.generate_content(
    model=MODEL_ID,
    contents=[
        Part.from_uri(file_uri=image_uri, mime_type=guess_type(image_uri)[0]),
            valuation_prompt,
    ],
    config=config_with_search,
)

Structured output and display

The Gemini API response is parsed, and now the estimated value, reasoning, and source URLs conform to a consistent data structure using controlled generation. HTMX then handles updating the results section dynamically.

class ValuationResponse(BaseModel):
    estimated_value: float
    currency: str
    reasoning: str
    search_urls: list[str]

# ...

config_for_parsing = GenerateContentConfig(
    response_mime_type="application/json",
    response_schema=ValuationResponse
)

Deployment to Cloud Run

Deploying AI Appraiser is very straightforward with Cloud Run. The Dockerfile is included in the repository, and deployment is as simple as running:

gcloud run deploy ai-appraiser \
  --source . \
  --region $LOCATION \
  --allow-unauthenticated \
  --set-env-vars="GOOGLE_CLOUD_PROJECT=$GOOGLE_CLOUD_PROJECT,STORAGE_BUCKET=$STORAGE_BUCKET,MODEL_ID=$MODEL_ID"

Cloud Run handles containerization, deployment, scaling, and infrastructure management, allowing you to focus on the application code.

Try it out and contribute!

AI Appraiser is a fun little project that demonstrates the power of combining Gemini with Google Cloud. It’s not meant for professional appraisals, but it can be a handy tool for getting a quick estimate or just satisfying your curiosity.

The code is available on GitHub: https://github.com/kweinmeister/ai-appraiser

Feel free to clone the repository, deploy it to your own Google Cloud project, and experiment with it. Contributions and feedback are welcome! What other creative applications can we build with Gemini and Google Cloud? Let’s continue the conversation on LinkedIn, X, and Bluesky.

Attention Evolved: How Multi-Head Latent Attention Works

Karl Weinmeister — Sun, 02 Mar 2025 08:48:32 +0000

Compressing keys and values to reduce the cache size is MLA’s key innovation

Attention is the fundamental mechanism that enables LLMs to capture long-range dependencies and context. It was introduced as part of the Transformer architecture in 2017 in Attention is All You Need. As models have grown in size, new variants have emerged to address the computational cost of standard Multi-Head attention.

Multi-Head Latent Attention (MLA), introduced in the Deepseek-V2 paper, represents a novel approach to efficient attention. MLA introduces a low-rank compression technique that reduces the memory footprint without sacrificing model performance. MLA is used by the DeepSeek-V3 and DeepSeek-R1 models available in the Vertex Model Garden and deployable to Google Kubernetes Engine.

In this blog post, we’ll start with the standard attention mechanism and build up to what makes Multi-Head Latent Attention special.

Attention

To illustrate how single-head attention works, consider this example sentence:

The animal didn’t cross the street, because it was too tired.

How can we understand what “it” means? We need to look at surrounding words, or more precisely, tokens. Attention allows us to analyze this context mathematically.

Self-attention for pronoun “it” (source)

There are three components that make attention work: queries, keys, and values. We compare the Query word (“it”) to each Key in the sentence using the dot product. The dot product measures how “similar” two vectors are. A higher dot product means more “attention” or relevance. This is reflected in the QKᵀ term of the attention calculation. A scaling factor, based on the dimension of the keys, is applied to keep the dot products from becoming too large.

We then use these scores to weigh the Value vectors, which hold the actual information associated with each key. Words that are more “relevant” get more weight in understanding our Query word. This weighted sum of these Value vectors becomes the Attention Output — a context-aware representation of our word “it”.

Attention mechanism overview (source)

Positional Encodings

Before we move on, there’s one more crucial piece to the puzzle: positional encodings. So far, we haven’t considered the order of words in the sentence. Without this information, “Cross the street” and “Street the cross” would be treated identically because attention, by itself, is order-agnostic.

To address this, we add positional encodings. These are special vectors that tell the attention mechanism the position of each word in the sequence. In Attention is All you Need, sinusoidal positional encodings were used. They create a unique “position signature” for each word with sine and cosine functions of different frequencies.

Visualization of positional encodings (source)

These positional encodings are added to the input word representations before we calculate Queries, Keys, and Values. Altogether, the attention mechanism is now position-aware.

While much of the original Transformers architecture has stood the test of time, a new approach to positional encodings is now commonly used. Rotary Position Embeddings, or RoPE, was introduced in 2021 in the RoFormer architecture. Rather than adding a position term, RoPE rotates the query and key vectors based on their relative positions. This rotation allows the model to understand the relationship between words based on their distance from each other.

Illustration of position embedded into a vector through rotation

For a more detailed understanding of positional encoding techniques, I recommend the Designing Positional Encoding blog post.

Multi-Head Attention

So far, we’ve heard about attention mechanics using one set of Query, Key, and Value projections. To capture more nuanced relationships, we can use Multi-Head Attention or MHA. MHA uses multiple sets of QKV projections — “Head 1”, “Head 2”, “Head 3”, and so on. Each head learns to focus on different aspects of the relationship between words. For example, one head might focus on grammatical relationships and another on semantic relationships like synonymy or antonymy. Each head calculates its own attention output, and MHA concatenates these outputs and projects the resulting vector to obtain the final output.

Multi-Head attention calculation (source)

As Key and Value tensors for earlier tokens in a sequence remain the same, they can be cached to avoid unnecessary computations. This key-value (KV) cache can become a memory bottleneck, slowing down inference, especially for longer texts.

Illustration of how the KV Cache grows with each token in the sequence

Multi-Query and Grouped-Query Attention

Fortunately, new techniques have emerged to address this issue, by reducing the number of keys and values.

Let’s look at Multi-Query Attention, or MQA, first. Instead of each Query head having its own Key and Value set, all Query heads share a single Key and Value set. Because the KV cache size is a function of the dimensions of each head, MQA can significantly reduce the cache size.

However, there’s a trade-off. By sharing a single Key and Value set, model performance can suffer. Grouped-Query Attention, or GQA, is a middle ground. Instead of one shared Key and Value in MQA, it uses a small number of shared Key and Value sets, called “groups”.

Introducing MLA

Ideally, we want to shrink the KV Cache without sacrificing performance. And that”s where MLA, or Multi-Head Latent Attention comes in.

KV Cache Comparison (source)

MLA tackles this KV Cache problem with compression. MLA compresses, or down-projects, keys and values into a smaller low-rank matrix. This compressed matrix is then up-projected during the attention calculation. We can see how a compressed latent matrix in lower-dimensional space is derived with a down-projection matrix Wᴰᴷⱽ. Keys and values can be up-converted with Wᵁᴷ and Wᵁⱽ, respectively.

Calculation of compressed latent matrix, keys, and values

MLA requires a modified approach to address token positions called decoupled RoPE. Standard RoPE directly modifies compressed keys and values with position information. MLA’s compression technique complicates this and would hinder inference efficiency. Instead, with decoupled RoPE, the relative position information is encoded in separate vectors that are then concatenated with the compressed keys and values before the up-projection. This allows for efficient application of positional information without interfering with the compression/decompression process.

Query (q) and Key (k) vectors are formed by concatenating their compressed (C) and positional (R) parts

In the end, Multi-Head Latent Attention provides faster inference and smaller memory usage. According to the Deepseek-V2 paper, this approach maintains or even improves performance compared to Multi-Head attention. Compressing the keys and values to a low-rank representation apparently doesn’t lose too much information, and may even aid generalization.

To try out MLA in action, you can deploy DeepSeek-R1 or DeepSeek-V3 671B on Google Kubernetes Engine (GKE) using graphical processing units (GPUs) across multiple nodes with this guide. You can also deploy DeepSeek from the Vertex Model Garden. Feel free to connect on LinkedIn, X, and Bluesky to continue the discussion!