DEV Community: Google AI

Build a Long-Running Agent in the Cloud for $5.70/Month

Shir Meir Lador — Thu, 03 Sep 2026 15:54:57 +0000

How do you run an autonomous AI agent in the cloud 24/7 for just $5.70 a month?

I recently wanted to build a background worker with persistent disk storage and an instant web dashboard, but I didn't want the headache of managing a virtual machine or paying a massive monthly bill.

If you are building long-running agents, you know this exact cloud hosting dilemma:

Standard serverless (like Cloud Run services or Lambda): When traffic stops, the container scales to zero — instantly killing your background loops and wiping your agent's active memory (RAM). On the flip side, a sudden traffic spike spins up multiple containers that can overwrite each other's state files and corrupt your data. (Note: Save state using JSON or Markdown files. Avoid SQLite, as Cloud Run volume mounts)
A regular virtual machine (like EC2 or Compute Engine): Keeps your agent running 24/7, but a standard 1-vCPU machine typically costs $15 to $25 a month even when idle. Even if you use a heavily-throttled fractional VM for $7/month, you are still stuck with the full infrastructure management overhead.

Last year, I built a multi-agent Trend Spotter with ADK. It worked well, but I wanted to make it fully autonomous: a continuous, long-running agent that scans and summarizes tech feeds in the background without manual triggers or high hosting costs.

Google Cloud's new Cloud Run instances primitive solves this exact problem. It gives you a single, always-on container that runs 24/7, costs $5.70 a month on a shared CPU, provides a free HTTPS endpoint, and lets you mount cloud storage like a normal local disk.

Here is how to build and deploy a production long-running agent with this setup (you can follow along with the complete source code in the repo.

What are we building?

I want to stay up to date with what is happening in AI and agent engineering. But instead of manually opening 20 browser tabs across different websites every morning, I wanted to build my own long-running agent that updates me on recent news anytime I want.

Personal tech briefing agent UI

Here is what the agent does:

Runs continuously as a background daemon: Wakes up automatically every 30 minutes to collect fresh news. Note that Cloud Run instances restart automatically up to every 7 days, so your agent just needs to gracefully resume its schedule when restarted.
Scans Hacker News and other curated AI and agent engineering sources.
Accepts real-time alerts & mobile shares: Includes an inbound webhook (POST /api/webhook) so you can push breaking tweets, iOS Share Sheet links, or GitHub releases straight into the agent for instant summarization.
Filters the noise: Strips out paywalls, ads, and low-substance articles.
Summarizes with Gemini 2.5 Flash: We use Gemini 2.5 Flash to keep costs low. You can swap in the newer Gemini 3.5 or 3.6 Flash models if you need advanced reasoning, but note that their input tokens cost 5x as much compared to 2.5 Flash ($1.50 vs $0.30 per 1M tokens). For simple daily summarization, 2.5 Flash (or the equally cheap Gemini 3.5 Flash-Lite) is fast, highly capable, and keeps the monthly API bill to just a few cents.
Saves data safely: Stores the daily markdown briefing and seen URLs directly in a mounted cloud storage folder (/data).
Serves a clean web dashboard: Gives an instant web page to read your briefing or trigger a fresh run whenever you want.

How the system works

The whole application runs inside one Cloud Run instance:

Tech briefing agent architechture

What else can you build with a long-running agent?

A tech briefing agent is just one example. Because Cloud Run instances give you an always-on background worker, a free web endpoint, and safe local disk storage, you can use this exact same pattern for many developer workflows:

Persistent Slack, Discord, or Telegram Bot: A bot that maintains long-lived connections to chat gateways, answers developer questions, and syncs unresolved issues to your backlog.
Security & Vulnerability Watchdog: An agent that runs on an internal timer to monitor dependencies and CVE security feeds, caching vulnerability signatures on local disk.
DevOps Incident Triage Co-Pilot: An agent that receives incoming webhook alerts from monitoring tools, runs background log queries without timing out, and renders an instant root-cause dashboard.
Pull-Based Queue Worker: An agent that continuously pulls complex tasks from Pub/Sub, Kafka, or RabbitMQ, performs multi-step LLM reasoning, and writes results to storage.
Nightly CI/CD & Flaky Test Fixer: A background daemon that runs overnight test suites, analyzes test logs to spot flaky tests, and opens pull requests with automated fixes.

Why Cloud Run instances are great for agents

Standard serverless platforms are designed for quick web requests. They wait for a user to click a button, run for one second, and shut down.

Long-running background agents have different needs:

Comparison — Standard Serverless / Regular VMs / Cloud Run Instances

With an instance, you get the simplicity of serverless with the stability of a VM. Because your instance is always hot with a public HTTPS endpoint, it easily handles three trigger styles in one container:

Periodic Background Polling: Runs autonomously on an internal asyncio schedule without needing external cron services.
Instant Web Dashboard: Zero cold starts when you open the reading dashboard.
Real-Time Push Webhooks: An inbound POST /api/webhook route that lets you push breaking tweets, iOS share sheet links, or GitHub release alerts straight into the agent for immediate summarization.

When NOT to use this

Cloud Run instances are great for single-worker background agents. You should pick a different tool if you need:

Massive parallel batch jobs: If you need to process 10,000 documents at once across 100 parallel workers, use Cloud Run Jobs or GKE. An instance is a single worker.
High-traffic, bursty web APIs: If your website gets sudden spikes of millions of requests, use standard Cloud Run services so your app can automatically autoscale to hundreds of containers and scale down to zero when traffic stops.
Heavy local GPU model hosting: If you want to host an open 70B model directly inside your container on a dedicated H100 GPU, use GKE or Compute Engine. Cloud Run instances are built for CPU applications that connect to hosted models like Gemini.

Alternative architecture: Decoupled Job + Service

Instead of a single instance, you could build an event-driven system: a Cloud Scheduler triggers a Cloud Run Job for polling, while a scale-to-zero Cloud Run Service hosts the dashboard and listens for webhooks.

While this decoupled approach drops compute costs to virtually $0.00 in the free tier, you lose single-container simplicity. You are forced to manage multiple cloud services and message queues (to prevent concurrent webhooks from corrupting your state), while accepting cold starts on your web dashboard.

Compare instances with Decoupled Job + Service for this task

Deploy your long-running agent in 6 simple steps

You can deploy this setup to Google Cloud in about five minutes.

1. Turn on the cloud services

export PROJECT_ID="your-project-id"
export REGION="us-west1"
export BUCKET_NAME="${PROJECT_ID}-agent-data"
export REPO_NAME="agent-repo"
gcloud config set project $PROJECT_ID
gcloud services enable run.googleapis.com storage.googleapis.com artifactregistry.googleapis.com cloudbuild.googleapis.com secretmanager.googleapis.com

Note: Cloud Run instances are not available in every region. Please pick a supported region near you from the Cloud Run instances locations page.

2. Create a storage bucket for your data

gcloud storage buckets create gs://$BUCKET_NAME \
  --location=$REGION \
  --uniform-bucket-level-access

3. Build your container

gcloud artifacts repositories create $REPO_NAME \
  --repository-format=docker \
  --location=$REGION
gcloud builds submit \
  --tag ${REGION}-docker.pkg.dev/${PROJECT_ID}/${REPO_NAME}/tech-briefing-agent:latest .

4. Create a service account

gcloud iam service-accounts create briefing-agent-sa \
  --display-name="Briefing Agent SA"
gcloud storage buckets add-iam-policy-binding gs://$BUCKET_NAME \
  --member="serviceAccount:briefing-agent-sa@${PROJECT_ID}.iam.gserviceaccount.com" \
  --role="roles/storage.objectUser"

5. Store your API key securely

Never pass API keys in plain text. Store your Gemini API key in Google Cloud Secret Manager and grant your service account permission to read it:

echo -n "YOUR_GEMINI_API_KEY" | gcloud secrets create gemini-api-key \
  --data-file=- \
  --replication-policy="automatic"
gcloud secrets add-iam-policy-binding gemini-api-key \
  --member="serviceAccount:briefing-agent-sa@${PROJECT_ID}.iam.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor"

6. Launch the instance

gcloud beta run instances create tech-briefing-agent \
  --image=${REGION}-docker.pkg.dev/${PROJECT_ID}/${REPO_NAME}/tech-briefing-agent:latest \
  --region=$REGION \
  --port=8080 \
  --cpu=1 \
  --memory=1Gi \
  --public \
  --service-account=briefing-agent-sa@${PROJECT_ID}.iam.gserviceaccount.com \
  --add-volume mount-path=/data,type=cloud-storage,mount-options="uid=1000;gid=1000;file-mode=0700;dir-mode=0700",bucket=$BUCKET_NAME \
  --set-secrets "GEMINI_API_KEY=gemini-api-key:latest" \
  --set-env-vars "DATA_DIR=/data,POLL_INTERVAL_MINUTES=30"

We set --cpu=1 and --memory=1Gi to keep the cost at $5.70. If you omit these, it defaults to 2 CPUs and 2 GiB (~$11.40/month, see pricing table). To improve load times, you can increase the CPU and memory.

[!TIP] Adjust uid=1000;gid=1000 in the mount-options flag to match the specific non-root user ID defined in your Dockerfile, if different.

When this command finishes, Cloud Run gives you a live HTTPS web address. Open it in your browser to see your briefing dashboard.

What does this cost in real life?

Here is the real monthly bill for running this 24/7:

Monthly cost breakdown

For less than the price of two cups of coffee, you have a private agent running day and night.

Learn more about Cloud Run instances

Want to dive deeper into Cloud Run Instances? Check out these official Google Cloud resources:

Official Launch Blog: Introducing Cloud Run instances
Official Documentation: Create and manage Cloud Run instances
Hands-on Codelab: Deploying to Cloud Run instances Codelab
Source Code & ADK Graph: Tech-briefing-agent on GitHub

What is coming next?

Now that the hosting problem is solved, how do you make the agent smart and resilient? How do you stop it from summarizing noise when it hits a paywall, or build self-correcting reflection loops?

Join us in the next part where we will dive into graph engineering and the architecture of the agent using ADK 2.0.

Happy building!

Your First Multi-agent system: A Beginner's Guide to Building an AI Trend finder with ADK

Shir Meir Lador — Thu, 03 Sep 2026 15:53:27 +0000

Welcome back to our series on building the ultimate AI research assistant for our AI agent podcast! In our first post, we built a fantastic agent that could search the web to find the latest AI agent news for the agent factory podcast. But what if we want to add more specialized skills, like getting the real pulse from developer communities on Reddit? To do that, we need to upgrade our agent's design.

In this guide, we are going to level up our skills and refactor our simple agent into a powerful multi-agent system. We will build a "Manager" agent that directs a team of specialists, including one with a custom-built Reddit tool, to gather richer, more diverse insights.

By the end of this post, you'll have an even more powerful Trend Spotter agent that gets information from multiple sources. More importantly, you will learn the advanced skills needed to build complex agents with ADK. You will know how to:

Build a scalable multi-agent system.
Build a custom tool from any Python function (like our new Reddit tool).
Create an orchestrator agent that delegates tasks to a team of specialists.
Write advanced prompts to manage a multi-step, multi-tool workflow.
Debug a multi-agent system using the ADK's powerful Trace view.

This architecture is the key to unlocking your agent's full potential. Let's get started!

Our Multi agent system flow

Step 1: Get Reddit API Credentials & Install Library

To allow our agent to access Reddit programmatically, we need to get API credentials. This is free and only takes a minute.

Navigate to Reddit Apps: Log in to your Reddit account and go to the app preferences page: https://www.reddit.com/prefs/apps.
Create a New App: Scroll to the bottom and click the button that says "are you a developer? create an app…".
Fill out the form:

name: Trend Spotter Agent
Select the script option for the application type.
about url: You can leave this blank.
redirect url: You must enter http://localhost:8080 for this field.

Click create app. You will now be taken to a new page showing your credentials.

5. Set Environment Variables: For security, we'll store these credentials as environment variables.

Your client ID is the string of characters right under "personal use script".
Your client secret is the long string next to the secret label.
Open your terminal and run the following export commands:

export REDDIT_CLIENT_ID=”YOUR_CLIENT_ID”  
export REDDIT_CLIENT_SECRET=”YOUR_CLIENT_SECRET”  
export REDDIT_USER_AGENT=”TrendSpotterAgent/0.1 by u/YourUsername”

6. Update Dependencies: Add the praw library to your requirements.txt file and install it.

# requirements.txt  
google-adk  
praw

Install it from your terminal:

pip install -r requirements.txt

Configure Your Cloud Environment

If you haven't done so already, in our previous blog we showed how to define the settings to tell ADK how to securely connect to your Google Cloud account to use services like Vertex AI and Google Search.

export GOOGLE_GENAI_USE_VERTEXAI=true  
export GOOGLE_CLOUD_PROJECT=<your-gcp-project-id>  
export GOOGLE_CLOUD_LOCATION=<your-gcp-project-location>

Run this one-time command. It will open a browser for you to sign in, allowing ADK to make authorized requests on your behalf.

gcloud auth application-default login

Step 2: Create the Project Folder

To organize our new team of agents, we'll create a sub_agents directory inside our main trend_spotter package.

No worries if you haven't gone through the first blog post, we got you! here is how you create the folder structure for your main agent:

# Create the main project folder  
mkdir trend-spotter && cd trend-spotter  
# Create the Python package folder that will hold our code  
mkdir trend_spotter  
touch trend_spotter/__init__.py  
touch trend_spotter/agent.py  
touch trend_spotter/prompt.py  
# Create the top-level configuration files  
touch pyproject.toml requirements.txt  
# Finally, create and activate a virtual environment  
python3 -m venv venv && source venv/bin/activate

(On Windows, use python -m venv venv && .\venv\Scripts\activate)

After you have the main agent and your folder structure defined, from your trend-spotter root folder, run:

# Create the sub-agents directory and its Python initializer  
mkdir trend_spotter/sub_agents  
touch trend_spotter/sub_agents/__init__.py

Step 3: Create the Specialist Sub-Agents

Now we'll build our two specialist agents by directly creating instances of the Agent class.

1. The Google Search Specialist:

Create a new file: trend_spotter/sub_agents/google_search_agent.py
Add this code. Note how we directly create the google_search_agent variable.

# trend_spotter/sub_agents/Google Search_agent.py  
from google.adk.agents import Agent  
from google.adk.tools import google_search  

MODEL = "gemini-2.5-pro-preview-05-06"  

# A specific, structured prompt to control the output format of this sub-agent.  
google_search_SUB_AGENT_PROMPT = """  
**Role:**  
- You are a specialist Research Assistant.  
- Your only purpose is to execute a Google Search based on instructions from your manager and return the raw, structured results.  

**Tools:**  
- You have access to one tool: `Google Search`.  

**Context:**  
- You will be given a query by a manager agent.  
- Your output will be read by another agent, so it must be clean, predictable, and structured.  
- You must not summarize, analyze, or interpret the search results. Your job is only to find and format the information directly from the tool's output.  

**Task:**  
1.  Take the search query provided to you.  
2.  Execute a search using the `Google Search` tool.  
3.  Format the raw output from the tool into a list, following the **exact** `Output Format` specified below.  

**Output Format:**  
For each search result, you MUST provide the Title, Link, and Snippet. Each complete result must be separated by '---'.  

---  
Title: [Title of the first search result]  
Link: [Full URL of the first search result]  
Snippet: [Snippet text of the first search result]  
---  
Title: [Title of the second search result]  
Link: [Full URL of the second search result]  
Snippet: [Snippet text of the second search result]  
---  
(and so on for all results)  
"""  

google_search_agent = Agent(  
    model=MODEL,  
    name="google_search_agent",  
    description="An expert at using google_search to find recent information and return a structured list of results including URLs.",  
    # We assign the new, structured instruction here.  
    instruction=google_search_SUB_AGENT_PROMPT,  
    tools=[google_search]  
)

2. The Reddit Specialist:

First, create a new file for our custom tool's code: trend_spotter/tools.py. Add the following function to it.

import os  
import praw  

# The function now accepts a LIST of subreddit names  
def search_hot_reddit_posts(subreddit_names: list[str], limit_per_subreddit: int = 5) -> str:  
    """  
    Searches a list of subreddits for their current hot posts and returns their titles and URLs.  

    Args:  
        subreddit_names: A list of subreddit names to search (e.g., ["LocalLLaMA", "MachineLearning"]).  
        limit_per_subreddit: The number of top posts to retrieve from each subreddit.  

    Returns:  
        A dictionary containing the status and a list of formatted post strings.  
    """  
    try:  
        print(f"\n🔎 Searching Reddit for hot posts in: {', '.join(subreddit_names)}...")  

        reddit = praw.Reddit(  
            client_id=os.environ["REDDIT_CLIENT_ID"],  
            client_secret=os.environ["REDDIT_CLIENT_SECRET"],  
            user_agent=os.environ["REDDIT_USER_AGENT"],  
            read_only=True,  
        )  

        all_posts = []  
        # Loop through each subreddit name provided in the list  
        for sub_name in subreddit_names:  
            print(f"  - Fetching from r/{sub_name}...")  
            subreddit = reddit.subreddit(sub_name)  
            for post in subreddit.hot(limit=limit_per_subreddit):  
                # We can add a simple filter here if we want, e.g., for score  
                if post.score > 5:  
                    all_posts.append(f"Title: {post.title}\nLink: {post.url}")  

        if not all_posts:  
            return "No hot posts found meeting the criteria in the specified subreddits."  

        print(f"✅ Reddit search complete. Found {len(all_posts)} qualifying posts.")  
        return "\n---\n".join(all_posts)  

    except Exception as e:  
        return f"Error searching Reddit: {e}"

Now, create the Reddit agent itself at trend_spotter/sub_agents/reddit_agent.py:

# trend_spotter/sub_agents/reddit_agent.py  
from google.adk.agents import Agent  
from trend_spotter.tools import search_hot_reddit_posts  
MODEL = "gemini-2.5-pro-preview-05-06"  

reddit_agent = Agent(  
    name="reddit_agent",  
    model=MODEL,  
    description="An expert at finding hot posts on specific Reddit subreddits using its tool.",  
    tools=[search_hot_reddit_posts]  
)

Step 4: Build the Main Orchestrator Agent

Now we'll modify our main agent from Part 1 to become the "manager" of our new specialist team.

Open trend_spotter/prompt.py and replace its contents with this new orchestrator prompt:

# trend_spotter/prompt.py  

ORCHESTRATOR_PROMPT = """  
**Role:**  
- You are the highly-capable manager of an AI research team.  
- Your purpose is to produce a high-quality, detailed intelligence report for the "The Agent Factory" podcast.  
- Your focus is exclusively on developments in AI agents that are impactful and relevant to software developers.  

**Tools:**  
- You have a team of two specialist agents available to you as tools:  
  1. `google_search_agent`: An expert at performing general web searches for news, releases, and technical articles.  
  2. `reddit_agent`: An expert at finding real, hands-on developer conversations on specific subreddits.  

**Context:**  
- You must synthesize information from BOTH the `google_search_agent` and the `reddit_agent` to form your conclusions.  
- Your primary filter for all information is its direct and significant impact on developers. Discard anything that is purely business-focused or marketing fluff.  
- Topics that appear in multiple sources (e.g., in both tech news and on Reddit) should be considered more important and prioritized in your report.  
- The final report must be structured exactly as described in the Task section.  

**Task:**  
1.  **Discover the Current Date:** Your very first action is to delegate to your `google_search_agent`. Instruct it to find the current date.  
2.  **Delegate Focused Research:**  
    - Based on the date, calculate the start and end dates for the last 7 days.  
    - Instruct the `google_search_agent` to find news about new open-source agent frameworks, updates to popular libraries (like LangChain, ADK, CrewAI or LlamaIndex), and technical tutorials about building agents within the calculated date range using `after:YYYY-MM-DD` and `before:YYYY-MM-DD` operators.  
    - Instruct the `reddit_agent` to find the hottest developer conversations about practical challenges, new techniques, and opinions on new tools from subreddits like "LocalLLaMA", "MachineLearning", "LangChain", "AI_Agents", "LLMDevs", and "singularity".  
3.  **Synthesize and Create the Final Report:**  
    - Review the information provided by **both** specialist agents.  
    - Combine, filter, and deduplicate the findings. Your primary filter is to **only select topics that have a direct and significant impact on developers.**  
    - **Pay special attention to topics that appear in multiple source in the general web search and on Reddit**, as these are likely the most important and should be prioritized.  
    - The report **must begin with a header** specifying the date range used.  
    - The body of the report must have exactly three sections as detailed below.  
    - For each item, you **must provide four pieces of information**: a 1-2 sentence explanation, an indented "Developer Impact" analysis, a "Prioritization Rationale", and a verifiable source URL.  

**Final Report Format:**  

**🔥 Top 5 Trends for Agent Developers**  
1.  **[Trend 1 Name]**: [A 1-2 sentence explanation of this trend.]  
    **(Source: [URL])**  
    * **Developer Impact**: [A 1-sentence explanation of why this matters to developers.]  
    * **Prioritization Rationale**: [A 1-sentence explanation of why this topic was selected, e.g., "High volume of discussion on Reddit and mentioned in multiple tech articles."]  
2.  ... (up to 5 total)  

**🚀 Top 5 Releases for Agent Developers**  
1.  **[Release 1 Name]**: [A 1-2 sentence explanation of the tool, framework, or model.]  
    **(Source: [URL])**  
    * **Developer Impact**: [A 1-sentence explanation of why this matters to developers.]  
    * **Prioritization Rationale**: [A 1-sentence explanation of why this topic was selected.]  
2.  ... (up to 5 total)  

**🤔 Top 5 Questions from Agent Developers**  
1.  **[Question 1 Topic]**: [A 1-2 sentence explanation of what developers are asking.]  
    **(Source: [URL])**  
    * **Developer Impact**: [A 1-sentence explanation of why this matters to developers.]  
    * **Prioritization Rationale**: [A 1-sentence explanation of why this topic was selected.]  
2.  ... (up to 5 total)  
"""

Open trend_spotter/agent.py and replace its contents to turn it into the orchestrator. Notice how we now import the agent instances we created.

# trend_spotter/agent.py  
from google.adk.agents import LlmAgent  
from google.adk.tools.agent_tool import AgentTool  
# Import the sub-agent INSTANCES  
from .sub_agents.google_search_agent import google_search_agent  
from .sub_agents.reddit_agent import reddit_agent  
from . import prompt  
MODEL = "gemini-2.5-pro-preview-05–06"  
# This is our main "manager" agent, now an LlmAgent  
root_agent = LlmAgent(  
model=MODEL,  
name="TrendSpotterOrchestrator",  
description="The manager of a team of specialist AI agents.",  
instruction=prompt.ORCHESTRATOR_PROMPT,  
# The Orchestrator's "tools" are its sub-agents, wrapped in AgentTool  
tools=[  
AgentTool(agent=google_search_agent),  
AgentTool(agent=reddit_agent)  
],  
)

Step 5: Running Your Multi-Agent System

The run process remains the same. The adk web tool will automatically load your root_agent, which is now the orchestrator.

Ensure your pyproject.toml file still correctly points to your main agent:

[project]  
name = "trend_spotter"  
version = "0.1.0"  
# Setuptools will now auto-discover your trend_spotter package.  
[tool.adk.agents]  
trend_spotter = "trend_spotter.agent:root_agent"

Install your updated package with the new dependencies:

pip install -e .

Launch the web interface:

adk web

In the "Event" tab of the web UI, you will now see the main orchestrator agent making calls to its new google_search_agent and reddit_agent sub-agents. You've successfully built a more robust and scalable AI system using the correct ADK patterns!

You should get an output like this:

Report for the week of June 4, 2025 - June 11, 2025  

🔥 Top 5 Trends for Agent Developers  

Enterprise Adoption of Agent Frameworks: Frameworks like LangChain are showing significant growth in enterprise contexts, indicated by surpassing the OpenAI SDK in Python downloads and new integrations with systems like SAP and Salesforce. (Source: news.knowledia.com - "LangChain Surpasses OpenAI SDK in Monthly Python Downloads" - This specific URL was not in the provided search but is representative of the type of news article that would contain this information from the search summary "LangChain: Surpassed the OpenAI SDK in monthly Python downloads as of June 2025")  
Developer Impact: Developers using or learning LangChain can expect more mature tooling and greater demand for their skills in enterprise settings.  
Prioritization Rationale: This trend is based on download statistics and enterprise integration news, indicating a major shift in the landscape relevant to developers' career opportunities.  
Enhanced Observability and Debugging in Agent Development: There's a clear move towards more sophisticated tools for monitoring, debugging, and managing the cost of AI agents, exemplified by Langfuse's integration with LlamaIndex and new features in LangSmith. (Source: https://news.llamaindex.ai/integrating-langfuse-for-observability-and-debugging-in-llamaindex-5f4f7f8c29ac [Example URL based on search "An article from June 6, 2025, discusses integrating Langfuse with LlamaIndex for observability and debugging."])  
Developer Impact: Developers will have better tools to understand agent behavior, optimize performance, and control operational costs.  
Prioritization Rationale: Addresses a critical developer need for building robust and maintainable agents, highlighted by multiple framework updates.  
Advanced Structured Data Handling by Agents: Agents are becoming more adept at working with structured data, with developments like Microsoft AutoGen's Structured Retrieval Augmentation and LlamaIndex's Spreadsheet Agent. (Source: Daily AI Agent News - "Microsoft AutoGen v0.4: Structured Retrieval Augmentation & MCP" - this URL was not in the search results but the information "Microsoft AutoGen v0.4 implements Structured Retrieval Augmentation" was.)  
Developer Impact: This allows developers to build agents for more complex, real-world use cases involving databases, spreadsheets, and other structured formats.  
Prioritization Rationale: Represents a significant expansion in agent capabilities, enabling new applications and increasing their utility.  
Growth of Local and Open-Source Agent Solutions: New open-source frameworks like "Goose" (from Block) and "OpenHands" emphasize local execution and customization, aligning with developer discussions on platforms like Reddit (r/LocalLLaMA) about gaining more control over their LLM setups. (Source: https://www.reddit.com/r/LocalLLaMA/comments/1l8pem0/i_finally_got_rid_of_ollama/ and news articles on Goose/OpenHands release)  
Developer Impact: Provides developers with more options for privacy-centric, cost-effective, and highly customizable agent development.  
Prioritization Rationale: Supported by both new tool releases in the general tech news and active discussions within the developer community (Reddit).  
Rise of Multi-Agent Systems and Interoperability Standards: The development of orchestrators like Fujitsu's and advancements in frameworks such as CrewAI and AutoGen (with MCP support) highlight a focus on complex systems where multiple agents collaborate. LangGraph is also central to this trend. (Source: Tech news article on "Fujitsu's Agentic Workflow Orchestrator" and https://www.reddit.com/r/LangChain/comments/1l8zy42/built_a_texttosql_multiagent_system_with/ )  
Developer Impact: Developers are increasingly tasked with designing, building, and managing interactions between multiple specialized agents, requiring new skill sets.  
Prioritization Rationale: This is a key area of innovation, mentioned in multiple framework updates and discussed by developers building sophisticated applications.  

🚀 Top 5 Releases for Agent Developers  

LangGraph Platform General Availability: LangChain's LangGraph, a library for creating stateful, multi-actor LLM applications, is now generally available, offering features like 1-click deployment. (Source: LangChain official blog/documentation - the search mentioned "LangGraph Platform is now generally available" as a LangChain update.)  
Developer Impact: Offers a production-ready path for developers to build complex, scalable agentic systems and multi-agent collaborations.  
Prioritization Rationale: Significant release for a popular ecosystem, addressing the need for robust multi-agent system development.  
AutoGen v0.4 (Microsoft): This version introduces Structured Retrieval Augmentation and full support for the Model Context Protocol (MCP), enhancing data handling and enabling interoperability with other agent platforms. (Source: GitHub releases page for Autogen or news articles covering the v0.4 release mentioned in the search: "Microsoft AutoGen v0.4 implements Structured Retrieval Augmentation and full support for the open-source MCP standard")  
Developer Impact: Allows developers to build more powerful AutoGen agents capable of complex data interactions and cross-platform collaboration.  
Prioritization Rationale: Major update to a key framework from Microsoft, improving core functionality and ecosystem compatibility.  
CrewAI v0.126.0: Features real-time task redistribution, Python 3.13 support, persisted tools from a Tool repository, streamable-http transport in MCP, and major documentation restructuring. (Source: CrewAI GitHub releases or their official blog - "CrewAI: Released v0.126.0 on June 5, 2025" was in the search results.)  
Developer Impact: Provides developers with improved performance, better tool management, enhanced interoperability, and easier onboarding through better documentation.  
Prioritization Rationale: A substantial update to a growing agent framework, focusing on efficiency and developer experience.  
LlamaIndex Spreadsheet Agent & Llama Cloud Updates: LlamaIndex launched a production-ready Spreadsheet Agent for natural language Q&A on spreadsheets and announced new MCP integration and Llama Cloud enhancements. (Source: LlamaIndex Newsletter/Blog - "LlamaIndex: ...announcing a new production-ready Spreadsheet Agent...new MCP (Model Context Protocol) integration..." was in the search results.)  
Developer Impact: Delivers a practical tool for a common business analytics task and improves the deployability and connectivity of LlamaIndex agents.  
Prioritization Rationale: Addresses specific, high-value developer use cases (spreadsheet interaction) and improves platform capabilities.  
Goose (Open-Source AI Agent Framework by Block): An extensible AI agent framework from Block designed to run entirely locally, capable of writing/executing code and interacting with the file system. (Source: News articles covering the Goose release - "Goose: Released by Block (formerly Square), Goose is an open-source AI agent framework..." was in the search results.)  
Developer Impact: Gives developers a new, powerful, and locally controllable tool for building agents, especially those focused on coding tasks.  
Prioritization Rationale: A new entrant in the open-source framework space from a well-known company, catering to local-first development.  
🤔 Top 5 Questions from Agent Developers  

Getting Started with AI Agent Development: Newcomers are actively seeking guidance on initial steps, choosing foundational frameworks (LangChain, CrewAI, AutoGen frequently cited), and identifying suitable beginner projects. (Source: Reddit thread from June 5, 2025, providing advice for beginners, or a general link like https://www.reddit.com/r/AI_Agents/ if specific links are too numerous. The search mentioned: "A Reddit thread from June 5, 2025, provides advice for beginners...")  
Developer Impact: Highlights a need for more structured learning paths and accessible resources for developers new to building AI agents.  
Prioritization Rationale: A foundational question indicating growing interest and the need for community/educational support, seen in both search and Reddit results.  
Optimizing and Controlling Local LLM Setups: Developers on subreddits like r/LocalLLaMA are discussing practical challenges and alternatives for local LLM environments (e.g., moving from Ollama to llama.cpp/OpenWebUI). (Source: https://www.reddit.com/r/LocalLLaMA/comments/1l92vr0/as_some_people_asked_me_to_share_some_details/)  
Developer Impact: Reflects the hands-on effort by developers to fine-tune their local development stacks for better performance, control, or feature sets.  
Prioritization Rationale: A practical, developer-driven discussion on Reddit about tooling and local environment optimization.  
Implementing Text-to-SQL Multi-Agent Systems: There is active exploration and sharing of projects on building multi-agent systems for complex database interaction tasks like Text-to-SQL, particularly using LangGraph. (Source: https://www.reddit.com/r/LangChain/comments/1l8zy42/built_a_texttosql_multiagent_system_with/)  
Developer Impact: Shows developers are pushing the capabilities of agents to tackle sophisticated, high-value enterprise tasks with multi-agent designs.  
Prioritization Rationale: Represents advanced application development discussed within the LangChain developer community.  
Seeking Open Source Alternatives for Observability Tools: Developers are interested in and building open-source options for agent observability, such as alternatives to LangSmith, including LangGraph visualization. (Source: https://www.reddit.com/r/LangChain/comments/1l93195/open_source_langsmith_alternative_with_langgraph/)  
Developer Impact: Indicates a community drive for more accessible and customizable tools for monitoring and understanding agent behavior.  
Prioritization Rationale: Highlights a specific tooling need and community-led solutions on Reddit, relevant for the LangChain ecosystem.  
Ensuring Code Quality in AI Training Data: A discussion on r/LLMDevs raises the question of how to ensure AI agents learn from high-quality code, suggesting approaches like using "gold standard files" rather than random code. (Source: https://www.reddit.com/r/LLMDevs/comments/1l8yweo/devs_stop_letting_ai_learn_from_random_code_use/)  
Developer Impact: This points to an evolving concern about data integrity and best practices when fine-tuning or training agents for coding tasks.  
Prioritization Rationale: An important discussion on Reddit regarding the quality and reliability of AI-assisted software development.

Part 6: Next Steps and Conclusion

Congratulations! You have successfully upgraded your simple agent into a powerful, multi-agent system using the Agent Development Kit's orchestrator pattern. This is a huge step in your journey as an agent developer.

You've now learned some of the most important skills for building complex AI applications:

How to create specialist sub-agents
Build a custom tool from any Python function
Design a manager agent that orchestrates an entire team to solve a problem.

This is how real-world, scalable agentic systems are built.

But this is just the beginning. You now have a truly powerful foundation that you can expand upon. Think about what other specialists you could add to your team — agents/tools to fetch additional sources of information? an agent that saves the report to a Google Doc? A tool that posts the summary to Slack or email? The possibilities are endless.

When you're ready to dive deeper and explore all the advanced features the framework has to offer, the best place to go is the official Google Cloud ADK documentation.

Happy building!

Your First AI Agent: A Beginner's Guide to Building an AI Trend finder with ADK

Shir Meir Lador — Thu, 03 Sep 2026 15:52:29 +0000

The world of AI agents is moving at lightning speed. Keeping up with the latest trends, new open-source tools, and important developer conversations can feel like a full-time job. We're facing this exact challenge as we prepare to launch our new podcast for agent developers, which will be dedicated to all things AI agents (stay tuned!). To make sure we're ready for each episode, we wanted to create an automated way to get up-to-date with the news that matters.

In this guide, we are going to walk you step-by-step through building your very first AI agent on Google Cloud using the open-source Agent Development Kit (ADK). We will design a "Trend Spotter" agent whose mission is to act as your personal AI analyst, teaching it to scan the web and sift through the noise to find what truly matters.

By the end of this post, you will have a practical, working tool that automatically creates a concise intelligence report to keep you up-to-date, saving you hours of manual research. More importantly, you will learn the fundamental skills to build your own agents with ADK. You will know how to:

Structure a simple, powerful ADK agent as a proper Python package.
Write a detailed prompt to define your agent's logic and workflow.
Provide your agent with tools like Google Search.
Set up, test, and run your agent locally using the adk web interface.
Deploy your agent to Cloud Run.

Let's get started.

Part 1: Setup and Configuration

This setup uses a standard package structure that allows the ADK tools to discover and run our agent without a main.py file.

Step 1.1: Prerequisites

Python 3.11+
Google Cloud CLI: Follow the official installation guide here.
Google Cloud Account.

Step 1.2: Create Your Project Structure

Open your terminal. Create the following folder structure and virtual environment.

# Create the main project folder  
mkdir trend-spotter && cd trend-spotter  
# Create the Python package folder that will hold our code  
mkdir trend_spotter  
touch trend_spotter/__init__.py  
touch trend_spotter/agent.py  
touch trend_spotter/prompt.py  
# Create the top-level configuration files  
touch pyproject.toml requirements.txt  
# Finally, create and activate a virtual environment  
python3 -m venv venv && source venv/bin/activate  
(On Windows, use python -m venv venv && .\venv\Scripts\activate)

Step 1.3: Install ADK

Open requirements.txt and add our single dependency:

google-adk

Install it from your terminal:

pip install -r requirements.txt

Step 1.4: Configure Your Cloud Environment

These settings tell ADK how to securely connect to your Google Cloud account to use services like Vertex AI and Google Search.

Set Environment Variables: In your terminal, run the following export commands. These tell ADK to use the Vertex AI platform in your specific Google Cloud project and region.

export GOOGLE_GENAI_USE_VERTEXAI=true  
export GOOGLE_CLOUD_PROJECT=<your-gcp-project-id>  
export GOOGLE_CLOUD_LOCATION=<your-gcp-project-location>

Log In to Your Account: Run this one-time command. It will open a browser for you to sign in, allowing ADK to make authorized requests on your behalf.

gcloud auth application-default login

Part 2: Building Your Agent

Now we'll write the code and place it inside our trend_spotter package directory.

Step 2.1: Define the Agent's Brain (The Prompt)

The prompt contains all the instructions for our agent.

Note that we are guiding the LLM to specify the date range in the call to the GoogleSearch tool to make sure we are focusing on trends from the last week.

Open trend_spotter/prompt.py and add these instructions:

# trend_spotter/prompt.py  

TREND_SPOTTER_PROMPT = """  
You are a helpful AI assistant and expert tech analyst for a new podcast called "The Agent Factory". Your goal is to generate a highly relevant and verifiable report about the latest developments in AI agents that specifically impact developers.  

**Your multi-step plan is as follows:**  

**Step 1: Discover the Current Date.**  
Your very first action must be to find the current date.  
- **Action**: Use the `Google Search` tool with a query like "what is today's date".  
- From the search result, identify the current year, month, and day.  

**Step 2: Formulate and Execute Search Queries with Date Operators.**  
Now, you must formulate your search queries by embedding the date range directly into the query string using Google's `after:YYYY-MM-DD` and `before:YYYY-MM-DD` operators. Calculate these dates to cover the last 7 days.  
- You must perform at least three initial searches to cover trends, releases, and questions.  
- **Example Query Format**: `"AI agent trends after:2025-06-01 before:2025-06-08"`  
- After the initial searches, you may perform 1-2 additional, more targeted searches if a category is missing information. **Do not perform more than 5 searches in total.**  

**Step 3: Analyze the Results and Create the Report.**  
Read through all the text and links from your searches. Your primary filter is to **only select topics, tools, and questions that have a direct and significant impact on developers building AI agents.**  

**Critical Rule for Sourcing:** For every trend, release, or question you identify, you must first pinpoint the **single best search result** that provides the evidence. You will then use the URL from that **exact search result** as the source link for that item. **If you cannot find a specific source link for an item, do not include that item in the report.**  

Based on these rules, create a report:  
1.  The report **must begin with a header** specifying the date range used.  
2.  The body of the report must have exactly three sections.  
3.  For each item, you **must provide three pieces of information**: a 1-2 sentence explanation, the "Developer Impact" analysis, and the **verifiable source URL**.  

The report format must be:  

**🔥 Top 5 Trends for Agent Developers**  
1.  **[Trend 1 Name]**: [A 1-2 sentence explanation of this trend.] (Source: [URL])  
    * **Developer Impact**: [A 1-sentence explanation of why this matters to developers.]  
2.  ... (up to 5 total)  

**🚀 Top 5 Releases for Agent Developers**  
1.  **[Release 1 Name]**: [A 1-2 sentence explanation of the tool, framework, or model.] (Source: [URL])  
    * **Developer Impact**: [A 1-sentence explanation of why this matters to developers.]  
2.  ... (up to 5 total)  

**🤔 Top 5 Questions from Agent Developers**  
1.  **[Question 1 Topic]**: [A 1-2 sentence explanation of what developers are asking.] (Source: [URL])  
    * **Developer Impact**: [A 1-sentence explanation of why this matters to developers.]  
2.  ... (up to 5 total)  

Begin your work now by executing your plan.  
"""

Step 2.2: Assemble the Agent

The agent.py file connects our prompt and the search tool to a new ADK Agent.

Open trend_spotter/agent.py and add this code:

# trend_spotter/agent.py  
from google.adk.agents import Agent  
from google.adk.tools import google_search  
from . import prompt  
# Use the "latest" tag to always get the most recent stable version of the model.  
MODEL = "gemini-2.5-pro-preview-05–06"  
# This single agent will perform all the work.  
trend_spotter_agent = Agent(  
model=MODEL,  
name="trend_spotter_agent",  
description="An agent that finds and reports on AI agent trends.",  
# The agent's entire logic comes from our detailed prompt.  
instruction=prompt.TREND_SPOTTER_PROMPT,  
# We give the agent a single tool: the ability to search Google.  
tools=[google_search],  
)  
# We assign it to `root_agent` by convention for ADK to discover.  
root_agent = trend_spotter_agent

Step 2.3: Making Your Agent Discoverable

To use the adk web command, we need to tell ADK where to find our agent. We do this in the pyproject.toml file.

Open pyproject.toml in your root directory and add the following configuration:

[project]  
name = "trend_spotter"  
version = "0.1.0"  
# This section tells the ADK how to find our agent.  
[tool.adk.agents]  
trend_spotter = "trend_spotter.agent:root_agent"

Part 3: Running Your Agent

Now for the exciting part!

Step 3.1: Install your agent:

Run this command from your project's root directory. The -e . command installs your project in "editable" mode so the adk tool can find it.

pip install -e .

Step 3.2: Launch the web interface:

adk web

Open the URL that appears in your terminal. In the web interface, select "trend_spotter" from the dropdown menu. You can now chat with your agent! Ask it: "Generate a report on the latest AI agent news." This might take a few minutes, depending on the amount of searches you instruct the agent to perform in your prompt.

The response will look similar to this:

Okay, I will begin by finding the current date.  


Okay, I have identified the current date as June 7, 2025.  

Now, I will formulate and execute search queries to find information from the last 7 days (June 1, 2025, to June 7, 2025) about AI agent trends, releases, and questions relevant to developers.  


Okay, I have analyzed the search results and will now generate the report on AI agent developments impacting developers for the period of June 1, 2025, to June 7, 2025.  

Report: AI Agent Developments for Developers (June 1, 2025 - June 7, 2025)  

🔥 Top 5 Trends for Agent Developers  

Maturation of Agentic AI: Agentic AI, which involves AI systems that can autonomously handle complex tasks, is moving beyond hype and into practical application, with companies beginning to launch pilot projects. This trend signifies a shift towards more sophisticated AI that can streamline various stages of software development, testing, and maintenance.  
Developer Impact: Developers will increasingly need to build, integrate, and manage these more autonomous AI agents, requiring skills in designing for complex, goal-oriented behaviors.  
Rise of Multi-Agent Systems: There is a growing interest in systems where multiple AI agents collaborate to solve larger, more complex problems than a single agent could handle. This involves individual agents with specific goals and tools working together within a broader workflow.  
Developer Impact: This trend requires developers to think about inter-agent communication protocols, task delegation, and overall system orchestration.  
Emphasis on AI Agent Evaluation and Benchmarking: As AI agents become more capable and undertake more complex tasks, there's a corresponding growth in the need for robust evaluation methods and standardized benchmarks. These benchmarks aim to assess core competencies like planning, reasoning, and tool usage.  
Developer Impact: Developers will need to leverage these evolving benchmarks to test, validate, and improve the reliability and performance of the agents they build.  
Enhanced NLP and Personalization in Conversational AI: Conversational AI agents are advancing to provide more human-like interactions, including better understanding of natural language, emotional cues, and offering highly personalized user experiences.  
Developer Impact: Developers creating conversational interfaces must focus on incorporating these advanced NLP and personalization techniques to build more engaging and effective AI agents.  
Shift Towards AI-Native Development Infrastructure: The rapid generation of code by AI agents is prompting a re-evaluation of existing development platforms like GitHub and GitLab, which were primarily designed for human development speeds. This points towards a need for infrastructure that can better support AI-driven development rhythms.  
Developer Impact: Developers should anticipate changes in their toolchains and workflows, potentially including new platforms designed to handle the increased volume and velocity of AI-generated code and the unique review processes required.  

🚀 Top 5 Releases for Agent Developers  

Embabel Agent Framework: Launched by Spring founder Rod Johnson, Embabel is a new JVM-based, type-safe framework for building AI-powered applications, particularly for integrating Large Language Models (LLMs) with Java applications using Goal-Oriented Action Planning (GOAP).  
Developer Impact: Java developers now have a dedicated, high-level framework that emphasizes production-readiness, type safety, and advanced planning capabilities for creating sophisticated AI agents within the Spring ecosystem.  
Digitate's ignio™ Agentic AI Platform Update: Digitate announced a significant update to its ignio™ platform, introducing a suite of purpose-built AI agents aimed at IT operations, SREs, and CIOs to accelerate the move towards autonomous enterprise and ticketless IT operations.  
Developer Impact: Developers working on enterprise IT and business process automation can leverage these new agents to handle complex tasks more autonomously, improving efficiency and system resiliency.  
Digital Twin Consortium's AI Agent Capabilities Periodic Table (AIA CPT): The DTC launched the AIA CPT, an industry-first standardized framework for evaluating AI agent systems based on their actual capabilities, designed to reduce market confusion.  
Developer Impact: This framework provides developers with a clear, objective way to assess and compare different AI agent technologies and vendor offerings, aiding in technology selection and expectation setting.  
Google's Agent Development Kit (mentioned with Agentspace): Alongside its Agentspace hub for managing AI agents, Google has noted the availability of a new Agent Development Kit designed to help developers build AI agents.  
Developer Impact: Developers within the Google Cloud ecosystem can expect new and refined tools to streamline the creation and deployment of AI agents, fostering more sophisticated agent-based solutions.  
MetaGPT Framework Highlighted: Although an existing open-source framework, MetaGPT continues to be recognized for its capability in allowing multi-agent systems to automate complex software engineering tasks by encoding Standard Operating Procedures (SOPs) into LLM prompts.  
Developer Impact: Developers can utilize MetaGPT to create collaborative multi-agent systems for various development tasks, potentially improving efficiency in areas like game development, web development, and data analysis.  

🤔 Top 5 Questions from Agent Developers  

Determining When to Use AI Agents: Developers are actively discussing the appropriate use cases for AI agents, cautioning against over-engineering solutions with complex agents when simpler AI workflows or even manual intervention would be more efficient.  
Developer Impact: This highlights the need for developers to critically evaluate task complexity and ROI before committing to building an AI agent, ensuring the chosen solution fits the problem.  
Managing Repository Strategy in an AI World: There's ongoing debate about how to structure code repositories when AI agents are involved—whether smaller, focused repositories are better for AI comprehension or if larger, comprehensive ones provide necessary context.  
Developer Impact: Developers need to consider how their repository strategy impacts AI agent performance and collaboration, potentially adapting practices to include AI-generated artifacts like prompts.  
Addressing the Impact of AI on Team Dynamics: As AI tools create a potential "productivity divide" where some engineers significantly increase output, questions arise about how to maintain team cohesion, collaboration, and equitable workload distribution.  
Developer Impact: This requires a conscious effort from development teams and managers to adapt team structures and processes to integrate AI assistance smoothly and support all team members.  
Ensuring Architectural Consistency with Multiple AI Agents: A key concern is how to ensure that different AI agents, possibly working across various repositories or parts of a system, adhere to consistent architectural principles and design patterns.  
Developer Impact: Developers may need to establish clearer architectural guidelines for AI agents or develop new mechanisms to enforce consistency when employing multiple autonomous agents.  
Need for AI-Native Development Infrastructure: Developers are questioning whether current development platforms and their associated workflows (e.g., for code review) are adequate for the speed and volume of code that AI agents can produce.  
Developer Impact: This points to an upcoming need for developers to adapt to, and possibly help shape, new tools and platforms specifically designed for an AI-assisted and AI-native development lifecycle.

Step 3.3: Debugging

The adk web interface is your best debugging tool. On the "Events" tab, you can see every step your agent takes, including which tools it calls and what the LLM is thinking. If the output isn't right, your first step should always be to adjust the instructions in prompt.py.

Part 4 - Deployment

The adk deploy cloud_run command deploys your agent code to Google Cloud Run.

Ensure you have authenticated with Google Cloud (gcloud auth login and gcloud config set project ) and setup your environment variables to deploy your agent to cloud run with one line command.

Step 4.1: Setup environment variables

Optional but recommended: Setting environment variables can make the deployment commands cleaner.

# Set your Google Cloud Project ID  
export GOOGLE_CLOUD_PROJECT="your-gcp-project-id"  
# Set your desired Google Cloud Location  
export GOOGLE_CLOUD_LOCATION="us-central1" # Example location  
# Set the path to your agent code directory  
export AGENT_PATH="./trend_spotter" # Assuming capital_agent is in the current directory  
# Set a name for your Cloud Run service (optional)  
export SERVICE_NAME="trend-spotter-service"  
# Set an application name (optional)  
export APP_NAME="trend-spotter-app"

Step 4.2: Deployment to Cloud Run

adk deploy cloud_run \  
 - project=$GOOGLE_CLOUD_PROJECT \  
 - region=$GOOGLE_CLOUD_LOCATION \  
 - service_name=$SERVICE_NAME \  
 - app_name=$APP_NAME \  
 - with_ui \  
$AGENT_PATH

(more options for Cloud run deployment can be found here)

Step 4.3: Testing your deployed agent

You can test your agent by simply navigating to the Cloud Run service URL provided after deployment in your web browser. (The URL should be similar to this: https://your-service-name-abc123xyz.a.run.app)

Part 5: Next Steps and Conclusion

Congratulations! You have successfully designed, built, tested and deployed your very first AI agent using the Agent Development Kit.

You've learned how to structure a proper agent package, how to write a detailed prompt to control an agent's logic, and how to run and interact with your agent using the adk web interface. We now have a working "researcher" for our AI agent podcast and you you now have a working foundation that you can expand upon. Try modifying the prompt to research a different topic, or explore adding new custom tools to give your agent more capabilities.

In our next post, we'll continue to build on this foundation and make our agent even more powerful by adding richer, more specialized tools.

When you're ready to dive deeper and explore all the powerful features the framework has to offer, the best place to go is the official Google Cloud ADK documentation.

Happy building!

How to Write Reliable Rubrics for LLM-as-a-Judge Evaluations

Jan-Felix Schmakeit — Wed, 02 Sep 2026 16:35:00 +0000

Follow up to Part 1: How to Design AI Evaluations You Can Actually Trust

At Google, we are publishing a suite of Agent Skills for Google products and technologies on GitHub. My team is interested in measuring their performance to understand how they perform. Deterministic tests, like checking if generated code compiles, are ideal. Unfortunately they can't be easily created at scale for nuanced, generative responses, such as answers to open-ended questions or information retrieval tasks.

In my previous post we looked at what you test, which means the evaluations that are the actions you ask the agent to perform. The next step is to look at how you assert whether the agent succeeded. This means creating reliable and accurate assessments of the agent's response.

To assess complex outputs at scale, especially when topics cover broad areas with nuanced parts, we use an "LLM-as-a-judge" approach. Responses are assessed against a structured rubric using a model based grader. The judge evaluates each response using a set of true/false questions. Aggregated, these answers provide an accuracy score for a response.

Giving an LLM a vague prompt or subjective questions leads to ambiguity in its response. This ambiguity introduces noisy data and causes inconsistent evaluations. Ultimately, it wastes your token budget on metrics that aren't useful.

To make these assessments more reliable, you must treat your rubrics like formal specifications. By constraining the judge to evaluate strict, objective boolean truths, you reduce the chance of hallucinations. Because evaluating strict boolean truths is a less complex task, you can even use smaller, faster models for grading.

Here are four lessons we learned to help you write robust rubric questions for your LLM-as-a-judge scorer.

Keep Questions Atomic and Distinct

Evaluating multiple requirements in a single question, such as "Does the response contain a metadata property and format the output as JSON?", forces the LLM judge to guess which clause is more important. This ambiguity leads to inconsistent grading and wasted tokens.

Split compound questions: Instead of writing one large check, divide your requirements into discrete, atomic TRUE/FALSE questions. (e.g., Check 1: Does it contain a metadata property? AND Check 2: Is the output JSON?)
Avoid overlapping questions: Never test the same underlying concept multiple times across your rubric. Overlap risks double-penalizing the evaluated model for a single mistake, which corrupts your accuracy score.
Reduce the reasoning load: Eliminate the need for the judge to weigh competing clauses. When every question assesses exactly one distinct fact, your grading becomes more consistent.

Constrain the Judge: Objective Facts Over Subjective Reasoning

The rubric-based approach exists because giving an LLM judge a full prose prompt to evaluate a complex response leads to inconsistent numbers. If you ask the judge subjective questions like "Was this a comprehensive answer?" or ask it to explain "Why did the agent do this?", you introduce ambiguity that creates noisy, unrepeatable data.

Focus on observable facts: Do not ask the judge to evaluate concepts that require interpretation, such as intent, quality, or reasoning. Only assess concrete facts that you expect to find in the response.
Write formal specifications: Use strict, objective language, such as RFC 2119 terminology (MUST, MUST NOT, REQUIRED), to test for observable outcomes. The judge should never have to guess what you mean.
Test for negative constraints: Explicitly verify what the agent should not do. Instead of asking if the agent "used best practices", check that it did not suggest a specific deprecated feature.
Require strict true/false answers: By forcing a strict TRUE/FALSE classification on objective facts, you reduce the reasoning load and reduce the variance in ratings.
Avoid opportunities for cheating: Agents will tailor answers to game your tests if given the chance. Keep scoring rubrics isolated in a separate system. Design rubrics that focus on strict functional outcomes or specific topics, rather than broad keywords matches.

Grade Only What You Asked For

When building rubrics, it is easy to accidentally evaluate an agent on requirements that were never stated in the prompt. Doing this creates false negatives and reduces the accuracy of your measurement.

Align the rubric to the prompt: Assess only what was explicitly requested. For example, do not penalize a model for failing to provide citations if the prompt never asked for them.
Grade the destination, not the journey: Avoid writing rubrics that check if the agent used a specific tool or followed a rigid sequence of steps. Pre-trained models might bypass custom tools entirely if they already know the answer.
Evaluate the final response: Grade the objective output. If you need to evaluate a step-by-step process, prompt the agent to output an execution plan and evaluate the plan itself.

Calibrate Your Judge

Even if you follow these rules and write perfectly atomic, objective questions, your LLM judge might still misinterpret your scoring instructions and rubrics. To guarantee your pipeline generates consistent ratings and reliable signals, you must prove that the judge's scoring aligns with how a human subject matter expert would evaluate the exact same response through calibration.

Establish a human baseline: Ask subject matter experts to manually grade a "golden set" of test responses.
Run the comparison: Run your LLM-as-a-judge against this golden set and compare the automated scores against the human scores.
Identify the delta: If the LLM judge disagrees with your human experts, it is usually a sign that your rubric or the grading instructions are too ambiguous.
Iterate until aligned: Tweak and calibrate your rubric questions or the scoring instructions until the LLM judge consistently aligns with human experts. Only then is your judge ready.

Summary

Once you have this reliable data, the next step is making it visible. In AI Evals at a Glance: Heatmaps for Stakeholders Joe Spiro explains how to take these raw measurements and visualize evaluations.

When building our agent skills, we learned that ambiguous evaluation rubrics do not provide useful signals and feedback. Forcing your LLM judge to evaluate strict, boolean facts eliminates this noise. It makes your tests repeatable, optimizes your token spend, and lets you confidently measure whether your AI tool is actually improving.

Photo by William Warby on Unsplash

What is harness engineering and why should I care?

Shir Meir Lador — Wed, 02 Sep 2026 15:28:05 +0000

How do you ship a software product with 0 lines of manually-written code?

A friend asked me this today, and I realized I didn't have a simple answer. So I dug deeper.

It turns out the answer is in how you engineer your harness.

Wait now, what? What is harness engineering?

There is a reason this is the most important trend right now around coding agents. The biggest question these days is how to validate AI-generated code without reading every single line. How do you make sure an agent doesn't break production or delete your data?

A blog by OpenAI shared an interesting experiment where a team of 3 engineers have built and shipped an internal beta of a software product with 0 lines of manually-written code. Every line of code: application logic, tests, CI configuration, documentation, observability, and internal tooling, has been written by Codex.

How did they do it? They didn't write the app. They designed the harness.

What exactly is a harness?

Think of an AI agent like a powerful racehorse. The harness is the track, the blinders, and the jockey's reins that keep it running in the right direction instead of jumping into the stands.

As my colleague Arthur Thompson explained today: for agents — the harness is composed of all the deterministic components that wrap the LLM.

Balaji Subramaniam details those deterministic components in his blog — the orchestration layer, execution sandboxing, state persistence, and verification tools.

If you want to build reliable agentic systems, your job shifts from writing the logic to designing the environment. Here is what you need to focus on:

Set strict boundaries: Don't let the agent guess what it can touch. Enforce strict access rules (like confining it to a specific sandbox) so it can't accidentally wipe out production data.
Build "Repair Loops": Agents will inevitably make mistakes. A great harness automatically traps errors, like a failed build or a test failure, and feeds those clean logs right back to the agent so it can fix its own code.
Give them a map, not a manual: As the OpenAI team discovered, don't overwhelm the agent with massive instruction files. Structure your repository logically so the agent can discover context progressively as it works.

Show me the code

What does this look like in practice? Here is a simple example using the Google Antigravity SDK with Google's ADK to configure a local harness. Notice how we are strictly bounding the agent to a specific workspace (workspaces=["./sandbox"]) and giving it a place to save its memory (save_dir="./trajectories" ) so it can learn from previous experience:

import os
from google.adk.labs.antigravity import AntigravityAgent
from google.antigravity import LocalAgentConfig
from google.antigravity.hooks import policy

# Ensure absolute paths for workspace containment
sandbox_dir = os.path.abspath("./sandbox")
os.makedirs(sandbox_dir, exist_ok=True)
save_dir = os.path.abspath("./trajectories")

# 1. Engineer the harness environment
sdk_config = LocalAgentConfig(
    system_instructions="You are a helpful local environment assistant.",
    workspaces=[sandbox_dir],
    # Let the agent write safely within the restricted sandbox boundary
    policies=[policy.allow_all()],
    save_dir=save_dir,
)

# 2. Wrap the config to run the agent inside the harness
root_agent = AntigravityAgent(
    name="antigravity_assistant",
    description="Runs an Antigravity SDK agent inside ADK.",
    config=sdk_config,
)

The policy keeps the agent access only in the Sandbox folder

With this design in place, you can drop your legacy code into the sandbox, write a simple loop to run unit tests against it, and let the agent iteratively fix its own bugs.

Adding Tests

So, how do we actually run tests against this sandboxed agent?

In modern harness engineering, tests are an active part of the agent's workflow graph. Using Google's ADK 2.0, which introduces graph-based workflows, you can define a test validation step as a simple routing node.

If the test passes, the job is done. If it fails, the harness automatically loops the error back to the agent to try again. Notice the built-in 'kill switch': we track the iteration count so if the agent gets stuck in an infinite loop of breaking and fixing code, the harness safely pulls the plug.

from google.adk.agents.context import Context
from google.adk import Event
from google.adk.events.event_actions import EventActions
from google.genai import types

# 3. Evaluate the code in the sandbox
def execution_test_node(ctx: Context):
    # Safely track our attempts to prevent infinite loops
    iteration_count = ctx.state.get("iteration_count", 0) + 1
    ctx.state["iteration_count"] = iteration_count

    test_passed = ctx.state.get("test_passed", False)
    feedback = ctx.state.get("feedback", "")

    if test_passed:
        # Success! End the workflow.
        return Event(actions=EventActions(route="END"))

    if iteration_count > 5:
        # The Kill Switch: The agent is stuck. Stop the loop.
        return Event(actions=EventActions(route="END"))

    # Failure! Feed the error trace back to the agent and loop it.
    feedback_msg = f"The unit tests failed with the following traceback:\n\n{feedback}"

    return Event(
        content=types.Content(role="user", parts=[types.Part(text=feedback_msg)]),
        actions=EventActions(route="loop_back")
    )

If you want to see this test routing pattern in action, you can check out an example with a full implementation in Balaji's ADK harness repository.

Wiring it all together using graph-based workflow

To connect the agent and the test node, you can use a Workflow graph to map out exactly how the execution should flow without needing complex, nested Python while loops.

Think of this as drawing the actual lanes on the racetrack:

from google.adk import Workflow

# 4. Wire the agent and the test node together into a loop
repair_loop = Workflow(
    name="repair_loop",
    edges=[
        # 1st Step: Define the main sequence (START -> agent -> test node)
        ("START", root_agent, execution_test_node),

        # 2nd Step: If the test returns "loop_back", go back to the agent
        (execution_test_node, {"loop_back": root_agent})
    ]
)

The test loop

Congratulations! you've built an autonomous system. The agent writes the code and hands it off to the test node. If the test fails and returns a loop_back route, the agent tries again with the error log in hand.

See more examples of loop patterns in ADK samples.

Try it yourself

You might wonder why you need a Python script to run an agent. In a normal chat window, you are the harness: you copy the error logs and babysit the model. A software harness lets the system babysit itself, allowing you to fully automate test-driven coding or safely refactor massive legacy codebases.

To run this self-healing loop on your own machine today, the setup takes less than five minutes:

Install the framework: Run pip install "google-adk[antigravity]" in your terminal to get the open-source Agent Development Kit and the Antigravity integration.
Set your API key: Grab a free Gemini API key from Google AI Studio and export it to your environment (export GEMINI_API_KEY="your-key").
Run the loop: Save the code blocks above as a Python script, drop a broken Python or Node file into your new ./sandbox directory, and run your script.
Expand your graph: Unit tests are just the baseline. To make your harness bulletproof, add a second AI agent to your workflow, like a SecurityAuditor, to review the code before it passes, or wire in custom linters to enforce strict architectural rules.

From there, you can swap out our simple test node for a subprocess that actually executes pytest or npm test against your sandbox, and you will have a fully functioning repair loop.

If you are ready to scale this up, you can download the full IDE and CLI at antigravity.google, explore the Antigravity managed agent for remote execution and google's ADK 2.0 for using graph based workflows.

Claude Fable 5.1 is now available on Agent Platform!

Jen Harvey — Tue, 01 Sep 2026 18:28:57 +0000

Claude Fable 5.1 is officially available in the Model Garden on Agent Platform.

Built for long-running, high-stakes work, Fable 5.1 puts frontier intelligence into production across your code, documents, and research.

👉 Try it today and let us know what you're building: Claude Fable 5.1

Elevating Antigravity agent skills, Part 2: Image generation

James O'Reilly — Tue, 01 Sep 2026 16:51:26 +0000

By coupling native image generation directly with structured prompt synthesis, you turn your AI agent from a text generator into an image studio producing clean, consistent images grounded in developer specs.

Here is what you will get out of this deep dive:

How to use the generate_image tool within agent skills.
Prompt tokenization strategies for consistent image generation.
Full breakdown of a real-world example skill you can use.
BONUS: Three visual asset anti-patterns to avoid in agentic workflows.

📝 About this series: Welcome to Elevating Antigravity Agent Skills series, a 5-part engineering guide to mastering the agent tools that reduce orchestration tax and transform AI agents into autonomous collaborators: ask_question, generate_image, define_subagent + invoke_subagent, send_message and manage_subagents.

Beyond the grey

Many of us building web applications, developer tools, or component libraries have used grey placeholder boxes while waiting for sample visual assets.

This is where the generate_image tool comes in. When an Antigravity agent executes generate_image, it synthesizes text prompts and visual parameters into images saved directly into the conversation artifact storage.

Passing un-processed prompts without structure directly to generate_image leads to inconsistency. By encoding lighting, camera, and scene specifications into your skill's prompt generator, your agent produces images that consistently match your intent.

Native artifact rendering and context storage

Before diving into skill authoring, let's review how Antigravity manages generated media.

When generate_image runs, the system automatically saves the output asset to the conversation's internal artifacts directory (<appDataDir>/brain/<conversation-id>/). The tool returns the absolute file path, allowing your skills to render the asset directly in chat responses using standard markdown syntax: ![Caption](file:///path/to/image.png).

A reference walkthrough of our image generation workflow

To see this architecture in action, let's examine an example generating-mock-images skill. Using a plant and flower e-commerce store as my example, this skill generates mock product assets, displays the resulting image for interactive review, and handles workspace asset persistence.

Figure 1: When we prompt "Create mock photo of white lilies." or "...Jasmine", or "...multi-colored roses", the generating-mock-images skill is auto-discovered by the agent and read into context. The skill instructs the agent to follow the playbook we created.

.agents/skills/generating-mock-images/
├── SKILL.md

1. Image prompt synthesis

The skill pairs the selected product subject (user provided houseplant or floral arrangement) with hardcoded photography parameters detailed in the skill:

### 1. Synthesize Prompt & Call `generate_image`
Construct the final image prompt by combining subject with photography specifications:

> "Professional studio product photograph of [SUBJECT]. [SCENE: smooth matte light gray surface, seamless background curve with soft infinite horizon, visible water line in vase]. [CAMERA: macro lens photography, sharp focus from front to back, f/11 aperture, crisp details, zero digital noise]. [LIGHTING: three-point soft-diffuse lighting, 48-inch octagonal softbox key light at 45 degrees, gentle fill light, backlit soft ambient glow, no harsh glare, premium e-commerce product catalog style]."

2. Tool execution and artifact rendering

Following the skill instructions, the agent automatically creates and issues a structured tool call to generate_image. The following is a mock representation of what that structured tool call would look like:

{
  "Prompt": "Professional studio product photograph of elegant floral arrangement of fresh seasonal blooms in a clear glass vase filled with crystal-clear water. SCENE: smooth matte light gray surface, seamless background curve with soft infinite horizon. CAMERA: macro lens photography, sharp focus, f/11 aperture, crisp details, zero digital noise. LIGHTING: three-point soft-diffuse lighting, 48-inch octagonal softbox key light at 45 degrees, gentle fill light, backlit soft ambient glow, premium e-commerce catalog style.",
  "ImageName": "mock_flower_plant",
  "AspectRatio": "1:1",
  "toolAction": "Generating floral arrangement mock product image",
  "toolSummary": "Generate mock flower product image"
}

Note, not shown in this example but generate_image natively supports visual composition. Passing up to three absolute file paths into the ImagePaths array parameter instructs the agent to blend existing logos, background textures, or reference wireframes into composite graphics.

Upon receiving the generated image path from generate_image, the agent renders the asset immediately in chat:

![Mock Flower/Plant Product](file://<GENERATED_IMAGE_PATH>)

3. Interactive review loop via `ask_question`

In my previous post, Elevating Antigravity Agent skills with interactive UI workflows, I established how to convert passive agents into active interviewers using the ask_question tool. By interrogating developers upfront, we eliminated prompt ambiguity.

Rather than assuming the generated asset is immediately production-ready, the agent prompts the developer using ask_question:

Invoke `ask_question` tool:
* Question: "What would you like to do with this generated mock image?"
* Options: ["Keep", "Regenerate", "Exit"]

4. Workspace lifecycle and asset persistence

If the developer approves the asset, the agent copies the file from temporary artifact storage to <workspace>/public/images/<image_filename> and returns a clickable markdown link:

[mock_flower_plant.png](file://<WORKSPACE_ROOT>/public/images/mock_flower_plant.png)

If the developer chooses to delete or regenerate, the agent purges the temporary artifact to prevent uncurated asset accumulation.

If your agent dumps uncurated artifacts directly into production asset paths, you aren't automating design, you're automating visual tech debt.

🎁 Bonus: Three anti-patterns in agent visual asset generation (and beyond)

As you integrate expensive tool calls, like generate_image into automated developer workflows, observe these three rules to maintain consistency and efficiency:

1. Avoid raw prompt injection

Avoid passing raw user inputs directly to generate_image. Build a prompt generator inside your skill that injects specific parameters (such as scene, lighting, camera, and style preferences) to ensure deterministic outputs across runs.

2. Avoid broken links

Use the absolute artifact path returned by generate_image and move approved images to a permanent location. Storing assets in temporary directories (/tmp) causes link decay when context windows clear or workspace sessions restart.

3. Avoid redundant calls

Before triggering generate_image, instruct your skill to check if a valid asset with the target ImageName already exists. Re-running image generation without parameter changes wastes compute budget and slows down workflow execution.

🚀 Builder Challenge: Take this workflow further!

Now that you have mastered parameterized image generation with generate_image, here is an engineering challenge to take your visual asset skills to the next level:

Build an Automated Multi-Aspect Ratio Asset Suite Generator.

Currently, our skill requires the user to include details in their prompt.

Your challenge:

Extend your skill's prompt generator to accept target layout presets (e.g., Hero Banner, Product Card, Mobile Thumbnail) or aspect ratios (e.g. 1:1, 2:1, 4:3, 16:9) using the ask_question tool.
Extend your skill's prompt generator to use pre-made reference images passed into generate_image using ImagePaths param. You can include up to 3 images.

Did you attempt the challenge? Share your skill's strategy in the comments below!

Conclusion

Static text descriptions and broken image icons belong to an earlier generation of developer tooling. By leveraging parameterized asset generation in generate_image, you empower your agents to produce clean, consistent and visually grounded technical artifacts.

Take 15 minutes today to audit your component scaffolding and asset generation skills.

📌 Elevating Antigravity Agent Skills Series Index

Part 1: Building interactive UI workflows with ask_question
Part 2: Automating Image Generation with generate_image (📍 You are here)
Part 3: Invoking Subagents with define_subagent, invoke_subagent
Part 4: Inter-Agent Communication with send_message
Part 5: Managing Active Agent Lifecycles with manage_subagents

Additional resources

Tool Reference: List of supported Antigravity tools
Example Source: source code

Help others find this post

Save this post to find it later.
Subscribe to my newsletter and don't miss an article.
Share this article across social media.
Follow me on LinkedIn or X for more agentic engineering insights.

Thanks for reading!

How to Design AI Evaluations You Can Actually Trust

Jan-Felix Schmakeit — Tue, 01 Sep 2026 16:35:00 +0000

As part of my work at Google, we are publishing a suite of Agent Skills for Google products and technologies on GitHub. These agent skills are designed to help AI agents interact with our technologies. But how do you test that these skills are useful and work as expected? My team in Developer Relations has been focused on this question, because having reliable signals on their performance is critical to help us improve them over time.

Just as you wouldn't deploy a production API without writing unit tests, you should apply the same standard to your AI agents. As Joe Spiro showed in the Designing AI Evals post series, scaling AI tools means moving beyond "vibe testing" in a terminal. Instead, you should set up a structured, automated evaluation pipeline to benchmark your integration. The evaluations (evals) are the actions you asked the agent to perform, which are graded using scorers (for example rubrics) that assert whether the agent succeeded. We'll focus on evaluations in this post and tackle tips for scoring rubrics in the next post.

However, AI evaluations cost real tokens. You need to make sure that you use these tokens as efficiently as possible. They need to provide real value that helps you build better tools. Writing good evaluations is critical. Poor evaluations provide false signals, waste your token budget, and create noise in your metrics.

Here are five rules we learned to design better evaluations you can trust. Follow them to ensure that every token you spend produces a useful metric.

Know Your Evaluation Environment

Before writing evaluations, you need to understand the setup and limitations of your chosen framework. This includes systems like Harbor, Inspect AI, or integrations in development tools like in the Agent Development Kit. Does it use an ephemeral sandbox? What tools are available? How is the output captured

Tailor your graders to the environment: For example, if you can deterministically access the sandbox to evaluate code, that's an option. Alternatively, ask the agent to print its response to the console. Your framework can capture this output and pass it directly to your scorers. Adjust your grader to handle these environments.
Be aware of dependency limitations and access to "real" resources: If an evaluation task requires access to "real" resources (for example an authenticated gcloud session with access to a Google Cloud project), create ephemeral resources or credentials that isolate and limit access so they don't impact other evaluations. Alternatively, you could provide mock tools instead of real test credentials.
Evaluate plans instead of tasks that are difficult to isolate: An easier approach might be to evaluate the plan to accomplish the task, rather than the actual execution.
Avoid interactive prompts: Multi-turn agent sessions are complex to evaluate. When getting started, design your evaluations using one-shot prompts.

Avoid the Ceiling Effect

If your evaluations show a high baseline accuracy (i.e., without your agent tool), it might not prove its value, or the evaluation prompts are too easy.

Write harder prompts: You cannot measure the impact of a new agent tool or skill if the baseline model already knows the answer.
Require multi-step reasoning: Design prompts that reflect complex, real-world use cases where your tool can actually differentiate itself from the model's pre-training.
Revisit the scope of your tool: If tests are repeatedly reporting a high accuracy without using your tool, it might be time to revisit it. The underlying model and agent may have improved and are able to accomplish the task without additional help. It might be time to refocus or deprecate your tool.

The Prompt-Grader Mismatch

You cannot grade an agent on something you did not explicitly ask it to do. Your evaluation prompts and graders should be complementary. This means that they should only test for things included in the prompt.

Avoid scope creep: If you asked a broad question, you can expect a similarly broad response. For example, if your prompt is "How do I secure Google Cloud Run?", your grader cannot penalize the agent for missing a specific, unprompted IAM role.
Be explicit: If you want to evaluate specific knowledge or exact implementation details, you must state those requirements clearly in the prompt.

Grade the Destination, Not the Journey

Agents possess inherent model knowledge and might skip your custom tools entirely to arrive at the correct answer. (That on its own is some useful feedback!)

Do not evaluate the trajectory: Avoid writing graders that check if the agent used a specific help command or followed a rigid sequence of steps.
Evaluate the final answer: Grade the objective output. If you absolutely must evaluate the agent's planning phase, explicitly ask it to output a detailed execution plan and evaluate the plan instead.

Curate Your Evaluation Dataset

A strong evaluation suite tests real, diverse use cases. But testing the same capability repeatedly causes overfitting and creates noisy metrics.

Use real-world examples: Evaluations should include real user journeys and focus on goals users want to achieve. Consider including additional context, such as sanitized sample data, to ground the evaluations.
Maximize your signal: Ensure every prompt in your evaluation suite tests a distinct concept or unique capability. Think of this like code coverage for traditional tests.
Remove overlap: Consolidate redundant prompts. A smaller, more curated data set provides clearer metrics, prevents overfitting and saves tokens.

Summary

You cannot improve AI tools if you can't measure them accurately. If you treat AI evaluations with the same focus as traditional unit tests, you improve the quality of your metrics and get more robust signals.

By applying these five rules, you eliminate false signals that waste your token budget. Instead of generating noise, your test suite gives you actionable feedback you can use to guide your engineering decisions and improve your tools.

Figuring out what to test is only the first step. A well-designed evaluation is only useful if the scorer grading answers is reliable and returns meaningful results. In my next post, we will look at how to test. You will learn how to write lean, atomic rubrics that minimize ambiguity for an LLM grader and make every token count.

Photo by William Warby on Unsplash

Step up to the Sheets: AI Eval Export and Illustrating Data

Katie McLaughlin — Tue, 01 Sep 2026 07:00:00 +0000

Welcome to part three of our series about designing, analyzing and visualizing AI Evals! As with the previous entries you are more than welcome to simply read over my exploration or for those of you looking for extra credit, feel free to play along at home.

Note: This blog series contains AI-generated diagrams alongside actual screenshots and hand-drawn edits of both. AI also assisted in minor copy editing.

Implementation prerequisites

If you intend to (re)implement the walkthrough, all you need to do right now is complete Part 1 (Designing AI Evals: Clarity Now and Visualization Next) and then install pandas; Part 2 (AI Evals at a Glance: Heatmaps for Stakeholders) is illustrative and will give you a deeper understanding going forward both of AI evals and why the following may be useful but it’s more elective than anything else.

Back to the story at hand

While engineering teams might evaluate agents using test automations, non-technical stakeholders (product managers, business leads, or executive sponsors) often use spreadsheets as their lingua franca and as such it would make sense to speak to them in a way they are almost certain to understand and which enables you all to collaboratively make visualizations to further get out the word.

To make this a reality, today we’ll use a pandas pipeline that transforms raw Inspect AI evaluation logs into a rich CSV optimized for Google Sheets, allowing you or anyone on your team with a little spreadsheet knowledge to make boardroom-ready visuals.

Drawing again from previous blog entries, the evals we are working on have three dimensions: model x skill x sample [and epoch] that cross to form the following matrix:

In Part 2 we created a simplified heatmap reflecting a planar slice of this space as reduced over the sample dimension. While we will ultimately do something similar in a future blog post using Data Studio, for the time being we actually want to keep all 32 combinations separate and pull more of their underlying stats from the inspect eval files and derive further information using pandas. This information will be exported as a flat csv file we can then import and view in Google Sheets.

The extractor `tocsv.py`

This script is used to parse the eval logs using Inspect AI’s python pandas interface. The script is relatively self documenting, and you can read up more on pandas and pandas’ csv IO if you are unfamiliar.

Heavy lifting in Python: Custom telemetry columns

While most of the script deals with mapping scalar metrics the like of which we’ve already discussed, there are two that are purely logical/boolean. They represent specific statistical cases that we will allow end users to partition the data set over for analysis later. These boolean conditions are derived and since we want to pare away excess information and therefore not supply the full eval files to Google Sheets, tocsv.py does the heavy lifting upstream during log ingestion.

It pre-computes two key boolean columns to simplify downstream dashboard filtering:

completed: Flags whether a sample finished cleanly (no API quota 429 errors, timeout truncations, or container crashes) with a valid score, cleanly isolating model capability from infrastructure noise.
is_baseline_or_has_activated_skill: Distinguishes Baseline control cohorts (always TRUE) and Active Skill interventions (TRUE when the skill tool was actually “activated” which in inspect means uptaken) from Dormant runs (FALSE, where skills were available but never activated).

In Google Sheets or Data Studio non-technical stakeholders can simply filter/toggle these pre-computed boolean keys to isolate clean, active performance (or otherwise) instantly.

Exporting to Google Sheets

In order to execute the pipeline I called:

python3 tocsv.py

The output file data_mega_export.csv is a standard comma-separated values file where each cell is surrounded by commas. You can preview the raw and unformatted contents in a standard text editor. While most of this is normal spreadsheet data such as strings and integers, you will also see fairly complicated Google Sheets formulas resulting from interpolating eval metrics into formula templates.

The above formula for instance will be transformed into a bar graph normalized against the range of values in the same column as its source data and tiered into colors by tertile.

How do you take this text and get it to show its stuff? By opening sheets.new of course, and importing data_mega_export.csv. When presented with the import dialog check Convert text to numbers, dates, and formulas.

On clicking Import data Google Sheets will instantly display the standard cells and render active, color-coded visual sparklines directly inside data cells!

What conclusions can you draw from what you see? One obvious one in this example is that the correlation between score and latency (how long an agent takes to answer a question) may not be very strict. But what if you wanted to get something deeper?

Well, since this is a spreadsheet, obviously you can make pivot tables, and custom functions or sort on multiple keys… you know, the whole accountant’s playbook. Let’s say though that you want custom views into the data rather than to permanently mutate it. How would you go about doing that?

Google Sheets, unlike inspect view and inspect viz heatmaps, allows you to dynamically sort AND filter on more than one key. Not only that but you can sort on ANY key, so I elected to do so with the previously unavailable score and latency fields. As shown in the image above, use Data > Create filter view which presents you with a new set of controls at the top of the main columns.

Note: Filter views do not mutate or change the underlying contents of a sheet; Instead they are (optionally named and shareable) configurations for restricting what rows are displayed and in what order.

If, as in the picture, you click the cost column > Filter by condition, select the Less than option and fill the comparison with 0.01, you can narrow the displayed rows to only those with costs less than a cent. This can be used (for instance) to only look over the samples that were “less costly” to try to analyze why.

Filter views (despite their name) also allow you to sort the display of the data. Because the sort operations are cumulative, the “primary” filter needs to be applied last. Thus, after sorting the filter view first by latency and then by score (both ‘Z to A’ aka decreasing) gives us the following:

Optionally you can name and save this filter view to toggle it on and off as well as share it with other collaborators. To learn more about Filter views, you can read Sort & filter your data.

This ability to sort and filter the display live without changing the underlying information, makes presenting the data and analyzing it during discussions with teammates significantly easier and faster.

Why leave the cradle?

Now, in spite of this tremendous flexibility, sheets have some drawbacks as visualization tools:

Spreadsheet formulas are cumbersome for nested IF statements and filters unless you define custom macros or JavaScript functions leading to code that's harder and harder to keep connected in one's mental model.
Flattened eval runs need to be inspected under multiple filter views to begin to understand complex trends or correlations across all of them.

So, how can we take this to the next level and circumvent those issues? What about an environment, which while it still offers you the ability to custom script anything you need, comes with a lot of WYSIWYG controllable visualizations and graphs, enabling you, without code, to bring data to an even more vibrant life? Better yet, what if that environment comes with built in visualizations for 3 dimensions or more and has broad compatibility with multiple data source types?

In Part 4, we’ll connect this Google Sheet directly to Data Studio to build a codeless executive summary dashboard and be able to quickly and intuitively display our findings!

Stop Wrestling with ASR: The Complete Guide to Gemini 3.5 Transcribe 🎙️

Guillaume Vernade — Fri, 28 Aug 2026 13:34:29 +0000

You’ve probably used Gemini to analyze hours of video, summarize podcasts, or answer questions from recorded meetings (if you didn't you should, it's extremely useful!). But when all you need is a clean, hyper-accurate, and structured transcript from audio, spinning up a huge reasoning model with complicated prompts often feels like using a sledgehammer to crack a nut.

Enter Gemini 3.5 Transcribe (gemini-3.5-transcribe).

It's Google's dedicated speech-to-text model built on Gemini's audio understanding core, optimized specifically for fast, accurate, and cost-effective transcription. Whether you want an exact court-reporter transcript with millisecond timestamps, or a reading-optimized summary that removes all your awkward "ums" and "uhs", this model handles it natively with zero prompt gymnastics.

🚀 Hands-on first: If you want to jump straight into running the code yourself, open the interactive Gemini Transcribe Colab notebook! It's ready to run so you can dirrectly experience how the model work.

Prefer a visual UI with zero coding? You can also test speech recognition directly in Google AI Studio.

Here's what you'll find in this guide:

0. Why a Dedicated Transcription Model? (Audio Understanding vs. Transcribe)
1. Setup & The Files API
2. Steer Languages & Code-Switching (85+ Locales)
3. Custom Vocabulary: Never Misspell Technical Jargon Again
4. The Killer Feature: Smart Transcription vs. Verbatim Mode
5. Speaker Diarization: Who Said What?
6. Word-Level Timestamps: Precise Time Offsets for Every Spoken Word
7. Decision Matrix: Which Configuration Should You Use?
8. What About Real-Time Live Streaming?

0. Why a Dedicated Transcription Model?

Before looking at the code, let's get the mental model straight. You might wonder: "Can't I just upload an MP3 to Gemini 3.7 and say 'Transcribe this'?"

You can, but here is why gemini-3.5-transcribe is different:

Feature	General Audio Understanding (e.g. Gemini 3.7)	Dedicated Transcribe (`gemini-3.5-transcribe`)
Primary Job	Reasoning, Q&A, sentiment analysis, audio chat	High-throughput, precise speech-to-text
Speaker Diarization	Prompt-dependent (can hallucinate turns)	Native segment labeling (`spk:0`, `spk:1`)
Timestamps	Approximate timecodes via text prompt	True word-level millisecond offsets in metadata
Vocabulary Biasing	System prompt instructions	Native acoustic biasing dictionary (up to 1,000 terms)
Cost & Latency	Full multimodal LLM generation overhead	Optimized lightweight speech pipeline

Pro tip: If you need to ask questions about what happened in an audio file ("What was the action item for Alice?"), use a multimodal model like Gemini 3.7. If you need the transcript itself, subtitles, or cleaned dictation notes, use Gemini Transcribe!

1. Setup & The Files API

The Gemini 3.5 Transcribe model runs on the modern Google GenAI SDK (google-genai v2.0+) using the Interactions API.

First, install the SDK:

pip install -U "google-genai>=2.0.0"

Make sure you have an API key from Google AI Studio, set it as GEMINI_API_KEY, and let's look at how audio gets passed to the model:

from google import genai

client = genai.Client()

# 1. Upload your audio file via the Files API
audio_file = client.files.upload(file="meeting_recap.mp3")

# 2. Request transcription using the uploaded file's URI
interaction = client.interactions.create(
    model="gemini-3.5-transcribe",
    input=[{"type": "audio", "uri": audio_file.uri}],
)

print(interaction.output_text)

Watch the demo video below to see the baseline transcription in action—handling natural speech and bilingual code-switching with ease:

Why use the Files API?

When dealing with audio and video, you never want to inline raw audio bytes as base64 in your API requests—it blows up the payload size by 33%, easily hits network timeouts, and requires re-uploading the same bytes if you want to rerun a query.

The Files API solves this cleanly:

Large file support: Upload audio and video files up to 2 GB per file (with 20 GB of total project storage).
Temporary lifecycle: Files are stored for 48 hours and automatically cleaned up afterwards.
It's completely free! Storage and uploads in the Files API incur zero additional cost—you only pay for token processing when you actually run inference against the model.

2. Steer Languages & Code-Switching (85+ Locales)

As you saw in the video above, Gemini Transcribe automatically identifies spoken languages out of the box and seamlessly handles code-switching (when someone mixes multiple languages in the same sentence—like switching between French and English mid-sentence, which happens to me all the time!).

However, if you know your audio is exclusively in a specific language or regional dialect, you can pass explicit BCP-47 language codes in transcription_config to bias recognition:

interaction = client.interactions.create(
    model="gemini-3.5-transcribe",
    input=[{"type": "audio", "uri": spanish_audio.uri}],
    generation_config={
        "transcription_config": {
            # Explicit language hint
            "language_codes": ["es-ES"],
        }
    },
)

print(interaction.output_text)

Note: Leaving language_codes=[] (or omitting it) enables full automatic detection across 85+ supported languages and locales. Check out the Audio Transcription Documentation for the complete list of language codes.

3. Custom Vocabulary: Never Misspell Technical Jargon Again

Every developer has suffered from an ASR model mangling proper names, confusing specialized libraries with everyday dictionary words (turning "ScaNN" into "scan", or "Qdrant" into "quadrant"), or inventing phonetically similar terms ("Sitsi" instead of "CitC", "Thiago" instead of "Tiago").

With custom_vocabulary, you can pass a list of up to 1,000 domain-specific terms that the model will bias towards:

interaction = client.interactions.create(
    model="gemini-3.5-transcribe",
    input=[{"type": "audio", "uri": team_briefing.uri}],
    generation_config={
        "transcription_config": {
            "custom_vocabulary": [
                "Guillaume Vernade",
                "ScaNN",
                "Qdrant",
                "Cilium",
                "Weaviate",
                "Milvus",
                "Buganizer",
                "Tiago",
                "CitC",
                "CL",
                "spaCy",
            ],
        }
    },
)

print(interaction.output_text)

Watch the side-by-side comparison video below to see how the model behaves with and without custom vocabulary biasing:

Without Custom Vocabulary (Default ASR)	With `custom_vocabulary` (100% Precision)
"For our vector benchmarks, sync with Guillaume Vernat* in Paris to compare Scan against Quadrant while Syllium handles the traffic."*	"For our vector benchmarks, sync with Guillaume Vernade* in Paris to compare ScaNN against Qdrant while Cilium handles the traffic."*
"We also need to evaluate Weaviate against Milvus, assign the buganizer ticket to Thiago, and test the changes in Sitsi* before submitting the CL."*	"We also need to evaluate Weaviate against Milvus, assign the Buganizer ticket to Tiago, and test the changes in CitC* before submitting the CL."*
"Finally, run a quick smoke test with Spacey* to validate the tokenization pipeline before deploying."*	"Finally, run a quick smoke test with spaCy* to validate the tokenization pipeline before deploying."*

Notice how default speech recognition falls back to phonetic dictionary guesses (Vernat, Scan, Quadrant, Syllium, Thiago, Sitsi, Spacey). By contrast, supplying custom_vocabulary guarantees that names of team members, niche tools, internal infrastructure, and open-source libraries are transcribed with 100% precision.

Pro tip: Don't just put acronyms in your custom vocabulary. Add proper names of team members, internal service codenames, GitHub repo handles, product brand names, and niche industry terminology.

4. The Killer Feature: Smart Transcription vs. Verbatim Mode

This is hands down my favorite capability of Gemini 3.5 Transcribe.

By default, speech-to-text models operate in verbatim mode: they write down everything, including every nervous stutter, throat clear, false start, and verbal tick.

When you're transcribing a speech rehearsal, interview, or voice memo, reading raw verbatim text is painful:

--- Verbatim output ---
"Uh, hello. Good evening, everyone. Um, I'd like to start by, well, first of all, thank you all for coming. Today is, um, a very special day, or rather, evening? No, afternoon? Right, evening. We are here to celebrate, uh, sorry, let me just find my notes. Ah, here. We are here to honor, no, not honor, but, um, to mark the launch of our new, sorry, my glasses are a bit foggy, the new marketing campaign. No, wait, product campaign? Product, yes. Um, where was I? Ah, yes. It has been a long journey, a very, uh, challenging, well, not challenging in a bad way, but, you know, difficult? No, rewarding. Rewarding is the word. So, um, yes, cheers to, wait, we don't have glasses yet. Thank you."

If you switch mode={"type": "smart"}, the model performs intelligent reading optimization:

Disfluency removal: Strips conversational filler words ("um", "uh", "you know").
Inline self-corrections: Automatically resolves verbal slip-ups ("Tuesday, wait no, Wednesday" $\rightarrow$ "Wednesday").
Structured formatting: Formats lists, bullet points, numbers, currencies ($26M), and natural paragraphs.

Here is how you turn it on:

interaction_smart = client.interactions.create(
    model="gemini-3.5-transcribe",
    input=[{"type": "audio", "uri": audio_file.uri}],
    generation_config={
        "transcription_config": {
            "mode": {
                "type": "smart",
            },
        }
    },
)

print(interaction_smart.output_text)

Look at the cleaned result on that exact same rehearsal audio:

--- Smart transcription output ---
Good evening everyone. First of all, thank you all for coming. Today is a very special evening. We are here to mark the launch of our new product campaign.

It has been a long journey, a very rewarding one. So, cheers to that.

Watch the side-by-side comparison video below to see how the raw disfluencies are stripped while listening:

(If the video doesn't load, you can listen to rehearsing.wav directly.)

Important caveat: Because Smart transcription uses language modeling to clean up disfluencies and structure the output, it might slightly rewrite, omit, or rephrase parts of what was said to make it sound natural and concise. If you are doing verbatim court reporting, medical transcription, or subtitle syncing where every exact syllable matters, stick with verbatim mode!

Also note that Smart mode is incompatible with word-level timestamps and speaker diarization (which require {"type": "verbatim"}).

5. Speaker Diarization: Who Said What?

Need to know who spoke during a multi-person meeting or podcast? Enable diarization with diarization_mode="speaker":

interaction = client.interactions.create(
    model="gemini-3.5-transcribe",
    input=[{"type": "audio", "uri": meeting_audio.uri}],
    generation_config={
        "transcription_config": {
            "mode": {
                "type": "verbatim",
                "diarization_mode": "speaker",
            },
        }
    },
)

To extract each speaker turn cleanly, iterate through the step annotations:

def print_diarized_transcript(interaction):
  words = []
  for step in getattr(interaction, "steps", []) or []:
    for content in getattr(step, "content", []) or []:
      for annotation in getattr(content, "annotations", []) or []:
        if getattr(annotation, "type", None) == "word_info":
          words.append(annotation)

  current_speaker = None
  current_turn = []

  for w in words:
    speaker = getattr(w, "speaker", "spk:0")
    if speaker != current_speaker:
      if current_turn:
        print(f"[{current_speaker}]: {' '.join(current_turn)}")
      current_speaker = speaker
      current_turn = [w.text]
    else:
      current_turn.append(w.text)

  if current_turn:
    print(f"[{current_speaker}]: {' '.join(current_turn)}")


print_diarized_transcript(interaction)

Output:

[spk:0]: One chocolatine, please.
[spk:1]: Tiago, arrête. It is a pain au chocolat.
[spk:0]: Wait, a guy from the south west told me it's chocolatine.
[spk:1]: Do not listen to them. 90% of France and the entire universe calls it pain au chocolat. Chocolatine is a myth.
[spk:0]: Meu Deus, you French are intense. In Brazil, people fight the exact same way over bolacha versus biscoito.
[spk:1]: Well, here pain au chocolat is the only real word.
[spk:0]: Fine. Two pain au chocolat, please. As long as it has chocolate, tá valendo.

Watch the demo video below where two colleagues debate pain au chocolat vs. chocolatine. Notice how the waveform line dynamically changes color (Cyan for Tiago, Orange for his colleague) as each speaker takes turns:

(Direct audio link: listen to pain_au_chocolat.wav)

6. Word-Level Timestamps: Precise Time Offsets for Every Spoken Word

When you need exact synchronization—for example, to jump to specific points in a video, build interactive transcripts, or align text with waveforms—you can request word-level millisecond start and end offsets.

Configure timestamp_granularities=["word"] (and optionally combine it with diarization_mode="speaker"):

interaction = client.interactions.create(
    model="gemini-3.5-transcribe",
    input=[{"type": "audio", "uri": audio_file.uri}],
    generation_config={
        "transcription_config": {
            "mode": {
                "type": "verbatim",
                "timestamp_granularities": ["word"],
                "diarization_mode": "speaker",
            },
        }
    },
)

Each recognized word comes back with its exact time offsets (and speaker turn) attached in the content annotations:

words = []
for step in getattr(interaction, "steps", []) or []:
  for content in getattr(step, "content", []) or []:
    for annotation in getattr(content, "annotations", []) or []:
      if getattr(annotation, "type", None) == "word_info":
        words.append(annotation)

for w in words[:6]:
  spk = getattr(w, "speaker", "spk:0")
  print(f"[{w.start_offset:>7} -> {w.end_offset:>7}] ({spk}) {w.text}")

Output:

[ 0.000s ->  0.400s] (spk:0) One
[ 0.400s ->  1.200s] (spk:0) chocolatine,
[ 1.200s ->  1.800s] (spk:0) please.
[ 3.200s ->  3.700s] (spk:1) Tiago,
[ 3.700s ->  4.200s] (spk:1) arrête.
[ 4.200s ->  4.500s] (spk:1) It

What can you do with word timestamps?

Having millisecond-level offsets for every individual word unlocks huge capabilities:

Instant subtitles (.srt / .ass): Group words into 3-5 second caption blocks for YouTube, Premiere, or Final Cut.
Karaoke & dynamic captions: Highlight each word in real-time as it's spoken (like TikTok / YouTube Shorts).
Click-to-play search: Build audio/video search indexes where clicking any search keyword immediately seeks the player to that exact millisecond.
Waveform & visual animations: Trigger visual events or highlight specific spoken phrases on screen.

💡 Behind the scenes: That's actually what I did to make the demo videos above! The word timestamps provided the exact millisecond timing to align the subtitle cards, highlight the custom terms ("oatmilk"), and trigger the color switch of the waveform line from Cyan to Orange when the speaker changed.

If you want the complete Python function to convert these word annotations into standard .srt subtitle files, you can find it directly in the interactive Cookbook Colab notebook.

7. Decision Matrix: Which Configuration Should You Use?

Here is a quick cheat sheet to pick the right settings for your use case:

Use Case	Mode	Diarization	Timestamps	Custom Vocab
Meeting Notes / Voice Memos	`smart`	No	No	Optional
Video Subtitles / Closed Captions	`verbatim`	Optional	`["word"]`	Highly recommended
Podcast / Multi-speaker Interview	`verbatim`	`speaker`	`["word"]`	Highly recommended
Legal / Compliance Audio Logs	`verbatim`	`speaker`	`["word"]`	Optional
Search Indexing & Embeddings	`smart`	No	No	Optional

8. What About Real-Time Live Streaming?

Everything we covered above is for pre-recorded audio files (unary mode via the Files API).

Gemini also supports real-time live streaming transcription over WebSockets using gemini-3.5-transcribe-live and the Live API. It lets you stream raw 16-bit PCM chunks (100ms each) directly from a microphone and receive instantaneous interim partial hypotheses (interim_input_transcription) and finalized text as speech occurs.

However, streaming real-time WebSockets with asynchronous Python workers (asyncio), handling audio chunking, and managing ephemeral valet tokens for secure client apps is quite a bit more complex and deserves its own dedicated tutorial.

If you want to dive straight into live streaming code right now:

📖 Open the Gemini Transcribe Colab Notebook (it includes runnable cells for live streaming and ephemeral token creation!)
📚 Read the official Gemini Live Transcription Documentation on ai.google.dev.

Wrapping Up

Gemini 3.5 Transcribe gives you the best of both worlds: strict, millisecond-accurate verbatim data when you need timestamps and diarization, and an intelligent, disfluency-stripping smart mode when you want clean text for human eyes.

Have you tried using smart mode on your own voice recordings or meetings? Drop your thoughts and edge cases in the comments below! 🚀🚀🚀

AI Evals at a Glance: Heatmaps for Stakeholders

Katie McLaughlin — Tue, 25 Aug 2026 07:00:00 +0000

Visualizing AI evals with Inspect Viz

Welcome back to our blog series on running, analyzing and visualizing AI evals. Last time we discussed how to design and run evals for analysis and visualization using inspect eval and inspect view. Many of these methods, such as rearranging the dashboard columns and sorting on them to determine model vs model and skill vs no-skill differences in metrics, can give you a broad overview of the field. While this can inspire further and deeper inquiry, it runs into issues with how predictive it is and how to communicate findings to other people. Imagine the following:

The 40-log gridlock

Imagine a data science lead needing to present a performance breakdown of four LLM candidates across ten internal tools during a high-stakes, time-crunched live meeting. Reordering and filtering the evals across multiple dimensions is hard to read on a presentation screen and requires doing mental math with an audience: riveting stuff that they definitely won’t fall asleep during of course.

What if instead you had a tool that allowed you to easily automate rendering a comparison of the different configurations? What if lining them up by model or by skill showed clear and legible patterns?

Well, let me introduce you to heatmaps! A heatmap is a 2D visual matrix where color intensity represents numerical magnitude—in our case, quadratic curved accuracy scores. By leveraging the human eye's natural ability to distinguish color gradients, heatmaps instantly reveal performance spectrums at a single glance. This makes trends across dependent and independent variables clear, intuitive, and highly legible.

While you can simply read along and learn about using inspect viz to generate heatmaps for inspect AI evals, if you intend to follow along and recreate the charts, please start by replicating the steps in the first entry and come back afterward.

Note: This blog series contains AI-generated diagrams alongside actual screenshots and hand-drawn edits of both. AI also assisted in minor copy editing.

Follow-along requirements

For those playing along at home, in order to perform the following, you will need the inspect_viz library installed in your environment (alongside inspect_ai and pandas).

From there, take a look at the heatmap script. While there are some intricacies to the script (such as establishing its CLI API) the script mainly wraps scores_heatmap(viz_data, orientation="horizontal", ...) and marshals our data to assemble a heatmap of it.

If you run the script

python3 inspect_viz_heatmap.py logs -o heatmap.html

you will get a visual that looks like the following.

This heatmap has two axes, one is the model (vertical) which allows you to visually break down your analysis to just skill inclusions for the same model, the other is the skills (horizontal) and going down the row allows you to demonstrate how changes to the model affect the evaluation.

Stakeholders (like investors or your boss) can now spot skill gains and model ceilings at a glance without reading line-by-line log values. Some quick takeaways:

Top-performing configurations (such as 3.6-flash + gemini_api_skill at 1.0) stand out in high-intensity shades
Within the aggregated 2D heatmap view of this dataset, 3.6-flash trended higher in score on average across the evaluated skill configurations compared to 3.5-flash-lite.
Skills tended to improve scores, all other things held equal.

Caption: The inspect viz heatmap projects our full Evaluation Matrix down to a 2D plane by collapsing individual task samples and epochs into aggregate accuracy cells along the Model and Skill axes.

Cognitive overhead and segue to BI analytics

While matrix scoreboards provide quick visibility into accuracy differences across models, raw heatmaps suffer from key limitations:

2D Constraint: Heatmaps map only two axes to a single color intensity (in our case Accuracy).
Missing Business Context: A single heatmap cannot simultaneously show token costs, system latency, or operational reliability.
Hidden Variance: Collapsing samples for a given configuration removes the information necessary to determine if a result is consistent or wildly erratic.

To bridge CLI logs with interactive, multidimensional and in-depth reporting, we need a pipeline that exports the raw telemetry metrics into spreadsheets and BI dashboards.

In Part 3, we’ll build a python script to convert output data for processing first in Google Sheets and ultimately Data Studio!

Elevating Antigravity agent skills, Part 1: Interactive UI workflows

James O'Reilly — Mon, 24 Aug 2026 19:55:23 +0000

By embedding native chat UI components directly into your SKILL.md instructions, you transform your agent from a passive text parser into an active technical interviewer with a delightful user experience.

Here is what you will get out of this deep dive:

How to use Google Antigravity interactive UI Primitives.
Round-trip state hydration for user selections with grounding.
Practical design rules for structuring finite selection lists without overwhelming developers or triggering visual clutter.

📝 About this series: Welcome to Elevating Antigravity Agent Skills series, a 5-part engineering guide to mastering the agent tools that reduce orchestration tax and transform AI agents into autonomous collaborators: ask_question, generate_image, define_subagent + invoke_subagent, send_message and manage_subagents.

Anatomy of an interactive skill

In Antigravity, agent capabilities are packaged as skills. These are dedicated directories containing a SKILL.md file with YAML frontmatter for discovery and Markdown instructions for execution. By default, agents interpret instructions sequentially and execute terminal commands or file edits. However, when an Antigravity agent encounters explicit instructions to invoke the ask_question tool, it suspends execution and renders a native, interactive UI modal in the chat window.

This simple tool invocation bridges the gap between agent autonomy and human-in-the-loop (HITL) alignment. Instead of guessing parameter values or parsing ambiguous prompts, the agent presents structured radio buttons, checkboxes, and write-in fields.

To eliminate repetitive questioning across sessions and subagents, I pair UI interrogation with persistent state hydration. My agent's lifecycle follows a structured sequence:

Workspace State Hydration (Read user_prefs.json) At the start of the workflow, the agent checks the workspace root for a namespaced configuration file. If found, it extracts saved defaults to use during UI presentation.
Interactive UI (Render ask_question modals) The agent invokes ask_question with finite option arrays and multi-select flags, prefixing hydrated defaults with "(Current Setting) " so the user sees active preferences immediately.
Selection Processing with Fallbacks (Clean and Apply) The agent strips UI signaling prefixes from selected strings and preserves existing JSON defaults if an optional question is left blank or skipped.
State Persistence (Write user_prefs.json) The agent serializes updated selections back to user_prefs.json in the workspace root, establishing an unshakeable source of truth for future subagent invocations. This is not a special Antigravity file, but rather one I've named and created for the use of my project. You should name and place this file in a location most suitable for your needs.

Built-in /grill-me command

Before diving into custom skill authoring, it is worth noting that Antigravity natively supports interactive alignment through the /grill-me slash command.

When you trigger /grill-me during a chat session, the agent temporarily pauses code execution to conduct a rigorous, multi-question interview. Instead of making silent architectural assumptions, the agent systematically interrogates you on design trade-offs, edge cases, and implementation boundaries until ambiguity is resolved.

While /grill-me is an exceptional built-in tool for ad-hoc session planning and general requirements gathering, custom Interactive Skills take this concept a step further. By authoring custom markdown skills that invoke UI selection modals, engineering teams can standardize domain-specific interview workflows (like form scaffolding or cloud deployments) and persist the resulting answers directly into workspace configuration files for subagent grounding.

You can learn more about the /grill-me command in Richard Seroter's article, Crafting an agent team that still includes me.

A reference walkthrough of our interactive UI workflow

To see this architecture in action, examine how I've implemented the exemplar making-forms-demo skill. This skill interviews developers before generating a web form, ensuring validation rules and label layouts match team standards.

Figure 1: When we prompt "Make a web form", our making-forms-demo skill is auto-discovered by the agent and read into context. The skill instructs the agent to present the user with a series of questions using the ask_question tool.

Stage 1: Workspace State Hydration

Before rendering UI components, the skill instructs the agent to read the existing state from user_prefs.json.

---
name: making-forms-demo
description: >
  Interviews the user for critical implementation details needed to
  generate a Web Form. Manages FORM namespace in `user_prefs.json`.
  Use when making Web Forms.
---

# Making forms
Interviews the user for critical implementation details needed to
generate a Web Form.

## Workflow Steps

1. **Confirm Intent & Hydrate State:**
   Check if `user_prefs.json` exists in the workspace root.
   If present, read the `"FORMS"` object to identify any previously
   saved preferences. Tell the user that a form has been identified
   and that a few questions must be answered (or confirmed) before
   building the form.

Stage 2: Interactive UI

If previous preferences exist, the agent prefixes those strings with "(Current Setting) " and places them at the top of the options array. When executed, this Markdown block generates an interactive modal supporting multi-selection checkboxes (is_multi_select: true). Because the options are explicitly enumerated, the agent never guesses validation requirements.

2. **Query Validation Type:**
   Invoke the `ask_question` tool to present the user with an option
   selection UI. If previously saved values exist in 
   `user_prefs.json`, prefix those option strings with 
   `"(Current Setting) "` and list them first:
   * **Question:** "How would you like to handle field validation?
     (select all that apply)"
   * **Options:** ["Client-side validation", 
     "Server-side validation"]

   Use the selected value as `FIELD_VALIDATION_TYPE`.

Stage 3: Selection Processing with Fallbacks

For mutually exclusive architectural choices, the skill configures single-selection lists. By omitting an explicit "other" string from the options array, the agent automatically enables Antigravity's native default write-in option in the UI modal.

3. **Query Validation Location:**
   Invoke the `ask_question` tool to present the user with an option
   selection UI. If a previously saved value exists in 
   `user_prefs.json`, prefix that option string with 
   `"(Current Setting) "` and list it first:
   * **Question:** "How would you like to handle validation
     messages?"
   * **Options:** ["Above the field", "Below the field", 
     "Summary Card", "Tooltip"]

   Use the selected value as `FIELD_VALIDATION_LOCATION`.

If a developer selects the write-in option in the modal and types "Inline below label", the agent captures that custom string directly into FIELD_VALIDATION_LOCATION without requiring custom syntax parsing.

Stage 4: State Persistence

Once the user submits the modal, the agent cleans the input strings by removing "(Current Setting) " prefixes and writes the finalized structure back to the workspace root:

4. **Persist User Preferences:**
   Write the user's selected form preferences to a `user_prefs.json`
   file in the workspace root directory using the `write_to_file`
   tool (or overwrite existing preferences). This guarantees that 
   future agent invocations and subagents are grounded in the user's 
   exact specifications without needing
   to re-interview them:
   ```json
   {
     "FORMS": {
       "FIELD_VALIDATION_TYPE": ["<SELECTED_VALUES>"],
       "FIELD_VALIDATION_LOCATION": "<SELECTED_VALUE>",
       "FIELD_LABEL_TYPE": "<SELECTED_VALUE>"
     }
   }
   ```

## Mock Skill Instructions
Since this skill is just a mock skill for illustrating how to
create interviewer UI within the chat window, don't actually create
a form. Instead:
1. Write the selections to `user_prefs.json` in the workspace root.
2. Output the form selections to the user as confirmation using the
   template below:

For each field:
**<FIELD_NAME>:** <SELECTED_VALUE>

By nesting form preferences under the "FORMS" namespace, the workspace configuration file remains clean and scalable. Other skills, such as deployment or database bootstrappers, can safely manage their own state within the same JSON root without collision.

Figure 2: The next time the skill is used it finds the user_prefs.json file previously created and uses the data to mark appropriate responses as "(Current Selection) ". The preferences file is updated should the user change their selections.

The 4 things to avoid when creating interactive skills

As you transition your team's custom skills from static text prompts to interactive interviewers, adhere to these four best practices to keep your workflows sharp and maintainable:

1. Avoid option-fatigue

Interactive modals are designed for fast scanability. If a selection list exceeds seven items the user can get overwhelmed with options and revert to guessing. When dealing with large datasets, such as hundreds of marketing campaigns or cloud regions, never dump the raw array directly into an ask_question block. Instead, instruct the agent to query an MCP tool first to filter the choices down to a top-5 list relevant to the current workspace before rendering the modal.

2. Avoid the write-in anti-pattern

A common mistake is explicitly adding "Other" or "Custom value" to the options array in SKILL.md. In Antigravity, the ask_question tool renders a default write-in text field automatically whenever an interactive question is presented. Explicitly including an "other" string creates redundant, confusing options and forces the agent to handle an unnecessary secondary text-prompting step.

3. Avoid flat root keys in config files

When persisting user selections to user_prefs.json, always nest key-value pairs under a high-level domain namespace (such as "FORMS", "DEPLOYMENT", or "TESTING"). If multiple skills write flat keys directly to the root of the JSON object, key collisions become inevitable as your team's skill pack grows. Namespacing guarantees strict boundary isolation across autonomous subagents.

4. Avoid third-person phrasing

Format option strings as the user's direct voice (e.g., "Floating labels (Labels sit inside the input field)") rather than third-person imperative commands (e.g., "Set form layout to floating"). Direct-response phrasing reduces cognitive friction and makes interactive modals read more naturally.

Conclusion: Stop prompting, start interviewing

The era of typing massive, brittle paragraphs into chat prompts to guide autonomous coding agents is over. By combining declarative Markdown instructions, native ask_question UI components, and round-trip JSON state hydration, skill authors can transform AI assistants into rigorous technical interviewers.

📌 Elevating Antigravity Agent Skills Series Index

Part 1: Building interactive UI workflows with ask_question (📍 You are here)
Part 2: Automating Image Generation with generate_image
Part 3: Invoking Subagents with define_subagent
Part 4: Inter-Agent Communication with send_message
Part 5: Managing Active Agent Lifecycles with manage_subagents

Additional resources

Tool Reference: List of supported Antigravity tools
Article: How to automate modernization with Antigravity and multi-agent orchestration
Article: Crafting an agent team that still includes me

Help others find this post

Save this post to find it later.
Subscribe to my The Agentic Developer newsletter.
Share this article across social media.
Follow me on LinkedIn or X for more agentic engineering insights.

Thanks for reading!

Without Custom Vocabulary (Default ASR)	With `custom_vocabulary` (100% Precision)
"For our vector benchmarks, sync with Guillaume Vernat* in Paris to compare Scan against Quadrant while Syllium handles the traffic."*	"For our vector benchmarks, sync with Guillaume Vernade* in Paris to compare ScaNN against Qdrant while Cilium handles the traffic."*
"We also need to evaluate Weaviate against Milvus, assign the buganizer ticket to Thiago, and test the changes in Sitsi* before submitting the CL."*	"We also need to evaluate Weaviate against Milvus, assign the Buganizer ticket to Tiago, and test the changes in CitC* before submitting the CL."*
"Finally, run a quick smoke test with Spacey* to validate the tokenization pipeline before deploying."*	"Finally, run a quick smoke test with spaCy* to validate the tokenization pipeline before deploying."*

DEV Community: Google AI

Build a Long-Running Agent in the Cloud for $5.70/Month

What are we building?

How the system works

What else can you build with a long-running agent?

Why Cloud Run instances are great for agents

When NOT to use this

Alternative architecture: Decoupled Job + Service

Deploy your long-running agent in 6 simple steps

1. Turn on the cloud services

2. Create a storage bucket for your data

3. Build your container

4. Create a service account

5. Store your API key securely

6. Launch the instance

What does this cost in real life?

Learn more about Cloud Run instances

What is coming next?

Your First Multi-agent system: A Beginner's Guide to Building an AI Trend finder with ADK

Step 1: Get Reddit API Credentials & Install Library

Configure Your Cloud Environment

Step 2: Create the Project Folder

Step 3: Create the Specialist Sub-Agents

Step 4: Build the Main Orchestrator Agent

Step 5: Running Your Multi-Agent System

Part 6: Next Steps and Conclusion

Your First AI Agent: A Beginner's Guide to Building an AI Trend finder with ADK

Part 1: Setup and Configuration

Step 1.1: Prerequisites

Step 1.2: Create Your Project Structure

Step 1.3: Install ADK

Step 1.4: Configure Your Cloud Environment

Part 2: Building Your Agent

Step 2.1: Define the Agent's Brain (The Prompt)

Step 2.2: Assemble the Agent

Step 2.3: Making Your Agent Discoverable

Part 3: Running Your Agent

Step 3.1: Install your agent:

Step 3.2: Launch the web interface:

Step 3.3: Debugging

Part 4 - Deployment

Step 4.1: Setup environment variables

Step 4.2: Deployment to Cloud Run

Step 4.3: Testing your deployed agent

Part 5: Next Steps and Conclusion

How to Write Reliable Rubrics for LLM-as-a-Judge Evaluations

Keep Questions Atomic and Distinct

Constrain the Judge: Objective Facts Over Subjective Reasoning

Grade Only What You Asked For

Calibrate Your Judge

Summary

What is harness engineering and why should I care?

What exactly is a harness?

Show me the code

Adding Tests

Wiring it all together using graph-based workflow

Try it yourself

Further reading

Claude Fable 5.1 is now available on Agent Platform!

Elevating Antigravity agent skills, Part 2: Image generation

Beyond the grey

Native artifact rendering and context storage

A reference walkthrough of our image generation workflow

1. Image prompt synthesis

2. Tool execution and artifact rendering

3. Interactive review loop via ask_question

4. Workspace lifecycle and asset persistence

🎁 Bonus: Three anti-patterns in agent visual asset generation (and beyond)

1. Avoid raw prompt injection

2. Avoid broken links

3. Avoid redundant calls

🚀 Builder Challenge: Take this workflow further!

Conclusion

📌 Elevating Antigravity Agent Skills Series Index

Additional resources

Help others find this post

How to Design AI Evaluations You Can Actually Trust

Know Your Evaluation Environment

Avoid the Ceiling Effect

The Prompt-Grader Mismatch

3. Interactive review loop via `ask_question`

The extractor `tocsv.py`