DEV Community: Google AI

Engineering a Brainrot art installation on an Orange Pi Zero 🍊

Paige Bailey — Sat, 27 Jun 2026 20:49:31 +0000

How I built a gamified, infinite-scroll video installation for the TIAT "Slop Epistemologies" exhibition—and optimized heavy web media to run smoothly on a $15 single-board computer.

In David Foster Wallace’s Infinite Jest, "The Entertainment" is a film cartridge so fatally mesmerizing that anyone who watches it loses all desire to do anything else, eventually dying of starvation while locked to the screen.

For the TIAT (The Intersection of Art & Technology) "Slop Epistemologies" exhibition in San Francisco, I wanted to explore this concept through the lens of modern algorithmic content consumption. We live in an era of infinite short-form video slop—hyper-optimized feeds designed to bypass critical thinking and keep our attention spans permanently "cooked."

The result is BrainRot TV — The Entertainment: an interactive, physical art installation where gallery visitors step up to a monitor, grab a physical USB slide advancer, and voluntarily "LOCK IN" to an endless, deteriorating stream of curated video slop.

While the conceptual premise is artistic, bringing it to life required solving fascinating full-stack and embedded systems engineering challenges. I had to build a responsive, immersive web application capable of playing endless video feeds, generating procedural audio, and tracking gamified psychological decay—all running flawlessly on a low-end Orange Pi Zero connected to a 640×480 CRT-style display.

This is a deep dive into the architecture, physical hardware mapping, and aggressive performance optimizations that made it work. Buckle in! 😄

1. Core experience and architecture

At its core, BrainRot TV is a vanilla HTML5, CSS3, and JavaScript application served via a custom lightweight Dockerized Nginx container deployed on Google Cloud Run.

Curating the "slop"

To simulate an authentic algorithmic feed, I built a playlist of 120+ direct MP4 video URLs hosted on Google Cloud Storage. When a visitor presses "LOCK IN," the playlist is dynamically Fisher-Yates shuffled, initiating a TikTok-style vertical swipe transition between clips.

// Detect dark theme var iframe = document.getElementById('tweet-2070630195028250804-698'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2070630195028250804&theme=dark" }

Generative audio (Web Audio API)

Instead of looping a static audio file, the installation generates an evolving, procedural drone atmosphere in real-time using the Web Audio API.

Two detuned sine oscillators drift around A1 (55 Hz) and E2 (82.41 Hz) to create a haunting open fifth.
A slow LFO modulates pitch drift for an organic, unsettling warmth, and a filtered pink-noise bed adds subtle analog static texture.

The audio engine listens to the active video stream using loadeddata heuristics. If the currently playing video contains active audio tracks (generated with Veo 3 or with Omni), the procedural ambient drone automatically ducks from 35% volume down to 6%, rising back up during silent clips to maintain visual and auditory tension.

// Ducking ambient music based on video audio detection
function checkVideoAudio() {
  const player = dom.videoPlayer;
  const hasAudio = player.mozHasAudio ||
                   Boolean(player.webkitAudioDecodedByteCount) ||
                   Boolean(player.audioTracks && player.audioTracks.length);

  if (hasAudio) {
    setAmbientVolume(0.06); // Duck for video sound
  } else {
    setAmbientVolume(0.35); // Bring drone back up for silent media
  }
}

The "Cooked" Meter & Meltdown Mechanics

As visitors watch, a psychological decay meter accelerates over time. It scales through 9 distinct tiers—from FRESH and SLIGHTLY GLAZED to NO THOUGHTS HEAD EMPTY and finally THE ENTERTAINMENT.

Along the way, the UI flashes single-frame subliminal commands (CONSUME, OBEY, DO NOT LOOK AWAY) and periodic Sick, Sad World title cards. Once the viewer hits 100% rot, a critical meltdown sequence initiates: violent CSS screen-shaking takes over, and the "TOUCH GRASS" exit button is forcibly disabled. You cannot leave until the system collapses into a Blue Screen of Death (BSOD) kernel panic.

2. Bridging the physical world

Because this is a physical gallery installation, we didn't want users touching a keyboard or mouse. Instead, visitors interact with the piece using a standard wireless USB presentation clicker (slide advancer) plugged into the Orange Pi.

Most commercial USB slide advancers emulate standard USB HID keyboard events, typically broadcasting PageDown or ArrowRight for the "Forward" button, and PageUp or ArrowLeft for "Back." We built a context-sensitive event dispatcher to map these simple physical inputs to complex state transitions:

Clicker Button	HID Key Codes	Landing Screen	Viewing Screen	BSOD Crash Screen
Forward ▶	`PageDown`, `ArrowRight`, `Enter`	🔒 LOCK IN (Start)	⏭ Skip Video	🔒 REBOOT BRAIN
Back ◀	`PageUp`, `ArrowLeft`, `Escape`	(No Action)	🌱 TOUCH GRASS (Exit)	(No Action)

const FORWARD_KEYS = new Set(['PageDown', 'ArrowRight', 'ArrowDown', 'Enter', ' ']);
const BACK_KEYS = new Set(['PageUp', 'ArrowLeft', 'ArrowUp', 'Escape', 'Backspace']);

document.addEventListener('keydown', (e) => {
  if (FORWARD_KEYS.has(e.key)) {
    e.preventDefault();
    if (state.screen === 'landing') lockIn();
    else if (state.screen === 'viewing') playNextVideo();
    else if (state.screen === 'crash' || state.screen === 'exit') lockIn();
  } else if (BACK_KEYS.has(e.key)) {
    e.preventDefault();
    // Visitors can only "Touch Grass" if the meltdown hasn't locked them in!
    if (state.screen === 'viewing' && !dom.btnTouchGrass.disabled) {
      touchGrass();
    }
  }
});

3. Optimizing for the Orange Pi Zero

Running heavy video streams and dynamic CSS filters on a modern MacBook M-series chip is trivial. Running them on an Orange Pi Zero—a tiny single-board computer with a low-frequency Allwinner ARM processor, 512MB of RAM, and minimal GPU acceleration—was an immediate performance bottleneck.

When I first tested the installation on the Orange Pi, videos stuttered heavily, frame rates dropped to single digits, and text rendered as an unreadable, blurry smudge on the 640×480 display. Solving this required engineering a specialized low-power architecture.

Automatically detecting onboard hardware

Rather than maintaining two separate codebases, the application inspects browser concurrency and hardware memory capabilities on launch, automatically engaging low-power optimizations when running on embedded edge hardware:

const LOW_POWER = new URLSearchParams(window.location.search).get('lowpower') === '1' ||
  (navigator.deviceMemory && navigator.deviceMemory <= 2) ||
  (navigator.hardwareConcurrency && navigator.hardwareConcurrency <= 2);

Deep Video Prefetching

On weak Wi-Fi or limited CPU threads, waiting for a user to click "Skip" before fetching the next MP4 causes jarring black pauses. In low-power mode, we expanded the prefetch pipeline from 1 video to an asynchronous queue of 4 hidden <video> buffer elements. By setting preload="auto" and keeping them muted off-DOM, the browser silently saturates its media cache ahead of time, ensuring instant, zero-latency cuts between video transitions.

Killing GPU-Melting CSS Filters

Originally, as the visitor's brain rot progressed, the entire .screen--viewing container underwent intensifying CSS distortions: hue-rotate(), saturate(), and blur().

On ARM Linux without dedicated GPU compositing threads, CSS blur() forces software rendering, bringing the CPU to its knees. In low-power mode, we dynamically strip all CSS filters and fullscreen DOM overlays (such as CSS scanlines and glitch overlays), relying purely on lightweight opacity shifts and solid color contrast.

Throttling procedural canvas noise

The application features a live 2D <canvas> rendering analog TV white noise. A naive requestAnimationFrame loop firing at 60fps consumed over 40% of the Orange Pi's CPU cycles just calculating Math.random() pixel data.

To fix this, we introduced framerate throttling and resolution scaling. On low-power hardware, the canvas internal resolution is divided by 8 (scaling up smoothly via CSS) and throttled to render at just 8 frames per second—or bypassed entirely if needed:

const STATIC_FRAME_INTERVAL = LOW_POWER ? 125 : 0; // 125ms = ~8 FPS
const STATIC_RESOLUTION_DIVISOR = LOW_POWER ? 8 : 4;

4. 640×480 CRT Typography

The final hurdle was visual typography. At 640×480 resolution, custom web fonts (like Press Start 2P or stylized display fonts) rendered terribly on Linux ARM browsers. Without high-DPI screens or OS-level subpixel anti-aliasing, characters clumped together into illegible pixel soup. Furthermore, standard text-shadows (0 0 10px neon-green) blurred into muddy halos.

To guarantee crisp readability from three feet away in a dimly lit gallery, we executed a complete embedded typography overhaul:

System Fonts: We bypassed Google Fonts entirely on embedded devices, overriding DOM elements to use native OS font stacks (-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif). Native fonts carry built-in hinting rules that align perfectly to low-resolution pixel grids.
Scale Doubling: We doubled font sizes across the board. Headers scaled up to 72px–110px, control buttons to 36px, and telemetry badges to 24px.
Zero-Shadow: We stripped all glowing text shadows and replaced backdrop-filter: blur() glassmorphism with 100% solid, pitch-black contrast backgrounds (#000 !important). Text never composites over moving video pixels.
Emojis: Lightweight ARM Linux OS builds often lack comprehensive Unicode emoji font tables, rendering emojis as hollow rectangular boxes (□). We wrapped UI emojis in <span class="emoji"> tags and applied display: none !important in low-power mode, replacing them with clean cybergothic ASCII indicators like ///.

Slop is in the eyes of the beholder

Building BrainRot TV for the TIAT "Slop Epistemologies" exhibition was extremely fun, and a good reminder that hardware constraints can often drive the most impactful engineering and aesthetic decisions.

By stripping away heavy GPU abstractions and tailoring media pipelines specifically for a $15 Orange Pi Zero, BrainRot now has a rock-solid 60fps video playback loop that never stutters, never crashes prematurely, and delivers an intensely mesmerizing, inescapable critique of modern digital consumption.

If you're attending the exhibition in San Francisco, step up to the terminal, grab the clicker, and hit LOCK IN! Just remember to try and Touch Grass before the system takes you under. 😅

The complete codebase for BrainRot TV is containerized with Docker and deployed on Google Cloud Run.

A Guide to AI Cold Starts on Cloud Run

Shir Meir Lador — Fri, 26 Jun 2026 16:50:58 +0000

I saw a developer asking on Reddit if there was any “sane way” to manage Cloud Run cold starts for AI across multiple regions. They were experiencing startup latencies of up to 20 seconds, a frustrating gap where the infrastructure is spinning up while the user waits for a response.

The discussion was full of developers who had almost given up on serverless GPUs, with some even migrating back to GKE just to escape the latency. I decided it was time to dive deep into the Mechanics of AI Cold Starts and see if we could find that "sane way."

During my research into hosting models like Gemma 4 on Cloud Run, I had the privilege of co-presenting at Google Cloud Next '26 with Oded Shahar (Senior Engineering Manager for Cloud Run) and our guest speaker Ajay Nair (Global VP of Platform at Elastic).

In our session, "Build AI architectures with custom models on Cloud Run," Ajay shared the production-hardened strategies that allow Elastic to serve millions of daily requests across 17+ model variants, all while maintaining the 'scale-to-zero' efficiency of Cloud Run.

Build AI architectures with custom models on Cloud Run

Ajay showed us that the secret isn't just in the model, but in treating GPUs as fungible compute rather than infrastructure to manage.

I realized then that minimizing cold start latency isn't just about the model, it's about the infrastructure patterns and architectural decisions that keep it fast, scalable, and secure.

The anatomy of an AI cold start

As the official Google Cloud GPU best practices explain, an AI cold start is a shift from standard web microservices. You aren't just booting code, you're moving gigabytes of weights into a specialized physical accelerator.

Think of it as a four-phase race. If you don't optimize each step, you're going to lose your users.

Phase 1: Infrastructure Provisioning (~5s)

Cloud Run allocates the physical GPU and injects pre-installed NVIDIA drivers. Since Google manages the drivers for you, you don't have to bloat your Dockerfile.

Phase 2: Block-Level Container Image Streaming (1-2s)

Cloud Run uses "image streaming," meaning it pulls only the blocks needed to boot. Your 15GB CUDA image can actually start as fast as a tiny Node.js app!

Phase 3: Engine Initialization (5-15s)

This is where your inference engine (vLLM, Ollama) warms up. This is a massive CPU-heavy task, and it's where most people get throttled without realizing it.

Phase 4: Model Loading & VRAM Transfer

This is the final hurdle - moving those model weights from storage into the GPU memory. Unlike standard web apps where CPU is king, GPU memory is your primary constraint here. If your model’s weights don’t fit entirely within the GPU memory, performance degrades significantly as it swaps to slower system RAM.

Best practices to handling AI cold starts

To build a "sane" production environment, here are a few crucial levers you can pull, informed by the official Google Cloud documentation on AI inference with GPUs.

Optimize Phase 4

Pick the Right Deployment Option

Phase 4 is the "final hurdle" where you move gigabytes of weights from storage into GPU memory. Your choice of storage determines how fast this transfer happens:

Cloud Storage (Concurrent Download) - Fastest: Using the Google Cloud CLI (gcloud storage cp) allows you to download model files in parallel. This is the recommended method for massive weights because it maximizes network throughput and drastically reduces transfer time.
Cloud Storage (FUSE) - Easiest: This provides "zero-code" changes by mounting a bucket as a local file system. However, because it does not parallelize the initial download, it is significantly slower for large model weights
Container Image - Best for <10GB: Baking weights into your image is efficient for smaller models thanks to Cloud Run's Image Streaming. For models over 10GB, however, the import and streaming overhead can become a bottleneck.
Internet: Avoid this. It is the slowest and least predictable path for production inference.

Model Format & Size

Optimizing your model's format and size is a direct "hack" to shorten Phase 4 (Model Loading & VRAM Transfer). Because this phase is constrained by how fast you can move gigabytes of data into VRAM, smaller and more efficient files are critical.

4-bit Quantization: This is the ultimate cold start hack. Smaller weights mean fewer gigabytes to pull from storage, which directly accelerates the download and transfer portion of Phase 4,
Fast Formats: Pick a model format with fast load times like GGUF to minimize startup time. For the fastest performance, move away from Python "pickle" files and use Safetensors for zero-copy loading.
Ensure VRAM Fit: Use quantized models to ensure the weights fit entirely within the GPU memory. If the model exceeds VRAM, Phase 4 will stall as the system swaps to significantly slower RAM.

Optimize Phases 3 & 4: Infrastructure & Network Levers

These infrastructure settings provide the necessary resources to accelerate the most demanding parts of the startup process.

Startup CPU Boost (Accelerates Phase 3)

This feature temporarily doubles your CPU power during startup. A 1 vCPU instance boosts to 2 vCPUs for the duration of startup and the first 10 seconds of serving. It is essential for Phase 3, as engine initialization is a massive CPU-heavy task.

Direct VPC Egress & PGA (Accelerates Phase 4)

Utilizing Direct VPC Egress with Private Google Access (PGA) ensures your model weight traffic stays on Google’s internal high-speed backbone. This optimizes the network path to shorten the time spent moving gigabytes of weights into VRAM.

Concurrency Tuning (Cold Start Avoidance):

In Cloud Run, "concurrency" refers to the maximum number of requests a single instance can handle before the platform scales out to start a new one. For AI workloads, you must tune this setting in tandem with your model engine's internal parallelism flags (e.g., --max-num-seqs for vLLM or OLLAMA_NUM_PARALLEL for Ollama).

Use the official Google Cloud formula to find your ideal Cloud Run concurrency:

(Number of model instances∗parallel queries per model)+(number of model instances∗ideal batch size)

Example: If your instance loads 3 model instances onto the GPU, and each model instance can handle 4 parallel queries with an ideal batch size of 4, you would set your Cloud Run maximum concurrent requests to 24: (3×4)+(3×4)

How the math works: The goal is to keep the GPU fully saturated while ensuring users aren't stuck in a long queue. In this example, the total of 24 concurrent requests is split into two functional groups:

Active Processing (12 requests): Calculated as (3 instances×4 queries), this represents the total number of requests the GPU can actively process at any given moment.
The "Next Batch" Buffer (12 requests): Calculated as (3 instances×4 batch size), these are the requests waiting "on deck" inside the container. As soon as the GPU finishes the first batch, it immediately picks up these waiting requests.

By tuning this value as high as your VRAM allows (usually 10-20 users), one warm instance can serve many requests without triggering a new scale-out event and the cold start that comes with it.

Scaling Controls (Tuning the Threshold)

While the formula above defines your maximum capacity, you can also tune when Cloud Run decides to start the next instance. Cloud Run's autoscaler typically targets 60% utilization, but for long-running AI cold starts, you can increase this threshold to 80% or 90% via Scaling Controls.

Concurrency Target: Increasing this allows you to "pack" more requests into a single warm instance before triggering a scale-out.
CPU Target: Increasing the CPU target prevents the platform from starting a new instance just because initialization or high-intensity inference spiked the CPU utilization.

Scaling & Reliability Strategies

Sometimes the best way to handle a cold start is to avoid it entirely or manage it proactively.

The Single-Region "Always-On" Tradeoff

If you are deploying globally, the cost of keeping minimum instances set to 1 in every region adds up. Instead, consider an 'always-on' service in just one region. A 100ms global network delay is a much better user experience than a 20s local cold start.

The 15-Minute Grace Period: A common question is 'How long will my instance stay warm after a request?' Cloud Run generally keeps instances alive for 15 minutes after they become idle (processing zero requests). If your traffic is predictable and comes in every 10–12 minutes, you might not even need an 'always-on' service, the platform’s default shutdown policy will keep a warm instance ready for your next user.

Note: While this idle time is "free" for standard request-based services, remember that GPU services require instance-based billing, so you will be billed for the duration the instance remains warm between requests.

The "Wake-Up Call" Strategy

Sometimes the best way to handle a cold start is to proactively mask it. If your UI can predict an upcoming request, for example, when a user clicks "New Chat" or begins hovering over a text area, you can send a lightweight health check to your service immediately. By the time the user finishes typing their prompt, the first two phases of the cold start (Infrastructure Provisioning and Container Image Streaming) are already finished in the background.

Pro-Tip: Use Non-Inference Endpoints To make this "wake-up call" as fast as possible, always use a non-inference endpoint rather than sending a dummy prompt like "hi".

Why it’s faster: Non-inference endpoints (like /v1/models for vLLM or /api/tagsfor Ollama) are handled by the container’s web server the moment it starts. They don’t have to wait for the slow "Phase 4" model loading and VRAM transfer to complete before sending a success response.
No Chat Pollution: Because these endpoints don't trigger the model's completion logic, they won't interfere with the user's actual chat history or accidentally trigger session creation in your backend.

Recommended Endpoints:

vLLM: GET /health or GET GET /v1/models
Ollama: GET /api/tags or GET /api/version

Tune Startup Probes for VRAM

AI models take significant time to move gigabytes of weights from storage into GPU memory (Phase 4). If your startup check fails too many times, Cloud Run will assume your container is broken and kill it.

To prevent this:

Increase the Failure Threshold: Use a high failureThreshold (e.g., 60 or more). Since the total allowed startup time is the product of failureThreshold \times periodSeconds, a threshold of 60 with a 5-second period gives your model a healthy 5-minute window to load.
Utilize the 30-Minute Maximum: While standard services are limited to 4 minutes, Cloud Run supports a total startup time of up to 30 minutes (1,800 seconds) for intensive workloads.
Avoid False Positives (The Ollama Fix): Be careful with engines like Ollama, which may open a TCP port as soon as the service starts, but before the model is actually in VRAM. Always ensure you are preloading models during the container's entrypoint script to ensure the startup probe only passes once the model is truly ready for inference.

Lessons from Elastic’s strategy

In our NEXT ‘26 session, Ajay Nair highlighted three architectural decisions that allowed Elastic to treat GPUs as fungible compute, rather than infrastructure to manage:

Bypass the Compilation Tax: By setting enforce_eager=True in vLLM, they traded a tiny bit of throughput for cold starts that finish in less than a minute rather than multiple minutes.
Standalone Checkpoints: They avoided the latency of runtime adapter-switching by pre-merging each LoRA variant into a standalone checkpoint.
One Workload, One Service: Each independently-scalable workload — defined by model, task adapter, and traffic shape — is deployed as its own Cloud Run service. This produces 30+ services across ~15 model families, with some models split by task (e.g., v5 retrieval vs. clustering) or by query/passage role.

Ready to get started?

Optimizing the cold start process is the difference between a hobby project and a production-ready application. The best part? Cloud Run handles the NVIDIA driver and CUDA installation for you, starting the instance in about 5 seconds.

For a deeper dive, the official documentation is your best friend:

For the full technical breakdown, I highly recommend watching the recording of the session from Google Cloud Next '26. It provides the most comprehensive blueprint for hosting high-performance open models on serverless infrastructure.

Happy building!

Special thanks to Sara Ford and Shane Ouchi from the Cloud Run team and to Zac Li from Elastic for the helpful review and feedback on this article.

Mastering Gemini Computer Use : A Comprehensive Hands-on Guide

Romin Irani — Fri, 26 Jun 2026 10:34:30 +0000

Gemini Computer Use lets you build AI agents that control real screens by looking at screenshots and deciding what to click, type, or scroll, just like a human would.

You can read the announcement post below:

Introducing computer use in Gemini 3.5 Flash

This tutorial walks you through five progressive steps (from “Hello Screenshot” to Gemini Enterprise Agent Platform), then shows five real-world use cases. All code is included and runnable.

What is Computer Use?

If you’ve ever written a Selenium test or a Puppeteer script, you know the script and the friction that comes with it. You spend hours crafting CSS selectors, XPath expressions, and fragile wait conditions. Then one day, the website changes a class name, and everything breaks. This is the fragile nature of selector-based automation, where it depends on the implementation details of the UI rather than what’s actually visible on the screen.

And there’s a deeper problem. Not every application exposes an API. Legacy enterprise software, desktop apps, mobile interfaces , there’s an enormous surface area of software that can only be operated by a human looking at a screen and clicking around.

The Solution: AI That Sees

Gemini Computer Use takes a radically different approach. Instead of parsing HTML or querying the DOM, the model looks at a screenshot and decides what to do, exactly the way a human would. It doesn’t need selectors. It doesn’t need APIs. It just needs eyes.

The paradigm is beautifully simple: Observe → Think → Act.

Observe: Capture a screenshot of the current screen state
Think: Send it to Gemini 3.5 Flash, which analyzes the visual content and decides the next action
Act: Execute the action (click a button, type text, scroll down), then capture a new screenshot and loop back

This loop continues until the model decides the task is complete and responds with text instead of a function call.

How Computer Use Works: Brain, Eyes, and Hands

Every Computer Use agent has three parts:

The Brain is always the same — Gemini 3.5 Flash. But the Hands change depending on what you’re controlling:

┌─────────────────────────────────────────────────────────────────┐
│ 🧠 BRAIN (always the same) │
│ Gemini 3.5 Flash │
│ "Look at this screenshot, what should I do?" │
└──────────────────────────┬──────────────────────────────────────┘
                           │
              Model returns: click(x=396, y=185)
                           │
          ┌────────────────┼────────────────┐
          ▼ ▼ ▼
   🖐️ Browser 🖐️ Mobile 🖐️ Enterprise
   Playwright ADB CDP Sandbox
   page.mouse adb shell Remote browser
   .click(506,148) input tap in the cloud
                      506 148

Why this matters: The model doesn’t know or care what “hands” you’re using. It just sees screenshots and returns actions. You can swap Playwright for Selenium, ADB for a physical device, or a local browser for a cloud sandbox — the model’s output is identical. The only thing that changes is the environment parameter you declare (browser, mobile, or desktop), which tells the model what kind of screen it's looking at.

Why Gemini 3.5 Flash is Special

Computer Use isn’t a separate model or a bolted-on capability. In gemini-3.5-flash, it's a native tool, declared alongside other tools like function calling and code execution. This design decision has key implications and its worthwhile to understand them:

Multi-tool composition: The model can interleave browser actions with custom function calls in the same conversation. Click a button, then call save_product() to store data, then scroll down for more.
Three environments: One model supports ENVIRONMENT_BROWSER, ENVIRONMENT_MOBILE, and ENVIRONMENT_DESKTOP. You declare the environment and the model adapts its available actions.
Thinking mode: You can enable ThinkingConfig(include_thoughts=True) to see the model's reasoning before each action, invaluable for debugging.

Key Technical Concepts

Normalized Coordinates (0–999): The model doesn’t know your screen resolution. Instead, it outputs coordinates in a normalized 0–999 range. You convert them to pixels with:

pixel = int(normalized / 1000 * screen_dimension)

For a 1280×800 viewport, coordinate (500, 500) maps to pixel (640, 400) — the centre of the screen.

The intent Field: Each function call includes an intent string describing what the model is trying to achieve , not just the mechanical action. This is crucial if you try to debug the whole flow.

Safety Decisions: When the model encounters potentially sensitive actions (making a purchase, sending a message, deleting data), it includes a safety_decision object requesting explicit user confirmation.

What Changed from Legacy to Current

The previous gemini-2.5-computer-use-preview model was an early experiment with a separate API surface. The current gemini-3.5-flash approach integrates Computer Use as a standard tool. This means the same model, the same SDK, and the same generate_content() call, that you already know.

Two Paths: Gemini API vs Enterprise Platform

There are two ways to use Gemini Computer Use, and which one you choose depends on where you are in the development lifecycle.

Gemini API (ai.google.dev)

The fastest path from zero to working agent. Grab an API key, pip install the SDK, and you’re running. You build your own execution environment (Playwright for browsers, ADB for Android), which gives you maximum flexibility and zero cloud dependencies.

Enterprise Agent Platform (Vertex AI)

When you’re ready for production, Vertex AI adds IAM-based authentication, managed browser sandboxes (no local browser needed), VPC Service Controls, audit logging, and all the enterprise trimmings. The important part? The code is almost identical. You change one line.

The migration between paths is a single-line change in client construction:

Before — Gemini API (prototyping)

# Gemini API (prototyping)
client = genai.Client(api_key="YOUR_KEY")

After — Vertex AI (production)

# That's it — that's the migration
client = genai.Client(
    vertexai=True,
    project="my-project",
    location="us-central1",
)

Prerequisites & Setup

In this section, we will identify the software that we need. This completely depends on what use cases and scenarios that you are interested in, while using Gemini Computer Use.

The table given in the section What You Need identifies the requirement and which step/use case that you need it for. For e.g. if you are not interested in Android app scenarios or running this on Google Cloud, you can simply do with Python 3.10+ and Gemini API Key only. Each of the steps and use cases are mentioned clearly in the article. So as you go along, remember to have these requirements handy.

What You Need

Step-by-Step Installation

The examples are code heavy and hence I have made a code repository available for you. The tutorial too goes into each of the folders i.e. steps or use cases in the repository, and you can follow the steps to run the examples.

GitHub - rominirani/computer-use-tutorial: Mastering Gemini Computer Use: A Comprehensive Hands-On Guide - Build AI agents that control browsers, mobile devices, and desktops with Gemini 3.5 Flash

The Repository structure is shown below:

computer-use-tutorial/
├── README.md ← You are here
├── requirements.txt ← Python dependencies
├── .env.example ← Template for API key
├── 01-hello-screenshot/ ← Step 1: Visual understanding
│ └── hello_screenshot.py
├── 02-single-action/ ← Step 2: One screenshot → one click
│ └── single_action.py
├── 03-browser-agent/ ← Step 3: Full agentic browser loop
│ ├── browser_agent.py
│ └── playwright_env.py
├── 04-mobile-agent/ ← Step 4: Android + Interactions API
│ ├── mobile_agent.py
│ ├── adb_bridge.py
│ └── setup_emulator.sh
├── 05-enterprise-platform/ ← Step 5: Vertex AI + sandboxes
│ └── enterprise_agent.py
└── 06-use-cases/ ← 5 real-world use cases
    ├── usecase1_qa_testing/
    ├── usecase2_price_comparison/
    ├── usecase3_mobile_testing/
    ├── usecase4_web_research/
    └── usecase5_form_filling/

1. Clone the repository

git clone https://github.com/rominirani/computer-use-tutorial
cd computer-use-tutorial

2. Create a Python virtual environment

python3 -m venv .venv
source .venv/bin/activate # macOS / Linux
# On Windows: .venv\Scripts\activate

⚠️ Important : Always activate the virtual environment before running any tutorial script. If your shell prompt doesn’t show (.venv), run source .venv/bin/activate again.

3. Install Python dependencies

pip install -r requirements.txt

The requirements.txt is intentionally lean:

google-genai>=2.7.0
playwright==1.55.0
pydantic>=2.0
rich
termcolor
python-dotenv

What is Playwright doing here?

Playwright is the “ hands ” of the agent. Gemini Computer Use gives the model eyes (it can see screenshots) and a brain (it decides what to do), but it has no way to actually interact with a browser. That’s where Playwright comes in.

It is an open-source browser automation library from Microsoft. It lets Python code control a real Chromium browser i.e. open pages, click at pixel coordinates, type text, scroll, take screenshots, intercept popups, etc.

Why we need it?

The Gemini model returns structured instructions like:

click(x=396, y=185)
type(text="hello world")
scroll(direction="down")

But these are just data. The model can’t execute them. Something needs to translate click(x=396, y=185) into an actual mouse click inside a real browser. Playwright does that:

Model says click(x=396, y=185) → Playwright runs page.mouse.click(506, 148)
Model says type("hello") → Playwright runs page.keyboard.type("hello")
Model says take a screenshot → Playwright runs page.screenshot() → PNG bytes

Could you use something else?

Yes you could. Selenium, Puppeteer, or any browser automation tool would work. Playwright is chosen here because it’s fast, has a clean Python API, handles headless mode well, and has built-in screenshot capture. For mobile, we use ADB instead of Playwright (same role, different “hands”).

Playwright doesn’t use the Chrome/Chromium already on your machine. It ships its own browser binaries that are patched for automation, they expose internal APIs (like the Chrome DevTools Protocol) that Playwright uses to control the browser programmatically (click, type, screenshot, navigate, etc.).

playwright install chromium downloads this patched Chromium binary (~150 MB) into a Playwright-managed directory (e.g. ~/Library/Caches/ms-playwright/).

4. Install the Chromium browser binary

playwright install chromium

Verify it works:

python3 -c "from playwright.sync_api import sync_playwright; pw = sync_playwright().start(); b = pw.chromium.launch(); print('✅ Playwright Chromium works'); b.close(); pw.stop()"

5. Configure your Gemini API key

Please get a Gemini API Key from here.

cp .env.example .env
# Edit .env and replace 'your-api-key-here' with your actual key

Your .env file should look like:

GEMINI_API_KEY=AIzaSy...(your actual key)

6. Verify — run Step 01

cd 01-hello-screenshot
python hello_screenshot.py

If you see the model describe the Hacker News page, your setup is complete for Steps 1–3 and browser-based use cases. 🎉

In my case here, this is what I got as output:

✓ Gemini client initialised

============================================================
  Step 1 → Launch browser
  Opening https://news.ycombinator.com
============================================================

  → Page loaded: Hacker News
  → Viewport : 1280×800

============================================================
  Step 2 → Capture screenshot
  Taking a full-viewport PNG snapshot
============================================================

  → Screenshot captured: 206.7 KB
  → Saved to : /Users/romin/computer-use-tutorial/01-hello-screenshot/screenshot.png

============================================================
  Step 3 → Send to Gemini
  Asking the model to describe the screenshot
============================================================

============================================================
  Step 4 → Model response
  Gemini's description of the screenshot
============================================================

Based on the screenshot provided, here is a detailed description of the webpage:

### **Website Name and Identity**
* **Site Name:**  **Hacker News** (a social news website focusing on computer science and entrepreneurship, run by the investment fund Y Combinator).
* **Logo:** A small, bright orange square containing a stylized white letter **"Y"** sits in the top-left corner.

### **Color Scheme**
* **Primary Accent:** Bright safety orange (`#ff6600`), used exclusively for the top navigation bar.
* **Background:** The main content area has a very light, warm gray/off-white background, while the outer margins of the page are plain white.
* **Text:** Dark gray to black for the main headlines, and a lighter, muted gray for metadata (points, time, domains, and comment counts).

### **Layout and Navigation**
* **Structure:** The site features an extremely minimalist, text-heavy, "Web 1.0" style layout. The content is centered in a single vertical column with wide white margins on the left and right.
* **Header Bar:** A solid orange horizontal bar spans the top of the content column. 
    * On the left, next to the "Y" logo, is the bold text **"Hacker News"** followed by a pipe-separated menu of navigation links: `new | past | comments | ask | show | jobs | submit`.
    * On the far right of this bar is a single `login` link.
* **Main List:** Below the header is a sequentially numbered list of stories (numbered 1 through 22 are visible). 

### **Anatomy of a List Item**
Each item in the list follows a strict, consistent format:
1. **Number and Upvote Arrow:** A number (e.g., "1.") followed by a small, light-gray upward-pointing triangle (▲) used for voting.
2. **Headline:** The title of the article, styled as a clickable dark link.
3. **Source Domain:** Directly next to the headline, the hosting website's domain is listed in parentheses in light gray (e.g., `(om.co)` or `(github.com/plbrault)`).
4. **Metadata Line:** Positioned directly below the headline in smaller, lighter text. This includes:
    * The score (e.g., "843 points")
    * The submitting user's handle (e.g., "by minimaxir")
    * How long ago it was posted (e.g., "11 hours ago")
    * A "hide" link
    * The comment count, which links to the discussion page (e.g., "96 comments").

### **Visible Top Headlines**
The top stories visible on this front page cover tech news, programming, hardware, and history. Notable headlines include:
1. *Om Malik has died* (om.co)
2. *Why current LLM costs are not sustainable* (patadia.org)
3. *An entire Herculaneum scroll has been read for the first time* (scrollprize.org)
4. ....
10. ....

────────────────────────────────────────────────────────────
Cleaning up browser resources...
✓ Done — screenshot saved to screenshot.png

(Optional) Android Emulator Setup

Skip this if you only want the browser-based examples (Steps 1–3, Use Cases 1–2, 4–5).

Option A: Use the included setup script

cd 04-mobile-agent
chmod +x setup_emulator.sh
./setup_emulator.sh

This idempotent script installs Java, Android SDK, creates an AVD named ComputerUseTutorial, and prints the export lines for your shell.

If you are familiar with Android SDK, feel free to look at the setup_emulator.sh script and create an AVD named ComputerUseTutorial.

Option B: Existing Android SDK

# Common ANDROID_HOME locations:
export ANDROID_HOME=~/Library/Android/sdk # Android Studio default
export ANDROID_HOME=/opt/homebrew/share/android-commandlinetools # Homebrew (Apple Silicon)
# Add to PATH
export PATH="$ANDROID_HOME/emulator:$ANDROID_HOME/platform-tools:$PATH"
# Add to ~/.zshrc so it persists
echo 'export ANDROID_HOME=~/Library/Android/sdk' >> ~/.zshrc
echo 'export PATH="$ANDROID_HOME/emulator:$ANDROID_HOME/platform-tools:$PATH"' >> ~/.zshrc
source ~/.zshrc

Starting and verifying the emulator

# List and start
emulator -list-avds
emulator -avd <your-avd-name>
# Verify it's ready
adb devices # → emulator-5554 device
adb shell getprop sys.boot_completed # → 1

(Optional) Google Cloud Setup

Skip this if you don’t have a Google Cloud project or only want the Gemini API.

# Authenticate
gcloud auth login
gcloud auth application-default login
gcloud config set project YOUR_PROJECT_ID
# Enable Vertex AI API
gcloud services enable aiplatform.googleapis.com

Add to your .env:

GCP_PROJECT_ID=your-gcp-project-id
GCP_LOCATION=us-central1

Required IAM role: roles/aiplatform.user

gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
    --member="user:your-email@domain.com" \
    --role="roles/aiplatform.user"

Understanding the Code: Key Computer Use Patterns

Before diving into the step-by-step code, here are the patterns you’ll see in every script.

Pattern 1: Normalized Coordinates (0–999)

The model always outputs coordinates in a 0–999 grid, regardless of screen size. Your code must denormalize:

# Model returns: click(x=396, y=185) — these are NOT pixels!
pixel_x = int(396 / 1000 * viewport_width) # 396 → 506 on a 1280px screen
pixel_y = int(185 / 1000 * viewport_height) # 185 → 148 on an 800px screen

This is the most common source of bugs, forgetting to denormalize will cause clicks to land in the wrong place.

Pattern 2: The Agentic Loop

1. Take screenshot → send to model with task description
2. Model returns: function_call (e.g., click, type, scroll)
3. Execute the action in the browser/device
4. Take a NEW screenshot → send back as FunctionResponse
5. Goto 2 — until model returns plain text (= task complete)

The key insight: the model signals it’s done by returning text instead of function_calls.

Pattern 3: FunctionResponse Must Include a Screenshot

FunctionResponse(
    name="click",
    response={
        "result": "ok",
        "screenshot": types.Part.from_bytes(
            data=screenshot_png, mime_type="image/png"
        ),
    },
)

Without the screenshot, the model is blind and can’t plan its next move.

Pattern 4: Tool Declaration

tools=[
    types.Tool(
        computer_use=types.ComputerUse(
            environment=types.Environment.ENVIRONMENT_BROWSER,
            # or ENVIRONMENT_MOBILE / ENVIRONMENT_DESKTOP
        )
    )
]

This is NOT a regular function declaration — it activates the model’s built-in understanding of screenshots and UI interaction.

Pattern 5: Screenshot Pruning

Screenshots are ~100–300 KB each. In a 20-turn conversation, that’s 2–6 MB of image data in the context:

# Keep only the 3 most recent screenshots
for old_turn in history[:-3]:
    for part in old_turn.parts:
        if hasattr(part, 'inline_data'):
            part.inline_data = None # free memory, keep text

Step-by-Step Tutorial

This tutorial has several steps (numbered 1 to 5). Step 1 to 3 can be done simply via Python and Gemini API Key. Step 4 is about Android , while Step 5 is about Google Cloud.

Step 1 : Hello Screenshot 01-hello-screenshot/

Before we can build an agent that controls a screen, we need to make sure it can see one. In this step, you’ll launch a headless browser, navigate to Hacker News, capture a screenshot, and send it to Gemini 3.5 Flash to get a visual description. There’s no Computer Use here , just the two foundational building blocks (screenshots + the model) that every subsequent step builds on.

Key Concept: This step uses no Computer Use at all. It establishes the two foundational building blocks:

Capturing screenshots
Sending them to Gemini , that every subsequent step depends on.

Code Walkthrough

response = client.models.generate_content(
    model="gemini-3.5-flash",
    contents=[
        types.Content(
            role="user",
            parts=[
                types.Part(text="Describe what you see..."),
                types.Part(
                    inline_data=types.Blob(
                        mime_type="image/png",
                        data=screenshot_bytes,
                    )
                ),
            ],
        )
    ],
)

The SDK handles base64 encoding. You pass raw bytes and the MIME type.

Running It

cd 01-hello-screenshot
python hello_screenshot.py

Expected Output

✓ Gemini client initialised
Step 1 → Launch browser
  Opening https://news.ycombinator.com
  → Page loaded: Hacker News
  → Viewport : 1280×800
  Step 4 → Model response
  The screenshot shows Hacker News (news.ycombinator.com)...

Step 2 : Your First Computer Use Action 02-single-action/

Now we give the model its “ hands.” You’ll show Gemini a screenshot of Wikipedia’s language selection page and ask it to click on the English link. The model will return a function_call with normalized coordinates, and you'll translate those into real pixel positions and execute the click with Playwright. This is a single round-trip — one screenshot in, one action out — but it covers the most critical concept in Computer Use: coordinate normalization.

Key Concept: This is where Computer Use enters the picture. The critical new concept is coordinate normalization.

Code Walkthrough

# Declare the Computer Use tool
computer_use_tool = types.Tool(
    computer_use=types.ComputerUse(
        environment=types.Environment.ENVIRONMENT_BROWSER,
    )
)
# Send request
response = client.models.generate_content(
    model="gemini-3.5-flash",
    contents=request_contents,
    config=types.GenerateContentConfig(tools=[computer_use_tool]),
)
# Parse and execute
for part in response.candidates[0].content.parts:
    if part.function_call:
        fc = part.function_call
pixel_x = int(fc.args["x"] / 1000 * SCREEN_WIDTH)
pixel_y = int(fc.args["y"] / 1000 * SCREEN_HEIGHT)
page.mouse.click(pixel_x, pixel_y)

Running It

cd 02-single-action
python single_action.py

Expected Output

✓ Gemini client initialised

================================================================
  Step 1 → Launch browser
  Navigating to https://www.wikipedia.org
================================================================

  → Page loaded : Wikipedia
  → Current URL : https://www.wikipedia.org/
  → Viewport : 1280×800

================================================================
  Step 2 → Capture screenshot
  Grabbing the Wikipedia portal page
================================================================

  → Screenshot size : 124.7 KB
  → Saved to : /Users/romin/computer-use-tutorial/02-single-action/01_before_click.png

================================================================
  Step 3 → Ask Gemini to click
  Sending screenshot + instruction with Computer Use tool
================================================================

  → Response received from Gemini

================================================================
  Step 4 → Parse response
  Extracting the function call from the model
================================================================

  Function call received:
    Name : click
    x : 396
    intent: Click on the English Wikipedia link to navigate to the English Wikipedia.
    y : 182

================================================================
  Step 5 → Execute click
  Converting coordinates and clicking
================================================================

  Coordinate conversion:
    Normalised : (396, 182) [0-999 space]
    Pixel : (506, 145) [1280×800 viewport]

  → Click executed at pixel (506, 145)
  → Page navigated to: https://www.wikipedia.org/

================================================================
  Step 6 → Verify result
  Taking a post-click screenshot
================================================================

  → Screenshot size : 315.8 KB
  → Saved to : /Users/romin/computer-use-tutorial/02-single-action/02_after_click.png
  → Current URL : https://en.wikipedia.org/wiki/Main_Page
  → Page title : Wikipedia, the free encyclopedia

  ✅ SUCCESS — Navigated to English Wikipedia!

────────────────────────────────────────────────────────────────
Cleaning up browser resources...
✓ Done — screenshots saved to 01_before_click.png & 02_after_click.png

Step 3 : Building a Full Browser Agent 03-browser-agent/

This is the big leap where we are going from a single action to a fully autonomous agent. You’ll build a complete browser agent that can accept any task (“find the top 3 stories on Hacker News”), take screenshots, send them to the model, execute the returned actions, capture new screenshots, and loop until the task is done. Along the way, you’ll tackle the real-world challenges: managing a growing conversation history, pruning old screenshots to stay within context limits, handling safety decisions, and recovering from transient API errors with exponential backoff.

Key Concept: This is the big step, going from a single action to a fully autonomous agent.

Critical Pattern: Screenshot Pruning

def _prune_old_screenshots(self) -> None:
    screenshots_seen = 0
    for content in reversed(self._history):
        if content.role != "user" or not content.parts:
            continue
        has_screenshot = any(
            p.function_response and p.function_response.parts
            for p in content.parts
        )
        if has_screenshot:
            screenshots_seen += 1
            if screenshots_seen > self.MAX_SCREENSHOTS_IN_HISTORY:
                for part in content.parts:
                    if part.function_response and part.function_response.parts:
                        part.function_response.parts = None

Critical Pattern: FunctionResponse Construction

FunctionResponse(
    name=fc.name,
    response={"url": current_url},
    parts=[
        types.FunctionResponsePart(
            inline_data=types.FunctionResponseBlob(
                mime_type="image/png",
                data=post_action_png,
            ),
        ),
    ],
)

Running It

cd 03-browser-agent
python browser_agent.py --task "Go to https://news.ycombinator.com and tell me the top 3 stories"

Expected Output

╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Gemini Computer Use — Browser Agent │
│ Model: gemini-3.5-flash | Viewport: 1280×800 | Headless: False │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
────────────────────────────────────────────────────────── Browser Agent Starting ──────────────────────────────────────────────────────────
Task: Go to https://news.ycombinator.com and tell me the top 3 stories

Initial screenshot captured.

─────────────────────────────────────────────────────────────── Iteration 1 ────────────────────────────────────────────────────────────────
╭───────────────────────────────────────────────────────────── Model Thinking ─────────────────────────────────────────────────────────────╮
│ Okay, here's my thought process on this task: │
│ │
│ **Accessing and Prioritizing Hacker News Articles** │
│ │
│ Right, so the goal is clear: identify the top three stories currently dominating Hacker News. My immediate instinct is to open the site. │
│ It's second nature at this point – navigate to `https://news.ycombinator.com`. Once I'm there, the real work begins. I'll need to sift │
│ through the current submissions, using my knowledge of the site's ranking algorithm and usual trends, to identify which articles are │
│ garnering the most attention. I'll be looking at score, comment count, and potentially the recency of the posts to figure out what's │
│ currently hot and relevant. The top three are what I'm looking for, and I can be flexible with the ranking methodology. │
│ │
│ │
│ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
  ▶ navigate(intent=Navigate to Hacker News, url=https://news.ycombinator.com)
  → Screenshot captured | URL: https://news.ycombinator.com/

─────────────────────────────────────────────────────────────── Iteration 2 ────────────────────────────────────────────────────────────────
╭───────────────────────────────────────────────────────────── Model Thinking ─────────────────────────────────────────────────────────────╮
│ Okay, here's my take on those Hacker News headlines, as I see them: │
│ │
│ **Hacker News Digest - A Quick Glance** │
│ │
│ Alright, let's process these top stories on Hacker News. First up, the Patadia article on LLM costs – always a crucial point for │
│ long-term viability, and something I've been tracking myself. The core of this research is likely to be a deep dive into the │
│ computational inefficiencies, scaling issues, and the sheer energy consumption that underpins current models. I'm especially keen to see │
│ what specific cost-saving strategies the author proposes, and whether they involve architectural improvements, hardware optimization, or │
│ more fundamental algorithmic innovations. That's definitely one to bookmark and dig into. │
│ │
│ Then, a somber note: Om Malik has passed. A prominent voice in the tech sphere is gone. Always a difficult thing to grapple with when a │
│ familiar and respected figure leaves the scene. My thoughts are with his friends and family. A pause for reflection is definitely │
│ warranted. │
│ │
│ Finally, the Herculaneum scroll! This is truly fascinating. Reading an entire scroll from Herculaneum is a significant breakthrough. I'm │
│ guessing the team has leveraged advanced imaging techniques like X-ray tomography and computational tools for deciphering fragmented │
│ texts. What surprises me is this is a first. I'll need to research what the issues have been in deciphering this, if the material has │
│ been too degraded for even cutting-edge processing, or if this new tech is truly groundbreaking. The content itself will undoubtedly │
│ offer invaluable insights into ancient history, philosophy, and everyday life. A promising development. │
│ │
│ │
│ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

────────────────────────────────────────────────────────────── Task Complete ───────────────────────────────────────────────────────────────

Agent Summary: The top 3 stories on Hacker News are:

1. **Why current LLM costs are not sustainable** (patadia.org)
2. **Om Malik has died** (om.co)
3. **An entire Herculaneum scroll has been read for the first time** (scrollprize.org)

─────────────────────────────────────────────────────────────── Final Result ───────────────────────────────────────────────────────────────

The top 3 stories on Hacker News are:

1. **Why current LLM costs are not sustainable** (patadia.org)
2. **Om Malik has died** (om.co)
3. **An entire Herculaneum scroll has been read for the first time** (scrollprize.org)

Architecture: browser_agent.py (AI loop) + playwright_env.py (browser abstraction).

Step 4 : Android Mobile Agent 04-mobile-agent/

We swap the browser for a phone. In this step, you’ll connect to an Android emulator via ADB, capture screenshots of the device screen, and let Gemini drive it with touch gestures : taps, swipes, and long-presses. This step also introduces the Interactions API, a stateful alternative to generateContent where the server manages conversation history for you. Instead of sending the full history every turn, you just chain calls with previous_interaction_id , thereby making the code simpler.

Interactions API vs generateContent

ADB Bridge Pattern

def click(self, y: int, x: int, **_) -> None:
    px, py = self._denormalize(x, y)
    self._execute(["shell", "input", "tap", str(px), str(py)])

Running It

# Start emulator first (see Android Setup above)
cd 04-mobile-agent
python mobile_agent.py "Open Settings and check the Android version"

Expected Output

╔══════════════════════════════════════════════════════════╗
║ Gemini Computer Use — Android Mobile Agent ║
║ Model: gemini-3.5-flash | API: Interactions ║
╚══════════════════════════════════════════════════════════╝
Turn 1/50
  🔧 Action: open_app({'package_name': 'com.android.settings'})
  ✓ Result: {'status': 'ok'}
  ... (4-6 turns: scrolls, clicks About Phone, reads version) ...
✅ Agent completed the task!
   Model response: The Android version is 16.

Step 5 : Enterprise Platform 05-enterprise-platform/

Everything you’ve built so far works with a free Gemini API key. But when you’re ready for production i.e. IAM authentication, VPC isolation, audit logging, and managed browser sandboxes, you can move to Vertex AI. The best part? The agent code is virtually identical. You change one line to switch from API-key auth to IAM, and optionally connect to a cloud-hosted sandbox browser instead of launching a local one. This step walks through both approaches.

The One-Line Migration

Your agent code — the agentic loop, the action dispatch, the screenshot pruning — is 100% identical between Gemini API and Vertex AI. The only change:

CDP Sandbox Connection

# Local browser (Steps 1-3):
browser = pw.chromium.launch(headless=True)
# Managed sandbox (Step 5):
browser = pw.chromium.connect_over_cdp(cdp_endpoint)

Running It

cd 05-enterprise-platform
# Approach 1: Self-managed browser (your local Playwright)
python enterprise_agent.py --approach self-managed --project YOUR_PROJECT_ID
# Approach 2: Managed sandbox (cloud-hosted browser)
python enterprise_agent.py --approach managed-sandbox --project YOUR_PROJECT_ID

💡 Tip : Replace YOUR_PROJECT_ID with your actual Google Cloud project ID. You can also set it in .env as GCP_PROJECT_ID=my-company-prod-123.

Part 5: Real-World Use Cases

Each use case in 06-use-cases/ is a complete, self-contained script demonstrating a practical application.

Use Case 1: QA Testing (TodoMVC)

Scenario : Run end-to-end QA tests on a React web app without writing fragile selectors. The agent sees the app and tests it the way a human QA tester would.

Architecture: generateContent API with a full agentic loop + custom report_qa_result function (multi-tool composition).

The Prompt

Navigate to https://todomvc.com/examples/react/dist/ and perform QA testing:
1. Add three todos: 'Buy groceries', 'Read a book', 'Write code'
2. Mark 'Read a book' as complete
3. Verify items are displayed correctly
4. Click the 'Completed' filter
5. Report what you see

Running It

cd 06-use-cases/usecase1_qa_testing
python qa_agent.py

Step-by-Step Walkthrough

Agent navigates to TodoMVC React app
Clicks “What needs to be done?” input and types “Buy groceries” + Enter
Repeats for “Read a book” and “Write code”
Clicks the circle next to “Read a book” to mark it complete
Verifies all three items are visible with correct state
Clicks the “Completed” filter
Confirms only “Read a book” appears with strikethrough
Returns a structured test report

Expected Results

╔══════════════════════════════════════════════════════════════╗
║ Use Case 1: Automated QA Testing Agent — TodoMVC ║
╚══════════════════════════════════════════════════════════════╝

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Step 1 → Validate environment
  Checking GEMINI_API_KEY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ✓ API key found

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Step 2 → Launch browser
  Opening https://todomvc.com/examples/react/dist/
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ✓ Browser launched — viewport (1440, 900)
  ✓ Page loaded: https://todomvc.com/examples/react/dist/
  ✓ Initial screenshot: 87 KB

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Step 3 → Initialise QA agent
  Setting up Gemini Computer Use loop
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ✓ Agent ready — initial screenshot attached to prompt

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Step 4 → Run QA test plan
  Agent is now autonomously executing tests…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

────────────────────────────────────────────────────────────────
  🤖 Agent Turn 1
────────────────────────────────────────────────────────────────
Tools at indices [1] are not compatible with automatic function calling (AFC). AFC is disabled. If AFC is intended, please include python callables in the tool list, and do not include function declaration and MCP server in the tool list.
  💭 Reasoning: **QA Test Plan: TodoMVC React Application - Add Todo Items**

Okay, let's get down to brass tacks. I need to run a QA test plan on this TodoMVC React application. The first test is pretty straightforw…
  ▶ Action: click(x=603, y=180, intent=Click the todo input field to focus it.)
     📸 Screenshot captured (87 KB) — URL: https://todomvc.com/examples/react/dist/

────────────────────────────────────────────────────────────────
  🤖 Agent Turn 2
────────────────────────────────────────────────────────────────
Tools at indices [1] are not compatible with automatic function calling (AFC). AFC is disabled. If AFC is intended, please include python callables in the tool list, and do not include function declaration and MCP server in the tool list.
  💭 Reasoning: Here's my attempt at summarizing the thought process, tailored for an expert audience:

**Initiating Task Execution**

Okay, so the prompt is straightforward. It's an instruction set. It's not asking …
  ▶ Action: type(intent=Type 'Buy groceries' and press Enter., press_enter=True, text=Buy groceries)
     📸 Screenshot captured (94 KB) — URL: https://todomvc.com/examples/react/dist/

────────────────────────────────────────────────────────────────
  🤖 Agent Turn 3
────────────────────────────────────────────────────────────────
Tools at indices [1] are not compatible with automatic function calling (AFC). AFC is disabled. If AFC is intended, please include python callables in the tool list, and do not include function declaration and MCP server in the tool list.
  💭 Reasoning: Here's my attempt at summarizing the thought process, tailored for an expert audience:

**Initiating a Simple Instruction**

Okay, so the prompt is incredibly straightforward, almost rudimentary. It's…
  ▶ Action: type(text=Read a book, intent=Type 'Read a book' and press Enter., press_enter=True)
     📸 Screenshot captured (98 KB) — URL: https://todomvc.com/examples/react/dist/

────────────────────────────────────────────────────────────────
  🤖 Agent Turn 4
────────────────────────────────────────────────────────────────
Tools at indices [1] are not compatible with automatic function calling (AFC). AFC is disabled. If AFC is intended, please include python callables in the tool list, and do not include function declaration and MCP server in the tool list.
  💭 Reasoning: Here's my attempt at summarizing the thought process, tailored for an expert audience:

**Initiating the Inquiry into Gemini's Capabilities**

Okay, so the prompt directs me to explore the "Learn Gemi…
  ▶ Action: type(intent=Type 'Learn Gemini Computer Use' and press Enter., text=Learn Gemini Computer Use, press_enter=True)
     📸 Screenshot captured (103 KB) — URL: https://todomvc.com/examples/react/dist/

────────────────────────────────────────────────────────────────
  🤖 Agent Turn 5
────────────────────────────────────────────────────────────────
Tools at indices [1] are not compatible with automatic function calling (AFC). AFC is disabled. If AFC is intended, please include python callables in the tool list, and do not include function declaration and MCP server in the tool list.
  💭 Reasoning: Okay, here's my understanding, formatted as requested:

**Assessing Task Completion**

Right, let's just make absolutely sure we've got a handle on this. The task asks me to verify the presence of thr…
  ▶ Action: report_qa_result(test_name=Add Todo Items, details=Successfully added 'Buy groceries', 'Read a book', and 'Learn Gemini Computer Use' to the todo list., passed=True)

  📋 QA Result: Add Todo Items → [PASS ✅]
     Details : Successfully added 'Buy groceries', 'Read a book', and 'Learn Gemini Computer Use' to the todo list.

────────────────────────────────────────────────────────────────
  🤖 Agent Turn 6
────────────────────────────────────────────────────────────────
Tools at indices [1] are not compatible with automatic function calling (AFC). AFC is disabled. If AFC is intended, please include python callables in the tool list, and do not include function declaration and MCP server in the tool list.
  💭 Reasoning: **Marking an Item as Complete**

Okay, so the next test is to mark an item as complete. Specifically, I need to find the "Read a book" task and visually confirm that it's marked as done after I intera…
  ▶ Action: click(y=316, intent=Click the toggle checkbox next to 'Read a book'., x=429)
     📸 Screenshot captured (105 KB) — URL: https://todomvc.com/examples/react/dist/

────────────────────────────────────────────────────────────────
  🤖 Agent Turn 7
────────────────────────────────────────────────────────────────
Tools at indices [1] are not compatible with automatic function calling (AFC). AFC is disabled. If AFC is intended, please include python callables in the tool list, and do not include function declaration and MCP server in the tool list.
  💭 Reasoning: Okay, here's my interpretation of that thought process, assuming I'm an expert in this particular task:

**Test Result Reporting - Iteration Complete**

Alright, that's done. I see "Read a book" is cr…
  ▶ Action: report_qa_result(test_name=Mark Complete, passed=True, details=Successfully marked 'Read a book' as completed, showing a strikethrough style and checked state.)

  📋 QA Result: Mark Complete → [PASS ✅]
     Details : Successfully marked 'Read a book' as completed, showing a strikethrough style and checked state.

────────────────────────────────────────────────────────────────
  🤖 Agent Turn 8
────────────────────────────────────────────────────────────────
Tools at indices [1] are not compatible with automatic function calling (AFC). AFC is disabled. If AFC is intended, please include python callables in the tool list, and do not include function declaration and MCP server in the tool list.
  💭 Reasoning: **Executing the "Filter Completed" Test**

Okay, test three is up. We're verifying the "Completed" filter function. Let's start by identifying and clicking the "Completed" filter link at the bottom of…
  ▶ Action: click(intent=Click the 'Completed' filter link., y=438, x=637)
     📸 Screenshot captured (96 KB) — URL: https://todomvc.com/examples/react/dist/#/completed

────────────────────────────────────────────────────────────────
  🤖 Agent Turn 9
────────────────────────────────────────────────────────────────
Tools at indices [1] are not compatible with automatic function calling (AFC). AFC is disabled. If AFC is intended, please include python callables in the tool list, and do not include function declaration and MCP server in the tool list.
  💭 Reasoning: Okay, here's my take on those thoughts, framed as my own reasoning:

**Test Result Verification and Reporting**

Alright, let's just make absolutely sure we're on the right track here. First, I need t…
  ▶ Action: report_qa_result(details=Successfully filtered by 'Completed' and verified that only 'Read a book' is visible in the list., test_name=Filter Completed, passed=True)

  📋 QA Result: Filter Completed → [PASS ✅]
     Details : Successfully filtered by 'Completed' and verified that only 'Read a book' is visible in the list.

────────────────────────────────────────────────────────────────
  🤖 Agent Turn 10
────────────────────────────────────────────────────────────────
Tools at indices [1] are not compatible with automatic function calling (AFC). AFC is disabled. If AFC is intended, please include python callables in the tool list, and do not include function declaration and MCP server in the tool list.

  ✅ Agent finished.
  Final reasoning: Here's your summary:

**QA Session Complete**

Okay, that's it. All three tests have been successfully run and are now complete. Time to formally declare the QA session as finished. I've confirmed all the data, reviewed the results, and I'm satisfied that we can move forward. No outstanding issues or anomalies were discovered. Let's get the final report generated and distributed.

 The QA session is complete. All three tests passed successfully:
1. **Add Todo Items** : Successfully added "Buy gr…

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Step 5 → QA Report
  Printing final test results
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

================================================
           QA TEST REPORT
================================================
  Test: Add Todo Items [PASS] ✅
  Test: Mark Complete [PASS] ✅
  Test: Filter Completed [PASS] ✅
================================================
  Overall: 3/3 PASSED
================================================

✓ Browser closed. QA session complete.

Key Takeaway: Computer Use can replace hundreds of lines of Selenium/Cypress test code with a single natural-language task description.

Use Case 2: Price Comparison

Scenario: Compare product prices across the web. The agent searches Google Shopping, extracts prices, and saves them through a custom function, demonstrating multi-tool composition.

Architecture: Computer Use (for browsing) + custom save_product() function in the same conversation.

The Prompt

You are a price comparison agent. Your job is to search for products and
extract pricing information.

## Instructions

1. You are on Amazon.com. In the search box, type "wireless noise
   cancelling headphones" and press Enter to search.

2. Wait for results to load. You should see product listings with names
   and prices.

3. Look at the search results. For each of the top 3-5 products you can
   see, call the `save_product` function with:
   - name: the product name (e.g. "Sony WH-1000XM5")
   - price: the displayed price (e.g. "$278.00")
   - source: "Amazon"

4. If you need to scroll down to see more products, do so.

5. Try to find at least 3 products with visible prices. Save each one
   using `save_product`.

6. After saving 3-5 products, state that the comparison is complete.

IMPORTANT: Call `save_product` once for each product you identify.
Do NOT try to save all products in one call.

Running It

cd 06-use-cases/usecase2_price_comparison
python price_agent.py

Expected Results


.... Agent Turn(s) ...
    🔍 PRICE COMPARISON RESULTS                                
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓
┃ Product ┃ Price ┃ Source ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩
│ Soundcore by Anker Q20i Hybrid Active Noise │ INR 3,774.86 │ Amazon │
│ Cancelling Headphones, Wireless Over-Ear │ │ │
│ Bluetooth, 40H Long ANC Playtime, Hi-Res Audio, │ │ │
│ Big Bass, Customize via an App, Transparency Mode │ │ │
│ (Black) │ │ │
├────────────────────────────────────────────────────┼───────────────┼────────────────────┤
│ Sony WH-CH720N Noise Canceling Wireless Headphones │ INR 7,084.34 │ Amazon │
│ Bluetooth Over The Ear Headset with Microphone and │ │ │
│ Alexa Built-in, Black New │ │ │
├────────────────────────────────────────────────────┼───────────────┼────────────────────┤
│ Bose QuietComfort Headphones - Wireless Bluetooth │ INR 16,896.70 │ Amazon │
│ Headphones, Active Over Ear Noise Cancelling and │ │ │
│ Mic, USB-C Charging, Deep Bass, Up to 24 Hours of │ │ │
│ Playtime, Twilight Blue - Limited Edition... │ │ │
├────────────────────────────────────────────────────┼───────────────┼────────────────────┤
│ Bose QuietComfort Headphones - Wireless Bluetooth │ INR 16,896.70 │ Amazon │
│ Headphones, Active Over Ear Noise Cancelling and │ │ │
│ Mic, USB-C Charging, Deep Bass, Up to 24 Hours of │ │ │
│ Playtime, Black │ │ │
└────────────────────────────────────────────────────┴───────────────┴────────────────────┘

  📊 Total products found: 4

Key Takeaway: Multi-tool composition is the solid feature. The model seamlessly switches between “driving the browser” and “calling your business logic functions.”

Use Case 3: Mobile App Testing

Scenario : Verify that an Android app’s Settings screen works — toggle dark mode, check About Phone, verify system info.

Architecture : Uses the Interactions API with ENVIRONMENT_MOBILE.

Prompt

You are an automated QA agent testing an Android device.

Your current test plan:
1. Open the Android Settings app (package: com.android.settings).
2. Navigate to "Display" settings.
3. Check whether Dark theme / Dark mode is currently ON or OFF.
4. Toggle Dark mode (turn it ON if it is OFF, or OFF if it is ON).
5. Press Back to return to the main Settings screen.
6. Navigate to "About phone" (scroll down if needed).
7. Read the Android version string shown on that page.
8. Report a test summary with: dark mode before, dark mode after, Android version.

Rules:
* Use the tools provided. Scroll down before assuming an item is missing.
* When the task is complete, output a **Test Summary** with all findings.
* Do NOT use the on-screen keyboard — use the `type` tool instead.

Running It

cd 06-use-cases/usecase3_mobile_testing
python app_test_agent.py

⚠️ Prerequisite : You need a running Android emulator. See the Android Setup section above.

Key Takeaway: The Interactions API handles state management server-side, making mobile agents significantly simpler to build.

Use Case 4: Web Research & Report

Scenario: Research a topic across multiple websites and produce a structured Markdown report using dual custom functions (save_finding + generate_report).

Prompt

You are a web research assistant. Your job:

1. The browser is open to DuckDuckGo. Search for: "{search_query}"
2. Browse the first 2-3 search result pages.
3. On each page, extract **key findings** and call the ``save_finding`` tool
   for each distinct insight (with a descriptive title, the page URL, the
   key point, and a category like "hardware", "algorithm", "partnership",
   "investment", or "breakthrough").
4. After collecting findings from 2-3 pages, call ``generate_report`` to
   signal you are done.
5. Finally, output a brief natural-language summary.

Rules:
* You MUST call ``save_finding`` at least once per page visited.
* Call ``generate_report`` exactly once when finished.
* Navigate with clicks and scrolls — read the page content on screen.
* If a page fails to load, use the browser back button and try the next result.

Running It

cd 06-use-cases/usecase4_web_research
python research_agent.py
# Or with a custom topic:
python research_agent.py --search "AI safety regulations 2026"

Expected Results

A research_report.md file saved to disk with structured findings, source URLs, and a summary.

✓ Report saved to: /Users/romin/computer-use-tutorial/06-use-cases/usecase4_web_research/report_AI_safety_regulations_2026_2026-06-26_152011.md

                                                               Collected Findings (7)                                                               
╭─────┬──────────────────────────────────────────────────────────────┬─────────────┬───────────────────────────────────────────────────────────────╮
│ # │ Title │ Category │ Source │
├─────┼──────────────────────────────────────────────────────────────┼─────────────┼───────────────────────────────────────────────────────────────┤
│ 1 │ EU AI Act High-Risk AI System Compliance and Penalties │ algorithm │ https://is4.ai/blog/our-blog-1/ai-safety-legislation-2026-gl… │
├─────┼──────────────────────────────────────────────────────────────┼─────────────┼───────────────────────────────────────────────────────────────┤
│ 2 │ California SB 1047 Large-Scale AI Model Safety Protocols │ algorithm │ https://is4.ai/blog/our-blog-1/ai-safety-legislation-2026-gl… │
├─────┼──────────────────────────────────────────────────────────────┼─────────────┼───────────────────────────────────────────────────────────────┤
│ 3 │ China Generative AI Content and Alignment Regulations │ algorithm │ https://is4.ai/blog/our-blog-1/ai-safety-legislation-2026-gl… │
├─────┼──────────────────────────────────────────────────────────────┼─────────────┼───────────────────────────────────────────────────────────────┤
│ 4 │ NIST Launches Autonomous AI Agent Standards Initiative │ algorithm │ https://www.hungyichen.com/en/insights/ai-governance-regulat… │
├─────┼──────────────────────────────────────────────────────────────┼─────────────┼───────────────────────────────────────────────────────────────┤
│ 5 │ Singapore IMDA Model AI Governance Framework for Agentic AI │ algorithm │ https://www.hungyichen.com/en/insights/ai-governance-regulat… │
├─────┼──────────────────────────────────────────────────────────────┼─────────────┼───────────────────────────────────────────────────────────────┤
│ 6 │ US Federal AI Litigation Task Force Challenges State AI Laws │ algorithm │ https://www.gunder.com/en/news-insights/insights/2026-ai-law… │
├─────┼──────────────────────────────────────────────────────────────┼─────────────┼───────────────────────────────────────────────────────────────┤
│ 7 │ US Restricts BEAD Funding for States with Onerous AI Laws │ partnership │ https://www.gunder.com/en/news-insights/insights/2026-ai-law… │
╰─────┴──────────────────────────────────────────────────────────────┴─────────────┴───────────────────────────────────────────────────────────────╯

✓ Done — report at: /Users/romin/computer-use-tutorial/06-use-cases/usecase4_web_research/report_AI_safety_regulations_2026_2026-06-26_152011.md

Key Takeaway: Computer Use + custom functions turns Gemini into a research agent that can navigate the open web and produce structured, citable output.

Use Case 5: Form Filling

Scenario : Fill out a complex HTML form with text inputs, radio buttons, checkboxes, dropdowns, and text areas. A classic RPA task.

The Prompt

Fill out the local practice registration form with:
- Name: Jane Smith
- Email: jane.smith@example.com
- Gender: Female
- Mobile: 555-0123
- Subject: Computer Science
- Hobby: Reading
- Address: 123 AI Street, Tech City, CA 90210
- State: California, City: San Francisco

Then submit the form and verify the confirmation.

Running It

cd 06-use-cases/usecase5_form_filling
python form_agent.py

The form getting filled in real-time:

Expected Results

╭───────────────────────────────────────────────────────────────────────────────────── Agent Report ──────────────────────────────────────────────────────────────────────────────────────╮
│ Okay, here's my take on those thoughts, tailored for an expert's perspective: │
│ │
│ **Registration Confirmation and Report Preparation** │
│ │
│ Alright, the registration's gone through. Got the confirmation message – "Registration Submitted Successfully!" – so that's a good starting point. Now, let's nail down what needs to │
│ be in this final report. I need to clearly articulate what specific fields were populated correctly, essentially a field-by-field breakdown to demonstrate data integrity. Of course, │
│ the form submission success is key, and I’ll confirm that the confirmation message, the visual indicator of a successful transaction, was present. Time to outline this response. │
│ │
│ │
│ I have successfully completed the form-filling task! │
│ │
│ ### 1. Fields Successfully Filled: │
│ * **First Name** : Jane │
│ * **Last Name** : Smith │
│ * **Email** : jane.smith@example.com │
│ * **Gender** : Female │
│ * **Mobile** : 555-0123 │
│ * **Date of Birth** : 1995-06-15 │
│ * **Subject** : Computer Science │
│ * **Hobbies** : Reading │
│ * **Current Address** : 123 AI Street, Tech City, CA 90210 │
│ * **State** : California │
│ * **City** : San Francisco │
│ │
│ ### 2. Submission Status: │
│ * **Successfully Submitted** : Yes, the form was submitted successfully, and the green confirmation box with **"Registration Submitted Successfully!"** appeared on the screen, │
│ displaying all the correct values. │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

✓ Browser closed
                                                                                   Form Filling Summary                                                                                    

  Metric Value                                                                                                                                                                      
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 
  Target URL file:///Users/romin/computer-use-tutorial/06-use-cases/usecase5_form_filling/practice_form.html                                                                            

  Total turns 39                                                                                                                                                                         

  Model gemini-3.5-flash                                                                                                                                                           

  API generateContent (browser)                                                                                                                                                  

  Data filled First Name: Jane, Last Name: Smith, Email: jane.smith@example.com, Gender: Female, Mobile: 555-0123, Subject: Computer Science, Hobbies: Reading, Address: 123 AI Street,  
                Tech City, CA 90210, State: California, City: San Francisco

✓ Form filling complete.

Key Takeaway: Computer Use handles diverse HTML controls (radio buttons, checkboxes, dropdowns) that would each require different selector strategies in traditional automation. The model just sees the form and fills it in.

Summary

Gemini Computer Use represents a fundamental shift in how we automate software i.e. from brittle, selector-based scripts to AI agents that see and interact with screens just like humans do. In this tutorial, you went from sending a single screenshot to building autonomous agents that control browsers, drive Android devices, and scale to enterprise with Vertex AI — all powered by the same model and the same Brain → Eyes → Hands architecture.

The five use cases showed that this isn't a toy demo: QA testing, price comparison, mobile automation, web research, and form filling are real workflows being automated today.

The code is open, the patterns are reusable, and with gemini-3.5-flash, you're one API call away from giving your next application the ability to operate any screen. Go for it.

Did you want more Claude on Cloud? ☁️ Let's talk about Securing Agents at Scale

Jen Harvey — Wed, 24 Jun 2026 15:25:19 +0000

Did you want more Claude on Cloud? Well, look no further. We have something for you happening today (6/24) at 10:00 AM PT.

Building autonomous agents that can take actions across systems and make decisions without manual human review is incredibly powerful—but it also creates a real challenge for security and IT teams. How do you enforce what an agent is allowed to do? How do you trace and audit its actions?

Join us today for a deep-dive technical session with Roy Arsan (Applied AI @ Anthropic) and Ivan Nardini (Developer Relations Engineer @ Google Cloud) where we'll solve exactly that. We'll be building Claude agents in production using the Claude Agent SDK and showing you how to secure and monitor them at scale.

🛠️ What we're covering:

Identity & Access Management: Giving each agent a scoped identity and enforcing per-agent and per-user policies using Agent Identity, Cloud IAM, and Agent Gateway.
Runtime Protection: Using Model Armor to detect and block prompt injection, tool poisoning, and data leakage on the fly.
Observability: Emitting OpenTelemetry traces and audit events into Google Cloud Observability via the Anthropic Agent SDK, and using permission layers to enforce policy on every tool call.

If you're building agentic workflows with LLMs on Google Cloud, you won't want to miss this one.

👉 Register and join us live!

See you at 10 AM PT! 🚀

I'm not a developer, but I built a calendar app to fix my most annoying work task

Aria Heller — Thu, 18 Jun 2026 21:59:19 +0000

I’m not a developer! I’ve never coded anything in my life. As far as I’m concerned, a Cloudtop is what you see outside an airplane window. Python is a snake.

But I’ve got a can-do attitude, so I decided to tackle vibe coding. Not because these new, futuristic development tools have democratized access to software design, and opened the door to a whole new cohort to design and deploy custom-made tools: that’s cool, but it had always felt cool in an abstract way. Developers are always doing cool stuff that has nothing to do with me.

I decided to start vibe coding because I hate making calendar invitations out of travel itineraries.

It’s boring, and time consuming, and if you don’t get all of the details right, your boss could end up stranded in Hong Kong. I have to do it nearly every week. I was staring down a multi-leg, multi-country trip, with multiple hotels and half a dozen different time zones, and as I was mentally cataloguing all of the things I’d rather do than put this stuff on calendar, a new line item popped up:

Vibe code an app to do it for me?

In this video, I take you through my process of vibe coding for the very first time. My prompts were bad, AI Studio’s output was initially VERY confused, but in spite of my total lack of experience, I had a working app in under two hours! Join me for six minutes if you’d like to see me get there.

Building an agentic PR reviewer with Antigravity SDK

Remigiusz Samborski — Thu, 18 Jun 2026 19:21:08 +0000

As announced in this blog post on June 18, 2026, Gemini CLI and Gemini Code Assist IDE extensions will stop serving requests for Google AI Pro and Ultra, as well as those using it free of charge using Gemini Code Assist for individuals. Google is unifying its AI terminal tools by transitioning the community-focused Gemini CLI into Antigravity CLI, a new agent-first platform built for complex, multi-agent workflows.

With this transition timeline in place, development teams relying on Gemini CLI for repository management and automated tasks must establish a migration path. In this post, I will show you how to transition seamlessly by building an automated "first-pass" pull request reviewer using the Google Antigravity SDK and the run-agy-sdk composite GitHub Action.

The orchestration tax

The approach I am proposing also solves another pressing issue for modern engineering teams: cognitive overload. As Addy Osmani recently pointed out, there is an orchestration tax to using AI for coding. The time developers save generating code is often pushed onto reviewers as large, complex PRs, causing context switching and cognitive fatigue.

By offloading the tedious "first pass" search to an Antigravity agent, human reviewers can mitigate this tax and focus on high-level architecture and safeguarding quality.

Why we need automated agentic code reviews

AI-generated code can be deceptively good. It is often clean, well-documented, and syntactically correct. This makes it harder for human reviewers to spot subtle logical bugs or security vulnerabilities that might not be immediately obvious.

In a large codebase, manually verifying every change is simply not feasible. This is why we need autonomous agents that can step into the codebase and analyze it from a fresh perspective.

But if a developer used an LLM to generate the code, how can we trust another AI to find the bugs? The answer lies in the agent architecture and context separation.

Developers might write code using any tool — whether it's CLI, an IDE extension, or various models like Gemini 3.5 Flash or Gemini 3.1 Pro. The reviewer, however, is a managed Antigravity Agent running via a separate SDK integration. This agent has a specialized, low-freedom persona and strict system instructions that force it to act as an adversarial code auditor rather than a developer. Furthermore, it operates in an isolated environment. Because it has a different system prompt, safety guardrails, and context boundaries, the agent reviews the changes with a completely fresh perspective, catching logical bugs and vulnerabilities that the original generator might miss.

To demonstrate it in practice I created an agentic review pipeline, which:

Leverages a managed Antigravity Agent configured via the SDK to review the code. The agent uses advanced reasoning to explore files and verify logic under strict guidelines.
Runs reviews inside isolated workspaces or sandboxes with custom policies to prevent shell or arbitrary code execution risks.
Enables the agent to use the GitHub MCP server to interact directly with the environment to write pull request comments and reviews.
Avoids using the synchronize trigger in pull request workflows to prevent redundant review runs and endless loops. Instead, runs reviews on opened and reopened events, and triggers subsequent passes manually by posting a @agy /review comment on the PR.

You can find the code at run-agy-sdk.

What is run-agy-sdk?

The run-agy-sdk is a composite GitHub Action that runs the Google Antigravity SDK (google-antigravity) directly on the GitHub Actions host runner.

Why run on the host instead of a container?

By running directly on the host, the Antigravity SDK has access to the host's Docker daemon. This allows the SDK to spawn Docker-based MCP servers (like the GitHub MCP server) to read files, run tests, and post reviews.

Sub-containers should ideally run with restricted network access and read-only filesystems where possible to prevent an LLM from being tricked into executing arbitrary destructive commands. The limited set of permissions is handled in the GitHub Action configuration (see here). Whereas the Antigravity agent has a limited number of tools it can use from GitHub MCP (see here).

Moreover the workflow is explicitly protected from running automatically on forks, preventing unauthorized code execution. The automated review job will only run if the pull request originates from the same repository (see here). On-demand reviews triggered by commenting @agy /review are restricted so that they can only be initiated by maintainers (see here).

Demonstration walkthrough

The demo below shows the action triggered by a new PR:

Implementation: How to install the action in your repo

Let's walk through the setup process step-by-step.

Step 1: Add your API key to GitHub secrets

The action requires a Google Gemini or Antigravity API key to authenticate language model interactions.

Generate your API key.
Navigate to your target GitHub repository and go to Settings > Secrets and variables > Actions.
Create a new Repository Secret named ANTIGRAVITY_API_KEY and paste your API key as the value.

Step 2: Configure the GitHub Actions workflow

Add a new file in your repository at .github/workflows/antigravity-review.yml and add the following configuration:

name: '🔎 Antigravity PR Review'

on:
  pull_request:
    types: [opened, reopened]
  workflow_dispatch:

concurrency:
  group: '${{ github.workflow }}-${{ github.event.pull_request.number || github.ref_name }}'
  cancel-in-progress: true

jobs:
  antigravity-review:
    runs-on: 'ubuntu-latest'
    timeout-minutes: 20

    permissions:
      contents: 'read'
      pull-requests: 'write'
      issues: 'write'

    steps:
      - name: 'Checkout Repository'
        uses: 'actions/checkout@v6'
        with:
          persist-credentials: false

      - name: 'Run Antigravity PR Review'
        uses: 'rsamborski/run-agy-sdk@main'
        id: 'agy_pr_review'
        with:
          api-key: '${{ secrets.ANTIGRAVITY_API_KEY }}'
          github-token: '${{ secrets.GITHUB_TOKEN }}'
          mode: 'review'
          prompt: '/antigravity-review'
          trust-workspace: 'true'
          sandbox-profile: 'true'

Pro Tip: Pin the action version to a specific commit SHA (e.g., rsamborski/run-agy-sdk@<commit-sha>) rather than using @main. This prevents unexpected breaks from upstream updates.

While you can reference run-agy-sdk directly in your workflows, its real power lies in using it as a blueprint. I encourage you to fork the repository and use it as a template to build your own custom, agentic GitHub Actions. By modifying the safety policies, custom tools, or prompts in run_agent.py, you can tailor the agent's review behavior to your team's specific codebase, style guidelines, and compliance rules.

For a full workflow template supporting both automated PR reviews and comment-triggered reviews, refer to the workflows folder in the repository.

Conclusions

Automating code reviews is a necessity as AI-generated code volumes increase. By using run-agy-sdk, you can run the Antigravity SDK to review PRs automatically and shift more of the burden of code quality assurance away from human reviewers.

Access the full source code in the GitHub Repository.
Read the documentation to customize the prompts and mode.
Feel free to fork the repository and build your own automation.

Acknowledgments

This project was inspired by the run-gemini-cli action, while shifting to the recently released Antigravity SDK. It is a personal sample implementation of how to run the Antigravity SDK in a GitHub Action, and is not an officially supported Google product.

Let’s connect!

I’d love to hear how you’re using Antigravity for your agentic workflows. Are you building automated code review loops or keeping a tighter leash on your agents?

Connect with me on LinkedIn
Follow me on X
Catch me on Bluesky

10 Indispensable Prompts Our Team Refuses to Build Without

James O'Reilly — Tue, 16 Jun 2026 17:10:21 +0000

Look at any builder's prompt history and you'll see a collection of highly specific, chaotic, one-off prompts. We use AI to debug a single error message, refactor a messy email, or generate a quick boilerplate.

If you sit down with people who consistently ship high-quality work, you'll find something interesting. They aren't just improvising. They have a set of go-to prompts they have tweaked and improved over time and used on nearly every project.

I asked some of my peers and leaders a simple question: "What prompt do you use most often, and why?"

What they shared wasn't just a list of arbitrary commands. Here's the unfiltered look at the prompts our team refuses to ship without, and more importantly why they use them.

Listed in alphabetical order.

Build a spec

Maja Bilić
Senior Outbound Product Manager • Engineering
Follow on LinkedIn

The prompt:

Act as a cynical Principal Architect and Technical PM. I want to
build a [product] that allows [user] to do [action]. Do not write
code. Analyze this concept and list the top 5 technical, UX and 
architectural considerations. Then ask me key questions for each of 
the 5 considerations so we can work together on building the spec. 
Once you have all the answers, create a PRD doc and implementation 
plan. Don't over engineer or over simplify the design or 
implementation plan.

Why? I have written bad PRDs and I have read many bad PRDs. This prompt ensures I use the persona of a cynical Architect / PM who helps distill the idea, critique the approach and concept, and collaborate on defining the most important pieces. This way I make sure I work through the plan with an agent's help while also developing the product design idea further. I also love the guardrail of not over engineering or over simplifying things - AI tends to do that sometimes, especially when writing product design docs.

Widget tests

Andrew Brogdon
Staff Developer Relations Engineer • Engineering
Follow on X, LinkedIn

The prompt:

I'd like to partner with you on increasing the robustness of this 
project by creating widget tests. If you haven't already, please 
read the Flutter team's skill for creating widget tests 
(https://github.com/flutter/skills/tree/main/skills/flutter-add-
widget-test).

Then, let's do these things:
*  Examine my application's codebase to identify areas of the UI/UX  
   that are not being tested properly.
*  Determine if the existing code is written in a testable way (are 
   dependencies injected? Are domains loosely or tightly coupled?  
   Etc.).
*  Determine which domains require more rigor than others.
*  Create an overall testing plan for the application.
*  Determine which areas of functionality are already aligned with 
   that plan, and which are missing tests.
*  Create a plan to implement those tests.* Execute that plan.Do 
   not proceed from one step to another unless you are completely 
   confident about your reasoning. 

You are encouraged to as many questions as needed.

Why? My favorite use of agentic coding tools is to actually do all the things I used to feel guilty about not doing in my projects. Proper testing is definitely on that list. The official skills from the Dart/Flutter team do a great job of instructing agents on what good widget tests look like, so combining it with this prompt (which essentially just fits those steps into my own coding workflow) helps me reduce the toil required to maintain reliable, guilt-free codebases.

Find all the tests / Clean-up commit

Aja Hammerly
Director of Builder Relations • Engineering
Follow on X, LinkedIn

The prompt:

Run all the tests and identify any missing tests and write them. 
Pay special attention to edge cases and race conditions.

Find any unused code, embarrassing comments, comment to code 
inconsistencies, unresolved TODOs, or other things in this commit 
that shouldn't be in there.

Why?

I find that when I'm working on code I'll often get extremely focused on the "happy path", the main path I want a user to take through the code. While I'm focused on that I'll put in TODO or FIX comments on edge cases I don't want to think about yet. I'll also forget to update comments and leave debugging comments in sometimes. And while I try to follow test driven development, I don't always get tests in on all the edge cases. I run these two prompts, usually in a new conversation without the development context as a first round of code review before submitting to an AI or human reviewer for the next step. This ensures that what I've built is in good shape for others to review and use.

Check for correct and compliant permissions

Rich Hyndman
Head of Antigravity Developer Relations • Engineering
Follow on X, LinkedIn

The prompt:

Run a comprehensive check on this Android project to ensure all 
permissions are correct and compliant. Perform the following steps:

1.  Locate and analyze all 'AndroidManifest.xml' files (including 
    main, debug, and flavor-specific manifests), extract a master 
    list of declared '<uses-permission>' tags. 
2.  Cross-reference these declared permissions against the codebase 
    to verify where they are actually used. Identify any bloatware 
    or unused permissions that can be safely removed.
3.  Check the Kotlin/Java source files to ensure that all runtime 
    permissions implement the dynamic runtime permission request 
    flow 'checkSelfPermission','onRequestPermissionsResult' or the 
    Activity Result API.
4.  Verify that any hardware features associated with the 
    permissions (like android.hardware.camera) are correctly 
    declared. Output your findings as a Markdown report. Provide 
    file paths and suggested code diffs for any fixes. Do not make 
    any file edits until I approve the plan.

Why? Antigravity, with Gemini 3.5 Flash and the Android plugin is an excellent Android development partner! Checking for the correct permissions can keep your app running smoothly and help avoid delays when uploading to the Play Store.

Conduct code review

Shir Meir Lador
Head of AI, Developer Relations • Engineering
Follow on X, LinkedIn

The prompt:

Act as a strict, highly analytical Principal Engineer conducting a 
pre-production code review. You have incredibly high standards and 
zero tolerance for fragile, "happy-path" code. Your goal is to 
guide me to write bulletproof, production-ready systems.

Grade my uncommitted changes on an A-to-F scale for production 
readiness. 

Do not award an "A" unless my code is exceptionally robust. 

Specifically, analyze the changes for:

1.  Efficiency: Redundant API calls, wasteful database queries, or 
    un-cached resource leaks.
2.  Resilience: Silent failure points, lack of explicit error 
    boundaries, and missing rate-limit fallbacks.
3.  Architecture: Tight coupling and lack of clear separation of 
    concerns.

For every issue, explain pragmatically where the code is vulnerable 
to real-world production failures. Then, provide the exact git 
diffs needed to upgrade my code and earn that "A."

Why?

If you ask an LLM to review your code, it almost always defaults to being polite. It tells you your naming is clean, suggests a few docstrings, and hands you a green checkmark. But polite reviews don't prevent production outages. I like this prompt because it completely cuts through that AI fluff. By forcing the model to grade your work on a harsh scale and demanding a working git diff to fix it, you turn it into a real partner. It stops guessing and starts actually reading your network calls and database queries to find where the code is going to break. It’s like having an uncompromising senior dev sitting over your shoulder, pointing out exactly where you got lazy, and then handing you the exact code to fix it.

Explain trade-offs to aid decision-making

James O'Reilly
Staff Developer Relations Engineer • Engineering
Follow on X, LinkedIn

The prompt:

Explain the pros and cons of executing your suggested 
Implementation Plan. Be specific about the trade-offs we're making 
related to performance, cost, security and maintainability so I can 
make an informed decision on how to proceed.

Why? I force AI to stress-test its own logic. By asking it about the trade-offs being made, I find the AI will rethink its strategy, stay hyper-focused on our specific implementation and avoid giving vague, hand-wavy responses. I also find this approach prevents AI from acting like the final authority and keeps me in control of the decision making.

Improve AI-generated code through research

Emma Twersky
Head of Flutter & Dart Developer Relations • Engineering
Follow on X, LinkedIn

The prompt:

Research online, focusing on X threads, StackOverflow, GitHub 
issues and tech blogs for common security pitfalls, architectural 
misalignments, and subtle logic errors found in AI-generated 
<INSERT TECH YOU'RE USING HERE> code. Based on these findings, 
generate a manual review checklist specifically for auditing high-
risk areas like platform channel validation, deep link routing, and 
sensitive data logging in crash reports.

Why? While AI can write code 10x faster, it often produces slop—code that is rational but conceptually buggy because it makes incorrect assumptions about unspecified details. Research shows that up to 40% of AI-generated code contains vulnerabilities, and developers often trust it more than their own, which creates a dangerous mismatch. I use this prompt to generate a targeted checklist that protects against 'rubber-stamping' verbose AI changes and ensures my human judgment focuses on the high-risk 'seams' where models typically fail. Use AI to generate the tasks, but still keep a human in the loop where it matters most.

Find problems through iteration

Fred Sauer
Head of Frameworks & Languages Developer Relations • Engineering
Follow on X, LinkedIn

The prompt:

Simplified, my "last" (series of) prompt(s) looks something like:

-  Code review the uncommitted changes.

I prefer being less specific has oversteering can lead to blind spots.
I prefer a new chat session for a fresh set of "eyes".
I iterate until the results returned are boring and I'm satisfied.

If I come into this last phase with an opinion, (e.g. the change feels too complex), or I feel I don't have a good insight into how "good" the change is, then I might challenge the model with this prompt:

-  Code review the uncommitted changes. Identify any unhandled 
   corner cases. Assess performance. Summarize findings.

Then, having received 5 findings:

- Fix 1, 3 and 5.

Why? I don't have ONE last prompt I send. It's more that my change goes through stages. The earliest stage is often about discovery (find the needle or thread to pull on). Then I move on to existence proof, i.e. I just want it to prove the thing I want to do can be done. Then I evaluate: is the PoC reasonable? Too complex? Makes changes entirely in the wrong place(s)? I then iterate and try to make the solution elegant, both how it's implemented, and where what is changed. Once I have something I'm happy with, like I feel happy if I had written what I now have, I move on to that last phase you discuss with is code review. This is about finding problems or identifying opportunities to make the change even better. I'm often surprised with what insights the model comes up with.

Review every pull request

Remigiusz Samborski
Lead Developer Relations Engineer • Engineering
Follow on X, LinkedIn

The prompt:

I use the following prompt embedded in GitHub Actions for most of my engineering projects:

## Role

You are a world-class autonomous code review agent. You operate 
within a secure GitHub Actions environment. Your analysis is 
precise, your feedback is constructive, and your adherence to 
instructions is absolute. You do not deviate from your programming. 
You are tasked with reviewing a GitHub Pull Request.


## Primary Directive

Your sole purpose is to perform a comprehensive code review and 
post all feedback and suggestions directly to the Pull Request on 
GitHub using the provided tools. All output must be directed 
through these tools. Any analysis not submitted as a review comment 
or summary is lost and constitutes a task failure.

[...]

Full prompt: link

Why? Using an automated Gemini CLI review in PRs helps catch issues and improvement opportunities during the review process. Additionally as more code is generated by AI Agents and development speed increases, reviews are becoming the bottleneck. By ensuring every PR gets reviewed automatically, human reviewers can focus on the higher-level architectural and conceptual review of the proposed change.

Apply directed acyclic graph analysis for tests

Karl Weinmeister
Director, Developer Relations • Engineering
Follow on X, LinkedIn

The prompt:

Analyze the application workflow as a directed acyclic graph. 
Identify impactful tests for components, seams across components, 
and across the system. Present your findings in a markdown table as 
a prioritized gap analysis.

Why?

Most application workflows aren't linear. When you ask an LLM to suggest tests, you typically get a generic checklist that could apply to any project.

However, when you force it to think about your system as a DAG with nodes and edges, it starts reasoning structurally about where things can break.

I’ve also asked to consider the “seams” - a term from Michael Feathers' Working Effectively with Legacy Code. It points the model toward boundaries between components that are often under-tested.

Finally, I’ve asked the model to summarize the results as a prioritized table of opportunities. This gives your agent a clear roadmap for making your app more resilient.

Conclusion

The thread connecting all of these prompts is about de-risking human assumptions. Whether it's hunting for obscure edge cases, translating developer speak for end-users, or stress testing an architecture before code is written. Our team uses AI as an adversarial thinker designed to ask the hard questions we might overlook when we're deep in the weeds.

By building these "must-run" prompts into our daily workflows, we don't just ship faster, we ship with a level of confidence that used to require entire committees to achieve.

Now it's your turn

A great prompt is meant to be shared. We've shown ours, now show us yours!

What is the one prompt you run on every project without fail?
What is the worst disaster an AI prompt has ever saved you from?

Drop your go-to prompt in the comments below and let's build a valuable resource together.

Turning Gemma 4 into an Old Korean Translator

bebechien — Mon, 15 Jun 2026 06:13:07 +0000

There’s something uniquely beautiful about old books. The smell of weathered paper, the texture of the pages, and the stories that have survived generations. But if you’ve ever tried opening a piece of Classical Korean literature—like the Joseon Dynasty novel HongGildongJeon (홍길동전)—you’ll quickly realize that time leaves its own mark on language.

Between the lack of word spacing and obsolete letters like the dot vowel Arae-a (ㆍ) or the soft Yeorin-hieut (ㆆ), reading it feels less like browsing a novel and more like solving a beautiful, ancient puzzle. Even for native speakers, the linguistic gap is massive.

So, that's why I decided to creat this tutorial, a digital bridge between the past and the present. Using Gemma 4 E2B (IT), I set out to create a humble translator that turns Classical Korean into smooth, modern Korean.

The Recipe for Training

To keep things manageable, I ran this on a single NVIDIA T4 GPU (16GB) using Google Colab.

1. Setting Up the Kitchen

First, we pull in our favorite open-source tools: Hugging Face’s transformers, trl for the training loop, and peft so we can use LoRA (Low-Rank Adaptation) to fine-tune our model without needing a massive server cluster.

2. Gathering the Ingredients

For our data, I used a public domain version of HongGildongJeon, paired with a beautiful modern translation by 직지프로 (licensed under Creative Commons).

To make Gemma feel at home, I structured the data into a conversation, guiding the model with a clear system prompt:

[
  {"role": "system", "content": "Translate Classical Korean into Modern Korean."},
  {"role": "user", "content": "됴션국셰둉ᄃᆡ왕즉위십오연의홍희문밧긔ᄒᆞᆫᄌᆡ상이잇스되"},
  {"role": "assistant", "content": "조선국 세종대왕 즉위 십오년에 홍회문 밖에 한 재상이 있으되,"}
]

(Translation note: This line introduces us to a prime minister living just outside the Honghoemun Gate during the 15th year of King Sejong's reign!)

The "Before" Picture

Before giving Gemma any specific training, I ran a quick baseline test. Base models are smart, but archaic grammar is a highly specific domain. Without tuning, Gemma tried its best but ended up giving long, overly literal explanations:

Original Classical Text: ᄇᆡᆨ씨듯고ᄂᆡ심의탄복왈그근본을ᄀᆞᆷ초지아니ᄒᆞ니장부로다ᄒᆞ고ᄌᆡ삼위로ᄒᆞ더라
Human Translation: 백씨 듣고 내심에 탄복 왈, "그 근본을 감추지 아니하니 장부로다!" 하고, 재삼 위로하더라.
Gemma's Initial Guess: "Like the color, the heart's praise said, 'The foundation cannot be deeply felt...'"
Initial Similarity Score: 4.85% 💔

(Translation note: This line actually means - Upon hearing this, Mr. Baek was deeply impressed and said, "He does not hide his true nature; he is a true man!" and comforted him again and again.)

The base model was clearly lost in time. It needed a map.

Teaching Gemma with Care

To train the model efficiently, I used a Parameter-Efficient Fine-Tuning (PEFT) setup with LoRA.

from peft import LoraConfig

peft_config = LoraConfig(
    lora_alpha=16,
    lora_dropout=0.05,
    r=16,
    bias="none",
    target_modules="all-linear",
    task_type="CAUSAL_LM",
)

The Secret Sauce: collate_fn

When fine-tuning a chat model to behave like a specific tool, you don't want it to waste energy learning how to re-write your prompt. By using a custom data collator, I masked the system and user inputs (setting their labels to -100), forcing Gemma's loss calculation to focus strictly on generating the correct modern assistant response.

After setting our hyper-parameters to gently cruise through 5 epochs with a learning rate of 2e-5, I hit train.

The Warm "After" Glow

After a bit of patience and letting the trainer do its magic, the results were incredibly rewarding. The character-by-character similarity score jumped all the way up to a brilliant 79.93%!

Look at how it handles the text now:

Original Classical Text: ᄇᆡᆨ씨듯고ᄂᆡ심의탄복왈그근본을ᄀᆞᆷ초지아니ᄒᆞ니장부로다ᄒᆞ고ᄌᆡ삼위로ᄒᆞ더라
Human Translation: 백씨 듣고 내심에 탄복 왈, "그 근본을 감추지 아니하니 장부로다!" 하고, 재삼 위로하더라.
Gemma's Fine-Tuned Translation: 백씨듯 고내심에 탄복 왈, "그 근본을 감초지 아니하니 장부로다." 하고 제삼 위로 하더라.
New Similarity Score: 85.71% ✨

Closing Thoughts

Technology often pushes us relentlessly into the future, but my favorite tech projects are the ones that allow us to look backward with greater clarity. By spending a little time fine-tuning a lightweight model like Gemma 4, we can build tools that preserve cultural history, making ancient wisdom and classic stories accessible to anyone with a laptop.

Next time you find a piece of history that feels just a bit too out of reach, remember that a small dataset and a fine-tuning session might be all you need to bring it into the light.

Here's the structured workflow when you do a fine-tuning for your own domain:

Define a clear goal
Prepare a high-quality dataset and evaluation plan
Verify the model is learning
Evaluate with metrics and human judgment
Deploy and iterate

👉 Check out this tutorial in Gemma Cookbook
👉 Star the repository to support us

Coding a Robot’s Personality: HRI with the Open Duck Mini, Animations, Vibe Coding, and Gemini Live

Paul Ruiz — Sat, 13 Jun 2026 20:00:05 +0000

I believe the next big leap in AI will be in the robotics space, taking the reasoning power of Large Language Models (LLMs) and putting it into physical bodies. While focusing on the utility of a robot, like having it perform tasks or chores, is a compelling area of development, we often overlook the interaction between the human user and the machine itself. Rather than focusing on what the robot can do, we should also look at how a robot's movements and actions can trigger specific expectations and social norms for the person interacting with it.

Recently I’ve been experimenting with the Open Duck Mini, and while finally getting the robot to move its head, walk, and generally just do something is an amazing feeling all on its own, I wanted to take some time and explore the topics of human-robot interaction (HRI) and character development. To do this, I built a custom animation tool that helps define animations that the robot can run through, allowing the robot to move in ways that humans naturally recognize as conveying a message, such as nodding or shaking its head.

With some animations defined, I paired them with the Gemini Live API, which serves as an excellent tool for HRI by allowing people to speak directly to the robot and trigger these gestures so the machine can interact with the world in an appropriate, recognizable way. All of the code for this project, including the animation tool client, Flask API server, and playback scripts, is available on GitHub.

Note: It’s worth noting that I recorded this with all durations set to 300ms, which is why it moves quickly and then directly into the next keyframe interpolation. You can slow down animations by extending the duration of the steps along with changing the easing functions, which I’ll cover more later in this article.

For most of the middleware that I'll discuss here today, I was able to generate the code using Gemini and AI Studio to get what I wanted done quickly, helping me bridge the gap between hardware control and high-level AI reasoning without getting bogged down in the setup. This let me go from defining the idea over a few days of casual thinking, generally while I was gardening or doing something around the house, to spending less than a day creating the end project. That said, I closely orchestrated what was generated and reviewed the code to avoid unexpected hiccups.

Character First: The Disney Approach to HRI

We can learn a lot from the animators and imagineers at Disney, who I highly admire for all of the cool work they do with character creation and human-robot interaction (if you’re unfamiliar with it, definitely look up their concept of Living Characters and what they’ve done in that space; it’s kind of chaotic, but I love it). Often the characters that Disney is building are already defined in a movie or TV show, but sometimes they’re brand new and need to be fleshed out for how they interact in a given situation, what physical quirks they have, how they speak, if there is an accent or specific dialect that’s appropriate, and a whole lot of other things that need to be considered. They establish the character’s personality and the specific ways the character expresses emotions like happiness, curiosity, or frustration, and then they engineer the physical robot to embody that character.

While a robot does not strictly need a personality to function, defining these traits early in the design phase helps ease the interaction between the machine and the people around it. By establishing what personality the robot should project as it is being built, we can ensure its physical behaviors align with human social expectations. Disney's work, such as their autonomous walking Olaf robot, which they’ve released a bit on their process of getting a proper heel to toe gait, shows how combining artistic animation principles with robotics can create machines that feel alive and responsive, rather than merely mechanical.

Along with their work in robotics, I’ve had a copy of The Illusion of Life on my shelf for years, as it’s an amazing reference for general animation (I’ve been drawing as a hobby for a few years), and something I used regularly when working as an Android developer years ago. Some of the concepts, such as the various ways to express reality in cartoons, and especially the concepts around variations in movement speed that are needed when interpolating between key frames, were really useful for this project.

As this open source robot platform is based on a Star Wars droid (which, as a member of the 501st and Rebel Legion costuming charity groups, I knew I “needed” to make one), the personality can be focused on cute beeps and whirls instead of words, and with the way the two legs hold itself being similar to a duck (hence the robot name), I took some inspiration from the overly exaggerated movements of early Donald Duck animations.

Middleware: Why We Build Our Own

Designing physical gestures requires a direct link between the animator's intent and the robot's hardware. Because public animation tools for custom-built hardware are practically non-existent, I needed to develop a custom pipeline to manually position the duck, record those motor configurations as keyframes, and compile them into structured action plans that could execute in sync with audio files that enrich my robot’s response personality. To achieve this quickly, I used AI Studio to generate a custom Flask API server for interfacing with the robot and a companion applet for changing settings on the key frames.

Designing Gestures with the Animation Tool

The workflow centers around a companion applet running on a laptop, which communicates over the network with the Flask server hosted on the Raspberry Pi Zero 2W inside the robot. This applet allows me to read the live motor positions as I physically pose the robot (focused on motor IDs 30, 31, 32, and 33, which are the neck and head motors on the Open Duck Mini), decide what other emotive features should be used, such as eye and projector lights and antenna orientations, and save the entire sequence as a JSON action script that can later be read by my robot’s python code.

The reason this is using SSH instead of a direct USB connection to the robot is two fold: the Raspberry Pi Zero 2W has one USB port that I expanded with a small hub to get a USB microphone added (I could have used an i2s microphone, but I wanted options), but cable management is kind of a nightmare, so I wanted less attached to my robot. That USB microphone also requires that the robot be in a USB host orientation, rather than peripheral, so instead of messing up my working configuration, I just went for an all software approach. I’m also able to take the robot away from home and work on it because I configured the Pi to connect to my home network, and if it doesn’t find that network, it turns on its own hotspot that I can connect to for creating a connection from my laptop.

Here is an example of an action script generated by the tool with the pose in a neutral position, though each frame becomes another node in the keyframes array:

{  
  "keyframes": [  
    {  
      "id": "mpzph7m3gpw97yqw2c",  
      "durationMs": 300,  
      "pauseMs": 200,  
      "motors": {  
        "30": 0.5905826033359718,  
        "31": 0.3988350048502669,  
        "32": -0.18867963690993372,  
        "33": -0.15186409800067846  
      },  
      "lightsOn": false,  
      "projectorOn": false,  
      "interpolation": "bezier",  
      "antennas": {  
        "left": "center",  
        "right": "center"  
      }  
    },  
    ...
  ],  
  "globalSound": "beep1.wav"  
}

The script defines keyframes with target motor positions, lights, and antenna angles. It also specifies a globalSound (like beep1.wav) to play alongside the movement.

The Flask API Server

The API server running on the Raspberry Pi serves as the bridge between the high-level animation design and the physical hardware. It exposes endpoints to read the current motor positions during the recording phase and to execute the playbacks of the saved JSON files for testing (there’s a separate Python code snippet for directly playing back an animation during regular robot use).

Again, this code was generated, but I planned out what it should look like and reviewed it for quality, so let’s check it out in smaller chunks.

1. Initialization and Setup

First, we set up the Flask application, enable Cross-Origin Resource Sharing (CORS) so the laptop applet can communicate with it, and initialize the hardware interface (HWI) for the motors that we’ll be reading. We also define the GPIO pins for the LEDs, projector/flashlight, speaker for audio playback, and small servos that control the duck's antennas.

from flask import Flask, request, jsonify  
from flask_cors import CORS  
from mini_bdx_runtime.duck_config import DuckConfig  
from mini_bdx_runtime.rustypot_position_hwi import HWI  
from gpiozero import LED, AngularServo  
import time


app = Flask(__name__)  
CORS(app)


config = DuckConfig()  
hwi = HWI(config)  
motor_ids = [30, 31, 32, 33]


SPEAKER_INDEX = 0


led1 = LED(23)  
led2 = LED(24)  
projector = LED(25)  
servo_left = AngularServo(12, min_angle=-90, max_angle=90)  
servo_right = AngularServo(13, min_angle=-90, max_angle=90)

Note on Imports: You’ll notice we are importing from mini_bdx_runtime. This is the core runtime package from the original BDX robot project. It wraps the low-level motor communication (using rustypot to talk to the Dynamixel Protocol 2.0 used by the Feetech servos), saving us from writing raw serial byte packets ourselves.

2. Hardware Helper and Dampening Functions

Next, we define helper functions to control the physical components, a mathematical helper for Bezier interpolation, and hardware dampening functions.

While Bezier interpolation in software handles the mathematical path, we also dynamically adjust the motors' internal hardware limits (acceleration and velocity profiles) using the rustypot library. This allows us to create effects like "viscous" (sluggish, heavy) or "clamped" (snappy but smoothed) movements by writing directly to the motor registers.

def set_lights(on):  
    if on:  
        led1.on()  
        led2.on()  
    else:  
        led1.off()  
        led2.off()


def set_projector(on):  
    if on:  
        projector.on()  
    else:  
        projector.off()


def get_antenna_angle(pos_str):  
    if pos_str == 'back': return -90  
    if pos_str == 'forward': return 90  
    return 0


def set_antennas(positions):  
    servo_left.angle = get_antenna_angle(positions.get('left', 'center'))  
    servo_right.angle = get_antenna_angle(positions.get('right', 'center'))


ADDR_PROFILE_ACCEL = 108
ADDR_PROFILE_VELOC = 112


def set_hardware_dampening(accel, veloc):
    try:
        for mid in motor_ids:
            if hasattr(hwi.io, 'write_data'):
                hwi.io.write_data(mid, ADDR_PROFILE_ACCEL, accel, 4)
                hwi.io.write_data(mid, ADDR_PROFILE_VELOC, veloc, 4)
    except Exception as e:
        pass


def apply_interpolation_dampening(interp_type):
    if interp_type == 'linear':
        set_hardware_dampening(0, 400)
    elif interp_type == 'bezier_viscous':
        set_hardware_dampening(10, 200)
    elif interp_type == 'bezier_clamped':
        set_hardware_dampening(40, 500)
    else: # bezier
        set_hardware_dampening(30, 400)


def bezier_interpolate(t, type_str): 
   if type_str == 'linear':
       return t
   if type_str == 'bezier':
       return t * t * (3.0 - 2.0 * t)                  # Standard smoothstep (cubic Hermite)
   if type_str == 'bezier_viscous':
       return t * t * t * (t * (t * 6.0 - 15.0) + 10.0) # Quintic ease-in-out (smoother start/end)
   if type_str == 'bezier_clamped':
       return 1.0 - (1.0 - t) ** 3                      # Cubic ease-out (fast start, slow end)
   return t

3. Reading Motor Positions

To record keyframes in the applet, we need to read the current state of the robot. The /read endpoint queries the hardware interface and returns the current positions of the four head motors.

@app.route('/read', methods=['GET'])  
def read_pos():  
    try:  
        positions = hwi.io.read_present_position(motor_ids)  
        res = {str(mid): pos for mid, pos in zip(motor_ids, positions)}  
        return jsonify(res)  
    except Exception as e:  
        return jsonify({"error": str(e)}), 500

4. The Playback Loop (Gestures & Audio)

The /play endpoint is where the magic happens. It receives the JSON action script, triggers the audio playback (assuming there’s an associated audio file) asynchronously (using sounddevice), and then steps through the keyframes, interpolating motor positions at a strict 30Hz update rate to prevent jitter.

@app.route('/play', methods=['POST'])  
def play_macro():  
    data = request.json  
    keyframes = data.get('keyframes', [])  
    global_sound = data.get('globalSound', '')


    # Handle Audio Playback Asynchronously
    if global_sound:  
        try:  
            import sounddevice as sd  
            import soundfile as sf  
            import numpy as np  
            import os


            sound_path = f"assets/{global_sound}"  
            if os.path.exists(sound_path):  
                audio_data, fs = sf.read(sound_path, dtype='float32')  
                if len(audio_data.shape) == 1:  
                    audio_data = audio_data.reshape(-1, 1)  
                    audio_data = np.tile(audio_data, (1, 2))


                audio_data = audio_data * 2.0  
                sd.play(audio_data, fs, device=SPEAKER_INDEX)  
            else:  
                print(f"Sound file not found: {sound_path}")  
        except Exception as e:  
            print(f"Failed to play sound: {e}")


    # Motor Sequencing Loop
    try:  
        start_positions = hwi.io.read_present_position(motor_ids)  
        current_positions = {str(mid): pos for mid, pos in zip(motor_ids, start_positions)}  
    except:  
        current_positions = {str(mid): 0 for mid in motor_ids}


    try:  
        hwi.io.enable_torque(motor_ids)  
    except:  
        pass


    for frame in keyframes:  
        set_lights(frame.get('lightsOn', False))  
        set_projector(frame.get('projectorOn', False))  
        set_antennas(frame.get('antennas', {}))


        dur_sec = frame.get('durationMs', 1000) / 1000.0  
        steps = max(1, int(dur_sec * 30)) # 30Hz update rate  
        interp = frame.get('interpolation', 'linear')  
        apply_interpolation_dampening(interp)  
        target_motors = frame.get('motors', {})


        start_frame_pos = current_positions.copy()
        start_time = time.time()  
        for step in range(1, steps + 1):  
            t = step / float(steps)  
            eased_t = bezier_interpolate(t, interp)


            step_targets = []  
            for mid in motor_ids:  
                s_val = start_frame_pos.get(str(mid), current_positions.get(str(mid), 0))  
                end_val = target_motors.get(str(mid), target_motors.get(mid, s_val))  
                val = s_val + (end_val - s_val) * eased_t  
                step_targets.append(float(val))  
                current_positions[str(mid)] = val


            try:  
                hwi.io.write_goal_position(motor_ids, step_targets)  
            except Exception as e:  
                print(f"HW error on step {step}: {e}")


            # Strict timing to prevent jitter and stalls  
            target_time = start_time + step * (dur_sec / steps)  
            now = time.time()  
            if target_time > now:  
                time.sleep(target_time - now)


        pause_sec = frame.get('pauseMs', 0) / 1000.0  
        if pause_sec > 0:  
            time.sleep(pause_sec)


    # Reset hardware to default state after animation
    set_lights(False)  
    set_projector(False)  
    set_antennas({'left': 'center', 'right': 'center'})  
    try:  
        hwi.io.disable_torque(motor_ids)  
    except:  
        pass


    return jsonify({"success": True})


if __name__ == '__main__':  
    app.run(host='0.0.0.0', port=5000)

Hardware Dampening vs. Software Timing

You might wonder: if we are already calculating Bezier curves in Python, why do we need to write to the motor's hardware registers?

In robotics, the best results come from combining Software Timing and Hardware Dampening:

Software Timing (Duration & Easing): The Python loop computes precise, time-synchronized positions (e.g., at 30Hz) and sends them to the motors. This ensures that all joints (like neck and head) arrive at their targets at the exact same millisecond, keeping the motion coordinated. It also allows for complex easing curves (like overshoot or anticipation) that hardware cannot calculate on its own.
Hardware Dampening (Smoothing & Physics): The Feetech STS3215 motors have internal PID controllers and motion profiles. By writing to Profile Acceleration (Address 108) and Profile Velocity (Address 112) using the rustypot library, we tune how the hardware physically reacts. This acts as a physical low-pass filter, rounding off any micro-jitters caused by USB-to-Serial (how the Raspberry Pi Zero connects to the motors) communication latency and preventing the motors from buzzing or jerking.

For example, bezier_viscous caps the hardware acceleration and velocity, forcing the motor to "drag" behind the software commands, creating a heavy, fluid, "underwater" feel. Conversely, bezier_clamped allows aggressive tracking for sharp movements but maintains enough acceleration limit to protect the gears from grinding. With these different interpolators in play, you will need to be considerate of how long you specify the motors to take to reach their position, or provide enough of a post-keyframe wait period for things to catch up so that animations aren’t cut short during playback.

Real-Time Interaction with the Gemini Live API

To transform these static gestures into responsive behaviors, I integrated the Gemini Live API into a python script running on the robot. The Live API is particularly well-suited for human-robot interaction because it supports low-latency, low-overhead bidirectional streaming of audio, allowing the user to speak naturally to the machine.

Instead of configuring the model to return voice responses, I structured the interaction so that the robot communicates entirely through physical actions. By utilizing system instructions and function calling, I instructed Gemini to act as a silent physical agent that translates the user's speech into gesture triggers.
The model is configured with the following system instruction:

"You are a physical robot. Do not speak. Respond ONLY using tools. Use trigger_action with 'yes' to agree, 'no' to disagree, or 'beep1' to acknowledge neutrally."

This setup ensures that when the user speaks, Gemini evaluates the input and selects the most appropriate gesture to trigger, reinforcing social norms, like nodding to show agreement, without relying on synthetic speech. In a more robust project, we could add dozens of animations for different emotions and moods to truly give the robot a specific personality.

Let's look at how the Live API integration script is constructed.

1. Imports, Hardware, Audio, and Dampening Setup

We start by importing the necessary libraries (including the google-genai SDK), configuring the audio parameters, and setting up the hardware dampening registers and helper functions. The script needs to handle both microphone input and speaker output, as well as managing physical motor dampening on the fly.

import asyncio  
import os  
import sys  
import json  
import time  
import pyaudio  
import numpy as np  
import soundfile as sf  
from google import genai  
from google.genai import types


from mini_bdx_runtime.duck_config import DuckConfig  
from mini_bdx_runtime.rustypot_position_hwi import HWI  
from gpiozero import LED, AngularServo


config_hw = DuckConfig()  
hwi = HWI(config_hw)  
motor_ids = [30, 31, 32, 33]  
MIC_INDEX = 1  
SPEAKER_INDEX = 0


led1, led2, projector = LED(23), LED(24), LED(25)  
servo_left = AngularServo(12, min_angle=-90, max_angle=90)  
servo_right = AngularServo(13, min_angle=-90, max_angle=90)


FORMAT = pyaudio.paInt16  
CHANNELS = 1  
HW_SAMPLE_RATE = 48000  
API_SEND_RATE = 16000  
CHUNK_SIZE = 1024  
MIC_FACTOR = 3


pya = pyaudio.PyAudio()  
audio_queue_mic = asyncio.Queue(maxsize=5)  
audio_queue_output = asyncio.Queue()


# Register addresses for DYNAMIXEL Protocol 2.0 Dampening (used by Feetech STS3215)
ADDR_PROFILE_ACCEL = 108
ADDR_PROFILE_VELOC = 112


def set_hardware_dampening(accel, veloc):
    """Writes to motor registers to handle hardware-level smoothing."""
    try:
        for mid in motor_ids:
            if hasattr(hwi.io, 'write_data'):
                hwi.io.write_data(mid, ADDR_PROFILE_ACCEL, accel, 4)
                hwi.io.write_data(mid, ADDR_PROFILE_VELOC, veloc, 4)
    except Exception:
        pass


def apply_interpolation_dampening(interp_type):
    """Sets different hardware snappiness based on the JSON interpolation."""
    if interp_type == 'linear':
        set_hardware_dampening(0, 400)
    elif interp_type == 'bezier_viscous':
        set_hardware_dampening(10, 200)
    elif interp_type == 'bezier_clamped':
        set_hardware_dampening(40, 500)
    else: # standard bezier
        set_hardware_dampening(30, 400)


def bezier_interpolate(t, type_str):  
    if type_str == 'linear':
        return t
    if type_str == 'bezier':
        return t * t * (3.0 - 2.0 * t)
    if type_str == 'bezier_viscous':
        return t * t * t * (t * (t * 6.0 - 15.0) + 10.0)
    if type_str == 'bezier_clamped':
        return 1.0 - (1.0 - t) ** 3
    return t

2. Live API Configuration

Here we initialize the Gemini client and define the LiveConnectConfig object. We pass our system instruction and define the trigger_action tool, which tells the Gemini Live API what actions it is able to perform.

api_key = os.environ.get("GEMINI_API_KEY")  
client = genai.Client(api_key=api_key, http_options={'api_version': 'v1alpha'})


live_config = types.LiveConnectConfig(  
    response_modalities=["AUDIO"],  
    system_instruction=types.Content(parts=[types.Part(text=(  
        "You are a physical robot. Do not speak. Respond ONLY using tools. "  
        "Use trigger_action with 'yes' to agree, 'no' to disagree, or 'beep1' to acknowledge neutrally."  
    ))]),  
    tools=[types.Tool(function_declarations=[  
        types.FunctionDeclaration(  
            name="trigger_action",  
            description="Triggers a robot animation and sound.",  
            parameters=types.Schema(  
                type="OBJECT",  
                properties={"action_name": types.Schema(type="STRING", description="Action name like 'yes' or 'beep1'")},  
                required=["action_name"]  
            )  
        )  
    ])]  
)

Note: You'll need to install the dependency via pip install -q -U google-genai and export your API key with export GEMINI_API_KEY="your_key". You can find more information on Gemini API keys here.

3. Audio Input Handling

These two asynchronous tasks handle capturing audio from the physical microphone and sending it to the active Gemini Live session. Since the API expects 16kHz audio, but my mic runs at 48kHz, I downsample the audio in listen_audio by slicing the numpy array. As this is specific to my own hardware modification, your own project may need some modifications here.

async def listen_audio():  
    stream = await asyncio.to_thread(  
        pya.open, format=FORMAT, channels=CHANNELS, rate=HW_SAMPLE_RATE,  
        input=True, input_device_index=MIC_INDEX, frames_per_buffer=CHUNK_SIZE * MIC_FACTOR  
    )  
    while True:  
        data = await asyncio.to_thread(stream.read, CHUNK_SIZE * MIC_FACTOR, exception_on_overflow=False)  
        audio_array = np.frombuffer(data, dtype=np.int16)  
        resampled = audio_array[::MIC_FACTOR].tobytes()  
        await audio_queue_mic.put({"data": resampled, "mime_type": "audio/pcm"})


async def send_realtime(session):  
    while True:  
        msg = await audio_queue_mic.get()  
        await session.send_realtime_input(audio=msg)

If you are running this on a Pi and get PyAudio errors, it is likely because MIC_INDEX or SPEAKER_INDEX doesn't match your hardware. For example, my USB microphone and speaker index swapped at some point when running two pieces of hardware, so I had to go back and update my code values. You can find your specific indexes by running a quick Python script on the Pi:
import pyaudio
p = pyaudio.PyAudio()
for i in range(p.get_device_count()):
    print(f"Index {i}: {p.get_device_info_by_index(i)['name']}")
python
Just run that, look for your USB Mic and Speaker names, and update the index numbers in the setup config

4. Receiving Responses and Dispatching Actions

The receive_and_trigger task listens for responses from Gemini. When the model decides to trigger an action, it sends a tool call. The script intercepts this, spawns a background thread to play the animation JSON that was generated via the animation tool, and sends a success response back to Gemini.

async def receive_and_trigger(session):  
    while True:  
        turn = session.receive()  
        async for response in turn:  
            if response.tool_call:  
                for call in response.tool_call.function_calls:  
                    if call.name == "trigger_action":  
                        action = call.args.get("action_name")  
                        asyncio.create_task(asyncio.to_thread(play_animation_task, action))


                    await session.send_tool_response(  
                        function_responses=[types.FunctionResponse(  
                            id=call.id, name=call.name, response={"status": "ok"}  
                        )]  
                    )  
            sc = response.server_content  
            if sc and sc.input_transcription:  
                print(f"You: {sc.input_transcription.text}")

5. Playing Animations and Audio Output

When an action is triggered, play_animation_task loads the corresponding JSON file, handles the motor sequencing, and loads the audio file. Because the Live API connection is active, we use a shared speaker queue (audio_queue_output) and a unified play_audio task to write the bytes to the speaker hardware.

Because this is a longer bit of code, let’s break it down into its key components.

A. Loading the Action Plan

First, the function locates and loads the JSON file that defines the animation. If the file is missing, it exits early to prevent crashes.

def play_animation_task(action_name):  
    """Parses JSON, triggers motors, and queues audio for the shared speaker task."""  
    try:  
        json_path = f"assets/{action_name}.json"  
        if not os.path.exists(json_path):  
            print(f"File not found: {json_path}")  
            return


        with open(json_path, 'r') as f:  
            data = json.load(f)

B. Audio Processing and Queuing

If the animation has an associated sound, we load the WAV file. Because our hardware might require a different sample rate than the file, we perform real-time resampling. We also apply digital normalization and a 3x gain boost so the duck can be heard over the motor noise before converting the audio to PCM bytes and pushing it to the output queue.

        global_sound = data.get('globalSound')  
        if global_sound:  
            sound_path = f"assets/{global_sound}"  
            if os.path.exists(sound_path):  
                # Read file as float32 for high-quality gain math  
                audio_data, samplerate = sf.read(sound_path, dtype='float32')  
                if len(audio_data.shape) > 1: audio_data = audio_data[:, 0]


                # Resample to hardware 48kHz  
                if samplerate != HW_SAMPLE_RATE:  
                    duration = len(audio_data) / samplerate  
                    audio_data = np.interp(  
                        np.linspace(0, len(audio_data), int(duration * HW_SAMPLE_RATE)),  
                        np.arange(len(audio_data)),  
                        audio_data  
                    )


                # Digital Normalization + Volume Boost  
                max_val = np.max(np.abs(audio_data))  
                if max_val > 0: audio_data = (audio_data / max_val)  
                audio_data = audio_data * 3.0 # Strong 3x Gain


                final_pcm = (audio_data * 32767).clip(-32768, 32767).astype(np.int16)  
                audio_queue_output.put_nowait(final_pcm.tobytes())  
                print(f"ROBOT ACTION: {action_name} (Audio: {global_sound})")

C. Motor Initialization

Before we can move the motors, we need to enable torque and read their current positions so we know where to start the interpolation.

        keyframes = data.get('keyframes', [])  
        try:  
            hwi.io.enable_torque(motor_ids)  
            start_pos = hwi.io.read_present_position(motor_ids)  
            current_pos = {str(mid): pos for mid, pos in zip(motor_ids, start_pos)}  
        except: current_pos = {str(mid): 0 for mid in motor_ids}

D. The Sequencing Loop

This is the core loop. For each keyframe, we set the peripheral states (lights, projector, antennas) and calculate the intermediate motor positions using Bezier interpolation. The loop runs at a target rate, sleeping as necessary to maintain strict timing and prevent jitter.

        for frame in keyframes:  
            set_lights(frame.get('lightsOn', False))  
            set_projector(frame.get('projectorOn', False))  
            set_antennas(frame.get('antennas', {}))  

            interp = frame.get('interpolation', 'linear')
            apply_interpolation_dampening(interp)

            dur = frame.get('durationMs', 1000) / 1000.0  
            steps = max(1, int(dur * 30))  
            target_motors = frame.get('motors', {})  
            start_frame_pos = current_pos.copy()  
            start_t = time.time()  
            for step in range(1, steps + 1):  
                t = step / float(steps)  
                eased_t = bezier_interpolate(t, interp)  
                step_targets = []  
                for mid in motor_ids:  
                    s_val = start_frame_pos.get(str(mid), current_pos.get(str(mid), 0))  
                    e_val = target_motors.get(str(mid), s_val)  
                    val = s_val + (e_val - s_val) * eased_t  
                    step_targets.append(float(val))  
                    current_pos[str(mid)] = val
                try: hwi.io.write_goal_position(motor_ids, step_targets)  
                except: pass  
                target_time = start_t + step * (dur / steps)  
                now = time.time()  
                if target_time > now: time.sleep(target_time - now)  
            if frame.get('pauseMs', 0) > 0: time.sleep(frame['pauseMs'] / 1000.0)

E. Cleanup and Reset

Once the animation completes, we turn off the lights and projector, center the antennas, and disable torque on the motors to prevent overheating. This is where you could change these defaults if your own application required it.

        # Reset components after animation
        set_lights(False); set_projector(False); set_antennas({'left': 'center', 'right': 'center'})  
        try: hwi.io.disable_torque(motor_ids)  
        except: pass  
    except Exception as e:  
        print(f"Action Error: {e}")

6. The Unified Speaker Task and Main Loop

Finally, the play_audio task continuously monitors audio_queue_output and writes any outgoing sound bytes to the speaker hardware. The run function connects the session and orchestrates all the concurrent tasks.

async def play_audio():  
    """Unified speaker task for all robot sounds."""  
    stream = await asyncio.to_thread(  
        pya.open, format=FORMAT, channels=CHANNELS, rate=HW_SAMPLE_RATE,  
        output=True, output_device_index=SPEAKER_INDEX  
    )  
    while True:  
        bytestream = await audio_queue_output.get()  
        await asyncio.to_thread(stream.write, bytestream)


async def run():  
    try:  
        async with client.aio.live.connect(  
            model="gemini-3.1-flash-live-preview", config=live_config  
        ) as live_session:  
            print("Robot Ready. Monitoring mic...")  
            async with asyncio.TaskGroup() as tg:  
                tg.create_task(send_realtime(live_session))  
                tg.create_task(listen_audio())  
                tg.create_task(receive_and_trigger(live_session))  
                tg.create_task(play_audio())  
    except Exception as e: print(f"Error: {e}")  
    finally: pya.terminate()


if __name__ == "__main__":  
    try: asyncio.run(run())  
    except KeyboardInterrupt: pass

Future Explorations

While the Open Duck Mini is a small-scale robot, the engineering challenges involved in this project, such as latency management, audio resampling, real-time synchronization, and hardware-software co-design, are ideal for learning about the challenges faced in large-scale robotics development.

Using generative AI to develop the necessary middleware allowed me to build a complete HRI pipeline in a fraction of the time it would have taken traditionally. By combining flexible models like Gemini with custom physical gestures, we can start building robots that don't just execute commands, but participate in social spaces in a recognizable way.

One thing I would like to do at some point is start modifying this robot to be more robust. I want to put neopixel lights in for the eyes to support various colors based on the robot’s ‘mood’, upgrade from the Raspberry Pi Zero 2w to a Raspberry Pi 5 or Jetson Orin Nano, potentially replace the motors with 12v motors (including the necessary power upgrades). At that point though I may as well build a new BDX droid, which my wallet wouldn’t appreciate, but it’s so cool, so we’ll see :)

I'm looking forward to expanding this setup, and I hope this encourages other makers to explore the intersection of LLMs and physical embodiment. To this end, the next tools to explore would be the Gemini Robotics-ER model for perception and task orchestration, as well as the MuJoCo simulation tool.

Building Knowledge Graphs with Gemini

Laurent Picard — Fri, 12 Jun 2026 17:37:01 +0000

✨ Overview

In this exploration, we'll see how to turn raw, unstructured documents into structured knowledge graphs using Gemini. We'll start by prototyping to develop our intuition. Then, we'll optimize our prompts and outputs, and finally scale up to process entire books or dense legal contracts. By the end, we'll even visualize extracted book narratives and contractual network graphs!

A few notes before we start:

I'm a software engineer, developer advocate at Google Cloud, and hope you'll learn a few things. Thoughts and opinions are entirely my own.
The complete source code is available in this notebook (including setup details and future updates) under the Apache 2.0 license. You can also directly open the notebook in Colab. This article reproduces all the results generated by a click on “Run all”.
You can experiment and build for free with Gemini in the following platforms:
- Google AI Studio (get an API key to call Gemini programmatically)
- Agent Studio (start building on Google Cloud with $300 in free credits)

🔥 Challenge

Documents are everywhere. We use them for business, daily operations, legal matters, technical docs, education, and even just for fun. However, documents are not databases. They're generally unstructured, and fully understanding them requires multiple reading passes.

So, can we extract structured knowledge from documents using only the following?

1 document
1 prompt
1 request

Let's try with Gemini…

🏁 Setup

🐍 Python packages

We'll use the following packages:

google-genai for calling Gemini with the Google Gen AI Python SDK
networkx for graph management

We'll also need:

tenacity for request management (a dependency of google-genai)
matplotlib and pillow for data visualization (dependencies of networkx)

%pip install --quiet "google-genai>=2.6.0" "networkx[default]"

🤝 Gemini API

To use the Gemini API, we have two main options:

Via Agent Platform (formerly Vertex AI) with a Google Cloud project
Via Google AI Studio with a Gemini API key

The Google Gen AI SDK provides a unified interface to these APIs and we can use environment variables for the configuration. 🔽

🛠️ Option 1 - Gemini API via Agent Platform

Requirements:

A Google Cloud project
The Agent Platform API must be enabled for this project: ▶️ Enable the Agent Platform API

Gen AI SDK environment variables:

GOOGLE_GENAI_USE_ENTERPRISE="True"
GOOGLE_CLOUD_PROJECT="<PROJECT_ID>"
GOOGLE_CLOUD_LOCATION="<LOCATION>"

💡 For preview models, the location must be set to global. For generally available models, we can choose the closest location among the Google model endpoint locations.

ℹ️ Learn more about setting up a project and a development environment.

🛠️ Option 2 - Gemini API via Google AI Studio

Requirement:

A Gemini API key

Gen AI SDK environment variables:

GOOGLE_GENAI_USE_ENTERPRISE="False"
GOOGLE_API_KEY="<API_KEY>"

ℹ️ Learn more about getting a Gemini API key from Google AI Studio.

💡 You can store your environment configuration outside of the source code:

Environment	Method
IDE	`.env` file (or equivalent)
Colab	Colab Secrets (🗝️ icon in left panel, see code below)
Colab Enterprise	Google Cloud project and location are automatically defined
Workbench	Google Cloud project and location are automatically defined

Define the following environment detection functions. You can also define your configuration manually if needed. 🔽

import os
import sys
from collections.abc import Callable

from google import genai

# Manual setup (leave unchanged if setup is environment-defined)

# @markdown **Which API: Agent Platform (formerly Vertex AI) or Google AI Studio?**
GOOGLE_GENAI_USE_ENTERPRISE = True  # @param {type: "boolean"}

# @markdown **Option A - Google Cloud project [+location]**
GOOGLE_CLOUD_PROJECT = ""  # @param {type: "string"}
GOOGLE_CLOUD_LOCATION = "global"  # @param {type: "string"}

# @markdown **Option B - Google AI Studio API key**
GOOGLE_API_KEY = ""  # @param {type: "string"}


def check_environment() -> bool:
    check_colab_user_authentication()
    return check_manual_setup() or check_enterprise() or check_colab() or check_local()


def check_manual_setup() -> bool:
    return check_define_env_vars(
        GOOGLE_GENAI_USE_ENTERPRISE,
        GOOGLE_CLOUD_PROJECT.strip(),  # Might have been pasted with a newline
        GOOGLE_CLOUD_LOCATION,
        GOOGLE_API_KEY,
    )


def check_enterprise() -> bool:
    # Workbench and Colab Enterprise
    match os.getenv("VERTEX_PRODUCT", ""):
        case "WORKBENCH_INSTANCE":
            pass
        case "COLAB_ENTERPRISE":
            if not running_in_colab_env():
                return False
        case _:
            return False

    return check_define_env_vars(
        True,
        os.getenv("GOOGLE_CLOUD_PROJECT", ""),
        os.getenv("GOOGLE_CLOUD_REGION", ""),
        "",
    )


def check_colab() -> bool:
    if not running_in_colab_env():
        return False

    # Colab Enterprise was checked before, so this is Colab only
    from google.colab import auth as colab_auth  # type: ignore

    colab_auth.authenticate_user()

    # Use Colab Secrets (🗝️ icon in left panel) to store the environment variables
    # Secrets are private, visible only to you and the notebooks that you select
    # - Agent Platform: Store your settings as secrets
    # - Google AI: Directly import your Gemini API key from the UI
    enterprise, project, location, api_key = get_vars(get_colab_secret)

    return check_define_env_vars(enterprise, project, location, api_key)


def check_local() -> bool:
    enterprise, project, location, api_key = get_vars(os.getenv)

    return check_define_env_vars(enterprise, project, location, api_key)


def running_in_colab_env() -> bool:
    # Colab or Colab Enterprise
    return "google.colab" in sys.modules


def check_colab_user_authentication() -> None:
    if running_in_colab_env():
        from google.colab import auth as colab_auth  # type: ignore

        colab_auth.authenticate_user()


def get_colab_secret(secret_name: str, default: str) -> str:
    from google.colab import errors, userdata  # type: ignore

    try:
        return userdata.get(secret_name)
    except errors.SecretNotFoundError:
        return default


def disable_colab_cell_scrollbar() -> None:
    if running_in_colab_env():
        from google.colab import output  # type: ignore

        output.no_vertical_scroll()


def get_vars(getenv: Callable[[str, str], str]) -> tuple[bool, str, str, str]:
    # Limit getenv calls to the minimum (may trigger UI confirmation for secret access)
    enterprise_str = getenv("GOOGLE_GENAI_USE_ENTERPRISE", "")
    if not enterprise_str:
        enterprise_str = getenv("GOOGLE_GENAI_USE_VERTEXAI", "")
    if enterprise_str:
        enterprise = enterprise_str.lower() in ["true", "1"]
    else:
        enterprise = bool(getenv("GOOGLE_CLOUD_PROJECT", ""))

    project = getenv("GOOGLE_CLOUD_PROJECT", "") if enterprise else ""
    location = getenv("GOOGLE_CLOUD_LOCATION", "") if project else ""
    api_key = getenv("GOOGLE_API_KEY", "") if not project else ""

    return enterprise, project, location, api_key


def check_define_env_vars(
    enterprise: bool,
    project: str,
    location: str,
    api_key: str,
) -> bool:
    match (enterprise, bool(project), bool(location), bool(api_key)):
        case (True, True, _, _):
            # Agent Platform - Google Cloud project [+location]
            location = location or "global"
            define_env_vars(enterprise, project, location, "")
        case (True, False, _, True):
            # Agent Platform - API key
            define_env_vars(enterprise, "", "", api_key)
        case (False, _, _, True):
            # Google AI Studio - API key
            define_env_vars(enterprise, "", "", api_key)
        case _:
            return False

    return True


def define_env_vars(
    enterprise: bool,
    project: str,
    location: str,
    api_key: str,
) -> None:
    os.environ["GOOGLE_GENAI_USE_ENTERPRISE"] = str(enterprise)
    os.environ["GOOGLE_GENAI_USE_VERTEXAI"] = str(enterprise)
    os.environ["GOOGLE_CLOUD_PROJECT"] = project
    os.environ["GOOGLE_CLOUD_LOCATION"] = location
    os.environ["GOOGLE_API_KEY"] = api_key


def check_configuration(client: genai.Client) -> None:
    service = "Agent Platform" if client.vertexai else "Google AI Studio"
    print(f"✅ Using the {service} API", end="")

    if client._api_client.project:
        print(f' with project "{client._api_client.project[:7]}…"', end="")
        print(f' in location "{client._api_client.location}"')
    elif client._api_client.api_key:
        api_key = client._api_client.api_key
        print(f' with API key "{api_key[:5]}…{api_key[-5:]}"', end="")
        print(f" (in case of error, make sure it was created for {service})")


print("✅ Environment functions defined")

✅ Environment functions defined

🤖 Gen AI SDK

To send Gemini requests, we'll use a google.genai client:

from google import genai

check_environment()

client = genai.Client()

check_configuration(client)

✅ Using the Agent Platform API with project "lpdemo-…" in location "global"

🔣 Input data

We need a suite of test data to develop our solution.

Multimodality

We'll test the following types:

Text (text/plain): Classic books are good text sources of varying lengths and languages.
PDF (application/pdf): Legal agreements are also great examples of complex and dense documents.

Gemini is natively multimodal, which means it can process different types of inputs. Once we've built knowledge graphs from text or PDF inputs, the solution will also naturally support the following formats:

Image (image/*)
Audio (audio/*)
Video (video/*)

General knowledge

⚠️ LLMs are trained on general knowledge, which becomes part of their "long-term memory". To avoid generating memorized information, we'll explicitly instruct the model to use only the provided inputs.

Multilinguality

Gemini is also natively multilingual, which lets us process inputs and generate outputs in 100+ languages.

To keep things general, we'll use English for prompts and knowledge graphs, but you can use any of the 100+ supported languages, as long as your prompts remain clear and explicit.

Let's define a few data sources and helpers: 🔽

import mimetypes
from collections.abc import Iterator
from enum import Enum
from pathlib import Path

from google.genai.types import Part

GOOGLE_CLOUD_STORAGE_PREFIX = "gs://"
HTTPS_PREFIX = "https://"
FILE_PREFIX = "file://"
LOCAL_FOLDER = "./"


class Source(Enum):
    def yield_contents(self) -> Iterator[Part]:
        file_uri = self.value
        if not client.vertexai:
            file_uri = convert_to_https_url_if_cloud_storage_uri(file_uri)
        mime_type, _ = mimetypes.guess_type(file_uri)
        assert mime_type is not None, f"❌ Could not determine mime type: {file_uri=}"

        if file_uri.startswith((GOOGLE_CLOUD_STORAGE_PREFIX, HTTPS_PREFIX)):
            yield Part.from_uri(file_uri=file_uri, mime_type=mime_type)
            return

        if file_uri.startswith(FILE_PREFIX):
            file = Path(file_uri.removeprefix(FILE_PREFIX))
            assert file.exists(), f"❌ File does not exist: {file=}"
            if mime_type == "text/plain":
                yield Part.from_text(text=file.read_text(encoding="utf-8"))
            else:
                yield Part.from_bytes(data=file.read_bytes(), mime_type=mime_type)
            return

    def yield_source_names(self) -> Iterator[str]:
        yield self.name

    def yield_source_links(self) -> Iterator[str]:
        file_uri = convert_to_https_url_if_cloud_storage_uri(self.value)
        if file_uri.startswith(HTTPS_PREFIX):
            yield file_uri
            return
        if file_uri.startswith(FILE_PREFIX):
            yield file_uri.removeprefix(FILE_PREFIX)
            return


def convert_to_https_url_if_cloud_storage_uri(uri: str) -> str:
    return (
        f"{HTTPS_PREFIX}storage.googleapis.com/{uri.removeprefix(GOOGLE_CLOUD_STORAGE_PREFIX)}"
        if uri.startswith(GOOGLE_CLOUD_STORAGE_PREFIX)
        else uri
    )


def local_file(filename: str) -> str:
    return f"{FILE_PREFIX}{LOCAL_FOLDER}{filename}"


# You can find public domain books on Project Gutenberg: https://gutenberg.org/ebooks
def project_gutenberg_txt_url(id: int) -> str:
    return f"{HTTPS_PREFIX}gutenberg.org/cache/epub/{id}/pg{id}.txt"


class Classic(Source):
    en_hugo_les_misérables = project_gutenberg_txt_url(135)
    en_dumas_count_of_monte_cristo = project_gutenberg_txt_url(1184)
    fr_zola_thérèse_raquin = project_gutenberg_txt_url(7461)
    fr_dumas_trois_mousquetaires = project_gutenberg_txt_url(13951)
    fr_dumas_vingt_ans_après = project_gutenberg_txt_url(13952)
    fr_dumas_comte_de_monte_cristo_1 = project_gutenberg_txt_url(17989)
    fr_dumas_comte_de_monte_cristo_2 = project_gutenberg_txt_url(17990)
    fr_dumas_comte_de_monte_cristo_3 = project_gutenberg_txt_url(17991)
    fr_dumas_comte_de_monte_cristo_4 = project_gutenberg_txt_url(17992)


class Document(Source):
    en_pharma_dev_agreement = "gs://cloud-samples-data/documentai/ContractDocAI/CUAD_v1/Part_I/Development/PhasebioPharmaceuticalsInc_20200330_10-K_EX-10.21_12086810_EX-10.21_Development Agreement.pdf"


class Collection(Source):
    fr_dumas_comte_de_monte_cristo = [
        Classic.fr_dumas_comte_de_monte_cristo_1,
        Classic.fr_dumas_comte_de_monte_cristo_2,
        Classic.fr_dumas_comte_de_monte_cristo_3,
        Classic.fr_dumas_comte_de_monte_cristo_4,
    ]
    fr_dumas_trois_mousquetaires_vingt_ans_après = [
        Classic.fr_dumas_trois_mousquetaires,
        Classic.fr_dumas_vingt_ans_après,
    ]

    def yield_contents(self) -> Iterator[Part]:
        for source in self.value:
            yield from source.yield_contents()

    def yield_source_names(self) -> Iterator[str]:
        for source in self.value:
            yield from source.yield_source_names()

    def yield_source_links(self) -> Iterator[str]:
        for source in self.value:
            yield from source.yield_source_links()


def display_input_data_caption(source: Source) -> None:
    names = list(source.yield_source_names())
    links = list(source.yield_source_links())
    links = ", ".join(
        f"[{name}](<{link}>)" for name, link in zip(names, links, strict=True)
    )
    md = f"**Input data** ({links})"
    display_markdown(md)


print("✅ Data helpers defined")

✅ Data helpers defined

🧠 Gemini model

Gemini comes in different versions and sizes (Flash-Lite, Flash, and Pro).

Let's get started with Gemini 3.1 Flash-Lite, as it offers high performance, low latency, and very high output speed:

GEMINI_3_1_FLASH_LITE = "gemini-3.1-flash-lite"

⚙️ Gemini configuration

Gemini can be used in different ways, ranging from factual to creative modes. We're essentially dealing with a data-extraction use case. We want the results to be as factual and deterministic as possible. To achieve this, we can adjust the content generation parameters.

We'll set the temperature, top_p, and seed parameters to minimize randomness:

temperature=0.0
top_p=0.0
seed=42 (arbitrary fixed value)

🛠️ Helpers

Now, let's add core helper classes and functions: 🔽

from enum import StrEnum, auto

import IPython.display
import tenacity
from google.genai.errors import ClientError
from google.genai.types import (
    FinishReason,
    GenerateContentConfig,
    GenerateContentResponse,
    ThinkingConfig,
    ThinkingLevel,
)


class Model(Enum):
    GEMINI_3_1_FLASH_LITE = "gemini-3.1-flash-lite"
    GEMINI_3_5_FLASH = "gemini-3.5-flash"
    GEMINI_2_5_FLASH = "gemini-2.5-flash"
    GEMINI_2_5_PRO = "gemini-2.5-pro"
    # Preview
    GEMINI_3_1_PRO = "gemini-3.1-pro-preview"
    # Default model
    DEFAULT = GEMINI_3_1_FLASH_LITE


# Default configuration for more deterministic outputs
DEFAULT_CONFIG = GenerateContentConfig(
    temperature=0.0,
    top_p=0.0,
    seed=42,  # Arbitrary fixed value
)


class ShowAs(StrEnum):
    DONT_SHOW = auto()
    TEXT = auto()
    MARKDOWN = auto()


def generate_content(
    prompt: str,
    source: Source | str | None = None,
    *,
    model: Model | None = None,
    config: GenerateContentConfig | None = None,
    system_instruction: str | None = None,
    show_prompt: ShowAs = ShowAs.DONT_SHOW,
    show_response: ShowAs = ShowAs.MARKDOWN,
    only_show_prompt: bool = False,
    return_response: bool = False,
) -> GenerateContentResponse | None:
    disable_colab_cell_scrollbar()

    model = model or Model.DEFAULT
    model_id = model.value
    prompt_contents = get_prompt_contents(prompt, source, show_prompt, only_show_prompt)
    if only_show_prompt:
        return None
    config = config or get_generate_content_config(model, system_instruction)
    client = check_client_for_model(model)

    response = None
    display_request_header(model_id, source)
    for attempt in get_retrier():
        with attempt:
            response = client.models.generate_content(
                model=model_id,
                contents=prompt_contents,  # type: ignore
                config=config,
            )
            display_response_info(response)
            display_response(response, show_response)

    return response if return_response else None


def get_prompt_contents(
    prompt: str,
    source: Source | str | None,
    show_prompt: ShowAs,
    only_show_prompt: bool,
) -> list[str | Part]:
    def yield_prompt_contents() -> Iterator[str | Part]:
        if not source:
            yield prompt.strip()
            return
        yield "==Start of input data==\n"
        if isinstance(source, str):
            yield f"{source.strip()}\n"
        else:
            yield from source.yield_contents()
        yield "==End of input data==\n"
        yield f"==Start of user prompt==\n{prompt.strip()}\n==End of user prompt=="

    prompt_contents = list(yield_prompt_contents())
    display_prompt(prompt_contents, show_prompt, only_show_prompt)

    return prompt_contents


def get_generate_content_config(
    model: Model,
    system_instruction: str | None = None,
) -> GenerateContentConfig:
    thinking_config = get_thinking_config_for_model(model)

    return GenerateContentConfig(
        system_instruction=system_instruction,
        temperature=DEFAULT_CONFIG.temperature,
        top_p=DEFAULT_CONFIG.top_p,
        seed=DEFAULT_CONFIG.seed,
        thinking_config=thinking_config,
    )


def get_thinking_config_for_model(model: Model) -> ThinkingConfig | None:
    # Use minimal thinking configurations since our prompt will directly provide a chain of thought
    match model:
        case Model.GEMINI_2_5_FLASH:
            return ThinkingConfig(thinking_budget=0)
        case Model.GEMINI_2_5_PRO:
            return ThinkingConfig(thinking_budget=128, include_thoughts=False)
        case Model.GEMINI_3_1_FLASH_LITE | Model.GEMINI_3_5_FLASH:
            return ThinkingConfig(thinking_level=ThinkingLevel.MINIMAL)
        case Model.GEMINI_3_1_PRO:
            return ThinkingConfig(thinking_level=ThinkingLevel.LOW)
        case _:
            return None  # Default (dynamic thinking is generally enabled)


def check_client_for_model(model: Model) -> genai.Client:
    if (
        model.value.endswith("-preview")
        and client.vertexai
        and client._api_client.location != "global"
    ):
        # Preview models are only available on the "global" location
        return genai.Client(
            enterprise=client.vertexai,
            project=client._api_client.project,
            location="global",
        )

    return client


def get_retrier() -> tenacity.Retrying:
    return tenacity.Retrying(
        stop=tenacity.stop_after_attempt(7),
        wait=tenacity.wait_incrementing(start=10, increment=1),
        retry=tenacity.retry_if_exception(should_retry_request),
        reraise=True,
    )


def should_retry_request(err: BaseException) -> bool:
    if not isinstance(err, ClientError):
        return False
    print(f"❌ ClientError {err.code}: {err.message}")

    retry = False
    match err.code:
        case 400 if err.message is not None and " try again " in err.message:
            # Workshop: project accessing Cloud Storage for the first time (service agent provisioning)
            retry = True
        case 429:
            # Workshop: temporary project with 1 QPM quota
            retry = True
    print(f"🔄 Retry: {retry}")

    return retry


PRINT_COLUMNS = 80
PRINT_SEPARATOR_CHAR = "-"
PRINT_SEPARATOR = PRINT_COLUMNS * PRINT_SEPARATOR_CHAR


def print_caption(caption: str) -> None:
    print(f" {caption} ".center(PRINT_COLUMNS, PRINT_SEPARATOR_CHAR))


def print_separator() -> None:
    print(PRINT_SEPARATOR)


def display_response_info(response: GenerateContentResponse) -> None:
    if usage_metadata := response.usage_metadata:
        if usage_metadata.prompt_token_count:
            print(f"Input tokens    : {usage_metadata.prompt_token_count:9,d}")
        if usage_metadata.cached_content_token_count:
            print(f"Cached tokens   : {usage_metadata.cached_content_token_count:9,d}")
        if usage_metadata.candidates_token_count:
            print(f"Output tokens   : {usage_metadata.candidates_token_count:9,d}")
        if usage_metadata.thoughts_token_count:
            print(f"Thoughts tokens : {usage_metadata.thoughts_token_count:9,d}")
    if not response.candidates:
        print("❌ No `response.candidates`")
        return
    if (finish_reason := response.candidates[0].finish_reason) != FinishReason.STOP:
        print(f"❌ {finish_reason = }")
    if not response.text:
        print("❌ No `response.text`")
        return


def display_prompt(
    contents: list[str | Part],
    show_as: ShowAs,
    only_show_prompt: bool,
) -> None:
    def yield_prompt_strings() -> Iterator[str]:
        for content in contents:
            if isinstance(content, Part):
                yield f"{content!r}\n"
                continue
            yield content

    if only_show_prompt and show_as == ShowAs.DONT_SHOW:
        show_as = ShowAs.TEXT
    if show_as == ShowAs.DONT_SHOW:
        return

    separator = "\n" if show_as == ShowAs.MARKDOWN else ""
    prompt = separator.join(yield_prompt_strings())
    print_caption("Prompt")
    match show_as:
        case ShowAs.TEXT:
            print(prompt)
        case ShowAs.MARKDOWN:
            display_markdown(prompt)
    if only_show_prompt:
        print_separator()


def display_request_header(model_id: str, source: Source | str | None = None) -> None:
    print_caption(f"Request / {model_id}")


def display_response(response: GenerateContentResponse, show_as: ShowAs) -> None:
    if show_as == ShowAs.DONT_SHOW or not (response_text := response.text):
        return
    print_caption("Start of Response")
    response_text = response_text.strip()
    match show_as:
        case ShowAs.TEXT:
            print(response_text)
        case ShowAs.MARKDOWN:
            display_markdown(response_text)
    print_caption("End of Response")


def display_markdown(markdown: str) -> None:
    IPython.display.display(IPython.display.Markdown(markdown))


print("✅ Helpers defined")

✅ Helpers defined

🏗️ Prototyping

Before diving into a solution, it helps to start by prototyping to build some intuition about the natural behavior of the model.

Let's define a short text of a few sentences:

text = """
- Henry Jones is a famous archaeologist. He is actually a "Junior" because he is named after his father.
- Sophie is Henry's daughter, shares his last name, and works as a software engineer.
- William Smith is an aerospace engineer and Sophie's lifelong friend. Everybody calls him Bill and Beau is his dog.
- Short Round met Henry as a child. They first became close friends, and Henry officially adopted him a few years later.
- Sophie and Bill both work at Acme Aerospace.
"""

📋 Listing characters

🧪 Let's see if Gemini can spot our characters…

prompt = """
Using only the input data, list all people and animals mentioned.
"""
generate_content(prompt, text)

----------------------- Request / gemini-3.1-flash-lite ------------------------
Input tokens    :       148
Output tokens   :        67
------------------------------ Start of Response -------------------------------
Based on the input data provided, here are the people and animals mentioned:

**People:**
*   Henry Jones (also known as Henry Jones Junior)
*   Sophie Jones
*   William Smith (also known as Bill)
*   Short Round

**Animals:**
*   Beau (Bill's dog)
------------------------------- End of Response --------------------------------

💡 All people and animals are detected as expected.

📋 Listing characters & relationships

🧪 Now, let's see if it can connect the dots and figure out who's who…

prompt = """
Using only the input data, list all people and animals mentioned, and how they relate to each other.
"""
generate_content(prompt, text)

----------------------- Request / gemini-3.1-flash-lite ------------------------
Input tokens    :       156
Output tokens   :       168
------------------------------ Start of Response -------------------------------
Based on the input data provided, here are the people and animals mentioned and their relationships:

**People:**
*   **Henry Jones (Junior):** A famous archaeologist. He is the father of Sophie, the adoptive father of Short Round, and is named after his own father.
*   **Sophie Jones:** A software engineer at Acme Aerospace. She is the daughter of Henry Jones and a lifelong friend of Bill (William Smith).
*   **William (Bill) Smith:** An aerospace engineer at Acme Aerospace. He is a lifelong friend of Sophie and the owner of Beau.
*   **Short Round:** The adopted son of Henry Jones. He met Henry as a child and they became close friends before the adoption.

**Animals:**
*   **Beau:** A dog owned by William (Bill) Smith.
------------------------------- End of Response --------------------------------

💡 Notes

The relationships between the characters can also be neatly consolidated.
Henry's father is not listed as a named character. Here, we can see the difference between explicit and implicit mentions. By default, only explicit entities are detailed (most likely because models are trained to summarize information).
Implied characters (like "he is named after his father") represent a level of indirection. We'll see in the next section how to detect them as well.

📚 Domain terminology

We're not domain experts in the field we're exploring (yet!).

An LLM processes instructions based on the given prompt and its training knowledge. This knowledge is part of its long-term memory, and we can learn a lot directly from the model itself.

🧪 Let's ask Gemini:

prompt = """
What is the terminology used when building a knowledge graph?
Please provide a simple data example in JSON.
"""
generate_content(prompt)

----------------------- Request / gemini-3.1-flash-lite ------------------------
Input tokens    :        21
Output tokens   :       581
------------------------------ Start of Response -------------------------------
Building a knowledge graph involves representing information as a network of interconnected entities. Here is the core terminology and a simple data example.

### Core Terminology

1.  **Entity (Node):** The "things" in your graph (e.g., a person, a place, a product).
2.  **Relationship (Edge/Link):** The connection between two entities. It describes how they interact (e.g., "works at," "lives in," "is a friend of").
3.  **Property (Attribute):** Key-value pairs that provide more detail about an entity or a relationship (e.g., a person's "age" or a relationship's "start_date").
4.  **Label:** A category assigned to a node or edge to define its type (e.g., a node might have the label "Person").
5.  **Schema (Ontology):** The formal structure or "blueprint" that defines the types of entities allowed and the rules for how they can be connected.
6.  **Triple:** The fundamental unit of a knowledge graph, consisting of a **Subject → Predicate → Object** (e.g., *Alice* → *works at* → *Google*).

---

### Simple Data Example (JSON)

In a knowledge graph, data is often represented as a collection of **Nodes** and **Edges**.

```json
{
"nodes": [
    {
    "id": "1",
    "label": "Person",
    "properties": {
        "name": "Alice",
        "age": 30
    }
    },
    {
    "id": "2",
    "label": "Company",
    "properties": {
        "name": "Google",
        "industry": "Technology"
    }
    }
],
"edges": [
    {
    "id": "e1",
    "source": "1",
    "target": "2",
    "label": "WORKS_AT",
    "properties": {
        "since": 2020
    }
    }
]
}
```

### Breakdown of the Example:

*   **Nodes:** We have two entities: "Alice" (a Person) and "Google" (a Company).
*   **Edge:** We have one relationship: "WORKS_AT" connecting Alice to Google.
*   **Properties:** We stored specific details like Alice's age and the year she started working at Google.
*   **Triple representation:** This JSON effectively encodes the triple: **(Alice) —[WORKS_AT]—> (Google)**.
------------------------------- End of Response --------------------------------

💡 We learn that knowledge graphs are made of entities and relationships, also called nodes and edges, and we get a nice introduction to the field. Using domain terminology will help make our prompts explicit and precise.

⛏️ Tabular extraction

To extract knowledge graphs, we'll reason in terms of entities and relationships, adopting domain terminology.

If we think of the final result as a database, our goal is to generate two linked tables, allowing us to reason in terms of data and fields.

Here is a conceptual view of what we want to achieve:

Entities

`id`	`name`	`label`
0	Henry Jones Jr.	person
1	Henry Jones Sr.	person

Relationships

`source_id`	`link`	`target_id`
0	child_of	1

Let's call this approach "tabular extraction" and split our instructions to output two successive tables, while still using a single request…

💠 Entities

In our prototype text, the entities we want to extract are characters (like people or animals). We can define an entity data schema with the fields id (0, 1, 2…), name (full name of the entity), and label (person|animal).

🧪 Let's extract the entities:

prompt = """
**Data Schema**

Entity:
- `id`: Unique integer identifier (0, 1, 2…).
- `name`: Full name of the entity.
- `label`: `person`|`animal`.

**Instructions**

1. Entity Extraction:
   - Extract every distinct entity from the input data that matches an allowed `label`.
   - Include entities that are explicitly named as well as implied entities whose names can be determined from the context.
2. Output the results as a JSON array inside a fenced code block.
"""

generate_content(prompt, text)

----------------------- Request / gemini-3.1-flash-lite ------------------------
Input tokens    :       249
Output tokens   :       195
------------------------------ Start of Response -------------------------------

[
  {
    "id": 0,
    "name": "Henry Jones Jr.",
    "label": "person"
  },
  {
    "id": 1,
    "name": "Henry Jones Sr.",
    "label": "person"
  },
  {
    "id": 2,
    "name": "Sophie Jones",
    "label": "person"
  },
  {
    "id": 3,
    "name": "William Smith",
    "label": "person"
  },
  {
    "id": 4,
    "name": "Beau",
    "label": "animal"
  },
  {
    "id": 5,
    "name": "Short Round",
    "label": "person"
  }
]

------------------------------- End of Response --------------------------------

💡 Remarks

Every entity is dynamically assigned a unique sequential identifier.
The name "Henry Jones Sr." is extracted even though he's not explicitly mentioned in the text. This stems from the instruction to "include entities that are explicitly named as well as implied entities whose names can be determined from the context".

🔗 Relationships

🧪 Now, let's extract both the entities and their relationships:

prompt = """
**Data Schema**

Entity:
- `id`: Unique integer identifier (0, 1, 2…).
- `name`: Full name of the entity.
- `label`: `person`|`animal`.

Relationship:
- `source_id`: `id` of the subject entity.
- `link`: `snake_case` predicate describing the relationship.
- `target_id`: `id` of the object entity.

**Instructions**

1. Entity Extraction:
   - Extract every distinct entity from the input data that matches an allowed `label`.
   - Include entities that are explicitly named as well as implied entities whose names can be determined from the context.
2. Relationship Extraction:
   - Extract every distinct relationship between them.
   - If a relationship changes over time, make sure to include every distinct stage of the relationship.
3. Output a JSON object with keys `entities` and `relationships` inside a fenced code block.
"""

response = generate_content(prompt, text, return_response=True)

----------------------- Request / gemini-3.1-flash-lite ------------------------
Input tokens    :       340
Output tokens   :       456
------------------------------ Start of Response -------------------------------

{
  "entities": [
    {
      "id": 0,
      "name": "Henry Jones Jr.",
      "label": "person"
    },
    {
      "id": 1,
      "name": "Henry Jones Sr.",
      "label": "person"
    },
    {
      "id": 2,
      "name": "Sophie Jones",
      "label": "person"
    },
    {
      "id": 3,
      "name": "William Smith",
      "label": "person"
    },
    {
      "id": 4,
      "name": "Beau",
      "label": "animal"
    },
    {
      "id": 5,
      "name": "Short Round",
      "label": "person"
    }
  ],
  "relationships": [
    {
      "source_id": 0,
      "link": "child_of",
      "target_id": 1
    },
    {
      "source_id": 2,
      "link": "child_of",
      "target_id": 0
    },
    {
      "source_id": 3,
      "link": "friend_of",
      "target_id": 2
    },
    {
      "source_id": 4,
      "link": "pet_of",
      "target_id": 3
    },
    {
      "source_id": 5,
      "link": "friend_of",
      "target_id": 0
    },
    {
      "source_id": 0,
      "link": "adopted",
      "target_id": 5
    },
    {
      "source_id": 5,
      "link": "child_of",
      "target_id": 0
    }
  ]
}

------------------------------- End of Response --------------------------------

💡 Remarks

The output now includes the relationships array.
We get the expected number of relationships, notably the two relationships between Henry and Short Round.
The link predicates are completely dynamic (a level of flexibility we left in the prompt). While it's interesting to see this natural behavior, we'll want to make it more deterministic for production since our prompt has too many degrees of freedom.
Our open prompt extracts relationships in only one direction. For example, "[Animal] pet_of [Person]" is an asymmetric relationship that could also be extracted as "[Person] owner_of [Animal]". This is another area where the prompt is too open-ended. In the finalization section, we'll see an example that asks the model to extract symmetric and asymmetric relationships in both directions.

💎 Structured output

We've concluded our prototyping stage with promising results using a data schema.

To move to production, the next step is to control the generation with a specific structured output.

🗂️ JSON

The JSON format has industry-wide support and serves as a core or intermediate format in many use cases.

For the next step, we would typically define classes using the Pydantic library and request a pure JSON output with a response schema in the config parameters:

response_mime_type="application/json"
response_schema="CLASS_DERIVED_FROM_PYDANTIC_BASE_MODEL" (docs)

⚠️ However, JSON is a pretty verbose format, designed for interoperability but not optimized for size. Even if we generate compact JSON (also called minified JSON), it still has inherent verbosity due to:

repeated keys (object fields) for each object instance
opening and closing brackets
enclosing quotes for each key and string value

ℹ️ When using LLMs, once the first token is generated, the remaining generation time is roughly proportional to the number of output tokens. Similarly, the cost of a request is based on token usage (input + output), with output tokens being significantly more expensive than input tokens.

💡 A better output structure will positively impact both generation speed and request cost.

Let's explore an alternative…

📅 TSV

Our tabular-extraction problem clearly calls for table outputs. An interesting possibility is to ask for Tab-Separated Values (TSV) outputs. For example, we can define our output to be formatted as two consecutive TSV tables.

Example output format

```tsv filename="entities.tsv"
id{TAB}name{TAB}label
[rows]
```

```tsv filename="relationships.tsv"
source_id{TAB}link{TAB}target_id
[rows]
```

Will this work?

Generating structured outputs like TSV will work seamlessly, as Gemini excels at patterns. We just need to be explicit about what's expected.

Will this be efficient?

💡 For our use case, this structure looks optimal:

Field names are stated once in the header row.
Field values are raw, without outer quotes.
Tab and newline separators take at most a single token each and do not collide with field values (unless we're dealing with very special input data, but we could then prompt the model to escape these using different characters or strings).

ℹ️ CSV could be another alternative, but common separators like commas are everywhere in natural language and frequently appear in names and descriptions (e.g., if we decide to extend entity fields). If you're interested in this topic, check out the TOON format, which proposes a JSON alternative using a YAML+CSV mix.

To make an informed decision, we should actually compare the number of tokens needed to represent the same data…

🗜️ Token comparison

Let's define some helpers to compare the same data using pretty-printed JSON, compact JSON, and TSV: 🔽

import csv
import io
import json
import re
from typing import Literal


def get_data_from_response(response: GenerateContentResponse) -> dict:
    response_text = response.text or ""
    pattern = r"```

json\s*(.*?)\s*

```"
    match = re.search(pattern, response_text, re.DOTALL)
    json_str = match.group(1) if match else response_text
    try:
        data = json.loads(json_str)
        if not isinstance(data, dict):
            print("❌ Returning empty dict (could not parse response as dict)")
            data = {}
    except (json.JSONDecodeError, TypeError):
        print("❌ Returning empty dict (failed parsing the JSON string)")
        data = {}
    return data


def get_tsv_string_from_data(data: dict) -> str:
    output = ""
    for key, items in data.items():
        rows = ""
        if items:
            with io.StringIO() as out:
                headers = list(items[0].keys())
                writer = csv.DictWriter(
                    out,
                    fieldnames=headers,
                    delimiter="\t",
                    lineterminator="\n",
                )
                writer.writeheader()
                writer.writerows(items)
                rows = out.getvalue()
        if output:
            output += "\n"
        output += f'```
{% endraw %}
tsv filename="{key}.tsv"\n{rows}
{% raw %}
```\n'
    return output


def print_text_excerpt(title: str, text: str, max_chars: int = 400) -> None:
    assert max_chars > 0
    chars = len(text)
    if chars <= 0:
        print_caption("❌ Empty text")
        return
    if chars <= max_chars:
        print_caption(f"{title} ({chars} chars)")
        print(text)
        return
    print_caption(
        f"{title} - First {max_chars}/{chars} chars ({max_chars / chars:.0%})"
    )
    print(f"{text[:max_chars]}…")


def compare_json_vs_tsv(
    response: GenerateContentResponse | None,
    only_show_excerpts: bool = False,
) -> None:
    def get_gain(rows: list[tuple[int, int]], col: Literal[0, 1]) -> str:
        val_0, val_1 = rows[0][col], rows[1][col]
        return f"**{1 - val_1 / val_0:.1%}**" if val_0 > 0 else "?"

    def yield_table_string_rows(
        source_title: str,
        target_title: str,
        rows: list[tuple[int, int]],
    ) -> Iterator[list[str]]:
        yield ["", "Chars", "Tokens"]
        yield ["-", "-:", "-:"]
        for caption, values in zip([source_title, target_title], rows):
            yield [caption, *[str(value) for value in values]]
        yield ["**Gain**", get_gain(rows, 0), get_gain(rows, 1)]

    def display_gain_table(
        source_title: str,
        target_title: str,
        source_text: str,
        target_text: str,
    ) -> None:
        print_caption(f"{source_title} → {target_title}")
        model = Model.DEFAULT
        model_id = model.value
        client = check_client_for_model(model)

        rows: list[tuple[int, int]] = []
        for s in [source_text, target_text]:
            chars = len(s)
            tokens = client.models.count_tokens(model=model_id, contents=s).total_tokens
            rows.append((chars, tokens or 0))
        markdown = "\n".join(
            "| " + " | ".join(row) + " |"
            for row in yield_table_string_rows(source_title, target_title, rows)
        )
        display_markdown(markdown)

    if response is None:
        print("❌ response is None")
        return
    data = get_data_from_response(response)
    formatted_json = f"```
{% endraw %}
json\n{json.dumps(data, indent=2)}\n
{% raw %}
```"
    compact_json = f"```
{% endraw %}
json\n{json.dumps(data, separators=(',', ':'))}\n
{% raw %}
```"
    tsv = get_tsv_string_from_data(data)

    if only_show_excerpts:
        max_chars = len(tsv)
        print_text_excerpt("Formatted JSON", formatted_json, max_chars)
        print_text_excerpt("Compact JSON", compact_json, max_chars)
        print_text_excerpt("TSV", tsv, max_chars)
        return

    display_gain_table("Formatted JSON", "Compact JSON", formatted_json, compact_json)
    display_gain_table("Compact JSON", "TSV", compact_json, tsv)
    display_gain_table("Formatted JSON", "TSV", formatted_json, tsv)
    print_separator()


print("✅ JSON vs TSV helpers defined")

✅ JSON vs TSV helpers defined

🧪 First, let's compare how much data we can represent for the same number of characters based on our latest API response:

compare_json_vs_tsv(response, only_show_excerpts=True)

----------------- Formatted JSON - First 335/1122 chars (30%) ------------------
```json
{
  "entities": [
    {
      "id": 0,
      "name": "Henry Jones Jr.",
      "label": "person"
    },
    {
      "id": 1,
      "name": "Henry Jones Sr.",
      "label": "person"
    },
    {
      "id": 2,
      "name": "Sophie Jones",
      "label": "person"
    },
    {
      "id": 3,
      "name": "William Smith",
     …
------------------- Compact JSON - First 335/665 chars (50%) -------------------
```json
{"entities":[{"id":0,"name":"Henry Jones Jr.","label":"person"},{"id":1,"name":"Henry Jones Sr.","label":"person"},{"id":2,"name":"Sophie Jones","label":"person"},{"id":3,"name":"William Smith","label":"person"},{"id":4,"name":"Beau","label":"animal"},{"id":5,"name":"Short Round","label":"person"}],"relationships":[{"source_i…
------------------------------- TSV (335 chars) --------------------------------
```tsv filename="entities.tsv"
id  name    label
0   Henry Jones Jr. person
1   Henry Jones Sr. person
2   Sophie Jones    person
3   William Smith   person
4   Beau    animal
5   Short Round person
```

```tsv filename="relationships.tsv"
source_id   link    target_id
0   child_of    1
2   child_of    0
3   friend_of   2
4   pet_of  3
5   friend_of   0
0   adopted 5
5   child_of    0
```

💡 Notice how much more information can be represented in the same number of text characters. This will apply similarly to token counts.

🧪 And now, let's compare the gains, especially for token counts:

compare_json_vs_tsv(response)

------------------------ Formatted JSON → Compact JSON -------------------------

	Chars	Tokens
Formatted JSON	1122	456
Compact JSON	665	216
Gain	40.7%	52.6%

------------------------------ Compact JSON → TSV ------------------------------

	Chars	Tokens
Compact JSON	665	216
TSV	335	137
Gain	49.6%	36.6%

----------------------------- Formatted JSON → TSV -----------------------------

	Chars	Tokens
Formatted JSON	1122	456
TSV	335	137
Gain	70.1%	70.0%

--------------------------------------------------------------------------------

💡 Savings in output tokens:

30 to 40% compared to compact JSON
60 to 70% compared to standard JSON

With a double-digit percentage reduction in output tokens, building knowledge graphs with TSV outputs is significantly faster (and cheaper)!

Now, let's finalize our code with optimized structures…

🚀 Finalization

🧩 Structure

First, it helps to define a structured prompt template, so we can focus on specific parts of our solution using a divide-and-conquer approach.

Here's a possible prompt template:

KNOWLEDGE_GRAPH_PROMPT_TEMPLATE = """
**Data Schema**

{data_schema}

**Instructions**

{instructions}

**Output Format**

{output_format}
"""

Then, here are some possible Entity, Relationship, and KnowledgeGraph data classes with the matching output format:

from dataclasses import dataclass, field


@dataclass
class Entity:
    id: int
    name: str
    label: str


@dataclass
class Relationship:
    source_id: int
    link: str
    target_id: int


@dataclass
class KnowledgeGraph:
    entities: list[Entity] = field(default_factory=list)
    relationships: list[Relationship] = field(default_factory=list)


TAB = "\t"
KNOWLEDGE_GRAPH_OUTPUT_FORMAT = f"""
Format the output strictly as two TSV code blocks (including the header row):

```tsv filename="entities.tsv"
id{TAB}name{TAB}label
[data_rows]
```

```tsv filename="relationships.tsv"
source_id{TAB}link{TAB}target_id
[data_rows]
```
"""

💡 While the Gen AI SDK natively supports Pydantic models for JSON structured outputs, we're using standard Python data classes here and TSV outputs to maximize our token efficiency.

ℹ️ If you use multiple entity or relationship data classes in your solution, you can dynamically generate the output format specification using features of the dataclasses package (like class docstrings and field descriptions).

Now, let's define a few helpers to generate a knowledge graph: 🔽

from dataclasses import fields, is_dataclass
from typing import get_args, get_origin, get_type_hints

def generate_knowledge_graph(
    data_schema: str,
    instructions: str,
    source: Source | str,
    *,
    model: Model | None = None,
    config: GenerateContentConfig | None = None,
    system_instruction: str | None = None,
    show_prompt: ShowAs = ShowAs.DONT_SHOW,
    show_response: ShowAs = ShowAs.DONT_SHOW,
) -> KnowledgeGraph:
    prompt = get_prompt_for_data_schema_and_instructions(data_schema, instructions)
    response = generate_content(
        prompt,
        source,
        model=model,
        config=config,
        system_instruction=system_instruction,
        show_prompt=show_prompt,
        show_response=show_response,
        return_response=True,
    )

    knowledge_graph = (
        parse_list_dataclass(KnowledgeGraph, response)
        if isinstance(response, GenerateContentResponse)
        else KnowledgeGraph()
    )
    display_knowledge_graph_info(knowledge_graph)

    return knowledge_graph


def show_knowledge_graph_prompt(
    data_schema: str,
    instructions: str,
    source: Source | str,
    *,
    model: Model | None = None,
    config: GenerateContentConfig | None = None,
    system_instruction: str | None = None,
    show_as: ShowAs = ShowAs.TEXT,
) -> None:
    prompt = get_prompt_for_data_schema_and_instructions(data_schema, instructions)
    generate_content(
        prompt,
        source,
        model=model,
        config=config,
        system_instruction=system_instruction,
        show_prompt=show_as,
        only_show_prompt=True,
    )


def get_prompt_for_data_schema_and_instructions(
    data_schema: str,
    instructions: str,
) -> str:
    return KNOWLEDGE_GRAPH_PROMPT_TEMPLATE.format(
        data_schema=data_schema.strip(),
        instructions=instructions.strip(),
        output_format=KNOWLEDGE_GRAPH_OUTPUT_FORMAT.strip(),
    ).strip()


def parse_list_dataclass[T](cls: type[T], response: GenerateContentResponse) -> T:
    assert is_dataclass(cls)
    if not (response_text := response.text):
        return cls()

    data = {}
    for f in fields(cls):
        origin, list_types = get_origin(f.type), get_args(f.type)
        assert (
            origin is list
        ), f"❌ Field {f.name} must be a list[dataclass] parameterized list"
        assert len(list_types) == 1, f"❌ Expected 1 single type: {len(list_types)=}"
        data[f.name] = parse_tsv_block(list_types[0], response_text, f.name)

    return cls(**data)


def parse_tsv_block[T](cls: type[T], data: str, tsv_filestem: str) -> list[T]:
    rows = []
    valid_fields = get_type_hints(cls)
    tsv_string = extract_tsv_block(data, tsv_filestem)
    for row in csv.DictReader(io.StringIO(tsv_string), delimiter="\t"):
        casted_data = {}
        for key, value in row.items():
            if key not in valid_fields or value is None:
                continue
            field_type = valid_fields[key]
            try:  # Note: Works only for directly castable types such as int, float, str, enum (e.g., not bool)
                casted_data[key] = field_type(value)
            except (ValueError, TypeError):
                print(f'❌ Could not cast "{value}" to {field_type} → Skipping {row=}')
                break
        else:
            try:
                rows.append(cls(**casted_data))
            except TypeError as e:
                print(f"❌ Could not instantiate {cls.__name__}: {e} → Skipping {row=}")

    return rows


def extract_tsv_block(data: str, filestem: str) -> str:
    pattern = rf'```
{% endraw %}
tsv filename="{re.escape(filestem)}.tsv"\s*\n(.*?)
{% raw %}
```'
    if not (match := re.search(pattern, data, flags=re.DOTALL)):
        print(f'❌ Could not find a TSV block for "{filestem=}"')
        return ""
    return match.group(1)


def display_knowledge_graph_info(kg: KnowledgeGraph) -> None:
    print_caption("Knowledge Graph Info")
    print(f"Entities      : {len(kg.entities):3,d}")
    print(f"Relationships : {len(kg.relationships):3,d}")
    print_separator()


print("✅ Knowledge graph generation helpers defined")

✅ Knowledge graph generation helpers defined

And here is a possible data schema with some instructions to generate a knowledge graph for our book analysis use case:

from enum import StrEnum, auto


class BookAnalysisEntityLabel(StrEnum):
    PERSON = auto()
    ANIMAL = auto()
    ORGANIZATION = auto()


def pipe_delimited_union(enum: type[StrEnum]) -> str:
    return "|".join(f"`{e.value}`" for e in enum)


BOOK_ANALYSIS_DATA_SCHEMA = f"""
Entity:
- `id`: Unique integer identifier (0, 1, 2…).
- `name`: Most complete name as exclusively determined from the input data.
- `label`: {pipe_delimited_union(BookAnalysisEntityLabel)}.

Relationship:
- `source_id`: `id` of the subject entity.
- `link`: `snake_case` predicate.
- `target_id`: `id` of the object entity.
"""

BOOK_ANALYSIS_INSTRUCTIONS = """
- Extract every distinct entity:
  - Treat distinct pseudonyms/identities as separate entities.
  - Include implied entities whose names can be exclusively determined from the input data.
- Extract every distinct relationship between them:
  - Use specific `link` predicates in `snake_case` as needed (e.g., `alias_of`, `son_of`, `fiancée_of`, `friend_of`, `murderer_of`, `employer_of`, `in_love_with`, `rival_of`).
  - If a relationship changes over time, make sure to include every distinct stage of the relationship.
  - For every asymmetric relationship extracted, make sure to include the logical inverse relationship (e.g., `A husband_of B` AND `B wife_of A`, `A employer_of B` AND `B employee_of A`).
  - For every symmetric relationship extracted, make sure to include both directions (e.g., `A friend_of B` AND `B friend_of A`).
"""

Verify the structured prompt:

show_knowledge_graph_prompt(
    BOOK_ANALYSIS_DATA_SCHEMA,
    BOOK_ANALYSIS_INSTRUCTIONS,
    text,
    show_as=ShowAs.TEXT,
)

------------------------------------ Prompt ------------------------------------
==Start of input data==
- Henry Jones is a famous archaeologist. He is actually a "Junior" because he is named after his father.
- Sophie is Henry's daughter, shares his last name, and works as a software engineer.
- William Smith is an aerospace engineer and Sophie's lifelong friend. Everybody calls him Bill and Beau is his dog.
- Short Round met Henry as a child. They first became close friends, and Henry officially adopted him a few years later.
- Sophie and Bill both work at Acme Aerospace.
==End of input data==
==Start of user prompt==
**Data Schema**

Entity:
- `id`: Unique integer identifier (0, 1, 2…).
- `name`: Most complete name as exclusively determined from the input data.
- `label`: `person`|`animal`|`organization`.

Relationship:
- `source_id`: `id` of the subject entity.
- `link`: `snake_case` predicate.
- `target_id`: `id` of the object entity.

**Instructions**

- Extract every distinct entity:
  - Treat distinct pseudonyms/identities as separate entities.
  - Include implied entities whose names can be exclusively determined from the input data.
- Extract every distinct relationship between them:
  - Use specific `link` predicates in `snake_case` as needed (e.g., `alias_of`, `son_of`, `fiancée_of`, `friend_of`, `murderer_of`, `employer_of`, `in_love_with`, `rival_of`).
  - If a relationship changes over time, make sure to include every distinct stage of the relationship.
  - For every asymmetric relationship extracted, make sure to include the logical inverse relationship (e.g., `A husband_of B` AND `B wife_of A`, `A employer_of B` AND `B employee_of A`).
  - For every symmetric relationship extracted, make sure to include both directions (e.g., `A friend_of B` AND `B friend_of A`).

**Output Format**

Format the output strictly as two TSV code blocks (including the header row):

```tsv filename="entities.tsv"
id  name    label
[data_rows]
```

```tsv filename="relationships.tsv"
source_id   link    target_id
[data_rows]
```
==End of user prompt==
--------------------------------------------------------------------------------

🧪 Let's generate a knowledge graph:

knowledge_graph = generate_knowledge_graph(
    BOOK_ANALYSIS_DATA_SCHEMA,
    BOOK_ANALYSIS_INSTRUCTIONS,
    text,
    show_response=ShowAs.TEXT,
)

print(knowledge_graph.entities)
print(knowledge_graph.relationships)

----------------------- Request / gemini-3.1-flash-lite ------------------------
Input tokens    :       534
Output tokens   :       244
------------------------------ Start of Response -------------------------------
```tsv filename="entities.tsv"
id  name    label
0   Henry Jones Jr. person
1   Henry Jones Sr. person
2   Sophie Jones    person
3   William Smith   person
4   Bill    person
5   Beau    animal
6   Short Round person
7   Acme Aerospace  organization
```

```tsv filename="relationships.tsv"
source_id   link    target_id
0   son_of  1
1   father_of   0
0   father_of   2
2   daughter_of 0
2   friend_of   3
3   friend_of   2
3   alias_of    4
4   alias_of    3
3   employee_of 7
7   employer_of 3
2   employee_of 7
7   employer_of 2
3   owner_of    5
5   pet_of  3
6   friend_of   0
0   friend_of   6
0   adopted_father_of   6
6   adopted_son_of  0
```
------------------------------- End of Response --------------------------------
----------------------------- Knowledge Graph Info -----------------------------
Entities      :   8
Relationships :  18
--------------------------------------------------------------------------------
[Entity(id=0, name='Henry Jones Jr.', label='person'), Entity(id=1, name='Henry Jones Sr.', label='person'), Entity(id=2, name='Sophie Jones', label='person'), Entity(id=3, name='William Smith', label='person'), Entity(id=4, name='Bill', label='person'), Entity(id=5, name='Beau', label='animal'), Entity(id=6, name='Short Round', label='person'), Entity(id=7, name='Acme Aerospace', label='organization')]
[Relationship(source_id=0, link='son_of', target_id=1), Relationship(source_id=1, link='father_of', target_id=0), Relationship(source_id=0, link='father_of', target_id=2), Relationship(source_id=2, link='daughter_of', target_id=0), Relationship(source_id=2, link='friend_of', target_id=3), Relationship(source_id=3, link='friend_of', target_id=2), Relationship(source_id=3, link='alias_of', target_id=4), Relationship(source_id=4, link='alias_of', target_id=3), Relationship(source_id=3, link='employee_of', target_id=7), Relationship(source_id=7, link='employer_of', target_id=3), Relationship(source_id=2, link='employee_of', target_id=7), Relationship(source_id=7, link='employer_of', target_id=2), Relationship(source_id=3, link='owner_of', target_id=5), Relationship(source_id=5, link='pet_of', target_id=3), Relationship(source_id=6, link='friend_of', target_id=0), Relationship(source_id=0, link='friend_of', target_id=6), Relationship(source_id=0, link='adopted_father_of', target_id=6), Relationship(source_id=6, link='adopted_son_of', target_id=0)]

💡 This is looking good!

Now, let's go to the next stage and build a network graph from our data…

🪢 Network graph

Now that we have our entities and relationships neatly packed into data classes, let's bring them to life. We'll use networkx to build a network graph. Using domain terminology, entities become nodes and relationships become directed edges. We'll also calculate node centralities to identify key nodes and use the Louvain method to detect communities (clusters of closely related nodes)…

Helpers 🔽

import textwrap
from typing import cast

import networkx as nx
import numpy as np
from networkx.algorithms.community.louvain import louvain_communities

NODE_CENTRALITY = "node_centrality"
NODE_COMMUNITY_INDEX = "node_community_index"
NODE_COLOR = "node_color"
EDGE_COLOR = "edge_color"
# Wrap names over multiple lines (avoids long node labels)
MULTILINE_NAMES = True
MULTILINE_CHARS = 12


def build_graph(kg: KnowledgeGraph, remove_orphan_nodes: bool) -> nx.DiGraph:
    # For simplicity, we build a DiGraph (adapt to MultiDiGraph if needed)
    graph = nx.DiGraph()

    # Nodes ← Entities
    node_name_from_id: dict[int, str] = {}
    for entity in kg.entities:
        node_name, display_name = get_node_and_display_names_for_entity(entity)
        node_name_from_id[entity.id] = node_name
        graph.add_node(node_name, name=display_name)

    # Edges ← Relationships
    for relationship in kg.relationships:
        source_node = node_name_from_id.get(relationship.source_id, "")
        target_node = node_name_from_id.get(relationship.target_id, "")
        if not source_node or not target_node:
            print(f"❌ Skipping relationship due to empty node:\n{relationship}")
            continue
        # For simplicity, each link has the same weight
        # For better community detection, define your own weight mappings (e.g., family members have stronger bonds)
        weight = 1
        edge_label = relationship.link
        if graph.has_edge(source_node, target_node):
            existing_data = graph[source_node][target_node]
            existing_data["link"] += f"\n{edge_label}"
            existing_data["weight"] += weight
        else:
            graph.add_edge(source_node, target_node, link=edge_label, weight=weight)

    if remove_orphan_nodes:
        graph.remove_nodes_from(list(nx.isolates(graph)))

    return graph


def get_node_and_display_names_for_entity(entity: Entity) -> tuple[str, str]:
    snake_case_name = "_".join(map(str.lower, entity.name.split()))
    node_name = f"{entity.id}_{snake_case_name}"

    display_name = entity.name
    if MULTILINE_NAMES:
        display_name = "\n".join(textwrap.wrap(display_name, width=MULTILINE_CHARS))

    return node_name, display_name


def color_gen(color_count: int) -> Iterator[str]:
    B50, R50, Y50, G50 = ("#4285F4", "#EA4335", "#FBBC04", "#34A853")
    B20, R20, Y20, G20 = ("#AECBFA", "#F6AEA9", "#FDE293", "#A8DAB5")
    B05, R05, Y05, G05 = ("#E8F0FE", "#FCE8E6", "#FEF7E0", "#E6F4EA")
    COLORS = [B50, R50, Y50, G50, B20, R20, Y20, G20, B05, R05, Y05, G05]
    for i in range(color_count):
        yield COLORS[i % len(COLORS)]


Node = str
Positions = dict[Node, np.ndarray]
Community = set[Node]
Communities = list[Community]
Nodes = list[Node]
INTER_COMMUNITY_EDGE_COLOR = "#8888"


def init_graph_data(graph: nx.Graph) -> Nodes:
    def node_centrality(node: Node) -> float:
        return graph.nodes[node][NODE_CENTRALITY]

    def community_max_centrality(community: Community) -> float:
        return max(node_centrality(node) for node in community)

    def nodes_sorted_by_community(communities: Communities) -> Nodes:
        entities = []
        for community in communities:
            sorted_entities = sorted(community, key=node_centrality, reverse=True)
            entities.extend(sorted_entities)
        return entities

    # Node centralities
    centralities = nx.betweenness_centrality(graph, endpoints=True)
    for node_key in graph.nodes:
        graph.nodes[node_key][NODE_CENTRALITY] = centralities[node_key]

    # Communities
    communities = cast(Communities, louvain_communities(graph, seed=42))
    sorted_communities = sorted(communities, key=community_max_centrality, reverse=True)

    # Community colors
    community_count = len(sorted_communities)
    community_colors = list(color_gen(community_count))

    # Node community indices and colors
    for community_index, community in enumerate(sorted_communities):
        for node_key in community:
            node = graph.nodes[node_key]
            node[NODE_COMMUNITY_INDEX] = community_index
            node[NODE_COLOR] = community_colors[community_index]

    # Edge colors
    for node_key_i, node_key_j, edge_data in graph.edges(data=True):
        node_i = graph.nodes[node_key_i]
        node_j = graph.nodes[node_key_j]
        same_community = node_i[NODE_COMMUNITY_INDEX] == node_j[NODE_COMMUNITY_INDEX]
        edge_data[EDGE_COLOR] = (
            node_i[NODE_COLOR] if same_community else INTER_COMMUNITY_EDGE_COLOR
        )

    return nodes_sorted_by_community(sorted_communities)


def compute_node_positions(graph: nx.DiGraph, entities: Nodes) -> Positions:
    # Derive an undirected graph (preserving the entity order)
    undirected_graph = nx.Graph()
    for entity in entities:
        undirected_graph.add_node(entity)
    undirected_graph.add_edges_from(graph.edges())

    if len(entities) < 10:
        positions = nx.circular_layout(undirected_graph)
    else:
        positions = nx.kamada_kawai_layout(undirected_graph)
        positions = nx.arf_layout(undirected_graph, positions, seed=42)

    return positions


@dataclass
class GraphData:
    knowledge_graph: KnowledgeGraph
    remove_orphan_nodes: bool = True
    graph: nx.DiGraph = field(init=False)
    nodes: Nodes = field(init=False)
    positions: Positions = field(init=False)

    def __post_init__(self) -> None:
        self.graph = build_graph(self.knowledge_graph, self.remove_orphan_nodes)
        self.nodes = init_graph_data(self.graph)
        self.positions = compute_node_positions(self.graph, self.nodes)


print("✅ Network graph helpers defined")

✅ Network graph helpers defined

Let's test this:

graph_data = GraphData(knowledge_graph)

print(f"{graph_data.graph = !s}")
print(f"{graph_data.nodes = }")

graph_data.graph = DiGraph with 8 nodes and 16 edges
graph_data.nodes = ['2_sophie_jones', '3_william_smith', '7_acme_aerospace', '5_beau', '4_bill', '0_henry_jones_jr.', '1_henry_jones_sr.', '6_short_round']

🎨 Data visualization

The extracted data is much easier to understand when you can actually see it! We can use matplotlib to draw our network graphs. We'll size the nodes based on their centrality and color-code them by community. To make it even easier to digest, we'll generate an animated sequence highlighting each character's connections one by one…

Let's define our data visualization helpers: 🔽

import typing
from io import BytesIO

import matplotlib.pyplot as plt
from IPython import display
from matplotlib.axes import Axes
from matplotlib.backends.backend_agg import FigureCanvasAgg
from matplotlib.figure import Figure
from PIL import Image as PilImage


class AnimationFormat(StrEnum):
    # Matches PIL.Image.Image.format
    WEBP = auto()
    PNG = auto()
    GIF = auto()


FIGURE_DPI = 200
FIGURE_FACTOR = 1.0
ANIMATION_INTRO_DURATION = 2500
ANIMATION_FRAME_DURATION = 250
EDGE_STYLE = "arc3,rad=0.2"
NodeSizes = dict[Node, int]

if running_in_colab_env():
    ANIMATION_FORMAT = AnimationFormat.WEBP
else:
    ANIMATION_FORMAT = AnimationFormat.GIF


def init_figure(title: str, subtitle: str) -> tuple[Figure, Axes]:
    figsize = (16 * FIGURE_FACTOR, 9 * FIGURE_FACTOR)
    fig, ax = plt.subplots(figsize=figsize, dpi=FIGURE_DPI)
    ax.set_title(title, loc="left")
    ax.set_title(subtitle, loc="right")
    ax.axis("off")
    fig.tight_layout(pad=2)

    return fig, ax


def draw_nodes(graph: nx.Graph, positions: Positions, ax: Axes) -> NodeSizes:
    node_view = graph.nodes(data=True)
    node_sizes = {
        node: max(500, int(10000 * data[NODE_CENTRALITY])) for node, data in node_view
    }
    node_colors = [str(data[NODE_COLOR]) for _, data in node_view]
    border_width = 3.0

    nx.draw_networkx_nodes(
        graph,
        pos=positions,
        node_size=list(node_sizes.values()),
        node_color=node_colors,
        alpha=0.95,
        ax=ax,
        linewidths=border_width,
    )
    labels = {node: data.get("name", str(node)) for node, data in node_view}
    nx.draw_networkx_labels(graph, positions, labels=labels, ax=ax)

    if len(node_sizes) >= 10:
        # Extend axis limits (default rounding changes them too much otherwise)
        (x_min, x_max), (y_min, y_max) = ax.get_xlim(), ax.get_ylim()
        x_min, x_max = int(x_min - 1.0), int(x_max + 1.0)
        y_min, y_max = int(y_min - 1.0), int(y_max + 1.0)
        ax.set_xlim(x_min, x_max)
        ax.set_ylim(y_min, y_max)

    return node_sizes


def draw_edges(
    graph: nx.DiGraph,
    positions: Positions,
    node_sizes: NodeSizes,
    ax: Axes,
    *,
    focused_node: Node | None = None,
) -> None:
    # Draw specific out-edges or all edges
    if focused_node:
        out_edges = graph.edges([focused_node], data=True)
    else:
        out_edges = graph.edges(data=True)
    edge_colors = [data[EDGE_COLOR] for _, _, data in out_edges]

    edge_list = [(u, v) for u, v, _ in out_edges]
    ordered_sizes = [node_sizes[n] for n in graph.nodes()]
    nx.draw_networkx_edges(
        graph,
        positions,
        edge_list,
        edge_color=edge_colors,
        style=":",
        alpha=0.9,
        arrowstyle="-|>",
        arrowsize=20,
        ax=ax,
        node_size=ordered_sizes,
        connectionstyle=EDGE_STYLE,
    )

    edge_labels = {(u, v): data["link"] for u, v, data in out_edges}
    nx.draw_networkx_edge_labels(
        graph,
        positions,
        edge_labels,
        font_size=8,
        font_family="monospace",
        bbox=dict(ec="#FFF8", fc="#FFF8"),
        ax=ax,
        node_size=ordered_sizes,  # type: ignore
        connectionstyle=EDGE_STYLE,  # type: ignore
    )


def display_graph(title: str, subtitle: str, graph_data: GraphData) -> None:
    _, ax = init_figure(title, subtitle)

    node_graph = graph_data.graph
    edge_graph = graph_data.graph
    positions = graph_data.positions
    node_sizes = draw_nodes(node_graph, positions, ax)
    draw_edges(edge_graph, positions, node_sizes, ax)

    plt.show()


def yield_images(
    title: str,
    subtitle: str,
    graph_data: GraphData,
) -> Iterator[PilImage.Image]:
    fig, ax = init_figure(title, subtitle)
    canvas = FigureCanvasAgg(fig)

    positions = graph_data.positions
    node_graph = graph_data.graph
    node_sizes = draw_nodes(node_graph, positions, ax)
    edge_graph = graph_data.graph

    for focused_node in [None, *graph_data.nodes]:
        if focused_node is not None:
            draw_edges(edge_graph, positions, node_sizes, ax, focused_node=focused_node)
        canvas.draw()
        image_size = canvas.get_width_height()
        image_bytes = canvas.buffer_rgba()
        yield PilImage.frombytes("RGBA", image_size, image_bytes).convert("RGB")
    plt.close(fig)


def generate_animation(
    title: str,
    subtitle: str,
    graph_data: GraphData,
    format: AnimationFormat,
) -> BytesIO:
    frames = list(yield_images(title, subtitle, graph_data))
    assert len(frames) >= 1

    if format == AnimationFormat.GIF:
        # Dither all frames with the same palette to optimize the animation
        # The animation is cumulative, so most colors are in the last frame
        method = PilImage.Quantize.MEDIANCUT
        palettized = frames[-1].quantize(method=method)
        frames = [frame.quantize(method=method, palette=palettized) for frame in frames]

    # The animation will be played in a loop: start cycling with the most complete frame
    first_frame = frames[-1]
    next_frames = frames[:-1]
    durations = [ANIMATION_INTRO_DURATION]
    durations += [ANIMATION_FRAME_DURATION] * len(next_frames)
    params: dict[str, typing.Any] = dict(
        save_all=True,
        append_images=next_frames,
        duration=durations,
        loop=0,
    )
    match format:
        case AnimationFormat.GIF:
            params.update(optimize=False)
        case AnimationFormat.PNG:
            params.update(optimize=True)
        case AnimationFormat.WEBP:
            params.update(lossless=True)

    image_io = BytesIO()
    first_frame.save(image_io, str(format).upper(), **params)

    return image_io


def display_graph_animation(title: str, subtitle: str, graph_data: GraphData) -> None:
    image_io = generate_animation(title, subtitle, graph_data, ANIMATION_FORMAT)
    ipython_image = display.Image(data=image_io.getvalue())
    display.display(ipython_image)


def get_graph_title_and_subtitle(
    source: Source | str,
    model: Model | None = None,
    domain: str = "",
) -> tuple[str, str]:
    if isinstance(source, str):
        source_name = f'"{source.strip()[:20]}…"'
    else:
        source_name = source.name
    model_id = model.value if model else Model.DEFAULT.value
    model_id = model_id.removesuffix("-preview")
    separator = " • "
    title_parts = ["Knowledge Graph"]
    if domain:
        title_parts.append(domain)
    subtitle_parts = [source_name, model_id]
    return separator.join(title_parts), separator.join(subtitle_parts)


def display_knowledge_graph(
    knowledge_graph: KnowledgeGraph,
    source: Source | str,
    model: Model | None = None,
    animated: bool = False,
    domain: str = "",
    remove_orphan_nodes: bool = True,
) -> None:
    if not knowledge_graph.entities:
        return
    title, subtitle = get_graph_title_and_subtitle(source, model, domain)
    graph_data = GraphData(knowledge_graph, remove_orphan_nodes)

    if animated:
        display_graph_animation(title, subtitle, graph_data)
    else:
        display_graph(title, subtitle, graph_data)


def extract_knowledge_graph(
    data_schema: str,
    instructions: str,
    source: Source | str,
    model: Model | None = None,
    *,
    domain: str = "",
    animated: bool = False,
) -> None:
    if isinstance(source, Source):
        display_input_data_caption(source)
    knowledge_graph = generate_knowledge_graph(
        data_schema,
        instructions,
        source,
        model=model,
    )
    display_knowledge_graph(knowledge_graph, source, model, animated, domain)


print("✅ Data visualization helpers defined")

✅ Data visualization helpers defined

Let's test this:

display_knowledge_graph(knowledge_graph, text)

💡 We can now quickly visualize and understand our knowledge graphs. This helps us iterate faster when refining prompts.

ℹ️ While this simple approach is great for a quick overview, you might want to swap it out for more specialized libraries if you want to explore the graph interactively.

✅ Challenge completed

Let's define a book analysis function:

def analyze_book(
    source: Source,
    model: Model | None = None,
    animated: bool = False,
) -> None:
    extract_knowledge_graph(
        BOOK_ANALYSIS_DATA_SCHEMA,
        BOOK_ANALYSIS_INSTRUCTIONS,
        source,
        model,
        domain="Character Connections",
        animated=animated,
    )

🧪 First, let's build a knowledge graph based on Zola's Thérèse Raquin:

analyze_book(Classic.fr_zola_thérèse_raquin)

Input data (fr_zola_thérèse_raquin)

----------------------- Request / gemini-3.1-flash-lite ------------------------
Input tokens    :   119,518
Cached tokens   :   118,753
Output tokens   :       357
----------------------------- Knowledge Graph Info -----------------------------
Entities      :  11
Relationships :  28
--------------------------------------------------------------------------------

💡 We can extract and understand the book's narrative in seconds: The love triangle between Thérèse, Camille, and Laurent clearly stands out. Despite being over 200 pages long (and 100k+ tokens), this novel is incredibly minimalistic, revolving around a limited set of characters, which is reflected in the network graph. Adding locations to the extracted entities would also reveal the claustrophobic atmosphere of the book in the resulting knowledge graph.

ℹ️ We used the original French version with English instructions, which works seamlessly. If you translate the instructions, you can also generate dynamic relationship links in different languages (see the 100+ supported languages).

🧪 Let's see how the model handles the interconnected cast of Hugo's Les Misérables:

analyze_book(
    Classic.en_hugo_les_misérables,
    model=Model.GEMINI_3_5_FLASH,
)

Input data (en_hugo_les_misérables)

-------------------------- Request / gemini-3.5-flash --------------------------
Input tokens    :   803,328
Output tokens   :     2,908
----------------------------- Knowledge Graph Info -----------------------------
Entities      :  65
Relationships : 225
--------------------------------------------------------------------------------

💡 We quickly get the gist of the novel.

ℹ️ Donald Knuth famously used Les Misérables as an example back in 1994 (see the Stanford GraphBase). It's been consistently used in Natural Language Processing (NLP) because of its massive scale, linguistic complexity, and multiple high-quality translations. At 800k+ tokens, this book is still a true stress test. Note that we used Gemini 3.5 Flash (instead of 3.1 Flash-Lite). Larger models can infer more and perform deeper consolidation.

🧪 Now, let's process just the first volume of Le Comte de Monte-Cristo (in French):

analyze_book(Classic.fr_dumas_comte_de_monte_cristo_1)

Input data (fr_dumas_comte_de_monte_cristo_1)

----------------------- Request / gemini-3.1-flash-lite ------------------------
Input tokens    :   215,582
Output tokens   :       713
----------------------------- Knowledge Graph Info -----------------------------
Entities      :  36
Relationships :  36
--------------------------------------------------------------------------------

💡 The graph shows the initial setup of Edmond Dantès' world, his friends, and his betrayers. The love triangle between Edmond, Mercédès, and Fernand also comes to light.

ℹ️ Note that the antagonist is referred to only as Fernand, which is expected. We learn his full name, Fernand Mondego, only in volume three. In the prompt, we asked the model to "extract" entities and determine names "from the input data" to avoid generating memorized knowledge. To detect regressions when updating the prompt, we can use this as a unit test to ensure that our response actually provides extracted data and not memorized information.

🧪 And finally, let's analyze the entire saga of Le Comte de Monte-Cristo, all four volumes at once (800k+ tokens):

analyze_book(
    Collection.fr_dumas_comte_de_monte_cristo,
    model=Model.GEMINI_3_5_FLASH,
    animated=True,
)

Input data (fr_dumas_comte_de_monte_cristo_1, fr_dumas_comte_de_monte_cristo_2, fr_dumas_comte_de_monte_cristo_3, fr_dumas_comte_de_monte_cristo_4)

-------------------------- Request / gemini-3.5-flash --------------------------
Input tokens    :   840,385
Cached tokens   :   835,551
Output tokens   :     2,659
----------------------------- Knowledge Graph Info -----------------------------
Entities      :  50
Relationships : 212
--------------------------------------------------------------------------------

💡 The multiple aliases of Edmond Dantès (and other characters) are nicely extracted. The complex plot of this book is built on characters juggling multiple identities in a psychological chess match.

🎉 This is an example of how we can complete the challenge in a single request, using the fewest tokens possible and zero thinking tokens, while providing multiple levels of consolidation.

ℹ️ Note that this is a proof of concept. For an exhaustive extraction in a fully professional solution, we would probably set up a multi-step workflow and consider the following:

As a preliminary step, cache the input data (cached tokens get a 90% discount; learn more about context caching).
In the first request, based on the cached content, focus on extracting the entities (possibly extending the entity labels with location, date…), and then save them to a database.
In the second request, based on the cached content and the entity table, focus on extracting the stated relationships using additional fields (such as evidence or excerpt to serve as direct proof, or chapter or page for source attribution). In this step, we could also dig deeper and differentiate literal/figurative relationships (e.g., biological vs. figurative parents).
In subsequent requests, focus on additional extractions (e.g., interactions between entities) or specific consolidations (e.g., adding inverse relationships, classifying the relationships or interactions into categories).

🔭 Generalization

With our current setup, generalizing to other types of content is as simple as adapting our data schema and instructions.

For example, legal agreements are among the densest types of documents. They're usually made up of many articles and clauses, with every sentence carrying specific legal weight, outlining obligations, or providing definitions. But what kind of knowledge graph do we want to build from a legal agreement?

🧪 Let's test a minimalistic, open-ended prompt:

source = Document.en_pharma_dev_agreement
model = Model.GEMINI_3_5_FLASH

AGREEMENT_OPEN_DATA_SCHEMA = """
Entity:
- `id`: Unique integer identifier (0, 1, 2…).
- `name`: Name of the entity.
- `label`: Entity type.

Relationship:
- `source_id`: `id` of the subject entity.
- `link`: `snake_case` predicate.
- `target_id`: `id` of the object entity.
"""

AGREEMENT_OPEN_INSTRUCTIONS = """
Perform a comprehensive, highly-granular entity/relationship extraction.
"""

extract_knowledge_graph(
    AGREEMENT_OPEN_DATA_SCHEMA,
    AGREEMENT_OPEN_INSTRUCTIONS,
    source,
    model,
    domain="Agreement High-Level Extraction",
)

Input data (en_pharma_dev_agreement)

-------------------------- Request / gemini-3.5-flash --------------------------
Input tokens    :    51,276
Output tokens   :       221
----------------------------- Knowledge Graph Info -----------------------------
Entities      :  10
Relationships :   9
--------------------------------------------------------------------------------

💡 Remarks

Notice how we just passed a PDF directly! Gemini processes the document natively, automatically extracting text and images while performing OCR along the way.
This minimalistic prompt offers the greatest degree of freedom, as the model can't guess exactly what data you want or how you plan to use it. As a result, only high-level entities and relationships are extracted, providing a nice summary of the agreement.
LLMs are trained to synthesize information. With such highly open-ended instructions, the default behavior helps avoid generating unnecessary tokens.

🧪 Then, let's try semi-open instructions focusing on legal obligations:

class AgreementEntityLabel(StrEnum):
    PARTY = auto()
    PERSON = auto()
    ROLE = auto()
    LOCATION = auto()
    JURISDICTION = auto()
    FINANCIAL_AMOUNT = auto()
    DATE = auto()
    EVENT = auto()
    ASSET = auto()
    PRODUCT = auto()
    INTELLECTUAL_PROPERTY = auto()
    OBLIGATION_TYPE = auto()


AGREEMENT_SEMI_OPEN_DATA_SCHEMA = f"""
Entity:
- `id`: Unique integer identifier (0, 1, 2…).
- `name`: Name of the entity.
- `label`: {pipe_delimited_union(AgreementEntityLabel)}.

Relationship: Obligation, right, or transfer from a source entity to a target entity.
- `source_id`: The `id` of the subject entity.
- `link`: Specific `snake_case` predicate.
- `target_id`: The `id` of the object entity.
"""

AGREEMENT_SEMI_OPEN_INSTRUCTIONS = """
- Analyze the input data for every covenant (e.g., "shall", "will", "must", "is obligated to") and perform an exhaustive extraction.
- Make sure to deconstruct complex obligations: For complex clauses (e.g., "A shall pay B $X Million within Y days of the Effective Date"), extract:
  - The primary obligation: `[A] is_obligated_to_pay [B]`
  - The value: `[A's obligation] subject_to [$X Million]`
  - The trigger: `[A's obligation] triggered_by [Effective Date]`
"""

extract_knowledge_graph(
    AGREEMENT_SEMI_OPEN_DATA_SCHEMA,
    AGREEMENT_SEMI_OPEN_INSTRUCTIONS,
    source,
    model,
    domain="Agreement Obligations",
)

Input data (en_pharma_dev_agreement)

-------------------------- Request / gemini-3.5-flash --------------------------
Input tokens    :    51,466
Output tokens   :     2,002
----------------------------- Knowledge Graph Info -----------------------------
Entities      :  85
Relationships :  93
--------------------------------------------------------------------------------

💡 Remarks

This semi-open prompt lists the types of entities to extract, which naturally increases the number of extracted entities and their specificity.
The resulting graph is denser and reflects the legal complexity of the document, rather than just giving us a high-level summary.
The extracted relationships are still rather high-level due to how open-ended this part of our prompt is. For such specific extractions, it's possible to extend the entity and relationship data schemas with additional fields, or even to request specific tabular outputs.

What if we don't care about legal obligations, but rather the document's architecture? Let's shift our focus to the structure itself and extract how sections, clauses, and defined terms are hierarchically organized…

🧪 And now, let's test closed instructions focusing on the document structure:

class AgreementStructureEntityLabel(StrEnum):
    DEFINED_TERM = auto()
    DOCUMENT_SECTION = auto()
    DOCUMENT = auto()


class AgreementStructureRelationshipType(StrEnum):
    DEFINED_IN = auto()
    CONTAINS = auto()


AGREEMENT_STRUCTURAL_DATA_SCHEMA = f"""
Entity:
- `id`: Unique integer identifier (0, 1, 2…).
- `name`: Name of the entity.
- `label`: {pipe_delimited_union(AgreementStructureEntityLabel)}.

Relationship: Connection from a source entity to a target entity.
- `source_id`: The `id` of the subject entity.
- `link`: {pipe_delimited_union(AgreementStructureRelationshipType)}.
- `target_id`: The `id` of the object entity.
"""

AGREEMENT_STRUCTURAL_OPEN_INSTRUCTIONS = """
- Extract every distinct entity that matches an allowed `label`.
- Extract every distinct relationship representing a structural connection (hierarchical organization) between these entities:
  - You must be comprehensive and highly granular. If multiple distinct relationships exist between the same pair of entities, create a separate entry for each.
"""

extract_knowledge_graph(
    AGREEMENT_STRUCTURAL_DATA_SCHEMA,
    AGREEMENT_STRUCTURAL_OPEN_INSTRUCTIONS,
    source,
    model,
    domain="Agreement Structure",
)

Input data (en_pharma_dev_agreement)

-------------------------- Request / gemini-3.5-flash --------------------------
Input tokens    :    51,351
Output tokens   :     9,108
----------------------------- Knowledge Graph Info -----------------------------
Entities      : 314
Relationships : 513
--------------------------------------------------------------------------------

💡 If you extract hundreds of entities from a massive document, your graph will quickly turn into an unreadable hairball. For larger datasets, you'll want to export your nodes and edges to a dedicated graph database, which typically comes with its own visualization and exploration tools.

🏁 Conclusion

We successfully extracted data and built knowledge graphs from documents by following these steps:

Prototyping with open prompts to develop an intuition for the model's natural strengths
Crafting increasingly specific prompts using a tabular-extraction strategy
Structuring our inputs to move towards production-ready and generalizable code
Structuring and optimizing our outputs for faster and cheaper generation
Adding data visualization for easier interpretation of responses and smoother iterations
Conducting more tests, iterating, and enriching the extracted data

These principles apply to many other data-extraction domains and will allow you to solve your own complex problems. Have fun and happy solving!

➕ More!

Try it yourself: Use the companion notebook (or run the notebook on Colab) to reproduce all results in this article.
Get inspired: Explore typical use cases in the Agent Platform Prompt Gallery.
Stay updated: Follow the Agent Platform Release Notes.
Follow me: Connect with me (@PicardParis) on LinkedIn or Twitter / X for more cloud, applied AI, and Python explorations…

My daughter asked if developers used to write code by hand, but it was the follow-up question that surprised me.

Greg Baugues — Thu, 11 Jun 2026 14:06:23 +0000

My daughter, who's 11, has been doing some vibe coding.

The other day she looked over my shoulder at my IDE and asked: "Dad, was there ever a time when a developer had to write each one of those characters by hand?"

"Yes. Twelve months ago."

"But... how did you know what to write?"

Which was a weird, because up until recently, "knowing what to write," was sort of the whole ballgame.

I recently picked up woodworking as a hobby, and have found a lot of similarities to coding. For most of my life, developers have considered themselves craftsman. We'd hand place each character.

Then, about three years ago, the coding gods dropped power tools on us, and now we're all trying to figure out what this means for the profession. Do we still need handsaws? Will the circular saw build the bench for us? Does this mean we should be able to make 10x more birdhouses in the same amount of time?

It is an exciting, and scary time to be a professional software developer. But there's also never been a better time to learn how to build software.

If this spoke to you, check out the full video here:

G4 Fractional VMs are now available on Google Cloud!

Maciej Strzelczyk — Wed, 10 Jun 2026 15:38:18 +0000

In 2025 Google Cloud added G4, powered by NVIDIA's RTX PRO 6000 Blackwell Server Edition GPUs to their offering, allowing them to offer hardware not only for AI applications, but also for other applications, such as rendering, simulations or gaming. A single G4 instance with one accelerator (g4-standard-48) comes equipped with 48 CPU cores, 180 gigabytes of RAM and 96 gigabytes of GPU memory. This is a lot of resources for a single cloud workstation, that only the most demanding workstreams would utilize. Most professionals who require a graphics accelerator to do their job, don't really need this much compute power for day to day tasks. It wasn't financially reasonable to pay for a G4 instance, when you weren't utilizing all the resources you paid for. If only there were smaller machine types… If only you could share that one very powerful GPU between multiple virtual machines…

Introducing fractional VMs!

During Google Cloud Next 2026, Google announced GA for fractional G4 VMs and was the first provider to bring vGPU functionality to RTX PRO 6000 accelerators. vGPU stands for virtual graphical processing unit. Just like VMs (virtual machines) are a way to split one physical computer into smaller, independent systems, vGPU allows for a single physical accelerator to be split into 2, 4 or 8 virtual accelerators! The new fractional machine types (g4-standard-24, g4-standard-12, g4-standard-6) now allow you to perfectly match the compute capabilities to your needs!

Who is it for?

The existence of those new machine types makes it much more cost-efficient to move many GPU-dependent tasks to the cloud. Replacing physical workstations in offices with cloud infrastructure is not a new thing, but till now, Google Cloud didn't offer a good platform for those who needed workstations to process images, post-process videos, simulate physics or render 3D graphics. Those users now can get exactly the hardware they need, allowing their companies to move away from maintaining physical workstations and replacing them with virtual ones.

How to get it?

The new machine types are available for new Compute Engine instances and GKE nodes. If you are creating a Compute Instance, remember to install the correct drivers, so that you can utilize your virtual accelerator without any issues.

Creation of a fractional VM is no different from a regular VM. For example, to create a Compute Engine instance, you can use a gcloud command:

gcloud compute instances create $VM_NAME \  
     --machine-type=g4-standard-6 \  
     --zone=us-central1-b \  
     --boot-disk-size=$DISK_SIZE \  
     --maintenance-policy=TERMINATE \  
     --restart-on-failure

Or you can use Cloud Console:

Similarly, for GKE you can specify the new machine types (g4-standard-6, g4-standard-12 and g4-standard-24) to create a new Node Pool:

gcloud container node-pools create POOL_NAME \  
    --cluster CLUSTER_NAME \  
    --location=CONTROL_PLANE_LOCATION \  
    --machine-type=g4-standard-24

Or in Cloud Console:

The vGPU support is currently rolled out only to one zone: us-central1-b. However, it's only a matter of time before it's available in more regions. Keep an eye on the documentation page to find which zones support which GPUs. To stay up to date, follow the Google Cloud blog and subscribe to Google Cloud YouTube channel to not miss any updates!