DEV Community: Connie Leung

Extending a Video with Angular, Veo 3.1 Lite, Firebase Cloud Functions, and Firebase Cloud Storage [GDE]

Connie Leung — Sat, 18 Apr 2026 06:50:21 +0000

Google released the Veo 3.1 Lite model for AI video generation in the Gemini API, Gemini in Vertex AI, and Gemini AI Studio. This model solves a common developer pain point: generating high-quality videos quickly and at a lower cost.

In this blog post, I migrate my application to use the Veo 3.1 Lite model and implement a new Firebase Cloud Function to extend a video using the GenAI TypeScript SDK.

The application supports image-to-video generation, video interpolation using the first and last frames, and extending Veo videos.

Prerequisites

The technical stack of the project:

Angular 21, the latest version as of April 2026.
Node LTS, the LTS version as of April 2026.
Firebase Remote Config: To manage dynamic parameters.
Firebase Cloud Functions: To be called by the frontend to generate a video, interpolate a video between two images, or extend a Veo video.
Firebase Cloud Storage: To host the generated video files in the default Firebase Storage bucket.
Firebase Local Emulator Suite: To test the functions locally at http://localhost:5001.
Gemini in Vertex AI: Use Gemini in Vertex AI to generate videos and store them in Firebase Cloud Storage.

The public Google AI Studio API is restricted in my region (Hong Kong). However, Vertex AI (Google Cloud) offers enterprise access that works reliably here, so I chose Vertex AI for this demo.

npm i -g firebase-tools

Install firebase-tools globally using npm.

firebase logout

firebase login

Log out of Firebase and log in again to perform proper Firebase authentication.

firebase init

Execute firebase init and follow the prompts to set up Firebase Cloud Functions, the Firebase Local Emulator Suite, Firebase Cloud Storage, and Firebase Remote Config.

If you have an existing project or multiple projects, you can specify the project ID on the command line.

firebase init --project <PROJECT_ID>

In both cases, the Firebase CLI automatically installs the firebase-admin and firebase-functions dependencies.

After completing the setup steps, the Firebase tools generate the functions emulator, functions, a storage rules file, remote config templates, and configuration files such as .firebaserc and firebase.json.

Angular dependency

npm i firebase

The Angular application requires the firebase dependency to initialize a Firebase app, load remote config, and invoke the Firebase Cloud Functions to generate videos.

Firebase dependencies

npm i @cfworker/json-schema @google/genai @modelcontextprotocol/sdk

Install the above dependencies to access Gemini in Vertex AI. @google/genai depends on @cfworker/json-schema and @modelcontextprotocol/sdk. Without these, the Cloud Functions cannot start.

With our project configured, let's look at how the frontend and backend communicate.

Architecture

The frontend application is built with Angular. It relies on Firebase AI Logic to generate images using the Gemini 3.1 Flash Image Preview model. Then, the text prompt and the image are submitted to a Firebase Cloud Function to generate a video, store it in a Firebase Cloud Storage bucket, and return the GCS URI, MIME type, and HTTP URL to the client.

When the client extends the Veo video, it provides the prompt, the GCS URI, and the MIME type to another Firebase Cloud Function to generate an extended video, store it in the Firebase Cloud Storage bucket, and return the GCS URI, MIME type, and HTTP URL to the client.

Similarly, the HTML video player element plays the HTTP URL in the client application.

Difference between Veo 3.1 Lite in Gemini AI Studio and Gemini in Vertex AI

Difference	Gemini AI Studio	Gemini in Vertex AI
Model Name	veo-3.1-lite-generate-preview	veo-3.1-lite-generate-001
Number of extensions	Extend by 7 seconds and up to 20 times	Extend by 7 seconds and up to 4 times
Video length	Extend 141 seconds and the total length is 148 seconds	Extend 28 seconds and the total length is 36 seconds

What Veo 3.1 Does Not Support in Gemini Vertex AI

Unlike Veo 3.1 and Veo 3.1 Fast (which only lack support for Reference style images), Veo 3.1 Lite does not support either Reference asset images or Reference style images.
Supported input resolution does not support 4K.
Supported output resolution does not support 4K.

This limitation does not impact our application, as the demo focuses exclusively on video extension and the resolution of generated videos is hardcoded to 720p.

Firebase Integration

1. Configure Environment Variables

I define the environment variables in the Firebase project. This ensures the functions know the regions for storage, function hosting, and the Veo model to use for video generation.

.env.example

GOOGLE_CLOUD_LOCATION="us-central1"
GEMINI_VIDEO_MODEL_NAME="veo-3.1-lite-generate-001"
IS_VEO31_USED="true"
POLLING_PERIOD_MS="10000"
GOOGLE_FUNCTION_LOCATION="us-central1"
WHITELIST="http://localhost:4200"
REFERER="http://localhost:4200/"

Variable	Description
GOOGLE_CLOUD_LOCATION	The location of the bucket. I chose `us-central1` because the bucket is always free in this region.
GEMINI_VIDEO_MODEL_NAME	The name of the Gemini video model.
IS_VEO31_USED	Whether Veo 3.1 is used. If false, it falls back to generating a video instead of interpolation.
POLLING_PERIOD_MS	The polling period of the video operation in milliseconds.
GOOGLE_FUNCTION_LOCATION	The region of the Cloud Functions. I chose `us-central1` so the functions and the bucket are in the same region.
WHITELIST	Requests must come from http://localhost:4200.
REFERER	Requests originate from http://localhost:4200/.

2. Validating Environment Variables

Before the Cloud Function proceeds with any AI calls, it is critical to ensure that all necessary environment variables are present. I implemented a VIDEO_CONFIG IIFE (Immediately Invoked Function Expression) to run once to validate environment variables such as the polling period, whether Veo 3.1 is used, the Veo model name, the project ID, and the location.

import logger from "firebase-functions/logger";

export function validate(value: string | undefined, fieldName: string, missingKeys: string[]) {
  const err = `${fieldName} is missing.`;
  if (!value) {
    logger.error(err);
    missingKeys.push(fieldName);
    return "";
  }

  return value;
}

export const VIDEO_CONFIG = (() => {
  process.loadEnvFile();

  const env = process.env;
  const isVeo31Used = (env.IS_VEO31_USED || "false") === "true";
  const pollingPeriod = Number(env.POLLING_PERIOD_MS || "10000");

  const missingKeys: string[] = [];
  const model = validate(env.GEMINI_VIDEO_MODEL_NAME, "Gemini Video Model Name", missingKeys);
  const project = validate(env.GCLOUD_PROJECT, "Project ID", missingKeys);
  const location = validate(env.GOOGLE_CLOUD_LOCATION, "Google Cloud Location", missingKeys);

  if (missingKeys.length > 0) {
    throw new Error(`Missing environment variables: ${missingKeys.join(", ")}`);
  }

  return {
    genAIOptions: {
      project,
      location,
      vertexai: true,
    },
    aiVideoOptions: {
      model,
      storageBucket: `${project}.firebasestorage.app`,
      isVeo31Used,
      pollingPeriod,
    },
  };
})();

I am using Node 24 as of April 2026. Since Node 20, we can use the built-in process.loadEnvFile function that loads environment variables from the .env file.

If you are using a Node version that does not support process.loadEnvfile, the alternative is to install dotenv to load the environment variables.

npm i dotenv

import dotenv from "dotenv";

dotenv.config();

Firebase provides the GCLOUD_PROJECT variable, so it is not defined in the .env file.

When the missingKeys array is not empty, VIDEO_CONFIG throws an error that lists all the missing variable names. If the validation is successful, the genAIOptions and aiVideoOptions are returned. The genAIOptions is used to initialize the GoogleGenAI and aiVideoOptions contains parameters for extending a Veo video.

3. Extending a Video and Storing in Firebase Storage

The extendVideo Cloud Function passes the payload to the extendVideoFunction function.

All Cloud Functions enforce App Check, CORS, and a timeout period of 600 seconds. If WHITELIST is unspecified, CORS defaults to true. It is acceptable in a demo, but it is safer to default to false or a specific domain in production.

const cors = process.env.WHITELIST ? process.env.WHITELIST.split(",").map((origin) => origin.trim()): true;
const options = {
  cors,
  enforceAppCheck: true,
  timeoutSeconds: 600,
};

export const extendVideo = onCall( options,
  ({ data }) => extendVideoFunction(data)
);

extendVideoFunction delegates to the extendVideoByPolling function to construct the video arguments and poll the video operation until it finishes. When the function completes successfully, it returns the GCS URI and the MIME type. In the error case, the function throws an error.

import { GoogleGenAI } from "@google/genai";
import { ExtendVideoRequest } from "./types/video.type";
import { extendVideoByPolling, VIDEO_CONFIG } from "./video.util";

export async function extendVideoFunction(data: ExtendVideoRequest) {
  const { genAIOptions, aiVideoOptions } = VIDEO_CONFIG;

  try {
    if (!aiVideoOptions.isVeo31Used) {
      throw new Error("Video extension is only supported for Veo 3.1 model");
    }

    const ai = new GoogleGenAI(genAIOptions);
    return await extendVideoByPolling({ ai, ...aiVideoOptions }, {
      prompt: data.prompt,
      video: data.video,
      config: data.config,
    });
  } catch (error) {
    console.error("Error generating video:", error);
    throw new Error("Error generating video");
  }
}

import { GenerateVideosConfig, GoogleGenAI, Video } from "@google/genai";

export type AIVideoBucket = {
  ai: GoogleGenAI;
  model: string;
  storageBucket: string;
  isVeo31Used: boolean;
  pollingPeriod: number;
}

export type ExtendVideoRequest = {
  prompt: string;
  video: Video;
  config?: GenerateVideosConfig;
}

export async function extendVideoByPolling(
  aiVideo: AIVideoBucket,
  request: ExtendVideoRequest,
) {
  return processVideoPolling(aiVideo, {
    prompt: request.prompt,
    config: request.config,
    video: request.video,
  });
}

async function processVideoPolling(
  { ai, model, storageBucket, pollingPeriod }: AIVideoBucket,
  mediaParams: VideoMediaParams
) {
  const genVideosParams: GenerateVideosParameters = {
    model,
    ...mediaParams,
    config: {
      ...mediaParams.config,
      numberOfVideos: 1,
      outputGcsUri: `gs://${storageBucket}`,
    },
  };

  return getVideoUri(ai, genVideosParams, pollingPeriod);
}

The extendVideoByPolling function invokes the processVideoPolling function to poll the video operation until it completes.

The processVideoPolling function constructs the video parameters and invokes the getVideoUri function to return the GCS URI and the MIME type. Most importantly, the config property specifies that the number of generated videos is one, and the outputGcsUri keeps the generated video in the storage bucket.

4. Asynchronous Polling

Video extension is a long-running task. Because Vertex AI processes them asynchronously, the functions must poll the operation status in a while loop until the done flag is true.

The Gemini API cannot see the Cloud Storage for Firebase Local Emulator, so it requires a real output GCS URI, which is gs://${storageBucket}.

When the done flag is true, the operation ends and one of three outcomes occurs:

Outcome 1: The error is true, and the video failed to generate. Therefore, the function throws an error.
Outcome 2: The video is successfully stored in the GCS bucket. The function returns the GCS URI and the MIME type to the client application.
Outcome 3: Neither happens. There is no error and no GCS URI, so the function returns an unknown error.

import { GenerateVideosParameters, GoogleGenAI } from "@google/genai";

async function getVideoUri(
  ai: GoogleGenAI,
  genVideosParams: GenerateVideosParameters,
  pollingPeriod: number,
): Promise<{ uri: string, mimeType: string }> {
  let operation = await ai.models.generateVideos(genVideosParams);

  while (!operation.done) {
    await new Promise((resolve) => setTimeout(resolve, pollingPeriod));
    operation = await ai.operations.getVideosOperation({ operation });
  }

  if (operation.error) {
    const strError = `Video generation failed: ${operation.error.message}`;
    console.error(strError);
    throw new Error(strError);
  }

  const video = operation.response?.generatedVideos?.[0]?.video || {};
  const { uri, mimeType } = video;
  if (uri && mimeType) {
    console.log("video uri", uri, mimeType);
    return { uri, mimeType };
  }

  const strError = "Video generation finished but no uri was provided.";
  console.error(strError);
  throw new Error(strError);
}

The environment variable, POLLING_PERIOD_MS, sets the polling period to 10 seconds. If the wait time is too long, the polling period can be decreased. If the polling cost is expensive or too frequent, the polling period can be increased.

Note: For demo purposes, polling is a decent solution to handle asynchronous video generation. However, it is expensive and creates unnecessary load and latency. For production usage, you may consider push notifications such as WebSockets and Server-Sent Events.

5. Firebase App Configuration and reCAPTCHA Site Key

getFirebaseConfig is a Firebase Cloud Function that returns both the Firebase app configuration and the reCAPTCHA site key.

import logger from "firebase-functions/logger";
import { validate } from "./validate";

function validateFirebaseConfigFields(env: NodeJS.ProcessEnv) {
  const missingKeys: string[] = [];
  const apiKey = validate(env.APP_API_KEY, "API Key", missingKeys);
  const appId = validate(env.APP_ID, "App Id", missingKeys);
  const messagingSenderId = validate(env.APP_MESSAGING_SENDER_ID, "Messaging Sender ID", missingKeys);
  const recaptchaSiteKey = validate(env.RECAPTCHA_ENTERPRISE_SITE_KEY, "Recaptcha site key", missingKeys);
  const projectId = validate(env.GCLOUD_PROJECT, "Project ID", missingKeys);

  if (missingKeys.length > 0) {
    throw new Error(`Missing environment variables: ${missingKeys.join(", ")}`);
  }

  return {
    app: {
      apiKey,
      appId,
      projectId,
      messagingSenderId,
      authDomain: `${projectId}.firebaseapp.com`,
      storageBucket: `${projectId}.firebasestorage.app`,
    },
    recaptchaSiteKey,
  };
}

export const getFirebaseConfigFunction = () => {
  logger.info("getFirebaseConfig called");

  process.loadEnvFile();

  const variables = validateFirebaseConfigFields(process.env);
  if (!variables) {
    return undefined;
  }

  return variables;
};

The Angular application receives the Firebase app configuration and reCAPTCHA site key from the Cloud Function to initialize Firebase AI Logic and protect resources from unauthorized access and abuse.

6. Local Development with Emulators

For local development, I used the Firebase Local Emulator Suite. In the bootstrapFirebase process, the app calls connectFunctionsEmulator to link to the Cloud Functions running at http://localhost:5001.

The port number defaulted to 5001 when firebase init was executed.

Note: While the Cloud Function runs locally (at zero cost), the Storage emulator is not used. This is because the Gemini API requires an actual accessible GCS bucket to store the generated videos.

export function connectEmulators({ remoteConfig, functions }: FirebaseObjects) {
  const useEmulators = getValue(remoteConfig, 'useEmulators').asBoolean();
  if (useEmulators) {
    console.log('Connecting to emulators...');
    const host = getValue(remoteConfig, 'functionEmulatorHost').asString();
    const port = getValue(remoteConfig, 'functionEmulatorPort').asNumber();
    console.log('functionEmulator', `${host}:${port}`);
    connectFunctionsEmulator(functions, host, port);
  }
}

loadFirebaseConfig is a helper function that makes request to the Cloud function to obtain the Firebase App configuration and the reCAPTCHA site key.

{
  "getFirebaseConfigUrl": "http://127.0.0.1:5001/vertexai-firebase-6a64f/us-central1/getFirebaseConfig"
}

import { HttpClient } from '@angular/common/http';
import { inject } from '@angular/core';
import { initializeAppCheck, ReCaptchaEnterpriseProvider } from 'firebase/app-check';
import { catchError, lastValueFrom, throwError } from 'rxjs';
import config from '../../public/config.json';
import { ConfigService } from './ai/services/config.service';
import { FirebaseConfigResponse } from './ai/types/firebase-config.type';
import { connectEmulators, initFirebaseApp } from './firebase.util';

async function loadFirebaseConfig() {
  const httpService = inject(HttpClient);
  const firebaseConfig$ =
    httpService.get<FirebaseConfigResponse>(`${config.getFirebaseConfigUrl}`)
      .pipe(catchError((e) => throwError(() => e)));
  return lastValueFrom(firebaseConfig$);
}

export async function bootstrapFirebase() {
    try {
      const configService = inject(ConfigService);
      const firebaseConfig = await loadFirebaseConfig();
      const { app, recaptchaSiteKey } = firebaseConfig;
      const firebaseObjects = await initFirebaseApp(app);
      const { firebaseApp } = firebaseObjects;

      initializeAppCheck(firebaseApp, {
        provider: new ReCaptchaEnterpriseProvider(recaptchaSiteKey),
        isTokenAutoRefreshEnabled: true,
      });

      connectEmulators(firebaseObjects);
      configService.loadConfig(firebaseObjects);
    } catch (err) {
      console.error(err);
    }
}

The AppConfig remains unchanged.

import { ApplicationConfig, provideAppInitializer } from '@angular/core';
import { bootstrapFirebase } from './app.bootstrap';

export const appConfig: ApplicationConfig = {
  providers: [
    provideAppInitializer(async () => bootstrapFirebase()),
  ]
};

7. Firebase Remote Configuration

New variables are introduced for extending a Veo video in Firebase Remote Config. For the Gemini API, a video can be extended as many as 20 times. For Gemini in Vertex AI, a video can be extended as many as 4 times because the video length exceeds 30 seconds. Therefore, this is externalized into the maxVideoExtendAllowed variable.

Variable	Description	Value
maxVideoExtendAllowed	Maximum number of extensions allowed	4

8. Video Player Component in Angular

The Angular frontend triggers the process using httpsCallable. Once the function returns the Cloud Storage path and the MIME type, the app fetches the download URL for playback.

The ConfigService stores the Firebase app, Remote Config, and functions to be used throughout the application.

import { FirebaseApp } from 'firebase/app';
import { Functions } from 'firebase/functions';
import { RemoteConfig } from 'firebase/remote-config';

export type FirebaseObjects = {
  firebaseApp: FirebaseApp;
  remoteConfig: RemoteConfig;
  functions: Functions;
}

import { Injectable } from '@angular/core';
import { FirebaseObjects } from '../types/firebase-objects';

@Injectable({
  providedIn: 'root'
})
export class ConfigService  {
    firebaseObjects: FirebaseObjects | undefined = undefined;

    loadConfig(firebaseObjects: FirebaseObjects) {
      this.firebaseObjects = firebaseObjects;
    }
}

The retrieveVideoUri method calls the Cloud Function directly to retrieve the GCS URI and the MIME type.

The downloadVideoUriAndUrl method resolves the URI to an HTTP URL so that an HTML video player can play it immediately.

export type VideoGenerationResponse = {
  uri: string;
  url: string;
  mimeType: string;
}

export type DownloadVideoResponse = {
  uri: string;
  mimeType: string;
}

import { inject, Injectable } from '@angular/core';
import { httpsCallable } from 'firebase/functions';
import { getDownloadURL, getStorage, ref } from 'firebase/storage';
import { GenerateVideoRequest } from '../types/video.type';
import { ConfigService } from './config.service';

@Injectable({
  providedIn: 'root'
})
export class VeoService {
  private readonly storage = getStorage();
  private readonly configService = inject(ConfigService);

  private async retrieveVideoUri<T = GenerateVideoRequest>(request: T, methodName: string): Promise<DownloadVideoResponse> {
    try {
      const { functions } = this.configService.firebaseObjects || {};
      if (!functions) {
        throw new Error('Functions does not exist.');
      }

      const downloadGcsUri = httpsCallable<T, DownloadVideoResponse>(
        functions, methodName, { timeout: 600000 }
      );
      const { data } = await downloadGcsUri(request);
      return data;
    } catch (err) {
        console.error(err);
        throw err;
    }
  }

  async downloadVideoUriAndUrl<T = GenerateVideoRequest>(request: T, methodName: CallableNames = 'videos-generateVideo'): Promise<VideoGenerationResponse> {
    const { uri, mimeType } = await this.retrieveVideoUri(request, methodName);

    if (!uri) {
      throw new Error('Video operation completed but no URI was returned.');
    }

    return getDownloadURL(ref(this.storage, uri))
      .then((url) => {
        console.log("download url", url);
        return { uri, url, mimeType };
      })
      .catch((error) => {
        switch (error.code) {
          case 'storage/object-not-found':
            throw new Error("File doesn't exist");
          case 'storage/unauthorized':
            throw new Error("User doesn't have permission to access the object");
          case 'storage/canceled':
            throw new Error("User canceled the upload");
          case 'storage/unknown':
            throw new Error("Unknown storage error occurred, inspect the server response");
        }
        throw new Error("Unknown error occurred");
      });
  }
}

The VideoPlayerComponent has a required videoUrl signal input that is assigned to the source of the video player. It has an "Extend Video" button that emits an extendVideo custom event when clicked. This event notifies the parent component to extend the video in the videoUrl signal input.

import { LoaderComponent } from '@/shared/loader/loader.component';
import { ChangeDetectionStrategy, Component, input, output } from '@angular/core';
import { ExtendVideoIconComponent } from '../../icons/extend-video-icon.component';

@Component({
  selector: 'app-video-player',
  imports: [LoaderComponent, ExtendVideoIconComponent],
  template: `
@if (isGeneratingVideo()) {
  <div class="mt-6">
    <app-loader [loadingText]="loadingText()">
      <p class="text-sm">This can take several minutes. Please be patient.</p>
    </app-loader>
  </div>
} @else if (videoUrl()) {
  <div class="mt-6 flex flex-col gap-4">
    <div class="video-container">
      <video [src]="videoUrl()" controls autoplay loop class="w-full rounded-md"></video>
    </div>

    <div class="flex justify-center">
      <button
        (click)="extendVideo.emit()"
        aria-label="Extend video"
        title="Extend video"
        class="extend-btn"
      >
        <app-extend-video-icon />
        <span>Extend Video</span>
      </button>
    </div>
  </div>
}
  `,
  styleUrl: './video-player.component.css',
  changeDetection: ChangeDetectionStrategy.OnPush,
})
export class VideoPlayerComponent {
  isGeneratingVideo = input(false);
  videoUrl = input.required<string>();
  extendVideo = output<void>();

  loadingText = input('Generating your video...');
}

9. Angular Integration with Gemini to Extend Video

import { VideoPlayerComponent } from './video-player/video-player.component';

@Component({
  selector: 'app-gen-media',
  imports: [
    VideoPlayerComponent,
  ],
  template: `
  <app-video-player
    [isGeneratingVideo]="isGeneratingVideo()" 
    [videoUrl]="videoUrl()"
    (extendVideo)="extendVideo()"
    [loadingText]="loadingVideoText()"
  />`,
  changeDetection: ChangeDetectionStrategy.OnPush,
})
export class GenMediaComponent {
  private readonly genMediaService = inject(GenMediaService);
  private readonly genVideoService = inject(GenVideoService);

  genMediaInput = input<GenMediaInput>();
  videoUrl = this.genVideoService.videoUrl;
  isGeneratingVideo = this.genVideoService.isGeneratingVideo.asReadonly();
  loadingVideoText = signal('Generating your video...');

  trimmedUserPrompt = computed(() => this.genMediaInput()?.userPrompt.trim() || '');

  async extendVideo() {
    if (this.trimmedUserPrompt()) {
      try {
        this.loadingVideoText.set('Extending your video...');
        await this.genVideoService.extendVideo(this.trimmedUserPrompt());
      } finally {
        this.loadingVideoText.set('Generating your video...');
      }
    }
  }
}

The GenMediaComponent calls extendVideo with the prompt and the Veo video URL to create an extended video.

import { ConfigService } from '@/ai/services/config.service';
import { VeoService } from '@/ai/services/veo.service';
import { ExtendVideoRequest, GenerateVideoFromFramesRequest, GenerateVideoRequest, VideoGenerationResponse } from '@/ai/types/video.type';
import { computed, inject, Injectable, signal, WritableSignal } from '@angular/core';
import { getValue } from 'firebase/remote-config';

@Injectable({
  providedIn: 'root'
})
export class GenVideoService {
  private readonly veoService = inject(VeoService);
  private readonly configService = inject(ConfigService);

  videoError = signal('');
  videoResponse = signal<VideoGenerationResponse>({ uri: '', url: '', mimeType: '' });
  #extendVideoCounter = signal(0);
  isGeneratingVideo = signal(false);

  videoUrl = computed(() => this.videoResponse().url);

  isVideoExtensionAllowed(counter: number) {
    const remoteConfig = this.configService.firebaseObjects?.remoteConfig;
    if (!remoteConfig) {
      console.warn('Remote config does not exist.');
      return false;
    }

    const max_extend_allowed = getValue(remoteConfig,'maxVideoExtendAllowed').asNumber();
    if (counter >= max_extend_allowed) {
      console.warn('Maximum extension limit reached.');
      return false;
    }

    return true;
  }

  async extendVideo(prompt: string): Promise<void> {
    try {
      const result = await this.extendInterpolatedVideo(
        prompt,
        this.#extendVideoCounter(),
        this.videoResponse()
      );

      if (!result) {
        return;
      }

      this.videoResponse.set(result);
      this.#extendVideoCounter.update(count => count + 1);
      console.log(`Video extended successfully. Current extension count: ${this.#extendVideoCounter()}`);
    } catch (e) {
      console.error(e);
      const errMsg = e instanceof Error ?
        e.message :
        'An unexpected error occurred in video generation.'
      this.videoError.set(errMsg);
    } finally {
      this.isGeneratingVideo.set(false);
    }
  }

  async extendInterpolatedVideo(
    prompt: string,
    counter: number,
    customVideo: Pick<VideoGenerationResponse, 'uri' | 'mimeType'>,
    generatingSignal?: WritableSignal<boolean>,
    error?: WritableSignal<string>
  ) {
    const { uri, mimeType } = customVideo;

    if (!mimeType || !uri) {
      console.warn('No video to extend. Please generate a video first.');
      return null;
    }

    if (!prompt) {
      console.warn('Prompt is required to extend the video.');
      return null;
    }

    if (!this.isVideoExtensionAllowed(counter)) {
      return null;
    }

    const actualErrorSignal = error || this.videoError;
    const actualGeneratingSignal = generatingSignal || this.isGeneratingVideo;
    try {
      actualErrorSignal.set('');
      actualGeneratingSignal.set(true);

      const extendVideoParams: ExtendVideoRequest = {
        prompt,
        video: { uri, mimeType },
      }

      return await this.veoService.downloadVideoUriAndUrl(extendVideoParams, 'videos-extendVideo');
    } catch (e) {
      console.error(e);
      throw e;
    } finally {
      actualGeneratingSignal.set(false);
    }
  }
}

The extendVideo and extendInterpolatedVideo methods validate that an extension is allowed by comparing the #extendVideoCounter signal value against the maxVideoExtendAllowed Remote Config value. If the check fails, nothing happens. If the check succeeds, the videos-extendVideo Cloud Function is called to extend the video. Then, the #extendVideoCounter is incremented by 1 until the extension limit is reached. When extending a Veo video that is too long, Gemini throws an error, and I want to block this from happening on the client side.

This is the end of the walkthrough for the demo. You should now be able to extend generated videos in a Cloud Function, store them securely in a bucket, and play them in a video player within a user interface.

Conclusion

Combining Veo 3.1 with the serverless scalability of Firebase is a powerful workflow.

First, the Angular application neither installs the genai dependency nor maintains the Vertex AI environment variables in a .env file. The client application calls the Cloud Functions to perform intensive tasks and waits for the results.

The Cloud Functions receive arguments from the client, execute complex AI operations like generation, interpolation, and video extension, and write the videos to the dedicated bucket securely. During local development, the Firebase Emulator calls the functions at http://localhost:5001 instead of the ones deployed on the Cloud Run platform.

Try cloning the GitHub repository, generating images, and using those images to generate and interpolate videos. Then, you can extend these Veo videos to create extended videos that are more than 7 seconds long.

Resources

Observability at Scale: Mastering ADK Callbacks for Cost, Latency, and Auditability

Connie Leung — Mon, 06 Apr 2026 16:08:17 +0000

Observability at Scale: Mastering ADK Callbacks for Cost, Latency, and Auditability

AI orchestrators receive significant attention; however, when deployments become latent and costly, developers often overlook a critical capability: ADK callback hooks. The design patterns and best practices of callback hooks enable developers to refactor logic from agents to callback hooks to add observability, reduce cost and latency, and modify session state dynamically.

This post explores how to create callback hooks at various stages of an ADK agent to demonstrate the following design patterns:

Logging and monitoring performance
Managing dynamic state
Modifying requests and responses
Skipping steps conditionally

Demo Overview

The orchestrator routes the project description to sequentialEvaluationAgent. The sequentialEvaluationAgent consists of project, anti-patterns, decision, recommendation, audit, upload, merger, and email subagents.

In the project, anti-patterns, decision, recommendation, and merger subagents, I implemented callback hooks to demonstrate their capabilities and practicality.

Architecture

Google ADK stands for Agent Development Kit. It is an open source agent framework that enables developers to build and deploy AI agents in a convenient way.

The project, anti-patterns, decision, recommendation, and synthesis agents are LLM agents. These agents require Gemini to reason and generate text responses.

The Audit Trail, Cloud Storage, and Email agents integrate with external APIs or resources that trigger deterministic actions.

Callback Types

ADK provides six types of callback hooks that are called before and after an agent is executed, before and after a model is executed, and before and after a tool is executed.

Callback Type	Description
beforeAgentCallback	Call before the new cycle of an agent
afterAgentCallback	Call after the agent cycle completes
beforeModelCallback	Call before the LLM is called
afterModelCallback	Call after the LLM returns a response
beforeToolCallback	Call before the tool is called
afterToolCallback	Call after the tool is called

When and why I started using ADK callback hooks

When I used ADK web for local testing and debugging, my first priority was correctness. Performance and token usage had lower priorities until the agents were deployed to a QA environment.

When product managers and the QA team recognized apparent slowness and high cost, they would inform me. Next, I examined the agent to identify performance bottlenecks and deterministic steps that could be handled in callback hooks or application code.

The Benefits of Callback

Using callback hooks offers several critical advantages in my subagents:

Type	Benefit
Short Circuit	Call `beforeModelCallback` to validate session data and skip LLM flow when data is invalid
Separation of Concerns	Reset session data in the `beforeAgentCallback` so that tool remains lean and focuses on business logic
Observability	Add logging in the `beforeAgentCallback` and the `afterAgentCallback` to log performance metrics
Dynamic State Management	Create a reusable `afterToolCallback` to increment validation attempts and modify response status to FATAL_ERROR

Prerequisites

Requirements

TypeScript 5.9.3 or later
Node.js 24.13.0 or later
npm 11.8.0 or later
Docker (For running MailHog locally)
Google ADK (For building the custom agent)
Gemini in Vertex AI (to call the model in LLM agents, although not required for the custom agent)

Note: I used Gemini in Vertex AI for authentication due to regional availability. Gemini is currently unavailable in Hong Kong; therefore, I utilized Gemini in Vertex AI instead.

Install npm dependencies

npm i --save-exact @google/adk
npm i --save-dev --save-exact @google/adk-devtools
npm i --save-exact nodemailer
npm i --save-dev --save-exact @types/nodemailer rimraf
npm i --save-exact marked
npm i --save-exact zod

I installed the required dependencies for building ADK agents, converting Markdown strings to HTML, and sending emails to MailHog during local testing.

Pinned dependencies ensure the versioning is the same in development and production environments for enterprise-level applications.

Environment variables

Copy .env.example to .env and fill in the credentials:

GEMINI_MODEL_NAME="gemini-3.1-flash-lite-preview"
GOOGLE_CLOUD_PROJECT="<Google Cloud Project ID>"
GOOGLE_CLOUD_LOCATION="global" 
GOOGLE_GENAI_USE_VERTEXAI=TRUE

# SMTP Settings (MailHog)
SMTP_HOST="localhost"
SMTP_PORT=1025
SMTP_USER=""
SMTP_PASS=""
SMTP_FROM="no-reply@test.local"
ADMIN_EMAIL="admin@test.local"

SMTP_HOST, SMTP_PORT, SMTP_USER, SMTP_PASS, are required to set up MailHog for local email testing.

SMTP_FROM is a from email address that can be any string in local testing.

ADMIN_EMAIL is an administrator email address receiving emails that the EmailAgent sends. It is an environment variable in my use case because it is the only recipient. If another scenario requires sending emails to customers, the environment variable should be removed.

ADK Callback Patterns in Action

Here is how I implemented these four callback design patterns in practice.

Logging and Monitoring

Callback used: beforeAgentCallback and afterAgentCallback

The agentStartCallback stores the current time in milliseconds in start_time variable in the session state.

import { SingleAgentCallback } from '@google/adk';

export const START_TIME_KEY = 'start_time';

export const agentStartCallback: SingleAgentCallback = (context) => {
  if (!context || !context.state) {
    return undefined;
  }

  context.state.set(START_TIME_KEY, Date.now());
  return undefined;
};

The agentEndCallback obtains the start time from the session state and calculates the duration. Then, it uses console.log to log the performance metrics. The callback returns undefined so that any subagent always flows to the next one in the sequential workflow.

export const agentEndCallback: SingleAgentCallback = (context) => {
  if (!context || !context.state) {
    return undefined;
  }

  const now = Date.now();
  const startTime = context.state.get<number>(START_TIME_KEY) || now;
  console.log(
    `Performance Metrics for Agent "${context.agentName}": Total Elapsed Time: ${(now - startTime) / 1000} seconds.`,
  );
  return undefined;
};

Next, I call both callback hooks in the subagents to know how long they take to execute. The following example shows how I log the performance metric of a project subagent.

export function createProjectAgent(model: string) {
  const projectAgent = new LlmAgent({
    name: 'ProjectAgent',
    model,
    beforeAgentCallback: agentStartCallback,
    instruction: (context) => {
      ... LLM instruction ....
    },
    afterAgentCallback: agentEndCallback,
    ... 
  });

  return projectAgent;
}

Reset Session State

Callback used: beforeAgentCallback

The orchestrator resets variables in the session state before the agent lifecycle begins.

export const AUDIT_TRAIL_KEY = 'auditTrail';
export const RECOMMENDATION_KEY = 'recommendation';
export const CLOUD_STORAGE_KEY = 'cloudStorage';
export const DECISION_KEY = 'decision';
export const PROJECT_KEY = 'project';
export const ANTI_PATTERNS_KEY = 'antiPatterns';
export const MERGED_RESULTS_KEY = 'mergedResults';
export const PROJECT_DESCRIPTION_KEY = 'project_description';
export const VALIDATION_ATTEMPTS_KEY = 'validation_attempts';

const resetNewEvaluationCallback: SingleAgentCallback = (context) => {
  if (!context || !context.state) {
    return undefined;
  }

  const state = context.state;

  // Clear all previous evaluation data
  state.set(PROJECT_KEY, null);
  state.set(ANTI_PATTERNS_KEY, null);
  state.set(DECISION_KEY, null);
  state.set(RECOMMENDATION_KEY, null);
  state.set(AUDIT_TRAIL_KEY, null);
  state.set(CLOUD_STORAGE_KEY, null);
  state.set(MERGED_RESULTS_KEY, null);
  state.set(VALIDATION_ATTEMPTS_KEY, 0);

  console.log(
    `beforeAgentCallback: Agent ${context.agentName} has reset the session state for a new evaluation cycle.`,
  );

  return undefined;
};

The orchestrator sets the variables to null in the beforeAgentCallback. In the prepareEvaluationTool, the orchestrator only replaces the description with the new project description.

const prepareEvaluationTool = new FunctionTool({
  name: 'prepare_evaluation',
  description: 'Stores the new project description to prepare for a fresh evaluation.',
  parameters: z.object({
    description: z.string().describe('The validated project description from the user.'),
  }),
  execute: ({ description }, context) => {
    if (!context || !context.state) {
      return { status: 'ERROR', message: 'No session state found.' };
    }

    // Set the new description for the ProjectAgent to find
    context.state.set(PROJECT_DESCRIPTION_KEY, description);

    return { status: 'SUCCESS', message: 'Description updated.' };
  },
});

The tool logic is more efficient and the token usage is reduced.

export const rootAgent = new LlmAgent({
  name: 'ProjectEvaluationAgent',
  model: 'gemini-3.1-flash-lite-preview',
  beforeAgentCallback: resetNewEvaluationCallback,
  instruction: `
    ... LLM instruction ....
  `,
  tools: [prepareEvaluationTool],
  subAgents: [sequentialEvaluationAgent],
});

The orchestrator uses the resetNewEvaluationCallback to reset the session variables, and the prepareEvaluationTool to replace the project description. Finally, the sequentialEvaluationAgent starts the agentic workflow to derive a decision and generate the recommendation.

Dynamic State Management

Prerequisite: The subagent uses tool calling to perform action
Callback used: afterToolCallback

The use case is to increment the value of validation_attempts in the session state. After consulting AI, the afterToolCallback stage of project and decision subagents is the perfect location to increment it. Therefore, I define a meta-function that creates a afterToolCallback to increment the validation attempts.

export const VALIDATION_ATTEMPTS_KEY = 'validation_attempts';

export const MAX_ITERATIONS = 3;

import { AfterToolCallback } from '@google/adk';
import { VALIDATION_ATTEMPTS_KEY } from '../output-keys.const.js';
import { MAX_ITERATIONS } from '../validation.const.js';

export function createAfterToolCallback(fatalErrorMessage: string, maxAttempts = MAX_ITERATIONS): AfterToolCallback {
  return ({ tool, context, response }) => {
    if (!tool || !context || !context.state) {
      return undefined;
    }

    const toolName = tool.name;
    const agentName = context.agentName;
    const state = context.state;

    if (!response || typeof response !== 'object' || !('status' in response)) {
      return undefined;
    }

    // [1] Dynamic state management
    const attempts = (state.get<number>(VALIDATION_ATTEMPTS_KEY) || 0) + 1;
    state.set(VALIDATION_ATTEMPTS_KEY, attempts);

    // [2] Response modification
    const status = response.status || 'ERROR';
    if (status === 'ERROR' && attempts >= maxAttempts) {
      context.actions.escalate = true;

      return {
        status: 'FATAL_ERROR',
        message: fatalErrorMessage,
      };
    }
  };
}

In both project and decision subagents, I invoke createAfterToolCallback to create their own afterToolCallback to increment the validation attempts.

const projectAfterToolCallback = createAfterToolCallback(
  `STOP processing immediately. Max validation attempts reached. Return the most accurate data found so far or empty strings if none.`,
);

const decisionAfterToolCallback = createAfterToolCallback(
  `STOP processing immediately and output the final JSON schema with verdict: "None".`,
);

Next, I call the projectAfterToolCallback in the afterToolCallback stage of the project subagent.

export function createProjectAgent(model: string) {
  const projectAgent = new LlmAgent({
    name: 'ProjectAgent',
    model,
    beforeAgentCallback: agentStartCallback,
    instruction: (context) => {
      ... LLM instruction ....
    },
    afterToolCallback: projectAfterToolCallback,
    afterAgentCallback: agentEndCallback,
    tools: [validateProjectTool],
    ... 
  });

  return projectAgent;
}

Next, I call the decisionAfterToolCallback in the afterToolCallback stage of the decision subagent.

export function createDecisionTreeAgent(model: string) {
  const decisionTreeAgent = new LlmAgent({
    name: 'DecisionTreeAgent',
    model,
    beforeAgentCallback: [resetAttemptsCallback, agentStartCallback],
    instruction: (context) => {
      ... instruction  of the LLM flow ...
    },
    afterToolCallback: decisionAfterToolCallback,
    afterAgentCallback: agentEndCallback,
    tools: [validateDecisionTool],
    ...
  });

  return decisionTreeAgent;
}

Request/Response Modification

Prerequisite: The subagent uses tool calling to perform action
Callback used: AfterToolCallback

The same AfterToolCallback also modifies the response when the number of validation attempts exceeds the maximum iterations.

When the status is ERROR and the value of validation_attempts is at least maximum_iterations, the context.actions.escalate flag is set to true to break out of the loop. Moreover, the status is changed to FATAL_ERROR and the custom fatal message is returned.

Conditional Skipping of Steps

This is an important design pattern for avoiding unnecessary LLM executions.

Callback used: beforeModelCallback.

In my subagents, I run this callback to validate the session data. When a specific condition is met, the callback returns content immediately to skip the subsequent LLM flow.

Example 1

If the project agent is able to break down a project description into task, problem, constraint, and goal, the agent will return the breakdown immediately. Otherwise, the agent prompts Gemini to use reasoning to perform the breakdown.

import { SingleBeforeModelCallback } from '@google/adk';

const beforeModelCallback: SingleBeforeModelCallback = ({ context }) => {
  const { project } = getEvaluationContext(context);
  const { isCompleted } = isProjectDetailsFilled(project);

  if (isCompleted) {
    return {
      content: {
        role: 'model',
        parts: [
          {
            text: JSON.stringify(project),
          },
        ],
      },
    };
  }

  return undefined;
};

Then, beforeModelCallback is hooked to the beforeModelCallback stage of the project subagent.

export function createProjectAgent(model: string) {
  const projectAgent = new LlmAgent({
    name: 'ProjectAgent',
    model,
    description:
      'Analyzes the user-provided project description to extract and structure its core components, including the primary task, underlying problem, ultimate goal, and architectural constraints.',
    beforeAgentCallback: agentStartCallback,
    beforeModelCallback,
    instruction: (context) => {
      const { projectDescription } = getEvaluationContext(context);
      if (!projectDescription) {
        return '';
      }

      return generateProjectBreakdownPrompt(projectDescription);
    },
    afterToolCallback: projectAfterToolCallback,
    afterAgentCallback: agentEndCallback,
    tools: [validateProjectTool],
    outputSchema: projectSchema,
    outputKey: PROJECT_KEY,
    disallowTransferToParent: true,
    disallowTransferToPeers: true,
  });

  return projectAgent;
}

Example 2

The decision agent verifies the verdict property in the beforeModelCallback. If the verdict is not None, the callback returns the valid decision immediately. The verdict is None and the callback examines the project breakdown and the anti-patterns. When project breakdown and anti-patterns are provided, the callback returns undefined to trigger the LLM flow. Otherwise, the decision agent does not have valid inputs to derive the verdict. The callback returns None in this edge case.

import { SingleBeforeModelCallback } from '@google/adk';

const beforeModelCallback: SingleBeforeModelCallback = ({ context }) => {
  const { decision } = getEvaluationContext(context);

  if (decision && decision.verdict !== 'None') {
    return {
      content: {
        role: 'model',
        parts: [
          {
            text: JSON.stringify(decision),
          },
        ],
      },
    };
  }

  const { project, antiPatterns } = getEvaluationContext(context);
  const { isCompleted } = isProjectDetailsFilled(project);

  if (isCompleted && antiPatterns) {
    return undefined;
  }

  return {
    content: {
      role: 'model',
      parts: [
        {
          text: JSON.stringify({
            verdict: 'None',
            nodes: [],
          }),
        },
      ],
    },
  };
};

Then, beforeModelCallback is hooked to the beforeModelCallback stage of the project subagent.

export function createDecisionTreeAgent(model: string) {
  const decisionTreeAgent = new LlmAgent({
    name: 'DecisionTreeAgent',
    model,
    beforeAgentCallback: [resetAttemptsCallback, agentStartCallback],
    beforeModelCallback,
    instruction: (context) => {
      ... instruction  of the LLM flow ...
    },
    afterToolCallback: decisionAfterToolCallback,
    afterAgentCallback: agentEndCallback,
    tools: [validateDecisionTool],
    outputSchema: decisionSchema,
    outputKey: DECISION_KEY,
    disallowTransferToParent: true,
    disallowTransferToPeers: true,
  });

  return decisionTreeAgent;
}

Example 3

The recommendation agent uses the beforeModelCallback to examine the project breakdown, anti-patterns and verdict. There are two scenarios that need LLM to generate the recommendation. The first scenario is valid project breakdown, anti-patterns, and non-None verdict. The second scenario is incomplete project breakdown and None verdict. The LLM is instructed to describe the missing field in the project breakdown and how important the missing field is to obtain a valid decision. For other cases, the callback returns a static recommendation immediately and skip the subsequent LLM flow.

function constructRecommendation(recommendation: string) {
  return {
    content: {
      role: 'model',
      parts: [
        {
          text: JSON.stringify({
            text: recommendation,
          }),
        },
      ],
    },
  };
}

const beforeModelCallback: SingleBeforeModelCallback = ({ context }) => {
  const { project, antiPatterns, decision } = getEvaluationContext(context);

  const { isCompleted } = isProjectDetailsFilled(project);
  const isDecisionNone = decision && decision.verdict === 'None';

  if ((isCompleted && antiPatterns && decision && decision.verdict !== 'None') || (!isCompleted && isDecisionNone)) {
    return undefined;
  } else if (isCompleted && isDecisionNone) {
    return constructRecommendation(
      '## Recommendation: Manual Review Required\n\n**Status:** Abnormal Case Detected\n\nThe provided project is complete and valid, but the decision tree could not reach a conclusive verdict (Result: `None`).\n\n**Possible Reasons:**\n- The requirements fall outside of known architectural patterns.\n- There are conflicting constraints and goals that cannot be resolved automatically.\n\n**Next Steps:**\n- Review and refine the constraints or goals.\n- Escalate for manual architectural review.',
    );
  }

  return constructRecommendation(
    '## Recommendation: Data Required\n\n**Status:** Abnormal Case Detected\n\nNo decision is reached.',
  );
};

Similar to the previous project and decision agents, the beforeModelCallback function is hooked to the beforeModelCallback stage of the recommendation agent.

export function createRecommendationAgent(model: string) {
  const recommendationAgent = new LlmAgent({
    name: 'RecommendationAgent',
    model,
    beforeModelCallback,
    beforeAgentCallback: agentStartCallback,
    instruction: (context) => {
      const { project, antiPatterns, decision } = getEvaluationContext(context);
      const { isCompleted, missingFields } = isProjectDetailsFilled(project);

      if (project) {
        if (!isCompleted && decision && decision.verdict === 'None') {
          console.log('RecommendationAgent -> generateFailedDecisionPrompt');
          return generateFailedDecisionPrompt(project, missingFields);
        } else if (isCompleted && antiPatterns && decision && decision.verdict !== 'None') {
          console.log('RecommendationAgent -> generateRecommendationPrompt');
          return generateRecommendationPrompt(project, antiPatterns, decision);
        }
      }
      return 'Skipping LLM due to missing data.';
    },
    afterAgentCallback: agentEndCallback,
    outputSchema: recommendationSchema,
    outputKey: RECOMMENDATION_KEY,
    disallowTransferToParent: true,
    disallowTransferToPeers: true,
  });

  return recommendationAgent;
}

These are the callback design patterns that the agent adopts. In the next section, I describe how to launch the agent to observe the logging messages in the terminal.

Environment Setup

I pulled the latest version of the MailHog Docker image from Docker Hub and started it locally to receive test emails and display them in the Web UI. The docker-compose.yml file contains the setup configuration.

services:
  mailhog:
    image: mailhog/mailhog
    container_name: mailhog
    ports:
      - '1025:1025' # SMTP port
      - '8025:8025' # HTTP (Web UI) port
    restart: always
    networks:
      - decision-tree-agent-network

networks:
  decision-tree-agent-network:

The SMTP server listens on port 1025, and the Web UI is accessible at http://localhost:8025.

docker compose up -d

Start MailHog in Docker.

Testing

Add scripts to package.json to build and start the ADK web interface.

  "scripts": {
    "prebuild": "rimraf dist",
    "build": "npx tsc --project tsconfig.json",
    "web": "npm run build && npx @google/adk-devtools web --host 127.0.0.1 dist/agent.js"
  },

Open a terminal and type npm run web to start the API server.
Open a new browser tab and type http://localhost:8000.
Paste the following text into the message box:

One of my favorite tech influencers just tweeted about a 'breakthrough in solid-state batteries.' Find which public company they might be referring to, check that company’s recent patent filings to see if it’s true, and then check their stock price to see if the market has already 'priced it in'.

Ensure the root agent executes and halts when the email agent terminates. The orchestrator and the project agent trigger the callback hooks to reset the session state, log performance metrics, validate session data, increment the validation attempts, and modify the response.

Similarly, both anti-patterns and decisions agents used the beforeAgentCallback and afterAgentCallback to log performance metrics. They also used the beforeModelCallback to validate session data before calling the LLM to generate a response. Moreover, the decision agent incremented the validation attempts and modify the status to FATAL_ERROR when the validation attempts exceeded or equal to maximum iterations in the afterToolCallback.

Similarly, both recommendation and merger agents used the beforeAgentCallback and afterAgentCallback to log performance metrics. The recommendation agent also used the beforeModelCallback to validate session data before calling the LLM to generate a recommendation.

Conclusion

ADK supports lifecycle callback hooks that are applicable to different use cases. In this post, I describe logging performance metrics in the beforeAgentCallback and afterAgentCallback. I refactored the reset session state logic in the orchestrator's beforeAgentCallback to make the tool lean and cost efficient. The afterToolCallback escalated to a higher agent and modified the response status to FATAL_ERROR when the loop count in the validation-retry loop exceeded the maximum iterations. When skipping LLM calls conditionally, the beforeModelCallback returned custom content immediately. Then, the agent did not add unnecessary time to the agent flow and did not consume tokens.

The takeaway: follow the design patterns and best practices of callback at various stages of an agent to log performance metrics, reduce cost and latency, and modify response.

Resources

Observability at Scale: Mastering ADK Callbacks for Cost, Latency, and Auditability [GDE]

Connie Leung — Mon, 06 Apr 2026 16:02:04 +0000

This post explores how to create callback hooks at various stages of an ADK agent to demonstrate the following design patterns:

Logging and monitoring performance
Managing dynamic state
Modifying requests and responses
Skipping steps conditionally

Demo Overview

In the project, anti-patterns, decision, recommendation, and merger subagents, I implemented callback hooks to demonstrate their capabilities and practicality.

Architecture

Google ADK stands for Agent Development Kit. It is an open source agent framework that enables developers to build and deploy AI agents in a convenient way.

The project, anti-patterns, decision, recommendation, and synthesis agents are LLM agents. These agents require Gemini to reason and generate text responses.

The Audit Trail, Cloud Storage, and Email agents integrate with external APIs or resources that trigger deterministic actions.

Callback Types

ADK provides six types of callback hooks that are called before and after an agent is executed, before and after a model is executed, and before and after a tool is executed.

Callback Type	Description
beforeAgentCallback	Call before the new cycle of an agent
afterAgentCallback	Call after the agent cycle completes
beforeModelCallback	Call before the LLM is called
afterModelCallback	Call after the LLM returns a response
beforeToolCallback	Call before the tool is called
afterToolCallback	Call after the tool is called

When and why I started using ADK callback hooks

When I used ADK web for local testing and debugging, my first priority was correctness. Performance and token usage had lower priorities until the agents were deployed to a QA environment.

The Benefits of Callback

Using callback hooks offers several critical advantages in my subagents:

Type	Benefit
Short Circuit	Call `beforeModelCallback` to validate session data and skip LLM flow when data is invalid
Separation of Concerns	Reset session data in the `beforeAgentCallback` so that tool remains lean and focuses on business logic
Observability	Add logging in the `beforeAgentCallback` and the `afterAgentCallback` to log performance metrics
Dynamic State Management	Create a reusable `afterToolCallback` to increment validation attempts and modify response status to FATAL_ERROR

Prerequisites

Requirements

TypeScript 5.9.3 or later
Node.js 24.13.0 or later
npm 11.8.0 or later
Docker (For running MailHog locally)
Google ADK (For building the custom agent)
Gemini in Vertex AI (to call the model in LLM agents, although not required for the custom agent)

Note: I used Gemini in Vertex AI for authentication due to regional availability. Gemini is currently unavailable in Hong Kong; therefore, I utilized Gemini in Vertex AI instead.

Install npm dependencies

npm i --save-exact @google/adk
npm i --save-dev --save-exact @google/adk-devtools
npm i --save-exact nodemailer
npm i --save-dev --save-exact @types/nodemailer rimraf
npm i --save-exact marked
npm i --save-exact zod

I installed the required dependencies for building ADK agents, converting Markdown strings to HTML, and sending emails to MailHog during local testing.

Pinned dependencies ensure the versioning is the same in development and production environments for enterprise-level applications.

Environment variables

Copy .env.example to .env and fill in the credentials:

GEMINI_MODEL_NAME="gemini-3.1-flash-lite-preview"
GOOGLE_CLOUD_PROJECT="<Google Cloud Project ID>"
GOOGLE_CLOUD_LOCATION="global" 
GOOGLE_GENAI_USE_VERTEXAI=TRUE

# SMTP Settings (MailHog)
SMTP_HOST="localhost"
SMTP_PORT=1025
SMTP_USER=""
SMTP_PASS=""
SMTP_FROM="no-reply@test.local"
ADMIN_EMAIL="admin@test.local"

SMTP_HOST, SMTP_PORT, SMTP_USER, SMTP_PASS, are required to set up MailHog for local email testing.

SMTP_FROM is a from email address that can be any string in local testing.

ADK Callback Patterns in Action

Here is how I implemented these four callback design patterns in practice.

Logging and Monitoring

Callback used: beforeAgentCallback and afterAgentCallback

The agentStartCallback stores the current time in milliseconds in start_time variable in the session state.

import { SingleAgentCallback } from '@google/adk';

export const START_TIME_KEY = 'start_time';

export const agentStartCallback: SingleAgentCallback = (context) => {
  if (!context || !context.state) {
    return undefined;
  }

  context.state.set(START_TIME_KEY, Date.now());
  return undefined;
};

export const agentEndCallback: SingleAgentCallback = (context) => {
  if (!context || !context.state) {
    return undefined;
  }

  const now = Date.now();
  const startTime = context.state.get<number>(START_TIME_KEY) || now;
  console.log(
    `Performance Metrics for Agent "${context.agentName}": Total Elapsed Time: ${(now - startTime) / 1000} seconds.`,
  );
  return undefined;
};

Next, I call both callback hooks in the subagents to know how long they take to execute. The following example shows how I log the performance metric of a project subagent.

export function createProjectAgent(model: string) {
  const projectAgent = new LlmAgent({
    name: 'ProjectAgent',
    model,
    beforeAgentCallback: agentStartCallback,
    instruction: (context) => {
      ... LLM instruction ....
    },
    afterAgentCallback: agentEndCallback,
    ... 
  });

  return projectAgent;
}

Reset Session State

Callback used: beforeAgentCallback

The orchestrator resets variables in the session state before the agent lifecycle begins.

export const AUDIT_TRAIL_KEY = 'auditTrail';
export const RECOMMENDATION_KEY = 'recommendation';
export const CLOUD_STORAGE_KEY = 'cloudStorage';
export const DECISION_KEY = 'decision';
export const PROJECT_KEY = 'project';
export const ANTI_PATTERNS_KEY = 'antiPatterns';
export const MERGED_RESULTS_KEY = 'mergedResults';
export const PROJECT_DESCRIPTION_KEY = 'project_description';
export const VALIDATION_ATTEMPTS_KEY = 'validation_attempts';

const resetNewEvaluationCallback: SingleAgentCallback = (context) => {
  if (!context || !context.state) {
    return undefined;
  }

  const state = context.state;

  // Clear all previous evaluation data
  state.set(PROJECT_KEY, null);
  state.set(ANTI_PATTERNS_KEY, null);
  state.set(DECISION_KEY, null);
  state.set(RECOMMENDATION_KEY, null);
  state.set(AUDIT_TRAIL_KEY, null);
  state.set(CLOUD_STORAGE_KEY, null);
  state.set(MERGED_RESULTS_KEY, null);
  state.set(VALIDATION_ATTEMPTS_KEY, 0);

  console.log(
    `beforeAgentCallback: Agent ${context.agentName} has reset the session state for a new evaluation cycle.`,
  );

  return undefined;
};

The orchestrator sets the variables to null in the beforeAgentCallback. In the prepareEvaluationTool, the orchestrator only replaces the description with the new project description.

const prepareEvaluationTool = new FunctionTool({
  name: 'prepare_evaluation',
  description: 'Stores the new project description to prepare for a fresh evaluation.',
  parameters: z.object({
    description: z.string().describe('The validated project description from the user.'),
  }),
  execute: ({ description }, context) => {
    if (!context || !context.state) {
      return { status: 'ERROR', message: 'No session state found.' };
    }

    // Set the new description for the ProjectAgent to find
    context.state.set(PROJECT_DESCRIPTION_KEY, description);

    return { status: 'SUCCESS', message: 'Description updated.' };
  },
});

The tool logic is more efficient and the token usage is reduced.

export const rootAgent = new LlmAgent({
  name: 'ProjectEvaluationAgent',
  model: 'gemini-3.1-flash-lite-preview',
  beforeAgentCallback: resetNewEvaluationCallback,
  instruction: `
    ... LLM instruction ....
  `,
  tools: [prepareEvaluationTool],
  subAgents: [sequentialEvaluationAgent],
});

Dynamic State Management

Prerequisite: The subagent uses tool calling to perform action
Callback used: afterToolCallback

export const VALIDATION_ATTEMPTS_KEY = 'validation_attempts';

export const MAX_ITERATIONS = 3;

import { AfterToolCallback } from '@google/adk';
import { VALIDATION_ATTEMPTS_KEY } from '../output-keys.const.js';
import { MAX_ITERATIONS } from '../validation.const.js';

export function createAfterToolCallback(fatalErrorMessage: string, maxAttempts = MAX_ITERATIONS): AfterToolCallback {
  return ({ tool, context, response }) => {
    if (!tool || !context || !context.state) {
      return undefined;
    }

    const toolName = tool.name;
    const agentName = context.agentName;
    const state = context.state;

    if (!response || typeof response !== 'object' || !('status' in response)) {
      return undefined;
    }

    // [1] Dynamic state management
    const attempts = (state.get<number>(VALIDATION_ATTEMPTS_KEY) || 0) + 1;
    state.set(VALIDATION_ATTEMPTS_KEY, attempts);

    // [2] Response modification
    const status = response.status || 'ERROR';
    if (status === 'ERROR' && attempts >= maxAttempts) {
      context.actions.escalate = true;

      return {
        status: 'FATAL_ERROR',
        message: fatalErrorMessage,
      };
    }
  };
}

In both project and decision subagents, I invoke createAfterToolCallback to create their own afterToolCallback to increment the validation attempts.

const projectAfterToolCallback = createAfterToolCallback(
  `STOP processing immediately. Max validation attempts reached. Return the most accurate data found so far or empty strings if none.`,
);

const decisionAfterToolCallback = createAfterToolCallback(
  `STOP processing immediately and output the final JSON schema with verdict: "None".`,
);

Next, I call the projectAfterToolCallback in the afterToolCallback stage of the project subagent.

export function createProjectAgent(model: string) {
  const projectAgent = new LlmAgent({
    name: 'ProjectAgent',
    model,
    beforeAgentCallback: agentStartCallback,
    instruction: (context) => {
      ... LLM instruction ....
    },
    afterToolCallback: projectAfterToolCallback,
    afterAgentCallback: agentEndCallback,
    tools: [validateProjectTool],
    ... 
  });

  return projectAgent;
}

Next, I call the decisionAfterToolCallback in the afterToolCallback stage of the decision subagent.

export function createDecisionTreeAgent(model: string) {
  const decisionTreeAgent = new LlmAgent({
    name: 'DecisionTreeAgent',
    model,
    beforeAgentCallback: [resetAttemptsCallback, agentStartCallback],
    instruction: (context) => {
      ... instruction  of the LLM flow ...
    },
    afterToolCallback: decisionAfterToolCallback,
    afterAgentCallback: agentEndCallback,
    tools: [validateDecisionTool],
    ...
  });

  return decisionTreeAgent;
}

Request/Response Modification

Prerequisite: The subagent uses tool calling to perform action
Callback used: AfterToolCallback

The same AfterToolCallback also modifies the response when the number of validation attempts exceeds the maximum iterations.

Conditional Skipping of Steps

This is an important design pattern for avoiding unnecessary LLM executions.

Callback used: beforeModelCallback.

In my subagents, I run this callback to validate the session data. When a specific condition is met, the callback returns content immediately to skip the subsequent LLM flow.

Example 1

import { SingleBeforeModelCallback } from '@google/adk';

const beforeModelCallback: SingleBeforeModelCallback = ({ context }) => {
  const { project } = getEvaluationContext(context);
  const { isCompleted } = isProjectDetailsFilled(project);

  if (isCompleted) {
    return {
      content: {
        role: 'model',
        parts: [
          {
            text: JSON.stringify(project),
          },
        ],
      },
    };
  }

  return undefined;
};

Then, beforeModelCallback is hooked to the beforeModelCallback stage of the project subagent.

export function createProjectAgent(model: string) {
  const projectAgent = new LlmAgent({
    name: 'ProjectAgent',
    model,
    description:
      'Analyzes the user-provided project description to extract and structure its core components, including the primary task, underlying problem, ultimate goal, and architectural constraints.',
    beforeAgentCallback: agentStartCallback,
    beforeModelCallback,
    instruction: (context) => {
      const { projectDescription } = getEvaluationContext(context);
      if (!projectDescription) {
        return '';
      }

      return generateProjectBreakdownPrompt(projectDescription);
    },
    afterToolCallback: projectAfterToolCallback,
    afterAgentCallback: agentEndCallback,
    tools: [validateProjectTool],
    outputSchema: projectSchema,
    outputKey: PROJECT_KEY,
    disallowTransferToParent: true,
    disallowTransferToPeers: true,
  });

  return projectAgent;
}

Example 2

import { SingleBeforeModelCallback } from '@google/adk';

const beforeModelCallback: SingleBeforeModelCallback = ({ context }) => {
  const { decision } = getEvaluationContext(context);

  if (decision && decision.verdict !== 'None') {
    return {
      content: {
        role: 'model',
        parts: [
          {
            text: JSON.stringify(decision),
          },
        ],
      },
    };
  }

  const { project, antiPatterns } = getEvaluationContext(context);
  const { isCompleted } = isProjectDetailsFilled(project);

  if (isCompleted && antiPatterns) {
    return undefined;
  }

  return {
    content: {
      role: 'model',
      parts: [
        {
          text: JSON.stringify({
            verdict: 'None',
            nodes: [],
          }),
        },
      ],
    },
  };
};

Then, beforeModelCallback is hooked to the beforeModelCallback stage of the project subagent.

export function createDecisionTreeAgent(model: string) {
  const decisionTreeAgent = new LlmAgent({
    name: 'DecisionTreeAgent',
    model,
    beforeAgentCallback: [resetAttemptsCallback, agentStartCallback],
    beforeModelCallback,
    instruction: (context) => {
      ... instruction  of the LLM flow ...
    },
    afterToolCallback: decisionAfterToolCallback,
    afterAgentCallback: agentEndCallback,
    tools: [validateDecisionTool],
    outputSchema: decisionSchema,
    outputKey: DECISION_KEY,
    disallowTransferToParent: true,
    disallowTransferToPeers: true,
  });

  return decisionTreeAgent;
}

Example 3

function constructRecommendation(recommendation: string) {
  return {
    content: {
      role: 'model',
      parts: [
        {
          text: JSON.stringify({
            text: recommendation,
          }),
        },
      ],
    },
  };
}

const beforeModelCallback: SingleBeforeModelCallback = ({ context }) => {
  const { project, antiPatterns, decision } = getEvaluationContext(context);

  const { isCompleted } = isProjectDetailsFilled(project);
  const isDecisionNone = decision && decision.verdict === 'None';

  if ((isCompleted && antiPatterns && decision && decision.verdict !== 'None') || (!isCompleted && isDecisionNone)) {
    return undefined;
  } else if (isCompleted && isDecisionNone) {
    return constructRecommendation(
      '## Recommendation: Manual Review Required\n\n**Status:** Abnormal Case Detected\n\nThe provided project is complete and valid, but the decision tree could not reach a conclusive verdict (Result: `None`).\n\n**Possible Reasons:**\n- The requirements fall outside of known architectural patterns.\n- There are conflicting constraints and goals that cannot be resolved automatically.\n\n**Next Steps:**\n- Review and refine the constraints or goals.\n- Escalate for manual architectural review.',
    );
  }

  return constructRecommendation(
    '## Recommendation: Data Required\n\n**Status:** Abnormal Case Detected\n\nNo decision is reached.',
  );
};

Similar to the previous project and decision agents, the beforeModelCallback function is hooked to the beforeModelCallback stage of the recommendation agent.

export function createRecommendationAgent(model: string) {
  const recommendationAgent = new LlmAgent({
    name: 'RecommendationAgent',
    model,
    beforeModelCallback,
    beforeAgentCallback: agentStartCallback,
    instruction: (context) => {
      const { project, antiPatterns, decision } = getEvaluationContext(context);
      const { isCompleted, missingFields } = isProjectDetailsFilled(project);

      if (project) {
        if (!isCompleted && decision && decision.verdict === 'None') {
          console.log('RecommendationAgent -> generateFailedDecisionPrompt');
          return generateFailedDecisionPrompt(project, missingFields);
        } else if (isCompleted && antiPatterns && decision && decision.verdict !== 'None') {
          console.log('RecommendationAgent -> generateRecommendationPrompt');
          return generateRecommendationPrompt(project, antiPatterns, decision);
        }
      }
      return 'Skipping LLM due to missing data.';
    },
    afterAgentCallback: agentEndCallback,
    outputSchema: recommendationSchema,
    outputKey: RECOMMENDATION_KEY,
    disallowTransferToParent: true,
    disallowTransferToPeers: true,
  });

  return recommendationAgent;
}

These are the callback design patterns that the agent adopts. In the next section, I describe how to launch the agent to observe the logging messages in the terminal.

Environment Setup

services:
  mailhog:
    image: mailhog/mailhog
    container_name: mailhog
    ports:
      - '1025:1025' # SMTP port
      - '8025:8025' # HTTP (Web UI) port
    restart: always
    networks:
      - decision-tree-agent-network

networks:
  decision-tree-agent-network:

The SMTP server listens on port 1025, and the Web UI is accessible at http://localhost:8025.

docker compose up -d

Start MailHog in Docker.

Testing

Add scripts to package.json to build and start the ADK web interface.

  "scripts": {
    "prebuild": "rimraf dist",
    "build": "npx tsc --project tsconfig.json",
    "web": "npm run build && npx @google/adk-devtools web --host 127.0.0.1 dist/agent.js"
  },

Open a terminal and type npm run web to start the API server.
Open a new browser tab and type http://localhost:8000.
Paste the following text into the message box:

One of my favorite tech influencers just tweeted about a 'breakthrough in solid-state batteries.' Find which public company they might be referring to, check that company’s recent patent filings to see if it’s true, and then check their stock price to see if the market has already 'priced it in'.

Ensure the root agent executes and halts when the email agent terminates. The orchestrator and the project agent trigger the callback hooks to reset the session state, log performance metrics, validate session data, increment the validation attempts, and modify the response.

Similarly, both anti-patterns and decisions agents used the beforeAgentCallback and afterAgentCallback to log performance metrics. They also used the beforeModelCallback to validate session data before calling the LLM to generate a response. Moreover, the decision agent incremented the validation attempts and modify the status to FATAL_ERROR when the validation attempts exceeded or equal to maximum iterations in the afterToolCallback.

Similarly, both recommendation and merger agents used the beforeAgentCallback and afterAgentCallback to log performance metrics. The recommendation agent also used the beforeModelCallback to validate session data before calling the LLM to generate a recommendation.

Conclusion

The takeaway: follow the design patterns and best practices of callback at various stages of an agent to log performance metrics, reduce cost and latency, and modify response.

Resources

Migrating to Nano Banana 2: Enhancing Your Angular Firebase AI App

Connie Leung — Tue, 31 Mar 2026 23:17:59 +0000

Migrating to Nano Banana 2: Enhancing Your Angular Firebase AI App

In the rapidly evolving world of AI, staying up-to-date with the latest models is crucial for performance and feature accessibility. In this guide, we’ll walk through the process of migrating an Angular application using Firebase AI Logic to the new Nano Banana 2 model. We will also explore how to implement advanced features like "Thinking Levels," custom aspect ratios, and the use of experimental Angular Signal Forms.

Introduction

Nano Banana 2 (often associated with the Gemini 3.1 Flash series) brings significant improvements to image and content generation. Specifically, it introduces:

New aspect ratios (8:1 and 1:8).
High-resolution options (512px and up).
Thinking Levels: Allowing you to toggle between "Minimal" and "High" reasoning for complex generation tasks.

Let’s dive into the architecture and the code changes required to make this transition.

Architecture Overview

The application follows a modern serverless AI architecture, leveraging the Google Cloud and Firebase ecosystem.

Diagram Description

Angular Frontend: Handles the user interface, prompt input, and displays generated media. It uses Signals for reactive state management.
Firebase Remote Config: Acts as the central brain for model parameters, allowing you to swap model names or reasoning levels without redeploying code.
Firebase AI Logic (Vertex AI for Firebase SDK): The bridge between the frontend and the Gemini/Nano Banana models.
Media Generation: Images generated via Nano Banana 2 can be piped into other models, such as Veo 3.1, to generate high-quality video content.

Architectural Advice

For a robust AI application, avoid hardcoding model parameters in your TypeScript files. Use Firebase Remote Config to manage modelName, thinkingLevel, and location. This allows for A/B testing and instant updates if a newer model version becomes available.

Step 1: Update Firebase Remote Config

The first step is updating your configuration in the Firebase Console. You need to point the application to the new model and define the thinking levels.

Key Changes:

geminiImageModelName: Update to gemini-3.1-flash-image-preview.
thinkingLevel: Set to HIGH or MINIMAL.

Remote Config JSON (Snippet)

{
  "geminiImageModelName": "gemini-3.1-flash-image-preview",
  "thinkingLevel": "HIGH",
  "vertexAlLocation": "us-central1"
}

After updating the console, download the remotemconfig.json and sync it with your local project's default template.

Step 2: Angular Service Configuration

In the Angular app, you need to map the Remote Config values to the AI Logic generation parameters.

`firebase.provider.ts`

We update the generation config to include the thinkingLevel. Since this is an enum in the SDK, we map the string from Remote Config.

function getGenerativeAIImageModel(configService: ConfigService) {
  const { firebaseApp, remoteConfig } = configService.firebaseObjects;

  const modelName = getValue(remoteConfig, 'geminiImageModelName').asString();
  const rawThinkingLevel = getValue(remoteConfig, 'thinkingLevel').asString();

  // Map string to ThinkingLevel enum
  const thinkingLevel = rawThinkingLevel as keyof typeof ThinkingLevel;

  const modelParams: ModelParams = {
    model: modelName,
    generationConfig: {
      candidateCount: 1,
      thinkingConfig: {
        thinkingLevel: ThinkingLevel[thinkingLevel]
      }
    }
  };

  return getGenerativeModel(getAI(firebaseApp), modelParams);
}

Step 3: Prompt Workaround for Aspect Ratio & Resolution

Currently, some Firebase AI Logic SDKs might not support aspectRatio or resolution as first-class parameters in the generationConfig. A reliable workaround is appending these instructions directly to the prompt.

`prompt-form.component.ts`

onGenerateClick(event: Event) {
  event.preventDefault();

  const inputValue = this.promptForm.value.prompt;
  const aspectRatio = this.genConfigValues().aspectRatio;
  const resolution = this.genConfigValues().resolution;

  let trimmedPrompt = inputValue;

  // Workaround: Append instructions to the prompt
  if (aspectRatio) {
    trimmedPrompt = `${trimmedPrompt} \nApply this aspect ratio to the image: ${aspectRatio}`;
  }

  if (resolution) {
    trimmedPrompt = `${trimmedPrompt} \nApply this resolution to the image: ${resolution}`;
  }

  this.generate.emit({ trimmedPrompt, inputValue });
}

Step 4: UI with Experimental Signal Forms

To handle the dropdown selections for Aspect Ratio and Resolution, the app uses Angular's experimental Signal Forms. This provides a more reactive approach compared to standard Reactive Forms.

`generate-options-form.component.html`

<form [formRoot]="generateConfigForm" class="flex flex-col gap-4">
  <label>Aspect Ratio</label>
  <select [formField]="aspectRatio">
    @for (opt of aspectRatios; track opt) {
      <option [value]="opt">{{ opt }}</option>
    }
  </select>

  <label>Resolution</label>
  <select [formField]="resolution">
    @for (opt of resolutions; track opt) {
      <option [value]="opt">{{ opt }}</option>
    }
  </select>
</form>

Conclusion

Migrating to Nano Banana 2 allows developers to tap into higher-quality image generation and nuanced reasoning levels. By combining Angular Signals with Firebase Remote Config, you create a flexible architecture that can adapt to the next model update with minimal code changes.

Happy coding!

Resources

Stop Wasting Tokens: Building Deterministic Custom Agents with Google ADK [GDE]

Connie Leung — Sun, 29 Mar 2026 16:35:36 +0000

In the world of AI orchestration, it's tempting to use a Large Language Model (LLM) for every step of a workflow. However, as applications scale, the "LLM-first" approach can introduce unnecessary latency, costs, and unpredictability. The Google Agent Development Kit (ADK) provides a powerful alternative: the BaseAgent.

This post explores how to create a custom, programmatic agent—specifically an Email Agent—that handles deterministic tasks while still participating seamlessly in an AI-driven ecosystem.

Why is the Email Agent a Custom Agent?

The agent is 100% deterministic and relies on external APIs to format text and send emails.

It uses the marked library to convert a Markdown string into HTML and the nodemailer library to send mail to an SMTP server.

On the contrary, if I created a LlmAgent, the instruction, tool, and structured output would introduce LLM latency, input, and output tokens.

The Benefits of Programmatic Execution

Using a custom agent instead of an LLM-wrapped prompt offers several critical advantages:

Agent Type	Llm Agent	Custom Agent
Latency	Time-intensive (generation delay)	Low-latency (direct execution)
Cost	Pay for tokens consumed	Zero cost
Determinism	Probabilistic and hallucinate when agent does not follow the instruction or the instruction is unclear	100% deterministic and does not require AI capabilities
Testability	Incorrect result when the model hallucinates	Test results are predictable
Integration	Delegate responsibility to tools	Call libraries directly in the class

Demo Overview

The EmailAgent is the final subagent of the Decision Tree multi-agent system. The Decision Tree agent evaluates the feasibility of applying an agent architecture to a project. It starts with a project description, determines whether an agent architecture should be used, generates a recommendation, returns the recommendation to the client, and sends an email to the administrator as a side effect.

The EmailAgent retrieves the recommendation and summary from the session state, returns the recommendation text to the frontend, and sends both to the administrator's email address.

The agent does not require an LLM or tool use; therefore, an LlmAgent is overkill. My solution was to use a custom agent to handle email to save time and cost.

Architecture

The project, anti-patterns, decision, recommendation, and synthesis agents are LLM agents. These agents require Gemini to reason and generate text responses.

The Audit Trail, Cloud Storage, and Email agents integrate with external APIs or resources that trigger deterministic actions.

Prerequisites

Requirements

TypeScript 5.9.3 or later
Node.js 24.13.0 or later
npm 11.8.0 or later
Docker (For running MailHog locally)
Google ADK (For building the custom agent)
Gemini in Vertex AI (to call the model in LLM agents, although not required for the custom agent)

Note: I used Gemini in Vertex AI for authentication due to regional availability. Gemini is blocked in Hong Kong currently, so I used Gemini in Vertex AI instead.

Install npm dependencies

npm i --save-exact @google/adk
npm i --save-dev --save-exact @google/adk-devtools
npm i --save-exact nodemailer
npm i --save-dev --save-exact @types/nodemailer rimraf
npm i --save-exact marked
npm i --save-exact zod

I installed the required dependencies for building ADK agents, converting Markdown string to HTML and sending emails to MailHog in local testing.

Pinned dependencies ensure the versioning is the same in development and production environments for enterprise-level applications.

Environment variables

Copy .env.example to .env and fill in the credentials:

GEMINI_MODEL_NAME="gemini-3-flash-preview"
GOOGLE_CLOUD_PROJECT="<Google Cloud Project ID>"
GOOGLE_CLOUD_LOCATION="global" 
GOOGLE_GENAI_USE_VERTEXAI=TRUE

# SMTP Settings (MailHog)
SMTP_HOST="localhost"
SMTP_PORT=1025
SMTP_USER=""
SMTP_PASS=""
SMTP_FROM="no-reply@test.local"
ADMIN_EMAIL="admin@test.local"

SMTP_HOST, SMTP_PORT, SMTP_USER, SMTP_PASS, are required to set up MailHog for local email testing.

SMTP_FROM is a from email address that can be any string in local testing.

Environment Setup

I pulled the latest version of the MailHog docker image from Docker Hub and started it locally to receive test emails and display them in the Web UI. The docker-compose.yml file contains the setup configuration.

services:
  mailhog:
    image: mailhog/mailhog
    container_name: mailhog
    ports:
      - "1025:1025" # SMTP port
      - "8025:8025" # HTTP (Web UI) port
    restart: always
    networks:
      - decision-tree-agent-network

networks:
  decision-tree-agent-network:

The SMTP port is 1025 and the URL of the Web UI port is http://localhost:8025.

docker compose up -d

Start MailHog in Docker.

ADK Multi-Agent Architecture

process.loadEnvFile();

const model = process.env.GEMINI_MODEL_NAME || '';
if (!model) {
  throw new Error('GEMINI_MODEL_NAME is not set');
}

The model variable specifies the gemini-3-flash-preview model. If model is undefined, an error is thrown.

The demo uses Node 20+, so the process.loadEnvFile() is provided to load variables from the environment file. Otherwise, developers should consider using dotenv to load environment variables.

export const RECOMMENDATION_KEY = 'recommendation';
export const MERGED_RESULTS_KEY = 'mergedResults';
export const PROJECT_DESCRIPTION_KEY = 'project_description';

import { FunctionTool, LlmAgent, SequentialAgent } from '@google/adk';
import { z } from 'zod';
import { initWorkflowAgent } from './init.js';
import {
  MERGED_RESULTS_KEY,
  PROJECT_DESCRIPTION_KEY,
  VALIDATION_ATTEMPTS_KEY,
} from './sub-agents/output-keys.js';

const prepareEvaluationTool = new FunctionTool({
  name: 'prepare_evaluation',
  parameters: z.object({
    description: z.string()
  }),
  execute: async ({ description }, context) => {
    if (!context || !context.state) {
      return { status: 'ERROR', message: 'No session state found.' };
    }

    const state = context.state;

    // Clear all previous evaluation data
    state.set(MERGED_RESULTS_KEY, null);
    state.set(VALIDATION_ATTEMPTS_KEY, 0);

    // Set the new description 
    state.set(PROJECT_DESCRIPTION_KEY, description);

    return { status: 'SUCCESS', message: 'State reset and description updated.' };
  },
});

export const SequentialEvaluationAgent = new SequentialAgent({
  name: 'SequentialEvaluationAgent',
  subAgents: initWorkflowAgent(model),
});

prepareEvaluationTool is a tool that reset the variables in the session state before the agent starts the project evaluation.

SequentialEvaluationAgent is a sequential agent that consists of subagents.

export function initWorkflowAgent(model: string) {
  return [
    createMergerAgent(model),
    createEmailAgent(),
  ];
}

The initWorkflowAgent calls createMergerAgent and createEmailAgent factory functions to return the merger agent and the email agent. The merger agent is a LlmAgent because it requires Gemini to generate a summary whereas the email agent is a custom agent that does not require Gemini.

export const rootAgent = new LlmAgent({
  name: 'ProjectEvaluationAgent',
  model,
  instruction: `... instruction...`,
  tools: [prepareEvaluationTool],
  subAgents: [SequentialEvaluationAgent],
});

The rootAgent is an orchestrator that routes the project description to SequentialEvaluationAgent to evaluate.

Email Agent Factory Function

export type SmtpConfig = {
  host: string;
  port: number;
  user?: string;
  pass?: string;
  from: string;
  email: string;
};

export function createEmailAgent(): BaseAgent {
  const email = process.env.ADMIN_EMAIL || 'admin@test.local';
  const host = process.env.SMTP_HOST || 'localhost';
  const port = parseInt(process.env.SMTP_PORT || '1025');
  const user = process.env.SMTP_USER || '';
  const pass = process.env.SMTP_PASS || '';
  const from = process.env.SMTP_FROM || 'no-reply@test.local';

  const smtpConfig: SmtpConfig = {
    host,
    port,
    user,
    pass,
    from,
    email,
  };

  return new EmailAgent(smtpConfig);
}

The createEmailAgent function retrieves the host, port, user, password, sender email, and administrator email from the environment variables to construct an instance of SmtpConfig.

Pass the SmtpConfig object to the constructor of the EmailAgent class to construct a custom agent. Finally, the function returns the email agent to become the last subagent of SequentialEvaluationAgent.

Implementing the Email Agent

Let’s look at the core structure of the EmailAgent.

1. The Class Definition and State

The EmailAgent has a readonly SmtpConfig instance member that is initialized in the constructor. The constructor calls super to initialize its name and description.

class EmailAgent extends BaseAgent {
  readonly smtpConfig: SmtpConfig;

  constructor(smtpConfig: SmtpConfig) {
    super({
      name: 'EmailAgent',
      description: 'Send a recommendation and summary email to the administrator.',
    });

    this.smtpConfig = smtpConfig;
  }
}

2. Accessing the Context

import { z } from 'zod';

export const recommendationSchema = z.object({
  text: z.string(),
});

export type Recommendation = z.infer<typeof recommendationSchema>;

export const mergerSchema = z.object({
  summary: z.string(),
});

export type Merger = z.infer<typeof mergerSchema>;

The text property stores the recommendation of the Recommendation type. Similarly, the summary property of the Merger type stores the summary.

export function getEvaluationContext(context: ReadonlyContext | undefined) {
  if (!context || !context.state) {
    return {
      recommendation: null,
    };
  }

  const state = context.state;
  return {
    recommendation: state.get<Recommendation>(RECOMMENDATION_KEY) ?? null,
  };
}

export function getMergerContext(context: ReadonlyContext | undefined) {
  if (!context || !context.state) {
    return {
      merger: null,
    };
  }

  const state = context.state;
  return {
    merger: state.get<Merger>(MERGED_RESULTS_KEY) ?? null,
  };
}

getEvaluationContext and getMergerContext are helper functions to obtain recommendation and summary from the session state.

3. Implement the Concrete Methods

The EmailAgent extends the BaseAgent and all subclasses must implement two abstract methods: runAsyncImpl and runLiveImpl.

  protected async *runAsyncImpl(context: InvocationContext): AsyncGenerator<Event, void, void> {
    for await (const event of this.runLiveImpl(context)) {
      yield event;
    }
  }

Implement runAsyncImpl by wrapping runLiveImpl. It invokes a for-await loop to iterate this.runLiveImpl(context) and yields each event.

protected async *runLiveImpl(context: InvocationContext): AsyncGenerator<Event, void, void> {
    const readonlyCtx = new ReadonlyContext(context);
    const { merger } = getMergerContext(readonlyCtx);
    const { recommendation } = getEvaluationContext(readonlyCtx);
    const recommendationText = recommendation?.text || 'No recommendation available.';

    const emit = (status: 'success' | 'error', author: string) =>
      createEmailStatusEvent({
        author,
        context,
        status,
        recommendationText,
      });

    if (!merger) {
      yield emit('error', this.name);
      return;
    }

    try {
      const emailContent = `${recommendation?.text || ''}\n\n## Summary\n\n${merger.summary}`;
      await sendEmail(this.smtpConfig, 'Project Evaluation Results', emailContent);
      yield emit('success', this.name);
    } catch (e) {
      console.error(e);
      yield emit('error', this.name);
    }
  }

The runLiveImpl method handles three cases: an undefined merger, a successful email dispatch, and an email dispatch failure. When email dispatch is successful, the inner emit arrow function yields a success event. Otherwise, the catch block logs the error and yields an error event.

import { marked } from 'marked';
import nodemailer from 'nodemailer';

export async function sendEmail(smtpConfig: SmtpConfig, subject: string, text: string) {
  const { host, port, user, pass, from, email: to } = smtpConfig;
  const transporter = nodemailer.createTransport({
    host,
    port,
    auth: user && pass ? { user, pass } : undefined,
    secure: false,
  });

  const html = await marked.parse(text);
  const mailOptions = {
    from,
    to,
    subject,
    text,
    html,
  };
  return transporter.sendMail(mailOptions);
}

The sendEmail function uses the nodemailer to create a transporter using the SMTP host, port, user, and password. The marked library parses the Markdown text and converts it to HTML. The transporter has a sendMail method that accepts mail options to send an email to the to email address in both text and html formats.

4. Returning Data in the Output

Next, implement createEmailStatusEvent to return an email status event.

export type EmailStatusOptions = {
  author: string;
  context: InvocationContext;
  status: 'success' | 'error';
  recommendationText: string;
};

export function createEmailStatusEvent(options: EmailStatusOptions): Event {
  return createEvent({
    invocationId: options.context.invocationId,
    author: options.author,
    branch: options.context.branch || '',
    content: {
      role: 'model',
      parts: [
        {
          text: JSON.stringify({
            status: options.status,
            recommendationText: options.recommendationText,
            sessionId: options.context.session.id,
            invocationId: options.context.invocationId,
          }),
        },
      ],
    },
  });
}

The EmailStatusOptions provide the invocation ID, branch, author (the agent name), status, and recommendation text.

The createEmailStatusEvent function reuses the createEvent function to return the invocation ID, branch, author, and a JSON-stringified object consisting of the status, session ID, invocation ID, and recommendation text.

The client can examine the status to determine whether or not to display the recommendation text.

Returning both session and invocation IDs to the client is recommended. The client can reuse the session ID to call the agent in a conversational flow to seek evaluations for additional project descriptions. The invocation ID allows querying an error-tracking service like Datadog to identify the server status.

Testing

Add scripts to package.json to build and start the ADK web interface.

  "scripts": {
    "prebuild": "rimraf dist",
    "build": "npx tsc --project tsconfig.json",
    "web": "npm run build && npx @google/adk-devtools web --host 127.0.0.1 dist/agent.js"
  },

Open a terminal and type npm run web to start the API server.
Open a new browser tab and type http://localhost:8000.
Paste the following into the message box:

One of my favorite tech influencers just tweeted about a 'breakthrough in solid-state batteries.' Find which public company they might be referring to, check that company’s recent patent filings to see if it’s true, and then check their stock price to see if the market has already 'priced it in'.

Ensure the root agent executes and halts when the email agent terminates.

Open another tab and navigate to http://localhost:8025 to open the MailHog web UI

The MailHog Web UI displays the recommendation and summary that summarizes the decision, any clear anti-patterns, and the URL of the cloud storage.

Conclusion

Custom agents are essential components of an ADK application. While ADK offers sequential, parallel, loop, and LLM agents, these agents may not meet the requirements in some occasions. BaseAgent is the generic Agent class that enables developers to write their logic and design their agentic workflow.

The takeaway: avoid using an LLM-based agent for tasks that do not require probabilistic reasoning, tool calling, or response generation.

Resources

Automating Technical Blog Localization with Gemini CLI Agent Skills [GDE]

Connie Leung — Thu, 19 Feb 2026 06:40:45 +0000

The Use Case: Scaling Content to Global Tech Communities

When content writers write English blog posts, they often want to distribute them to different tech communities to share knowledge, increase impact, and reach. However, many readers in these communities do not read, write, or speak English. When the blog post is posted in these localized, language-specific Facebook Groups, their followers do not read it and miss out on valuable insights.

I often write English blog posts for global tech communities, but I encountered an obstacle when sharing them in localized groups, like Taiwan's Traditional Chinese Facebook communities. My readers there missed out because of the language barrier, and I lacked the technical vocabulary to translate them manually. My solution? I built a custom Agent Skill for the Gemini CLI.

The skill identifies the target language in the user prompt and loads the terminology files from the target language folder to perform the translation. Then, the skill runs a Node.js script to verify that the source and target URLs match and that the Alt-text and link text have been translated to improve accessibility and boost SEO.

This guide demonstrates how to build the translation agent skill for the Gemini CLI.

Prerequisites

Before we begin, ensure you have the following:

Install the gemini-cli.
Generate a Google AI Studio API Key or a Vertex AI Credential.
Ensure you have access to a terminal (macOS, Linux, or WSL).
Install Node.js v18+ to run the script.

Setting Up the Environment

Install the Gemini CLI globally.

npm install -g @google/gemini-cli

To access your extension and the API key globally, create a .env file in the ~/.gemini global workspace, and add the API key to the .env file as follows:

# Your Google AI Studio API Key
GEMINI_API_KEY=<Gemini API Key>

We use the ~/.gemini path, which is standard for macOS/Linux systems. This guide assumes familiarity with a Unix-like environment. Windows (PowerShell/CMD) users might need to adjust paths or commands slightly.

Gemini CLI automatically loads variables from the first .env file it finds, searching up from the current directory, then in ~/.gemini/.env or ~/.env.

When the CLI loads the GEMINI_API_KEY from the .gemini/.env file, it works in projects automatically. Moreover, it acts as a single source of truth for key rotation. When the key expires, users only need to update a single file, rather than locating multiple copies scattered across projects.

Source Code

The full source code for this skill is available in the Technical Localization Expert Agent Skill, however, the following section examines how the key components work.

The Architecture: Orchestrating a Translation Agent Skill

A user enters a prompt in the Gemini CLI to use the technical-localization-expert skill to translate an English blog to Traditional Chinese.
The CLI activates the skill, loads the Taiwanese technical terminology files from the references folder.
The skill directs the CLI to execute the Linux cat command to read the source blog post.
The skill builds a terminology mapping and injects it, along with the source text, into the translation prompt context to generate the translated text.
The Node.js link validation script in the scripts folder validates the links of the translated text are valid.
The skill saves the translated text to a markdown file in the chinese-blogs folder.
The audit report lists the localization process's steps and each step produces the expected outcome.

Structure of an Agent Skill

An Agent Skill is a self-contained directory with a required SKILL.md Markdown file, and optional folders such as scripts, references, and assets. When the directory is placed in the skills folder of the global .gemini folder, it is available after restart.

my-skill/
├── SKILL.md          # Required: instructions + metadata
├── scripts/          # Optional: executable code
├── references/       # Optional: documentation
└── assets/           # Optional: templates, resources

In our translate blog post skill, the top folder name (my-skill) is named technical-localization-expert to match the skill name to avoid confusion and facilitate discovery.

The references folder contains terminology files grouped under terminology-en-[code] subfolders where [code] is the target language code. The markdown file has a mapping table to map the English terms to the target language. For example, this table maps common AI terms from English to Traditional Chinese.

# AI

| English | zh-TW (Taiwan Standard) | Notes |
| :--- | :--- | :--- |
| Artificial Intelligence | 人工智慧 | Never use 人工智能 |
| Training | 訓練 | |
| Inference | 推論 | Not 推理 |
| Fine-tuning | 微調 | |
| Model | 模型 | |
| Hyperparameter | 超參數 | |
| Prompt | 提示詞 | |
| Embedding | 嵌入 | |
| Large Language Model | 大型語言模型 | LLM |

Finally, the scripts folder contains a Node.js script to perform validations on the assets and links in the translated text.

Let's create the folder structure of the agent skill.

Create a Basic Agent Skill

The following commands create all the folders and empty files for the translate blog post skill.

cd ~    # Switch to the home directory
mkdir technical-localization-expert
touch SKILL.md

# Create the references folder, subfolder, and terminology files
mkdir technical-localization-expert/references
mkdir technical-localization-expert/references/terminology-en-zhtw

mkdir technical-localization-expert/references/terminology-en-zhtw/ai.md
mkdir technical-localization-expert/references/terminology-en-zhtw/backend.md
mkdir technical-localization-expert/references/terminology-en-zhtw/cloud.md
mkdir technical-localization-expert/references/terminology-en-zhtw/frontend.md
mkdir technical-localization-expert/references/terminology-en-zhtw/general.md
mkdir technical-localization-expert/references/terminology-en-zhtw/internet.md

# Create the scripts folder and an empty Node.js file
mkdir technical-localization-expert/scripts
touch technical-localization-expert/scripts/check-links.js

The following section analyzes each component of the skill in detail.

The Reference Files

For maintainability and scalability, the files are placed under terminology-en-zhtw for Traditional Chinese. The terminology-en-zhtw directory contains ai.md, frontend.md, backend.md, web.md, cloud.md, general.md, and internet.md Markdown files to keep the standard Taiwan terminology.

Extend this modular design to Simplified Chinese by simply adding a terminology-en-zhcn folder. This ensures the agent uses correct regional terms, such as 軟件 (Simplified) instead of 軟體 (Traditional).

We can find the markdown files in terminology-en-zhtw and copy the markdown tables to our empty files.

After the CLI activates the skill, it loads these files to build a single map of English-to-Traditional-Chinese technical terms. Moreover, it injects the map and the source blog post into the user prompt to generate the translated Traditional Chinese blog post and saves it to a destination folder.

The validation script verifies URLs, Alt-text, and links before we finalize the post for publication.

Automated Asset Validation (The Node.js Script)

The check-links.js script validates the original and translated blog posts. While Gemini is powerful, LLMs can occasionally hallucinate URLs or accidentally translate technical paths. This script provides a deterministic check to ensure the blog remains functional.

First, the script ensures the Alt-text and link texts are translated to improve accessibility.

assetRegex is a regular expression to match the images with Alt-text and URLs with link text. The extract function extracts the assets to verify the texts are not translated by accident.

const assetRegex = /(!?)\[([^\]]*)\]\(([^)]+)\)/g;

const extract = (content) => {
  const results = [];
  let match;
  while ((match = assetRegex.exec(content)) !== null) {
    results.push({
      isImage: match[1] === '!',
      text: match[2].trim(),
      url: match[3].trim()
    });
  }
  return results;
};

const sourceAssets = extract(sourceContent);
const targetAssets = extract(targetContent);

When the text is neither in a single-backtick block nor a triple-backticks block, the source and target texts should not be the same. If they are the same, an error occurs, and an error message is pushed to the errors array.

const isCode = src.text.startsWith('`') && src.text.endsWith('`');
if (src.text && src.text === trg.text && !isCode) {
  errors.push(`Untranslated Text: Asset #${i+1} still says "${src.text}"`);
}

If the target URL is a relative path, it should not contain non-English characters. This is valid for ASCII, but some valid URLs might contain encoded UTF-8 characters. Please ensure the paths in the blog posts remain ASCII.

// For relative paths, we ensure they aren't accidentally translated into words
// (e.g., ../assets/ changed to ../activos/)
if (trg.url.match(/[^\x00-\x7F]/)) { 
    errors.push(`Relative Path Error: Path "${trg.url}" contains non-English characters.`);
}

The script prints the error messages if the errors array is not empty. Upon successful validation, the script displays a success message.

if (errors.length > 0) {
console.log('❌ Integrity & Translation Errors:');
errors.forEach(err => console.log(`  - ${err}`));
} else {
console.log('✅ Asset integrity and translation checks passed.');
}

When the externalLinksToCheck array is not empty, the target blog post contains external URLs that require connectivity checking. The validateConnectivity makes at least one fetch call on the target URL and waits for the response result. When the response is not ok, the script detects broken links and logs the error to the console.

async function validateConnectivity(urls) {
  const results = await Promise.all(urls.map(async (url) => {
    try {
      let response = await fetch(url, { method: 'HEAD', signal: AbortSignal.timeout(5000) });
      if (!response.ok) response = await fetch(url, { method: 'GET', signal: AbortSignal.timeout(5000) });
      return { url, ok: response.ok, status: response.status };
    } catch (e) {
      return { url, ok: false, status: 'Error/Timeout' };
    }
  }));

  results.forEach(res => {
    if (!res.ok) console.log(`  ❌ [${res.status}] ${res.url}`);
  });
}

if (src.url.startsWith('http')) {
  if (src.url !== trg.url) {
    errors.push(`URL Mismatch: Source "${src.url}" vs Target "${trg.url}"`);
  }
  externalLinksToCheck.push(trg.url);
}

if (externalLinksToCheck.length > 0) {
  console.log(`\nChecking connectivity for ${externalLinksToCheck.length} external links...`);
  await validateConnectivity(externalLinksToCheck);
}

We can find the script in the scripts folder.

Next, make the script executable using the Linux chmod command in a macOS/Linux environment.

chmod a+x check-links.js

The Specification

The skill specification is defined in the SKILL.md file. The markdown file starts with the metadata. The metadata has the skill name and description.

Gemini CLI loads the skill name and description at first. When the description matches the keywords in a user prompt, the CLI activates the skill and loads the entire file in the context.

---
name: technical-localization-expert
description: A high-precision localization engine that scans categorized terminology folders in `references/` to ensure domain-specific accuracy. Optimized for English-to-Global technical content with strict code-asset protection.
---

The next few sections are the persona and the logic to perform blog translation.
Category 1 asks Gemini to validate the source is English.

Category 2 asks the model to scan the user prompt to identify the target language (Traditional Chinese) and derives the language code (zhtw). The skill finds references/terminology-en-zhtw folder and loads all the markdown files in it.

# Technical Localization Expert

## PERSONA

You are a Senior Technical Translator and Localization Specialist. You specialize in converting developer-focused English content into various target languages. You prioritize technical accuracy, character set integrity, and the absolute preservation of all technical assets.

## LINGUISTIC & MECHANICAL LOGIC

1. **Category 1: Source Language Validation**
    - **Rule:** This skill is strictly optimized for translating **English source text**.
    - **Action:** Analyze the source content before proceeding.
    - **Stop Condition:** If the source language is NOT English, stop processing and notify the user: "⚠️ Notification: This localization engine requires English source text to ensure terminology-to-glossary consistency."

2. **Category 2: Modular Glossary Scanning & Fallback**
    - **Step 1:** Identify the Target Language Code (e.g., Traditional Chinese -> `zhtw`, Spanish -> `es`, Brazilian Portuguese -> `ptbr`).
    - **Step 2:** Locate the folder: `references/terminology-en-[code]/`.
    - **Step 3 (Success):** If the folder exists, read all Markdown files within it (e.g., `cloud.md`, `frontend.md`, `general.md`) to build a comprehensive terminology map.
    - **Step 4 (Fallback):** If the folder does NOT exist, notify the user: "Notice: No custom glossary folder found for [code]. Proceeding with standard regional technical terminology." Revert to internal high-precision standards for that locale.

Categories 3, 4, 5, and 6 are instructions ensuring that the model understands what to do and what not to do. For example, the Alt-text and link texts must be translated to the target language whereas relative paths should be untranslated. Moreover, any text, code, and variables in the single-backtick and triple-backtick blocks should not be translated.

3. **Category 3: Immutable Code Assets (Zero-Touch Policy)**
    - **Hard Rule:** Any text wrapped in single backticks or triple backticks is a **Literal String Block**.
    - **Preservation Instructions:**
        1. **DO NOT** translate any text inside, including code logic, variable names, or comments.
        2. **DO NOT** modify the casing, indentation, or spacing.
        3. **DO NOT** interpret escape characters. If the source contains a backslash followed by a letter (such as n, t, or r), you **MUST** output those literal characters. Do not convert them into actual newlines or tabs.
        4. **Even if the block contains English instructions or prose, it must be treated as raw, immutable data.**
    - **Verification:** Before outputting, check that every character within the backticks matches the source 1:1.

4. **Category 4: Image & Link Asset Protection & Path Adjustment**
    - **External Links:** For `(https://...)`, translate the `Link Text` but leave the `(URL)` verbatim.
    - **Absolute Internal Paths:** For paths starting with `/` (e.g., `/assets/images/logo.png`), leave the path verbatim as it is root-relative.
    - **Relative Paths (Strategy 3 - Path Re-calculation):**
        1. If the target localization file is stored at a different directory depth than the English source (e.g., moving from `blog/post.md` to `zh/blog/post.md`), you **must** recalculate the relative path.
        2. Adjust the number of "upward" steps (e.g., changing `../assets/` to `../../assets/`) to ensure the link remains functional from the new file location.
    - **Alt-Text:** Always translate the `Alt Text` for images and `Link Text` for hyperlinks to the target language, while applying the rules above to the `(URL)` portion.

5. **Category 5: Regional Standards & zh-TW Integrity**
    - **Traditional Chinese (zh-TW):** Strictly use Traditional Chinese characters. Use Taiwan-specific industry standards (e.g., Software -> 軟體, Instance -> 執行個體, Project -> 專案). Ensure zero "bleed" from Simplified Chinese vocabulary.
    - **General:** For any target language, use the professional IT vocabulary of that specific region. Avoid literal "machine" translations.

6. **Category 6: Formatting & Raw String Integrity**
    - **Markdown:** Maintain all Markdown formatting integrity (#, **, *, -, >).
    - **Escape Characters:** Treat code as **raw text**. Do not render `\n` as a physical newline within code blocks or backticks; output the literal characters.
    - **Tone:** Professional, technical "developer-to-developer" tone.
    - **Address:** Use the polite/formal form of address in the target language (e.g., "您" for Chinese, "Usted" for Spanish).

Category 7 specifies the order of execution so that the English blog is translated systematically and with high accuracy.

Category 8 validates the assets to ensure external links are not broken. Only the Alt-text and link texts are translated, and relative paths are excluded.

7. **Category 7: Order of Execution**
    1. **Validate:** Confirm the source is English. (Stop if not).
    2. **Map:** Identify the target ISO code.
    3. **Scan:** Look for the folder `references/terminology-en-[code]/`.
       - **If folder exists:** Ingest all modular glossary files.
       - **If folder missing:** Trigger Fallback Notification and use internal standards.
    4. **Translate:** Execute the translation while strictly protecting all backticks, URLs, and image paths.
    5. **Verify:** **Execute `./scripts/check-links.js`** on the result to identify broken or mismatched URLs.
    6. **Remediate:** If links are broken, fix them based on the English source and re-verify.
    7. **Deliver:** Provide the finalized, verified translation.

8. **Category 8: Quality Assurance & Link Integrity (Mandatory)**
    - **Verification Tool:** `./scripts/check-links.js`.
    - **Logic for Broken Links:**
        1. If the script reports a link as broken, compare it to the original English source.
        2. **Branch A (Translation Error):** If the URL in your translation differs from the source, fix it to match the source exactly and re-run the script.
        3. **Branch B (Source Error):** If the URL matches the source exactly but the script still reports it as broken, categorize it as a **"Pre-existing Source Error."**
        4. **Maximum Retries:** Do not attempt more than 2 correction cycles per link.
    - **Action Audit Report:** At the end of the output, provide a concise summary of the localization actions taken, including:
        - **Source Validation:** Confirmation that the source was English.
        - **Glossary Status:** Identification of the glossary folder used (e.g., `references/terminology-en-[code]/`) or if the "Fallback" was triggered.
        - **Asset Protection:** Confirmation that all text within single and triple backticks was preserved 1:1 as per Category 3.
        - **Link Integrity:** Results of the link verification (e.g., "All links verified" or "Pre-existing Source Errors identified").
    - **Final Action:** If errors persist after retries, deliver the translation and list all persistent errors within the Link Integrity section of the Audit Report.

Installing the Skill

With the specification defined, static terminology files, and validation script for the translate blog post skill in the technical-localization-expert directory. We can copy the directory to ~/.gemini/skills to make it available to the Gemini CLI.

The technical-localization-expert folder contains the following files: SKILL.md, references, and scripts folders.

technical-localization-expert
├── references
│   └── terminology-en-zhtw
│       ├── ai.md
│       ├── backend.md
│       ├── cloud.md
│       ├── frontend.md
│       ├── general.md
│       ├── internet.md
│       └── web.md
├── scripts
│   └── check-links.js
└── SKILL.md

Component	Description
SKILL markdown file	The specification of the Translation Agent Skill.
references	A folder that stores English-to-language code subfolders.
references/terminology-en-zhtw	This folder contains the English-to-Traditional Chinese terminology files.
terminology-en-zhtw/ai.md	This file maintains mapping of AI terms from English to Traditional Chinese.
terminology-en-zhtw/backend.md	This file maintains mapping of backend terms from English to Traditional Chinese.
terminology-en-zhtw/cloud.md	This file maintains mapping of cloud terms from English to Traditional Chinese.
terminology-en-zhtw/frontend.md	This file maintains mapping of frontend terms from English to Traditional Chinese.
terminology-en-zhtw/general.md	This file maintains mapping of general technology terms from English to Traditional Chinese.
terminology-en-zhtw/internet.md	This file maintains mapping of Internet terms from English to Traditional Chinese.
scripts	A folder that contains standalone scripts that the agent can run.
check-links.js	It is a Node.js script that tests the links in the translated text are reachable.

Copy the entire technical-localization-expert folder to ~/.gemini/skills. Then, list the content of ~/.gemini/skills/technical-localization-expert to verify.

mkdir -p ~/.gemini/skills/technical-localization-expert

cd ~/technical-localization-expert 
cp -r ./technical-localization-expert/* ~/.gemini/skills/technical-localization-expert
ls ~/.gemini/skills/technical-localization-expert

After copying the directory successfully, I restart the Gemini CLI to translate a blog post to Traditional Chinese.

The Workflow in Action

Change the directory to the folder where my blog post is located. Then, launch the Gemini CLI on the terminal

cd markdown
gemini

Enter the following command in the Gemini CLI:

use the technical-localization-expert skill to translate @mastering_live_sports_data_with_gemini_3.md to Traditional Chinese and write the translation to chinese-blogs/translated3.md

This user prompt instructs the CLI to use the skill to translate a blog post in the current directory to Traditional Chinese. The Traditional Chinese blog post is saved to chinese-blogs/translated3.md.

The CLI discovers the technical-localization-expert skill and prompts me to activate it.

Then, the CLI loads the terminology files from the references/terminology-en-zhtw folder and the skill executes the cat linux command to injest the English blog post into the model's context window dynamically.

Next, the skill extracts the technical terms from the files, injects them and the English blog text into the prompt to generate a Traditional Chinese blog. It activates the link validation script to extract and validate the links in the translated text.

The links in the translated text are valid and match the ones in the English blog post. The content is saved to chinese-blogs/translated3.md for further review and publication.

Finally, the skill prints the audit report to list all the completed steps of that localization process. The skill detects English content, applies Taiwan technology terminology during the translation, preserves any text in the triple backticks, and verifies the links in the translated text match the original and reachable.

Conclusion

By orchestrating the terminology files and the link validation script in an agent skill, I can now translate English posts into Traditional Chinese with a single skill in Gemini CLI. This ensures every technical term is accurate and every link works. This workflow allows me to publish confidently to Taiwan Facebook Groups and global platforms in the future.

How would you use Gemini CLI and Agent Skill to increase your impact in the communities globally?

Resources

Gemini 3 Agentic Vision: Moving Beyond Simple Image Prompting

Connie Leung — Sun, 15 Feb 2026 02:49:30 +0000

The Pain Point: The Invisible World of the "Granular" Masterpiece
When a postcard miniaturizes a traditional Chinese painting, the reduced scale obscures details. Consider a standard postcard depicting workers and horses pulling wagons in a crowded street during the Qing dynasty. Adults and children walk on a busy bridge while shops on both sides sell goods to pedestrians. For aging viewers, these granular details—a few millimeters wide—become an inaccessible blur.

Chinese art's "density" becomes a barrier, turning detailed scenes into indistinct shapes. When the eyes no longer resolve fine brushstrokes, the artwork loses its narrative.

The Solution: Agentic Vision as a Digital Docent
Gemini 3's Agentic Vision bridges the gap between the postcard's physical limitations and the original art's vast detail. It surpasses basic image understanding using a Think-Act-Observe loop to look where aging eyes struggle.

Given a command—like "Zoom in on the postcard to find the figures on the bridge"—the model becomes an active investigator:

Think: It scans the painting's complex composition, identifying the requested detail's coordinates amidst mountains, trees, buildings, and people.
Act: It autonomously writes and executes Python code to "crop and enhance," isolating the area with high accuracy.
Observe: It reviews the image, answers the prompt, and reasons before presenting the results to the user.

The Result: Visual Accessibility Improvement
Agentic Vision enables users with low vision to resolve fine details. Users can "walk through" a Chinese painting, moving from broad landscape strokes to fine details of a face or bird.

This tool does more than zoom; it makes art accessible by clarifying details, regardless of postcard size.

This post walks through the Colab notebook to inspect artwork postcards, answer visual questions, and identify the manufacturer.

Demo Overview

Let's look at how we can implement this 'Digital Docent' using Python and the Gemini SDK.

Run the Colab notebook to inspect the postcard front for visual answers, and the back for metadata like the title, dimensions, and manufacturer.

The public agentic_visions GitHub folder hosts the raw images used in this demo.

I load these URLs and use the Gemini 3 Flash preview model, the code execution tool, and a text prompt for image cropping and data extraction.

The model receives a prompt to zoom into the postcard and count the horses, mules, and donkeys in three groups. The model uses the code execution tool to find the answer. I parse the JSON response and iterate through the response's parts array to display text and images.

Although the prompt is a single sentence, the model repeats the think, act, and observe steps until it derives an answer.

Prerequisites

To run the Colab notebook, ensure you have:

A valid personal Gmail account to sign in to Google Colab
Vertex AI in Express Mode: I use Gemini via Vertex AI due to regional availability (Hong Kong), but these features function identically in the public Gemini API. To use this for free, create a Google Cloud project, enable the Vertex AI API, and opt into Express Mode using your Gmail account.
Google Cloud API key: Generate an API key in the Google Cloud Console. Configure this key by following the authentication guide.

Setup

Clone the notebook

Save a copy of the Colab notebook to Google Drive to run the test cases.

Client Secrets

Add a GOOGLE_CLOUD_API_KEY secret variable in the Colab notebook by clicking the Secrets (key icon) tab in the left sidebar.

GOOGLE_CLOUD_API_KEY=<GOOGLE CLOUD API KEY>

In the notebook, import userdata to get the Vertex AI Gemini API Key.

from google.colab import userdata

cloud_api_key = userdata.get('GOOGLE_CLOUD_API_KEY')

Code Execution

Install and Import Libraries

The notebook installs Python libraries and imports classes.

%pip install google-genai
%pip install matplotlib
%pip install os
%pip install dotenv
%pip install pydantic

from google import genai
from matplotlib import pyplot as plt
from google.genai import types
from pydantic import BaseModel, Field
from urllib.error import URLError
import requests
from PIL import Image
from io import BytesIO
import os
from typing import Union

Test Images

The front of the postcard: The National Museum of Taiwan exhibits Up the River.

I prompt the model to zoom in on the bottom of the postcard to count horses.

The back of the postcard: It contains metadata like the title and dimensions.

The prompt asks the model to zoom into the label to extract the title, dimensions, manufacturing date, manufacturing location, manufacturer, phone number, address, serial number, and price.

Define the Pydantic Models

I define a Pydantic model to store data.

ArtworkDiscovery stores the code execution's answer and reasoning.

class ArtworkDiscovery(BaseModel):
    answer: str = Field(..., description="The answer to the prompt")
    reasoning: str = Field(..., description="The reasoning behind the answer")

PostcardLabel stores the postcard's back label information.

class PostcardLabel(BaseModel):
    title: str = Field(..., description="The title of the artwork on the postcard")
    dimensions: str = Field(..., description="The dimensions of the postcard")
    manufacturing_date: str = Field(..., description="The manufacturing date of the postcard")
    manufacturing_location: str = Field(..., description="The manufacturing location of the postcard")
    brand: str = Field(..., description="The brand of the postcard")
    manufacturer: str = Field(..., description="The manufacturer of the postcard")
    phone_number: str = Field(..., description="The phone number of the postcard manufacturer")
    address: str = Field(..., description="The address of the postcard manufacturer")
    serial_number: str = Field(..., description="The serial number of the postcard")
    price: float = Field(..., description="The price of the postcard")
    reasoning: str = Field(..., description="The reasoning behind the answer")

The VisionTestCase class defines the image URL, text prompt, and response model type. The response model uses either the ArtworkDiscovery or PostcardLabel class.

class VisionTestCase(BaseModel):
    image_url: str = Field(..., description="The URL of the image to be analyzed")
    prompt: str = Field(..., description="The prompt to be answered based on the image")
    response_model: Union[type[ArtworkDiscovery], type[PostcardLabel]]  = Field(..., description="The Pydantic model that defines the expected response format")

Prompt Engineering

Agentic Vision struggles with counting. When I ask it to count horses or houses on the postcards, the answers deviate by +/- 5 from my count. I add a reasoning field to the Pydantic models to capture the logic. The field forces the model to induce Chain-of-Thought (CoT) processing, which significantly improves arithmetic and logic tasks like counting.

Create a Gemini client

I use the client to call the Gemini 3 Flash Preview model to execute code on the provided images, find the requested object's coordinates, review the cropped regions, and return the text and reasoning.

def create_vertexai_client():
    cloud_api_key = userdata.get('GOOGLE_CLOUD_API_KEY')
    if not cloud_api_key:
        raise ValueError("GOOGLE_CLOUD_API_KEY not found in .env file")

    # Configure the client with your API key
    client = genai.Client(
        vertexai=True,
        api_key=cloud_api_key,
    )

    return client

client = create_vertexai_client()

The Agentic Loop in Action

The curate_artwork_postcard function calls generate_content, prompting the model to crop images within an agentic loop to derive the final answer.

Please be clear that the loop occurs on the server-side within the generated_content calls. The Python client waits for the result while the model iterates.

Zoom into the Postcards

curate_artwork_postcard sends the image URL and prompt to Gemini 3 Flash Preview for a JSON answer. The configuration includes a tools array enabling code execution, along with settings for high thinking and media resolution.

tools=[types.Tool(code_execution=types.ToolCodeExecution())]
config=types.GenerateContentConfig(
    response_mime_type="application/json",
    response_json_schema=test_case.response_model.model_json_schema(),
    thinking_config=types.ThinkingConfig(
        thinking_level=types.ThinkingLevel.HIGH
    ),
    media_resolution=types.MediaResolution.MEDIA_RESOLUTION_HIGH,
    tools=tools, 
)

def curate_artwork_postcard(test_case: VisionTestCase) -> types.GenerateContentResponse:
    response = client.models.generate_content(
        model="gemini-3-flash-preview",
        contents=[
            types.Content(
                role="user",
                parts=[
                    types.Part.from_uri(file_uri=test_case.image_url),
                    types.Part(text=test_case.prompt),
                ]
            )
        ],
        config=config
    )

    return response

Display the Results

The print_parts function receives a response and iterates through the response.candidates[0].content.parts list to print the JSON object as a string, the executable code, the executable code result, and the cropped image.

def print_parts(response: types.GenerateContentResponse):
    for part in response.candidates[0].content.parts:
        if part.text is not None and part.text.strip():
            print("part.text -> ", part.text.strip())
        if part.executable_code is not None:
            print("part.executable_code -> ", part.executable_code)
        if part.code_execution_result is not None:
            print("part.code_execution_result -> ", part.code_execution_result)
        if part.as_image() is not None:
            # display() is a standard function in Jupyter/Colab notebooks
            display(Image.open(BytesIO(part.as_image().image_bytes)))

print_artwork_result calls curate_artwork_postcard. If the model defines response.parsed, the function validates it via model_validate to validate it and obtain the model instance. Otherwise, the function calls the model_validate_json method to validate response.text to obtain the model instance. print_artwork_result then calls print_parts to display the response parts and the result object.

clean_json_string is a helper that is a fallback for raw text responses.

def clean_json_string(raw_string):
    # Remove the markdown code blocks
    clean_str = raw_string.strip()
    if clean_str.startswith("```

json"):
        clean_str = clean_str[7:]
    if clean_str.endswith("

```"):
        clean_str = clean_str[:-3]
    return clean_str.strip()

def print_artwork_result(test_case: VisionTestCase):
    response = curate_artwork_postcard(test_case=test_case)

    if response.parsed:
        result = test_case.response_model.model_validate(response.parsed) 
    else:
        result = test_case.response_model.model_validate_json(
            clean_json_string(response.text)
        )

    print_parts(response=response)
    if isinstance(result, ArtworkDiscovery):
        print("Final Answer: ", result.answer, "\nReasoning: ", result.reasoning)
    elif isinstance(result, PostcardLabel):
        print("Title: ", result.title,
              "\nDimensions: ", result.dimensions, 
              "\nManufacturing Date: ", result.manufacturing_date, 
              "\nManufacturing Location: ", result.manufacturing_location, 
              "\nBrand: ", result.brand, 
              "\nManufacturer: ", result.manufacturer, 
              "\nPhone Number: ", result.phone_number, 
              "\nAddress: ", result.address, 
              "\nSerial Number: ", result.serial_number, 
              "\nPrice: ", result.price, 
              "\nReasoning: ", result.reasoning
        )

def print_test_cases(heading: str, cases: list[VisionTestCase]):
    print(heading)
    for test_case in cases:
        print_artwork_result(test_case=test_case)

Construct the Test Cases

I use show_the_postcard_front_and_back to construct test image URLs and display them. The load_image_from_url uses the libraries to load the image from an URL and display it.

def load_image_from_url(url: str):
    try:
        response = requests.get(url=url)
        img = Image.open(BytesIO(response.content))
        plt.imshow(img)
        plt.axis('off')
        plt.show()
    except requests.HTTPError as e:
        # e.code contains the status code (e.g., 404)
        if e.code == 404:
            print("Error: URL not found (404).")
        else:
            print(f"HTTP Error: {e.code}")

    except URLError:
        print(f"Error: The file at '{url}' was not found.")
    except Exception as e:
        print(f"An error occurred: {e}")

def show_the_postcard_front_and_back(front: str, back: str):
    agent_vision_base_url = "https://raw.githubusercontent.com/railsstudent/colab_images/refs/heads/main/agentic_visions"
    front_url = f"{agent_vision_base_url}/{front}"
    back_url = f"{agent_vision_base_url}/{back}"

    load_image_from_url(url=front_url)
    load_image_from_url(url=back_url)

front_url, back_url = show_the_postcard_front_and_back(front="up-the-river-front.jpg", back="up-the-river-back.jpg")

I ask the model to count animals at the bottom of up-the-river-front.jpg.

up_the_river_test_cases = [
    VisionTestCase(
        image_url=front_url,
        prompt="Zoom to bottom of the postcard to find the number of horses/donkeys/mules near the wagons",
        response_model=ArtworkDiscovery
    ),
]

print_test_cases(heading="Up the River test cases", cases=up_the_river_test_cases)

The following outputs appear:

executable_code generates Python code to load the image, crop parts (for example, bottom_view = get_crop(img, [730, 80, 920, 950])) and save them as PNG files (for example, bottom_view.png). code_execution_result displayed OUTCOME_OK and a null output. The model generates the get_crop function itself, not the developer.

When CodeExecution is enabled, the model automatically downloads the URL and renames it to f_https___raw.githubusercontent.com_railsstudent_colab_images...up_the_river_front.jpg into the local sandbox. The sandbox executes the generated code and the external assets to find the answer.

Note: Libraries like cv2 and PIL are pre-installed in the Gemini model's secure execution sandbox.

part.executable_code ->  code='import PIL.Image\nimport PIL.ImageDraw\n\n# Load the image to get dimensions\nimg = PIL.Image.open(\'f_https___raw.githubusercontent.com_railsstudent_colab_images_refs_heads_main_agentic_visions_up_the_river_front.jpg\')\nwidth, height = img.size\n\n# Identify the bottom area with wagons and animals\n# Wagon 1 area: bottom left/center\n# Wagon 2 area: bottom right/center\n# Far right animals\n# Let\'s crop the bottom third of the image first to see clearly\nbottom_strip = [700, 100, 950, 900] # [ymin, xmin, ymax, xmax] in normalized coordinates\n\n# Better crop: focusing on the animal teams\n# Wagon team 1 (left): approx [750, 100, 900, 400]\n# Wagon team 2 (center): approx [750, 400, 900, 700]\n# Far right: [780, 720, 880, 820]\n\ndef get_crop(img, box_norm):\n    ymin, xmin, ymax, xmax = box_norm\n    left = xmin * width / 1000\n    top = ymin * height / 1000\n    right = xmax * width / 1000\n    bottom = ymax * height / 1000\n    return img.crop((left, top, right, bottom))\n\nbottom_view = get_crop(img, [730, 80, 920, 950])\nbottom_view.save(\'bottom_view.png\')\n\n# Detailed crops for counting\nteam1_crop = get_crop(img, [750, 100, 900, 400])\nteam1_crop.save(\'team1.png\')\n\nteam2_crop = get_crop(img, [720, 400, 900, 720])\nteam2_crop.save(\'team2.png\')\n\nright_animals = get_crop(img, [780, 720, 880, 820])\nright_animals.save(\'right_animals.png\')\n\n# Output objects for verification\n# [{box_2d: [750, 150, 880, 360], label: "animal team 1"},\n#  {box_2d: [750, 400, 885, 545], label: "animal team 2"},\n#  {box_2d: [795, 730, 861, 805], label: "right animals"}]\n' language=<Language.PYTHON: 'PYTHON'>
part.code_execution_result ->  outcome=<Outcome.OUTCOME_OK: 'OUTCOME_OK'> output=None

part.text displays the answer and reasoning. The model discovers 24 animals whereas I found 23. The reasoning field describes the process leading to the answer.

part.text ->  {
 "answer": "24",
 "reasoning": "By zooming into the bottom of the postcard, a large wagon can be seen being pulled by multiple teams of horses, donkeys, or mules. There are two main teams in front of the wagon, each consisting of 8 animals (arranged in pairs). Behind the wagon, there is another group of 6 animals (3 pairs) that appear to be pushing or following closely. Additionally, there is a separate pair of 2 animals to the right of the wagon. Adding these groups together (8 + 8 + 6 + 2) results in a total of 24 animals near the wagon."
}

Final Answer:  24 
Reasoning:  By zooming into the bottom of the postcard, a large wagon can be seen being pulled by multiple teams of horses, donkeys, or mules. There are two main teams in front of the wagon, each consisting of 8 animals (arranged in pairs). Behind the wagon, there is another group of 6 animals (3 pairs) that appear to be pushing or following closely. Additionally, there is a separate pair of 2 animals to the right of the wagon. Adding these groups together (8 + 8 + 6 + 2) results in a total of 24 animals near the wagon.

I then ask the model to extract information from the postcard's back label.

The make_postcard_label_testcase function creates a test case that prompts the model to extract information from back_url.

def make_postcard_label_testcase(back_url: str) -> types.GenerateContentResponse:
  return VisionTestCase(
    image_url=back_url,
    prompt=(
        "Zoom to the label and find:\n"
        "1) the title of the artwork\n"
        "2) the dimensions\n"
        "3) the manufacturing date of the postcard\n"
        "4) the manufacturing location of the postcard\n"
        "5) the brand of the postcard\n"
        "6) the manufacturer of the postcard\n"
        "7) the phone number of the postcard manufacturer\n"
        "8) the address of the postcard manufacturer\n"
        "9) the serial number of the postcard\n"
        "10) the price of the postcard\n"
    ),
    response_model=PostcardLabel
)

print_test_cases(heading="Up the River test cases", 
    cases=[
        make_postcard_label_testcase(back_url=back_url),
    ]
)

The executable_code generates Python code to crop the left label area and save it to price_crop.jpg. The code printed Original image size: 2500x1875. The code_execution_result displayed OUTCOME_OK and Original image size: 2500x1875.

part.executable_code ->  code="import cv2\nimport PIL.Image\n\nimg = cv2.imread('f_https___raw.githubusercontent.com_railsstudent_colab_images_refs_heads_main_agentic_visions_up_the_river_back.jpg')\nheight, width, _ = img.shape\n\n# Crop the left label area\n# Roughly [100, 50, 850, 400] in normalized coordinates\nlabel_crop = img[int(0.1*height):int(0.85*height), int(0.05*width):int(0.4*width)]\ncv2.imwrite('label_crop.jpg', label_crop)\n\n# Crop the price area at the bottom left\nprice_crop = img[int(0.65*height):int(0.85*height), int(0.15*width):int(0.35*width)]\ncv2.imwrite('price_crop.jpg', price_crop)\n\n# Output for visual confirmation\nprint(f'Original image size: {width}x{height}')\n" language=<Language.PYTHON: 'PYTHON'>
part.code_execution_result ->  outcome=<Outcome.OUTCOME_OK: 'OUTCOME_OK'> output='Original image size: 2500x1875\n'

The model stores the JSON object in part.text and displays the title, dimensions, manufacturing date, manufacturing location, brand, manufacturer, phone number, address, serial number, and price. The model renders English and Traditional Chinese characters.

part.text ->  {
  "title": "(Qing Court Version of) Up the River During Qingming",
  "dimensions": "W14.8 x H10 x D0.1 cm",
  "manufacturing_date": "2025.07.01",
  "manufacturing_location": "Taiwan",
  "brand": "臻印藝術",
  "manufacturer": "興台彩色印刷股份有限公司",
  "phone_number": "+886-(4)-2287-1181",
  "address": "台中市南區忠孝路 64 號",
  "serial_number": "2928833300961",
  "price": 47,
  "reasoning": "The information was extracted from the product label on the back of the postcard. The title is derived from the description mentioning the '(Qing Court Version of) Up the River During Qingming'. Dimensions, manufacturing date (2025.07.01), location (Taiwan), brand (臻印藝術), manufacturer (興台彩色印刷股份有限公司), phone number, and address were all clearly printed on the label. The serial number and price ($47) were identified from the barcode sticker at the bottom."
}

Because the result uses the PostcardLabel type, I access its fields and display their values.

Title:  (Qing Court Version of) Up the River During Qingming 
Dimensions:  W14.8 x H10 x D0.1 cm 
Manufacturing Date:  2025.07.01 
Manufacturing Location:  Taiwan 
Brand:  臻印藝術 
Manufacturer:  興台彩色印刷股份有限公司 
Phone Number:  +886-(4)-2287-1181 
Address:  台中市南區忠孝路 64 號 
Serial Number:  2928833300961 
Price:  47.0 
Reasoning:  The information was extracted from the product label on the back of the postcard. The title is derived from the description mentioning the '(Qing Court Version of) Up the River During Qingming'. Dimensions, manufacturing date (2025.07.01), location (Taiwan), brand (臻印藝術), manufacturer (興台彩色印刷股份有限公司), phone number, and address were all clearly printed on the label. The serial number and price ($47) were identified from the barcode sticker at the bottom.

This concludes the Gemini 3 Agentic Vision code walkthrough.

Conclusion

Agentic Vision uses Code Execution to generate Python code for precise, high-resolution postcard cropping. The code_execution_result in the workflow demonstrates how Gemini 3 surpasses standard object detection and uses the model's new vision capability to transform a low-legibility postcard area into a series of cropped images.

Technical Takeaways:

Active Investigation: The agent does not rely on a single look at the image; it identifies areas of interest and interacts with the data to extract hidden details.
Deterministic Accuracy: Code execution extracts details more precisely than standard bounding boxes.
User Experience (UX): This workflow demonstrates how generative AI can act as a bridge between dense physical media and specific end-user resolution requirements.

What will you build when your application can act on what it sees? Start experimenting with the Gemini API and observe actively rather than passively.

Resources

Gemini 3 Agentic Vision: Moving Beyond Simple Image Prompting [GDE]

Connie Leung — Thu, 12 Feb 2026 01:04:56 +0000

Chinese art's "density" becomes a barrier, turning detailed scenes into indistinct shapes. When the eyes no longer resolve fine brushstrokes, the artwork loses its narrative.

Given a command—like "Zoom in on the postcard to find the figures on the bridge"—the model becomes an active investigator:

Think: It scans the painting's complex composition, identifying the requested detail's coordinates amidst mountains, trees, buildings, and people.
Act: It autonomously writes and executes Python code to "crop and enhance," isolating the area with high accuracy.
Observe: It reviews the image, answers the prompt, and reasons before presenting the results to the user.

This tool does more than zoom; it makes art accessible by clarifying details, regardless of postcard size.

This post walks through the Colab notebook to inspect artwork postcards, answer visual questions, and identify the manufacturer.

Demo Overview

Let's look at how we can implement this 'Digital Docent' using Python and the Gemini SDK.

Run the Colab notebook to inspect the postcard front for visual answers, and the back for metadata like the title, dimensions, and manufacturer.

The public agentic_visions GitHub folder hosts the raw images used in this demo.

I load these URLs and use the Gemini 3 Flash preview model, the code execution tool, and a text prompt for image cropping and data extraction.

Although the prompt is a single sentence, the model repeats the think, act, and observe steps until it derives an answer.

Prerequisites

To run the Colab notebook, ensure you have:

A valid personal Gmail account to sign in to Google Colab
Vertex AI in Express Mode: I use Gemini via Vertex AI due to regional availability (Hong Kong), but these features function identically in the public Gemini API. To use this for free, create a Google Cloud project, enable the Vertex AI API, and opt into Express Mode using your Gmail account.
Google Cloud API key: Generate an API key in the Google Cloud Console. Configure this key by following the authentication guide.

Setup

Clone the notebook

Save a copy of the Colab notebook to Google Drive to run the test cases.

Client Secrets

Add a GOOGLE_CLOUD_API_KEY secret variable in the Colab notebook by clicking the Secrets (key icon) tab in the left sidebar.

GOOGLE_CLOUD_API_KEY=<GOOGLE CLOUD API KEY>

In the notebook, import userdata to get the Vertex AI Gemini API Key.

from google.colab import userdata

cloud_api_key = userdata.get('GOOGLE_CLOUD_API_KEY')

Code Execution

Install and Import Libraries

The notebook installs Python libraries and imports classes.

%pip install google-genai
%pip install matplotlib
%pip install os
%pip install dotenv
%pip install pydantic

from google import genai
from matplotlib import pyplot as plt
from google.genai import types
from pydantic import BaseModel, Field
from urllib.error import URLError
import requests
from PIL import Image
from io import BytesIO
import os
from typing import Union

Test Images

The front of the postcard: The National Museum of Taiwan exhibits Up the River.

I prompt the model to zoom in on the bottom of the postcard to count horses.

The back of the postcard: It contains metadata like the title and dimensions.

The prompt asks the model to zoom into the label to extract the title, dimensions, manufacturing date, manufacturing location, manufacturer, phone number, address, serial number, and price.

Define the Pydantic Models

I define a Pydantic model to store data.

ArtworkDiscovery stores the code execution's answer and reasoning.

class ArtworkDiscovery(BaseModel):
    answer: str = Field(..., description="The answer to the prompt")
    reasoning: str = Field(..., description="The reasoning behind the answer")

PostcardLabel stores the postcard's back label information.

class PostcardLabel(BaseModel):
    title: str = Field(..., description="The title of the artwork on the postcard")
    dimensions: str = Field(..., description="The dimensions of the postcard")
    manufacturing_date: str = Field(..., description="The manufacturing date of the postcard")
    manufacturing_location: str = Field(..., description="The manufacturing location of the postcard")
    brand: str = Field(..., description="The brand of the postcard")
    manufacturer: str = Field(..., description="The manufacturer of the postcard")
    phone_number: str = Field(..., description="The phone number of the postcard manufacturer")
    address: str = Field(..., description="The address of the postcard manufacturer")
    serial_number: str = Field(..., description="The serial number of the postcard")
    price: float = Field(..., description="The price of the postcard")
    reasoning: str = Field(..., description="The reasoning behind the answer")

The VisionTestCase class defines the image URL, text prompt, and response model type. The response model uses either the ArtworkDiscovery or PostcardLabel class.

class VisionTestCase(BaseModel):
    image_url: str = Field(..., description="The URL of the image to be analyzed")
    prompt: str = Field(..., description="The prompt to be answered based on the image")
    response_model: Union[type[ArtworkDiscovery], type[PostcardLabel]]  = Field(..., description="The Pydantic model that defines the expected response format")

Prompt Engineering

Create a Gemini client

def create_vertexai_client():
    cloud_api_key = userdata.get('GOOGLE_CLOUD_API_KEY')
    if not cloud_api_key:
        raise ValueError("GOOGLE_CLOUD_API_KEY not found in .env file")

    # Configure the client with your API key
    client = genai.Client(
        vertexai=True,
        api_key=cloud_api_key,
    )

    return client

client = create_vertexai_client()

The Agentic Loop in Action

The curate_artwork_postcard function calls generate_content, prompting the model to crop images within an agentic loop to derive the final answer.

Please be clear that the loop occurs on the server-side within the generated_content calls. The Python client waits for the result while the model iterates.

Zoom into the Postcards

tools=[types.Tool(code_execution=types.ToolCodeExecution())]
config=types.GenerateContentConfig(
    response_mime_type="application/json",
    response_json_schema=test_case.response_model.model_json_schema(),
    thinking_config=types.ThinkingConfig(
        thinking_level=types.ThinkingLevel.HIGH
    ),
    media_resolution=types.MediaResolution.MEDIA_RESOLUTION_HIGH,
    tools=tools, 
)

def curate_artwork_postcard(test_case: VisionTestCase) -> types.GenerateContentResponse:
    response = client.models.generate_content(
        model="gemini-3-flash-preview",
        contents=[
            types.Content(
                role="user",
                parts=[
                    types.Part.from_uri(file_uri=test_case.image_url),
                    types.Part(text=test_case.prompt),
                ]
            )
        ],
        config=config
    )

    return response

Display the Results

def print_parts(response: types.GenerateContentResponse):
    for part in response.candidates[0].content.parts:
        if part.text is not None and part.text.strip():
            print("part.text -> ", part.text.strip())
        if part.executable_code is not None:
            print("part.executable_code -> ", part.executable_code)
        if part.code_execution_result is not None:
            print("part.code_execution_result -> ", part.code_execution_result)
        if part.as_image() is not None:
            # display() is a standard function in Jupyter/Colab notebooks
            display(Image.open(BytesIO(part.as_image().image_bytes)))

clean_json_string is a helper that is a fallback for raw text responses.

def clean_json_string(raw_string):
    # Remove the markdown code blocks
    clean_str = raw_string.strip()
    if clean_str.startswith("<triple backticks>json"):
        clean_str = clean_str[7:]
    if clean_str.endswith("<triple backticks>"):
        clean_str = clean_str[:-3]
    return clean_str.strip()

def print_artwork_result(test_case: VisionTestCase):
    response = curate_artwork_postcard(test_case=test_case)

    if response.parsed:
        result = test_case.response_model.model_validate(response.parsed) 
    else:
        result = test_case.response_model.model_validate_json(
            clean_json_string(response.text)
        )

    print_parts(response=response)
    if isinstance(result, ArtworkDiscovery):
        print("Final Answer: ", result.answer, "\nReasoning: ", result.reasoning)
    elif isinstance(result, PostcardLabel):
        print("Title: ", result.title,
              "\nDimensions: ", result.dimensions, 
              "\nManufacturing Date: ", result.manufacturing_date, 
              "\nManufacturing Location: ", result.manufacturing_location, 
              "\nBrand: ", result.brand, 
              "\nManufacturer: ", result.manufacturer, 
              "\nPhone Number: ", result.phone_number, 
              "\nAddress: ", result.address, 
              "\nSerial Number: ", result.serial_number, 
              "\nPrice: ", result.price, 
              "\nReasoning: ", result.reasoning
        )

def print_test_cases(heading: str, cases: list[VisionTestCase]):
    print(heading)
    for test_case in cases:
        print_artwork_result(test_case=test_case)

Construct the Test Cases

I use show_the_postcard_front_and_back to construct test image URLs and display them. The load_image_from_url uses the libraries to load the image from an URL and display it.

def load_image_from_url(url: str):
    try:
        response = requests.get(url=url)
        img = Image.open(BytesIO(response.content))
        plt.imshow(img)
        plt.axis('off')
        plt.show()
    except requests.HTTPError as e:
        # e.code contains the status code (e.g., 404)
        if e.code == 404:
            print("Error: URL not found (404).")
        else:
            print(f"HTTP Error: {e.code}")

    except URLError:
        print(f"Error: The file at '{url}' was not found.")
    except Exception as e:
        print(f"An error occurred: {e}")

def show_the_postcard_front_and_back(front: str, back: str):
    agent_vision_base_url = "https://raw.githubusercontent.com/railsstudent/colab_images/refs/heads/main/agentic_visions"
    front_url = f"{agent_vision_base_url}/{front}"
    back_url = f"{agent_vision_base_url}/{back}"

    load_image_from_url(url=front_url)
    load_image_from_url(url=back_url)

front_url, back_url = show_the_postcard_front_and_back(front="up-the-river-front.jpg", back="up-the-river-back.jpg")

I ask the model to count animals at the bottom of up-the-river-front.jpg.

up_the_river_test_cases = [
    VisionTestCase(
        image_url=front_url,
        prompt="Zoom to bottom of the postcard to find the number of horses/donkeys/mules near the wagons",
        response_model=ArtworkDiscovery
    ),
]

print_test_cases(heading="Up the River test cases", cases=up_the_river_test_cases)

The following outputs appear:

Note: Libraries like cv2 and PIL are pre-installed in the Gemini model's secure execution sandbox.

part.executable_code ->  code='import PIL.Image\nimport PIL.ImageDraw\n\n# Load the image to get dimensions\nimg = PIL.Image.open(\'f_https___raw.githubusercontent.com_railsstudent_colab_images_refs_heads_main_agentic_visions_up_the_river_front.jpg\')\nwidth, height = img.size\n\n# Identify the bottom area with wagons and animals\n# Wagon 1 area: bottom left/center\n# Wagon 2 area: bottom right/center\n# Far right animals\n# Let\'s crop the bottom third of the image first to see clearly\nbottom_strip = [700, 100, 950, 900] # [ymin, xmin, ymax, xmax] in normalized coordinates\n\n# Better crop: focusing on the animal teams\n# Wagon team 1 (left): approx [750, 100, 900, 400]\n# Wagon team 2 (center): approx [750, 400, 900, 700]\n# Far right: [780, 720, 880, 820]\n\ndef get_crop(img, box_norm):\n    ymin, xmin, ymax, xmax = box_norm\n    left = xmin * width / 1000\n    top = ymin * height / 1000\n    right = xmax * width / 1000\n    bottom = ymax * height / 1000\n    return img.crop((left, top, right, bottom))\n\nbottom_view = get_crop(img, [730, 80, 920, 950])\nbottom_view.save(\'bottom_view.png\')\n\n# Detailed crops for counting\nteam1_crop = get_crop(img, [750, 100, 900, 400])\nteam1_crop.save(\'team1.png\')\n\nteam2_crop = get_crop(img, [720, 400, 900, 720])\nteam2_crop.save(\'team2.png\')\n\nright_animals = get_crop(img, [780, 720, 880, 820])\nright_animals.save(\'right_animals.png\')\n\n# Output objects for verification\n# [{box_2d: [750, 150, 880, 360], label: "animal team 1"},\n#  {box_2d: [750, 400, 885, 545], label: "animal team 2"},\n#  {box_2d: [795, 730, 861, 805], label: "right animals"}]\n' language=<Language.PYTHON: 'PYTHON'>
part.code_execution_result ->  outcome=<Outcome.OUTCOME_OK: 'OUTCOME_OK'> output=None

part.text displays the answer and reasoning. The model discovers 24 animals whereas I found 23. The reasoning field describes the process leading to the answer.

part.text ->  {
 "answer": "24",
 "reasoning": "By zooming into the bottom of the postcard, a large wagon can be seen being pulled by multiple teams of horses, donkeys, or mules. There are two main teams in front of the wagon, each consisting of 8 animals (arranged in pairs). Behind the wagon, there is another group of 6 animals (3 pairs) that appear to be pushing or following closely. Additionally, there is a separate pair of 2 animals to the right of the wagon. Adding these groups together (8 + 8 + 6 + 2) results in a total of 24 animals near the wagon."
}

Final Answer:  24 
Reasoning:  By zooming into the bottom of the postcard, a large wagon can be seen being pulled by multiple teams of horses, donkeys, or mules. There are two main teams in front of the wagon, each consisting of 8 animals (arranged in pairs). Behind the wagon, there is another group of 6 animals (3 pairs) that appear to be pushing or following closely. Additionally, there is a separate pair of 2 animals to the right of the wagon. Adding these groups together (8 + 8 + 6 + 2) results in a total of 24 animals near the wagon.

I then ask the model to extract information from the postcard's back label.

The make_postcard_label_testcase function creates a test case that prompts the model to extract information from back_url.

def make_postcard_label_testcase(back_url: str) -> types.GenerateContentResponse:
  return VisionTestCase(
    image_url=back_url,
    prompt=(
        "Zoom to the label and find:\n"
        "1) the title of the artwork\n"
        "2) the dimensions\n"
        "3) the manufacturing date of the postcard\n"
        "4) the manufacturing location of the postcard\n"
        "5) the brand of the postcard\n"
        "6) the manufacturer of the postcard\n"
        "7) the phone number of the postcard manufacturer\n"
        "8) the address of the postcard manufacturer\n"
        "9) the serial number of the postcard\n"
        "10) the price of the postcard\n"
    ),
    response_model=PostcardLabel
)

print_test_cases(heading="Up the River test cases", 
    cases=[
        make_postcard_label_testcase(back_url=back_url),
    ]
)

part.executable_code ->  code="import cv2\nimport PIL.Image\n\nimg = cv2.imread('f_https___raw.githubusercontent.com_railsstudent_colab_images_refs_heads_main_agentic_visions_up_the_river_back.jpg')\nheight, width, _ = img.shape\n\n# Crop the left label area\n# Roughly [100, 50, 850, 400] in normalized coordinates\nlabel_crop = img[int(0.1*height):int(0.85*height), int(0.05*width):int(0.4*width)]\ncv2.imwrite('label_crop.jpg', label_crop)\n\n# Crop the price area at the bottom left\nprice_crop = img[int(0.65*height):int(0.85*height), int(0.15*width):int(0.35*width)]\ncv2.imwrite('price_crop.jpg', price_crop)\n\n# Output for visual confirmation\nprint(f'Original image size: {width}x{height}')\n" language=<Language.PYTHON: 'PYTHON'>
part.code_execution_result ->  outcome=<Outcome.OUTCOME_OK: 'OUTCOME_OK'> output='Original image size: 2500x1875\n'

part.text ->  {
  "title": "(Qing Court Version of) Up the River During Qingming",
  "dimensions": "W14.8 x H10 x D0.1 cm",
  "manufacturing_date": "2025.07.01",
  "manufacturing_location": "Taiwan",
  "brand": "臻印藝術",
  "manufacturer": "興台彩色印刷股份有限公司",
  "phone_number": "+886-(4)-2287-1181",
  "address": "台中市南區忠孝路 64 號",
  "serial_number": "2928833300961",
  "price": 47,
  "reasoning": "The information was extracted from the product label on the back of the postcard. The title is derived from the description mentioning the '(Qing Court Version of) Up the River During Qingming'. Dimensions, manufacturing date (2025.07.01), location (Taiwan), brand (臻印藝術), manufacturer (興台彩色印刷股份有限公司), phone number, and address were all clearly printed on the label. The serial number and price ($47) were identified from the barcode sticker at the bottom."
}

Because the result uses the PostcardLabel type, I access its fields and display their values.

Title:  (Qing Court Version of) Up the River During Qingming 
Dimensions:  W14.8 x H10 x D0.1 cm 
Manufacturing Date:  2025.07.01 
Manufacturing Location:  Taiwan 
Brand:  臻印藝術 
Manufacturer:  興台彩色印刷股份有限公司 
Phone Number:  +886-(4)-2287-1181 
Address:  台中市南區忠孝路 64 號 
Serial Number:  2928833300961 
Price:  47.0 
Reasoning:  The information was extracted from the product label on the back of the postcard. The title is derived from the description mentioning the '(Qing Court Version of) Up the River During Qingming'. Dimensions, manufacturing date (2025.07.01), location (Taiwan), brand (臻印藝術), manufacturer (興台彩色印刷股份有限公司), phone number, and address were all clearly printed on the label. The serial number and price ($47) were identified from the barcode sticker at the bottom.

This concludes the Gemini 3 Agentic Vision code walkthrough.

Conclusion

Technical Takeaways:

Active Investigation: The agent does not rely on a single look at the image; it identifies areas of interest and interacts with the data to extract hidden details.
Deterministic Accuracy: Code execution extracts details more precisely than standard bounding boxes.
User Experience (UX): This workflow demonstrates how generative AI can act as a bridge between dense physical media and specific end-user resolution requirements.

What will you build when your application can act on what it sees? Start experimenting with the Gemini API and observe actively rather than passively.

Resources

[GDE] How to Refactor a Complex Blog Review Prompt into Reusable AI Agents

Connie Leung — Sat, 31 Jan 2026 03:03:49 +0000

How to Refactor a Complex Blog Review Prompt into Reusable AI Agents

Technical writers and ESL professionals often rely on complex prompts to polish our work. However, as prompts grow into monolithic walls of text, they become brittle, hard to debug, and expensive to run. Modularity addresses this challenge.

In this post, I transition from a single Custom Command to a modular, portable Gemini CLI extension implemented as a set of specialized Agent Skills.

Prerequisites

Before we begin, ensure you have the following:

Install the gemini-cli.
Create a Google AI Studio API Key or a Vertex AI Credential.
Ensure you have access to a terminal (macOS, Linux, or WSL).

Setting Up the Environment

Install the Gemini CLI globally.

npm install -g @google/gemini-cli

To access your extension and the API key globally, create a .env file in the ~/.gemini global workspace, and add the API key to the .env file as follows:

# Your Google AI Studio API Key
GEMINI_API_KEY=<Gemini API Key>

I use the ~/.gemini path, which is standard for macOS/Linux systems. The audience is either using or is familiar with a Unix-like environment.

Gemini CLI automatically loads variables from the first .env file it finds, searching up from the current directory, then in ~/.gemini/.env or ~/.env.

When the CLI loads the GEMINI_API_KEY from the .gemini/.env file, it works in projects automatically. Moreover, it acts as a single source of truth for key rotation. When the key expires, users only need to update a single file, rather than hunting down multiple copies scattered across projects.

Comparison between Custom Commands and Agent Skills

While both Agent Skills and Custom Commands add functionality to the Gemini CLI, they serve fundamentally different architectural roles. A Custom Command is a shortcut that can run in a terminal or in a Gemini CLI chat. On the other hand, an Agent Skill acts as a plugin or a domain expert to provide expertise to the AI model.

There are four key differences between Custom Commands and Agent Skills.

Feature	Custom Command	Agent Skill
Activation	Explicit (Slash command)	Automatic (Model-decided)
Structure	Single `.toml` file	Directory with `SKILL.md` & assets
Context	Prompt loaded immediately	Instructions loaded on-demand
Use Case	One-shot transformation	Multi-step / Stateful workflow

Custom commands are slash commands that users must type explicitly into the terminal. The AI model does not use them unless the user explicitly requests them. In contrast, the model analyzes a user prompt to identify the skill gap and activates a skill based on the skill description. This relies on Gemini's function calling capabilities to autonomously select the right tool.
A Custom Command is a TOML file containing a description and a prompt. When a user places a command in the commands folder of the global .gemini folder, it is available after restarting the Gemini CLI. An Agent Skill is a self-contained directory with a SKILL.md Markdown file, instructions, metadata, and external assets such as scripts, templates, or example data. When the directory is placed in the skills folder of the global .gemini folder, it is available after restart.
When a user runs a Custom Command, the model loads the prompt from the TOML file into its current context. The CLI loads the name and description of all the skills into the model's context. This ensures the model is aware of the tools without consuming context tokens on their instructions until they are actually invoked. The instructions and external assets are loaded into the context window when the model explicitly calls a specific skill. This allows users to have hundreds of specialized skills without exhausting the token limit.
A Custom Command is primarily used for one-shot transformations. Users provide the inputs and the Custom Command calls the prompt to generate the results. An Agent Skill is designed for a multi-step procedural workflow. A skill can define a complex thought process or a series of actions to solve a complex problem. It represents a stateful capability rather than a text shortcut.

Understanding the Architecture

Custom Command Architecture

In the blog-review.toml file, I initially defined the description and a prompt that instructs the model to read a blog post and audit the content against the nine focus areas.

Modular Agent Skills

I extracted each focus area into a separate Agent Skill that could be executed individually or as part of the Custom Command.

Portable CLI Extension Packaging

I created a new CLI extension project to package the Custom Command and skills. Then, I installed the extension in the Gemini CLI.

The Starting Point: A Monolithic Custom Command

I started with a single, exhaustive prompt that detected every linguistic error in a Markdown file.

description = "Comprehensive Markdown Prose Auditor: Scans for grammar, syntax, tone, and technical clarity with list-based reporting and a summary table."

prompt = """
You are a Senior Technical Copyeditor. Your goal is to identify and provide specific fixes for linguistic and structural errors in Markdown content without rewriting the entire document.

### YOUR MISSION:
Perform a word-by-word audit of the input. Group all identified issues by their category so the user can review them systematically.

**Focus Areas:**
1. **Grammar & Syntax:** Subject-verb agreement, tense consistency, and prepositions.
2. **Active Voice:** Converting passive descriptions into direct, active instructions.

### CONSTRAINTS:
- **DO NOT** output the full polished text.
- **DO NOT** report errors found inside code blocks (<triple ticks>).
- **FORMAT:** Use a **List** for the audit details (for console readability) and a **Table** for the final summary.
- **LOCATION:** Specify the nearest Heading or Paragraph number for every find.

### CONTENT TO AUDIT:
<content>
{{args}}
</content>

### OUTPUT FORMAT:

#### 🔬 LINGUISTIC AUDIT DETAILS

### [Category Name]
- **Location:** [Heading/Paragraph]
  - **Original:** "..."
  - **Fixed:** "..."
  - **Rationale:** [Brief explanation of the fix]

---

#### 📊 AUDIT SUMMARY
| Category | Issues Found |
| :--- | :--- |
| 1. Grammar & Syntax | [Count] |
| 2. Active Voice | [Count] |
| **TOTAL** | **[Sum]** |
"""

{{args}} is the placeholder for the user's input file or text.

I copied blog-review.toml to ~/.gemini/commands.

mkdir -p ~/.gemini/commands
cp blog-review.toml ~/.gemini/commands

Then, I started gemini to verify the blog-review command's availability.

gemini

/blog-review

How to Execute the Command

I created a dummy file named test_agent_skills.md containing intentional errors to test the system. To execute this command in the standard CLI, I passed it as an argument:

# In a terminal
gemini /blog-review @test_agent_skills.md

Or, in the Gemini CLI chat:

gemini

/blog-review @test_agent_skills.md

The Custom Command identified 25 issues and displayed the results in the console. While handling fewer than 10 issues was manageable, the 25 identified issues exceeded a viable threshold.

To fix this, I needed to decouple the detection logic. I decided to build specialized Agents for each category.

Instead of listing every issue, I focused on one area and fixed all related issues. Using Agent Skills established the modular architecture. For example, I used natural language to identify grammar and syntax errors in a blog post.

In the following sections, I created grammar-syntax-reviewer, missing-article-reviewer, lexical-precision-reviewer, technical-punctuation-reviewer, and active-voice-reviewer skills.

Build the Skill Suite

To improve accuracy, extract the grammar logic into a specialized "Agent Skill."

I created a test-project/skills folder recursively.

mkdir -p test-project/skills

I created a file named SKILL.md in the test-project/skills/grammar-syntax-reviewer directory.

The Markdown file has the following structure:

YAML Metadata
PERSONA
REVIEW PROTOCOL
STRATEGIC RULES
OUTPUT FORMAT

Below is the full content of SKILL.md of grammar-syntax-reviewer skill:

Grammar & Syntax Reviewer (7-Category Exhaustive Edition)

---
name: grammar-syntax-reviewer
description: A high-precision editorial skill focused on subject-verb agreement, tense consistency, prepositional accuracy, and structural integrity in technical writing.
---

# Grammar & Syntax Reviewer (7-Category Exhaustive Edition)

## PERSONA

You are a Senior Technical Copyeditor and ESL Writing Coach. You possess an obsessive eye for the mechanical "gears" of the English language. Your goal as a Grammar & Syntax Reviewer is to ensure that every sentence follows the formal rules of English grammar to provide maximum clarity for a global audience, with special attention to technical nuances.

## REVIEW PROTOCOL

Execute seven distinct passes over the text and group findings into these categories:

1. **CATEGORY 1: Subject-Verb Agreement (including Intervening Phrases)**
    - Target: Verbs that do not match their subjects, especially when separated by long phrases (e.g., "The list of all available API endpoints are" vs "The list... is").

[Other categories omitted for brevity]

## STRATEGIC RULES

- **No Omissions:** Identify EVERY instance. If a document has 20 errors, list all 20.
- **Searchability Priority:** Provide the nearest **Heading** and the **Full Sentence (Verbatim)** so the user can use Ctrl+F.
- **Code-Adjacent Logic:** Pay extra attention to the grammar of sentences that wrap around inline code backticks (e.g., "Using the `GET` methods" where it should be "method").
- **Code Immunity:** Ignore everything inside triple backticks (<triple ticks>).

## OUTPUT FORMAT

### 🔬 EXHAUSTIVE GRAMMAR & SYNTAX REVIEW

### Category 1: Subject-Verb Agreement

- **Location:** [Heading]
- **Search String:** "[Full sentence containing the error]"
- **Fixed:** "[The corrected full sentence]"
- **Rationale:** [Briefly explain the rule, e.g., "The collective noun 'group' is singular and requires the verb 'is'."]

---

### [Continue for other Categories...]

---

#### 📊 GRAMMAR & SYNTAX REVIEW SUMMARY

| Category | Issues Found |
| :--- | :--- |
| 1. Subject-Verb Agreement | [Count] |
| **TOTAL ERRORS** | **[Sum]** |

The other skills are similar and are omitted for brevity. The full content of the skills can be found in: Custom Blog Review Agent Skills

Active Voice Reviewer

The Active Voice Reviewer skill is designed to ensure technical content is direct and authoritative. Create skills/active-voice-reviewer/SKILL.md:

---
name: active-voice-reviewer
description: A high-precision editorial skill designed to eliminate passive voice, weak 'to-be' constructions, and nominalizations in technical prose. 
---

# Active Voice Reviewer

## PERSONA

You are a Senior Technical Copyeditor specializing in Instructional Clarity. Your goal is to eliminate passive voice, weak "to-be" constructions, and indirect phrasing to make technical prose direct, authoritative, and easier to follow.

## REVIEW PROTOCOL

1. **CATEGORY 1: Formal Passive Voice**
  - Target: "To-be" verb + Past Participle (e.g., "is clicked by").

## STRATEGIC RULES

- **No Omissions:** Identify EVERY instance. If a document has 50 issues, list all 50.

## OUTPUT FORMAT

### ⚡ EXHAUSTIVE ACTIVE VOICE AUDIT

### Category 1: Formal Passive Voice

- **Location:** ## Authentication Flow
- **Search String:** "The security token is generated by the middleware layer after the request arrives."
- **Fixed (Active):** "The middleware layer generates the security token after the request arrives."
- **Rationale:** The actor (middleware layer) was placed at the end of the sentence. Moving it to the subject position makes the sentence more direct.

[Other categories omitted for brevity]

#### 📊 ACTIVE VOICE SUMMARY

| Category | Issues Found |
| :--- | :--- |
| 1. Formal Passive Voice | [Count] |
| **TOTAL IMPROVEMENTS** | **[Sum]** |

Run Skills in the Gemini CLI Chat

The skills folder did not exist in ~/.gemini, so I created the folder and copied all the skills to it.

mkdir -p ~/.gemini/skills
cp -r ~/test-project/skills/*  ~/.gemini/skills

To use these skills interactively, start the chat REPL:

gemini

Enable the experimental Agent Skills in the CLI.

/settings

The CLI displays a settings screen featuring a search filter to narrow down the settings.

Users input "Agent" in the search filter and scroll the list until "Enable Agent Skills" is visible. Users can set the flag to true and apply the changes to User Settings.

Restart gemini to enable the Agent Skills.

Inside the chat, manage your skills:

/skills — List all detected skills.
/skills enable <skill> — Activate a specific skill.
/skills disable <skill-name> — Deactivate a specific skill.
/skills reload — Refresh the list if you've added new files.

I instructed the CLI to identify active-voice issues in the blog post. The AI model used the active-voice-reviewer skill to find all issues.

Find the active voice issues in @test_agent_skills.md

The CLI prompted me to activate the skill.

✓  Activate Skill "active-voice-reviewer": A high-precision editorial skill designed to eliminate passive voice, weak 'to-be' constructions, and nom… │
│                                                                                                                                                       │
│ Skill active-voice-reviewer activated. Resources loaded from ~/.gemini/skills/active-voice-reviewer:                                 │
│                                                                                                                                                       │
│ Showing up to 200 items (files + folders).                                                                                                            │
│                                                                                                                                                       │
│ ~/.gemini/skills/active-voice-reviewer/                                                                                              │
│ └───SKILL.md

The CLI displayed the issues and the skill summary.

Category 1: Formal Passive Voice

   - Location: ## Demo Overview
   - Search String: "the Google Search Tool is used."
   - Fixed (Active): "the model uses the Google Search Tool."
   - Rationale: The original sentence obscures the actor (the model).

  ---

  Category 2: Hidden Actors (Agentless Passive)

   - Location: ## Introduction
   - Search String: "and the response JSON schema is provided"
   - Fixed (Active): "and you provide the response JSON schema"
   - Rationale: The original implies the schema appears by magic.

  ---

  Category 3: Weak "To-Be" Wind-ups (Deadwood)

   - Location: ## Lessons Learned (4. Valid URI...)
   - Search String: "The solution was to provide instructions for categorizing"
   - Fixed (Active): "I solved this by instructing the model to categorize"
   - Rationale: "The solution was to" is a weak, noun-heavy wind-up. "I solved this by" is dynamic.

  ---

  Category 4: Nominalizations (Smothered Verbs)

   - Location: ## Demo Overview
   - Search String: "Name verification is performed before data retrieval."
   - Fixed (Active): "The script verifies the name before retrieving data."
   - Rationale: "Verification" and "retrieval" are smothered verbs. "Verifies" and "retrieving" restore the energy.

   - Location: ## Lessons Learned (4. Valid URI...)
   - Search String: "The solution was to provide instructions for categorizing"
   - Fixed (Active): "I solved this by instructing the model to categorize"
   - Rationale: "Provide instructions" is a weak nominalization of "instruct".

  ---

  Category 5: Modal Hesitation (Soft Commands)

   - Location: ## Architecture
   - Search String: "it can access an external system"
   - Fixed (Active): "it accesses an external system"
   - Rationale: "Can access" implies capability; "accesses" describes what it actually does in this architecture.

  ---

  📊 ACTIVE VOICE SUMMARY


  ┌──────────────────────────┬──────────────┐
  │ Category                 │ Issues Found │
  ├──────────────────────────┼──────────────┤
  │ 1. Formal Passive Voice  │ 7            │
  │ 2. Hidden Actors         │ 6            │
  │ 3. Weak "To-Be" Wind-ups │ 1            │
  │ 4. Nominalizations       │ 2            │
  │ 5. Modal Hesitation      │ 1            │
  │ TOTAL IMPROVEMENTS       │ 17           │
  └──────────────────────────┴──────────────┘

The skill listed the issues by categories. The location pointed to the nearest heading, and the search string helped me locate the sentence in the blog post, where I fixed it. The rationale explained the error to help the ESL writer learn.

The table summarized my weak areas, enabling simultaneous learning and editing.

Update the Command to use Agent Skills

Next, I refactored the original blog-review.toml Custom Command to call these specific skills. This centralized the logic and ensured the AI used our high-precision rules.

description = "Exhaustive 9-Point Markdown Auditor: Powered by specialized high-precision editorial skills."

prompt = """
You are a Senior Technical Copyeditor and ESL Writing Coach. Your goal is to identify EVERY SINGLE error in the provided Markdown content by executing specialized editorial skills.

### EXHAUSTIVE AUDIT PROTOCOL:
1. **Multi-Pass Execution:** Scan the text 9 times—once for each focus area. 
2. **Skill-Specific Logic:** 
   - For Area 1: Execute the **grammar-syntax-reviewer** (7-category check).
   - For Area 9: Execute the **active-voice-reviewer** (Eliminate nominalizations and hesitation).
3. **No Omissions:** List every single occurrence. If an error appears 10 times, list all 10.
4. **Searchability Priority:** Provide the nearest Heading and the **Full Sentence** containing the error so the user can use Ctrl+F.

### FOCUS AREAS:

1. **[Skill: grammar-syntax-reviewer]:** Audit for (a) Subject-verb agreement with intervening phrases, (b) Tense consistency, (c) Preposition precision, (d) Pronoun clarity, (e) Comma splices, (f) Count vs. Non-count nouns (Code, Software, Data), and (g) Technical Plurals vs. Possessives.
... Other focus areas ...
9. **[Skill: active-voice-reviewer]:** Audit for (a) Formal passive voice, (b) Hidden actors, (c) "To-be" wind-ups (e.g., 'There are'), (d) Nominalizations (e.g., 'Perform an analysis' -> 'Analyze'), and (e) Modal hesitation (e.g., 'You should click' -> 'Click').

### CONSTRAINTS:
- **DO NOT** output the full polished text.
- **DO NOT** report errors found inside code blocks (<triple ticks>).
- **LOCATION:** You MUST provide:
    1. The nearest **Heading** (e.g., ## Setup).
    2. A **Search String**: The full sentence containing the error (for easy Ctrl+F).

### CONTENT TO AUDIT:
<content>
{{args}}
</content>

### OUTPUT FORMAT:

#### 🔬 EXHAUSTIVE LINGUISTIC AUDIT

### [Category Name]

- **Skill Applied:** [Insert Skill Name, e.g., technical-punctuation-reviewer]
- **Location:** [Nearest Heading Name]
- **Search String:** "[The full sentence where the error exists]"
- **Fixed:** "[The corrected full sentence]"
- **Rationale:** **[Formal Rule Name]**: [Brief explanation of the rule to help the writer learn.]

---

#### 📊 AUDIT SUMMARY
| Category | Issues Found |
| :--- | :--- |
| 1. Grammar & Syntax (Deep) | [Count] |
| 9. Active Voice (Instructional) | [Count] |
| **TOTAL** | **[Sum]** |
"""

{{args}} is the placeholder for the user's input file or text.

The skills are enabled in the CLI, the model perceives them as available functions or tools and maps the text instruction "Execute..." to the appropriate tool definition.

The name field in the SKILL.md frontmatter registers the skill as a tool that the model can invoke by name.

In the Gemini CLI chat, I ran the blog-review command, and the activated skills scanned the content to output the errors and summarize the count in a table.

/blog-review @test_agent_skills.md

🔬 EXHAUSTIVE LINGUISTIC AUDIT

  1. Grammar & Syntax (Deep)

   - Skill Applied: grammar-syntax-reviewer
   - Location: ## Introduction
   - Search String: "In this post, I will explore retrieving the Premier League 2025/2026 Player statistics..."
   - Fixed: "In this post, I explore retrieving the Premier League 2025/2026 Player statistics..."
   - Rationale: [Tense Consistency]: Use the present tense ("I explore") for the document's narrative to match "I walk through" and "describe".

  ... other issues ...

  2. Articles & Determiners (Noun-First)

   - Skill Applied: missing-article-reviewer
   - Location: ## Environment Variables
   - Search String: "I use the client to call Gemini 3 Flash Preview model to retrieve data..."
   - Fixed: "I use the client to call the Gemini 3 Flash Preview model to retrieve data..."
   - Rationale: [Missing Article]: "Model" is a singular countable noun unmodified by a determiner in the original phrase.

  ... other issues ...

  3. Word Choice

   - Skill Applied: lexical-precision-reviewer
   - Location: ## Lessons Learned
   - Search String: "When the player was Kaoru Mitoma, the model worked hard to find the results via Google Search..."
   - Fixed: "When the player was Kaoru Mitoma, the model executed a deep search via Google Search..."
   - Rationale: [Anthropomorphism]: "Worked hard" assigns human effort to software. Use precise technical terms like "executed a deep search."

  ... other issues ...

  4. Punctuation

   - Skill Applied: technical-punctuation-reviewer
   - Location: ## Architecture
   - Search String: "![High level architecture of Retrieving Premier League Player Stats...]"
   - Fixed: "![High-level architecture of Retrieving Premier League Player Stats...]"
   - Rationale: [Compound Modifier]: "High-level" modifies "architecture" and must be hyphenated.

  ... other issues ....

  6. Parallelism

   - Skill Applied: parallelism-check
   - Location: ## Prerequisites
   - Search String: "Python: Have Python 3.12+ and the google-genai SDK installed."
   - Fixed: "Python: Python 3.12+ and the google-genai SDK."
   - Rationale: [List Structure]: The other items in the list are Noun Phrases ("Vertex AI...", "A Google Cloud API Key..."). This item uses an
     imperative ("Have...").

  9. Active Voice (Instructional)

   - Skill Applied: active-voice-reviewer
   - Location: ## Introduction
   - Search String: "When the URL is invalid or the data is missing... the Google Search Tool is used."
   - Fixed: "When the URL is invalid or the data is missing... the model uses the Google Search tool."
   - Rationale: [Formal Passive Voice]: "Is used" hides the actor (the model).

   ... other issues ...

The table displayed the number of issues that the skills and the rest of the Custom Command identified.

📊 AUDIT SUMMARY

  ┌────────────────────────────────────────┬──────────────┐
  │ Category                               │ Issues Found │
  ├────────────────────────────────────────┼──────────────┤
  │ 1. Grammar & Syntax (Deep)             │ 6            │
  │ 2. Articles & Determiners (Noun-First) │ 4            │
  │ 3. Word Choice                         │ 2            │
  │ 4. Punctuation                         │ 5            │
  │ 5. Technical Accuracy                  │ 0            │
  │ 6. Parallelism                         │ 1            │
  │ 7. Expert Blindness                    │ 0            │
  │ 8. Conciseness                         │ 0            │
  │ 9. Active Voice (Instructional)        │ 7            │
  │ TOTAL                                  │ 25           │
  └────────────────────────────────────────┴──────────────┘

The test blog post had 25 issues; I used the heading and the text to search for and find issues to resolve.

I prototyped the skills locally, so I decided to package them into a distributable Extension so others can install them easily.

Packaging for Distribution: Creating a CLI Extension

To package the commands and skills for portability, I created an extension by running the gemini extensions new command.

gemini extensions new blog-post-pro-editor

The command created a new extension named blog-post-pro-editor. I copied the skills and commands folders from ~/.gemini to blog-post-pro-editor.

Structure:

├── commands
│   ├── blog-review.toml
│   └── critique.toml
├── gemini-extension.json
└── skills
    ├── active-voice-reviewer
    │   └── SKILL.md
    ├── grammar-syntax-reviewer
    │   └── SKILL.md
    ├── lexical-precision-reviewer
    │   └── SKILL.md
    ├── missing-articles-reviewer
    │   └── SKILL.md
    └── technical-punctuation-reviewer
        └── SKILL.md

I started each file in skills/ with YAML frontmatter:

---
name: grammar-syntax-reviewer
description: A high-precision editorial skill focused on subject-verb agreement, tense consistency, prepositional accuracy, and structural integrity in technical writing.
---

# <Skill Name>

## PERSONA

<Persona of the skill> 

(The refactored prompt goes here)

I started each file in commands/ with a description and a prompt:

description = "<Description of the blog review command>"

prompt = """
<Prompt of the blog review command and the skill instructions>
""""

Install the Extension and Run Locally

During development, I installed the skills and commands to test the Gemini CLI extension locally.

cd blog-post-pro-editor
gemini extension link .

The CLI linked and enabled the 'blog-post-pro-editor' extension in the terminal.

Extension "blog-post-pro-editor" linked successfully and enabled.

Next, I started gemini and typed the following to activate the skills and find grammar issues in the Markdown file.

> Find the grammar errors in @test_agent_skills.md

The CLI activated the grammar-syntax-reviewer and missing-articles-reviewer skills to identify structural and article-related errors in my blog post.

The skills output the following results in a table. The Grammar & Syntax skill found 3 errors, and the Missing Articles skill found 7 errors.

  ┌──────────────────────────────┬──────────────┐
  │ Category                     │ Issues Found │
  ├──────────────────────────────┼──────────────┤
  │ Grammar & Syntax             │              │
  │ 2. Tense Consistency         │ 1            │
  │ 3. Preposition Precision     │ 1            │
  │ 5. Fragments/Run-ons/Splices │ 1            │
  │ Articles & Determiners       │              │
  │ 1. Missing Articles          │ 7            │
  │ TOTAL ERRORS                 │ 10           │
  └──────────────────────────────┴──────────────┘

I packaged the skills and the commands into an extension to install them on any machine with the Gemini CLI.

I verified the new extension in the ~/.gemini/extensions folder. The command showed blog-post-pro-editor in the extension list.

ls ~/.gemini/extensions

Conclusion

By breaking the blog-review Custom Command into Agent Skills and packaging them into an Extension, we have built a comprehensive editorial suite. Execute individual skills in a Gemini CLI chat to locate relevant linguistic errors in a blog post using natural language. Similarly, enter the Custom Command blog-review in the same chat to scan the blog post and identify all linguistic errors explicitly. This modular approach ensures higher accuracy for ESL writers and makes your AI tools portable across any machine.

Resources

Fetching Live Sports Data with Gemini 3: A Guide to Grounded, Structured JSON

Connie Leung — Wed, 21 Jan 2026 00:17:53 +0000

Retrieving accurate, up-to-date sports statistics with LLMs is notoriously difficult due to hallucinations and outdated training data. This post explores extracting the Premier League 2025/2026 player statistics using the Gemini 3 Flash Preview model, URL Context, Grounding with Google Search, and structured output. I will describe the lessons learned, how I extracted data from the official Premier League player profile pages when available, and how I used Google Search to find missing data on the websites.

Introduction

The Gemini 3 Flash Preview model supports structured output, URL Context, and Grounding with Google Search. When the response MIME type is application/json and the response JSON schema is provided, Gemini 3 is remarkably consistent at returning valid JSON objects. Moreover, the Pydantic library has functions to convert a generic JSON object to a Pydantic model.

The post walks through the code in the Colab notebook, and demonstrate how to engineer a prompt that uses the provided URLs as the primary source. When the URL is invalid or the data is missing (e.g., net worth, preferred foot, and height), the model uses the the Google Search tool. When an answer is generated for each field, I apply grounding to verify that the model does not hallucinate and correctly cites reputable sources.

Prerequisites

To run the provided Colab notebook, you will need:

Vertex AI in Express Mode: I utilize Gemini via Vertex AI due to regional availability (Hong Kong), but these features function identically in the public Gemini API. If you want to use Gemini in Vertex AI for free, you can sign up for Vertex AI in Express Mode using your Gmail account.
Google Cloud API Key: Ensure the API key is properly configured within your environment.
Google Colab VS Code Extension - Visit the Visual Studio Marketplace to install the extension to run the demo in the VS Code environment.
Python: Have Python 3.12+ and the google-genai SDK installed.

Demo Overview

We will see how to force the model to prioritize retrieval over generation. The demo attempts to retrieve player statistics for the Premier League 2025/2026 season.

The Gemini 3 Flash Preview model is given some pages:

 url_list = [
    "https://www.premierleague.com/en/players/141746/bruno-fernandes/stats",
    "https://www.premierleague.com/en/players/223094/erling-haaland/stats",
    "https://www.premierleague.com/en/players/97032/virgil-van-dijk/stats",
    "https://www.premierleague.com/en/players/244851/cole-palmer/stats"
]

urls = "\n".join(url_list)

When the requested player is Bruno Fernandes, Erling Haaland, Virgil Van Dijk, or Cole Palmer, the model will execute the URL Context tool to extract data in the above URLs. Otherwise, the model will execute the Google Search tool to find the facts from reputable sources.

The system performs name verification before data retrieval. If the subject is not a player or not a Premier League player in the 2025/2026 season, the structured output assigns false to is_professional_player and an explanation to verification_status. When it is a Premier League player, the prompt instructs the model to return a JSON object containing the name, net worth, is_professional_player, verification_status, height, shirt_number, preferred_foot, goals, goal_assists, appearances, and minutes_played. Then, the notebook converts the generic JSON object to a Pydantic PlayerStats model, and displays the value, source quotation, uri, and source type to ensure accuracy.

Architecture

The URL Context and Google Search tools enable the Gemini 3 Flash Preview model to extract the performance metrics for the Premier League 2025/2026 season from the Premier League player profile pages, while retrieving personal information, such as net worth, preferred foot, and height from the web. Even though the model's internal data is outdated, it can access an external system to obtain up-to-date information.

Environment Variable

Copy .env.example to .env and replace the GOOGLE_CLOUD_API_KEY with your API key.

GOOGLE_CLOUD_API_KEY=<GOOGLE CLOUD API KEY>

The create_vertexai_client function constructs and returns a Gemini client. I use the client to call the Gemini 3 Flash Preview model to retrieve data from the provided URLs, and fall back to the Google Search tool when a URL is not provided or if it does not provide the details after deep search.

The create_vertexai_client result is assigned to a global client variable.

genai is a unified SDK for Gemini API and Gemini in Vertex AI. The vertexai=True parameter specifies usage of a Gemini in Vertex AI in the Colab notebook.

from google import genai
from dotenv import load_dotenv
from google.genai import types
from pydantic import BaseModel, Field
from typing import Literal
import os

def create_vertexai_client():

    cloud_api_key = os.getenv("GOOGLE_CLOUD_API_KEY")
    if not cloud_api_key:
        raise ValueError("GOOGLE_CLOUD_API_KEY not found in .env file")

    # Configure the client with your API key
    client = genai.Client(
        vertexai=True, 
        api_key=cloud_api_key, 
    )

    return client

load_dotenv()

client = create_vertexai_client()

Installation

The demo uses the newer Google Gen AI SDK, so I install the google-genai library.

%pip install google-genai
%pip install dotenv
%pip install pydantic

Lessons Learned

I will describe the challenges faced in this demo and how I tackled each one.

1. Lazy Behavior of Gemini 3 Flash Preview Model

Problem:

Gemini prioritized its internal training data (or generic search) over the specific url_context tool unless explicitly constrained. It relied on Google Search or internal knowledge to find facts for each field instead of using the given Premier League player profile pages.

Solution:

I added a DYNAMIC SOURCE IDENTIFICATION section to the prompt to always execute the URL Context tool when a player's profile page is available. When a player's profile page is not available, the model uses the Grounding with Google Search tool to retrieve information for each field in the JSON object. Finally, I emphasized the priority of the official URL, web citations, and internal training data.

### **1. DYNAMIC SOURCE IDENTIFICATION**
1.  **IF a Premier League URL is provided:**
    *   You **MUST** execute the `url_context` tool first. This is your **Primary Source**.
2.  **IF NO URL is provided (or if the player is non-Premier League):**
    *   The **Web Citations** (Google Search results) become your **Primary Source**. 
3.  **PRIORITY:** Official URL > Web Citations > Internal Training Data.

2. Chunk ID Hallucination and Low Confidence in Grounding Metadata

Problem:

I wanted to verify that the AI did not hallucinate and assign incorrect value to the fields in the JSON object. Therefore, I declared a Pydantic model, PlayerField, to store the chunk id that provided the answer for the field.

class PlayerField(BaseModel):
    value: str | int | float
    chunk_id: int = Field(..., description="Index of the chunk in the grounding metadata that provide the value")

class PlayerStats(BaseModel):
    name: str
    net_worth: PlayerField | None = Field(..., description="Net worth of the Premier League player.")
    is_professional_player: bool = Field(..., description="Must be True if found in Premier League records, False otherwise")
    verification_status: str = Field(..., description="Explanation of where the data was found or why it failed")
    height: PlayerField
    shirt_number: PlayerField
    preferred_foot: PlayerField
    goals: PlayerField
    goal_assists: PlayerField
    appearances: PlayerField
    minutes_played: PlayerField

Moreover, I wrote functions to obtain the web and retrieved context URIs from the grounding metadata to verify the Chunk ID.

def get_citations(response: types.GenerateContentResponse) -> tuple[int, list[dict]]:

    citations: list[dict] = []
    if response.candidates is not None and len(response.candidates) > 0:
        candidate = response.candidates[0]
        if candidate.grounding_metadata:
            grounding_chunks = candidate.grounding_metadata.grounding_chunks or []
            num_chunks = len(grounding_chunks)
            for i, chunk in enumerate(grounding_chunks):
                if chunk and chunk.web and chunk.web.uri:
                    citations.append({
                        "chunk_id": i,
                        "uri": chunk.web.uri,
                        "title": chunk.web.title
                    })
                elif chunk and chunk.retrieved_context and chunk.retrieved_context.uri:
                    citations.append({
                        "chunk_id": i,
                        "uri": chunk.retrieved_context.uri,
                        "title": chunk.retrieved_context.text
                    })

    return num_chunks, citations

def print_citations_by_response(response: types.GenerateContentResponse):
    num_chunks, citations = get_citations(response)

    print("num_chunks ->", num_chunks)
    for i, citation in enumerate(citations):
        print(f"Citation {i}: {citation}")

The chunk_id in the PlayerField was larger than num_chunks, which indicated that the model hallucinated and created a chunk_id that did not exist in the grounding metadata.

Bad Chunk ID

Number of chunks: 5. Chunk IDs range from 0 to 4.

{
  "value":  "Right",
  "chunk_id": 10    (Incorrect chunk id)
}

Good Chunk ID

Number of chunks: 5. Chunk IDs range from 0 to 4.

{
  "value":  "Right",
  "chunk_id": 2    (Correct chunk id)
}

Solution:

I considered the grounding metadata to be an unreliable source of truth, and did not use it to verify the chunk id. I modified the PlayerField model, removed the chunk_id field, and added the source_quote, uri, and source_type fields.

This architectural shift involved moving from relying on grounding metadata to asking the model to generate the citation inline within the uri field.

class PlayerField(BaseModel):
    value: str | int | float
    source_quote: str
    uri: str | None = Field(None, description="The EXACT, UNEDITED URL provided by the tool. Do not guess or shorten. None if the source is internal training data")
    source_type: Literal["GOOGLE_SEARCH", "URL_CONTEXT", "INTERNAL_KNOWLEDGE"] = Field(None, description="Categorize the source of this specific field. Must not be None when uri is not null.")

This forces Gemini to read the pages to cite specific information rather than hallucinating an answer for each field. The uri field stores either one of the provided URLs or a URL of the Google Search result. If Gemini possesses the fact and provides the answer, the uri field is None. When uri is None, the source_type must be INTERNAL_KNOWLEDGE. When uri contains vertexaisearch, the source_type is GOOGLE_SEARCH. Otherwise, the source_type is URL_CONTEXT.

3. Broken or Invalid URIs in the JSON object

Problem:

The uri in the JSON object was often a vertexaisearch proxy link or a hallucinated string, rather than the direct source URL. When uri is neither None nor one of the provided Premier League player profile pages, it is a Google Search result. Since I am using Gemini in Vertex AI, the base URL for search results is https://vertexaisearch.cloud.google.com. If the URI has a different base URL, it is very likely broken or invalid.

Solution:

I added a URI EXTRACTION RULES (STRICT) section to specify the rules to determine the uri field in the PlayerField model.

### **4. URI EXTRACTION RULES (STRICT):**
1.  **NO GUESSING:** You are strictly forbidden from constructing, autocompleting, or guessing a URL based on the website name. 
2.  **LITERAL COPY:** You must copy the `uri` exactly as it appears in the search result that provided the `source_quote`. 
3.  **THE JOIN RULE:** Before finalizing the JSON, verify that the `source_quote` actually appears in the content/snippet associated with the `uri` you provided.
4.  **IF IN DOUBT:** If you found a fact in your training data but cannot find a specific, working URI for it in the search results, you MUST set `source_type` to `INTERNAL_KNOWLEDGE` and `uri` to `null`.

Upon re-running the Colab notebook, the uri contained vertexaisearch. When I pasted a vertexaisearch URI into the browser, it redirected me to the real page.

Pro Tip: The vertexaisearch.cloud.google.com links are specific to Grounding with Google Search in Vertex AI and act as a bridge to the original source.

This was the result of Erling Haaland (Incorrect Output before the next fix).

{
  "name": "Erling Haaland",
  "net_worth": {
    "value": "$80 million",
    "source_quote": "Haaland's net worth, which according to Forbes is $80m (£59.5m)",
    "uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwe5-b036A0QaSzU9G0uAgOVZs1OJ0NMe0MmnMy-wRh4Qh9PtR_PhO6UV4S4-6ZJXJL4cicMEt0CFm_hwEER-d5WI5uY3H6DySLRcQZOmbeXF0PQU68ny6xyWF-64jJ_2Jnht_hkr7Kk3FasVjBki_2Q-n8jvr6PIcYUkTrFyJa2wZjPt4jkkdrFDo4Y6A2_OahUrg7unsUbzWJBxPp6e52tzg9w==",
    "source_type": null
  },
  "is_professional_player": true,
  "verification_status": "Confirmed active for Manchester City in the Premier League for the 2025/26 Season via official Premier League statistics.",
  "height": {
    "value": 1.95,
    "source_quote": "Height, 1.95 m (6 ft 5 in).",
    "uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHFUkt6uBYUgbC5fgMofMPjKps_9QszoJClzsHz1zkpovNWK1-OueQEj_2oFq--1dOaL7L4X9Aef7iLamfT1f4iN2aH3BxaYfJUznngJ2vXuawp-SAWl4x3R1nxHpiSCKF_pjnPg-rL",
    "source_type": null
  },
  "shirt_number": {
    "value": 9,
    "source_quote": "Man City• 9",
    "uri": "https://www.premierleague.com/en/players/223094/erling-haaland/stats",
    "source_type": null
  },
  "preferred_foot": {
    "value": "left",
    "source_quote": "Foot: left",
    "uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHf-kM-at4W05tsLLdo3dzthpWxntkWhnfwd4fWKBHxvl8qWHioz9jaDrI5ZkmhVlo71D7RQhctvDTXFqywdvl40_Q6CC2S24FujOcmdg1rhVGJiJudqtzVqLtpQ-kIWlJIkevddD68Fe40se8gRmljvgmqx4O5ATDve4F23gRmljvgmqx4O5ATDve4F23g==",
    "source_type": null
  },
  "goals": {
    "value": 20,
    "source_quote": "Goals 20",
    "uri": "https://www.premierleague.com/en/players/223094/erling-haaland/stats",
    "source_type": null
  },
  "goal_assists": {
    "value": 4,
    "source_quote": "Assists 4",
    "uri": "https://www.premierleague.com/en/players/223094/erling-haaland/stats",
    "source_type": null
  },
  "appearances": {
    "value": 22,
    "source_quote": "Appearances 22",
    "uri": "https://www.premierleague.com/en/players/223094/erling-haaland/stats",
    "source_type": null
  },
  "minutes_played": {
    "value": 1906,
    "source_quote": "Minutes Played 1,906",
    "uri": "https://www.premierleague.com/en/players/223094/erling-haaland/stats",
    "source_type": null
  }
}

One of the redirected URLs led me to https://www.sofascore.com/football/player/erling-haaland/839956 to view the preferred foot of the football player.

4. Valid URI has null Source Type

Problem:

When the player is Erling Haaland, the source_type is null even though the uri is https://www.premierleague.com/en/players/223094/erling-haaland/stats or a valid Google Search result.

When the player is Kaoru Mitoma, the model worked correctly and assigned GOOGLE_SEARCH to the source_type field of all the URIs.

This occurred because the model did not classify the source_type after successfully using the URL Context tool. Therefore, the source type is null for Erling Haaland.

When the player was Kaoru Mitoma, the model exhaustively queried Google Search and classified the source_type as GOOGLE_SEARCH.

Solution:

The solution was to provide instructions for categorizing the source_type field in the PlayerField model.

The prompt had a new MANDATORY SOURCE_TYPE CLASSIFICATION RULES section to derive the source_type field based on the uri field.

### **2. MANDATORY SOURCE_TYPE CLASSIFICATION RULES**
You are strictly forbidden from returning `null` for `source_type` if a `uri` is present.
*   **MATCHING RULE:** If the `uri` matches one of the URLs provided below, you MUST use "URL_CONTEXT".
*   **SEARCH RULE:** If the `uri` is a search result (e.g., Transfermarkt, Wikipedia, vertexaisearch links), you MUST use "GOOGLE_SEARCH".
*   **FALLBACK RULE:** If no tool found the data and you use internal memory, `uri` must be `null` and `source_type` must be "INTERNAL_KNOWLEDGE".

I re-ran the Colab notebook, and the source_type field was None when the uri field was None.

{
  "name": "Erling Haaland",
  "net_worth": {
    "value": "$80 million",
    "source_quote": "Haaland's net worth, which according to Forbes is $80m (£59.5m)",
    "uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwe5-b036A0QaSzU9G0uAgOVZs1OJ0NMe0MmnMy-wRh4Qh9PtR_PhO6UV4S4-6ZJXJL4cicMEt0CFm_hwEER-d5WI5uY3H6DySLRcQZOmbeXF0PQU68ny6xyWF-64jJ_2Jnht_hkr7Kk3FasVjBki_2Q-n8jvr6PIcYUkTrFyJa2wZjPt4jkkdrFDo4Y6A2_OahUrg7unsUbzWJBxPp6e52tzg9w==",
    "source_type": "GOOGLE_SEARCH"
  },
  "is_professional_player": true,
  "verification_status": "Confirmed active for Manchester City in the Premier League for the 2025/26 season via official Premier League statistics.",
  "height": {
    "value": 1.95,
    "source_quote": "Height, 1.95 m (6 ft 5 in).",
    "uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHFUkt6uBYUgbC5fgMofMPjKps_9QszoJClzsHz1zkpovNWK1-OueQEj_2oFq--1dOaL7L4X9Aef7iLamfT1f4iN2aH3BxaYfJUznngJ2vXuawp-SAWl4x3R1nxHpiSCKF_pjnPg-rL",
    "source_type": "GOOGLE_SEARCH"
  },
  "shirt_number": {
    "value": 9,
    "source_quote": "Man City• 9",
    "uri": "https://www.premierleague.com/en/players/223094/erling-haaland/stats",
    "source_type": "URL_CONTEXT"
  },
  "preferred_foot": {
    "value": "left",
    "source_quote": "Foot: left",
    "uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHf-kM-at4W05tsLLdo3dzthpWxntkWhnfwd4fWKBHxvl8qWHioz9jaDrI5ZkmhVlo71D7RQhctvDTXFqywdvl40_Q6CC2S24FujOcmdg1rhVGJiJudqtzVqLtpQ-kIWlJIkevddD68Fe40se8gRmljvgmqx4O5ATDve4F23gRmljvgmqx4O5ATDve4F23g==",
    "source_type": "GOOGLE_SEARCH"
  },
  "goals": {
    "value": 20,
    "source_quote": "Goals 20",
    "uri": "https://www.premierleague.com/en/players/223094/erling-haaland/stats",
    "source_type": "URL_CONTEXT"
  },
  "goal_assists": {
    "value": 4,
    "source_quote": "Assists 4",
    "uri": "https://www.premierleague.com/en/players/223094/erling-haaland/stats",
    "source_type": "URL_CONTEXT"
  },
  "appearances": {
    "value": 22,
    "source_quote": "Appearances 22",
    "uri": "https://www.premierleague.com/en/players/223094/erling-haaland/stats",
    "source_type": "URL_CONTEXT"
  },
  "minutes_played": {
    "value": 1906,
    "source_quote": "Minutes Played 1,906",
    "uri": "https://www.premierleague.com/en/players/223094/erling-haaland/stats",
    "source_type": "URL_CONTEXT"
  }
}

The Final Prompt Template

This is the complete prompt after incorporating these lessons. It has the objective (Retrieve player statistics), instructions (execute URL context tool first, and Google search second), constraints (must be an active Premier League player in 2025/2026 Season), context (the provided URLS and player name), and the output format (JSON object).

**OBJECTIVE:**
Search and identify the Premier League 2025/2026 Player Statistics of {player}.

---

### **1. DYNAMIC SOURCE IDENTIFICATION**
1.  **IF a Premier League URL is provided:**
    *   You **MUST** execute the `url_context` tool first. This is your **Primary Source**.
2.  **IF NO URL is provided (or if the player is non-PL):**
    *   The **Web Citations** (Google Search results) become your **Primary Source**. 
3.  **PRIORITY:** Official URL > Web Citations > Internal Training Data.

### **2. MANDATORY SOURCE_TYPE CLASSIFICATION RULES**
You are strictly forbidden from returning `null` for `source_type` if a `uri` is present.
*   **MATCHING RULE:** If the `uri` matches one of the URLs provided below, you MUST use "URL_CONTEXT".
*   **SEARCH RULE:** If the `uri` is a search result (e.g., Transfermarkt, Wikipedia, vertexaisearch links), you MUST use "GOOGLE_SEARCH".
*   **FALLBACK RULE:** If no tool found the data and you use internal memory, `uri` must be `null` and `source_type` must be "INTERNAL_KNOWLEDGE".

### **3. INACTIVE / NON-PROFESSIONAL PLAYER LOGIC**
If the player cannot be found in active professional records for the 2025/26 Season:
*   `is_professional_player`: `false`.
*   **All Numeric Fields:** `{{ "value": 0, "source_quote": null, "uri": null, "source_type": null }}`.
*   **All String Fields:** `{{ "value": "n/a", "source_quote": null, "uri": null, "source_type": null }}`.
*   **Verification Status:** "Player not found in active professional databases."

### **4. URI EXTRACTION RULES (STRICT):**
1.  **NO GUESSING:** You are strictly forbidden from constructing, autocompleting, or guessing a URL based on the website name. 
2.  **LITERAL COPY:** You must copy the `uri` exactly as it appears in the search result that provided the `source_quote`. 
3.  **THE JOIN RULE:** Before finalizing the JSON, verify that the `source_quote` actually appears in the content/snippet associated with the `uri` you provided.
4.  **IF IN DOUBT:** If you found a fact in your training data but cannot find a specific, working URI for it in the search results, you MUST set `source_type` to `INTERNAL_KNOWLEDGE` and `uri` to `null`.

### **5. DATA VALIDATION & AUDIT**
*   **`net_worth`**: Must be a string (e.g., `100 million dollars`).
*   **`height`**: Must be a float (e.g., `1.85`).

### PROVIDED URLS:
{ urls }

### OUTPUT FORMAT:
Return a JSON object exactly as follows:
{{
    "name": "string",
    "net_worth": {{ "value": "string", "source_quote": "...", "uri": "...", "source_type": "Google Search" }},
    "is_professional_player": boolean,
    "verification_status": "Detailed confirmation of Premier League status for 2025/26",
    "height": {{ "value": float, "source_quote": "...", "uri": "...", "source_type": "URL Context" }},
    "shirt_number": {{ "value": int, "source_quote": "...", "uri": "...", "source_type": "URL Context" }},
    "preferred_foot": {{ "value": "string", "source_quote": "...", "uri": "...", "source_type": "URL Context" }},
    "goals": {{ "value": int, "source_quote": "...", "uri": "...", "source_type": "URL Context" }},
    "goal_assists": {{ "value": int, "source_quote": "...", "uri": "...", "source_type": "URL Context" }},
    "appearances": {{ "value": int, "source_quote": "...", "uri": "...", "source_type": "URL Context" }},
    "minutes_played": {{ "value": int, "source_quote": "...", "uri": "...", "source_type": "URL Context" }}
}}

The above is a Python F-string, where the double braces {{ }} escape the JSON structures so that Python does not treat them as variables.

After formulating a prompt that generates structured output, I move on to defining Pydantic models that align with it.

Final Pydantic Models

The PlayerField model has a value field that is either a string, an integer, or a floating-point number.

The PlayerStats model represents the structured output containing performance metrics and personal information such as net worth, height, and name. The is_professional_player and verification_status fields confirm whether the player is playing in the Premier League in the 2025/2026 season.

These field descriptions are passed to the model as part of the JSON schema and they act as "micro-prompts".

class PlayerField(BaseModel):
    value: str | int | float
    source_quote: str
    uri: str | None = Field(None, description="The EXACT, UNEDITED URL provided by the tool. Do not guess or shorten. None if the source is internal training data")
    source_type: Literal["GOOGLE_SEARCH", "URL_CONTEXT", "INTERNAL_KNOWLEDGE"] = Field(None, description="Categorize the source of this specific field. Must not be None when uri is not null.")

class PlayerStats(BaseModel):
    name: str
    net_worth: PlayerField = Field(..., description="Net worth of the PL player.")
    is_professional_player: bool = Field(..., description="Must be True if found in PL records, False otherwise")
    verification_status: str = Field(..., description="Explanation of where the data was found or why it failed")
    height: PlayerField
    shirt_number: PlayerField
    preferred_foot: PlayerField
    goals: PlayerField
    goal_assists: PlayerField
    appearances: PlayerField
    minutes_played: PlayerField

Generate Structured Output with Tools

The get_player_stats function substitutes the urls and player variables in the prompt with the concatenated URLs and player name. Then, I pass the prompt to the Gemini 3 Flash Preview model to obtain a response.

The configuration specifies that the response_mime_type is application/json and the response_json_schema is the JSON schema of the PlayerStats model. The thinking level is set to High. The model also includes the URL Context tool to read Premier League player pages and the Google Search tool to query for missing details on web pages.

ThinkingLevel.High also adds latency and cost. A high thinking level causes the model to take more reasoning steps and use more thinking tokens before generating the JSON object.

from google.genai import types

def get_player_stats(player: str) -> types.GenerateContentResponse:

    prompt = f"""<original prompt from the above section>"""

    response = client.models.generate_content(
        model='gemini-3-flash-preview',
        contents=types.Content(
            role="user",
            parts=[types.Part(text=prompt)]
        ),
        config=types.GenerateContentConfig(
            response_mime_type="application/json",
            response_json_schema=PlayerStats.model_json_schema(),
            thinking_config=types.ThinkingConfig(
                thinking_level=types.ThinkingLevel.HIGH,
            ),
            tools=[
                types.Tool(url_context=types.UrlContext()),
                types.Tool(google_search=types.GoogleSearch()),
            ]
        )
    )

    return response

Get the Results

The model is proficient at returning a JSON object and it can be found in response.parsed. Then, I use PlayerStats.model_validate(...) to parse response.parsed into a player_stats instance.

When response_mime_type="application/json" is specified, the parsed field is the expected path, and parsing text is a fallback for edge cases.

If response.parsed is None, the fallback is to parse the response.text with PlayerStats.model_validate_json(...). Gemini in Vertex AI wraps the text response in a Markdown JSON code block, so the clean_json_string function removes the enclosed code block to ensure proper parsing.

def clean_json_string(raw_string):
    # Remove the markdown code blocks
    clean_str = raw_string.strip()
    if clean_str.startswith("<JSON code block>"):
        clean_str = clean_str[7:]
    if clean_str.endswith("<JSON code block>"):
        clean_str = clean_str[:-3]
    return clean_str.strip()

def print_player_stats(response: types.GenerateContentResponse):
    if response.parsed:
        player_stats = PlayerStats.model_validate(response.parsed)
    else:
        player_stats = PlayerStats.model_validate_json(clean_json_string(response.text))

    print(player_stats.model_dump_json(indent=2))

When the get_player_stats function is called with Erling Haaland, the model triggers the URL Context tool first, and then the Google Search tool. In contrast, the model triggers the Google Search tool when the player is Kaoru Mitoma and his profile page is not included in the provided URL list.

response = get_player_stats(player="Erling Haaland")
print_player_stats(response=response)

response = get_player_stats(player="Kaoru Mitoma")
print_player_stats(response)

Conclusion

I provided the URL Context and Google Search tools to allow Gemini 3 to interact with external knowledge. The internal knowledge of the Gemini 3 Flash Preview model lacks the latest player statistics of the Premier League 2025/2026 season, so it must call tools to find the statistics from the official Premier League player profile pages, and the missing net worth, height, and preferred foot from other sports pages.

I hope you find this content useful and appreciate the capabilities of the Gemini API.

Thank you.

Resources

GitHub Example: Retrieve Premier League Player Stats Colab.
Use Vertex AI for free: Vertex AI in express mode.
Run the demo in VS Code: Google Colab VS Code Extension

[GDE] Fetching Live Sports Data with Gemini 3: A Guide to Grounded, Structured JSON

Connie Leung — Wed, 21 Jan 2026 00:03:56 +0000

Introduction

Prerequisites

To run the provided Colab notebook, you will need:

Vertex AI in Express Mode: I utilize Gemini via Vertex AI due to regional availability (Hong Kong), but these features function identically in the public Gemini API. If you want to use Gemini in Vertex AI for free, you can sign up for Vertex AI in Express Mode using your Gmail account.
Google Cloud API Key: Ensure the API key is properly configured within your environment.
Google Colab VS Code Extension - Visit the Visual Studio Marketplace to install the extension to run the demo in the VS Code environment.
Python: Have Python 3.12+ and the google-genai SDK installed.

Demo Overview

We will see how to force the model to prioritize retrieval over generation. The demo attempts to retrieve player statistics for the Premier League 2025/2026 season.

The Gemini 3 Flash Preview model is given some pages:

 url_list = [
    "https://www.premierleague.com/en/players/141746/bruno-fernandes/stats",
    "https://www.premierleague.com/en/players/223094/erling-haaland/stats",
    "https://www.premierleague.com/en/players/97032/virgil-van-dijk/stats",
    "https://www.premierleague.com/en/players/244851/cole-palmer/stats"
]

urls = "\n".join(url_list)

Architecture

Environment Variable

Copy .env.example to .env and replace the GOOGLE_CLOUD_API_KEY with your API key.

GOOGLE_CLOUD_API_KEY=<GOOGLE CLOUD API KEY>

The create_vertexai_client result is assigned to a global client variable.

genai is a unified SDK for Gemini API and Gemini in Vertex AI. The vertexai=True parameter specifies usage of a Gemini in Vertex AI in the Colab notebook.

from google import genai
from dotenv import load_dotenv
from google.genai import types
from pydantic import BaseModel, Field
from typing import Literal
import os

def create_vertexai_client():

    cloud_api_key = os.getenv("GOOGLE_CLOUD_API_KEY")
    if not cloud_api_key:
        raise ValueError("GOOGLE_CLOUD_API_KEY not found in .env file")

    # Configure the client with your API key
    client = genai.Client(
        vertexai=True, 
        api_key=cloud_api_key, 
    )

    return client

load_dotenv()

client = create_vertexai_client()

Installation

The demo uses the newer Google Gen AI SDK, so I install the google-genai library.

%pip install google-genai
%pip install dotenv
%pip install pydantic

Lessons Learned

I will describe the challenges faced in this demo and how I tackled each one.

1. Lazy Behavior of Gemini 3 Flash Preview Model

Problem:

Solution:

### **1. DYNAMIC SOURCE IDENTIFICATION**
1.  **IF a Premier League URL is provided:**
    *   You **MUST** execute the `url_context` tool first. This is your **Primary Source**.
2.  **IF NO URL is provided (or if the player is non-Premier League):**
    *   The **Web Citations** (Google Search results) become your **Primary Source**. 
3.  **PRIORITY:** Official URL > Web Citations > Internal Training Data.

2. Chunk ID Hallucination and Low Confidence in Grounding Metadata

Problem:

class PlayerField(BaseModel):
    value: str | int | float
    chunk_id: int = Field(..., description="Index of the chunk in the grounding metadata that provide the value")

class PlayerStats(BaseModel):
    name: str
    net_worth: PlayerField | None = Field(..., description="Net worth of the Premier League player.")
    is_professional_player: bool = Field(..., description="Must be True if found in Premier League records, False otherwise")
    verification_status: str = Field(..., description="Explanation of where the data was found or why it failed")
    height: PlayerField
    shirt_number: PlayerField
    preferred_foot: PlayerField
    goals: PlayerField
    goal_assists: PlayerField
    appearances: PlayerField
    minutes_played: PlayerField

Moreover, I wrote functions to obtain the web and retrieved context URIs from the grounding metadata to verify the Chunk ID.

def get_citations(response: types.GenerateContentResponse) -> tuple[int, list[dict]]:

    citations: list[dict] = []
    if response.candidates is not None and len(response.candidates) > 0:
        candidate = response.candidates[0]
        if candidate.grounding_metadata:
            grounding_chunks = candidate.grounding_metadata.grounding_chunks or []
            num_chunks = len(grounding_chunks)
            for i, chunk in enumerate(grounding_chunks):
                if chunk and chunk.web and chunk.web.uri:
                    citations.append({
                        "chunk_id": i,
                        "uri": chunk.web.uri,
                        "title": chunk.web.title
                    })
                elif chunk and chunk.retrieved_context and chunk.retrieved_context.uri:
                    citations.append({
                        "chunk_id": i,
                        "uri": chunk.retrieved_context.uri,
                        "title": chunk.retrieved_context.text
                    })

    return num_chunks, citations

def print_citations_by_response(response: types.GenerateContentResponse):
    num_chunks, citations = get_citations(response)

    print("num_chunks ->", num_chunks)
    for i, citation in enumerate(citations):
        print(f"Citation {i}: {citation}")

The chunk_id in the PlayerField was larger than num_chunks, which indicated that the model hallucinated and created a chunk_id that did not exist in the grounding metadata.

Bad Chunk ID

Number of chunks: 5. Chunk IDs range from 0 to 4.

{
  "value":  "Right",
  "chunk_id": 10    (Incorrect chunk id)
}

Good Chunk ID

Number of chunks: 5. Chunk IDs range from 0 to 4.

{
  "value":  "Right",
  "chunk_id": 2    (Correct chunk id)
}

Solution:

This architectural shift involved moving from relying on grounding metadata to asking the model to generate the citation inline within the uri field.

class PlayerField(BaseModel):
    value: str | int | float
    source_quote: str
    uri: str | None = Field(None, description="The EXACT, UNEDITED URL provided by the tool. Do not guess or shorten. None if the source is internal training data")
    source_type: Literal["GOOGLE_SEARCH", "URL_CONTEXT", "INTERNAL_KNOWLEDGE"] = Field(None, description="Categorize the source of this specific field. Must not be None when uri is not null.")

3. Broken or Invalid URIs in the JSON object

Problem:

Solution:

I added a URI EXTRACTION RULES (STRICT) section to specify the rules to determine the uri field in the PlayerField model.

### **4. URI EXTRACTION RULES (STRICT):**
1.  **NO GUESSING:** You are strictly forbidden from constructing, autocompleting, or guessing a URL based on the website name. 
2.  **LITERAL COPY:** You must copy the `uri` exactly as it appears in the search result that provided the `source_quote`. 
3.  **THE JOIN RULE:** Before finalizing the JSON, verify that the `source_quote` actually appears in the content/snippet associated with the `uri` you provided.
4.  **IF IN DOUBT:** If you found a fact in your training data but cannot find a specific, working URI for it in the search results, you MUST set `source_type` to `INTERNAL_KNOWLEDGE` and `uri` to `null`.

Upon re-running the Colab notebook, the uri contained vertexaisearch. When I pasted a vertexaisearch URI into the browser, it redirected me to the real page.

Pro Tip: The vertexaisearch.cloud.google.com links are specific to Grounding with Google Search in Vertex AI and act as a bridge to the original source.

This was the result of Erling Haaland (Incorrect Output before the next fix).

{
  "name": "Erling Haaland",
  "net_worth": {
    "value": "$80 million",
    "source_quote": "Haaland's net worth, which according to Forbes is $80m (£59.5m)",
    "uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwe5-b036A0QaSzU9G0uAgOVZs1OJ0NMe0MmnMy-wRh4Qh9PtR_PhO6UV4S4-6ZJXJL4cicMEt0CFm_hwEER-d5WI5uY3H6DySLRcQZOmbeXF0PQU68ny6xyWF-64jJ_2Jnht_hkr7Kk3FasVjBki_2Q-n8jvr6PIcYUkTrFyJa2wZjPt4jkkdrFDo4Y6A2_OahUrg7unsUbzWJBxPp6e52tzg9w==",
    "source_type": null
  },
  "is_professional_player": true,
  "verification_status": "Confirmed active for Manchester City in the Premier League for the 2025/26 Season via official Premier League statistics.",
  "height": {
    "value": 1.95,
    "source_quote": "Height, 1.95 m (6 ft 5 in).",
    "uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHFUkt6uBYUgbC5fgMofMPjKps_9QszoJClzsHz1zkpovNWK1-OueQEj_2oFq--1dOaL7L4X9Aef7iLamfT1f4iN2aH3BxaYfJUznngJ2vXuawp-SAWl4x3R1nxHpiSCKF_pjnPg-rL",
    "source_type": null
  },
  "shirt_number": {
    "value": 9,
    "source_quote": "Man City• 9",
    "uri": "https://www.premierleague.com/en/players/223094/erling-haaland/stats",
    "source_type": null
  },
  "preferred_foot": {
    "value": "left",
    "source_quote": "Foot: left",
    "uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHf-kM-at4W05tsLLdo3dzthpWxntkWhnfwd4fWKBHxvl8qWHioz9jaDrI5ZkmhVlo71D7RQhctvDTXFqywdvl40_Q6CC2S24FujOcmdg1rhVGJiJudqtzVqLtpQ-kIWlJIkevddD68Fe40se8gRmljvgmqx4O5ATDve4F23gRmljvgmqx4O5ATDve4F23g==",
    "source_type": null
  },
  "goals": {
    "value": 20,
    "source_quote": "Goals 20",
    "uri": "https://www.premierleague.com/en/players/223094/erling-haaland/stats",
    "source_type": null
  },
  "goal_assists": {
    "value": 4,
    "source_quote": "Assists 4",
    "uri": "https://www.premierleague.com/en/players/223094/erling-haaland/stats",
    "source_type": null
  },
  "appearances": {
    "value": 22,
    "source_quote": "Appearances 22",
    "uri": "https://www.premierleague.com/en/players/223094/erling-haaland/stats",
    "source_type": null
  },
  "minutes_played": {
    "value": 1906,
    "source_quote": "Minutes Played 1,906",
    "uri": "https://www.premierleague.com/en/players/223094/erling-haaland/stats",
    "source_type": null
  }
}

One of the redirected URLs led me to https://www.sofascore.com/football/player/erling-haaland/839956 to view the preferred foot of the football player.

4. Valid URI has null Source Type

Problem:

When the player is Erling Haaland, the source_type is null even though the uri is https://www.premierleague.com/en/players/223094/erling-haaland/stats or a valid Google Search result.

When the player is Kaoru Mitoma, the model worked correctly and assigned GOOGLE_SEARCH to the source_type field of all the URIs.

This occurred because the model did not classify the source_type after successfully using the URL Context tool. Therefore, the source type is null for Erling Haaland.

When the player was Kaoru Mitoma, the model exhaustively queried Google Search and classified the source_type as GOOGLE_SEARCH.

Solution:

The solution was to provide instructions for categorizing the source_type field in the PlayerField model.

The prompt had a new MANDATORY SOURCE_TYPE CLASSIFICATION RULES section to derive the source_type field based on the uri field.

### **2. MANDATORY SOURCE_TYPE CLASSIFICATION RULES**
You are strictly forbidden from returning `null` for `source_type` if a `uri` is present.
*   **MATCHING RULE:** If the `uri` matches one of the URLs provided below, you MUST use "URL_CONTEXT".
*   **SEARCH RULE:** If the `uri` is a search result (e.g., Transfermarkt, Wikipedia, vertexaisearch links), you MUST use "GOOGLE_SEARCH".
*   **FALLBACK RULE:** If no tool found the data and you use internal memory, `uri` must be `null` and `source_type` must be "INTERNAL_KNOWLEDGE".

I re-ran the Colab notebook, and the source_type field was None when the uri field was None.

{
  "name": "Erling Haaland",
  "net_worth": {
    "value": "$80 million",
    "source_quote": "Haaland's net worth, which according to Forbes is $80m (£59.5m)",
    "uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwe5-b036A0QaSzU9G0uAgOVZs1OJ0NMe0MmnMy-wRh4Qh9PtR_PhO6UV4S4-6ZJXJL4cicMEt0CFm_hwEER-d5WI5uY3H6DySLRcQZOmbeXF0PQU68ny6xyWF-64jJ_2Jnht_hkr7Kk3FasVjBki_2Q-n8jvr6PIcYUkTrFyJa2wZjPt4jkkdrFDo4Y6A2_OahUrg7unsUbzWJBxPp6e52tzg9w==",
    "source_type": "GOOGLE_SEARCH"
  },
  "is_professional_player": true,
  "verification_status": "Confirmed active for Manchester City in the Premier League for the 2025/26 season via official Premier League statistics.",
  "height": {
    "value": 1.95,
    "source_quote": "Height, 1.95 m (6 ft 5 in).",
    "uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHFUkt6uBYUgbC5fgMofMPjKps_9QszoJClzsHz1zkpovNWK1-OueQEj_2oFq--1dOaL7L4X9Aef7iLamfT1f4iN2aH3BxaYfJUznngJ2vXuawp-SAWl4x3R1nxHpiSCKF_pjnPg-rL",
    "source_type": "GOOGLE_SEARCH"
  },
  "shirt_number": {
    "value": 9,
    "source_quote": "Man City• 9",
    "uri": "https://www.premierleague.com/en/players/223094/erling-haaland/stats",
    "source_type": "URL_CONTEXT"
  },
  "preferred_foot": {
    "value": "left",
    "source_quote": "Foot: left",
    "uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHf-kM-at4W05tsLLdo3dzthpWxntkWhnfwd4fWKBHxvl8qWHioz9jaDrI5ZkmhVlo71D7RQhctvDTXFqywdvl40_Q6CC2S24FujOcmdg1rhVGJiJudqtzVqLtpQ-kIWlJIkevddD68Fe40se8gRmljvgmqx4O5ATDve4F23gRmljvgmqx4O5ATDve4F23g==",
    "source_type": "GOOGLE_SEARCH"
  },
  "goals": {
    "value": 20,
    "source_quote": "Goals 20",
    "uri": "https://www.premierleague.com/en/players/223094/erling-haaland/stats",
    "source_type": "URL_CONTEXT"
  },
  "goal_assists": {
    "value": 4,
    "source_quote": "Assists 4",
    "uri": "https://www.premierleague.com/en/players/223094/erling-haaland/stats",
    "source_type": "URL_CONTEXT"
  },
  "appearances": {
    "value": 22,
    "source_quote": "Appearances 22",
    "uri": "https://www.premierleague.com/en/players/223094/erling-haaland/stats",
    "source_type": "URL_CONTEXT"
  },
  "minutes_played": {
    "value": 1906,
    "source_quote": "Minutes Played 1,906",
    "uri": "https://www.premierleague.com/en/players/223094/erling-haaland/stats",
    "source_type": "URL_CONTEXT"
  }
}

The Final Prompt Template

**OBJECTIVE:**
Search and identify the Premier League 2025/2026 Player Statistics of {player}.

---

### **1. DYNAMIC SOURCE IDENTIFICATION**
1.  **IF a Premier League URL is provided:**
    *   You **MUST** execute the `url_context` tool first. This is your **Primary Source**.
2.  **IF NO URL is provided (or if the player is non-PL):**
    *   The **Web Citations** (Google Search results) become your **Primary Source**. 
3.  **PRIORITY:** Official URL > Web Citations > Internal Training Data.

### **2. MANDATORY SOURCE_TYPE CLASSIFICATION RULES**
You are strictly forbidden from returning `null` for `source_type` if a `uri` is present.
*   **MATCHING RULE:** If the `uri` matches one of the URLs provided below, you MUST use "URL_CONTEXT".
*   **SEARCH RULE:** If the `uri` is a search result (e.g., Transfermarkt, Wikipedia, vertexaisearch links), you MUST use "GOOGLE_SEARCH".
*   **FALLBACK RULE:** If no tool found the data and you use internal memory, `uri` must be `null` and `source_type` must be "INTERNAL_KNOWLEDGE".

### **3. INACTIVE / NON-PROFESSIONAL PLAYER LOGIC**
If the player cannot be found in active professional records for the 2025/26 Season:
*   `is_professional_player`: `false`.
*   **All Numeric Fields:** `{{ "value": 0, "source_quote": null, "uri": null, "source_type": null }}`.
*   **All String Fields:** `{{ "value": "n/a", "source_quote": null, "uri": null, "source_type": null }}`.
*   **Verification Status:** "Player not found in active professional databases."

### **4. URI EXTRACTION RULES (STRICT):**
1.  **NO GUESSING:** You are strictly forbidden from constructing, autocompleting, or guessing a URL based on the website name. 
2.  **LITERAL COPY:** You must copy the `uri` exactly as it appears in the search result that provided the `source_quote`. 
3.  **THE JOIN RULE:** Before finalizing the JSON, verify that the `source_quote` actually appears in the content/snippet associated with the `uri` you provided.
4.  **IF IN DOUBT:** If you found a fact in your training data but cannot find a specific, working URI for it in the search results, you MUST set `source_type` to `INTERNAL_KNOWLEDGE` and `uri` to `null`.

### **5. DATA VALIDATION & AUDIT**
*   **`net_worth`**: Must be a string (e.g., `100 million dollars`).
*   **`height`**: Must be a float (e.g., `1.85`).

### PROVIDED URLS:
{ urls }

### OUTPUT FORMAT:
Return a JSON object exactly as follows:
{{
    "name": "string",
    "net_worth": {{ "value": "string", "source_quote": "...", "uri": "...", "source_type": "Google Search" }},
    "is_professional_player": boolean,
    "verification_status": "Detailed confirmation of Premier League status for 2025/26",
    "height": {{ "value": float, "source_quote": "...", "uri": "...", "source_type": "URL Context" }},
    "shirt_number": {{ "value": int, "source_quote": "...", "uri": "...", "source_type": "URL Context" }},
    "preferred_foot": {{ "value": "string", "source_quote": "...", "uri": "...", "source_type": "URL Context" }},
    "goals": {{ "value": int, "source_quote": "...", "uri": "...", "source_type": "URL Context" }},
    "goal_assists": {{ "value": int, "source_quote": "...", "uri": "...", "source_type": "URL Context" }},
    "appearances": {{ "value": int, "source_quote": "...", "uri": "...", "source_type": "URL Context" }},
    "minutes_played": {{ "value": int, "source_quote": "...", "uri": "...", "source_type": "URL Context" }}
}}

The above is a Python F-string, where the double braces {{ }} escape the JSON structures so that Python does not treat them as variables.

After formulating a prompt that generates structured output, I move on to defining Pydantic models that align with it.

Final Pydantic Models

The PlayerField model has a value field that is either a string, an integer, or a floating-point number.

These field descriptions are passed to the model as part of the JSON schema and they act as "micro-prompts".

class PlayerField(BaseModel):
    value: str | int | float
    source_quote: str
    uri: str | None = Field(None, description="The EXACT, UNEDITED URL provided by the tool. Do not guess or shorten. None if the source is internal training data")
    source_type: Literal["GOOGLE_SEARCH", "URL_CONTEXT", "INTERNAL_KNOWLEDGE"] = Field(None, description="Categorize the source of this specific field. Must not be None when uri is not null.")

class PlayerStats(BaseModel):
    name: str
    net_worth: PlayerField = Field(..., description="Net worth of the PL player.")
    is_professional_player: bool = Field(..., description="Must be True if found in PL records, False otherwise")
    verification_status: str = Field(..., description="Explanation of where the data was found or why it failed")
    height: PlayerField
    shirt_number: PlayerField
    preferred_foot: PlayerField
    goals: PlayerField
    goal_assists: PlayerField
    appearances: PlayerField
    minutes_played: PlayerField

Generate Structured Output with Tools

ThinkingLevel.High also adds latency and cost. A high thinking level causes the model to take more reasoning steps and use more thinking tokens before generating the JSON object.

from google.genai import types

def get_player_stats(player: str) -> types.GenerateContentResponse:

    prompt = f"""<original prompt from the above section>"""

    response = client.models.generate_content(
        model='gemini-3-flash-preview',
        contents=types.Content(
            role="user",
            parts=[types.Part(text=prompt)]
        ),
        config=types.GenerateContentConfig(
            response_mime_type="application/json",
            response_json_schema=PlayerStats.model_json_schema(),
            thinking_config=types.ThinkingConfig(
                thinking_level=types.ThinkingLevel.HIGH,
            ),
            tools=[
                types.Tool(url_context=types.UrlContext()),
                types.Tool(google_search=types.GoogleSearch()),
            ]
        )
    )

    return response

Get the Results

The model is proficient at returning a JSON object and it can be found in response.parsed. Then, I use PlayerStats.model_validate(...) to parse response.parsed into a player_stats instance.

When response_mime_type="application/json" is specified, the parsed field is the expected path, and parsing text is a fallback for edge cases.

def clean_json_string(raw_string):
    # Remove the markdown code blocks
    clean_str = raw_string.strip()
    if clean_str.startswith("<JSON code block>"):
        clean_str = clean_str[7:]
    if clean_str.endswith("<JSON code block>"):
        clean_str = clean_str[:-3]
    return clean_str.strip()

def print_player_stats(response: types.GenerateContentResponse):
    if response.parsed:
        player_stats = PlayerStats.model_validate(response.parsed)
    else:
        player_stats = PlayerStats.model_validate_json(clean_json_string(response.text))

    print(player_stats.model_dump_json(indent=2))

response = get_player_stats(player="Erling Haaland")
print_player_stats(response=response)

response = get_player_stats(player="Kaoru Mitoma")
print_player_stats(response)

Conclusion

I hope you find this content useful and appreciate the capabilities of the Gemini API.

Thank you.

Resources

GitHub Example: Retrieve Premier League Player Stats Colab.
Use Vertex AI for free: Vertex AI in express mode.
Run the demo in VS Code: Google Colab VS Code Extension

[GDE] Simplifying Multimodal Inputs: Using Public URLs with the Gemini API

Connie Leung — Sat, 17 Jan 2026 12:22:34 +0000

In this blog post, I will explore the enhanced inputs in the Gemini API and how they simplify my Colab demo. I will outline the recent updates to the Gemini API, walk through the legacy implementation, and finally demonstrate how to replace a complex utility function with just two lines of code.

Introduction

As of January 2026, the Gemini API has significantly increased the limit of inline data and made data ingestion more flexible. The maximum size of the inline data has increased from 20 MB to 100 MB per file. While the increase in inline data size is significant, my demo is about adopting external public URLs instead of local files.

Moreover, the Gemini API supports external public HTTPS URLs and signed URLs. Examples of signed URLs are S3 Presigned URLs or Azure Blob Storage shared access signatures (SAS).

If the files are ready in Google Cloud Storage (GCS), their URIs can be registered and used across multiple requests without re-uploading.

The primary advantage of public URLs is eliminating the client-side upload step. The model fetches data directly from the source, removing the need to manage local byte streams. While re-uploading works fine for prototyping, it becomes a bottleneck in production where data persistence is key.

In summary, these are the changes:

Limit / Capability

Inline data limit increased to 100 MB
Support external URLs (Public HTTPS or Signed)
Support for registered Google Cloud Storage URIs and use them in the requests

Type	Supported File Types
Text file types	text/html, text/css, text/plain, text/xml, text/csv, text/rtf, text/javascript
Application file types	application/json, application/pdf
Image file types	image/bmp, image/jpeg, image/png, image/webp

Prerequisites

To run the provided Colab notebook, you will need:

Vertex AI in Express Mode: Due to regional availability (I am based in Hong Kong), I utilize Gemini via Vertex AI. However, these features apply to both. Therefore, all my demos use Gemini in Vertex AI. If you want to use Gemini in Vertex AI for free, you can sign up for Vertex AI in express mode using your Gmail account.
Google Cloud API Key: Properly configured within your environment.
Google Colab VS Code Extension - Visit Visual Studio Marketplace to install the extension to run the demo in the VS Code environment.
Python: Have Python 3.12+ and the google-genai SDK installed

Demo Overview

The demo performs image understanding by comparing pictures of two people to identify their facial similarities and differences. The result is a structured output containing a similarity score, a list of similarities, and a list of differences.

Initially, the images were local to my demo, and I wrote a utility function to open an image file, read the raw bytes, and wrap them in a Gemini Part object. Then, the text prompt, and two inline data parts are passed to the Gemini API to generate the structured output.

After the announcement of the enhancements, I moved the images to a public GitHub Repo, and refactored the code to accept public image URLs instead of local files. This mimics a production environment where assets are hosted remotely (e.g., S3, GCS, remote server) rather than bundled locally.

Note: This demo is for educational purposes to demonstrate multimodal capabilities. Always review the AI provider's Acceptable Use Policy regarding biometric categorization and facial recognition before deploying such features in production.

Architecture

The Gemini API ingests two public image URLs, and passes them along with the text prompt to the Gemini 3 Flash Preview model. The model compares the images to generate a similarity score, produce a list of similarities, and a list of differences.

Environment Variable

Copy env.example to .env and replace the placeholder of GOOGLE_CLOUD_API_KEY with the API key.

GOOGLE_CLOUD_API_KEY=<GOOGLE CLOUD API KEY>

The create_vertexai_client function constructs and returns a Gemini client. I use the client to call Gemini 3 Flash Preview model to analyze images and generate a structured output.

import os
from google import genai
from matplotlib import pyplot as plt
from dotenv import load_dotenv
from google.genai import types
from pydantic import BaseModel, Field

def create_vertexai_client():
    cloud_api_key = os.getenv("GOOGLE_CLOUD_API_KEY")
    if not cloud_api_key:
        raise ValueError("GOOGLE_CLOUD_API_KEY not found in .env file")

    # Configure the client with your API key
    client = genai.Client(
        vertexai=True, 
        api_key=cloud_api_key, 
    )

    return client

Installation

The demo uses the newer Google Gen AI SDK, so I install the google-genai library.

%pip install google-genai
%pip install matplotlib
%pip install os
%pip install dotenv
%pip install pydantic

The Legacy Approach: Local File Uploads

1. Image Understanding with Thinking Level and Media Resolution (Facial Analysis)

This utility function opens an image file, reads the image bytes, and returns an inline data part.

def get_inline_data_part(image_path: str):
    import mimetypes

    try:
        mime_type, _ = mimetypes.guess_type(image_path)
        if mime_type is None:
            mime_type = 'application/octet-stream'
            print(f"Warning: Could not determine MIME type for {image_path}. Defaulting to {mime_type}.")

        file_bytes: bytes | None = None
        with open(image_path, "rb") as image_file:
            file_bytes = image_file.read()

        if file_bytes is None:
            raise Exception(f"Unable to read the bytes from {image_path}")

        return types.Part(
            inline_data=types.Blob(
                mime_type=mime_type,
                data=file_bytes
            ),
            media_resolution={"level": "media_resolution_high"}
        ) 
    except FileNotFoundError:
        print (f"Error: The file was not found at {image_path}")
    except Exception as e:
        print (f"An error occurred: {e}")

The structured output of the prompt adheres to the schema of the FacialAnalysis model.

class FacialAnalysis(BaseModel):
    percentage: int = Field(description="Integer similarity score.")
    similarities: list[str] = Field(description="Key similarities.")
    differences: list[str] = Field(description="Key differences.")

With response_mime_type='application/json', the model is very good at returning JSON object, and it can be found in response.parse. If response.parse fails, the fallback is to parse reponse.text to an instance of FacialAnalysis model.

The Gemini model often wraps the response.text in Markdown code blocks, so the clean_json_string function removes the code blocks to ensure clean parsing.

def clean_json_string(raw_string):
    # Remove the markdown code blocks
    clean_str = raw_string.strip()
    if clean_str.startswith("<triple backticks>json"):
        clean_str = clean_str[7:]
    if clean_str.endswith("<triple ticks>"):
        clean_str = clean_str[:-3]
    return clean_str.strip()

Here, I use the Gemini 3 Flash Preview model to compare two images, applying a high thinking level and high media resolution to ensure a detailed analytical response.

config=types.GenerateContentConfig(
    response_mime_type='application/json',
    response_json_schema=FacialAnalysis.model_json_schema(),
    media_resolution=types.MediaResolution.MEDIA_RESOLUTION_HIGH,
    thinking_config=types.ThinkingConfig(
        include_thoughts=True,
        thinking_level=types.ThinkingLevel.HIGH
    )
)

response = client.models.generate_content(
    model='gemini-3-flash-preview',
    contents=[
        types.Content(
            role="user",
            parts=[
                types.Part(text=prompt),
                get_inline_data_part(person_a_image),
                get_inline_data_part(person_b_image),
            ],
        ),
    ],
    config=config
)

The complete implementation of analyze_facial_images is below:

def analyze_facial_images(person_a_image: str, person_b_image: str):
    prompt = """
    Role: You are an Expert Biometric Analyst and Facial Recognition Specialist. Your expertise lies in anthropometric comparison, analyzing facial landmarks, bone structure, and morphological traits.

    Task: Analyze the attached image(s). Compare the physical appearance of the two individuals shown. If two separate images are provided, treat the first as Person A and the second as Person B. If one image containing two people is provided, treat the person on the left as Person A and the person on the right as Person B.

    Analysis Criteria:
    Focus strictly on biological and physical traits (facial geometry, feature shapes, and proportions). Ignore clothing, lighting, camera angles, or accessories (glasses, hats) unless they obscure features.

    The Rule of Similarity (Visual Scoring Rubric):
    You must assign a final similarity percentage based strictly on this scale:

    0 - 10 (Dissimilar): Polar opposites. No shared facial geometry or features; different phenotypes.
    11 - 20 (Faint Connection): Very weak link; perhaps a shared broad head shape, but all specific features differ.
    21 - 30 (Vague Resemblance): Superficial similarities only (e.g., similar eye color or hair texture only), but the faces look unrelated.
    31 - 40 (Partial Overlap): Noticeable but minor similarities. Maybe the nose or mouth is similar, but the overall bone structure is different.
    41 - 50 (Moderate Association): Reminiscent. One major anatomical zone aligns (e.g., eyes/brows), but the rest of the face is distinct.
    51 - 60% (Balanced Similarity): Comparable. Distinct individuals, but there is enough overlap in bone structure to suggest a relation.
    61 - 70 (Strong Resemblance): "Cousin" status. Strong resemblance in the "map" of the face; differences are only in the details/nuance.
    71 - 80 (Kindred Spirits): High resemblance. They share facial ratios that suggest a blood relation (e.g., parent/child or first cousins), but are clearly distinguishable as different people.
    81 - 90 (Biological Sibling / Doppelgänger): Extremely high correlation. This tier represents biological sisters/brothers who look very similar, or unrelated high-grade lookalikes. They share almost all facial features, with only minor structural variances.
    91 - 100 (Identical Match): The same person or identical twins. Facial geometry and bone structure are exact matches. Any visible differences are strictly superficial (e.g., age, styling, or weight) and not structural.

    Output Format:
    Please provide the result strictly in the following format:

    Similarity Score: [Insert Integer]%

    Key Similarities:
    - [Detail 1]
    - [Detail 2]
    - [Detail 3]
    - [Detail 4]
    - [Detail 5]

    Key Differences:
    - [Detail 1]
    - [Detail 2]
    - [Detail 3]
    - [Detail 4]
    - [Detail 5]
    """

    response = client.models.generate_content(
        model="gemini-3-flash-preview",
        contents=[
            types.Content(
                role="user",
                parts=[
                    types.Part(text=prompt),
                    get_inline_data_part(person_a_image),
                    get_inline_data_part(person_b_image),
                ]
            )
        ],
        config=types.GenerateContentConfig(
            response_mime_type="application/json",
            response_json_schema=FacialAnalysis.model_json_schema(),
            media_resolution=types.MediaResolution.MEDIA_RESOLUTION_HIGH,
            thinking_config=types.ThinkingConfig(
                thinking_level=types.ThinkingLevel.HIGH
            )
        )
    )

    if response.parsed:
        result = FacialAnalysis.model_validate(response.parsed)
    else:
        result = FacialAnalysis.model_validate_json(
            clean_json_string(response.text)
        )
    return result

2. Prompt Gemini 3 and Print results

The print_result function invokes analyze_facial_images to generate the response. It displays the similarity score in percentage. Then, it iterates the similarities list to display all similarities. Finally, it iterates the differences list to display all differences.

def print_result(person_a_image: str, person_b_image: str):
    result = analyze_facial_images(person_a_image=person_a_image, person_b_image=person_b_image)

    print("Percentage: ", result.percentage)
    print ("Similarities:")
    for s in result.similarities:
        print("- ", s)

    print ("Differences:")
    for d in result.differences:
        print("-",  d)
    print("========================================================================")

def print_test_cases(heading: str, cases: list[list[str]]):
    print(heading)
    for case in cases:
        print_result(person_a_image=case[0], person_b_image=case[1])

3. Prepare a Test Case

I ask Gemini 3 to find the similarity score between the Olsen twins (Mary Kate and Ashley).

identical_cases = [
    ['./images/Ashley_Olsen.jpg', './images/Mary_Kate_Olsen.jpg'],
]

print_test_cases(heading="Identical twin cases", cases=identical_cases)

The output is as follows:

Percentage:  88
Similarities:
-  Identical large, almond-shaped green eye morphology and orbital positioning.
-  Consistent thick, arched eyebrow structure and density.
-  Nearly identical nasal bridge width and rounded tip shape.
-  Similar lip proportions, specifically the defined cupid's bow and lower lip fullness.
-  Matching heart-shaped facial geometry and prominent cheekbone structure.
Differences:
- Slightly different buccal fat distribution, with Person A showing more defined facial hollowing.
- Minor variance in chin projection and tapering at the mandibular symphysis.
- Subtle differences in the vertical height of the forehead and hairline positioning.
- Person A displays slightly more pronounced nasolabial folds in the mid-face region.
- Slight variations in the shape and protrusion of the ears relative to the jawline.
========================================================================

This is the code walkthrough of the old code.

The Latest Approach: Direct URL Ingestion

1. Move the images to a public GitHub Repository

I created a new repository, added an images folder, and upload the images to it.

2. Replace inline data with public URLs

The analyze_facial_images function uses types.Part.from_uri to create a part from a public image URL. This is crucial for types.Part.from_uri to work with raw GitHub URLs. MIME type is optional in types.Part.from_uri. The API infers it from the URL when it is not provided.

However, if you are using signed URLs that do not end in a file extension, you should explicitly pass the mime_type parameter to from_uri to ensure the model processes it correctly.

The function expects URLs and the parameters are named from person_a_image and person_b_image to person_a_image_url and person_b_image_url. In this version, the function uses the same configuration to generate content.

client, types and FacialAnalysis model are imported from the previous steps.

def analyze_facial_images(person_a_image_url: str, person_b_image_url: str):
    prompt = """
    <original prompt>
    """

    response = client.models.generate_content(
        model="gemini-3-flash-preview",
        contents=[
            types.Content(
                role="user",
                parts=[
                    types.Part(text=prompt),
                    types.Part.from_uri(file_uri=person_a_image_url),
                    types.Part.from_uri(file_uri=person_b_image_url),
                ]
            )
        ],
        config=types.GenerateContentConfig(
            response_mime_type="application/json",
            response_json_schema=FacialAnalysis.model_json_schema(),
            media_resolution=types.MediaResolution.MEDIA_RESOLUTION_HIGH,
            thinking_config=types.ThinkingConfig(
                thinking_level=types.ThinkingLevel.HIGH
            )
        )
    )

    if response.parsed:
        result = FacialAnalysis.model_validate(response.parsed)
    else:
        result = FacialAnalysis.model_validate_json(
            clean_json_string(response.text)
        )

    return result

3. URLs construction in `print_result`

The print_result constructs the full image URLs from a base URL, and invokes analyze_facial_images.
In the demo, the base URL is https://raw.githubusercontent.com/railsstudent/colab_images/refs/heads/main/images. GitHub.com URLs return an HTML page whereas raw.githubusercontent.com returns the actual image bytes, which is what the model needs.

No change to the print logic in print_result and the print_test_cases function stays unchanged.

def print_result(person_a_file_name: str, person_b_file_name: str):
    base_url = "https://raw.githubusercontent.com/railsstudent/colab_images/refs/heads/main/images"
    person_a_image_url = f"{base_url}/{person_a_file_name}"
    person_b_image_url = f"{base_url}/{person_b_file_name}"

    result = analyze_facial_images(
        person_a_image_url=person_a_image_url, person_b_image_url=person_b_image_url
    )

    ... print the score, a list of similarities, and a list of differences ...

4. Delete the unused function

Delete get_inline_data_part because it is redundant now.

5. Fix the test case

Modify the test cases because the images are not in the local images folder.

identical_cases = [
    ["Ashley_Olsen.jpg", f"Mary_Kate_Olsen.jpg"],
]

print_test_cases(heading="Identical twin cases", cases=identical_cases)

Rerun the code and the same result should be achieved.

The solution works because the GitHub repo is public, which makes the images in the images folder publicly accessible.

Conclusion

Gemini File API streamlines data ingestion by increasing the inline data size to 100 MB, allowing public HTTPS URLs, signed URLs, and registered files in Google Cloud Storage. Developers can fetch data from the URLs or GCS URIs, and pass them to a Gemini 3 model to generate text output or structured output. This update deprecates the need for boilerplate utility functions that convert images to inline data parts before passing them to the Gemini API for image analysis and generating structured output.

I hope you enjoy the content and appreciate the ease of use of the Gemini API.

Thank you.

Resources

GitHub Example: Look-alike Colab.
Use VertexAI for free: Vertex AI in express mode.
Run the demo in VS Code: Google Colab VS Code Extension