DEV Community: Connie Leung

Build Firebase AI Logic Application with Antigravity CLI and Stitch MCP Server [GDE]

Connie Leung — Wed, 15 Jul 2026 06:50:10 +0000

Build Firebase AI Logic with Antigravity CLI

Note: Google Cloud credits are provided for this project.

In this blog post, I demonstrate how to use the Antigravity CLI (an agentic AI assistant integrating directly with development workflows via skills and servers) to build an image analysis demo using Angular, the Firebase Hybrid & On-device Inference Web SDK, and Gemini models. Users upload an image and use a Gemini model to analyze it to generate a few alternative texts, tags, recommendations, and CSS tips to enhance the image quality.

When the demo is running in Chrome 148+, the Hybrid & On-device SDK leverages the Prompt API of the on-device Gemini Nano model to perform the image-to-text tasks, and the token usage is 0. When other browsers, such as Safari or Firefox, execute the same tasks, the SDK falls back to Cloud AI (Gemini 3.5 Flash model), which consumes tokens.

Next, I describe how to install the skills in my Angular project and register the Angular and Stitch MCP servers in the Antigravity CLI to develop the infrastructure, services, and UI design of my demo.

1. Workflow

This is my entire workflow from implementing features, generating UI screens, and mapping the screens to Angular components.

2. Skills

I installed the grill-with-docs, angular, and firebase skills in my project for the following reasons:

grill-with-docs: Conduct a rigid Q&A session to generate a specification for a feature, refactor, or critical fix. AI is responsible for performing thorough analysis, and putting in more efforts to generate code to achieve the task.
domain-modeling: The skill is referenced in the SKILL.md of the grill-with-docs skill, so a copy of it is required.
code-review: Spawn two sub-agents to review changes to detect code smells and verify that the changes align with the specification.
angular: Provide the best practices of modern Angular architecture, such as using signals and signal forms.
firebase: Provide the skills for Firebase AI Logic, Firebase Remote, etc.

3. Wire the Stitch MCP Server with the Antigravity CLI

Stitch is a Google design-to-code tool that lets developers and designers generate interactive screens using natural language prompts.

Hong Kong users need to enable a VPN to access Stitch with AI due to geofencing. Then, I can visit https://stitch.withgoogle.com/ to create a Stitch project and generate new screens in it using natural language.

On the other hand, there is no geofencing restriction on the Antigravity CLI, so I can register the Stitch MCP server to its global MCP configuration file.

This is how I made the CLI talk to the Stitch MCP server after trial and error:

Navigate to https://github.com/davideast/stitch-mcp and read the Quick Start to initialize the MCP server.
Visit https://stitch.withgoogle.com/settings, scroll to API Key and create a new key. Copy the key because we need it in the mcp configuration file.

Open ~/.gemini/config/mcp_config.json to register the Stitch MCP server globally. Replace <Stitch API Key> with your actual API key.

"mcpServers": {
   "stitch": {
        "command": "npx",
        "args": ["-y", "@_davideast/stitch-mcp@latest", "proxy"],
        "env": {
            "STITCH_API_KEY": "<Stitch API Key>",
            "STITCH_USE_SYSTEM_GCLOUD": "1",
            "GOOGLE_CLOUD_PROJECT": "<google cloud project id>",
            "CI": "1",
            "DOTENV_CONFIG_QUIET": "true"
        }
   }
}

Quit the Antigravity CLI and launch again to verify the integration is established correctly.
In the terminal, ask a question such as "List all my Stitch Projects", and wait for the response. If the CLI returns a correct response, then it can communicate with the Stitch MCP server.

The CLI is equipped with the skills and the MCP server. Next, I can use the CLI and the Gemini model to implement features and generate Angular components.

4. Wire the Angular MCP Server

Similarly, I also registered the Angular MCP server to find Angular best practices and search documentation.

"mcpServers": {
  "angular-cli": {
       "command": "npx",
       "args": ["-y", "@angular/cli", "mcp"]
    }
}

The server exposes the following tools to ease Angular development:

ai_tutor: Asks the Angular AI tutor for help with code generation or questions.
get_best_practices: Retrieves recommended guidelines and architectural practices for Angular.
search_documentation: Searches the official Angular documentation.
list_projects: Lists the projects defined in your Angular workspace configuration (angular.json).
onpush_zoneless_migration : Helps migrate parts of your application to OnPush change detection and zoneless mode.
run_target : Runs any configured architect target (e.g., build, test, lint) using the Angular CLI.
devserver.start, devserver.stop , and devserver.wait_for_build : Manage and interact with the local Angular development server.

Now, the CLI is equipped with the agent skills and MCP servers to coordinate the development of the new features in the Angular application.

5. Implement features with skills and MCP servers

After implementing a few features in my Firebase AI Logic application, I developed two workflows and chose either one depending on whether I wanted to develop muscle memory.

Workflow 1: Use /grill-with-docs to generate an Architecture Decision Record (ADR), and save it to a path for human review and documentation. Then, use /goal to ask the coding assistant to implement the ADR, and accept the changes.

For example, I type this prompt to ask the coding assistant to implement an ADR and include relevant skills.

Prompt: "/goal implement <ADR file name>, use Angular skill and Angular MCP server to adopt modern Angular architecture in the code"

Workflow 2: Use /grill-with-docs to generate an ADR, and save it to a path for human review and documentation. I implement the ADR to build up muscle memory, and commit the changes to a git branch. Next, use /code-review to fetch a commit or all changes of a Git branch, and request the coding assistant to spawn two sub-agents to review changes. The first sub-agent detects code smells, and the second sub-agent validates the changes against the ADR to find scope creep, missing, or incorrect details.

For example, I type this prompt to ask the coding assistant to fetch a Git branch, and validate against an ADR.

Prompt: "/code-review fetch all the commits of <Git branch>, and validate the changes against <ADR filename>."

The sub-agents run and print out a report with violations. I fix the violations, rebase the branch, and repeat the workflow until the report shows 0 violations. Finally, I merge the feature branch to main, push the main branch to the remote repository to deploy to Firebase App Hosting.

After implementing the features, I have the enums, types, interfaces, utilities, and services. In the next section, I use the Stitch MCP server to generate UI screens and map them to Angular components.

6. Generate UI Screens with Stitch

This web demo allows users to upload an image and prompt a Gemini model to analyze it to generate tags, alternative text, CSS filters, and image crops.

Note: Enable VPN to access the Stitch site

I use natural language to prompt Stitch to generate two pages for my web application. The web application has a landing page and an image analysis page.

First, I created a new Stitch project to host the UI screens.

Then, I wrote prompts to generate both the home page and the image analysis page

After clicking the Analysis Image button, the page displays the tab group with some tabs.

Next, I need to use the Antigravity CLI to communicate with the Stitch MCP server to generate the Design.md files.

Prompt: "Use the Stitch MCP server to retrieve a Stitch UI Screen from my Stitch Project. Analyze its visual layout, extract the design tokens (colors, typography, and spacing), and generate a DESIGN.md file in the project folder path."

For example, I saved the DESIGN.md of the home screen to the root directory and the DESIGN.md of the image analysis page in the image-analysis folder.

After reviewing the DESIGN.md and ensuring everything looks good, I use /goal again to map the DESIGN.md to the Angular components. In the prompt, I try to provide as much context as possible, so that the coding assistant does not generate all the CSS styles, HTML template, and logic in a single component.

Prompt: "/goal You are an expert Angular developer. Your goal is to implement the UI layout defined in DESIGN.md into production-ready Angular components. Fully read DESIGN.md first to extract all design tokens, spacing guidelines, colors, and layout instructions, and apply them using the following rules: Insert the context and rules...."

7. Live Site Testing

The use case is to upload an image, perform image analysis, and display the token consumption of the analysis.

On Chrome 148+,

Step 1: Upload an image
Step 2: Click the button to generate tags and alternative texts
Step 3: Click the Token usage tab to observe how tokens are spent to achieve the tasks.

On Firefox, repeat the same procedure

Step 1: Upload an image
Step 2: Click the button to generate tags and alternative texts
Step 3: Click the Token usage tab to observe how tokens are spent to achieve the tasks.

8. Conclusion

This concludes the journey of using the Antigravity CLI, agent skills, and Stitch MCP server to build an image analysis application.

When the demo is run in Chrome 148+, the Gemini Nano model is used, and the token usage of the cloud AI is 0. When the demo is run on other browsers, the Gemini 3.5 Flash model is used to generate the response, and tokens are consumed in the cloud.

Please examine the Firebase AI Hybrid & On-Device Web SDK, and leverage the on-device model to perform simple AI tasks to reduce costs.

Resources

Build Firebase AI Logic Application with Antigravity CLI

Connie Leung — Fri, 10 Jul 2026 15:45:18 +0000

Note: Google Cloud credits are provided for this project.

1. Workflow

This is my entire workflow from implementing features, generating UI screens, and mapping the screens to Angular components.

2. Skills

I installed the grill-with-docs, angular, and firebase skills in my project for the following reasons:

grill-with-docs: Conduct a rigid Q&A session to generate a specification for a feature, refactor, or critical fix. AI is responsible for performing thorough analysis, and putting in more efforts to generate code to achieve the task.
domain-modeling: The skill is referenced in the SKILL.md of the grill-with-docs skill, so a copy of it is required.
code-review: Spawn two sub-agents to review changes to detect code smells and verify that the changes align with the specification.
angular: Provide the best practices of modern Angular architecture, such as using signals and signal forms.
firebase: Provide the skills for Firebase AI Logic, Firebase Remote, etc.

3. Wire the Stitch MCP Server with the Antigravity CLI

Stitch is a Google design-to-code tool that lets developers and designers generate interactive screens using natural language prompts.

On the other hand, there is no geofencing restriction on the Antigravity CLI, so I can register the Stitch MCP server to its global MCP configuration file.

This is how I made the CLI talk to the Stitch MCP server after trial and error:

Navigate to https://github.com/davideast/stitch-mcp and read the Quick Start to initialize the MCP server.
Visit https://stitch.withgoogle.com/settings, scroll to API Key and create a new key. Copy the key because we need it in the mcp configuration file.

Open ~/.gemini/config/mcp_config.json to register the Stitch MCP server globally. Replace <Stitch API Key> with your actual API key.

"mcpServers": {
   "stitch": {
        "command": "npx",
        "args": ["-y", "@_davideast/stitch-mcp@latest", "proxy"],
        "env": {
            "STITCH_API_KEY": "<Stitch API Key>",
            "STITCH_USE_SYSTEM_GCLOUD": "1",
            "GOOGLE_CLOUD_PROJECT": "<google cloud project id>",
            "CI": "1",
            "DOTENV_CONFIG_QUIET": "true"
        }
   }
}

Quit the Antigravity CLI and launch again to verify the integration is established correctly.
In the terminal, ask a question such as "List all my Stitch Projects", and wait for the response. If the CLI returns a correct response, then it can communicate with the Stitch MCP server.

The CLI is equipped with the skills and the MCP server. Next, I can use the CLI and the Gemini model to implement features and generate Angular components.

4. Wire the Angular MCP Server

Similarly, I also registered the Angular MCP server to find Angular best practices and search documentation.

"mcpServers": {
  "angular-cli": {
       "command": "npx",
       "args": ["-y", "@angular/cli", "mcp"]
    }
}

The server exposes the following tools to ease Angular development:

ai_tutor: Asks the Angular AI tutor for help with code generation or questions.
get_best_practices: Retrieves recommended guidelines and architectural practices for Angular.
search_documentation: Searches the official Angular documentation.
list_projects: Lists the projects defined in your Angular workspace configuration (angular.json).
onpush_zoneless_migration : Helps migrate parts of your application to OnPush change detection and zoneless mode.
run_target : Runs any configured architect target (e.g., build, test, lint) using the Angular CLI.
devserver.start, devserver.stop , and devserver.wait_for_build : Manage and interact with the local Angular development server.

Now, the CLI is equipped with the agent skills and MCP servers to coordinate the development of the new features in the Angular application.

5. Implement features with skills and MCP servers

After implementing a few features in my Firebase AI Logic application, I developed two workflows and chose either one depending on whether I wanted to develop muscle memory.

For example, I type this prompt to ask the coding assistant to implement an ADR and include relevant skills.

Prompt: "/goal implement <ADR file name>, use Angular skill and Angular MCP server to adopt modern Angular architecture in the code"

For example, I type this prompt to ask the coding assistant to fetch a Git branch, and validate against an ADR.

Prompt: "/code-review fetch all the commits of <Git branch>, and validate the changes against <ADR filename>."

6. Generate UI Screens with Stitch

This web demo allows users to upload an image and prompt a Gemini model to analyze it to generate tags, alternative text, CSS filters, and image crops.

Note: Enable VPN to access the Stitch site

I use natural language to prompt Stitch to generate two pages for my web application. The web application has a landing page and an image analysis page.

First, I created a new Stitch project to host the UI screens.

Then, I wrote prompts to generate both the home page and the image analysis page

After clicking the Analysis Image button, the page displays the tab group with some tabs.

Next, I need to use the Antigravity CLI to communicate with the Stitch MCP server to generate the Design.md files.

For example, I saved the DESIGN.md of the home screen to the root directory and the DESIGN.md of the image analysis page in the image-analysis folder.

7. Live Site Testing

The use case is to upload an image, perform image analysis, and display the token consumption of the analysis.

On Chrome 148+,

Step 1: Upload an image
Step 2: Click the button to generate tags and alternative texts
Step 3: Click the Token usage tab to observe how tokens are spent to achieve the tasks.

On Firefox, repeat the same procedure

Step 1: Upload an image
Step 2: Click the button to generate tags and alternative texts
Step 3: Click the Token usage tab to observe how tokens are spent to achieve the tasks.

8. Conclusion

This concludes the journey of using the Antigravity CLI, agent skills, and Stitch MCP server to build an image analysis application.

Please examine the Firebase AI Hybrid & On-Device Web SDK, and leverage the on-device model to perform simple AI tasks to reduce costs.

Resources

How to Securely Manage AI Prompts in Angular with Firebase [GDE]

Connie Leung — Mon, 15 Jun 2026 18:31:32 +0000

Migrate to Firebase Server Prompt Template in Angular using Dependency Injection

Firebase released Server Prompt Templates to host prompt templates in its infrastructure. The template follows the DotPrompt format and syntax, so the content can have one or more of the following:

Model name
Model configuration
Input validation and schema
Output schema
Tool user
System instruction
User prompt

Moreover, the team offers the TemplateGenerativeModel class, which allows engineers to call either the generateContent or generateContentStream method with a template ID and optional template variables to generate responses. This simplifies the process of constructing text and inline data parts programmatically, passing the parts array and the generation configuration to GenerativeModel to obtain the same results.

Server Prompt Templates resolve several key enterprise AI pain points.

Pain Point	Description
Better Security	The prompt text is stored in the server side, so it cannot be exposed in the network call. Users cannot open the Network tab of the Chrome browser and inspect the prompt text in the payload.
Better Guardrail	Prompt texts are not revealed, so malicious users cannot modify the prompt easily to trigger prompt injection and other attacks to Gemini models
No Prompt Drift	Engineer A edits a prompt locally, forgets to commit, and deploys the code changes. Engineer B uses the old prompt for development, and there are two versions scattered around. Server prompt templates ensure engineers use the same version for development. When the prompt is updated on the server, it is propagated to all instances of the client application.
Testing in Console	Engineers can verify the prompts are working in the Firebase Console before writing a line of code.
Less Deployments	When prompts are updated in the server side, client applications receive the prompt updates without redeployment.

I have listed the benefits of Firebase AI Logic Server Prompt Templates. Next, I will demonstrate how to migrate an existing prompt to use Server Prompt Templates in Angular using Dependency Injection.

Note: Currently, Firebase AI Logic Server Prompt Template is in Preview, please do not use it in production until it reaches General Availability (GA) status. However, it is an interesting technology to explore.

1. Prerequisites

Angular 19
TailwindCSS
Node 22
gemini-3.1-flash-image (also known as Nano Banana 2)
Firebase AI Logic
Firebase Cloud Functions
Firebase Remote Config
Firebase Local Emulator Suite

npm i -g firebase-tools

Install firebase-tools globally using npm.

firebase logout

firebase login

Log out of Firebase and re-login to perform proper Firebase authentication.

firebase init

Execute firebase init and follow the screens to set up Firebase Cloud Function, Firebase Local Emulator Suite and Firebase Remote Config.

If you have an existing project or multiple projects, you can specify the project ID on the command line.

firebase init --project <PROJECT_ID>

After completing the step-by-step, the Firebase tools will generate function and remote config templates, and configuration files such as .firebaserc and firebase.json.

The next section has the details of the implementation repository.

2. Source Code

The full source code for this project is available in the NG Firebase AI Nano Banana, however, the following sections describe the code changes made to migrate to Firebase Server Prompt Templates.

3. Architecture

The application matches the URL paths and routes to different components. When the URL path matches template-prompt/:featureId, the route creates GenMediaService at the route level and injects IMAGE_GENERATOR_TOKEN using the route's injection context. The token is mapped to ServerTemplateService. On the other hand, other routes use the GenMediaService in the root injector and inject a global IMAGE_GENERATOR_TOKEN that maps to FirebaseService. The implementation will be shown later in the blog.

4. Server Prompt Template Creation

You can create a server prompt template in the Firebase Console. This guide assumes an existing Firebase project named vertexai-firebase. Click "AI Logic" from the left sidebar, and click the "Prompt templates (PREVIEW)" tab.

Users can click the Create Template button to create a new prompt on the server side.

A template is configured to generate a glass bottle image from inline image data. The unique template ID is glass-bottle-souvenir-v0-0-1, and the template name is glass-bottle-souvenir.

4.1. Model Configuration

---
model: "gemini-3.1-flash-image"
config:
  candidateCount: 1
  safetySettings:
    - category: HARM_CATEGORY_HARASSMENT
      threshold: BLOCK_ONLY_HIGH
    - category: HARM_CATEGORY_HATE_SPEECH
      threshold: BLOCK_ONLY_HIGH
    - category: HARM_CATEGORY_SEXUALLY_EXPLICIT
      threshold: BLOCK_ONLY_HIGH
    - category: HARM_CATEGORY_DANGEROUS_CONTENT
      threshold: BLOCK_ONLY_HIGH
input:
  schema:
    inlineImages?(array, inline image data):
      type: object
      properties:
        mimeType: string
        data: string  # inline data must be base64-encoded
    aspectRatio?: string, the aspect ratio of the image
    resolution?: string, the resolution of the image
---

The configuration specifies the model name, model configuration, and input schema and validations.

Section	Configuration	Description
model	gemini-3.1-flash-image	The Gemini model name of Nano Banana 2.
config	candidateCount: 1	The model returns at most 1 image
safetySettings	BLOCK_ONLY_HIGH	Safety category of harassment, hate speech, sexually explicit content, and dangerous content
input	schema	Input schema and validation

This prompt expects an array of inlineImages of type object. Each inline image contains a MIME type and inline data. Moreover, the prompt accepts an optional aspect ratio and resolution.

4.2. System Instructions

The prompt parts has {{role "system"}} syntax to specify the system instructions, and {{role "user"}} to specify the user prompt.

{{role "user"}}
A 1/7 scale commercialized collectible ... with realistic lighting and shadows.
{{#if aspectRatio}}
Apply this aspect ratio to the image: {{aspectRatio}}.
{{/if}}
{{#if resolution}}
Apply this resolution to the image: {{resolution}}.
{{/if}}

{{#each inlineImages}}
  {{media type="mimeType" data="data"}}
{{/each}}

The user prompt generates a souvenir glass bottle image from the uploaded inline image.

When the aspect ratio is provided, "Apply this aspect ratio to the image: {{aspectRatio}}." is appended to the prompt.

When the resolution is provided, "Apply this resolution to the image: {{resolution}}." is appended to the prompt.

The loop iterates the inlineImages list to specify the mime type and the inline data.

4.3. Testing the Prompt in Firebase Console

// Prompt Input
{
   "inline_images": [{
    "mime_type": "image/png",
    "contents": "iVBORw0KGgoAAAANSUhEUgAAARAAAABcCAYAAACm+q2AAAXGElEQVR4Ae1dC5QcVZm..."
  }],
   "aspectRatio": "4:1",
   "resolution": "512"
}

The prompt input includes an image, aspect ratio, and resolution for testing before writing a line of code.

In the Firebase UI Console, choose the Gemini API provider from the dropdown list. The Create formatted test request button allows users to verify the request is correct before the actual execution. The Run prompt text button executes the request to generate a 512px and 4:1 image.

The test request generates a souvenir glass bottle with the expected aspect ratio.

Next, I will define two new injection tokens: the first one injects an image generator and the second one injects a TemplateGenerativeModel. I also create a new Server Prompt Template service to generate an image based on the template ID and template variables.

5. Server Prompt Template Service Implementation

5.1. Image Generator Interface

export type BaseGenerateParam = {
  aspectRatio?: string;
  resolution?: string;
  imageFiles: File[];
}

export type GenerateImageParam = BaseGenerateParam &  {
  prompt?: string;
  templateId?: string;
}

The GenerateImageParam type provides aspect ratio, resolution, uploaded images, and template ID to the Gemini model to generate an image.

export type ImageResponseWithoutId = {
  data: string;
  mimeType: string;
  inlineData: string;
}

export type ImageResponse = ImageResponseWithoutId & {
  id: number;
}

export type ImageTokenUsage = {
  image: ImageResponse,
}

The ImageTokenUsage type stores inline image data, mime type, and a dummy image ID.

import { GenerateImageParam } from '@/features/ai/types/generate-image-param.type';
import { ImageTokenUsage } from '@/features/ai/types/image-response.type';

export interface ImageGenerator {
  generateImage(param: GenerateImageParam): Promise<ImageTokenUsage | undefined>;
}

ImageGenerator interface is a contract that must implement a generateImage method to accept a GenerateImageParam parameter and output a promise of ImageTokenUsage or undefined.

5.2. Injection Token for Image Generator

import { FirebaseService } from '@/features/ai/services/firebase.service';
import { ImageGenerator } from '@/shared/ui/gen-media/interfaces/image-generator.interface';
import { InjectionToken, inject } from '@angular/core';

export const IMAGE_GENERATOR_TOKEN = new InjectionToken<ImageGenerator>('IMAGE_GENERATOR_TOKEN', {
  providedIn: 'root',
  factory: () => inject(FirebaseService)
});

The IMAGE_GENERATOR_TOKEN injection token uses the factory function to inject FirebaseService by default. It can be overridden to use the ServerTemplateService when the URL path is template-prompt/:featureId.

5.3. Injection Token for Server Template Model

import { InjectionToken } from '@angular/core';
import { AI, TemplateGenerativeModel } from 'firebase/ai';

export const SERVER_TEMPLATE_MODEL = new InjectionToken<TemplateGenerativeModel>('SERVER_TEMPLATE_MODEL');

The SERVER_TEMPLATE_MODEL injection token injects an instance of TemplateGenerativeModel

Then, the provideFirebase function is updated to instantiate a TemplateGenerativeModel and provide it.

export function provideFirebase() {
    return makeEnvironmentProviders([
        {
          provide: VERTEX_AI_BACKEND,
          useFactory: () => {
            const configService = inject(ConfigService);
            const vertexAILocation = getValue(configService.remoteConfig, 'vertexAILocation').asString();
            const ai = getAI(configService.app, {
              backend: new VertexAIBackend(vertexAILocation)
            });

            return ai;
          }
        },
        {
          provide: SERVER_TEMPLATE_MODEL,
          useFactory: () => {
            const ai = inject(VERTEX_AI_BACKEND); 
            return getTemplateGenerativeModel(ai);
          }
        }
    ]);
}

5.4. Server Prompt Template Service

export async function makeTemplateVariables({ imageFiles, aspectRatio, resolution }: GenerateImageParam) {
  const imageParts = await resolveImageParts(imageFiles);
  const inlineImages = imageParts.map(part => part.inlineData);
  return {
    inlineImages,
    aspectRatio,
    resolution
  }
}

The makeTemplateVariables function converts Files[] to an array of inline image data before returning an object of inline images, aspect ratio, and resolution.

function processImageGeneratedContent(result: GenerateContentResult): ImageTokenUsage {
  const response = result.response;
  const inlineDataParts = response.inlineDataParts();

  if (inlineDataParts?.length) {
    const images = inlineDataParts.map(({inlineData}, index) => {
      const { data, mimeType } = inlineData;
      return {
        id: index,
        mimeType,
        data,
        inlineData: `data:${mimeType};base64,${data}`
      };
    });

    if (images.length <= 0) {
      throw new Error('Error in generating the image.');
    }

    return {
      image: images[0],
    };
  }

  throw new Error('Error in generating the image.');
}

export async function getTemplateBase64Images({ model, templateId, templateVariables }: TemplateImageOptions): Promise<ImageTokenUsage> {
  const result = await model.generateContent(templateId, templateVariables);
  return processImageGeneratedContent(result);
}

The getTemplateBase64Images function uses the model to generate an image, calls processImageGeneratedContent to post-process the result, and returns the ID, MIME type, inline data, and Base64-encoded string.

import { SERVER_TEMPLATE_MODEL } from '@/features/ai/constants/firebase.constant';
import { GenerateImageParam } from '@/features/ai/types/generate-image-param.type';
import { ImageTokenUsage } from '@/features/ai/types/image-response.type';
import { getTemplateBase64Images } from '@/features/ai/utils/generate-image.util';
import { makeTemplateVariables } from '@/features/ai/utils/inline-image-data.util';
import { inject, Injectable } from '@angular/core';

@Injectable({
  providedIn: 'root'
})
export class ServerTemplateService  {
    private readonly serverTemplateModel = inject(SERVER_TEMPLATE_MODEL);

    async generateImage(genImageParameter: GenerateImageParam): Promise<ImageTokenUsage | undefined> {
        const { templateId } = genImageParameter;
        if (!templateId) {
          return undefined;
        }

        const templateVariables = await makeTemplateVariables(genImageParameter);
        return getTemplateBase64Images({
          model: this.serverTemplateModel,
          templateId,
          templateVariables,
        });
    }
}

The ServerTemplateService fulfills the contract of ImageGenerator and implements generateImage to call serverTemplateModel.

6. Angular Route Definition

import { ServerTemplateService } from '@/features/ai/services/server-template.service';
import { IMAGE_GENERATOR_TOKEN } from '@/shared/ui/gen-media/constants/image-generator.token';
import { GenMediaService } from '@/shared/ui/gen-media/services/gen-media.service';
import { Routes } from '@angular/router';

export const routes: Routes = [
  {
    path: 'predefined-prompt/:featureId',
    loadComponent: () => import('./features/predefined-prompt-editor/predefined-prompt-editor.component'),
  },
  {
    path: 'template-prompt/:featureId',
    loadComponent: () => import('./features/predefined-prompt-editor/predefined-prompt-editor.component'),
    providers: [
      GenMediaService,
      { provide: IMAGE_GENERATOR_TOKEN, useExisting: ServerTemplateService }
    ],
  },
  ... other routes ...
];

The routes array specifies a list of paths to route to different components to demonstrate use cases of image generation. The PredefinedPromptEditorComponent consists of an uploader that allows users to upload at least one image to prompt gemini-3.1-flash-image to generate a new image.

Use this component in two scenarios: programmatically passing the prompt text, or using Firebase Server Prompt Templates.

When the path is predefined-prompt/:featureId, the prompt text is submitted to gemini-3.1-flash-image directly. When the path is template-prompt/:featureId, the server prompt template is used.

In the former case, the component uses the FirebaseService that IMAGE_GENERATOR_TOKEN provides in its factory function. In the latter case, the route creates an instance of GenMediaService and does not use the global one. It also provides ServerTemplateService to IMAGE_GENERATOR_TOKEN.

@Injectable({
  providedIn: 'root'
})
export class GenMediaService {
  private readonly imageGenerator = inject(IMAGE_GENERATOR_TOKEN);

  ... the rest of the service ...
}

When GenMediaService injects IMAGE_GENERATOR_TOKEN, imageGenerator is mapped to the ServerTemplateService instead of FirebaseService.

Next, update the navigation menu to use /template-prompt/bottle to call the new template.

7. Update the Navigation Menu

"modeling": {
    "figurine": {
      "path": "/predefined-prompt/figurine",
      "customPrompt": "... custom prompt ..."
    },
    "bottle": {
      "path": "/template-prompt/bottle",
      "templateConfigName": "glassBottleSouvenirTemplateId"
    },
  }

In the features JSON file, the path of bottle is updated to /template-prompt/bottle. Delete customPrompt and add templateConfigName to store the Firebase Remote Config name.

glassBottleSouvenirTemplateId references the template Id, glass-bottle-souvenir-v0-0-1, to load the template to generate the image.

When the Angular application makes the request to Firebase AI Logic, the network payload does not reveal the prompt text.

8. Verify the Network Request

The network payload includes the aspect ratio, resolution, and inline image data. Firebase hides the prompt text, preventing it from being stored as a static value in the JSON file. If prompt text is sensitive data of an application, it is secured in the Firebase's infrastructure.

9. Conclusion

This concludes the journey of migrating the static prompt text to Firebase AI Logic Server Prompt Template.

After the migration, the Angular application does not require redeployment when the server prompt is modified. Users reload the page and they can use the latest prompt to generate images.

Engineers can build AI applications with Firebase AI Logic Server Prompt Templates to perform tasks beyond image generation, such as summarization, text generation, and tool use via Google Search and Google Maps.

Resources

How to Securely Manage AI Prompts in Angular with Firebase

Connie Leung — Mon, 15 Jun 2026 18:29:52 +0000

Firebase released Server Prompt Templates to host prompt templates in its infrastructure. The template follows the DotPrompt format and syntax, so the content can have one or more of the following:

Model name
Model configuration
Input validation and schema
Output schema
Tool user
System instruction
User prompt

Server Prompt Templates resolve several key enterprise AI pain points.

Pain Point	Description
Better Security	The prompt text is stored in the server side, so it cannot be exposed in the network call. Users cannot open the Network tab of the Chrome browser and inspect the prompt text in the payload.
Better Guardrail	Prompt texts are not revealed, so malicious users cannot modify the prompt easily to trigger prompt injection and other attacks to Gemini models
No Prompt Drift	Engineer A edits a prompt locally, forgets to commit, and deploys the code changes. Engineer B uses the old prompt for development, and there are two versions scattered around. Server prompt templates ensure engineers use the same version for development. When the prompt is updated on the server, it is propagated to all instances of the client application.
Testing in Console	Engineers can verify the prompts are working in the Firebase Console before writing a line of code.
Less Deployments	When prompts are updated in the server side, client applications receive the prompt updates without redeployment.

1. Prerequisites

Angular 19
TailwindCSS
Node 22
gemini-3.1-flash-image (also known as Nano Banana 2)
Firebase AI Logic
Firebase Cloud Functions
Firebase Remote Config
Firebase Local Emulator Suite

npm i -g firebase-tools

Install firebase-tools globally using npm.

firebase logout

firebase login

Log out of Firebase and re-login to perform proper Firebase authentication.

firebase init

Execute firebase init and follow the screens to set up Firebase Cloud Function, Firebase Local Emulator Suite and Firebase Remote Config.

If you have an existing project or multiple projects, you can specify the project ID on the command line.

firebase init --project <PROJECT_ID>

After completing the step-by-step, the Firebase tools will generate function and remote config templates, and configuration files such as .firebaserc and firebase.json.

The next section has the details of the implementation repository.

2. Source Code

The full source code for this project is available in the NG Firebase AI Nano Banana, however, the following sections describe the code changes made to migrate to Firebase Server Prompt Templates.

3. Architecture

4. Server Prompt Template Creation

Users can click the Create Template button to create a new prompt on the server side.

A template is configured to generate a glass bottle image from inline image data. The unique template ID is glass-bottle-souvenir-v0-0-1, and the template name is glass-bottle-souvenir.

4.1. Model Configuration

---
model: "gemini-3.1-flash-image"
config:
  candidateCount: 1
  safetySettings:
    - category: HARM_CATEGORY_HARASSMENT
      threshold: BLOCK_ONLY_HIGH
    - category: HARM_CATEGORY_HATE_SPEECH
      threshold: BLOCK_ONLY_HIGH
    - category: HARM_CATEGORY_SEXUALLY_EXPLICIT
      threshold: BLOCK_ONLY_HIGH
    - category: HARM_CATEGORY_DANGEROUS_CONTENT
      threshold: BLOCK_ONLY_HIGH
input:
  schema:
    inlineImages?(array, inline image data):
      type: object
      properties:
        mimeType: string
        data: string  # inline data must be base64-encoded
    aspectRatio?: string, the aspect ratio of the image
    resolution?: string, the resolution of the image
---

The configuration specifies the model name, model configuration, and input schema and validations.

Section	Configuration	Description
model	gemini-3.1-flash-image	The Gemini model name of Nano Banana 2.
config	candidateCount: 1	The model returns at most 1 image
safetySettings	BLOCK_ONLY_HIGH	Safety category of harassment, hate speech, sexually explicit content, and dangerous content
input	schema	Input schema and validation

This prompt expects an array of inlineImages of type object. Each inline image contains a MIME type and inline data. Moreover, the prompt accepts an optional aspect ratio and resolution.

4.2. System Instructions

The prompt parts has {{role "system"}} syntax to specify the system instructions, and {{role "user"}} to specify the user prompt.

{{role "user"}}
A 1/7 scale commercialized collectible ... with realistic lighting and shadows.
{{#if aspectRatio}}
Apply this aspect ratio to the image: {{aspectRatio}}.
{{/if}}
{{#if resolution}}
Apply this resolution to the image: {{resolution}}.
{{/if}}

{{#each inlineImages}}
  {{media type="mimeType" data="data"}}
{{/each}}

The user prompt generates a souvenir glass bottle image from the uploaded inline image.

When the aspect ratio is provided, "Apply this aspect ratio to the image: {{aspectRatio}}." is appended to the prompt.

When the resolution is provided, "Apply this resolution to the image: {{resolution}}." is appended to the prompt.

The loop iterates the inlineImages list to specify the mime type and the inline data.

4.3. Testing the Prompt in Firebase Console

// Prompt Input
{
   "inline_images": [{
    "mime_type": "image/png",
    "contents": "iVBORw0KGgoAAAANSUhEUgAAARAAAABcCAYAAACm+q2AAAXGElEQVR4Ae1dC5QcVZm..."
  }],
   "aspectRatio": "4:1",
   "resolution": "512"
}

The prompt input includes an image, aspect ratio, and resolution for testing before writing a line of code.

The test request generates a souvenir glass bottle with the expected aspect ratio.

5. Server Prompt Template Service Implementation

5.1. Image Generator Interface

export type BaseGenerateParam = {
  aspectRatio?: string;
  resolution?: string;
  imageFiles: File[];
}

export type GenerateImageParam = BaseGenerateParam &  {
  prompt?: string;
  templateId?: string;
}

The GenerateImageParam type provides aspect ratio, resolution, uploaded images, and template ID to the Gemini model to generate an image.

export type ImageResponseWithoutId = {
  data: string;
  mimeType: string;
  inlineData: string;
}

export type ImageResponse = ImageResponseWithoutId & {
  id: number;
}

export type ImageTokenUsage = {
  image: ImageResponse,
}

The ImageTokenUsage type stores inline image data, mime type, and a dummy image ID.

import { GenerateImageParam } from '@/features/ai/types/generate-image-param.type';
import { ImageTokenUsage } from '@/features/ai/types/image-response.type';

export interface ImageGenerator {
  generateImage(param: GenerateImageParam): Promise<ImageTokenUsage | undefined>;
}

ImageGenerator interface is a contract that must implement a generateImage method to accept a GenerateImageParam parameter and output a promise of ImageTokenUsage or undefined.

5.2. Injection Token for Image Generator

import { FirebaseService } from '@/features/ai/services/firebase.service';
import { ImageGenerator } from '@/shared/ui/gen-media/interfaces/image-generator.interface';
import { InjectionToken, inject } from '@angular/core';

export const IMAGE_GENERATOR_TOKEN = new InjectionToken<ImageGenerator>('IMAGE_GENERATOR_TOKEN', {
  providedIn: 'root',
  factory: () => inject(FirebaseService)
});

5.3. Injection Token for Server Template Model

import { InjectionToken } from '@angular/core';
import { AI, TemplateGenerativeModel } from 'firebase/ai';

export const SERVER_TEMPLATE_MODEL = new InjectionToken<TemplateGenerativeModel>('SERVER_TEMPLATE_MODEL');

The SERVER_TEMPLATE_MODEL injection token injects an instance of TemplateGenerativeModel

Then, the provideFirebase function is updated to instantiate a TemplateGenerativeModel and provide it.

export function provideFirebase() {
    return makeEnvironmentProviders([
        {
          provide: VERTEX_AI_BACKEND,
          useFactory: () => {
            const configService = inject(ConfigService);
            const vertexAILocation = getValue(configService.remoteConfig, 'vertexAILocation').asString();
            const ai = getAI(configService.app, {
              backend: new VertexAIBackend(vertexAILocation)
            });

            return ai;
          }
        },
        {
          provide: SERVER_TEMPLATE_MODEL,
          useFactory: () => {
            const ai = inject(VERTEX_AI_BACKEND); 
            return getTemplateGenerativeModel(ai);
          }
        }
    ]);
}

5.4. Server Prompt Template Service

export async function makeTemplateVariables({ imageFiles, aspectRatio, resolution }: GenerateImageParam) {
  const imageParts = await resolveImageParts(imageFiles);
  const inlineImages = imageParts.map(part => part.inlineData);
  return {
    inlineImages,
    aspectRatio,
    resolution
  }
}

The makeTemplateVariables function converts Files[] to an array of inline image data before returning an object of inline images, aspect ratio, and resolution.

function processImageGeneratedContent(result: GenerateContentResult): ImageTokenUsage {
  const response = result.response;
  const inlineDataParts = response.inlineDataParts();

  if (inlineDataParts?.length) {
    const images = inlineDataParts.map(({inlineData}, index) => {
      const { data, mimeType } = inlineData;
      return {
        id: index,
        mimeType,
        data,
        inlineData: `data:${mimeType};base64,${data}`
      };
    });

    if (images.length <= 0) {
      throw new Error('Error in generating the image.');
    }

    return {
      image: images[0],
    };
  }

  throw new Error('Error in generating the image.');
}

export async function getTemplateBase64Images({ model, templateId, templateVariables }: TemplateImageOptions): Promise<ImageTokenUsage> {
  const result = await model.generateContent(templateId, templateVariables);
  return processImageGeneratedContent(result);
}

import { SERVER_TEMPLATE_MODEL } from '@/features/ai/constants/firebase.constant';
import { GenerateImageParam } from '@/features/ai/types/generate-image-param.type';
import { ImageTokenUsage } from '@/features/ai/types/image-response.type';
import { getTemplateBase64Images } from '@/features/ai/utils/generate-image.util';
import { makeTemplateVariables } from '@/features/ai/utils/inline-image-data.util';
import { inject, Injectable } from '@angular/core';

@Injectable({
  providedIn: 'root'
})
export class ServerTemplateService  {
    private readonly serverTemplateModel = inject(SERVER_TEMPLATE_MODEL);

    async generateImage(genImageParameter: GenerateImageParam): Promise<ImageTokenUsage | undefined> {
        const { templateId } = genImageParameter;
        if (!templateId) {
          return undefined;
        }

        const templateVariables = await makeTemplateVariables(genImageParameter);
        return getTemplateBase64Images({
          model: this.serverTemplateModel,
          templateId,
          templateVariables,
        });
    }
}

The ServerTemplateService fulfills the contract of ImageGenerator and implements generateImage to call serverTemplateModel.

6. Angular Route Definition

import { ServerTemplateService } from '@/features/ai/services/server-template.service';
import { IMAGE_GENERATOR_TOKEN } from '@/shared/ui/gen-media/constants/image-generator.token';
import { GenMediaService } from '@/shared/ui/gen-media/services/gen-media.service';
import { Routes } from '@angular/router';

export const routes: Routes = [
  {
    path: 'predefined-prompt/:featureId',
    loadComponent: () => import('./features/predefined-prompt-editor/predefined-prompt-editor.component'),
  },
  {
    path: 'template-prompt/:featureId',
    loadComponent: () => import('./features/predefined-prompt-editor/predefined-prompt-editor.component'),
    providers: [
      GenMediaService,
      { provide: IMAGE_GENERATOR_TOKEN, useExisting: ServerTemplateService }
    ],
  },
  ... other routes ...
];

Use this component in two scenarios: programmatically passing the prompt text, or using Firebase Server Prompt Templates.

When the path is predefined-prompt/:featureId, the prompt text is submitted to gemini-3.1-flash-image directly. When the path is template-prompt/:featureId, the server prompt template is used.

@Injectable({
  providedIn: 'root'
})
export class GenMediaService {
  private readonly imageGenerator = inject(IMAGE_GENERATOR_TOKEN);

  ... the rest of the service ...
}

When GenMediaService injects IMAGE_GENERATOR_TOKEN, imageGenerator is mapped to the ServerTemplateService instead of FirebaseService.

Next, update the navigation menu to use /template-prompt/bottle to call the new template.

7. Update the Navigation Menu

"modeling": {
    "figurine": {
      "path": "/predefined-prompt/figurine",
      "customPrompt": "... custom prompt ..."
    },
    "bottle": {
      "path": "/template-prompt/bottle",
      "templateConfigName": "glassBottleSouvenirTemplateId"
    },
  }

In the features JSON file, the path of bottle is updated to /template-prompt/bottle. Delete customPrompt and add templateConfigName to store the Firebase Remote Config name.

glassBottleSouvenirTemplateId references the template Id, glass-bottle-souvenir-v0-0-1, to load the template to generate the image.

When the Angular application makes the request to Firebase AI Logic, the network payload does not reveal the prompt text.

8. Verify the Network Request

9. Conclusion

This concludes the journey of migrating the static prompt text to Firebase AI Logic Server Prompt Template.

After the migration, the Angular application does not require redeployment when the server prompt is modified. Users reload the page and they can use the latest prompt to generate images.

Resources

Building Dynamic Audio with Emotion & Pace: Gemini 3.1 Flash TTS, Angular & Firebase Cloud Functions

Connie Leung — Sun, 07 Jun 2026 13:26:05 +0000

Google released the Gemini 3.1 Flash TTS Preview model for AI audio generation in the Gemini API, Gemini in Vertex AI, and Gemini AI Studio. This model introduces a new Audio tags feature to exhibit expressive human emotion, pace, and style.

This application explores Firebase AI Logic to analyze an uploaded image to generate recommendations, description, alternative tags, and an obscure fact. The obscure fact is sent to a Firebase Cloud Function to generate an audio using a Gemini TTS model. The Cloud Function returns the stream to an Angular application that converts it to a Blob URL object. An audio player sets the URL to the source that users can click the Play button to play the stream.

In this blog post, I migrate my application to use the Gemini 3.1 Flash TTS Preview model and create a signal form in Angular to input a scene, emotion, and pace. Then, the Angular application provides the form values and the obscure fact to the Firebase Cloud Function to generate an expressive voice using the GenAI TypeScript SDK.

Prerequisites

The technical stack of the project:

Angular 21: The latest version as of May 2026.
Node.js LTS: The LTS version as of May 2026.
Firebase Remote Config: To manage dynamic parameters.
Firebase Cloud Functions: To generate an expressive human voice when called by the frontend.
Firebase Local Emulator Suite: To test the functions locally at http://localhost:5001.
Gemini in Vertex AI: To generate videos and store them in Firebase Cloud Storage.

The public Google AI Studio API is restricted in my region (Hong Kong). However, Vertex AI (Google Cloud) offers enterprise access that works reliably here, so I chose Vertex AI for this demo.

npm i -g firebase-tools

Install firebase-tools globally using npm.

firebase logout

firebase login

Log out of Firebase and log in again to perform proper Firebase authentication.

firebase init

Execute firebase init and follow the prompts to set up Firebase Cloud Functions, the Firebase Local Emulator Suite, Firebase Cloud Storage, and Firebase Remote Config.

If you have an existing project or multiple projects, you can specify the project ID on the command line.

firebase init --project <PROJECT_ID>

In both cases, the Firebase CLI automatically installs the firebase-admin and firebase-functions dependencies.

After completing the setup steps, the Firebase tools generate the functions emulator, functions, a storage rules file, remote config templates, and configuration files such as .firebaserc and firebase.json.

Angular dependency

npm i firebase

The Angular application requires the firebase dependency to initialize a Firebase app, load remote config, and invoke the Firebase Cloud Functions to generate videos.

Firebase dependencies

npm i @cfworker/json-schema @google/genai @modelcontextprotocol/sdk

Install the above dependencies to access Gemini in Vertex AI. @google/genai depends on @cfworker/json-schema and @modelcontextprotocol/sdk. Without these, the Cloud Functions cannot start.

With our project configured, let's look at how the frontend and backend communicate.

Architecture

A user uploads an image in an Angular application and prompts the Gemini 3.1 Flash Lite Preview model to generate a few recommendations for improving the image, a description, and alternative tags. The user also uses the same model and the Google Search tool to find an obscure fact related to the image.

A user inputs a scene, an emotion, and a pace in an experimental signal form. When a user clicks the generate audio button, the Angular application sends the form values and the obscure fact to the Firebase Cloud Function to generate an expressive voice using the GenAI TypeScript SDK and Gemini 3.1 Flash TTS Preview model.

Limitations of Gemini 3.1 Flash TTS Preview Model

The model can only accept text inputs and generate audio outputs.
The context window is 32K tokens
TTS does not support streaming.
The supported languages can be found in https://ai.google.dev/gemini-api/docs/speech-generation#languages. My mother tongue, Cantonese, is currently unsupported.

Firebase Integration

1. Configure Environment Variables

Defining the environment variables in the Firebase project ensures the functions know the region of the Google Cloud project, the Firebase Cloud Function location, and the required TTS model.

.env.example

GOOGLE_CLOUD_LOCATION="global"
GOOGLE_FUNCTION_LOCATION="asia-east2"
GEMINI_TTS_MODEL_NAME="gemini-3.1-flash-tts-preview"
WHITELIST="http://localhost:4200"
REFERER="http://localhost:4200/"

Variable	Description
GOOGLE_CLOUD_LOCATION	The region of the Google Cloud project. I chose `global` so that the Firebase project has access to the newest Gemini 3.1 Flash TTS preview model.
GOOGLE_FUNCTION_LOCATION	The region of the Firebase Cloud Functions. I chose `asia-east2` because this is the region where I live.
WHITELIST	Requests must come from http://localhost:4200.
REFERER	Requests originate from http://localhost:4200/.

http://localhost:4200 is the host and port number of my local Angular application.

2. Validating Environment Variables

Before the Cloud Function proceeds with any AI calls, it is critical to ensure that all necessary environment variables are present. I implemented an AUDIO_CONFIG IIFE (Immediately Invoked Function Expression) to validate environment variables like the TTS model name, Google Cloud Project ID, and location.

import logger from "firebase-functions/logger";

export function validate(value: string | undefined, fieldName: string, missingKeys: string[]) {
    const err = `${fieldName} is missing.`;
    if (!value) {
        logger.error(err);
        missingKeys.push(fieldName);
        return "";
    }

    return value;
}

export const AUDIO_CONFIG = (() => {
    logger.info("AUDIO_CONFIG initialization: Loading environment variables and validating configuration...");

    const env = process.env;

    const missingKeys: string[] = [];
    const location = validate(env.GOOGLE_CLOUD_LOCATION, "Vertex Location", missingKeys);
    const model = validate(env.GEMINI_TTS_MODEL_NAME, "Gemini TTS Model Name", missingKeys);
    const project = validate(env.GCLOUD_PROJECT, "Google Cloud Project", missingKeys);

    if (missingKeys.length > 0) {
        throw new HttpsError("failed-precondition", `Missing environment variables: ${missingKeys.join(", ")}`);
    }

    return {
        genAIOptions: {
            project,
            location,
            vertexai: true,
        },
        model,
    };
})();

I am using Node 24 as of May 2026. Since Node 20, we can use the built-in process.loadEnvFile function that loads environment variables from the .env file.

In env.ts, the try-catch block attempts to load the environment variables from the .env file.

try {
    process.loadEnvFile();
} catch {
    // Ignore error if .env file is not found (e.g., in production where env vars are set by the platform)
}

In src/index.ts, the first statement imports the env.ts before importing other files and libraries.

import "./env";

... other import statements ...

If you are using a Node version that does not support process.loadEnvfile, the alternative is to install dotenv to load the environment variables.

npm i dotenv

import dotenv from "dotenv";

dotenv.config();

Firebase provides the GCLOUD_PROJECT variable, so it is not defined in the .env file.

When the missingKeys array is not empty, AUDIO_CONFIG throws an error that lists all the missing variable names. If the validation is successful, the genAIOptions and model are returned. The genAIOptions is used to initialize the GoogleGenAI and model is the selected TTS model name.

3. Sanitize the Prompt Inputs

The Cloud Function sanitizes the scene and transcript before composing the audio prompt.

The sanitizeScene function accepts the scene by escaping the newline character ('\n') with the '\\n'. The newline character creates a blank line and often signals the end of a block. The sanitization effectively flattens the scene into one continuous line of data and the LLM's Markdown parser recognizes it as a single, safe paragraph. The sanitization also removes all Markdown headers that are injected into the scene.

function sanitizeScene(text: string): string {
    return (text || "").trim().replace(/\r?\n/g, "\\n").replace(/^[#\s]+/gm, "");
}

The sanitizeTranscript function accepts the transcript by removing all Markdown headers and triple quotes that are injected into it.

function sanitizeTranscript(text: string): string {
    return (text || "").trim().replace(/^#+/gm, "").replace(/"""/g, '"');
}

4. Build an Audio Prompt

The AudioPrompt interface encapsulates the scene, emotion, pace, transcript, and voice option to set the location, audio tags, text, and persona of the audio.

export type AudioPrompt = {
  scene: string;
  emotion: string;
  pace: string;
  transcript: string;
  voiceOption: string;
}

The SCENE_DICTIONARY is an array of scenes. When the user does not provide a scene, a scene is randomly selected from the array.

export const SCENE_DICTIONARY = [
    "A dimly lit, dusty library filled with ancient leather-bound books.\n" +
        "The air is thick with history. A scholarly archivist is leaning closely into a warm, vintage ribbon microphone.\n" +
        "They speak with an infectious, hushed intensity, eager to share a forgotten secret they just uncovered in a decaying manuscript.",

    "It is 10:00 PM in a glass-walled studio overlooking the moonlit London skyline, but inside, it is blindingly bright.\n" +
        "The red 'ON AIR' tally light is blazing. The speaker is standing up, bouncing on the balls of their heels to the rhythm of a thumping backing track.\n" +
        "It is a chaotic, caffeine-fueled cockpit designed to wake up an entire nation.",

    "A meticulously sound-treated bedroom in a suburban home.\n" +
        "The space is deadened by plush velvet curtains and a heavy rug, creating an intimate, close-up acoustic environment.\n" +
        "The speaker delivers the information like a trusted friend sharing an inside joke.",

    "A high-tech, minimalist laboratory humming with servers.\n" +
        "Crisp, clean acoustics reflect off glass and steel.\n" +
        "A brilliant but eccentric scientist is pacing back and forth, speaking rapidly and enthusiastically into a headset microphone, excited to explain a complex phenomenon.",
];

I define a buildAudioPrompt function to construct the advanced audio prompt.
When an emotion is defined, the tag is [<emotion>]. When a pace is defined, the tag is [<pace>]. The combined audio tag is [<emotion>] [<pace>]<a space> to create a proper token boundary.

The insertAudioTagsToTranscript uses a regular expression to split the transcript into lines, inserts the combined audio tag before each line, and then joins them with an empty string.

The buildAudioPrompt concatenates the scene and the expressive transcript into a string before returning it.

import { SCENE_DICTIONARY } from './constants/scenes.const';
import { AudioPrompt } from './types/audio-prompt.type';

function makeTag(value: string) {
    const trimmedValue = value.trim();
    return trimmedValue ? `[${trimmedValue}] ` : "";
}

function insertAudioTagsToTranscript({ transcript, pace, emotion }: AudioPrompt): string {
    const audioTags = `${makeTag(emotion)}${makeTag(pace)}`;
    const cleanedTranscript = sanitizeTranscript(transcript);

    const parts = cleanedTranscript.split(/(?<!\b(?:Mr|Mrs|Ms|Dr|St|i\.e|e\.g))([.!?\n\r]+[”"’']*\s*)/);
    return parts
        .map((text, i, arr) => {
            if (i % 2 !== 0) {
                return ""; // Skip delimiters, they are appended to the text blocks
            }
            const delimiter = arr[i + 1] || "";
            return text.trim() ? `${audioTags}${text.trim()}${delimiter}` : delimiter;
        })
        .join("");
}

export function buildAudioPrompt(data: AudioPrompt): string {
    const randomIndex = Math.floor(Math.random() * SCENE_DICTIONARY.length);
    const selectedScene = SCENE_DICTIONARY[randomIndex];

    const trimmedScene = (data.scene || "").trim() || selectedScene;
    const escapedScene = sanitizeScene(trimmedScene);
    const transcript = insertAudioTagsToTranscript(data);

    return `## Scene:
${escapedScene}

## Transcript:
"""
${transcript}
"""
`;
}

The output of the prompt looks like:

## Scene:
<scene>

## Transcript:
[<emotion>] [<pace>] <sentence 1>[<emotion>] [<pace>] <sentence 2>...[<emotion>] [<pace>] <sentence N>

5. Generating an Expression Human Audio in a Firebase Cloud Function

The createVoiceConfig function constructs an instance of GenerateContentConfig that outputs a speech narrated by the given voice name.

import { GenerateContentConfig } from "@google/genai";

export function createVoiceConfig(voiceName = "Kore"): GenerateContentConfig {
    return {
        responseModalities: ["audio"],
        speechConfig: {
            voiceConfig: {
                prebuiltVoiceConfig: {
                    voiceName,
                },
            },
        },
    };
}

const splitList = (whitelist?: string) => (whitelist || "").split(",").map((origin) => origin.trim());

export const whitelist = splitList(process.env.WHITELIST);
export const cors = whitelist.length > 0 ? whitelist : true;
export const refererList = splitList(process.env.REFERER);

All Cloud Functions enforce App Check, CORS, and a timeout period of 600 seconds. If WHITELIST is unspecified, CORS defaults to true. While acceptable in a demo environment, configure CORS to a specific domain or false in production to prevent unauthorized access.

The readFact cloud function delegates to readFactStreamFunction when isStreaming is true. Otherwise, it is delegated to readFactFunction.

The readFactFunction function returns a Promise<string> that is the encoded base64 string.

The readFactStreamFunction functions returns a Promise<number[] | undefined> that represents a buffer of WAV header bytes.

import { onCall } from "firebase-functions/v2/https";
import { cors } from "../auth";
import { buildAudioPrompt } from './audio-prompt';
import { readFactFunction, readFactFunctionStream } from "./read-fact";
import { createVoiceConfig } from './voice-config';

const options = {
    cors,
    enforceAppCheck: true,
    timeoutSeconds: 600,
};

export const readFact = onCall(options, (request, response) => {
    const { data, acceptsStreaming } = request;
    const isStreaming = acceptsStreaming && !!response;
    const prompt = buildAudioPrompt(data);
    const voiceOption = createVoiceConfig(data.voiceOption);

    return isStreaming
        ? readFactStreamFunction(prompt, voiceOption, response)
        : readFactFunction(prompt, voiceOption);
});

The withAIAudio function is a high-order function that calls the callback to generate an audio stream.

async function withAIAudio(callback: (ai: GoogleGenAI, model: string) => Promise<string | number[] | undefined>) {
    try {
        const variables = AUDIO_CONFIG;
        if (!variables) {
            return "";
        }

        const { genAIOptions, model } = variables;
        const ai = new GoogleGenAI(genAIOptions);
        return await callback(ai, model);
    } catch (e) {
        if (e instanceof HttpsError) {
            throw e;
        }
        throw new HttpsError("internal", "An internal error occurred while setting up the AI client.", {
            originalError: (e as Error).message,
        });
    }
}

generateAudio is a callback function that uses the Gemini 3.1 Flash TTS Preview model to generate a response. getBase64DataUrl invokes extractInlineAudioData to extract the raw data and the mime type from the response. The encodeBase64String function first converts the raw data to WAV format, then encodes it to base64 format, and finally returns the base64 string.

The createAudioParams function constructs a parameter with the Gemini TTS model, the audio prompt, and the speech configuration.

async function generateAudio(aiTTS: AIAudio, prompt: string, voiceOption: GenerateContentConfig) {
    try {
        const { ai, model } = aiTTS;
        const response = await ai.models.generateContent(createAudioParams(model, prompt, voiceOption));
        return getBase64DataUrl(response);
    } catch (error) {
        console.error(error);
        throw error;
    }
}

function createAudioParams(model: string, prompt: string, config?: GenerateContentConfig) {
    return {
        model,
        contents: [
            {
                role: "user",
                parts: [
                    {
                        text: prompt,
                    },
                ],
            },
        ],
        config,
    };
}

function extractInlineAudioData(response: GenerateContentResponse): {
    rawData: string | undefined;
    mimeType: string | undefined;
} {
    const { data: rawData, mimeType } = response.candidates?.[0]?.content?.parts?.[0]?.inlineData ?? {};

    return { rawData, mimeType };
}

function getBase64DataUrl(response: GenerateContentResponse) {
    const { rawData, mimeType } = extractInlineAudioData(response);

    if (!rawData || !mimeType) {
        throw new Error("Audio generation failed: No audio data received.");
    }

    return encodeBase64String({ rawData, mimeType });
}

export function encodeBase64String({ rawData, mimeType }: RawAudioData) {
    const wavBuffer = convertToWav(rawData, mimeType);
    const base64Data = wavBuffer.toString("base64");
    return `data:audio/wav;base64,${base64Data}`;
}

generateAudioStream is a callback function that uses the Gemini 3.1 Flash TTS Preview model to stream a list of audio chunks. The chunks are iterated so that each chunk is passed to the extractInlineAudioData function to extract the raw data and the mime type. The function converts the chunk's raw data into a buffer and sends it to the client; the byte length accumulates to determine the total size of all chunks.

After all the chunks are sent to the client, the createWavHeader function uses the total byte length and the audio options to construct a WAV header and returns it.

async function generateAudioStream(
    aiTTS: AIAudio,
    prompt: string,
    voiceOption: GenerateContentConfig,
    response: CallableResponse<unknown>,
): Promise<number[] | undefined> {
    try {
        const { ai, model } = aiTTS;
        const chunks = await ai.models.generateContentStream(createAudioParams(model, prompt, voiceOption));
        let byteLength = 0;
        let options: WavConversionOptions | undefined = undefined;
        for await (const chunk of chunks) {
            const { rawData, mimeType } = extractInlineAudioData(chunk);
            if (!options && mimeType) {
                options = parseMimeType(mimeType);
                response.sendChunk({
                    type: "metadata",
                    payload: {
                        sampleRate: options.sampleRate,
                    },
                });
            }

            if (rawData && mimeType) {
                const buffer = Buffer.from(rawData, "base64");
                byteLength = byteLength + buffer.length;
                response.sendChunk({
                    type: "data",
                    payload: {
                        buffer,
                    },
                });
            }
        }

        if (options && byteLength > 0) {
            const header = createWavHeader(byteLength, options);
            return [...header];
        }

        return undefined;
    } catch (error) {
        console.error(error);
        throw error;
    }
}

The readFactFunction invokes the withAIAudio high-order function to generate a base64-encoded string.

The readFactStreamFunction function calls the withAIAudio high-order function to write chunks to the response body and send them to the client. Then, the generateAudioStream function returns the bytes of the WAV header.

export async function readFactFunction(prompt: string, voiceOption: GenerateContentConfig) {
    return withAIAudio((ai, model) => generateAudio({ ai, model }, prompt, voiceOption));
}

export async function readFactStreamFunction(prompt: string, voiceOption: GenerateContentConfig, response: CallableResponse<unknown>) {
    return withAIAudio((ai, model) => generateAudioStream({ ai, model }, prompt, voiceOption, response));
}

6. Firebase App Configuration and reCAPTCHA Site Key

I implemented a FIREBASE_APP_CONFIG IIFE (Immediately Invoked Function Expression) to run once to validate the environment variables of the Firebase app.

export const FIREBASE_APP_CONFIG = (() => {
    const env = process.env;
    const missingKeys: string[] = [];
    const apiKey = validate(env.APP_API_KEY, "API Key", missingKeys);
    const appId = validate(env.APP_ID, "App Id", missingKeys);
    const messagingSenderId = validate(env.APP_MESSAGING_SENDER_ID, "Messaging Sender ID", missingKeys);
    const recaptchaSiteKey = validate(env.RECAPTCHA_ENTERPRISE_SITE_KEY, "Recaptcha site key", missingKeys);
    const projectId = validate(env.GCLOUD_PROJECT, "Project ID", missingKeys);

    if (missingKeys.length > 0) {
        throw new Error(`Missing environment variables: ${missingKeys.join(", ")}`);
    }

    return {
        app: {
            apiKey,
            appId,
            projectId,
            messagingSenderId,
            authDomain: `${projectId}.firebaseapp.com`,
            storageBucket: `${projectId}.firebasestorage.app`,
        },
        recaptchaSiteKey,
    };
})();

The getFirebaseConfig function caches the FIREBASE_APP_CONFIG for an hour before returning it to the Angular application.

The Angular application receives the Firebase app configuration and reCAPTCHA site key from the Cloud Function to initialize Firebase AI Logic and protect resources from unauthorized access and abuse.

export const getFirebaseConfig = onRequest({ cors }, (request, response) => {
    if (!validateRequest(request, response)) {
        return;
    }

    try {
        response.set("Cache-Control", "public, max-age=3600, s-maxage=3600");
        response.json(FIREBASE_APP_CONFIG);
    } catch (err) {
        console.error(err);
        response.status(500).send("Internal Server Error");
    }
});

7. Local Development with Emulators

For local development, I used the Firebase Local Emulator Suite to save cost and time. In the bootstrapFirebase process, the application calls connectFunctionsEmulator to link to the Cloud Functions running at http://localhost:5001.

The port number defaulted to 5001 when firebase init was executed.

function connectEmulators(functions: Functions, remoteConfig: RemoteConfig) {
  if (location.hostname === 'localhost') {
    const host = getValue(remoteConfig, 'functionEmulatorHost').asString();
    const port = getValue(remoteConfig, 'functionEmulatorPort').asNumber();
    connectFunctionsEmulator(functions, host, port);
  }
}

loadFirebaseConfig is a helper function that makes request to the Cloud function to obtain the Firebase App configuration and the reCAPTCHA site key.

{
  "getFirebaseConfigUrl": "http://127.0.0.1:5001/vertexai-firebase-6a64f/us-central1/getFirebaseConfig"
}

export type FirebaseConfigResponse = {
  app: FirebaseOptions;
  recaptchaSiteKey: string
}

import { HttpClient } from '@angular/common/http';
import { inject } from '@angular/core';
import { catchError, lastValueFrom, throwError } from 'rxjs';
import config from '../../public/config.json';
import { FirebaseConfigResponse } from './ai/types/firebase-config.type';

async function loadFirebaseConfig() {
  const httpService = inject(HttpClient);
  const firebaseConfig$ =
    httpService.get<FirebaseConfigResponse>(config.getFirebaseConfigUrl)
      .pipe(catchError((e) => throwError(() => e)));
  return lastValueFrom(firebaseConfig$);
}

The bootstrapFirebase function initializes the FirebaseApp and App Check, loads the Firebase remote configuration and cloud functions, and stores them in the config service for later use.

export async function bootstrapFirebase() {
    try {
      const configService = inject(ConfigService);
      const firebaseConfig = await loadFirebaseConfig();
      const { app, recaptchaSiteKey } = firebaseConfig;
      const firebaseApp = initializeApp(app);
      const remoteConfig = await fetchRemoteConfig(firebaseApp);

      initializeAppCheck(firebaseApp, {
        provider: new ReCaptchaEnterpriseProvider(recaptchaSiteKey),
        isTokenAutoRefreshEnabled: true,
      });

      const functionRegion = getValue(remoteConfig, 'functionRegion').asString();
      const functions = getFunctions(firebaseApp, functionRegion);
      connectEmulators(functions, remoteConfig);

      configService.loadConfig(firebaseApp, remoteConfig, functions);
    } catch (err) {
      console.error(err);
    }
}

The AppConfig remains unchanged.

import { ApplicationConfig, provideAppInitializer } from '@angular/core';
import { bootstrapFirebase } from './app.bootstrap';

export const appConfig: ApplicationConfig = {
  providers: [
    provideAppInitializer(async () => bootstrapFirebase()),
  ]
};

8. Angular Implementation

8.1 Audio Tags Component

I create an AudioTagsComponent and a new signal form to input the scene, emotion, pace, and voice name in the Angular frontend.

<div>
  <h3>
    <span class="text-xl">🎙️</span> Customize Audio Generation
  </h3>

  <div class="grid grid-cols-1 md:grid-cols-2 gap-4">
    <!-- Scene -->
    <div class="flex flex-col gap-1.5 md:col-span-2">
      <label for="scene">Scene Description</label>
      <textarea id="scene" [formField]="audioPromptForm.scene"
      ></textarea>
    </div>

    <!-- Emotion -->
    <div class="flex flex-col gap-1.5">
      <label for="emotion">Vocal Emotion</label>
      <input type="text" id="emotion" [formField]="audioPromptForm.emotion"
        placeholder="e.g., panicked, whispers"
      />
    </div>

    <!-- Pace -->
    <div class="flex flex-col gap-1.5">
      <label for="pace">Speaking Pace</label>
      <input type="text" id="pace" [formField]="audioPromptForm.pace"
        placeholder="e.g., very slow, rapid"
      />
    </div>

    <!-- Voice Option -->
    <div class="flex flex-col gap-1.5 md:col-span-2">
      <label for="voiceOption">AI Voice Model</label>
      <select id="voiceOption" [formField]="audioPromptForm.voiceOption"
      >
        <option value="" disabled selected>Select a voice...</option>
        @for (option of sortedVoiceOptions(); track option.name) {
          <option [value]="option.name" class="bg-slate-800">{{ option.label }}</option>
        }
      </select>
    </div>
  </div>
</div>

import { ChangeDetectionStrategy, Component, computed, signal } from '@angular/core';
import { form, FormField } from '@angular/forms/signals';
import { VOICE_OPTIONS } from './constants/voice-options.const';
import { AudioPromptData } from './types/audio-prompt-data.type';

@Component({
  selector: 'app-audio-tags',
  imports: [FormField],
  templateUrl: './audio-tags.component.html',
  changeDetection: ChangeDetectionStrategy.OnPush,
})
export class AudioTagsComponent {
    #audioPromptModel = signal<AudioPromptData>({
      scene: 'A news anchor reading the news in a busy newsroom',
      emotion: 'professional, slightly serious',
      pace: 'moderate, clear enunciation',
      voiceOption: 'Kore'
    });
    audioPromptForm = form(this.#audioPromptModel);

    sortedVoiceOptions = computed(() => {
      const sortedList = VOICE_OPTIONS.sort((a, b) => a.name.localeCompare(b.name));

      return sortedList.map(option => ({
        name: option.name,
        label: `${option.name} - ${option.description}`
      }));
    });

    audioPromptModel = this.#audioPromptModel.asReadonly();
}

The AudioTagsComponent is imported into ObscureFactComponent such that users can input values into the experimental signal form.

In the HTML template of ObscureFactComponent, the <app-audio-tags> has a template variable audioTags, and audioTags.audioPromptModel() resolves to an instance of AudioPromptData. The data is assigned to the audioTags property of the generateSpeech method.

<div class="w-full mt-6">
    <app-audio-tags #audioTags />

    <h3>A surprising or obscure fact about the tags</h3>
    @if (interestingFact()) {
      <p>{{ interestingFact() }}</p>

      <app-error-display [error]="ttsError()" />

      <app-text-to-speech
        [isLoadingSync]="isLoadingSync()"
        [isLoadingStream]="isLoadingStream()"
        [isLoadingWebAudio]="isLoadingWebAudio()"
        [audioUrl]="audioUrl()"
        (generateSpeech)="generateSpeech({ mode: $event, audioTags: audioTags.audioPromptModel() })"
        [playbackRate]="playbackRate()"
      />
    } @else {
      <p>The tag(s) does not have any interesting or obscure fact.</p>
    }
</div>

import { AudioPromptData } from './audio-prompt-data.type';
import { GenerateSpeechMode } from '../../generate-audio.util';

export type ModeWithAudioTags = {
  mode: GenerateSpeechMode;
  audioTags: AudioPromptData;
};

export type AudioPrompt = {
  scene: string;
  emotion: string;
  pace: string;
  transcript: string;
  voiceOption: string;
};

The generateSpeech method uses the fact and audioTags to contruct an instance of AudioPrompt. When the mode is stream, the SpeechService calls generateAudioBlobURL to use the audioPrompt to construct a blob URL. When the mode is sync, the SpeechService calls generateAudio to use the audioPrompt to generate an encoded base64 string. When the mode is web_audio_api, the AudioPlayerService calls playStream to stream the audio.

import { SpeechService } from '@/ai/services/speech.service';
import { AudioPrompt } from '@/ai/types/audio-prompt.type';
import { ChangeDetectionStrategy, Component, inject, input, OnDestroy, signal } from '@angular/core';
import { revokeBlobURL } from '../blob.util';
import { AudioTagsComponent } from './audio-tags/audio-tags.component';
import { ModeWithAudioTags } from './audio-tags/types/mode-audio-tags.type';
import { generateSpeechHelper, streamSpeechWithWebAudio, ttsError } from './generate-audio.util';
import { AudioPlayerService } from './services/audio-player.service';

@Component({
  selector: 'app-obscure-fact',
  templateUrl: './obscure-fact.component.html',
  imports: [
    TextToSpeechComponent,
  ],
  changeDetection: ChangeDetectionStrategy.OnPush,
})
export class ObscureFactComponent implements OnDestroy {
  interestingFact = input<string | undefined>(undefined);

  speechService = inject(SpeechService);
  audioPlayerService = inject(AudioPlayerService);

  isLoadingSync = signal(false);
  isLoadingStream = signal(false);
  isLoadingWebAudio = signal(false);

  audioUrl = signal<string | undefined>(undefined);

  ttsError = ttsError;

  async generateSpeech({ mode, audioTags }: ModeWithAudioTags) {
    const fact = this.interestingFact();

    if (fact) {
      revokeBlobURL(this.audioUrl);
      this.audioUrl.set(undefined);

      const audioPrompt = {
          ...audioTags,
          transcript: fact,
      };
      if (mode === 'sync' || mode === 'stream') {
        const loadingSignal = mode === 'stream' ? this.isLoadingStream : this.isLoadingSync;
        const speechFn = (audioPrompt: AudioPrompt) => mode === 'stream' ?
            this.speechService.generateAudioBlobURL(audioPrompt) :
            this.speechService.generateAudio(audioPrompt);

        await generateSpeechHelper(audioPrompt, loadingSignal, this.audioUrl, speechFn);
      } else if (mode === 'web_audio_api') {
        await streamSpeechWithWebAudio(
          audioPrompt,
          this.isLoadingWebAudio,
          (audioPrompt: AudioPrompt) => this.audioPlayerService.playStream(audioPrompt));
      }
    }
  }

  ngOnDestroy(): void {
    revokeBlobURL(this.audioUrl);
  }
}

8.2 Call Firebase Cloud Functions directly

The SpeechService has a generateAudio method that calls the readFact cloud function to obtain the encoded base64 string.

Similarly, the service has a generateAudioBlobURL method that streams the chunks to create a buffer and prepend it with the WAV header. The constructBlobURL creates a blob URL from the Blob Part array.

export function constructBlobURL(parts: BlobPart[]) {
  return URL.createObjectURL(new Blob(parts, { type: 'audio/wav' }));
}

import { AudioPrompt } from '@/ai/types/audio-prompt.type';
import { constructBlobURL } from '@/photo-panel/blob.util';
import { inject, Injectable } from '@angular/core';
import { Functions, httpsCallable } from 'firebase/functions';
import { StreamMessage } from '../types/stream-message.type';
import { ConfigService } from './config.service';

@Injectable({
  providedIn: 'root'
})
export class SpeechService  {
    private configService = inject(ConfigService);

    private get functions(): Functions {
      if (!this.configService.functions) {
        throw new Error('Firebase Functions has not been initialized.');
      }
      return this.configService.functions;
    }

    async generateAudio(audioPrompt: AudioPrompt) {
      const readFactFunction = httpsCallable<AudioPrompt, string>(
        this.functions, 'textToAudio-readFact'
      );

      const { data: audioUri } = await readFactFunction(audioPrompt);
      return audioUri;
    }

    async generateAudioStream(audioPrompt: AudioPrompt) {
      const readFactStreamFunction = httpsCallable<AudioPrompt, number[] | undefined, StreamMessage>(
        this.functions, 'textToAudio-readFact'
      );

      return readFactStreamFunction.stream(audioPrompt);
    }

    async generateAudioBlobURL(audioPrompt: AudioPrompt) {
      const { stream, data } = await this.generateAudioStream(audioPrompt);

      const audioParts: BlobPart[] = [];
      for await (const audioChunk of stream) {
        if (audioChunk && audioChunk.type === 'data') {
          audioParts.push(new Uint8Array(audioChunk.payload.buffer.data));
        }
      }

      const wavHeader = await data;
      if (wavHeader && wavHeader.length) {
        audioParts.unshift(new Uint8Array(wavHeader));
      }

      return constructBlobURL(audioParts);
    }
}

Similar to SpeechService.generateAudioBlobURL, the playStream method of AudioPlayerService also calls generateAudioStream to get a stream of chunks and play each of them immediately.

import { SpeechService } from '@/ai/services/speech.service';
import { AudioPrompt } from '@/ai/types/audio-prompt.type';
import { inject, Injectable, OnDestroy, signal } from '@angular/core';

@Injectable({
  providedIn: 'root'
})
export class AudioPlayerService implements OnDestroy  {

  async playStream(audioPrompt: AudioPrompt) {
      const { stream } = await this.speechService.generateAudioStream(audioPrompt);
      for await (const audioChunk of stream) {
        ... process each chunk ...
      }
  }


  ngOnDestroy(): void {
    ... release resources to prevent memory leak ...
  }
}

This is the end of the walkthrough for the demo. You should now be able to input different combinations of scene, emotion, and pace to create a unique personality to say the given text in an audio clip.

9. Caveats and Lessons Learned: Avoiding the Dynamic Prompt Trap

The examples in Gemini AI Studio and Vertex AI Studio use static audio tags and transcripts and they work correctly for me.

When I applied dynamic audio tags and transcripts in the demo, the Gemini 3.1 TTS Flash Preview model ignored the audio tags. The issue was resolved after debugging in Gemini CLI for hours.

Here are the Caveats and Lessons Learned:

The Token Boundary Trap. The code originally concatenated tags and transcript without a space (for example, "[giggle][slow]Before"). The LLM tokenizer failed to recognize the instruction to change the behavior and pace of the audio. My fix was to insert a space between the tags and the transcript, which was "[giggle] [slow] Before".
Sanitize inputs before injecting into the prompt template. The sanitize functions remove Markdown headers (#) and triple quotes from the scene and transcript. The cleansed scene and transcript are injected into the prompt template to construct the final audio prompt.
LLM does not understand idiom. I typed "at a snail's pace" in the signal form and inserted "[at a snail's pace]" before the line. However, the model vocalized the tag literally, and no pace change occurred.
"Repetitive Weighting" is a Real Strategy. If standard tags like [slow] and [fast] are not dramatic enough, prepend the pace with "very" to increase the dramatic effect of the pace. It was evident when [very, very, very slow] generated a longer audio than [slow].
Replace newline character (\n) with \\n. to flatten the lines into a single paragraph. When the scene and transcript are cleansed and escaped, they are injected into the prompt template while the structure is preserved for the LLM parser.

Conclusion

The integration of text-to-speech with Firebase's serverless scalability empowers Angular applications for real-time audio generation.

First, the Angular application neither requires the genai dependency nor stores the Vertex AI environment variables in a .env file. The client application calls the Cloud Functions to perform the text to speech tasks to generate an audio stream.

The Cloud Functions receive arguments from the client, and execute a TTS operation to either return the entire audio as an encoded base64 string or stream the audio bytes in chunks. During local development, the Firebase Emulator calls the functions at http://localhost:5001 instead of the ones deployed on the Cloud Run platform to save cost.

Try cloning the GitHub repository, uploading an image to generate an obscure fact, and using the Gemini 3.1 Flash TTS preview model to speak it with the specified scene, emotion, and pace.

Resources

Building Dynamic Audio with Emotion & Pace: Gemini 3.1 Flash TTS, Angular & Firebase Cloud Functions [GDE]

Connie Leung — Sat, 02 May 2026 14:11:45 +0000

Prerequisites

The technical stack of the project:

Angular 21: The latest version as of May 2026.
Node.js LTS: The LTS version as of May 2026.
Firebase Remote Config: To manage dynamic parameters.
Firebase Cloud Functions: To generate an expressive human voice when called by the frontend.
Firebase Local Emulator Suite: To test the functions locally at http://localhost:5001.
Gemini in Vertex AI: To generate videos and store them in Firebase Cloud Storage.

The public Google AI Studio API is restricted in my region (Hong Kong). However, Vertex AI (Google Cloud) offers enterprise access that works reliably here, so I chose Vertex AI for this demo.

npm i -g firebase-tools

Install firebase-tools globally using npm.

firebase logout

firebase login

Log out of Firebase and log in again to perform proper Firebase authentication.

firebase init

Execute firebase init and follow the prompts to set up Firebase Cloud Functions, the Firebase Local Emulator Suite, Firebase Cloud Storage, and Firebase Remote Config.

If you have an existing project or multiple projects, you can specify the project ID on the command line.

firebase init --project <PROJECT_ID>

In both cases, the Firebase CLI automatically installs the firebase-admin and firebase-functions dependencies.

Angular dependency

npm i firebase

The Angular application requires the firebase dependency to initialize a Firebase app, load remote config, and invoke the Firebase Cloud Functions to generate videos.

Firebase dependencies

npm i @cfworker/json-schema @google/genai @modelcontextprotocol/sdk

Install the above dependencies to access Gemini in Vertex AI. @google/genai depends on @cfworker/json-schema and @modelcontextprotocol/sdk. Without these, the Cloud Functions cannot start.

With our project configured, let's look at how the frontend and backend communicate.

Architecture

Limitations of Gemini 3.1 Flash TTS Preview Model

The model can only accept text inputs and generate audio outputs.
The context window is 32K tokens
TTS does not support streaming.
The supported languages can be found in https://ai.google.dev/gemini-api/docs/speech-generation#languages. My mother tongue, Cantonese, is currently unsupported.

Firebase Integration

1. Configure Environment Variables

Defining the environment variables in the Firebase project ensures the functions know the region of the Google Cloud project, the Firebase Cloud Function location, and the required TTS model.

.env.example

GOOGLE_CLOUD_LOCATION="global"
GOOGLE_FUNCTION_LOCATION="asia-east2"
GEMINI_TTS_MODEL_NAME="gemini-3.1-flash-tts-preview"
WHITELIST="http://localhost:4200"
REFERER="http://localhost:4200/"

Variable	Description
GOOGLE_CLOUD_LOCATION	The region of the Google Cloud project. I chose `global` so that the Firebase project has access to the newest Gemini 3.1 Flash TTS preview model.
GOOGLE_FUNCTION_LOCATION	The region of the Firebase Cloud Functions. I chose `asia-east2` because this is the region where I live.
WHITELIST	Requests must come from http://localhost:4200.
REFERER	Requests originate from http://localhost:4200/.

http://localhost:4200 is the host and port number of my local Angular application.

2. Validating Environment Variables

import logger from "firebase-functions/logger";

export function validate(value: string | undefined, fieldName: string, missingKeys: string[]) {
    const err = `${fieldName} is missing.`;
    if (!value) {
        logger.error(err);
        missingKeys.push(fieldName);
        return "";
    }

    return value;
}

export const AUDIO_CONFIG = (() => {
    logger.info("AUDIO_CONFIG initialization: Loading environment variables and validating configuration...");

    const env = process.env;

    const missingKeys: string[] = [];
    const location = validate(env.GOOGLE_CLOUD_LOCATION, "Vertex Location", missingKeys);
    const model = validate(env.GEMINI_TTS_MODEL_NAME, "Gemini TTS Model Name", missingKeys);
    const project = validate(env.GCLOUD_PROJECT, "Google Cloud Project", missingKeys);

    if (missingKeys.length > 0) {
        throw new HttpsError("failed-precondition", `Missing environment variables: ${missingKeys.join(", ")}`);
    }

    return {
        genAIOptions: {
            project,
            location,
            vertexai: true,
        },
        model,
    };
})();

I am using Node 24 as of May 2026. Since Node 20, we can use the built-in process.loadEnvFile function that loads environment variables from the .env file.

In env.ts, the try-catch block attempts to load the environment variables from the .env file.

try {
    process.loadEnvFile();
} catch {
    // Ignore error if .env file is not found (e.g., in production where env vars are set by the platform)
}

In src/index.ts, the first statement imports the env.ts before importing other files and libraries.

import "./env";

... other import statements ...

If you are using a Node version that does not support process.loadEnvfile, the alternative is to install dotenv to load the environment variables.

npm i dotenv

import dotenv from "dotenv";

dotenv.config();

Firebase provides the GCLOUD_PROJECT variable, so it is not defined in the .env file.

3. Sanitize the Prompt Inputs

The Cloud Function sanitizes the scene and transcript before composing the audio prompt.

function sanitizeScene(text: string): string {
    return (text || "").trim().replace(/\r?\n/g, "\\n").replace(/^[#\s]+/gm, "");
}

The sanitizeTranscript function accepts the transcript by removing all Markdown headers and triple quotes that are injected into it.

function sanitizeTranscript(text: string): string {
    return (text || "").trim().replace(/^#+/gm, "").replace(/"""/g, '"');
}

4. Build an Audio Prompt

The AudioPrompt interface encapsulates the scene, emotion, pace, transcript, and voice option to set the location, audio tags, text, and persona of the audio.

export type AudioPrompt = {
  scene: string;
  emotion: string;
  pace: string;
  transcript: string;
  voiceOption: string;
}

The SCENE_DICTIONARY is an array of scenes. When the user does not provide a scene, a scene is randomly selected from the array.

export const SCENE_DICTIONARY = [
    "A dimly lit, dusty library filled with ancient leather-bound books.\n" +
        "The air is thick with history. A scholarly archivist is leaning closely into a warm, vintage ribbon microphone.\n" +
        "They speak with an infectious, hushed intensity, eager to share a forgotten secret they just uncovered in a decaying manuscript.",

    "It is 10:00 PM in a glass-walled studio overlooking the moonlit London skyline, but inside, it is blindingly bright.\n" +
        "The red 'ON AIR' tally light is blazing. The speaker is standing up, bouncing on the balls of their heels to the rhythm of a thumping backing track.\n" +
        "It is a chaotic, caffeine-fueled cockpit designed to wake up an entire nation.",

    "A meticulously sound-treated bedroom in a suburban home.\n" +
        "The space is deadened by plush velvet curtains and a heavy rug, creating an intimate, close-up acoustic environment.\n" +
        "The speaker delivers the information like a trusted friend sharing an inside joke.",

    "A high-tech, minimalist laboratory humming with servers.\n" +
        "Crisp, clean acoustics reflect off glass and steel.\n" +
        "A brilliant but eccentric scientist is pacing back and forth, speaking rapidly and enthusiastically into a headset microphone, excited to explain a complex phenomenon.",
];

The insertAudioTagsToTranscript uses a regular expression to split the transcript into lines, inserts the combined audio tag before each line, and then joins them with an empty string.

The buildAudioPrompt concatenates the scene and the expressive transcript into a string before returning it.

import { SCENE_DICTIONARY } from './constants/scenes.const';
import { AudioPrompt } from './types/audio-prompt.type';

function makeTag(value: string) {
    const trimmedValue = value.trim();
    return trimmedValue ? `[${trimmedValue}] ` : "";
}

function insertAudioTagsToTranscript({ transcript, pace, emotion }: AudioPrompt): string {
    const audioTags = `${makeTag(emotion)}${makeTag(pace)}`;
    const cleanedTranscript = sanitizeTranscript(transcript);

    const parts = cleanedTranscript.split(/(?<!\b(?:Mr|Mrs|Ms|Dr|St|i\.e|e\.g))([.!?\n\r]+[”"’']*\s*)/);
    return parts
        .map((text, i, arr) => {
            if (i % 2 !== 0) {
                return ""; // Skip delimiters, they are appended to the text blocks
            }
            const delimiter = arr[i + 1] || "";
            return text.trim() ? `${audioTags}${text.trim()}${delimiter}` : delimiter;
        })
        .join("");
}

export function buildAudioPrompt(data: AudioPrompt): string {
    const randomIndex = Math.floor(Math.random() * SCENE_DICTIONARY.length);
    const selectedScene = SCENE_DICTIONARY[randomIndex];

    const trimmedScene = (data.scene || "").trim() || selectedScene;
    const escapedScene = sanitizeScene(trimmedScene);
    const transcript = insertAudioTagsToTranscript(data);

    return `## Scene:
${escapedScene}

## Transcript:
"""
${transcript}
"""
`;
}

The output of the prompt looks like:

## Scene:
<scene>

## Transcript:
[<emotion>] [<pace>] <sentence 1>[<emotion>] [<pace>] <sentence 2>...[<emotion>] [<pace>] <sentence N>

5. Generating an Expression Human Audio in a Firebase Cloud Function

The createVoiceConfig function constructs an instance of GenerateContentConfig that outputs a speech narrated by the given voice name.

import { GenerateContentConfig } from "@google/genai";

export function createVoiceConfig(voiceName = "Kore"): GenerateContentConfig {
    return {
        responseModalities: ["audio"],
        speechConfig: {
            voiceConfig: {
                prebuiltVoiceConfig: {
                    voiceName,
                },
            },
        },
    };
}

const splitList = (whitelist?: string) => (whitelist || "").split(",").map((origin) => origin.trim());

export const whitelist = splitList(process.env.WHITELIST);
export const cors = whitelist.length > 0 ? whitelist : true;
export const refererList = splitList(process.env.REFERER);

The readFact cloud function delegates to readFactStreamFunction when isStreaming is true. Otherwise, it is delegated to readFactFunction.

The readFactFunction function returns a Promise<string> that is the encoded base64 string.

The readFactStreamFunction functions returns a Promise<number[] | undefined> that represents a buffer of WAV header bytes.

import { onCall } from "firebase-functions/v2/https";
import { cors } from "../auth";
import { buildAudioPrompt } from './audio-prompt';
import { readFactFunction, readFactFunctionStream } from "./read-fact";
import { createVoiceConfig } from './voice-config';

const options = {
    cors,
    enforceAppCheck: true,
    timeoutSeconds: 600,
};

export const readFact = onCall(options, (request, response) => {
    const { data, acceptsStreaming } = request;
    const isStreaming = acceptsStreaming && !!response;
    const prompt = buildAudioPrompt(data);
    const voiceOption = createVoiceConfig(data.voiceOption);

    return isStreaming
        ? readFactStreamFunction(prompt, voiceOption, response)
        : readFactFunction(prompt, voiceOption);
});

The withAIAudio function is a high-order function that calls the callback to generate an audio stream.

async function withAIAudio(callback: (ai: GoogleGenAI, model: string) => Promise<string | number[] | undefined>) {
    try {
        const variables = AUDIO_CONFIG;
        if (!variables) {
            return "";
        }

        const { genAIOptions, model } = variables;
        const ai = new GoogleGenAI(genAIOptions);
        return await callback(ai, model);
    } catch (e) {
        if (e instanceof HttpsError) {
            throw e;
        }
        throw new HttpsError("internal", "An internal error occurred while setting up the AI client.", {
            originalError: (e as Error).message,
        });
    }
}

The createAudioParams function constructs a parameter with the Gemini TTS model, the audio prompt, and the speech configuration.

async function generateAudio(aiTTS: AIAudio, prompt: string, voiceOption: GenerateContentConfig) {
    try {
        const { ai, model } = aiTTS;
        const response = await ai.models.generateContent(createAudioParams(model, prompt, voiceOption));
        return getBase64DataUrl(response);
    } catch (error) {
        console.error(error);
        throw error;
    }
}

function createAudioParams(model: string, prompt: string, config?: GenerateContentConfig) {
    return {
        model,
        contents: [
            {
                role: "user",
                parts: [
                    {
                        text: prompt,
                    },
                ],
            },
        ],
        config,
    };
}

function extractInlineAudioData(response: GenerateContentResponse): {
    rawData: string | undefined;
    mimeType: string | undefined;
} {
    const { data: rawData, mimeType } = response.candidates?.[0]?.content?.parts?.[0]?.inlineData ?? {};

    return { rawData, mimeType };
}

function getBase64DataUrl(response: GenerateContentResponse) {
    const { rawData, mimeType } = extractInlineAudioData(response);

    if (!rawData || !mimeType) {
        throw new Error("Audio generation failed: No audio data received.");
    }

    return encodeBase64String({ rawData, mimeType });
}

export function encodeBase64String({ rawData, mimeType }: RawAudioData) {
    const wavBuffer = convertToWav(rawData, mimeType);
    const base64Data = wavBuffer.toString("base64");
    return `data:audio/wav;base64,${base64Data}`;
}

After all the chunks are sent to the client, the createWavHeader function uses the total byte length and the audio options to construct a WAV header and returns it.

async function generateAudioStream(
    aiTTS: AIAudio,
    prompt: string,
    voiceOption: GenerateContentConfig,
    response: CallableResponse<unknown>,
): Promise<number[] | undefined> {
    try {
        const { ai, model } = aiTTS;
        const chunks = await ai.models.generateContentStream(createAudioParams(model, prompt, voiceOption));
        let byteLength = 0;
        let options: WavConversionOptions | undefined = undefined;
        for await (const chunk of chunks) {
            const { rawData, mimeType } = extractInlineAudioData(chunk);
            if (!options && mimeType) {
                options = parseMimeType(mimeType);
                response.sendChunk({
                    type: "metadata",
                    payload: {
                        sampleRate: options.sampleRate,
                    },
                });
            }

            if (rawData && mimeType) {
                const buffer = Buffer.from(rawData, "base64");
                byteLength = byteLength + buffer.length;
                response.sendChunk({
                    type: "data",
                    payload: {
                        buffer,
                    },
                });
            }
        }

        if (options && byteLength > 0) {
            const header = createWavHeader(byteLength, options);
            return [...header];
        }

        return undefined;
    } catch (error) {
        console.error(error);
        throw error;
    }
}

The readFactFunction invokes the withAIAudio high-order function to generate a base64-encoded string.

export async function readFactFunction(prompt: string, voiceOption: GenerateContentConfig) {
    return withAIAudio((ai, model) => generateAudio({ ai, model }, prompt, voiceOption));
}

export async function readFactStreamFunction(prompt: string, voiceOption: GenerateContentConfig, response: CallableResponse<unknown>) {
    return withAIAudio((ai, model) => generateAudioStream({ ai, model }, prompt, voiceOption, response));
}

6. Firebase App Configuration and reCAPTCHA Site Key

I implemented a FIREBASE_APP_CONFIG IIFE (Immediately Invoked Function Expression) to run once to validate the environment variables of the Firebase app.

export const FIREBASE_APP_CONFIG = (() => {
    const env = process.env;
    const missingKeys: string[] = [];
    const apiKey = validate(env.APP_API_KEY, "API Key", missingKeys);
    const appId = validate(env.APP_ID, "App Id", missingKeys);
    const messagingSenderId = validate(env.APP_MESSAGING_SENDER_ID, "Messaging Sender ID", missingKeys);
    const recaptchaSiteKey = validate(env.RECAPTCHA_ENTERPRISE_SITE_KEY, "Recaptcha site key", missingKeys);
    const projectId = validate(env.GCLOUD_PROJECT, "Project ID", missingKeys);

    if (missingKeys.length > 0) {
        throw new Error(`Missing environment variables: ${missingKeys.join(", ")}`);
    }

    return {
        app: {
            apiKey,
            appId,
            projectId,
            messagingSenderId,
            authDomain: `${projectId}.firebaseapp.com`,
            storageBucket: `${projectId}.firebasestorage.app`,
        },
        recaptchaSiteKey,
    };
})();

The getFirebaseConfig function caches the FIREBASE_APP_CONFIG for an hour before returning it to the Angular application.

export const getFirebaseConfig = onRequest({ cors }, (request, response) => {
    if (!validateRequest(request, response)) {
        return;
    }

    try {
        response.set("Cache-Control", "public, max-age=3600, s-maxage=3600");
        response.json(FIREBASE_APP_CONFIG);
    } catch (err) {
        console.error(err);
        response.status(500).send("Internal Server Error");
    }
});

7. Local Development with Emulators

The port number defaulted to 5001 when firebase init was executed.

function connectEmulators(functions: Functions, remoteConfig: RemoteConfig) {
  if (location.hostname === 'localhost') {
    const host = getValue(remoteConfig, 'functionEmulatorHost').asString();
    const port = getValue(remoteConfig, 'functionEmulatorPort').asNumber();
    connectFunctionsEmulator(functions, host, port);
  }
}

loadFirebaseConfig is a helper function that makes request to the Cloud function to obtain the Firebase App configuration and the reCAPTCHA site key.

{
  "getFirebaseConfigUrl": "http://127.0.0.1:5001/vertexai-firebase-6a64f/us-central1/getFirebaseConfig"
}

export type FirebaseConfigResponse = {
  app: FirebaseOptions;
  recaptchaSiteKey: string
}

import { HttpClient } from '@angular/common/http';
import { inject } from '@angular/core';
import { catchError, lastValueFrom, throwError } from 'rxjs';
import config from '../../public/config.json';
import { FirebaseConfigResponse } from './ai/types/firebase-config.type';

async function loadFirebaseConfig() {
  const httpService = inject(HttpClient);
  const firebaseConfig$ =
    httpService.get<FirebaseConfigResponse>(config.getFirebaseConfigUrl)
      .pipe(catchError((e) => throwError(() => e)));
  return lastValueFrom(firebaseConfig$);
}

The bootstrapFirebase function initializes the FirebaseApp and App Check, loads the Firebase remote configuration and cloud functions, and stores them in the config service for later use.

export async function bootstrapFirebase() {
    try {
      const configService = inject(ConfigService);
      const firebaseConfig = await loadFirebaseConfig();
      const { app, recaptchaSiteKey } = firebaseConfig;
      const firebaseApp = initializeApp(app);
      const remoteConfig = await fetchRemoteConfig(firebaseApp);

      initializeAppCheck(firebaseApp, {
        provider: new ReCaptchaEnterpriseProvider(recaptchaSiteKey),
        isTokenAutoRefreshEnabled: true,
      });

      const functionRegion = getValue(remoteConfig, 'functionRegion').asString();
      const functions = getFunctions(firebaseApp, functionRegion);
      connectEmulators(functions, remoteConfig);

      configService.loadConfig(firebaseApp, remoteConfig, functions);
    } catch (err) {
      console.error(err);
    }
}

The AppConfig remains unchanged.

import { ApplicationConfig, provideAppInitializer } from '@angular/core';
import { bootstrapFirebase } from './app.bootstrap';

export const appConfig: ApplicationConfig = {
  providers: [
    provideAppInitializer(async () => bootstrapFirebase()),
  ]
};

8. Angular Implementation

8.1 Audio Tags Component

I create an AudioTagsComponent and a new signal form to input the scene, emotion, pace, and voice name in the Angular frontend.

<div>
  <h3>
    <span class="text-xl">🎙️</span> Customize Audio Generation
  </h3>

  <div class="grid grid-cols-1 md:grid-cols-2 gap-4">
    <!-- Scene -->
    <div class="flex flex-col gap-1.5 md:col-span-2">
      <label for="scene">Scene Description</label>
      <textarea id="scene" [formField]="audioPromptForm.scene"
      ></textarea>
    </div>

    <!-- Emotion -->
    <div class="flex flex-col gap-1.5">
      <label for="emotion">Vocal Emotion</label>
      <input type="text" id="emotion" [formField]="audioPromptForm.emotion"
        placeholder="e.g., panicked, whispers"
      />
    </div>

    <!-- Pace -->
    <div class="flex flex-col gap-1.5">
      <label for="pace">Speaking Pace</label>
      <input type="text" id="pace" [formField]="audioPromptForm.pace"
        placeholder="e.g., very slow, rapid"
      />
    </div>

    <!-- Voice Option -->
    <div class="flex flex-col gap-1.5 md:col-span-2">
      <label for="voiceOption">AI Voice Model</label>
      <select id="voiceOption" [formField]="audioPromptForm.voiceOption"
      >
        <option value="" disabled selected>Select a voice...</option>
        @for (option of sortedVoiceOptions(); track option.name) {
          <option [value]="option.name" class="bg-slate-800">{{ option.label }}</option>
        }
      </select>
    </div>
  </div>
</div>

import { ChangeDetectionStrategy, Component, computed, signal } from '@angular/core';
import { form, FormField } from '@angular/forms/signals';
import { VOICE_OPTIONS } from './constants/voice-options.const';
import { AudioPromptData } from './types/audio-prompt-data.type';

@Component({
  selector: 'app-audio-tags',
  imports: [FormField],
  templateUrl: './audio-tags.component.html',
  changeDetection: ChangeDetectionStrategy.OnPush,
})
export class AudioTagsComponent {
    #audioPromptModel = signal<AudioPromptData>({
      scene: 'A news anchor reading the news in a busy newsroom',
      emotion: 'professional, slightly serious',
      pace: 'moderate, clear enunciation',
      voiceOption: 'Kore'
    });
    audioPromptForm = form(this.#audioPromptModel);

    sortedVoiceOptions = computed(() => {
      const sortedList = VOICE_OPTIONS.sort((a, b) => a.name.localeCompare(b.name));

      return sortedList.map(option => ({
        name: option.name,
        label: `${option.name} - ${option.description}`
      }));
    });

    audioPromptModel = this.#audioPromptModel.asReadonly();
}

The AudioTagsComponent is imported into ObscureFactComponent such that users can input values into the experimental signal form.

<div class="w-full mt-6">
    <app-audio-tags #audioTags />

    <h3>A surprising or obscure fact about the tags</h3>
    @if (interestingFact()) {
      <p>{{ interestingFact() }}</p>

      <app-error-display [error]="ttsError()" />

      <app-text-to-speech
        [isLoadingSync]="isLoadingSync()"
        [isLoadingStream]="isLoadingStream()"
        [isLoadingWebAudio]="isLoadingWebAudio()"
        [audioUrl]="audioUrl()"
        (generateSpeech)="generateSpeech({ mode: $event, audioTags: audioTags.audioPromptModel() })"
        [playbackRate]="playbackRate()"
      />
    } @else {
      <p>The tag(s) does not have any interesting or obscure fact.</p>
    }
</div>

import { AudioPromptData } from './audio-prompt-data.type';
import { GenerateSpeechMode } from '../../generate-audio.util';

export type ModeWithAudioTags = {
  mode: GenerateSpeechMode;
  audioTags: AudioPromptData;
};

export type AudioPrompt = {
  scene: string;
  emotion: string;
  pace: string;
  transcript: string;
  voiceOption: string;
};

import { SpeechService } from '@/ai/services/speech.service';
import { AudioPrompt } from '@/ai/types/audio-prompt.type';
import { ChangeDetectionStrategy, Component, inject, input, OnDestroy, signal } from '@angular/core';
import { revokeBlobURL } from '../blob.util';
import { AudioTagsComponent } from './audio-tags/audio-tags.component';
import { ModeWithAudioTags } from './audio-tags/types/mode-audio-tags.type';
import { generateSpeechHelper, streamSpeechWithWebAudio, ttsError } from './generate-audio.util';
import { AudioPlayerService } from './services/audio-player.service';

@Component({
  selector: 'app-obscure-fact',
  templateUrl: './obscure-fact.component.html',
  imports: [
    TextToSpeechComponent,
  ],
  changeDetection: ChangeDetectionStrategy.OnPush,
})
export class ObscureFactComponent implements OnDestroy {
  interestingFact = input<string | undefined>(undefined);

  speechService = inject(SpeechService);
  audioPlayerService = inject(AudioPlayerService);

  isLoadingSync = signal(false);
  isLoadingStream = signal(false);
  isLoadingWebAudio = signal(false);

  audioUrl = signal<string | undefined>(undefined);

  ttsError = ttsError;

  async generateSpeech({ mode, audioTags }: ModeWithAudioTags) {
    const fact = this.interestingFact();

    if (fact) {
      revokeBlobURL(this.audioUrl);
      this.audioUrl.set(undefined);

      const audioPrompt = {
          ...audioTags,
          transcript: fact,
      };
      if (mode === 'sync' || mode === 'stream') {
        const loadingSignal = mode === 'stream' ? this.isLoadingStream : this.isLoadingSync;
        const speechFn = (audioPrompt: AudioPrompt) => mode === 'stream' ?
            this.speechService.generateAudioBlobURL(audioPrompt) :
            this.speechService.generateAudio(audioPrompt);

        await generateSpeechHelper(audioPrompt, loadingSignal, this.audioUrl, speechFn);
      } else if (mode === 'web_audio_api') {
        await streamSpeechWithWebAudio(
          audioPrompt,
          this.isLoadingWebAudio,
          (audioPrompt: AudioPrompt) => this.audioPlayerService.playStream(audioPrompt));
      }
    }
  }

  ngOnDestroy(): void {
    revokeBlobURL(this.audioUrl);
  }
}

8.2 Call Firebase Cloud Functions directly

The SpeechService has a generateAudio method that calls the readFact cloud function to obtain the encoded base64 string.

export function constructBlobURL(parts: BlobPart[]) {
  return URL.createObjectURL(new Blob(parts, { type: 'audio/wav' }));
}

import { AudioPrompt } from '@/ai/types/audio-prompt.type';
import { constructBlobURL } from '@/photo-panel/blob.util';
import { inject, Injectable } from '@angular/core';
import { Functions, httpsCallable } from 'firebase/functions';
import { StreamMessage } from '../types/stream-message.type';
import { ConfigService } from './config.service';

@Injectable({
  providedIn: 'root'
})
export class SpeechService  {
    private configService = inject(ConfigService);

    private get functions(): Functions {
      if (!this.configService.functions) {
        throw new Error('Firebase Functions has not been initialized.');
      }
      return this.configService.functions;
    }

    async generateAudio(audioPrompt: AudioPrompt) {
      const readFactFunction = httpsCallable<AudioPrompt, string>(
        this.functions, 'textToAudio-readFact'
      );

      const { data: audioUri } = await readFactFunction(audioPrompt);
      return audioUri;
    }

    async generateAudioStream(audioPrompt: AudioPrompt) {
      const readFactStreamFunction = httpsCallable<AudioPrompt, number[] | undefined, StreamMessage>(
        this.functions, 'textToAudio-readFact'
      );

      return readFactStreamFunction.stream(audioPrompt);
    }

    async generateAudioBlobURL(audioPrompt: AudioPrompt) {
      const { stream, data } = await this.generateAudioStream(audioPrompt);

      const audioParts: BlobPart[] = [];
      for await (const audioChunk of stream) {
        if (audioChunk && audioChunk.type === 'data') {
          audioParts.push(new Uint8Array(audioChunk.payload.buffer.data));
        }
      }

      const wavHeader = await data;
      if (wavHeader && wavHeader.length) {
        audioParts.unshift(new Uint8Array(wavHeader));
      }

      return constructBlobURL(audioParts);
    }
}

Similar to SpeechService.generateAudioBlobURL, the playStream method of AudioPlayerService also calls generateAudioStream to get a stream of chunks and play each of them immediately.

import { SpeechService } from '@/ai/services/speech.service';
import { AudioPrompt } from '@/ai/types/audio-prompt.type';
import { inject, Injectable, OnDestroy, signal } from '@angular/core';

@Injectable({
  providedIn: 'root'
})
export class AudioPlayerService implements OnDestroy  {

  async playStream(audioPrompt: AudioPrompt) {
      const { stream } = await this.speechService.generateAudioStream(audioPrompt);
      for await (const audioChunk of stream) {
        ... process each chunk ...
      }
  }


  ngOnDestroy(): void {
    ... release resources to prevent memory leak ...
  }
}

9. Caveats and Lessons Learned: Avoiding the Dynamic Prompt Trap

The examples in Gemini AI Studio and Vertex AI Studio use static audio tags and transcripts and they work correctly for me.

When I applied dynamic audio tags and transcripts in the demo, the Gemini 3.1 TTS Flash Preview model ignored the audio tags. The issue was resolved after debugging in Gemini CLI for hours.

Here are the Caveats and Lessons Learned:

The Token Boundary Trap. The code originally concatenated tags and transcript without a space (for example, "[giggle][slow]Before"). The LLM tokenizer failed to recognize the instruction to change the behavior and pace of the audio. My fix was to insert a space between the tags and the transcript, which was "[giggle] [slow] Before".
Sanitize inputs before injecting into the prompt template. The sanitize functions remove Markdown headers (#) and triple quotes from the scene and transcript. The cleansed scene and transcript are injected into the prompt template to construct the final audio prompt.
LLM does not understand idiom. I typed "at a snail's pace" in the signal form and inserted "[at a snail's pace]" before the line. However, the model vocalized the tag literally, and no pace change occurred.
"Repetitive Weighting" is a Real Strategy. If standard tags like [slow] and [fast] are not dramatic enough, prepend the pace with "very" to increase the dramatic effect of the pace. It was evident when [very, very, very slow] generated a longer audio than [slow].
Replace newline character (\n) with \\n. to flatten the lines into a single paragraph. When the scene and transcript are cleansed and escaped, they are injected into the prompt template while the structure is preserved for the LLM parser.

Conclusion

The integration of text-to-speech with Firebase's serverless scalability empowers Angular applications for real-time audio generation.

Try cloning the GitHub repository, uploading an image to generate an obscure fact, and using the Gemini 3.1 Flash TTS preview model to speak it with the specified scene, emotion, and pace.

Resources

Extending a Video with Angular, Veo 3.1 Lite, Firebase Cloud Functions, and Firebase Cloud Storage

Connie Leung — Fri, 01 May 2026 03:17:34 +0000

Google released the Veo 3.1 Lite model for AI video generation in the Gemini API, Gemini in Vertex AI, and Gemini AI Studio. This model solves a common developer pain point: generating high-quality videos quickly and at a lower cost.

In this blog post, I migrate my application to use the Veo 3.1 Lite model and implement a new Firebase Cloud Function to extend a video using the GenAI TypeScript SDK.

The application supports image-to-video generation, video interpolation using the first and last frames, and extending Veo videos.

Prerequisites

The technical stack of the project:

Angular 21, the latest version as of April 2026.
Node LTS, the LTS version as of April 2026.
Firebase Remote Config: To manage dynamic parameters.
Firebase Cloud Functions: To be called by the frontend to generate a video, interpolate a video between two images, or extend a Veo video.
Firebase Cloud Storage: To host the generated video files in the default Firebase Storage bucket.
Firebase Local Emulator Suite: To test the functions locally at http://localhost:5001.
Gemini in Vertex AI: Use Gemini in Vertex AI to generate videos and store them in Firebase Cloud Storage.

The public Google AI Studio API is restricted in my region (Hong Kong). However, Vertex AI (Google Cloud) offers enterprise access that works reliably here, so I chose Vertex AI for this demo.

npm i -g firebase-tools

Install firebase-tools globally using npm.

firebase logout

firebase login

Log out of Firebase and log in again to perform proper Firebase authentication.

firebase init

Execute firebase init and follow the prompts to set up Firebase Cloud Functions, the Firebase Local Emulator Suite, Firebase Cloud Storage, and Firebase Remote Config.

If you have an existing project or multiple projects, you can specify the project ID on the command line.

firebase init --project <PROJECT_ID>

In both cases, the Firebase CLI automatically installs the firebase-admin and firebase-functions dependencies.

Angular dependency

npm i firebase

The Angular application requires the firebase dependency to initialize a Firebase app, load remote config, and invoke the Firebase Cloud Functions to generate videos.

Firebase dependencies

npm i @cfworker/json-schema @google/genai @modelcontextprotocol/sdk

Install the above dependencies to access Gemini in Vertex AI. @google/genai depends on @cfworker/json-schema and @modelcontextprotocol/sdk. Without these, the Cloud Functions cannot start.

With our project configured, let's look at how the frontend and backend communicate.

Architecture

The frontend application is built with Angular. It relies on Firebase AI Logic to generate images using the Gemini 3.1 Flash Image Preview model. Then, the text prompt and the image are submitted to a Firebase Cloud Function to generate a video, store it in a Firebase Cloud Storage bucket, and return the GCS URI, MIME type, and HTTP URL to the client.

When the client extends the Veo video, it provides the prompt, the GCS URI, and the MIME type to another Firebase Cloud Function to generate an extended video, store it in the Firebase Cloud Storage bucket, and return the GCS URI, MIME type, and HTTP URL to the client.

Similarly, the HTML video player element plays the HTTP URL in the client application.

Difference between Veo 3.1 Lite in Gemini AI Studio and Gemini in Vertex AI

Difference	Gemini AI Studio	Gemini in Vertex AI
Model Name	veo-3.1-lite-generate-preview	veo-3.1-lite-generate-001
Number of extensions	Extend by 7 seconds and up to 20 times	Extend by 7 seconds and up to 4 times
Video length	Extend 141 seconds and the total length is 148 seconds	Extend 28 seconds and the total length is 36 seconds

What Veo 3.1 Does Not Support in Gemini Vertex AI

Unlike Veo 3.1 and Veo 3.1 Fast (which only lack support for Reference style images), Veo 3.1 Lite does not support either Reference asset images or Reference style images.
Supported input resolution does not support 4K.
Supported output resolution does not support 4K.

This limitation does not impact our application, as the demo focuses exclusively on video extension and the resolution of generated videos is hardcoded to 720p.

Firebase Integration

1. Configure Environment Variables

I define the environment variables in the Firebase project. This ensures the functions know the regions for storage, function hosting, and the Veo model to use for video generation.

.env.example

GOOGLE_CLOUD_LOCATION="us-central1"
GEMINI_VIDEO_MODEL_NAME="veo-3.1-lite-generate-001"
IS_VEO31_USED="true"
POLLING_PERIOD_MS="10000"
GOOGLE_FUNCTION_LOCATION="us-central1"
WHITELIST="http://localhost:4200"
REFERER="http://localhost:4200/"

Variable	Description
GOOGLE_CLOUD_LOCATION	The location of the bucket. I chose `us-central1` because the bucket is always free in this region.
GEMINI_VIDEO_MODEL_NAME	The name of the Gemini video model.
IS_VEO31_USED	Whether Veo 3.1 is used. If false, it falls back to generating a video instead of interpolation.
POLLING_PERIOD_MS	The polling period of the video operation in milliseconds.
GOOGLE_FUNCTION_LOCATION	The region of the Cloud Functions. I chose `us-central1` so the functions and the bucket are in the same region.
WHITELIST	Requests must come from http://localhost:4200.
REFERER	Requests originate from http://localhost:4200/.

2. Validating Environment Variables

Before the Cloud Function proceeds with any AI calls, it is critical to ensure that all necessary environment variables are present. I implemented a VIDEO_CONFIG IIFE (Immediately Invoked Function Expression) to run once to validate environment variables such as the polling period, whether Veo 3.1 is used, the Veo model name, the project ID, and the location.

import logger from "firebase-functions/logger";

export function validate(value: string | undefined, fieldName: string, missingKeys: string[]) {
  const err = `${fieldName} is missing.`;
  if (!value) {
    logger.error(err);
    missingKeys.push(fieldName);
    return "";
  }

  return value;
}

export const VIDEO_CONFIG = (() => {
  process.loadEnvFile();

  const env = process.env;
  const isVeo31Used = (env.IS_VEO31_USED || "false") === "true";
  const pollingPeriod = Number(env.POLLING_PERIOD_MS || "10000");

  const missingKeys: string[] = [];
  const model = validate(env.GEMINI_VIDEO_MODEL_NAME, "Gemini Video Model Name", missingKeys);
  const project = validate(env.GCLOUD_PROJECT, "Project ID", missingKeys);
  const location = validate(env.GOOGLE_CLOUD_LOCATION, "Google Cloud Location", missingKeys);

  if (missingKeys.length > 0) {
    throw new Error(`Missing environment variables: ${missingKeys.join(", ")}`);
  }

  return {
    genAIOptions: {
      project,
      location,
      vertexai: true,
    },
    aiVideoOptions: {
      model,
      storageBucket: `${project}.firebasestorage.app`,
      isVeo31Used,
      pollingPeriod,
    },
  };
})();

I am using Node 24 as of April 2026. Since Node 20, we can use the built-in process.loadEnvFile function that loads environment variables from the .env file.

In env.ts, the try-catch block attempts to load the environment variables from the .env file.

try {
    process.loadEnvFile();
} catch {
    // Ignore error if .env file is not found (e.g., in production where env vars are set by the platform)
}

In src/index.ts, the first statement imports the env.ts before importing other files and libraries.

import "./env";

... other import statements ...

If you are using a Node version that does not support process.loadEnvfile, the alternative is to install dotenv to load the environment variables.

npm i dotenv

import dotenv from "dotenv";

dotenv.config();

Firebase provides the GCLOUD_PROJECT variable, so it is not defined in the .env file.

When the missingKeys array is not empty, VIDEO_CONFIG throws an error that lists all the missing variable names. If the validation is successful, the genAIOptions and aiVideoOptions are returned. The genAIOptions is used to initialize the GoogleGenAI and aiVideoOptions contains parameters for extending a Veo video.

3. Extending a Video and Storing in Firebase Storage

The extendVideo Cloud Function passes the payload to the extendVideoFunction function.

All Cloud Functions enforce App Check, CORS, and a timeout period of 600 seconds. If WHITELIST is unspecified, CORS defaults to true. It is acceptable in a demo, but it is safer to default to false or a specific domain in production.

const cors = process.env.WHITELIST ? process.env.WHITELIST.split(",").map((origin) => origin.trim()): true;
const options = {
  cors,
  enforceAppCheck: true,
  timeoutSeconds: 600,
};

export const extendVideo = onCall( options,
  ({ data }) => extendVideoFunction(data)
);

extendVideoFunction delegates to the extendVideoByPolling function to construct the video arguments and poll the video operation until it finishes. When the function completes successfully, it returns the GCS URI and the MIME type. In the error case, the function throws an error.

import { GoogleGenAI } from "@google/genai";
import { ExtendVideoRequest } from "./types/video.type";
import { extendVideoByPolling, VIDEO_CONFIG } from "./video.util";

export async function extendVideoFunction(data: ExtendVideoRequest) {
  const { genAIOptions, aiVideoOptions } = VIDEO_CONFIG;

  try {
    if (!aiVideoOptions.isVeo31Used) {
      throw new Error("Video extension is only supported for Veo 3.1 model");
    }

    const ai = new GoogleGenAI(genAIOptions);
    return await extendVideoByPolling({ ai, ...aiVideoOptions }, {
      prompt: data.prompt,
      video: data.video,
      config: data.config,
    });
  } catch (error) {
    console.error("Error generating video:", error);
    throw new Error("Error generating video");
  }
}

import { GenerateVideosConfig, GoogleGenAI, Video } from "@google/genai";

export type AIVideoBucket = {
  ai: GoogleGenAI;
  model: string;
  storageBucket: string;
  isVeo31Used: boolean;
  pollingPeriod: number;
}

export type ExtendVideoRequest = {
  prompt: string;
  video: Video;
  config?: GenerateVideosConfig;
}

export async function extendVideoByPolling(
  aiVideo: AIVideoBucket,
  request: ExtendVideoRequest,
) {
  return processVideoPolling(aiVideo, {
    prompt: request.prompt,
    config: request.config,
    video: request.video,
  });
}

async function processVideoPolling(
  { ai, model, storageBucket, pollingPeriod }: AIVideoBucket,
  mediaParams: VideoMediaParams
) {
  const genVideosParams: GenerateVideosParameters = {
    model,
    ...mediaParams,
    config: {
      ...mediaParams.config,
      numberOfVideos: 1,
      outputGcsUri: `gs://${storageBucket}`,
    },
  };

  return getVideoUri(ai, genVideosParams, pollingPeriod);
}

The extendVideoByPolling function invokes the processVideoPolling function to poll the video operation until it completes.

The processVideoPolling function constructs the video parameters and invokes the getVideoUri function to return the GCS URI and the MIME type. Most importantly, the config property specifies that the number of generated videos is one, and the outputGcsUri keeps the generated video in the storage bucket.

4. Asynchronous Polling

Video extension is a long-running task. Because Vertex AI processes them asynchronously, the functions must poll the operation status in a while loop until the done flag is true.

The Gemini API cannot see the Cloud Storage for Firebase Local Emulator, so it requires a real output GCS URI, which is gs://${storageBucket}.

When the done flag is true, the operation ends and one of three outcomes occurs:

Outcome 1: The error is true, and the video failed to generate. Therefore, the function throws an error.
Outcome 2: The video is successfully stored in the GCS bucket. The function returns the GCS URI and the MIME type to the client application.
Outcome 3: Neither happens. There is no error and no GCS URI, so the function returns an unknown error.

import { GenerateVideosParameters, GoogleGenAI } from "@google/genai";

async function getVideoUri(
  ai: GoogleGenAI,
  genVideosParams: GenerateVideosParameters,
  pollingPeriod: number,
): Promise<{ uri: string, mimeType: string }> {
  let operation = await ai.models.generateVideos(genVideosParams);

  while (!operation.done) {
    await new Promise((resolve) => setTimeout(resolve, pollingPeriod));
    operation = await ai.operations.getVideosOperation({ operation });
  }

  if (operation.error) {
    const strError = `Video generation failed: ${operation.error.message}`;
    console.error(strError);
    throw new Error(strError);
  }

  const video = operation.response?.generatedVideos?.[0]?.video || {};
  const { uri, mimeType } = video;
  if (uri && mimeType) {
    console.log("video uri", uri, mimeType);
    return { uri, mimeType };
  }

  const strError = "Video generation finished but no uri was provided.";
  console.error(strError);
  throw new Error(strError);
}

The environment variable, POLLING_PERIOD_MS, sets the polling period to 10 seconds. If the wait time is too long, the polling period can be decreased. If the polling cost is expensive or too frequent, the polling period can be increased.

Note: For demo purposes, polling is a decent solution to handle asynchronous video generation. However, it is expensive and creates unnecessary load and latency. For production usage, you may consider push notifications such as WebSockets and Server-Sent Events.

5. Firebase App Configuration and reCAPTCHA Site Key

getFirebaseConfig is a Firebase Cloud Function that returns both the Firebase app configuration and the reCAPTCHA site key.

import logger from "firebase-functions/logger";
import { validate } from "./validate";

function validateFirebaseConfigFields(env: NodeJS.ProcessEnv) {
  const missingKeys: string[] = [];
  const apiKey = validate(env.APP_API_KEY, "API Key", missingKeys);
  const appId = validate(env.APP_ID, "App Id", missingKeys);
  const messagingSenderId = validate(env.APP_MESSAGING_SENDER_ID, "Messaging Sender ID", missingKeys);
  const recaptchaSiteKey = validate(env.RECAPTCHA_ENTERPRISE_SITE_KEY, "Recaptcha site key", missingKeys);
  const projectId = validate(env.GCLOUD_PROJECT, "Project ID", missingKeys);

  if (missingKeys.length > 0) {
    throw new Error(`Missing environment variables: ${missingKeys.join(", ")}`);
  }

  return {
    app: {
      apiKey,
      appId,
      projectId,
      messagingSenderId,
      authDomain: `${projectId}.firebaseapp.com`,
      storageBucket: `${projectId}.firebasestorage.app`,
    },
    recaptchaSiteKey,
  };
}

export const getFirebaseConfigFunction = () => {
  logger.info("getFirebaseConfig called");

  process.loadEnvFile();

  const variables = validateFirebaseConfigFields(process.env);
  if (!variables) {
    return undefined;
  }

  return variables;
};

6. Local Development with Emulators

For local development, I used the Firebase Local Emulator Suite. In the bootstrapFirebase process, the app calls connectFunctionsEmulator to link to the Cloud Functions running at http://localhost:5001.

The port number defaulted to 5001 when firebase init was executed.

Note: While the Cloud Function runs locally (at zero cost), the Storage emulator is not used. This is because the Gemini API requires an actual accessible GCS bucket to store the generated videos.

export function connectEmulators({ remoteConfig, functions }: FirebaseObjects) {
  const useEmulators = getValue(remoteConfig, 'useEmulators').asBoolean();
  if (useEmulators) {
    console.log('Connecting to emulators...');
    const host = getValue(remoteConfig, 'functionEmulatorHost').asString();
    const port = getValue(remoteConfig, 'functionEmulatorPort').asNumber();
    console.log('functionEmulator', `${host}:${port}`);
    connectFunctionsEmulator(functions, host, port);
  }
}

loadFirebaseConfig is a helper function that makes request to the Cloud function to obtain the Firebase App configuration and the reCAPTCHA site key.

{
  "getFirebaseConfigUrl": "http://127.0.0.1:5001/vertexai-firebase-6a64f/us-central1/getFirebaseConfig"
}

import { HttpClient } from '@angular/common/http';
import { inject } from '@angular/core';
import { initializeAppCheck, ReCaptchaEnterpriseProvider } from 'firebase/app-check';
import { catchError, lastValueFrom, throwError } from 'rxjs';
import config from '../../public/config.json';
import { ConfigService } from './ai/services/config.service';
import { FirebaseConfigResponse } from './ai/types/firebase-config.type';
import { connectEmulators, initFirebaseApp } from './firebase.util';

async function loadFirebaseConfig() {
  const httpService = inject(HttpClient);
  const firebaseConfig$ =
    httpService.get<FirebaseConfigResponse>(`${config.getFirebaseConfigUrl}`)
      .pipe(catchError((e) => throwError(() => e)));
  return lastValueFrom(firebaseConfig$);
}

export async function bootstrapFirebase() {
    try {
      const configService = inject(ConfigService);
      const firebaseConfig = await loadFirebaseConfig();
      const { app, recaptchaSiteKey } = firebaseConfig;
      const firebaseObjects = await initFirebaseApp(app);
      const { firebaseApp } = firebaseObjects;

      initializeAppCheck(firebaseApp, {
        provider: new ReCaptchaEnterpriseProvider(recaptchaSiteKey),
        isTokenAutoRefreshEnabled: true,
      });

      connectEmulators(firebaseObjects);
      configService.loadConfig(firebaseObjects);
    } catch (err) {
      console.error(err);
    }
}

The AppConfig remains unchanged.

import { ApplicationConfig, provideAppInitializer } from '@angular/core';
import { bootstrapFirebase } from './app.bootstrap';

export const appConfig: ApplicationConfig = {
  providers: [
    provideAppInitializer(async () => bootstrapFirebase()),
  ]
};

7. Firebase Remote Configuration

New variables are introduced for extending a Veo video in Firebase Remote Config. For the Gemini API, a video can be extended as many as 20 times. For Gemini in Vertex AI, a video can be extended as many as 4 times because the video length exceeds 30 seconds. Therefore, this is externalized into the maxVideoExtendAllowed variable.

Variable	Description	Value
maxVideoExtendAllowed	Maximum number of extensions allowed	4

8. Video Player Component in Angular

The Angular frontend triggers the process using httpsCallable. Once the function returns the Cloud Storage path and the MIME type, the app fetches the download URL for playback.

The ConfigService stores the Firebase app, Remote Config, and functions to be used throughout the application.

import { FirebaseApp } from 'firebase/app';
import { Functions } from 'firebase/functions';
import { RemoteConfig } from 'firebase/remote-config';

export type FirebaseObjects = {
  firebaseApp: FirebaseApp;
  remoteConfig: RemoteConfig;
  functions: Functions;
}

import { Injectable } from '@angular/core';
import { FirebaseObjects } from '../types/firebase-objects';

@Injectable({
  providedIn: 'root'
})
export class ConfigService  {
    firebaseObjects: FirebaseObjects | undefined = undefined;

    loadConfig(firebaseObjects: FirebaseObjects) {
      this.firebaseObjects = firebaseObjects;
    }
}

The retrieveVideoUri method calls the Cloud Function directly to retrieve the GCS URI and the MIME type.

The downloadVideoUriAndUrl method resolves the URI to an HTTP URL so that an HTML video player can play it immediately.

export type VideoGenerationResponse = {
  uri: string;
  url: string;
  mimeType: string;
}

export type DownloadVideoResponse = {
  uri: string;
  mimeType: string;
}

import { inject, Injectable } from '@angular/core';
import { httpsCallable } from 'firebase/functions';
import { getDownloadURL, getStorage, ref } from 'firebase/storage';
import { GenerateVideoRequest } from '../types/video.type';
import { ConfigService } from './config.service';

@Injectable({
  providedIn: 'root'
})
export class VeoService {
  private readonly storage = getStorage();
  private readonly configService = inject(ConfigService);

  private async retrieveVideoUri<T = GenerateVideoRequest>(request: T, methodName: string): Promise<DownloadVideoResponse> {
    try {
      const { functions } = this.configService.firebaseObjects || {};
      if (!functions) {
        throw new Error('Functions does not exist.');
      }

      const downloadGcsUri = httpsCallable<T, DownloadVideoResponse>(
        functions, methodName, { timeout: 600000 }
      );
      const { data } = await downloadGcsUri(request);
      return data;
    } catch (err) {
        console.error(err);
        throw err;
    }
  }

  async downloadVideoUriAndUrl<T = GenerateVideoRequest>(request: T, methodName: CallableNames = 'videos-generateVideo'): Promise<VideoGenerationResponse> {
    const { uri, mimeType } = await this.retrieveVideoUri(request, methodName);

    if (!uri) {
      throw new Error('Video operation completed but no URI was returned.');
    }

    return getDownloadURL(ref(this.storage, uri))
      .then((url) => {
        console.log("download url", url);
        return { uri, url, mimeType };
      })
      .catch((error) => {
        switch (error.code) {
          case 'storage/object-not-found':
            throw new Error("File doesn't exist");
          case 'storage/unauthorized':
            throw new Error("User doesn't have permission to access the object");
          case 'storage/canceled':
            throw new Error("User canceled the upload");
          case 'storage/unknown':
            throw new Error("Unknown storage error occurred, inspect the server response");
        }
        throw new Error("Unknown error occurred");
      });
  }
}

The VideoPlayerComponent has a required videoUrl signal input that is assigned to the source of the video player. It has an "Extend Video" button that emits an extendVideo custom event when clicked. This event notifies the parent component to extend the video in the videoUrl signal input.

import { LoaderComponent } from '@/shared/loader/loader.component';
import { ChangeDetectionStrategy, Component, input, output } from '@angular/core';
import { ExtendVideoIconComponent } from '../../icons/extend-video-icon.component';

@Component({
  selector: 'app-video-player',
  imports: [LoaderComponent, ExtendVideoIconComponent],
  template: `
@if (isGeneratingVideo()) {
  <div class="mt-6">
    <app-loader [loadingText]="loadingText()">
      <p class="text-sm">This can take several minutes. Please be patient.</p>
    </app-loader>
  </div>
} @else if (videoUrl()) {
  <div class="mt-6 flex flex-col gap-4">
    <div class="video-container">
      <video [src]="videoUrl()" controls autoplay loop class="w-full rounded-md"></video>
    </div>

    <div class="flex justify-center">
      <button
        (click)="extendVideo.emit()"
        aria-label="Extend video"
        title="Extend video"
        class="extend-btn"
      >
        <app-extend-video-icon />
        <span>Extend Video</span>
      </button>
    </div>
  </div>
}
  `,
  styleUrl: './video-player.component.css',
  changeDetection: ChangeDetectionStrategy.OnPush,
})
export class VideoPlayerComponent {
  isGeneratingVideo = input(false);
  videoUrl = input.required<string>();
  extendVideo = output<void>();

  loadingText = input('Generating your video...');
}

9. Angular Integration with Gemini to Extend Video

import { VideoPlayerComponent } from './video-player/video-player.component';

@Component({
  selector: 'app-gen-media',
  imports: [
    VideoPlayerComponent,
  ],
  template: `
  <app-video-player
    [isGeneratingVideo]="isGeneratingVideo()" 
    [videoUrl]="videoUrl()"
    (extendVideo)="extendVideo()"
    [loadingText]="loadingVideoText()"
  />`,
  changeDetection: ChangeDetectionStrategy.OnPush,
})
export class GenMediaComponent {
  private readonly genMediaService = inject(GenMediaService);
  private readonly genVideoService = inject(GenVideoService);

  genMediaInput = input<GenMediaInput>();
  videoUrl = this.genVideoService.videoUrl;
  isGeneratingVideo = this.genVideoService.isGeneratingVideo.asReadonly();
  loadingVideoText = signal('Generating your video...');

  trimmedUserPrompt = computed(() => this.genMediaInput()?.userPrompt.trim() || '');

  async extendVideo() {
    if (this.trimmedUserPrompt()) {
      try {
        this.loadingVideoText.set('Extending your video...');
        await this.genVideoService.extendVideo(this.trimmedUserPrompt());
      } finally {
        this.loadingVideoText.set('Generating your video...');
      }
    }
  }
}

The GenMediaComponent calls extendVideo with the prompt and the Veo video URL to create an extended video.

import { ConfigService } from '@/ai/services/config.service';
import { VeoService } from '@/ai/services/veo.service';
import { ExtendVideoRequest, GenerateVideoFromFramesRequest, GenerateVideoRequest, VideoGenerationResponse } from '@/ai/types/video.type';
import { computed, inject, Injectable, signal, WritableSignal } from '@angular/core';
import { getValue } from 'firebase/remote-config';

@Injectable({
  providedIn: 'root'
})
export class GenVideoService {
  private readonly veoService = inject(VeoService);
  private readonly configService = inject(ConfigService);

  videoError = signal('');
  videoResponse = signal<VideoGenerationResponse>({ uri: '', url: '', mimeType: '' });
  #extendVideoCounter = signal(0);
  isGeneratingVideo = signal(false);

  videoUrl = computed(() => this.videoResponse().url);

  isVideoExtensionAllowed(counter: number) {
    const remoteConfig = this.configService.firebaseObjects?.remoteConfig;
    if (!remoteConfig) {
      console.warn('Remote config does not exist.');
      return false;
    }

    const max_extend_allowed = getValue(remoteConfig,'maxVideoExtendAllowed').asNumber();
    if (counter >= max_extend_allowed) {
      console.warn('Maximum extension limit reached.');
      return false;
    }

    return true;
  }

  async extendVideo(prompt: string): Promise<void> {
    try {
      const result = await this.extendInterpolatedVideo(
        prompt,
        this.#extendVideoCounter(),
        this.videoResponse()
      );

      if (!result) {
        return;
      }

      this.videoResponse.set(result);
      this.#extendVideoCounter.update(count => count + 1);
      console.log(`Video extended successfully. Current extension count: ${this.#extendVideoCounter()}`);
    } catch (e) {
      console.error(e);
      const errMsg = e instanceof Error ?
        e.message :
        'An unexpected error occurred in video generation.'
      this.videoError.set(errMsg);
    } finally {
      this.isGeneratingVideo.set(false);
    }
  }

  async extendInterpolatedVideo(
    prompt: string,
    counter: number,
    customVideo: Pick<VideoGenerationResponse, 'uri' | 'mimeType'>,
    generatingSignal?: WritableSignal<boolean>,
    error?: WritableSignal<string>
  ) {
    const { uri, mimeType } = customVideo;

    if (!mimeType || !uri) {
      console.warn('No video to extend. Please generate a video first.');
      return null;
    }

    if (!prompt) {
      console.warn('Prompt is required to extend the video.');
      return null;
    }

    if (!this.isVideoExtensionAllowed(counter)) {
      return null;
    }

    const actualErrorSignal = error || this.videoError;
    const actualGeneratingSignal = generatingSignal || this.isGeneratingVideo;
    try {
      actualErrorSignal.set('');
      actualGeneratingSignal.set(true);

      const extendVideoParams: ExtendVideoRequest = {
        prompt,
        video: { uri, mimeType },
      }

      return await this.veoService.downloadVideoUriAndUrl(extendVideoParams, 'videos-extendVideo');
    } catch (e) {
      console.error(e);
      throw e;
    } finally {
      actualGeneratingSignal.set(false);
    }
  }
}

The extendVideo and extendInterpolatedVideo methods validate that an extension is allowed by comparing the #extendVideoCounter signal value against the maxVideoExtendAllowed Remote Config value. If the check fails, nothing happens. If the check succeeds, the videos-extendVideo Cloud Function is called to extend the video. Then, the #extendVideoCounter is incremented by 1 until the extension limit is reached. When extending a Veo video that is too long, Gemini throws an error, and I want to block this from happening on the client side.

This is the end of the walkthrough for the demo. You should now be able to extend generated videos in a Cloud Function, store them securely in a bucket, and play them in a video player within a user interface.

Conclusion

Combining Veo 3.1 with the serverless scalability of Firebase is a powerful workflow.

First, the Angular application neither installs the genai dependency nor maintains the Vertex AI environment variables in a .env file. The client application calls the Cloud Functions to perform intensive tasks and waits for the results.

The Cloud Functions receive arguments from the client, execute complex AI operations like generation, interpolation, and video extension, and write the videos to the dedicated bucket securely. During local development, the Firebase Emulator calls the functions at http://localhost:5001 instead of the ones deployed on the Cloud Run platform.

Try cloning the GitHub repository, generating images, and using those images to generate and interpolate videos. Then, you can extend these Veo videos to create extended videos that are more than 7 seconds long.

Resources

Extending a Video with Angular, Veo 3.1 Lite, Firebase Cloud Functions, and Firebase Cloud Storage [GDE]

Connie Leung — Sat, 18 Apr 2026 06:50:21 +0000

In this blog post, I migrate my application to use the Veo 3.1 Lite model and implement a new Firebase Cloud Function to extend a video using the GenAI TypeScript SDK.

The application supports image-to-video generation, video interpolation using the first and last frames, and extending Veo videos.

Prerequisites

The technical stack of the project:

Angular 21, the latest version as of April 2026.
Node LTS, the LTS version as of April 2026.
Firebase Remote Config: To manage dynamic parameters.
Firebase Cloud Functions: To be called by the frontend to generate a video, interpolate a video between two images, or extend a Veo video.
Firebase Cloud Storage: To host the generated video files in the default Firebase Storage bucket.
Firebase Local Emulator Suite: To test the functions locally at http://localhost:5001.
Gemini in Vertex AI: Use Gemini in Vertex AI to generate videos and store them in Firebase Cloud Storage.

The public Google AI Studio API is restricted in my region (Hong Kong). However, Vertex AI (Google Cloud) offers enterprise access that works reliably here, so I chose Vertex AI for this demo.

npm i -g firebase-tools

Install firebase-tools globally using npm.

firebase logout

firebase login

Log out of Firebase and log in again to perform proper Firebase authentication.

firebase init

Execute firebase init and follow the prompts to set up Firebase Cloud Functions, the Firebase Local Emulator Suite, Firebase Cloud Storage, and Firebase Remote Config.

If you have an existing project or multiple projects, you can specify the project ID on the command line.

firebase init --project <PROJECT_ID>

In both cases, the Firebase CLI automatically installs the firebase-admin and firebase-functions dependencies.

Angular dependency

npm i firebase

The Angular application requires the firebase dependency to initialize a Firebase app, load remote config, and invoke the Firebase Cloud Functions to generate videos.

Firebase dependencies

npm i @cfworker/json-schema @google/genai @modelcontextprotocol/sdk

Install the above dependencies to access Gemini in Vertex AI. @google/genai depends on @cfworker/json-schema and @modelcontextprotocol/sdk. Without these, the Cloud Functions cannot start.

With our project configured, let's look at how the frontend and backend communicate.

Architecture

Similarly, the HTML video player element plays the HTTP URL in the client application.

Difference between Veo 3.1 Lite in Gemini AI Studio and Gemini in Vertex AI

Difference	Gemini AI Studio	Gemini in Vertex AI
Model Name	veo-3.1-lite-generate-preview	veo-3.1-lite-generate-001
Number of extensions	Extend by 7 seconds and up to 20 times	Extend by 7 seconds and up to 4 times
Video length	Extend 141 seconds and the total length is 148 seconds	Extend 28 seconds and the total length is 36 seconds

What Veo 3.1 Does Not Support in Gemini Vertex AI

Unlike Veo 3.1 and Veo 3.1 Fast (which only lack support for Reference style images), Veo 3.1 Lite does not support either Reference asset images or Reference style images.
Supported input resolution does not support 4K.
Supported output resolution does not support 4K.

This limitation does not impact our application, as the demo focuses exclusively on video extension and the resolution of generated videos is hardcoded to 720p.

Firebase Integration

1. Configure Environment Variables

I define the environment variables in the Firebase project. This ensures the functions know the regions for storage, function hosting, and the Veo model to use for video generation.

.env.example

GOOGLE_CLOUD_LOCATION="us-central1"
GEMINI_VIDEO_MODEL_NAME="veo-3.1-lite-generate-001"
IS_VEO31_USED="true"
POLLING_PERIOD_MS="10000"
GOOGLE_FUNCTION_LOCATION="us-central1"
WHITELIST="http://localhost:4200"
REFERER="http://localhost:4200/"

Variable	Description
GOOGLE_CLOUD_LOCATION	The location of the bucket. I chose `us-central1` because the bucket is always free in this region.
GEMINI_VIDEO_MODEL_NAME	The name of the Gemini video model.
IS_VEO31_USED	Whether Veo 3.1 is used. If false, it falls back to generating a video instead of interpolation.
POLLING_PERIOD_MS	The polling period of the video operation in milliseconds.
GOOGLE_FUNCTION_LOCATION	The region of the Cloud Functions. I chose `us-central1` so the functions and the bucket are in the same region.
WHITELIST	Requests must come from http://localhost:4200.
REFERER	Requests originate from http://localhost:4200/.

2. Validating Environment Variables

import logger from "firebase-functions/logger";

export function validate(value: string | undefined, fieldName: string, missingKeys: string[]) {
  const err = `${fieldName} is missing.`;
  if (!value) {
    logger.error(err);
    missingKeys.push(fieldName);
    return "";
  }

  return value;
}

export const VIDEO_CONFIG = (() => {
  process.loadEnvFile();

  const env = process.env;
  const isVeo31Used = (env.IS_VEO31_USED || "false") === "true";
  const pollingPeriod = Number(env.POLLING_PERIOD_MS || "10000");

  const missingKeys: string[] = [];
  const model = validate(env.GEMINI_VIDEO_MODEL_NAME, "Gemini Video Model Name", missingKeys);
  const project = validate(env.GCLOUD_PROJECT, "Project ID", missingKeys);
  const location = validate(env.GOOGLE_CLOUD_LOCATION, "Google Cloud Location", missingKeys);

  if (missingKeys.length > 0) {
    throw new Error(`Missing environment variables: ${missingKeys.join(", ")}`);
  }

  return {
    genAIOptions: {
      project,
      location,
      vertexai: true,
    },
    aiVideoOptions: {
      model,
      storageBucket: `${project}.firebasestorage.app`,
      isVeo31Used,
      pollingPeriod,
    },
  };
})();

I am using Node 24 as of April 2026. Since Node 20, we can use the built-in process.loadEnvFile function that loads environment variables from the .env file.

In env.ts, the try-catch block attempts to load the environment variables from the .env file.

try {
    process.loadEnvFile();
} catch {
    // Ignore error if .env file is not found (e.g., in production where env vars are set by the platform)
}

In src/index.ts, the first statement imports the env.ts before importing other files and libraries.

import "./env";

... other import statements ...

If you are using a Node version that does not support process.loadEnvfile, the alternative is to install dotenv to load the environment variables.

npm i dotenv

import dotenv from "dotenv";

dotenv.config();

Firebase provides the GCLOUD_PROJECT variable, so it is not defined in the .env file.

3. Extending a Video and Storing in Firebase Storage

The extendVideo Cloud Function passes the payload to the extendVideoFunction function.

const cors = process.env.WHITELIST ? process.env.WHITELIST.split(",").map((origin) => origin.trim()): true;
const options = {
  cors,
  enforceAppCheck: true,
  timeoutSeconds: 600,
};

export const extendVideo = onCall( options,
  ({ data }) => extendVideoFunction(data)
);

import { GoogleGenAI } from "@google/genai";
import { ExtendVideoRequest } from "./types/video.type";
import { extendVideoByPolling, VIDEO_CONFIG } from "./video.util";

export async function extendVideoFunction(data: ExtendVideoRequest) {
  const { genAIOptions, aiVideoOptions } = VIDEO_CONFIG;

  try {
    if (!aiVideoOptions.isVeo31Used) {
      throw new Error("Video extension is only supported for Veo 3.1 model");
    }

    const ai = new GoogleGenAI(genAIOptions);
    return await extendVideoByPolling({ ai, ...aiVideoOptions }, {
      prompt: data.prompt,
      video: data.video,
      config: data.config,
    });
  } catch (error) {
    console.error("Error generating video:", error);
    throw new Error("Error generating video");
  }
}

import { GenerateVideosConfig, GoogleGenAI, Video } from "@google/genai";

export type AIVideoBucket = {
  ai: GoogleGenAI;
  model: string;
  storageBucket: string;
  isVeo31Used: boolean;
  pollingPeriod: number;
}

export type ExtendVideoRequest = {
  prompt: string;
  video: Video;
  config?: GenerateVideosConfig;
}

export async function extendVideoByPolling(
  aiVideo: AIVideoBucket,
  request: ExtendVideoRequest,
) {
  return processVideoPolling(aiVideo, {
    prompt: request.prompt,
    config: request.config,
    video: request.video,
  });
}

async function processVideoPolling(
  { ai, model, storageBucket, pollingPeriod }: AIVideoBucket,
  mediaParams: VideoMediaParams
) {
  const genVideosParams: GenerateVideosParameters = {
    model,
    ...mediaParams,
    config: {
      ...mediaParams.config,
      numberOfVideos: 1,
      outputGcsUri: `gs://${storageBucket}`,
    },
  };

  return getVideoUri(ai, genVideosParams, pollingPeriod);
}

The extendVideoByPolling function invokes the processVideoPolling function to poll the video operation until it completes.

4. Asynchronous Polling

Video extension is a long-running task. Because Vertex AI processes them asynchronously, the functions must poll the operation status in a while loop until the done flag is true.

The Gemini API cannot see the Cloud Storage for Firebase Local Emulator, so it requires a real output GCS URI, which is gs://${storageBucket}.

When the done flag is true, the operation ends and one of three outcomes occurs:

Outcome 1: The error is true, and the video failed to generate. Therefore, the function throws an error.
Outcome 2: The video is successfully stored in the GCS bucket. The function returns the GCS URI and the MIME type to the client application.
Outcome 3: Neither happens. There is no error and no GCS URI, so the function returns an unknown error.

import { GenerateVideosParameters, GoogleGenAI } from "@google/genai";

async function getVideoUri(
  ai: GoogleGenAI,
  genVideosParams: GenerateVideosParameters,
  pollingPeriod: number,
): Promise<{ uri: string, mimeType: string }> {
  let operation = await ai.models.generateVideos(genVideosParams);

  while (!operation.done) {
    await new Promise((resolve) => setTimeout(resolve, pollingPeriod));
    operation = await ai.operations.getVideosOperation({ operation });
  }

  if (operation.error) {
    const strError = `Video generation failed: ${operation.error.message}`;
    console.error(strError);
    throw new Error(strError);
  }

  const video = operation.response?.generatedVideos?.[0]?.video || {};
  const { uri, mimeType } = video;
  if (uri && mimeType) {
    console.log("video uri", uri, mimeType);
    return { uri, mimeType };
  }

  const strError = "Video generation finished but no uri was provided.";
  console.error(strError);
  throw new Error(strError);
}

5. Firebase App Configuration and reCAPTCHA Site Key

getFirebaseConfig is a Firebase Cloud Function that returns both the Firebase app configuration and the reCAPTCHA site key.

import logger from "firebase-functions/logger";
import { validate } from "./validate";

function validateFirebaseConfigFields(env: NodeJS.ProcessEnv) {
  const missingKeys: string[] = [];
  const apiKey = validate(env.APP_API_KEY, "API Key", missingKeys);
  const appId = validate(env.APP_ID, "App Id", missingKeys);
  const messagingSenderId = validate(env.APP_MESSAGING_SENDER_ID, "Messaging Sender ID", missingKeys);
  const recaptchaSiteKey = validate(env.RECAPTCHA_ENTERPRISE_SITE_KEY, "Recaptcha site key", missingKeys);
  const projectId = validate(env.GCLOUD_PROJECT, "Project ID", missingKeys);

  if (missingKeys.length > 0) {
    throw new Error(`Missing environment variables: ${missingKeys.join(", ")}`);
  }

  return {
    app: {
      apiKey,
      appId,
      projectId,
      messagingSenderId,
      authDomain: `${projectId}.firebaseapp.com`,
      storageBucket: `${projectId}.firebasestorage.app`,
    },
    recaptchaSiteKey,
  };
}

export const getFirebaseConfigFunction = () => {
  logger.info("getFirebaseConfig called");

  process.loadEnvFile();

  const variables = validateFirebaseConfigFields(process.env);
  if (!variables) {
    return undefined;
  }

  return variables;
};

6. Local Development with Emulators

The port number defaulted to 5001 when firebase init was executed.

export function connectEmulators({ remoteConfig, functions }: FirebaseObjects) {
  const useEmulators = getValue(remoteConfig, 'useEmulators').asBoolean();
  if (useEmulators) {
    console.log('Connecting to emulators...');
    const host = getValue(remoteConfig, 'functionEmulatorHost').asString();
    const port = getValue(remoteConfig, 'functionEmulatorPort').asNumber();
    console.log('functionEmulator', `${host}:${port}`);
    connectFunctionsEmulator(functions, host, port);
  }
}

loadFirebaseConfig is a helper function that makes request to the Cloud function to obtain the Firebase App configuration and the reCAPTCHA site key.

{
  "getFirebaseConfigUrl": "http://127.0.0.1:5001/vertexai-firebase-6a64f/us-central1/getFirebaseConfig"
}

import { HttpClient } from '@angular/common/http';
import { inject } from '@angular/core';
import { initializeAppCheck, ReCaptchaEnterpriseProvider } from 'firebase/app-check';
import { catchError, lastValueFrom, throwError } from 'rxjs';
import config from '../../public/config.json';
import { ConfigService } from './ai/services/config.service';
import { FirebaseConfigResponse } from './ai/types/firebase-config.type';
import { connectEmulators, initFirebaseApp } from './firebase.util';

async function loadFirebaseConfig() {
  const httpService = inject(HttpClient);
  const firebaseConfig$ =
    httpService.get<FirebaseConfigResponse>(`${config.getFirebaseConfigUrl}`)
      .pipe(catchError((e) => throwError(() => e)));
  return lastValueFrom(firebaseConfig$);
}

export async function bootstrapFirebase() {
    try {
      const configService = inject(ConfigService);
      const firebaseConfig = await loadFirebaseConfig();
      const { app, recaptchaSiteKey } = firebaseConfig;
      const firebaseObjects = await initFirebaseApp(app);
      const { firebaseApp } = firebaseObjects;

      initializeAppCheck(firebaseApp, {
        provider: new ReCaptchaEnterpriseProvider(recaptchaSiteKey),
        isTokenAutoRefreshEnabled: true,
      });

      connectEmulators(firebaseObjects);
      configService.loadConfig(firebaseObjects);
    } catch (err) {
      console.error(err);
    }
}

The AppConfig remains unchanged.

import { ApplicationConfig, provideAppInitializer } from '@angular/core';
import { bootstrapFirebase } from './app.bootstrap';

export const appConfig: ApplicationConfig = {
  providers: [
    provideAppInitializer(async () => bootstrapFirebase()),
  ]
};

7. Firebase Remote Configuration

Variable	Description	Value
maxVideoExtendAllowed	Maximum number of extensions allowed	4

8. Video Player Component in Angular

The Angular frontend triggers the process using httpsCallable. Once the function returns the Cloud Storage path and the MIME type, the app fetches the download URL for playback.

The ConfigService stores the Firebase app, Remote Config, and functions to be used throughout the application.

import { FirebaseApp } from 'firebase/app';
import { Functions } from 'firebase/functions';
import { RemoteConfig } from 'firebase/remote-config';

export type FirebaseObjects = {
  firebaseApp: FirebaseApp;
  remoteConfig: RemoteConfig;
  functions: Functions;
}

import { Injectable } from '@angular/core';
import { FirebaseObjects } from '../types/firebase-objects';

@Injectable({
  providedIn: 'root'
})
export class ConfigService  {
    firebaseObjects: FirebaseObjects | undefined = undefined;

    loadConfig(firebaseObjects: FirebaseObjects) {
      this.firebaseObjects = firebaseObjects;
    }
}

The retrieveVideoUri method calls the Cloud Function directly to retrieve the GCS URI and the MIME type.

The downloadVideoUriAndUrl method resolves the URI to an HTTP URL so that an HTML video player can play it immediately.

export type VideoGenerationResponse = {
  uri: string;
  url: string;
  mimeType: string;
}

export type DownloadVideoResponse = {
  uri: string;
  mimeType: string;
}

import { inject, Injectable } from '@angular/core';
import { httpsCallable } from 'firebase/functions';
import { getDownloadURL, getStorage, ref } from 'firebase/storage';
import { GenerateVideoRequest } from '../types/video.type';
import { ConfigService } from './config.service';

@Injectable({
  providedIn: 'root'
})
export class VeoService {
  private readonly storage = getStorage();
  private readonly configService = inject(ConfigService);

  private async retrieveVideoUri<T = GenerateVideoRequest>(request: T, methodName: string): Promise<DownloadVideoResponse> {
    try {
      const { functions } = this.configService.firebaseObjects || {};
      if (!functions) {
        throw new Error('Functions does not exist.');
      }

      const downloadGcsUri = httpsCallable<T, DownloadVideoResponse>(
        functions, methodName, { timeout: 600000 }
      );
      const { data } = await downloadGcsUri(request);
      return data;
    } catch (err) {
        console.error(err);
        throw err;
    }
  }

  async downloadVideoUriAndUrl<T = GenerateVideoRequest>(request: T, methodName: CallableNames = 'videos-generateVideo'): Promise<VideoGenerationResponse> {
    const { uri, mimeType } = await this.retrieveVideoUri(request, methodName);

    if (!uri) {
      throw new Error('Video operation completed but no URI was returned.');
    }

    return getDownloadURL(ref(this.storage, uri))
      .then((url) => {
        console.log("download url", url);
        return { uri, url, mimeType };
      })
      .catch((error) => {
        switch (error.code) {
          case 'storage/object-not-found':
            throw new Error("File doesn't exist");
          case 'storage/unauthorized':
            throw new Error("User doesn't have permission to access the object");
          case 'storage/canceled':
            throw new Error("User canceled the upload");
          case 'storage/unknown':
            throw new Error("Unknown storage error occurred, inspect the server response");
        }
        throw new Error("Unknown error occurred");
      });
  }
}

import { LoaderComponent } from '@/shared/loader/loader.component';
import { ChangeDetectionStrategy, Component, input, output } from '@angular/core';
import { ExtendVideoIconComponent } from '../../icons/extend-video-icon.component';

@Component({
  selector: 'app-video-player',
  imports: [LoaderComponent, ExtendVideoIconComponent],
  template: `
@if (isGeneratingVideo()) {
  <div class="mt-6">
    <app-loader [loadingText]="loadingText()">
      <p class="text-sm">This can take several minutes. Please be patient.</p>
    </app-loader>
  </div>
} @else if (videoUrl()) {
  <div class="mt-6 flex flex-col gap-4">
    <div class="video-container">
      <video [src]="videoUrl()" controls autoplay loop class="w-full rounded-md"></video>
    </div>

    <div class="flex justify-center">
      <button
        (click)="extendVideo.emit()"
        aria-label="Extend video"
        title="Extend video"
        class="extend-btn"
      >
        <app-extend-video-icon />
        <span>Extend Video</span>
      </button>
    </div>
  </div>
}
  `,
  styleUrl: './video-player.component.css',
  changeDetection: ChangeDetectionStrategy.OnPush,
})
export class VideoPlayerComponent {
  isGeneratingVideo = input(false);
  videoUrl = input.required<string>();
  extendVideo = output<void>();

  loadingText = input('Generating your video...');
}

9. Angular Integration with Gemini to Extend Video

import { VideoPlayerComponent } from './video-player/video-player.component';

@Component({
  selector: 'app-gen-media',
  imports: [
    VideoPlayerComponent,
  ],
  template: `
  <app-video-player
    [isGeneratingVideo]="isGeneratingVideo()" 
    [videoUrl]="videoUrl()"
    (extendVideo)="extendVideo()"
    [loadingText]="loadingVideoText()"
  />`,
  changeDetection: ChangeDetectionStrategy.OnPush,
})
export class GenMediaComponent {
  private readonly genMediaService = inject(GenMediaService);
  private readonly genVideoService = inject(GenVideoService);

  genMediaInput = input<GenMediaInput>();
  videoUrl = this.genVideoService.videoUrl;
  isGeneratingVideo = this.genVideoService.isGeneratingVideo.asReadonly();
  loadingVideoText = signal('Generating your video...');

  trimmedUserPrompt = computed(() => this.genMediaInput()?.userPrompt.trim() || '');

  async extendVideo() {
    if (this.trimmedUserPrompt()) {
      try {
        this.loadingVideoText.set('Extending your video...');
        await this.genVideoService.extendVideo(this.trimmedUserPrompt());
      } finally {
        this.loadingVideoText.set('Generating your video...');
      }
    }
  }
}

The GenMediaComponent calls extendVideo with the prompt and the Veo video URL to create an extended video.

import { ConfigService } from '@/ai/services/config.service';
import { VeoService } from '@/ai/services/veo.service';
import { ExtendVideoRequest, GenerateVideoFromFramesRequest, GenerateVideoRequest, VideoGenerationResponse } from '@/ai/types/video.type';
import { computed, inject, Injectable, signal, WritableSignal } from '@angular/core';
import { getValue } from 'firebase/remote-config';

@Injectable({
  providedIn: 'root'
})
export class GenVideoService {
  private readonly veoService = inject(VeoService);
  private readonly configService = inject(ConfigService);

  videoError = signal('');
  videoResponse = signal<VideoGenerationResponse>({ uri: '', url: '', mimeType: '' });
  #extendVideoCounter = signal(0);
  isGeneratingVideo = signal(false);

  videoUrl = computed(() => this.videoResponse().url);

  isVideoExtensionAllowed(counter: number) {
    const remoteConfig = this.configService.firebaseObjects?.remoteConfig;
    if (!remoteConfig) {
      console.warn('Remote config does not exist.');
      return false;
    }

    const max_extend_allowed = getValue(remoteConfig,'maxVideoExtendAllowed').asNumber();
    if (counter >= max_extend_allowed) {
      console.warn('Maximum extension limit reached.');
      return false;
    }

    return true;
  }

  async extendVideo(prompt: string): Promise<void> {
    try {
      const result = await this.extendInterpolatedVideo(
        prompt,
        this.#extendVideoCounter(),
        this.videoResponse()
      );

      if (!result) {
        return;
      }

      this.videoResponse.set(result);
      this.#extendVideoCounter.update(count => count + 1);
      console.log(`Video extended successfully. Current extension count: ${this.#extendVideoCounter()}`);
    } catch (e) {
      console.error(e);
      const errMsg = e instanceof Error ?
        e.message :
        'An unexpected error occurred in video generation.'
      this.videoError.set(errMsg);
    } finally {
      this.isGeneratingVideo.set(false);
    }
  }

  async extendInterpolatedVideo(
    prompt: string,
    counter: number,
    customVideo: Pick<VideoGenerationResponse, 'uri' | 'mimeType'>,
    generatingSignal?: WritableSignal<boolean>,
    error?: WritableSignal<string>
  ) {
    const { uri, mimeType } = customVideo;

    if (!mimeType || !uri) {
      console.warn('No video to extend. Please generate a video first.');
      return null;
    }

    if (!prompt) {
      console.warn('Prompt is required to extend the video.');
      return null;
    }

    if (!this.isVideoExtensionAllowed(counter)) {
      return null;
    }

    const actualErrorSignal = error || this.videoError;
    const actualGeneratingSignal = generatingSignal || this.isGeneratingVideo;
    try {
      actualErrorSignal.set('');
      actualGeneratingSignal.set(true);

      const extendVideoParams: ExtendVideoRequest = {
        prompt,
        video: { uri, mimeType },
      }

      return await this.veoService.downloadVideoUriAndUrl(extendVideoParams, 'videos-extendVideo');
    } catch (e) {
      console.error(e);
      throw e;
    } finally {
      actualGeneratingSignal.set(false);
    }
  }
}

Conclusion

Combining Veo 3.1 with the serverless scalability of Firebase is a powerful workflow.

Resources

Observability at Scale: Mastering ADK Callbacks for Cost, Latency, and Auditability

Connie Leung — Mon, 06 Apr 2026 16:08:17 +0000

Observability at Scale: Mastering ADK Callbacks for Cost, Latency, and Auditability

AI orchestrators receive significant attention; however, when deployments become latent and costly, developers often overlook a critical capability: ADK callback hooks. The design patterns and best practices of callback hooks enable developers to refactor logic from agents to callback hooks to add observability, reduce cost and latency, and modify session state dynamically.

This post explores how to create callback hooks at various stages of an ADK agent to demonstrate the following design patterns:

Logging and monitoring performance
Managing dynamic state
Modifying requests and responses
Skipping steps conditionally

Demo Overview

The orchestrator routes the project description to sequentialEvaluationAgent. The sequentialEvaluationAgent consists of project, anti-patterns, decision, recommendation, audit, upload, merger, and email subagents.

In the project, anti-patterns, decision, recommendation, and merger subagents, I implemented callback hooks to demonstrate their capabilities and practicality.

Architecture

Google ADK stands for Agent Development Kit. It is an open source agent framework that enables developers to build and deploy AI agents in a convenient way.

The project, anti-patterns, decision, recommendation, and synthesis agents are LLM agents. These agents require Gemini to reason and generate text responses.

The Audit Trail, Cloud Storage, and Email agents integrate with external APIs or resources that trigger deterministic actions.

Callback Types

ADK provides six types of callback hooks that are called before and after an agent is executed, before and after a model is executed, and before and after a tool is executed.

Callback Type	Description
beforeAgentCallback	Call before the new cycle of an agent
afterAgentCallback	Call after the agent cycle completes
beforeModelCallback	Call before the LLM is called
afterModelCallback	Call after the LLM returns a response
beforeToolCallback	Call before the tool is called
afterToolCallback	Call after the tool is called

When and why I started using ADK callback hooks

When I used ADK web for local testing and debugging, my first priority was correctness. Performance and token usage had lower priorities until the agents were deployed to a QA environment.

When product managers and the QA team recognized apparent slowness and high cost, they would inform me. Next, I examined the agent to identify performance bottlenecks and deterministic steps that could be handled in callback hooks or application code.

The Benefits of Callback

Using callback hooks offers several critical advantages in my subagents:

Type	Benefit
Short Circuit	Call `beforeModelCallback` to validate session data and skip LLM flow when data is invalid
Separation of Concerns	Reset session data in the `beforeAgentCallback` so that tool remains lean and focuses on business logic
Observability	Add logging in the `beforeAgentCallback` and the `afterAgentCallback` to log performance metrics
Dynamic State Management	Create a reusable `afterToolCallback` to increment validation attempts and modify response status to FATAL_ERROR

Prerequisites

Requirements

TypeScript 5.9.3 or later
Node.js 24.13.0 or later
npm 11.8.0 or later
Docker (For running MailHog locally)
Google ADK (For building the custom agent)
Gemini in Vertex AI (to call the model in LLM agents, although not required for the custom agent)

Note: I used Gemini in Vertex AI for authentication due to regional availability. Gemini is currently unavailable in Hong Kong; therefore, I utilized Gemini in Vertex AI instead.

Install npm dependencies

npm i --save-exact @google/adk
npm i --save-dev --save-exact @google/adk-devtools
npm i --save-exact nodemailer
npm i --save-dev --save-exact @types/nodemailer rimraf
npm i --save-exact marked
npm i --save-exact zod

I installed the required dependencies for building ADK agents, converting Markdown strings to HTML, and sending emails to MailHog during local testing.

Pinned dependencies ensure the versioning is the same in development and production environments for enterprise-level applications.

Environment variables

Copy .env.example to .env and fill in the credentials:

GEMINI_MODEL_NAME="gemini-3.1-flash-lite-preview"
GOOGLE_CLOUD_PROJECT="<Google Cloud Project ID>"
GOOGLE_CLOUD_LOCATION="global" 
GOOGLE_GENAI_USE_VERTEXAI=TRUE

# SMTP Settings (MailHog)
SMTP_HOST="localhost"
SMTP_PORT=1025
SMTP_USER=""
SMTP_PASS=""
SMTP_FROM="no-reply@test.local"
ADMIN_EMAIL="admin@test.local"

SMTP_HOST, SMTP_PORT, SMTP_USER, SMTP_PASS, are required to set up MailHog for local email testing.

SMTP_FROM is a from email address that can be any string in local testing.

ADMIN_EMAIL is an administrator email address receiving emails that the EmailAgent sends. It is an environment variable in my use case because it is the only recipient. If another scenario requires sending emails to customers, the environment variable should be removed.

ADK Callback Patterns in Action

Here is how I implemented these four callback design patterns in practice.

Logging and Monitoring

Callback used: beforeAgentCallback and afterAgentCallback

The agentStartCallback stores the current time in milliseconds in start_time variable in the session state.

import { SingleAgentCallback } from '@google/adk';

export const START_TIME_KEY = 'start_time';

export const agentStartCallback: SingleAgentCallback = (context) => {
  if (!context || !context.state) {
    return undefined;
  }

  context.state.set(START_TIME_KEY, Date.now());
  return undefined;
};

The agentEndCallback obtains the start time from the session state and calculates the duration. Then, it uses console.log to log the performance metrics. The callback returns undefined so that any subagent always flows to the next one in the sequential workflow.

export const agentEndCallback: SingleAgentCallback = (context) => {
  if (!context || !context.state) {
    return undefined;
  }

  const now = Date.now();
  const startTime = context.state.get<number>(START_TIME_KEY) || now;
  console.log(
    `Performance Metrics for Agent "${context.agentName}": Total Elapsed Time: ${(now - startTime) / 1000} seconds.`,
  );
  return undefined;
};

Next, I call both callback hooks in the subagents to know how long they take to execute. The following example shows how I log the performance metric of a project subagent.

export function createProjectAgent(model: string) {
  const projectAgent = new LlmAgent({
    name: 'ProjectAgent',
    model,
    beforeAgentCallback: agentStartCallback,
    instruction: (context) => {
      ... LLM instruction ....
    },
    afterAgentCallback: agentEndCallback,
    ... 
  });

  return projectAgent;
}

Reset Session State

Callback used: beforeAgentCallback

The orchestrator resets variables in the session state before the agent lifecycle begins.

export const AUDIT_TRAIL_KEY = 'auditTrail';
export const RECOMMENDATION_KEY = 'recommendation';
export const CLOUD_STORAGE_KEY = 'cloudStorage';
export const DECISION_KEY = 'decision';
export const PROJECT_KEY = 'project';
export const ANTI_PATTERNS_KEY = 'antiPatterns';
export const MERGED_RESULTS_KEY = 'mergedResults';
export const PROJECT_DESCRIPTION_KEY = 'project_description';
export const VALIDATION_ATTEMPTS_KEY = 'validation_attempts';

const resetNewEvaluationCallback: SingleAgentCallback = (context) => {
  if (!context || !context.state) {
    return undefined;
  }

  const state = context.state;

  // Clear all previous evaluation data
  state.set(PROJECT_KEY, null);
  state.set(ANTI_PATTERNS_KEY, null);
  state.set(DECISION_KEY, null);
  state.set(RECOMMENDATION_KEY, null);
  state.set(AUDIT_TRAIL_KEY, null);
  state.set(CLOUD_STORAGE_KEY, null);
  state.set(MERGED_RESULTS_KEY, null);
  state.set(VALIDATION_ATTEMPTS_KEY, 0);

  console.log(
    `beforeAgentCallback: Agent ${context.agentName} has reset the session state for a new evaluation cycle.`,
  );

  return undefined;
};

The orchestrator sets the variables to null in the beforeAgentCallback. In the prepareEvaluationTool, the orchestrator only replaces the description with the new project description.

const prepareEvaluationTool = new FunctionTool({
  name: 'prepare_evaluation',
  description: 'Stores the new project description to prepare for a fresh evaluation.',
  parameters: z.object({
    description: z.string().describe('The validated project description from the user.'),
  }),
  execute: ({ description }, context) => {
    if (!context || !context.state) {
      return { status: 'ERROR', message: 'No session state found.' };
    }

    // Set the new description for the ProjectAgent to find
    context.state.set(PROJECT_DESCRIPTION_KEY, description);

    return { status: 'SUCCESS', message: 'Description updated.' };
  },
});

The tool logic is more efficient and the token usage is reduced.

export const rootAgent = new LlmAgent({
  name: 'ProjectEvaluationAgent',
  model: 'gemini-3.1-flash-lite-preview',
  beforeAgentCallback: resetNewEvaluationCallback,
  instruction: `
    ... LLM instruction ....
  `,
  tools: [prepareEvaluationTool],
  subAgents: [sequentialEvaluationAgent],
});

The orchestrator uses the resetNewEvaluationCallback to reset the session variables, and the prepareEvaluationTool to replace the project description. Finally, the sequentialEvaluationAgent starts the agentic workflow to derive a decision and generate the recommendation.

Dynamic State Management

Prerequisite: The subagent uses tool calling to perform action
Callback used: afterToolCallback

The use case is to increment the value of validation_attempts in the session state. After consulting AI, the afterToolCallback stage of project and decision subagents is the perfect location to increment it. Therefore, I define a meta-function that creates a afterToolCallback to increment the validation attempts.

export const VALIDATION_ATTEMPTS_KEY = 'validation_attempts';

export const MAX_ITERATIONS = 3;

import { AfterToolCallback } from '@google/adk';
import { VALIDATION_ATTEMPTS_KEY } from '../output-keys.const.js';
import { MAX_ITERATIONS } from '../validation.const.js';

export function createAfterToolCallback(fatalErrorMessage: string, maxAttempts = MAX_ITERATIONS): AfterToolCallback {
  return ({ tool, context, response }) => {
    if (!tool || !context || !context.state) {
      return undefined;
    }

    const toolName = tool.name;
    const agentName = context.agentName;
    const state = context.state;

    if (!response || typeof response !== 'object' || !('status' in response)) {
      return undefined;
    }

    // [1] Dynamic state management
    const attempts = (state.get<number>(VALIDATION_ATTEMPTS_KEY) || 0) + 1;
    state.set(VALIDATION_ATTEMPTS_KEY, attempts);

    // [2] Response modification
    const status = response.status || 'ERROR';
    if (status === 'ERROR' && attempts >= maxAttempts) {
      context.actions.escalate = true;

      return {
        status: 'FATAL_ERROR',
        message: fatalErrorMessage,
      };
    }
  };
}

In both project and decision subagents, I invoke createAfterToolCallback to create their own afterToolCallback to increment the validation attempts.

const projectAfterToolCallback = createAfterToolCallback(
  `STOP processing immediately. Max validation attempts reached. Return the most accurate data found so far or empty strings if none.`,
);

const decisionAfterToolCallback = createAfterToolCallback(
  `STOP processing immediately and output the final JSON schema with verdict: "None".`,
);

Next, I call the projectAfterToolCallback in the afterToolCallback stage of the project subagent.

export function createProjectAgent(model: string) {
  const projectAgent = new LlmAgent({
    name: 'ProjectAgent',
    model,
    beforeAgentCallback: agentStartCallback,
    instruction: (context) => {
      ... LLM instruction ....
    },
    afterToolCallback: projectAfterToolCallback,
    afterAgentCallback: agentEndCallback,
    tools: [validateProjectTool],
    ... 
  });

  return projectAgent;
}

Next, I call the decisionAfterToolCallback in the afterToolCallback stage of the decision subagent.

export function createDecisionTreeAgent(model: string) {
  const decisionTreeAgent = new LlmAgent({
    name: 'DecisionTreeAgent',
    model,
    beforeAgentCallback: [resetAttemptsCallback, agentStartCallback],
    instruction: (context) => {
      ... instruction  of the LLM flow ...
    },
    afterToolCallback: decisionAfterToolCallback,
    afterAgentCallback: agentEndCallback,
    tools: [validateDecisionTool],
    ...
  });

  return decisionTreeAgent;
}

Request/Response Modification

Prerequisite: The subagent uses tool calling to perform action
Callback used: AfterToolCallback

The same AfterToolCallback also modifies the response when the number of validation attempts exceeds the maximum iterations.

When the status is ERROR and the value of validation_attempts is at least maximum_iterations, the context.actions.escalate flag is set to true to break out of the loop. Moreover, the status is changed to FATAL_ERROR and the custom fatal message is returned.

Conditional Skipping of Steps

This is an important design pattern for avoiding unnecessary LLM executions.

Callback used: beforeModelCallback.

In my subagents, I run this callback to validate the session data. When a specific condition is met, the callback returns content immediately to skip the subsequent LLM flow.

Example 1

If the project agent is able to break down a project description into task, problem, constraint, and goal, the agent will return the breakdown immediately. Otherwise, the agent prompts Gemini to use reasoning to perform the breakdown.

import { SingleBeforeModelCallback } from '@google/adk';

const beforeModelCallback: SingleBeforeModelCallback = ({ context }) => {
  const { project } = getEvaluationContext(context);
  const { isCompleted } = isProjectDetailsFilled(project);

  if (isCompleted) {
    return {
      content: {
        role: 'model',
        parts: [
          {
            text: JSON.stringify(project),
          },
        ],
      },
    };
  }

  return undefined;
};

Then, beforeModelCallback is hooked to the beforeModelCallback stage of the project subagent.

export function createProjectAgent(model: string) {
  const projectAgent = new LlmAgent({
    name: 'ProjectAgent',
    model,
    description:
      'Analyzes the user-provided project description to extract and structure its core components, including the primary task, underlying problem, ultimate goal, and architectural constraints.',
    beforeAgentCallback: agentStartCallback,
    beforeModelCallback,
    instruction: (context) => {
      const { projectDescription } = getEvaluationContext(context);
      if (!projectDescription) {
        return '';
      }

      return generateProjectBreakdownPrompt(projectDescription);
    },
    afterToolCallback: projectAfterToolCallback,
    afterAgentCallback: agentEndCallback,
    tools: [validateProjectTool],
    outputSchema: projectSchema,
    outputKey: PROJECT_KEY,
    disallowTransferToParent: true,
    disallowTransferToPeers: true,
  });

  return projectAgent;
}

Example 2

The decision agent verifies the verdict property in the beforeModelCallback. If the verdict is not None, the callback returns the valid decision immediately. The verdict is None and the callback examines the project breakdown and the anti-patterns. When project breakdown and anti-patterns are provided, the callback returns undefined to trigger the LLM flow. Otherwise, the decision agent does not have valid inputs to derive the verdict. The callback returns None in this edge case.

import { SingleBeforeModelCallback } from '@google/adk';

const beforeModelCallback: SingleBeforeModelCallback = ({ context }) => {
  const { decision } = getEvaluationContext(context);

  if (decision && decision.verdict !== 'None') {
    return {
      content: {
        role: 'model',
        parts: [
          {
            text: JSON.stringify(decision),
          },
        ],
      },
    };
  }

  const { project, antiPatterns } = getEvaluationContext(context);
  const { isCompleted } = isProjectDetailsFilled(project);

  if (isCompleted && antiPatterns) {
    return undefined;
  }

  return {
    content: {
      role: 'model',
      parts: [
        {
          text: JSON.stringify({
            verdict: 'None',
            nodes: [],
          }),
        },
      ],
    },
  };
};

Then, beforeModelCallback is hooked to the beforeModelCallback stage of the project subagent.

export function createDecisionTreeAgent(model: string) {
  const decisionTreeAgent = new LlmAgent({
    name: 'DecisionTreeAgent',
    model,
    beforeAgentCallback: [resetAttemptsCallback, agentStartCallback],
    beforeModelCallback,
    instruction: (context) => {
      ... instruction  of the LLM flow ...
    },
    afterToolCallback: decisionAfterToolCallback,
    afterAgentCallback: agentEndCallback,
    tools: [validateDecisionTool],
    outputSchema: decisionSchema,
    outputKey: DECISION_KEY,
    disallowTransferToParent: true,
    disallowTransferToPeers: true,
  });

  return decisionTreeAgent;
}

Example 3

The recommendation agent uses the beforeModelCallback to examine the project breakdown, anti-patterns and verdict. There are two scenarios that need LLM to generate the recommendation. The first scenario is valid project breakdown, anti-patterns, and non-None verdict. The second scenario is incomplete project breakdown and None verdict. The LLM is instructed to describe the missing field in the project breakdown and how important the missing field is to obtain a valid decision. For other cases, the callback returns a static recommendation immediately and skip the subsequent LLM flow.

function constructRecommendation(recommendation: string) {
  return {
    content: {
      role: 'model',
      parts: [
        {
          text: JSON.stringify({
            text: recommendation,
          }),
        },
      ],
    },
  };
}

const beforeModelCallback: SingleBeforeModelCallback = ({ context }) => {
  const { project, antiPatterns, decision } = getEvaluationContext(context);

  const { isCompleted } = isProjectDetailsFilled(project);
  const isDecisionNone = decision && decision.verdict === 'None';

  if ((isCompleted && antiPatterns && decision && decision.verdict !== 'None') || (!isCompleted && isDecisionNone)) {
    return undefined;
  } else if (isCompleted && isDecisionNone) {
    return constructRecommendation(
      '## Recommendation: Manual Review Required\n\n**Status:** Abnormal Case Detected\n\nThe provided project is complete and valid, but the decision tree could not reach a conclusive verdict (Result: `None`).\n\n**Possible Reasons:**\n- The requirements fall outside of known architectural patterns.\n- There are conflicting constraints and goals that cannot be resolved automatically.\n\n**Next Steps:**\n- Review and refine the constraints or goals.\n- Escalate for manual architectural review.',
    );
  }

  return constructRecommendation(
    '## Recommendation: Data Required\n\n**Status:** Abnormal Case Detected\n\nNo decision is reached.',
  );
};

Similar to the previous project and decision agents, the beforeModelCallback function is hooked to the beforeModelCallback stage of the recommendation agent.

export function createRecommendationAgent(model: string) {
  const recommendationAgent = new LlmAgent({
    name: 'RecommendationAgent',
    model,
    beforeModelCallback,
    beforeAgentCallback: agentStartCallback,
    instruction: (context) => {
      const { project, antiPatterns, decision } = getEvaluationContext(context);
      const { isCompleted, missingFields } = isProjectDetailsFilled(project);

      if (project) {
        if (!isCompleted && decision && decision.verdict === 'None') {
          console.log('RecommendationAgent -> generateFailedDecisionPrompt');
          return generateFailedDecisionPrompt(project, missingFields);
        } else if (isCompleted && antiPatterns && decision && decision.verdict !== 'None') {
          console.log('RecommendationAgent -> generateRecommendationPrompt');
          return generateRecommendationPrompt(project, antiPatterns, decision);
        }
      }
      return 'Skipping LLM due to missing data.';
    },
    afterAgentCallback: agentEndCallback,
    outputSchema: recommendationSchema,
    outputKey: RECOMMENDATION_KEY,
    disallowTransferToParent: true,
    disallowTransferToPeers: true,
  });

  return recommendationAgent;
}

These are the callback design patterns that the agent adopts. In the next section, I describe how to launch the agent to observe the logging messages in the terminal.

Environment Setup

I pulled the latest version of the MailHog Docker image from Docker Hub and started it locally to receive test emails and display them in the Web UI. The docker-compose.yml file contains the setup configuration.

services:
  mailhog:
    image: mailhog/mailhog
    container_name: mailhog
    ports:
      - '1025:1025' # SMTP port
      - '8025:8025' # HTTP (Web UI) port
    restart: always
    networks:
      - decision-tree-agent-network

networks:
  decision-tree-agent-network:

The SMTP server listens on port 1025, and the Web UI is accessible at http://localhost:8025.

docker compose up -d

Start MailHog in Docker.

Testing

Add scripts to package.json to build and start the ADK web interface.

  "scripts": {
    "prebuild": "rimraf dist",
    "build": "npx tsc --project tsconfig.json",
    "web": "npm run build && npx @google/adk-devtools web --host 127.0.0.1 dist/agent.js"
  },

Open a terminal and type npm run web to start the API server.
Open a new browser tab and type http://localhost:8000.
Paste the following text into the message box:

One of my favorite tech influencers just tweeted about a 'breakthrough in solid-state batteries.' Find which public company they might be referring to, check that company’s recent patent filings to see if it’s true, and then check their stock price to see if the market has already 'priced it in'.

Ensure the root agent executes and halts when the email agent terminates. The orchestrator and the project agent trigger the callback hooks to reset the session state, log performance metrics, validate session data, increment the validation attempts, and modify the response.

Similarly, both anti-patterns and decisions agents used the beforeAgentCallback and afterAgentCallback to log performance metrics. They also used the beforeModelCallback to validate session data before calling the LLM to generate a response. Moreover, the decision agent incremented the validation attempts and modify the status to FATAL_ERROR when the validation attempts exceeded or equal to maximum iterations in the afterToolCallback.

Similarly, both recommendation and merger agents used the beforeAgentCallback and afterAgentCallback to log performance metrics. The recommendation agent also used the beforeModelCallback to validate session data before calling the LLM to generate a recommendation.

Conclusion

ADK supports lifecycle callback hooks that are applicable to different use cases. In this post, I describe logging performance metrics in the beforeAgentCallback and afterAgentCallback. I refactored the reset session state logic in the orchestrator's beforeAgentCallback to make the tool lean and cost efficient. The afterToolCallback escalated to a higher agent and modified the response status to FATAL_ERROR when the loop count in the validation-retry loop exceeded the maximum iterations. When skipping LLM calls conditionally, the beforeModelCallback returned custom content immediately. Then, the agent did not add unnecessary time to the agent flow and did not consume tokens.

The takeaway: follow the design patterns and best practices of callback at various stages of an agent to log performance metrics, reduce cost and latency, and modify response.

Resources

Observability at Scale: Mastering ADK Callbacks for Cost, Latency, and Auditability [GDE]

Connie Leung — Mon, 06 Apr 2026 16:02:04 +0000

This post explores how to create callback hooks at various stages of an ADK agent to demonstrate the following design patterns:

Logging and monitoring performance
Managing dynamic state
Modifying requests and responses
Skipping steps conditionally

Demo Overview

In the project, anti-patterns, decision, recommendation, and merger subagents, I implemented callback hooks to demonstrate their capabilities and practicality.

Architecture

Google ADK stands for Agent Development Kit. It is an open source agent framework that enables developers to build and deploy AI agents in a convenient way.

The project, anti-patterns, decision, recommendation, and synthesis agents are LLM agents. These agents require Gemini to reason and generate text responses.

The Audit Trail, Cloud Storage, and Email agents integrate with external APIs or resources that trigger deterministic actions.

Callback Types

ADK provides six types of callback hooks that are called before and after an agent is executed, before and after a model is executed, and before and after a tool is executed.

Callback Type	Description
beforeAgentCallback	Call before the new cycle of an agent
afterAgentCallback	Call after the agent cycle completes
beforeModelCallback	Call before the LLM is called
afterModelCallback	Call after the LLM returns a response
beforeToolCallback	Call before the tool is called
afterToolCallback	Call after the tool is called

When and why I started using ADK callback hooks

When I used ADK web for local testing and debugging, my first priority was correctness. Performance and token usage had lower priorities until the agents were deployed to a QA environment.

The Benefits of Callback

Using callback hooks offers several critical advantages in my subagents:

Type	Benefit
Short Circuit	Call `beforeModelCallback` to validate session data and skip LLM flow when data is invalid
Separation of Concerns	Reset session data in the `beforeAgentCallback` so that tool remains lean and focuses on business logic
Observability	Add logging in the `beforeAgentCallback` and the `afterAgentCallback` to log performance metrics
Dynamic State Management	Create a reusable `afterToolCallback` to increment validation attempts and modify response status to FATAL_ERROR

Prerequisites

Requirements

TypeScript 5.9.3 or later
Node.js 24.13.0 or later
npm 11.8.0 or later
Docker (For running MailHog locally)
Google ADK (For building the custom agent)
Gemini in Vertex AI (to call the model in LLM agents, although not required for the custom agent)

Note: I used Gemini in Vertex AI for authentication due to regional availability. Gemini is currently unavailable in Hong Kong; therefore, I utilized Gemini in Vertex AI instead.

Install npm dependencies

npm i --save-exact @google/adk
npm i --save-dev --save-exact @google/adk-devtools
npm i --save-exact nodemailer
npm i --save-dev --save-exact @types/nodemailer rimraf
npm i --save-exact marked
npm i --save-exact zod

I installed the required dependencies for building ADK agents, converting Markdown strings to HTML, and sending emails to MailHog during local testing.

Pinned dependencies ensure the versioning is the same in development and production environments for enterprise-level applications.

Environment variables

Copy .env.example to .env and fill in the credentials:

GEMINI_MODEL_NAME="gemini-3.1-flash-lite-preview"
GOOGLE_CLOUD_PROJECT="<Google Cloud Project ID>"
GOOGLE_CLOUD_LOCATION="global" 
GOOGLE_GENAI_USE_VERTEXAI=TRUE

# SMTP Settings (MailHog)
SMTP_HOST="localhost"
SMTP_PORT=1025
SMTP_USER=""
SMTP_PASS=""
SMTP_FROM="no-reply@test.local"
ADMIN_EMAIL="admin@test.local"

SMTP_HOST, SMTP_PORT, SMTP_USER, SMTP_PASS, are required to set up MailHog for local email testing.

SMTP_FROM is a from email address that can be any string in local testing.

ADK Callback Patterns in Action

Here is how I implemented these four callback design patterns in practice.

Logging and Monitoring

Callback used: beforeAgentCallback and afterAgentCallback

The agentStartCallback stores the current time in milliseconds in start_time variable in the session state.

import { SingleAgentCallback } from '@google/adk';

export const START_TIME_KEY = 'start_time';

export const agentStartCallback: SingleAgentCallback = (context) => {
  if (!context || !context.state) {
    return undefined;
  }

  context.state.set(START_TIME_KEY, Date.now());
  return undefined;
};

export const agentEndCallback: SingleAgentCallback = (context) => {
  if (!context || !context.state) {
    return undefined;
  }

  const now = Date.now();
  const startTime = context.state.get<number>(START_TIME_KEY) || now;
  console.log(
    `Performance Metrics for Agent "${context.agentName}": Total Elapsed Time: ${(now - startTime) / 1000} seconds.`,
  );
  return undefined;
};

Next, I call both callback hooks in the subagents to know how long they take to execute. The following example shows how I log the performance metric of a project subagent.

export function createProjectAgent(model: string) {
  const projectAgent = new LlmAgent({
    name: 'ProjectAgent',
    model,
    beforeAgentCallback: agentStartCallback,
    instruction: (context) => {
      ... LLM instruction ....
    },
    afterAgentCallback: agentEndCallback,
    ... 
  });

  return projectAgent;
}

Reset Session State

Callback used: beforeAgentCallback

The orchestrator resets variables in the session state before the agent lifecycle begins.

export const AUDIT_TRAIL_KEY = 'auditTrail';
export const RECOMMENDATION_KEY = 'recommendation';
export const CLOUD_STORAGE_KEY = 'cloudStorage';
export const DECISION_KEY = 'decision';
export const PROJECT_KEY = 'project';
export const ANTI_PATTERNS_KEY = 'antiPatterns';
export const MERGED_RESULTS_KEY = 'mergedResults';
export const PROJECT_DESCRIPTION_KEY = 'project_description';
export const VALIDATION_ATTEMPTS_KEY = 'validation_attempts';

const resetNewEvaluationCallback: SingleAgentCallback = (context) => {
  if (!context || !context.state) {
    return undefined;
  }

  const state = context.state;

  // Clear all previous evaluation data
  state.set(PROJECT_KEY, null);
  state.set(ANTI_PATTERNS_KEY, null);
  state.set(DECISION_KEY, null);
  state.set(RECOMMENDATION_KEY, null);
  state.set(AUDIT_TRAIL_KEY, null);
  state.set(CLOUD_STORAGE_KEY, null);
  state.set(MERGED_RESULTS_KEY, null);
  state.set(VALIDATION_ATTEMPTS_KEY, 0);

  console.log(
    `beforeAgentCallback: Agent ${context.agentName} has reset the session state for a new evaluation cycle.`,
  );

  return undefined;
};

The orchestrator sets the variables to null in the beforeAgentCallback. In the prepareEvaluationTool, the orchestrator only replaces the description with the new project description.

const prepareEvaluationTool = new FunctionTool({
  name: 'prepare_evaluation',
  description: 'Stores the new project description to prepare for a fresh evaluation.',
  parameters: z.object({
    description: z.string().describe('The validated project description from the user.'),
  }),
  execute: ({ description }, context) => {
    if (!context || !context.state) {
      return { status: 'ERROR', message: 'No session state found.' };
    }

    // Set the new description for the ProjectAgent to find
    context.state.set(PROJECT_DESCRIPTION_KEY, description);

    return { status: 'SUCCESS', message: 'Description updated.' };
  },
});

The tool logic is more efficient and the token usage is reduced.

export const rootAgent = new LlmAgent({
  name: 'ProjectEvaluationAgent',
  model: 'gemini-3.1-flash-lite-preview',
  beforeAgentCallback: resetNewEvaluationCallback,
  instruction: `
    ... LLM instruction ....
  `,
  tools: [prepareEvaluationTool],
  subAgents: [sequentialEvaluationAgent],
});

Dynamic State Management

Prerequisite: The subagent uses tool calling to perform action
Callback used: afterToolCallback

export const VALIDATION_ATTEMPTS_KEY = 'validation_attempts';

export const MAX_ITERATIONS = 3;

import { AfterToolCallback } from '@google/adk';
import { VALIDATION_ATTEMPTS_KEY } from '../output-keys.const.js';
import { MAX_ITERATIONS } from '../validation.const.js';

export function createAfterToolCallback(fatalErrorMessage: string, maxAttempts = MAX_ITERATIONS): AfterToolCallback {
  return ({ tool, context, response }) => {
    if (!tool || !context || !context.state) {
      return undefined;
    }

    const toolName = tool.name;
    const agentName = context.agentName;
    const state = context.state;

    if (!response || typeof response !== 'object' || !('status' in response)) {
      return undefined;
    }

    // [1] Dynamic state management
    const attempts = (state.get<number>(VALIDATION_ATTEMPTS_KEY) || 0) + 1;
    state.set(VALIDATION_ATTEMPTS_KEY, attempts);

    // [2] Response modification
    const status = response.status || 'ERROR';
    if (status === 'ERROR' && attempts >= maxAttempts) {
      context.actions.escalate = true;

      return {
        status: 'FATAL_ERROR',
        message: fatalErrorMessage,
      };
    }
  };
}

In both project and decision subagents, I invoke createAfterToolCallback to create their own afterToolCallback to increment the validation attempts.

const projectAfterToolCallback = createAfterToolCallback(
  `STOP processing immediately. Max validation attempts reached. Return the most accurate data found so far or empty strings if none.`,
);

const decisionAfterToolCallback = createAfterToolCallback(
  `STOP processing immediately and output the final JSON schema with verdict: "None".`,
);

Next, I call the projectAfterToolCallback in the afterToolCallback stage of the project subagent.

export function createProjectAgent(model: string) {
  const projectAgent = new LlmAgent({
    name: 'ProjectAgent',
    model,
    beforeAgentCallback: agentStartCallback,
    instruction: (context) => {
      ... LLM instruction ....
    },
    afterToolCallback: projectAfterToolCallback,
    afterAgentCallback: agentEndCallback,
    tools: [validateProjectTool],
    ... 
  });

  return projectAgent;
}

Next, I call the decisionAfterToolCallback in the afterToolCallback stage of the decision subagent.

export function createDecisionTreeAgent(model: string) {
  const decisionTreeAgent = new LlmAgent({
    name: 'DecisionTreeAgent',
    model,
    beforeAgentCallback: [resetAttemptsCallback, agentStartCallback],
    instruction: (context) => {
      ... instruction  of the LLM flow ...
    },
    afterToolCallback: decisionAfterToolCallback,
    afterAgentCallback: agentEndCallback,
    tools: [validateDecisionTool],
    ...
  });

  return decisionTreeAgent;
}

Request/Response Modification

Prerequisite: The subagent uses tool calling to perform action
Callback used: AfterToolCallback

The same AfterToolCallback also modifies the response when the number of validation attempts exceeds the maximum iterations.

Conditional Skipping of Steps

This is an important design pattern for avoiding unnecessary LLM executions.

Callback used: beforeModelCallback.

In my subagents, I run this callback to validate the session data. When a specific condition is met, the callback returns content immediately to skip the subsequent LLM flow.

Example 1

import { SingleBeforeModelCallback } from '@google/adk';

const beforeModelCallback: SingleBeforeModelCallback = ({ context }) => {
  const { project } = getEvaluationContext(context);
  const { isCompleted } = isProjectDetailsFilled(project);

  if (isCompleted) {
    return {
      content: {
        role: 'model',
        parts: [
          {
            text: JSON.stringify(project),
          },
        ],
      },
    };
  }

  return undefined;
};

Then, beforeModelCallback is hooked to the beforeModelCallback stage of the project subagent.

export function createProjectAgent(model: string) {
  const projectAgent = new LlmAgent({
    name: 'ProjectAgent',
    model,
    description:
      'Analyzes the user-provided project description to extract and structure its core components, including the primary task, underlying problem, ultimate goal, and architectural constraints.',
    beforeAgentCallback: agentStartCallback,
    beforeModelCallback,
    instruction: (context) => {
      const { projectDescription } = getEvaluationContext(context);
      if (!projectDescription) {
        return '';
      }

      return generateProjectBreakdownPrompt(projectDescription);
    },
    afterToolCallback: projectAfterToolCallback,
    afterAgentCallback: agentEndCallback,
    tools: [validateProjectTool],
    outputSchema: projectSchema,
    outputKey: PROJECT_KEY,
    disallowTransferToParent: true,
    disallowTransferToPeers: true,
  });

  return projectAgent;
}

Example 2

import { SingleBeforeModelCallback } from '@google/adk';

const beforeModelCallback: SingleBeforeModelCallback = ({ context }) => {
  const { decision } = getEvaluationContext(context);

  if (decision && decision.verdict !== 'None') {
    return {
      content: {
        role: 'model',
        parts: [
          {
            text: JSON.stringify(decision),
          },
        ],
      },
    };
  }

  const { project, antiPatterns } = getEvaluationContext(context);
  const { isCompleted } = isProjectDetailsFilled(project);

  if (isCompleted && antiPatterns) {
    return undefined;
  }

  return {
    content: {
      role: 'model',
      parts: [
        {
          text: JSON.stringify({
            verdict: 'None',
            nodes: [],
          }),
        },
      ],
    },
  };
};

Then, beforeModelCallback is hooked to the beforeModelCallback stage of the project subagent.

export function createDecisionTreeAgent(model: string) {
  const decisionTreeAgent = new LlmAgent({
    name: 'DecisionTreeAgent',
    model,
    beforeAgentCallback: [resetAttemptsCallback, agentStartCallback],
    beforeModelCallback,
    instruction: (context) => {
      ... instruction  of the LLM flow ...
    },
    afterToolCallback: decisionAfterToolCallback,
    afterAgentCallback: agentEndCallback,
    tools: [validateDecisionTool],
    outputSchema: decisionSchema,
    outputKey: DECISION_KEY,
    disallowTransferToParent: true,
    disallowTransferToPeers: true,
  });

  return decisionTreeAgent;
}

Example 3

function constructRecommendation(recommendation: string) {
  return {
    content: {
      role: 'model',
      parts: [
        {
          text: JSON.stringify({
            text: recommendation,
          }),
        },
      ],
    },
  };
}

const beforeModelCallback: SingleBeforeModelCallback = ({ context }) => {
  const { project, antiPatterns, decision } = getEvaluationContext(context);

  const { isCompleted } = isProjectDetailsFilled(project);
  const isDecisionNone = decision && decision.verdict === 'None';

  if ((isCompleted && antiPatterns && decision && decision.verdict !== 'None') || (!isCompleted && isDecisionNone)) {
    return undefined;
  } else if (isCompleted && isDecisionNone) {
    return constructRecommendation(
      '## Recommendation: Manual Review Required\n\n**Status:** Abnormal Case Detected\n\nThe provided project is complete and valid, but the decision tree could not reach a conclusive verdict (Result: `None`).\n\n**Possible Reasons:**\n- The requirements fall outside of known architectural patterns.\n- There are conflicting constraints and goals that cannot be resolved automatically.\n\n**Next Steps:**\n- Review and refine the constraints or goals.\n- Escalate for manual architectural review.',
    );
  }

  return constructRecommendation(
    '## Recommendation: Data Required\n\n**Status:** Abnormal Case Detected\n\nNo decision is reached.',
  );
};

Similar to the previous project and decision agents, the beforeModelCallback function is hooked to the beforeModelCallback stage of the recommendation agent.

export function createRecommendationAgent(model: string) {
  const recommendationAgent = new LlmAgent({
    name: 'RecommendationAgent',
    model,
    beforeModelCallback,
    beforeAgentCallback: agentStartCallback,
    instruction: (context) => {
      const { project, antiPatterns, decision } = getEvaluationContext(context);
      const { isCompleted, missingFields } = isProjectDetailsFilled(project);

      if (project) {
        if (!isCompleted && decision && decision.verdict === 'None') {
          console.log('RecommendationAgent -> generateFailedDecisionPrompt');
          return generateFailedDecisionPrompt(project, missingFields);
        } else if (isCompleted && antiPatterns && decision && decision.verdict !== 'None') {
          console.log('RecommendationAgent -> generateRecommendationPrompt');
          return generateRecommendationPrompt(project, antiPatterns, decision);
        }
      }
      return 'Skipping LLM due to missing data.';
    },
    afterAgentCallback: agentEndCallback,
    outputSchema: recommendationSchema,
    outputKey: RECOMMENDATION_KEY,
    disallowTransferToParent: true,
    disallowTransferToPeers: true,
  });

  return recommendationAgent;
}

These are the callback design patterns that the agent adopts. In the next section, I describe how to launch the agent to observe the logging messages in the terminal.

Environment Setup

services:
  mailhog:
    image: mailhog/mailhog
    container_name: mailhog
    ports:
      - '1025:1025' # SMTP port
      - '8025:8025' # HTTP (Web UI) port
    restart: always
    networks:
      - decision-tree-agent-network

networks:
  decision-tree-agent-network:

The SMTP server listens on port 1025, and the Web UI is accessible at http://localhost:8025.

docker compose up -d

Start MailHog in Docker.

Testing

Add scripts to package.json to build and start the ADK web interface.

  "scripts": {
    "prebuild": "rimraf dist",
    "build": "npx tsc --project tsconfig.json",
    "web": "npm run build && npx @google/adk-devtools web --host 127.0.0.1 dist/agent.js"
  },

Open a terminal and type npm run web to start the API server.
Open a new browser tab and type http://localhost:8000.
Paste the following text into the message box:

One of my favorite tech influencers just tweeted about a 'breakthrough in solid-state batteries.' Find which public company they might be referring to, check that company’s recent patent filings to see if it’s true, and then check their stock price to see if the market has already 'priced it in'.

Ensure the root agent executes and halts when the email agent terminates. The orchestrator and the project agent trigger the callback hooks to reset the session state, log performance metrics, validate session data, increment the validation attempts, and modify the response.

Similarly, both anti-patterns and decisions agents used the beforeAgentCallback and afterAgentCallback to log performance metrics. They also used the beforeModelCallback to validate session data before calling the LLM to generate a response. Moreover, the decision agent incremented the validation attempts and modify the status to FATAL_ERROR when the validation attempts exceeded or equal to maximum iterations in the afterToolCallback.

Similarly, both recommendation and merger agents used the beforeAgentCallback and afterAgentCallback to log performance metrics. The recommendation agent also used the beforeModelCallback to validate session data before calling the LLM to generate a recommendation.

Conclusion

The takeaway: follow the design patterns and best practices of callback at various stages of an agent to log performance metrics, reduce cost and latency, and modify response.

Resources

Migrating to Nano Banana 2: Enhancing Your Angular Firebase AI App

Connie Leung — Tue, 31 Mar 2026 23:17:59 +0000

Migrating to Nano Banana 2: Enhancing Your Angular Firebase AI App

In the rapidly evolving world of AI, staying up-to-date with the latest models is crucial for performance and feature accessibility. In this guide, we’ll walk through the process of migrating an Angular application using Firebase AI Logic to the new Nano Banana 2 model. We will also explore how to implement advanced features like "Thinking Levels," custom aspect ratios, and the use of experimental Angular Signal Forms.

Introduction

Nano Banana 2 (often associated with the Gemini 3.1 Flash series) brings significant improvements to image and content generation. Specifically, it introduces:

New aspect ratios (8:1 and 1:8).
High-resolution options (512px and up).
Thinking Levels: Allowing you to toggle between "Minimal" and "High" reasoning for complex generation tasks.

Let’s dive into the architecture and the code changes required to make this transition.

Architecture Overview

The application follows a modern serverless AI architecture, leveraging the Google Cloud and Firebase ecosystem.

Diagram Description

Angular Frontend: Handles the user interface, prompt input, and displays generated media. It uses Signals for reactive state management.
Firebase Remote Config: Acts as the central brain for model parameters, allowing you to swap model names or reasoning levels without redeploying code.
Firebase AI Logic (Vertex AI for Firebase SDK): The bridge between the frontend and the Gemini/Nano Banana models.
Media Generation: Images generated via Nano Banana 2 can be piped into other models, such as Veo 3.1, to generate high-quality video content.

Architectural Advice

For a robust AI application, avoid hardcoding model parameters in your TypeScript files. Use Firebase Remote Config to manage modelName, thinkingLevel, and location. This allows for A/B testing and instant updates if a newer model version becomes available.

Step 1: Update Firebase Remote Config

The first step is updating your configuration in the Firebase Console. You need to point the application to the new model and define the thinking levels.

Key Changes:

geminiImageModelName: Update to gemini-3.1-flash-image-preview.
thinkingLevel: Set to HIGH or MINIMAL.

Remote Config JSON (Snippet)

{
  "geminiImageModelName": "gemini-3.1-flash-image-preview",
  "thinkingLevel": "HIGH",
  "vertexAlLocation": "us-central1"
}

After updating the console, download the remotemconfig.json and sync it with your local project's default template.

Step 2: Angular Service Configuration

In the Angular app, you need to map the Remote Config values to the AI Logic generation parameters.

`firebase.provider.ts`

We update the generation config to include the thinkingLevel. Since this is an enum in the SDK, we map the string from Remote Config.

function getGenerativeAIImageModel(configService: ConfigService) {
  const { firebaseApp, remoteConfig } = configService.firebaseObjects;

  const modelName = getValue(remoteConfig, 'geminiImageModelName').asString();
  const rawThinkingLevel = getValue(remoteConfig, 'thinkingLevel').asString();

  // Map string to ThinkingLevel enum
  const thinkingLevel = rawThinkingLevel as keyof typeof ThinkingLevel;

  const modelParams: ModelParams = {
    model: modelName,
    generationConfig: {
      candidateCount: 1,
      thinkingConfig: {
        thinkingLevel: ThinkingLevel[thinkingLevel]
      }
    }
  };

  return getGenerativeModel(getAI(firebaseApp), modelParams);
}

Step 3: Prompt Workaround for Aspect Ratio & Resolution

Currently, some Firebase AI Logic SDKs might not support aspectRatio or resolution as first-class parameters in the generationConfig. A reliable workaround is appending these instructions directly to the prompt.

`prompt-form.component.ts`

onGenerateClick(event: Event) {
  event.preventDefault();

  const inputValue = this.promptForm.value.prompt;
  const aspectRatio = this.genConfigValues().aspectRatio;
  const resolution = this.genConfigValues().resolution;

  let trimmedPrompt = inputValue;

  // Workaround: Append instructions to the prompt
  if (aspectRatio) {
    trimmedPrompt = `${trimmedPrompt} \nApply this aspect ratio to the image: ${aspectRatio}`;
  }

  if (resolution) {
    trimmedPrompt = `${trimmedPrompt} \nApply this resolution to the image: ${resolution}`;
  }

  this.generate.emit({ trimmedPrompt, inputValue });
}

Step 4: UI with Experimental Signal Forms

To handle the dropdown selections for Aspect Ratio and Resolution, the app uses Angular's experimental Signal Forms. This provides a more reactive approach compared to standard Reactive Forms.

`generate-options-form.component.html`

<form [formRoot]="generateConfigForm" class="flex flex-col gap-4">
  <label>Aspect Ratio</label>
  <select [formField]="aspectRatio">
    @for (opt of aspectRatios; track opt) {
      <option [value]="opt">{{ opt }}</option>
    }
  </select>

  <label>Resolution</label>
  <select [formField]="resolution">
    @for (opt of resolutions; track opt) {
      <option [value]="opt">{{ opt }}</option>
    }
  </select>
</form>

Conclusion

Migrating to Nano Banana 2 allows developers to tap into higher-quality image generation and nuanced reasoning levels. By combining Angular Signals with Firebase Remote Config, you create a flexible architecture that can adapt to the next model update with minimal code changes.

Happy coding!

Resources

Stop Wasting Tokens: Building Deterministic Custom Agents with Google ADK [GDE]

Connie Leung — Sun, 29 Mar 2026 16:35:36 +0000

In the world of AI orchestration, it's tempting to use a Large Language Model (LLM) for every step of a workflow. However, as applications scale, the "LLM-first" approach can introduce unnecessary latency, costs, and unpredictability. The Google Agent Development Kit (ADK) provides a powerful alternative: the BaseAgent.

This post explores how to create a custom, programmatic agent—specifically an Email Agent—that handles deterministic tasks while still participating seamlessly in an AI-driven ecosystem.

Why is the Email Agent a Custom Agent?

The agent is 100% deterministic and relies on external APIs to format text and send emails.

It uses the marked library to convert a Markdown string into HTML and the nodemailer library to send mail to an SMTP server.

On the contrary, if I created a LlmAgent, the instruction, tool, and structured output would introduce LLM latency, input, and output tokens.

The Benefits of Programmatic Execution

Using a custom agent instead of an LLM-wrapped prompt offers several critical advantages:

Agent Type	Llm Agent	Custom Agent
Latency	Time-intensive (generation delay)	Low-latency (direct execution)
Cost	Pay for tokens consumed	Zero cost
Determinism	Probabilistic and hallucinate when agent does not follow the instruction or the instruction is unclear	100% deterministic and does not require AI capabilities
Testability	Incorrect result when the model hallucinates	Test results are predictable
Integration	Delegate responsibility to tools	Call libraries directly in the class

Demo Overview

The EmailAgent is the final subagent of the Decision Tree multi-agent system. The Decision Tree agent evaluates the feasibility of applying an agent architecture to a project. It starts with a project description, determines whether an agent architecture should be used, generates a recommendation, returns the recommendation to the client, and sends an email to the administrator as a side effect.

The EmailAgent retrieves the recommendation and summary from the session state, returns the recommendation text to the frontend, and sends both to the administrator's email address.

The agent does not require an LLM or tool use; therefore, an LlmAgent is overkill. My solution was to use a custom agent to handle email to save time and cost.

Architecture

The project, anti-patterns, decision, recommendation, and synthesis agents are LLM agents. These agents require Gemini to reason and generate text responses.

The Audit Trail, Cloud Storage, and Email agents integrate with external APIs or resources that trigger deterministic actions.

Prerequisites

Requirements

TypeScript 5.9.3 or later
Node.js 24.13.0 or later
npm 11.8.0 or later
Docker (For running MailHog locally)
Google ADK (For building the custom agent)
Gemini in Vertex AI (to call the model in LLM agents, although not required for the custom agent)

Note: I used Gemini in Vertex AI for authentication due to regional availability. Gemini is blocked in Hong Kong currently, so I used Gemini in Vertex AI instead.

Install npm dependencies

npm i --save-exact @google/adk
npm i --save-dev --save-exact @google/adk-devtools
npm i --save-exact nodemailer
npm i --save-dev --save-exact @types/nodemailer rimraf
npm i --save-exact marked
npm i --save-exact zod

I installed the required dependencies for building ADK agents, converting Markdown string to HTML and sending emails to MailHog in local testing.

Pinned dependencies ensure the versioning is the same in development and production environments for enterprise-level applications.

Environment variables

Copy .env.example to .env and fill in the credentials:

GEMINI_MODEL_NAME="gemini-3-flash-preview"
GOOGLE_CLOUD_PROJECT="<Google Cloud Project ID>"
GOOGLE_CLOUD_LOCATION="global" 
GOOGLE_GENAI_USE_VERTEXAI=TRUE

# SMTP Settings (MailHog)
SMTP_HOST="localhost"
SMTP_PORT=1025
SMTP_USER=""
SMTP_PASS=""
SMTP_FROM="no-reply@test.local"
ADMIN_EMAIL="admin@test.local"

SMTP_HOST, SMTP_PORT, SMTP_USER, SMTP_PASS, are required to set up MailHog for local email testing.

SMTP_FROM is a from email address that can be any string in local testing.

Environment Setup

I pulled the latest version of the MailHog docker image from Docker Hub and started it locally to receive test emails and display them in the Web UI. The docker-compose.yml file contains the setup configuration.

services:
  mailhog:
    image: mailhog/mailhog
    container_name: mailhog
    ports:
      - "1025:1025" # SMTP port
      - "8025:8025" # HTTP (Web UI) port
    restart: always
    networks:
      - decision-tree-agent-network

networks:
  decision-tree-agent-network:

The SMTP port is 1025 and the URL of the Web UI port is http://localhost:8025.

docker compose up -d

Start MailHog in Docker.

ADK Multi-Agent Architecture

process.loadEnvFile();

const model = process.env.GEMINI_MODEL_NAME || '';
if (!model) {
  throw new Error('GEMINI_MODEL_NAME is not set');
}

The model variable specifies the gemini-3-flash-preview model. If model is undefined, an error is thrown.

The demo uses Node 20+, so the process.loadEnvFile() is provided to load variables from the environment file. Otherwise, developers should consider using dotenv to load environment variables.

export const RECOMMENDATION_KEY = 'recommendation';
export const MERGED_RESULTS_KEY = 'mergedResults';
export const PROJECT_DESCRIPTION_KEY = 'project_description';

import { FunctionTool, LlmAgent, SequentialAgent } from '@google/adk';
import { z } from 'zod';
import { initWorkflowAgent } from './init.js';
import {
  MERGED_RESULTS_KEY,
  PROJECT_DESCRIPTION_KEY,
  VALIDATION_ATTEMPTS_KEY,
} from './sub-agents/output-keys.js';

const prepareEvaluationTool = new FunctionTool({
  name: 'prepare_evaluation',
  parameters: z.object({
    description: z.string()
  }),
  execute: async ({ description }, context) => {
    if (!context || !context.state) {
      return { status: 'ERROR', message: 'No session state found.' };
    }

    const state = context.state;

    // Clear all previous evaluation data
    state.set(MERGED_RESULTS_KEY, null);
    state.set(VALIDATION_ATTEMPTS_KEY, 0);

    // Set the new description 
    state.set(PROJECT_DESCRIPTION_KEY, description);

    return { status: 'SUCCESS', message: 'State reset and description updated.' };
  },
});

export const SequentialEvaluationAgent = new SequentialAgent({
  name: 'SequentialEvaluationAgent',
  subAgents: initWorkflowAgent(model),
});

prepareEvaluationTool is a tool that reset the variables in the session state before the agent starts the project evaluation.

SequentialEvaluationAgent is a sequential agent that consists of subagents.

export function initWorkflowAgent(model: string) {
  return [
    createMergerAgent(model),
    createEmailAgent(),
  ];
}

The initWorkflowAgent calls createMergerAgent and createEmailAgent factory functions to return the merger agent and the email agent. The merger agent is a LlmAgent because it requires Gemini to generate a summary whereas the email agent is a custom agent that does not require Gemini.

export const rootAgent = new LlmAgent({
  name: 'ProjectEvaluationAgent',
  model,
  instruction: `... instruction...`,
  tools: [prepareEvaluationTool],
  subAgents: [SequentialEvaluationAgent],
});

The rootAgent is an orchestrator that routes the project description to SequentialEvaluationAgent to evaluate.

Email Agent Factory Function

export type SmtpConfig = {
  host: string;
  port: number;
  user?: string;
  pass?: string;
  from: string;
  email: string;
};

export function createEmailAgent(): BaseAgent {
  const email = process.env.ADMIN_EMAIL || 'admin@test.local';
  const host = process.env.SMTP_HOST || 'localhost';
  const port = parseInt(process.env.SMTP_PORT || '1025');
  const user = process.env.SMTP_USER || '';
  const pass = process.env.SMTP_PASS || '';
  const from = process.env.SMTP_FROM || 'no-reply@test.local';

  const smtpConfig: SmtpConfig = {
    host,
    port,
    user,
    pass,
    from,
    email,
  };

  return new EmailAgent(smtpConfig);
}

The createEmailAgent function retrieves the host, port, user, password, sender email, and administrator email from the environment variables to construct an instance of SmtpConfig.

Pass the SmtpConfig object to the constructor of the EmailAgent class to construct a custom agent. Finally, the function returns the email agent to become the last subagent of SequentialEvaluationAgent.

Implementing the Email Agent

Let’s look at the core structure of the EmailAgent.

1. The Class Definition and State

The EmailAgent has a readonly SmtpConfig instance member that is initialized in the constructor. The constructor calls super to initialize its name and description.

class EmailAgent extends BaseAgent {
  readonly smtpConfig: SmtpConfig;

  constructor(smtpConfig: SmtpConfig) {
    super({
      name: 'EmailAgent',
      description: 'Send a recommendation and summary email to the administrator.',
    });

    this.smtpConfig = smtpConfig;
  }
}

2. Accessing the Context

import { z } from 'zod';

export const recommendationSchema = z.object({
  text: z.string(),
});

export type Recommendation = z.infer<typeof recommendationSchema>;

export const mergerSchema = z.object({
  summary: z.string(),
});

export type Merger = z.infer<typeof mergerSchema>;

The text property stores the recommendation of the Recommendation type. Similarly, the summary property of the Merger type stores the summary.

export function getEvaluationContext(context: ReadonlyContext | undefined) {
  if (!context || !context.state) {
    return {
      recommendation: null,
    };
  }

  const state = context.state;
  return {
    recommendation: state.get<Recommendation>(RECOMMENDATION_KEY) ?? null,
  };
}

export function getMergerContext(context: ReadonlyContext | undefined) {
  if (!context || !context.state) {
    return {
      merger: null,
    };
  }

  const state = context.state;
  return {
    merger: state.get<Merger>(MERGED_RESULTS_KEY) ?? null,
  };
}

getEvaluationContext and getMergerContext are helper functions to obtain recommendation and summary from the session state.

3. Implement the Concrete Methods

The EmailAgent extends the BaseAgent and all subclasses must implement two abstract methods: runAsyncImpl and runLiveImpl.

  protected async *runAsyncImpl(context: InvocationContext): AsyncGenerator<Event, void, void> {
    for await (const event of this.runLiveImpl(context)) {
      yield event;
    }
  }

Implement runAsyncImpl by wrapping runLiveImpl. It invokes a for-await loop to iterate this.runLiveImpl(context) and yields each event.

protected async *runLiveImpl(context: InvocationContext): AsyncGenerator<Event, void, void> {
    const readonlyCtx = new ReadonlyContext(context);
    const { merger } = getMergerContext(readonlyCtx);
    const { recommendation } = getEvaluationContext(readonlyCtx);
    const recommendationText = recommendation?.text || 'No recommendation available.';

    const emit = (status: 'success' | 'error', author: string) =>
      createEmailStatusEvent({
        author,
        context,
        status,
        recommendationText,
      });

    if (!merger) {
      yield emit('error', this.name);
      return;
    }

    try {
      const emailContent = `${recommendation?.text || ''}\n\n## Summary\n\n${merger.summary}`;
      await sendEmail(this.smtpConfig, 'Project Evaluation Results', emailContent);
      yield emit('success', this.name);
    } catch (e) {
      console.error(e);
      yield emit('error', this.name);
    }
  }

The runLiveImpl method handles three cases: an undefined merger, a successful email dispatch, and an email dispatch failure. When email dispatch is successful, the inner emit arrow function yields a success event. Otherwise, the catch block logs the error and yields an error event.

import { marked } from 'marked';
import nodemailer from 'nodemailer';

export async function sendEmail(smtpConfig: SmtpConfig, subject: string, text: string) {
  const { host, port, user, pass, from, email: to } = smtpConfig;
  const transporter = nodemailer.createTransport({
    host,
    port,
    auth: user && pass ? { user, pass } : undefined,
    secure: false,
  });

  const html = await marked.parse(text);
  const mailOptions = {
    from,
    to,
    subject,
    text,
    html,
  };
  return transporter.sendMail(mailOptions);
}

The sendEmail function uses the nodemailer to create a transporter using the SMTP host, port, user, and password. The marked library parses the Markdown text and converts it to HTML. The transporter has a sendMail method that accepts mail options to send an email to the to email address in both text and html formats.

4. Returning Data in the Output

Next, implement createEmailStatusEvent to return an email status event.

export type EmailStatusOptions = {
  author: string;
  context: InvocationContext;
  status: 'success' | 'error';
  recommendationText: string;
};

export function createEmailStatusEvent(options: EmailStatusOptions): Event {
  return createEvent({
    invocationId: options.context.invocationId,
    author: options.author,
    branch: options.context.branch || '',
    content: {
      role: 'model',
      parts: [
        {
          text: JSON.stringify({
            status: options.status,
            recommendationText: options.recommendationText,
            sessionId: options.context.session.id,
            invocationId: options.context.invocationId,
          }),
        },
      ],
    },
  });
}

The EmailStatusOptions provide the invocation ID, branch, author (the agent name), status, and recommendation text.

The createEmailStatusEvent function reuses the createEvent function to return the invocation ID, branch, author, and a JSON-stringified object consisting of the status, session ID, invocation ID, and recommendation text.

The client can examine the status to determine whether or not to display the recommendation text.

Returning both session and invocation IDs to the client is recommended. The client can reuse the session ID to call the agent in a conversational flow to seek evaluations for additional project descriptions. The invocation ID allows querying an error-tracking service like Datadog to identify the server status.

Testing

Add scripts to package.json to build and start the ADK web interface.

  "scripts": {
    "prebuild": "rimraf dist",
    "build": "npx tsc --project tsconfig.json",
    "web": "npm run build && npx @google/adk-devtools web --host 127.0.0.1 dist/agent.js"
  },

Open a terminal and type npm run web to start the API server.
Open a new browser tab and type http://localhost:8000.
Paste the following into the message box:

One of my favorite tech influencers just tweeted about a 'breakthrough in solid-state batteries.' Find which public company they might be referring to, check that company’s recent patent filings to see if it’s true, and then check their stock price to see if the market has already 'priced it in'.

Ensure the root agent executes and halts when the email agent terminates.

Open another tab and navigate to http://localhost:8025 to open the MailHog web UI

The MailHog Web UI displays the recommendation and summary that summarizes the decision, any clear anti-patterns, and the URL of the cloud storage.

Conclusion

Custom agents are essential components of an ADK application. While ADK offers sequential, parallel, loop, and LLM agents, these agents may not meet the requirements in some occasions. BaseAgent is the generic Agent class that enables developers to write their logic and design their agentic workflow.

The takeaway: avoid using an LLM-based agent for tasks that do not require probabilistic reasoning, tool calling, or response generation.