[Agent Platform in Action (6)] Towards Production-Grade AI Agent Ops: Building Full-Stack Observable LINE Agents with agents-cli

Recap: The Five-Stage Evolution from Local Prototype to Cloud Production-Grade AI Agent

Welcome to the sixth (and final) article in the "Agent Platform in Action" series! Looking back at the past five articles, we have taken LINE AI Card Assistant (kkdai/line-card-assistant) through the four major stages of evolution from a local prototype to an enterprise-grade AI Agent architecture:

1. [Agent Platform in Action (1)] Building LINE AI Card Assistant with Go and Google ADK 2.0 Graph Workflow: Implemented a business card workflow graph using google.golang.org/adk/v2/workflow covering Vision OCR, AddFanOut / AddFanIn parallel background research, and NewRequestInputEvent HITL interaction loops.
2. [Agent Platform in Action (2)] Moving from Monolithic Cloud Run to Google Cloud Agent Runtime: Refactoring LINE AI Card Assistant with agents-cli and ADK Go: Completed Phase 1 (PR #10), standardizing the project into an agents-cli (adk_go) dual-channel architecture, supporting LINE Webhook, ADK WebUI Playground, and A2A protocols.
3. [Agent Platform in Action (3)] Solving HITL Amnesia in Stateless Containers: Implementing Cross-Instance Card Recognition and Pause/Resume with ADK 2.0 GCS Artifact Service and Agent Platform Sessions: Completed the first half of Phase 2 (PR #11), introducing artifact/gcsartifact and session/vertexai to solve cross-instance HITL recovery and image storage issues in stateless containers.
4. [Agent Platform in Action (4)] Towards Cloud Dual-Layer Decoupling and Full Persistence: Completing the LINE AI Card Assistant Architecture with Cloud Run LINE Webhook Gateway, Agent Runtime, and Cloud SQL PostgreSQL: Completed the second half of Phase 2 (Issue #5) and Phase 3 (Issue #6, PR #12), migrating the card library to Cloud SQL PostgreSQL (PostgresStore) and building a stateless Cloud Run LINE Webhook Gateway (internal/gateway).
5. [Agent Platform in Action (5)] Building the AI Agent Quality Flywheel: Establishing Automated Evaluation Benchmarks for LINE AI Card Assistant with agents-cli eval and ADK 2.0 Go: Completed Phase 4 (Issue #7, PR #13), using agents-cli eval to establish an automated quality flywheel covering 6 golden cases and 4 major evaluation metrics, all scoring 1.0000.

In this final chapter today, we will complete the last piece of the puzzle for Roadmap #9 — Phase 5 (Issue #8): Official Agent Runtime Deployment, Terraform IaC, GitHub Actions CI/CD, and Full-Stack Cloud Trace Observability!

---

I. What Problem Does Phase 5 Solve? Moving from "Manual gcloud Commands" to "Declarative AgentOps"

In many AI projects, even if the code and evaluation tests are written, production environments often become operational nightmares due to the following four issues:

1. Infrastructure relies on manual commands (ClickOps / Shell Script drift): If Vertex AI Reasoning Engine, Service Account IAM permissions, GCS Buckets, Secret Manager secrets, and Cloud SQL are all created manually via gcloud, configuration inconsistencies easily occur between Staging and Production environments.
2. PR merges lack an AI quality gatekeeper (No Eval Gate in CI): Running go test can catch syntax and logic errors, but if someone accidentally breaks a Graph Node's state_delta or Prompt, how can we automatically intercept it before the PR is merged into main?
3. Cross-environment deployment and high-concurrency verification (Staging $\rightarrow$ Load Test $\rightarrow$ Prod Approval): After pushing to main, how can we use keyless Workload Identity Federation (WIF) to automatically deploy to Staging, run high-concurrency pressure tests (Vegeta Load Test), archive the reports to GCS, and finally push to Production only after manual approval (GitHub Environment Approval Gate)?
4. Graph Workflow node-level observability (Node-Level Distributed Tracing): When processing a business card from upload to Flex Message preview takes 2.8 seconds, is the time spent in VisionOCRNode or in the parallel Fan-Out's CompanyEnrichNode (Google Search Grounding)? We need every node of the ADK 2.0 Graph Workflow to expand into a clear nested Span tree in Google Cloud Trace, and automatically import it into BigQuery for P95/P99 latency and Token cost analysis!

This is exactly what the Phase 5 Full-Stack Architecture Diagram at the beginning of the article depicts through four core pillars:

• Top-left (GitHub Actions CI/CD & Quality Gate): .github/workflows/pr_checks.yaml, staging.yaml, deploy-to-prod.yaml
• Bottom-left (Terraform Infrastructure as Code): deployment/terraform/single-project/, cicd/, shared/
• Center (Vertex AI Agent Runtime + OpenTelemetry Node Spans): agents-cli deploy + internal/workflow/telemetry.go
• Right (Full-Stack Observability & Analytics): observability.go $\rightarrow$ Cloud Trace + Cloud Logging $\rightarrow$ telemetry.tf BigQuery Sink

---

II. Step 1: Generate Terraform IaC and CI/CD Scaffolding with agents-cli scaffold enhance

In Phase 1, we introduced the Prototype-First philosophy of agents-cli: keep the project lightweight first, and then execute agents-cli scaffold enhance to layer on production infrastructure when ready for the cloud.

We execute the following directly in the project root directory:

uvx google-agents-cli@1.10.0 scaffold enhance . \
  --cicd-runner github_actions \
  --region asia-east1 \
  --yes

agents-cli automatically detected our agents-cli-manifest.yaml (template: adk_go, deployment_target: agent_runtime) and, without breaking any existing app/, internal/, eval/, or sessions/ code, added three major modules to the project:

1. deployment/terraform/single-project/ and deployment/terraform/cicd/: Complete Terraform declarative infrastructure modules.
2. deployment/terraform/shared/telemetry.tf and storage.tf: Cloud Logging linked to Buckets, Datasets, and BigQuery telemetry analysis pipelines.
3. .github/workflows/ and e2e/: GitHub Actions CI/CD pipelines and A2A / AppInfo / Vegeta Load Test end-to-end test suites.

Next, we perform customized enhancements for the specific requirements of LINE AI Card Assistant (LINE credentials, Cloud SQL PostgreSQL permissions, Taiwan asia-east1 region)!

---

III. Step 2: Deep Dive into Terraform IaC (Vertex AI Reasoning Engine + Secret Manager + BigQuery Telemetry Sink)

Let's open deployment/terraform/ and see how Terraform declaratively builds the entire Google Cloud Agent Platform:

1. Declaratively Defining Vertex AI Reasoning Engine (service.tf)

In deployment/terraform/single-project/service.tf, we use the google_vertex_ai_reasoning_engine resource from the google-beta provider to package the Go source code and declare it as a managed Agent Runtime instance. We also set lifecycle { ignore_changes = [spec[0].source_code_spec] } so that daily code updates are handled by the CI/CD's agents-cli deploy for fast rolling deployment, while infrastructure settings (IAM, Buckets, environment variables) are unified under Terraform control:

resource "google_vertex_ai_reasoning_engine" "app" {
  provider = google-beta
  display_name = var.project_name
  project = var.project_id
  region = var.region

  labels = {
    "agent-framework" = var.agent_framework
  }

  spec {
    identity_type = "AGENT_IDENTITY"
    agent_framework = var.agent_framework

    source_code_spec {
      inline_source {
        source_archive = data.archive_file.source.output_path
      }
      developer_connect_source {}
    }

    container_spec {
      image_uri = "us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-15:latest"
    }
  }

  lifecycle {
    ignore_changes = [
      spec[0].source_code_spec,
      spec[0].container_spec,
    ]
  }
}

2. Least Privilege IAM and Secret Manager Integration (iam.tf & variables.tf)

To prevent LINE_CHANNEL_SECRET, LINE_CHANNEL_ACCESS_TOKEN, and DATABASE_URL from appearing in plain text in configuration files, we declare Google Cloud Secret Manager secrets in deployment/terraform/single-project/iam.tf and variables.tf. We grant roles/secretmanager.secretAccessor and roles/cloudsql.client to both the Vertex AI Service Agent (gcp-sa-aiplatform-re) and the Agent Identity Principal (principal://${google_vertex_ai_reasoning_engine.app.spec[0].effective_identity}):

# Secret Manager secrets for LINE Messaging API credentials and Cloud SQL DSN
locals {
  line_card_secrets = [
    "line-channel-secret",
    "line-channel-access-token",
    "line-card-database-url",
  ]
}

resource "google_secret_manager_secret" "app_secrets" {
  for_each = toset(local.line_card_secrets)
  project = var.project_id
  secret_id = each.value

  replication {
    auto {}
  }

  depends_on = [resource.google_project_service.services]
}

This way, during deployment, you only need to pass --secrets="LINE_CHANNEL_SECRET=line-channel-secret,LINE_CHANNEL_ACCESS_TOKEN=line-channel-access-token" to agents-cli deploy, and the Agent Runtime will securely mount the secrets from Secret Manager when starting the container!

3. Automatically Creating Cloud Logging $\rightarrow$ BigQuery Telemetry Analysis Dataset (telemetry.tf)

In deployment/terraform/shared/telemetry.tf, Terraform automatically creates a Cloud Logging Bucket with a 365-day retention period, enables enable_analytics = true, and directly maps telemetry logs to a BigQuery Dataset (telemetry_logs_dataset) via google_logging_linked_dataset:

resource "google_logging_project_bucket_config" "telemetry_logs_bucket" {
  project = each.value
  location = "global"
  bucket_id = "${var.project_name}-telemetry"
  retention_days = 365
  enable_analytics = true
}

resource "google_logging_linked_dataset" "telemetry_logs_dataset" {
  bucket = google_logging_project_bucket_config.telemetry_logs_bucket[each.key].bucket_id
  link_id = replace("${var.project_name}_telemetry", "-", "_")
  project = each.value
  location = "global"
}

This means all structured telemetry data output by OpenTelemetry can be analyzed directly in BigQuery using standard SQL without writing additional ETL scripts!

---

IV. Step 3: Injecting OpenTelemetry Distributed Tracing into the 14 Nodes of ADK 2.0 Go Graph Workflow (internal/workflow/telemetry.go)

In the observability.go generated in Phase 1, the system was already configured with an OTLP HTTP Exporter pointing to telemetry.googleapis.com and Google ADC automatic authentication. However, by default, we only see top-level traces for the entire HTTP request or LLM call, but not the execution time and routing decisions of each FunctionNode within the ADK 2.0 Graph Workflow.

To present the processing of each business card as a complete waterfall span tree on Google Cloud Trace, we added internal/workflow/telemetry.go:

package workflow

import (
    "context"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/codes"
    "go.opentelemetry.io/otel/trace"
)

const workflowTracerName = "github.com/kkdai/line-card-assistant/internal/workflow"

const (
    AttrWorkflowNode = attribute.Key("adk.workflow.node")
    AttrWorkflowRoute = attribute.Key("adk.workflow.route")
    AttrInvocationID = attribute.Key("adk.invocation.id")
    AttrSessionID = attribute.Key("adk.session.id")
    AttrUserID = attribute.Key("adk.user.id")
    AttrCardID = attribute.Key("adk.card.id")
    AttrCardCompany = attribute.Key("adk.card.company")
    AttrCardDuplicate = attribute.Key("adk.card.is_duplicate")
    AttrHITLInterrupt = attribute.Key("adk.hitl.interrupt_id")
    AttrHITLAction = attribute.Key("adk.hitl.action")
    AttrSearchMatches = attribute.Key("adk.search.matches")
    AttrSearchKeywords = attribute.Key("adk.search.keywords")
)

func startNodeSpan(ctx context.Context, nodeName string, attrs ...attribute.KeyValue) (context.Context, trace.Span) {
    baseAttrs := make([]attribute.KeyValue, 0, len(attrs)+1)
    baseAttrs = append(baseAttrs, AttrWorkflowNode.String(nodeName))
    baseAttrs = append(baseAttrs, attrs...)
    return otel.Tracer(workflowTracerName).Start(
        ctx,
        "workflow.node."+nodeName,
        trace.WithAttributes(baseAttrs...),
    )
}

Next, we inject startNodeSpan and endNodeSpan into all nodes in internal/workflow/nodes_ingest.go and internal/workflow/nodes_search.go. For example, in HybridSearchNode:

func newHybridSearchNode(cardStore store.CardStore) *workflow.FunctionNode {
    return workflow.NewFunctionNode(
        NodeHybridSearch,
        func(ctx agent.Context, _ any) (ev *session.Event, err error) {
            spanCtx, span := startNodeSpan(ctx, NodeHybridSearch, AttrInvocationID.String(ctx.InvocationID()))
            defer func() {
                routeAttr := ""
                if ev != nil && len(ev.Routes) > 0 {
                    routeAttr = ev.Routes[0]
                }
                endNodeSpan(span, err, AttrWorkflowRoute.String(routeAttr))
            }()
            // ... execute cardStore.HybridSearch(spanCtx, filter) ...
            span.SetAttributes(AttrSearchMatches.Int(len(cards)))
            // ...
        },
        workflow.NodeConfig{},
    )
}

When a user uploads a business card or performs a hybrid search, Cloud Trace automatically draws a hierarchical Span tree like the one below, including the time spent in the three child nodes of the parallel Fan-Out and the business attributes of each node:

flowchart TD
    Root["invoke_workflow (line_card_assistant)<br/>trace_id: 4bf92f3577b34da6a3ce929d0e0e4736"] --> R["workflow.node.RouterNode<br/>adk.workflow.route = 'ingest' (1.2ms)"]
    R --> V["workflow.node.VisionOCRNode<br/>adk.card.company = 'Taiwan AI Labs' (640ms)"]
    V --> C1["workflow.node.CleanAndDedupNode<br/>adk.card.is_duplicate = false (8.5ms)"]
    V --> C2["workflow.node.CompanyEnrichNode<br/>GoogleSearch Grounding (780ms)"]
    V --> C3["workflow.node.ContextMatchNode<br/>inferMetContext (310ms)"]
    C1 & C2 & C3 --> B["workflow.node.BuildPreviewNode<br/>Flex Bubble JSON (2.1ms)"]
    B --> H["workflow.node.HITLConfirmNode<br/>adk.hitl.interrupt_id = 'hitl_confirm_1'<br/>adk.hitl.action = 'pause' (0.8ms)"]

We also wrote automated unit tests in internal/workflow/telemetry_test.go using the OpenTelemetry SDK's tracetest.NewInMemoryExporter(), ensuring that "card recognition + HITL modification loop + confirmation save" and "hybrid search with 0 results automatically degrading to fuzzy search" 100% generate the expected Span nodes and attribute tags!

---

V. Step 4: Building a GitHub Actions CI/CD Pipeline Integrated with agents-cli eval Quality Gatekeeper (.github/workflows/)

With Terraform IaC and OpenTelemetry tracing in place, we officially embedded the agents-cli eval quality flywheel created in Phase 4 into the GitHub Actions CI/CD pipeline, forming a two-stage automated delivery:

flowchart LR
    subgraph PR["1. Pull Request Quality Gate (pr_checks.yaml)"]
        P1["go test -race -v ./...<br/>(Unit + OTel Span + Trajectory Tests)"] --> P2["go test -tags=e2e<br/>(A2A + /app-info E2E Tests)"]
        P2 --> P3["agents-cli eval run<br/>(4 major metrics must = 1.0000)"]
    end

    subgraph CD["2. Staging & Production CD (staging.yaml -> deploy-to-prod.yaml)"]
        S1["WIF Keyless Auth<br/>(google-github-actions/auth)"] --> S2["agents-cli deploy<br/>Deploy to Staging Agent Runtime"]
        S2 --> S3["Vegeta Load Test<br/>30s pressure test + report upload to GCS"]
        S3 --> S4["GitHub Environment<br/>Production Manual Approval Gate"]
        S4 --> S5["agents-cli deploy<br/>Deploy to Production Agent Runtime"]
    end

    PR -->|Merge to main| CD

1. PR Quality Gatekeeper (.github/workflows/pr_checks.yaml)

In .github/workflows/pr_checks.yaml, whenever a PR modifies app/**, internal/**, cmd/**, sessions/**, or eval/**, GitHub Actions will execute in order:

1. go test -race -v ./...: Verify all Go unit tests, cross-instance HITL recovery, OpenTelemetry Node Spans, and trajectory tests.
2. go test -v -tags=e2e ./e2e/integration/...: Start a real ADK Universal Launcher subprocess to verify /.well-known/agent-card.json, /a2a/invoke, and /apps/app/app-info endpoints.
3. agents-cli eval run Quality Gatekeeper: Execute 6 golden evaluation cases, verifying that the four major metrics (workflow_trajectory_accuracy, ocr_and_enrichment_quality, duplicate_and_hitl_precision, search_and_email_grounding) must maintain a perfect score!

2. Staging Automated Deployment and Vegeta Load Testing (.github/workflows/staging.yaml)

After the PR is merged into the main branch, .github/workflows/staging.yaml will:

1. Obtain short-term GCP credentials keylessly via Workload Identity Federation (WIF).
2. Call uvx google-agents-cli@1.10.0 deploy to deploy the latest commit to the Staging Vertex AI Agent Runtime, while mounting LINE_CHANNEL_SECRET and LINE_CHANNEL_ACCESS_TOKEN from Secret Manager and tagging OTEL_RESOURCE_ATTRIBUTES=service.version=$.
3. Start e2e/load_test/load_test.go (based on tsenart/vegeta/v12) to perform a 30-second continuous pressure test on the Staging Agent Runtime, checking HTTP 200 success rate, P50/P95/P99 latency, and 429 Too Many Requests ratio, and automatically archiving the results to gs://${LOGS_BUCKET_NAME_STAGING}/load-test-results/.
4. After the pressure test passes, automatically trigger .github/workflows/deploy-to-prod.yaml. After passing the Required Reviewers audit for the GitHub production environment, it is officially pushed to Production!

---

VI. Practical Verification: agents-cli deploy --dry-run and Full E2E Tests All Passed!

Before submitting the code as a PR, we fully executed three major verifications locally:

1. agents-cli deploy --dry-run Cloud Deployment Preflight

$ uvx google-agents-cli@1.10.0 deploy \
    --project line-vertex \
    --region asia-east1 \
    --dry-run \
    --no-confirm-project

Preflight output results:

• aiplatform.googleapis.com API: Enabled ✅
• Target Agent Runtime Engine: line-card-assistant (asia-east1) ✅
• IAM Roles & Service Account Preflight: Passed ✅

2. Go Unit Tests + OpenTelemetry Span Tests + E2E Integration Tests

$ go test -v -tags=e2e ./...

Includes the newly added TestWorkflowOpenTelemetrySpans_IngestAndHITL, TestWorkflowOpenTelemetrySpans_SearchFallback, and all tests under e2e/integration such as TestA2AAgentCard, TestA2AMessageSend, TestA2AErrorHandling, and TestAppInfo — all PASS!

3. Phase 4 agents-cli eval Quality Flywheel Regression Verification

$ DRY_RUN=true make eval

The 6 golden cases across the 4 custom metrics (workflow_trajectory_accuracy, ocr_and_enrichment_quality, duplicate_and_hitl_precision, search_and_email_grounding) still maintain 1.0000 (100% Pass)!

---

VII. "Agent Platform in Action" Series Summary: A Complete Five-Stage Evolution Map

With the completion of Phase 5 (Issue #8), the five stages planned in Roadmap Issue #9 have been 100% achieved and closed! Let's conclude with a summary table reviewing this journey from 0 to 1 with Google ADK 2.0 Go + Google Cloud Agent Platform:

Stage	Corresponding Issue / PR	Core Technical Breakthroughs	Corresponding Blog Post
Prototype	Initial Commit	Go adk/v2/workflow 12-node Graph Workflow, parallel AddFanOut/AddFanIn, NewRequestInputEvent HITL loop	In Action (1)
Phase 1	Issue #4 / PR #10	agents-cli (adk_go) standardization, app.NewRootAgent dual-channel factory, ADK WebUI Playground, and A2A protocol support	In Action (2)
Phase 2 (Part 1)	Issue #5 / PR #11	artifact/gcsartifact zero-state image bloat, session/vertexai cross-instance stateless HITL pause and resume	In Action (3)
Phase 2 (Part 2) & Phase 3	Issue #5, #6 / PR #12	Cloud SQL PostgreSQL (PostgresStore) card persistence, Cloud Run LINE Webhook Gateway (internal/gateway) and Agent Runtime dual-layer decoupling	In Action (4)
Phase 4	Issue #7 / PR #13	agents-cli eval quality flywheel, /app-info reflection introspection, 6 golden cases, and setNodeTrace trajectory repair (all 4 metrics at 1.0000)	In Action (5)
Phase 5	Issue #8 / PR #14	agents-cli deploy, Terraform IaC (deployment/terraform/), GitHub Actions CI/CD + WIF + Vegeta pressure testing, OpenTelemetry node-level Cloud Trace & BigQuery observation	This article (In Action 6)

The complete open-source code, Terraform modules, and evaluation datasets have been synchronized to the GitHub Repo: kkdai/line-card-assistant. If you are also building enterprise-grade AI Agents with Go and Google ADK 2.0, I hope this series serves as your best practical blueprint for moving from local development all the way to the Google Cloud Agent Platform production environment!