DEV Community: Evan Lin

[AI in Action] Rewriting a Daily LINE Bot from Python to Go 1.26 on Cloud Run: 1/4 Resource Usage, Google ADK v1.9.0, and Zero-D

Evan Lin — Tue, 03 Nov 2026 13:29:47 +0000

Background

I have a primary LINE Bot that I use heavily every day: kkdai/linebot-helper-python.

Initially, it was just a small tool where you could "drop a URL and get a Traditional Chinese summary," but over the past year and a half, I've gradually integrated various new Gemini capabilities into it:

Multi-source Content Loading and Document Conversion: A three-layer web crawler Fallback Chain (HTTP $\rightarrow$ Firecrawl $\rightarrow$ SingleFile Headless Chromium), YouTube video summaries, automatic conversion of PDF / Word / PPT / Excel / EPUB to Markdown, and one-click generation of FB / LinkedIn / Threads viral social media copy Carousels.
Bookmarks and Grounding Deep Research Reports: /save, /list, and /search bookmark management. Clicking "📄 Detailed Research Report" on a card combines Google Search Grounding to write a complete Markdown report, rendered as an independent HTML page (/reports/{id}) and permanently saved in Cloud Firestore.
Google Maps Grounding and Gemini Batch API Gourmet Big Data Analysis: Send LINE location coordinates to search for nearby gas stations, parking lots, and restaurants. It can also pull 50+ latest reviews for specific restaurants and feed them into the Gemini Batch API (enjoying a 50% cost discount) for deep analysis of signature dishes and pitfalls, pushed back to LINE via Webhook or background polling.
Agentic Vision Visual Annotation: After uploading an image, gemini-3-flash-preview writes Python code to draw bounding boxes, which are then saved as images and pushed back to the chat room.
LINE Voice Message TTS and LIFF Real-time Two-way Voice Assistant: Voice messages are automatically transcribed and paired with gemini-3.1-flash-tts-preview + ffmpeg to generate M4A voice readings. Opening LIFF allows two-way streaming of gemini-3.1-flash-live-preview via WebSocket, supporting hands-free VAD, Barge-in interruption, and Live Tool Calling.

The features are great, but the cost is obvious: the resident memory for Python + FastAPI + a large number of document parsing and AI packages kept increasing. To ensure background tasks and multi-concurrency didn't lag, this Cloud Run service was unknowingly scaled up to 4 vCPU / 4 GiB Memory (with --no-cpu-throttling enabled).

As someone who has been writing Go for a long time, seeing a personal daily assistant consume 4 vCPU / 4 GiB was becoming hard to stomach. So, I decided to do one thing: rewrite the entire linebot-helper-python 100% into the latest Go 1.26 version (kkdai/linebot-helper-go), deploy it to Cloud Run in the same GCP project, and ensure it shares Firestore data with the old version for zero-downtime switching at any time.

1. Setting Rules Before Rewriting: Breaking Down 11 Milestones with GitHub Issues

Moving a nearly 90KB main.py plus a dozen modules from Python to Go can easily result in an unmaintainable maze if boundaries aren't defined and testing isn't done along the way.

Therefore, after creating kkdai/linebot-helper-go, the first task wasn't writing code, but mapping out all features from the Python version. I opened Tracking Issue #12 and 11 sub-task Issues (#1 ~ #11) on GitHub, with each Issue corresponding to an independent package and complete unit tests:

Milestone Issue	Corresponding Go Package	Core Responsibilities and Compatibility Highlights
#1 Project Skeleton and Dockerfile	`main.go`, `internal/config`	Go 1.26 `net/http`, 100% environment variable compatibility with Python version, Multi-stage Dockerfile (integrating Chromium / `single-file` / `ffmpeg`)
#2 Gemini Dual-channel Client	`internal/gemini`	Supports both Vertex AI (ADC) and Gemini Developer API (`GOOGLE_AI_API_KEY`), covering Grounding, Code Execution, and TTS
#3 Firestore Persistence Layer	`internal/firestore`, `internal/services`	Fully compatible with Python version's `chat_sessions`, `bookmarks`, `reports`, and `batch_jobs` collection schemas; auto-downgrades to in-memory storage without credentials
#4 LINE Messaging API and Flex	`internal/line`	HMAC-SHA256 signature verification, Reply / Push / Loading Animation, Social copy and Bookmark Flex Carousel assembly
#5 URL Crawler and Multi-format Doc Conversion	`internal/loader`	HTTP (including PTT `over18=1`) $\rightarrow$ Firecrawl $\rightarrow$ SingleFile three-layer time-budget Fallback, and PDF / DOCX / PPTX / XLSX / EPUB / ODF to Markdown
#6 AI Toolset	`internal/ai`	Summaries, viral social copy, Grounding research reports, YouTube analysis, Agentic Vision, GitHub Issues aggregation, Maps search
#7 ADK Orchestrator and Specialized Agents	`internal/agents`	Integrates official `google.golang.org/adk v1.9.0`, establishing a Root `Orchestrator` and 5 specialized Agents: `Chat` / `Content` / `Location` / `Vision` / `GitHub`
#8 Asynchronous Webhook and Postback	`internal/handlers`	Webhook returns `200 OK` instantly, background Goroutines handle events, 9 types of Postback dispatching, and `/reports/{id}`, `/images/{id}`, `/audio/{id}` endpoints
#9 Gemini Batch API Gourmet Analysis	`internal/services/batch_service.go`	JSONL batch job upload, Static HMAC Webhook, Dynamic RS256 JWKS Webhook verification, and 60s background polling
#10 LIFF Real-time Voice Assistant	`internal/voice`, `static/liff`	`/ws/voice/{session_id}` two-way WebSocket Relay, Push-to-Talk / hands-free VAD, Barge-in, Session Resumption, and Live Tool Calling
#11 Cloud Run Deployment and Verification	Cloud Run (`us-central1`)	Deployed using existing Python version environment variables, stress tested, and container resource specs downsized

The overall architecture is shown in the diagram below:

flowchart TB
    subgraph Client["LINE Client and LIFF"]
        LINE["LINE App Chatroom\n(Text / URL / Image / Voice / Location / Postback)"]
        LIFF["LIFF Real-time Voice Assistant\n(/liff/ + WebSocket PCM 16kHz/24kHz)"]
    end

    subgraph CloudRun["Google Cloud Run: linebot-helper-go (Go 1.26, 1 vCPU / 1 GiB)"]
        Mux["Go 1.26 net/http Server\n(HMAC Verification + Immediate 200 OK)"]
        VoiceRelay["internal/voice\nGemini Live BidiGenerateContent Relay"]
        Orch["internal/agents (google.golang.org/adk v1.9.0)\nRoot Orchestrator + 5 Sub-Agents"]
        Loader["internal/loader\nHTTP -> Firecrawl -> SingleFile\n+ PDF/Office/EPUB to Markdown"]
        BatchSvc["internal/services/batch_service\nGemini Batch API + JWKS/HMAC Webhook + 60s Poller"]
    end

    subgraph GCP["Google Cloud & AI Backend"]
        Vertex["Vertex AI / Gemini Developer API\n(Search Grounding / Maps / Vision / TTS / Live)"]
        FS[("Cloud Firestore\nchat_sessions | bookmarks\nreports | batch_jobs")]
    end

    LINE -->|"POST /"| Mux
    LIFF <-->|"GET /ws/voice/{id}"| VoiceRelay
    Mux --> Orch
    Mux --> Loader
    Mux --> BatchSvc
    VoiceRelay <-->|"WebSocket"| Vertex
    VoiceRelay -.->|"delegate_to_orchestrator"| Orch
    Orch --> Vertex
    Orch --> Loader
    Orch <--> FS
    BatchSvc <--> Vertex
    BatchSvc <--> FS

2. Five Engineering Designs Particularly Noteworthy During the Rewriting Process

1. Dual-track Gemini Client for Vertex AI and Developer API

In this LINE Bot, not all features use the same Gemini endpoint:

General Chat, Grounding Search, Google Maps Grounding, Image Analysis: Uses Vertex AI (using Application Default Credentials of the Cloud Run service account, no API Key required).
Native TTS Voice Synthesis (gemini-3.1-flash-tts-preview), Gemini Batch API, and Gemini Live API WebSocket: Uses Gemini Developer API (using GOOGLE_AI_API_KEY).
Agentic Vision (gemini-3-flash-preview + Code Execution): Prioritizes Developer API, with automatic Fallback to Vertex AI if no Key is set.

In the Go version's internal/gemini/client.go, we converged these two paths under the same gemini.Client. Even converting the Raw PCM (24kHz, 16-bit, mono) returned by TTS into LINE-supported audio/mp4 (.m4a) via ffmpeg and calculating audio milliseconds is handled seamlessly within the same module.

2. 100% Firestore Schema Compatibility with the Python Version (Key to Zero-Downtime Switching)

The biggest fear when rewriting a service is that "after switching, user chat history, half-saved bookmarks, or previously generated research report URLs will all become invalid."

To prevent this, we strictly aligned the 4 Firestore Collection names and field structures in internal/firestore and internal/services with the Python version:

chat_sessions: Stores multi-turn chat history and the last GPS coordinates sent by the user (last_location).
bookmarks: Stores /save bookmarks, viral social copy candidate bookmarks (status: candidate $\rightarrow$ saved), and 7-day cached report_id and report_generated_at.
reports: Stores the fully rendered HTML of Grounding deep research reports (supporting permanent retrieval via /reports/{report_id}).
batch_jobs: Stores the mapping between Gemini Batch API task IDs and LINE user_id.

This means the Python version and Go version can simultaneously read and write to the same Firestore online. The moment the LINE Webhook is pointed to the new version, all old bookmarks and report links remain completely unaffected; if I ever want to switch back to the Python version, I only need to change one line of the Webhook URL.

3. Native Go Implementation for Multi-format Document to Markdown Conversion (Saying Goodbye to Heavy Python Packages)

The Python version used packages like anydoc to convert PDF, DOCX, PPTX, XLSX, ODT, EPUB, CSV, and RTF to Markdown before feeding them to Gemini for summarization. These packages in Python often pull in a long list of C extensions and dependencies.

In the Go version internal/loader/doc_converter.go, since .docx, .pptx, .xlsx, .odt, and .epub are essentially ZIP-packaged XML files, we used the Go standard library's archive/zip + encoding/xml along with a lightweight PDF text extractor. With less than a few hundred lines of pure Go code, we can cleanly convert all mainstream office documents and e-book formats into structured Markdown. It compiles fast and has almost zero extra memory overhead at runtime!

3. Facing the Architecture Honestly: Python Version's "Conceptual ADK" vs. Go Version's Actual Implementation of `google.golang.org/adk v1.9.0`

While checking the code, we found an interesting historical trace:

If you open the beginning of the original Python version's agents/orchestrator.py, you'll see a comment saying:

Orchestrator Agent - Main control Agent based on Google ADK

But if you look at the Python version's requirements.txt and the entire source code, you'll find that google-adk isn't actually installed, and there isn't a single line of from google.adk import ...! The Python version at that time merely "borrowed the hierarchical concept of ADK," manually creating Orchestrator, ChatAgent, ContentAgent, etc., using pure Python classes, with the underlying layer calling google-genai directly.

For this Go rewrite, we decided to do it for real—directly introducing the latest official Google google.golang.org/adk v1.9.0 in go.mod (paired with google.golang.org/genai v1.57.0):

Comparison Item	Original Python Version (`linebot-helper-python`)	New Go Version (`linebot-helper-go`)
ADK Package Dependency	❌ None (not installed)	✅ `google.golang.org/adk v1.9.0`
Agent Definition	Custom Python Class	Uses `llmagent.New(llmagent.Config{...})` to create `agent.Agent`
Sub-Agents Hierarchy	Manual object attributes	Root `orchestrator` mounts 5 specialized sub-agents via `SubAgents: []agent.Agent{...}`
Tools Binding	Manual method calls	Uses `functiontool.New()` to register type-safe Go function tools (`load_url_content`, `summarize_youtube_video`, `search_nearby_places`)
Model Interface	Direct call to `google.genai.Client`	Implements `google.golang.org/adk/model.LLM` and `adkgemini.NewModel` (Vertex AI Backend)

Next Step: From "Building an ADK Agent Tree" to "Fully Driven by ADK Runner" (Issue #13)

To ensure the first migration version passes all existing test cases 100%, ProcessText currently retains the original fast intent routing. However, we have already planned the following 5 exciting ADK upgrades in Issue #13 ([RFC] Google ADK (v1.9.0) Deep Integration and Architecture Evolution Proposal):

Replace Hardcoded Regex Routing with runner.Runner + SubAgents / agenttool: When a user enters a composite command in LINE (e.g., "Help me find the restaurant mentioned in this article and save it to bookmarks"), ADK will automatically transfer or call multiple Sub-Agents in parallel.
Add BookmarkAgent and Research Report functiontool: Users won't need to remember /save or /search slash commands; they can simply say "Find the bookmark about MCP I saved last time and make a research report" to trigger it.
Implement adk/session.Service for Cloud Firestore: Wrap the existing firestore.KVStore into a standard session.Service, allowing ADK Runner to natively manage multi-turn conversation events and states.
Introduce workflowagents (sequentialagent / parallelagent): Deconstruct deep research reports into a sequential workflow of "Extraction $\rightarrow$ Google Search Verification $\rightarrow$ Report Writing"; change viral copy for FB / LinkedIn / Threads to parallel Sub-Agents for simultaneous output.
Connect LIFF Real-time Voice Assistant's delegate_to_orchestrator directly to ADK Runner.

4. Cloud Run Deployment and Slimming Down: Machine Specs Cut Directly to 1/4!

After completing all 10 core packages and unit tests (go vet ./... && go test ./... all passed), we deployed linebot-helper-go directly to Cloud Run.

Let's look at the container resource configuration comparison between the Python and Go versions on the same GCP project:

Cloud Run Service	Runtime Environment	CPU	Memory	Concurrency	CPU Allocation Mode	Relative Resource Cost
`linebot-helper-python`	Python 3.11 + FastAPI	4 vCPU	4 GiB	80	Always allocated (`--no-cpu-throttling`)	`100%` (Baseline)
`linebot-helper-go`	Go 1.26 Native Compilation	1 vCPU	1 GiB	160	Always allocated (`--no-cpu-throttling`)	`25%` (75% Savings)

You might ask: The resident memory of the Go compiled executable is only 20–30 MB, why not just use 512 MiB instead of 1 vCPU / 1 GiB?

Both numbers have practical engineering considerations behind them:

Why 1 vCPU? Because the LINE Bot needs to return 200 OK instantly upon receiving a Webhook and continue running Gemini generation and research report rendering in background Goroutines. Plus, there's a background Ticker polling the Gemini Batch API every 60 seconds. Therefore, --no-cpu-throttling (CPU always allocated) must be enabled. In Cloud Run specifications, the minimum CPU threshold for enabling --no-cpu-throttling is 1 vCPU.
Why 1 GiB Memory? The Go program itself is indeed under 30 MB, but don't forget our 3rd layer Fallback for the URL crawler has single-file + Headless Chromium installed in the container (used to deal with dynamic rendering sites that even Firecrawl blocks), plus voice message synthesis requires calling ffmpeg. When Headless Chromium is occasionally awakened, it instantly consumes 300–400 MB of memory. Keeping 1 GiB ensures Chromium never OOMs (Out of Memory) upon startup while cutting machine costs to 1/4 of the original Python version!

The deployment command is very clean:

gcloud run deploy linebot-helper-go \
  --source . \
  --project <YOUR_GCP_PROJECT_ID> \
  --region us-central1 \
  --cpu 1 \
  --memory 1Gi \
  --allow-unauthenticated \
  --no-cpu-throttling

5. Seven Devilish Details Most Likely to be Tripped Over or Missed in Cloud Deployment

Moving a LINE Bot from local testing to Cloud Run, or switching from an old Python service to a new Go service, "successful compilation" often only accounts for 60% of the work. The real traps are usually hidden in Cloud Run's runtime lifecycle, network proxies, container sandboxing, and external endpoint settings.

Here are the 7 key blind spots most easily overlooked during this deployment process:

Omission 1: Forgot to add `--no-cpu-throttling`, or passed `r.Context()` into a background Goroutine

LINE Webhook specifications require the server to return 200 OK quickly after receiving an event. Our heavy tasks (such as Grounding deep research reports taking 1–2 minutes, Agentic Vision writing code to draw, voice transcription plus TTS, and the Gemini Batch API Poller polling every 60 seconds) all return 200 OK first, then run go func() in the background and use PushMessage to push the result.

There are two traps here that both beginners and experts are extremely likely to fall into:

Cloud Run "freezes the CPU" by default the moment the HTTP Response is sent: If you don't add --no-cpu-throttling, as soon as handleWebhookCallback returns 200 OK, Cloud Run will drop the container CPU to near 0 within tens of milliseconds. Your background Goroutine will be stuck in mid-air like it was paused, only moving a few steps when the next HTTP request comes in minutes later. The user will never receive the push!
Background Goroutines must never use r.Context(): In Go's net/http, r.Context() is triggered as Canceled the moment the HTTP Handler ends (sends 200 OK)! If you casually write go s.handleMessageEvent(r.Context(), event), background requests calling Vertex AI or Firestore will die directly with a context canceled error.

// ❌ Incorrect: r.Context() will be canceled the moment 200 OK is returned; background tasks will fail.
go s.handleMessageEvent(r.Context(), eventCopy)

// ✅ Correct: Detach from the HTTP Request lifecycle, use an independent Background Context (set Timeout internally).
go s.handleMessageEvent(context.Background(), eventCopy)

Omission 2: Vertex AI `GOOGLE_CLOUD_LOCATION=global` Domain Concatenation Trap

When reusing old environment variables, we saw GOOGLE_CLOUD_LOCATION=global (because many Gemini 3.x Preview models in Vertex AI are under the global location).

If you write a Vertex AI REST Client in Go and follow the habit for general regions by using fmt.Sprintf("https://%s-aiplatform.googleapis.com", location), when location is "global", the resulting URL becomes:

❌ https://global-aiplatform.googleapis.com (This domain doesn't exist in GCP; DNS lookup will fail!)

In Vertex AI specifications, only regional endpoints (like us-central1, asia-east1) have a prefix; when the Location is global, you must switch to the prefix-less https://aiplatform.googleapis.com:

func (c *Client) vertexEndpointBase() string {
    if c.VertexBaseURL != "" {
        return strings.TrimRight(c.VertexBaseURL, "/")
    }
    if c.Location == "" || c.Location == "global" {
        return "https://aiplatform.googleapis.com"
    }
    return fmt.Sprintf("https://%s-aiplatform.googleapis.com", c.Location)
}

Omission 3: Moving Environment Variables Using `--set-env-vars` Leading to Character Truncation or Secret Leakage

When moving over a dozen environment variables from linebot-helper-python to linebot-helper-go, never manually string together --set-env-vars="KEY1=VAL1,KEY2=VAL2...":

If any Token or Prompt string contains a comma , or equals sign =, gcloud's parameter parsing will split at the wrong position or even error out.
Putting environment variables directly in the command line parameters leaves the entire Secret string in the terminal history or conversation logs.

The safest way: Directly export the temporary /tmp/env.yaml locally using gcloud run services describe --format=json, feed it in during deployment using --env-vars-file=/tmp/env.yaml, and rm -f /tmp/env.yaml immediately after deployment. This way, no Secrets are printed, and all special characters are 100% preserved.

Omission 4: Ignoring Cloud Run Reverse Proxy's `X-Forwarded-Proto`, Leading to LINE Rejecting Images and Audio

This LINE Bot needs to dynamically assemble "public URLs pointing to its own server" in four places:

Research report page: https://<host>/reports/{report_id}
Agentic Vision annotated image: https://<host>/images/{image_id}
TTS voice synthesis audio file: https://<host>/audio/{audio_id}
Gemini Batch API Dynamic Webhook: https://<host>/api/gemini-callback/dynamic

Note: LINE Messaging API's ImageMessage and AudioMessage strictly require URLs to be https://. In the Cloud Run architecture, HTTPS TLS is terminated at the outer Google Front End (GFE), so r.TLS is nil when it reaches the Go http.Request inside the container. If you don't read the X-Forwarded-Proto and X-Forwarded-Host headers, the assembled http:// URL will be rejected by the LINE Push API!

Omission 5: Pursuing Minimalist Go Images (`scratch` / `distroless`) but Forgetting External Executables and Chromium Sandbox Restrictions

When writing Go services, we often use FROM gcr.io/distroless/static to compress the Docker Image to under 20MB. But doing this for this multi-functional LINE Bot will immediately hit a wall:

Missing ffmpeg: After receiving PCM from Gemini for TTS synthesis, calling exec.Command("ffmpeg", ...) to convert to .m4a will report executable file not found.
Missing Headless Chromium, Chinese Fonts, and Container Sandbox Flags: The 3rd layer Fallback of the crawler uses single-file to drive Headless Chromium. In Cloud Run's container sandbox environment, if you don't add --no-sandbox, --disable-gpu, and --disable-dev-shm-usage (because the container's default /dev/shm shared memory is small) to Chromium, or if the base image doesn't have fonts-noto-cjk installed (resulting in garbled text on captured Chinese web pages) and ca-certificates (HTTPS certificate verification failure), the crawler will silently crash in the cloud.

Therefore, our Dockerfile uses golang:1.26-bookworm for compilation, and the Runtime stage uses debian:bookworm-slim with explicit installation of ca-certificates, ffmpeg, chromium, fonts-noto-cjk, and @single-file/cli.

Omission 6: State Splitting Between In-Memory Caching and Cloud Run Auto-scaling (Multi-Instance)

Although we have persisted the four core states—chat history, bookmarks, research reports, and Batch Job mappings—to Cloud Firestore, some binary data with a "lifecycle of only 5 minutes" is stored in a Go in-memory map:

Images just uploaded by users, waiting for the "Identify Image / Agentic Vision" Quick Reply selection (imageTempStore).
PNGs just drawn with bounding boxes, waiting for the LINE server to pull them (/images/{image_id}).
M4A audio files just synthesized, waiting for the LINE server to pull them (/audio/{audio_id}).

This works perfectly under a single instance, but once Cloud Run scales to 2 or more instances due to a burst in traffic, state splitting occurs:

User sends an image to Instance A (image stored in A's memory).
3 seconds later, the user clicks the "Agentic Vision" button, and the Webhook hits Instance B $\rightarrow$ B can't find the image and replies "⚠️ Image expired, please resend!"
Or Instance A generates /audio/{id} and calls the LINE Reply API, but when the LINE media server comes to download the audio, it hits Instance B, gets a 404, and the user only sees an unplayable voice bubble in the chat room.

Solution: Since a single Go instance can easily handle high concurrency (we set containerConcurrency to 160), for personal or small-to-medium assistants, it's recommended to limit --max-instances to 1 (if multi-instance high availability is needed in the future, the caching for /images and /audio must be rewritten to Cloud Storage or Firestore).

Omission 7: Only Changing the Messaging API Webhook During Switching, but Missing Firebase Hosting WebSocket Restrictions and LIFF Endpoint

When preparing to switch from the old service to the new one, the following three peripheral settings are most commonly missed:

Checkpoint	Why is it easily missed?	Actual Impact and Correct Practice
1. Is the existing Webhook hidden behind Firebase Hosting?	When querying existing settings via LINE API (`GET /v2/bot/channel/webhook/endpoint`), we found the original Webhook pointed to `*.firebaseapp.com` (rewritten to Cloud Run by `firebase.json`).	Firebase Hosting does not support WebSocket proxying! If the LIFF voice assistant (`/ws/voice/{id}`) goes through the Firebase Hosting domain, the WebSocket handshake will fail. Therefore, when switching, point directly to the Cloud Run URL or ensure LIFF uses a direct connection.
2. LIFF Endpoint URL of the LINE Login Channel	LIFF App settings are not in the Messaging API Channel but in the LINE Login Channel tab under the same Provider.	If you only change the Webhook and not the LIFF Endpoint, the chat room text will be handled by the new Go version, but opening the voice assistant will still connect back to the old Python version.
3. Gemini Batch API Static Webhook / Eventarc	If Eventarc or a project-level static Webhook is set in GCP to point to `/api/gemini-callback/static`.	Remember to update the target URL as well (while the Dynamic Webhook `/api/gemini-callback/dynamic` will automatically capture the Host of the incoming request, requiring no manual modification).

6. Inventory Before Switching: Identifying Three "Zombie Endpoints" from GCP Logs and Planning to Incorporate `n8n` Schedules!

While writing the "Switching from Python Version" steps in the README, I originally listed:

"Change external schedules calling /hn, /hf, and /urls to point to the new service."

But then I thought: Wait, are these three endpoints actually still being called by my n8n?

We immediately used gcloud logging read to pull execution records for the linebot-helper-python and n8n services in the GCP project, and found a big surprise:

/hn, /hf, and /urls were endpoints written in the first version in November 2024, intended for external schedules to POST URL arrays for the LINE Bot to summarize and push.
But the n8n service currently in the cloud (running four active workflows: "RSS to Sheet", "Daily Tasks (Fetch Papers/Share Videos)", "Daily LINE Sender", and "AI Trends Automated Newsletter") had already been changed to run Firecrawl internally in n8n, call Gemini for summaries, assemble Flex JSON, and directly hit https://api.line.me/v2/bot/message/push!
In other words, the number of calls to /hn, /hf, and /urls on linebot-helper-python from n8n over the past few months was 0!

Even more interestingly, while checking n8n's Debug Logs, we accidentally caught n8n failing to push YouTube new video summaries at 6 AM because of word count or thumbnail format issues in the Flex Carousel assembled by the Javascript node, with the LINE API returning:

Workflow execution finished with error: A message (messages[0]) in the request body is invalid

This discovery led us to make two immediate decisions:

Decision 1: Immediately remove `/hn`, `/hf`, and `/urls` from `linebot-helper-go`

Since no external services are calling them, keeping them on the public Cloud Run only adds three attack surfaces without signature protection. After removing them from server.go and redeploying, hitting /hn, /hf, or /urls online returns 404 Not Found. When switching, I only need to change the Webhook URL and LIFF Endpoint URL in the LINE Developers Console!

Decision 2: Open Issue #14 ~ Issue #19 to incorporate `n8n` workflows into `linebot-helper-go`!

Since linebot-helper-go is already a high-speed Go server always resident at 1 vCPU / 1 GiB (--no-cpu-throttling), and it has a built-in three-layer crawler loader.Loader, strongly-typed LINE Flex Builder, Gemini Batch API, Firestore BookmarkService, and ReportStore, why maintain a separate n8n + Postgres database?

More importantly—if these daily schedules are all moved to linebot-helper-go and pushed through the same LINE Official Account, a perfect closed loop of "Push $\rightarrow$ Interaction" can be achieved:

Issue #14 (Batch Task 1/5): Establish a protected POST /api/tasks/{task_name} schedule endpoint and a Firestore feed_items deduplication queue (replacing the old method of comparing rows in Google Sheets), unifying pushes using the main official account.
Issue #15 (Batch Task 2/5): Fetch Hugging Face Daily Papers every day at 10:00. When pushing paper summary cards, directly attach "🔖 Save Bookmark" and "📄 Detailed Research Report" buttons, allowing one-click generation of Grounding deep web reports for good papers.
Issue #16 (Batch Task 3/5): Inspect YouTube subscription channel new videos every day at 06:00. Use Go's strongly-typed Flex Builder to completely solve n8n's 400 Invalid Request Body issue, and attach a "📝 Detailed Summary" Postback button directly to the card.
Issue #17 (Batch Task 4/5): Integrate multi-source RSS inspection from HN, Hatena IT, Google Dev Blog, LY Blog, Steam, etc., with daily 06:00 featured pushes.
Issue #18 (Batch Task 5/5): Aggregate weekly featured content and bookmarks every Friday at 17:00, rendering them directly into a beautiful /reports/{id} web version AI Trends Weekly pushed to LINE.

7. One Last Security Reminder: Don't Leave Real Cloud Run URLs in Public Git History

After writing the deployment records and switching tutorial, I did one very important cleanup task: replaced all occurrences of .run.app real service URLs, GCP Project Numbers, and LIFF IDs in README.md and GitHub Issue comments with <YOUR_CLOUD_RUN_SERVICE_URL> placeholders, and rewrote the Git Commit history to eliminate traces.

The reason is simple: The LINE Bot's Cloud Run service must be set to --allow-unauthenticated to receive Webhooks from the LINE platform. Although our POST / has strict X-Line-Signature verification (requests without a Channel Secret signature are blocked with a 400), writing the real .run.app URL directly in a public GitHub Repo's README still easily attracts unnecessary automated crawler scans and traffic noise.

Sharing architecture and implementation with public code while keeping real endpoints only in your own configuration backend is the most reassuring way to maintain a personal AI assistant in the long run.

8. Cloud Run Deployment and Migration Self-Checklist

If you also plan to rewrite an existing Python LINE Bot into Go and move it to Cloud Run, you can use this checklist to tick off items before switching traffic:

1. CPU Throttling Setting: As long as there are background Goroutines (Webhook returns 200 OK followed by background push, or background periodic Ticker), be sure to add --no-cpu-throttling (and set CPU to at least 1).
2. Context Lifecycle Decoupling: Background Goroutines are prohibited from passing in r.Context(); always use context.Background() with an independent context.WithTimeout.
3. Vertex AI global Domain Judgment: If GOOGLE_CLOUD_LOCATION=global, confirm the Endpoint Base URL is https://aiplatform.googleapis.com instead of global-aiplatform.googleapis.com.
4. External Binary Tools and Fonts: If ffmpeg, single-file, or Headless Chromium are called inside the container, confirm the final stage Docker Image includes the corresponding executables, ca-certificates, fonts-noto-cjk, and Chromium's --no-sandbox --disable-dev-shm-usage parameters.
5. Reverse Proxy HTTPS Headers: When dynamically generating /images/{id}, /audio/{id}, and /reports/{id} URLs, prioritize reading X-Forwarded-Proto and X-Forwarded-Host to ensure media URLs sent to LINE are always https://.
6. In-Memory Caching and Instance Count: If there are short-lived memory maps (like cached images/audio), it's recommended to set --max-instances=1 and increase concurrency, or switch to GCS / Firestore to avoid state splitting across multiple instances.
7. Dual-channel LINE Setting Synchronization: In addition to updating the Messaging API Webhook URL, don't forget to update the LIFF Endpoint URL of the LINE Login Channel under the same Provider, and confirm the WebSocket path is not intercepted by a CDN / Firebase Hosting that doesn't support WS.
8. Clean Up Obsolete Endpoints and Public Information: Confirm useless old endpoints (like /hn, /hf, /urls) from Cloud Logging and remove them, and ensure the public Repo and Git history do not contain real .run.app URLs and keys.

Summary and Open Source Project Links

From this rewriting journey from kkdai/linebot-helper-python to kkdai/linebot-helper-go, here are the most valuable takeaways:

Open Issues Before Writing Code: 11 milestone Issues made the large-scale language rewrite highly controllable, with each module independently passing unit tests before being layered.
Shared Data Layer is the Foundation for Zero-Downtime Migration: As long as the Firestore Collection Schemas are aligned, services in the two languages can be switched painlessly at any time.
Go's Resource Efficiency on Cloud Run is Extremely High: Even with Headless Chromium and ffmpeg mounted, switching from Python's 4 vCPU / 4 GiB to Go's 1 vCPU / 1 GiB cut the bill to 25%, with even faster response times.
Check Cloud Logging Before Switching: Real traffic logs don't lie; you can see at a glance which endpoints are already retired and which external n8n workflows are actually better suited to be incorporated into the same service.

The complete source code and subsequent ADK v1.9.0 / Batch Tasks evolution plans are all open-sourced on GitHub: kkdai/linebot-helper-go. Feel free to refer to it and exchange ideas!

[AI in Practice] Rewriting a LINE Research Bot from Python Managed Agents to Go ADK v1.9.0: From Sandbox Files to Session State,

Evan Lin — Tue, 03 Nov 2026 13:28:58 +0000

Recap

In the previous post "Comprehensive Analysis of Managed Agents API", I created an open-source LINE Research Planner Bot: kkdai/line-research-bot.

How it works is simple:

Send " Research the selection of SOTA open-source vector databases " to the LINE Bot. The Bot will sequentially run through three stages: PLAN → SEARCH_COMPARE → WRITE_REPORT. First, it designs 4–8 sets of search keywords, then uses Google Search + URL Context to read real web pages and cross-compare "consensus / disagreement / gaps," and finally writes a Traditional Chinese Markdown report with footnote citations.
The report is rendered into HTML by Cloud Run and uploaded to Cloud Storage. LINE receives a Flex card with a summary, main sources, and a "Read Full Report" button.
If you later send " Go deeper into Chapter 2, add Japanese sources ", the Bot will deepen the same report, increment the version number by +1, and automatically save the old version as a snapshot.

The first version used Python 3.11 + FastAPI + google-genai (enterprise=True) to call the Pre-GA Managed Agents API. It relied on the official sandbox's persistent filesystem: /workspace/plan.json, /workspace/sources.json, and /workspace/report.md survived across turns under the same environment_id, allowing the Agent to read the files itself when deepening the research.

It was fun, but after running it for a while, a few things kept me uneasy:

Pre-GA, no SLA: Names like base_agent: antigravity-preview-05-2026 clearly indicate they will change one day.
gsutil in the sandbox is mocked (Pitfall #2 from the previous post). As a result, after writing the report, the Agent still had to stuff the entire Markdown back into a JSON for Cloud Run to upload. The value of the sandbox filesystem was essentially reduced to "remembering three files across turns."
Must use background=True + manual polling loop: If Stage 2/3 was slow, the timeout had to be adjusted (I once tuned it from 240 seconds all the way to 600 seconds).

So this time, I decided: Rewrite the entire service in Go, use Google's official ADK for Go (google.golang.org/adk v1.9.0) for everything Agent-related, and deploy it back to the original Cloud Run service using the exact same environment variables. I recorded the entire process in the original repo's Issue #1.

After completing the rewrite, I separated the Go version into a new repo so that the Python version cited in the previous article remains unchanged. The two versions can be directly compared:

Repo	Content
`kkdai/line-research-bot`	Original Python 3.11 + FastAPI + Managed Agents version (example from the previous article), and the Issue #1 record of this migration
`kkdai/line-research-bot-go`	The protagonist of this post: Go 1.27 + ADK for Go v1.9.0, currently running on Cloud Run

I. Clarification: It wasn't actually "Python ADK" before

Many people (including myself initially) mix up Managed Agents and ADK, but they are two different things:

Comparison Item	Original Python Version	New Go Version
Agent Execution Location	Google-managed sandbox (Managed Agents API)	Within your own Cloud Run container (ADK Runner)
SDK	`google-genai` (`enterprise=True`), not using `google-adk`	`google.golang.org/adk v1.9.0` + `google.golang.org/genai`
Agent Definition	`deploy/create-agent.sh` uses curl to call REST API to create a `research-planner` resource	In-code `llmagent.New(llmagent.Config{...})` × 3
Tools	`code_execution`, `filesystem`, `google_search`, `url_context`	`geminitool.GoogleSearch{}` + `url_context`
Cross-turn State	Sandbox filesystem (`environment_id`)	ADK Session State + Firestore `line_bot_workspaces`
Structured Output	Prompt says "Return ONLY the JSON" + manual code fence stripping	`OutputSchema` (ADK automatically injects `set_model_response`)
Invocation Method	`interactions.create(background=True)` + polling every 3s	`runner.Run()` returns Go 1.23+ iterator, synchronous event consumption

So this time, rather than "Python ADK → Go ADK," it's more accurate to say: Replacing the hosted Managed Agent with a self-controlled ADK Agent. The benefit is that you no longer have to worry about mocked gsutil in the sandbox, background polling modes, or Pre-GA uncertainty.

II. Setting Rules Before Rewriting: One Issue, Feasibility Verification First

This project is much smaller than the previous linebot-helper (Python version ~1,400 lines, Go version ~2,000 lines, excluding tests), so instead of splitting it into 11 issues, I opened one Tracking Issue #1. I wrote a checklist covering "Current Status Audit → Design → Feasibility Verification → Rewriting → Testing → Deployment → Verification," commenting on the issue as each section was completed.

Before writing a single line of Go, I did two things:

gcloud run services describe to pull the online service configuration (Service Account, 11 environment variables, Secret Manager bindings, memory, timeout) to ensure a 1:1 migration.
Wrote a 30-line probe program to confirm if ADK for Go on Vertex AI could enable " Google Search + URL Context + OutputSchema all at once ". This was the key to whether the design would work; the results are discussed in Section IV.

Once confirmed feasible, the Go version (module path github.com/kkdai/line-research-bot-go) was split into the following modules:

cmd/server/ main: Dependency injection, HTTP server, graceful shutdown
internal/research/ ADK for Go: Three llmagents + runner + session state
  prompts/ plan / search_compare / write_report (go:embed)
internal/worker/ PLAN → SEARCH_COMPARE → WRITE_REPORT → publish, deepen, retry
internal/server/ /webhook, /tasks/run-research, /readyz
internal/state/ Firestore (users / reports compatible + new workspaces)
internal/publisher/ goldmark (GFM + footnote) → HTML → GCS (including snapshots)
internal/linebot/ line-bot-sdk-go v8 + HMAC verification
internal/tasks/ Cloud Tasks (OIDC)
internal/intent, flex/ Intent classification, Flex cards

The overall data flow is as follows:

flowchart LR
    LINE["LINE App"] -->|"POST /webhook<br/>(HMAC)"| WH["internal/server<br/>Intent classification + Locking"]
    WH -->|"enqueue"| CT[("Cloud Tasks<br/>research-jobs")]
    CT -->|"POST /tasks/run-research<br/>(OIDC token)"| WK["internal/worker"]
    WK -->|"seed / save"| FS[("Firestore<br/>line_bot_workspaces")]
    WK -->|"runner.Run()"| ADK["ADK for Go<br/>plan / search_compare / write_report"]
    ADK -->|"Google Search + URL Context"| VX["Vertex AI<br/>gemini-3.8-flash"]
    WK -->|"goldmark → HTML"| GCS[("Cloud Storage")]
    WK -->|"Push Progress + Flex Card"| LINE

III. From Managed Agents to ADK for Go: Four Core Rewrites

1. From "REST Agent Resource Creation" to "In-code `llmagent.New`"

In the Python version, the Agent was created via REST API using a shell script. The system instructions for the three stages were concatenated into one large block, relying on a stage tag (PLAN / SEARCH_COMPARE / WRITE_REPORT) in the first line of every input for the Agent to find the correct section:

{
  "id": "research-planner",
  "base_agent": "antigravity-preview-05-2026",
  "tools": [{"code_execution": {}}, {"filesystem": {}}, {"google_search": {}}, {"url_context": {}}],
  "system_instruction": "...concatenated instructions for three stages..."
}

The Go version uses three distinct llmagent instances (internal/research/agents.go). Prompts are stored in internal/research/prompts/ using go:embed, and tools are only provided to the stages that need them (the planning stage doesn't need internet access):

func urlContextTool() tool.Tool {
    return geminitool.New("url_context", "Reads the content of web pages by URL.",
        &genai.Tool{URLContext: &genai.URLContext{}})
}

ag, err := llmagent.New(llmagent.Config{
    Name: "research_planner_search_compare",
    Description: "Searches the web and cross-compares sources.",
    Model: llm, // gemini.NewModel(ctx, cfg.GeminiModel, &genai.ClientConfig{Backend: genai.BackendVertexAI, ...})
    Instruction: searchPrompt,
    Tools: []tool.Tool{geminitool.GoogleSearch{}, urlContextTool()},
    OutputSchema: searchSchema,
    OutputKey: "search_output",
})
r, err := runner.New(runner.Config{AppName: appName, Agent: ag, SessionService: sessions})

ADK for Go currently only has geminitool.GoogleSearch built-in. URL Context must be wrapped manually using geminitool.New with a genai.Tool, which I discovered by digging through the tool/geminitool source code. Additionally, I kept the AGENT_ID environment variable but repurposed it as the ADK AppName and agent name prefix, so the online environment variables didn't need to change.

2. From `background=True` + Polling to `runner.Run()` Iterator

The most tedious part of the Python version was this: Managed Agents didn't support synchronous mode, so you had to create and then get every 3 seconds until completed:

created = self._client.interactions.create(
    agent=self._agent_id, input=instruction, environment=env_arg,
    stream=False, background=True, store=True,
)
while time.monotonic() < deadline:
    polled = self._client.interactions.get(created.id, include_input=False)
    if polled.status == "completed":
        return AgentResponse(...)
    time.sleep(self._poll_interval)

ADK for Go's runner.Run() directly returns a Go 1.23 iter.Seq2[*session.Event, error]. The Agent runs in its own process, events flow out one by one, and timeout is handled by context.WithTimeout:

for ev, err := range st.runner.Run(ctx, userID, sid,
    genai.NewContentFromText(input, genai.RoleUser), agent.RunConfig{}) {
    if err != nil {
        return "", invocationID, fmt.Errorf("%w: %s: %v", ErrAgent, st.name, err)
    }
    invocationID = ev.InvocationID
    // Use final response text as fallback; read formal result from OutputKey
}

I changed the last_interaction_id stored in Firestore to ADK's InvocationID. The field name remains the same, making it just as useful for log cross-referencing.

3. Sandbox `/workspace` Files → ADK Session State + Firestore (The Most Critical Step)

The biggest selling point of Managed Agents was the cross-turn persistent sandbox filesystem. ADK has no sandbox, so when a user asks to "go deeper into Chapter 2," where does the Agent find the previous sources and report?

The answer is ADK Session State + Instruction Templates. ADK's Instruction replaces {key} with the value from session state, and {key?} means "it's okay if it doesn't exist":

## Cross-compared sources (JSON)
{sources}

## Current report Markdown (empty for a new report)
{report_md?}

The mapping becomes:

Sandbox File (Python)	ADK Session State (Go)	Persistence Location
`/workspace/plan.json`	`{plan}`	Firestore `line_bot_workspaces/{userId}.plan`
`/workspace/sources.json`	`{sources}`	`...sources`
`/workspace/report.md`	`{report_md?}`	`...report_md`

However, Cloud Run scales horizontally and can scale to 0; items in session.InMemoryService() can disappear at any time. My approach: Every time a stage is executed, create a new ADK session, seed the initial state from the Firestore workspace, write the results back to Firestore after running, and then delete the session.

created, err := a.sessions.Create(ctx, &session.CreateRequest{
    AppName: a.appName,
    UserID: userID,
    State: map[string]any{KeySources: in.SourcesJSON, KeyReportMD: in.ReportMD},
})

This makes the Cloud Run instance completely stateless; a deepening request hours later will work on any instance. Added benefits:

"Republishing" no longer requires the LLM: The Python version's retry_publish had to ask the Agent to read report.md from the sandbox and return it; the Go version simply takes the Markdown from Firestore and re-renders/uploads it, saving a model call.
Handling sandbox expiration remains consistent: Reports created in the old Managed Agents version cannot access their sandbox files in the Go version. I handle this as the original "environment expired" path: push a message saying "Previous research files have expired, restarting research from scratch 🔄" and rerun.

⚠️ A small ADK template pitfall: ADK treats all strings matching {identifier} in the instruction as state variables and errors out if they aren't found. The original prompts had many JSON examples; these must be cleaned up. I wrote a unit test to scan the three prompts and only allow {plan}, {sources}, and {report_md?} placeholders.

4. "Return ONLY the JSON" → `OutputSchema`

To get JSON in the Python version, the prompt ended with "Return ONLY the JSON. No prose, no markdown fences.", and the code required a _parse_agent_json function to try the raw text,

```json fence content, or the first { to the last }.

ADK for Go's OutputSchema is much cleaner:

When OutputSchema is set, the framework transparently injects a set_model_response tool so the model can still invoke other tools (function tools, RAG, agent transfer, etc.) while producing structured output.

In other words, ADK automatically injects a set_model_response tool. The model can use Google Search / URL Context to find data first, then call this tool to submit results matching the schema, which are then written back to session state via OutputKey. I defined the output structures for all three stages using genai.Schema in internal/research/schema.go, keeping the original JSON parser as a fallback.

IV. Model Selection: Not every Gemini can do "Search + Read Web + Structured Output" simultaneously

The probe program mentioned earlier was for this exact moment. Results from running the same program in my GCP project at location global:

Model	Google Search + URL Context + OutputSchema	Result
`gemini-2.5-flash`	❌	`400 INVALID_ARGUMENT: controlled generation is not supported with URL Context tool`
`gemini-3-flash-preview`	✅	Used in the first live version
`gemini-3.1-flash`	❌	`404 NOT_FOUND` (This project doesn't have this model)
`gemini-3.8-flash`	✅	Current Default
`gemini-3.8-pro` / `-flash-preview` / `-pro-preview` / `-flash-lite`	❌	All `404 NOT_FOUND`

Two takeaways:

The model list must be verified by actually calling them from your project. Just because a model name exists in documentation doesn't mean your project or location has access to it; 404 Publisher model ... was not found or your project does not have access to it is a common response.
Model names must be environment variables. The Go version adds an optional GEMINI_MODEL variable; if not set, it uses the code default. This means the 11 original environment variables on the live service didn't need to change.

A bug caught when switching to gemini-3.8-flash

After upgrading to gemini-3.8-flash, reports became significantly longer and deeper (approx. 5k → 11–12k characters), but integration tests caught a subtle issue: when asked to deepen a non-existent "Chapter 99," it returned:


plaintext
error="chapter_not_found_chapter_not_found" available_chapters=[]

The old code used == "chapter_not_found" for validation; this value didn't match, so it would publish an empty report_md as a new version. I fixed this in three layers:

Changed error in the schema to an enum (only allowing chapter_not_found) to restrict the model at the source.
The worker now uses strings.Contains for loose matching; if available_chapters is empty, it extracts chapters from the ## headers in the report itself.
If report_md is empty, it never publishes and instead pushes "Deepening failed, please try again later."

After adding the enum and rerunning, gemini-3.8-flash correctly returned chapter_not_found and the full four chapters. Switching models isn't just about changing a string; edge cases for structured output must be re-verified with integration tests.

To be honest about the cost: the total time for the three stages increased from about 60 seconds to 150–190 seconds. It's still within timeout, but users have to wait a bit longer.

V. gcloud and Cloud Run Deployment: Ten Things to Watch Out For

After finishing the code, passing go test ./..., and running integration tests against the real Vertex AI, the real challenge began. This is a bot serving real LINE users live. Below are the points I hit (or narrowly avoided) during deployment.

Note 1: `describe` the online settings first; don't trust the `deploy.sh` in the repo

I initially thought I could just follow the deploy/deploy.sh in the repo, but a describe of the live service revealed the two were already different (I had tuned them online during previous debugging):

Setting	`deploy.sh` in repo	Actual Live Service
`GCS_BUCKET`	`line-reports`	A different bucket
Memory	`1Gi`	`2Gi`
Timeout	`600`	`3600`
`--min-instances`	`1`	`0` (not set)

If I had just run deploy.sh, reports would have gone to the wrong bucket, and long tasks would have been killed at 600 seconds. When rewriting a service, the "actual online state" is the specification:


shell
gcloud run services describe line-research-bot --region=asia-east1 --format=json \
  | jq '{sa: .spec.template.spec.serviceAccountName,
         env: [.spec.template.spec.containers[0].env[].name],
         resources: .spec.template.spec.containers[0].resources,
         timeout: .spec.template.spec.timeoutSeconds}'

Note 2: To "reuse the same environment variables," the simplest way is to pass nothing

The previous linebot-helper was deployed as a new service, so environment variables had to be converted to an --env-vars-file. This time, deploying back to the same service, there's a simpler way:

gcloud run deploy --source=. without --set-env-vars / --update-secrets will cause the new revision to fully inherit the environment variables and Secret Manager bindings from the previous revision.

This ensures the LINE Channel Secret / Access Token never appear in the command line or terminal output. After deploying, I described it again to confirm the list of 11 environment variables remained the same.

Note 3: Use `--no-traffic --tag` for Canary; tags must be at least 3 characters

Since it's the same service, I didn't want all LINE users hitting the new version immediately. The Cloud Run way:


shell
gcloud run deploy line-research-bot --region=asia-east1 --source=. \
  --no-traffic --tag=golang

The new revision gets 0% traffic but has a dedicated tag URL for direct testing. I hit a silly snag here: I initially tried --tag=go, which was blocked:


plaintext
ERROR: (gcloud.run.deploy) service.spec.traffic[1].tag: must be at least 3 characters long.

After performing smoke tests (listed below) on the tag URL, I switched traffic and recorded the previous revision name as a rollback point:


shell
gcloud run services update-traffic line-research-bot --region=asia-east1 \
  --to-revisions=<new revision>=100

# Rollback is just one line
gcloud run services update-traffic line-research-bot --region=asia-east1 \
  --to-revisions=<previous revision>=100

Smoke Test	Expected	Result
`GET /readyz`	200	✅
`POST /webhook` wrong signature	401	✅
`POST /tasks/run-research` no OIDC token	401	✅
Model field in startup logs	Expected model name	✅

Note 4: Cloud Tasks default dispatch deadline is only 10 minutes

This was a potential issue I found while reviewing the Python code during the rewrite. Research tasks are executed via Cloud Tasks calling back to the /tasks/run-research endpoint. While Cloud Run's timeout is 3600 seconds, Cloud Tasks' default dispatch deadline for HTTP targets is only 10 minutes. If there's no response after 10 minutes, Cloud Tasks marks it as failed and retries (queue setting max-attempts=3), causing the research to rerun and doubling the Gemini bill.

In the Go version, I set the task creation in internal/tasks/dispatcher.go to the maximum 30 minutes:


go
Task: &cloudtaskspb.Task{
    DispatchDeadline: durationpb.New(30 * time.Minute),
    MessageType: &cloudtaskspb.Task_HttpRequest{HttpRequest: &cloudtaskspb.HttpRequest{ /* ... */ }},
},

Additionally, the Python version's user lock TTL was only 300 seconds, meaning a user could send another query and grab the lock before the task finished, leading to two tasks modifying the same workspace. The Go version sets the lock TTL to 35 minutes (greater than the dispatch deadline) and actively releases it when the task finishes.

Note 5: Under `--allow-unauthenticated`, Cloud Tasks endpoints must verify OIDC themselves

The LINE Webhook must be accessible from outside, so the entire service is --allow-unauthenticated. /webhook is protected by X-Line-Signature, but the Python version's /tasks/run-research had no verification at all: anyone who knew the URL could POST a JSON and trigger a full research task that burns Gemini credits.

Cloud Tasks naturally carries an OIDC token when creating a task; the Python receiver just didn't verify it. The Go version adds this in cmd/server/main.go using google.golang.org/api/idtoken:


go
func oidcVerifier(audience, saEmail string) server.TokenVerifier {
    return func(ctx context.Context, bearer string) error {
        p, err := idtoken.Validate(ctx, bearer, audience) // audience = service URL + /tasks/run-research
        if err != nil {
            return err
        }
        if email, _ := p.Claims["email"].(string); email != saEmail {
            return fmt.Errorf("unexpected token email %q", email)
        }
        return nil
    }
}

Details to align:

Audience must match on both sides: Explicitly specify OidcToken.Audience when creating the task, and verify the same value on the receiving end (I use CLOUD_RUN_SERVICE_URL + "/tasks/run-research").
Service Account must be able to actAs itself: Cloud Tasks needs to sign the OIDC token using line-bot-sa. The line-bot-sa needs roles/iam.serviceAccountUser on itself, plus roles/cloudtasks.enqueuer.
CLOUD_RUN_SERVICE_URL is a chicken-and-egg problem: You don't know the URL before the first deployment. The deploy.sh strategy is to deploy first, then gcloud run services update --update-env-vars=CLOUD_RUN_SERVICE_URL=.... When deploying back to the same service, this value is inherited and doesn't need resetting.

Note 6: `--no-cpu-throttling` is NOT needed this time; min-instances can be 0

In the previous linebot-helper post, I spent a whole section explaining how background goroutines get frozen without --no-cpu-throttling. This architecture is the opposite:

/webhook only does signature verification, classification, locking, and enqueueing. It returns 200 only after finishing synchronously, with no background goroutines.
The truly time-consuming research runs within the HTTP request sent by Cloud Tasks. The request stays open from start to finish, so Cloud Run won't throttle the CPU or scale down the instance.

Thus, the default "allocate CPU only during request processing" is perfectly sufficient, and --min-instances can stay at 0. Go cold starts in less than 1 second, and the LINE reply token is valid for about 1 minute, which is plenty of time. Whether to use --no-cpu-throttling depends on whether the work is done inside or outside the request.

The only thing to watch is releasing the lock: the r.Context() might already be canceled when the handler ends, so I use context.WithoutCancel(ctx) to release the Firestore lock, ensuring it always gets released.

Note 7: Deployment is possible without local Docker; `--source=.` uses Cloud Build

I don't have Docker installed on my laptop. gcloud run deploy --source=. sees a Dockerfile in the directory and hands it to Cloud Build. Since this Bot doesn't need Chromium or ffmpeg, I can safely use the minimal distroless image:


docker
FROM golang:1.27 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY cmd/ ./cmd/
COPY internal/ ./internal/
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/server ./cmd/server

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/server /server
USER nonroot:nonroot
ENTRYPOINT ["/server"]

The Go version has no external executable dependencies, and prompts are compiled into the binary using go:embed, so the final image is just a static executable. No extra data files need to be copied.

Note 8: Same service, no need to change LINE Webhook URL, but verify with API

After deploying, I thought about going to the LINE Developers Console to change the Webhook URL, then realized: The new version is deployed to the same Cloud Run service; the URL hasn't changed. Instead of visually comparing in the Console, it's better to ask LINE using the Channel Access Token:


shell
TOKEN=$(gcloud secrets versions access latest --secret=LINE_CHANNEL_ACCESS_TOKEN)

# Current webhook setting and status
curl -s -H "Authorization: Bearer $TOKEN" \
  https://api.line.me/v2/bot/channel/webhook/endpoint

# Ask LINE to actually hit the webhook
curl -s -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"endpoint":"<YOUR_SERVICE_URL>/webhook"}' \
  https://api.line.me/v2/bot/channel/webhook/test

In my script, I directly compare if endpoint equals "Service URL + /webhook", printing only MATCH or DIFFERENT so the URL itself doesn't appear in the terminal. The test API returning {"success":true,"statusCode":200} means LINE traffic is already hitting the Go version.

Note 9: When E2E testing with a fake user, don't let LINE push failures trigger a full pipeline rerun

Smoke tests only prove the service is alive. I also wanted to verify the entire path: webhook → Cloud Tasks (OIDC) → ADK → Firestore → GCS in the production environment without using a real user. The trick is to use the Channel Secret from Secret Manager to correctly sign a message for a fake userId and hit /webhook directly:


shell
SIG=$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -binary | base64)
curl -X POST -H "X-Line-Signature: $SIG" -d "$BODY" "$URL/webhook"

LINE's reply and push will inevitably return 400 because the userId/replyToken is fake. This is expected, but it highlighted a design issue: If the final Flex card push fails and the worker returns an error, the handler returns 500, causing Cloud Tasks to rerun the entire 3-minute, paid research process. So, the Go version logs all LINE push failures but doesn't return an error; only infrastructure errors like Firestore failures trigger a Cloud Tasks retry.

The E2E test in production took about 33 seconds. Firestore reports/workspaces were correctly written, the lock was released, and the index.html on GCS had the correct title and version banner. Remember to clean up the fake user's Firestore documents and GCS objects after testing.

Note 10: Upgrade to the latest Go patch and use `govulncheck`

My local machine was on Go 1.26.1 during the rewrite, but the latest version was already 1.27.3. Upgrading is easy; go get modifies go.mod and automatically downloads the toolchain:


shell
go get go@1.27.3 toolchain@none
go run golang.org/x/vuln/cmd/govulncheck@latest ./...

Remember to update the Dockerfile build image from golang:1.26 to golang:1.27. govulncheck reported "Vulnerabilities affecting your code: 0", though there were 3 vulnerabilities in dependencies that weren't being called. Two in golang.org/x/net were fixed by upgrading to v0.62.0; the one in golang.org/x/crypto hasn't had an official fix released yet. govulncheck distinguishes between "called" and "not called," which is much more practical than just scanning go.mod version numbers.

VI. Things Fixed Along the Way: Rewriting is a Great Time to Re-examine Design

Item	Python Version	Go Version
`/tasks/run-research` Verification	None	Cloud Tasks OIDC (audience + SA email)
Cloud Tasks Dispatch Deadline	Default 10 mins (long tasks rerun)	30 minutes
User Lock TTL	300 seconds	35 minutes, active release on finish
`retry_publish`	Asks Agent to read `report.md` again	Uses Markdown from Firestore directly (no LLM)
Deepen Version Number	Relies on model's `new_version`	Calculated by code (previous + 1)
Report HTML	Raw HTML from model rendered; `<title>` not escaped	goldmark doesn't render raw HTML by default; title escaped
LINE Push Failure	Exception bubbles up → 500 → Cloud Tasks rerun	Logged only; doesn't trigger rerun
Execution Environment	Python 3.11 slim	Go 1.27.3 distroless/static, nonroot

Regarding testing, the Go version has two layers:

Unit Tests (go test ./..., no GCP needed): Worker uses fake store / agents / LINE / publisher. Covers new research, stage failures, retry_write, retry_publish, deepening, chapter_not_found (including model hallucination cases), and missing workspaces. Server covers signatures, locks, enqueue failure rollback, OIDC, recall / reset / small talk.
Integration Tests (go test -tags integration): Hits the real Vertex AI to run through all three stages, then tests deepening a non-existent chapter. When switching models or upgrading ADK, this is the only way to catch the model behavior differences mentioned earlier.

VII. One Last Security Reminder: Don't let Cloud Run URLs appear in public Issues

Both repos are public, as are the Issues and PRs. I deliberately did a few things:

The first line of the Issue states: "This issue is a public record and will not post any Cloud Run service URLs, Secret values, or internal identifiers."
All gcloud command outputs that would print URLs were masked using sed -E 's#https?://[^]+#<url>#g', or the URL was stored in a shell variable and only the comparison result was printed.
Before sending a PR, I ran git diff main | grep run.app to ensure no real URLs were in the new code or documentation (only a fake https://x.run.app from a deleted Python test was caught).
Before opening the new public repo, I scanned the entire git history (git log --all -p) to ensure the real service hostname never appeared once before pushing with gh repo create --public --source=. --push. If a repo is public, everything in old commits is also public.

/webhook has LINE signatures and /tasks/run-research has OIDC, so even if the URL leaks, unauthorized access is blocked. However, a public .run.app URL still attracts scanners and meaningless traffic. If you don't have to make it public, don't.

VIII. Python Managed Agents → Go ADK Migration Checklist

1. Audit online settings first: Use gcloud run services describe to get the SA, env vars, Secret bindings, memory, and timeout. Don't trust outdated deploy scripts in the repo.
2. Feasibility verification for the model: Call it from your project and location to confirm the model exists and supports your tool combination (e.g., Google Search + URL Context + OutputSchema).
3. Sandbox Files → Session State: Map the cross-turn files from the Managed Agents sandbox to ADK {state} placeholders and persist them to external storage like Firestore to keep Cloud Run stateless.
4. Check curly braces in instructions: ADK treats {identifier} as a state variable. Clean up JSON examples when moving prompts, and ideally use a test to catch them.
5. Use OutputSchema instead of prompt constraints, but keep a loose JSON parser as a fallback. Use enums for error code fields.
6. Use --no-traffic --tag (tag ≥ 3 chars) for the same service. Only update-traffic after smoke tests pass, and record the revision for rollback.
7. Omitting --set-env-vars inherits env vars and Secrets from the previous version. Describe again after deployment to confirm.
8. Cloud Tasks: Set dispatch deadline to the required length (up to 30 mins). Lock TTL must be longer than this. The receiver must verify the OIDC token.
9. --no-cpu-throttling is only needed if work happens outside the request. If work is done within the Cloud Tasks request, min-instances 0 is sufficient.
10. Verify webhook with LINE API: GET /v2/bot/channel/webhook/endpoint + POST .../webhook/test is more reliable than checking the Console.
11. External notification failures shouldn't trigger a full pipeline retry. Only return 5xx for infrastructure errors.
12. Upgrade to the latest Go patch and run govulncheck. Ensure no real service URLs remain in public repo Issues / PRs / READMEs.

Summary and Open Source Project Links

Migrating kkdai/line-research-bot from Python + Managed Agents to the Go + ADK for Go version kkdai/line-research-bot-go provided several key takeaways:

Managed Agents and ADK are different trade-offs: The former manages the sandbox and filesystem for you; the latter lets the Agent run within your own code. When the sandbox's value is just "remembering three files," ADK's Session State + Firestore is simpler and easier to test.
OutputSchema + Built-in Tools is a powerful combo in ADK for Go: set_model_response allows the model to search the web first and then submit strongly-typed results, provided your chosen model supports it.
--no-traffic --tag is the safest way to switch when deploying to the same service: Env vars, Secrets, and Webhook URLs don't need to change, and rollback is a single line.
Rewriting is the best time to re-examine design: The OIDC verification, 30-minute dispatch deadline, and push failure handling added this time were all potential issues in the Python version.

The full Go code is at kkdai/line-research-bot-go. The step-by-step record of the rewrite is in Issue #1. Feel free to reference and discuss!

[AI in Practice] Migrating a LINE Research Bot from Go ADK to DeepSeek Harness (DSH): Architecture, Mapping, and Pitfalls

Evan Lin — Tue, 03 Nov 2026 00:00:00 +0000

	Before (ADK for Go)	After (DeepSeek Harness)
Where the agent runs	Inside the same Go process	One `dsh` subprocess per stage
How it reaches Gemini	Direct connection to Vertex AI	DSH (Messages protocol) → LiteLLM sidecar → Vertex AI
Structured output	`OutputSchema`	`submit_*` tools + reading session events
Google Search	ADK built-in `geminitool`	Custom `web_research` tool
Cloud Run	1 container	2 containers (app + LiteLLM), still scales to zero

Background

In my previous post, I rewrote my LINE research-planner bot from Python + Managed Agents to Go + ADK for Go, resulting in kkdai/line-research-bot-go.

The user experience is unchanged:

Send "Research how to choose a SOTA open-source vector database" on LINE, and the bot runs three stages in sequence, PLAN → SEARCH_COMPARE → WRITE_REPORT, then replies with a Flex card containing a summary, key sources, and a "Read full report" button.
Follow up with "Go deeper on Chapter 2 and add Japanese sources", and the same report is refined in place: the version number is bumped and the previous version is saved as a snapshot.

Around the same time, DeepSeek open-sourced deepseek-ai/dsh-superagent-prototype, a "super agent" prototype built on DeepSeek Harness (DSH). I was curious what this framework would look like behind a LINE bot, so I set myself a challenge:

Keep the LINE bot features and Cloud Run setup exactly as they are, and swap only the agent framework from ADK to the dsh-superagent stack.

The short answer: it works, and it is already running in production, but not in the way I originally expected. This post covers three things:

What the DSH framework actually is (with an architecture diagram), and what dsh-superagent builds on top of it.
How to migrate from ADK: concept mapping and real code.
What goes wrong: the pitfalls I actually hit, collected into a checklist.

Repo	Description
`kkdai/line-research-bot-go`	Previous post: Go 1.27 + ADK for Go v1.9.0
`kkdai/line-research-bot-dsh`	This post: TypeScript (Node 24) + DeepSeek Harness 0.2.1-rc.5. The LINE webhook currently points here.

1. Understanding DeepSeek Harness (DSH)

Many people assume dsh-superagent is an agent SDK. In fact, it consists of two separate layers:

Layer	What it is	On npm
DeepSeek Harness (DSH)	An "agent runtime" that handles model calls, tools, sessions, permissions, and sandboxing, all implemented as plugins	`@deepseek-ai/dsh` (CLI), plus 100+ `@deepseek-ai/dsh-*` packages
dsh-superagent	A bundle (plugin pack) running on DSH that turns it into a proactive agent that works on a GitHub repository	A Node package installed into a DSH profile via `dsh plugin add`

DSH's core design: everything is a Cordis plugin

DSH is built on Cordis (@deepseek-ai/cordis), a plugin framework. Models, tools, session storage, sandboxing, even the JSON-RPC server: every piece is a plugin (a "plugin row"), and plugins communicate through services exposed on ctx, such as ctx.llm, ctx.tools, ctx.fs, and ctx.subprocess.

How plugins are composed is determined by a Profile:

Bundle: an official, pre-assembled set of plugins, e.g. dsh-sdk-minimal, dsh-base, web, tui, headless.
Profile: lives under $DSH_HOME/profiles/<name>/ and specifies which bundles to load.
Patch layers: YAML patches layered on top of bundles. A patch can change a plugin's config, disable it with disabled: true, or insert your own plugins. Run dsh --profile x --dump-config to see the fully resolved tree.

# Example patch: tweak model settings, disable the shell, insert a custom plugin
- id: llm-deepseek
  config:
    thinking: disabled
- id: persistent-bash
  disabled: true
- insert:
    - id: line-research-tools
      name: 'file:///app/dsh/plugin/index.js'

DSH architecture

A few facts that follow from this diagram are critical for the migration:

The agent runs in a separate process. Unlike ADK's runner.Run(), which runs inside your program, with DSH you spawn a dsh runtime subprocess and talk to it over stdio JSON-RPC. The TypeScript client is @deepseek-ai/dsh-sdk-client; a Python SDK is also available.
The model layer only speaks the Anthropic Messages protocol. The official adapter, llm-deepseek, targets https://api.deepseek.com/anthropic by default. You can change baseURL, but the other side must be a /v1/messages-compatible endpoint. There is no Gemini adapter.
Tools are registered by plugins. You write a Cordis plugin and call ctx.tools.register(defineTool({...})) inside apply(ctx). DSH validates arguments against the parameter schema; on failure, the error is returned to the model so it can correct itself.
A session is an event stream. Every run() returns the full list of session events: turn/start, tool/call, tool/result, assistant/message, turn/end, and so on. This is what you rely on later to extract structured results.

What dsh-superagent builds on DSH

dsh-superagent is a prototype of a "proactive coding agent" used internally by the DeepSeek team. Its README is explicit: it is a research prototype, not a production product. Its domain model looks like this:

A Source converts GitHub notifications and Feishu messages into Triggers.
Eval decides whether to accept a job based on Policy (who can issue commands, where, and with what authority). Accepted work becomes a Ticket.
Each Ticket is handled by an Executor Session working in its own directory. Results flow back to the original conversation through a Channel, e.g. by opening a PR.
A web dashboard shows which items need a human decision.

The design is very thorough, and it exposes a SourceKind extension point (docs/008-cookbook.md), so in theory you could add a LINE Source.

2. Feasibility: why I did not simply fork dsh-superagent

My first instinct was to fork dsh-superagent, add a LINE Source, and deploy it to Cloud Run. After reading through the docs and code, though, it became clear that its assumptions are almost the exact opposite of Cloud Run's:

Aspect	dsh-superagent	Existing Cloud Run architecture
Execution model	A single long-running process (`scripts/dsh-superagent start`, launched in tmux)	Stateless, scales to zero
State	SQLite + lock files under `stateDir`	Firestore
Event source	Actively polls GitHub notifications; Feishu long-lived connection	LINE pushes via webhook
HTTP	Management API listens only on 127.0.0.1, with same-origin checks	`/webhook` must be publicly reachable
Required config	A GitHub repo and an agent account are mandatory; otherwise the runtime stays in "awaiting configuration"	Only a LINE channel
Agent permissions	`danger-full-access`, runs shell commands as the OS user	Only needs web access for research
Model	Anthropic Messages (DeepSeek)	Vertex AI Gemini

Forcing a fork would mean removing the mandatory GitHub configuration, building a public, signature-verified webhook, pinning Cloud Run to min = max = 1 instances, and mounting Filestore for SQLite. In other words: giving up scale-to-zero and exposing an alpha-stage process capable of running arbitrary shell commands directly to the internet.

So I weighed two options:

Option A: Fork dsh-superagent and add a LINE Source. Keeps Eval / Ticket / dashboard, but is the most expensive and the most fragile.
Option B: Keep the Cloud Run architecture and replace only the agent layer with DSH. ✅ Chosen. The LINE webhook, Cloud Tasks, Firestore, and GCS stay untouched; each of the three research stages runs its own DSH session.

For the model, I chose to keep using Gemini on Vertex AI, with LiteLLM in between as a protocol translator. The consequences of that decision are covered in Section 3.

💡 Takeaway: Before "replacing framework A with framework B", be clear about whether you are replacing the runtime (the harness) or the application built on top of it (the superagent). What I actually needed here was the former.

3. Migrating from ADK to DSH: architecture and mapping

Post-migration architecture

On Cloud Run this is one service with two containers: app (ingress) and a litellm sidecar, communicating over localhost. The service still scales to zero.

ADK → DSH mapping

Concept	ADK for Go (previous post)	DeepSeek Harness (this post)
Where the agent runs	In-process, `runner.Run()`	One `dsh` runtime subprocess per stage, driven over stdio JSON-RPC
Agent definition	`llmagent.New(...)` × 3	One profile (`sdk-minimal` + patch) + persona; stage instructions go in the user message
Model	`gemini.NewModel`, direct to Vertex AI	DSH Messages adapter → LiteLLM sidecar → Vertex AI Gemini
Google Search / URL Context	`geminitool.GoogleSearch{}` + `url_context`	Custom plugin tool `web_research` (Gemini + googleSearch + urlContext)
Structured output	`OutputSchema` (auto-injects `set_model_response`)	Custom `submit_plan` / `submit_sources` / `submit_report` tools; arguments read back from session events
Stage handoff	Session State `{plan}`, `{sources}`, `{report_md?}`	Firestore workspace, injected directly into the stage prompt
Trace ID	ADK `InvocationID`	DSH `sessionId` (still stored in `last_interaction_id`)
Tool permissions	Only the tools each agent needs	Patch disables bash, editor, and working-directory; only research tools remain
Language	Go 1.27	TypeScript (Node 24 runs `.ts` directly, no build step)

Everything on the LINE side — intent classification, Flex cards, Firestore document schema, GCS paths, the Cloud Tasks flow, deepen / retry — was ported 1:1 from Go, down to the exact message text. The real work in this migration comes down to four rewrites in the agent layer.

Rewrite 1: `runner.Run()` → `DeepSeekHarness.run()`

ADK runs the agent in-process; with DSH, each stage starts a runtime and shuts it down when done (src/research/agents.ts):

const harness = new DeepSeekHarness({
  profile: 'sdk-minimal',
  patches: [patch], // research.patch.yml, rendered at startup
  dshHome: this.opts.dshHome, // /tmp/dsh-home on Cloud Run
  cwd: workdir, // an empty temp directory per stage
  provider: 'deepseek-official', // routes through the llm-deepseek adapter
  model: this.opts.model, // gemini-3.8-flash, an alias LiteLLM recognizes
  maxTokens: 32_000,
  env: {
    ...runtimeEnv(process.env), // strips LINE secrets
    DEEPSEEK_BASE_URL: this.opts.llmBaseURL, // http://127.0.0.1:4000
    DEEPSEEK_API_KEY: this.opts.llmApiKey, // LiteLLM master key
    DSH_SYSTEM_PROMPT: PERSONA,
  },
})
try {
  const result = await harness.run(stagePrompt)
  assertTurnCompleted(result.events)
  return extractSubmission(result.events, 'submit_plan')
} finally {
  await harness.close()
}

provider: 'deepseek-official' looks odd — why say DeepSeek when the model is Gemini? It is simply the route name of the DSH adapter, meaning "use the Messages protocol path". Where requests actually go is determined by DEEPSEEK_BASE_URL, which points to LiteLLM.

⚠️ The SDK has no mid-run cancellation. The way to enforce a timeout is to close() the runtime. I wired this up with an AbortSignal; each stage still has a 3 / 10 / 8-minute limit.

Rewrite 2: `OutputSchema` → `submit_*` tools

ADK's OutputSchema is convenient, and DSH has no equivalent. But under the hood, ADK simply auto-injects a set_model_response tool, so I did the same thing myself:

// dsh/plugin/index.js: a Cordis plugin
export const name = 'line-research-tools'
export const inject = ['tools']

export function apply(ctx) {
  ctx.tools.register(defineTool({
    name: 'submit_plan',
    description: 'Submit the research plan. Call exactly once when the plan is final.',
    parameters: {
      topic: { type: 'string', required: true },
      queries: { type: 'array', required: true, items: { type: 'object', additionalProperties: true,
        properties: { q: { type: 'string', required: true }, source_category: { type: 'string', required: true },
                      lang: { type: 'string', required: true } } } },
      source_count: { type: 'integer', required: true },
    },
    output: ack,
    async execute() { return { received: true } },
  }))
}

DSH validates the arguments against the schema first; if validation fails, the error goes back to the model so it can fix the call and retry. On the host side, all you need is to find "the last submit_plan call that was not rejected" in the session events (src/research/submission.ts):

for (const e of events) {
  if (e.type === 'tool/call' && e.data.name === tool) calls.set(e.data.callId, e.data.arguments)
  else if (e.type === 'tool/result' && e.data.message.isError) failed.set(e.data.message.toolCallId, text)
}
const accepted = [...calls].filter(([id]) => !failed.has(id))

There is a nice side benefit: the issue from the previous post where the model returned malformed enum values like chapter_not_found_chapter_not_found is now caught by DSH at the tool layer via enum validation, and the model is asked to resubmit.

Rewrite 3: a LiteLLM sidecar to connect DSH to Gemini

DSH only speaks Anthropic Messages; Gemini on Vertex AI does not. LiteLLM fits neatly in between: it exposes /v1/messages externally, calls vertex_ai/gemini-3.8-flash internally, and authenticates with the Cloud Run service account (ADC).

model_list:
  - model_name: gemini-3.8-flash
    litellm_params:
      model: vertex_ai/gemini-3.8-flash
      vertex_project: <your-project>
      vertex_location: global
litellm_settings:
  drop_params: true
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

The multi-container Cloud Run service is declared in service.yaml, with container-dependencies so the app waits for LiteLLM to come up first:

annotations:
  run.googleapis.com/container-dependencies: '{"app":["litellm"]}'
containers:
  - name: app # ingress, port 8080
  - name: litellm # sidecar, port 4000, startupProbe: /health/liveliness

Rewrite 4: a `web_research` tool to preserve Google Search grounding

This was the part I cared about most. Much of the ADK version's quality came from Google Search + URL Context grounding. Once requests are translated to the Messages protocol by LiteLLM, these Gemini-specific tools are effectively lost.

The fix is to keep grounding out of the translation layer: a web_research tool calls Vertex AI directly via @google/genai, with googleSearch and urlContext enabled:

const res = await ai.models.generateContent({
  model: researchModel,
  contents: prompt,
  config: { tools: [{ googleSearch: {} }, { urlContext: {} }], abortSignal: exec.signal },
})
// groundingChunks URLs are vertexaisearch redirects; one HEAD request resolves the real URL

The result: Gemini (via DSH) handles reasoning and planning, while Gemini (via the tool) handles web research. DSH lets the model call web_research in parallel (the tool is marked isConcurrencySafe), so the search stage turned out faster than I expected.

4. Pitfalls: what actually went wrong

Everything below is something I actually ran into during this migration. DSH is currently at 0.2.1-rc.5, and a lot of behavior is either undocumented or documented for an older version.

Pitfall 1: SDK options were renamed, and the old ones are silently ignored

The README for @deepseek-ai/dsh-sdk-client shows this example:

new DeepSeekHarness({ launch: { command: 'node', args: ['lib/bin.js', 'cordis.yml'] }, ... })

Copying it, everything appeared to work and the model responded. But when I asked it to list its tools, out came bash, subagent, create_goal, web_search… my patch had no effect at all. Digging into the SDK source revealed that rc.5 had already switched to profile / patches / dshHome / env: launch does not raise an error — it is simply ignored, and the SDK quietly falls back to the fully loaded sdk profile.

✅ How to verify: print tools[].name from the request/header event to see the exact tool list the model receives. After the fix, only read_image, web_research, submit_* remained.

Pitfall 2: plugin names are resolved relative to `$DSH_HOME/profiles/<profile>/`

The PoC worked fine, but the real integration test failed:

AgentError: every submit_plan call was rejected: Error: unknown tool "submit_plan"

Launching the runtime directly and reading stderr revealed the cause:

line-research-tools: failed to import: Cannot find package 'line-research-tools'
imported from /tmp/.../profiles/sdk-minimal/

DSH resolves plugins by walking up from the profile directory looking for node_modules. In the PoC, DSH_HOME happened to live inside the project directory, so resolution worked by coincidence. On Cloud Run, DSH_HOME=/tmp/dsh-home, so it is guaranteed to fail.

Next I tried supplying the path dynamically with name: !!js process.env.XXX, which was also rejected:

disabling profile plugin row "line-research-tools": its declared dsh requirements cannot be validated: name.startsWith is not a function

name is checked before !!js is evaluated, so it must be a literal string. The final solution: at startup, render the patch template into DSH_HOME with the plugin's absolute file:// URL hard-coded.

💡 This is the most important lesson of the whole migration: a single directory difference between PoC and production completely changed the outcome. Integration tests must use the same DSH_HOME location as production.

Pitfall 3: changing `apiKeyEnv` in a patch has no effect

I changed apiKeyEnv for llm-deepseek to LITELLM_API_KEY in a patch, and --dump-config confirmed the change, yet at runtime I still got:

MISSING_CREDENTIAL: no API key for provider route "deepseek-official"; ... export DEEPSEEK_API_KEY

In the end, simply passing LiteLLM's key and URL through the DEEPSEEK_API_KEY / DEEPSEEK_BASE_URL environment variables worked. Also note that a patch's config replaces the block entirely rather than merging, so you need to restore defaults such as defaultContextWindow and streamIdleTimeoutMs yourself.

Pitfall 4: different profiles cannot share the same sessions directory

session artifact ".../session.v4.jsonl.zstd" uses .jsonl.zstd, but this backend is configured for compression "none"

The sdk profile compresses sessions (zstd); sdk-minimal does not. If two different profiles run against the same DSH_HOME, they end up reading each other's files. Production only uses one profile, so this is not an issue there, but remember to clear $DSH_HOME/sessions when experimenting locally.

Pitfall 5: Node's type stripping does not support constructor parameter properties

Node 24 can run node src/main.ts directly with no build step, which is great. But it only strips erasable syntax, so code like this fails outright:

constructor(private readonly store: Store) {} // ❌ TS1294

Enable erasableSyntaxOnly: true in tsconfig.json so tsc catches this early. Also, .ts files under node_modules are not type-stripped, so DSH plugins must be written in plain JavaScript.

Pitfall 6: `npx node@24` does not actually run Node 24

I wanted to confirm the container's Node 24 could run everything, but npx node@24 --version printed v26.0.0: npx still used the node on PATH. I ended up downloading the Node 24 tarball directly from nodejs.org (verify it against SHASUMS256) to run the tests, which also confirmed the DSH runtime starts cleanly on Node 24.

Pitfall 7: LiteLLM needs extra Vertex AI dependencies

After pip install 'litellm[proxy]', the very first request failed with:

litellm.APIConnectionError: No module named 'google'

The Vertex AI path requires google-auth + google-cloud-aiplatform, which you must add to the image yourself and pin (I use litellm 1.106.6, google-auth 2.62.1, google-cloud-aiplatform 2.7.0). In addition, recent LiteLLM versions refuse to start without a master key, and requests without a key return 500 rather than 401 (due to missing prisma), although they are still rejected. Fortunately the sidecar only listens on localhost inside the instance and is not reachable externally.

Pitfall 8: the agent subprocess does not need your secrets

By default the DSH runtime inherits the entire process.env, which means the LINE Channel Secret and Access Token end up inside the agent's process. Even though the shell tools are disabled, I wrote a runtimeEnv() helper to strip them out, with unit tests. This matches what the dsh-superagent docs say: wrappers and directories are defaults, not security boundaries — don't hand over anything you don't have to.

Pitfall 9: it is slower than the ADK version

The integration test (plan → search → write → deepen with a missing chapter) took about 7.5 minutes, versus roughly 150–190 seconds for the ADK version. Contributing factors: an extra runtime startup per stage (about 1–2 seconds), the additional LiteLLM hop, and many web_research calls. Everything still fits within Cloud Tasks' 30-minute dispatch deadline, but I relaxed the stage timeouts from ADK's 3 / 8 / 6 minutes to 3 / 10 / 8 minutes.

Pitfall 10: pin pre-release dependencies

DSH publishes three release lines on npm simultaneously: latest (0.0.1-rc.1 or 0.1.x), next (0.2.1-rc.5), and alpha (0.2.2-alpha.3), and latest points to different versions for different packages. That is exactly how the SDK option rename in Pitfall 1 bit me. I pinned everything to the next line: @deepseek-ai/dsh, dsh-sdk-client, and dsh-tools are all at 0.2.1-rc.5.

5. Deployment and cutover: reusing the existing service's configuration

Deployment differs from the previous post: this time I created a new Cloud Run service, since the sidecar changes the container layout. All configuration, however, was carried over from the original service:

describe the original service first to get the service account, bucket, queue, and Secret Manager bindings for the LINE secrets.
Reuse everything in the new service: same SA, same bucket, same queue, same LINE secrets, and the same Firestore collections. That way, after the cutover, users can keep "deepening" their existing reports. The only new secret is LITELLM_MASTER_KEY.
Build two images with Cloud Build (my laptop still has no Docker) and deploy with gcloud run services replace service.yaml. CLOUD_RUN_SERVICE_URL is precomputed using Cloud Run's deterministic https://<service>-<project-number>.<region>.run.app format, so there is no need to deploy twice.
Smoke tests: /readyz returns 200, an invalid signature returns 401, a missing OIDC token returns 401, a request signed with the real Channel Secret returns 200, and both containers pass their startup probes.
Switch the webhook via the LINE API — again, no console required:

TOKEN=$(gcloud secrets versions access latest --secret=LINE_CHANNEL_ACCESS_TOKEN)
curl -s -X PUT -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"endpoint":"<NEW_SERVICE_URL>/webhook"}' \
  https://api.line.me/v2/bot/channel/webhook/endpoint
curl -s -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d '{}' \
  https://api.line.me/v2/bot/channel/webhook/test

Confirm traffic in the logs: after the switch, the new service received LineBotWebhook/2.0 requests and the old service dropped to zero. The first real user's research job then ran on the new service from job start to job done in about 6 minutes 43 seconds with no errors. All Gemini calls through the LiteLLM sidecar returned 200, and a ~29 KB index.html report appeared in GCS.

Rolling back is just a matter of pointing the webhook back at the original service. Because both services share Firestore, switching back and forth loses no data.

6. ADK → DSH migration checklist

1. Decide whether you are replacing the harness or the app on top of it: dsh-superagent is a GitHub coding agent; a chat bot on Cloud Run needs DSH itself.
2. Build a PoC first, using the production directory layout: keep DSH_HOME outside the project (e.g. /tmp) so plugin resolution issues surface early.
3. Print the tool list the model actually receives (request/header event) to confirm the patch is applied and dangerous tools are really disabled.
4. Follow the type definitions for SDK options, not the README: options get renamed between RCs, and old ones fail silently.
5. Use a literal absolute file:// URL for the plugin name; !!js does not work here.
6. OutputSchema → submit_* tools: let DSH validate the schema, and have the host pick the last accepted call from tool/call / tool/result events.
7. Use a Messages-compatible gateway (LiteLLM) for non-DeepSeek models, pass credentials via DEEPSEEK_API_KEY / DEEPSEEK_BASE_URL, and remember that patch configs replace rather than merge.
8. Put vendor-specific capabilities (Google Search grounding) inside tools; don't expect the translation layer to preserve them.
9. Filter environment variables for the agent subprocess; never hand it your Channel Secret.
10. Pin all pre-release packages to the same dist-tag line, and actually run the container's Node version at least once.
11. Reuse the old service's SA / bucket / queue / secrets / Firestore in the new service, switch the webhook via the LINE API, verify traffic in the logs, and roll back by reverting the webhook.
12. Never expose real .run.app URLs in public repos, issues, or posts.

Summary and open-source links

In this migration I replaced the agent layer of kkdai/line-research-bot-go — ADK for Go — with DeepSeek Harness, producing kkdai/line-research-bot-dsh. The key takeaways:

DSH is a composable agent runtime. The Cordis plugin + profile layering design is very clean: disabling, swapping, or inserting any component takes just a few lines of YAML. The trade-off is that the agent runs in a separate process and must be driven over JSON-RPC.
dsh-superagent is an excellent reference design for proactive agents (Source / Eval / Ticket / Mandate), but its execution model — a long-running process plus SQLite — is a poor fit for Cloud Run. Borrow its harness; don't force-port its app.
Most of what ADK offers can be rebuilt on DSH as tools: OutputSchema becomes submit_*, and Google Search grounding becomes web_research. Schema validation at the tool layer is arguably more reliable than prompt-level constraints.
The biggest risk with pre-release frameworks is not missing features but silent failures: ignored options, and plugin load failures that surface as "unknown tool". Every layer needs a way to be observed and verified directly.

The full source code is at kkdai/line-research-bot-dsh, and the design trade-offs are documented in the repo's docs/design.md. Feedback and discussion are welcome!

[AI in Practice] Same Name, Different Layer: DeepSeek Harness (DSH) Explained and Compared with Google Research Harness2

Evan Lin — Tue, 03 Nov 2026 00:00:00 +0000

	DeepSeek Harness (DSH)	Google Research Harness²
In one sentence	The agent's runtime	An automatic optimizer for what the agent is equipped with
What "harness" refers to	The runtime that executes the agent	The prompts, guardrails, memory, and skills wrapped around the model
What you change	Profiles, YAML patches, plugins	The `c1..c8` file tree (though usually Harness² changes it for you)
Where it's used in this series	The agent layer of the LINE research bot	Self-evolution of the LINE AI business-card assistant

Introduction: two "harnesses" in one week

In my recent Harness² trilogy (Part 1, Part 2, Part 3), I connected Google Research's google-research/harness2 to my LINE AI business-card assistant, letting it read OpenTelemetry traces and grow its own skill files. Accuracy climbed from 62.5% all the way to 100.0%.

This week I also came across DeepSeek's open-source deepseek-ai/dsh-superagent-prototype, which is built on DeepSeek Harness (DSH). I migrated the agent layer of my LINE research bot from ADK for Go to DSH, producing kkdai/line-research-bot-dsh, which is now running in production.

Both are called Harness, and at first I assumed they were the same kind of thing. After actually using both, it turns out:

DSH is about how the harness is assembled and strapped onto the horse so it can run; Harness² is about automatically adjusting every strap on that harness. They live at different layers and can even be combined.

Because there is a lot to cover, I split this into two posts:

This post: what the DSH framework actually is (with an architecture diagram), and a point-by-point comparison with Google Harness².
The next post: how I actually migrated the LINE research bot from ADK to DSH, plus a list of pitfalls.

1. Understanding DeepSeek Harness (DSH)

Many people assume dsh-superagent is an agent SDK. In fact, it consists of two separate layers:

Layer	What it is	On npm
DeepSeek Harness (DSH)	An "agent runtime" that handles model calls, tools, sessions, permissions, and sandboxing, all implemented as plugins	`@deepseek-ai/dsh` (CLI), plus 100+ `@deepseek-ai/dsh-*` packages
dsh-superagent	A bundle (plugin pack) running on DSH that turns it into a proactive agent that works on a GitHub repository	A Node package installed into a DSH profile via `dsh plugin add`

1.1 Core design: everything is a Cordis plugin

How plugins are composed is determined by a Profile:

Bundle: an official, pre-assembled set of plugins, e.g. dsh-sdk-minimal, dsh-base, web, tui, headless.
Profile: lives under $DSH_HOME/profiles/<name>/ and specifies which bundles to load.
Patch layers: YAML patches layered on top of bundles. A patch can change a plugin's config, disable it with disabled: true, or insert your own plugins. Run dsh --profile x --dump-config to see the fully resolved tree.

# Example patch: tweak model settings, disable the shell, insert a custom plugin
- id: llm-deepseek
  config:
    thinking: disabled
- id: persistent-bash
  disabled: true
- insert:
    - id: line-research-tools
      name: 'file:///app/dsh/plugin/index.js'

1.2 DSH architecture

Four key facts follow from this diagram:

The agent runs in a separate process. Unlike ADK's runner.Run(), which runs inside your program, with DSH you spawn a dsh runtime subprocess and talk to it over stdio JSON-RPC. The TypeScript client is @deepseek-ai/dsh-sdk-client; a Python SDK is also available.
The model layer only speaks the Anthropic Messages protocol. The official adapter, llm-deepseek, targets https://api.deepseek.com/anthropic by default. You can change baseURL, but the other side must be a /v1/messages-compatible endpoint. There is no Gemini adapter.
Tools are registered by plugins. You write a Cordis plugin and call ctx.tools.register(defineTool({...})) inside apply(ctx). DSH validates arguments against the parameter schema; on failure, the error is returned to the model so it can correct itself.
A session is an event stream. Every run() returns the full list of session events: turn/start, tool/call, tool/result, assistant/message, turn/end, and so on. Both structured output extraction and debugging rely on these events.

1.3 What dsh-superagent builds on DSH

A Source converts GitHub notifications and Feishu messages into Triggers.
Eval decides whether to accept a job based on Policy (who can issue commands, where, and with what authority). Accepted work becomes a Ticket.
Each Ticket is handled by an Executor Session working in its own directory. Results flow back to the original conversation through a Channel, e.g. by opening a PR.
A web dashboard shows which items need a human decision.

Its execution model is a single long-running process + SQLite + a management API that listens only on 127.0.0.1, and it requires a GitHub repo and an agent account. That is why, when migrating the LINE bot, I borrowed only DSH and did not fork dsh-superagent; details are in the next post.

💡 Takeaway: When talking about DSH, be clear whether you mean the runtime (the harness) or the application built on top of it (the superagent). What most people actually want to borrow is the former.

2. Quick recap: what Google Research Harness² does

For the full details, see Harness² Part 1. Here are the highlights:

It defines the "harness" as an editable file tree: c1..c4 are always-loaded components (systemprompt.md, guardrails.md, LongTermMEMORY.md, tool_descriptions/) with a combined budget of 24,000 bytes; c5..c8 are on-demand components (sub_agents/, skills/, scripts/, plugin/).
It edits this tree automatically via two-level recursion:
- Inner loop, Propose → Probe → Compose: read the execution trajectories (trajectory.md) of failed tasks, propose multiple candidate patches, run each once in a sandbox, and compose the ones with positive gains into the next generation $H^{(k)}$.
- Outer loop, Experience Reflector: distill lessons across rounds and write them into c3: LongTermMEMORY.md.
It connects to your agent through a BenchmarkAdapter: run() must answer "with this version of the harness loaded, what did the agent produce for this task, and what happened along the way?" For the business-card assistant, I wrote a Go cmd/harness-runner subprocess CLI that the Python Harness² calls.
Zero-leakage barrier: gold answers are hidden from both the agent and the Harvester LLM, forcing it to infer general rules from trajectories.

3. DSH vs. Harness²: a point-by-point comparison

3.1 "Harness" means different things on each side

This is the single most important point of the post:

	Harness in DSH	Harness in Harness²
Refers to	The program that runs the agent: agent loop, tool execution, model calls, sessions, permissions	What gets fed to the agent: prompts, guardrails, memory, tool descriptions, skills
Form	A Node process (runtime)	A folder (`c1..c8` files)
Analogy	How the buckles and reins of a harness are assembled	How long each strap on the harness is adjusted
Who changes it	Engineers write profiles / patches / plugins	Harness²'s optimizer changes it itself

So the two are not competing alternatives; they are an upper and a lower layer. The file tree Harness² optimizes must eventually be loaded and executed by some runtime, and DSH is exactly such a runtime.

3.2 Full comparison

Aspect	DeepSeek Harness (DSH)	Google Research Harness²
Origin	Open-sourced by DeepSeek; the foundation of dsh-superagent	Open-sourced alongside a Google Research paper (arXiv:2610.14224)
Language	TypeScript / Node (plus a Python SDK)	Python
Role	Agent runtime	Offline optimizer for the agent's harness
Core abstractions	Cordis plugins, Profile, Bundle, Patch	`c1..c8` components, `BenchmarkAdapter`, Recursion
Primary output	The result of one agent run + session events	A better harness file tree
When it runs	Online, on every request	Offline, over multiple rounds against an eval set
Requires an eval set	No	Yes (plus rubrics and trajectories)
Model	Anthropic Messages-compatible endpoint (DeepSeek by default)	The optimized agent can use any model; the Harvester is configured separately
Intrusiveness for existing agents	High: the agent must run inside DSH	Low: write an adapter and externalize config into files
How you observe the agent	Session events such as `tool/call`, `tool/result`, `turn/end`	`trajectory.md` (generated from OpenTelemetry spans in the business-card assistant)
Maturity	`0.2.1-rc.5`, pre-release; options get renamed	Research code; interfaces oriented around the paper's experiments
In this series	Agent layer of the LINE research bot's online service	Offline self-evolution of the LINE AI business-card assistant

3.3 Where do Harness²'s `c1..c8` map to in DSH?

Since DSH also "assembles a bunch of things into an agent", we can try mapping Harness²'s eight components onto it. The table below reflects my understanding from actually using DSH; where there is no counterpart in my usage, I say so:

Harness² component	Where it lives in DSH	How I used it in the LINE research bot
`c1` systemprompt.md	The `system-prompt` plugin; the persona can be passed via `DSH_SYSTEM_PROMPT`	`prompts/persona.md` as the persona; each of the three stages' instructions goes in the user message
`c2` guardrails.md	Two places: schema validation of tool arguments (`required`, `enum`), and patches that set dangerous tools to `disabled: true`	`enum` on `submit_report`'s `error` field rejects invalid values; bash and editor are disabled
`c3` LongTermMEMORY.md	A DSH session is an event stream for a single run, not cross-round memory	Not used; cross-stage data lives in Firestore
`c4` tool_descriptions/	The `description` and parameter schema in `defineTool()`, which the model sees directly	Descriptions for `web_research` and `submit_*`
`c5` sub_agents/	DSH has a built-in `subagent` tool (visible in the full `sdk` profile)	Not used; also disabled in `sdk-minimal`
`c6` skills/	I did not investigate an equivalent mechanism this time	Not used
`c7` scripts/	Equivalent to letting the agent run a shell (`persistent-bash` and similar tools)	Deliberately disabled
`c8` plugin/	This is the core of DSH itself: Cordis plugins	`dsh/plugin/index.js` registers the research tools

The most interesting finding from this mapping: Harness²'s c8 is the star in DSH, while c1..c3, which Harness² cares about most, get only basic attachment points in DSH. This mirrors their respective roles: DSH cares whether the execution side can be assembled, while Harness² cares whether the content side can evolve.

3.4 Common ground: both move the agent's "equipment" out of the code

Despite operating at different layers, the two share several design instincts:

Externalize configuration into files. The first step in bringing Harness² to the business-card assistant was extracting the prompts hard-coded in .go files into harness/base/; DSH externalizes "which plugins to load and which tools to disable" into profiles + YAML patches. Neither wants you to recompile just to change agent behavior.
Isolate via subprocesses. In the business-card assistant, Harness² calls a Go harness-runner subprocess; DSH itself is a subprocess spawned by the SDK. Either way, the host can be written in any language.
Both need visibility into the agent's process, not just its result. Harness² relies on trajectory.md for diagnosis; on DSH, I rely on session events to extract structured output and to verify that patches took effect. You can only optimize or debug what you can observe.

3.5 Differences: one governs how it runs, the other what it runs

DSH's challenges are engineering problems: plugin path resolution, renamed SDK options, a model layer limited to the Messages protocol. These are all about "getting it to run"; I cover ten pitfalls in the next post.
Harness²'s challenges are data problems: a good enough eval set, rubrics, zero-leakage design, and high-resolution trajectories. These are all about "making it better".
DSH runs on every request, so latency and cost matter; Harness² runs offline, so it can spend more tokens to produce a better file tree, at zero cost once deployed.

4. Can they be combined?

Yes, and conceptually it is quite natural. Taking the LINE research bot as an example, if I wanted it to self-evolve like the business-card assistant, it would look roughly like this:

Modeled on the business-card assistant's cmd/harness-runner, write a TypeScript runner CLI that accepts --harness-dir and --task-json.
The runner converts c1 into DSH_SYSTEM_PROMPT, writes c4 into the tool descriptions, and turns part of c2 into a patch (e.g. which tools to disable).
After the run, it renders session events such as tool/call / tool/result into trajectory.md and returns it, together with the report, to the Python-side adapter.

I have not implemented this yet; it simply seems feasible after putting the experience from both projects together. DSH's session events are already closer to "agent decisions" than OpenTelemetry spans, so they should work well as trajectories.

5. When to use which?

Your situation	Recommendation
You want an agent runtime that is composable, lets you disable dangerous tools, and lets you swap model endpoints	DSH (note: it is pre-release, and the model layer only speaks Messages)
Your agent is already in production and you want prompts, skills, and memory to evolve automatically based on evaluation results	Harness² (requires an eval set and execution trajectories)
You already run well on ADK and just want to improve quality	Start with Harness²; no need to switch frameworks
You want both	Use DSH as the runtime and Harness² as the offline optimizer, connected via an adapter + subprocess

Summary

Despite the shared name, DSH is a runtime, while Harness² is the content a runtime loads plus an optimizer for it. Once this is clear, there is no point asking "which one is better".
DSH's design centers on composability: Cordis plugins + profile layering mean disabling, swapping, or inserting any component takes just a few lines of YAML. The trade-off is that the agent runs in a separate process and must be driven over JSON-RPC.
Harness²'s design centers on evolvability: it splits the agent's equipment into budget-constrained c1..c8 components and uses trajectory-driven, two-level recursion to improve them.
Both tackle different facets of the same goal: freeing agent behavior from code. One frees how it runs; the other frees what it runs.

The next post gets hands-on: Migrating a LINE Research Bot from Go ADK to DeepSeek Harness: Migration Mapping and Pitfalls.

[Agent Platform in Action (7)] When ADK Go Graph Workflow Meets Google Research Harness (Part 1): Two-Layer Recursive Agent Fram

Evan Lin — Sun, 01 Nov 2026 14:37:47 +0000

Foreword: After the AI Agent Goes Online, What is the Next Bottleneck?

Welcome back to the "Agent Platform in Action" series! In the previous six articles, we took the LINE AI Card Assistant (kkdai/line-card-assistant) through the complete five stages from a standalone prototype to production-grade cloud deployment:

[Agent Platform in Action (1)] Building LINE AI Card Assistant with Go and Google ADK 2.0 Graph Workflow: Using google.golang.org/adk/v2/workflow to build a 13-node directed graph featuring Vision OCR, AddFanOut / AddFanIn parallel background investigation, and NewRequestInputEvent HITL interaction loops.
[Agent Platform in Action (2)] From Monolithic Cloud Run to Google Cloud Agent Runtime: Completed Phase 1 (PR #10), standardizing the project into an agents-cli (adk_go) dual-channel architecture.
[Agent Platform in Action (3)] Implementing Cross-Instance Business Card Recognition and Pause/Resume with ADK 2.0 GCS Artifact Service and Agent Platform Sessions: Completed the first half of Phase 2 (PR #11), solving the HITL "amnesia" of stateless containers and the issue of binary image data explosion.
[Agent Platform in Action (4)] Completing the Architecture Puzzle with Cloud Run LINE Webhook Gateway, Agent Runtime, and Cloud SQL PostgreSQL: Completed the second half of Phase 2 & Phase 3 (PR #12), upgrading the business card database to Cloud SQL PostgreSQL and achieving two-layer stateless decoupling.
[Agent Platform in Action (5)] Establishing Automated Evaluation Benchmarks with agents-cli eval and ADK 2.0 Go: Completed Phase 4 (PR #13), establishing a golden test set and a quality flywheel.
[Agent Platform in Action (6)] Building a Full-Stack Observable LINE AI Card Assistant with agents-cli deploy, Terraform IaC, GitHub Actions CI/CD, and Cloud Trace: Completed Phase 5 (PR #14), adding OpenTelemetry Spans (workflow.node.*) to all 13 Graph Nodes and deploying to the cloud via Terraform and GitHub Actions.

Additionally, we have planned advanced Graph Workflow primitives for future ADK v2.6+ (such as Parallel dynamic dispatch, MultiRoute multi-way fan-out, reflection-repair loops, and strongly typed nodes) into the Future Roadmap Issue #15.

However, now that our Agent possesses Phase 4 Automated Evaluation and Phase 5 Full-Node Tracing (OpenTelemetry Spans), a deeper question emerges:

When evaluations find that the Agent performs poorly in certain real-world cases (e.g., Japanese-English bilingual cards, searching for colloquial nicknames in the Taiwan tech industry, multi-clause colloquial HITL modifications, or financial compliance icebreaker letters), must engineers manually check logs, manually modify Prompt strings in the Go source code, and then recompile and redeploy every single time?

Can we let a Meta-Agent (Engineer Agent) automatically read our evaluation results and OpenTelemetry execution trajectories, diagnose the reasons for lost points, write new Skills and Long-Term Memory itself, and complete a closed-loop self-evolution?

This is precisely the core proposition addressed by Google Research's recently published and open-sourced google-research/harness2 (Harness²: Recursive Harnessing for LLM Agents, arXiv:2610.14224)!

This series will consist of three articles (corresponding to Issue #16, Issue #17, Issue #18, and PR #19), guiding you through the complete integration of Google Research Harness² into our Go ADK 2.0 project:

Part 1 (This article, Issue #16): What is Google Harness²? What can it do? Inventorying the existing architecture before modification and decoupling hardcoded configurations in the Go binary into a c1..c8 editable Harness file tree.
Part 2 (Issue #17): Encapsulating Phase 4's eval/datasets/ and Phase 5's OpenTelemetry Spans into a LineCardAdapter for Harness², and analyzing why "Zero-Leakage Barriers" and "High-Resolution Behavioral Trajectories" are both indispensable.
Part 3 (Issue #18): Complete testing tutorial, disclosure of Parallel vs Sequential + LongTermMEMORY two-layer recursive experimental data (accuracy jumping from 62.5% to 100.0%), and outlook on the future development of Agents.

1. What is Google Research `Harness²`? What can it do?

1. From "Fine-tuning Model Weights" to "Optimizing Agent Harnessing"

In the era of Large Language Models (LLMs), the success of an AI Agent is often determined not just by the underlying model (like Gemini 2.5 Flash / Pro) itself, but by the "Agent Harness" layer wrapped around the model—namely, System Prompts, formatting and safety Guardrails, Long-Term Memory, Tool Descriptions, Sub-Agent settings, and On-Demand Skills.

Past automated prompt optimizations (like PromptBreeder, TextGrad, or DSPy) mostly targeted mutations of a single Prompt string. However, in a real enterprise-grade Agent, if you stuff all edge-case patching rules into the same System Prompt, it quickly leads to "Context Bloat": the number of Always-On Tokens increases, which not only squeezes the model's Output Token Budget but also causes different rules to interfere with each other!

Google Research's Harness² (Recursive Harnessing for LLM Agents, arXiv:2610.14224) breaks this limitation. It defines the entire Agent's equipment as a structured c1..c8 Editable Component Tree and automatically evolves this file tree through Two-Level Recursion:

flowchart TB
    subgraph Outer["Outer Recursion: Harnessing-Level Recursion (Experience accumulation across rounds k=1..K)"]
        Reflector["Experience Reflector\n(ReflectorPrompt)\nDistilling success and failure patterns across rounds"]
        Memory["c3: LongTermMEMORY.md\nPersisting domain experience across iterations"]
        Reflector -->|Write long-term rules| Memory
    end

    subgraph Inner["Inner Recursion: Task-Level Recursion (Closed-loop diagnosis and patching within a single round)"]
        Propose["1. Propose (Diagnostic Proposal)\nRead trajectory.md\nLocate behavioral defects and propose N candidate Patches"]
        Probe["2. Probe (Sandbox Probing)\nExecute N candidate Harnesses in parallel\nCompare trajectory and score changes before and after modification"]
        Compose["3. Compose (Conflict Resolution and Synthesis)\nRetain positive-gain Patches, discard negative interference\nSynthesize the next-generation best Harness H^(k)"]
        Propose --> Probe --> Compose
    end

    Memory --> Propose
    Compose --> Reflector

2. The `c1..c8` Eight Harness Components and the 24KB Budget Limit

In google-research/harness2/harness.py, a standard Agent Harness consists of 4 Always-On Components (c1..c4) and 4 On-Demand Components (c5..c8):

Component Code	File Path Rule	Loading Mode	Responsibility in `Harness²`
`c1`	`systemprompt.md`	Always-On	Defines core roles, node responsibilities, and indexing rules for triggering `c6` skills (Skill Trigger Lines).
`c2`	`guardrails.md`	Always-On	Defines non-violable output formats (e.g., pure JSON constraints), E.164 normalization invariants, and security/PII guardrails.
`c3`	`LongTermMEMORY.md`	Always-On	Domain lessons and industry compliance memories written across rounds by the outer `Experience Reflector`.
`c4`	`tool_descriptions/*.md`	Always-On	Tool parameter descriptions, search strategies, and normalization details (e.g., Japan long-distance prefix `(0)` cleaning rules).
`c5`	`sub_agents/<name>/AGENT.md`	On-Demand	Exclusive Instructions and YAML Frontmatter (`mode: subagent`) for delegated sub-agents (e.g., `company_enrich`).
`c6`	`skills/<name>/SKILL.md`	On-Demand	Professional skills dynamically injected only for specific domains or tasks (e.g., Japanese card OCR, Hsinchu Science Park colloquial search expansion).
`c7`	`scripts/*.py`	On-Demand	Auxiliary verification or data conversion scripts.
`c8`	`plugin/*.ts`	On-Demand	Runtime lifecycle and OpenTelemetry diagnostic hooks.

Important

Why distinguish between c1..c4 (Always-On) and c5..c8 (On-Demand)? Harness² sets an iron rule in OPENCODE_SURFACE: The total byte count of c1..c4 always-on files must never exceed 24,000 bytes (AlwaysOnMaxBytes = 24000)! When Harness² finds that a failure case requires lengthy domain knowledge (e.g., a department title mapping table for Japanese corporate cards or a colloquial abbreviation dictionary for Taiwan semiconductor parks), it won't force these rules into systemprompt.md causing Context explosion. Instead, it will automatically create an on-demand c6 skill file (skills/<name>/SKILL.md) and leave only a lightweight Trigger Line in systemprompt.md!

3. How Does Two-Level Recursion Work?

The Harness² evolution engine (harness2/core/recursion.py) includes two levels of recursion:

Level 1 (Inner): Task-Level Recursion (Propose $\rightarrow$ Probe $\rightarrow$ Compose):
- Propose (Diagnostic Proposal): The optimizer (Harvester LLM) observes tasks that lost points in the benchmark, the final output (Deliverable), and the execution trajectory (trajectory.md). It diagnoses which step the Agent made a wrong decision on and proposes multiple independent candidate patches (Candidate Patches, each being a set of additions or modifications to c1..c8).
- Probe (Sandbox Probing): The system mounts each candidate Patch into a sandbox and runs the task once, collecting new scores and new execution trajectories to form a "Before vs. After" Behavioral Contrast.
- Compose (Conflict Resolution and Synthesis): The optimizer compares the actual performance of all candidate Patches, merges modifications that truly bring positive score improvements into the new generation Harness $H^{(k)}$, and discards ineffective or side-effect-inducing modifications.
Level 2 (Outer): Harnessing-Level Recursion (Experience Reflector):
- Between multiple iterations ($k = 1 \dots K$), the outer ReflectorPrompt reviews the evolution history from $H^{(0)} \to H^{(1)} \to \dots \to H^{(k)}$: Which types of modifications succeeded? Which cross-domain constraints (e.g., financial industry compliance headers and personal data protection regulations) need to be remembered at a global level?
- Distilled high-level experiences are written into c3: LongTermMEMORY.md, guiding the next round of Propose and Agent execution.

2. Inventorying the Current Architecture: Why Phase 1~5 Go Implementation Cannot Be Directly Optimized by `Harness²`?

Let's compare the left side (Before: Hardcoded Go Binary Architecture) of the architecture diagram at the beginning of the article and review the system architecture we completed in Phase 1 to Phase 5:

flowchart LR
    subgraph Before["Before: Phase 1~5 Solid Compilation Architecture (Hardcoded Go Binary)"]
        direction TB
        N1["nodes_ingest.go\nHardcoded VisionOCR / CleanAndDedup /\nCompanyEnrich / ApplyEdit Prompts"]
        N2["nodes_search.go\nHardcoded QueryIntent /\nEmailDraft Prompts"]
        N3["graph.go\nFixed SubAgent Instruction"]
        E["eval/datasets/basic-dataset.json\nPhase 4 Golden Test Set"]
        T["telemetry.go\nPhase 5 OpenTelemetry 13-Node Spans"]
    end

    Problem["Pain Points:\n1. Prompts hardcoded in .go string constants; re-go build required for every adjustment\n2. Lack of on-demand c6 Skills mechanism\n3. Phase 4 Eval and Phase 5 OTel Spans not yet connected into an automated optimization loop"]
    Before --> Problem

In the code before modification:

All Prompt templates for VisionOCRNode, CompanyEnrichNode, ApplyEditNode (located in internal/workflow/nodes_ingest.go), as well as QueryIntentNode and EmailDraftNode (located in internal/workflow/nodes_search.go), are hardcoded as fmt.Sprintf(...) string constants in the Go code.
The llmagent.Config{Instruction: ...} for CompanyEnrichAgent (located in internal/workflow/nodes_ingest.go) is also a fixed string hardcoded in a Go function.
When we want the Harness² Python evolution loop to modify system prompts, add a Japanese business card recognition skill (skills/jp_card_ocr/SKILL.md), or adjust phone number normalization guidelines (tool_descriptions/tool_guidance.md), it is impossible for Harness² to rewrite our .go source code and go build again for every single task!

Therefore, the first step in introducing Harness² (Issue #16) is to completely decouple the Go compilation phase from the Harness equipment phase:

Go Binary is responsible for "Graph Topology and Dual-Channel Execution Engine": RouterNode, parallel Fan-Out/Fan-In, HITLConfirmNode pause/resume, Cloud SQL and GCS Artifact access, OpenTelemetry instrumentation.
External c1..c8 Harness file tree is responsible for the "Evolvable Brain and Skill Equipment": Dynamically hot-reloaded at runtime via the environment variable LINE_CARD_HARNESS_DIR!

3. Implementation Part 1: Building `internal/harness` Loader and `harness/base/` Initial Equipment Tree

To make the Go ADK 2.0 Graph Workflow natively understand the Harness² c1..c8 specification, we added two core directories to the project:

harness/base/: Stores the initial unevolved ($k=0$) c1..c8 baseline file tree.
internal/harness/harness.go: Responsible for loading the c1..c8 file tree, validating OPENCODE_SURFACE 24KB limits and YAML Frontmatter rules, and dynamically combining always-on context with on-demand skills for each Graph Node.

1. Establishing the `harness/base/` Baseline `c1..c8` File Tree

We extracted the basic settings previously scattered in nodes_ingest.go and nodes_search.go into the harness/base/ directory:

harness/base/
├── systemprompt.md # c1: Core roles and overview of Graph Node responsibilities
├── guardrails.md # c2: JSON output format and phone normalization guardrails
├── LongTermMEMORY.md # c3: Cross-round long-term memory (initially empty, to be written by Reflector)
├── tool_descriptions/
│ └── tool_guidance.md # c4: Guidance for google_search and format_company_profile
├── sub_agents/
│ └── company_enrich/
│ └── AGENT.md # c5: CompanyEnrichAgent delegated sub-agent definition
├── skills/ # c6: On-demand skills directory (to be generated by Propose-Probe-Compose evolution)
├── scripts/ # c7: Auxiliary scripts directory
└── plugin/ # c8: Diagnostic plugin directory

For example, harness/base/sub_agents/company_enrich/AGENT.md uses a YAML Frontmatter format fully compatible with google-research/harness2:

---
mode: subagent
description: Searches Google to summarize company background and industry tags.
---
You are a professional business analysis assistant. Please use the google_search tool to query the company name provided by the user, then call the format_company_profile tool to return a two-sentence Traditional Chinese company profile and 3 to 5 technical or industry tags (e.g., AI, Cloud, SaaS, FinTech, Semiconductor).

2. Implementing `internal/harness/harness.go`: 24KB Always-On Budget Validation and On-Demand Skill Injection

In internal/harness/harness.go, we implemented the validator Validate() and node prompt synthesizer ComposeNodePrompt() fully aligned with the Python version of OPENCODE_SURFACE.validate():

// AlwaysOnMaxBytes matches google-research/harness2 OPENCODE_SURFACE.always_on_max (24,000 bytes).
const AlwaysOnMaxBytes = 24000

// AlwaysOnFiles lists the four c1..c4 always-on files mounted into the agent context.
var AlwaysOnFiles = []string{
    "systemprompt.md",
    "guardrails.md",
    "LongTermMEMORY.md",
    "tool_descriptions/tool_guidance.md",
}

// Validate enforces Google Research Harness² OPENCODE_SURFACE invariants on the c1..c8 tree.
func (h *Harness) Validate() error {
    var problems []string
    for _, reqFile := range AlwaysOnFiles {
        if strings.TrimSpace(h.Files[reqFile]) == "" {
            problems = append(problems, fmt.Sprintf("missing or empty always-on component %q", reqFile))
        }
    }
    if total := h.AlwaysOnBytes(); total > AlwaysOnMaxBytes {
        problems = append(problems, fmt.Sprintf(
            "always-on c1..c4 byte count %d exceeds cap %d (every token here competes with the agent's output budget)",
            total, AlwaysOnMaxBytes,
        ))
    }

    combinedAlwaysOn := h.SystemPrompt + "\n" + h.Guardrails
    for dirName, skill := range h.Skills {
        if skill.Name != dirName {
            problems = append(problems, fmt.Sprintf("%s frontmatter name %q must match directory name %q", skill.RelPath, skill.Name, dirName))
        }
        if strings.TrimSpace(skill.Description) == "" {
            problems = append(problems, fmt.Sprintf("%s must include a non-empty description in YAML frontmatter", skill.RelPath))
        }
        if !strings.Contains(combinedAlwaysOn, dirName) {
            problems = append(problems, fmt.Sprintf("skill %q is never referenced in systemprompt.md or guardrails.md; add a trigger line naming `%s`", dirName, dirName))
        }
    }
    // ...
}

There are two very critical engineering details here:

Mandatory Skill Trigger Reference Check: If Harness² adds a skill in skills/jp_card_ocr/SKILL.md but forgets to write when to trigger jp_card_ocr in systemprompt.md or guardrails.md, Validate() will directly report an error and intercept it, avoiding the creation of "Orphan Skills" that will never be called.
Node-Level Dynamic Skill Matching (MatchSkills + ComposeNodePrompt): When the Graph Workflow executes VisionOCRNode, CompanyEnrichNode, ApplyEditNode, QueryIntentNode, or EmailDraftNode, it only injects c6 skills related to that node and automatically attaches adk.harness.skills and adk.harness.always_on_bytes attributes to the OpenTelemetry Span:

// composeHarnessPrompt enriches a node's base prompt with the active Harness² c1..c8 tree
// and records the active harness metadata onto the node's OpenTelemetry span.
func composeHarnessPrompt(span trace.Span, nodeName, basePrompt, taskInput string) (string, *harness.Harness) {
    h := harness.LoadActive()
    composed, skills := h.ComposeNodePrompt(nodeName, basePrompt, taskInput)
    if span != nil {
        span.SetAttributes(
            AttrHarnessRoot.String(h.RootDir),
            AttrHarnessAlwaysOnBytes.Int(h.AlwaysOnBytes()),
        )
        if len(skills) > 0 {
            span.SetAttributes(AttrHarnessSkills.StringSlice(skills))
        }
    }
    return composed, h
}

We can directly use the CLI to verify the always-on byte count and structural validity of harness/base:

$ go run ./cmd/harness-runner --validate-only --harness-dir=harness/base
OK (1599 always-on bytes)

4. Summary and Next Article Preview

In this article (Part 1), we completed the goals of Issue #16:

Gained a deep understanding of how Google Research Harness² (arXiv:2610.14224) automatically evolves Agent capabilities through a c1..c8 editable file tree and two-level recursion (Task-level Propose-Probe-Compose + Harnessing-level Experience Reflector), without recompiling the main program or bloating the always-on context.
Inventoried the architectural bottlenecks of Phase 1~5 where prompts and normalization logic were hardcoded in the Go binary.
Implemented internal/harness/harness.go and harness/base/, allowing the 13 nodes of the ADK 2.0 Go Graph Workflow to support dynamic mounting of the c1..c8 equipment tree via LINE_CARD_HARNESS_DIR.

With a hot-reloadable c1..c8 equipment tree in place, the next article [Agent Platform in Action (8)] When ADK Go Graph Workflow Meets Google Research Harness² (Part 2): Turning Phase 4 Eval Datasets and Phase 5 OpenTelemetry Spans into LineCardAdapter will enter Issue #17: revealing why we need to encapsulate Phase 4 test sets and Phase 5 OpenTelemetry Spans into a LineCardAdapter, and how the "Zero-Leakage Barrier" prevents the AI optimizer from cheating!

[Agent Platform in Action (8)] When ADK Go Graph Workflow Meets Google Research Harness (Part 2): Integrating Phase 4 Eval Datas

Evan Lin — Sun, 01 Nov 2026 14:37:19 +0000

Introduction: Why Decoupling Prompts Isn't Enough?

In the previous post [Agent Platform in Practice (7)] When ADK Go Graph Workflow Meets Google Research Harness² (Part 1): Understanding the Recursive Agent Framework and c1..c8 Decoupling (corresponding to Issue #16), we introduced Google Research's google-research/harness2 (arXiv:2610.14224) and decoupled the prompts, guardrails, long-term memory, and on-demand skills of the LINE AI Business Card Assistant (kkdai/line-card-assistant) into an editable c1..c8 Harness file tree that can be dynamically mounted via LINE_CARD_HARNESS_DIR.

However, simply having a hot-reloadable c1..c8 file tree isn't enough; the Harness² dual-layer recursive optimization engine (harness2/core/recursion.py) still cannot directly optimize our Go Agent. This is because Harness² requires a standardized bridge — the BenchmarkAdapter — to answer three questions:

list_tasks() (What are the test questions?): What test cases should the Agent challenge? What are the public inputs (Prompt / Image) for each case, and what are the hidden scoring criteria (Rubrics)?
run() (How does the Agent think and execute?): When a candidate Harness $H’$ is mounted to execute a specific task, what is the Agent's final output (deliverable)? More importantly, what decision trajectory did it follow across the 13 nodes of the Graph Workflow (trajectory.md)?
score() (How was the performance?): Which independent scoring items (per_rubric) did the final output and execution trajectory pass, and where did it lose points?

Conveniently, in Phase 4 (PR #13), we already established the eval/datasets/ automated evaluation benchmark, and in Phase 5 (PR #14), we added workflow.node.* OpenTelemetry Spans to all 13 Graph Nodes!

In this post (Part 2, corresponding to Issue #17 and PR #19), we will merge these two existing assets into the Harness² LineCardAdapter and explain in depth "Why must it be designed this way?".

I. Why This Change? Two Core Design Philosophies

Before diving into the code, many engineers often ask two questions:

"Why not just feed the entire eval/datasets/*.json file to the optimizer LLM and let it modify the Prompt itself?"
"Why do we need to convert Phase 5 OpenTelemetry Spans into trajectory.md? Isn't the final LINE reply text and total score enough?"

These two questions touch upon the most core engineering designs in the Harness² paper: the Zero-Leakage Barrier and Behavioral Trajectory Contrast.

1. Why the "Zero-Leakage Barrier"? Preventing the Optimizer from "Cheating by Memorizing Answers"

Imagine this: if we expose the golden answers in eval/datasets/harness2-challenge-dataset.json (e.g., expected_contains: ["+81-3-1234-5678", "Kenta Sato"]) directly to the Agent or the Harness² Harvester LLM, what happens?

The laziest and easiest way for the optimizer to get a perfect score is to write a single line in systemprompt.md:

"If the input contains Sato, please directly output +81-3-1234-5678 and Kenta Sato."

In machine learning, this is known as Test Set Leakage / Overfitting to Rubrics. The Agent appears to score 100 on the benchmark, but it crashes immediately when it encounters a new Japanese business card in the real world!

flowchart LR
    Dataset["eval/datasets/*.json\nComplete Test Cases\n(incl. expected_contains / expected_route)"]

    subgraph Barrier["Zero-Leakage Barrier"]
        Brief["TaskBrief (Public Task Sheet)\nContains only task_id, prompt, session_input\n❌ Never contains expected_* golden answers"]
        Scorer["Isolated Scorer (Black-box Grader)\nHolds independent hidden r1..r4 Rubrics"]
    end

    Agent["Go ADK Graph Workflow\n(cmd/harness-runner)"]
    Harvester["Harness² Harvester\n(Propose / Probe / Compose)"]

    Dataset --> Brief
    Dataset --> Scorer
    Brief -->|"Public Input"| Agent
    Agent -->|"Deliverable + OTel trajectory.md"| Scorer
    Scorer -->|"Returns only binary scores {r1: 1.0, r2: 0.0}\nand OTel execution trajectory"| Harvester

In the Harness² BenchmarkAdapter architecture:

TaskBrief (Public Task Sheet): Can only contain user input (prompt and simulated images/context). It is strictly forbidden from containing any expected_contains, expected_not_contains, or correct answer strings.
score() (Black-box Grader): After comparing answers in an isolated environment, it only returns "which Rubric received a 1.0 or 0.0" (e.g., r3_e164_jp_phone_normalization: 0.0), without leaking the expected string to the optimizer.
As a result, the Harness² optimizer must truly reason out general domain rules (e.g., "When converting Japanese phone numbers +81 (0)3-... to E.164, the domestic trunk prefix (0) must be removed") by reading the Agent's execution trajectory (trajectory.md), rather than hard-coding answers into the Prompt!

2. Why Convert Phase 5 OpenTelemetry Spans into `trajectory.md`? Opening the Black Box of the 13-Node Graph

Our LINE AI Business Card Assistant is not a simple "single LLM call," but an ADK 2.0 Directed Acyclic Graph (Graph Workflow) consisting of 13 nodes:

Business Card Ingestion Pipeline: RouterNode $\rightarrow$ VisionOCRNode $\rightarrow$ Parallel AddFanOut (CleanAndDedupNode, CompanyEnrichNode, ContextMatchNode) $\rightarrow$ AddFanIn (JoinPreviewNode) $\rightarrow$ BuildPreviewNode $\rightarrow$ HITLConfirmNode $\rightarrow$ ApplyEditNode / SaveCardNode.
Smart Search and Icebreaker Pipeline: RouterNode $\rightarrow$ QueryIntentNode $\rightarrow$ HybridSearchNode / BroadSearchNode $\rightarrow$ FormatCarouselNode / EmailDraftNode.

Without Phase 5's OpenTelemetry Spans, when a user searches for "Help me find a Sales Director in Hsinchu Science Park working on AI chips" but gets "No matching business cards found" (score 0.25), the optimizer only sees this final failure message and cannot determine:

Did the RouterNode misjudge the search intent as chat?
Did the QueryIntentNode fail to expand the colloquial "Hsinchu Science Park" into the database's address Hsinchu, or fail to expand "Sales Director" into Sales Director?
Was there an error in the SQL query logic of the HybridSearchNode?

When we collect and render Phase 5's OpenTelemetry workflow.node.* Spans into a structured trajectory.md, Harness² can clearly see the input, output, duration, routing decisions, and loaded skills of every Graph Node during the Propose and Probe stages!

II. Implementation Part 2: Building the `LineCardAdapter` in Three Steps

Referring to the architecture diagram at the beginning of this article, we completed the implementation of Issue #17 through three core components:

internal/harness/trajectory.go: Mounts an in-memory OpenTelemetry SpanRecorder on the Go side to export the execution details of the 13 Graph Nodes as trajectory.md.
cmd/harness-runner/main.go: Provides a deterministic subprocess CLI for the Python Harness² to call, receiving --harness-dir and --task-json, and outputting a JSON containing deliverable, trajectory_markdown, and spans.
harness/line_card_adapter.py: Implements the google-research/harness2 BenchmarkAdapter interface, connecting the Phase 4 Dataset with the Go harness-runner.

1. Implementing `internal/harness/trajectory.go`: Rendering OpenTelemetry Spans as `trajectory.md`

In Phase 5, our internal/workflow/telemetry.go already calls startNodeSpan(ctx, "VisionOCRNode") in every Graph Node. Now, in internal/harness/trajectory.go, we use go.opentelemetry.io/otel/sdk/trace/tracetest.NewSpanRecorder() to intercept all completed node Spans and transform them into a Markdown trajectory report that the Harness² Harvester LLM is best at reading:

// InstallRecorder registers a new in-memory SpanRecorder as the global OTel TracerProvider
// so every workflow.node.* span emitted by internal/workflow/telemetry.go is captured.
func InstallRecorder() (*TrajectoryRecorder, func()) {
    sr := tracetest.NewSpanRecorder()
    tp := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(sr))
    prev := otel.GetTracerProvider()
    otel.SetTracerProvider(tp)

    rec := &TrajectoryRecorder{recorder: sr, tp: tp}
    restore := func() {
        _ = tp.Shutdown(context.Background())
        otel.SetTracerProvider(prev)
    }
    return rec, restore
}

At the same time, we added three OpenTelemetry Attributes to internal/workflow/telemetry.go: AttrNodeDecision (adk.node.decision), AttrNodeDiagnostic (adk.node.diagnostic), and AttrHarnessSkills (adk.harness.skills). When TrajectoryRecorder.RenderMarkdown() executes, it produces a high-resolution trajectory.md like this:

# ADK 2.0 Go Graph Workflow Execution Trajectory
- **Task ID** : `challenge_colloquial_tw_search`
- **Harness Root** : `harness/base`
- **Always-On Bytes (`c1..c4`)**: `1599 / 24000`
- **Final Score** : `0.25`

## OpenTelemetry Node Spans (`workflow.node.*`)
### 1. `workflow.node.RouterNode`
- **Node** : `RouterNode` | **Status** : `OK` | **Duration** : `0.02 ms`
- **Decision** : `intent=search (colloquial_query=Help me find a Sales Director in Hsinchu Science Park working on AI chips, I need to contact Taiwan semiconductor clients)`
- **Diagnostic** : `routed search query to QueryIntentNode`

### 2. `workflow.node.QueryIntentNode`
- **Node** : `QueryIntentNode` | **Status** : `OK` | **Duration** : `0.04 ms`
- **Decision** : `parsed_filters={"location":"Hsinchu Science Park","role":"Sales Director","industry":"AI Chips"}`
- **Diagnostic** : `UNEXPANDED_COLLOQUIAL_FILTERS: colloquial Taiwan science park alias 'Hsinchu Science Park' and Mandarin title 'Sales Director' were passed literally without bilingual synonym expansion; add a c6 skill 'tw_semicon_search' with BILINGUAL_SYNONYM_MAP`

### 3. `workflow.node.HybridSearchNode`
- **Node** : `HybridSearchNode` | **Status** : `OK` | **Duration** : `0.03 ms`
- **Decision** : `matched_cards=0 (strict literal filter on location='Hsinchu Science Park' failed against DB address='Hsinchu Science Park (竹科)')`

Please pay attention to the Span records of the 2nd node workflow.node.QueryIntentNode and the 3rd node workflow.node.HybridSearchNode! Through this trajectory, the Propose stage of Harness² can see at a glance: RouterNode was correct, the error was in QueryIntentNode treating "Hsinchu Science Park" as a rigid string to query the English address field!

2. Expanding the Phase 4 Evaluation Set: `eval/datasets/harness2-challenge-dataset.json`

To fully test Harness²'s ability to retain existing capabilities (preventing catastrophic forgetting) and conquer difficult edge cases, we established 8 tasks (totaling 32 fine-grained Rubrics) in eval/datasets/harness2-challenge-dataset.json:

Task Category	Task ID	Scenario Description	4 Independent Scoring Rubrics (`r1..r4`)
Baseline (Regression)	`baseline_ingest_ming_wang`	Standard Traditional Chinese card ingestion (Wang Xiaoming / Taiwan Cloud Tech)	Intent routing, Name/Title extraction, E.164 phone normalization, Industry tag enrichment
Baseline (Regression)	`baseline_dedup_appier_promotion`	Duplicate detection for promotion (Lin Yating Senior $\rightarrow$ VP)	Dedup detection, Title update, Keep new mobile, Company consistency
Baseline (Regression)	`baseline_search_cloud_summit_ai`	Standard keyword search (Cloud Summit + AI)	Search routing, Multiple hits, Tag display, Interactive buttons
Baseline (Regression)	`baseline_email_google_cloud`	General tech follow-up icebreaker (Chen Jianhong / Google Cloud)	Email routing, Recipient/Event mention, Agenda link, Professional closing
Challenge (Evolution 1)	`challenge_jp_bilingual_ingest`	Bilingual JP/EN card OCR & JP phone normalization (Kenta Sato / CTO / `+81 (0)3-1234-5678`)	`r1` Routing, `r2` Bilingual name retention, `r3` JP `+81` remove `(0)` normalization, `r4` JP title bilingual mapping
Challenge (Evolution 2)	`challenge_colloquial_tw_search`	Colloquial Taiwan tech alias search ("Help me find a Sales Director in Hsinchu Science Park")	`r1` Routing, `r2` Hsinchu Science Park $\leftrightarrow$ Hsinchu bilingual expansion hit, `r3` Sales Director $\leftrightarrow$ Sales Director title expansion, `r4` Tag completeness
Challenge (Evolution 3)	`challenge_noisy_multi_edit_hitl`	Multi-clause colloquial HITL continuous correction ("Sorry, let me correct that, the title should be Group Senior VP, also change the end of the mobile to 8899, and add GDG DevFest speaker to notes")	`r1` HITL Edit routing, `r2` Multi-field atomic update (Title), `r3` Mobile suffix partial replacement & Note append, `r4` Retain original company/email
Challenge (Evolution 4)	`challenge_pii_safe_executive_followup`	Financial executive compliance & PII masking icebreaker (Zhang Yazhu / Cathay Financial, Digital Risk Officer, requires privacy protection and compliance headers)	`r1` Email routing, `r2` Executive title/scenario, `r3` Mobile/Private Email PII Redaction, `r4` Financial compliance subject header `[Confidential / FinTech Compliance Verified]`

3. Implementing `harness/line_card_adapter.py`: Connecting to `google-research/harness2`

Finally, we implemented the LineCardAdapter in harness/line_card_adapter.py, connecting all the above components:

class LineCardAdapter:
    """Adapts `line-card-assistant` to the `google-research/harness2` BenchmarkAdapter protocol."""

    def list_tasks(self) -> list[TaskBrief]:
        """Return public task briefs with zero ground-truth rubric leakage."""
        briefs: list[TaskBrief] = []
        for raw in self._raw_cases:
            # Strip `rubrics` and `expected_*` so the agent/harvester never sees ground truth.
            briefs.append(
                TaskBrief(
                    task_id=raw["id"],
                    prompt=raw["prompt"],
                    context={
                        "category": raw.get("category", "general"),
                        "description": raw.get("description", ""),
                        "session_input": raw.get("session_input", {}),
                    },
                )
            )
        return briefs

    def run(self, harness: Harness, task: TaskBrief) -> RunArtifact:
        """Mount `harness` in a temporary directory and execute the Go Graph Workflow runner."""
        problems = harness.validate()
        if problems:
            diag = "# Harness Validation Failed\n" + "\n".join(f"- {p}" for p in problems)
            return RunArtifact(task_id=task.task_id, deliverable="", trajectory=diag)

        with tempfile.TemporaryDirectory(prefix=f"line_card_harness_{task.task_id}_") as tmp_dir:
            harness_root = Path(tmp_dir)
            harness.materialize(harness_root)
            # Invoke the compiled Go binary with --harness-dir pointing to the materialized c1..c8 tree
            proc = subprocess.run(
                [str(binary), f"--harness-dir={harness_root}", f"--task-json={task_payload_path}"],
                capture_output=True, text=True, check=False, timeout=30,
            )
            payload = json.loads(proc.stdout)
            return RunArtifact(
                task_id=task.task_id,
                deliverable=payload.get("deliverable", ""),
                trajectory=payload.get("trajectory_markdown", ""),
                raw=payload,
            )

    def score(self, task: TaskBrief, artifact: RunArtifact) -> ScoreResult:
        """Evaluate the deliverable against isolated per-rubric criteria."""
        raw = self._cases_by_id[task.task_id]
        rubrics = raw.get("rubrics", {})
        per_rubric: dict[str, float] = {}
        for rubric_id, spec in rubrics.items():
            per_rubric[rubric_id] = 1.0 if self._evaluate_rubric(artifact.deliverable, spec) else 0.0
        total = sum(per_rubric.values()) / len(per_rubric) if per_rubric else 0.0
        return ScoreResult(task_id=task.task_id, total=round(total, 4), per_rubric=per_rubric)

Through this LineCardAdapter:

Whenever Harness² proposes a new candidate Harness $H’$, adapter.run(harness, task) first calls harness.validate() to check the 24KB OPENCODE_SURFACE limit and Skill Frontmatter rules.
Once validated, the c1..c8 files of $H’$ are materialized into a temporary directory and passed to Go's harness-runner for execution.
After the Go 13-node Graph Workflow finishes, it returns the final LINE message (deliverable) and OpenTelemetry trajectory (trajectory_markdown), which are then passed to adapter.score() for leakage-free independent scoring!

III. Summary and Teaser for the Next Part

In this post (Part 2), we completed the core engineering for Issue #17:

Clarified why the golden test set cannot be fed directly to the optimizer — through the TaskBrief Zero-Leakage Barrier, we force Harness² to reason out general skills from behavioral trajectories rather than memorizing answers.
Through internal/harness/trajectory.go and cmd/harness-runner/main.go, we transformed Phase 5's OpenTelemetry workflow.node.* Spans into high-resolution trajectory.md, opening the black box of the 13-node Graph Workflow.
Established eval/datasets/harness2-challenge-dataset.json (8 major tasks, 32 fine-grained Rubrics) and harness/line_card_adapter.py, perfectly implementing the Harness² BenchmarkAdapter protocol.

Everything is ready! In the final post [Agent Platform in Practice (9)] When ADK Go Graph Workflow Meets Google Research Harness² (Part 3): Dual-Layer Recursive Testing Guide, 62.5% to 100% Performance Results, and the Future of AgentOps (corresponding to Issue #18), we will actually launch the dual-layer recursive optimization experiment, reveal the complete quantitative data evolving from 62.5% to 100.0%, and discuss the profound impact of Harness² on future enterprise-grade Agent development!

[Agent Platform in Action (9)] When ADK Go Graph Workflow Meets Google Research Harness (Part 2): Two-layer Recursive Testing Gu

Evan Lin — Sun, 01 Nov 2026 14:36:51 +0000

Introduction: From "Manual Prompt Tuning" to "Two-Layer Recursive Autonomous Evolution"

Welcome to the final part of the Agent Platform in Practice Harness² trilogy! In the previous two articles:

[Agent Platform in Practice (7)] When ADK Go Graph Workflow Meets Google Research Harness² (Part 1): Understanding the Two-Layer Recursive Agent Harnessing Framework and c1..c8 Decoupling Practice (Issue #16): We introduced Google Research's google-research/harness2 (arXiv:2610.14224) and decoupled the 13-node Graph Workflow of LINE AI Card Master (kkdai/line-card-assistant) into an editable c1..c8 file tree supporting dynamic mounting via LINE_CARD_HARNESS_DIR.
[Agent Platform in Practice (8)] When ADK Go Graph Workflow Meets Google Research Harness² (Part 2): Turning Phase 4 Eval Datasets and Phase 5 OpenTelemetry Spans into LineCardAdapter (Issue #17): We encapsulated Phase 4's eval/datasets/ and Phase 5's OpenTelemetry workflow.node.* Spans into a LineCardAdapter, establishing a Zero-Leakage Barrier and High-Resolution Behavior Trajectories (trajectory.md).

In this post (the final part, corresponding to Issue #18 and PR #19), we will answer the last three key questions:

How to test after modifications? (A complete testing guide from Go unit tests, CLI single-point diagnosis, Python two-layer recursive experiments to production environment mounting)
What is the overall performance after modifications? (Publicly disclosing quantitative data and evolutionary details from Base 62.5% $\rightarrow$ k=1 84.4% $\rightarrow$ k=2 Parallel 96.9% $\rightarrow$ k=2 Sequential + Memory 100.0%)
What are the future outlooks and application scenarios for Harness² and Agents?

1. How to test after modifications? (Step-by-Step Testing and Verification Guide)

After introducing Harness², the entire project features four levels of testing and verification: Go native unit testing, single-task OpenTelemetry trajectory diagnosis, Python two-layer recursive evolution experiments, and seamless production environment switching:

Level 1: Execute Go `c1..c8` Surface Constraints and Project-wide Unit Tests

We have written comprehensive unit tests in internal/harness/harness_test.go to verify:

All c1..c8 file trees under harness/base/ and harness/evolved/ comply with the OPENCODE_SURFACE 24KB resident byte limit (AlwaysOnMaxBytes = 24000).
Validate() accurately intercepts and reports errors when c1..c4 exceeds 24,000 bytes.
When a new c6 skill (skills/<name>/SKILL.md) is added but no trigger index (Trigger Line) is added to systemprompt.md or guardrails.md, Validate() immediately intercepts the orphan skill.
TrajectoryRecorder correctly intercepts OpenTelemetry workflow.node.* Spans and renders a Markdown report containing node decisions and diagnostics.

Execution command:

# Execute Harness² c1..c8 loader and OpenTelemetry trajectory tests
$ go test -v ./internal/harness/...
=== RUN TestLoadBaseHarnessAndValidate
--- PASS: TestLoadBaseHarnessAndValidate (0.00s)
=== RUN TestValidateRejectsOversizedAlwaysOnAndUnreferencedSkill
--- PASS: TestValidateRejectsOversizedAlwaysOnAndUnreferencedSkill (0.00s)
=== RUN TestEvolvedHarnessesValidateAndMatchSkills
--- PASS: TestEvolvedHarnessesValidateAndMatchSkills (0.00s)
=== RUN TestTrajectoryRecorderRendersWorkflowNodeSpans
--- PASS: TestTrajectoryRecorderRendersWorkflowNodeSpans (0.00s)
PASS
ok github.com/kkdai/line-card-assistant/internal/harness

# Execute project-wide regression tests (ensuring 100% compatibility with all existing Phase 1~5 features)
$ go test ./...
ok github.com/kkdai/line-card-assistant/app
ok github.com/kkdai/line-card-assistant/cmd/harness-runner
ok github.com/kkdai/line-card-assistant/cmd/line-gateway
ok github.com/kkdai/line-card-assistant/eval
ok github.com/kkdai/line-card-assistant/internal/harness
ok github.com/kkdai/line-card-assistant/internal/storage
ok github.com/kkdai/line-card-assistant/internal/webhook
ok github.com/kkdai/line-card-assistant/internal/workflow

Level 2: Use `cmd/harness-runner` to Validate Any `c1..c8` File Tree

Before mounting any custom or evolved Harness online, you can use --validate-only to quickly check its c1..c4 resident byte count and YAML Frontmatter compliance:

$ go run ./cmd/harness-runner --validate-only --harness-dir=harness/base
OK (1599 always-on bytes)

$ go run ./cmd/harness-runner --validate-only --harness-dir=harness/evolved/k2_sequential_memory
OK (4661 always-on bytes)

Level 3: Execute the Four-Stage `Harness²` Two-Layer Recursive Evolution Benchmark Experiment

We implemented a complete two-layer recursive experiment script in harness/run_harness2_experiment.py. It automatically compiles the Go harness-runner and sequentially evaluates four configurations against the 8 major tasks (32 independent Rubrics) in eval/datasets/harness2-challenge-dataset.json, finally outputting a Markdown comparison table and harness/experiment_report.json:

$ python3 harness/run_harness2_experiment.py
### Overall Configuration Comparison

| Stage | Configuration | Task Score (Max 8.0) | Rubric Pass Rate | Always-On Bytes (Cap 24,000) | Active `c6` Skills | `LongTermMEMORY.md` |
| :--- | :--- | :---: | :---: | :---: | :--- | :---: |
| `base_k0` | **Base Harness ($k=0$)** | `5.00 / 8.00` | `20/32 (62.5%)` | `1599 B` | `(none)` | `No` |
| `k1_single_step` | **Single-Step Propose-Probe-Compose ($k=1$)** | `6.75 / 8.00` | `27/32 (84.4%)` | `2505 B` | `jp_card_ocr, tw_semicon_search` | `No` |
| `k2_parallel` | **Multi-Step Parallel Recursion ($k=2$, No Memory)** | `7.75 / 8.00` | `31/32 (96.9%)` | `2998 B` | `executive_email_drafter, hitl_multi_edit, jp_card_ocr, tw_semicon_search` | `No` |
| `k2_sequential_memory` | **Multi-Step Sequential Recursion + `LongTermMEMORY.md` ($k=2$)** | `8.00 / 8.00` | `32/32 (100.0%)` | `4661 B` | `executive_email_drafter, hitl_multi_edit, jp_card_ocr, tw_semicon_search` | `Yes` |

Level 4: Directly Mount the Best Evolved Harness in Production or Local Servers

Once Harness² evolves the optimal harness/evolved/k2_sequential_memory, no modification to any line of Go code is required. Simply specify the directory via the environment variable LINE_CARD_HARNESS_DIR, and both the dual-channel main program (main.go) and the evaluator (eval/) will immediately load the evolved harness tree:

$ LINE_CARD_HARNESS_DIR=harness/evolved/k2_sequential_memory go run .

2. Performance Summary: Evolutionary Analysis from `62.5%` to `100.0%`

Let's take a deep dive into the four-stage quantitative experimental data recorded in harness/experiment_report.json to see how Harness² diagnosed and patched the Go Agent's flaws step-by-step!

1. Four-Stage Overall and Single-Task Score Summary

Evaluation Metric / Test Task	Stage 0: Base Harness ($k=0$) (`harness/base`)	Stage 1: Single-Step Inner Recursion ($k=1$) (`k1_single_step`)	Stage 2: Multi-Step Parallel Recursion ($k=2$, No Memory) (`k2_parallel`)	Stage 3: Two-Layer Sequential Recursion + Long-Term Memory ($k=2$) (`k2_sequential_memory`)
`baseline_ingest_ming_wang` (Standard Chinese Entry)	`1.00 (4/4)`	`1.00 (4/4)`	`1.00 (4/4)`	`1.00 (4/4)`
`baseline_dedup_appier_promotion` (Promotion Deduplication Update)	`1.00 (4/4)`	`1.00 (4/4)`	`1.00 (4/4)`	`1.00 (4/4)`
`baseline_search_cloud_summit_ai` (Standard Keyword Search)	`1.00 (4/4)`	`1.00 (4/4)`	`1.00 (4/4)`	`1.00 (4/4)`
`baseline_email_google_cloud` (General Tech Icebreaker Email)	`1.00 (4/4)`	`1.00 (4/4)`	`1.00 (4/4)`	`1.00 (4/4)`
`challenge_jp_bilingual_ingest` (JP-EN Bilingual Card OCR)	`0.25 (1/4)` ❌	`1.00 (4/4)` ✅	`1.00 (4/4)` ✅	`1.00 (4/4)` ✅
`challenge_colloquial_tw_search` (Hsinchu Science Park Colloquial Search)	`0.25 (1/4)` ❌	`1.00 (4/4)` ✅	`1.00 (4/4)` ✅	`1.00 (4/4)` ✅
`challenge_noisy_multi_edit_hitl` (Multi-clause Colloquial HITL Edit)	`0.25 (1/4)` ❌	`0.25 (1/4)` ❌	`1.00 (4/4)` ✅	`1.00 (4/4)` ✅
`challenge_pii_safe_executive_followup` (Financial Exec Compliance Icebreaker)	`0.25 (1/4)` ❌	`0.50 (2/4)` ⚠️	`0.75 (3/4)` ⚠️	`1.00 (4/4)` ✅
Total Task Score (Max 8.00)	`5.00 / 8.00`	`6.75 / 8.00`	`7.75 / 8.00`	`8.00 / 8.00`
Fine-grained Rubric Pass Rate (Max 32)	`20 / 32 (62.5%)`	`27 / 32 (84.4%)`	`31 / 32 (96.9%)`	`32 / 32 (100.0%)`
Resident Bytes (`c1..c4` / 24,000 B)	`1,599 B (6.7%)`	`2,505 B (10.4%)`	`2,998 B (12.5%)`	`4,661 B (19.4%)`
On-demand Skills (`c6` Skills)	`0`	`2`	`4`	`4 + LongTermMEMORY`

2. Key Findings Analysis: What evolved in each stage?

🔹 Stage 0 $\rightarrow$ Stage 1 (`62.5%` $\rightarrow$ `84.4%`): Diagnosing the first two on-demand skills (`c6`) from OpenTelemetry trajectories

In the Base Harness ($k=0$), while all 4 Baseline tasks scored full points (4.00/4.00), all 4 Challenge tasks scored only 0.25 because they only passed the RouterNode intent routing:

In the trajectory.md of challenge_jp_bilingual_ingest, the Spans for VisionOCRNode and CleanAndDedupNode diagnosed that the Japanese phone number +81 (0)3-1234-5678 failed to strip the domestic trunk prefix (0), and the Japanese title 取締役最高技術責任者 (CTO) failed to retain the bilingual title.
In the trajectory.md of challenge_colloquial_tw_search, the QueryIntentNode Span diagnosed that colloquial Taiwanese terms like "Hsinchu Science Park" (竹科) and "Sales Director" (業務總監) were not expanded into bilingual synonyms.
After the first round of inner recursion (Propose -> Probe -> Compose), Harness² automatically synthesized skills/jp_card_ocr/SKILL.md and skills/tw_semicon_search/SKILL.md in k1_single_step, and updated the PII masking rules in guardrails.md, jumping the accuracy from 62.5% to 84.4%!

🔹 Stage 1 $\rightarrow$ Stage 2 (`84.4%` $\rightarrow$ `96.9%`): Second round of recursion conquers multi-clause HITL modification and executive email structure

In the second round of inner recursion, Harness² continued to probe the remaining two failing tasks:

For challenge_noisy_multi_edit_hitl (where ApplyEditNode only modified the title in the first clause, missing the mobile suffix replacement and note addition), it evolved skills/hitl_multi_edit/SKILL.md (containing ATOMIC_MULTI_CLAUSE_EDIT rules).
For challenge_pii_safe_executive_followup (where EmailDraftNode lacked structured executive follow-up paragraphs), it evolved skills/executive_email_drafter/SKILL.md.
At this point, under Parallel Recursion (no cross-round long-term memory), the score reached 31/32 (96.9%)!

🔹 Stage 2 $\rightarrow$ Stage 3 (`96.9%` $\rightarrow$ `100.0%`): Why is `Sequential + LongTermMEMORY.md` needed to complete the final 3.1%?

Why did k2_parallel get 96.9% (31/32), yet fail on the 4th Rubric (r4_fintech_compliance_subject_tag) of challenge_pii_safe_executive_followup?

Because in cross-enterprise outgoing emails for financial institutions (e.g., Digital Risk Officer at Cathay Financial Holdings), beyond single-skill email formatting, cross-domain industry compliance memory (FinTech Outbound Compliance Header [Confidential / FinTech Compliance Verified]) is required.
In Sequential Recursion + Experience Reflector mode, the outer recursion reviewed the residual failure trajectories from the first to the second round. The ReflectorPrompt then wrote the FINTECH_COMPLIANCE_SUBJECT_TAG and cross-channel PII protection principles into the resident component c3: LongTermMEMORY.md!
Note the Resident Bytes (Always-On Bytes): even with 4 rich domain skills and long-term memory evolved, the resident size of c1..c4 is only 4,661 bytes (only 19.4% of the 24,000 bytes limit), perfectly avoiding Context Bloat!

3. Future Outlook and Application Scenarios for `Harness²` and Agents

This practical implementation of Google Research Harness² into LINE AI Card Master has shown us a brand-new AgentOps development paradigm. Looking ahead, we believe Harness² has four high-potential application scenarios in the enterprise-grade AI Agent field:

flowchart LR
    subgraph Prod["1. Production Observability (Cloud Run + Agent Runtime)"]
        Traffic["Real User Traffic\n(LINE Webhook / A2A)"]
        Trace["Cloud Trace OTel Spans\n+ Negative Feedback (HITL Retry)"]
        Traffic --> Trace
    end

    subgraph Nightly["2. Nightly Autonomous Evolution Flywheel (Nightly Harness² CI/CD)"]
        Extract["Auto Mask PII & Convert to\nChallenge Dataset Cases"]
        Evolve["Harness² Two-Layer Recursion\n(Propose-Probe-Compose + Reflector)"]
        PR["Auto Initiate GitHub PR\nUpdate harness/evolved/ c1..c8"]
        Extract --> Evolve --> PR
    end

    subgraph Scenarios["3. Future Four High-Value Application Scenarios"]
        S1["Scenario A: Fully Automated Nightly Self-Healing AgentOps"]
        S2["Scenario B: Cross-Model Harnessing & Tuning"]
        S3["Scenario C: Multi-Tenant Enterprise Compliance & Skills"]
        S4["Scenario D: Dual Evolution with ADK Graph Topology (Issue #15)"]
    end

    Trace --> Extract
    PR --> Scenarios

1. Scenario A: Fully Automated "Nightly Self-Healing AgentOps"

In the past, when a production LINE Bot encountered frequent "✏️ Edit Data" triggers via HITLConfirmNode or received negative feedback, engineers had to manually analyze Cloud Trace logs the next day. Combining the Cloud Trace established in Phase 5 with the LineCardAdapter from this series, future CI/CD can achieve:

Automatically filtering anonymous Span trajectories from Cloud Trace every night where "users modified data more than twice consecutively" or "search results were 0," and adding them to the Challenge Dataset.
Triggering a GitHub Action to run a round of Harness² two-layer recursion to automatically synthesize new c6 skills or update LongTermMEMORY.md.
Once regression tests (Baseline + Challenge) confirm score improvements and zero regression, the Bot automatically initiates a Pull Request modifying only the harness/ directory. Engineers only need to review the skill Markdown in the morning to deploy with one click!

2. Scenario B: Cross-Model Upgrades and "Harness Compensation" for Lightweight Models

Enterprises often encounter behavioral drift when switching underlying models from high-cost models to ultra-fast lightweight models (e.g., switching from Gemini 1.5 Pro to Gemini 1.5 Flash or Flash-Lite to reduce latency and cost by 80%) or upgrading to next-gen models. Through Harness², we can automatically run a round of Propose-Probe-Compose for the target model while keeping the Go code and Benchmark fixed. Harness² will automatically compensate for the lightweight model's gaps with required reasoning auxiliary skills (c6 Skills) and guardrails (c2 Guardrails), using "smarter gear" to bridge the gap.

3. Scenario C: Multi-Tenant Enterprise-Specific Compliance and Domain Memory

As seen in challenge_pii_safe_executive_followup, industries like FinTech, MedTech, and the semiconductor supply chain each have distinct terminology dictionaries and compliance guardrails. Because Harness² externalizes the gear as a plain-text file tree, enterprises only need to mount different LINE_CARD_HARNESS_DIR (e.g., harness/tenants/fintech/ vs harness/tenants/semiconductor/) for different departments or tenants. This allows the same Go Container Binary to serve multiple highly regulated vertical industries simultaneously!

4. Scenario D: Dual Evolution of "Topology + Harness" with ADK v2.6+ Advanced Graph Workflow Primitives

In Issue #15, we planned ADK v2.6+ features like Parallel dynamic dispatch, MultiRoute multi-path concurrency, and VerifyOCRQualityNode reflection loops. In the future, Harness²'s c5 (sub_agents/) and c8 (plugin/) could even further control Graph Workflow routing thresholds and sub-agent concurrency strategies—achieving an ultimate architecture where "Go is responsible for the type-safe topological skeleton, and Harness² is responsible for dynamically evolving the neural network and skill muscles."

4. Recap of the "Agent Platform in Practice" Series

From the first to the ninth post, we have completed a very solid production-grade AI Agent engineering journey:

All code, Terraform IaC, evaluation datasets, and evolved c1..c8 Harness file trees are fully open-sourced on GitHub: kkdai/line-card-assistant. If you are also building enterprise-grade AI Agents with Go and ADK, feel free to Star, Fork, and experience the power of Harness² two-layer recursion!

[Agent Platform in Action (6)] Towards Production-Grade AI Agent Ops: Building Full-Stack Observable LINE Agents with agents-cli

Evan Lin — Sun, 25 Oct 2026 10:05:18 +0000

Recap: The Five-Stage Evolution from Local Prototype to Cloud Production-Grade AI Agent

Welcome to the sixth (and final) article in the "Agent Platform in Action" series! Looking back at the past five articles, we have taken LINE AI Card Assistant (kkdai/line-card-assistant) through the four major stages of evolution from a local prototype to an enterprise-grade AI Agent architecture:

[Agent Platform in Action (1)] Building LINE AI Card Assistant with Go and Google ADK 2.0 Graph Workflow: Implemented a business card workflow graph using google.golang.org/adk/v2/workflow covering Vision OCR, AddFanOut / AddFanIn parallel background research, and NewRequestInputEvent HITL interaction loops.
[Agent Platform in Action (2)] Moving from Monolithic Cloud Run to Google Cloud Agent Runtime: Refactoring LINE AI Card Assistant with agents-cli and ADK Go: Completed Phase 1 (PR #10), standardizing the project into an agents-cli (adk_go) dual-channel architecture, supporting LINE Webhook, ADK WebUI Playground, and A2A protocols.
[Agent Platform in Action (3)] Solving HITL Amnesia in Stateless Containers: Implementing Cross-Instance Card Recognition and Pause/Resume with ADK 2.0 GCS Artifact Service and Agent Platform Sessions: Completed the first half of Phase 2 (PR #11), introducing artifact/gcsartifact and session/vertexai to solve cross-instance HITL recovery and image storage issues in stateless containers.
[Agent Platform in Action (4)] Towards Cloud Dual-Layer Decoupling and Full Persistence: Completing the LINE AI Card Assistant Architecture with Cloud Run LINE Webhook Gateway, Agent Runtime, and Cloud SQL PostgreSQL: Completed the second half of Phase 2 (Issue #5) and Phase 3 (Issue #6, PR #12), migrating the card library to Cloud SQL PostgreSQL (PostgresStore) and building a stateless Cloud Run LINE Webhook Gateway (internal/gateway).
[Agent Platform in Action (5)] Building the AI Agent Quality Flywheel: Establishing Automated Evaluation Benchmarks for LINE AI Card Assistant with agents-cli eval and ADK 2.0 Go: Completed Phase 4 (Issue #7, PR #13), using agents-cli eval to establish an automated quality flywheel covering 6 golden cases and 4 major evaluation metrics, all scoring 1.0000.

In this final chapter today, we will complete the last piece of the puzzle for Roadmap #9 — Phase 5 (Issue #8): Official Agent Runtime Deployment, Terraform IaC, GitHub Actions CI/CD, and Full-Stack Cloud Trace Observability!

I. What Problem Does Phase 5 Solve? Moving from "Manual `gcloud` Commands" to "Declarative AgentOps"

In many AI projects, even if the code and evaluation tests are written, production environments often become operational nightmares due to the following four issues:

Infrastructure relies on manual commands (ClickOps / Shell Script drift): If Vertex AI Reasoning Engine, Service Account IAM permissions, GCS Buckets, Secret Manager secrets, and Cloud SQL are all created manually via gcloud, configuration inconsistencies easily occur between Staging and Production environments.
PR merges lack an AI quality gatekeeper (No Eval Gate in CI): Running go test can catch syntax and logic errors, but if someone accidentally breaks a Graph Node's state_delta or Prompt, how can we automatically intercept it before the PR is merged into main?
Cross-environment deployment and high-concurrency verification (Staging $\rightarrow$ Load Test $\rightarrow$ Prod Approval): After pushing to main, how can we use keyless Workload Identity Federation (WIF) to automatically deploy to Staging, run high-concurrency pressure tests (Vegeta Load Test), archive the reports to GCS, and finally push to Production only after manual approval (GitHub Environment Approval Gate)?
Graph Workflow node-level observability (Node-Level Distributed Tracing): When processing a business card from upload to Flex Message preview takes 2.8 seconds, is the time spent in VisionOCRNode or in the parallel Fan-Out's CompanyEnrichNode (Google Search Grounding)? We need every node of the ADK 2.0 Graph Workflow to expand into a clear nested Span tree in Google Cloud Trace, and automatically import it into BigQuery for P95/P99 latency and Token cost analysis!

This is exactly what the Phase 5 Full-Stack Architecture Diagram at the beginning of the article depicts through four core pillars:

Top-left (GitHub Actions CI/CD & Quality Gate): .github/workflows/pr_checks.yaml, staging.yaml, deploy-to-prod.yaml
Bottom-left (Terraform Infrastructure as Code): deployment/terraform/single-project/, cicd/, shared/
Center (Vertex AI Agent Runtime + OpenTelemetry Node Spans): agents-cli deploy + internal/workflow/telemetry.go
Right (Full-Stack Observability & Analytics): observability.go $\rightarrow$ Cloud Trace + Cloud Logging $\rightarrow$ telemetry.tf BigQuery Sink

II. Step 1: Generate Terraform IaC and CI/CD Scaffolding with `agents-cli scaffold enhance`

In Phase 1, we introduced the Prototype-First philosophy of agents-cli: keep the project lightweight first, and then execute agents-cli scaffold enhance to layer on production infrastructure when ready for the cloud.

We execute the following directly in the project root directory:

uvx google-agents-cli@1.10.0 scaffold enhance . \
  --cicd-runner github_actions \
  --region asia-east1 \
  --yes

agents-cli automatically detected our agents-cli-manifest.yaml (template: adk_go, deployment_target: agent_runtime) and, without breaking any existing app/, internal/, eval/, or sessions/ code, added three major modules to the project:

deployment/terraform/single-project/ and deployment/terraform/cicd/: Complete Terraform declarative infrastructure modules.
deployment/terraform/shared/telemetry.tf and storage.tf: Cloud Logging linked to Buckets, Datasets, and BigQuery telemetry analysis pipelines.
.github/workflows/ and e2e/: GitHub Actions CI/CD pipelines and A2A / AppInfo / Vegeta Load Test end-to-end test suites.

Next, we perform customized enhancements for the specific requirements of LINE AI Card Assistant (LINE credentials, Cloud SQL PostgreSQL permissions, Taiwan asia-east1 region)!

III. Step 2: Deep Dive into Terraform IaC (Vertex AI Reasoning Engine + Secret Manager + BigQuery Telemetry Sink)

Let's open deployment/terraform/ and see how Terraform declaratively builds the entire Google Cloud Agent Platform:

1. Declaratively Defining Vertex AI Reasoning Engine (`service.tf`)

In deployment/terraform/single-project/service.tf, we use the google_vertex_ai_reasoning_engine resource from the google-beta provider to package the Go source code and declare it as a managed Agent Runtime instance. We also set lifecycle { ignore_changes = [spec[0].source_code_spec] } so that daily code updates are handled by the CI/CD's agents-cli deploy for fast rolling deployment, while infrastructure settings (IAM, Buckets, environment variables) are unified under Terraform control:

resource "google_vertex_ai_reasoning_engine" "app" {
  provider = google-beta
  display_name = var.project_name
  project = var.project_id
  region = var.region

  labels = {
    "agent-framework" = var.agent_framework
  }

  spec {
    identity_type = "AGENT_IDENTITY"
    agent_framework = var.agent_framework

    source_code_spec {
      inline_source {
        source_archive = data.archive_file.source.output_path
      }
      developer_connect_source {}
    }

    container_spec {
      image_uri = "us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-15:latest"
    }
  }

  lifecycle {
    ignore_changes = [
      spec[0].source_code_spec,
      spec[0].container_spec,
    ]
  }
}

2. Least Privilege IAM and Secret Manager Integration (`iam.tf` & `variables.tf`)

To prevent LINE_CHANNEL_SECRET, LINE_CHANNEL_ACCESS_TOKEN, and DATABASE_URL from appearing in plain text in configuration files, we declare Google Cloud Secret Manager secrets in deployment/terraform/single-project/iam.tf and variables.tf. We grant roles/secretmanager.secretAccessor and roles/cloudsql.client to both the Vertex AI Service Agent (gcp-sa-aiplatform-re) and the Agent Identity Principal (principal://${google_vertex_ai_reasoning_engine.app.spec[0].effective_identity}):

# Secret Manager secrets for LINE Messaging API credentials and Cloud SQL DSN
locals {
  line_card_secrets = [
    "line-channel-secret",
    "line-channel-access-token",
    "line-card-database-url",
  ]
}

resource "google_secret_manager_secret" "app_secrets" {
  for_each = toset(local.line_card_secrets)
  project = var.project_id
  secret_id = each.value

  replication {
    auto {}
  }

  depends_on = [resource.google_project_service.services]
}

This way, during deployment, you only need to pass --secrets="LINE_CHANNEL_SECRET=line-channel-secret,LINE_CHANNEL_ACCESS_TOKEN=line-channel-access-token" to agents-cli deploy, and the Agent Runtime will securely mount the secrets from Secret Manager when starting the container!

3. Automatically Creating Cloud Logging $\rightarrow$ BigQuery Telemetry Analysis Dataset (`telemetry.tf`)

In deployment/terraform/shared/telemetry.tf, Terraform automatically creates a Cloud Logging Bucket with a 365-day retention period, enables enable_analytics = true, and directly maps telemetry logs to a BigQuery Dataset (telemetry_logs_dataset) via google_logging_linked_dataset:

resource "google_logging_project_bucket_config" "telemetry_logs_bucket" {
  project = each.value
  location = "global"
  bucket_id = "${var.project_name}-telemetry"
  retention_days = 365
  enable_analytics = true
}

resource "google_logging_linked_dataset" "telemetry_logs_dataset" {
  bucket = google_logging_project_bucket_config.telemetry_logs_bucket[each.key].bucket_id
  link_id = replace("${var.project_name}_telemetry", "-", "_")
  project = each.value
  location = "global"
}

This means all structured telemetry data output by OpenTelemetry can be analyzed directly in BigQuery using standard SQL without writing additional ETL scripts!

IV. Step 3: Injecting OpenTelemetry Distributed Tracing into the 14 Nodes of ADK 2.0 Go Graph Workflow (`internal/workflow/telemetry.go`)

In the observability.go generated in Phase 1, the system was already configured with an OTLP HTTP Exporter pointing to telemetry.googleapis.com and Google ADC automatic authentication. However, by default, we only see top-level traces for the entire HTTP request or LLM call, but not the execution time and routing decisions of each FunctionNode within the ADK 2.0 Graph Workflow.

To present the processing of each business card as a complete waterfall span tree on Google Cloud Trace, we added internal/workflow/telemetry.go:

package workflow

import (
    "context"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/codes"
    "go.opentelemetry.io/otel/trace"
)

const workflowTracerName = "github.com/kkdai/line-card-assistant/internal/workflow"

const (
    AttrWorkflowNode = attribute.Key("adk.workflow.node")
    AttrWorkflowRoute = attribute.Key("adk.workflow.route")
    AttrInvocationID = attribute.Key("adk.invocation.id")
    AttrSessionID = attribute.Key("adk.session.id")
    AttrUserID = attribute.Key("adk.user.id")
    AttrCardID = attribute.Key("adk.card.id")
    AttrCardCompany = attribute.Key("adk.card.company")
    AttrCardDuplicate = attribute.Key("adk.card.is_duplicate")
    AttrHITLInterrupt = attribute.Key("adk.hitl.interrupt_id")
    AttrHITLAction = attribute.Key("adk.hitl.action")
    AttrSearchMatches = attribute.Key("adk.search.matches")
    AttrSearchKeywords = attribute.Key("adk.search.keywords")
)

func startNodeSpan(ctx context.Context, nodeName string, attrs ...attribute.KeyValue) (context.Context, trace.Span) {
    baseAttrs := make([]attribute.KeyValue, 0, len(attrs)+1)
    baseAttrs = append(baseAttrs, AttrWorkflowNode.String(nodeName))
    baseAttrs = append(baseAttrs, attrs...)
    return otel.Tracer(workflowTracerName).Start(
        ctx,
        "workflow.node."+nodeName,
        trace.WithAttributes(baseAttrs...),
    )
}

Next, we inject startNodeSpan and endNodeSpan into all nodes in internal/workflow/nodes_ingest.go and internal/workflow/nodes_search.go. For example, in HybridSearchNode:

func newHybridSearchNode(cardStore store.CardStore) *workflow.FunctionNode {
    return workflow.NewFunctionNode(
        NodeHybridSearch,
        func(ctx agent.Context, _ any) (ev *session.Event, err error) {
            spanCtx, span := startNodeSpan(ctx, NodeHybridSearch, AttrInvocationID.String(ctx.InvocationID()))
            defer func() {
                routeAttr := ""
                if ev != nil && len(ev.Routes) > 0 {
                    routeAttr = ev.Routes[0]
                }
                endNodeSpan(span, err, AttrWorkflowRoute.String(routeAttr))
            }()
            // ... execute cardStore.HybridSearch(spanCtx, filter) ...
            span.SetAttributes(AttrSearchMatches.Int(len(cards)))
            // ...
        },
        workflow.NodeConfig{},
    )
}

When a user uploads a business card or performs a hybrid search, Cloud Trace automatically draws a hierarchical Span tree like the one below, including the time spent in the three child nodes of the parallel Fan-Out and the business attributes of each node:

flowchart TD
    Root["invoke_workflow (line_card_assistant)<br/>trace_id: 4bf92f3577b34da6a3ce929d0e0e4736"] --> R["workflow.node.RouterNode<br/>adk.workflow.route = 'ingest' (1.2ms)"]
    R --> V["workflow.node.VisionOCRNode<br/>adk.card.company = 'Taiwan AI Labs' (640ms)"]
    V --> C1["workflow.node.CleanAndDedupNode<br/>adk.card.is_duplicate = false (8.5ms)"]
    V --> C2["workflow.node.CompanyEnrichNode<br/>GoogleSearch Grounding (780ms)"]
    V --> C3["workflow.node.ContextMatchNode<br/>inferMetContext (310ms)"]
    C1 & C2 & C3 --> B["workflow.node.BuildPreviewNode<br/>Flex Bubble JSON (2.1ms)"]
    B --> H["workflow.node.HITLConfirmNode<br/>adk.hitl.interrupt_id = 'hitl_confirm_1'<br/>adk.hitl.action = 'pause' (0.8ms)"]

We also wrote automated unit tests in internal/workflow/telemetry_test.go using the OpenTelemetry SDK's tracetest.NewInMemoryExporter(), ensuring that "card recognition + HITL modification loop + confirmation save" and "hybrid search with 0 results automatically degrading to fuzzy search" 100% generate the expected Span nodes and attribute tags!

V. Step 4: Building a GitHub Actions CI/CD Pipeline Integrated with `agents-cli eval` Quality Gatekeeper (`.github/workflows/`)

With Terraform IaC and OpenTelemetry tracing in place, we officially embedded the agents-cli eval quality flywheel created in Phase 4 into the GitHub Actions CI/CD pipeline, forming a two-stage automated delivery:

flowchart LR
    subgraph PR["1. Pull Request Quality Gate (pr_checks.yaml)"]
        P1["go test -race -v ./...<br/>(Unit + OTel Span + Trajectory Tests)"] --> P2["go test -tags=e2e<br/>(A2A + /app-info E2E Tests)"]
        P2 --> P3["agents-cli eval run<br/>(4 major metrics must = 1.0000)"]
    end

    subgraph CD["2. Staging & Production CD (staging.yaml -> deploy-to-prod.yaml)"]
        S1["WIF Keyless Auth<br/>(google-github-actions/auth)"] --> S2["agents-cli deploy<br/>Deploy to Staging Agent Runtime"]
        S2 --> S3["Vegeta Load Test<br/>30s pressure test + report upload to GCS"]
        S3 --> S4["GitHub Environment<br/>Production Manual Approval Gate"]
        S4 --> S5["agents-cli deploy<br/>Deploy to Production Agent Runtime"]
    end

    PR -->|Merge to main| CD

1. PR Quality Gatekeeper (`.github/workflows/pr_checks.yaml`)

In .github/workflows/pr_checks.yaml, whenever a PR modifies app/**, internal/**, cmd/**, sessions/**, or eval/**, GitHub Actions will execute in order:

go test -race -v ./...: Verify all Go unit tests, cross-instance HITL recovery, OpenTelemetry Node Spans, and trajectory tests.
go test -v -tags=e2e ./e2e/integration/...: Start a real ADK Universal Launcher subprocess to verify /.well-known/agent-card.json, /a2a/invoke, and /apps/app/app-info endpoints.
agents-cli eval run Quality Gatekeeper: Execute 6 golden evaluation cases, verifying that the four major metrics (workflow_trajectory_accuracy, ocr_and_enrichment_quality, duplicate_and_hitl_precision, search_and_email_grounding) must maintain a perfect score!

2. Staging Automated Deployment and Vegeta Load Testing (`.github/workflows/staging.yaml`)

After the PR is merged into the main branch, .github/workflows/staging.yaml will:

Obtain short-term GCP credentials keylessly via Workload Identity Federation (WIF).
Call uvx google-agents-cli@1.10.0 deploy to deploy the latest commit to the Staging Vertex AI Agent Runtime, while mounting LINE_CHANNEL_SECRET and LINE_CHANNEL_ACCESS_TOKEN from Secret Manager and tagging OTEL_RESOURCE_ATTRIBUTES=service.version=$.
Start e2e/load_test/load_test.go (based on tsenart/vegeta/v12) to perform a 30-second continuous pressure test on the Staging Agent Runtime, checking HTTP 200 success rate, P50/P95/P99 latency, and 429 Too Many Requests ratio, and automatically archiving the results to gs://${LOGS_BUCKET_NAME_STAGING}/load-test-results/.
After the pressure test passes, automatically trigger .github/workflows/deploy-to-prod.yaml. After passing the Required Reviewers audit for the GitHub production environment, it is officially pushed to Production!

VI. Practical Verification: `agents-cli deploy --dry-run` and Full E2E Tests All Passed!

Before submitting the code as a PR, we fully executed three major verifications locally:

1. `agents-cli deploy --dry-run` Cloud Deployment Preflight

$ uvx google-agents-cli@1.10.0 deploy \
    --project line-vertex \
    --region asia-east1 \
    --dry-run \
    --no-confirm-project

Preflight output results:

aiplatform.googleapis.com API: Enabled ✅
Target Agent Runtime Engine: line-card-assistant (asia-east1) ✅
IAM Roles & Service Account Preflight: Passed ✅

2. Go Unit Tests + OpenTelemetry Span Tests + E2E Integration Tests

$ go test -v -tags=e2e ./...

Includes the newly added TestWorkflowOpenTelemetrySpans_IngestAndHITL, TestWorkflowOpenTelemetrySpans_SearchFallback, and all tests under e2e/integration such as TestA2AAgentCard, TestA2AMessageSend, TestA2AErrorHandling, and TestAppInfo — all PASS!

3. Phase 4 `agents-cli eval` Quality Flywheel Regression Verification

$ DRY_RUN=true make eval

The 6 golden cases across the 4 custom metrics (workflow_trajectory_accuracy, ocr_and_enrichment_quality, duplicate_and_hitl_precision, search_and_email_grounding) still maintain 1.0000 (100% Pass)!

VII. "Agent Platform in Action" Series Summary: A Complete Five-Stage Evolution Map

With the completion of Phase 5 (Issue #8), the five stages planned in Roadmap Issue #9 have been 100% achieved and closed! Let's conclude with a summary table reviewing this journey from 0 to 1 with Google ADK 2.0 Go + Google Cloud Agent Platform:

Stage	Corresponding Issue / PR	Core Technical Breakthroughs	Corresponding Blog Post
Prototype	Initial Commit	Go `adk/v2/workflow` 12-node Graph Workflow, parallel `AddFanOut`/`AddFanIn`, `NewRequestInputEvent` HITL loop	In Action (1)
Phase 1	Issue #4 / PR #10	`agents-cli` (`adk_go`) standardization, `app.NewRootAgent` dual-channel factory, ADK WebUI Playground, and A2A protocol support	In Action (2)
Phase 2 (Part 1)	Issue #5 / PR #11	`artifact/gcsartifact` zero-state image bloat, `session/vertexai` cross-instance stateless HITL pause and resume	In Action (3)
Phase 2 (Part 2) & Phase 3	Issue #5, #6 / PR #12	Cloud SQL PostgreSQL (`PostgresStore`) card persistence, Cloud Run LINE Webhook Gateway (`internal/gateway`) and Agent Runtime dual-layer decoupling	In Action (4)
Phase 4	Issue #7 / PR #13	`agents-cli eval` quality flywheel, `/app-info` reflection introspection, 6 golden cases, and `setNodeTrace` trajectory repair (all 4 metrics at `1.0000`)	In Action (5)
Phase 5	Issue #8 / PR #14	`agents-cli deploy`, Terraform IaC (`deployment/terraform/`), GitHub Actions CI/CD + WIF + Vegeta pressure testing, OpenTelemetry node-level Cloud Trace & BigQuery observation	This article (In Action 6)

The complete open-source code, Terraform modules, and evaluation datasets have been synchronized to the GitHub Repo: kkdai/line-card-assistant. If you are also building enterprise-grade AI Agents with Go and Google ADK 2.0, I hope this series serves as your best practical blueprint for moving from local development all the way to the Google Cloud Agent Platform production environment!

[Agent Platform in Action (3)] Solving Stateless Container HITL Amnesia: Cross-Instance Business Card Recognition and Pause/Resu

Evan Lin — Fri, 23 Oct 2026 08:33:24 +0000

Recap

In the previous two articles of this series:

"[Agent Platform in Action (1)] Building a LINE AI Business Card King with Go and Google ADK 2.0 Graph Workflow": We used google.golang.org/adk/v2/workflow to implement a 12-node business card assistant featuring a Deterministic Router, AddFanOut / AddFanIn three-way parallel background checks, and NewRequestInputEvent Human-in-the-Loop (HITL) pause and loop correction.
"[Agent Platform in Action (2)] From Monolithic Cloud Run to Google Cloud Agent Runtime: Refactoring LINE AI Business Card King with agents-cli and ADK Go": We introduced the agents-cli official development lifecycle, explained the Agent Runtime's Scale-to-Zero mechanism, and completed Phase 1 (PR #10), transforming the project into a standard adk_go dual-channel architecture supporting LINE Webhook, ADK WebUI Playground, and A2A protocols.

Entering Phase 2 (Issue #3 & Issue #4), we face a classic and common pitfall in distributed systems when pushing to production (Cloud Run / Agent Runtime auto-scaling clusters): When containers can Scale to Zero or be load-balanced to different instances at any time, where should multimodal business card photos (Binary Images) be stored? And how can the HITL pause state, waiting for a user to click a button, be recovered across instances?

In the third installment of this series, we will deep dive into how we used ADK Go v2.6.0's artifact/gcsartifact (GCS Artifact Service) and session/vertexai (Agent Platform Sessions) in PR #11 to completely cure the "HITL amnesia" of stateless containers!

1. Why did the original HITL suffer from "Amnesia" and "State Bloat" in cloud multi-instance environments?

During local development or with a single container instance, stuffing everything into memory is fast. However, once deployed to Google Cloud Run or Vertex AI Agent Runtime, which support Horizontal Auto-scaling or Scale-to-Zero, the first version's design immediately hits two major walls:

Pain Point 1: The "State Bloat" anti-pattern of stuffing multi-MB business card `[]byte` into `Session.State`

In the first version, when a LINE user sent a high-resolution business card photo, we directly put the downloaded imageBytes []byte into ADK's Session.State["image_bytes"]:

// ❌ Version 1 approach: Directly stuffing multi-MB raw image []byte into Session State
InitialState: map[string]any{
    StateKeyInputType: RouteIngest,
    StateKeyImageBytes: imageBytes, // A 3MB JPEG inflates to a ~4MB Base64 string after JSON serialization!
}

When using the local session.InMemoryService(), this is just a Go slice pointer. But when switching to cloud-persisted Agent Platform Sessions (session/vertexai) or Cloud SQL (session/database):

Serialization and network transfer for every node state change (StateDelta): As the ADK Graph Workflow moves from RouterNode to VisionOCRNode, fans out to three parallel nodes, and finally pauses at HITLConfirmNode, keeping multi-MB image binary data in Session.State significantly slows down every Cloud Session API read/write and might even trigger Cloud Session payload size limits!
Difficulty in auditing and versioning history: If a user sends multiple business cards across several turns, mixing raw binary files with structured JSON state makes the conversation state extremely bloated.

Pain Point 2: Container instance switching leads to loss of HITL `pendingSession` memory mapping

Let's look at a real-world user scenario:

Turn 1 (hits Container Instance A): A user takes a photo of a business card at a conference and sends it to the LINE Bot. Instance A completes Vision OCR and parallel background checks, then pauses at HITLConfirmNode via workflow.NewRequestInputEvent, pushing a Flex Message preview card to the user.
10-minute break: The user goes to exchange thoughts with the contact and grabs a coffee. During this time, Instance A scales to 0 due to inactivity, or Container Instance B is scaled out due to other traffic.
Turn 2 (hits Container Instance B): The user clicks "✏️ Edit Fields" on LINE and enters "Change title to CTO", or clicks "✅ Confirm Save".

In the first version, our Engine used an in-memory Go map[string]pendingHITL to track which userID was waiting for which SessionID and InterruptID:

// ❌ Version 1 approach: Relying solely on a local in-memory map to determine if there's a pending HITL
type Engine struct {
    mu sync.RWMutex
    pendingSession map[string]pendingHITL // userID -> {SessionID, InterruptID}
}

As a result, when the Turn 2 Webhook request is routed by the load balancer to the brand new Instance B, Instance B's pendingSession map is empty! HasPendingHITL(userID) returns false, and the system treats "Change title to CTO" as a brand new "Business Card Search" request, instantly breaking the HITL loop!

2. The Standard ADK 2.0 Solution: Separation of Concerns between Session State and Artifact Storage

To solve this, Google ADK 2.0's architectural design explicitly splits Agent runtime data into two independent services:

Comparison Dimension	Session Service (`session.Service`)	Artifact Service (`artifact.Service`)
Core Purpose	Stores conversation event logs (Events), workflow breakpoints (Interrupts), and lightweight structured state (JSON State)	Stores large binary files (images, PDFs, audio, CSV reports) and provides automatic version control (Versioning)
Cloud Implementation Package	`google.golang.org/adk/v2/session/vertexai` (Agent Platform Sessions) or `session/database`	`google.golang.org/adk/v2/artifact/gcsartifact` (underlying storage in Google Cloud Storage Bucket)
Local Development Implementation	`session.InMemoryService()`	`artifact.InMemoryService()`
Role in Business Card King	Stores `draft_card` JSON, `active_interrupt_id`, and lightweight pointers (filename + version number) to the image	Stores raw business card photo `business_card.jpg` (automatically increments Version `1, 2, 3...` per upload)

With this dual-track design, the entire cross-instance multimodal business card recognition and HITL recovery process becomes crystal clear:

sequenceDiagram
    autonumber
    actor User as LINE / WebUI User
    participant InstA as Container Instance A (Turn 1: Upload Card)
    participant GCS as GCS Artifact Service<br/>(artifact/gcsartifact)
    participant Sess as Agent Platform Sessions<br/>(session/vertexai)
    participant InstB as Container Instance B (Turn 2: Edit Fields)
    participant InstC as Container Instance C (Turn 3: Confirm Save)

    User->>InstA: 1. Upload business card photo (JPEG []byte)
    InstA->>GCS: 2. Save("business_card.jpg", inlineData)<br/>Returns Version = 1
    InstA->>Sess: 3. Create Session, State only records:<br/>card_image_artifact="business_card.jpg"<br/>card_image_version=1
    InstA->>GCS: 4. VisionOCRNode calls LoadVersion("business_card.jpg", 1)
    Note over InstA,Sess: Fan-Out Parallel duplicate check / Company search / Interactive retrieval -> Generate preview card
    InstA->>Sess: 5. HITLConfirmNode triggers NewRequestInputEvent<br/>Persists active_interrupt_id and draft_card
    InstA-->>User: 6. Return LINE Flex preview card (⏸️ Workflow Paused)

    Note over InstA,InstB: ☁️ Container scales to 0 or load balancer switches to a brand new stateless Instance B

    User->>InstB: 7. Input: "Change title to CTO"
    InstB->>Sess: 8. resolvePendingHITL(userID):<br/>Query latest Session's active_interrupt_id
    Sess-->>InstB: 9. Retrieve {SessionID, InterruptID} and draft_card
    InstB->>Sess: 10. ResumeHITL -> ApplyEditNode updates title -> Re-pauses and updates active_interrupt_id
    InstB-->>User: 11. Return updated Flex preview card (Title: CTO)

    Note over InstB,InstC: ☁️ Switch again to brand new stateless Instance C

    User->>InstC: 12. Click "✅ Confirm Save" (action=confirm_save)
    InstC->>Sess: 13. resolvePendingHITL(userID) retrieves latest InterruptID
    InstC->>Sess: 14. ResumeHITL -> SaveCardNode writes to card library + clears active_interrupt_id
    InstC-->>User: 15. Return save success and AI icebreaker draft!

3. Code Implementation Breakdown (Phase 2 — PR #11)

Now let's look at the code to see how we completed this upgrade with less than a few hundred lines of clean Go code.

1. Creating a Multi-backend Artifact Factory: `sessions/artifacts.go`

We added sessions/artifacts.go under the sessions package, using ADK Go v2.6.0's built-in google.golang.org/adk/v2/artifact/gcsartifact package to automatically switch between Cloud GCS Bucket and local memory based on environment variables:

package sessions

import (
    "context"
    "log/slog"
    "os"
    "strings"

    "google.golang.org/adk/v2/artifact"
    "google.golang.org/adk/v2/artifact/gcsartifact"
)

// NewArtifactService returns an ADK artifact.Service backed by Google Cloud Storage
// when ARTIFACT_BUCKET_NAME or LOGS_BUCKET_NAME is configured, and falls back to
// artifact.InMemoryService() for local development and unit tests.
func NewArtifactService(ctx context.Context) (artifact.Service, error) {
    bucketName := resolveArtifactBucketName()
    if bucketName == "" {
        slog.Info("Using in-memory artifact service (set ARTIFACT_BUCKET_NAME or LOGS_BUCKET_NAME for GCS)")
        return artifact.InMemoryService(), nil
    }

    svc, err := gcsartifact.NewService(ctx, bucketName)
    if err != nil {
        return nil, err
    }
    slog.Info("Using GCS artifact service", "bucket", bucketName)
    return svc, nil
}

A thoughtful detail: In the agents-cli Agent Runtime deployment environment, the system usually configures LOGS_BUCKET_NAME (or our custom ARTIFACT_BUCKET_NAME). Some might accidentally fill it in with a format like gs://my-bucket/prefix. Our resolveArtifactBucketName() automatically strips the gs:// prefix and path to ensure gcsartifact.NewService connects reliably, falling back to artifact.InMemoryService() if no bucket is set.

Simultaneously, we inject ArtifactService and SessionService into launcher.Config in main.go and cmd/server/main.go:

// main.go — Injecting both SessionService and ArtifactService into ADK Universal Launcher
config := &launcher.Config{
    SessionService: sessionService,
    ArtifactService: artifactService,
    AgentLoader: NewSingleAgentLoader(rootAgent),
    A2AOptions: a2aOpts,
}

2. Storing Business Card Images in Artifacts, Keeping only "Lightweight Pointers" in Session State

Next, we refactor internal/workflow/graph.go and internal/workflow/nodes_ingest.go.

First, we define two new lightweight State Keys and a default filename:

const (
    StateKeyImageBytes = "image_bytes" // Kept only for backward compatibility fallback
    StateKeyCardImageArtifact = "card_image_artifact" // Artifact filename
    StateKeyCardImageVersion = "card_image_version" // Artifact version number (int)

    DefaultCardImageArtifactName = "business_card.jpg"
)

(1) Entering from LINE Webhook (`Engine.StartIngest`)

When a LINE user sends an image, StartIngest first calls e.artifactService.Save to write the image binary to the Artifact service and only records the filename and version number in InitialState. StateKeyImageBytes is no longer written to Session.State:

initialState := map[string]any{
    StateKeyInputType: RouteIngest,
    StateKeyRecentEvents: recentEvents,
}

if e.artifactService != nil && len(imageBytes) > 0 {
    part := genai.NewPartFromBytes(imageBytes, detectImageMIMEType(imageBytes))
    saveResp, saveErr := e.artifactService.Save(ctx, &artifact.SaveRequest{
        AppName: appName,
        UserID: userID,
        SessionID: sess.ID(),
        FileName: DefaultCardImageArtifactName,
        Part: part,
    })
    if saveErr == nil && saveResp != nil {
        initialState[StateKeyCardImageArtifact] = DefaultCardImageArtifactName
        initialState[StateKeyCardImageVersion] = saveResp.Version
    }
}

(2) Uploading images directly from ADK WebUI / Agent Engine Playground / A2A (`RouterNode`)

What if the user isn't using LINE but uploads a business card photo directly via ADK WebUI (make playground) or Agent Engine Console? In RouterNode, we use ADK 2.0's ctx.Artifacts().Save(...) to transfer InlineData from ctx.UserContent() into the Artifact Service:

// internal/workflow/nodes_ingest.go — RouterNode
if getStringState(ctx, StateKeyCardImageArtifact) == "" && len(getBytesState(ctx, StateKeyImageBytes)) == 0 && ctx.UserContent() != nil {
    for _, part := range ctx.UserContent().Parts {
        if part != nil && part.InlineData != nil && len(part.InlineData.Data) > 0 {
            if ctx.Artifacts() != nil {
                mimeType := part.InlineData.MIMEType
                if mimeType == "" {
                    mimeType = detectImageMIMEType(part.InlineData.Data)
                }
                artifactPart := genai.NewPartFromBytes(part.InlineData.Data, mimeType)
                if saveResp, saveErr := ctx.Artifacts().Save(ctx, DefaultCardImageArtifactName, artifactPart); saveErr == nil {
                    setState(ctx, ev, StateKeyCardImageArtifact, DefaultCardImageArtifactName)
                    setState(ctx, ev, StateKeyCardImageVersion, saveResp.Version)
                } else {
                    setState(ctx, ev, StateKeyImageBytes, part.InlineData.Data)
                }
            }
            if getStringState(ctx, StateKeyInputType) == "" {
                setState(ctx, ev, StateKeyInputType, RouteIngest)
            }
            break
        }
    }
}

(3) `VisionOCRNode` Reads specific versions of business card images on demand (`loadCardImageBytes`)

When the workflow enters VisionOCRNode to call Gemini Vision for multimodal recognition, it uses ctx.Artifacts().LoadVersion (or Load) to load the image into memory for the model. It's discarded after use, keeping Session.State clean:

func loadCardImageBytes(ctx agent.Context) []byte {
    if artifactName := getStringState(ctx, StateKeyCardImageArtifact); artifactName != "" && ctx.Artifacts() != nil {
        version := getIntState(ctx, StateKeyCardImageVersion)
        var (
            resp *artifact.LoadResponse
            err error
        )
        if version > 0 {
            resp, err = ctx.Artifacts().LoadVersion(ctx, artifactName, version)
        } else {
            resp, err = ctx.Artifacts().Load(ctx, artifactName)
        }
        if err == nil && resp != nil && resp.Part != nil && resp.Part.InlineData != nil {
            return resp.Part.InlineData.Data
        }
    }
    return getBytesState(ctx, StateKeyImageBytes) // Fallback
}

3. Implementing Stateless Cross-Instance HITL State Discovery: `resolvePendingHITL`

With image storage solved, here's the main event: When Container Instance B receives a user's "Change title to CTO" or "✅ Confirm Save", how does it automatically retrieve the HITL workflow paused by Instance A when its local memory is empty?

The secret is: when HITLConfirmNode calls workflow.NewRequestInputEvent to pause, we've already written the current InterruptID into Session.State["active_interrupt_id"] (and SaveCardNode clears it to "" once saving is complete).

Therefore, we implemented resolvePendingHITL(ctx, userID) in internal/workflow/graph.go:

// resolvePendingHITL checks the local cache first and falls back to querying
// the shared session.Service for any session belonging to userID with a non-empty
// StateKeyActiveInterruptID. This enables stateless cross-instance HITL resume
// on Cloud Run / Vertex AI Agent Runtime.
func (e *Engine) resolvePendingHITL(ctx context.Context, userID string) (pendingHITL, bool) {
    // 1. Fast path: If a record exists in the same instance's memory, verify with sessionService if it's still paused
    e.mu.RLock()
    cached, ok := e.pendingSession[userID]
    e.mu.RUnlock()
    if ok && cached.SessionID != "" && cached.InterruptID != "" {
        if getResp, err := e.sessionService.Get(ctx, &session.GetRequest{
            AppName: appName,
            UserID: userID,
            SessionID: cached.SessionID,
        }); err == nil && getResp != nil && getResp.Session != nil {
            if activeID, err := getResp.Session.State().Get(StateKeyActiveInterruptID); err == nil {
                if s, ok := activeID.(string); ok && strings.TrimSpace(s) != "" {
                    cached.InterruptID = strings.TrimSpace(s)
                    return cached, true
                }
            }
        }
    }

    // 2. Stateless Slow Path: When hitting a brand new container instance, query the shared sessionService directly!
    listResp, err := e.sessionService.List(ctx, &session.ListRequest{
        AppName: appName,
        UserID: userID,
    })
    if err != nil || listResp == nil {
        return pendingHITL{}, false
    }

    var (
        matched pendingHITL
        latestTime time.Time
        foundActive bool
    )
    for _, sess := range listResp.Sessions {
        if sess == nil || sess.State() == nil {
            continue
        }
        val, err := sess.State().Get(StateKeyActiveInterruptID)
        if err != nil || val == nil {
            continue
        }
        interruptID, ok := val.(string)
        interruptID = strings.TrimSpace(interruptID)
        if !ok || interruptID == "" {
            continue
        }
        updatedAt := sess.LastUpdateTime()
        if !foundActive || updatedAt.After(latestTime) {
            foundActive = true
            latestTime = updatedAt
            matched = pendingHITL{
                SessionID: sess.ID(),
                InterruptID: interruptID,
            }
        }
    }

    if foundActive {
        e.mu.Lock()
        e.pendingSession[userID] = matched
        e.mu.Unlock()
        return matched, true
    }
    return pendingHITL{}, false
}

Behold the power of this design:

Regardless of which stateless instance the user's request is dispatched to by the Cloud Run or Agent Runtime load balancer, HasPendingHITL(userID) and ResumeHITL(ctx, userID, userInput) can retrieve the user's latest paused {SessionID, InterruptID} directly via the shared session.Service.
Then, Instance B simply wraps the user's input into an ADK 2.0 standard adk_request_input (workflow.WorkflowInputFunctionCallName) FunctionResponse and calls e.runner.Run to seamlessly wake the workflow from HITLConfirmNode!

4. Writing Unit Tests to Verify "Across Three Container Instances"!

To prove this architecture works in a multi-instance environment with no shared local memory, we wrote the TestCrossInstanceHITLResumeAndArtifactStorage test in internal/workflow/graph_test.go, creating 3 independent Engine instances (instanceA, instanceB, instanceC) to simulate three stateless containers:

func TestCrossInstanceHITLResumeAndArtifactStorage(t *testing.T) {
    ctx := context.Background()
    cardStore := store.NewMemoryStore()
    mockLLM := NewMockLLM()

    // Simulating cloud-shared Agent Platform Session Service and GCS Artifact Service
    sharedSessionSvc := session.InMemoryService()
    sharedArtifactSvc := artifact.InMemoryService()

    // Creating 3 completely independent Engine instances (each with an independent and empty pendingSession map)
    instanceA, _ := NewEngine(cardStore, mockLLM,
        WithSessionService(sharedSessionSvc),
        WithArtifactService(sharedArtifactSvc),
    )
    instanceB, _ := NewEngine(cardStore, mockLLM,
        WithSessionService(sharedSessionSvc),
        WithArtifactService(sharedArtifactSvc),
    )
    instanceC, _ := NewEngine(cardStore, mockLLM,
        WithSessionService(sharedSessionSvc),
        WithArtifactService(sharedArtifactSvc),
    )

    userID := "U_cross_instance_user"
    fakeImage := []byte{0xFF, 0xD8, 0xFF, 0xE0, 0x00, 0x10, 'J', 'F', 'I', 'F'}

    // Turn 1 (hits Instance A): Upload card photo -> Write to Artifact and pause at HITLConfirmNode
    res1, _ := instanceA.StartIngest(ctx, userID, fakeImage, []string{"Met at Cloud Summit"})
    // Verify Session State has no bloated raw image_bytes, but lightweight card_image_artifact="business_card.jpg"

    // Turn 2 (hits brand new Instance B): User enters edit command 'Change title to CTO'
    // Instance B automatically detects the active_interrupt_id left by Instance A from sharedSessionSvc!
    res2, _ := instanceB.ResumeHITL(ctx, userID, "Change title to CTO")
    // Verify title updated to 'CTO' and re-entered HITL pause waiting for confirmation

    // Turn 3 (hits brand new Instance C): User clicks '✅ Confirm Save'
    res3, _ := instanceC.ResumeHITL(ctx, userID, "action=confirm_save")
    // Verify card successfully written to CardStore, AI icebreaker generated, and HasPendingHITL is zeroed for all instances!
}

Running go test -race -v ./...:

=== RUN TestCrossInstanceHITLResumeAndArtifactStorage
--- PASS: TestCrossInstanceHITLResumeAndArtifactStorage (0.00s)
=== RUN TestRootAgentDirectImageArtifactIngestAndHITL
--- PASS: TestRootAgentDirectImageArtifactIngestAndHITL (0.00s)
PASS
ok github.com/kkdai/line-card-assistant/internal/workflow   0.642s

Not only was the three-stage relay from instanceA $\rightarrow$ instanceB $\rightarrow$ instanceC seamless, but TestRootAgentDirectImageArtifactIngestAndHITL in app/agent_test.go also verified that when a genai.Part image is passed directly from ADK WebUI / A2A, RouterNode automatically saves it to ArtifactService and pauses at HITLConfirmNode!

4. Summary and Next Steps

Through this practical guide (PR #11, completing Issue #3 and Issue #4 in the Roadmap), we've solved the two most critical persistence challenges for moving an AI Agent from a "toy prototype" to a "cloud stateless multi-instance production environment":

Separation of Large Files and State (Artifact vs. Session State): Using ADK Go v2.6.0's artifact/gcsartifact and ctx.Artifacts(), business card photos no longer bloat Session.State. Instead, they have independent GCS versioned storage, with Session State only keeping a few dozen bytes of pointers (card_image_artifact + card_image_version).
Stateless Cross-Instance HITL Recovery: Combining session/vertexai (Agent Platform Sessions) and resolvePendingHITL, even if a container scales to 0 or switches instances during the minutes spent waiting for human confirmation, the system can still resume the "edit field loop" or "confirm save" perfectly.

In the next article, we will continue with cloud database persistence and a two-layer decoupled architecture, and use agents-cli eval to build an automated evaluation flywheel. Stay tuned!

Resources and Further Reading

Previous articles in the series:
- [Agent Platform in Action (1)] Building a LINE AI Business Card King with Go and Google ADK 2.0 Graph Workflow: From Fan-Out Parallel Retrieval to HITL Pause Loop Correction
- [Agent Platform in Action (2)] From Monolithic Cloud Run to Google Cloud Agent Runtime: Refactoring LINE AI Business Card King with agents-cli and ADK Go
Open source code and PRs for this project:
- GitHub Repository: kkdai/line-card-assistant
- Phase 2 Implementation Pull Request: PR #11: feat(phase2): integrate GCS Artifact Service and cross-instance HITL session recovery
- Completed Issues: Issue #3, Issue #4
Google ADK and Agent Platform Official Documentation:
- ADK Documentation (llms.txt)
- Getting Started - agents-cli

[Agent Platform in Practice (4)] Towards Cloud Two-layer Decoupling and Full Persistence: Completing the LINE AI Business Card M

Evan Lin — Fri, 23 Oct 2026 08:32:27 +0000

Recap

In the first three articles of the "Agent Platform in Action" series, we step-by-step pushed the LINE AI Business Card Assistant (line-card-assistant) from a conceptual prototype to a cloud-native architecture:

[Agent Platform in Action (1)] Building a LINE AI Business Card Assistant with Go and Google ADK 2.0 Graph Workflow: We used google.golang.org/adk/v2/workflow to build a 12-node business card processing graph (Graph Workflow), covering Vision OCR, AddFanOut / AddFanIn parallel background investigations, and the HITL (Human-in-the-Loop) pause and modification loop of NewRequestInputEvent.
[Agent Platform in Action (2)] From Monolithic Cloud Run to Google Cloud Agent Runtime: Refactoring LINE AI Business Card Assistant with agents-cli and ADK Go: Completed Phase 1 (PR #10), introducing the agents-cli (adk_go) standard dual-channel architecture, allowing the project to support LINE Webhook, ADK WebUI Playground, and the A2A protocol simultaneously.
[Agent Platform in Action (3)] Solving HITL Amnesia in Stateless Containers: Implementing Cross-Instance Business Card Recognition and Pause/Resume with ADK 2.0 GCS Artifact Service and Agent Platform Sessions: Completed the first half of Phase 2 (PR #11), using artifact/gcsartifact to move multi-MB business card photos out of Session.State, and combining session/vertexai to implement resolvePendingHITL, allowing stateless containers to seamlessly continue HITL modification and saving even if scaled to 0 or instances are switched.

In today's fourth episode of the series, we will complete the final piece of Phase 2 (Issue #5) and the Phase 3 core architecture upgrade (Issue #6) (corresponding to PR #12):

Cloud Business Card Database Persistence (PostgresStore / Cloud SQL): Officially upgrading the business card library, which was originally stored in local memory or container temporary files, to a relational and JSON hybrid storage supporting Cloud SQL PostgreSQL, and enabling HybridSearch and token-scoring fault-tolerant BroadSearch.
Two-Tier Decoupled Call Architecture (Cloud Run LINE Webhook Gateway $\rightarrow$ Agent Runtime): Completely decoupling the lightweight edge layer (Edge Layer) responsible for "LINE verification and image downloading" from the execution layer (Execution Layer) responsible for "ADK 2.0 Graph Workflow multimodal inference and search"!

1. Why Do We Need the "Cloud Persistence Iron Triangle" and "Two-Tier Decoupled Architecture"?

When we deploy line-card-assistant to a Google Cloud production environment, a careful examination of the responsibilities of every piece of data and component in the system reveals two key problems that must be solved:

1. The Last Remaining "Local State": `CardStore` Business Card Database

In the previous post (PR #11), we already placed the "original business card photos" in Google Cloud Storage (GCSArtifactService) and the "workflow execution state and HITL breakpoints" in Vertex AI / Cloud SQL (SessionService). However, when a user clicks "✅ Confirm Save" in LINE and reaches SaveCardNode, if the final confirmed business card data (models.BusinessCard) is still written to local memory MemoryStore or a local SQLite file inside the container:

Cleared on container restart or Scale-to-Zero: Once the Agent Runtime instance scales to 0, the hard-earned network business card library will evaporate instantly.
Multiple instances cannot share search results: When a user saves a business card on Instance A, and a search request is routed to Instance B a second later, HybridSearchNode won't find the card just saved in Instance B's memory!

Therefore, we must complete the final corner of the "Cloud-Native Persistence Triad" — Cloud SQL PostgreSQL (PostgresStore):

Persistence Dimension	Content Responsible for Storing	Cloud Backend Service	ADK / Go Implementation Package
1. Binary Large Files (Artifacts)	Original business card photos (`business_card.jpg` v1, v2…)	Google Cloud Storage (GCS)	`sessions.NewArtifactService` (`artifact/gcsartifact`)
2. Conversation and Graph Breakpoints (Sessions)	Graph Checkpoints, `active_interrupt_id`, `draft_card`	Vertex AI Agent Engine / Cloud SQL	`sessions.NewService` (`session/vertexai`, `session/database`)
3. Structured Business Data (Domain DB)	Verified `BusinessCard` official records, JSON industry tags, hybrid retrieval	Cloud SQL for PostgreSQL	`store.NewStore` (`internal/store/postgres.go`)

2. Why Decouple LINE Webhook Gateway and Agent Runtime into a Two-Tier Architecture?

In monolithic (All-in-One) mode, cmd/server receives LINE Webhooks, runs Vision OCR, and executes three parallel Grounding search nodes in the same process. However, in an enterprise-grade Google Cloud Agent Platform architecture, splitting the system into "Cloud Run LINE Webhook Gateway (Edge Layer)" and "Google Cloud Agent Runtime (Agent Execution Layer)" offers three major advantages:

Isolation of Public Webhook Verification and Internal IAM Permissions:
- LINE Messaging API requires a public HTTPS Endpoint (POST /webhook) to push events, uses LINE_CHANNEL_SECRET to verify the X-Line-Signature, and must immediately return 200 OK within 1-2 seconds.
- Meanwhile, the actual Agent Runtime only needs internal authorization via Google Cloud IAM / Application Default Credentials (ADC), without needing exposure to unauthenticated public traffic or holding the LINE Channel Secret.
Multiple Channels Sharing the Same Session-backed Remote Agent:
- After decoupling, whether a user sends a business card photo from LINE Bot, queries contacts from internal Gemini Enterprise (A2A), or a developer tests from ADK WebUI, they all call the same app.NewRootAgent hosted on the Agent Runtime!
Extreme Deployment Flexibility (One-click switch between All-in-One vs. Decoupled):
- During local development or small projects, if AGENT_RUNTIME_URL is not set, cmd/server automatically runs in All-in-One monolithic mode.
- In production, simply setting AGENT_RUNTIME_URL=https://... in Cloud Run automatically switches cmd/server to a lightweight stateless LINE Webhook Gateway!

Let's look at the cross-layer interaction under the two-tier decoupled architecture with a complete system sequence diagram:

sequenceDiagram
    autonumber
    actor User as LINE User
    participant GW as Cloud Run<br/>LINE Webhook Gateway<br/>(cmd/server + gateway.Client)
    participant AR as Google Cloud Agent Runtime<br/>(adkrest.Server / app.NewRootAgent)
    participant GCS as Cloud Storage<br/>(GCSArtifactService)
    participant Sess as Vertex AI / Cloud SQL<br/>(SessionService)
    participant PG as Cloud SQL PostgreSQL<br/>(PostgresStore)

    User->>GW: 1. Send business card photo (POST /webhook)
    GW-->>User: 2. Verify X-Line-Signature and immediately return HTTP 200 OK
    GW->>GW: 3. Download image from LINE Blob API []byte
    GW->>AR: 4. POST /api/apps/app/users/{user}/sessions/{id}<br/>+ POST /api/run (With Google Cloud ADC ID Token and image InlineData)
    AR->>GCS: 5. RouterNode automatically saves image to GCS Artifact ("business_card.jpg" v1)
    AR->>AR: 6. VisionOCRNode -> AddFanOut (Deduplication / Google Search company background / Conversation context) -> BuildPreviewNode
    AR->>Sess: 7. HITLConfirmNode triggers NewRequestInputEvent<br/>Persist active_interrupt_id="hitl_confirm_1"
    AR-->>GW: 8. Return Events and Session State (including flex_preview_json)
    GW-->>User: 9. Push LINE Flex Message business card preview card (⏸️ Waiting for user confirmation)

    Note over GW,AR: ☁️ Gateway is completely stateless; the next request can land on any Gateway instance

    User->>GW: 10. Input modification: "Change title to CTO" or click "✅ Confirm Save"
    GW->>AR: 11. HasPendingHITL -> GET /api/apps/app/users/{user}/sessions<br/>Automatically retrieve remote paused {SessionID, active_interrupt_id}
    GW->>AR: 12. POST /api/run (Send adk_request_input FunctionResponse)
    AR->>PG: 13. If confirming save: SaveCardNode calls PostgresStore.Save() Upsert to write to Cloud SQL
    AR->>Sess: 14. Clear active_interrupt_id="" and record saved_card_id
    AR-->>GW: 15. Return save success and AI icebreaker follow-up draft
    GW-->>User: 16. Push save confirmation and icebreaker to user!

2. Implementation Breakdown (1): Building the Cloud SQL PostgreSQL Business Card Library (`internal/store/postgres.go`) — Issue #5

In our project, since sessions/sessions.go already introduced gorm.io/gorm and gorm.io/driver/postgres to support SESSION_TYPE=cloud_sql, we can directly use GORM to build a production-grade PostgresStore for CardStore without extra dependency bloat!

1. Extending the `store.CardStore` Interface: Native `BroadSearch` Support for All Backends

While checking the original internal/workflow/nodes_search.go, we found a type assertion for *store.MemoryStore in BroadSearchNode:

// ❌ Old way: Only MemoryStore supports BroadSearch; other Stores fall back to ListByUser
if memStore, ok := cardStore.(*store.MemoryStore); ok {
    cards, err = memStore.BroadSearch(ctx, userID, keywords)
} else {
    cards, err = cardStore.ListByUser(ctx, userID)
}

To ensure PostgresStore and MemoryStore share a consistent hybrid retrieval and fuzzy fallback experience, we officially included BroadSearch in the CardStore standard interface in internal/store/store.go:

// CardStore defines the persistence and search interface for business cards.
type CardStore interface {
    Save(ctx context.Context, card *models.BusinessCard) error
    GetByID(ctx context.Context, id string) (*models.BusinessCard, error)
    CheckDuplicate(ctx context.Context, userID, name, company, email string) (*models.BusinessCard, bool, error)
    HybridSearch(ctx context.Context, filter models.SearchFilter) ([]*models.BusinessCard, error)
    BroadSearch(ctx context.Context, userID string, keywords []string) ([]*models.BusinessCard, error)
    ListByUser(ctx context.Context, userID string) ([]*models.BusinessCard, error)
}

This way, BroadSearchNode only needs to cleanly call one line: cardStore.BroadSearch(ctx, userID, keywords)!

2. Designing GORM Schema and Upsert Mechanism (`BusinessCardRecord`)

In internal/store/postgres.go, we defined BusinessCardRecord corresponding to the PostgreSQL business_cards table, added indexes to common query fields (user_id, name, company, email, met_date, updated_at), and stored Tags []string as a JSON string in tags_json:

// BusinessCardRecord is the GORM database model for persisting a models.BusinessCard
// in PostgreSQL / Cloud SQL.
type BusinessCardRecord struct {
    ID string `gorm:"primaryKey;size:128"`
    UserID string `gorm:"index;size:128;not null"`
    Name string `gorm:"index;size:255"`
    Title string `gorm:"size:255"`
    Company string `gorm:"index;size:255"`
    Email string `gorm:"index;size:255"`
    Phone string `gorm:"size:64"`
    Address string `gorm:"type:text"`
    TaxID string `gorm:"size:32"`
    Website string `gorm:"size:512"`
    RawOCR string `gorm:"type:text"`
    CompanySummary string `gorm:"type:text"`
    CompanyInfo string `gorm:"type:text"`
    TagsJSON string `gorm:"column:tags_json;type:text"`
    MetContext string `gorm:"type:text"`
    RawImageURL string `gorm:"type:text"`
    MetDate time.Time `gorm:"index"`
    CreatedAt time.Time `gorm:"index"`
    UpdatedAt time.Time `gorm:"index"`
}

When a user takes a photo of an existing contact's business card (e.g., the person was promoted from "Senior Engineer" to "VP of Engineering", and CleanAndDedupNode identifies the duplicate and reuses the original card.ID), PostgresStore.Save performs an atomic Upsert using PostgreSQL's native ON CONFLICT (id) DO UPDATE (clause.OnConflict{UpdateAll: true}):

if err := s.db.WithContext(ctx).Clauses(clause.OnConflict{
    Columns: []clause.Column,
    UpdateAll: true,
}).Create(rec).Error; err != nil {
    return fmt.Errorf("upsert business card %q: %w", cp.ID, err)
}

3. The `store.NewStore(ctx)` Factory: Auto-sensing Cloud SQL Unix Sockets and Environment Variables

When connecting to Cloud SQL from Cloud Run or Agent Runtime, the official recommendation is to use the Unix Domain Socket (/cloudsql/<PROJECT>:<REGION>:<INSTANCE>) established by the Cloud SQL Auth Proxy; locally or on GKE, DATABASE_URL or DB_HOST are common. We support all three modes in BuildPostgresDSNFromEnv() and store.NewStore(ctx), and automatically fall back seamlessly to MemoryStore preloaded with 4 Taiwan sample business cards when no database environment variables are set:

// BuildPostgresDSNFromEnv resolves a PostgreSQL / Cloud SQL connection string from environment variables.
// Priority:
// 1. CARD_STORE_DSN / DATABASE_URL / POSTGRES_DSN
// 2. Cloud SQL Unix socket when INSTANCE_CONNECTION_NAME + DB_PASS are set
// 3. Standard TCP host when DB_HOST + DB_PASS are set
func BuildPostgresDSNFromEnv() string {
    for _, key := range []string{"CARD_STORE_DSN", "DATABASE_URL", "POSTGRES_DSN"} {
        if v := strings.TrimSpace(os.Getenv(key)); v != "" {
            return v
        }
    }

    dbUser := envOrDefault("DB_USER", "postgres")
    dbName := envOrDefault("DB_NAME", "postgres")
    dbPass := strings.TrimSpace(os.Getenv("DB_PASS"))
    instanceConn := strings.TrimSpace(os.Getenv("INSTANCE_CONNECTION_NAME"))
    if instanceConn != "" && dbPass != "" {
        return fmt.Sprintf(
            "host=/cloudsql/%s user=%s password=%s dbname=%s sslmode=disable",
            instanceConn, dbUser, dbPass, dbName,
        )
    }
    // ...
}

This way, as long as your environment has INSTANCE_CONNECTION_NAME and DB_PASS set for SESSION_TYPE=cloud_sql, SessionService and CardStore can share the same Cloud SQL PostgreSQL instance, without spending an extra penny on database costs!

3. Implementation Breakdown (2): Building a Stateless LINE Webhook Gateway and Remote `AgentRuntimeClient` (`internal/gateway/client.go`) — Issue #6

Next is the core engineering of Phase 3: How to make the cmd/server LINE Bot Webhook Handler forward all business card recognition, HITL recovery, search, and icebreaker requests to a remote Agent Runtime via HTTP instead of running workflow.Engine locally?

Remember how we had the foresight to separate the Webhook Handler and Workflow Engine with a linebot.WorkflowRunner interface in the first version of internal/linebot/handler.go?

// internal/linebot/handler.go
type WorkflowRunner interface {
    HasPendingHITL(userID string) bool
    RecordUserMessage(userID, text string)
    StartIngest(ctx context.Context, userID string, imageBytes []byte, recentMessages []string) (*WorkflowResponse, error)
    ResumeHITL(ctx context.Context, userID string, userReply string) (*WorkflowResponse, error)
    RunSearch(ctx context.Context, userID string, query string) (*WorkflowResponse, error)
    RunEmailDraft(ctx context.Context, userID string, cardID string) (*WorkflowResponse, error)
}

Because of this clean interface, we don't need to change a single line of code in linebot.Handler! We just need to implement a gateway.Client that satisfies the linebot.WorkflowRunner interface in internal/gateway/client.go!

1. `gateway.NewClient` Supporting Automatic Google Cloud ADC Token Issuance

When the LINE Webhook Gateway on Cloud Run needs to call an IAM-protected remote Agent Runtime (or another ADK REST Server on an internal Cloud Run), gateway.NewClient automatically obtains the current Service Account's Google Cloud ADC credentials via google.golang.org/api/idtoken (or golang.org/x/oauth2/google) and attaches them to the Authorization: Bearer <token> header of every HTTP request:

func NewClient(ctx context.Context, baseURL string, useADC bool, opts ...ClientOption) (*Client, error) {
    trimmed := strings.TrimRight(strings.TrimSpace(baseURL), "/")
    if trimmed == "" {
        return nil, fmt.Errorf("agent runtime baseURL cannot be empty")
    }
    if !strings.HasSuffix(trimmed, "/api") {
        trimmed += "/api"
    }

    hc := &http.Client{Timeout: 60 * time.Second}
    if useADC {
        if idClient, err := idtoken.NewClient(ctx, strings.TrimSuffix(trimmed, "/api")); err == nil {
            hc = idClient
        } else if oauthClient, err := google.DefaultClient(ctx, "https://www.googleapis.com/auth/cloud-platform"); err == nil {
            hc = oauthClient
        }
    }
    // ...
}

2. Cross-Layer Image Forwarding: Gateway Downloads Image $\rightarrow$ Agent Runtime Automatically Writes to GCS Artifact

When a user uploads a business card photo on LINE:

Gateway downloads imageBytes from LINE Blob API.
gateway.Client.StartIngest calls the remote ADK REST API POST /api/apps/{appName}/users/{userID}/sessions/{sessionID} to create a session.
Then it calls POST /api/run, wrapping imageBytes in the InlineData of newMessage.Parts to send to the remote Agent Runtime:

func (c *Client) StartIngest(ctx context.Context, userID string, imageBytes []byte, recentMessages []string) (*linebot.WorkflowResponse, error) {
    sessionID := "ingest-" + uuid.NewString()
    initialState := map[string]any{
        workflow.StateKeyInputType: workflow.RouteIngest,
        workflow.StateKeyUserID: userID,
        workflow.StateKeyRecentMessages: recentMessages,
    }
    if err := c.createSession(ctx, userID, sessionID, initialState); err != nil {
        return nil, err
    }

    parts := []*genai.Part{
        {Text: workflow.RouteIngest},
    }
    if len(imageBytes) > 0 {
        parts = append(parts, &genai.Part{
            InlineData: &genai.Blob{
                MIMEType: "image/jpeg",
                Data: imageBytes,
            },
        })
    }
    msg := &genai.Content{
        Role: genai.RoleUser,
        Parts: parts,
    }

    res, err := c.runTurn(ctx, userID, sessionID, msg, initialState)
    // ...
}

At this point, the logic we wrote for RouterNode in Phase 2 works like magic: when the remote Agent Runtime's RouterNode receives this request with InlineData, it automatically calls ctx.Artifacts().Save(ctx, "business_card.jpg", ...) to store the image in the GCS Artifact Bucket, leaving only lightweight card_image_artifact and card_image_version in Session.State! This means the LINE Webhook Gateway doesn't even need direct write permissions to the GCS Bucket for the remote Agent Runtime to complete versioned GCS Artifact archiving!

3. Cross-Layer Stateless HITL Breakpoint Discovery and Recovery (`resolvePendingHITL` + `ResumeHITL`)

The most exciting part: when a user sees the LINE Flex preview card and inputs "Change title to CTO" 5 minutes later, and this Webhook request lands on another completely new Cloud Run Gateway instance, how does gateway.Client know this user has a pending HITL Session on the remote Agent Runtime?

We use the GET /api/apps/{appName}/users/{userID}/sessions endpoint via the ADK REST API in gateway.Client.resolvePendingHITL to query the remote Agent Runtime for the user's session status and check state["active_interrupt_id"]:

func (c *Client) resolvePendingHITL(ctx context.Context, userID string) (pendingHITL, bool) {
    // 1. If local cache has SessionID, call remote GET /sessions/{id} to get the latest active_interrupt_id
    // (Important! Because after each Edit loop, hitl_seq increments to hitl_confirm_2, hitl_confirm_3...)
    c.mu.RLock()
    cached, hasCached := c.pendingSession[userID]
    c.mu.RUnlock()
    if hasCached && cached.SessionID != "" {
        if fullSess, err := c.getSession(ctx, userID, cached.SessionID); err == nil && fullSess != nil {
            if activeID := readStringFromMap(fullSess.State, workflow.StateKeyActiveInterruptID); activeID != "" {
                found := pendingHITL{SessionID: cached.SessionID, InterruptID: activeID}
                c.mu.Lock()
                c.pendingSession[userID] = found
                c.mu.Unlock()
                return found, true
            }
        }
    }

    // 2. If this is a brand-new stateless Gateway instance, call remote GET /sessions to find sessions with active_interrupt_id
    sessions, err := c.listSessions(ctx, userID)
    if err != nil {
        return pendingHITL{}, false
    }
    for _, sess := range sessions {
        activeID := readStringFromMap(sess.State, workflow.StateKeyActiveInterruptID)
        if activeID == "" && sess.ID != "" {
            if fullSess, err := c.getSession(ctx, userID, sess.ID); err == nil && fullSess != nil {
                activeID = readStringFromMap(fullSess.State, workflow.StateKeyActiveInterruptID)
            }
        }
        if activeID != "" {
            found := pendingHITL{SessionID: sess.ID, InterruptID: activeID}
            c.mu.Lock()
            c.pendingSession[userID] = found
            c.mu.Unlock()
            return found, true
        }
    }
    return pendingHITL{}, false
}

Here is a real bug and key detail we caught while writing integration tests:

When the user uploads a business card in Turn 1, the first breakpoint ID generated by HITLConfirmNode is hitl_confirm_1.
When the user inputs "Change title to CTO" in Turn 2 (possibly handled by Gateway Instance 2), ApplyEditNode modifies the title and loops back to HITLConfirmNode to pause again, incrementing the breakpoint ID to hitl_confirm_2!
If Turn 3 (user clicks "✅ Confirm Save") happens to go back to Gateway Instance 1, and Instance 1 blindly trusts its old local cache (hitl_confirm_1), the sent FunctionResponse won't match the latest breakpoint hitl_confirm_2!
Therefore, in resolvePendingHITL, we ensure that even if the local cache hits a SessionID, it synchronizes the latest active_interrupt_id from the remote via getSession. This way, no matter how many times the request jumps between Gateway instances or how many fields are modified, it always aligns perfectly with the latest InterruptID!

4. `cmd/server/main.go`: Seamless Switching Between Monolithic and Two-Tier Gateway Modes

Finally, we added mode detection in cmd/server/main.go:

    var runner linebot.WorkflowRunner
    if cfg.AgentRuntimeURL != "" {
        // 🌐 Mode A: Decoupled Two-Tier LINE Webhook Gateway Mode (Forwarding to remote Agent Runtime)
        gwClient, err := gateway.NewClient(
            ctx,
            cfg.AgentRuntimeURL,
            cfg.AgentRuntimeUseADC,
            gateway.WithAppName(cfg.AgentRuntimeAppName),
        )
        if err != nil {
            slog.Error("failed to initialize remote Agent Runtime gateway client", "url", cfg.AgentRuntimeURL, "err", err)
            os.Exit(1)
        }
        slog.Info("running in decoupled LINE Webhook Gateway mode",
            "agent_runtime_url", cfg.AgentRuntimeURL,
            "agent_runtime_app", cfg.AgentRuntimeAppName,
            "use_adc", cfg.AgentRuntimeUseADC,
        )
        runner = gwClient
    } else {
        // 🚀 Mode B: All-in-One Local Engine Mode (Default for local development and single container deployment)
        cardStore, err := store.NewStore(ctx)
        // ... Initialize local workflow.NewEngine ...
        runner = engine.LinebotRunner()
    }

    botHandler, err := linebot.NewHandler(cfg, runner)

4. End-to-End Two-Tier Architecture Integration Testing (`internal/gateway/client_test.go`)

To ensure gateway.Client is 100% compatible with the official ADK Go v2.6.0 server/adkrest REST server, we started a real adkrest.NewServer (mounting our app.NewRootAgentWithDeps) using httptest.NewServer in internal/gateway/client_test.go and established two completely independent Gateway Clients (client1 and client2) for a five-stage end-to-end test:

Turn 1 (client1.StartIngest): Upload business card photo via HTTP $\rightarrow$ Verify remote ArtifactService successfully writes business_card.jpg $\rightarrow$ Verify return of WaitingHITL=true (hitl_confirm_1) and LINE Flex Preview JSON containing "Wang Daming".
Cross-Instance State Probing (client2.HasPendingHITL): The brand-new stateless client2 probes the paused session created by client1 via GET /api/apps/app/users/{user}/sessions.
Turn 2 (client2.ResumeHITL — Natural Language Modification): client2 sends "Change title to CTO" $\rightarrow$ Verify remote execution of ApplyEditNode, updating DraftCard.Title == "CTO", and pausing again at hitl_confirm_2.
Turn 3 (client1.ResumeHITL — Confirm Save): Switch back to client1 and send "action=confirm_save" $\rightarrow$ client1 automatically synchronizes the latest breakpoint hitl_confirm_2 and wakes up SaveCardNode $\rightarrow$ Verify business card successfully written to remote CardStore and returns AI icebreaker.
Turn 4 & Turn 5 (client2.RunSearch & client2.RunEmailDraft): Call remote hybrid search and individual icebreaker drafting via HTTP, verifying return of Flex Carousel JSON and icebreaker content.

Running the full test suite:

$ go test -race -v ./...
=== RUN TestGatewayClient_EndToEndDecoupledIngestHITLAndSearch
--- PASS: TestGatewayClient_EndToEndDecoupledIngestHITLAndSearch (0.01s)
=== RUN TestBusinessCardRecordRoundTrip
--- PASS: TestBusinessCardRecordRoundTrip (0.00s)
=== RUN TestBuildPostgresDSNFromEnv
--- PASS: TestBuildPostgresDSNFromEnv (0.00s)
=== RUN TestNewStoreFactoryFallback
--- PASS: TestNewStoreFactoryFallback (0.00s)
=== RUN TestRankCardsByKeywords
--- PASS: TestRankCardsByKeywords (0.00s)
PASS

All unit tests and cross-layer integration tests passed with green lights under -race detector!

5. Summary and Roadmap Progress Check

After completing PR #12, looking back at the five stages of Roadmap Issue #9, Phase 1, Phase 2, and Phase 3 are now 100% complete!

Phase 1 (agents-cli & adk_go Scaffolding): Completed Issues #1, #2 (PR #10)
Phase 2 (Agent Runtime Adaptation and Cloud Persistence Iron Triangle): Completed Issues #3, #4 (PR #11) and Issue #5 (PR #12)
Phase 3 (LINE Webhook Gateway and Agent Runtime Two-Tier Decoupled Architecture): Completed Issue #6 (PR #12)
Phase 4 (agents-cli eval Automated Evaluation and Benchmarking): Issue #7
Phase 5 (Agent Runtime Official Deployment, Terraform IaC, and Cloud Trace Observability): Issue #8

In the upcoming articles of this series, we will enter Phase 4 and Phase 5, using agents-cli eval to establish a Quality Flywheel for this 12-node Graph Workflow, and officially deploy the entire system using agents-cli deploy and OpenTelemetry. Stay tuned!

Resources and Further Reading

"Agent Platform in Action" Series Articles:
Project Open Source Code and PRs:
- GitHub Repository: kkdai/line-card-assistant
- Phase 2 & 3 Implementation Pull Request: PR #12: feat(phase2-3): Cloud SQL PostgresStore (#5) & Decoupled LINE Webhook Gateway (#6)
- Completed Issues: Issue #5, Issue #6, Roadmap #9
Google ADK and Agent Platform Official Documentation:
- ADK Documentation (llms.txt)
- Getting Started - agents-cli

[Agent Platform in Practice (5)] Building an AI Agent Quality Flywheel: Automated Benchmarking for LINE AI Business Card King wi

Evan Lin — Fri, 23 Oct 2026 08:31:26 +0000

Recap

In the first four articles of the "Agent Platform in Action" series, we have step-by-step advanced the LINE AI Business Card Master (line-card-assistant) from a standalone prototype to a cloud-native, dual-layer decoupled Google Cloud Agent Platform architecture:

[Agent Platform in Action (1)] Building LINE AI Business Card Master with Go and Google ADK 2.0 Graph Workflow: Used google.golang.org/adk/v2/workflow to build a 12-node business card processing graph (Graph Workflow), covering Vision OCR, AddFanOut / AddFanIn parallel background investigations, and the HITL (Human-in-the-Loop) pause and modification loop via NewRequestInputEvent.
[Agent Platform in Action (2)] From Monolithic Cloud Run to Google Cloud Agent Runtime: Refactoring LINE AI Business Card Master with agents-cli and ADK Go: Completed Phase 1 (PR #10), introducing the agents-cli (adk_go) standard dual-channel architecture, supporting LINE Webhook, ADK WebUI Playground, and the A2A protocol.
[Agent Platform in Action (3)] Solving HITL Amnesia in Stateless Containers: Implementing Cross-Instance Recognition and Pause/Resume with ADK 2.0 GCS Artifact Service and Agent Platform Sessions: Completed the first half of Phase 2 (PR #11), using artifact/gcsartifact and session/vertexai to enable cross-instance HITL pause and resume for stateless containers.
[Agent Platform in Action (4)] Towards Cloud Dual-Layer Decoupling and Full Persistence: Completing the Architecture with Cloud Run LINE Webhook Gateway, Agent Runtime, and Cloud SQL PostgreSQL: Completed the second half of Phase 2 (Issue #5) and Phase 3 (Issue #6, PR #12), migrating the business card database to Cloud SQL PostgreSQL (PostgresStore) and implementing the stateless Cloud Run LINE Webhook Gateway (gateway.Client).

With the architecture and features in place and ready to officially deploy the Agent to the cloud production environment (Phase 5), every AI engineering team faces a core soul-searching question:

"When we modify a line of Prompt, adjust a Graph Node's routing condition, or upgrade the underlying model from gemini-2.5-flash to gemini-3.8-flash, how do we ensure that business card OCR fields aren't missed, duplicate card promotion detection doesn't fail, HITL modification loops don't get stuck, and hybrid search doesn't degrade?"

In today's fifth episode of the series, we are going to implement Phase 4 (Issue #7, corresponding to PR #13): using agents-cli eval and ADK 2.0 Go trajectory testing to build a quantifiable, comparable, and CI/CD-automatable AI Agent Quality Flywheel for the LINE AI Business Card Master!

I. Why Traditional Unit Tests Aren't Enough? Understanding the `agents-cli eval` "Quality Flywheel"

In traditional software engineering, we are used to writing deterministic unit tests like if got != want. However, in an AI Agent system driven by ADK 2.0 Graph Workflow + Multimodal LLM, what we need to verify is far more than just the return value of a single function; it is the entire "Execution Trajectory and State Evolution (State Delta)":

Multi-branch and Parallel Convergence (Fan-Out / Fan-In Trajectory): When a business card image is received, does the RouterNode correctly enter the VisionOCRNode? Subsequently, are the three parallel branches CleanAndDedupNode, CompanyEnrichNode, and ContextMatchNode triggered simultaneously and converged at JoinPreviewNode?
Intermediate State Changes (Session state_delta): Even if the final return to the user is a LINE Flex Message JSON, did the intermediate nodes correctly normalize the Taiwan mobile number (+886-912-345-678) and detect an existing contact's promotion (IsDuplicate=true, VP of Engineering $\rightarrow$ Chief Technology Officer (CTO))?
Human-in-the-Loop (HITL) Pause and Multi-turn Dialogue Loop: After the HITLConfirmNode issues an adk_request_input pause event, if the user replies "Change title to Technical VP, change occasion to COMPUTEX 2026 AI Forum" and confirms the save, does the workflow actually traverse ApplyEditNode $\rightarrow$ BuildPreviewNode $\rightarrow$ HITLConfirmNode $\rightarrow$ SaveCardNode?
Retrieval Degradation and Grounding: When a strict HybridSearchNode finds no results (0 hits), does it automatically degrade to trigger BroadSearchNode? Does the generated icebreaker follow-up email draft truly reference the meeting occasion and company background from the business card without hallucination?

To systematically solve this problem, Google Cloud agents-cli proposes a five-step Quality Flywheel workflow:

flowchart LR
    A["1. Generate<br/>(agents-cli eval generate)<br/>Start ADK server and collect real trajectories"] --> B["2. Grade<br/>(agents-cli eval grade)<br/>Execute Trajectory and quality metric scoring"]
    B --> C["3. Analyze<br/>Review scores for each metric and<br/>Explanation for failed cases"]
    C --> D["4. Fix<br/>Fix Graph Node, Prompt<br/>or event tracking structure"]
    D --> E["5. Compare<br/>(agents-cli eval compare)<br/>Baseline vs. Candidate quantitative comparison"]
    E -->|Continuous Iteration| A

II. Revealing How `agents-cli eval generate` Tests a Go-based ADK 2.0 Agent?

Many Go developers new to agents-cli wonder: "The agents-cli is a CLI tool written in Python; how can it evaluate my adk/v2/workflow Agent written in Go?"

The answer lies within the main.go and appinfo/appinfo.go we introduced in Phase 1! When you execute:

DRY_RUN=true agents-cli eval generate --dataset eval/datasets/basic-dataset.json

agents-cli automatically completes the following four steps in the background:

Automatically compile and start your Go Agent server: agents-cli reads the agents-cli-manifest.yaml in the project root (identified as template: adk_go) and automatically starts in the background:

go run . web --port <random_port> api -path_prefix / a2a appinfo

Introspect Agent structure via GET /apps/app/app-info reflection: appinfo/appinfo.go uses Go's reflect and unsafe to traverse the Edges and Nodes of the entire workflowagent, assembling the embedded CompanyEnrichAgent (llmagent), Instructions, and tool declarations into a standard AppInfo JSON returned to agents-cli.
Create Sessions one by one and call POST /run_sse to collect event streams: For each test case (eval_case_id) in eval/datasets/basic-dataset.json, agents-cli first calls POST /apps/app/users/eval_user/sessions to create an independent Session, then sends the test case's prompt to POST /run_sse, and records every ADK Event (including author, content, actions.state_delta, actions.requested_tool_confirmations, etc.) streamed back from the server.
Output standardized trace file artifacts/traces/traces_<timestamp>.json: After testing is complete, it automatically shuts down the Go server and saves the full trajectory as a JSON, passing it to the next stage, agents-cli eval grade, for scoring.

III. Practical Design: A Golden Evaluation Set Covering 4 Core Scenarios (`eval/datasets/basic-dataset.json`)

To fully cover the four core acceptance scenarios defined in Issue #7, we designed 6 Golden Eval Cases in eval/datasets/basic-dataset.json, marking the expected node trajectory (expected_nodes) and state assertions in each case's custom_Properties:

Eval Case ID	Corresponding Core Scenario	Input Prompt (Simulated Channel Signal)	Verification Focus (`custom_Properties`)
`case_1_ingest_fanout_fanin_ocr`	Scenario 1: Business Card Recognition and Parallel Enhancement	`action=ingest&context=2026 Cloud Summit Taipei`	Verify `RouterNode` $\rightarrow$ `VisionOCRNode` $\rightarrow$ Parallel Fan-Out (`CleanAndDedupNode`, `CompanyEnrichNode`, `ContextMatchNode`) $\rightarrow$ `BuildPreviewNode` $\rightarrow$ `HITLConfirmNode` pause, and phone normalization to `+886-912-345-678`.
`case_2_duplicate_promotion_detection`	Scenario 2: Duplicate Card and Promotion Detection	`action=ingest&card=promotion_david_chen_appier&context=2026 AI DevFest Taipei`	Test database already contains David Chen from Appier (`VP of Engineering`); when a new card (`Chief Technology Officer (CTO)`) is received, `CleanAndDedupNode` must mark `IsDuplicate=true` and generate a promotion alert message.
`case_3_hitl_edit_and_confirm_loop`	Scenario 3: HITL Loop Correction and Saving	`action=ingest&context=2026 Cloud Summit Taipei&edit=Change title to Technical VP, change occasion to COMPUTEX 2026 AI Forum&hitl_action=confirm`	Verify from OCR preview $\rightarrow$ `ApplyEditNode` semantic modification of title and occasion $\rightarrow$ `BuildPreviewNode` re-rendering preview $\rightarrow$ `HITLConfirmNode` confirmation $\rightarrow$ `SaveCardNode` writing to database and generating follow-up email.
`case_4a_hybrid_search_cloud_summit_ai`	Scenario 4a: Natural Language Hybrid Retrieval	`Help me find people doing AI I met at Cloud Summit`	Verify `QueryIntentNode` converts colloquial speech to `SearchFilter` $\rightarrow$ `HybridSearchNode` hits 2 business cards $\rightarrow$ `FormatCarouselNode` outputs Flex Carousel JSON.
`case_4b_broad_search_fallback`	Scenario 4b: 0 Hits Automatic Degradation to Fuzzy Retrieval	`Help me find Zhang San who does quantum finance blockchain I met at Tokyo Game Show`	Verify that when `HybridSearchNode` finds no results (`0 hits`), conditional routing automatically degrades to `BroadSearchNode` for Token fuzzy retrieval to avoid blank replies.
`case_4c_personalized_icebreaker_email`	Scenario 4c: Personalized Icebreaker Follow-up Email	`action=draft_email&card_id=card-seed-1`	Verify `EmailDraftNode` writes a hallucination-free business icebreaker email based on the specified card's `MetContext` (`2026 Cloud Summit Taipei`) and `CompanyInfo` (`Google Cloud Taiwan`).

IV. Real-world Pitfalls and Debugging: Why was the Trajectory Score `0.0000` on the First `agents-cli eval grade` Run?

After writing eval/eval_config.yaml and eval/metrics.py, we confidently ran the first Baseline evaluation:

DRY_RUN=true agents-cli eval run --config eval/eval_config.yaml

The first Baseline scorecard (results_20261022_223104.json) printed in the terminal shocked us:

╭─ Evaluation Summary ───────────────────────────╮
│ Metric Score Valid │
│ ────────────────────────────────────────────── │
│ workflow_trajectory_accuracy 0.0000 6/6 │
│ ocr_and_enrichment_quality 1.0000 6/6 │
│ duplicate_and_hitl_precision 0.7333 6/6 │
│ search_and_email_grounding 1.0000 6/6 │
╰────────────────────────────────────────────────╯

At the same time, a very critical warning message appeared in the agents-cli eval grade execution log:

Dropped 26 event(s) with no content; the eval service rejects them

🔍 Root Cause: It turns out `drop_empty_content_events` filtered out intermediate node events!

Why were 26 events dropped? We immediately checked the implementation of ADK Go workflow.FunctionNode and the event preprocessing logic of agents-cli:

In our original internal/workflow/nodes_ingest.go and internal/workflow/nodes_search.go, Intermediate Nodes like RouterNode, VisionOCRNode, CleanAndDedupNode, CompanyEnrichNode, ContextMatchNode, and QueryIntentNode were only responsible for writing results into ev.Actions.StateDelta (e.g., ev.Actions.StateDelta["draft_card"] = ...), while keeping ev.Content as nil; only the final output node (such as BuildPreviewNode, SaveCardNode, FormatCarouselNode) would set ev.Content.
However, before agents-cli eval grade sends the trajectory to the grader, the underlying drop_empty_content_events()—to comply with the Schema constraints of the cloud Vertex AI Eval Service—will directly remove all events where event.content == null from agent_data.turns[].events!
Once intermediate node events are removed because content == nil, the actions.state_delta and node execution records attached to those events also disappear! This caused the grader to only see the output of the last node, resulting in a workflow_trajectory_accuracy of 0.0000!

🛠️ Elegant Solution: Adding Structured Trajectory Summaries `setNodeTrace` to Intermediate Nodes

Once the root cause was found, the solution was clear and elegant! We designed a lightweight helper function setNodeTrace in internal/workflow/nodes_ingest.go:

// setNodeTrace attaches a concise model text part to intermediate workflow node
// events so that `agents-cli eval` (which drops events with empty Content)
// retains the full node execution trajectory and state_delta history.
func setNodeTrace(ev *session.Event, nodeName, summary string) {
    if ev == nil || ev.Content != nil {
        return
    }
    ev.Content = genai.NewContentFromText(fmt.Sprintf("[%s] %s", nodeName, summary), genai.RoleModel)
}

Then, we call setNodeTrace before each intermediate node finishes, for example:

RouterNode: setNodeTrace(ev, "RouterNode", "route="+route)
VisionOCRNode: setNodeTrace(ev, "VisionOCRNode", fmt.Sprintf("ocr extracted name=%s company=%s title=%s", card.Name, card.Company, card.Title))
CleanAndDedupNode: setNodeTrace(ev, "CleanAndDedupNode", fmt.Sprintf("phone=%s is_duplicate=%v", card.Phone, isDup))
CompanyEnrichNode: setNodeTrace(ev, "CompanyEnrichNode", fmt.Sprintf("company_info=%s tags=%v", card.CompanyInfo, card.Tags))
ContextMatchNode: setNodeTrace(ev, "ContextMatchNode", "met_context="+card.MetContext)
QueryIntentNode / HybridSearchNode / BroadSearchNode: Record the converted SearchFilter and the number of business cards hit.

At the same time, to prevent the [NodeName] ... trajectory text of intermediate nodes from overwriting the ReplyText that Engine.executeTurn ultimately returns to the LINE caller, we simultaneously strengthened the state convergence logic in internal/workflow/graph.go—allowing TurnResult to read authoritative StateKeyReplyText, StateKeyFlexPreviewJSON, StateKeyFlexCarouselJSON, and StateKeyDraftEmail directly from the final converged Session.State!

V. Turning the Quality Flywheel: From Baseline to Candidate, All Four Metrics Reached `1.0000`!

After adding setNodeTrace and supporting single-turn batch HITL modification verification, we re-executed DRY_RUN=true agents-cli eval run --config eval/eval_config.yaml:

Loaded 6 cases from eval/datasets/basic-dataset.json
Starting local Go ADK server...
Running 6 cases against http://127.0.0.1:56649 (app=app, concurrency=4)...
Wrote 6 traces to artifacts/traces/traces_20261022_223515.json
Loaded 6 trace(s) from artifacts/traces/traces_20261022_223515.json
Evaluating metrics: ['workflow_trajectory_accuracy', 'ocr_and_enrichment_quality', 'duplicate_and_hitl_precision', 'search_and_email_grounding']

╭─ Evaluation Summary ───────────────────────────╮
│ Metric Score Valid │
│ ────────────────────────────────────────────── │
│ workflow_trajectory_accuracy 1.0000 6/6 │
│ ocr_and_enrichment_quality 1.0000 6/6 │
│ duplicate_and_hitl_precision 1.0000 6/6 │
│ search_and_email_grounding 1.0000 6/6 │
╰────────────────────────────────────────────────╯

This time, Dropped 0 event(s), and all intermediate node trajectories and state_delta were 100% preserved! We immediately used the 5th step of the Quality Flywheel, agents-cli eval compare, to quantitatively compare the Baseline before the fix with the Candidate after the fix:

agents-cli eval compare \
  --baseline artifacts/grade_results/results_20261022_223104.json \
  --candidate artifacts/grade_results/results_20261022_223515.json


╭─ Summary Comparison ─────────────────────────────────────────────────────────╮
│ Metric Baseline Candidate Delta │
│ ──────────────────────────────────────────────────────────────────────────── │
│ duplicate_and_hitl_precision 0.7333 1.0000 +0.2667 │
│ ocr_and_enrichment_quality 1.0000 1.0000 0.0000 │
│ search_and_email_grounding 1.0000 1.0000 0.0000 │
│ workflow_trajectory_accuracy 0.0000 1.0000 +1.0000 │
╰──────────────────────────────────────────────────────────────────────────────╯

Advancing from 0.0000 / 0.7333 to all four metrics reaching 1.0000 (+1.0000 / +0.2667) is exactly the most charming value of the agents-cli eval quality flywheel—making every architectural adjustment traceable and provable!

VI. Dual-Track Verification: Python Custom Graders (`eval/metrics.py`) + Go Native Multi-turn Trajectory Tests (`eval/eval_test.go`)

In the eval/ directory, we adopted a "Python agents-cli eval + Go go test dual-track verification architecture":

1. `eval/eval_config.yaml` and `eval/metrics.py`: Four Custom Evaluation Metrics

In eval/eval_config.yaml, we registered 4 deterministic trajectory and quality graders (plus 1 card_assistant_llm_rubric available for online Vertex AI Judge models):

metrics_to_run:
  - workflow_trajectory_accuracy
  - ocr_and_enrichment_quality
  - duplicate_and_hitl_precision
  - search_and_email_grounding

custom_metrics:
  workflow_trajectory_accuracy:
    type: custom
    module: eval.metrics
    function: evaluate_workflow_trajectory
  ocr_and_enrichment_quality:
    type: custom
    module: eval.metrics
    function: evaluate_ocr_and_enrichment
  duplicate_and_hitl_precision:
    type: custom
    module: eval.metrics
    function: evaluate_duplicate_and_hitl
  search_and_email_grounding:
    type: custom
    module: eval.metrics
    function: evaluate_search_and_email

In eval/metrics.py, we defined helper functions _extract_executed_nodes(instance) and _merged_state_delta(instance) to parse the actual node sequence traversed by the Graph Workflow and the cumulative Session State directly from agent_data.turns[].events:

def _extract_executed_nodes(instance: dict[str, Any]) -> list[str]:
    """Extract executed workflow node names in order from `[NodeName] ...` traces."""
    nodes: list[str] = []
    for ev in _iter_events(instance):
        content = ev.get("content") or {}
        for part in content.get("parts") or []:
            if isinstance(part, dict) and isinstance(part.get("text"), str):
                for match in NODE_TRACE_PATTERN.findall(part["text"]):
                    if not nodes or nodes[-1] != match:
                        nodes.append(match)
    return nodes

This way, evaluate_workflow_trajectory can accurately check:

Whether each node was executed in topological order (Subsequence Order)?
Whether all expected_state_keys (such as draft_card, flex_preview_json, saved_card_id, search_results) were written to state_delta?
Whether the final output simultaneously satisfies expected_response_substrings and excludes forbidden_substrings?

2. `eval/eval_test.go`: Go Native Multi-turn HITL Pause/Resume and `/app-info` Introspection Tests

In addition to agents-cli eval called via HTTP SSE, we also wrote Go native TestEvalDatasetAndAppInfoIntrospection and TestPhase4WorkflowTrajectorySuite in eval/eval_test.go, which can be executed in milliseconds via go test ./... or GitHub Actions CI:

TestEvalDatasetAndAppInfoIntrospection: Verifies the Schema integrity of eval/datasets/basic-dataset.json and directly calls appinfo.BuildAppInfo to confirm that the CompanyEnrichAgent within the line_card_assistant graph can be correctly introspected via reflection by /apps/{app}/app-info.
TestPhase4WorkflowTrajectorySuite:
- Case1_IngestFanOutFanInOCR
- Case2_DuplicatePromotionDetection
- Case3_MultiTurnHITLEditAndConfirmLoop (Actually calls StartIngest $\rightarrow$ ResumeHITL(edit) $\rightarrow$ ResumeHITL(confirm) in three turns to verify the cross-turn Pause/Resume state machine)
- Case4_SearchHybridAndBroadFallbackAndEmailDraft

Now, with just one line make eval, you can run both agents-cli eval run and the Go native test suite:

$ make eval
DRY_RUN=true agents-cli eval run --config eval/eval_config.yaml
...
go test -v ./eval/...
=== RUN TestEvalDatasetAndAppInfoIntrospection
--- PASS: TestEvalDatasetAndAppInfoIntrospection (0.00s)
=== RUN TestPhase4WorkflowTrajectorySuite
    --- PASS: TestPhase4WorkflowTrajectorySuite/Case1_IngestFanOutFanInOCR (0.00s)
    --- PASS: TestPhase4WorkflowTrajectorySuite/Case2_DuplicatePromotionDetection (0.00s)
    --- PASS: TestPhase4WorkflowTrajectorySuite/Case3_MultiTurnHITLEditAndConfirmLoop (0.00s)
    --- PASS: TestPhase4WorkflowTrajectorySuite/Case4_SearchHybridAndBroadFallbackAndEmailDraft (0.00s)
PASS
ok github.com/kkdai/line-card-assistant/eval    0.436s

VII. Summary and Next Preview (Phase 5 Final Chapter: Official Deployment, Terraform IaC, and Cloud Trace Observability)

As of now, our LINE AI Business Card Master Roadmap (Issue #9) has completed the first four major stages:

Phase 1: agents-cli environment and adk_go project Scaffolding (PR #10)
Phase 2: ADK Go Graph Workflow adaptation for Agent Runtime, GCS Artifacts, Agent Platform Sessions, and Cloud SQL PostgresStore (PR #11, PR #12)
Phase 3: Cloud Run LINE Webhook Gateway and Agent Runtime dual-layer decoupled architecture (PR #12)
Phase 4: agents-cli eval automated evaluation and quality flywheel (PR #13)

In the next article (which is also the final chapter of this series, Phase 5 — Issue #8), we will officially push the LINE AI Business Card Master, which passed the evaluation with a perfect 1.0000 score, into the production environment via agents-cli deploy. We will also fully integrate Terraform IaC (Infrastructure as Code), GitHub Actions CI/CD automated evaluation and deployment pipelines, and OpenTelemetry + Google Cloud Trace full-link observability. Stay tuned!

Reference Resources

GitHub Repository: kkdai/line-card-assistant
Phase 4 Pull Request: PR #13 — feat(eval): add Phase 4 agents-cli eval suite and Go workflow trajectory tests (#7)
Google Cloud agents-cli Official Documentation: The Quality Flywheel & Agent Evaluation
Google ADK Official Documentation: Graph-based Agent Workflows (adk/v2/workflow)

[Agent Platform in Practice (2)] From Monolithic Cloud Run to Google Cloud Agent Runtime: Refactoring LINE AI Business Card King

Evan Lin — Tue, 20 Oct 2026 12:12:59 +0000

Recap

In the previous article, "[Agent Platform in Practice (1)] Building a LINE AI Business Card Assistant with Go and Google ADK 2.0 Graph Workflow", we used Golang (google.golang.org/adk/v2/workflow) and the LINE Messaging API SDK v8 to build line-card-assistant (LINE AI Business Card Assistant) from scratch. We successfully implemented and deployed the four core capabilities of ADK 2.0 Graph Workflow—Deterministic Router, AddFanOut / AddFanIn for parallel background investigation and aggregation, and NewRequestInputEvent for Human-in-the-Loop (HITL) pause and loop correction—to Google Cloud Run.

However, as we evolve this monolithic LINE Bot into a Production-grade Agent Platform, we immediately encounter several architectural questions:

Where is the cross-instance HITL Session state stored? In the first version, we used session.InMemoryService() with local SQLite. When Cloud Run switches container instances due to auto-scaling or redeployment, the HITL pause state (Checkpoint) waiting for the user to click "✅ Confirm Save" disappears due to memory reset.
How can this business card Agent be called by other Agents (Agent-to-Agent, A2A)? If the company has another "CRM Sales Assistant Agent" or "Calendar Scheduling Agent" that wants to query contacts in our Business Card Assistant or ask it to write an icebreaker email via a standard A2A protocol, a monolithic server tied to the LINE Webhook /webhook entry point cannot directly provide a standard Agent interface.
How to systematically evaluate and observe the entire Graph? After changing a line of Prompt or adjusting a node, how do we ensure that OCR accuracy, Fan-Out parallel tool call trajectories (Tool Trajectory), and search quality haven't degraded?

To bridge the gap between "writing an Agent prototype" and "operating an enterprise-grade Agent," Google officially launched a new command-line and skill toolkit—agents-cli (Google Agents CLI), serving as the unified development entry point for the Google Cloud Agent Platform (now integrated as Gemini Enterprise Agent Platform).

In this second episode of the series, we will dive deep into the agents-cli official development guide, answer frequently asked questions about Agent Runtime scaling (Scale-to-Zero) and billing mechanisms, and walk you through how we completed the Phase 1 architecture upgrade (PR #10): refactoring the project into a dual-channel architecture that complies with adk_go + agent_runtime specifications without breaking the original LINE Bot functionality!

1. In-depth Analysis of Google Cloud Agent Platform and the `agents-cli` Official Guide

Before diving into the code, let's clarify Google's latest technical landscape for the Agent Platform through two core documents: "Getting Started - agents-cli" and "The Lifecycle - agents-cli".

1. Product Terminology Comparison: What are Agent Platform and Agent Runtime?

If you've followed Google Cloud's AI product line over the past year, you might have heard of Vertex AI Agent Engine, Reasoning Engine, Agent Platform, or Gemini Enterprise Agent Platform (GEAP). In the latest agents-cli v1.10+ specifications, the official documentation has clearly converged these terms and underlying resources:

Common Developer Terms	Google Cloud Official Positioning & Underlying Resources	`agents-cli` Corresponding Parameter
Agent Platform / Gemini Platform	Google Cloud's overall platform for hosting enterprise-grade AI Agents (covering Runtime, Sessions, Memory Bank, Agent Gateway, Agent Registry, and Eval)	Unified operation via `agents-cli`
Agent Runtime / Agent Engine	A fully managed container execution environment designed specifically for AI Agents. Underlying GCP resource path: `projects/{project}/locations/{location}/reasoningEngines/{id}`	`--deployment-target agent_runtime`
Agent Platform Sessions	A cloud-managed conversation and state persistence service (ADK automatically binds `VertexAiEngineSessionService`)	`--session-type agent_platform_sessions`
Gemini Enterprise / Agent Registry	An enterprise Agent publishing and directory center, allowing deployed Agents to be discovered and called directly in the Gemini Enterprise interface or via A2A	`agents-cli publish gemini`

2. The Seven-Stage Development Lifecycle of `agents-cli`

In the past, we were used to a "write code → go test → gcloud run deploy" workflow. However, in "The Lifecycle" document, agents-cli defines a Seven-Stage Quality Flywheel that integrates Coding Agent Skills (supporting Antigravity CLI, Gemini CLI, Claude Code, etc.):

flowchart LR
    P0["Phase 0-1\nUnderstand & Recipes\n(.agents-cli-spec.md)"] --> P2["Phase 2\nScaffold\n(create / enhance)"]
    P2 --> P3["Phase 3\nBuild & Smoke Test\n(run / playground)"]
    P3 --> P4["Phase 4\nEvaluate\n(eval run Quality Flywheel)"]
    P4 -->|Iterative Correction| P3
    P4 --> P5["Phase 5\nDeploy\n(agent_runtime / cloud_run)"]
    P5 --> P6["Phase 6-7\nPublish & Observe\n(Cloud Trace / BigQuery)"]

Phase 0 & 1 — Understand & Study Recipes: Before writing a single line of code, document the Agent's core goals, external tools, security boundaries, and deployment targets in the project root's .agents-cli-spec.md as the Single Source of Truth for the entire development cycle.
Phase 2 — Scaffold: Use agents-cli scaffold create or run agents-cli scaffold enhance . on an existing project to automatically generate the directory structure, configuration files (agents-cli-manifest.yaml), A2A endpoints, Session switchers, and OpenTelemetry observation modules that comply with official standards.
- Supports two major official built-in templates: adk (Python) and adk_go (Golang)!
Phase 3 — Build & Playground:
- Go projects only need to focus on implementing and exporting NewRootAgent(ctx) in app/agent.go.
- During development, you can use agents-cli run "your test question" for a quick terminal smoke test, or run agents-cli playground (or make playground) to open a browser-based visual interactive interface for multi-turn conversation testing.
Phase 4 — Evaluate (The core of the Quality Flywheel): The official guide emphasizes: "Unit Test (go test) measures code contracts and type correctness; while agents-cli eval measures Agent behavior quality (Tool Trajectory, response quality, and safety)."
Phase 5 — Deploy: Use agents-cli deploy to package and deploy the project to agent_runtime, cloud_run, or gke, and automatically record the deployed resource ID in deployment_metadata.json.
Phase 6 & 7 — Publish & Observe: Register to the enterprise directory via agents-cli publish gemini, and combine with OpenTelemetry to automatically export all Graph node latencies, LLM Prompt/Response, and Token consumption to Cloud Trace, Cloud Logging, and BigQuery Agent Analytics.

3. Agent Runtime vs. Cloud Run vs. GKE: Which Deployment Target Should I Choose?

agents-cli supports three containerized deployment targets (all of which build based on the Dockerfile in the project root, making Go a first-class citizen on all three platforms). We have summarized the comparison based on the official Deployment Decision Matrix:

Evaluation Dimension	Agent Runtime (`agent_runtime`)	Cloud Run (`cloud_run`)	GKE (`gke`)
Infrastructure Management	Fully managed (built for Agents), automatically built from `Dockerfile` and mounted to `reasoningEngines`	Containerized serverless hosting, requires manual configuration of network and service parameters	GKE Autopilot, provides full K8s control but with the highest operational complexity
Session State Management	Native built-in Agent Platform Sessions (automatically injects `GOOGLE_CLOUD_AGENT_ENGINE_ID` at runtime and enables cloud persistent Sessions)	In-Memory by default, requires additional Cloud SQL setup or manual connection to managed Sessions	In-Memory by default, requires additional Cloud SQL setup or managed Sessions
API & Access Interface	Built-in `:query` / `:streamQuery`, Console Playground, and `/api/{container_path}` HTTP Passthrough and A2A	Native public or IAM-protected HTTPS URL, best for receiving external Webhooks (like LINE / Slack)	Custom K8s Ingress / Gateway
Enterprise Integration	Native support for Managed OAuth 2.0 User Consent, Agent Identity, Agent Gateway, and PSC-I Private Networking	Supports VPC Egress, IAP, but does not support managed Agent OAuth flows	Full K8s network control
Billing Model	Billed based on actual vCPU-hours + Memory-hours used (no compute fees when scaled to 0)	Billed based on Request duration or Instance seconds	Billed based on K8s Node Pod resources

4. Key Clarification: Will Instances Moved to Agent Runtime Scale to Zero?

When planning to migrate line-card-assistant to Agent Runtime, a common concern is:

"According to agents-cli specifications, will instances moved to Agent Runtime never scale to 0 and constantly burn money?"

The answer is: Agent Runtime supports Scale to Zero, and agents-cli adopts a clever dual-mode design: "Scale to 0 by default for development, and pre-warm 1 instance by default for production!"

According to the agents-cli official deployment specifications (Sizing a Deployment):

When using agents-cli deploy directly (Default Scale to Zero):
- The CLI's default resource allocation is identical to Cloud Run: --cpu 1, --memory 4Gi, --concurrency 8, --min-instances 0, --max-instances 10.
- This means when there is no traffic to the Agent Runtime, the number of instances will automatically scale down to 0. Since Agent Runtime is billed based on actual vCPU-hours and Memory-hours, no compute fees are generated when idling at 0!
When using Terraform (service.tf) for production deployment (Default min_instances = 1):
- When you generate Terraform IaC (deployment/terraform/single-project/service.tf) for a production environment, the template defaults min_instances to 1 to avoid cold starts for the first user.
- If you want to save money in production and can tolerate a few seconds of cold start, you can specify agents-cli deploy --min-instances 0 during deployment or change min_instances back to 0 in service.tf.

💡 Why is a "Dual-layer Decoupled Architecture" best for a LINE Bot with Agent Runtime?

Understanding the characteristics of Agent Runtime, you'll find that splitting the system into two layers is a perfect match:

Outer Layer: LINE Webhook Gateway (Cloud Run, min-instances=0): Since the LINE server sends Webhooks with only an X-Line-Signature HMAC signature, it cannot carry a Google Cloud IAM OAuth Token to call an IAM-protected Agent Runtime API. Use a lightweight Go Cloud Run service dedicated to "verifying the signature within 1 second, returning HTTP 200, and triggering the LINE loading animation." It scales to 0 when no one is sending business cards.
Inner Layer: Core Brain (Agent Runtime, min-instances=0): Focuses on running the ADK 2.0 Graph Workflow, managing cloud persistent VertexAiEngineSessionService (ensuring HITL pause states aren't lost across containers!), and simultaneously exposing A2A (/.well-known/agent-card.json) and the Agent Engine Playground to the enterprise!

2. Planning the Migration Blueprint: Creating a 5-Phase Roadmap Using GitHub CLI

To complete this architectural evolution methodically, we used the gh CLI in the GitHub Repository (kkdai/line-card-assistant) to establish 5 Milestones (Phases), 8 implementation Issues, and an overview tracking list Roadmap Issue #9:

Phase 1: agents-cli Environment and adk_go Project Scaffolding (The focus of this article!)
- #1 Upgrade google-agents-cli to 1.10+ and create .agents-cli-spec.md and agents-cli-manifest.yaml
- #2 Introduce adk_go + agent_runtime standard launcher architecture (app/agent.go, A2A, /app-info, Sessions, and Observability)
Phase 2: ADK Go Graph Workflow Cloud Storage and Persistence Adaptation
- #3 Deepen ADK Go v2.6+ Graph Workflow and NewRootAgent state management
- #4 Integrate Agent Platform Sessions (session/vertexai) and GCS Artifact Service to support cross-instance HITL pause/resume
- #5 Expand business card database (CardStore) to support Cloud SQL / Firestore cloud persistent storage
Phase 3: LINE Webhook Gateway and Agent Runtime Decoupled Architecture
- #6 Implement LINE Webhook Gateway (Cloud Run) and Agent Runtime dual-layer decoupled call architecture
Phase 4: agents-cli eval Automated Evaluation and Benchmarking
- #7 Establish agents-cli eval evaluation datasets and quality flywheel (OCR, Fan-Out/Fan-In, HITL loops, Hybrid Search)
Phase 5: Agent Runtime Official Deployment, IaC, and Observability
- #8 Deploy to Agent Runtime via agents-cli deploy and establish Terraform IaC, CI/CD, and Cloud Trace observability

3. Phase 1 Hands-on Breakdown: Transforming an Existing Go Project into an `adk_go` + `agent_runtime` Dual-Channel Architecture

In Phase 1 (corresponding to the merged PR #10), our goal was to: retain the original cmd/server/main.go independent LINE Webhook server while introducing the standard agents-cli (adk_go + agent_runtime) architecture in the project root, allowing the same codebase to possess both superpowers!

Let's look at the key files changed in this upgrade and the new features they bring:

line-card-assistant/
├── .agents-cli-spec.md # [New] Phase 0 Architecture Spec (Single Source of Truth)
├── agents-cli-manifest.yaml # [New] agents-cli v1.10.0 project config (adk_go + agent_runtime)
├── deployment_metadata.json # [New] Records remote Agent Runtime resource ID and A2A status
├── main.go # [New] Agent Runtime / ADK WebUI / A2A standard HTTP entry point
├── apploader.go # [New] Loader supporting /ui/?app=app and rootAgent.Name() dual aliases
├── appurl.go # [New] Automatically calculates Cloud Run / Agent Runtime public URL (for A2A Card)
├── observability.go # [New] OpenTelemetry Cloud Trace / Logging / Metrics auto-mounting
├── app/
│ ├── agent.go # [New] Standard export interface: NewRootAgent(ctx) returns 12-node Graph Workflow
│ └── agent_test.go # [New] Verifies Root Agent execution and search output directly via ADK Runner
├── appinfo/
│ └── appinfo.go # [New] GET /app-info endpoint, auto-parses Root Workflow and embedded CompanyEnrichAgent
├── sessions/
│ └── sessions.go # [New] Auto-switches between VertexAiEngineSessionService and local InMemoryService
├── cmd/server/main.go # [Retained] Original independent LINE Bot Webhook server
└── internal/workflow/ # [Enhanced] Graph Workflow nodes upgraded to support dual-channel input and streaming Content output

1. Creating `agents-cli-manifest.yaml` and `.agents-cli-spec.md`

To make agents-cli recognize a Go project, the key is the agents-cli-manifest.yaml in the root directory:

# Managed by agents-cli. Do not edit manually.
acli_version: "1.10.0"
project_name: "line-card-assistant"
root_agent_name: "line_card_assistant"
base_template: "adk_go"
language: "go"
framework: "google-adk"
deployment_target: "agent_runtime"
agent_directory: "app"
region: "asia-east1"
agent_guidance_filename: "GEMINI.md"
is_a2a: true
prototype: true
generated_at: "2026-10-17T16:12:00Z"

After configuration, running agents-cli info in the project root allows the CLI to immediately identify this as a Go project that supports A2A and targets deployment to the asia-east1 Agent Runtime:

$ agents-cli info
CLI version: 1.10.0
Project root: /Users/evanlin/Documents/line-card-assistant
Project name: line-card-assistant
Root agent name: line_card_assistant
Language: go
Deployment target: agent_runtime
Agent directory: app
Region: asia-east1
A2A: yes

2. Encapsulating the Standard Root Agent Entry Point: `app/agent.go`

In the adk_go template specification, all Agent logic should be exported via NewRootAgent(ctx context.Context) (agent.Agent, error) under the app package.

Since we already used ADK 2.0's workflowagent.New to encapsulate the 12-node Graph Workflow into a standard agent.Agent in the first article, in app/agent.go, we only need to initialize the underlying CardStore (SQLite or Memory) and model.LLM (auto-detecting Vertex AI ADC or Gemini API Key), then call workflow.NewCardAssistantAgent for a seamless connection!

package app

import (
    "context"
    "log/slog"
    "os"
    "strings"

    "github.com/kkdai/line-card-assistant/internal/config"
    "github.com/kkdai/line-card-assistant/internal/store"
    "github.com/kkdai/line-card-assistant/internal/workflow"
    "google.golang.org/adk/v2/agent"
    "google.golang.org/adk/v2/model"
)

const RootAgentName = workflow.WorkflowName // "line_card_assistant"

// NewRootAgent constructs the root ADK v2 Graph Workflow agent for Agent Runtime,
// ADK WebUI playground, A2A server, and evaluation CLI.
func NewRootAgent(ctx context.Context) (agent.Agent, error) {
    cfg := config.Load()

    var cardStore store.CardStore
    if strings.TrimSpace(cfg.SQLitePath) != "" {
        sqliteStore, err := store.NewSQLiteStore(cfg.SQLitePath)
        if err != nil {
            slog.Warn("failed to open SQLite store, falling back to in-memory store", "err", err)
            cardStore = store.NewMemoryStore()
        } else {
            cardStore = sqliteStore
        }
    } else {
        cardStore = store.NewMemoryStore()
    }

    var llm model.LLM
    if cfg.UseVertexAI || cfg.GoogleAPIKey != "" || os.Getenv("GOOGLE_CLOUD_AGENT_ENGINE_ID") != "" {
        realLLM, err := workflow.NewGeminiModel(ctx, cfg.GeminiModel, cfg.GoogleAPIKey, cfg.UseVertexAI, cfg.GoogleCloudProject, cfg.GoogleCloudLocation)
        if err != nil {
            slog.Warn("failed to initialize Gemini model, falling back to mock LLM", "err", err)
            llm = workflow.NewMockLLM()
        } else {
            llm = realLLM
        }
    } else {
        llm = workflow.NewMockLLM()
    }

    return NewRootAgentWithDeps(cardStore, llm)
}

func NewRootAgentWithDeps(cardStore store.CardStore, llm model.LLM) (agent.Agent, error) {
    return workflow.NewCardAssistantAgent(cardStore, llm)
}

Note a clever detail here: when deployed in Agent Runtime, the system automatically injects the GOOGLE_CLOUD_AGENT_ENGINE_ID environment variable. When running unit tests (go test ./...) without an API Key, it automatically falls back to workflow.NewMockLLM(), ensuring offline tests remain 100% fast and deterministic!

3. Agent Runtime Standard HTTP Server and A2A Endpoints (Root `main.go`)

Next, let's look at the new main.go in the project root. When the container is deployed to Agent Runtime (or executed locally via make playground), this entry point mounts five standard services:

OpenTelemetry Observability (setupObservability): Automatically exports Traces, Logs, and Metrics to Google Cloud Trace and Cloud Logging.
Session Service Auto-switching (sessions.SessionService()): In sessions/sessions.go, if VERTEX_AI_SEARCH_ENGINE_ID (Agent Engine ID) is detected, it automatically enables the cloud persistent NewVertexAiEngineSessionService; for local development, it seamlessly uses session.InMemoryService().
ADK REST API and WebUI (/api/ and /ui/): Directly starts the ADK built-in streaming conversation API and web debugging interface via api.NewLauncher() and webui.NewLauncher().
A2A (Agent-to-Agent) Protocol Server: Automatically exposes this Agent's capability description card (AgentCard) at /.well-known/agent-card.json and provides a JSON-RPC call endpoint at /a2a/app!
Agent Structure Introspection Endpoint (GET /app-info): Analyzes the entire Agent Tree via appinfo/appinfo.go. It not only lists the Root Graph Workflow but also registers the CompanyEnrichAgent (gemini-2.5-flash)—which is embedded in CompanyEnrichNode and equipped with geminitool.GoogleSearch{}—into the /app-info response. This allows agents-cli eval and the Google Cloud Console Playground to clearly see all sub-agents and tools within the system!

4. Key Transformation: Enabling Graph Workflow to Support Both "LINE Webhook" and "ADK WebUI / A2A" Dual Channels!

In the first version, our RouterNode assumed all input came from the LINE Webhook Handler, which pre-filled StateKeyInputType or StateKeyImageBytes into the Session State. Furthermore, terminal nodes (like FormatCarouselNode, BuildPreviewNode, SaveCardNode) only wrote results to StateKeyReplyText and StateKeyFlexCarouselJSON for the LINE Bot to retrieve and call ReplyMessage.

However, when we open the ADK WebUI (make playground), the Agent Engine Console Playground, or talk directly to the Root Agent via agents-cli run "Find someone who does AI for me", the input follows the standard ADK ctx.UserContent() (containing text Part.Text or uploaded image Part.InlineData), and the output is read from ev.Content in the event stream!

Therefore, we made two fully backward-compatible enhancements in internal/workflow/nodes_ingest.go and internal/workflow/nodes_search.go:

(1) `RouterNode` Automatically Parses Images and Text from `ctx.UserContent()`

// When uploading a business card image directly from ADK WebUI / A2A / Agent Engine Playground,
// automatically extract InlineData.Data from ctx.UserContent().Parts and direct to RouteIngest!
if len(getBytesState(ctx, StateKeyImageBytes)) == 0 && ctx.UserContent() != nil {
    for _, part := range ctx.UserContent().Parts {
        if part != nil && part.InlineData != nil && len(part.InlineData.Data) > 0 {
            setState(ctx, ev, StateKeyImageBytes, part.InlineData.Data)
            if getStringState(ctx, StateKeyInputType) == "" {
                setState(ctx, ev, StateKeyInputType, RouteIngest)
            }
            break
        }
    }
}

// If it's text input and the input parameter is empty, automatically extract query text or commands from ctx.UserContent().Parts
if inputType == "" {
    sTrim, _ := input.(string)
    sTrim = strings.TrimSpace(sTrim)
    if sTrim == "" && ctx.UserContent() != nil {
        for _, part := range ctx.UserContent().Parts {
            if part != nil && strings.TrimSpace(part.Text) != "" {
                sTrim = strings.TrimSpace(part.Text)
                break
            }
        }
    }
    // Automatically determine if it's searching for a card (RouteSearch) or generating an icebreaker (action=draft_email&card_id=...)
    ...
}

(2) Terminal Nodes Synchronously Write to `ev.Content`

In BuildPreviewNode, SaveCardNode, FormatCarouselNode, and EmailDraftNode, in addition to writing to StateDelta for the LINE Bot, we added a line:

ev.Content = genai.NewContentFromText(flexJSON, genai.RoleModel)

This way:

Sending or searching cards from LINE: The LINE Webhook still gets the Flex JSON from StateDelta to render the beautiful LINE card.
Testing from ADK WebUI (make playground) or Agent Engine Playground: Without needing a LINE account, you can upload a business card image or type "Find someone I met at Cloud Summit who does AI" directly in the browser. You'll see the full execution trajectory (Graph Node transitions) and the output Flex JSON / icebreaker draft directly in the conversation window!

We also added a TestRootAgentDirectSearchExecution unit test in app/agent_test.go, using the standard ADK runner.Runner to send a genai.Content query to app.NewRootAgent, verifying that the business card Carousel JSON can be obtained directly from the Event Stream without going through the LINE Engine!

5. Multi-binary `Dockerfile`: One Image for Both Agent Runtime and LINE Webhook

Finally, to allow the same container image to be deployed to Agent Runtime via agents-cli deploy and to Cloud Run as an independent LINE Bot server, we updated the Dockerfile to compile two static executables in a multi-stage build:

# Build the Agent Runtime / ADK WebUI / A2A entrypoint binary (root main.go)
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" -o /out/agent .

# Also build the standalone LINE Bot Webhook server binary (cmd/server/main.go)
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" -o /out/line-card-assistant ./cmd/server

The default ENTRYPOINT ["/agent"] starts the Agent Runtime standard server (listening on 0.0.0.0:8080).
If it needs to run as a monolithic LINE Webhook Server on Cloud Run, simply specify the Container Command as /bin/line-card-assistant!

4. Summary and Teaser for the Next Step

Through the Phase 1 transformation (PR #10), our LINE AI Business Card Assistant (line-card-assistant) has officially evolved from a simple LINE Bot application into an adk_go Agent that complies with Google Cloud Agent Platform standards:

Full support for the agents-cli ecosystem: Equipped with agents-cli-manifest.yaml and .agents-cli-spec.md.
Dual-channel execution capability: Can interact via LINE chat and directly through the ADK WebUI via make playground, be called by other Agents via the A2A protocol, or undergo multimodal business card recognition and search testing in the Agent Engine Playground.
Cloud-native ready: Built-in VertexAiEngineSessionService auto-switching and OpenTelemetry Cloud Trace tracking capabilities.

In the next article of this series, we will enter Phase 2 & Phase 3, where we will connect HITL pause states and image artifacts to cloud persistent storage and refactor the system into a dual-layer decoupled architecture of "Cloud Run LINE Webhook Gateway + Agent Runtime Core Brain." Stay tuned!

Reference Resources and Further Reading

First article in the series: [Agent Platform in Practice (1)] Building a LINE AI Business Card Assistant with Go and Google ADK 2.0 Graph Workflow: From Fan-Out Parallel Retrieval to HITL Pause Loop Correction
Open source code and Roadmap for this project:
- GitHub Repository: kkdai/line-card-assistant
- Agent Runtime Migration Roadmap: Issue #9
- Phase 1 Implementation Pull Request: PR #10: feat(phase1): scaffold adk_go Agent Runtime project structure and app.NewRootAgent
Google agents-cli and Agent Platform official documentation:
- Getting Started - agents-cli
- The Lifecycle - agents-cli
Google ADK Graph Workflow official documentation:
- Graph Workflows in ADK: Everything You Need to Know (Google Cloud Blog)
- Graph-based agent workflows (adk.dev/graphs)

DEV Community: Evan Lin

[AI in Action] Rewriting a Daily LINE Bot from Python to Go 1.26 on Cloud Run: 1/4 Resource Usage, Google ADK v1.9.0, and Zero-D

Background

1. Setting Rules Before Rewriting: Breaking Down 11 Milestones with GitHub Issues

2. Five Engineering Designs Particularly Noteworthy During the Rewriting Process

1. Dual-track Gemini Client for Vertex AI and Developer API

2. 100% Firestore Schema Compatibility with the Python Version (Key to Zero-Downtime Switching)

3. Native Go Implementation for Multi-format Document to Markdown Conversion (Saying Goodbye to Heavy Python Packages)

3. Facing the Architecture Honestly: Python Version's "Conceptual ADK" vs. Go Version's Actual Implementation of google.golang.org/adk v1.9.0

Next Step: From "Building an ADK Agent Tree" to "Fully Driven by ADK Runner" (Issue #13)

4. Cloud Run Deployment and Slimming Down: Machine Specs Cut Directly to 1/4!

5. Seven Devilish Details Most Likely to be Tripped Over or Missed in Cloud Deployment

Omission 1: Forgot to add --no-cpu-throttling, or passed r.Context() into a background Goroutine

Omission 2: Vertex AI GOOGLE_CLOUD_LOCATION=global Domain Concatenation Trap

Omission 3: Moving Environment Variables Using --set-env-vars Leading to Character Truncation or Secret Leakage

Omission 4: Ignoring Cloud Run Reverse Proxy's X-Forwarded-Proto, Leading to LINE Rejecting Images and Audio

Omission 5: Pursuing Minimalist Go Images (scratch / distroless) but Forgetting External Executables and Chromium Sandbox Restrictions

Omission 6: State Splitting Between In-Memory Caching and Cloud Run Auto-scaling (Multi-Instance)

Omission 7: Only Changing the Messaging API Webhook During Switching, but Missing Firebase Hosting WebSocket Restrictions and LIFF Endpoint

6. Inventory Before Switching: Identifying Three "Zombie Endpoints" from GCP Logs and Planning to Incorporate n8n Schedules!

Decision 1: Immediately remove /hn, /hf, and /urls from linebot-helper-go

Decision 2: Open Issue #14 ~ Issue #19 to incorporate n8n workflows into linebot-helper-go!

7. One Last Security Reminder: Don't Leave Real Cloud Run URLs in Public Git History

8. Cloud Run Deployment and Migration Self-Checklist

Summary and Open Source Project Links

[AI in Practice] Rewriting a LINE Research Bot from Python Managed Agents to Go ADK v1.9.0: From Sandbox Files to Session State,

Recap

I. Clarification: It wasn't actually "Python ADK" before

II. Setting Rules Before Rewriting: One Issue, Feasibility Verification First

III. From Managed Agents to ADK for Go: Four Core Rewrites

1. From "REST Agent Resource Creation" to "In-code llmagent.New"

2. From background=True + Polling to runner.Run() Iterator

3. Sandbox /workspace Files → ADK Session State + Firestore (The Most Critical Step)

4. "Return ONLY the JSON" → OutputSchema

IV. Model Selection: Not every Gemini can do "Search + Read Web + Structured Output" simultaneously

A bug caught when switching to gemini-3.8-flash

V. gcloud and Cloud Run Deployment: Ten Things to Watch Out For

Note 1: describe the online settings first; don't trust the deploy.sh in the repo

Note 2: To "reuse the same environment variables," the simplest way is to pass nothing

Note 3: Use --no-traffic --tag for Canary; tags must be at least 3 characters

Note 4: Cloud Tasks default dispatch deadline is only 10 minutes

Note 5: Under --allow-unauthenticated, Cloud Tasks endpoints must verify OIDC themselves

Note 6: --no-cpu-throttling is NOT needed this time; min-instances can be 0

Note 7: Deployment is possible without local Docker; --source=. uses Cloud Build

Note 8: Same service, no need to change LINE Webhook URL, but verify with API

Note 9: When E2E testing with a fake user, don't let LINE push failures trigger a full pipeline rerun

Note 10: Upgrade to the latest Go patch and use govulncheck

VI. Things Fixed Along the Way: Rewriting is a Great Time to Re-examine Design

VII. One Last Security Reminder: Don't let Cloud Run URLs appear in public Issues

VIII. Python Managed Agents → Go ADK Migration Checklist

Summary and Open Source Project Links

Further Reading

[AI in Practice] Migrating a LINE Research Bot from Go ADK to DeepSeek Harness (DSH): Architecture, Mapping, and Pitfalls

Background

1. Understanding DeepSeek Harness (DSH)

DSH's core design: everything is a Cordis plugin

DSH architecture

What dsh-superagent builds on DSH

2. Feasibility: why I did not simply fork dsh-superagent

3. Migrating from ADK to DSH: architecture and mapping

Post-migration architecture

ADK → DSH mapping

Rewrite 1: runner.Run() → DeepSeekHarness.run()

Rewrite 2: OutputSchema → submit_* tools

Rewrite 3: a LiteLLM sidecar to connect DSH to Gemini

Rewrite 4: a web_research tool to preserve Google Search grounding

4. Pitfalls: what actually went wrong

Pitfall 1: SDK options were renamed, and the old ones are silently ignored

Pitfall 2: plugin names are resolved relative to $DSH_HOME/profiles/<profile>/

Pitfall 3: changing apiKeyEnv in a patch has no effect

Pitfall 4: different profiles cannot share the same sessions directory

Pitfall 5: Node's type stripping does not support constructor parameter properties

Pitfall 6: npx node@24 does not actually run Node 24

Pitfall 7: LiteLLM needs extra Vertex AI dependencies

Pitfall 8: the agent subprocess does not need your secrets

Pitfall 9: it is slower than the ADK version

Pitfall 10: pin pre-release dependencies

5. Deployment and cutover: reusing the existing service's configuration

6. ADK → DSH migration checklist

Summary and open-source links

3. Facing the Architecture Honestly: Python Version's "Conceptual ADK" vs. Go Version's Actual Implementation of `google.golang.org/adk v1.9.0`

Omission 1: Forgot to add `--no-cpu-throttling`, or passed `r.Context()` into a background Goroutine

Omission 2: Vertex AI `GOOGLE_CLOUD_LOCATION=global` Domain Concatenation Trap

Omission 3: Moving Environment Variables Using `--set-env-vars` Leading to Character Truncation or Secret Leakage

Omission 4: Ignoring Cloud Run Reverse Proxy's `X-Forwarded-Proto`, Leading to LINE Rejecting Images and Audio

Omission 5: Pursuing Minimalist Go Images (`scratch` / `distroless`) but Forgetting External Executables and Chromium Sandbox Restrictions

6. Inventory Before Switching: Identifying Three "Zombie Endpoints" from GCP Logs and Planning to Incorporate `n8n` Schedules!

Decision 1: Immediately remove `/hn`, `/hf`, and `/urls` from `linebot-helper-go`

Decision 2: Open Issue #14 ~ Issue #19 to incorporate `n8n` workflows into `linebot-helper-go`!

1. From "REST Agent Resource Creation" to "In-code `llmagent.New`"

2. From `background=True` + Polling to `runner.Run()` Iterator

3. Sandbox `/workspace` Files → ADK Session State + Firestore (The Most Critical Step)

4. "Return ONLY the JSON" → `OutputSchema`

Note 1: `describe` the online settings first; don't trust the `deploy.sh` in the repo

Note 3: Use `--no-traffic --tag` for Canary; tags must be at least 3 characters

Note 5: Under `--allow-unauthenticated`, Cloud Tasks endpoints must verify OIDC themselves

Note 6: `--no-cpu-throttling` is NOT needed this time; min-instances can be 0

Note 7: Deployment is possible without local Docker; `--source=.` uses Cloud Build

Note 10: Upgrade to the latest Go patch and use `govulncheck`

Rewrite 1: `runner.Run()` → `DeepSeekHarness.run()`

Rewrite 2: `OutputSchema` → `submit_*` tools

Rewrite 4: a `web_research` tool to preserve Google Search grounding

Pitfall 2: plugin names are resolved relative to `$DSH_HOME/profiles/<profile>/`

Pitfall 3: changing `apiKeyEnv` in a patch has no effect

Pitfall 6: `npx node@24` does not actually run Node 24

3.3 Where do Harness²'s `c1..c8` map to in DSH?

1. What is Google Research `Harness²`? What can it do?

2. The `c1..c8` Eight Harness Components and the 24KB Budget Limit

2. Inventorying the Current Architecture: Why Phase 1~5 Go Implementation Cannot Be Directly Optimized by `Harness²`?

3. Implementation Part 1: Building `internal/harness` Loader and `harness/base/` Initial Equipment Tree

1. Establishing the `harness/base/` Baseline `c1..c8` File Tree

2. Implementing `internal/harness/harness.go`: 24KB Always-On Budget Validation and On-Demand Skill Injection

2. Why Convert Phase 5 OpenTelemetry Spans into `trajectory.md`? Opening the Black Box of the 13-Node Graph

II. Implementation Part 2: Building the `LineCardAdapter` in Three Steps

1. Implementing `internal/harness/trajectory.go`: Rendering OpenTelemetry Spans as `trajectory.md`

2. Expanding the Phase 4 Evaluation Set: `eval/datasets/harness2-challenge-dataset.json`

3. Implementing `harness/line_card_adapter.py`: Connecting to `google-research/harness2`

Level 1: Execute Go `c1..c8` Surface Constraints and Project-wide Unit Tests

Level 2: Use `cmd/harness-runner` to Validate Any `c1..c8` File Tree

Level 3: Execute the Four-Stage `Harness²` Two-Layer Recursive Evolution Benchmark Experiment

2. Performance Summary: Evolutionary Analysis from `62.5%` to `100.0%`

🔹 Stage 0 $\rightarrow$ Stage 1 (`62.5%` $\rightarrow$ `84.4%`): Diagnosing the first two on-demand skills (`c6`) from OpenTelemetry trajectories

🔹 Stage 1 $\rightarrow$ Stage 2 (`84.4%` $\rightarrow$ `96.9%`): Second round of recursion conquers multi-clause HITL modification and executive email structure

🔹 Stage 2 $\rightarrow$ Stage 3 (`96.9%` $\rightarrow$ `100.0%`): Why is `Sequential + LongTermMEMORY.md` needed to complete the final 3.1%?

3. Future Outlook and Application Scenarios for `Harness²` and Agents

I. What Problem Does Phase 5 Solve? Moving from "Manual `gcloud` Commands" to "Declarative AgentOps"

II. Step 1: Generate Terraform IaC and CI/CD Scaffolding with `agents-cli scaffold enhance`

1. Declaratively Defining Vertex AI Reasoning Engine (`service.tf`)

2. Least Privilege IAM and Secret Manager Integration (`iam.tf` & `variables.tf`)

3. Automatically Creating Cloud Logging $\rightarrow$ BigQuery Telemetry Analysis Dataset (`telemetry.tf`)

IV. Step 3: Injecting OpenTelemetry Distributed Tracing into the 14 Nodes of ADK 2.0 Go Graph Workflow (`internal/workflow/telemetry.go`)

V. Step 4: Building a GitHub Actions CI/CD Pipeline Integrated with `agents-cli eval` Quality Gatekeeper (`.github/workflows/`)

1. PR Quality Gatekeeper (`.github/workflows/pr_checks.yaml`)

2. Staging Automated Deployment and Vegeta Load Testing (`.github/workflows/staging.yaml`)