[Agent Platform in Practice (5)] Building an AI Agent Quality Flywheel: Automated Benchmarking for LINE AI Business Card King wi

Recap

In the first four articles of the "Agent Platform in Action" series, we have step-by-step advanced the LINE AI Business Card Master (line-card-assistant) from a standalone prototype to a cloud-native, dual-layer decoupled Google Cloud Agent Platform architecture:

• [Agent Platform in Action (1)] Building LINE AI Business Card Master with Go and Google ADK 2.0 Graph Workflow: Used google.golang.org/adk/v2/workflow to build a 12-node business card processing graph (Graph Workflow), covering Vision OCR, AddFanOut / AddFanIn parallel background investigations, and the HITL (Human-in-the-Loop) pause and modification loop via NewRequestInputEvent.
• [Agent Platform in Action (2)] From Monolithic Cloud Run to Google Cloud Agent Runtime: Refactoring LINE AI Business Card Master with agents-cli and ADK Go: Completed Phase 1 (PR #10), introducing the agents-cli (adk_go) standard dual-channel architecture, supporting LINE Webhook, ADK WebUI Playground, and the A2A protocol.
• [Agent Platform in Action (3)] Solving HITL Amnesia in Stateless Containers: Implementing Cross-Instance Recognition and Pause/Resume with ADK 2.0 GCS Artifact Service and Agent Platform Sessions: Completed the first half of Phase 2 (PR #11), using artifact/gcsartifact and session/vertexai to enable cross-instance HITL pause and resume for stateless containers.
• [Agent Platform in Action (4)] Towards Cloud Dual-Layer Decoupling and Full Persistence: Completing the Architecture with Cloud Run LINE Webhook Gateway, Agent Runtime, and Cloud SQL PostgreSQL: Completed the second half of Phase 2 (Issue #5) and Phase 3 (Issue #6, PR #12), migrating the business card database to Cloud SQL PostgreSQL (PostgresStore) and implementing the stateless Cloud Run LINE Webhook Gateway (gateway.Client).

With the architecture and features in place and ready to officially deploy the Agent to the cloud production environment (Phase 5), every AI engineering team faces a core soul-searching question:

"When we modify a line of Prompt, adjust a Graph Node's routing condition, or upgrade the underlying model from gemini-2.5-flash to gemini-3.8-flash, how do we ensure that business card OCR fields aren't missed, duplicate card promotion detection doesn't fail, HITL modification loops don't get stuck, and hybrid search doesn't degrade?"

In today's fifth episode of the series, we are going to implement Phase 4 (Issue #7, corresponding to PR #13): using agents-cli eval and ADK 2.0 Go trajectory testing to build a quantifiable, comparable, and CI/CD-automatable AI Agent Quality Flywheel for the LINE AI Business Card Master!

---

I. Why Traditional Unit Tests Aren't Enough? Understanding the agents-cli eval "Quality Flywheel"

In traditional software engineering, we are used to writing deterministic unit tests like if got != want. However, in an AI Agent system driven by ADK 2.0 Graph Workflow + Multimodal LLM, what we need to verify is far more than just the return value of a single function; it is the entire "Execution Trajectory and State Evolution (State Delta)":

1. Multi-branch and Parallel Convergence (Fan-Out / Fan-In Trajectory): When a business card image is received, does the RouterNode correctly enter the VisionOCRNode? Subsequently, are the three parallel branches CleanAndDedupNode, CompanyEnrichNode, and ContextMatchNode triggered simultaneously and converged at JoinPreviewNode?
2. Intermediate State Changes (Session state_delta): Even if the final return to the user is a LINE Flex Message JSON, did the intermediate nodes correctly normalize the Taiwan mobile number ([phone]) and detect an existing contact's promotion (IsDuplicate=true, VP of Engineering $\rightarrow$ Chief Technology Officer (CTO))?
3. Human-in-the-Loop (HITL) Pause and Multi-turn Dialogue Loop: After the HITLConfirmNode issues an adk_request_input pause event, if the user replies "Change title to Technical VP, change occasion to COMPUTEX 2026 AI Forum" and confirms the save, does the workflow actually traverse ApplyEditNode $\rightarrow$ BuildPreviewNode $\rightarrow$ HITLConfirmNode $\rightarrow$ SaveCardNode?
4. Retrieval Degradation and Grounding: When a strict HybridSearchNode finds no results (0 hits), does it automatically degrade to trigger BroadSearchNode? Does the generated icebreaker follow-up email draft truly reference the meeting occasion and company background from the business card without hallucination?

To systematically solve this problem, Google Cloud agents-cli proposes a five-step Quality Flywheel workflow:

flowchart LR
    A["1. Generate<br/>(agents-cli eval generate)<br/>Start ADK server and collect real trajectories"] --> B["2. Grade<br/>(agents-cli eval grade)<br/>Execute Trajectory and quality metric scoring"]
    B --> C["3. Analyze<br/>Review scores for each metric and<br/>Explanation for failed cases"]
    C --> D["4. Fix<br/>Fix Graph Node, Prompt<br/>or event tracking structure"]
    D --> E["5. Compare<br/>(agents-cli eval compare)<br/>Baseline vs. Candidate quantitative comparison"]
    E -->|Continuous Iteration| A

---

II. Revealing How agents-cli eval generate Tests a Go-based ADK 2.0 Agent?

Many Go developers new to agents-cli wonder: "The agents-cli is a CLI tool written in Python; how can it evaluate my adk/v2/workflow Agent written in Go?"

The answer lies within the main.go and appinfo/appinfo.go we introduced in Phase 1! When you execute:

DRY_RUN=true agents-cli eval generate --dataset eval/datasets/basic-dataset.json

agents-cli automatically completes the following four steps in the background:

1. Automatically compile and start your Go Agent server: agents-cli reads the agents-cli-manifest.yaml in the project root (identified as template: adk_go) and automatically starts in the background:

go run . web --port <random_port> api -path_prefix / a2a appinfo

1. Introspect Agent structure via GET /apps/app/app-info reflection: appinfo/appinfo.go uses Go's reflect and unsafe to traverse the Edges and Nodes of the entire workflowagent, assembling the embedded CompanyEnrichAgent (llmagent), Instructions, and tool declarations into a standard AppInfo JSON returned to agents-cli.
2. Create Sessions one by one and call POST /run_sse to collect event streams: For each test case (eval_case_id) in eval/datasets/basic-dataset.json, agents-cli first calls POST /apps/app/users/eval_user/sessions to create an independent Session, then sends the test case's prompt to POST /run_sse, and records every ADK Event (including author, content, actions.state_delta, actions.requested_tool_confirmations, etc.) streamed back from the server.
3. Output standardized trace file artifacts/traces/traces_<timestamp>.json: After testing is complete, it automatically shuts down the Go server and saves the full trajectory as a JSON, passing it to the next stage, agents-cli eval grade, for scoring.

---

III. Practical Design: A Golden Evaluation Set Covering 4 Core Scenarios (eval/datasets/basic-dataset.json)

To fully cover the four core acceptance scenarios defined in Issue #7, we designed 6 Golden Eval Cases in eval/datasets/basic-dataset.json, marking the expected node trajectory (expected_nodes) and state assertions in each case's custom_Properties:

Eval Case ID	Corresponding Core Scenario	Input Prompt (Simulated Channel Signal)	Verification Focus (custom_Properties)
case_1_ingest_fanout_fanin_ocr	Scenario 1: Business Card Recognition and Parallel Enhancement	action=ingest&context=2026 Cloud Summit Taipei	Verify RouterNode $\rightarrow$ VisionOCRNode $\rightarrow$ Parallel Fan-Out (CleanAndDedupNode, CompanyEnrichNode, ContextMatchNode) $\rightarrow$ BuildPreviewNode $\rightarrow$ HITLConfirmNode pause, and phone normalization to [phone].
case_2_duplicate_promotion_detection	Scenario 2: Duplicate Card and Promotion Detection	action=ingest&card=promotion_david_chen_appier&context=2026 AI DevFest Taipei	Test database already contains David Chen from Appier (VP of Engineering); when a new card (Chief Technology Officer (CTO)) is received, CleanAndDedupNode must mark IsDuplicate=true and generate a promotion alert message.
case_3_hitl_edit_and_confirm_loop	Scenario 3: HITL Loop Correction and Saving	action=ingest&context=2026 Cloud Summit Taipei&edit=Change title to Technical VP, change occasion to COMPUTEX 2026 AI Forum&hitl_action=confirm	Verify from OCR preview $\rightarrow$ ApplyEditNode semantic modification of title and occasion $\rightarrow$ BuildPreviewNode re-rendering preview $\rightarrow$ HITLConfirmNode confirmation $\rightarrow$ SaveCardNode writing to database and generating follow-up email.
case_4a_hybrid_search_cloud_summit_ai	Scenario 4a: Natural Language Hybrid Retrieval	Help me find people doing AI I met at Cloud Summit	Verify QueryIntentNode converts colloquial speech to SearchFilter $\rightarrow$ HybridSearchNode hits 2 business cards $\rightarrow$ FormatCarouselNode outputs Flex Carousel JSON.
case_4b_broad_search_fallback	Scenario 4b: 0 Hits Automatic Degradation to Fuzzy Retrieval	Help me find Zhang San who does quantum finance blockchain I met at Tokyo Game Show	Verify that when HybridSearchNode finds no results (0 hits), conditional routing automatically degrades to BroadSearchNode for Token fuzzy retrieval to avoid blank replies.
case_4c_personalized_icebreaker_email	Scenario 4c: Personalized Icebreaker Follow-up Email	action=draft_email&card_id=card-seed-1	Verify EmailDraftNode writes a hallucination-free business icebreaker email based on the specified card's MetContext (2026 Cloud Summit Taipei) and CompanyInfo (Google Cloud Taiwan).

---

IV. Real-world Pitfalls and Debugging: Why was the Trajectory Score 0.0000 on the First agents-cli eval grade Run?

After writing eval/eval_config.yaml and eval/metrics.py, we confidently ran the first Baseline evaluation:

DRY_RUN=true agents-cli eval run --config eval/eval_config.yaml

The first Baseline scorecard (results_20261022_223104.json) printed in the terminal shocked us:

╭─ Evaluation Summary ───────────────────────────╮
│ Metric Score Valid │
│ ────────────────────────────────────────────── │
│ workflow_trajectory_accuracy 0.0000 6/6 │
│ ocr_and_enrichment_quality 1.0000 6/6 │
│ duplicate_and_hitl_precision 0.7333 6/6 │
│ search_and_email_grounding 1.0000 6/6 │
╰────────────────────────────────────────────────╯

At the same time, a very critical warning message appeared in the agents-cli eval grade execution log:

Dropped 26 event(s) with no content; the eval service rejects them

🔍 Root Cause: It turns out drop_empty_content_events filtered out intermediate node events!

Why were 26 events dropped? We immediately checked the implementation of ADK Go workflow.FunctionNode and the event preprocessing logic of agents-cli:

1. In our original internal/workflow/nodes_ingest.go and internal/workflow/nodes_search.go, Intermediate Nodes like RouterNode, VisionOCRNode, CleanAndDedupNode, CompanyEnrichNode, ContextMatchNode, and QueryIntentNode were only responsible for writing results into ev.Actions.StateDelta (e.g., ev.Actions.StateDelta["draft_card"] = ...), while keeping ev.Content as nil; only the final output node (such as BuildPreviewNode, SaveCardNode, FormatCarouselNode) would set ev.Content.
2. However, before agents-cli eval grade sends the trajectory to the grader, the underlying drop_empty_content_events()—to comply with the Schema constraints of the cloud Vertex AI Eval Service—will directly remove all events where event.content == null from agent_data.turns[].events!
3. Once intermediate node events are removed because content == nil, the actions.state_delta and node execution records attached to those events also disappear! This caused the grader to only see the output of the last node, resulting in a workflow_trajectory_accuracy of 0.0000!

🛠️ Elegant Solution: Adding Structured Trajectory Summaries setNodeTrace to Intermediate Nodes

Once the root cause was found, the solution was clear and elegant! We designed a lightweight helper function setNodeTrace in internal/workflow/nodes_ingest.go:

// setNodeTrace attaches a concise model text part to intermediate workflow node
// events so that `agents-cli eval` (which drops events with empty Content)
// retains the full node execution trajectory and state_delta history.
func setNodeTrace(ev *session.Event, nodeName, summary string) {
    if ev == nil || ev.Content != nil {
        return
    }
    ev.Content = genai.NewContentFromText(fmt.Sprintf("[%s] %s", nodeName, summary), genai.RoleModel)
}

Then, we call setNodeTrace before each intermediate node finishes, for example:

• RouterNode: setNodeTrace(ev, "RouterNode", "route="+route)
• VisionOCRNode: setNodeTrace(ev, "VisionOCRNode", fmt.Sprintf("ocr extracted name=%s company=%s title=%s", card.Name, card.Company, card.Title))
• CleanAndDedupNode: setNodeTrace(ev, "CleanAndDedupNode", fmt.Sprintf("phone=%s is_duplicate=%v", card.Phone, isDup))
• CompanyEnrichNode: setNodeTrace(ev, "CompanyEnrichNode", fmt.Sprintf("company_info=%s tags=%v", card.CompanyInfo, card.Tags))
• ContextMatchNode: setNodeTrace(ev, "ContextMatchNode", "met_context="+card.MetContext)
• QueryIntentNode / HybridSearchNode / BroadSearchNode: Record the converted SearchFilter and the number of business cards hit.

At the same time, to prevent the [NodeName] ... trajectory text of intermediate nodes from overwriting the ReplyText that Engine.executeTurn ultimately returns to the LINE caller, we simultaneously strengthened the state convergence logic in internal/workflow/graph.go—allowing TurnResult to read authoritative StateKeyReplyText, StateKeyFlexPreviewJSON, StateKeyFlexCarouselJSON, and StateKeyDraftEmail directly from the final converged Session.State!

---

V. Turning the Quality Flywheel: From Baseline to Candidate, All Four Metrics Reached 1.0000!

After adding setNodeTrace and supporting single-turn batch HITL modification verification, we re-executed DRY_RUN=true agents-cli eval run --config eval/eval_config.yaml:

Loaded 6 cases from eval/datasets/basic-dataset.json
Starting local Go ADK server...
Running 6 cases against http://127.0.0.1:56649 (app=app, concurrency=4)...
Wrote 6 traces to artifacts/traces/traces_20261022_223515.json
Loaded 6 trace(s) from artifacts/traces/traces_20261022_223515.json
Evaluating metrics: ['workflow_trajectory_accuracy', 'ocr_and_enrichment_quality', 'duplicate_and_hitl_precision', 'search_and_email_grounding']

╭─ Evaluation Summary ───────────────────────────╮
│ Metric Score Valid │
│ ────────────────────────────────────────────── │
│ workflow_trajectory_accuracy 1.0000 6/6 │
│ ocr_and_enrichment_quality 1.0000 6/6 │
│ duplicate_and_hitl_precision 1.0000 6/6 │
│ search_and_email_grounding 1.0000 6/6 │
╰────────────────────────────────────────────────╯

This time, Dropped 0 event(s), and all intermediate node trajectories and state_delta were 100% preserved! We immediately used the 5th step of the Quality Flywheel, agents-cli eval compare, to quantitatively compare the Baseline before the fix with the Candidate after the fix:

agents-cli eval compare \
  --baseline artifacts/grade_results/results_20261022_223104.json \
  --candidate artifacts/grade_results/results_20261022_223515.json


╭─ Summary Comparison ─────────────────────────────────────────────────────────╮
│ Metric Baseline Candidate Delta │
│ ──────────────────────────────────────────────────────────────────────────── │
│ duplicate_and_hitl_precision 0.7333 1.0000 +0.2667 │
│ ocr_and_enrichment_quality 1.0000 1.0000 0.0000 │
│ search_and_email_grounding 1.0000 1.0000 0.0000 │
│ workflow_trajectory_accuracy 0.0000 1.0000 +1.0000 │
╰──────────────────────────────────────────────────────────────────────────────╯

Advancing from 0.0000 / 0.7333 to all four metrics reaching 1.0000 (+1.0000 / +0.2667) is exactly the most charming value of the agents-cli eval quality flywheel—making every architectural adjustment traceable and provable!

---

VI. Dual-Track Verification: Python Custom Graders (eval/metrics.py) + Go Native Multi-turn Trajectory Tests (eval/eval_test.go)

In the eval/ directory, we adopted a "Python agents-cli eval + Go go test dual-track verification architecture":

1. eval/eval_config.yaml and eval/metrics.py: Four Custom Evaluation Metrics

In eval/eval_config.yaml, we registered 4 deterministic trajectory and quality graders (plus 1 card_assistant_llm_rubric available for online Vertex AI Judge models):

metrics_to_run:
  - workflow_trajectory_accuracy
  - ocr_and_enrichment_quality
  - duplicate_and_hitl_precision
  - search_and_email_grounding

custom_metrics:
  workflow_trajectory_accuracy:
    type: custom
    module: eval.metrics
    function: evaluate_workflow_trajectory
  ocr_and_enrichment_quality:
    type: custom
    module: eval.metrics
    function: evaluate_ocr_and_enrichment
  duplicate_and_hitl_precision:
    type: custom
    module: eval.metrics
    function: evaluate_duplicate_and_hitl
  search_and_email_grounding:
    type: custom
    module: eval.metrics
    function: evaluate_search_and_email

In eval/metrics.py, we defined helper functions _extract_executed_nodes(instance) and _merged_state_delta(instance) to parse the actual node sequence traversed by the Graph Workflow and the cumulative Session State directly from agent_data.turns[].events:

def _extract_executed_nodes(instance: dict[str, Any]) -> list[str]:
    """Extract executed workflow node names in order from `[NodeName] ...` traces."""
    nodes: list[str] = []
    for ev in _iter_events(instance):
        content = ev.get("content") or {}
        for part in content.get("parts") or []:
            if isinstance(part, dict) and isinstance(part.get("text"), str):
                for match in NODE_TRACE_PATTERN.findall(part["text"]):
                    if not nodes or nodes[-1] != match:
                        nodes.append(match)
    return nodes

This way, evaluate_workflow_trajectory can accurately check:

• Whether each node was executed in topological order (Subsequence Order)?
• Whether all expected_state_keys (such as draft_card, flex_preview_json, saved_card_id, search_results) were written to state_delta?
• Whether the final output simultaneously satisfies expected_response_substrings and excludes forbidden_substrings?

2. eval/eval_test.go: Go Native Multi-turn HITL Pause/Resume and /app-info Introspection Tests

In addition to agents-cli eval called via HTTP SSE, we also wrote Go native TestEvalDatasetAndAppInfoIntrospection and TestPhase4WorkflowTrajectorySuite in eval/eval_test.go, which can be executed in milliseconds via go test ./... or GitHub Actions CI:

• TestEvalDatasetAndAppInfoIntrospection: Verifies the Schema integrity of eval/datasets/basic-dataset.json and directly calls appinfo.BuildAppInfo to confirm that the CompanyEnrichAgent within the line_card_assistant graph can be correctly introspected via reflection by /apps/{app}/app-info.
• TestPhase4WorkflowTrajectorySuite:
  • Case1_IngestFanOutFanInOCR
  • Case2_DuplicatePromotionDetection
  • Case3_MultiTurnHITLEditAndConfirmLoop (Actually calls StartIngest $\rightarrow$ ResumeHITL(edit) $\rightarrow$ ResumeHITL(confirm) in three turns to verify the cross-turn Pause/Resume state machine)
  • Case4_SearchHybridAndBroadFallbackAndEmailDraft

Now, with just one line make eval, you can run both agents-cli eval run and the Go native test suite:

$ make eval
DRY_RUN=true agents-cli eval run --config eval/eval_config.yaml
...
go test -v ./eval/...
=== RUN TestEvalDatasetAndAppInfoIntrospection
--- PASS: TestEvalDatasetAndAppInfoIntrospection (0.00s)
=== RUN TestPhase4WorkflowTrajectorySuite
    --- PASS: TestPhase4WorkflowTrajectorySuite/Case1_IngestFanOutFanInOCR (0.00s)
    --- PASS: TestPhase4WorkflowTrajectorySuite/Case2_DuplicatePromotionDetection (0.00s)
    --- PASS: TestPhase4WorkflowTrajectorySuite/Case3_MultiTurnHITLEditAndConfirmLoop (0.00s)
    --- PASS: TestPhase4WorkflowTrajectorySuite/Case4_SearchHybridAndBroadFallbackAndEmailDraft (0.00s)
PASS
ok github.com/kkdai/line-card-assistant/eval    0.436s

---

VII. Summary and Next Preview (Phase 5 Final Chapter: Official Deployment, Terraform IaC, and Cloud Trace Observability)

As of now, our LINE AI Business Card Master Roadmap (Issue #9) has completed the first four major stages:

• Phase 1: agents-cli environment and adk_go project Scaffolding (PR #10)
• Phase 2: ADK Go Graph Workflow adaptation for Agent Runtime, GCS Artifacts, Agent Platform Sessions, and Cloud SQL PostgresStore (PR #11, PR #12)
• Phase 3: Cloud Run LINE Webhook Gateway and Agent Runtime dual-layer decoupled architecture (PR #12)
• Phase 4: agents-cli eval automated evaluation and quality flywheel (PR #13)

In the next article (which is also the final chapter of this series, Phase 5 — Issue #8), we will officially push the LINE AI Business Card Master, which passed the evaluation with a perfect 1.0000 score, into the production environment via agents-cli deploy. We will also fully integrate Terraform IaC (Infrastructure as Code), GitHub Actions CI/CD automated evaluation and deployment pipelines, and OpenTelemetry + Google Cloud Trace full-link observability. Stay tuned!

---

Reference Resources

• GitHub Repository: kkdai/line-card-assistant
• Phase 4 Pull Request: PR #13 — feat(eval): add Phase 4 agents-cli eval suite and Go workflow trajectory tests (#7)
• Google Cloud agents-cli Official Documentation: The Quality Flywheel & Agent Evaluation
• Google ADK Official Documentation: Graph-based Agent Workflows (adk/v2/workflow)