A Layered Evaluation Strategy for LLM Agents (Google ADK's 12 Criteria)

Google's Agent Development Kit (ADK) ships 12 evaluation criteria for testing agent behavior: tool-call trajectories, final response quality, hallucination detection, safety, multi-turn task success, and more. The natural instinct is to wire all 12 into your PR pipeline and call it done. That instinct is wrong, and the reason why is worth understanding before you build an eval suite for any agent, ADK or not.

The 12 criteria, grouped by what they actually measure

ADK's criteria split cleanly along two axes: does it need a golden reference or rubric, and does it call an LLM-as-a-Judge.

Criteria	Golden Reference	Rubric	LLM Judge	User Simulation
tool_trajectory_avg_score	Yes	No	No	No
response_match_score	Yes	No	No	No
final_response_match_v2	Yes	No	Yes	No
rubric_based_final_response_quality_v1	No	Yes	Yes	Yes
rubric_based_tool_use_quality_v1	No	Yes	Yes	Yes
rubric_based_multi_turn_trajectory_quality_v1	No	Yes	Yes	Yes
hallucinations_v1	No	No	Yes	Yes
safety_v1	No	No	Yes	Yes
per_turn_user_simulator_quality_v1	No	No	Yes	Yes
multi_turn_task_success_v1	No	No	Yes	Yes
multi_turn_trajectory_quality_v1	No	No	Yes	Yes
multi_turn_tool_use_quality_v1	No	No	Yes	Yes

Ten out of twelve depend on an LLM Judge. That single fact is why "run everything on every PR" doesn't survive contact with a real pipeline.

Why full coverage in CI breaks down

Three structural constraints show up as soon as you try:

• Latency. LLM-as-a-Judge and User Simulation cases take tens of seconds to minutes each. Multiply by a real test matrix and PR turnaround goes from minutes to an hour-plus.
• Cost. For N test cases and T turns, a User Simulation + Judge suite costs roughly O(N × T) in tokens. Running that on every commit is not something FinOps will sign off on.
• Flakiness. Even at temperature=0.0, Judge and Simulator outputs drift. A red CI run stops meaning "the code is broken" — it might just mean the Judge had an off moment. Once developers can't trust red, they start ignoring it.

There's also a subtler problem with the two deterministic criteria (tool_trajectory_avg_score, response_match_score). response_match_score uses ROUGE-1 — raw unigram overlap — which is brittle against formatting or phrasing changes and produces false negatives whenever you touch a prompt. And tool_trajectory_avg_score in EXACT or IN_ORDER mode punishes an agent for finding a more efficient (e.g., parallelized) tool-call order after a model upgrade. Strict matching modes actively discourage improvement.

The fix: layer by execution frequency and determinism

The resolution is a three-layer pyramid, ordered by how often it runs and how deterministic it is.

                 Layer 3: E2E / Scenario Eval        [weekly / pre-release]
                 - User Simulation                    high cost / non-deterministic
                 - multi_turn_task_success
            ------------------------------------
              Layer 2: Nightly / Quality & Safety     [daily]
              - hallucinations_v1 / safety_v1          medium cost / LLM Judge
              - rubric_based_final_response
            ------------------------------------
            Layer 1: PR / Pre-commit (deterministic CI) [per commit/PR]
            - tool_trajectory_avg_score (ANY_ORDER)     low cost / deterministic

Layer 1 — PR gate: no LLM calls at all

The only criterion here is tool_trajectory_avg_score, run in ANY_ORDER (not EXACT) mode — asserting only that the required tools fired, not the exact sequence. External APIs are mocked. Zero LLM calls, zero dollars, sub-minute runtime.

import pytest
from google.adk.evaluation import AgentEvaluator
from my_agent.agent import root_agent

@pytest.mark.asyncio
async def test_layer1_tool_trajectory():
    """PR/CI gate: deterministic tool-call check, no LLM involved."""
    evaluator = AgentEvaluator(
        agent=root_agent,
        config_path="tests/eval_cases/layer1_ci.json"
    )
    results = await evaluator.run_evaluation()
    for result in results.case_results:
        score = result.metrics["tool_trajectory_avg_score"].score
        assert score == 1.0, f"Failed case {result.case_name}: score={score}"

layer1_ci.json pins the config that makes this deterministic:

{
  "criteria": {
    "tool_trajectory_avg_score": {
      "threshold": 1.0,
      "match_type": "ANY_ORDER"
    }
  }
}

Layer 2 — Nightly: quality, hallucination, safety

This is where LLM-as-a-Judge enters, against a fixed, versioned dataset (hundreds of cases, not the full corpus). Two things matter for keeping this layer trustworthy: pin the Judge model and its temperature, and track score deltas over time rather than just absolute thresholds.

evaluator = AgentEvaluator(
    agent=root_agent,
    config_path="tests/eval_cases/layer2_nightly.json",
    judge_model="gemini-1.5-pro",
    judge_model_config={"temperature": 0.0},
)
results = await evaluator.run_evaluation()

for result in results.case_results:
    assert result.metrics["hallucinations_v1"].score >= 0.9
    assert result.metrics["safety_v1"].score == 1.0
    rubric = result.metrics["rubric_based_final_response_quality_v1"]
    assert rubric.score >= 0.8, rubric.reasoning

A rubric criterion needs an explicit, weighted rubric — not a vague adjective. This is the part teams get wrong most often:

{
  "rubric_based_final_response_quality_v1": {
    "threshold": 0.8,
    "rubrics": [
      {"name": "conclusion_first", "description": "Does the first 1-2 sentences state a clear conclusion?", "weight": 0.3},
      {"name": "formatting_structure", "description": "Is the response structured with Markdown lists/headings?", "weight": 0.3},
      {"name": "technical_explanation", "description": "Are technical terms briefly defined in context?", "weight": 0.4}
    ]
  }
}

Write rubric items as objectively checkable conditions, not subjective adjectives:

• Bad: "explained in an easy-to-understand way"
• Good: "each technical term is followed by a one-sentence definition, either in parentheses or the next sentence"

The Judge LLM receives the query, the response, and the weighted rubric list, and returns per-item scores plus a weighted average — so a failing assertion comes with a reasoning string you can read directly, not just a number.

Layer 3 — pre-release E2E: multi-turn simulation

This layer runs a User Simulator LLM against the agent across several turns and checks goal completion (multi_turn_task_success_v1) plus process quality (multi_turn_trajectory_quality_v1, multi_turn_tool_use_quality_v1).

evaluator = AgentEvaluator(
    agent=root_agent,
    config_path="tests/eval_cases/layer3_e2e.json",
    user_simulator_model="gemini-1.5-flash",
    judge_model="gemini-1.5-pro",
    judge_model_config={"temperature": 0.0},
)
results = await evaluator.run_evaluation()

for result in results.case_results:
    sim_quality = result.metrics["per_turn_user_simulator_quality_v1"].score
    if sim_quality < 0.8:
        pytest.skip(f"Simulator degraded in {result.case_name}")
    assert result.metrics["multi_turn_task_success_v1"].score == 1.0
    assert result.metrics["multi_turn_trajectory_quality_v1"].score >= 0.8

Note the pytest.skip guard on per_turn_user_simulator_quality_v1. This layer has a structural gotcha worth calling out explicitly.

The gotcha: meta-evaluation error propagation

per_turn_user_simulator_quality_v1 exists because this layer has three LLMs talking past each other: a Simulator LLM playing the user, the Agent under test, and a Judge LLM scoring the outcome. That's a "Judge judging a Judge" structure — if the Simulator goes off-script (loops, ignores its own goal, hallucinates a constraint), the resulting task_success score reflects the Simulator's failure, not the agent's. Skipping low-quality-simulation sessions before asserting on task success is not optional defensive coding — without it, a bad night for the Simulator model silently fails your agent's release gate.

Wiring it into CI/CD

name: Agent Evaluation Pipeline
on:
  pull_request:
    branches: [ main ]
  schedule:
    - cron: '0 18 * * *'  # nightly

jobs:
  layer1-ci:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.11' }
      - run: pip install -r requirements.txt
      - run: pytest tests/test_layer1_ci.py

  layer2-layer3-nightly:
    if: github.event_name == 'schedule'
    needs: layer1-ci
    runs-on: ubuntu-latest
    env:
      GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.11' }
      - run: pip install -r requirements.txt
      - run: pytest tests/test_layer2_nightly.py
      - run: pytest tests/test_layer3_e2e.py

Only Layer 1 blocks the PR. Layers 2 and 3 run on schedule, against main, and their failures route to a human triage step rather than a red PR check.

Where to put the eval code

A recurring question: does eval code belong under tests/? The honest answer is that tests/ and evals/ are different disciplines and benefit from being separated at the top level:

.
├── src/
│   └── my_agent/
├── tests/                 # deterministic, mocked, no API calls
│   ├── test_tools.py
│   └── test_state_management.py
└── evals/                 # probabilistic, real API/Simulator calls
    ├── datasets/
    ├── metrics/            # custom rubric definitions
    └── run_evals.py

tests/ should stay fast and free so a developer can run it locally without worrying about API spend. If eval code does live under tests/, mark it explicitly (pytest.mark.eval) so pytest -m "not eval" stays the default local loop.

Summary

Only 1-2 of ADK's 12 criteria are realistic PR-blocking gates — tool_trajectory_avg_score in ANY_ORDER mode, backed by deterministic assertions and mocks. Everything that touches an LLM Judge or a User Simulator belongs in a nightly or pre-release layer, versioned against a fixed dataset, with the Judge model and temperature pinned for reproducibility. If your eval suite tries to be a single flat list run on every commit, expect flaky red builds and a PR queue nobody trusts — the fix isn't a better assertion, it's a different pyramid.

Comments

[Reid Marlow]

I like the PR versus nightly split here. The extra trap is making the cheap layer too golden-path, so it only proves the demo still works. I’d keep a few boring deterministic fixtures for tool contracts, then reserve judge runs for cases where the failure cost is genuinely ambiguous.