Migrating Google ADK to Telemetry Schema v2 Silently Broke Every Analytics Query

Upgrading Google ADK from 2.7 to 2.13, I also flipped telemetry from ADK's own schema v1 to schema v2, which follows the OpenTelemetry GenAI semantic conventions.

The upgrade itself was uneventful. What was not: every BigQuery query reading our Cloud Trace spans started returning zero rows. No errors, no warnings. WHERE name = 'call_llm' simply stopped matching anything, because that span no longer exists.

Here are the six traps I hit, in the order they cost me time.

The setup: spans read from BigQuery

Some context on the pipeline, because it explains why the breakage was silent.

Agent spans go to Cloud Trace, and we query them from BigQuery with SQL. We want to aggregate "which agent, with what instruction, what the user said, what the model answered" after the fact, so eyeballing the trace UI is not enough.

  Agent (Cloud Run)
        │  OTel spans
        ▼
  SpanProcessor wrapper  ──  shrinks attributes over 64 KiB
        │  export
        ▼
  Cloud Trace  _Trace bucket  (30-day retention)
        │
        │  google_observability_link
        ▼
  BigQuery  dataset: cloud_trace_spans
        └── _AllSpans   (a VIEW — no copy of the data)
                │  bq query --location=us
                ▼
          analytics queries

A common misreading: this is not a Logging sink. Nothing is routed through the log router into BigQuery. It is a linked dataset attached to Cloud Trace's _Trace bucket:

resource "google_observability_link" "cloud_trace_spans" {
  location = "us"
  bucket   = "_Trace"
  dataset  = "Spans"
  link_id  = "cloud_trace_spans"
}

# The link needs BigQuery permission on the observability service agent
resource "google_project_iam_member" "observability_bigquery_admin" {
  project = var.project_id
  role    = "roles/bigquery.admin"
  member  = "serviceAccount:service-${data.google_project.project.number}@gcp-sa-observability.iam.gserviceaccount.com"
}

Three properties of this setup matter later:

• _AllSpans is a VIEW, not a table. No data is copied, so retention follows Cloud Trace (30 days by default). There is no "dig into last year's data".
• The dataset lives in us. Without --location=us, bq query tells you the dataset does not exist.
• All span attributes arrive in a single JSON column called attributes. So every analytics query looks like this:

SELECT JSON_VALUE(attributes, '$."gen_ai.input.messages"') AS input
FROM `<project>.cloud_trace_spans._AllSpans`
WHERE start_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)

That last property is the direct cause of the incident. JSON paths have no schema and no type checking. Point JSON_VALUE at a key that does not exist and it returns NULL without complaint. Misspell a real column name and BigQuery stops you instantly; misspell a JSON path and it says nothing at all.

So when schema v2 renamed every attribute, the queries kept succeeding and started returning zero rows. That is how an entire analytics pipeline breaks without anyone noticing.

What actually changes

Schema v2 is gated behind ADK_TELEMETRY_SCHEMA_VERSION_OPT_IN=2. The mapping:

	v1	v2
LLM call span	call_llm	generate_content <model>
Agent span	invocation / agent_run [name]	invoke_agent <name> / invoke_workflow <name>
Tool span	execute_tool <name>	execute_tool <name> (unchanged)
LLM request body	gcp.vertex.agent.llm_request	gen_ai.input.messages
LLM response body	gcp.vertex.agent.llm_response	gen_ai.output.messages
System instruction	inside llm_request	gen_ai.system_instructions
Tool definitions	inside llm_request	gen_ai.tool.definitions
Tool arguments	gcp.vertex.agent.tool_call_args	gen_ai.tool.call.arguments
Tool result	gcp.vertex.agent.tool_response	gen_ai.tool.call.result
Agent name	embedded in span name	gen_ai.agent.name

v1 stuffs whole JSON blobs into single attributes; v2 splits them by meaning. The direction is right, but every reference you have breaks at once.

Note the dots in gen_ai.tool.definitions. I wrote gen_ai.tool_definitions in a design doc and nobody caught it until someone ran it. Pull the constant instead of trusting your memory:

from opentelemetry.semconv._incubating.attributes import gen_ai_attributes as g
print(g.GEN_AI_TOOL_DEFINITIONS)  # gen_ai.tool.definitions

Trap 1: without SPAN_ONLY, no content is recorded at all

This one cost the most. After switching to v2, spans appeared but every content attribute was empty.

In v2, content capture is governed by the OTel-side variable OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT:

Value	Behavior
NO_CONTENT (default)	Records nothing
SPAN_ONLY	Records into span attributes
EVENT_ONLY	Records into log records (OTel events)
SPAN_AND_EVENT	Both

ADK's own ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=true is a v1-only knob. If you migrated from v1, that one is probably already set to true, which makes the failure look especially confusing: the setting is right there, and nothing is captured.

The tempting choice is SPAN_AND_EVENT — why not have both? Because the event path needs a LoggerProvider, and ADK only creates one when log record processors exist:

# google/adk/telemetry/setup.py
if log_record_processors:
    # LoggerProvider is created only here
    ...

If you call get_gcp_exporters(enable_cloud_tracing=True) and nothing else, you have zero log record processors, so there is no LoggerProvider and nowhere for those events to go.

For a trace-only setup, SPAN_ONLY is the only sensible value:

env {
  name  = "OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT"
  value = "SPAN_ONLY"
}

env {
  name  = "ADK_TELEMETRY_SCHEMA_VERSION_OPT_IN"
  value = "2"
}

Trap 2: v1 and v2 bodies live in different ROWS, not different columns

During a migration you have both schemas in the same table, so you reach for a fallback:

-- This does nothing useful
COALESCE(
  JSON_VALUE(attributes, '$."gcp.vertex.agent.llm_request"'),
  JSON_VALUE(attributes, '$."gen_ai.input.messages"')
) AS llm_input

COALESCE picks the first non-NULL within one row. But the v1 body sits on a call_llm row and the v2 body sits on a generate_content ... row. The two attributes never coexist in a single row, so this expression only ever returns whichever one that row already had. It is a fallback that cannot fall back.

Branch on the span name and treat them as separate rows:

WHERE name = 'call_llm' OR STARTS_WITH(name, 'generate_content ')

And extract differently per shape:

CASE
  WHEN name = 'call_llm' THEN
    -- v1: llm_request.contents[last].parts[].text
    (SELECT STRING_AGG(JSON_VALUE(pt, '$.text'), ' / ')
     FROM UNNEST(JSON_QUERY_ARRAY(last_content, '$.parts')) AS pt)
  ELSE
    -- v2: input_messages[last].parts[].content
    (SELECT STRING_AGG(JSON_VALUE(pt, '$.content'), ' / ')
     FROM UNNEST(JSON_QUERY_ARRAY(last_message, '$.parts')) AS pt)
END AS user_text

The inner keys differ too: a v1 part uses text, a v2 part uses content. The assistant role is model in v1 and assistant in v2, per semconv.

Meanwhile, tool arguments and results do share a row, because execute_tool <name> keeps the same span name across both schemas. There, COALESCE is exactly right:

COALESCE(
  JSON_VALUE(attributes, '$."gcp.vertex.agent.tool_call_args"'),
  JSON_VALUE(attributes, '$."gen_ai.tool.call.arguments"')
) AS tool_args

So the same migration needs two different fallback strategies depending on which span you are reading. You will not deduce this from the docs; you have to capture real spans and look.

Trap 3: resolving the agent name needs parent traversal, not trace_id

v1 embedded the agent in the span name (agent_run [SomeLeafAgent]). v2 exposes gen_ai.agent.name as its own attribute — but it is not on v1's call_llm. To cover both during migration, walk up the parent chain:

SELECT b.*,
  COALESCE(
    JSON_VALUE(b.attributes, '$."gen_ai.agent.name"'),
    JSON_VALUE(p.attributes, '$."gen_ai.agent.name"'),
    JSON_VALUE(g.attributes, '$."gen_ai.agent.name"')
  ) AS agent_name
FROM base b
LEFT JOIN base p ON b.parent_span_id = p.span_id   -- parent
LEFT JOIN base g ON p.parent_span_id = g.span_id   -- grandparent

Do not join on trace_id. A single trace contains the router agent and any AgentTool children, so a trace-wide join happily attributes one agent's spans to another.

In practice, every v2 generate_content span in my capture carried gen_ai.agent.name directly. The parent traversal is insurance for the v1 rows during the overlap window.

Trap 4: the 64 KiB attribute limit rejects the whole batch

GCP's Telemetry API refuses any attribute over 64 KiB — and it does not drop just that attribute. The entire batch comes back HTTP 400. One oversized system instruction erases every span that shipped with it.

v1 was easy to trip because it packed everything into one attribute. v2 does not really help: gen_ai.system_instructions and gen_ai.tool.definitions get large on their own. Mine reached 63,427 bytes.

The obvious lever, OTel SDK's OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT, is the wrong tool. It truncates by character count with no UTF-8 byte guarantee, and it cuts JSON mid-structure, so what survives no longer parses. Unparseable bodies are worthless for analytics.

I wrapped ADK's span processors instead, shrinking attributes with knowledge of the JSON structure just before export:

wrapped_hooks = OTelHooks(
    span_processors=[
        SpanAttributeLimitSpanProcessor(p) for p in hooks.span_processors
    ],
    metric_readers=list(hooks.metric_readers),
    log_record_processors=list(hooks.log_record_processors),
)
maybe_set_otel_providers([wrapped_hooks], resource)

Wrapping rather than replacing the exporter keeps ADK's native Cloud Trace batching. The shrink order goes by analytic value: incidental fields like grounding_metadata and logprobs_result first, then tool definitions, then the system instruction, then oldest conversation turns. A __adk_span_truncated__ marker always goes in so downstream readers know.

In production it looks like this:

WARNING Truncated span attribute: span=generate_content gemini-3.8-flash
        attribute=gen_ai.system_instructions original_bytes=63427

63 KB shrank to 27,826 characters with the tail (the most recent context) preserved, and zero 400s since.

One known weakness: this approach handles one huge string well and many medium strings badly. When no single removal gets you under budget, the depth-first shrink gives up and collapses everything to the marker. I reproduced 30 parts x 2.1 KB = 63,520 bytes — only 3.5 KB over budget — degrading to 47 bytes. v2 splits content into more, array-shaped attributes, so this path gets more likely after migration, not less.

Trap 5: exclude-newer-span in uv.lock blocked the new version

Not schema-related, but it happened in the same change.

Dependabot's ADK 2.13 PR was failing CI. Trying to reproduce locally, I could not even install 2.13:

Because only google-adk<=2.12.0 is available ...

The cause lives at the top of uv.lock:

exclude-newer = "0001-01-01T00:00:00Z"
exclude-newer-span = "P1W"

exclude-newer-span refuses anything published within a week of the lock's resolution timestamp — a reasonable supply-chain measure. In this case 2.13.0 was published about five and a half hours past the cutoff. A few hours of timing, and the error message tells you none of this.

Worse, my own ~/.config/uv/uv.toml carried exclude-newer = "1 week", which overrode the PR's lockfile. That is why checking out the branch and running uv sync kept installing 2.7.0.

# A stale VIRTUAL_ENV will also make `uv pip list` lie to you
unset VIRTUAL_ENV
UV_EXCLUDE_NEWER=2026-12-31T00:00:00Z uv sync

Reproducibility settings manufacture exactly as much irreproducibility for anyone who did not write them, because the failure never points back at the setting.

Trap 6: a sys.modules stub leaking across test files

The Dependabot CI failure turned out to have nothing to do with production code. Excluding one broken test file, all 1,243 remaining tests passed.

Two independent causes, both test scaffolding.

First, a test module installed stubs into sys.modules at import time and never cleaned up:

# tests/test_leaf_agent.py
google_genai_types = types.ModuleType("google.genai.types")
google_genai_types.GenerateContentConfig = MagicMock(name="GenerateContentConfig")
sys.modules["google.genai"] = google_genai
sys.modules["google.genai.types"] = google_genai_types

_MCP_CLIENT_STUB = _build_stub_modules()   # runs at import, no teardown

sys.modules is process-global. Depending on collection order, another test file imports google.genai.types, gets the stub, and dies with ImportError during collection. It happened to work on ADK 2.7 and broke on 2.13 when the import path changed.

Second, contents.types no longer exists in ADK 2.13:

original_contents_types = contents.types   # AttributeError on ADK 2.13

The nasty part: a plain python -c import succeeds. It only fails under pytest, because only pytest collects the other module that plants the stub. I lost a good while to "the import works but CI fails."

If an import error will not reproduce outside pytest, suspect sys.modules pollution first.

Bonus: an all-NULL column is not evidence of zero

While verifying the migration I noticed a new ADK 2.13 warning:

App "router" can transfer between agents but has no context_cache_config.
Every transfer swaps the system instruction and the tool set, so the request
prefix changes and the whole prompt is re-sent uncached after each transfer.

So I went to measure how much caching was actually happening:

JSON_VALUE(attributes, '$."gen_ai.usage.cached_input_tokens"') AS cached_tok

All NULL. I concluded caching had never once engaged, and filed an issue saying so.

That was wrong. The real attribute is gen_ai.usage.cache_read.input_tokens. The key I queried does not exist at all:

# google/adk/telemetry/_token_usage.py
GEN_AI_USAGE_CACHE_READ_INPUT_TOKENS = 'gen_ai.usage.cache_read.input_tokens'
GEN_AI_USAGE_CACHE_CREATION_INPUT_TOKENS = 'gen_ai.usage.cache_creation.input_tokens'

Re-run with the right key, and caching had been working before the fix:

Time	Agent	input_tokens	cache_read.input_tokens
10-31 07:41	leaf_agent	19,290	—
10-31 07:41	leaf_agent	23,599	—
10-31 07:41	leaf_agent	24,472	19,410
10-31 07:41	leaf_agent	26,465	23,359

Vertex AI's implicit caching was already kicking in from the third call. My premise of "zero caching" never held.

What makes this trap sharp is that ADK sets usage attributes conditionally:

if self.cache_read_input_tokens is not None:
    attrs[GEN_AI_USAGE_CACHE_READ_INPUT_TOKENS] = self.cache_read_input_tokens

When the value is None, the attribute is absent entirely. So a NULL from a JSON path means "the value was zero", or "you typed the key wrong", or "this span never carries that attribute" — three very different things that look identical.

The cheap guard is to check whether the attribute exists on any row before drawing a conclusion:

SELECT COUNTIF(JSON_VALUE(attributes, '$."gen_ai.usage.cache_read.input_tokens"') IS NOT NULL) AS has_attr,
       COUNT(*) AS n
FROM spans WHERE name LIKE 'generate_content%'

If has_attr = 0, suspect the key before you suspect reality.

The context_cache_config fix was still worth doing — explicit caching genuinely was unset. After enabling it:

Time	Agent	input_tokens	cache_read.input_tokens
11-01 08:05	router	2,875	—
11-01 08:06	router	3,010	—
11-01 08:06	leaf_agent	19,410	—
11-01 08:06	leaf_agent	23,655	19,391
11-01 08:06	leaf_agent	24,554	19,391
11-01 08:06	leaf_agent	24,757	19,391
11-01 08:07	leaf_agent	24,958	19,391

Hit rate improved (2/4 to 4/5), but the honest description is "the implicit caching that already existed became more consistent", not "caching started working".

The router agent never caches, by design. Per ContextCacheConfig's docstring, a cacheable prefix needs 2,048 tokens on Gemini 2.5 and 4,096 on Gemini 3; the router's 2,875–3,010 tokens never reach it. Agents with small instructions stay outside explicit caching no matter what you configure.

Takeaways

• Schema v2 changes span names and attribute keys. Nothing errors — you just get zero rows. Capture real spans and diff them before you migrate queries.
• Set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT, or no content is recorded. For trace-only setups, SPAN_ONLY.
• v1 and v2 LLM bodies live on different rows, so COALESCE cannot bridge them. Tool arguments and results share a row, so there it works.
• The 64 KiB attribute limit kills the whole batch. SDK character truncation corrupts JSON; shrink with structural knowledge.
• exclude-newer-span produces failures whose cause is unreadable from the error.
• Tests that plant stubs in sys.modules must tear them down. Classic symptom: an import error that will not reproduce under python -c.
• Never read an all-NULL column as proof that something did not happen. Wrong key names and conditionally-set attributes look exactly the same.

For version deltas, google-adk on PyPI and the adk-python releases are the fastest reference. For attribute names, read the semconv constants directly rather than any blog post — including this one. I got them wrong twice.