DEV Community

Hiroshi Toyama
Hiroshi Toyama

Posted on

Tracing Google ADK Agents on Vertex AI Agent Engine, and the Five Gotchas Nobody Warns You About

#googlecloud #ai #python #observability

Turning on tracing for a Google ADK agent deployed to Vertex AI Agent Engine is one keyword argument:

from vertexai.agent_engines import AdkApp

agent_engine = AdkApp(agent=root_agent, enable_tracing=True)
Enter fullscreen mode Exit fullscreen mode

Deploy that, trigger your agent, open Cloud Trace, and there is a decent chance you see nothing at all — with no error anywhere in your application logs. Every failure mode in this post is silent by design: the SDK treats telemetry as best-effort, so a misconfiguration costs you observability, never availability. That is the right call for production and a miserable one for debugging.

This is what I found while wiring it up, verified against google-adk 2.7.0 and google-cloud-aiplatform 1.165.1.

Gotcha 1: it does not export to the Cloud Trace API

The obvious move is to enable cloudtrace.googleapis.com and move on. That is not where the spans go.

The exporter is an OTLP/HTTP exporter pointed at the Telemetry API:

# vertexai/agent_engines/templates/adk.py
_DEFAULT_TELEMETRY_ENDPOINT = "https://telemetry.googleapis.com/v1/traces"
Enter fullscreen mode Exit fullscreen mode

Spans are POSTed there using the credentials from google.auth.default() — in practice, the Agent Engine service account. The SDK even probes the API at startup and warns you:

Tracing integration for Agent Engine has migrated to a new API.
The 'telemetry.googleapis.com' has not been enabled in project <project>.
**Impact:** Until this API is enabled, telemetry data will not be stored.
Enter fullscreen mode Exit fullscreen mode

That check is wrapped in a bare except and is explicitly non-fatal, so your container boots normally and serves traffic. The warning scrolls past in the startup logs and your traces quietly go nowhere.

gcloud services enable telemetry.googleapis.com --project=my-project
Enter fullscreen mode Exit fullscreen mode

Check what you actually have enabled, because this is the single most likely reason you are reading this post:

gcloud services list --enabled --project=my-project \
  --filter="config.name:(telemetry.googleapis.com OR cloudtrace.googleapis.com)" \
  --format="value(config.name)"
Enter fullscreen mode Exit fullscreen mode

Keep cloudtrace.googleapis.com — that is what backs the read side and the console. But enabling it alone buys you nothing on the write path. You want both names in that output.

Gotcha 2: the IAM failure is a 403 in a log you are probably not reading

Enable the API, redeploy, still nothing. The error exists, but not where you are looking. It is not in your Cloud Run service logs and not in your application output. It is in the Reasoning Engine resource logs:

ERROR: Failed to export span batch code: 403, reason: Forbidden
Enter fullscreen mode Exit fullscreen mode

Find it with:

gcloud logging read \
  'resource.type="aiplatform.googleapis.com/ReasoningEngine"
   AND textPayload=~"Failed to export span batch"' \
  --project=my-project --freshness=1h --limit=20
Enter fullscreen mode Exit fullscreen mode

The batch span processor logs the failure and drops the batch. Your agent is completely unaffected — correct answers, normal latency, zero traces.

Which role? The permission you need is telemetry.traces.write. Two roles grant it, and the naming is misleading:

$ gcloud iam roles describe roles/cloudtrace.agent \
    --format='value(includedPermissions)'
cloudtrace.traces.patch;telemetry.traces.write

$ gcloud iam roles describe roles/telemetry.tracesWriter \
    --format='value(includedPermissions)'
telemetry.traces.write
Enter fullscreen mode Exit fullscreen mode

roles/cloudtrace.agent is a superset. Despite the legacy-sounding name, it already covers the Telemetry API, so granting both is pointless:

gcloud projects add-iam-policy-binding my-project \
  --member="serviceAccount:my-agent@my-project.iam.gserviceaccount.com" \
  --role="roles/cloudtrace.agent"
Enter fullscreen mode Exit fullscreen mode

Confirm it landed:

gcloud projects get-iam-policy my-project \
  --flatten="bindings[].members" \
  --filter="bindings.role=roles/cloudtrace.agent" \
  --format="value(bindings.members)"
Enter fullscreen mode Exit fullscreen mode

One useful property of this failure mode: once you fix the IAM, the 403s stop at a clean timestamp boundary without a redeploy. If you see errors up to T and traces appearing after T, the binding propagated. That is your confirmation signal.

Gotcha 3: do not set ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS yourself

By default ADK records prompts, model responses, and tool arguments into spans. The documented knob is ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS, and ADK's own default is on:

# google/adk/telemetry/context.py
env_value = os.getenv(ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS, 'true').strip().lower()
Enter fullscreen mode Exit fullscreen mode

The natural instinct is to pin it explicitly in your deployment config so a future upstream default change cannot silently strip your payloads. On Agent Engine that accomplishes nothing, because AdkApp.set_up() overwrites it unconditionally:

# vertexai/agent_engines/templates/adk.py — inside set_up()
if self._tmpl_attrs.get("enable_tracing"):
    os.environ["ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS"] = "true"
else:
    os.environ["ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS"] = "false"
Enter fullscreen mode Exit fullscreen mode

Whatever you put in env_vars is clobbered at startup. Content capture is a pure function of enable_tracing here. (The adk deploy CLI does inject false — but that only applies if you deploy through that CLI. Create the engine from the SDK or any IaC tool and the injection never runs.)

The practical consequence is a security one: on Agent Engine, enable_tracing=True means full prompt and tool payloads land in your traces, and you cannot opt out through this variable. If your agent handles anything sensitive, that is a deliberate decision about who can read traces, not an afterthought.

Gotcha 4: enable_tracing is not the only switch

Tracing is an AND of two controls, and the truth table is in the source:

# vertexai/agent_engines/templates/adk.py
def _tracing_enabled(self) -> bool:
    """
    | enable_tracing | enable_telemetry(env) | tracing_actually_enabled |
    |----------------|-----------------------|--------------------------|
    | true           | false                 | false                    |
    | true           | true                  | true                     |
    | true           | None                  | true                     |
    | None(default)  | true                  | adk_version >= 1.17      |
    | None(default)  | None                  | false                    |
    """
Enter fullscreen mode Exit fullscreen mode

enable_telemetry comes from the GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY env var, which is also what the Cloud Console toggle writes. Set it to false and your enable_tracing=True is dead on arrival. Leaving it unset (None) is fine.

One precision point worth getting right: the SDK deprecates enable_tracing=False, not the parameter in general.

if self._tmpl_attrs.get("enable_tracing") is False:
    _warn("Your 'enable_tracing=False' setting is being deprecated ...")
Enter fullscreen mode Exit fullscreen mode

enable_tracing=True emits no warning. If you want to disable tracing, drop the argument and use the env var instead — passing False desynchronizes the Console toggle from reality.

Gotcha 5: the root span is not called invoke_agent

You will go looking for invoke_agent and not find it at the root. ADK has two telemetry schema versions, and Agent Engine defaults to v2:

# google/adk/telemetry/_schema_version.py
#   Default resolution: 2 on Agent Engine, detected via the presence of
#   the GOOGLE_CLOUD_AGENT_ENGINE_ID env var.
Enter fullscreen mode Exit fullscreen mode

Under v2 the tree looks like this:

invoke_workflow my_agent          <- root
└─ invoke_agent my_agent
   ├─ call_llm
   │  └─ generate_content gemini-x.y
   ├─ execute_tool transfer_to_agent
   └─ invoke_agent my_sub_agent
      └─ execute_tool my_sub_tool
Enter fullscreen mode Exit fullscreen mode

invoke_agent {name} is a child. execute_tool carries the tool name. Content lives on call_llm as gcp.vertex.agent.llm_request / llm_response, and on execute_tool as tool_call_args / tool_response.

Filtering by root span name works against the v1 read API:

GET https://cloudtrace.googleapis.com/v1/projects/<PROJECT>/traces
    ?startTime=<RFC3339>&endTime=<RFC3339>
    &filter=root:invoke_workflow
    &view=COMPLETE
Enter fullscreen mode Exit fullscreen mode

Note that the default listing is ordered by trace ID, not time — without a filter you get an arbitrary slice and may conclude nothing was recorded when it was.

The structural problem: one request is not one trace

This is the part that changes how you design, not just how you configure.

Each create_session() + stream_query() pair is an independent invocation and produces its own trace. If your handler fans out — say, three parallel checks against the same business record:

await asyncio.gather(*(
    query_agent_engine(message=m, user_id=f"check-{i}-{record_id}")
    for i, m in enumerate(messages)
))
Enter fullscreen mode Exit fullscreen mode

you get three unrelated traces. There is no parent linking them, because the trace context never crosses the Agent Engine boundary.

ADK's standard span attributes are invocation_id, session_id, and event_id — all internal identifiers. Nothing ties a trace back to your domain object, and user_id is not exported at all.

So you have to add it. A BasePlugin runs inside the root span's context, which makes it the right hook:

import logging
from google.adk.agents.invocation_context import InvocationContext
from google.adk.plugins.base_plugin import BasePlugin
from opentelemetry import trace

logger = logging.getLogger(__name__)


class RecordTracingPlugin(BasePlugin):
    def __init__(self) -> None:
        super().__init__(name="record_tracing")

    async def before_run_callback(
        self, *, invocation_context: InvocationContext
    ) -> None:
        try:
            identity = parse_user_id(invocation_context.user_id)
            if identity is None:
                return
            span = trace.get_current_span()
            if not span.is_recording():
                return
            span.set_attribute("app.record_id", identity.record_id)
            span.set_attribute("app.flow", identity.flow)
        except Exception:
            logger.warning("failed to tag span with record identity", exc_info=True)
        return None


agent_engine = AdkApp(
    agent=root_agent,
    enable_tracing=True,
    plugins=[RecordTracingPlugin()],
)
Enter fullscreen mode Exit fullscreen mode

Now app.record_id = "12345" retrieves all three traces as a set.

Four things about that snippet are load-bearing:

It must be async def. The base signature is a coroutine; a sync method breaks when PluginManager awaits it.

It must return None. Any non-None return value is treated as an early exit and short-circuits the run:

# google/adk/plugins/plugin_manager.py
result = await callback_method(**kwargs)
if result is not None:
    # Early exit: A plugin has returned a value. We stop
    # processing further plugins and return this value immediately.
Enter fullscreen mode Exit fullscreen mode

The try / except is not defensive padding. Plugin exceptions are not contained — PluginManager._run_callbacks wraps them in a RuntimeError, and Runner._exec_with_plugin re-raises:

except Exception as e:
    await _notify_run_error(plugin_manager, invocation_context, e)
    raise
Enter fullscreen mode Exit fullscreen mode

An attribute-tagging bug takes down the agent run itself. Observability code has no business doing that, so catch Exception (not BaseException — leave CancelledError and KeyboardInterrupt alone) and log.

invocation_context.user_id is a property. It is not a model field, so it will not appear if you introspect model_fields:

# google/adk/agents/invocation_context.py
@property
def user_id(self) -> str:
    return self.session.user_id
Enter fullscreen mode Exit fullscreen mode

Note what this implies: user_id is the only general-purpose channel for smuggling correlation data across the Agent Engine boundary, since you control it at create_session() time. Encode it carefully, and keep the encode and decode in one module — they execute in different processes, and a format change applied to only one side fails silently in exactly the way this whole post is about.

Two limits worth knowing before you design around this

Attribute values cap at 64 KiB, and excess is dropped non-deterministically. If an agent takes a large document as input, gcp.vertex.agent.llm_request can exceed that and simply not be there. Do not build a workflow that assumes the prompt is always recoverable from the trace.

Retention is 30 days, fixed, in the _Trace bucket. Cloud Trace is an incident-investigation tool here, not an analytics store. If you want months of prompt history for offline evaluation, export to logging and BigQuery separately.

Summary

Symptom Cause
No traces, no errors telemetry.googleapis.com not enabled
No traces, 403 in Reasoning Engine logs SA missing roles/cloudtrace.agent
Payloads missing or present unexpectedly set_up() overwrites ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS from enable_tracing
enable_tracing=True does nothing GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=false
Cannot find invoke_agent at the root Agent Engine uses schema v2; root is invoke_workflow {name}
One request produces N unrelated traces Each stream_query is its own invocation; correlate with a plugin attribute

The configuration is genuinely one line. The surrounding API enablement, IAM, and span model are where the time goes — and because every one of them fails quietly, the only efficient way through is to know which log holds the 403.

Top comments (0)