DEV Community

Cover image for From Local Traces to Production Observability for Google AI Agents
Raju Dandigam
Raju Dandigam

Posted on AI-assisted

From Local Traces to Production Observability for Google AI Agents

#ai #observability #typescript #googlecloud

Most difficult agent incidents begin with one question:

Why did the agent do that?

Why did it call this tool? Why did it retry? Why did it skip the notification? Why did it trust stale data? Why was an action blocked even though every API call succeeded?

Traditional logs often answer a narrower question: what executed?

agent started
model called
tool called
tool completed
response sent
Enter fullscreen mode Exit fullscreen mode

That timeline is useful, but it loses causation. Agent systems are decision workflows. A run may include routing, model calls, tools, validation, memory, approval checks, retries, suppressions, and user feedback.

Production observability must reconstruct that decision path without turning your telemetry system into a second database of sensitive prompts.

Begin with an execution tree

During local development, I want to see the run as a tree before I want to search a production dashboard.

proactive-hotel-agent                         1,842 ms
├─ load-user-policy                             18 ms
├─ detect-intent                               312 ms
├─ search-hotels                               486 ms
├─ compare-price                               201 ms
├─ notification-policy                          11 ms
│  └─ blocked: quiet-hours
└─ final-response                              604 ms
Enter fullscreen mode Exit fullscreen mode

The tree immediately exposes parent-child relationships, missing steps, unexpected retries, and the point where the run changed direction.

This is the local-to-production path I aim for:

ADK / Genkit / Gemini application
             │
             ├── model and tool spans
             ├── policy decision events
             ├── metrics and safe logs
             ▼
      OpenTelemetry pipeline
             │
       ┌─────┴───────────┐
       ▼                 ▼
Local trace view   Cloud Trace / Logging / Monitoring
                         │
                         ▼
              alerts, dashboards, and analytics
Enter fullscreen mode Exit fullscreen mode

The tools can differ between development and production. The event shape should not.

Use spans for work and events for decisions

Create a span for an operation with measurable duration: an agent run, model request, tool execution, memory lookup, or policy evaluation.

Attach an event when something meaningful happens inside that operation: a retry is scheduled, an action is blocked, confirmation is requested, or a fallback is selected.

import { SpanStatusCode, trace } from "@opentelemetry/api";

const tracer = trace.getTracer("travel-agent");

async function tracedToolCall<T>(options: {
  runId: string;
  toolName: string;
  risk: "read" | "write" | "irreversible";
  execute: () => Promise<T>;
}): Promise<T> {
  return tracer.startActiveSpan(`agent.tool.${options.toolName}`, async (span) => {
    span.setAttributes({
      "app.agent.run_id": options.runId,
      "app.agent.tool.name": options.toolName,
      "app.agent.tool.risk": options.risk,
    });

    try {
      const result = await options.execute();
      span.setStatus({ code: SpanStatusCode.OK });
      return result;
    } catch (error) {
      span.recordException(error as Error);
      span.setStatus({
        code: SpanStatusCode.ERROR,
        message: error instanceof Error ? error.message : "Tool failed",
      });
      throw error;
    } finally {
      span.end();
    }
  });
}
Enter fullscreen mode Exit fullscreen mode

The app.agent.* attributes are deliberately application-owned. Adopt standard semantic attributes where they fit, but do not wait for every agent-specific convention to stabilize before creating a consistent internal taxonomy.

Record reason codes, not hidden reasoning

Observability does not require private chain-of-thought. It requires an explanation produced by your application at consequential boundaries.

span.addEvent("action.blocked", {
  "app.agent.reason_code": "QUIET_HOURS",
  "app.agent.policy_version": "notifications-v4",
  "app.agent.next_state": "suppressed",
});
Enter fullscreen mode Exit fullscreen mode

A small vocabulary of reason codes is easier to aggregate than arbitrary text:

type ReasonCode =
  | "USER_REQUEST_MATCHED"
  | "MISSING_REQUIRED_DETAIL"
  | "CONFIRMATION_REQUIRED"
  | "QUIET_HOURS"
  | "DUPLICATE_ACTION"
  | "TOOL_TIMEOUT"
  | "LOW_CONFIDENCE"
  | "POLICY_DENIED";
Enter fullscreen mode Exit fullscreen mode

You can still include a redacted human-readable summary for debugging. The code is what makes dashboards and alerts reliable.

Make privacy a telemetry configuration, not a promise

Raw prompts and tool results may contain personal data, retrieved memory, internal identifiers, or confidential business context. "We will be careful" is not a control.

Prefer metadata that answers operational questions:

{
  "promptTemplate": "hotel-price-drop-v3",
  "model": "gemini-family",
  "tool": "search_hotels",
  "inputClassification": "travel-preferences",
  "piiSentToModel": false,
  "reasonCode": "USER_REQUEST_MATCHED",
  "status": "success"
}
Enter fullscreen mode Exit fullscreen mode

Genkit automatically instruments AI features and makes traces available locally in its Developer UI. Its Google Cloud telemetry configuration can also collect logs, traces, and metrics. Because input and output capture may be enabled, review the configuration rather than assuming payloads are excluded.

For privacy-sensitive systems, disable input/output logging and add only approved metadata:

import { enableFirebaseTelemetry } from "@genkit-ai/firebase";

enableFirebaseTelemetry({
  disableLoggingInputAndOutput: true,
});
Enter fullscreen mode Exit fullscreen mode

Also establish retention, sampling, and access controls. Redaction performed after export may already be too late.

Measure agent behavior, not just infrastructure

CPU, memory, HTTP errors, and container latency still matter. They do not tell you whether the agent is useful or safe.

Add agent-level metrics:

  • tool calls per run;
  • model calls per run;
  • retries and repeated-tool rate;
  • blocked-action count by reason;
  • human-confirmation rate;
  • clarification rate;
  • terminal outcome;
  • latency by model and tool;
  • estimated cost per successful outcome;
  • user acceptance, dismissal, or correction rate.

Be careful with averages. A mean of 2.1 tool calls can hide a small population of 40-call loops. Use distributions and set budgets.

type RunBudget = {
  maxModelCalls: number;
  maxToolCalls: number;
  maxDurationMs: number;
};
Enter fullscreen mode Exit fullscreen mode

When a budget is reached, record a terminal state such as budget_exhausted; do not let the trace simply disappear after a timeout.

Join telemetry with product outcomes

An agent trace can be technically successful and still produce no value.

A proactive travel agent might complete every model and tool call, send a notification, and receive an immediate dismissal. That is not necessarily an infrastructure failure. It may indicate poor timing, weak relevance, or insufficient personalization.

Connect operational traces to privacy-safe outcome events:

run completed
  → recommendation delivered
    → opened
      → accepted / dismissed / ignored
Enter fullscreen mode Exit fullscreen mode

This makes better questions possible:

  • Which reason codes correlate with user corrections?
  • Which tools dominate latency without improving acceptance?
  • Which low-confidence actions still reach users?
  • Which prompt or policy version increased suppression?
  • Did a cost reduction also reduce successful outcomes?

Observability should help improve the product, not merely explain incidents.

Promote real traces into regression tests

Production observability and testing should form a loop.

  1. Detect a failed or surprising trajectory.
  2. Sanitize the relevant input, state, and tool results.
  3. Save the required and forbidden transitions.
  4. Replay the case before changing prompts, tools, or policies.
  5. Compare the new trajectory with the original failure.

This local evidence loop is also the motivation behind AgentInspect, an open-source project I created for inspecting TypeScript agent trajectories. A local debugger does not replace Cloud Trace, Genkit Monitoring, or a production observability platform. It shortens the path from "something looks wrong" to a reproducible engineering artifact.

A practical rollout sequence

Do not begin by designing fifty dashboards.

Start with one root agent.run span, child spans for models and tools, policy-decision events, terminal outcomes, and strict payload controls. Confirm that one run can be followed end to end. Then add metrics and alerts for the failure modes that matter.

A useful production trace should answer:

  • What did the agent understand?
  • Which tools and models did it use?
  • Which state and policy versions influenced the path?
  • What was allowed, blocked, retried, or skipped?
  • How much time and budget did the run consume?
  • What happened after the user received the result?

AI-agent observability is not more logging. It is preserved causation.

Start locally, keep the trace shape stable, minimize sensitive payloads, and promote important failures into tests. That is how agent debugging becomes an engineering discipline instead of an exercise in guessing.

References

Top comments (0)