AI Agent Observability Made Simple: Using CloudWatch EMF and X-Ray to Trace Claude Tool Calls

AI agents can feel like black boxes—when a Claude‑driven tool call fails you’re left guessing why. By pairing CloudWatch’s Embedded Metrics Format with X‑Ray tracing you get real‑time, zero‑overhead visibility into every plan‑act‑observe loop. This post shows exactly how to instrument your Node.js Lambda agent and read the signals.

Why AI Agents Need Observability

When a language model decides to call a tool—say a weather API—there are three hidden steps:

1. Planning – the model produces a plan string.
2. Acting – the code actually invokes the external service.
3. Observing – the response is fed back into the model for the next turn.

If any step stalls, you only see a generic “Lambda timed out” or a vague error message. Without observability you cannot answer questions like:

• Did the model produce the correct plan?
• Did the HTTP request succeed, or did the Bedrock service throw a ServiceUnavailableException?
• How long did each step take, and where did the latency come from?

Think of an AI agent as a kitchen where a chef (Claude) writes a recipe, a sous‑chef (your code) fetches ingredients, and a taste‑tester (the next model turn) decides whether to continue. Observability is the kitchen’s order‑ticket system and timer—without it you never know which dish caused the delay.

In plain English: Adding metrics and traces is like putting a stopwatch and a notebook on each step of the recipe. You can later look back and see exactly where the problem happened.

Key concepts

• Observability – the practice of exposing internal state (logs, metrics, traces) so you can understand a system’s behavior from the outside.
• Metric – a numeric value that you can graph over time (e.g., “tool‑call‑latency”).
• Trace – a record of the path a request follows through many services, broken into segments and subsegments.

Embedded Metrics Format: Zero‑Cost Custom Metrics

CloudWatch’s Embedded Metrics Format (EMF) lets you write JSON inside a log event that CloudWatch automatically extracts as a metric. The benefit is two‑fold:

1. Zero extra API calls – you just write a log line, CloudWatch does the heavy lifting.
2. Near‑real‑time graphs – metrics appear in the console within seconds.

Why EMF instead of the classic PutMetricData API?

The classic API requires a separate network call for each batch of metrics, adding latency and cost. EMF piggybacks on the log stream you already have for Lambda, so you pay only for the log storage you’d use anyway.

Tip: Keep the metric name short and consistent (e.g., tool_call_latency_ms). CloudWatch treats each unique name as a separate time‑series, which can affect cost.

Code: Minimal EMF logger for a Lambda

// emfLogger.ts
import { CloudWatchClient, PutLogEventsCommand } from "@aws-sdk/client-cloudwatch";
import { v4 as uuidv4 } from "uuid";

/**
 * Simple EMF helper that builds a JSON log line that CloudWatch
 * will turn into a custom metric.
 *
 * @param metricName   The name that will appear in the CloudWatch console.
 * @param value        Numeric value of the metric.
 * @param dimensions   Optional key/value pairs to slice the metric (e.g., tool name).
 */
export async function emitEmfMetric(
  metricName: string,
  value: number,
  dimensions: Record<string, string> = {}
) {
  // Build the EMF JSON payload.
  const emfPayload = {
    _aws: {
      // Required fields for EMF.
      Timestamp: Date.now(),
      CloudWatchMetrics: [
        {
          Namespace: "AI/ClaudeAgent",
          Dimensions: [Object.keys(dimensions)], // e.g., ["ToolName"]
          MetricName: metricName,
          Unit: "Milliseconds",
        },
      ],
    },
    // The actual metric value.
    [metricName]: value,
    // Any extra dimensions you want to filter on.
    ...dimensions,
  };

  // Emit the JSON as a log line. In Lambda the console.log output is automatically
  // sent to CloudWatch Logs, which then parses the EMF block.
  console.log(JSON.stringify(emfPayload));
}

/**
 * Helper to force the Lambda runtime to finish flushing any pending
 * async log writes. Without this the function may exit before the
 * EMF line reaches CloudWatch.
 */
export async function flushLogs() {
  // Lambda's console.log is synchronous, but when using async libraries
  // (e.g., winston) you may need an explicit flush. Here we simply
  // return a resolved promise to keep the pattern clear.
  return Promise.resolve();
}

Why we call flushLogs at the end: Lambda may stop the process as soon as the handler returns. If the EMF line hasn't been flushed, CloudWatch never sees the metric, creating a blind spot.

Key takeaway: EMF lets you turn a console.log into a metric without extra network hops, but you must make sure the log actually reaches CloudWatch before the function exits.

Adding X‑Ray Subsegments to Tool Calls

AWS X‑Ray records a trace for each request that touches your Lambda. A segment represents the Lambda invocation itself; subsegments are child pieces you create manually (e.g., a call to a Bedrock tool). Subsegments let you see latency, errors, and custom annotations for each step.

Why subsegments matter

If Claude decides to call a calendar‑lookup tool, you want to know:

• How long the HTTP request took.
• Whether the request succeeded or threw a ServiceUnavailableException.
• Which tool name caused the latency spike.

Without subsegments you only see the total Lambda duration, making it impossible to pinpoint the problematic step.

In plain English: A subsegment is like a sticky note on a specific part of your kitchen order ticket, saying “this sauce simmered for 5 seconds” or “the oven failed to preheat”.

Code: X‑Ray client and subsegment wrapper

// xrayHelper.ts
import {
  XRayClient,
  PutTelemetryRecordsCommand,
  PutTraceSegmentsCommand,
} from "@aws-sdk/client-xray";

/**
 * Minimal wrapper that creates an X‑Ray subsegment around an async
 * operation. It records start/end time, any error, and optional annotations.
 *
 * @param name          Human‑readable name of the subsegment (e.g., "weatherTool").
 * @param fn            Async function representing the tool call.
 * @param annotations   Optional key/value pairs to attach to the subsegment.
 */
export async function withXRaySubsegment<T>(
  name: string,
  fn: () => Promise<T>,
  annotations: Record<string, string> = {}
): Promise<T> {
  // X‑Ray automatically creates a segment for the Lambda handler.
  // The SDK expects a JSON segment document; we construct a simple one.
  const segment = {
    name,
    id: crypto.randomUUID().replace(/-/g, "").substring(0, 16), // 16‑char hex ID
    start_time: Date.now() / 1000,
    // Annotations are searchable in the X‑Ray console.
    annotations,
  };

  try {
    const result = await fn(); // Run the actual tool call.
    segment.end_time = Date.now() / 1000;
    segment.http = { response: { status: 200 } };
    // Send the completed subsegment.
    await sendSubsegment(segment);
    return result;
  } catch (err: any) {
    segment.end_time = Date.now() / 1000;
    segment.error = true;
    segment.cause = { exceptions: [{ message: err.message, type: err.name }] };
    await sendSubsegment(segment);
    throw err; // Re‑throw so the Lambda handler sees the error.
  }
}

/**
 * Sends a completed subsegment to the X‑Ray daemon.
 * In Lambda the daemon is already running and reachable via the
 * AWS_XRAY_DAEMON_ADDRESS environment variable.
 */
async function sendSubsegment(segment: any) {
  const client = new XRayClient({});
  const command = new PutTraceSegmentsCommand({
    TraceSegmentDocuments: [JSON.stringify(segment)],
  });
  await client.send(command);
}

Gotcha: The X‑Ray SDK adds ~50‑100 ms to cold starts because it starts a UDP daemon connection. To keep start‑up time low, import the helper lazily (inside the handler) or use the aws-xray-sdk-core library’s captureAsyncFunc which re‑uses the daemon across invocations.

Putting It All Together in a Lambda‑Hosted Claude Agent

Now we combine EMF metrics and X‑Ray subsegments into a single Lambda function that:

1. Receives an event with the user query.
2. Calls Claude (via Bedrock) to get a plan.
3. Executes a tool based on the plan.
4. Sends the observation back to Claude for the next turn.

We will instrument the tool call part with both EMF and X‑Ray.

Full Lambda handler

// index.ts
import {
  BedrockRuntimeClient,
  InvokeModelCommand,
} from "@aws-sdk/client-bedrock-runtime";
import { emitEmfMetric, flushLogs } from "./emfLogger";
import { withXRaySubsegment } from "./xrayHelper";

// Types for clarity.
type ToolResult = {
  success: boolean;
  data?: any;
  error?: string;
};

/**
 * Example tool: fetches current weather from a fake HTTP endpoint.
 * In real life you would use fetch/axios. Here we just simulate latency.
 */
async function callWeatherTool(location: string): Promise<ToolResult> {
  // Simulated network latency.
  await new Promise((r) => setTimeout(r, Math.random() * 200 + 100));

  // Randomly throw an error to illustrate error handling.
  if (Math.random() < 0.1) {
    const err = new Error("ServiceUnavailableException");
    (err as any).name = "ServiceUnavailableException";
    throw err;
  }

  return { success: true, data: { location, temperature: 72 } };
}

/**
 * Main Lambda entry point.
 * The handler is deliberately small; most logic lives in helper functions.
 */
export const handler = async (event: any) => {
  const query = event.query ?? "What is the weather in Paris?";
  const bedrock = new BedrockRuntimeClient({});

  // 1️⃣ Ask Claude for a plan. (Simplified request.)
  const planResponse = await bedrock.send(
    new InvokeModelCommand({
      ModelId: "anthropic.claude-v2",
      ContentType: "application/json",
      Accept: "application/json",
      Body: JSON.stringify({ prompt: `Plan a tool call for: ${query}` }),
    })
  );

  // Extract the tool name and argument from Claude's response.
  // In a real implementation you'd parse the model's JSON output.
  const toolName = "weatherTool";
  const toolArg = "Paris";

  // 2️⃣ Wrap the tool call with X‑Ray and EMF.
  const startMs = Date.now();
  const result = await withXRaySubsegment(
    toolName,
    async () => await callWeatherTool(toolArg),
    { tool_name: toolName, location: toolArg }
  );
  const latencyMs = Date.now() - startMs;

  // Emit EMF metric for the latency.
  await emitEmfMetric("tool_call_latency_ms", latencyMs, {
    ToolName: toolName,
  });

  // 3️⃣ Feed the observation back to Claude (omitted for brevity).
  // ... imagine another InvokeModelCommand here ...

  // Ensure the EMF line has been flushed before Lambda exits.
  await flushLogs();

  // Return a simple JSON payload to the caller.
  return {
    statusCode: 200,
    body: JSON.stringify({
      query,
      toolResult: result,
      latencyMs,
    }),
  };
};

What the code does, step by step

1. Ask Claude for a plan – we send a prompt to Bedrock. The response tells us which tool to call.
2. Create a subsegment – withXRaySubsegment records start/end time, adds an annotation with the tool name, and sends the subsegment to the X‑Ray daemon.
3. Emit an EMF metric – emitEmfMetric writes a JSON log line that CloudWatch extracts as tool_call_latency_ms.
4. Flush logs – flushLogs guarantees the metric reaches CloudWatch before the function ends. Without it you would see a missing data point in the graph.

Tip: If you ever notice a gap in your latency graph, check that flushLogs() was awaited. The Lambda runtime can shut down the process in the middle of an async console.log.

Gotchas specific to this setup

Area	Common mistake	Result
EMF	Forgetting to await flushLogs()	Metric never appears, trace shows no latency info
X‑Ray	Not adding annotations	You can’t filter subsegments by tool name in the console
Bedrock	Not handling ServiceUnavailableException	Lambda crashes, Lambda logs show error but no metric or trace for the failed call
Lambda timeout	Metric emission takes longer than remaining time	Lambda exits early, leaving both metric and subsegment incomplete

Reading the Signals in CloudWatch Console

Once your Lambda runs, you have two places to look:

1. Metrics pane (EMF) – Navigate to CloudWatch → Metrics → Browse → AI / ClaudeAgent. You’ll see a line chart for tool_call_latency_ms. Use the Dimensions dropdown to slice by ToolName.

2. X‑Ray console – Open X‑Ray → Traces. Each trace shows the Lambda segment with a child subsegment named weatherTool. Click the subsegment to view:

   • Annotations (tool name, location).
   • Error flag if the call threw an exception.
   • Latency (the same number you emitted as a metric).

How to spot a problem

• High latency spikes on the EMF chart → drill into X‑Ray trace for the matching timestamp; the subsegment will reveal whether the delay was inside the tool call or in the Bedrock response.
• Missing data points → verify that your Lambda didn’t exit before flushing. The X‑Ray console will still show the segment, but the metric line will have a gap.
• Frequent error: true flags → filter X‑Ray subsegments by the error annotation; you’ll see a pattern (e.g., ServiceUnavailable every 5 minutes) that points to a downstream API issue.

Key takeaway: Metrics give you the “what” (latency numbers), traces give you the “where” (exact code path). Together they let you answer “why” in minutes instead of hours.

The Takeaway

• Observability works for AI agents just like it does for any microservice.
• EMF turns a log line into a metric with no extra network call; remember to flush before the Lambda exits.
• X‑Ray subsegments let you see each tool call as its own piece of a trace, complete with annotations and error flags.
• Combine the two: EMF for aggregate latency graphs, X‑Ray for per‑call detail.
• Watch out for common pitfalls: missing flushLogs(), cold‑start overhead from X‑Ray, and cost implications of high‑resolution metrics or dense traces.

By treating Claude‑driven tool calls as first‑class operations—instrumented with metrics and traces—you gain the same confidence you have when debugging a REST API. The black‑box becomes a transparent, measurable component, and you spend less time guessing and more time improving. Happy tracing!

---

Transparency notice

This article was written with the help of an AI system — Groq (GPT OSS 120B).

Published: 2026-10-15 · Primary focus: CloudWatch

All code blocks are intended to be correct and runnable, but please verify them
against the official docs for the tools mentioned before using in production.

Find an error? Drop a comment — corrections are always welcome.