Introduction
In part 6, we introduced our sample application spring-ai-2.0-ac-conference-app-agent-bedrock-agentcore-runtime using Spring AI AgentCore. Later, in parts 7 and 8, we added AgentCore short- and long-term Memory, respectively, using Spring AI AgentCore Memory.
In this part of the series, we'll explore how to configure AgentCore Observability. I've already covered this topic in my other article series when we used the Strands Agents SDK. I refer to my following articles:
Even though the CloudWatch Generative AI Observability service UI now looks different and has more capabilities, CloudWatch Logging and Monitoring configuration looks the same for our application and is provided out of the box. For the CloudWatch metrics, please review the article Amazon Bedrock AgentCore generated observability data for the current state of the exposed metrics. Also, make sure to enable Log delivery for Application and Usage Logs and Tracing for all AgentCore Services involved. In our case, these are:
- Runtime (2 of them, as we host the MCP server and the Agent there)
- Gateway
- Memory (both short- and long-term).
- Identity (please make sure to do so for the Runtime and Gateway Identity)
Enabling AgentCore Tracing
If we follow the steps described above and enable tracing for all AgentCore services in use, we'll only see the basic AgentCore metrics, but completely miss Sessions and Traces. The reason for this is that we provided the examples using the Strands Agents SDK. It works well with AgentCore Observability (baked by CloudWatch Generative AI Observability). We only had to add the dependency to aws-opentelemetry-distro and instrument our code, as shown below in the Dockerfile. Strands Agent has all the information on where to send the metrics and traces to the default OTEL provider, AWS CloudWatch. But how does it work for Java applications based on Spring AI and hosted on AgentCore Runtime?
To view the metrics in CloudWatch Generative AI observability, we need to add the AWS Distro for OpenTelemetry (ADOT) SDK to our agent code. ADOT is a secure, production-ready, AWS-supported distribution of the OpenTelemetry project. Part of the Cloud Native Computing Foundation, OpenTelemetry provides open source APIs, libraries, and agents to collect distributed traces and metrics for application monitoring. With ADOT, we can instrument our applications just once to send correlated metrics and traces to multiple AWS and Partner monitoring solutions. In our case, we will send the metrics to the CloudWatch GenAI Observability service.
AWS offers AWS Distro for OpenTelemetry Java with the AWS Distro for OpenTelemetry (ADOT). To get started, see the AWS Distro for OpenTelemetry Java documentation. Let's add it to our sample application. To do so, we need to modify the Dockerfile of our sample application. We first have to download the aws-opentelemetry-agent and then run it as the Java agent to instrument the code on the fly:
ADD https://github.com/aws-observability/aws-otel-java-instrumentation/releases/latest/download/aws-opentelemetry-agent.jar /opt/aws-opentelemetry-agent.jar
ENV JAVA_TOOL_OPTIONS=-javaagent:/opt/aws-opentelemetry-agent.jar
The documentation also says that the second component is required to receive the metrics and traces: the AWS Distro for OpenTelemetry Collector. In all the examples AWS provides, the collector is a sidecar application deployed with Docker Compose. Unfortunately, it's not possible to use Docker Compose for the AgentCore Runtime. We only provide the reference to the image in the Amazon Elastic Container Registry (ECR) repository that the AgentCore Runtime pulls and runs for us.
It took me a while to figure out how to achieve this, and I even created the issue for it. There is a so-called collector-less Observability for the Amazon Bedrock AgentCore resources. As of now, unfortunately, not all parameters to be configured are described in this article. But I combined this information with the article Exporting collector-less telemetry using AWS Distro for OpenTelemetry (ADOT) SDK to achieve the goal:
ADD https://github.com/aws-observability/aws-otel-java-instrumentation/releases/latest/download/aws-opentelemetry-agent.jar /opt/aws-opentelemetry-agent.jar
ENV JAVA_TOOL_OPTIONS=-javaagent:/opt/aws-opentelemetry-agent.jar \
AGENT_OBSERVABILITY_ENABLED=true \
OTEL_RESOURCE_ATTRIBUTES=spring_ai_ac_conference_application_runtime,aws.log.group.names=
/aws/bedrock-agentcore/runtimes/spring_ai_ac_conference_application_runtime-a00QWV3i7t \
OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=/aws/bedrock-agentcore/runtimes/spring_ai_ac_conference_application_runtime-a00QWV3i7t,x-aws-log-stream=runtime-logs,x-aws-metric-namespace=bedrock-agentcore \
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf \
OTEL_TRACES_EXPORTER=otlp \
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://xray.us-east-1.amazonaws.com/v1/traces \
OTEL_EXPORTER_OTLP_LOGS_PROTOCOL=http/protobuf \
OTEL_LOGS_EXPORTER=otlp \
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://logs.us-east-1.amazonaws.com/v1/logs
Besides the already described steps to download the aws-opentelemetry-agent and run it as the Java agent to instrument the code, we configured the following:
- AGENT_OBSERVABILITY_ENABLED=true to indicate that we use Agent Observability and would like to view the traces in the CloudWatch Generative AI Observability and not in X-Ray.
- OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_EXPORTER_OTLP_TRACES_PROTOCOL, and OTEL_EXPORTER_OTLP_LOGS_PROTOCOL to all be http/protobuf.
- OTEL_EXPORTER_OTLP_TRACES_ENDPOINT and OTEL_EXPORTER_OTLP_LOGS_ENDPOINT as regional endpoints for traces and logs. If you deploy your application in another region other than us-east-1, you need to adjust the URL.
- OTEL_RESOURCE_ATTRIBUTES to be service.name=spring_ai_ac_conference_application_runtime,aws.log.group.names=/aws/bedrock-agentcore/runtimes/spring_ai_ac_conference_application_runtime-a00QWV3i7t. Please adjust service.name value to how you named the service in AgentCore Runtime. I called it spring_ai_ac_conference_application_runtime. For the suffix of the aws.log.group.names use your AgentCore Runtime ID (in my case spring_ai_ac_conference_application_runtime-a00QWV3i7t). AWS Log Group Name for AgentCore Runtime always follows the pattern: /aws/bedrock-agentcore/runtimes/{RUNTIME_ID}.
- OTEL_EXPORTER_OTLP_LOGS_HEADERS to be x-aws-log-group=/aws/bedrock-agentcore/runtimes/spring_ai_ac_conference_application_runtime-a00QWV3i7t,x-aws-log-stream=runtime-logs,x-aws-metric-namespace=bedrock-agentcore. The same as above: for the suffix of the x-aws-log-group use your AgentCore Runtime ID again.
Then, after rebuilding and redeploying the application, we can see similar metrics and traces as provided in the articles using Strands Agents SDK above. There are, of course, some differences in the collected metadata. This is because we use the AWS OpenTelemetry Agent distribution for Java and not for Python, as in the articles above. Here are some selected screenshots taken from the CloudWatch GenAI Observability: Bedrock AgentCore Observability for the prompt "Please provide me with the list of conferences, including their IDs, with the Java topic happening in 2027, with the call for papers open today. Also, provide me with the list of my talks with this topic in the title. Finally, for each conference and talk retrieved, apply individually for the conference.", which I sent :
All sessions view:
All traces view:
All spans view:
Session ID view:
Tree view of the Trace ID:
Trajectory view of the Trace ID:
Timeline view of the Trace ID:
This is how to enable observability for Amazon Bedrock AgentCore Runtime hosted agents. If you don't host your application on AgentCore, you can still use CloudWatch to enable observability for non-Amazon Bedrock AgentCore-hosted agents.
Conclusion
In this article, we described how to configure AgentCore Observability in a collector-less way. This involves running the Java aws-opentelemetry-agent agent to instrument the code and set a bunch of environment variables in the Dockerfile. Please also make sure you have activated logging and tracing for both AgentCore Gateway and Runtime.
In the next article, we'll explore how to configure the same AgentCore observability with the help of spring-ai-agentcore-otel-extension.
If you like my content, please follow me on GitHub and give my repositories a star!
Please also check out my website for more technical content and upcoming public speaking activities.
Top comments (0)