Introduction
In the article Amazon Bedrock AgentCore Runtime - Part 3 AgentCore Observability, we introduced AgentCore Observability for the agents hosted on the AgentCore Runtime and explained how to enable this observability. Moreover, we looked into the sessions and especially traces and introduced spans, timeline and trajectory views. This example didn't use any AgentCore short or long-term Memory capabilities which we introduced in the part 6 for the short-term AgentCore Memory and in the part 7 for the long-term AgentCore Memory. Let's re-visit AgentCore Observability and explore what metrics, traces and logs we additionally have for the AgentCore Memory.
AgentCore Memory Metrics
CloudWatch Metrics provides the following metrics by default for different operations with the AgentCore Memory (CreateMemory, RetrieveMemory, GetMemory, ListMemory DeleteMemory, UpdateMemory, CreateEvent, GetEvent, ListEvents and so on)
Latency : The total time elapsed between receiving the request and sending the final response token. Represents complete end-to-end processing of the request. For a create event, this represents the end to end time taken from last createEvent that met strategy criteria to the memory stored completed.
Invocations : The total number of API requests made to the data plane and control plane. This metric also tracks the number of memory ingestion events.
System Errors : Number of invocations that result in AWS server-side errors.
User Errors : Number of invocations that result in client-side errors.
Errors : Total number of errors that occur while processing API requests in the data plane and control plane. This metric also tracks the total errors that occur during memory ingestion.
Throttles : Number of invocations that the system throttled. Throttled requests don't count as invocations or errors.
Creation Count : Counts the number of created memory events and memory records.
Here are some examples of such CloudWatch metrics for the Custon Agent with short and long-term memory create and list event requests:
and for retrieve and get memory requests:
If we navigate to the AgentCore console, select Memory, then select some particular created memory by its id we also see some operational metrics for the last 24 or 48 hours in the "Observability" section.
This is the example for the short-term memory:
This is the example for the long-term memory:
AgentCore Memory Tracing
If we navigate to the AgentCore console, select Memory, then select some particular created memory by its id we see that the tracing is disabled by default:
In order to the information about the AgentCore Memory to be present in the traces we first need to enable tracing for all created memories:
After we have done this, let's send some prompts with short and long-erm memory and then navigate to the CloudWatch Gen AI Observability, then to the "Bedrock AgentCore" panel and select "Traces View" (see part 3 for the detailed explanation).
For the example with short-term memory we see in the "Trajectory View" memory operation like ListEvents and CreateEvent :
In the span data in the example of CreateEvent we see such attributes like memory id, actor id, and session id:
I refer to the complete list of the AgentoCore Memory provided span data.
In the "Timeline View" and in the spans data we can also see the latency of the operation with memory:
For the example with long-term memory we see partially different events in the "Trajectory View" memory operation like GetMemory, RetrieveMemory and CreateEvent :
Here we see CreateEvent in red, which means that it runs into an error. I see that the total context to be stored in the event became to big and exceeded the limit, as I see an error in the CloudWatch log "failed to satisfy constraint: Member must have length less than or equal to 9000" in the log. This is the reason that's why the observability operational statistics for the long-term memory previously showed 0 CreateEvents. I'll later fix it.
"Timeline View" and span data basically show similar information for the long-term memory operation like memory id, actor id, and session id and overall latency.
AgentCore Memory Logs
AgentCore can output logs to CloudWatch Logs, Amazon S3, or Firehose stream. If we use a CloudWatch Logs destination, these logs are stored under the default log group /aws/vendedlogs/bedrock-agentcore/memory/APPLICATION_LOGS/{memory_id} or under a custom log group starting with /aws/vendedlogs/. See Enable observability for AgentCore memory, gateway, and built-in tool resources to learn more.
I created log delivery to the CloudWatch Logs for both short and log-term memory:
But unfortunately in both cases I don't see any log streams after having multiple conversations with my agents for several days:
I'll dig depper into this (maybe I'm doing something wrong) or wait until it's fixed as the same smoothly works for the AgentCore Gateway log delivery as described in my article Amazon Bedrock AgentCore Gateway - Part 4 AgentCore Gateway Observability.
Conclusion
In this article, we explored Amazon Bedrock AgentCore short and long-term Memory observability and how to combine available information in the CloudWatch (metrics) and CloudWatch GenAI Observability (agent, session and traces views) for the agents hosted on the AgentCore Runtime. This gives us also required additional observability in case we use AgentCore Memory.
I hope that AgentCore Memory log delivery will also work soon.
Top comments (0)