Agentic security is still in an emerging state, and there’s a lot of things we need to keep an eye on. The OWASP GenAI Project has published some great resources and one of them contains this graphic, outlining the Agentic SecOps lifecycle.
AI Security Solutions Landscape for Agentic AI Q2 2026
So, a multitude of steps and activities to keep our eyeballs on, right? Observability and telemetry are incredibly important in every situation, whether we are using it for troubleshooting functionality and operations, to security and cost. With OpenTelemetry as the foundational layer, we can have some visibility into how agents interact with one another and our tools. Some of the newer protocols, like A2A, MCP, and ACP are standardizing agent collaboration so we can use a variety of different implementations to create our ecosystems.
GenAI is moving so fast, though, I am still working out how I can get the data I want out of my workflows. I played with an official implementation of A2A locally, and really liked what I saw. I ran into problems deploying beyond my local environment, however, so that is an undertaking for the future. For this project, I used application logs to capture some of the interagent communication I wanted to look at.
Concept
This project is based on an AWS Labs sample — https://github.com/awslabs/agentcore-samples/tree/main/04-infrastructure-as-code/terraform/multi-agent-runtime. I tried to add increased observability and ran into a number of problems, but also some successes. Here’s what I built- https://github.com/mgbec/AgentCoreObservabilityInterAgent. Questions are submitted to the Orchestrator Agent, who decides how complex the question is. If it is simple, it will answer it directly. If it is more complex, it gets sent off to the Specialist agent. If the Orchestrator thinks it needs verification of the answer, it gets sent to the Fact Checker agent. Both the Specialist and the Fact Checker agents can reach out to Tavily for updated information. The model I am using is a little older and was missing some current knowledge, so the web search became necessary.
My big focus here was building in enhanced observability. Some of the issues I ran into:
- Code changes get baked into a new Docker image (via CodeBuild), but AgentCore won’t pull the new image under the same latest tag unless the runtime is recreated. Run Terraform apply, but you may need to taint or mark as “replace” the current runtime to make sure you get the new one. For example, terraform taint aws_bedrockagentcore_agent_runtime.orchestrator
- If you look at the Bedrock Marketplace, you will see that serverless foundation models are automatically enabled across Bedrock. When you first invoke an Amazon Bedrock serverless model in an account, Bedrock attempts to automatically enable the model for your account. For this auto-enablement to work, AWS Marketplace permissions are required. Once enabled, all users in the account can invoke the model without needing AWS Marketplace permissions. I needed to enable a model using a higher permissioned account before I could use it here, in this project.
- If the runtimes were recreated (new ARNs), the tracing toggle needs to be re-enabled. I ran into this problem repeatedly, so I ended up documenting the process (see below). I do have tracing enabled account wide but this seems to be the current way it works.
Turn on tracing manually for agent runtime: go to Amazon Bedrock AgentCore>Runtime>Identity to enable tracing.
Go Time
So, we have our workflow built and functional. I have a script here called ask.py that lets you submit the question, and then invokes the orchestrator. Again, the orchestrator should decide the complexity of the question and either answer it without calling another agent, or call on the specialist for a more complex question.
Here are some examples with the associated trace.
Simple question that the orchestrator can answer itself-” Can you give me a recipe for molasses cookies?” (I probably should have used the recipe it gave me instead of the one I chose.)
A more complex question that needs to be routed to the specialist agent and needs a web search tool invoked- “Can you tell me security issues with agent protocol A2A?”
You may ask yourself, how is the Orchestrator agent deciding if a question is complex enough to need a specialist? The LLM (Claude Sonnet 4.5) reads the user’s question and its system prompt, then decides on its own whether to:
Respond directly (e.g., “Hello” → no tool call)
Call one or both tools (e.g., “Is it true that…” → fact checker)
Strands handles the mechanics — if the LLM’s response contains a toolUse block, Strands executes that tool function. If not, it just returns the text response.
-There’s no threshold, classifier, or rules engine. It’s entirely the LLM’s judgment based on the system prompt guidelines. This means:
-It’s flexible (handles edge cases naturally)
-It’s non-deterministic (the same question might route differently on rare occasions)
-It’s only as good as the system prompt (if the instructions are vague, routing gets inconsistent)
-If you wanted deterministic routing, you’d replace the LLM decision with code, possibly keyword matching or a lightweight classifier before calling any agent.
The web search tool gets invoked when we ask about something our model doesn’t know about. It is a model from 09/29/2025, so it is trained on data before that time period. Anything past that point will need a web search.
Observability
This is where having things in the AWS ecosystem rocks. We have a ton of logs and metrics:
Agent to Agent Observability
I was particularly looking for inter-agent communication in this project, and I am able to see some details in the execute_tool call_specialist_agent span. A2A_CALL_START, A2A_CALL_PAYLOAD, A2A_CALL_END, and A2A_CALL_RESPONSE show up with details like target, query length, response length, latency, and similar attributes.
This script focuses on checking the “A2A” communication- check_a2a.py. It gets “A2A” call logs from the Orchestrator agent. These are application logs that were written manually in the Orchestrator’s agent.py. The A2A_CALL_START/END/PAYLOAD/RESPONSE messages were produced by explicit logger.info() calls in agent.py. These contain payload data so they can be useful for security analysis, among other things.
This script monitor_a2a.py does some basic security testing on the interagent logs. Things it checks for:
Unauthorized Targets
Verifies that the Orchestrator only calls agents in the approved list (specialist, factchecker). An unapproved target could indicate prompt injection causing the agent to call an unintended endpoint, or a misconfiguration routing traffic to the wrong agent. It is triggered by an A2A_CALL_START log entry with a target name not in the ALLOWED_TARGETS set.
If an attacker manipulates the Orchestrator into calling an unauthorized agent, they could exfiltrate data or escalate privileges. IAM policies provide a hard guardrail, but this check catches the attempt at the application layer.
Prompt Injection
Scans the payloads sent between agents for patterns commonly used in prompt injection attacks. These patterns attempt to override the agent’s system prompt or extract sensitive information. It is triggered by an A2A_CALL_PAYLOAD log entry containing text matching known injection patterns (e.g., ‘ignore previous instructions’, ‘reveal system prompt’, ‘you are now’, etc.). This is triggered by a query_length > 5000 characters (outbound) or response_length > 50000 characters (inbound).
An attacker could use prompt injection to cause the Orchestrator to embed sensitive context, conversation history, or system information into the query sent to a downstream agent — effectively using “A2A” as a data exfiltration channel.
Latency Anomalies
Detects “A2A” calls with unusual timing. Very slow calls may indicate resource abuse, stuck agents, or denial-of-service conditions. Very fast calls may indicate the agent returned without processing (bypassed logic, cached/replayed responses). It is triggered by a latency_ms > 120000ms (too slow) or latency_ms < 100ms (too fast).
Slow calls could indicate an agent caught in a loop or being abused for compute. Fast calls could indicate a compromised agent returning hardcoded responses without invoking the LLM.
Error Rates
Monitors the failure rate of “A2A” calls. A sudden spike in errors could indicate permission changes, resource exhaustion, agent crashes, or an ongoing attack causing repeated failures. The trigger is more than 10% of “A2A” calls returning errors, or any individual A2A_CALL_ERROR log entry.
Elevated error rates impact system reliability and may indicate an attacker probing for vulnerabilities (e.g., sending malformed payloads to cause crashes).
Call Frequency
Detects bursts of “A2A” calls within short time windows. An abnormally high number of calls in a single minute could indicate a runaway loop, automated abuse, or denial-of-service attack against downstream agents. The trigger is more than 20 A2A_CALL_START events within any single 1-minute window.
High-frequency A2A calls consume compute resources on downstream agents, incur LLM costs, and could be used to exhaust quotas or create billing attacks.
Bedrock, AgentCore, OpenTelemetry and CloudWatch Observability
AgentCore provides these automatically:
-CloudWatch metrics — invocation count, latency, throttles, errors, session count per runtime
-AgentCore.Runtime.Invoke spans — one span per inbound invocation showing the target agent ARN, latency, HTTP status, session ID, request ID (appears in aws/spans when tracing is enabled)
-CloudWatch log group creation — automatically creates a log group per runtime for application stdout/stderr
AWS Distro for OpenTelemetry:
The ADOT wrapper in the Dockerfile automatically hooks into boto3 and Strands to produce standardized trace spans. The instrumentation gives you structured trace data (latency, trace IDs, span hierarchy) for dashboards and trace visualization.
-boto3 call spans (InvokeAgentRuntime client-side)
-LLM call spans (ConverseStream, token counts)
-Strands spans (tool calls, agent lifecycle) strands-agents[otel] package
GenAI Observability dashboard visibility:
We can look at many elements of token usage and model invocation, as well as detailed information regarding AgentCore Agents, Tools, Gateways, Identity, Memory, and Payments. I haven’t used the Payments Observability feature yet, but it has Payment managers, API invocation, sessions, transactions, and more.
Wrap Up
Agentic observability is important for troubleshooting (believe me), but it is also crucial for agentic systems because it transforms our agents’ autonomous decisions into traceable, measurable, and governable workflows. It provides transparency into an agent’s context evaluation, intermediate reasoning, tool selection, and execution chains. We will need proof of decision integrity more and more to comply with business policies and regulatory requirements, as they evolve and mature. It also can flag inefficiencies, like excessive unnecessary token usage or model drift over time. Finally, I find it completely fascinating to see the “behind the scenes” interactions between all the moving parts. Big thanks to the Cloud Native Computing Foundation and all the open source creators out there everywhere, helping make this possible.
Top comments (0)