This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.
Project Overview
I built an AWS Security Posture Agent: five specialist AI agents that scan your AWS account for security misconfigurations, map findings to CIS benchmarks, score risk, and generate copy-paste fix commands.
The agents run sequentially on CrewAI with Amazon Bedrock Nova Pro as the LLM:
1. ResourceDiscovery → inventories EC2, S3, Lambda, IAM, SGs, API GW, DynamoDB
2. SecurityScanner → finds open ports, public buckets, admin roles, insecure configs
3. ComplianceChecker → maps to CIS AWS Foundations Benchmark
4. RiskScorer → severity × blast radius × exploitability
5. RemediationPlanner → generates AWS CLI fix commands
Each agent has custom boto3 tools that make real AWS API calls against a live account with 90 IAM roles, 14 S3 buckets, 9 security groups, and 7 Lambda functions. Not test data. Real findings.
simplynadaf
/
aws-security-posture-agent
Multi-agent AI system that scans AWS accounts for security misconfigurations using CrewAI + Amazon Bedrock, instrumented with Sentry AI Agent Monitoring. 5 specialist agents discover resources, analyze security, map to CIS benchmarks, score risk, and generate fix commands.
AWS Security Posture Agent
Five AI agents scan your AWS account. Find misconfigurations. Score risk. Get fix commands.
The Problem
Most AWS accounts accumulate security debt silently. Open SSH ports from testing. S3 buckets without encryption. IAM roles with full admin access that nobody remembers creating. Manual audits miss things. AWS Config rules cost money. SecurityHub is noisy.
This agent scans your account in 60 seconds, finds real issues, maps them to CIS benchmarks, scores risk, and gives you the exact AWS CLI command to fix each one.
What It Finds
On a real AWS account (90 IAM roles, 14 S3 buckets, 9 security groups, 7 Lambda functions):
97 security findings
├── CRITICAL: open SSH/RDP from 0.0.0.0/0, IAM users without MFA
├── HIGH: admin roles, unencrypted EBS, missing public access blocks
├── MEDIUM: deprecated Lambda runtimes, missing versioning, stale SGs
└──Demo
Here's the full scan running against my AWS account. The scan portion is sped up 4x, results walkthrough is at normal speed:
Bug Fix or Performance Improvement
The SecurityScanner agent was taking 22.6 seconds while every other agent averaged 5-10 seconds. Without visibility into what was happening inside each agent's execution, I would have blamed Bedrock latency and moved on.
The root cause: my IAMAnalyzer tool was fetching all 90 IAM roles from the account (59 after filtering service-linked roles), serializing them into a 27KB JSON blob, and handing that entire payload to the LLM as tool output. The context window got overwhelmed. CrewAI's internal retry logic kicked in, burning tokens on a second attempt with even more context.
One tool. Wrong default. The entire pipeline suffered.
Code
PR with the fix:
fix: paginate IAM analysis and add token budget guard
#1
The IAMAnalyzer tool was fetching every single role in the account (90 of them, 59 after filtering service-linked ones) and dumping the full details into a single JSON blob. That blob hit 26,980 characters. The LLM choked on it, CrewAI retried the task, and the SecurityScanner agent ended up taking 22.6s while every other agent finished in 5-10s.
Added Sentry spans to each agent and tool. The trace waterfall made it obvious: SecurityScanner was twice as wide as everything else. Drilled into the tool spans and saw result_length_chars: 26980 on iam_analyzer vs ~4000 on the other tools.
7x output disproportion. That was the problem.
Three things in iam_analyzer.py:
- Paginate and sort roles by
RoleLastUseddate, only analyze the top 20 most active ones. The rest are stale roles nobody has touched in months. - Skip 31 service-linked roles upfront (you cannot modify them anyway, auditing them is noise).
- Token budget guard at the end: if the JSON output exceeds 4000 chars, trim the role_summary down to just names and policy lists.
- Tool output: 26,980 chars down to 15,532 (42% smaller)
- SecurityScanner: 22.6s down to 17.8s (21% faster, no more LLM retry)
- API calls to IAM: 59 down to 20
- Total pipeline: 62s down to 57.7s
- Findings: still 27. No coverage loss because the top-20 most active roles contain all the problematic ones (AdminRole, Bedrock-Lambda-Role, etc are all heavily used).
The core change lives in src/security_posture/tools/iam_analyzer.py. Here's the before and after:
Before (the bug):
# Fetches ALL roles without pagination limit
roles = iam.list_roles(MaxItems=100)
role_details = []
for role in roles["Roles"]:
role_name = role["RoleName"]
if role.get("Path", "").startswith("/aws-service-role/"):
continue
# Analyzes every single role...
attached = iam.list_attached_role_policies(RoleName=role_name)
# ...builds massive JSON output
This produces 26,980 characters of JSON for an account with 90 roles. The LLM chokes on it.
After (the fix):
# FIX: Paginate and sort by relevance
all_roles = []
paginator = iam.get_paginator("list_roles")
for page in paginator.paginate():
all_roles.extend(page["Roles"])
# Filter service-linked roles (31 roles, can't modify anyway)
auditable_roles = [
r for r in all_roles
if not r.get("Path", "").startswith("/aws-service-role/")
]
# Sort by last used date (most active first)
auditable_roles.sort(key=_last_used_sort_key, reverse=True)
# Take only top 20 roles
roles_to_analyze = auditable_roles[:max_roles]
Plus a token budget guard at the end:
# Token budget guard: truncate if output exceeds threshold
if len(output) > 4000:
result["role_summary"] = [
{"role_name": r["role_name"], "policies": r.get("attached_policies", [])}
for r in role_details
]
result["note"] = "Role details truncated to stay within token budget"
output = json.dumps(result, indent=2, default=str)
Three changes. Pagination, relevance sorting, and a safety valve. The SecurityScanner stopped retrying.
My Improvements
The fix itself is simple. The interesting part is how I found it.
Without Sentry's trace waterfall, all I would have seen is "pipeline takes 62 seconds." Maybe I'd profile the Python code. Maybe I'd add time.time() calls around each agent. Honestly my first instinct was to blame Bedrock latency. Every time something is slow you blame the LLM, right? The trace said otherwise. The real problem wasn't Python execution time. It was the LLM getting a context payload it couldn't process cleanly on the first attempt.
The approach:
- Wrap each agent execution in a
gen_ai.invoke_agentspan - Wrap each tool call in a
gen_ai.execute_toolspan - Run the pipeline and look at the trace waterfall
- The SecurityScanner span was visually obvious: twice the width of everything else
- Inside it, the
iam_analyzertool span showed aresult_length_charsof 26,980 - Compare to
security_group_analyzerat 4,200 chars ands3_config_checkerat 3,800 chars
The disproportion was the clue. The fix followed naturally: if the tool output is 7x larger than its siblings, reduce it.
Results:
| Metric | Before | After | Improvement |
|---|---|---|---|
| IAM tool output | 26,980 chars | 15,532 chars | 42% smaller |
| SecurityScanner time | 22.6s | 17.8s | 21% faster |
| IAM API calls | 59 | 20 | 66% fewer |
| Total pipeline | 62.0s | 57.7s | 7% faster |
| Security findings | 97 | 97 | No coverage loss |
The 21% improvement on the SecurityScanner came from the LLM completing analysis in a single pass instead of retrying.
Best Use of Sentry
I used Sentry's AI Agent Monitoring to instrument a multi-agent CrewAI pipeline from scratch. This isn't a web app or API. It's five autonomous AI agents making LLM calls and executing custom tools. Standard APM wouldn't help here.
What I instrumented
Every pipeline run creates a Sentry transaction with this span hierarchy:
Transaction: "Security Posture Scan" (57s)
├── gen_ai.invoke_agent: ResourceDiscovery
│ └── gen_ai.execute_tool: aws_resource_scanner
├── gen_ai.invoke_agent: SecurityScanner
│ ├── gen_ai.execute_tool: security_group_analyzer
│ ├── gen_ai.execute_tool: s3_config_checker
│ ├── gen_ai.execute_tool: iam_analyzer
│ ├── gen_ai.execute_tool: ec2_security_checker
│ └── gen_ai.execute_tool: lambda_security_checker
├── gen_ai.invoke_agent: ComplianceChecker
├── gen_ai.invoke_agent: RiskScorer
└── gen_ai.invoke_agent: RemediationPlanner
The instrumentation code
For each agent (in main.py):
with start_span(
op="gen_ai.invoke_agent",
name=f"invoke_agent {agent_name}",
) as agent_span:
agent_span.set_data("gen_ai.operation.name", "invoke_agent")
agent_span.set_data("gen_ai.agent.name", agent_name)
agent_span.set_data("gen_ai.request.model", "amazon.nova-pro-v1:0")
agent_span.set_data("gen_ai.pipeline.name", "security-posture-scan")
task_output = task_obj.execute_sync(
agent=task_obj.agent,
context=context,
tools=task_obj.agent.tools,
)
agent_span.set_data("duration_seconds", round(elapsed, 2))
agent_span.set_data("output_length_chars", len(str(task_output)))
For each tool (decorator in monitoring.py):
def trace_tool(tool_name: str):
def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
with start_span(
op="gen_ai.execute_tool",
name=f"execute_tool {tool_name}",
) as span:
span.set_data("gen_ai.tool.name", tool_name)
result = func(*args, **kwargs)
span.set_data("result_length_chars", len(result))
try:
data = json.loads(result)
if "findings_count" in data:
span.set_data("findings_count", data["findings_count"])
except (json.JSONDecodeError, KeyError):
pass
return result
return wrapper
return decorator
Sentry features used
| Feature | How I Used It |
|---|---|
| Distributed Tracing | Full pipeline trace from start to final report |
| AI Agent Monitoring |
gen_ai.invoke_agent spans for all 5 agents |
| Tool Execution Tracing |
gen_ai.execute_tool spans for all 5 boto3 tools |
| Custom Span Data | Token counts, output sizes, duration, finding counts |
| Error Monitoring | Exception capture with sentry_sdk.capture_exception()
|
| Breadcrumbs | Agent completion events via task callbacks |
| Transaction Metadata | Model name, pipeline config, agent count |
What Sentry revealed
The trace waterfall made the bottleneck visually obvious. The SecurityScanner span was nearly twice the width of any other agent 22.6 seconds while others averaged 5-10s. Inside it, individual AWS API calls like s3 get_public_access_block (206ms per call × 14 buckets) and the iam_analyzer tool span showed result_length_chars: 26980 while the other tools like s3_config_checker showed 3,800-4,200. The granularity goes down to individual boto3 calls every GetPublicAccessBlock, ListRoles, and ListAttachedRolePolicies shows up as its own span.
After the fix, all agent spans are proportional to their actual task complexity. No single agent dominates the trace. The iam_analyzer output dropped from 26,980 chars to 15,532, and the SecurityScanner stopped triggering retry logic on the first pass.
Why this matters for AI agent observability
Standard logging tells you an agent "finished." It doesn't tell you which tool inside which agent returned a 27KB payload that triggered a retry you never asked for.
With five agents each making their own LLM calls and tool executions, you need span-level visibility. Sentry's gen_ai.invoke_agent and gen_ai.execute_tool conventions gave me exactly that. I could see the problem in the trace waterfall before I even looked at the code.
That's the difference between "add some logging" and actual AI observability.
If you're running multi-agent pipelines, what's your observability setup? Curious if anyone else has hit similar tool-output-size problems with CrewAI or LangGraph.
Built during DEV's Summer Bug Smash 2026. The agent found 97 real security findings in my AWS account, including open SSH ports, missing MFA, roles with full admin access, unencrypted EBS volumes, and Lambda functions on deprecated runtimes. The fix is in production.
Top comments (1)
Took me way longer than it should have to figure out the problem wasnt Bedrock being slow. Kept blaming the model when it was my own tool dumping 27kb into the context window lol. If you're running multiagent pipelines with crewai or langgraph how do you handle tool output sizes? I couldnt find much guidance on what happens when one tool returns 7x more data than the others and the LLM just silently retries............