DEV Community: Dmitriy Trunov

I Gave Three Broken AWS Environments to an AI On-Call Engineer. Here's What AWS DevOps Agent Found

Dmitriy Trunov — Sun, 11 Oct 2026 19:50:16 +0000

It’s 2:00 AM. A CloudWatch alert triggers. The on-call engineer wakes up, opens five console tabs, and begins reconstructing the timeline of events from scratch.

The AWS DevOps Agent is designed to handle that initial hour of painstaking analysis. AWS refers to it as a "frontier agent"- one that not only resolves incidents but prevents them before they occur. It analyzes CloudWatch metrics, CloudTrail data, VPC Flow Logs, RDS Performance Insights, and IaC stacks, subsequently providing a root-cause analysis and an action plan for remediation.

I was curious to see how this works in practice, so I attended an AWS DevOps Agent workshop focused on incident investigation. The workshop consisted of four labs: one on system setup and three involving specially prepared environments with pre-injected faults. I attended this engaging event - organized by the AWS User Group 3City community in collaboration with AWS-some time ago.

The AWS DevOps Agent console: "Resolve. Prevent. Improve."

Scenario	What I broke	What the agent concluded	Time to root cause
EC2 / Auto Scaling	Ran a CPU-burn script on the ASG through SSM	The CPU alarm had no actions attached and the ASG had no scaling policy, so the group couldn't scale out	~10 min
Network security	Port scan + HTTP flood from my laptop	A security group open to `0.0.0.0/0` on 22/80/443/8080 since stack creation. It also found that 3 of 4 earlier alarm firings were false positives caused by S3 return traffic	~20–25 min
RDS MySQL	140 connections + a terrible `CROSS JOIN … ORDER BY RAND()`	A runaway workload from the bastion host using the `admin` user. It also spotted broken CloudWatch log export	~16 min

In all three labs the agent did more than restate the alarm. It found the configuration gap underneath it, tied the fix back to the CloudFormation stack so the fix wouldn't become drift, and wrote a mitigation plan in five phases: prepare → pre-validate → apply → post-validate → rollback.

Part 1: Setting it up takes about 5 minutes

You create an Agent Space. This is the boundary for what the agent can see and do. In the setup wizard you:

name the space and optionally choose a response language (Russian, Japanese, Portuguese and others are available)
let it auto-create a read role for your primary AWS account
optionally turn on Agent Actions, a separate role for write actions. Every change needs your explicit approval.
turn on the web app, which is the operator UI your on-call team works in

Once the space exists, you can add more capabilities: secondary AWS accounts, Azure, GitHub/GitLab/Azure DevOps pipelines, Datadog/Dynatrace/New Relic/Splunk/Grafana telemetry, Slack/ServiceNow/PagerDuty, any MCP server, and even remote agents over the A2A protocol.

👉 Full walkthrough: Part 1: Setting up AWS DevOps Agent

Part 2: "Why didn't my Auto Scaling Group scale?"

A Lambda used SSM Run Command to push one t3.micro instance to 100% CPU. The CloudWatch alarm went red, and the group stayed at one instance.

About 1m48s in, the agent had its first finding: ScalingPolicies: [] and AlarmActions: []. The alarm worked. It just wasn't wired to anything.

It also:

ruled out CPU credit throttling, since CPUCreditBalance stayed at the t3.micro maximum of 288
showed the spike wasn't user traffic, because the ALB stayed at 0–1 req/min
found the exact SSM Run Command ID that started the CPU burn
said plainly what it couldn't do: CloudTrail was blocked in the workshop account, so it couldn't attribute the IAM principal

👉 Part 2: EC2 CPU spike investigation

Part 3: Investigating a network attack using VPC Flow Logs

I ran a script from my laptop that port-scanned the web server, enumerated admin pages, tried SSH logins and flooded it with HTTP requests.

The agent rated it Severity: high. Using VPC Flow Logs, it identified the source IP, listed the scanned ports, and confirmed that 2,764 accepted packets on port 80 were the real attack.

The part that stood out was historical context I never asked for. Looking at earlier firings of the same alarm, it worked out that 3 of the 4 were benign S3 return traffic, which means the alarm itself needs tuning.

👉 Part 3: Network security investigation

Part 4: RDS connection exhaustion, and where the agent contradicted itself

A script on the bastion host opened about 140 connections against a max_connections=150 limit and pinned CPU with Cartesian joins.

The investigation was excellent. It split the load using Performance Insights into 108 sessions sitting in SLEEP() and 38 AAS of the CROSS JOIN query, and traced all of it to a single host and DB user.

Then I asked follow-up questions in chat and found two contradictions between the chat and the mitigation plan. The two disagreed on whether 150 is above or below the default max_connections, and on whether wait_timeout would help. Part 4 covers which one was right.

👉 Part 4: RDS performance investigation

What I liked

It answers why, not only what. None of the root causes was simply "CPU was high". Each was a missing link: no scaling policy, a world-open security group, a host with no connection limit.
It knows about IaC. Every mitigation plan warned that an API-only fix would drift from the CloudFormation stack, and every one included a code change specification to update the template as well.
Mitigation plans include rollback. Every plan had prepare, pre-validate, apply, post-validate and rollback steps with copy-paste CLI commands.
It reports its gaps. Each investigation had an "Investigation gaps" section listing what it couldn't access (CloudTrail, GuardDuty, RDS log files).
It respects guardrails. Its safe-deployment policy refused to auto-run destructive security group rule removals and marked them for a human to run.
Agent-ready spec. You can copy the remediation as a spec for a coding agent to implement.

What to watch out for

Verify the chat answers against the investigation. The structured findings were more careful than the conversational answers (see Part 4).
IAM scope determines what it can find. The agent can only investigate what its role and your SCPs allow. In the workshop account, CloudTrail lookups, GuardDuty and RDS log downloads were blocked, and those gaps showed up in the results.
Investigations take time. Expect 8–16 minutes. That's quick compared with a human starting at 2 AM, but it isn't instant.
Ignore the "Unable to describe support level" error. The workshop says it's expected.
AWS DevOps Agent Pricing

Should you try it?

I'd put it in the on-call path as the first responder that hands a human a pre-built timeline, not as an autonomous fixer… yet

Screenshots are from my own run of the AWS DevOps Agent workshop on 3 October 2026. All resources were in a temporary workshop account that has since been deleted.

AWS DevOps Agent, Part 4: RDS Connection Exhaustion, and Why You Should Still Check the AI's Answers

Dmitriy Trunov — Sun, 11 Oct 2026 19:50:01 +0000

Part 4 of 4. Overview · Setup · EC2 · Network · RDS

"Too many connections" is one of the most common database incidents, and diagnosing it usually takes a lot of manual work: Performance Insights, SHOW PROCESSLIST, parameter groups and CloudWatch, all in separate tabs. This was the most interesting lab in the workshop. The agent's investigation was the best of the three, and then the follow-up chat taught me something about how far to trust it.

The setup

Lab architecture (redrawn from the AWS DevOps Agent workshop)

ecommerce-db: RDS MySQL on db.t3.medium (2 vCPU, 4 GB), with Performance Insights and Enhanced Monitoring, and max_connections=150 in a custom parameter group
Lab4-BastionHost: the only host that can reach the database, accessed through SSM Session Manager (no SSH keys)
Lab4-RDS-HighConnections: an alarm on DatabaseConnections above 100

From the bastion:

sudo su - ec2-user
cat baseline_complete.txt         # Connection Status: SUCCESS
cd ~/scripts
./simulate-db-performance-issue.sh

The script:

creates an ecommerce database (10K customers, 5K products, 20K orders)
starts 100 concurrent inefficient queries (full scans, complex JOINs)
opens 130 connections, then 140, close to the 150 limit
keeps the CPU busy with continuous heavy queries

The alarm fired within a couple of minutes.

The investigation: about 16 minutes

I triggered it at 11:40:04 and the banner said Investigation completed at 11:56:03.

Severity: high. The summary:

ecommerce-db experienced connection exhaustion: DatabaseConnections stepped from 0 to ~148 (flat against the max_connections=150 ceiling) and CPUUtilization pinned at ~100% in the same minute, causing "Too many connections" errors and elevated query latency.

And the cause:

A runaway, manually launched workload from the Lab4-BastionHost client (10.10.20.201), authenticating as MySQL superuser admin, started right after an interactive login by principal Participant at 09:36 UTC. It combined ~108 concurrent SELECT SLEEP(?) sessions holding connections open with a pathological customers CROSS JOIN customers … ORDER BY RAND() query (~38 active sessions) pinning the 2 vCPUs at ~100%.

This is close to what an experienced DBA would write. It used Performance Insights to split the load into "User sleep" (108 AAS) and executing CROSS JOIN (38 AAS). It traced the workload to one host, one DB user, and one interactive login. It explained all three symptoms (connection exhaustion, CPU and latency) with a single cause. It also listed what it ruled out: no infrastructure, parameter group, security group or scaling change; not I/O-bound; not memory-bound; no lock contention; no gradual leak.

The connections-and-CPU chart shows the incident in one view:

It found a second problem I didn't plant

Under Investigation gaps:

RDS log file download blocked by IAM permissions (rds:DownloadDBLogFilePortion denied to the agent role); the CloudWatch error log stopped ingesting after 2026-09-30 and the general/slowquery log groups were never created despite being enabled for export.

So the log export was broken even though the configuration said it was enabled. In a real incident you'd go looking for the slow query log and find nothing. The agent added this to the mitigation plan as its own item ("Repair CloudWatch log export so future diagnosis is not hampered").

The mitigation plan

Kill the runaway MySQL sessions from the Lab4-BastionHost client, stop the source process, and raise max_connections from 150 back toward the db.t3.medium engine default (~341).

The steps are specific to RDS:

List the offending sessions with information_schema.PROCESSLIST WHERE HOST LIKE '10.10.20.201%'
Kill them with CALL mysql.rds_kill(<id>), because on RDS you can't use plain KILL for other users' threads
A helpful warning: "With connections near the 150 cap, you may need to connect using the RDS reserved superuser slot… if 'Too many connections' blocks login, briefly raise max_connections first."
Stop the source process on the bastion through SSM. Without this, "mysql.rds_kill alone will not hold; connections will be re-opened within seconds."
Raise max_connections to 341. It's a dynamic parameter, so no reboot is needed. It also warns you to watch FreeableMemory and SwapUsage.

Then post-validate (parameter value, connections dropping, alarm back to OK) and a rollback command.

Agent-ready spec

At the end there's a "Next steps → Agent-ready spec" block with Copy spec. It's a list of change requirements written for a coding agent (or a developer) to implement in source code and IaC:

Introduce bounded, pooled database connectivity (RDS Proxy or an app-side pool)
Stop using the all-privileges admin account, and cap per-user connections with MAX_USER_CONNECTIONS
Fix the Cartesian ORDER BY RAND() query
Set max_connections durably in IaC
Repair CloudWatch log export

Each item has acceptance criteria, for example: "A load test reproducing the 09:38 workload no longer trips the Lab4-RDS-HighConnections alarm."

Where the chat contradicted the investigation

Next I asked the workshop's follow-up questions in chat.

The answers were fluent and mostly good. "Should I increase max_connections or implement connection pooling?" got a clear pooling first answer with sensible reasoning. But two claims contradict the agent's own mitigation plan:

❌ Contradiction 1: is 150 above or below the default?

Mitigation plan: 150 is "artificially low". The db.t3.medium engine default is ~341, so raise it.
Chat: "max_connections=150 was manually **raised above* the db.t3.medium default (~90–100)."*

The plan is correct. The RDS MySQL default is {DBInstanceClassMemory/12582880}. With about 4 GiB that comes to roughly 341, so 150 is well below the default.

❌ Contradiction 2: will `wait_timeout` help?

Mitigation plan: "wait_timeout/interactive_timeout do **NOT* reap SLEEP() sessions (they count as active), idle-timeout reaping is not an effective lever here."*
Chat: "wait_timeout: this is **the big one* for your SLEEP() problem… Lower this significantly so idle sessions get auto-killed."* It ranked this as priority #1.

The plan is correct again. SELECT SLEEP(n) is an executing query. In the processlist it shows Command = Query, not Command = Sleep. wait_timeout only closes connections that are idle between statements. The chat mixed up "connection in Sleep state" with "connection running SLEEP()". (In the previous answer it also told me to look for command = 'Sleep' to find these sessions, which is the same mix-up.)

✅ What the chat did well

To be fair, the answer to "Would migrating to Aurora MySQL help?" was well calibrated. It said the root cause is a workload problem, not an engine limitation, that Aurora would not have prevented this incident, and it added: "I haven't verified current Aurora MySQL connection-handling specifics… against AWS documentation. I'd want to confirm those before you make a migration decision." That is the kind of honesty I want from an agent.

What I take from this

The structured investigation (timeline, findings, root cause and mitigation plan) is built from evidence: actual API calls, metrics and Performance Insights data. The chat answers sound just as confident but rely more on general knowledge, and that's where the mistakes appeared.

Treat the investigation output as the source of truth, and check chat answers against it, especially before changing parameters in production.

Takeaways

🧠 Performance Insights used well: it separated idle-holding sessions from CPU-burning sessions, which is the key to the right fix
🎯 It traced the problem to one host, one DB user and one login session without being told where to look
🪵 It found broken log export, a problem I didn't plant, that would hurt the next investigation
🤖 The agent-ready spec with acceptance criteria is a good handoff to development
⚠️ The chat contradicted the investigation twice. Verify before you act.

That's the end of the series. For the overall verdict and what I'd tell a team considering AWS DevOps Agent, see the overview post.

AWS DevOps Agent, Part 3: I Attacked My Own Web Server. Here's What the AI Found in the VPC Flow Logs.

Dmitriy Trunov — Sun, 11 Oct 2026 19:49:41 +0000

Part 3 of 4. Overview · Setup · EC2 · Network · RDS

Part 2 was an operations problem. This one is a security problem: a public web server gets scanned and flooded. Can AWS DevOps Agent work out who did it, how, and what to fix, using only VPC Flow Logs and CloudWatch?

The setup

Lab architecture (redrawn from the AWS DevOps Agent workshop)

Lab3-WebServer: a public EC2 instance
VPC Flow Logs going to CloudWatch Logs (/aws/vpc/flowlogs/Lab3-networksec)
Lab3-UnusualConnections: an alarm on NetworkPacketsIn with a sum above 500 per 5 minutes

From my own laptop I ran the workshop's simulate-attack.sh against the server's public IP:

chmod +x simulate-attack.sh
./simulate-attack.sh <WEB_SERVER_PUBLIC_IP>

The script runs four phases over about 2 minutes:

Port scan of 15 common ports
Web enumeration of admin and login pages
SSH brute force with common usernames
Connection flood of 450+ HTTP requests

About 5 minutes later, Lab3-UnusualConnections went to In alarm.

Starting the investigation

Details: Suspicious network traffic detected from external source - need to analyze attack patterns and assess security posture
Starting point: CloudWatch alarm Lab3-UnusualConnections triggered due to high incoming network traffic. Investigate source, attack methodology, and recommend security improvements.

The workshop says to expect 8–12 minutes, because querying Flow Logs takes longer than reading metrics.

The verdict: Severity high

The summary had three layers, and that structure is what I'd want from a human analyst as well:

Root cause: security group sg-… exposes TCP 22/80/443/8080 to 0.0.0.0/0. This was baked into the CloudFormation template at stack creation… never tightened since. CloudTrail confirmed there were no later manual changes.
Direct trigger: because port 80 was open to everyone, an external IP (my laptop) ran a recon port scan and then a Layer-7 HTTP flood on TCP/80: 2,764 accepted packets, about 80% of inbound traffic. It also found a similar flood from a different IP on 30 September, three days before my test and before I had run anything, so someone else had found the same open host.
Contributing cause: the NetworkPacketsIn alarm also fires on benign AWS/S3 return traffic to ephemeral ports.

It described the overall pattern as "opportunistic, rotating-IP internet scanning/flooding of an exposed public host, not a single persistent actor, and not a control-plane compromise."

The false-positive finding

I only asked about the latest alarm. The agent went back through every firing of Lab3-UnusualConnections since 30 September and broke each one down:

3 of the 4 'recurring spike' windows were dominated by AWS service IP ranges (52.217.x, 52.216.x, 16.182.x) sending large packet bursts to high ephemeral destination ports, the signature of legitimate S3/download return traffic, not an attack.

In other words, the alarm is mostly noise. A plain packet-count threshold can't tell an aws s3 cp apart from a flood. If an on-call engineer was paged for all four of those, they would soon start ignoring it, and that's how a real attack gets missed.

Findings and evidence

The findings list shows how the agent labels its own reasoning:

🔴 root cause: the world-open security group since stack creation
🟠 cause: open ports enable unsolicited scans; benign S3 traffic triggers the alarm
⚪ hypothesis, ruled out: "Security group modified after creation to open SSH/HTTP/HTTPS". CloudTrail showed it was never modified.
🔵 symptom: recurring short spikes since 30 September
⚪ gap: GuardDuty access denied by SCP
⚪ observations: no misuse of instance-role credentials, VPC Flow Logs confirm the flood and scan, no control-plane changes
⚪ topology: instance, VPC, subnet, security group, flow log ID and log group, all mapped

Keeping the ruled-out hypotheses in the list is useful. In a postmortem, what it wasn't matters as much as what it was.

Mitigation plan, with guardrails

The plan: remove the 0.0.0.0/0 rules for 22 and 8080 completely, and scope 80/443 to trusted clients or an ALB+WAF, then reconcile the CloudFormation stack so the fix lasts.

The pre-validate steps are careful:

Save the current rule set as the rollback baseline
Confirm VPC Flow Logs are active so you can see the effect of each removal
Check recent ACCEPTed flows on each port so you don't cut off a legitimate client
Check that the instance is reachable through SSM Session Manager before removing port 22, so you don't lock yourself out

One more detail: the apply steps that revoke security group rules were flagged "Automated execution of this removal is denied by the safe-deployment deletion-operation policy, must be run manually by an operator." The agent can propose destructive changes, but it won't run them itself. That's the right default.

Follow-up chat

I worked through the workshop's suggested questions in chat.

"What specific ports were targeted?" It returned the full scan list (1433, 139, 21, 5432, 3306, 9200, 23, 53, 135, 3389, 8080, 443), then pointed out that the attack that actually succeeded was only on port 80 (2,764 packets) and port 22 (57 packets, "too small to be a brute-force attempt"). It explicitly ruled out SYN flood and SSH brute force as the attack type.

"How can I prevent similar attacks?" Lock down the security group, put WAF with rate limiting in front of 80/443, enable GuardDuty (which was blocked by an SCP in this account), and fix the alarm's false-positive rate.

"Should I implement AWS WAF?" It said yes, and then added the catch: "WAF attaches to CloudFront, ALB, or API Gateway, not directly to an EC2 instance… That's a bigger architectural change than just 'turning on WAF'." It suggested a path (ALB → WAF rate-based rule → security group only from the ALB → managed rule groups) and asked for confirmation before building anything.

Takeaways

🕵️ It turned Flow Logs into a clear attack story: the source IP, the phases, which ports were only probed and which were actually hit
🔕 It found an alarm-quality problem I hadn't asked about. 3 of 4 firings were S3 noise.
🧪 It kept ruled-out hypotheses visible, which is useful for audits and postmortems
🛑 Destructive changes are blocked from auto-execution by policy
🧱 It noticed the misconfiguration came from the IaC template, and the fix targets the template

Finally, in Part 4: an RDS MySQL instance that runs out of connections, and a case where the agent's chat answers contradicted its own mitigation plan.

AWS DevOps Agent, Part 2: Why Didn't My Auto Scaling Group Scale?

Dmitriy Trunov — Sun, 11 Oct 2026 19:49:07 +0000

Part 2 of 4. Overview · Setup · EC2 · Network · RDS

This is a common incident: CPU hits 100%, the alarm goes red, MaxSize is 3, and the Auto Scaling Group still runs one instance. I gave this scenario to AWS DevOps Agent to see how far it got without help.

The setup

Lab architecture (source: AWS DevOps Agent workshop)

The workshop pre-deploys:

Lab2-ASG: an Auto Scaling Group of t3.micro instances (min 1, max 3) behind Lab2-ALB
Lab2-ASG-CPU-High: a CloudWatch alarm on the ASG's average CPU above 60%
Lab2-RemoteLoadTrigger: a Lambda that uses SSM Run Command to run a CPU-burn script on every instance for 5 minutes, with no SSH needed

To start the incident, I opened the Lambda, created a test event with {} as the payload, and clicked Test. About 2–3 minutes later the alarm went from OK to In alarm.

Starting the investigation

In the web app, Start investigation opens a short form. It also offers quick prompts such as Latest alarm, High CPU usage and Error rate spike.

Investigation details: ASG CPU utilization spiked to near 100% during load testing - need to determine root cause and assess infrastructure scaling
Name: Workshop_ASG_CPU_Investigation
Priority: Medium (default)

⚠️ You may see "Unable to describe support level" in the bottom-left corner. The workshop says this is expected and safe to ignore.

Watching it think

The Summary tab updates live. Within seconds the agent had found the ASG and described its current focus:

The Investigation timeline tab shows the reasoning step by step. Each entry lists the tool calls behind it (use_aws, datetime), and you can expand them to see the actual API calls:

Here is what happened in the first two minutes:

Time	What the agent did
+11s	Checked the current time and listed Auto Scaling Groups
+34s	Found `Lab2-ASG`: t3.micro, Min=1/Max=3/Desired=1, part of CloudFormation stack `devops-agent-combined`, group metrics not enabled
+1m32s	"Very revealing findings already": `ScalingPolicies: []`, alarm has `AlarmActions: []`, and scaling history has only one event ever (the initial launch)
+1m32s	Started a sub-agent (`alb-traffic-analysis`) in parallel to check whether real traffic caused the spike
+1m48s	Finding: No Auto Scaling policies configured on Lab2-ASG

So in under two minutes it had the core answer: the alarm fires, but nothing listens to it.

Ruling out the alternatives

A good investigation also eliminates other explanations, and the agent kept checking after its first finding:

When exactly did it happen? About 0.4% CPU baseline for 2+ days, then a single anomaly. At 1-minute resolution: 08:15 ramp (avg 19%), 08:20 peak (avg 81%, max 100%), 08:25 back to idle.
Was it CPU credit throttling? No. "CPUCreditBalance held at the t3.micro maximum of 288 throughout… the 100% CPU was genuine compute demand, not a burst-credit starvation artifact." This is the kind of t3-specific check a person can easily forget.
Was it user traffic? No. The ALB sub-agent showed Lab2-ALB at a baseline of 0–1 req/min during the spike.
So what caused it? It found the SSM Run Command (AWS-RunShellScript, with the exact CommandId) that ran a yes > /dev/null loop per vCPU for 300 s. That is the mechanism the workshop uses, and the agent identified it without being told.

It also recorded a gap: it couldn't tell who sent that SSM command, because cloudtrail lookup_events was blocked in the workshop account. It noted this is "not material to the root cause… but needed for full accountability." I value an agent that says what it couldn't check.

The root cause

At +10m22s:

No Auto Scaling policy wired to the CPU alarm. Lab2-ASG has no scaling policies attached (ScalingPolicies: []). The alarm Lab2-ASG-CPU-High exists and would enter ALARM during a CPU spike, but has no AlarmActions configured, so it cannot trigger scale-out.

The Root cause tab separates Impact, Root causes and Investigation gaps, and has Copy/Download buttons, so you can paste it straight into a postmortem.

The mitigation plan

Clicking Generate mitigation plan took a few minutes and produced this:

Attach a target-tracking scaling policy (CPU target 55%) to Lab2-ASG *and codify it in the devops-agent-combined CloudFormation template.*

Every step comes with a ready-to-run command:

Prepare. Save the current ASG config and the empty policy list as a rollback baseline.
Pre-validate. Confirm there's no conflicting policy.
Apply.

   aws autoscaling put-scaling-policy --auto-scaling-group-name Lab2-ASG \
     --policy-name Lab2-ASG-TargetTracking-CPU55 --policy-type TargetTrackingScaling \
     --estimated-instance-warmup 120 \
     --target-tracking-configuration '{"PredefinedMetricSpecification":{"PredefinedMetricType":"ASGAverageCPUUtilization"},"TargetValue":55.0,"DisableScaleIn":false}' \
     --region us-east-1

Post-validate. Check the policy and the CloudWatch alarms it creates.
Rollback. Remove the policy.

Three details here are worth pointing out:

⚠️ Drift warning: "Making this change directly via the API registers as drift against CloudFormation stack devops-agent-combined and may be reverted on the next stack update." The agent knows the ASG is managed by IaC.
ℹ️ Reactive scaling has lag: "For a burst as short as ~10 minutes, new capacity may only become healthy near the end." It suggests scheduled scaling or a higher MinSize ahead of known load tests.
💰 Cost note: more instances under load means a higher EC2 bill.

The finished summary

The final Summary shows status complete, Severity: low (correct, since there was no user-facing impact: zero 5XX errors and no latency increase), and an annotated incident timeline that marks the CPU ramp, the SSM command, the peak, the ALARM transition and the return to OK.

Below that is a Code change specification: add an AWS::AutoScaling::ScalingPolicy to the Lab2AutoScalingGroup logical resource, decide what to do with the now-redundant alarm, and reconsider whether t3.micro is the right size.

Asking follow-up questions

While the investigation runs you can ask questions in chat. For example: "Why didn't my ASG scale during the CPU spike?" My question went into a Queued requests list, and the investigation addressed it as part of the work instead of starting a separate thread.

Takeaways

⏱ About 10 minutes to a root cause I would have accepted from a senior engineer
🔍 It ruled out credit throttling and user traffic with evidence, instead of only confirming the first hypothesis
🧱 It recognised that the resource is managed by CloudFormation and wrote the fix for the template, not only for the API
🙋 It listed what it couldn't verify

Next, in Part 3, I attack a web server from my laptop and see whether the agent can work out what happened using VPC Flow Logs.

AWS DevOps Agent, Part 1: Setting Up Your First Agent Space

Dmitriy Trunov — Sun, 11 Oct 2026 19:48:09 +0000

Part 1 of 4. Overview · Setup · EC2 · Network · RDS

AWS DevOps Agent is an AI agent that investigates operational incidents in your AWS (and multicloud) environment. Before it can investigate anything, you need to give it a scope. That scope is called an Agent Space.

This post covers the setup I went through in the AWS DevOps Agent Workshop for Incident Investigation. It takes about 5 minutes.

What is an Agent Space?

An Agent Space is a logical container that defines:

what the agent can see, meaning which AWS accounts, telemetry tools, repositories and pipelines
what the agent can do, meaning read-only investigation, plus optional write actions that need your approval
who can use it, through the web app that your operators work in

You can have several Agent Spaces, for example one per product team or per environment.

Step 1: Open the service

In the AWS Console (region us-east-1), search for DevOps Agent and click Begin setup.

The left nav has two items: Agent Spaces and Capability Providers. Capability providers are registered at the account level and then attached to individual spaces.

Step 2: Create the Agent Space

The form has five sections:

Section	What I chose	Why it matters
Agent Space details	Name `enterprise-workshop-space`, description	Shown to your operators in the web app
AWS resource access	Auto-create a new DevOps Agent role	The read role the agent uses to investigate the primary account
Agent Actions	Enabled, auto-create role	A separate role for write actions. "Every change requires your explicit approval before it runs."
Tags	none	Optional
Enable web app	Auto-create a new DevOps Agent role	The role behind the operator web UI

For each role you get three options: auto-create, assign an existing role, or create one from a policy template. In production you'll likely want the template option so the role goes through your normal IAM review.

There's also an optional Agent response language setting. Findings and investigation output can come back in 17 languages:

Click Create.

Step 3: The agent starts mapping your environment

Right after creation, the banner says "DevOps Agent is Mapping entities: Analyzing resources and their relationships." By default the agent builds a topology of resources managed by CloudFormation, or of resources that carry tags you specify. That topology grows as it:

runs investigations
learns from what you tell it in the web app
gets new capabilities

This topology is what lets it work out later that "this alarm is on this ASG, which is behind this target group, which belongs to this stack".

The space page has tabs for Capabilities, Access, Configuration, Summary report and Environments. On the Capabilities tab, the Primary source account shows both a read-role status and an Actions role status. Both should say Valid.

Step 4: Explore the capabilities

This is where the agent gets more than a basic CloudWatch reader. Each capability type adds a new data source or a new way to interact:

Capability	What it adds
Cloud	Secondary AWS accounts for cross-account investigations, and Azure
Sandbox (Preview)	Controls outbound network access and pre-installed packages for the agent's sandbox
Telemetry	Dynatrace, Datadog, New Relic, Splunk, Grafana
Pipeline	GitHub, GitLab, Azure DevOps, so the agent can link incidents to code changes
Communications	Slack, ServiceNow, PagerDuty (with bidirectional support)
MCP Server	Any MCP server (OAuth 3LO, client credentials, or API key)
Remote Agent	Other agents over the A2A protocol
Webhooks	Let third-party systems start investigations (HMAC or API-key auth)

The Capability Providers page shows the full list. 14 of 17 were available to register in my account:

💡 The MCP and A2A options stood out to me. You can plug in your own internal tooling (a runbook service, a CMDB, a home-grown deploy tool) without waiting for AWS to build a native integration.

Step 5: Open the web app

The console is for admin tasks: creating spaces, wiring integrations and managing permissions. Day-to-day incident work happens in the web app, which you open with Launch via IAM (the workshop calls this Operator access).

The web app gives you:

a chat with prompts such as "Give me a health summary" (with a Balanced mode selector)
Incidents, the investigation dashboard
Improvements, proactive recommendations
Releases (Preview), release-readiness checks on code changes
Artifacts and Topology
Agents, Knowledge and Settings

The Incident Response Dashboard is where you start investigations. It also shows an Investigation hours used counter, which is worth watching from a cost point of view:

Checklist

By the end of setup you should have:

✅ An Agent Space with primary account status Valid
✅ The Actions role Valid, if you enabled Agent Actions
✅ The web app opening through Launch via IAM
✅ An empty Incident Response Dashboard, ready for the first incident

In Part 2 I break something on purpose: an Auto Scaling Group at 100% CPU that doesn't scale. Then I let the agent work out why.

The SDLC is dead. Long live the AI-DLC!?

Dmitriy Trunov — Tue, 22 Sep 2026 09:33:18 +0000

AI-DLC vs SDLC: What I Kept, What I Changed

Progress never stands still; it is constantly evolving, bringing changes to our daily work. Today, I want to describe the changes that have taken place in my own workflows. For years, we relied on the SDLC (Software Development Life Cycle) approach; now, it is time to adopt the AI-DLC approach.

The first question is: what exactly is AI-DLC? AI-DLC does not replace the traditional SDLC; instead, it shifts task execution from humans to AI while retaining the human role of approving the results.

I incorporated the AWS-developed AI-DLC system into my own task-planning application—treating it as a core product feature rather than just a personal coding workflow. (After all, every self-respecting programmer should have their own to-do app😂)
You articulate an "Intent" in your own words; this is broken down into "Units"—large, epic-level tasks with defined acceptance criteria and dependencies. These units are then executed via "Bolts"—execution cycles lasting anywhere from a few hours to several days. At various points throughout the process, execution pauses to await human approval, and a clear history of changes (artifacts) is preserved in the target repository.

For those considering whether to adopt this approach, the most interesting aspect isn't its novelty. What matters is how much of the traditional SDLC remains virtually unchanged, despite the shift to this new model of task execution.

The 16 stages are the old SDLC phases, renamed

The catalog runs three phases — Inception → Construction (per unit) → Operations — across sixteen stages. Every one of them is a classic SDLC activity, just regrouped:

SDLC activity	AI-DLC stage(s)	Phase
Discovery / requirements gathering	Workspace Detection, Requirements Analysis, User Stories	Inception
Planning / estimation	Workflow Planning, Units Generation, Delivery Planning	Inception
System design	Application Design, Functional Design, Infrastructure Design	Inception / Construction
Non-functional requirements	NFR Requirements, NFR Design	Construction
Implementation	Code Generation Planning, Code Generation	Construction
Testing	Build and Test	Construction
Release	Deployment	Operations
Ops / observability	Monitoring	Operations

Nothing was dropped. What changed is who performs each stage and what gates the move to the next one.

What I retained from the SDLC

The sequence of stages itself. The "requirements → design → build → test → deploy → monitor" flow remains linear for each unit of work. AI-DLC did not invent a new lifecycle model but replicated the old one, substituting the human actors involved.
Human approval stages. Moving to the next stage still requires approval of the previous one (analogous to code reviews or checkpoints in traditional SDLC), but this is now enforced at the system architecture level rather than merely through procedural compliance: a stage launches only after all preceding stages in its scope are complete; a conditional write in DynamoDB prevents state conflicts during concurrent executions; and no runner process contains code to set the "approved" status—this is possible only via an HTTP request initiated by a human.
Process documentation. The results of each stage are mirrored in Markdown format (within the aidlc-docs/<intent-slug>/... directory of the target repository) alongside an audit.md file, which logs executions, approvals, change requests, and skipped stages. These are the same artifacts and audit logs always required in SDLC management—only now they are generated automatically rather than created manually.
Work decomposition structure. Units of work—complete with acceptance criteria and dependencies—serve as epics; "Bolts" function as sprints with a clear Definition of Done: a Bolt cannot be closed if even one of its constituent units of work has an unapproved mandatory stage.
Separation of concerns. Design and analysis stages operate in a read-only planning mode; file write permissions are granted exclusively to implementation stages (code_generation, build_and_test, deployment, monitoring). The design phase cannot silently refactor the repository while the document is being prepared—this is the same principle underlying the rule that "reviewers cannot merge their own pull requests."

What has actually changed

Execution has shifted from humans to AI; the human role is now limited to approving results. In a classic SDLC, a human writes the requirements document. Here, however, stages such as ask-executor force an LLM tool call to generate the document, while the human's task is narrowed down to reviewing and either accepting or rejecting the output. There are two types of executors: claude (autonomous execution of Claude within a specific intent's workspace—for stages requiring repository read/write operations) and ask (forcing a tool call via a configured provider—for stages where the output must be displayed as a form, such as clarifying questions or a list of proposed work units). Stage output can also be entered manually via the PUT /stages/:id/output endpoint, allowing the entire process to function even without AI configuration.
The lifecycle is initialized for each unit of work rather than once for the entire project. The stage catalog is data-driven: creating an "Intent" triggers its own "Inception" stages, while accepting a "Unit" launches its own "Construction" stages. Traditional SDLC phases are typically project-level rituals performed once; here, they are generated anew for each Intent and Unit. Furthermore, Units can be processed in parallel and independently, as a specific Unit's stages depend only on preceding stages within that same Unit.
Two artifact stores instead of one. DynamoDB serves as the source of truth (authoritative store); this is necessary because the web application and an agent connected via MCP might write data simultaneously, and only a robust database supports conditional update operations. The Markdown files in the aidlc-docs/ directory constitute a read-only mirror that is updated after every transition. In a traditional SDLC, no such separation exists; the document itself serves as the record—the single source of truth.

Even the "implementation of AI-DLC" did not mean blindly following the specification.

This warrants a separate section: implementing the AWS methodology entailed deliberate and documented deviations from it.

Upstream AI-DLC	This implementation	Why
`aidlc-state.md` is the state	DynamoDB is authoritative; markdown is a mirror	Two writers (web app + MCP) plus conditional-update locking need a real store — a markdown file can't express a `ConditionExpression`
Every stage runs through the coding agent	Form-shaped stages run on the LLM provider directly (`ask` executor)	Keeps file-write authority away from stages that have no use for it
Mob Elaboration (synchronous group review)	Single-player: an editable unit list plus request-changes	This app has profiles, not accounts — the intent of the practice survives, the "whole team" part isn't modelled
33 stages across five phases	16 stages across three	The catalog is data, so growing it back is a data change, not new code
Implicit approval authority	`approvedBy` records who acted, not that they were allowed to	There's no authentication in this app, so approval is a record, not a claim

The point is not that these deviations suit everyone, but rather that "implementing AI-DLC" is not a binary process. Any team adopting a known methodology will adapt it to their system's actual capabilities; documenting these changes—rather than letting them devolve into imperceptible process "drift"—is, in itself, a valuable SDLC discipline.

A second layer of governance atop the first.

The AWS AI-DLC methodology provided me with stages, modules, and an approval checkpoint. It differs from Anthropic’s approach (the AI-native SDLC playbook), which outlines its own six-step model (Planning → Design → Development → Testing → Deployment → Support) based on three governance layers: Skills (advisory rules and policies), Hooks (deterministic runtime safeguards), and human approval stages. Comparing my implementation against this second model revealed gaps that the AWS specification alone did not address, and I resolved all four:

Skills (Rules/Guidelines). Markdown-based policy files (e.g., security checklists or compliance rules) placed by the user in the target workspace; these are automatically attached as context to stages marked as relevant (nfr_design, infrastructure_design, deployment). They are purely advisory in nature, with a level of trust equal to that of any other context passed to the stage.
Hooks. The AIDLC_POST_RUN_HOOK variable executes a custom shell command after the successful completion of the code_generation or build_and_test stages, but before the stage transitions to the awaiting_approval state. A non-zero exit code causes the stage to fail immediately—this serves as a deterministic counterpart to the advisory approach used by Skills and is the mechanism most similar to Claude Code hooks.
AI-based verification extended to code-related stages. The previously existing optional verification step—formerly limited to the requirements or design phases—now also analyzes diffs during the code_generation stage and the results (success/failure) of the build_and_test stage. These checks are purely advisory and never block the process, thereby upholding the inviolable principle that "no executor can approve their own work."
An evaluation suite. Tests located in apps/api/tests/*.eval.ts verify that stage prompts align with the intents defined in the test data (fixtures); this ensures that any changes to prompts or the directory structure—which might otherwise silently break Workspace Detection or Units Generation mechanisms—are caught during the CI phase rather than in production.

All these features are optional and disabled by default—including two that I developed but did not bring into full compliance with the specification: AIDLC_PARALLEL_WORKTREES assigns a separate Git worktree and branch to each unit to ensure complete build isolation, yet it intentionally omits the automatic merge step following approval that was envisioned in the original playbook design. Automatically merging a branch into the actual Git history is too critical an action to entrust to the stage runner without human oversight; Consequently, the branch remains available for the user to review and merge manually—reflecting the general principle that final approval always rests with a human. The POST /aidlc/alerts endpoint closes the operational loop by creating a draft standard Intent based on a monitoring system alert; however, there is no automatic approval or execution involved here either: the generated Intent enters the pipeline at the workspace detection stage alongside other Intents and undergoes the same validation checks.

Conclusion

When creating something similar, it is more useful to view the situation not as a clash between two lifecycle models—SDLC and AI-DLC—but from a different perspective. AI-DLC is essentially the same as SDLC, except that the execution vector is redirected toward AI, while the validation vector is intentionally retained by humans. Control is maintained through structural mechanisms (such as blocks, checkpoints, and access control systems) rather than team discipline; after all, discipline cannot be relied upon when the work between checkpoints is performed by an agent.

Everything else—stages, approval procedures, documentation, audit logs, and backlog structure—is carried over from the old model rather than reinvented. The methodology’s name changes, but the fundamental nature of the work remains largely the same.

Are there any among you who have fully switched to AI-DLC? What methodology do you use? Have you made any changes to the used AI-DLC specifications?

The Database I Deleted: Migrating an Agentic RAG App to AWS Serverless #3

Dmitriy Trunov — Sat, 19 Sep 2026 19:54:30 +0000

Part 3 of 3 — bugs, numbers, and what generalizes

Part 1 covered why an agentic RAG assistant moved off a single EC2 box onto AWS serverless, and the resulting architecture. Part 2 walked through five decisions that didn't survive contact with AWS's fine print: a database that turned out to be a file, why there's no streaming, an auth design CloudFront can't do, what Bedrock batch inference can't give you, and where the vector text actually lives. This part is the payoff — what the migration found, what it cost, and what's still unverified.

Two bugs the migration found in the existing system

Re-deriving the corpus surfaced two problems that had nothing to do with AWS.

303 chunks that could never be retrieved

The original derives a chunk id as {repo}::{file_path}::{section}. Two chunks in one file can share a composed heading — a repeated "Usage", an "Example" under two parents. I measured it:

doc_id collisions (repo::file::section): 303   (of 24,775 chunks)

RRF dedupes by doc_id. So 303 chunks were silently shadowing each other during fusion — one of each pair could never surface, and which one survived depended on iteration order. A pre-existing retrieval bug, invisible from the outside, sitting in a system whose evaluation numbers I'd published.

Adding a per-file ordinal to the hash takes it to zero.

A chunk that could not be embedded at all

Chunk sizes look fine at the median and awful in the tail:

          chunks    p50     p90      p99      max
before    24,775    602   2,797   10,483  119,786   bytes
after     25,482    621   3,096    7,905    7,998

That 119 KB chunk is roughly 30,000 tokens. Titan Text Embeddings caps input at 8,192. It would not have embedded — it would have failed, or been silently truncated, and simply been missing from the vector index.

The fix splits on paragraph boundaries, with a hard fallback for unbroken tables and code blocks, because "never split mid-paragraph" is a nice principle right up until it leaves you with a chunk that no embedding model will accept.

The rate limiter had to change shape, not providers

The crawler paces itself against GitHub's quota and sleeps until reset when it runs low. That's correct for a local run that owns the whole crawl. Inside a Lambda it is wrong twice over:

Sleeping burns billed duration, and a GitHub reset window runs up to an hour — well past the 15-minute limit. The function gets killed mid-sleep and loses its work.
Under a Map, each child sees only its own responses while the quota is account-global. Every child independently concludes it has headroom. maxConcurrency bounds concurrent executions, not requests per hour.

So the crawler now raises instead of sleeping, carrying the reset time, and the state machine catches it into a Wait — which costs nothing and can span an hour:

crawl_task.add_catch(
    wait_for_quota.next(task("CrawlRepoRetryTask", crawl_repo)),
    errors=["RateLimitedError"],
    result_path="$.error",
)

GITHUB_RATE_LIMIT_MODE keeps the sleeping behaviour for local runs. And there is a test asserting the class name — Step Functions matches on the error name the runtime reports, so renaming the exception would silently break the Wait path with no import error to warn you.

The spend cap, ported and improved

The original's proudest feature was an atomic budget reservation: not "read the total, compare, spend" — which races under concurrent traffic — but one UPDATE ... WHERE ... RETURNING that Postgres serializes.

That property transfers exactly onto a DynamoDB conditional write:

get_table().update_item(
    Key={"pk": PK_BUDGET, "sk": window},          # window = today, UTC
    UpdateExpression="ADD total_spent :amt",
    ConditionExpression=(
        "attribute_not_exists(total_spent) OR total_spent <= :headroom"
    ),
    ExpressionAttributeValues={
        ":amt": amt,
        ":headroom": Decimal(str(cap)) - amt,
    },
)

Same one-round-trip check-and-deduct, single-digit milliseconds, zero idle cost. Verified against a real DynamoDB implementation (via moto) rather than reasoned about — 40 concurrent requests against a cap allowing 5 reservations grant exactly 5.

The attribute_not_exists branch is not decoration: ADD creates the attribute, but the condition still has to pass before it exists. Without that clause, every new day starts permanently blocked.

And one deliberate behaviour change, which is really a bug fix. The original cap was a lifetime total. That means one scraper — or one enthusiastic afternoon — exhausts it forever, and every future visitor sees "budget used up" for the life of the demo. Keying the budget item by UTC date makes the window structural: yesterday's spend is simply a different item, nothing has to reset anything, and the worst case is bounded to one day.

What the interface had to become

Streamlit is a WebSocket server holding per-session state. That doesn't survive contact with Lambda, so the UI became a static SPA on S3 behind CloudFront.

It is three files — index.html, styles.css, app.js — with no build step and no framework. That's a deliberate match to the rest of the architecture: nothing idles, few moving parts, CI syncs three files. The API is the real contract, so React later changes nothing server-side.

The design has exactly one idea. The assistant answers two fundamentally different ways, and that distinction is the point of the project, so colour and structure encode which mechanism answered: amber for exact metadata queries, teal for document retrieval, split down the bar when a question needed both. The same coding runs through the question index, so you can see before clicking which path a question will take.

I built it, then looked at it in a browser, which caught four things reasoning had not:

.view { display: grid } beats the browser's [hidden] { display: none }, so the History view rendered underneath the chat. Invisible in review, obvious in a screenshot.
On mobile the question index came before the input, so a phone user scrolled past fifteen questions to reach the box.
The entry header crushed question titles to one word per line on narrow screens.
IBM Plex Mono gives every glyph a full em, so "25,482" rendered with a visible gap around the comma. No figure setting fixes that in a monospace face.

None of those are hard. All of them needed eyes.

The numbers

	Original (EC2)	Serverless
Idle cost	~$15/month	~$2/month
At ~500 questions/month	~$15/month	~$5–8/month
Deployable artifact	multi-GB image	~7 MB zip
Runtime dependencies	torch, sentence-transformers, minsearch, streamlit, pandas, psycopg, openai	boto3 + scrape/parse stack
Cold start	n/a (always on)	artifact download + init
Infrastructure as code	none	5 CDK stacks

The ~7 MB bundle is the single most satisfying number. Dropping the two local torch models — Titan for embeddings, Amazon Rerank for reranking — is what turns a container image into a zip.

Unit prices are estimates and need re-checking before anyone quotes them.

What I have not verified

The original write-up was honest that its deep evaluation was scaffolded but unrun. This one has more to admit, and burying it would make everything above less trustworthy:

Neither quality gate has produced a number. And these are the two that matter:

The routing gate compares which tools fired, question by question, against the OpenAI baseline. It needs Bedrock model access to run.
The retrieval gate re-measures hit-rate and MRR after swapping MiniLM + minsearch for Titan + S3 Vectors + FTS5. It needs a ground-truth set the batch pipeline generates.

Until they run, the claim that this migration preserved answer quality is unmeasured. The original's published retrieval numbers are fiction for this stack.

What is verified: the DynamoDB behaviour, against a real implementation, including the concurrency property above; the SQL port, against the real 279-project artifact (all 11 tools, with the leaderboard and top-projects rankings coming back genuinely different — which is the distinction the router exists to preserve); and the retrieval path end to end over the real 25,482-chunk corpus at 3–8 ms per keyword query.

The risk I'm most worried about

Swapping OpenAI for Nova Pro is not a like-for-like substitution, and the place it will show is routing.

The router makes a 12-way decision with genuinely subtle distinctions. "Top project score" and "leaderboard total score" are different numbers that must not be conflated. A person lookup is required to chain into a follow-up document search, because a bare theme label is an uninformative answer. Those instructions were tuned against a stronger model.

Here's why the obvious evaluation won't catch a regression: ask "who's the top scorer on the leaderboard?", have the model call top_projects_by_score instead of top_authors_by_total_score, and you get a fluent, confident, well-grounded-sounding answer built on the wrong number. An LLM judge scoring answer-against-context will happily mark it RELEVANT. It is consistent with its context. It's just answering a different question.

Only the tool trace exposes that. Which is why the gate compares routing rather than relevance, and why the pass bar gets set before the eval runs rather than after.

What generalizes

Price the floor, not the feature. Aurora Serverless v2 is the right answer to a lot of questions. It was the wrong answer to "where do I put 88 KB of read-only data that a pipeline rebuilds," and the only way to see that was to look at what the database actually held rather than what it was called.

Check what your runtime can actually do before designing around it. Python Lambda can't stream. CloudFront OAC can't sign a POST body. Bedrock batch can't constrain output. Each of those invalidated a design I'd already committed to on paper, and each took minutes to discover once I looked.

A migration re-derives your data, which audits it. Two real bugs in the original — 303 unreachable chunks and one chunk too large to embed — surfaced not from careful review but from rebuilding the corpus and measuring the result. If you migrate something, measure the thing you rebuilt against the thing you replaced.

When two systems must agree, assert it in a test. The vector filter and the SQL filter accepting the same fields is not something either system's own tests can catch. Neither is a Step Functions error-name match. Cross-system invariants need a test that exists specifically to hold them together.

Say what you haven't verified. A migration that "works" in the sense of compiling, synthesizing, and passing 142 tests is still a migration where no model has answered a real question. Those are very different claims, and only one of them is true here.

The Database I Deleted: Migrating an Agentic RAG App to AWS Serverless #2

Dmitriy Trunov — Wed, 16 Sep 2026 16:43:58 +0000

Part 2 — the five decisions that fought back

Part 1 covered why an agentic RAG assistant — built 284 LLM Zoomcamp capstone submissions — moved off a single EC2 box onto AWS serverless, and the resulting architecture: CloudFront and S3 for a static SPA, one Lambda behind a shared-secret header, Bedrock for inference/embeddings/reranking, S3 Vectors and a SQLite artifact for retrieval, DynamoDB for state, Step Functions for ingestion. This part is the five places where the obvious design turned out to be wrong.

Quick recap of the shape: before, everything lived on one t4g.small — Postgres, a Streamlit process holding two torch models in RAM, minsearch indexes, Grafana — idling at ~$15/month. After, nothing idles: a Lambda behind CloudFront calls Bedrock for inference and DynamoDB/S3/S3-Vectors for state, and an offline Step Functions pipeline does the ingestion. Below is why each piece of the "after" ended up where it did.

Decision 1: I deleted the database

The obvious port of Postgres is Aurora Serverless v2 with the RDS Data API — HTTPS-based, so the Lambdas stay outside the VPC and you avoid a NAT gateway. I planned exactly that.

Then I priced the floor. Aurora Serverless v2 at a 0.5-ACU minimum is ~$43/month to sit there doing nothing — three times what the whole EC2 box cost. Dropping to a 0-ACU floor helps, but the auto-pause interval is 300 seconds minimum and the resume takes ~15 seconds, or 30+ after a day idle. For a portfolio demo visited twice a week, "30 seconds before the first byte" is the user experience.

So I looked at what the database actually held, and the answer was embarrassing:

278 projects and a few thousand library rows. 88 KB. Rebuilt from scratch by the ingestion pipeline, and read-only at query time.

That is not a database workload. That is a file.

The read path became a projects.sqlite artifact — the projects and libraries tables, the corpus, and an FTS5 index over it — built by the pipeline, published to S3, and pulled into the Lambda's /tmp on cold start. The write path (conversations, feedback, the spend ledger) went to DynamoDB. ~$43/month → ~$1/month.

The part I didn't expect: it is also better. Postgres full-text search has no IDF at all — ts_rank_cd scores on term frequency and position, corpus-blind. SQLite's bm25() does weight rare terms. The thing I moved to is closer to the minsearch TF-IDF I was replacing than the "proper" database would have been.

And the original project made it cheap. Its own schema.sql opens with a note that these queries already ran on SQLite before the Postgres migration, and that the SQL had been kept portable. Past me left a door open.

Two things did need real work, not placeholder substitution:

-- Postgres
SELECT DISTINCT ON (author) repo, author, github_url, score, author_total_score
FROM projects WHERE author_total_score IS NOT NULL
ORDER BY author, author_total_score DESC

-- SQLite: no DISTINCT ON. GROUP BY with MAX() returns the row that produced
-- the max, which is exactly the row DISTINCT ON would have kept.
SELECT repo, author, github_url, score, MAX(author_total_score) AS author_total_score
FROM projects WHERE author_total_score IS NOT NULL AND author IS NOT NULL
GROUP BY author ORDER BY author_total_score DESC LIMIT ?

And ARRAY_AGG(repo ORDER BY attempt) has no portable equivalent — SQLite only gained ordered aggregates in 3.44, and the Lambda runtime may ship older. That grouping moved into Python, which is duller and correct.

Decision 2: no streaming, and the reason isn't performance

I had planned a streaming Function URL. Then: Lambda response streaming is supported on Node.js managed runtimes only. Python needs a custom runtime or the Lambda Web Adapter.

I could have fought that. I didn't, because streaming buys less here than it looks. This router runs up to six turns of tool calls, plus a query rewrite, plus a rerank — streaming only helps the final call, so the user waits through the entire tool chain either way.

There's also a billing trap worth naming: a client that disconnects mid-stream still bills the full function duration and the full token spend. On a public demo with a shared budget cap, that's a hole someone can walk through by closing tabs.

Buffered JSON. Moving on.

Decision 3: the auth I designed doesn't work

Plan: S3 + CloudFront with Origin Access Control, Lambda Function URL on AuthType: AWS_IAM, OAC signing requests to the Lambda. Clean, no shared secrets.

It does not work for POST. CloudFront signs the request but does not compute the body payload hash, and Lambda rejects unsigned payloads. /api/ask and /api/feedback are both POSTs. Every one would fail with InvalidSignatureException.

API Gateway solves auth — and reintroduces a ~29-second integration timeout that a six-turn loop can exceed.

So: Function URL on AuthType: NONE, with CloudFront attaching a shared secret header, and the Lambda comparing it with secrets.compare_digest. Which leads to an asymmetry I find genuinely interesting:

The Lambda reads the secret from Secrets Manager at cold start, because a Lambda environment variable is stored in plaintext in the CloudFormation template.
CloudFront cannot. Custom origin headers have no runtime secret reference. The value is resolved at deploy time into the distribution config, where anyone with console read access can see it.

That is inherent to the feature, not something I could engineer around. So I stopped pretending it was a credential and wrote down what it actually is: a proof-of-origin token. The rate limiter and the daily budget cap sit behind it precisely because it is not a strong boundary. Both halves are now pinned by tests, so the weakness stays a known property rather than a surprise.

Decision 4: Bedrock batch can't do the thing the whole design rests on

The original enforces a hard rule: only the live chat router makes synchronous LLM calls. Everything bulk — theme classification, ground-truth generation, RAG evaluation, judge scoring, about 11,000 requests — goes through the OpenAI Batch API, at roughly half price.

Two discoveries, in increasing order of annoyance:

The Anthropic/OpenAI-style Message Batches API doesn't exist on Bedrock. Fine — Bedrock has its own batch inference: JSONL to S3, CreateModelInvocationJob, poll, JSONL back. Same discount, same shape, different API.
Bedrock batch inference supports neither tool calling nor structured output. That one hurts. Four of the five bulk stages relied on text_format=<pydantic model> for guaranteed-schema-valid JSON.

So structure moved from a guarantee into a request:

def json_instruction(schema: dict) -> str:
    """The prompt fragment that stands in for structured output."""
    lines = []
    for name, spec in (schema.get("properties") or {}).items():
        bits = [f'  "{name}": {spec.get("type", "string")}']
        if spec.get("enum"):
            bits.append(f'one of {", ".join(map(str, spec["enum"]))}')
        if spec.get("description"):
            bits.append(spec["description"])
        lines.append(" -- ".join(bits))
    return JSON_INSTRUCTION.format(schema="\n".join(lines))

…and parsing became forgiving on the way back — fenced blocks, preambles, trailing explanations. Not because sloppiness is fine, but because discarding an otherwise-correct answer over a stray sentence means re-running a paid batch job. A record that yields nothing parseable returns None for a retry list rather than raising and killing the run.

The live judge is unaffected: Converse's toolConfig does constrain output. Two code paths, deliberately, and the reason is written at both call sites.

The nice part: I kept the seam in one module. _batch.py retained its function names and request shapes and became a translation layer onto Bedrock, so the prompts, parsing, and stage logic in four consumer modules never changed.

Decision 5: what S3 Vectors should actually store

The obvious design puts chunk text in vector metadata, so one query returns everything the answer needs.

Two constraints argue against it, and both are unforgiving:

Filterable metadata is capped at 2 KB per vector, and keys are filterable by default. The p90 chunk here is 2.8 KB — leaving content filterable rejects most of the corpus.
nonFilterableMetadataKeys is fixed at index creation and cannot be changed. Get it wrong and you rebuild.

But the deciding argument was simpler. The SQLite artifact is already open in /tmp. Hydrating text from it is a local file read. Duplicating 31 MB of text into vector metadata means paying to write and store a second copy, and living against the 40 KB-per-vector total cap forever.

S3 Vectors stores identity and filters. SQLite stores text. Vector metadata is now a few hundred bytes.

One correctness trap fell out of splitting retrieval across two systems, and it's the kind that no test of either half would catch: the S3 Vectors metadata filter and the SQLite WHERE clause must accept exactly the same fields. If they diverge, RRF fuses two candidate pools selected under different predicates, and the ranking is quiet wrong. There is now a test whose entire job is asserting those two tuples are equal.

Five decisions were overridden by AWS's terms and conditions: a pricing floor, a runtime limit, a signing gap, the batch API's capabilities, and a metadata limit—rules that conflicted with my original plan.

Migrating an Agentic RAG App to AWS Serverless. #1

Dmitriy Trunov — Fri, 11 Sep 2026 06:49:13 +0000

A previous post covered building an agentic RAG router over **LLM Zoomcamp* capstone submissions. This series is about moving it off a single EC2 box and onto AWS serverless, where nearly every obvious choice turned out to be wrong, and the most valuable decision was deleting a database rather than migrating it. This part covers the starting point and the resulting architecture;

The course on designing and creating a RAG project has concluded. The final project is working as expected. Now it's time to optimize and use the developed concept as a production system. In the previous article, I described creating an agent-based RAG router for processing LLM theses submitted at Zoomcamp. I decided to write a series of articles dedicated to migrating the project from a single EC2 server to a serverless AWS environment. This section examines the starting point and the resulting architecture.

Starting point and its drawbacks

The initial assistant is working. It answers fundamentally different types of questions about these repositories.
For example, "What does this project do?" using a hybrid search of README files, and "various questions about performance statistics and libraries used in projects?" using a precise parameterized SQL query. The agent router chooses between them.

In my capstone, it runs as six Docker Compose containers on a single t4g.small instance, deployed via SSH, with a 2GB swap file as a cheap safety net so that Postgres, Streamlit (which contains the Torch embedder and Torch cross-encoder in RAM), and Grafana can coexist on a 2GB RAM instance.

This isn't bad for a capstone project. But it has properties I'd like to eliminate for a real system.

The corpus - 24,775 fragments, 115MB of JSON-is loaded and rebuilt into the process's RAM on each run.
Two local machine learning models require a 1.7GB virtual environment and an image. Several gigabytes in size.
There's no infrastructure as code at all. This approach is completely unsuitable for a real system. Deployment is done via a bash script.
The cost of downtime is about $15 per month, regardless of whether anyone visits it.

So: rebuild it as a fully serverless AWS application using real infrastructure as code (IaC) as a learning path. All development is done using Amazon's own resources—Nova for output, Titan for embeddings, Amazon Rerank for reranking, and boto3—the only SDK. No OpenAI, no Anthropic.

Here is what actually happened.

Two architectures:

Before — everything on one box, everything in RAM:

flowchart LR
    User["User"] --> ST["Streamlit :8501<br/>torch embedder + reranker<br/>24.8k chunks in RAM"]
    ST --> KW[("minsearch.Index<br/>keyword, in-process")]
    ST --> Vec[("minsearch.VectorSearch<br/>MiniLM, in-process")]
    ST --> PG[("Postgres 17<br/>projects · conversations<br/>feedback · budget")]
    ST --> OAI["OpenAI<br/>Responses + Batch API"]
    PG --> Graf["Grafana :3000"]
    Cache[("vector_index_cache.npz<br/>37MB on disk")] -.-> Vec

    subgraph EC2 ["one t4g.small — 6 containers, ~$15/mo idle"]
        ST
        KW
        Vec
        PG
        Graf
        Cache
    end

    style PG fill:#336791,color:#fff
    style EC2 fill:#f6f8fa,stroke:#8b949e

After — nothing idling, and the corpus is an artifact rather than a runtime cost:

flowchart LR
    User["User"] --> CF["CloudFront<br/>+ edge origin check"]
    CF --> S3W[("S3: static SPA<br/>3 files, no build step")]
    CF --> L["api Lambda<br/>~7MB zip · no VPC"]

    L --> BR["Bedrock<br/>Nova Pro / Nova Lite<br/>Titan · Rerank"]
    L --> SV[("S3 Vectors<br/>25,482 vectors<br/>identity + filters")]
    L --> SQ[("S3 artifact → /tmp<br/>projects.sqlite<br/>projects · chunks · FTS5")]
    L --> DDB[("DynamoDB<br/>conversations · feedback<br/>budget/day · rate limits")]

    SFN["Step Functions<br/>scrape · Map(crawl + Wait)<br/>embed · build artifact"] --> SV
    SFN --> SQ

    style BR fill:#232F3E,color:#fff
    style DDB fill:#4053D6,color:#fff
    style L fill:#0C8C7D,color:#fff

Two Ways of Knowing: Building an Agentic RAG Explorer for LLM Zoomcamp Capstones.

Dmitriy Trunov — Mon, 17 Aug 2026 14:37:21 +0000

A technical deep-dive into an agentic RAG assistant built for the LLM Zoomcamp 2026 capstone — the architecture, five real production bugs, and what a race-condition stress test taught me about spend caps.

The problem with many GitHub repos

The LLM Zoomcamp 2026 cohort produced over 300 capstone submissions — real RAG projects, each with its own GitHub repo, README, dependency manifest, and a whole course leaderboard score. Browsing that by hand doesn’t scale. So I built a chatbot over the whole corpus to obbtain the all themes, used technology, and library.

_But the questions people actually ask about a corpus like this split cleanly into two kinds, and they need fundamentally different retrieval:

”What does project X do? How does it implement reranking?” — a content question. Answer it by finding and grounding on the right README excerpt.
”Which libraries are most common? What’s the top-scored project? Who’s the leaderboard’s top scorer?” — an aggregation question over the corpus’s structured metadata: score, votes, pass/fail, theme, declared dependencies, each author’s total course score._

The second category is the interesting one, because it’s exactly where naive RAG quietly breaks. Ask an LLM to “count” or “rank” over a pile of retrieved text chunks, and it will confidently produce a plausible-sounding wrong answer — not because the model is bad, but because counting isn’t a retrieval problem. It’s a database problem wearing a chat interface.

And there’s a subtler trap hiding in that second category: “top-scored project” and “top of the leaderboard” sound like the same question. They aren’t. A project’s own score and its author’s total course score (homeworks + project + learning-in-public) are genuinely different numbers, and conflating them is a real, plausible failure mode — one I actually hit and had to fix (more on that below).

Solving this well means routing each question to the right retrieval mechanism, not picking one strategy and forcing every question through it.

Architecture: one router, two ways of knowing

The ingestion side is what you’d expect from a RAG pipeline — scrape the listing pages, crawl every submitted repo’s markdown via the GitHub API, chunk by heading, cache everything to disk. Theme and tech-stack classification run as OpenAI Batch API jobs (bulk, offline, ~50% cheaper than synchronous calls) rather than one API call per project in the hot path.

The interesting part is the runtime query path. Instead of one retrieval strategy, there’s an agentic router sitting in front of two completely different tools: search_docs (hybrid keyword+vector search, RRF-fused, cross-encoder reranked, LLM-query-rewritten) and eleven parameterized, hand-written SQL functions-never LLM-generated SQL.

A “which libraries are most common” question never touches the model’s judgment about counting. It runs COUNT(*) … GROUP BY.

Why an agentic router, not “just RAG plus a calculator tool”?

The tool-calling loop itself is unremarkable — it’s the same shape as any OpenAI function-calling agent: feed the model a question and a set of tool schemas, let it pick zero or more tools, feed the results back, repeat until it produces a final answer instead of another tool call.

for _turn in range(self.max_turns):
    response = self.llm_client.responses.create(
        model=self.model,
        input=input_items,
        tools=self._tool_schemas(),
    )

    function_calls = [item for item in response.output if item.type == "function_call"]

    if not function_calls:
        return {"answer": response.output_text, "sources": all_sources, "tool_calls": tool_calls_log}

    # ... execute each function_call, append results, loop again

What actually matters is what’s in those eleven tool schemas — specifically, how much implicit intent you have to make explicit before the model reliably picks it up. Here’s a real example from this project. most_common_libraries takes an optional filter_top_scored_n parameter that restricts the aggregation to the top-N scored projects instead of the whole corpus. The first version of its description just documented what the parameter did. Ask “what are the most common libraries among the top-scored projects?” and the model would call the tool — but leave filter_top_scored_n unset, silently answering for the entire corpus instead of the subset the question actually asked about. The count was real, correctly computed, exactly matching the SQL — and still the wrong answer, because it answered a different question than the one asked.

The fix wasn’t code. It was making the qualifier explicit in the tool description itself:

IMPORTANT: if the question says ‘top-scored’, ‘top projects’, ‘best projects’, or similar, you MUST set filter_top_scored_n — leaving it null answers for ALL projects instead, which is a different (wrong) answer to that question.

Tool descriptions are prompts. Implicit qualifiers a human reader would catch instantly — ‘top-scored’, clearly means ‘restrict the scope’, — need to be spelled out just as explicitly as any other instruction, or the model will happily run the wrong exact query with complete confidence.

Five bugs that actually happened:

Everything above is the design that looks right on a whiteboard. Here’s what broke in practice.

1. CUDA wheels on a code path that never touches a GPU

Every embedding call in this project forces device=”cpu” explicitly, in code, everywhere — sentence-transformers on CPU, CrossEncoder reranking on CPU. And yet uv sync was still resolving torch’s full CUDA build by default: cuda-bindings, cuda-toolkit, half a dozen nvidia-* packages — several gigabytes of GPU tooling for a code path that never runs on a GPU, on any platform, including ARM (the cuda-toolkit extras aren’t gated to exclude aarch64).

The fix is a standard uv pattern most people don’t reach for until it bites them:


[tool.uv.sources]
torch = [{ index = "pytorch-cpu" }]

[[tool.uv.index]]
name = "pytorch-cpu"
url = "https://download.pytorch.org/whl/cpu"
explicit = true

One subtlety: this silently did nothing until torch was also listed as a direct dependency in pyproject.toml, not just a transitive one pulled in via sentence-transformers. Once it was explicit, re-locking dropped 18 CUDA/nvidia/triton packages from uv.lock, and uv sync actually uninstalled them from the local environment. Smaller image, faster builds, no wasted disk on a code path that was never going to touch a GPU.

2. A killed connection took the whole app down until restart

The router keeps one Postgres connection open for the process’s entire lifetime — avoiding a fresh-connection-per-query cost for a chat app that might field hundreds of questions. That’s a reasonable optimization, and it worked fine until a killed backend connection (psycopg.errors.AdminShutdown) took down every subsequent query_projects call until the whole process was manually restarted. One dead connection, and the assistant stopped answering any structured question for anyone.

The fix was reconnect-and-retry-once logic wrapped around every SQL tool call. But there was a second, quieter bug hiding underneath it: every one of those SQL tools is a read-only SELECT that never calls .commit(), and Postgres’s default autocommit=False means each one silently opens a transaction that never closes. Left alone, the shared connection accumulates “idle in transaction” state — which then blocks unrelated DROP TABLE statements running elsewhere in the same database. It happened twice before the pattern was obvious. The real fix was one line: roll back (never commit — nothing was written) after every single tool call, closing the implicit transaction immediately instead of leaving it open indefinitely.

3. Three judge false-negatives, one root cause

The live chat has an LLM-as-judge step that scores every answer’s relevance against the context it was actually grounded in. Three separate times, it flagged a correct answer as unsupported, and all three traced back to the same underlying mistake: the judge was being shown a derived view of the grounding data, not the real thing.

Round one: aggregation answers (no retrieved document chunks) were judged against a hardcoded empty context string, so every purely-SQL-grounded answer looked ungrounded by construction. Round two: after fixing that by building context from the sources list, a score-only field left the judge unable to verify an author_total_score field the answer legitimately cited, because that field simply wasn’t in the sources view. Round three, the sharpest one: aggregation-only tool results (a library count, say) have no github_url to become a sources entry — correctly, there’s nothing to link to — but that left the judge with nothing at all to check a fully correct count against, so it flagged genuinely accurate answers as hallucinated.

All three converged on the same fix: stop trying to derive judge context from a UI-facing view of the answer (the sources list), and instead hand the judge the actual tool-call results the answering model itself saw — the real data, not a lossy projection of it built for a different purpose.

4. The budget-cap race condition

Write on Medium
This is the one I’m proudest of catching before it mattered in production, because the naive version looked completely reasonable.

A public demo needs a hard spend cap — otherwise it’s a free, unlimited ChatGPT proxy to anyone with the URL. The first version did the obvious thing: read the total spent so far from Postgres, compare it to the cap in Python, and only call the LLM if there was room left.

That’s a classic time-of-check-to-time-of-use race. Under concurrent traffic, every simultaneous request reads “total is under the cap” before any of them commits their own cost. With enough concurrent visitors, the real total can blow straight past the cap — a $0.05 limit could, in the worst case, cost many times that if enough requests land in the same window.

The fix replaces “read, compare, spend” with one atomic Postgres operation — a conservative reservation made before the LLM call, not a check made after:

def try_reserve_budget(cap: float, amount: float = RESERVATION_USD) -> bool:
    conn = get_connection()
    try:
        with conn.cursor() as cur:
            cur.execute(
                """
                UPDATE budget_ledger
                SET total_spent = total_spent + %s
                WHERE id = 1 AND total_spent + %s <= %s
                RETURNING total_spent
                """,
                (amount, amount, cap),
            )
            row = cur.fetchone()
        conn.commit()
        return row is not None
    finally:
        conn.close()

Because a turn’s real cost isn’t known until after the LLM responds, this reserves a conservative fixed upper bound up front and trues it up to the real cost afterward (or fully refunds it if the call fails) — but the cap enforcement itself is a single atomic UPDATE … WHERE … RETURNING, which Postgres serializes across concurrent transactions the same way it serializes any other concurrent row update. There’s no gap between “check” and “spend” for a race to land in.

I didn’t just reason about this — I stress-tested it against the real database. Fifty concurrent reservation attempts against a $0.05 cap, with $0.01 reservations:

concurrent attempts: 50
succeeded reservations: 5
final ledger total: $0.0500 (cap was $0.05)
PASS: ledger never exceeded the cap under concurrency

Exactly five succeeded. The ledger landed at exactly the cap, not a cent over. The naive read-then-check version would have let far more than five through under the same load.

5. The disk-exhaustion saga that took two fixes to actually fix

Deploying to a small (2GB RAM, 20GB disk) EC2 instance, docker compose up — build failed with “no space left on device” mid-build. The instance had 8GB free at the time — that should have been plenty for one ~1.6GB Python image.

The first diagnosis looked right: docker-compose.yaml built five separate images from the identical Dockerfile (one per service — init-db, load-projects, build-index, streamlit, dashboard), each independently unpacking a multi-GB layer in parallel. Giving all five services a shared image: tag seemed like the obvious fix — one name instead of five.

It wasn’t enough. The retry failed with the exact same error. Compose was still building and exporting the image once per service definition, just landing on the same final tag — five redundant multi-gigabyte export operations racing for disk, instead of five distinct named images. The tag was shared; the work wasn’t.

The actual fix: only one service keeps a build: block. The other four reference the same image: with no build: of their own. Compose’s build phase — triggered by — build — runs to completion for every service that owns a build: block before it starts any containers at all; depends_on only orders container start, not the build phase. So by the time the other four services start, the image the first one built already exists on disk, and they just use it — no rebuild attempted, no redundant export.

The lesson, generalized past Docker: when a fix makes the symptom go away sometimes but the same failure mode returns under slightly different conditions, look for whether you actually eliminated the redundant work, or just gave the redundant work a nicer name.

Evaluating an agentic router

Retrieval-quality evaluation for a fixed single-strategy RAG pipeline is well-trodden: hit-rate, MRR, precision@k against a ground-truth question set. An agentic router adds a layer on top of that — it’s not just “did retrieval find the right chunk,” it’s “did the router pick the right tool.”

This project runs two evaluations at different depths. The deep one — comparing keyword/vector/hybrid/hybrid+rerank retrieval strategies and two answer-generation prompts against an LLM-generated ground-truth set, via the Batch API — is fully scaffolded but wasn’t run for this write-up; it needs a real API key budget I hadn’t allocated yet. No numbers invented to fill that gap.

The one that did run is faster and, in practice, more useful day to day: a 30-question smoke eval, running every question from the app’s own example-questions sidebar through the live assistant — real retrieval, real SQL, real LLM calls — judged the same way a live chat turn is judged. It’s a regression check on the router’s tool-routing, not a substitute for the deeper eval, and it re-runs in under three minutes:

30 questions — RELEVANT: 30 PARTLY_RELEVANT: 0 NON_RELEVANT: 0 missing: 0

relevance_score: 1.0 | total cost: $0.0248 | avg response time: 5.64s

This eval set earned its keep twice during development — it’s what caught the filter_top_scored_n bug above, and separately caught a one-shot Docker Compose loader container silently reverting an author_total_score field to NULL after a stale image was reused. A fast, always-current smoke eval that’s genuinely wired into your deploy process catches real regressions a slower, less-frequently-run ground-truth suite won’t.

Hardening for a public demo

Three small features made the difference between “a demo I can run locally” and “a demo I can actually hand someone a URL for”:

A production flag that hides the ingestion trigger from public visitors — nobody browsing a public demo should be able to kick off a live re-crawl of 278 GitHub repos.
The atomic budget cap described above.
Answer-from-history caching: an exact-match repeat of a previously-asked question (unsurprisingly common when people click the same example question twice) answers instantly from the logged prior turn — no new LLM call, no new cost, and no budget reservation at all, since nothing is actually being spent.
None of these are exotic. All three are the kind of thing that’s easy to skip when a project is “just a demo” — right up until it’s a public URL.

What generalizes

A few things from this build feel true beyond this specific project:

Tool descriptions are prompts, not documentation. An implicit qualifier a human reader catches instantly — “top-scored” clearly means “restrict the scope” — has to be spelled out as explicitly as any other instruction, or the model will run the wrong exact query with complete confidence and no visible error.
A judge needs the real grounding data, not a UI-shaped view of it. Every judge false-negative in this project traced back to feeding the judge something derived from the actual answer-generation context, built for a different purpose (usually, rendering source links), rather than the real tool output the answering model itself saw.
Anything with a spend cap under concurrent traffic needs an atomic reservation, not a read-then-check. This is true well beyond LLM cost caps — any shared, capped resource checked by “read the total, compare, then act” has the same race, and it’s worth stress-testing with real concurrent load, not just reasoning about it on paper.

The code is on GitHub:

Built as a capstone project for [LLM Zoomcamp] 2026 cohort

Begin your journey with Amazon Q Developer

Dmitriy Trunov — Sun, 28 Sep 2025 19:07:04 +0000

Last year, we heard too much about AI and AI technology. Names such as ChatGPT, Claude, and Deepseek are well-known in the internet community. Many people don't see themselves without their hints and help in everyday life.

Almost all software developers are using tools like Copilot, Cursor, and Windsurf in their working routines. However, in the development community, I encountered the ignorance of one tool that is worth paying attention to.
This tool is named "Amazon Q Developer".

A former Amazon product, "CodeWhisperer", became a part of Amazon Q Developer. It is a generative artificial intelligence (AI) powered conversational assistant that can help you understand, build, extend, and operate software applications. This product was made generally available on April 30, 2024. Over the years of constant development and improvement, it has received many improvements and updates. It was full integration into the AWS ecosystem, with major updates and milestones occurring throughout 2025. Amazon Q Developer is built on Amazon Bedrock and includes automated abuse detection implemented in Amazon Bedrock to enforce safety, security, and the responsible use of AI.

Amazon Q was provided in two versions - Amazon Q CLI and Amazon Q extension for development tools such as Visual Studio Code and JetBrains IDE.

Let's start with Amazon Q Developer extension for free.

To do this, go to the website [AWS Skill Builder]https://skillbuilder.aws/ and create a free AWS Builder ID (no AWS account required).
Today, we are planning to start using Amazon Q as an IDE extension. I think you already have something installed on your computer. Download the extension for Visual Studio Code. Install the Amazon Q extension in VS Code.

This procedure does not require you to have a Builder ID. If you have not yet signed up for Builder ID, you will have the opportunity to do so during the sign-in process.

Choose the Amazon Q icon in your IDE.

The icon will be on the side of the interface by default.

Follow the instructions in your browser to authenticate with Builder ID.

To begin using Amazon Q, choose the Amazon Q icon to chat with Amazon Q, or choose Amazon Q from the navigation bar at the bottom of your IDE.

Nowadays, many developers are bound by NDA rules and are concerned that their code may end up on the other side. I came across a situation where some very experienced Amazon Q users don't know some subtleties of these settings. Let's tune some extension settings.

This setting allows you to customize the ability to send tracking information in AWS

Let's explore some additional settings related to context. This is the most important part of Amazon Q Developer that allows us to use our own context to provide AI with the most relevant questions.
The local workspace context created by these settings helps Amazon Q deliver more accurate answers and speed up the work capabilities. Amazon Q maintains the context of your conversation within a given session to inform future responses. You can ask follow-up questions or refer to previous questions and responses throughout the duration of your session. To start a new conversation with Amazon Q, open a new tab in the panel. You can open up to 10 tabs at a time. Amazon Q doesn't retain context across different conversations.

Now you can ask Amazon Q questions, update your code, and initiate actions with quick commands all from the Amazon Q chat panel in your IDE.

To start chatting with Amazon Q, choose the Amazon Q icon from the navigation bar in your IDE and enter your question in the text bar. To start chatting with Amazon Q in Visual Studio, choose View from the main menu and then choose Amazon Q chat.

When you ask Amazon Q a question, it uses the current file that is open in your IDE as context, including the programming language and the file path.

If Amazon Q includes code in its response, you can copy the code or insert it directly into your file by choosing Insert at cursor. Amazon Q might include inline references to its sources in its response. To view a list of sources, expand the Sources section at the bottom of a response.

Chat commands

You can enter the following commands in the chat panel to access Amazon Q features, depending on your IDE.

/transform - Use this command to update the code language version of an entire project.

/dev - Use this command to get an implementation plan to develop a feature.

/test - Use this command to generate unit tests for your code.

/review - Use this command to review your codebase for security vulnerabilities and code quality issues.

/doc - Use this command to generate READMEs for your project base.

/clear - Use this command to clear a current conversation. This removes all previous conversations from the chat panel and clears the context that Amazon Q has about your previous conversation.

/compact - Use this command to compact your chat history when the context window approaches its capacity limit. This creates a concise summary of your conversation while preserving essential information.

/help - Use this command to see an overview of what Amazon Q can and can't do, example questions, and quick commands

Try out the Amazon Q developer capabilities and share your first impressions.

In the next articles, I will try to contribute to you some advanced interaction techniques with Amazon Q Developer.

DEV Community: Dmitriy Trunov

I Gave Three Broken AWS Environments to an AI On-Call Engineer. Here's What AWS DevOps Agent Found

Part 1: Setting it up takes about 5 minutes

Part 2: "Why didn't my Auto Scaling Group scale?"

Part 3: Investigating a network attack using VPC Flow Logs

Part 4: RDS connection exhaustion, and where the agent contradicted itself

What I liked

What to watch out for

Should you try it?

I'd put it in the on-call path as the first responder that hands a human a pre-built timeline, not as an autonomous fixer… yet

AWS DevOps Agent, Part 4: RDS Connection Exhaustion, and Why You Should Still Check the AI's Answers

The setup

The investigation: about 16 minutes

It found a second problem I didn't plant

The mitigation plan

Agent-ready spec

Where the chat contradicted the investigation

❌ Contradiction 1: is 150 above or below the default?

❌ Contradiction 2: will wait_timeout help?

✅ What the chat did well

What I take from this

Takeaways

AWS DevOps Agent, Part 3: I Attacked My Own Web Server. Here's What the AI Found in the VPC Flow Logs.

The setup

Starting the investigation

The verdict: Severity high

The false-positive finding

Findings and evidence

Mitigation plan, with guardrails

Follow-up chat

Takeaways

AWS DevOps Agent, Part 2: Why Didn't My Auto Scaling Group Scale?

The setup

Starting the investigation

Watching it think

Ruling out the alternatives

The root cause

The mitigation plan

The finished summary

Asking follow-up questions

Takeaways

AWS DevOps Agent, Part 1: Setting Up Your First Agent Space

What is an Agent Space?

Step 1: Open the service

Step 2: Create the Agent Space

Step 3: The agent starts mapping your environment

Step 4: Explore the capabilities

Step 5: Open the web app

Checklist

The SDLC is dead. Long live the AI-DLC!?

The 16 stages are the old SDLC phases, renamed

What I retained from the SDLC

What has actually changed

Even the "implementation of AI-DLC" did not mean blindly following the specification.

A second layer of governance atop the first.

Conclusion

The Database I Deleted: Migrating an Agentic RAG App to AWS Serverless #3

Part 3 of 3 — bugs, numbers, and what generalizes

Two bugs the migration found in the existing system

303 chunks that could never be retrieved

A chunk that could not be embedded at all

The rate limiter had to change shape, not providers

The spend cap, ported and improved

What the interface had to become

The numbers

What I have not verified

The risk I'm most worried about

What generalizes

The Database I Deleted: Migrating an Agentic RAG App to AWS Serverless #2

Part 2 — the five decisions that fought back

Decision 1: I deleted the database

Decision 2: no streaming, and the reason isn't performance

Decision 3: the auth I designed doesn't work

Decision 4: Bedrock batch can't do the thing the whole design rests on

Decision 5: what S3 Vectors should actually store

Migrating an Agentic RAG App to AWS Serverless. #1

Starting point and its drawbacks

Two architectures:

Two Ways of Knowing: Building an Agentic RAG Explorer for LLM Zoomcamp Capstones.

Begin your journey with Amazon Q Developer

❌ Contradiction 2: will `wait_timeout` help?