Dmitriy Trunov

Posted on Oct 11

AWS DevOps Agent, Part 2: Why Didn't My Auto Scaling Group Scale?

#agents #aws #devops #infrastructure

Part 2 of 4. Overview · Setup · EC2 · Network · RDS

This is a common incident: CPU hits 100%, the alarm goes red, MaxSize is 3, and the Auto Scaling Group still runs one instance. I gave this scenario to AWS DevOps Agent to see how far it got without help.

The setup

Lab architecture (source: AWS DevOps Agent workshop)

The workshop pre-deploys:

Lab2-ASG: an Auto Scaling Group of t3.micro instances (min 1, max 3) behind Lab2-ALB
Lab2-ASG-CPU-High: a CloudWatch alarm on the ASG's average CPU above 60%
Lab2-RemoteLoadTrigger: a Lambda that uses SSM Run Command to run a CPU-burn script on every instance for 5 minutes, with no SSH needed

To start the incident, I opened the Lambda, created a test event with {} as the payload, and clicked Test. About 2–3 minutes later the alarm went from OK to In alarm.

Starting the investigation

In the web app, Start investigation opens a short form. It also offers quick prompts such as Latest alarm, High CPU usage and Error rate spike.

Investigation details: ASG CPU utilization spiked to near 100% during load testing - need to determine root cause and assess infrastructure scaling
Name: Workshop_ASG_CPU_Investigation
Priority: Medium (default)

⚠️ You may see "Unable to describe support level" in the bottom-left corner. The workshop says this is expected and safe to ignore.

Watching it think

The Summary tab updates live. Within seconds the agent had found the ASG and described its current focus:

The Investigation timeline tab shows the reasoning step by step. Each entry lists the tool calls behind it (use_aws, datetime), and you can expand them to see the actual API calls:

Here is what happened in the first two minutes:

Time	What the agent did
+11s	Checked the current time and listed Auto Scaling Groups
+34s	Found `Lab2-ASG`: t3.micro, Min=1/Max=3/Desired=1, part of CloudFormation stack `devops-agent-combined`, group metrics not enabled
+1m32s	"Very revealing findings already": `ScalingPolicies: []`, alarm has `AlarmActions: []`, and scaling history has only one event ever (the initial launch)
+1m32s	Started a sub-agent (`alb-traffic-analysis`) in parallel to check whether real traffic caused the spike
+1m48s	Finding: No Auto Scaling policies configured on Lab2-ASG

So in under two minutes it had the core answer: the alarm fires, but nothing listens to it.

Ruling out the alternatives

A good investigation also eliminates other explanations, and the agent kept checking after its first finding:

When exactly did it happen? About 0.4% CPU baseline for 2+ days, then a single anomaly. At 1-minute resolution: 08:15 ramp (avg 19%), 08:20 peak (avg 81%, max 100%), 08:25 back to idle.
Was it CPU credit throttling? No. "CPUCreditBalance held at the t3.micro maximum of 288 throughout… the 100% CPU was genuine compute demand, not a burst-credit starvation artifact." This is the kind of t3-specific check a person can easily forget.
Was it user traffic? No. The ALB sub-agent showed Lab2-ALB at a baseline of 0–1 req/min during the spike.
So what caused it? It found the SSM Run Command (AWS-RunShellScript, with the exact CommandId) that ran a yes > /dev/null loop per vCPU for 300 s. That is the mechanism the workshop uses, and the agent identified it without being told.

It also recorded a gap: it couldn't tell who sent that SSM command, because cloudtrail lookup_events was blocked in the workshop account. It noted this is "not material to the root cause… but needed for full accountability." I value an agent that says what it couldn't check.

The root cause

At +10m22s:

No Auto Scaling policy wired to the CPU alarm. Lab2-ASG has no scaling policies attached (ScalingPolicies: []). The alarm Lab2-ASG-CPU-High exists and would enter ALARM during a CPU spike, but has no AlarmActions configured, so it cannot trigger scale-out.

The Root cause tab separates Impact, Root causes and Investigation gaps, and has Copy/Download buttons, so you can paste it straight into a postmortem.

The mitigation plan

Clicking Generate mitigation plan took a few minutes and produced this:

Attach a target-tracking scaling policy (CPU target 55%) to Lab2-ASG *and codify it in the devops-agent-combined CloudFormation template.*

Every step comes with a ready-to-run command:

Prepare. Save the current ASG config and the empty policy list as a rollback baseline.
Pre-validate. Confirm there's no conflicting policy.
Apply.

   aws autoscaling put-scaling-policy --auto-scaling-group-name Lab2-ASG \
     --policy-name Lab2-ASG-TargetTracking-CPU55 --policy-type TargetTrackingScaling \
     --estimated-instance-warmup 120 \
     --target-tracking-configuration '{"PredefinedMetricSpecification":{"PredefinedMetricType":"ASGAverageCPUUtilization"},"TargetValue":55.0,"DisableScaleIn":false}' \
     --region us-east-1

Post-validate. Check the policy and the CloudWatch alarms it creates.
Rollback. Remove the policy.

Three details here are worth pointing out:

⚠️ Drift warning: "Making this change directly via the API registers as drift against CloudFormation stack devops-agent-combined and may be reverted on the next stack update." The agent knows the ASG is managed by IaC.
ℹ️ Reactive scaling has lag: "For a burst as short as ~10 minutes, new capacity may only become healthy near the end." It suggests scheduled scaling or a higher MinSize ahead of known load tests.
💰 Cost note: more instances under load means a higher EC2 bill.

The finished summary

The final Summary shows status complete, Severity: low (correct, since there was no user-facing impact: zero 5XX errors and no latency increase), and an annotated incident timeline that marks the CPU ramp, the SSM command, the peak, the ALARM transition and the return to OK.

Below that is a Code change specification: add an AWS::AutoScaling::ScalingPolicy to the Lab2AutoScalingGroup logical resource, decide what to do with the now-redundant alarm, and reconsider whether t3.micro is the right size.

Asking follow-up questions

While the investigation runs you can ask questions in chat. For example: "Why didn't my ASG scale during the CPU spike?" My question went into a Queued requests list, and the investigation addressed it as part of the work instead of starting a separate thread.

Takeaways

⏱ About 10 minutes to a root cause I would have accepted from a senior engineer
🔍 It ruled out credit throttling and user traffic with evidence, instead of only confirming the first hypothesis
🧱 It recognised that the resource is managed by CloudFormation and wrote the fix for the template, not only for the API
🙋 It listed what it couldn't verify

Next, in Part 3, I attack a web server from my laptop and see whether the agent can work out what happened using VPC Flow Logs.

DEV Community