Part 2 was an operations problem. This one is a security problem: a public web server gets scanned and flooded. Can AWS DevOps Agent work out who did it, how, and what to fix, using only VPC Flow Logs and CloudWatch?
The setup
Lab architecture (redrawn from the AWS DevOps Agent workshop)
- Lab3-WebServer: a public EC2 instance
-
VPC Flow Logs going to CloudWatch Logs (
/aws/vpc/flowlogs/Lab3-networksec) -
Lab3-UnusualConnections: an alarm on
NetworkPacketsInwith a sum above 500 per 5 minutes
From my own laptop I ran the workshop's simulate-attack.sh against the server's public IP:
chmod +x simulate-attack.sh
./simulate-attack.sh <WEB_SERVER_PUBLIC_IP>
The script runs four phases over about 2 minutes:
- Port scan of 15 common ports
- Web enumeration of admin and login pages
- SSH brute force with common usernames
- Connection flood of 450+ HTTP requests
About 5 minutes later, Lab3-UnusualConnections went to In alarm.
Starting the investigation
- Details: Suspicious network traffic detected from external source - need to analyze attack patterns and assess security posture
- Starting point: CloudWatch alarm Lab3-UnusualConnections triggered due to high incoming network traffic. Investigate source, attack methodology, and recommend security improvements.
The workshop says to expect 8–12 minutes, because querying Flow Logs takes longer than reading metrics.
The verdict: Severity high
The summary had three layers, and that structure is what I'd want from a human analyst as well:
-
Root cause: security group
sg-…exposes TCP 22/80/443/8080 to0.0.0.0/0. This was baked into the CloudFormation template at stack creation… never tightened since. CloudTrail confirmed there were no later manual changes. - Direct trigger: because port 80 was open to everyone, an external IP (my laptop) ran a recon port scan and then a Layer-7 HTTP flood on TCP/80: 2,764 accepted packets, about 80% of inbound traffic. It also found a similar flood from a different IP on 30 September, three days before my test and before I had run anything, so someone else had found the same open host.
-
Contributing cause: the
NetworkPacketsInalarm also fires on benign AWS/S3 return traffic to ephemeral ports.
It described the overall pattern as "opportunistic, rotating-IP internet scanning/flooding of an exposed public host, not a single persistent actor, and not a control-plane compromise."
The false-positive finding
I only asked about the latest alarm. The agent went back through every firing of Lab3-UnusualConnections since 30 September and broke each one down:
3 of the 4 'recurring spike' windows were dominated by AWS service IP ranges (52.217.x, 52.216.x, 16.182.x) sending large packet bursts to high ephemeral destination ports, the signature of legitimate S3/download return traffic, not an attack.
In other words, the alarm is mostly noise. A plain packet-count threshold can't tell an aws s3 cp apart from a flood. If an on-call engineer was paged for all four of those, they would soon start ignoring it, and that's how a real attack gets missed.
Findings and evidence
The findings list shows how the agent labels its own reasoning:
- 🔴 root cause: the world-open security group since stack creation
- 🟠 cause: open ports enable unsolicited scans; benign S3 traffic triggers the alarm
- ⚪ hypothesis, ruled out: "Security group modified after creation to open SSH/HTTP/HTTPS". CloudTrail showed it was never modified.
- 🔵 symptom: recurring short spikes since 30 September
- ⚪ gap: GuardDuty access denied by SCP
- ⚪ observations: no misuse of instance-role credentials, VPC Flow Logs confirm the flood and scan, no control-plane changes
- ⚪ topology: instance, VPC, subnet, security group, flow log ID and log group, all mapped
Keeping the ruled-out hypotheses in the list is useful. In a postmortem, what it wasn't matters as much as what it was.
Mitigation plan, with guardrails
The plan: remove the 0.0.0.0/0 rules for 22 and 8080 completely, and scope 80/443 to trusted clients or an ALB+WAF, then reconcile the CloudFormation stack so the fix lasts.
The pre-validate steps are careful:
- Save the current rule set as the rollback baseline
- Confirm VPC Flow Logs are active so you can see the effect of each removal
- Check recent ACCEPTed flows on each port so you don't cut off a legitimate client
- Check that the instance is reachable through SSM Session Manager before removing port 22, so you don't lock yourself out
One more detail: the apply steps that revoke security group rules were flagged "Automated execution of this removal is denied by the safe-deployment deletion-operation policy, must be run manually by an operator." The agent can propose destructive changes, but it won't run them itself. That's the right default.
Follow-up chat
I worked through the workshop's suggested questions in chat.
"What specific ports were targeted?" It returned the full scan list (1433, 139, 21, 5432, 3306, 9200, 23, 53, 135, 3389, 8080, 443), then pointed out that the attack that actually succeeded was only on port 80 (2,764 packets) and port 22 (57 packets, "too small to be a brute-force attempt"). It explicitly ruled out SYN flood and SSH brute force as the attack type.
"How can I prevent similar attacks?" Lock down the security group, put WAF with rate limiting in front of 80/443, enable GuardDuty (which was blocked by an SCP in this account), and fix the alarm's false-positive rate.
"Should I implement AWS WAF?" It said yes, and then added the catch: "WAF attaches to CloudFront, ALB, or API Gateway, not directly to an EC2 instance… That's a bigger architectural change than just 'turning on WAF'." It suggested a path (ALB → WAF rate-based rule → security group only from the ALB → managed rule groups) and asked for confirmation before building anything.
Takeaways
- 🕵️ It turned Flow Logs into a clear attack story: the source IP, the phases, which ports were only probed and which were actually hit
- 🔕 It found an alarm-quality problem I hadn't asked about. 3 of 4 firings were S3 noise.
- 🧪 It kept ruled-out hypotheses visible, which is useful for audits and postmortems
- 🛑 Destructive changes are blocked from auto-execution by policy
- 🧱 It noticed the misconfiguration came from the IaC template, and the fix targets the template
Finally, in Part 4: an RDS MySQL instance that runs out of connections, and a case where the agent's chat answers contradicted its own mitigation plan.
Top comments (2)
The scanner/flooder scenario is a nice stress test for the difference between detection and comprehension. Flow logs are perfect for the first — the agent can say "portscan pattern, SYN without follow-up ACK, 200+ sources" — but they structurally can't answer the question an incident lead actually asks: what were they looking for? The payload lives in packet bodies that flow logs never see. An agent that confidently narrates intent from five-tuple data is doing literature, not forensics, and the confident tone is the same failure mode you flagged in Part 2 with the ASG.
My two cents from running collection-layer pipelines: the eval design matters more than the model here. If you hand the agent the flow logs and the ground truth in the same exercise, it will pattern-match to the answer you planted and you'll both feel great. The honest harness is one-way: agent sees only the raw export, writes its finding card first, and only then does the human reveal what was injected. Publishing the failure cases (findings the agent asserted that the raw lines don't support) is the part of the series that'll outlive the product news — that's the receipt layer nobody else writes.
Curious what Part 4's RDS case will show: connection-exhaustion is one of the few failure classes where flow logs alone can mislead even a perfect human analyst, because SYN-SENT buildup looks identical under scan flood and pool exhaustion. If the agent separates those two on flow data alone, that's a genuinely useful result worth the writeup.
Thank you for such valuable comments! I am currently just beginning to implement Agent DevOps capabilities, with plans to test the system in a real-world environment later on. At this stage, I would recommend using it solely as an auxiliary tool for gathering incoming data and processing logs. The information obtained should be passed on to a qualified specialist capable of drawing well-founded conclusions. I'm going to inform you about my future steps.