✓ Human-authored analysis; AI used for formatting and proofreading.
NCC Group built SadCloud to test cloud security tools. It deploys intentionally vulnerable AWS infrastructure with 84 misconfigurations across 22 services so you can measure exactly what your scanner catches and what it misses.
We pointed a static analyzer at a SadCloud deployment. It does not use credentials or make API calls against the live environment. Just a JSON snapshot of the AWS account state, evaluated on a laptop.
57 findings. 12 services. 1 compound risk chain. Zero false positives.
This post walks through the results, including what we missed and why.
The setup
SadCloud deploys resources via Terraform. Each service module has flags that enable specific misconfigurations. These are things like CloudTrail with no log validation, KMS keys with no rotation, IAM roles with wildcard trust policies, security groups open to the internet.
We enabled everything: CloudTrail, IAM, KMS, EC2, EBS, ELBv2, CloudWatch, AWS Config, CloudFormation, OpenSearch, S3, and SES. Terraform created the resources in a dedicated AWS account. We captured the state using standard AWS CLI calls (describe-trails, list-users, list-keys, describe-instances, etc.), saved the output as JSON files, and destroyed the infrastructure.
Total time infrastructure was live: under 2 hours. Total cost: under $10.
The analysis ran against the JSON files after the infrastructure was already gone.
The progression
We got to 57 in four iterations, each one exposing a gap that we fixed before the next run.
| Iteration | Services | Assets | Findings | What changed |
|---|---|---|---|---|
| 1 | 3 | 9 | 12 | CloudTrail + KMS + IAM baseline |
| 2 | 3 | 11 | 19 | Fixed Terraform flag conflict, authored 3 new controls |
| 3 | 7 | 21 | 26 | Added S3, EBS, EC2 security groups |
| 4 | 12 | 35 | 57 | Fixed property path mismatches, added remaining services |
Every iteration followed the same cycle: run the analyzer, read the gaps, determine whether the gap is a missing observation, a missing control, or a property path mismatch, fix it, re-run.
What fired
57 findings across 28 controls. Here's the breakdown by service and severity:
Critical (3 findings)
The three highest-severity findings represent the most dangerous misconfigurations in the deployment:
An IAM role with Principal: * in its trust policy. Any AWS account in the world can assume it. A group with full administrator access attached. And a CloudTrail configuration that's single-region only, meaning activity in other regions goes unlogged.
High (22 findings)
CloudTrail with no log file validation and no S3 data event logging. A KMS key with an open key policy (Principal: *). An EC2 instance with a public IP, IMDSv2 not enforced, and secrets in user data. An ALB with an HTTP listener (no TLS). An OpenSearch domain with an open access policy. Security groups with high ports and restricted ports open to 0.0.0.0/0. S3 buckets with public prefixes and no object ownership controls. Shadow IAM policies (inline policies that duplicate or conflict with managed policies).
Each of these maps to a SadCloud misconfiguration flag. The analyzer identified the correct asset, the correct property, and produced a remediation specific to the finding.
Medium (24 findings)
Password policy weaknesses (short minimum length, no complexity requirements, no reuse prevention). KMS key rotation disabled. S3 with no versioning and no access logging. Inline IAM policies on users and groups. CloudFormation stack with termination protection disabled. ALB with no deletion protection and no access logging. OpenSearch with no logging. Seven security groups with 0.0.0.0/0 CIDR blocks and seven with overly broad CIDR ranges.
Low and Info (8 findings)
Empty IAM groups (members assigned to a group that has a policy, but the group has no members or vice versa). Incomplete CloudTrail and CloudFormation configurations. S3 governance gaps.
The compound chain
The most important finding:
[critical] Chain: cloudwatch_detection_broken
Detection pipeline broken at the metric-filter, alarm, or alarm-action
layer. The alarm exists in the console with the expected name, but the
path from log event to operator notification is severed.
Failing: CTL.CLOUDWATCH.ALARM.NOACTION.001
Fix any of: CTL.CLOUDWATCH.GHOST.METRICFILTER.LOGGROUP.001,
CTL.CLOUDWATCH.ALARM.DISABLED.001
Score: 75.0
Stages: detection_evasion
SadCloud deploys a CloudWatch alarm with no actions configured. On its own, that's a medium-severity finding with an alarm that doesn't notify anyone. Every scanner catches it.
The compound chain links this to the detection pipeline: CloudTrail logs → CloudWatch metric filters → CloudWatch alarms → notification actions. If any stage in that pipeline is broken, the entire detection path is severed. The team believes monitoring is in place because the alarms exist in the console with the expected names. The reality is that an attacker's activity flows through CloudTrail, hits the metric filter, triggers the alarm and nothing happens. No page, Slack message or incident.
This is the class of finding that individual-check scanners structurally cannot produce. The alarm-without-actions check exists in every tool. The insight that this specific alarm is the only detection path for a specific class of attacker activity, and that its failure creates a detection blind spot that requires evaluating the composition, not the setting.
What we missed
S3 public bucket policies (3 misconfigs): The AWS account has BucketOwnerEnforced and BlockPublicPolicy enabled at the account level. SadCloud's S3 module was written for older AWS defaults that allowed public bucket ACLs and policies. The module's public-bucket resources fail to deploy. This isn't a Stave gap. The misconfigurations don't exist in the deployed infrastructure because AWS prevents them. We'll revisit when SadCloud updates its S3 module for current AWS defaults.
9 services with no deployed resources: ACM, ECR, EKS, ELB classic, Lightsail, RDS, Redshift, SNS, and SQS modules either don't create resources by default or were blocked by sandbox restrictions. Nothing to scan means nothing to find. These services have controls in the catalog. They'll be validated when infrastructure exists.
SES (2 misconfigs): Domain identity and DKIM are deployed but no matching controls exist in the catalog yet. Noted as a gap for future work.
4 Prowler parity checks: The coverage posture shows 44 of 47 Prowler IAM checks and 20 of 21 S3 checks covered. The 4 uncovered checks are prescriptive presence-checks (does a specific named role exist, is a root account feature enabled, are S3 event notifications turned on). These are compliance checkbox items, not risk-based invariants.
The numbers
Infrastructure deployed: Full SadCloud (all_findings = true)
Services with resources: 12 of 22
Total misconfigurations: ~60 deployable (of 84 catalog)
Findings produced: 57
False positives: 0
Compound chains: 1
Controls fired: 28
Assets evaluated: 35
Attack surface: 28
Analysis time: 4.7 seconds
Credentials required: 0
Network access required: none
What this validates
SadCloud exists to test security tools. NCC Group uses it to benchmark ScoutSuite, Prowler, CloudMapper, CloudSploit, and tfsec. The published sample reports show what each tool finds against the same corpus.
Running a static analyzer against the same corpus and getting 57 findings with zero false positives from a JSON snapshot validates three things:
First, the observation contract works. Raw AWS CLI output transforms into a structured format that the analyzer's controls can evaluate. The property paths match what the controls expect. When they didn't match (iterations 2 and 3), fixing the paths immediately produced the missing findings.
Second, the control catalog covers real misconfigurations. Every finding maps to an actual SadCloud flag. The controls weren't written for SadCloud. They were written from incident reports, compliance frameworks, and AWS security best practices. They fired against SadCloud because the misconfigurations are real patterns that appear in production environments.
Third, compound chain detection works on third-party infrastructure. The cloudwatch_detection_broken chain wasn't authored for SadCloud. It was authored from the observation that detection pipelines fail silently when any stage is broken. SadCloud happened to deploy that failure mode with an alarm with no actions and the chain caught it.
What's next
This is the third vendor lab completed. Datadog Pathfinding Labs validated IAM privilege escalation detection. Bishop Fox IAM Vulnerable validated 30 escalation paths. NCC Group SadCloud validated broad CSPM coverage across 12 services.
The hardest test is still ahead: Rhino Security's CloudGoat, which deploys multi-service compound attack chains where the vulnerability isn't in any single resource but in the path between them. That's where compound chain detection either proves out at scale or doesn't.
The tool is Stave. Static analysis. No credentials. No agents. Files in, findings out.
Top comments (1)
tr.ee/dev-to