✓ Human-authored analysis; AI used for formatting and proofreading.
SadCloud is an open-source Terraform project by NCC Group that deploys misconfigured AWS resources. Security teams use it to test their tooling: if your scanner can't find the misconfigurations in SadCloud, it can't find them in production.
We pointed Stave at SadCloud. The numbers tell a story about what happens when you move from per-resource scanning to deterministic evaluation with explicit data-gap tracking.
The raw numbers
| Metric | SadCloud (NCC Group) |
|---|---|
| Assets evaluated | 35 |
| Attack surface | 28 |
| Violations | 51 |
| Indeterminate | 76 |
A per-resource scanner (Prowler, ScoutSuite, Checkov) would show findings and move on. Stave shows 51 violations AND 76 controls it could not evaluate because the observation data was missing. That second number is the one scanners never report.
What the 51 violations cover
The violations cluster across five AWS service areas:
CloudTrail (5 findings). Trail not enabled, log validation off, global events not captured, S3 data events not logged, validation missing. These are the audit foundation — without them, every other security control operates blind.
IAM (5 findings). A superuser with admin policy, two shadow admin policies, a wildcard trust policy, and inline policies on users, roles, and groups. The trust wildcard (TRUST.WILDCARD.001) is critical — any AWS principal can assume that role.
EC2 (4 findings). IMDSv1 enabled, public instance, unrestricted security group ports, secrets in user data. The IMDSv1 + public IP combination is the Capital One breach pattern.
S3 (5 findings). Public access policy, missing encryption, no access logging, object ownership misconfigured, governance controls absent.
Security Groups (15 findings). Broad CIDR ranges, unrestricted ingress, high port exposure, ICMP open, wide port ranges. These are the network perimeter — individually medium severity, collectively they mean the VPC has no meaningful network segmentation.
Other (17 findings). CloudWatch alarm with no action, AWS Config recorder not capturing all resources, KMS key policy issues, CloudFormation incomplete stacks, password policy gaps.
The 76 indeterminate controls
This is the number that matters most for understanding the gap between what Stave evaluates and what the observation data contains.
76 controls could not produce a verdict because the SadCloud observation snapshot doesn't include the fields they check. These aren't false negatives — they're explicit "I don't know" verdicts. Examples:
- CloudTrail alarms (9 controls): Does an alarm fire when someone deletes a trail? Stops logging? Modifies event selectors? The snapshot doesn't include CloudWatch alarm configurations for these API calls.
- IAM boundaries (9 controls): Do permission boundaries constrain role creation? Does a boundary require boundaries on created roles? The snapshot doesn't include boundary policy details.
- Detection blindness (6 controls): Are Amplify, AppRunner, Cloud9, EVS, Lightsail, MWAA data events logged? The snapshot doesn't include per-service event selector configuration.
- EC2 account defaults (3 controls): Is IMDSv2 the account default? Are AMI and snapshot public sharing blocked at the account level? Account-level settings aren't in the snapshot.
Each indeterminate finding names the missing field: audit.cloudtrail.alarm_delete_trail, identity.role.has_permissions_boundary, compute.instance_metadata_defaults.http_tokens. The operator knows exactly what to collect to convert each "I don't know" into a pass or fail.
A scanner that lacks these fields silently skips the check. Stave fails loud: 76 controls could not run. Here's why. Here's what to collect.
Running this yourself
The SadCloud snapshot is in the CTF fixture repository. To reproduce:
stave apply --observations ~/work/ctf/05-raw-data/observations/nccgroup/
Add --format json for machine-readable output. Add --eval-time for deterministic timestamps.
Same snapshot, same catalog, same findings, every time. The evaluation is deterministic — there's no model, no heuristic, no probabilistic scoring. The findings are a logical consequence of the control predicates applied to the observation data. Run it on a different machine and the output is byte-identical.
What this means for your environment
SadCloud is deliberately broken. Your environment probably isn't. The numbers will be different — fewer violations, but likely a similar ratio of indeterminate controls, because most observation collection pipelines don't capture every field Stave's catalog checks.
The indeterminate count is the roadmap. Each one names a field. Collect that field in your next snapshot, and the control either passes (the configuration is correct) or fires (it isn't). Either way, you know. The worst state is the one scanners leave you in: silence where the check should be.
51 violations is what's broken. 76 indeterminate is what you can't see yet. Same data. Two outputs. Two actions.
The data is in the lab. The evaluation is deterministic. Run it and see what your tools miss.
Top comments (0)