✓ Human-authored analysis; AI used for formatting and proofreading.
In JavaScript, when you want to compare frameworks, you go to TodoMVC. Same app, every framework. You compare React and Vue and Svelte against identical requirements. The differences such as bundle size, rendering approach, state management, verbosity reveal themselves on uniform ground.
Cloud security has never had this.
Every vendor demos against their own scenario. Prowler shows you their best findings. Wiz shows you their graph. AWS Config shows you its rules. Nobody runs them all against the same deliberately misconfigured environment and publishes what each one found and what each one missed.
If you're evaluating cloud security tools such as during a POC, a vendor comparison, or an internal assessment, you're comparing demos, not data.
This should change.
The problem with vendor-controlled comparisons
When a vendor runs a demo, they choose the environment. They choose which misconfigurations to show. They choose the narrative. Every tool looks good when the author picks the example.
A POC is better. You run the tool in your own environment. But your environment has whatever it has. You can't control what misconfigurations exist, so you can't tell the difference between "the tool didn't find it" and "it doesn't exist to find." If Tool A reports 200 findings and Tool B reports 150, is Tool A better or just noisier?
What you need is a controlled environment. Known misconfigurations. Documented ground truth. A scorecard. Run every candidate tool against the same environment, fill in FOUND or MISSED for each ground truth item, and compare.
TodoMVC gave that for JavaScript ecosystem. Cloud security doesn't have an equivalent.
What we built
A Terraform-deployed AWS environment containing 30 documented misconfigurations across 8 services, plus 5 multi-resource attack paths. Each misconfiguration has a unique ID, a description, a severity rating, and manual verification steps so you can confirm it's real by checking the AWS console yourself.
The environment deploys in about 10 minutes and costs roughly $2/day to run. You tear it down when you're done evaluating.
The misconfigurations cannot be argued. Not "this IAM role might be overprivileged depending on your risk threshold." Rather: "this S3 bucket allows public read access." Binary. Objectively wrong. Any tool claiming to cover S3 should find it.
The 5 compound paths are the interesting part. These connect misconfigurations across services into attack chains. A publicly reachable instance with an overbroad IAM role that can read from an unencrypted S3 bucket isn't three independent findings, it's one attack path. Some tools find the individual resources. Fewer tools connect them into the path. The scorecard distinguishes between the two.
How it works
Deploy the environment. Run your candidate tools against it. Fill in the scorecard.
The scorecard is a CSV with every ground truth ID pre-filled. For each tool, you mark FOUND, MISSED, PARTIAL, or N/A. At the bottom: an atomic score (out of 30) and a compound score (out of 5).
No software to install beyond the tools you're evaluating. No vendor to contact or account to create.
We include one completed scorecard as a reference. Stave's results against the environment so you can see what a filled-in scorecard looks like and verify the environment is working before running your own tools. It's there as an example, not a claim.
What's in the ground truth
Eight AWS services: S3, IAM, CloudTrail, KMS, EC2, ELBv2, OpenSearch, and Config. The misconfigurations span:
Public exposure such as resources accessible from the internet that shouldn't be. Encryption gaps such as data at rest or in transit without encryption. Logging deficiencies such as audit trails that are disabled or incomplete. Identity issues such as overprivileged roles, unused credentials, missing MFA conditions. Network configuration such as security groups with overly permissive rules.
The compound paths combine these into realistic attack chains. Each path documents the entry point, the pivot, and the target. Each step references a specific ground truth atomic finding, so you can see exactly which individual misconfigurations compose into the chain.
Why compound paths matter
The CSA Top Threats to Cloud Computing 2026 report says: "attackers exploit the seams between products." They use a compromised third-party API to pivot to an overprivileged identity to exfiltrate data from a misconfigured bucket. No single product sees the whole chain.
Most security tools check one resource at a time. Security group: ✓ checked. IAM role: ✓ checked. S3 bucket: ✓ checked. Three passing checks. One exploitable path.
The compound paths in the eval kit test whether a tool can connect the checks. Finding the individual misconfigurations gets you partial credit. Connecting them into the attack path gets you the full score. The scorecard makes this visible.
Standarized environment
This is a standardized environment for running your own evaluations. You deploy it, run your tools, see what they find. The ground truth is public. Your results are yours.
This is not a public ranking, a vendor-shaming exercise, or a definitive verdict on which tool is best. Different tools have different strengths. Some focus on compliance mapping. Some focus on runtime detection. Some focus on code scanning. The eval kit tests configuration posture such as one dimension of a multi-dimensional problem. A tool that scores 25/30 on atomic findings might be exactly right for your use case, and a tool that scores 30/30 might miss what matters to you.
This is not a replacement for evaluating tools in your own environment. The eval kit gives you a controlled comparison. Your production environment gives you the real one. Use both.
Contributing results
If you run the eval kit during a real evaluation and want to share your results, you can submit a completed scorecard to the repo. This is optional. The kit works just as well if you never share anything.
Over time, if enough people share results, a picture emerges. Not a ranking but a coverage map. Which tools find which categories of misconfigurations. Where the compound path detection drops off. What the industry is good at and where the gaps are located.
That picture is missing in security industry now. Every organization runs POCs but because nobody runs them on the same ground.
Try it
The repo is open. The environment is Terraform. The ground truth is documented. The scorecard is a CSV.
If you're evaluating cloud security tools this quarter, run them against the same environment and see what you see.
The eval kit is open source and vendor-neutral. Stave's results are included as a reference implementation. The first completed scorecard, not a claim of superiority. The environment, ground truth, and scoring template stand on their own regardless of which tools you choose to evaluate.
Top comments (0)