AWS ships a policy simulator: paste a principal, an action, a resource, and it tells you whether that specific request would be allowed or denied. It runs the same evaluation logic AWS uses at runtime, including identity policies, resource policies, permission boundaries, and SCPs. Teams reach for it before shipping a policy change, treat a clean result as a green light, and move on.
That's a narrower guarantee than it looks like.
What the Simulator Is Actually Answering
The simulator answers one question at a time: given this exact principal, this exact action, this exact resource, what's the decision? That's genuinely useful — it's the same logic AWS applies at runtime, so a simulated "Allow" is a real allow, not a guess.
But "is this one request permitted" and "is this policy safe to ship" are different questions. A policy can pass every simulated check you think to run and still grant far more than you intended, because the simulator only evaluates the specific combinations you feed it.
Where the Gap Shows Up
You have to already know what to ask. The simulator doesn't enumerate what a principal can do — it confirms or denies whatever action/resource pair you type in. If you don't think to test iam:PassRole against every role in the account, or s3:GetObject against every bucket a wildcard resource might match, the simulator won't surface it for you. It's a lookup, not an audit.
It doesn't follow chains. Say the policy under test grants sts:AssumeRole on a single role. The simulator will correctly tell you that's allowed. It won't tell you what that assumed role can then do, because that's a second policy, evaluated by a second simulator call you'd have to think to make. Multiply that by every AssumeRole and PassRole statement in a real account and the number of calls needed to actually cover the graph gets large fast.
Wildcards pass silently. "Resource": "arn:aws:s3:::*/*" simulated against one specific bucket returns "Allow" — correctly. It doesn't flag that the same statement matches every other bucket in every other account the identity might reach. The simulator isn't wrong here; it's just not built to say "by the way, this also matches 40 other things you didn't ask about."
Boundaries and SCPs compute correctly, but only for the query you gave it. If a permission boundary caps an identity policy down to two actions, the simulator will show that correctly for those two actions. It won't proactively tell you the boundary is even attached, or that removing it would change the outcome, unless you already knew to check.
Why This Matters for Pre-Deploy Review
None of this makes the simulator unreliable. The decision it returns for a given query is exactly what AWS would do at runtime. The failure mode is procedural: teams run a handful of simulated checks against the changes they're consciously aware of, get clean results, and treat that as equivalent to "this policy is safe." It answers the questions you ask. It doesn't tell you which questions you forgot to ask.
That gap matters most on new roles and CI/CD-authored policies, where nobody has built the mental model of what the role should and shouldn't reach yet — there's no experienced reviewer's intuition to fill in the queries the simulator won't generate on its own.
What Closes the Gap
Closing it means generating the queries instead of hand-picking them: walking every AssumeRole and PassRole statement to see what's reachable two and three hops out, expanding wildcard resources against the actual resources in the account instead of one example, and checking whether a boundary or SCP is present at all before assuming the identity policy is the whole story. That's the same evaluation logic the simulator runs, just applied exhaustively instead of one query at a time.
This is where AI-Powered analysis earns its keep: Shieldly's free demo at shieldly.io/app/iam takes a pasted policy and runs that exhaustive pass automatically, surfacing over-privileged wildcards, chained AssumeRole/PassRole paths, and missing boundaries without requiring you to already know which combination to test. No signup needed for the demo.
For teams: the CLI (@shieldly/cli), GitHub Action, CDK Construct (@shieldly/cdk-guard), and VS Code extension run the same analysis in CI and in your editor, so the check happens on every policy change, not just the ones someone remembers to simulate by hand. Team plan adds scheduled auto-scanning across a full account.
The policy simulator tells you the truth about the exact question you asked. The risk lives in the questions a wildcard, a chain, or an unfamiliar new role never prompted you to ask in the first place.
Top comments (0)