Originally published on kuryzhev.cloud
The 3 a.m. Bedrock Bill
Bedrock IAM least privilege sounds like a compliance checkbox until a CI job proves otherwise. A common setup: a pull-request summarizer or changelog bot runs on every push, calling bedrock:InvokeModel through a shared IAM role someone created months ago "to get the demo working." Nobody has touched the policy since.
The role's trust and permission policy use "Resource": "*" and "Action": "bedrock:*". That's fine for a demo. It becomes a liability the day a malformed prompt or a broken rate-limit backoff sends the pipeline into a retry loop — hundreds of invocations in a short window, often against the most expensive model tier the account has access to, because nothing in the policy stops the call from reaching for it.
There's no per-model spending guardrail and no CloudTrail alert wired to Bedrock data events. The first signal isn't a failed pipeline step — the job might even report success on retry. The first signal is an AWS Budgets email a day or two later, or the invoice itself. By then the retry loop has already run its course, and reconstructing what happened means digging through CloudTrail logs that, in a lot of accounts, were never configured to capture InvokeModel calls at all.
This isn't an edge case someone got unlucky with. It's the predictable outcome of a permission model scoped for "make it work" and never revisited for "safe to leave running unattended."
Why Bedrock IAM Ends Up Over-Permissioned
This pattern repeats across teams for structural reasons, not because any one engineer got careless. The Bedrock quickstart documentation and the console's "test invoke" wizard both default to broad bedrock:* on Resource: "*". Scoping to a specific model ARN or inference profile is an extra step most getting-started guides skip entirely, so the copy-paste path leads straight to a wildcard.
CI and bot roles also tend to get provisioned once and reused indefinitely. A role created for one project quietly becomes "the AI role" for three others. As new models or regions get enabled on the account, nobody circles back to narrow the original policy — permission creep happens through convenience, not through any single bad decision.
The third factor is credential hygiene. Many pipelines still authenticate to AWS with a long-lived IAM user's access key pair stored as a CI secret, instead of short-lived OIDC federation. A static key has no built-in expiry and no binding to a specific repository, branch, or job. If that key leaks — through a misconfigured fork PR, a logging bug, or a compromised dependency — the blast radius is whatever the role can do, indefinitely, until someone notices and rotates it by hand.
None of these three shortcuts looks dangerous on its own. Stacked together they produce a role with wildcard model access, no regional boundary, and a credential that never expires — exactly the setup a runaway retry loop needs to turn into a five-figure bill.
Scoping the Policy — Model, Region, and Identity
The fix has three parts: scope the actions and resources explicitly, constrain region and identity with condition keys, and replace static keys with short-lived OIDC-federated credentials.
Start by restricting the Bedrock actions to only what the bot needs — typically bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream — and point Resource at an explicit foundation-model ARN instead of a wildcard. Model ARNs follow the pattern arn:aws:bedrock:<region>::foundation-model/<model-id>. If the application uses cross-region inference profiles rather than direct model invocation, those carry a different ARN shape (arn:aws:bedrock:<region>:<account-id>:inference-profile/<profile-id>) — a policy written only for direct model ARNs will silently deny, or fall back in unexpected ways, if the app later switches to profiles. Verify with the Bedrock inference profile documentation which ARN format the SDK call actually targets before you write the policy.
Layer in condition keys: aws:RequestedRegion blocks cross-region invocation when Bedrock is enabled in more than one region on the account, and aws:PrincipalTag binds the policy to a specifically tagged bot identity rather than any role that happens to look similar. Pair this with GitHub or GitLab OIDC federation so the CI job assumes a role scoped to a specific repo, branch, and job — which removes long-lived keys from CI secrets entirely.
The Terraform below defines a GitHub Actions bot role trusted only for pushes to main, with an inline policy scoped to one model family, one region, and one principal tag.
# Terraform: least-privilege Bedrock invoke role for a GitHub Actions bot
resource "aws_iam_role" "bedrock_ci_bot" {
name = "ci-bedrock-summarizer-bot"
# OIDC federation trust — scoped to one repo, one branch, one job
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = { Federated = aws_iam_openid_connect_provider.github.arn }
Action = "sts:AssumeRoleWithWebIdentity"
Condition = {
StringEquals = {
"token.actions.githubusercontent.com:sub" = "repo:org/repo:ref:refs/heads/main"
}
}
}]
})
tags = { Purpose = "bedrock-ci" }
}
resource "aws_iam_role_policy" "bedrock_invoke_scoped" {
name = "bedrock-invoke-scoped"
role = aws_iam_role.bedrock_ci_bot.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Action = [
"bedrock:InvokeModel",
"bedrock:InvokeModelWithResponseStream"
]
# Model-family scoped ARN, not an account-wide wildcard
Resource = "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-*"
# Both keys belong in ONE StringEquals block — a second block with
# the same operator name silently overwrites the first in jsonencode
Condition = {
StringEquals = {
"aws:RequestedRegion" = "us-east-1"
"aws:PrincipalTag/Purpose" = "bedrock-ci"
}
}
}]
})
}
Before attaching a policy like this to a production role, dry-run it. aws iam simulate-principal-policy checks whether specific actions and resources resolve to allow or deny without granting anything, which makes it usable as a policy-as-code test step in CI.
# Dry-run the policy before attaching it — catches accidental over-scope
aws iam simulate-principal-policy \
--policy-source-arn arn:aws:iam::123456789012:role/ci-bedrock-summarizer-bot \
--action-names bedrock:InvokeModel bedrock:ListFoundationModels \
--resource-arns "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-*" \
--region us-east-1
# Expect: InvokeModel -> allowed, ListFoundationModels -> implicitDeny
# If ListFoundationModels comes back "allowed," the policy is broader than intended.
Watch out for one specific overreach: don't grant model-access management actions like bedrock:PutFoundationModelEntitlement to CI roles. Those control which models an account can reach at all, and belong to human admin roles, not automated pipelines — a bot role that can grant itself access to new models effectively has no ceiling left to hit.
Prevention Checklist
Scoping one policy fixes one bot. Auditing every existing Bedrock role against a checklist is what stops the next retry-loop bill before it starts. Each item below closes a specific way permission creep or credential exposure re-enters an account over time.
- One role per bot or pipeline. A shared "AI automation" role across projects means a bug in one pipeline spends budget attributed to an unrelated one, and makes the blast radius of a leak impossible to scope.
- Deny-by-default on high-cost model tiers. Require an explicit allow plus a matching tag for anything above a defined cost class, so a retry loop can't silently escalate to the priciest model available.
- OIDC trust scoped to repo, branch, and job. No static access keys in CI secrets — a token leaked from a fork PR shouldn't be able to assume a production-scoped role.
-
CloudTrail data events enabled for
bedrock:InvokeModel*. Management events alone won't log invocation calls. Pair the log with a metric filter and alarm, not just a cost-based Budgets alert that fires days late. -
Quarterly policy simulation. Run IAM Access Analyzer or
aws iam simulate-principal-policyagainst every bot role to catch drift as new models or regions get enabled. - Permission boundaries on bot roles. A boundary caps what the role can ever be granted, even if someone later attaches a broader inline policy by mistake.
- Bedrock Guardrails as a second, independent layer. IAM scoping controls what the bot can call; Guardrails controls what it can generate. Scoped IAM alone does nothing against prompt injection or unsafe output, so treat the two controls as separate problems.
None of this requires exotic tooling — most items are a Terraform diff and a CloudTrail trail away. What actually prevents the 3 a.m. bill is treating Bedrock IAM least privilege as a recurring audit rather than a one-time setup step, especially in accounts where new models and regions get switched on faster than anyone remembers to revisit the roles that reach them. For broader patterns on locking down AWS automation identities, see the related guidance on kuryzhev.cloud.
Top comments (0)