Forbidden Configuration — Operationalizing Policy as Code to guard against misconfigurations, vulnerabilities, and more on an evolving planet
A few weeks ago I added a new AI agent runtime to an infrastructure-as-code template, ran a trusted security scanner over it, and got a clean result. No misconfigurations, no secrets, nothing. The scanner was working perfectly. The problem was that it had never heard of the resource I’d just deployed.
That gap between a tool reporting “clean” and a deployment actually being safe is the thing most policy-as-code programs aren’t built to handle. We tend to treat policy-as-code as a setup task: drop a scanner into CI, write a few rules, watch for the green check. That’s fine until the pace of change outruns the tool’s knowledge, which, in the age of AI services shipping monthly, is now the normal operating condition rather than the exception.
The teams that stay ahead aren’t the ones who picked the best scanner. They’re the ones who run IaC and policy-as-code as a continuously operated practice with intake gates, layered enforcement, managed coverage, and feedback loops. This post is about that operating model, grounded in a concrete example I hit in practice.
This is one example I encountered recently. Amazon Bedrock AgentCore is a managed runtime for deploying AI agents. It has first-class CloudFormation support AWS::BedrockAgentCore::Runtime and more. It is also recent enough that security scanners don’t all model it yet, which makes it a perfect stand-in for “the new thing we all adopted this quarter.”
Here’s an insecure runtime in a CloudFormation template:
Resources:
InsecureAgentRuntime:
Type: AWS::BedrockAgentCore::Runtime
Properties:
AgentRuntimeName: insecure\_agent
NetworkConfiguration:
NetworkMode: PUBLIC # not network-isolated
EnvironmentVariables:
OPENAI\_API\_KEY: sk-plaintext-do-not-do-this # plaintext secret
DB\_PASSWORD: hunter2
# no AuthorizerConfiguration -> unauthenticated inbound
# no RoleArn -> no explicit execution role
Four things are wrong: no inbound authorization, public networking, no execution role, and plaintext secrets in the environment. Now watch what a default scan does with it.
trivy config agentcore-insecure.yaml — scanners misconfig
Trivy reports nothing. Not because the template is safe, but because Trivy normalizes CloudFormation into its built-in cloud model a representation that only contains services it ships a provider for. AgentCore isn’t in that model, so the resource effectively vanishes before any rule runs. A clean scan here means “unknown,” not “safe.” If your process stops at the green check, you just shipped all four problems to production.Everything below is about building a process so that doesn’t happen and so it keeps not happening as the next new service arrives.
Before the processes, four ideas that make them hold together:
- Fail closed on the unknown. A resource type no policy evaluated is a finding, not a pass. Silence is the most dangerous output a scanner produces.
- Portability over tool loyalty. Keep policy intent movable across engines. Coverage gaps in one tool shouldn’t become gaps in your program.
- Policy lives with the infrastructure it governs. Versioned, reviewed, in the same repo and lifecycle as the IaC itself.
- Coverage is a managed asset. Someone owns “what we scan, what we skip, and why,” and reviews it on a cadence.
Process One: Adoption intake: a gate for every new technology
The intake gate is the single highest-leverage process, because it’s the one that would have caught the AgentCore gap. The trigger is simple: any new service, resource type, model, or framework entering your IaC has to pass a checklist before it reaches production.
Some of the items on the new-tech intake checklist might be:
-Does our scanner actually model this resource type?
-If not, who writes the baseline policy, and by when?
-What are the non-negotiable controls for this class of tech?
(identity / network / data handling / logging)
-Is a fail-closed rule registered so unknown instances alert, not pass?
-Example pass/fail IaC committed alongside the policy?
You check whether the tool sees the resource before you trust its verdict. For Trivy, you can confirm the blindspot directly: a custom check scoped to the normalized cloud model only ever sees the services Trivy models (in my case config, iam, rds) AgentCore is simply absent from the input. That five-minute check is the difference between knowing you have a gap and discovering it in an incident review.
Wire the trigger to something that can’t be skipped, like a PR label, a new provider block, a module-registry entry. You want intake that runs automatically when new tech appears rather than relying on someone remembering.
Process Two: Authoring and review: treat policy like product code
Once intake flags that AgentCore needs coverage, you write the policy. The point of operationalizing is that this isn’t ad hoc: a policy is proposed in a PR, tested against known-good and known-bad fixtures, peer reviewed, and merged exactly like application code.
With Trivy, the catch is that you can’t use its normalized model (the resource isn’t there), so you scan the template as raw data instead and write the rule against input.Resources directly, at least at this point in time. I wrote and tested policies for AgentCore using cfn-guard, OPA, Trivy, and Checkov. See https://github.com/mgbec/AgentCore-Policy-as-Code--multiple-options/blob/main/TOOL_COMPARISON.md for more details.
Three tools, same four controls, same verdict on the same template. The reason to express the intent portably is that your coverage no longer depends on any one vendor’s release schedule. When cfn-guard reads the resource natively and Trivy needs a flag, that’s data for Process 4, not a crisis.
Whichever engine you choose, the policy ships with example pass/fail templates so the rule is provably correct, and the secure version produces zero findings. We really need a human in the loop here. Automation enforces the controls you already understand; reviewers reason about the novel resource type and the risks the rules don’t cover yet. For a brand-new agent runtime, that human judgment is most of the value. See, we aren’t useless, yet.
Process Three: Enforcement across the lifecycle
Map each control to where it runs, and expect many to run in more than one place. Some of the details we’ll need to manage are:
-Stage
-What runs
-What it catches
-Author time
-AWS Config rules (or similar) for evaluating live resources. We want to catch out-of-band changes and configuration drift.
Decide fail-open versus fail-closed per stage deliberately, and write the choice down. For a new, poorly understood resource class, fail closed.
Process Four: Coverage and drift management
Maintain a living inventory: resource types in use versus resource types with policy coverage. If a resource type goes on the list as “covered via workaround”, we can revisit this type and review all coverage on a predetermined schedule. Scanners add providers over time; the raw-JSON Trivy workaround you wrote this quarter may be replaceable by native support next quarter, at which point you retire the hack and simplify. Pin your tool, model, and provider versions, and treat a scanner upgrade as a change that can alter results because a new Trivy release that models AgentCore natively would change what your existing check sees.
Drift detection closes the loop: periodically reconcile what’s actually running against the IaC and policies that were supposed to govern it. A resource that exists in the account but matches no policy is the same silent-pass risk as an unscanned template, just discovered later.
Process Five: Feedback loops
Treat findings as data. Track two numbers in particular: silent passes (deployments containing resource types no policy evaluated) and false positives(rules that fire on compliant config). The first tells you where coverage is missing; the second tells you where rules need tuning before people start ignoring them.
Every miss becomes an artifact. An incident involving an unauthenticated agent runtime becomes a new policy and a tightened intake trigger so the next unmodeled service is caught at the gate. Run the review cadence against the ecosystem’s pace, not the audit calendar when the tech moves, re-check the controls. Share rules internally as a policy library so a team adopting AgentCore next inherits your coverage instead of rediscovering the blind spot.
Getting Our Shield Base Operational
Ownership and exceptions
Oh dang, someone has to be in charge. Assign names to five things: the policy library, the intake gate, the coverage inventory, the enforcement pipeline, and the exceptions process. Shared ownership with no owner is how coverage silently rots. Exceptions should be a managed process; time-boxed, documented, auto-expiring. “We’re shipping this AgentCore runtime with public networking for a two-week pilot” is a legitimate exception as a tracked item with an end date, not a quietly disabled rule. Your auditors or the compliance team will thank you. Well, they should anyway.
Metrics that prove the practice works
**Coverage ratio** resource types governed / resource types in use.
**Time-to-policy** new service adopted → first enforced policy.
**Silent-pass rate** deployments with resource types no policy evaluated.
**Mean time to remediate** a policy failure.
**Exception count and age.**
**Drift incidents** caught at runtime versus missed until audit.
Silent-pass rate is the one to watch first. It’s the direct measure of the exact failure I started with; a clean scan that didn’t actually look at anything.
90-day rollout
**Days 0–30:** Inventory current coverage and find your silent passes. Run the “does our scanner model this?” check against your real resource types. Stand up the intake checklist. Pick a portable policy approach (Conftest/OPA travels well; cfn-guard if you’re CloudFormation-only).
**Days 30–60:** Move policies into the repo with pass/fail fixtures. Add PR gating. Define fail-closed defaults. Assign the five owners.
**Days 60–90:** Add deploy-time (CloudFormation Hooks) and runtime (AWS Config)enforcement. Start the coverage-review cadence. Publish the first metrics.
Start small and make it repeatable. Add coverage every sprint rather than overwhelming yourself.
Lesson Learned
The scanner that told me my AgentCore template was clean wasn’t lying. It just didn’t know what it was looking at and nothing in the process forced the question of whether it should have. The tools are commodities and they’re improving fast; the operating model around them is the durable advantage.
As AI increasingly generates and deploys infrastructure, the compliance that keeps up is the kind that’s continuously operated: intake gates that assume nothing, enforcement at every stage, coverage tracked as an asset, and feedback loops that turn every miss into a rule. Measure your silent-pass rate this week. It’s the fastest signal of whether you have a practice or just a green check.
— -
*Tool versions and service details reflect the time of writing; verify current scanner coverage and resource schemas before relying on any specific behavior.*
Top comments (0)