<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Dmytro Bot</title>
    <description>The latest articles on DEV Community by Dmytro Bot (@777dimas).</description>
    <link>https://dev.to/777dimas</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4159798%2Fff39b092-ed62-49a1-ade3-7c23df0daff2.jpg</url>
      <title>DEV Community: Dmytro Bot</title>
      <link>https://dev.to/777dimas</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/777dimas"/>
    <language>en</language>
    <item>
      <title>Policy as code for Terraform: the gate is easy, the pipeline is not</title>
      <dc:creator>Dmytro Bot</dc:creator>
      <pubDate>Sat, 03 Oct 2026 00:00:00 +0000</pubDate>
      <link>https://dev.to/777dimas/policy-as-code-for-terraform-the-gate-is-easy-the-pipeline-is-not-378c</link>
      <guid>https://dev.to/777dimas/policy-as-code-for-terraform-the-gate-is-easy-the-pipeline-is-not-378c</guid>
      <description>&lt;p&gt;Most teams that run Terraform have had this meeting.&lt;/p&gt;

&lt;p&gt;Someone opened a pull request. The plan said &lt;code&gt;3 to destroy&lt;/code&gt;. Four hundred lines of diff scrolled past. Two people approved it. A queue that another service had been writing to for two years stopped existing.&lt;/p&gt;

&lt;p&gt;Nobody was careless. The warning was on the screen. It was on line 312, and people do not read line 312.&lt;/p&gt;

&lt;p&gt;Asking reviewers to try harder does not fix this. What fixes it is a program that reads the plan, spots the dangerous change, and stops the merge until a human says in writing that they meant it.&lt;/p&gt;

&lt;p&gt;We built that gate for about twenty Terraform root modules. This article explains how the mechanism works, and what went wrong while we set it up. The examples are simplified and the infrastructure is generic. The mistakes are real.&lt;/p&gt;

&lt;p&gt;One thing to clear up first, because people arrive at this problem holding the wrong tool. &lt;a href="https://dmytrobot.dev/blog/opa-gatekeeper/" rel="noopener noreferrer"&gt;OPA Gatekeeper&lt;/a&gt; will not help you: it is a Kubernetes admission controller, it lives in the cluster's API server, and it cannot read HCL or a Terraform plan. What it shares with the setup below is Rego, the policy language, which is worth something if your cluster policies are already written in it. The tool that reads a Terraform plan is Conftest, a small wrapper around OPA that takes a structured file, runs Rego against it, and returns an exit code.&lt;/p&gt;

&lt;h2&gt;
  
  
  How the gate works
&lt;/h2&gt;

&lt;p&gt;This part is short, and that is the point. The policy engine took two days. Everything after it took weeks.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Check the plan, not the code.&lt;/strong&gt; Conftest can read HCL directly, which needs no credentials and covers the whole repository at once. It is nearly useless for the rules that matter, because source code does not know what exists today. You delete a resource by removing its block, and a policy cannot match on an absence. State has the opposite problem: it describes what exists, not what is about to change. Only the plan carries the intent, with values resolved:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;terraform plan &lt;span class="nt"&gt;-out&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;tfplan
terraform show &lt;span class="nt"&gt;-json&lt;/span&gt; tfplan &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; tfplan.json

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That choice sets the boundary of the whole system, so say it out loud before anyone assumes otherwise.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;A policy only protects what is planned.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A module with no plan in CI has no rules applied to it, however many rules you have written.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What the plan file says.&lt;/strong&gt; Under &lt;code&gt;resource_changes&lt;/code&gt; there is one entry per resource, and &lt;code&gt;change.actions&lt;/code&gt; carries the decision:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"address"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"aws_s3_bucket.exports"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"aws_s3_bucket"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"change"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"actions"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"delete"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"before"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"bucket"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"..."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"force_destroy"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"after"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The vocabulary is small: &lt;code&gt;["no-op"]&lt;/code&gt; for most of the file, &lt;code&gt;["read"]&lt;/code&gt; for data sources, the obvious &lt;code&gt;["create"]&lt;/code&gt;, &lt;code&gt;["update"]&lt;/code&gt; and &lt;code&gt;["delete"]&lt;/code&gt;, and then the two that catch people out. &lt;code&gt;["delete", "create"]&lt;/code&gt; is a replacement. &lt;code&gt;["create", "delete"]&lt;/code&gt; is the same thing where the new resource is built first.&lt;/p&gt;

&lt;p&gt;A replacement destroys the old resource. For anything holding data that is the same loss as a deletion, but it reads differently in a diff and gets described in pull requests as a rename or a move. So test for membership:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;delete&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;resource&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;change&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;actions&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;not equality against &lt;code&gt;["delete"]&lt;/code&gt;. We found this because someone asked what happens when a key changes. In our case one system keys its resources by strings in a map, some of them old identifiers that look untidy. Cleaning one up would have destroyed years of history behind a diff that looks cosmetic.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;before&lt;/code&gt; and &lt;code&gt;after&lt;/code&gt; hold resolved values, which gives you the second kind of rule: not "is this being destroyed" but "is this setting moving the wrong way". A flag that takes a record out from behind a proxy. Deletion protection switched off. A retention period shortened. One word each, and none of them look urgent.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A rule, in full.&lt;/strong&gt; They are all about this size.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rego"&gt;&lt;code&gt;&lt;span class="ow"&gt;package&lt;/span&gt; &lt;span class="n"&gt;main&lt;/span&gt;

&lt;span class="ow"&gt;import&lt;/span&gt; &lt;span class="n"&gt;rego&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;v1&lt;/span&gt;

&lt;span class="n"&gt;stateful_types&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"aws_dynamodb_table"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"aws_s3_bucket"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"aws_sqs_queue"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;destroyed_stateful&lt;/span&gt; &lt;span class="n"&gt;contains&lt;/span&gt; &lt;span class="n"&gt;resource&lt;/span&gt; &lt;span class="n"&gt;if&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="ow"&gt;some&lt;/span&gt; &lt;span class="n"&gt;resource&lt;/span&gt; &lt;span class="n"&gt;in&lt;/span&gt; &lt;span class="n"&gt;input&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;resource_changes&lt;/span&gt;
    &lt;span class="n"&gt;resource&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;type&lt;/span&gt; &lt;span class="n"&gt;in&lt;/span&gt; &lt;span class="n"&gt;stateful_types&lt;/span&gt;
    &lt;span class="s2"&gt;"delete"&lt;/span&gt; &lt;span class="n"&gt;in&lt;/span&gt; &lt;span class="n"&gt;resource&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;change&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;actions&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;deny&lt;/span&gt; &lt;span class="n"&gt;contains&lt;/span&gt; &lt;span class="n"&gt;msg&lt;/span&gt; &lt;span class="n"&gt;if&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;destroy_allowed&lt;/span&gt;
    &lt;span class="ow"&gt;some&lt;/span&gt; &lt;span class="n"&gt;resource&lt;/span&gt; &lt;span class="n"&gt;in&lt;/span&gt; &lt;span class="n"&gt;destroyed_stateful&lt;/span&gt;
    &lt;span class="n"&gt;msg&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;sprintf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="s2"&gt;"%s (%s) will be destroyed by this plan. Add the allow-destroy label to the pull request if that is intended."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;resource&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;address&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;concat&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;", "&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;resource&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;change&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;actions&lt;/span&gt;&lt;span class="p"&gt;)],&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Conftest binds the plan to &lt;code&gt;input&lt;/code&gt;. &lt;code&gt;deny&lt;/code&gt; is a set, not a function: Rego tries every match and collects every message, so ten bad resources produce ten messages in one pass and no loop is written. An empty set is what passing looks like from the inside.&lt;/p&gt;

&lt;p&gt;If that syntax looks unfamiliar, you have read older material. OPA 1.0 made &lt;code&gt;if&lt;/code&gt; and &lt;code&gt;contains&lt;/code&gt; mandatory, so the pre-1.0 form still shown in most tutorials, &lt;code&gt;deny[msg] { ... }&lt;/code&gt; without either keyword, no longer compiles without the &lt;code&gt;--v0-compatible&lt;/code&gt; flag. Iteration with &lt;code&gt;[_]&lt;/code&gt; still works; &lt;code&gt;some ... in&lt;/code&gt; just reads better.&lt;/p&gt;

&lt;p&gt;The sentence inside &lt;code&gt;sprintf&lt;/code&gt; is the part anyone will actually read. "Policy violation: rule 7 failed" is an obstacle. A line that names the resource, says what is lost and gives the next step is help. We spend more time on the message than on the logic.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The escape hatch.&lt;/strong&gt; Conftest also has &lt;code&gt;warn&lt;/code&gt;, which reports without failing. For a week of watching that is fine; as a permanent setting it is a trap, because a warning on every pull request stops being read within a month.&lt;/p&gt;

&lt;p&gt;Deny by default instead, with a deliberate way to say yes. Ours is a label on the pull request, passed to the policy as data:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;destroy_allowed if {
    some label in data.pr.labels
    label == "allow-destroy"
}

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With no label the list is empty, &lt;code&gt;destroy_allowed&lt;/code&gt; is undefined, and the rule denies. The policy does not change between a blocked run and an approved one, which is an easy thing to explain to an auditor. The label also stays visible in the pull request list and in the history, so "when did we drop that table, and who agreed" is a search rather than an excavation.&lt;/p&gt;

&lt;p&gt;Use separate labels for separate decisions. We have three: one for destroying a resource, one for taking a DNS record out from behind the proxy, one for changing quality thresholds that apply to every project. Someone who thought about the first has not necessarily thought about the second, and a single allow-everything label turns three decisions into one habit.&lt;/p&gt;

&lt;h2&gt;
  
  
  Five things that went wrong
&lt;/h2&gt;

&lt;p&gt;None of these are about Rego. Every one of them showed up in a real run, after the rules were written and working, and each took longer to find than the rule it broke.&lt;/p&gt;

&lt;p&gt;Here is the whole pipeline, with the numbers marking where each one lives.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;  pull request opened, updated or labelled
                   │
                   ▼
  ┌─────────────────────────────────┐ ① the job holds credentials, and runs
  │ terraform init &amp;amp;&amp;amp; plan -out │ configuration written in the branch
  └────────────────┬────────────────┘
                   ▼
  ┌─────────────────────────────────┐
  │ terraform show -json tfplan │──▶ tfplan.json sensitive values in clear
  │ terraform show tfplan │──▶ plan.txt sensitive values redacted
  └────────────────┬────────────────┘
                   ▼
  ┌─────────────────────────────────┐ ③ the override labels are read here,
  │ conftest test tfplan.json │ and the payload is the wrong source
  │ --data pr_data.json │
  └────────────────┬────────────────┘
                   ▼
  ┌─────────────────────────────────┐ ④ comment 65,536 chars, summary 1 MiB
  │ comment · run summary · artifact│ ② the upload is skipped after a failure
  └────────────────┬────────────────┘
                   ▼
        status check on the pull request

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Number five is not on the diagram, because it is about which modules reach it at all.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. The credentials, and what a plan can do with them
&lt;/h3&gt;

&lt;p&gt;A plan needs to read state, and usually the infrastructure too. Our first version used a long-lived access key stored as a repository secret.&lt;/p&gt;

&lt;p&gt;Replacing that with OIDC federation was the biggest single improvement. The pipeline gets a short-lived token from the CI provider. The cloud provider checks its signature and its claims, then hands back credentials that live for one job. There is no stored key to leak.&lt;/p&gt;

&lt;p&gt;The trust condition is the whole security boundary, so make it exact. Matching the identity claim with a wildcard across an organization lets every repository in that organization assume the role. Match the full claim. If you want an approval step, use the CI provider's environment claim, because anything written only in the workflow file can be edited by whoever opened the pull request.&lt;/p&gt;

&lt;p&gt;In practice that is two pieces. The workflow asks for a token and trades it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;permissions&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;contents&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;read&lt;/span&gt;
  &lt;span class="na"&gt;id-token&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;write&lt;/span&gt; &lt;span class="c1"&gt;# without this no token is minted&lt;/span&gt;

&lt;span class="na"&gt;steps&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;aws-actions/configure-aws-credentials@v6&lt;/span&gt;
    &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;role-to-assume&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;arn:aws:iam::&amp;lt;account&amp;gt;:role/terraform-plan&lt;/span&gt;
      &lt;span class="na"&gt;aws-region&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;us-east-1&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And the role decides who may do that:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"Effect"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Allow"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"Action"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"sts:AssumeRoleWithWebIdentity"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"Principal"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"Federated"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"arn:aws:iam::&amp;lt;account&amp;gt;:oidc-provider/token.actions.githubusercontent.com"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"Condition"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"StringEquals"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"token.actions.githubusercontent.com:aud"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"sts.amazonaws.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"token.actions.githubusercontent.com:sub"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"repo:&amp;lt;org&amp;gt;/&amp;lt;repo&amp;gt;:pull_request"&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note &lt;code&gt;StringEquals&lt;/code&gt; on the whole &lt;code&gt;sub&lt;/code&gt;, not &lt;code&gt;StringLike&lt;/code&gt; with a wildcard. &lt;code&gt;repo:&amp;lt;org&amp;gt;/*&lt;/code&gt; would let any repository in the organization assume this role. If you want a human approval in front of a sensitive module, gate the job on a CI environment and match &lt;code&gt;...:environment:&amp;lt;name&amp;gt;&lt;/code&gt; instead: that claim only appears when the job really declares the environment, so removing the declaration from the workflow file does not get you the role.&lt;/p&gt;

&lt;p&gt;Now the uncomfortable part, and the one your security reviewer will find:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;code&gt;terraform plan&lt;/code&gt; runs configuration from the branch under review.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Someone can add a data source that reads the job's environment and sends it somewhere. Every credential in that job is reachable by anyone who can open a pull request, and so is every state file the job's role can read.&lt;/p&gt;

&lt;p&gt;Most CI systems do not give secrets to pull requests from forks, so in practice the exposure is to people who already have write access. That is not nothing: "already trusted" is not the same as "cannot be phished". Give the role read-only access, scope it to the state it needs, and decide before you add your most sensitive module whether its plan should sit behind an approval.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. A condition that skipped the step we needed most
&lt;/h3&gt;

&lt;p&gt;Our pipeline uploads the readable plan as an artifact. The step had a condition: only run if the plan succeeded.&lt;/p&gt;

&lt;p&gt;In GitHub Actions, a condition that does not mention a status function gets an implicit &lt;code&gt;success()&lt;/code&gt; added to it. So as soon as the policy check failed, the upload was skipped. Meanwhile the pull request comment still pointed at an artifact that did not exist, exactly when a reviewer most needed to read the plan.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# skipped as soon as any earlier step fails, artifact and all&lt;/span&gt;
&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Upload plan&lt;/span&gt;
  &lt;span class="na"&gt;if&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;steps.plan.outcome == 'success'&lt;/span&gt;

&lt;span class="c1"&gt;# runs on its own terms&lt;/span&gt;
&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Upload plan&lt;/span&gt;
  &lt;span class="na"&gt;if&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;always() &amp;amp;&amp;amp; steps.plan.outcome == 'success'&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The fix is one word, &lt;code&gt;always()&lt;/code&gt;. The lesson is bigger: after a step fails, every condition below it means something different.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Labels the pipeline could not see
&lt;/h3&gt;

&lt;p&gt;The override label was read from the event that started the run.&lt;/p&gt;

&lt;p&gt;A reviewer's natural sequence is: check fails, read the message, add the label, re-run the failed job. That does not work. Re-running replays the original event, labels included. The new label is invisible and the check fails again in exactly the same way. From the outside this looks like "the override is broken".&lt;/p&gt;

&lt;p&gt;Two changes fixed it. The job now asks the API for the pull request's current labels instead of trusting the stored event. And the workflow also triggers when a label is added or removed, so applying the label starts a fresh run by itself.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;on:
  pull_request:
    &lt;span class="c"&gt;# the default is [opened, synchronize, reopened]: labelling fires nothing&lt;/span&gt;
    types: &lt;span class="o"&gt;[&lt;/span&gt;opened, synchronize, reopened, labeled, unlabeled]


&lt;span class="c"&gt;# not github.event.pull_request.labels: that is the payload the run started with&lt;/span&gt;
gh &lt;span class="nb"&gt;pr &lt;/span&gt;view &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$PR_NUMBER&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--repo&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$REPO&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--json&lt;/span&gt; labels &lt;span class="nt"&gt;--jq&lt;/span&gt; &lt;span class="s1"&gt;'{pr: {labels: [.labels[].name]}}'&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; pr_data.json

&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Labels seen by the policies: &lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;jq &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s1"&gt;'.pr.labels'&lt;/span&gt; pr_data.json&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;

conftest &lt;span class="nb"&gt;test &lt;/span&gt;tfplan.json &lt;span class="nt"&gt;--policy&lt;/span&gt; policy &lt;span class="nt"&gt;--data&lt;/span&gt; pr_data.json

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The job also prints the labels it saw. That turns an argument into a five-second check.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Output that outgrew the comment
&lt;/h3&gt;

&lt;p&gt;Our first version pasted the whole plan into a pull request comment and cut it off when it got too long. Fine for a small environment. Not fine for a real one: the comment limit is 65,536 characters, and reviewers got a plan that stopped mid-resource.&lt;/p&gt;

&lt;p&gt;What works is three layers.&lt;/p&gt;

&lt;p&gt;The comment holds the summary line and one line per changed resource, built from the plan JSON instead of the text output:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;jq &lt;span class="nt"&gt;-r&lt;/span&gt; &lt;span class="s1"&gt;'.resource_changes[]
       | select(.change.actions != ["no-op"] and .change.actions != ["read"])
       | "\(.change.actions | join("+")) \(.address)"'&lt;/span&gt; tfplan.json | &lt;span class="nb"&gt;sort&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;which gives one line per real change:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight terraform"&gt;&lt;code&gt;&lt;span class="nx"&gt;delete&lt;/span&gt; &lt;span class="nx"&gt;aws_s3_bucket&lt;/span&gt;&lt;span class="err"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;exports&lt;/span&gt;
&lt;span class="nx"&gt;delete&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="nx"&gt;create&lt;/span&gt; &lt;span class="nx"&gt;aws_dynamodb_table&lt;/span&gt;&lt;span class="err"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;sessions&lt;/span&gt;
&lt;span class="nx"&gt;update&lt;/span&gt; &lt;span class="nx"&gt;aws_sqs_queue&lt;/span&gt;&lt;span class="err"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;jobs&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Five hundred resources is about 25 KB, which fits easily.&lt;/p&gt;

&lt;p&gt;The full readable plan goes into the pipeline's run summary page, which renders in the browser with a 1 MiB budget and needs no download.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# readable plan, no refresh noise, safe to publish&lt;/span&gt;
terraform show &lt;span class="nt"&gt;-no-color&lt;/span&gt; tfplan &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; plan.txt

&lt;span class="o"&gt;{&lt;/span&gt;
  &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"## Plan for &lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;DIR&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
  &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s1"&gt;'```

hcl'&lt;/span&gt;
  &lt;span class="nb"&gt;head&lt;/span&gt; &lt;span class="nt"&gt;-c&lt;/span&gt; 900000 plan.txt
  &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s1"&gt;'

```'&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$GITHUB_STEP_SUMMARY&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The same file is attached as an artifact for anyone who wants to search it locally.&lt;/p&gt;

&lt;p&gt;One detail worth flagging to your security and legal reviewers: attach the &lt;strong&gt;human-readable&lt;/strong&gt; plan, not the JSON. Terraform hides sensitive attributes in its rendered output. The JSON contains them. An artifact can be downloaded by everyone with read access and stays for its retention period, which by default is far longer than any review needs. Shorten that, and remember each run leaves its own copy instead of replacing the last one.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Coverage that looked better than it was
&lt;/h3&gt;

&lt;p&gt;The pull request comment was green long before the rules protected anything important, because most modules produced no plan at all.&lt;/p&gt;

&lt;p&gt;If you take one operational point from this article, take this one. Track which modules are actually planned in CI. Publish the list. Treat it as the real coverage number.&lt;/p&gt;

&lt;p&gt;Ours is two commands and a diff. Every root module declares a backend, so the first list is free:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# every root module in the repository&lt;/span&gt;
&lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-rl&lt;/span&gt; &lt;span class="s1"&gt;'backend "s3"'&lt;/span&gt; &lt;span class="nt"&gt;--include&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'*.tf'&lt;/span&gt; &lt;span class="nb"&gt;.&lt;/span&gt; | xargs &lt;span class="nt"&gt;-n1&lt;/span&gt; &lt;span class="nb"&gt;dirname&lt;/span&gt; | &lt;span class="nb"&gt;sort&lt;/span&gt; &lt;span class="nt"&gt;-u&lt;/span&gt;

&lt;span class="c"&gt;# the ones the pipeline actually plans&lt;/span&gt;
yq &lt;span class="s1"&gt;'.on.pull_request.paths[]'&lt;/span&gt; .github/workflows/terraform-plan.yaml

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The gap between those two lists is the honest answer to "is production covered". Ours started at two modules out of twenty. Nobody would have guessed that from the green checkmarks.&lt;/p&gt;

&lt;p&gt;Rules are cheap. Plans are the expensive part, because each new module needs credentials, network access, and sometimes an approval design.&lt;/p&gt;

&lt;h2&gt;
  
  
  Test the rules, then test the plumbing
&lt;/h2&gt;

&lt;p&gt;Policy code is code, and it has an unpleasant failure mode: a broken rule does not crash. It just stops denying, and everything turns green. The thing you have to design against is silence.&lt;/p&gt;

&lt;p&gt;That needs two layers.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Unit tests, in Rego.&lt;/strong&gt; Conftest runs files named &lt;code&gt;*_test.rego&lt;/code&gt;. A test hands the policy a small hand-written document instead of a real plan:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rego"&gt;&lt;code&gt;&lt;span class="n"&gt;replace_plan&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"resource_changes"&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[{&lt;/span&gt;
    &lt;span class="s2"&gt;"address"&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"aws_s3_bucket.media"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"type"&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"aws_s3_bucket"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"change"&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"actions"&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"delete"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"create"&lt;/span&gt;&lt;span class="p"&gt;]},&lt;/span&gt;
&lt;span class="p"&gt;}]}&lt;/span&gt;

&lt;span class="n"&gt;test_replacement_is_denied&lt;/span&gt; &lt;span class="n"&gt;if&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;deny&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt; &lt;span class="ow"&gt;with&lt;/span&gt; &lt;span class="n"&gt;input&lt;/span&gt; &lt;span class="ow"&gt;as&lt;/span&gt; &lt;span class="n"&gt;replace_plan&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;test_label_overrides_deny&lt;/span&gt; &lt;span class="n"&gt;if&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;deny&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt; &lt;span class="ow"&gt;with&lt;/span&gt; &lt;span class="n"&gt;input&lt;/span&gt; &lt;span class="ow"&gt;as&lt;/span&gt; &lt;span class="n"&gt;replace_plan&lt;/span&gt;
        &lt;span class="ow"&gt;with&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pr&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;labels&lt;/span&gt; &lt;span class="ow"&gt;as&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"allow-destroy"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No Terraform runs. No credentials. These finish in about a second, and they cover the logic: the right resource types, replacement treated as deletion, the override working, the wrong override not working.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;One test of the command itself.&lt;/strong&gt; Unit tests run inside the policy engine, so they cannot see how it gets called. In our pipeline it is called with a policy directory, a data file, and an exit code that a shell script reads. All three can break without touching a single rule.&lt;/p&gt;

&lt;p&gt;So we keep one hand-written plan that breaks every rule at once. The pipeline checks two things:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The real command rejects it.&lt;/li&gt;
&lt;li&gt;The same command with every override label accepts it.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The first check is inverted: if the command &lt;em&gt;passes&lt;/em&gt;, the step fails, because this file is supposed to be rejected. That inversion caught a folder rename that had quietly stopped half our rules from loading.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;conftest verify &lt;span class="nt"&gt;--policy&lt;/span&gt; policy

&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s1"&gt;'{"pr": {"labels": []}}'&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; no_labels.json
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s1"&gt;'{"pr": {"labels": ["allow-destroy", "allow-unproxied", "allow-threshold-change"]}}'&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; all_labels.json

&lt;span class="c"&gt;# inverted on purpose: the fixture breaks every rule, so passing is the failure&lt;/span&gt;
&lt;span class="k"&gt;if &lt;/span&gt;conftest &lt;span class="nb"&gt;test &lt;/span&gt;tests/plan_violations.json &lt;span class="se"&gt;\&lt;/span&gt;
     &lt;span class="nt"&gt;--policy&lt;/span&gt; policy &lt;span class="nt"&gt;--data&lt;/span&gt; no_labels.json&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then
  &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"The fixture violates every rule, but conftest passed it."&lt;/span&gt;
  &lt;span class="nb"&gt;exit &lt;/span&gt;1
&lt;span class="k"&gt;fi

&lt;/span&gt;conftest &lt;span class="nb"&gt;test &lt;/span&gt;tests/plan_violations.json &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--policy&lt;/span&gt; policy &lt;span class="nt"&gt;--data&lt;/span&gt; all_labels.json

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Print a line before it saying the failures below are expected. Otherwise the next person to open that log sees six red FAIL lines in a green job and files a bug.&lt;/p&gt;

&lt;p&gt;Two practical notes. Keep that file outside the policy folder, so nothing can mistake a test document for a policy input. And make it a rule that every new policy adds its own violation to the file, otherwise the file slowly stops covering everything and the check goes green for the wrong reason.&lt;/p&gt;

&lt;p&gt;Before this we did something else: when a policy changed, the pipeline ran a real plan against one environment as a smoke test. It worked, and everyone disliked it, because a pull request that touched only policy files would start planning infrastructure it had never touched. The fixture replaced it. It is faster, needs no credentials, and tests more.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this does not do
&lt;/h2&gt;

&lt;p&gt;Being clear about the limits is what keeps the gate trusted.&lt;/p&gt;

&lt;p&gt;It does not see changes made outside Terraform. Someone deleting a resource in a web console is invisible here.&lt;/p&gt;

&lt;p&gt;It does not cover modules that produce no plan.&lt;/p&gt;

&lt;p&gt;It is not a security scanner. The established scanners cover unencrypted storage, public buckets and open security groups better than anything you will write by hand, and they need no policy authoring at all. Use them for known bad configuration. Use custom policy for facts only your organization knows: which resources cannot be recreated, which identifiers other systems depend on, which thresholds were chosen deliberately.&lt;/p&gt;

&lt;p&gt;It does not stop a determined person with merge rights. It makes an accident expensive and a deliberate act recorded.&lt;/p&gt;

&lt;p&gt;And it does not survive neglect. Write twenty rules in a planning meeting and half of them will be dead within a quarter, with everyone adding the override label without reading the message. Ours grow one at a time. Each one traces back to an incident, a near miss, or a review comment where someone asked what stops this from happening again. Nine rules across four providers. Each has a story.&lt;/p&gt;

&lt;h2&gt;
  
  
  If you are starting on Monday
&lt;/h2&gt;

&lt;p&gt;Write the tests before the second rule, not after the tenth.&lt;/p&gt;

&lt;p&gt;Keep one fixture that breaks everything, and keep it up to date.&lt;/p&gt;

&lt;p&gt;Start with a single rule about destruction. Everyone agrees with it, so nobody argues in review.&lt;/p&gt;

&lt;p&gt;Make the message a sentence you would be glad to read at the moment it stops you.&lt;/p&gt;

&lt;p&gt;Put the override behind something that sticks around and says what was approved.&lt;/p&gt;

&lt;p&gt;Federate your credentials before you widen what the pipeline can reach, not after.&lt;/p&gt;

&lt;p&gt;And write down which modules are covered, so that when someone asks whether the gate protects production, the answer is a list and not a hope.&lt;/p&gt;




&lt;p&gt;Originally published at &lt;a href="https://dmytrobot.dev/blog/terraform-policy-as-code/" rel="noopener noreferrer"&gt;dmytrobot.dev&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>terraform</category>
      <category>opa</category>
      <category>conftest</category>
      <category>ci</category>
    </item>
    <item>
      <title>Adopting OPA Gatekeeper: mechanics and failure modes</title>
      <dc:creator>Dmytro Bot</dc:creator>
      <pubDate>Fri, 02 Oct 2026 00:00:00 +0000</pubDate>
      <link>https://dev.to/777dimas/adopting-opa-gatekeeper-mechanics-and-failure-modes-3nb4</link>
      <guid>https://dev.to/777dimas/adopting-opa-gatekeeper-mechanics-and-failure-modes-3nb4</guid>
      <description>&lt;h2&gt;
  
  
  Why admission control
&lt;/h2&gt;

&lt;p&gt;Every cluster runs on rules that exist only in people's heads. Images come from our registry. Nothing runs privileged. Tags are pinned. Everyone agrees, and then a debugging session at two in the morning mounts the node's filesystem into a pod, and the rule quietly stops being true.&lt;/p&gt;

&lt;p&gt;Code review catches some of this. It cannot catch what never reaches review: a &lt;code&gt;kubectl run&lt;/code&gt; from a laptop, a Helm chart whose values nobody read, a manifest applied by a script written three years ago by someone who has since left. The gap is not discipline. The rules are simply not enforced at the point where objects actually arrive.&lt;/p&gt;

&lt;p&gt;Admission control closes that gap by moving the rules into the API server's request path. Every create and update is offered to a policy engine before it is persisted. OPA Gatekeeper is one implementation: policies are written in Rego, shipped as ordinary Kubernetes resources, and evaluated both at admission time and continuously against what already exists.&lt;/p&gt;

&lt;p&gt;This article covers how Gatekeeper works in the places that matter operationally, then the four failure modes encountered while rolling it out across three clusters over several weeks. The mechanics are documented upstream. The failure modes are the expensive part, and most of them are not.&lt;/p&gt;

&lt;p&gt;Examples are anonymised. Registry names, namespaces and hostnames below are placeholders; the shapes of the problems are not.&lt;/p&gt;

&lt;h2&gt;
  
  
  The moving parts
&lt;/h2&gt;

&lt;p&gt;Gatekeeper splits a policy into two objects: a template that defines the rule, and a constraint that decides where the rule applies. The separation is the whole design, and it is what makes a policy library reusable across clusters that share nothing else.&lt;/p&gt;

&lt;p&gt;A &lt;code&gt;ConstraintTemplate&lt;/code&gt; carries the logic and, as a side effect, mints a new CRD:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;templates.gatekeeper.sh/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ConstraintTemplate&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;k8sblocknodeport&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;crd&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;names&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;K8sBlockNodePort&lt;/span&gt; &lt;span class="c1"&gt;# the CRD this creates&lt;/span&gt;
  &lt;span class="na"&gt;targets&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;target&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;admission.k8s.gatekeeper.sh&lt;/span&gt;
      &lt;span class="na"&gt;rego&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;|&lt;/span&gt;
        &lt;span class="s"&gt;package k8sblocknodeport&lt;/span&gt;

        &lt;span class="s"&gt;violation[{"msg": msg}] {&lt;/span&gt;
          &lt;span class="s"&gt;input.review.kind.kind == "Service"&lt;/span&gt;
          &lt;span class="s"&gt;input.review.object.spec.type == "NodePort"&lt;/span&gt;
          &lt;span class="s"&gt;msg := "Service of type NodePort is not allowed"&lt;/span&gt;
        &lt;span class="s"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Applying that template creates a CRD named &lt;code&gt;K8sBlockNodePort&lt;/code&gt;. A constraint is an instance of it, and instances carry scope and severity:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;constraints.gatekeeper.sh/v1beta1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;K8sBlockNodePort&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;block-nodeport-services&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;enforcementAction&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;deny&lt;/span&gt;
  &lt;span class="na"&gt;match&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;kinds&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;apiGroups&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
        &lt;span class="na"&gt;kinds&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Service"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
    &lt;span class="na"&gt;excludedNamespaces&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;kube-system&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;ingress&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Five things about the Rego explain most of the surprises later. They are worth learning even if you never write a policy yourself.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;violation&lt;/code&gt; is a set, not a yes/no.&lt;/strong&gt; You never write "allow". You list reasons to reject. An empty set means the object passes. Each element becomes one line in a report, so one bad container produces one violation and three produce three.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Lines in a rule are joined by AND.&lt;/strong&gt; There is no operator for it. The rule fires only if every line holds.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Two rules with the same name are OR.&lt;/strong&gt; That is how you write "or" in Rego: not with an operator, but by defining the rule twice. This matters more than it sounds, see failure mode 1.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A missing field usually means "allowed".&lt;/strong&gt; In Rego, reading a field that is not there gives &lt;code&gt;undefined&lt;/code&gt;, not &lt;code&gt;false&lt;/code&gt;. The rule stops and produces no violation. So a policy about &lt;code&gt;spec.type&lt;/code&gt; ignores an object that has no &lt;code&gt;spec.type&lt;/code&gt;, unless the author added a rule for that case.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;input&lt;/code&gt; is what you are given.&lt;/strong&gt; &lt;code&gt;input.review.object&lt;/code&gt; is the object under review. &lt;code&gt;input.parameters&lt;/code&gt; is whatever the constraint put in &lt;code&gt;spec.parameters&lt;/code&gt;. Looping is written &lt;code&gt;[_]&lt;/code&gt;, and every match produces its own violation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two loops that do not talk to each other
&lt;/h2&gt;

&lt;p&gt;Gatekeeper evaluates the same policies in two completely separate places, and conflating them is the single most common source of confusion.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;  kubectl / CI / controller
            │
            ▼
     ┌──────────────┐   admission review   ┌──────────────┐
     │  API server  │ ───────────────────▶ │  Gatekeeper  │
     │              │ ◀─────────────────── │   webhook    │
     └──────┬───────┘     allow / deny     └──────────────┘
            │
            ▼
        ( etcd )
            ┆  every N seconds
            ▼
     ┌──────────────┐         ┌───────────────────┐
     │    Audit     │ ┄┄┄┄┄┄▶ │ constraint status │
     │  controller  │         │ + metrics         │
     └──────────────┘         └───────────────────┘
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;strong&gt;admission webhook&lt;/strong&gt; is synchronous and sits in the request path. It sees an object once, at the moment someone tries to create or change it, and answers allow or deny. It never sees anything that already exists.&lt;/p&gt;

&lt;p&gt;The &lt;strong&gt;audit controller&lt;/strong&gt; is asynchronous. On an interval it walks objects already in the cluster, evaluates them against every constraint, and writes the results to each constraint's status and to metrics. It blocks nothing, ever.&lt;/p&gt;

&lt;p&gt;Three consequences follow, and all three bite in practice.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Zero violations does not mean nothing would be blocked.&lt;/strong&gt; It means the audit found nothing. Objects created since the last audit run, and objects that were refused and so never created, are both missing from that number.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A new constraint reports zero until its first audit finishes.&lt;/strong&gt; Zero-because-clean and zero-because-not-computed-yet look identical. Tell them apart with &lt;code&gt;status.auditTimestamp&lt;/code&gt;: when every constraint shows the same recent timestamp, a full cycle has run and the numbers mean something.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Exemptions are not symmetric.&lt;/strong&gt; The &lt;code&gt;--exempt-namespace&lt;/code&gt; flag (&lt;code&gt;controllerManager.exemptNamespaces&lt;/code&gt; in the Helm chart) silences the webhook only. The audit controller ignores it and walks every namespace anyway, &lt;a href="https://open-policy-agent.github.io/gatekeeper/website/docs/exempt-namespaces/" rel="noopener noreferrer"&gt;as the upstream docs state&lt;/a&gt;. To exclude a namespace from both, either repeat the list in each constraint's &lt;code&gt;match.excludedNamespaces&lt;/code&gt;, or list it once in the cluster-wide &lt;code&gt;Config&lt;/code&gt; resource with &lt;code&gt;processes: ["*"]&lt;/code&gt;. The &lt;code&gt;Config&lt;/code&gt; route still sends every request to the webhook, so a Gatekeeper outage still reaches that namespace; the flag does not. In review either one looks redundant next to the flag. It is not.&lt;/p&gt;

&lt;h2&gt;
  
  
  Enforcement, and what happens when the engine is down
&lt;/h2&gt;

&lt;p&gt;Nothing is blocked until you add a constraint. Every restriction is opt-in, so the rollout can go one policy at a time.&lt;/p&gt;

&lt;p&gt;Each constraint has an &lt;code&gt;enforcementAction&lt;/code&gt;:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Action&lt;/th&gt;
&lt;th&gt;Admission&lt;/th&gt;
&lt;th&gt;Audit&lt;/th&gt;
&lt;th&gt;Use it for&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;dryrun&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;allows&lt;/td&gt;
&lt;td&gt;records&lt;/td&gt;
&lt;td&gt;measuring a policy before you commit to it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;warn&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;allows, warns the author&lt;/td&gt;
&lt;td&gt;records&lt;/td&gt;
&lt;td&gt;telling people without breaking them&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;deny&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;rejects&lt;/td&gt;
&lt;td&gt;records&lt;/td&gt;
&lt;td&gt;enforcement&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;One detail catches people out: &lt;strong&gt;if you leave the field out, the default is &lt;code&gt;deny&lt;/code&gt;&lt;/strong&gt; , not &lt;code&gt;dryrun&lt;/code&gt;. A constraint written without it starts blocking the moment it lands. Always set it explicitly.&lt;/p&gt;

&lt;p&gt;The next question is what happens when Gatekeeper itself is down. The webhook sits in front of every create in the cluster, so this matters.&lt;/p&gt;

&lt;p&gt;The answer depends on &lt;code&gt;failurePolicy&lt;/code&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;Ignore&lt;/code&gt;&lt;/strong&gt; : the API server carries on when the webhook does not answer. Policies stop being enforced during an outage. The cluster keeps working.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;Fail&lt;/code&gt;&lt;/strong&gt; : the API server rejects the request. Policies hold. An engine outage becomes a cluster-wide write outage.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Most teams want &lt;code&gt;Ignore&lt;/code&gt;, with a short webhook timeout. A policy engine that can take the cluster down with it is a bad trade when the policies are hygiene rules.&lt;/p&gt;

&lt;p&gt;But be honest about what that means. &lt;code&gt;Ignore&lt;/code&gt; makes the guarantee best-effort. Anyone who can disrupt the webhook can bypass the policy. If a policy is part of your security boundary, &lt;code&gt;Fail&lt;/code&gt; is the right choice, and you handle the availability risk by running the engine properly, not by weakening the policy.&lt;/p&gt;

&lt;p&gt;Either way, run the controllers with several replicas, a pod disruption budget and anti-affinity. Under &lt;code&gt;Ignore&lt;/code&gt; an outage silently switches enforcement off, which is exactly when you want it on.&lt;/p&gt;

&lt;h2&gt;
  
  
  Policies match Pods; humans apply Deployments
&lt;/h2&gt;

&lt;p&gt;Almost every interesting workload policy is about a pod spec, so the constraint matches &lt;code&gt;Pod&lt;/code&gt;. Almost nobody applies a Pod. They apply a Deployment, a StatefulSet or a CronJob, and the pod appears later, created by a controller.&lt;/p&gt;

&lt;p&gt;Match only &lt;code&gt;Pod&lt;/code&gt; and the rejection lands in the wrong place. The Deployment is admitted happily; its ReplicaSet then fails to create pods, and the author sees a healthy-looking Deployment with zero ready replicas and an error buried in events.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;ExpansionTemplate&lt;/code&gt; fixes this. It tells Gatekeeper how to derive the pod a workload would produce, so policies evaluate against that derived pod at the moment the workload is applied:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;expansion.gatekeeper.sh/v1beta1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ExpansionTemplate&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;expand-workloads&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;applyTo&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;groups&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;apps"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
      &lt;span class="na"&gt;kinds&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Deployment"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;StatefulSet"&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;DaemonSet"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
      &lt;span class="na"&gt;versions&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;v1"&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;templateSource&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;spec.template"&lt;/span&gt;
  &lt;span class="na"&gt;generatedGVK&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Pod"&lt;/span&gt;
    &lt;span class="na"&gt;version&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;v1"&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With expansion in place, the rejection arrives on &lt;code&gt;kubectl apply&lt;/code&gt; of the Deployment, and the message is prefixed to say it was implied by expansion. Jobs and CronJobs need their own templates, since the path to the pod spec differs.&lt;/p&gt;

&lt;p&gt;Two details are easy to miss.&lt;/p&gt;

&lt;p&gt;Expansion evaluates &lt;code&gt;spec.template&lt;/code&gt; regardless of replica count. A Deployment scaled to zero still fails admission, because the pod spec is still being declared. Dormant workloads left at zero replicas are a real source of violations in clusters that keep a full service catalogue in every environment.&lt;/p&gt;

&lt;p&gt;Deliberately leaving ReplicaSet out of the expansion chain avoids double-reporting. A Deployment and the ReplicaSet it owns would otherwise each produce the same violation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Order policies by cost, not by severity
&lt;/h2&gt;

&lt;p&gt;The instinct is to rank candidate policies by how bad the thing they prevent is, adopt the worst first, and work down. That ordering is wrong, and following it is how Gatekeeper rollouts stall.&lt;/p&gt;

&lt;p&gt;What determines whether a policy ever reaches enforcement is not its value. It is how many objects already violate it. A policy that cannot be driven to zero stays in &lt;code&gt;dryrun&lt;/code&gt; forever, and a dashboard full of permanent violations teaches everyone to ignore the dashboard, including for the policies that did reach zero.&lt;/p&gt;

&lt;p&gt;So measure first. Render every overlay and chart the cluster deploys, evaluate the candidate policies against the rendered output offline, and count. The spread is usually enormous.&lt;/p&gt;

&lt;p&gt;These are real counts from one production cluster's rendered manifests, across roughly 50 workloads:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Candidate policy&lt;/th&gt;
&lt;th&gt;Violating containers&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Block privileged containers&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Block hostPID / hostIPC&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Block hostNetwork / hostPort&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Block hostPath volumes&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Block NodePort / LoadBalancer services&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Require resource requests&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Require liveness and readiness probes&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Require memory limits&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Require &lt;code&gt;allowPrivilegeEscalation: false&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;80&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Require read-only root filesystem&lt;/td&gt;
&lt;td&gt;80&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Require dropping all capabilities&lt;/td&gt;
&lt;td&gt;80&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Require running as non-root&lt;/td&gt;
&lt;td&gt;80&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The top group is free. Nothing violates it, so it can go straight to &lt;code&gt;deny&lt;/code&gt; on the day it is written, and it closes exactly the escape routes a container breakout would use.&lt;/p&gt;

&lt;p&gt;The bottom group touches every workload in the estate, because no base manifest sets &lt;code&gt;securityContext&lt;/code&gt; at all. Same security literature, same "best practice" list, and the cost runs from zero to every workload.&lt;/p&gt;

&lt;p&gt;The ordering that works is value divided by violation count. Ship the free policies immediately. Schedule the expensive ones as projects with owners, and be honest that they are projects.&lt;/p&gt;

&lt;p&gt;For anything not free, the ladder is: &lt;code&gt;dryrun&lt;/code&gt;, measure, fix the causes, confirm a fresh full audit reports zero, then promote. Confirm means checking that every constraint shares one recent &lt;code&gt;auditTimestamp&lt;/code&gt;, not glancing at a zero.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure mode 1: policies that fault absence
&lt;/h2&gt;

&lt;p&gt;Two policies from the same upstream library, both described as pod hardening, differ by a single Rego rule, and that difference is the whole reason one costs nothing and the other costs a quarter.&lt;/p&gt;

&lt;p&gt;The privileged-container policy flags only an explicit setting:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rego"&gt;&lt;code&gt;&lt;span class="n"&gt;violation&lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="s2"&gt;"msg"&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="p"&gt;}]&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;input_containers&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;securityContext&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;privileged&lt;/span&gt; &lt;span class="c1"&gt;# only true when explicitly set&lt;/span&gt;
  &lt;span class="n"&gt;msg&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;sprintf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Privileged container is not allowed: %v"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The privilege-escalation policy adds a second rule:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rego"&gt;&lt;code&gt;&lt;span class="n"&gt;input_allow_privilege_escalation&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;has_field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"securityContext"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# no securityContext at all&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;input_allow_privilege_escalation&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;securityContext&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;allowPrivilegeEscalation&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Remember that several rules with the same name are OR. The first rule means &lt;strong&gt;a container with no &lt;code&gt;securityContext&lt;/code&gt; block is a violation&lt;/strong&gt;. That is deliberate upstream: the field defaults to permissive, so silence really is insecure. It is also why this policy flagged 80 containers where the privileged check flagged zero.&lt;/p&gt;

&lt;p&gt;The lesson generalises past this one policy. Before adopting anything from a policy library, read the Rego and answer one question: does an absent field count as a violation?&lt;/p&gt;

&lt;p&gt;If yes, the policy is not a rule you switch on. It is a migration across every workload you own, and it needs the schedule and the owner that implies. The same applies to read-only root filesystems, dropping all capabilities, running as non-root and seccomp profiles. Each of them flagged the entire estate for the same reason.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure mode 2: image matching is string matching
&lt;/h2&gt;

&lt;p&gt;The registry allowlist policy compares the image string exactly as written. There is no normalisation, no registry resolution, no awareness that a bare name means Docker Hub. Given an allowlist containing &lt;code&gt;docker.io/library/&lt;/code&gt;:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Image as written&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;docker.io/library/postgres:18-alpine&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;allowed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;postgres:18-alpine&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;rejected&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;registry.example.com/team/app:v1.2.3&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;allowed if that prefix is listed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;docker.io/team/app:v1.2.3&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;rejected unless that exact prefix is listed&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;This is defensible. A bare name is an implicit Docker Hub pull, which is both a supply-chain exposure and a rate-limit exposure, and forcing it to be written out makes that explicit. But it produces two traps.&lt;/p&gt;

&lt;p&gt;The first is the trailing slash. An allowlist entry of &lt;code&gt;acme&lt;/code&gt; without the slash also permits &lt;code&gt;acme-evil/backdoor:v1&lt;/code&gt;, because prefix matching does not know where a registry name ends. Every entry needs the trailing slash, and a test case should guard it. This is the kind of thing that is obvious once stated and invisible in review.&lt;/p&gt;

&lt;p&gt;The second is asymmetry between the forms your team writes. If the allowlist contains &lt;code&gt;myorg/&lt;/code&gt; for your own images and &lt;code&gt;docker.io/library/&lt;/code&gt; for upstream ones, then &lt;code&gt;myorg/app:v1&lt;/code&gt; is correct and &lt;code&gt;docker.io/myorg/app:v1&lt;/code&gt; is rejected, while &lt;code&gt;postgres:18&lt;/code&gt; is rejected and &lt;code&gt;docker.io/library/postgres:18&lt;/code&gt; is correct. Both conventions are defensible; having both in one list is a trap that fires months later, when someone writes the fully-qualified form out of tidiness.&lt;/p&gt;

&lt;p&gt;Write the rule down beside the allowlist, and test both spellings.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure mode 3: the objects nothing can see
&lt;/h2&gt;

&lt;p&gt;This is the one that cost real money, and it is the reason to read the rest of this article.&lt;/p&gt;

&lt;p&gt;A nightly job backed up the cluster's databases. For each database it created a short-lived helper pod, ran a dump through it, streamed the result to object storage and deleted the pod. The pod was created imperatively, from a shell script stored in a ConfigMap:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;IMAGE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"postgres:18-alpine"&lt;/span&gt; &lt;span class="c"&gt;# note: no registry prefix&lt;/span&gt;
kubectl run &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;pod_name&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;IMAGE&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt; &lt;span class="nt"&gt;-n&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;NAMESPACE&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; ...

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The registry allowlist went to &lt;code&gt;deny&lt;/code&gt;. Every helper pod was rejected at admission from that moment on. Backups produced nothing for twelve days.&lt;/p&gt;

&lt;p&gt;What makes this worth dwelling on is that &lt;strong&gt;every check we had was clean&lt;/strong&gt; , and each was clean for a different and individually reasonable reason.&lt;/p&gt;

&lt;p&gt;The rendered manifests were clean, because that pod is not in any manifest. It exists only as a string inside a shell script inside a ConfigMap. Rendering the overlay produces the ConfigMap, not the pod.&lt;/p&gt;

&lt;p&gt;The policy unit tests were clean, for the same reason. They test manifests. There is no manifest.&lt;/p&gt;

&lt;p&gt;The audit controller was clean, because audit only evaluates objects that exist. These pods were rejected at creation, so they never existed to be audited. A policy working perfectly and a policy matching nothing produce the same zero.&lt;/p&gt;

&lt;p&gt;And the CronJob reported success throughout, which is the subject of the next section.&lt;/p&gt;

&lt;p&gt;The general statement is uncomfortable: &lt;strong&gt;anything that creates pods at runtime is invisible to every static check in your pipeline.&lt;/strong&gt; That includes backup and restore jobs, operators that spawn workers, CI runners that start build pods, &lt;code&gt;kubectl debug&lt;/code&gt;, Helm hooks, and any tool shelling out to &lt;code&gt;kubectl run&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;There is no clever fix. There is a checklist item. Before promoting any policy to &lt;code&gt;deny&lt;/code&gt;, grep the repository for imperative pod creation and evaluate each site by hand:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-rn&lt;/span&gt; &lt;span class="s1"&gt;'kubectl run\|--image='&lt;/span&gt; &lt;span class="nt"&gt;--include&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'*.sh'&lt;/span&gt; &lt;span class="nt"&gt;--include&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'*.yaml'&lt;/span&gt; &lt;span class="nb"&gt;.&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In our case that search turned up six more sites beyond the one that broke, including the database restore script, which would have failed at the exact moment someone needed it most.&lt;/p&gt;

&lt;p&gt;One more thing is worth checking at the same time. A rejection has to be received by something, and that something was written before anyone thought about admission control. Look at what the caller does with a failure: a loop that logs the error and carries on will turn a policy rejection into a silent no-op, and the run will still report success.&lt;/p&gt;

&lt;p&gt;That is worth knowing before you promote a policy, not after. In our case the rejection was sent to &lt;code&gt;/dev/null&lt;/code&gt; and the job exited zero, so the breakage stayed invisible far longer than it needed to.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure mode 4: two engines in one template
&lt;/h2&gt;

&lt;p&gt;Kubernetes now has its own policy mechanism, &lt;a href="https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/" rel="noopener noreferrer"&gt;ValidatingAdmissionPolicy&lt;/a&gt;. It runs CEL expressions inside the API server, with no webhook. Gatekeeper can compile constraints down to it, and many upstream templates now carry both implementations, a Rego block and a CEL block, in one &lt;code&gt;ConstraintTemplate&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;That causes two problems. One is cosmetic. One is not.&lt;/p&gt;

&lt;p&gt;The cosmetic one: the two engines word their messages differently. Offline test tools run the Rego path. The cluster may run the CEL path. So a test that asserts on message text can pass locally and describe something the cluster never says. Assert that a violation happened. Do not assert its wording unless you know which engine answers.&lt;/p&gt;

&lt;p&gt;The serious one is about &lt;code&gt;failurePolicy&lt;/code&gt;. A native policy runs inside the API server. If it has &lt;code&gt;failurePolicy: Fail&lt;/code&gt; and cannot be evaluated while the API server is starting, because the resources it needs are not loaded yet, then the API server starts rejecting requests it makes of itself. The node never finishes coming up.&lt;/p&gt;

&lt;p&gt;We hit this on a control-plane node. Three properties make it nasty:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Namespace exemptions do not help. The failing requests are not namespaced.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;dryrun&lt;/code&gt; does not help. The failure is in evaluating the policy, not in its verdict.&lt;/li&gt;
&lt;li&gt;Recovery runs through the API server that is refusing to start.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is a known upstream issue, not something exotic we did: see &lt;a href="https://github.com/open-policy-agent/gatekeeper/issues/4530" rel="noopener noreferrer"&gt;Gatekeeper #4530, "Control plane deadlock via Gatekeeper-generated ValidatingAdmissionPolicies"&lt;/a&gt;. The general webhook version of the same trap is described in the Kubernetes docs, which warn that a cluster can reach a state where self-healing is impossible.&lt;/p&gt;

&lt;p&gt;Recovery means removing the policy objects out of band, through the control plane's own configuration rather than &lt;code&gt;kubectl&lt;/code&gt;. That is a bad thing to be learning at the time.&lt;/p&gt;

&lt;p&gt;Kubernetes 1.36 added &lt;a href="https://kubernetes.io/blog/2026/05/04/kubernetes-v1-36-manifest-based-admission-control/" rel="noopener noreferrer"&gt;manifest-based admission control&lt;/a&gt; partly to close this bootstrap window. Until you are on it, treat &lt;code&gt;failurePolicy: Fail&lt;/code&gt; on a native policy as a control-plane change, not a policy change. Roll it to one node. Restart that node on purpose. Only then continue.&lt;/p&gt;

&lt;h2&gt;
  
  
  Testing, and the edge of what testing covers
&lt;/h2&gt;

&lt;p&gt;Gatekeeper ships &lt;code&gt;gator&lt;/code&gt;, a CLI that embeds the OPA engine, so policies can be tested with no cluster at all. Two modes matter.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;gator verify&lt;/code&gt; runs unit tests: a suite pairs a template with a constraint and a set of fixture objects, each with an expected outcome. &lt;code&gt;gator test&lt;/code&gt; takes rendered manifests and a set of policies and reports what would be rejected.&lt;/p&gt;

&lt;p&gt;A suite entry is unremarkable, which is the point:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;privileged-containers&lt;/span&gt;
  &lt;span class="na"&gt;template&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;../templates/k8spspprivilegedcontainer.yaml&lt;/span&gt;
  &lt;span class="na"&gt;constraint&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;fixtures/constraint-deny.yaml&lt;/span&gt;
  &lt;span class="na"&gt;expansion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;../templates/expansion-workloads.yaml&lt;/span&gt;
  &lt;span class="na"&gt;cases&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;privileged-container-is-rejected&lt;/span&gt;
      &lt;span class="na"&gt;object&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;fixtures/pod-privileged.yaml&lt;/span&gt;
      &lt;span class="na"&gt;assertions&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;violations&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;yes&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;absent-security-context-is-accepted&lt;/span&gt;
      &lt;span class="na"&gt;object&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;fixtures/pod-no-security-context.yaml&lt;/span&gt;
      &lt;span class="na"&gt;assertions&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;violations&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;no&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That second case is worth copying. It pins the property from failure mode 1, that this particular policy tolerates a missing &lt;code&gt;securityContext&lt;/code&gt;, so that a future template upgrade changing the behaviour shows up as a failing test rather than a broken deployment.&lt;/p&gt;

&lt;p&gt;Two more habits proved their worth. Keep a negative control for prefix matching, a fixture like &lt;code&gt;acme-evil/backdoor:v1&lt;/code&gt; asserting that the trailing slash in the allowlist does its job. And run &lt;code&gt;gator test&lt;/code&gt; in CI against rendered manifests with enforcement forced to &lt;code&gt;deny&lt;/code&gt;, regardless of what the cluster is running, so a bad manifest fails a pull request rather than a deploy.&lt;/p&gt;

&lt;p&gt;In a GitOps shop this is where enforcement actually lives. Objects reach the cluster from the repository, so CI is the gate that authors experience. Admission &lt;code&gt;deny&lt;/code&gt; is the backstop for everything that does not come from the repository, which, as failure mode 3 showed, is the interesting part.&lt;/p&gt;

&lt;p&gt;The boundary is worth stating plainly, because it is easy to over-trust a green pipeline. &lt;code&gt;gator&lt;/code&gt; sees manifests. It does not see Helm releases you did not render, objects created by operators, or anything constructed at runtime. The gap between "CI is green" and "the cluster is clean" is exactly the set of objects nobody wrote down.&lt;/p&gt;

&lt;h2&gt;
  
  
  What changes the day you switch to deny
&lt;/h2&gt;

&lt;p&gt;Enforcement changes how people work, and the changes are worth anticipating rather than discovering.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Debugging gets harder.&lt;/strong&gt; &lt;code&gt;kubectl debug node/&amp;lt;name&amp;gt;&lt;/code&gt; creates a pod with host namespaces and the node's root filesystem mounted. Under a host-escape policy set it is rejected outright. &lt;code&gt;kubectl debug&lt;/code&gt; on a pod defaults to a bare upstream image, which a registry allowlist rejects. Both are correct behaviour and both will be reported as "Gatekeeper is broken" the first time.&lt;/p&gt;

&lt;p&gt;Decide the break-glass path in advance. A dedicated namespace excluded from the relevant constraints is cleaner than teaching people to flip policies to &lt;code&gt;dryrun&lt;/code&gt;, because it is one known exception in one known place rather than a habit of disabling enforcement under pressure.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Most debugging does not need elevation.&lt;/strong&gt; Worth knowing, because the instinct is to reach for host access. Kubernetes leaves &lt;code&gt;CAP_NET_RAW&lt;/code&gt; in the default capability set, so &lt;code&gt;tcpdump&lt;/code&gt;, &lt;code&gt;ping&lt;/code&gt;, &lt;code&gt;mtr&lt;/code&gt; and port scanning all work in an ordinary pod against the pod's own network. Elevation is needed only for the node's netfilter tables, other pods' traffic, tracing node processes, and reading node files.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Scope your policies deliberately, and know the difference.&lt;/strong&gt; A policy scoped by &lt;code&gt;namespaces&lt;/code&gt; covers a listed set; one scoped by &lt;code&gt;excludedNamespaces&lt;/code&gt; covers everything else, including namespaces created next month by someone who has not read your policy. The second is the safer default for host-escape rules and the riskier one for anything opinionated about images.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Keep the back-out one line.&lt;/strong&gt; Setting a single constraint to &lt;code&gt;dryrun&lt;/code&gt; should be a one-line change in one file, reviewable in seconds. Anything more elaborate will not be used at three in the morning.&lt;/p&gt;

&lt;p&gt;Finally, watch the webhook itself. Constraint count, rejection rate by constraint, and webhook latency all belong on a dashboard. Rejections are the interesting signal: a spike means either an attack, or far more likely, that someone's routine workflow just stopped working and they have not told you yet.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it buys, and what it does not
&lt;/h2&gt;

&lt;p&gt;Admission control is worth adopting. It is the only place a cluster rule becomes a fact rather than an agreement, and it costs one afternoon. The free tier of policies (no privileged containers, no host namespaces, no hostPath, no NodePort or LoadBalancer services, no arbitrary external IPs) closes the routes a container escape actually uses.&lt;/p&gt;

&lt;p&gt;What it does not buy is the thing the marketing implies. It does not make a cluster compliant, and it does not replace the work of fixing what the policies find. The expensive policies stay expensive; a tool that tells you 80 workloads run as root has not made any of them stop.&lt;/p&gt;

&lt;p&gt;The adoption sequence that worked:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Install the engine with no constraints. Confirm the webhook is healthy and &lt;code&gt;failurePolicy&lt;/code&gt; is deliberate.&lt;/li&gt;
&lt;li&gt;Render every manifest you deploy and measure candidate policies offline. Rank by violations, not by severity.&lt;/li&gt;
&lt;li&gt;Adopt the zero-violation set straight to &lt;code&gt;deny&lt;/code&gt;. Write unit tests, including a negative control per policy.&lt;/li&gt;
&lt;li&gt;Grep for imperative pod creation and evaluate every site by hand. This is the step that gets skipped.&lt;/li&gt;
&lt;li&gt;Put everything else in &lt;code&gt;dryrun&lt;/code&gt;. Fix causes. Promote only after a full audit cycle reports zero.&lt;/li&gt;
&lt;li&gt;Alert on outcomes, not on the engine. A backup that stops producing backups matters more than a constraint count.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;One distinction is worth carrying away. Admission control makes a rule true at the boundary of the cluster. It does not make what sits behind that boundary correct, and it does not decide what happens when a rule fires.&lt;/p&gt;

&lt;p&gt;That second part is the one teams underestimate. A policy engine will do exactly what you asked of it, correctly, from the first night. Everything downstream of a rejection will behave the way it already behaved. Enforcement is very good at turning an unwritten assumption into a fact, and it is no substitute for knowing how your own systems fail.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://open-policy-agent.github.io/gatekeeper/website/docs/violations/" rel="noopener noreferrer"&gt;Handling Constraint Violations&lt;/a&gt;: enforcement actions and the &lt;code&gt;deny&lt;/code&gt; default&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://open-policy-agent.github.io/gatekeeper/website/docs/exempt-namespaces/" rel="noopener noreferrer"&gt;Exempting Namespaces&lt;/a&gt;: webhook exemption versus audit&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/open-policy-agent/gatekeeper/issues/4530" rel="noopener noreferrer"&gt;Gatekeeper #4530: control plane deadlock via generated ValidatingAdmissionPolicies&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/" rel="noopener noreferrer"&gt;ValidatingAdmissionPolicy&lt;/a&gt;: Kubernetes documentation&lt;/li&gt;
&lt;li&gt;&lt;a href="https://kubernetes.io/blog/2026/05/04/kubernetes-v1-36-manifest-based-admission-control/" rel="noopener noreferrer"&gt;Kubernetes v1.36: manifest-based admission control&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;Originally published at &lt;a href="https://dmytrobot.dev/blog/opa-gatekeeper/" rel="noopener noreferrer"&gt;dmytrobot.dev&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>devops</category>
      <category>security</category>
      <category>devsecops</category>
    </item>
  </channel>
</rss>
