Originally published on kuryzhev.cloud
Context
Our terraform compliance scanning in GitLab CI looked like a success story for the better part of a year. We had roughly 40 Terraform modules spread across a multi-account AWS setup, a growing team of engineers who all had merge rights, and zero policy enforcement before this project started. Anyone could ship a public S3 bucket, an open security group, or an unencrypted EBS volume straight to prod, and nobody would know until an audit — or an incident — surfaced it.
The initial goal was simple: wire up checkov into GitLab CI and block non-compliant infrastructure before merge. Public buckets, wide-open ingress rules, unencrypted volumes — stop them at the pull request, not after `terraform apply`. On paper, this was a two-week project. We added a job, wired it into the pipeline, and declared victory.
It worked. Just not the way we thought it would. Eight months later, during a security review, we discovered the pipeline had been effectively decorative for most of that time. This post is the honest version of that story — three mistakes that quietly undid the entire point of the scan, and what we changed afterward.
Mistake 1: We made the scan advisory "just for the rollout period"
When we first rolled out checkov, we knew some existing modules would fail checks immediately. Rather than block every team on day one, we added allow_failure: true to the compliance job. The plan was to run it in "observe mode" for a sprint or two, let teams clean up their modules, then flip it to blocking.
Nobody created a ticket for that flip. Nobody owned the decision. There was no metric tracked — no dashboard showing "percentage of MRs now passing cleanly" that would tell us when it was safe to enforce. The rollout period just… continued. Indefinitely.
Eight months in, we pulled pipeline stats and found the checkov job was red on more than 60% of merge requests. Engineers had stopped looking at it entirely — a red job that never blocks anything is just noise, and people are very good at tuning out noise. We had built a compliance pipeline that scanned everything and stopped nothing.
The root cause wasn't technical. It was organizational: allow_failure: true is a perfectly reasonable transitional state, but it needs an expiry date and an owner from day one, not a vague intention to "come back to it." We didn't have that, so it never happened.
Mistake 2: We scanned raw .tf source instead of the plan output
The second mistake was more subtle, and honestly more embarrassing once we understood it. We were running checkov directly against our .tf files — the raw HCL source — instead of the resolved plan. That gave us a false sense of security, because static source scanning only sees the template, not what actually gets created.
We had a real incident that exposed this. A public S3 bucket passed scanning cleanly because its acl attribute was set through a variable that resolved differently per environment. In the staging .tfvars it was private; in production it resolved to public-read. Checkov, scanning the source file, saw a variable reference and had no idea what it would actually become at plan time. The bucket shipped to production, public, with a green pipeline behind it.
This is a known limitation of source-level scanning, and worth calling out explicitly: anything resolved through variables, module defaults, or provider-level defaults is invisible to a tool that only reads HCL text. The fix — which I'll get into in the last section — is to scan the actual resolved plan, generated with:
terraform show -json plan.tfplan > plan.jsonThat JSON is what the provider will actually create. It's the only thing worth scanning if you care about what ends up in AWS, not what the template author intended.
Mistake 3: We let policy exceptions grow with zero audit trail
Checkov supports inline skip comments — #checkov:skip=CKV_AWS_20:reason — and we let engineers add them freely, with no review requirement and no expiry. It seemed harmless at the time. A team needed to ship something fast, added a skip comment with a half-written reason, and moved on.
Six months in, a security review counted more than 40 skip comments scattered across our repos. Several were for checks that genuinely mattered: unencrypted RDS instances, security groups open to 0.0.0.0/0. Nobody could explain why half of them existed. The original author had either left the company or simply forgotten the context. One skip had disabled an RDS encryption check for five months with no ticket link and no owner attached — a silent gap that could easily have gone unnoticed for years.
Watch out for this specifically: a skip comment without a reason, owner, and expiry isn't an exception — it's permanent technical debt disguised as a temporary workaround. If your policy tooling lets people self-approve exceptions with a code comment, you don't actually have a policy. You have a suggestion.
What we do differently now
We rebuilt the pipeline around four principles, and this is what runs in production today, not an aspirational roadmap.
Staged enforcement with a real deadline. New rules start as warn for two sprints. The removal date — when it flips to deny — goes directly into the MR template that introduced the rule, so it's tracked like any other commitment, not a vague intention.
Scan the plan, not the source. We moved from checkov-on-HCL to Conftest and OPA/Rego against terraform show -json output, run as a distinct stage after terraform plan. This catches what actually gets created, including anything resolved through variables or module defaults. We also pinned Terraform to 1.7.x across every scanned repo — a mismatched local vs. CI version produced inconsistent plan JSON structure and gave us false negatives more than once.
One exceptions file, validated in CI. All skips now live in compliance/exceptions.yaml with mandatory reason, owner, and expires_on fields. A dedicated job fails the pipeline if any entry is expired. No more inline comments nobody reviews.
One shared policy template. Rules and severity thresholds live in a central included template so every repo inherits the same baseline, rather than each team maintaining its own drifted copy.
Here's the pipeline as it actually runs today:
# .gitlab-ci.yml — staged Terraform compliance pipeline
# Scans the resolved plan JSON, not raw HCL, and enforces exception TTLs
stages:
- plan
- compliance
- apply
variables:
TF_VERSION: "1.7.5"
CHECKOV_VERSION: "3.2.6" # pinned — unpinned versions caused silent rule changes
CONFTEST_VERSION: "0.55.0"
terraform-plan:
stage: plan
image: hashicorp/terraform:${TF_VERSION}
script:
- terraform init -input=false
- terraform plan -out=plan.tfplan -lock=false # ephemeral runner, avoid lock conflicts
- terraform show -json plan.tfplan > plan.json # scan THIS, not *.tf files
artifacts:
paths:
- plan.json
expire_in: 1 hour
rules:
- changes:
- "**/*.tf"
validate-exceptions:
stage: compliance
image: python:3.11-slim
script:
- pip install pyyaml
- python compliance/validate_exceptions.py compliance/exceptions.yaml
# fails if any exception's expires_on has passed
conftest-scan:
stage: compliance
image: openpolicyagent/conftest:${CONFTEST_VERSION}
needs: ["terraform-plan"]
script:
- conftest test plan.json -p policy/critical --output json > critical-results.json
- conftest test plan.json -p policy/warn --output json > warn-results.json || true
# critical policies BLOCK, warn policies report only
artifacts:
paths:
- critical-results.json
- warn-results.json
when: always
allow_failure: false # deliberately NOT true — this was the mistake we fixedAnd the exceptions file that replaced scattered skip comments:
# compliance/exceptions.yaml
# Every exception must have owner + expiry, or the CI validation job fails the pipeline
exceptions:
- rule_id: CKV_AWS_20
resource: aws_s3_bucket.logs
reason: "Public read required for CloudFront OAC migration, tracked in INFRA-482"
owner: "j.smith"
expires_on: "2025-03-01"
- rule_id: CKV_AWS_8
resource: aws_instance.legacy_bastion
reason: "Legacy host scheduled for decommission, see INFRA-410"
owner: "team-platform"
expires_on: "2025-01-15"
# --- example conftest output that triggers a pipeline failure ---
# FAIL - plan.json - main - S3 bucket "logs" allows public ACL (CKV_AWS_20)
# FAIL - plan.json - main - Security group "app-sg" allows 0.0.0.0/0 on port 22
#
# 2 tests, 0 passed, 2 failed, 0 warningsOne more gotcha worth flagging: we originally scanned every Terraform directory on every MR, which was slow and wasted GitLab SaaS runner minutes we're billed for per-minute. Scoping the plan job with GitLab CI's rules: changes: to only touch modified directories cut pipeline time from around 7 minutes to under 90 seconds on our monorepo, and cut CI minutes by roughly 80%. It's a small config change with an outsized cost impact.
None of this was rocket science. It was mostly about admitting that "temporary" pipeline flags and self-service exceptions need the same rigor as production code — an owner, a deadline, and something automated checking that the deadline was respected. If you're setting up terraform compliance scanning in GitLab CI from scratch, save yourself the eight months and build the enforcement deadline into the pipeline on day one. If you want more on how we structure GitLab CI pipelines generally, we've covered related patterns over at kuryzhev.cloud.
Top comments (0)