Originally published on kuryzhev.cloud
The scenario
Someone opened the AWS console during an incident and added an inbound rule to a security group to unblock a debugging session. The incident closes, the ticket gets marked resolved, and nobody touches Terraform. Three weeks later a scheduled terraform apply runs, sees that the security group no longer matches the `.tf` config, and either reverts the fix silently or fails with a diff nobody expected. This is terraform drift detection's core problem: state stops reflecting reality, and the gap is invisible until something breaks.
Drift comes from more than console fixes. Another team's automation might tag resources directly through the AWS SDK. Auto-scaling groups rewrite instance counts. Cloud providers backfill default attributes — a default security group rule, a provider-assigned ARN suffix, KMS key rotation metadata — that Terraform never wrote but will happily "fix" on the next apply. Each of these is a small divergence between the state file and the actual infrastructure, and none of them show up until a plan or an incident forces the question.
The cost of unmanaged drift is twofold. First, the next apply can revert an intentional emergency change because Terraform has no way to know it was intentional — it only knows the config doesn't match. Second, plans become noisy and untrustworthy: engineers start ignoring diffs because "there's always something," which is exactly the habit that lets a real, dangerous change slip through unnoticed. The goal here is a scheduled CI job that runs a plan, classifies what it finds, opens a PR or issue for human review, and — only for narrowly scoped, low-risk cases — remediates automatically.
Prerequisites
A few things need to be true before wiring drift detection into CI, or the scheduled job will produce false positives or, worse, false confidence.
Remote state with locking is non-negotiable. Terraform Cloud/HCP, GCS with its native locking, or an S3 backend using native locking (use_lockfile = true, available from Terraform 1.10+) all work. Local state gives no locking and no shared visibility across a team, so it's a bad fit for scheduled runs that might overlap a human-triggered plan or apply. Locking exists specifically to stop a collision from corrupting anything — a lock collision just fails the operation with an error, and terraform plan never writes to the state file in the first place; only apply does. See the Terraform S3 backend documentation for the locking configuration; the older dynamodb_table argument is deprecated as of Terraform 1.11 in favor of native S3 locking.
Pin the Terraform CLI version with required_version and commit .terraform.lock.hcl. If the scheduled CI run resolves a newer provider version than the one used for the last manual plan, the diff will include provider-attribute changes that have nothing to do with real infrastructure drift — a classic false positive that erodes trust in the whole pipeline.
Finally, the CI platform needs scheduled triggers — GitHub Actions schedule:, GitLab CI pipeline schedules, or equivalent — plus credentials for opening PRs or issues, and cloud credentials scoped to read/plan-only for the detection run itself. Even a plan-only role needs write access to the lock: dynamodb:PutItem/dynamodb:DeleteItem on the lock table for the legacy backend, or s3:PutObject/s3:DeleteObject on the lockfile object for native S3 locking. A role with only s3:GetObject and read-only DynamoDB permissions will fail on every single run with a lock-acquisition error, not a clean read-only plan. Apply-capable credentials should be a separate role, gated behind approval, introduced later in the pipeline.
Step 1 — Run scheduled drift detection
The detection job is terraform plan -refresh-only -detailed-exitcode, not a plain plan. A plain plan's exit code 2 fires for both real drift and any unapplied change to the `.tf` config — two very different situations, and lumping them together defeats the point of a drift report. The -refresh-only flag isolates the first case: it refreshes state against real infrastructure and reports only what changed outside Terraform, without folding in pending config edits.
Documented Terraform CLI behavior for -detailed-exitcode returns 0 for no changes, 1 for an error, and 2 when changes are found. That exit code is the branching point for everything downstream — but the workflow has to capture it carefully. GitHub Actions runs step scripts with bash -e by default, and exit code 2 will abort the script before a naive echo "exitcode=$?" on the next line ever runs, leaving every if: condition downstream silently false.
# .github/workflows/drift-detection.yml
name: Terraform Drift Detection
on:
schedule:
- cron: "0 6 * * *" # daily at 06:00 UTC, tune per workspace criticality
workflow_dispatch: {} # allow manual trigger for verification
permissions:
id-token: write # required for OIDC role assumption
contents: write # required to push the remediation branch
pull-requests: write # needed to open the remediation PR
jobs:
detect-drift:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.DRIFT_READONLY_ROLE_ARN }}
aws-region: us-east-1
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: "1.13.1" # pin to match lockfile expectations
- name: Init
run: terraform init -input=false
# -refresh-only isolates real drift from unapplied config changes;
# -detailed-exitcode: 0=no drift, 1=error, 2=drift found
- name: Plan and capture exit code
id: plan
run: |
set +e
terraform plan -refresh-only -detailed-exitcode -out=drift.tfplan
code=$?
echo "exitcode=$code" >> "$GITHUB_OUTPUT"
exit 0
- name: Fail loudly on a real plan error
if: steps.plan.outputs.exitcode == '1'
run: |
echo "terraform plan failed with exit code 1 — this is not drift, it's a broken plan"
exit 1
- name: Export plan as JSON for parsing
if: steps.plan.outputs.exitcode == '2'
run: terraform show -json drift.tfplan > drift.json
- name: Persist the plan for later review and apply
if: steps.plan.outputs.exitcode == '2'
uses: actions/upload-artifact@v4
with:
name: drift-plan-${{ github.run_id }}
path: |
drift.tfplan
drift.json
retention-days: 14
- name: Open remediation PR if drift detected
if: steps.plan.outputs.exitcode == '2'
run: ./scripts/open-drift-pr.sh drift.json drift.tfplan
Run this with a read-only or plan-only IAM role assumed via OIDC through aws-actions/configure-aws-credentials, not the same admin credentials used for apply. A common mistake in early drift-detection setups is reusing broad, apply-capable credentials for the scheduled job — if that role leaks, or the workflow gets misconfigured, a "detection" run could mutate production. terraform show -json against the saved plan gives structured, parseable output instead of scraping plan text with regex, and uploading the plan as a build artifact means the exact plan that was inspected is still around days later, instead of existing only inside one ephemeral runner that's already gone.
Step 2 — Classify and report drift
A raw plan diff dumped into a Slack channel is not actionable — it gets scrolled past. Running terraform show -json against a refresh-only plan produces a resource_drift array specifically for changes found during the refresh; the resource_changes array in the same output mixes those in with changes driven by edits to the `.tf` config, which is not what this pipeline is trying to surface. Read resource_drift, and each entry's change.actions field (create, update, delete, or delete, create for a replacement) is enough to build a structured report without parsing plan text.
Not every non-empty refresh-only plan is drift that needs fixing. Provider-computed or eventually-consistent attributes — certain AWS timestamps, KMS key rotation metadata, some load balancer attributes — generate diffs that are noise, not signal. Filtering these out, or scoping them with ignore_changes, keeps the report trustworthy. If every scheduled run reports "drift" on the same cosmetic field, the team will start ignoring the report entirely, which defeats the purpose.
The report itself should separate cosmetic changes (tags, descriptions) from structural ones (security group rules, IAM policies, instance types, anything touching data stores or networking). That severity tag is what determines whether the change goes through auto-remediation or requires a human to look at it.
# Example classified drift report attached to the remediation PR
Drift summary — workspace: prod-network (2026-01-14T06:00Z)
aws_security_group.app[0] UPDATE severity: structural
ingress: [+] 0.0.0.0/0:22 (added outside Terraform)
aws_instance.web[2] UPDATE severity: cosmetic
tags.CostCenter: "eng" -> "eng-platform"
aws_s3_bucket.logs CREATE severity: structural
(resource deleted outside Terraform — plan wants to recreate it)
Decision:
- structural changes -> require manual review + approval before apply
- cosmetic-only changes -> eligible for auto-merge/auto-apply path
- CREATE drift on a "missing" resource -> confirm intent before applying (recreation risk)
Posting this as a PR comment or a filed issue, rather than raw plan output to chat, gives owners a durable, auditable trail tied to version control — chat messages scroll away, PRs don't.
Step 3 — Remediate via a gated pipeline
The fix for drift should go through the same review path as any other infrastructure change, not another console click. There are two distinct outcomes here, and they need different plan types — collapsing them into one "apply the drift plan" step is how remediation pipelines quietly do the wrong thing.
If the drift should be reverted, applying the refresh-only plan from Step 1 won't do it: terraform apply against a refresh-only plan only updates the state file to match what's actually running, it does not touch real infrastructure. See the refresh-only mode documentation for the exact behavior. Reverting requires a standard terraform plan -out=revert.tfplan run against the now-refreshed state, which computes the actual create/update/destroy actions needed to bring infrastructure back in line with the `.tf` config. Upload that plan with actions/upload-artifact, the same way the detection job does, so the apply job triggered from the remediation PR downloads and applies the exact plan a human reviewed, instead of re-running terraform plan at apply time and reopening a race against whatever else might change the account in the meantime.
If the drift should be kept, don't apply anything — update the `.tf` config to match reality (Step 4 covers this) and let a follow-up plan confirm the diff is gone.
For narrowly scoped, pre-approved categories — tag-only diffs are the usual example — an auto-merge or auto-apply path for the revert plan can be allowed, but only with a strict resource-type allowlist. Anything touching IAM, networking, or a data store should require a human, regardless of how small the diff looks. Gate the apply job behind a protected environment or required reviewers in GitHub, or the equivalent manual gate in whatever CI platform is running this.
Step 4 — Handle unmanageable or intentional drift
Not all drift should be reverted. If the manual security group fix from the incident should stay, the correct move is updating the `.tf` config to match reality and confirming terraform plan comes back clean — not applying over the fix.
Watch out for resources that will always show noise from provider-managed defaults. The `lifecycle { ignore_changes = [...] }` block suppresses drift reporting for the listed attributes, but scope it to specific fields rather than `all`. Ignoring everything on a resource hides real future drift along with the noise, and that resource effectively drops out of drift detection entirely.
Watch out for drift caused by a resource deleted outside Terraform — the next plan shows a `create` action, and reapplying it recreates something that may have been intentionally decommissioned. Run terraform state list and terraform state show <address> to confirm whether the resource is still tracked before deciding whether to recreate, import, or remove it from state. Any `terraform state rm` or import action changes the state file and should go through the same PR review as code, for auditability.
Verify and test
Before trusting the pipeline, induce drift deliberately in a sandbox environment — change a security group rule via CLI or console — and confirm the scheduled job flags it, exits with code 2, and posts the report with the expected resource address and severity tag.
Check the remediation PR's apply step against the plan summary counts (add/change/destroy) captured when the revert plan was generated, to confirm it changes exactly the drifted resource with no unrelated diffs. A revert plan that touches more resources than expected usually points to state that shifted between plan and apply, or to a workspace mix-up in multi-workspace setups — run detection per workspace rather than as one aggregate plan, since an aggregate plan can mask which specific environment actually drifted.
Finally, confirm the read-only detection role genuinely cannot apply. Attempt an intentional apply from the detection credentials and expect an access-denied error. Catching a privilege-scoping mistake in a test run is far cheaper than discovering it because a scheduled job silently mutated production.
Drift detection only pays off if remediation flows through the same reviewed, versioned pipeline as every other infrastructure change — otherwise a team just trades console drift for a new kind of pipeline drift, where auto-applies happen with no one watching. Start with detection-and-report-only for a few weeks, let the team see what kinds of drift actually occur in the environment, and only enable an auto-apply path for the narrow, low-blast-radius categories that show up repeatedly and safely. More CI and infrastructure patterns like this one are covered under DevOps_DayS on kuryzhev.cloud.
Top comments (1)
The
set +earoundterraform plan -detailed-exitcodeis the kind of detail that decides whether the whole pipeline works: exit code 2 is the success path of the detection job, yet it's the same code that aborts a default bash step — so the naive version fails closed into silence and every downstreamif:reads false. Classifying drift into structural vs cosmetic before anyone touches a PR is the other half; "engineers start ignoring diffs because there's always something" is exactly the failure we've seen in alerting, transplanted into IaC.Two questions from running automation against a shared live system: how noisy has the cosmetic bucket been in practice, and did auto-merge for cosmetic-only drift survive first contact — say, a provider backfilling an ARN suffix the week your schema version moved? And the deliberately-induced-drift test you prescribe: does it run continuously against staging, or was it a one-time validation? The day detection breaks silently is the day it matters.