Drift Detection and Remediation: Keeping Production Honest

Terraform state is a snapshot. Reality moves on. The gap between what Terraform thinks exists and what actually exists is called drift, and every team running Terraform in production deals with it. This post covers how drift happens, how to detect it automatically, how to fix it when you find it, and how to prevent most of it in the first place.

What drift actually is

When you run terraform apply, Terraform writes the real-world state of every resource into the state file. From that moment, the state file starts going stale. Anything that changes infrastructure outside of Terraform creates drift: the state says one thing, the cloud says another.

A little drift is normal. A lot of drift is dangerous, because the next terraform apply will try to reconcile the difference, and that reconciliation can surprise you. If someone manually resized a database in the console and your next apply scales it back down, you get an unplanned outage wearing the disguise of a routine deploy.

How drift happens

Drift has three usual sources.

Console ClickOps. Someone fixes something urgent at 2 AM in the cloud console. The fix works. Nobody writes it down. The Terraform code still describes the old configuration. This is the most common source and the hardest to eliminate, because the console exists for a reason: sometimes you need it.

Out-of-band automation. A Lambda function that adjusts autoscaling limits. A cron job that rotates something. Another team's pipeline that touches shared resources. These are not people making mistakes. They are systems doing their jobs. But if they modify resources Terraform manages, the result is drift all the same.

Other teams touching shared infrastructure. The security team tightens a security group. The networking team updates a route table. Both changes are legitimate. Neither team knew Terraform owned those resources, because nothing told them.

Not all drift is bad. Sometimes the manual change was the right call and the code was wrong. The problem is never the change itself. The problem is undocumented change that nobody reconciles.

Using plan as a drift detector

Every terraform plan already does drift detection. Before comparing config to state, Terraform refreshes: it queries the real infrastructure and updates its picture. Any difference between the refreshed reality and the state file shows up in the plan output.

For dedicated drift checks, use refresh-only mode:

terraform plan -refresh-only

This updates the state to match reality and shows you what changed, without proposing any config-driven modifications. It answers one question: what moved outside of Terraform since the last apply?

For automation, add -detailed-exitcode:

terraform plan -refresh-only -detailed-exitcode -no-color

The exit codes are the whole interface: 0 means no drift, 1 means an error occurred, 2 means drift was detected. CI pipelines branch on these codes. That is all you need to build scheduled detection.

Scheduled drift detection in CI

Drift comes from outside Terraform, so it will never be caught by your normal pull request pipeline. You need a separate scheduled job whose only purpose is asking: does reality still match the code?

Here is a nightly workflow using GitHub Actions. It runs a read-only refresh plan against every environment and reports the result:

name: Nightly Drift Detection

on:
  schedule:
    - cron: '0 6 * * *'  # 6 AM UTC daily
  workflow_dispatch:      # allow manual runs too

permissions:
  contents: read
  id-token: write        # for OIDC auth to cloud

jobs:
  drift-check:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        environment: [dev, staging, prod]
    steps:
      - uses: actions/checkout@v4

      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.9.8

      - name: Configure cloud credentials (read-only role)
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/terraform-drift-reader
          aws-region: us-east-1

      - name: Terraform init
        working-directory: envs/${{ matrix.environment }}
        run: terraform init -input=false

      - name: Refresh-only plan
        id: drift
        working-directory: envs/${{ matrix.environment }}
        run: |
          set +e
          terraform plan -refresh-only -detailed-exitcode -no-color -input=false \
            > drift-output.txt 2>&1
          echo "exit_code=$?" >> "$GITHUB_OUTPUT"
          set -e
          cat drift-output.txt

      - name: Upload drift report
        if: steps.drift.outputs.exit_code == '2'
        uses: actions/upload-artifact@v4
        with:
          name: drift-report-${{ matrix.environment }}
          path: envs/${{ matrix.environment }}/drift-output.txt

      - name: Notify Slack
        if: steps.drift.outputs.exit_code == '2'
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.DRIFT_SLACK_WEBHOOK }}
        run: |
          curl -s -X POST "$SLACK_WEBHOOK_URL" \
            -H 'Content-Type: application/json' \
            -d "{\"text\":\":warning: Drift detected in *${{ matrix.environment }}*. See the workflow run for the full report.\"}"

A few design decisions worth noting:

Read-only credentials. The drift job only needs to read infrastructure and state. Give it a dedicated IAM role with read permissions. If the job can only read, it cannot make anything worse, which matters when it runs unattended at 6 AM.

Matrix over environments. One job definition, every environment checked. Prod drift and dev drift both get reported. You decide which ones page someone.

Artifacts, not just logs. When drift is found, the full plan output is saved as a workflow artifact. The person investigating gets the complete report, not a Slack message saying "something changed."

Never auto-remediate from this job. It is tempting to follow detection with an automatic terraform apply. Do not. Remediation is a judgment call (see below). The job detects and reports. A human decides.

Reading a drift report

Refresh-only output has a distinctive header. Learn to recognize it:

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply":

  # aws_instance.web has changed
  ~ instance_type = "t3.medium" -> "t3.large"

  # aws_security_group.app has changed
  ~ ingress = [
      + {
          + cidr_blocks = ["10.0.0.0/8"]
          + from_port   = 443
          + to_port     = 443
          + protocol    = "tcp"
        },
      ]

Read it like this: the left side is what Terraform last recorded, the right side is what exists now. Someone resized the instance from medium to large and added an ingress rule.

The skill is telling meaningful drift from noise. Some attributes always show diffs because the provider computes them fresh on every refresh (timestamps, generated IDs, default values the API fills in). If you see the same handful of attributes flagging every night with no real change, that is provider noise, not drift. Note which ones they are so the team stops investigating them. Real drift is a value someone deliberately changed: sizes, counts, rules, policies.

For machine-readable reports, terraform show -json on a saved plan file gives you structured output you can feed into ticketing or dashboards. Most teams do not need this until drift volume gets high. Start with the human-readable report.

Remediation: three options

Finding drift is the easy part. Deciding what to do about it is the real work. There are three options, and the right one depends on whether the manual change was correct.

Option 1: Apply to converge. If the manual change was wrong or temporary, run a normal terraform apply. Terraform reverts reality to match the code. This is the right call for accidental console edits and for emergency fixes whose emergency is over. Announce it first: reverting someone's change without warning creates a different kind of incident.

Option 2: Import the unmanaged resource. If someone created something entirely outside Terraform that should be managed going forward, bring it into state instead of destroying it:

# Write the resource block first, matching reality as closely as you can
resource "aws_s3_bucket" "logs" {
  bucket = "myorg-manual-logs-bucket"
}

terraform import aws_s3_bucket.logs myorg-manual-logs-bucket
terraform plan

The plan after import shows you every attribute where your config differs from reality. Adjust the config until the plan is clean. The resource is now managed, and the next apply changes nothing. Import is for things that should have been in code all along.

Option 3: Update the code to accept the change. Sometimes the console change was right and the code was wrong. The instance needed to be larger. The firewall rule was genuinely missing. In that case, update the Terraform configuration to describe the new reality, then apply. The apply is a no-op, the drift is resolved, and the change is now documented in version control where it belongs.

The decision framework is one question: was the manual change correct? If yes, codify it (option 2 or 3). If no, revert it (option 1). If nobody knows, find out before touching anything. Reverting a change you do not understand is how you turn drift into an outage.

Preventing drift with guardrails

Detection and remediation are necessary. Prevention is better. The most effective guardrail is also the simplest: most humans should not be able to change production infrastructure directly.

IAM scoping. Engineers get read access to production. The CI/CD pipeline role gets write access. This single change eliminates the majority of accidental drift, because the console becomes a viewing tool rather than an editing tool for everyday work. Break-glass admin access still exists for real emergencies, with the understanding that anything done through it gets codified afterward.

AWS Service Control Policies. For destructive actions, add an organization-level backstop that denies them to everyone except the pipeline role:

resource "aws_organizations_policy" "prod_guardrail" {
  name = "deny-prod-changes-outside-pipeline"
  type = "SERVICE_CONTROL_POLICY"

  content = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "DenyDestructiveActionsOutsidePipeline"
        Effect = "Deny"
        Action = [
          "ec2:TerminateInstances",
          "rds:DeleteDBInstance",
          "s3:DeleteBucket",
        ]
        Resource = "*"
        Condition = {
          StringNotEquals = {
            "aws:PrincipalArn" = "arn:aws:iam::123456789012:role/terraform-pipeline"
          }
        }
      },
    ]
  })
}

SCPs cannot distinguish Terraform from the console. They distinguish principals. The pattern is: only the pipeline role may perform these actions, everyone else is denied at the organization level. Be honest about the tradeoff: this also blocks legitimate emergency work, which is why break-glass procedures must exist alongside it.

Azure equivalents. Resource locks (CanNotDelete) on critical scopes, Azure Policy deny assignments for specific actions, and RBAC scoped so humans hold Reader on production while the pipeline service principal holds Contributor. Note the same tradeoff as SCPs: locks block Terraform too, so the pipeline must manage lock lifecycle or you scope locks to the resources where deletion would be catastrophic.

Guardrails do not replace the drift job. They reduce the volume. Run both.

Alerting without the noise

A drift alert that fires constantly gets ignored, and an ignored alert is worse than no alert because it creates false confidence. A few rules:

Alert on prod immediately, every time. Aggregate dev and staging into a daily digest. Nobody needs a 3 AM page for dev drift.

Include the environment, a link to the workflow run, and the artifact with the full report. "Drift detected in prod" with no link is an alert that creates work instead of directing it.

Track who owns remediation. The team that owns the Terraform code owns the drift in it. If the drift came from another team's automation, the alert should route to both teams, because the fix might be a conversation, not a command.

The cultural piece

Tools catch drift. Culture prevents it. The goal is a team where console changes to Terraform-managed infrastructure feel wrong, not because anyone gets punished, but because everyone has seen what undocumented changes cost.

Blameless matters here. If someone fixes prod in the console at 2 AM and the response is blame, the next person will fix it in the console and not tell anyone. If the response is "good catch, let's get it into code today," people report their own drift.

Have an explicit break-glass procedure: who can do it, how it gets recorded, and who codifies it afterward. If the only way to fix production in an emergency is the console, that is a process gap, not a people problem. Close the gap and the culture follows.

Pre-flight checklist

• [ ] Nightly (or daily) refresh-only plan job running against every environment
• [ ] Drift job uses dedicated read-only credentials
• [ ] -detailed-exitcode drives branching: 0 clean, 1 error, 2 drift
• [ ] Drift reports saved as artifacts, linked from alerts
• [ ] Prod drift alerts immediately; non-prod aggregates into a digest
• [ ] Team knows the three remediation options and when each applies
• [ ] IAM scoped so humans read prod and only the pipeline writes it
• [ ] Guardrails (SCPs, Policy, locks) on destructive actions, with break-glass documented
• [ ] Break-glass procedure written down: who, how it is recorded, who codifies after

Closing

Drift is not a sign your team is careless. It is a sign your infrastructure is alive: people and systems touch it for good reasons. The mature response is not zero drift. It is detected drift, understood drift, and remediated drift, with guardrails shrinking the volume over time.

Next week: Terraform workspaces versus directory-per-environment. Both organize multiple environments, both have sharp edges, and picking the wrong one for your team size creates the kind of mess that takes a quarter to unwind.