Terraform state describes what you think is deployed. Drift is the gap between that and reality: someone changes a security group in the console during an incident, an autoscaling event edits a tag, or a teammate runs apply from a stale branch. Drift is rarely dramatic on the day it happens. It hurts later, when an unrelated change produces a plan full of surprises and nobody is sure which side is right.
The fix is cheap: run terraform plan on a schedule, read the exit code, and open an issue when something differs. This post builds that pipeline in GitHub Actions, authenticates to AWS with OIDC so there are no stored access keys, and fans out across multiple workspaces with a matrix.
What we are building
- An IAM OIDC identity provider for GitHub and a read-only IAM role that only this repository's workflow can assume.
- A scheduled workflow that runs
terraform plan -detailed-exitcode -lock=falsenightly. - A matrix that checks several environments (directories or workspaces) in parallel.
- An issue per environment when drift is detected, updated instead of duplicated on later runs.
Step 1: The OIDC provider and drift-check role
GitHub Actions can request a short-lived OIDC token that AWS STS exchanges for temporary credentials. You define the trust once, in Terraform:
data "tls_certificate" "github" {
url = "https://token.actions.githubusercontent.com/.well-known/openid-configuration"
}
resource "aws_iam_openid_connect_provider" "github" {
url = "https://token.actions.githubusercontent.com"
client_id_list = ["sts.amazonaws.com"]
thumbprint_list = [data.tls_certificate.github.certificates[0].sha1_fingerprint]
}
data "aws_iam_policy_document" "drift_trust" {
statement {
actions = ["sts:AssumeRoleWithWebIdentity"]
principals {
type = "Federated"
identifiers = [aws_iam_openid_connect_provider.github.arn]
}
condition {
test = "StringEquals"
variable = "token.actions.githubusercontent.com:aud"
values = ["sts.amazonaws.com"]
}
# Only the main branch of this one repository.
condition {
test = "StringEquals"
variable = "token.actions.githubusercontent.com:sub"
values = ["repo:my-org/infra:ref:refs/heads/main"]
}
}
}
resource "aws_iam_role" "drift_check" {
name = "github-terraform-drift-check"
assume_role_policy = data.aws_iam_policy_document.drift_trust.json
max_session_duration = 3600
}
If an OIDC provider for GitHub already exists in the account, reference it with a data "aws_iam_openid_connect_provider" block instead of creating a second one; AWS allows only one provider per URL. AWS also now validates GitHub's certificate chain itself, so the thumbprint is mostly a formality, but the provider resource still requires a value in older provider versions.
The sub condition is the important part. Without it, any repository on GitHub could assume the role. Pin it to the exact repo and ref. If you trigger from a GitHub environment instead, the subject becomes repo:my-org/infra:environment:prod.
Permissions: plan needs to read, not write
terraform plan refreshes every resource, so the role needs broad read access plus access to the state backend:
resource "aws_iam_role_policy_attachment" "read_only" {
role = aws_iam_role.drift_check.name
policy_arn = "arn:aws:iam::aws:policy/ReadOnlyAccess"
}
data "aws_iam_policy_document" "state_read" {
statement {
actions = ["s3:GetObject", "s3:ListBucket"]
resources = [
"arn:aws:s3:::my-org-tf-state",
"arn:aws:s3:::my-org-tf-state/*",
]
}
statement {
actions = ["dynamodb:GetItem", "dynamodb:DescribeTable"]
resources = ["arn:aws:dynamodb:us-east-1:123456789012:table/tf-locks"]
}
}
resource "aws_iam_role_policy" "state_read" {
name = "terraform-state-read"
role = aws_iam_role.drift_check.id
policy = data.aws_iam_policy_document.state_read.json
}
ReadOnlyAccess is wide, and it includes reads of things like S3 objects. If that is too much for your account, start with it, then tighten using the IAM Access Analyzer policy generation covered in our least-privilege post. Because the role cannot write, a compromised workflow can look but not change anything. We run plan with -lock=false so the role never needs to write lock entries; a read-only plan cannot corrupt state.
Note that some data sources and providers read secrets (for example aws_secretsmanager_secret_version). If your configuration uses them, the role needs that access too, and the plan output may contain sensitive values. Keep plan output out of public issues.
Step 2: Understanding -detailed-exitcode
By default terraform plan exits 0 whether or not changes exist. With -detailed-exitcode:
| Exit code | Meaning |
|---|---|
0 |
Succeeded, no changes |
1 |
Error |
2 |
Succeeded, changes present (drift or unapplied code) |
This is what lets a pipeline distinguish "drift" from "broken". One subtlety: exit code 2 also fires when main contains merged code that was never applied. That is still worth an alert, since it means main doesn't match production.
Step 3: The scheduled workflow with a matrix
Assume a repository layout with one root module per environment:
envs/
dev/
staging/
prod/
.github/workflows/drift.yml:
name: terraform-drift
on:
schedule:
- cron: "17 2 * * *" # 02:17 UTC daily; off the hour to avoid queue spikes
workflow_dispatch: {}
permissions:
id-token: write # required to request the OIDC token
contents: read
issues: write
concurrency:
group: terraform-drift
cancel-in-progress: false
jobs:
plan:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
env: [dev, staging, prod]
defaults:
run:
working-directory: envs/${{ matrix.env }}
steps:
- uses: actions/checkout@v4
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789012:role/github-terraform-drift-check
aws-region: us-east-1
role-session-name: drift-${{ matrix.env }}-${{ github.run_id }}
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.9.8
terraform_wrapper: false # keep real exit codes
- name: terraform init
run: terraform init -input=false
- name: terraform plan
id: plan
run: |
set +e
terraform plan -detailed-exitcode -lock=false -input=false -no-color -out=tfplan > plan.txt 2>&1
code=$?
set -e
echo "exitcode=$code" >> "$GITHUB_OUTPUT"
if [ "$code" -eq 1 ]; then
cat plan.txt
exit 1
fi
terraform show -no-color tfplan | grep -E '^\s*# .* (will|must) be' > summary.txt || true
cat summary.txt
- name: Open or update drift issue
if: steps.plan.outputs.exitcode == '2'
env:
GH_TOKEN: ${{ github.token }}
ENV_NAME: ${{ matrix.env }}
run: |
title="Terraform drift detected: ${ENV_NAME}"
{
echo "Scheduled plan found changes in \`envs/${ENV_NAME}\`."
echo
echo "Run: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
echo
echo '```
{% endraw %}
'
head -c 60000 summary.txt
echo '
{% raw %}
```'
} > body.md
existing=$(gh issue list --repo "$GITHUB_REPOSITORY" --state open \
--label drift --search "in:title \"${title}\"" --json number --jq '.[0].number')
if [ -n "$existing" ]; then
gh issue comment "$existing" --repo "$GITHUB_REPOSITORY" --body-file body.md
else
gh issue create --repo "$GITHUB_REPOSITORY" --title "$title" \
--label drift --body-file body.md
fi
Create the drift label once (gh label create drift --color B60205) or gh issue create --label will fail.
A few details that matter:
-
terraform_wrapper: false. The setup-terraform wrapper intercepts output and can mask exit codes in some versions. Turning it off keeps$?honest. -
set +earound plan. GitHub's default bash runs with-e, so exit code2would fail the step before you can read it. -
Summary, not full plan. The issue gets only the resource-level lines (
# aws_security_group.web will be updated in-place). The full plan can contain sensitive attributes. Engineers can rerun locally or read the job log. -
fail-fast: false. A brokendevshouldn't cancel theprodcheck. - Dedupe. The workflow comments on an existing open issue rather than opening a new one every night. Close the issue when drift is resolved and the next detection opens a fresh one.
Using Terraform workspaces instead of directories
If you use one root module with terraform workspace, swap the matrix and select the workspace after init:
strategy:
fail-fast: false
matrix:
workspace: [dev, staging, prod]
steps:
# ...checkout, credentials, setup-terraform as above...
- run: terraform init -input=false
- run: terraform workspace select ${{ matrix.workspace }}
- name: terraform plan
run: terraform plan -detailed-exitcode -lock=false -input=false -no-color -var-file=${{ matrix.workspace }}.tfvars
Alternatively, set TF_WORKSPACE: ${{ matrix.workspace }} as an env var on the job, which Terraform reads automatically.
If environments live in separate AWS accounts (recommended), use include to give each its own role:
matrix:
include:
- env: dev
role: arn:aws:iam::111111111111:role/github-terraform-drift-check
- env: prod
role: arn:aws:iam::222222222222:role/github-terraform-drift-check
and pass role-to-assume: ${{ matrix.role }}. Each account gets its own OIDC provider and role with the same trust policy.
Triage: what to do when the issue opens
Drift has three common causes, and each has a different fix:
- Out-of-band change that should stay. Codify it in Terraform, open a PR, and the next plan goes clean.
-
Out-of-band change that should not stay. Run
terraform applythrough your normal pipeline to revert it. -
Noise. Attributes that AWS or another controller manages (autoscaling desired count, tags added by other tools). Use
lifecycle { ignore_changes = [...] }narrowly, on the specific attribute, with a comment explaining why.
To see only drift without your pending code changes, run terraform plan -refresh-only. It shows what changed in the real world compared to state, without diffing against configuration.
Limits worth knowing
- Drift detection only covers resources Terraform manages. Things created entirely by hand are invisible to it; AWS Config or a tagging policy covers that gap.
- A nightly schedule means drift can exist for up to a day. For security-sensitive resources, pair this with real-time alerts (EventBridge on CloudTrail events).
- GitHub disables scheduled workflows in repositories with no activity for 60 days. A quiet infra repo can silently stop checking, so watch for that.
Wrap-up
With an OIDC role scoped to one repo and branch, a read-only permission set, and -detailed-exitcode, drift detection becomes a small, boring workflow that runs every night and only speaks up when something is off. There are no keys to rotate, and each environment's status shows up as an issue your team already knows how to triage.
Written by the ByteBridge Talent LLC software engineering team.
ByteBridge Talent LLC is an E-Verify enrolled tech staffing firm in Grand Island, Nebraska: bytebridgetalent.com
Top comments (0)