Versioned modules mean nothing if engineers still run apply from their laptops. Laptop applies are unreviewed, unlogged, and unrepeatable. The fix is a pipeline: every change goes through plan on pull request, review, then apply from CI with short-lived credentials. This post builds that pipeline on GitHub Actions from scratch.
Why CI/CD runs apply
Four reasons, in order of how much pain each one prevents.
Review. A plan posted as a PR comment lets reviewers see exactly what will change before it changes. No more "I applied and something broke, let me check what I did."
Audit trail. Every apply is tied to a commit, a PR, and a GitHub Actions run. When something breaks at 2 AM, you know exactly which change did it and who approved it.
No credential sprawl. Engineers do not need cloud credentials on their laptops. The pipeline assumes a role with OIDC. Credentials live for the duration of the run, then vanish.
Repeatability. The same code, the same Terraform version, the same provider versions, every time. "Works on my machine" stops being a sentence anyone says.
Pipeline stages
A solid Terraform pipeline has four stages. Each one gates the next.
-
fmt.
terraform fmt -checkfails the run if any file is misformatted. Cheap, fast, non-negotiable. Formatting arguments are not worth human review time. -
validate.
terraform init -backend=falsefollowed byterraform validate. Catches syntax errors and invalid references without touching state or cloud APIs. -
plan. Full init against the real backend, then
terraform plan -out=tfplan. The plan file is saved as an artifact. On pull requests, the human-readable plan is posted as a PR comment. - apply. Only on merge to main. Downloads the exact plan artifact from the plan stage and applies it. Never re-plans at apply time. You apply what was reviewed, not something subtly different.
Linting with tflint slots in between validate and plan. Security scanning with tfsec or Checkov goes there too.
The plan artifact pattern
The critical detail: apply consumes the saved plan file, not a fresh plan. Here is why it matters.
Between plan and apply, time passes. A reviewer takes an hour. Someone merges another PR. The cloud state drifts. If apply re-plans, it applies something nobody reviewed. By saving the plan with -out=tfplan and applying that exact file, the reviewed change is the applied change. If the world moved underneath, the apply fails loudly instead of silently doing something unexpected. That failure is correct behavior.
Posting plan output as a PR comment
A plan artifact is a binary file. Reviewers cannot read it on GitHub. So the pipeline also renders the plan as text and posts it as a comment:
- name: Render plan
run: terraform show -no-color tfplan > tfplan.txt
working-directory: ${{ env.WORKING_DIR }}
- name: Post plan comment
if: github.event_name == 'pull_request'
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const plan = fs.readFileSync(process.env.WORKING_DIR + '/tfplan.txt', 'utf8');
const body = `## Terraform Plan\n\n\`\`\`\n${plan.substring(0, 60000)}\n\`\`\``;
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: body
});
Long plans get truncated at 60,000 characters, which fits GitHub's comment limits. For most changes, the full plan fits. Reviewers see adds, changes, and destroys inline, right where they already review code.
OIDC auth: no stored secrets
The old way: a long-lived AWS access key or Azure service principal secret sitting in GitHub Secrets, rotating never, shared by everyone. The new way: OpenID Connect. GitHub mints a short-lived token per run. The cloud trusts GitHub's identity provider and hands back temporary credentials scoped to one IAM role.
AWS setup. Create an IAM role with a trust policy that allows GitHub's OIDC provider to assume it, scoped to your repo:
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
},
"StringLike": {
"token.actions.githubusercontent.com:sub": "repo:myorg/infra:*"
}
}
}]
}
The sub condition restricts assumption to your infrastructure repo. Tighten it further with branch conditions if you want: only main can assume the production role, while any branch can assume the dev role.
Azure setup. Register an app, add a federated credential trusting repo:myorg/infra:ref:refs/heads/main, and log in with azure/login@v2 using client ID, tenant ID, and subscription ID. No secret needed.
The workflow needs one permission block:
permissions:
id-token: write
contents: read
pull-requests: write
id-token: write is what lets the run mint the OIDC token. Without it, the credential step fails with an unhelpful error.
Apply gates
Not every merge should auto-apply to production. GitHub Environments give you gates.
Define an environment per target: dev, staging, production. Attach rules:
- dev: auto-apply on merge. Fast feedback, low blast radius.
- staging: apply on merge, but the job waits for no one. Actually, make staging auto-apply too. Staging exists to catch what dev missed, and manual gates on staging just slow the signal.
- production: required reviewers. The apply job pauses until a designated approver clicks approve. The approver sees the plan diff from the PR. This is the gate that matters.
apply-prod:
runs-on: ubuntu-latest
needs: plan-prod
if: github.ref == 'refs/heads/main'
environment: production
concurrency:
group: terraform-prod
cancel-in-progress: false
The environment: production line is what triggers the approval gate. Configure the required reviewers in the repo settings under Environments.
Full working workflow
Here is a complete workflow for one environment. It runs fmt and validate on every PR, plans and comments, and applies only on merge to main behind the production gate.
name: Terraform Prod
on:
push:
branches: [main]
paths: ['envs/prod/**']
pull_request:
paths: ['envs/prod/**']
permissions:
id-token: write
contents: read
pull-requests: write
env:
TF_VERSION: "1.9.0"
WORKING_DIR: envs/prod
jobs:
fmt:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: ${{ env.TF_VERSION }}
- run: terraform fmt -check -recursive
working-directory: ${{ env.WORKING_DIR }}
validate:
runs-on: ubuntu-latest
needs: fmt
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: ${{ env.TF_VERSION }}
- run: terraform init -backend=false
working-directory: ${{ env.WORKING_DIR }}
- run: terraform validate
working-directory: ${{ env.WORKING_DIR }}
plan:
runs-on: ubuntu-latest
needs: validate
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: ${{ env.TF_VERSION }}
terraform_wrapper: false
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789012:role/terraform-ci-prod
aws-region: us-east-1
- run: terraform init -input=false
working-directory: ${{ env.WORKING_DIR }}
- run: terraform plan -out=tfplan -input=false
working-directory: ${{ env.WORKING_DIR }}
- run: terraform show -no-color tfplan > tfplan.txt
working-directory: ${{ env.WORKING_DIR }}
- uses: actions/upload-artifact@v4
with:
name: tfplan-prod
path: ${{ env.WORKING_DIR }}/tfplan
- name: Post plan comment
if: github.event_name == 'pull_request'
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const plan = fs.readFileSync(process.env.WORKING_DIR + '/tfplan.txt', 'utf8');
const body = `## Terraform Plan (prod)\n\n\`\`\`\n${plan.substring(0, 60000)}\n\`\`\``;
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: body
});
apply:
runs-on: ubuntu-latest
needs: plan
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
environment: production
concurrency:
group: terraform-prod
cancel-in-progress: false
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: ${{ env.TF_VERSION }}
terraform_wrapper: false
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789012:role/terraform-ci-prod
aws-region: us-east-1
- uses: actions/download-artifact@v4
with:
name: tfplan-prod
path: ${{ env.WORKING_DIR }}
- run: terraform init -input=false
working-directory: ${{ env.WORKING_DIR }}
- run: terraform apply -input=false tfplan
working-directory: ${{ env.WORKING_DIR }}
Two details worth calling out. First, terraform_wrapper: false on the setup-terraform action. The wrapper captures output for PR comments in the action's own way, which conflicts with the manual plan-comment pattern. Disable it when you post comments yourself.
Second, the apply job downloads the plan artifact and applies that file. It does not run plan again. Reviewed plan in, applied plan out.
Multi-environment promotion
One repo, one workflow per environment, or one workflow with a matrix. The promotion flow:
- Merge to main. Dev applies automatically. No gate, fast feedback.
- Staging applies automatically after dev succeeds. Use
needs: [apply-dev]to serialize. - Production waits. The
environment: productiongate pauses for a human approver.
Sequential, not parallel. Prod never applies before staging succeeds. If staging apply fails, prod never starts, and the team fixes forward instead of debating whether prod is safe.
Keep one workflow file per environment rather than one mega-workflow with conditionals. terraform-prod.yml, terraform-staging.yml, terraform-dev.yml. Each is readable on its own. Shared steps go into composite actions or reusable workflows if duplication bothers you, but duplication across three files is cheaper than one clever file nobody understands.
Concurrency groups: serialize applies per environment
Two merges in quick succession can queue two applies against the same state. State locking makes the second one wait or fail, but it is cleaner to serialize at the workflow level:
concurrency:
group: terraform-prod
cancel-in-progress: false
cancel-in-progress: false is the important part. You want the second run to wait, not to cancel the first run mid-apply. Cancelling mid-apply is how you get tainted resources and half-created infrastructure.
One concurrency group per environment. Dev, staging, and prod apply independently. Two environments never block each other.
State locking and CI
State locking still applies in CI. If a run crashes mid-apply, the lock can stick. The next run fails with a lock error. The fix is the same as locally: terraform force-unlock, but run it deliberately, not in the pipeline.
Do not add auto-unlock to the pipeline. An automatic force-unlock can release a lock held by a genuinely running apply, recreating the race condition locking exists to prevent. When CI reports a stuck lock, a human checks that nothing is actually applying, runs force-unlock once, and re-runs the workflow.
When apply fails mid-run
It happens. A network blip, a cloud API quota, a resource that fails to create. Terraform marks the failed resource as tainted. The next apply destroys and recreates it. Usually that is the right behavior.
When it is not, for example a database that took twenty minutes to provision and failed on a tagging step, you have options:
-
terraform apply -replace="aws_db_instance.main"recreates one specific resource on the next run, without waiting for a failure to taint it. -
terraform state rmfollowed byterraform importdetaches a resource from management when you need to stop Terraform from touching it while you fix things by hand. Use sparingly. - As a last resort,
-targetapplies to a subset of resources. It bypasses dependency ordering guarantees, so treat it as emergency surgery, not routine practice. Every-targetapply should be followed by a full, untargeted plan to confirm the state is coherent.
After any recovery, run a full plan. If the plan is empty, you are back to a clean state. If it is not, keep going until it is.
Checklist
- [ ] fmt, validate, plan, apply as separate gated stages
- [ ] Plan saved with
-out=tfplan, applied from the artifact. No re-planning at apply time. - [ ] Plan output posted as a PR comment for review
- [ ] OIDC auth configured. No long-lived credentials in GitHub Secrets.
- [ ]
id-token: writepermission set on the workflow - [ ] Production apply behind a GitHub Environment approval gate
- [ ] Concurrency group per environment,
cancel-in-progress: false - [ ] Promotion order enforced: dev, then staging, then prod
- [ ] No auto-unlock in the pipeline. Stuck locks get human attention.
- [ ] Terraform, provider, and module versions pinned (see article 2)
Next
The pipeline applies cleanly and every change is reviewed. But infrastructure does not stay as Terraform left it. Someone tweaks a setting in the console. A cloud provider changes a default. The next plan shows diffs nobody made.
Next article: drift detection and remediation. Scheduled scans, telling real drift from cosmetic noise, and the workflow for bringing reality back in line with code.
Top comments (1)
One extra caution with the saved plan artifact: it can contain sensitive values even when Terraform hides them in terminal output. Worth calling out alongside the upload step.