Building a Terraform CI/CD Pipeline With GitHub Actions (And Debugging Every Single Thing That Broke)

After getting comfortable with Terraform modules across multiple environments, I gave myself a new scenario. Imagine the CTO of the company sends down a new rule: no engineer runs terraform apply manually against production ever again. Every infrastructure change goes through a pull request. A pipeline validates it, plans it, and only applies once that change is actually merged to main. If something is wrong, it fails inside the pipeline, not in front of customers.

This post is the build of that pipeline, and honestly it turned into one of the most instructive things I have done, mostly because of how many small, sharp mistakes I made along the way and had to actually understand before I could fix them. If you are trying to build your first real CI/CD pipeline for Terraform, I want this to save you some of the pain.

The plan

Two jobs matter most here. A validate and plan job that runs on every pull request, and an apply job that only runs after code lands on main. I reused the module and one environment, payments, from an earlier project of mine, copied into a fresh repo built specifically for this pipeline.

terraform-cicd-pipeline/
├── .github/workflows/terraform.yml
├── environments/payments/
│   ├── providers.tf
│   ├── main.tf
│   └── terraform.tfvars
└── modules/infrastructure/
    ├── main.tf
    ├── variables.tf
    ├── outputs.tf
    └── cloud-init.yaml

Setting up the backend and authentication

Same manual backend pattern as always, since Terraform cannot create the storage it depends on.

az group create --name ci-cd-pipeline-backend-rg --location eastus
az storage account create --name cicdpipestorageacc --resource-group ci-cd-pipeline-backend-rg --location eastus --sku Standard_LRS --kind StorageV2
az storage container create --name ci-cd-container --account-name cicdpipestorageacc --auth-mode login

For authentication, a pipeline should never use a personal login. It needs its own identity, a Service Principal, scoped to what it actually needs to do.

MSYS_NO_PATHCONV=1 az ad sp create-for-rbac \
  --name "github-actions-terraform" \
  --role Contributor \
  --scopes /subscriptions/<subscription-id> \
  --sdk-auth

That MSYS_NO_PATHCONV bit came from a genuinely confusing early mistake. On Git Bash for Windows, any argument starting with a single slash gets silently rewritten into a Windows filesystem path before Azure CLI even sees it. So /subscriptions/xxxx quietly became C:/Program Files/Git/subscriptions/xxxx, and Azure returned a MissingSubscription error that had nothing obviously to do with slashes. That one environment variable prefix tells Git Bash to leave this specific command alone.

The four values that command outputs go into GitHub as repository secrets.

Two different kinds of authentication living in the same workflow

This tripped me up conceptually before it tripped me up technically. There are two separate things reading those same four secrets, doing two different jobs.

The azure/login action logs the Azure CLI itself into the runner. Useful if any step needs to run a plain az command.

Separately, Terraform's own provider and backend need ARM_CLIENT_ID, ARM_CLIENT_SECRET, ARM_SUBSCRIPTION_ID, and ARM_TENANT_ID set as environment variables. Terraform does not automatically inherit whatever the Azure CLI is logged in as. Without these four set explicitly, terraform init fails with a very specific error: authenticating using the Azure CLI is only supported as a user, not a service principal. Both pieces are needed, and neither replaces the other.

The workflow file, and everything wrong with my first draft

I want to be honest about how many small mistakes stacked up in the first version of this file, because I think seeing the actual failures is more useful than seeing a clean finished workflow with no context.

The very first error was a missing comma inside the JSON block that azure/login reads. Plain JSON needs a comma after every property except the last one, and I had none. GitHub Actions caught this before the workflow even ran, which is honestly the best possible time for it to fail.

Other things I found and fixed one at a time across several pushes: capitalized Terraform subcommands like terraform Init that Terraform's CLI does not recognize since subcommands are lowercase only, a case mismatch between a job named Terraform-validate and another job's needs field referencing terraform-validate lowercase, since job IDs are case sensitive, and a typo in an action name, acttions/download-artifact instead of actions, which fails to resolve entirely since GitHub reads that as a literal, nonexistent repository.

The six minute silent hang

This one was the strangest to diagnose because there was no error at all, just a plan step that sat there doing nothing.

The cause was a typo in one of my TF_VAR environment variables, admin_usename instead of admin_username. Since that variable had deliberately been given no default value, based on a rule I had already learned about identity related variables never defaulting silently, Terraform tried to do the right thing and ask for a value interactively. On my own terminal that would show a prompt I could just answer. On a GitHub Actions runner, there is no one there to answer it, so it just waits, forever, with zero error output.

The fix was adding -input=false to both the plan and apply commands. This tells Terraform never to wait for interactive input, ever, and instead fail immediately and loudly if something required is missing. That single flag turned an invisible hang into an instant, readable error the next time a typo happened.

Cancelling that hung run left a real problem behind

Cancelling a stuck workflow does not run Terraform's normal cleanup. It just kills the process. Whatever lock Terraform had acquired on the state file right before it started waiting for input never got released.

The important thing to understand here is that the lock lives on the actual blob in Azure Storage, not inside GitHub Actions itself. So clearing it does not require anything special about CI, it just requires a client authenticated to the same backend.

terraform force-unlock <lock-id>

Run locally, against the same backend config, and the lock clears the same way it would if the stuck operation had come from a teammate's laptop instead of a runner.

A file that only existed on my machine

Once the workflow started actually running plan correctly, it immediately failed on missing required variables I definitely had values for. address_space, subnet_prefix, environment, backend_state_key, all of them living happily in my local terraform.tfvars.

The lesson here is one of the most important things to understand about any CI system. GitHub Actions runs on a completely fresh, disposable machine that has never seen your laptop and never will. It only ever sees what is actually committed and pushed to the repository. My terraform.tfvars was sitting quietly excluded by a default *.tfvars rule in .gitignore, so it existed locally and nowhere else. Since nothing in that particular file was sensitive, just config values, the fix was removing that line from .gitignore and actually committing the file.

SSH keys, and the difference between a path and content

The VM needed an SSH public key, and my module originally read it with file(pathexpand(var.ssh_public_key_path)), the same approach that worked fine on my own machine. It cannot work in CI. That function reads a real file from disk at plan time, and the disposable runner has no ssh folder with my keys in it at all.

The fix was changing the module to accept the key's actual text content as a variable, rather than a path to read. The public key itself is safe to store as a GitHub secret and pass in directly as TF_VAR_ssh_public_key, no file reading involved.

Getting the pipeline to actually deploy a live website

For the VM's own provisioning, update, upgrade, install nginx, clone a portfolio site, I used cloud-init rather than having the pipeline SSH in after the fact. Cloud-init runs automatically the moment the VM boots, which keeps the whole thing declarative instead of needing a private key sitting in my pipeline's secrets.

#cloud-config
package_update: true
package_upgrade: true

packages:
  - nginx
  - git

runcmd:
  - rm -rf /var/www/html
  - git clone https://github.com/highpee1991/cloud-devops-portfolio-repo.git /var/www/html
  - systemctl enable nginx
  - systemctl restart nginx

Two small but completely blocking typos hid in here for a while. First, my header line read hash cloud config with a space, which cloud-init reads as an ordinary comment rather than recognizing the file as a real cloud-config document at all, so it silently did nothing. Second, once that was fixed, I had package-update and package-upgrade with hyphens. Cloud-init's real key names use underscores. That mismatch failed schema validation and quietly skipped the whole final provisioning stage, no nginx, no clone, with almost nothing useful in the general log to point at why.

Tracing an invisible failure all the way to its root cause

Even after fixing both typos, the site still would not load. This is the part I am most proud of in this whole project, because instead of stopping at the first plausible explanation, I actually traced the failure the whole way down.

Browser, to nginx, to an empty var www html folder, to no git repository present, to a cloud-init schema error, to a systemd failure, to the actual root cause: the Linux kernel's OOM killer.

My VM size, Standard B1ls, only has about 393 megabytes of RAM. Asking it to run a full apt upgrade and install two packages unattended, all in one shot, was simply too much. The kernel killed the heaviest process mid operation to protect the system, which is exactly why cloud-init reported finishing with no obvious errors while accomplishing nothing. Sizing up to Standard B2s, with about 3.8 gigabytes of RAM, gave it enough headroom to actually finish the job.

It finally worked

A pipeline that validates, plans, and applies entirely through pull requests and merges, provisioning a real VM that installs its own software and serves a real website, with zero manual terraform apply anywhere in the loop. I tore the infrastructure down after confirming it worked, since there was no reason to keep it running once the point was proven.

What actually stuck with me

Almost every failure in this build came down to the same underlying idea in different clothes. A CI runner is a stranger to your machine. It cannot see your files, your SSH keys, your local Azure login, or anything you have not explicitly given it through a commit or a secret. Once that idea genuinely clicked, most of these bugs stopped being mysterious and started being predictable.

If you want to see the full pipeline, including the workflow file and the cloud-init script:

highpee1991 / terraform-cicd-pipeline

terraform-cicd-pipeline — Runbook

A GitHub Actions pipeline that manages Azure infrastructure through Terraform, so no engineer ever runs terraform apply by hand against production. Every infrastructure change goes through a pull request; the pipeline validates and plans it, and only applies once that change is merged to main.

Scenario

No engineer manually runs terraform apply in production ever again. Every infrastructure change goes through a pull request. The pipeline validates it, posts the plan as a comment, and applies it only when the PR is merged to main. If it fails, it fails in the pipeline, not in production.

Repository layout

terraform-cicd-pipeline/
├── .github/
│   └── workflows/
│       └── terraform.yml
├── environments/
│   └── payments/
│       ├── providers.tf      # backend + provider blocks
│       ├── main.tf           # module call + variable declarations + outputs
│       └── terraform.tfvars  # non-sensitive config values
└── modules/
    └── infrastructure/
        ├── main.tf           # resource

…

View on GitHub

If you are building something similar and hit a wall, drop a comment, happy to help however I can.