An environment was two settings that had to agree

At 10:20 on a Wednesday we resized a production database down to two vCPUs. The pull request was a staging change. The plan attached to it was a staging plan. The apply ran against production, and nothing in any log looked anomalous, because every component did precisely what it had been configured to do.

Our infrastructure repository had one root module and selected an environment through two independent inputs. A backend config chose which state file to load. A variables file chose the values to put in it. A refactor of the CI script three weeks earlier had moved the backend selection into a step that read a repository variable, while the variables file was still derived from the directory that appeared in the diff. For every staging change since, the two had agreed. That morning the repository variable had been left pointing at prod by an unrelated manual run the evening before.

So Terraform loaded production state and staging values and produced exactly the plan you would expect. Shrink two instance classes, take a node group from twelve to three, remove a pair of alarms. Thirty-one changes. The reviewer scrolled it, saw the word staging in the pull request title, and approved. Twenty minutes of degraded checkout later we worked out what we were looking at.

Three things changed. Environments are separate directories now, each with its backend hardcoded in the file: no dynamic init, no workspace selection, no variables file switching. It costs some duplication and it removes a whole category of mistake. Plan and apply became two jobs, with apply consuming the plan file as an artifact rather than planning again, so what was reviewed is what runs. And each environment directory carries a committed file naming its account id and region, which a preflight step compares against the credentials the job is actually holding, failing the run if they disagree.

That last check is the one I would add first if I could only add one. Fifteen lines, and it turns an invisible mismatch into a red job before anything touches the world.

An environment was never one thing in our setup. It was two things that had to agree, and nothing anywhere was comparing them.

– Sergey Shinder