During an incident in the spring we needed to raise a connection limit in a hurry, and the full plan for that workspace proposed sixty changes we had not asked for and did not understand at eleven at night. So we ran the apply with a target flag, touched one module, fixed the incident, and went to bed. It was the right call that night.
It was also the last unrestricted apply anybody ran in that workspace for eight weeks.
Nobody decided this. The targeted apply worked, it was fast, and it let each of us change our own corner without reading sixty lines about somebody else's. The flag spread from a runbook step into the pipeline, and then into how the team talked: I'll just target my bit. Every individual run was small, reviewed and correct.
What accumulates behind that is not drift in the usual sense. The code was fine. The state was fine. What we had was a growing set of changes that were merged, approved and unapplied, sitting in the repository looking exactly like changes that had shipped. Two of them had been written by people who had since moved teams.
We found out in July, when we needed to add capacity quickly and the honest version of that work required a full apply. The plan was a hundred and thirty changes, including a replacement of a load balancer nobody could account for, and we could not safely run it during an incident. So we made the change by hand in the console instead, which is the thing infrastructure as code exists to stop, and I want to be clear that I am the one who did it.
We spent two days afterwards applying the backlog in a maintenance window with two people watching, and it broke twice. The pipeline now refuses the target flag unless a break glass input is set, which requires a written reason and posts it to the team channel. Every workspace records the timestamp of its last unrestricted apply, and anything past seven days raises a ticket against the owning team. Two workspaces were already past thirty days when we switched it on.
An escape hatch you use twice is a tool. An escape hatch you use every day is the door, and the door you stopped using is the one you will need in an emergency.
– Sergey Shinder
Top comments (0)