The apply we interrupted and the resources state never heard about

Our apply job had a thirty minute timeout because that had always been enough. The afternoon we added a managed NAT gateway and two subnet groups to the networking workspace, it was not. The runner was killed at minute thirty with the apply roughly two thirds through, and everything after that was our own doing.

Terraform does not roll anything back when you kill it. Resources that were created before the process died exist in the cloud; whether they exist in state depends on whether the state write completed. Our lock was still held in DynamoDB, because the process that owned it never got to release it. The next run refused to start, someone ran terraform force-unlock from a laptop, and the plan that followed proposed creating a NAT gateway and a subnet group that were already sitting in the account.

Half of that apply failed loudly with "already exists", which is the good case. The NAT gateway is the bad case: it has no unique name to collide on, so a second one was created quite happily, attached to the same subnet, doing nothing. It ran for nine days before it turned up in a cost review at roughly forty dollars a week, and the only reason we found the orphan and not the duplicate-in-use was that we had tagged everything with the workspace name.

What changed. Apply timeouts are now set from the measured p99 of that workspace's applies plus a wide margin, and the apply step is marked so a cancelled workflow does not kill it mid-flight. force-unlock is not something anybody runs to make a red pipeline go away: after any interrupted apply the procedure is to refresh, diff the plan against the account, and import anything that exists but is not in state, before a single further change is applied.

We also run a weekly sweep that lists resources carrying our workspace tag and compares them against state, which is how you find things Terraform has forgotten it owns.

An interrupted apply is not a no-op that you retry. It is a partial change to the world, and the record of it is the one thing that did not finish.

– Sergey Shinder