The monitoring collector lost its connection to the payments subnet on a Monday. Someone re-added the ingress rule, monitoring came back, and by mid afternoon it was gone again. That went on for four days before anyone treated it as a fault rather than an inconvenience.
The provider audit log showed two identities taking turns. One was the platform team's automation role, the other was the payments team's. Both were us. The platform repository declared the security group with its full set of ingress rules inline. The payments repository declared a standalone rule resource attached to the same group, added a year ago because waiting on a platform pull request was slow. Each state file was internally consistent and each plan was small and correct from where it stood. The platform apply ran nightly and removed a rule it had no record of. The payments drift job ran every three hours and put it back.
Inside a single state, the dependency graph is the arbiter and it is a good one. Across two states there is no arbiter at all. Neither tool was wrong, neither plan looked suspicious, and the only place the conflict was visible was in a cloud provider log nobody reads on a normal day.
We fixed the rule by moving it into the platform module behind an input variable, which took twenty minutes and is not the interesting part. The interesting part was finding the others. We added a scheduled plan only run in every workspace that reports any non empty plan to the owning team, on the principle that a workspace which has converged should propose no changes at all. Two more collisions turned up in a fortnight: an IAM policy attachment declared in two places, and a DNS record that a deployment pipeline was writing over the top of Terraform.
We also wrote an ownership file. One line per resource type per environment, naming the repository that owns it, and a review rule that a pull request introducing a resource type not in that file needs a second approver.
We split our state a year ago to reduce blast radius, and it worked. What we never wrote down was where one state stopped and the next one started.
– Sergey Shinder
Top comments (0)