In February a routine audit found that deletion protection was off on our orders database. The change that turned it off had been through a pull request, a plan and two approvals in November. Both reviewers had read the plan, and both had scrolled past it.
The plan in question listed thirty one resources to update. Thirty of them had looked like that on every plan of that module for four months, and that was the real problem.
In the summer a colleague fixed an ordering problem between our application module and our network module by adding depends_on = [module.network] to the module block. It worked, and it changed more than it seemed to. A depends_on on a module makes everything inside it depend on everything in the other module, including the data sources. Terraform reads a data source during plan unless it depends on something with changes pending, in which case it defers the read until apply, and every value derived from it becomes known after apply. Our network module had a small permanent diff, a tag the provider kept wanting to rewrite on two route tables. So on every plan, every data source in the application module was deferred, with the note that it depends on a resource or a module with changes pending, and thirty resources showed updates to values Terraform could not yet know. Applies changed nothing that mattered, because the values came out the same.
People learn quickly. After a few weeks of plans that always said thirty and always did nothing, a plan that said thirty one looked exactly the same.
We turned deletion protection back on the day the audit found it. Then we removed the module depends_on and passed the network module's outputs, subnet and security group ids, in as variables, which gives Terraform the exact dependency instead of a blanket one. The route table tags are now ignored where the provider insists on rewriting them. Our pipeline fails any plan that contains a deferred data source read, and a nightly plan of main fails if it is not empty. And changes to a short list of attributes, deletion protection among them, are posted as a separate comment on the pull request, above the plan.
A plan with noise in it does not just hide one change. It teaches the people reading it that plans are not worth reading.
– Sergey Shinder
Top comments (0)