In August an engineer ran terraform destroy in what he thought was his own sandbox workspace. It was the shared QA environment, and among the things it removed was the QA database. Nobody worried at first, because our database module sets deletion_protection to true, and AWS refuses to delete an RDS instance with that flag set. The instance was gone anyway.
The platform team owns the database module, and its variable for the flag has a default of true. Most teams do not call it directly. They call a wrapper module that adds their naming and tags, and the wrapper exposes the same setting with a default of null. The intention, written in a comment in the wrapper, was simple: if a team says nothing, let the platform module decide.
That is not what null does. A variable in Terraform is nullable unless it says otherwise, and passing null into a nullable variable does not mean the caller left it out. It means the caller set it to null, and the default is never consulted. The platform module received null and handed it straight to the aws_db_instance resource, where a null argument is treated as not set at all. The provider then used its own default, which for deletion protection is false.
So every database built through the wrapper had been created without protection. Each create plan had said deletion_protection = false, one line among sixty in a module everyone trusted. AWS Config has a managed rule, rds-instance-deletion-protection-enabled, and when we finally switched it on it reported fourteen of our twenty two instances, all built through the wrapper since its first version. We restored QA from the nightly snapshot copy and lost an afternoon of test data.
Every variable in the platform module now sets nullable to false. With that set, a null coming in is replaced by the module's own default, which is the behaviour the wrapper's author had expected all along. The wrapper drops its own defaults and passes values only when a team sets them. The Config rule stays on in every account, and the database resource has a precondition that fails the plan unless the resolved value is true.
Null in Terraform means unset, and an unset value goes to whoever is next in line. Ours was a provider that had never heard of our standards.
– Sergey Shinder
Top comments (0)