Our disaster recovery environment is built from the same modules as production, in a second region, from nothing. We do the exercise twice a year. In March the apply finished in twenty-six minutes, three hundred and eighty resources, no errors, and forty minutes later we were still working out why nothing in it functioned.
Every pod started. Every pod then failed its first outbound call. Licence checks, our payment partner's token endpoint, the package registry two services pull from at boot: all of them timed out. Inside the VPC everything was fine. Services reached each other, the database answered, internal DNS resolved.
The private subnets had no route to a NAT gateway. Our networking module creates one gateway per availability zone and one route table per subnet, and associates the two by matching on a tag. In the second region the subnet tags are assembled from a different variable, and one of them produced private-1 where the association expected private-01. The for_each over that map simply yielded nothing for those subnets, so the associations were never declared. Terraform did precisely what we asked. Every resource in the configuration existed with the attributes we had written, and the plan was clean, because from the plan's point of view nothing was missing.
What was missing was a relationship nobody had ever stated as a requirement.
That is the part I had not properly taken on board. An apply proves the resources exist. It proves nothing whatsoever about what they allow you to do, and we had spent three years getting good at reviewing plans without ever adding a step that asks the built thing a question.
So the pipeline has a final stage now. From a pod in every subnet it resolves a public name, makes an outbound HTTPS call, opens a connection to the database, writes and reads one object in the bucket, and assumes the role the application uses to call one real API with it. Eleven assertions, about ninety seconds, and a failure fails the run.
The cheap invariants went into the code itself as postconditions: the number of route table associations must equal the number of private subnets. That one line would have caught this before anything was created.
Running the assertions against our existing environments turned up two more faults, both in places that had been live for months.
– Sergey Shinder
Top comments (0)