We had an incident at 19:20, fixed it at 19:35, wrote it up, and went home. The same incident started again at 02:07 the next morning with nobody awake to recognise it, and the on-call engineer spent forty minutes rediscovering a root cause we had already documented seven hours earlier.
During the first incident, connections to the primary database were exhausted. The fastest available fix was to widen the security group so the read replica could take a chunk of reporting traffic, and the on-call engineer made that change in the AWS console because that takes ninety seconds and a Terraform pull request does not. It worked. It also existed nowhere in code.
We ran a drift-correction pipeline nightly: terraform apply -auto-approve at 02:00 against every production workspace, because we had been burned by configuration drift and decided the answer was to make the repository continuously true. At 02:04 it noticed an unmanaged security group rule, and at 02:05 it removed it. The reporting traffic went back to the primary, connections exhausted again, and the alert fired at 02:07. The automation did precisely what we had asked, at the worst possible moment, against the one change that was keeping the system up.
Two things were wrong and only one of them was the console change. An unattended apply is a deploy with no human looking at a plan, and we had scheduled it for the hour when nobody could react to it. That pipeline is now terraform plan -detailed-exitcode only. Drift opens a ticket with the plan attached and pages nobody; applying is still a reviewed action a human starts.
The other half was making the emergency path legitimate instead of forbidden. Production IAM is read-only by default. There is a break-glass role that anyone on-call can assume, which lasts an hour, announces itself in the incident channel, and automatically opens a ticket titled with the resources touched. Closing an incident now requires either reverting the manual change or landing the code that codifies it, and the drift report is the thing that reminds you.
Infrastructure as code does not remove manual changes. It decides what happens to them, and "silently undone at two in the morning" is the worst option available.
– Sergey Shinder
Top comments (0)