On a Monday in June new pods in our production cluster started failing with ImagePullBackOff. ECR answered 403. Pods already running were fine, and so were the nodes that had been up since the weekend. Only the nodes the autoscaler added for the morning peak could not pull anything.
EKS worker nodes pull images with their instance role, and that role needs the AWS managed policy AmazonEC2ContainerRegistryReadOnly. Ours had it, attached in our cluster module with aws_iam_role_policy_attachment. The week before, a pull request to the CI module gave our new build runners the same policy so they could pull base images. It used a different resource, aws_iam_policy_attachment, with one role in its roles list. The plan showed one resource to create.
That resource does not attach a policy to a role. It owns every attachment of that policy in the whole account, and the provider documentation says so in a warning at the top of its page. After creating it, Terraform read the policy's attachments back into state, which now listed our runner role and also the node roles of three clusters and two Lambda roles. On Monday morning a routine change to the runner instance size went through the CI module, and its plan included an update in place to the attachment, with five roles to remove from its list. Nobody reviewing a runner change expected it to touch the cluster.
Running nodes had their images cached. At nine the autoscaler added eleven nodes, and every pod scheduled on them waited for an image it was no longer allowed to fetch. We applied the cluster module again, which put the attachment back, and the pods started within minutes. Checkout ran at reduced capacity for about forty minutes. Before anyone touched the CI module again we removed that resource from its state, or its next apply would have taken the policy away a second time.
The CI module now uses aws_iam_role_policy_attachment like everything else. A policy check in our pipeline rejects aws_iam_policy_attachment and the other resources that manage a whole set from one place, such as aws_iam_group_membership. And a scheduled job compares the roles each managed policy we depend on is really attached to with what our modules declare, and reports any difference within the hour.
The name of a resource says what it touches. The documentation says what it owns, and those were very different sizes.
– Sergey Shinder
Top comments (0)