Personally, I think cutting corners is inevitably linked to how we function as human beings. It's all about following the desired path. In my head, I often go over a problem and its solutions and in the meantime, I consider the tradeoffs of all options. Usually, this process is about finding the right balance between value and effort. For example, for a one-time job it's often not worth to put tremendous effort into automation.
However, from time to time, you stumble upon something which you thought you'd well considered. A few weeks later, giving your solution a second thought, you could feel like you're being hit by lightning.
One of the first things you learn on AWS is to follow the standard security advice of granting least privilege. It's a simple rule, but that doesn't mean it's easy to implement. When we started creating AWS CodePipelines a while ago we came up with the following CloudFormation code describing our infrastructure:
DeliveryPipeline: Type: AWS::CodePipeline::Pipeline Properties: ArtefactStores: - Region: eu-west-1 ArtifactStore: Location: !Ref ArtefactBucketName Type: S3 Name: some-project-pipeline RoleArn: !GetAtt CodePipelineServicenRole.Arn Stages: - Name: Source Actions: - Name: GitSource ActionTypeId: Category: Source Owner: ThirdParty Provider: GitHub Version: "1" Configuration: Owner: !Ref GitHubOwner Repo: !Ref GitRepo Branch: !Ref GitBranch PollForSourceChanges: False OAuthToken: !Ref GitHubOAuthToken OutputArtifacts: - Name: SourceZip ... - Name: Deploy Actions: - Name: DeployPersistantStack ActionTypeId: Category: Deploy Owner: AWS Provider: CloudFormation Version: "1" Configuration: ActionMode: CREATE_UPDATE Capabilities: CAPABILITY_NAMED_IAM RoleArn: !GetAtt CloudFormationExecutionRole.Arn StackName: some-stack-cfn TemplatePath: BuildArtifactAsZip::cfn-template.yaml TemplateConfiguration: BuildArtifactAsZip::dist/config/cloudformation/stack-config.json InputArtifacts: - Name: BuildArtifactAsZip RunOrder: 1 ... CloudFormationExecutionRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: Action: - sts:AssumeRole Effect: Allow Principal: Service: - cloudformation.amazonaws.com Path: / ManagedPolicyArns: - arn:aws:iam::aws:policy/AdministratorAccess
I think the corner that has been cut in the code above is easy to find. A
AdministratorAccess isn't compliant the least privilege rule. The reason to take a shortcut -in this case- was the fact that CloudFormation stacks often do a lot of different things. Applying the least privilege rule in this context would mean a long and well-thought-out policy. So for simplicity and speed, we decided to cut a corner. The trade-off? Allowing the pipeline a tad more access then it should, no big deal, I could live with that.
When it comes to security, I think we made a pretty good fortress of our AWS environment: a good IDP, AWS WAF, AWS Security Hub, AWS Inspector and an endless list of other tools to close the gates. We made AWS a safe place to go to. A Git repository, however, often lives outside the AWS ecosystem. GitHub, GitLab and Bitbucket are the most common places to put your code. Next question: is your Git repository set up like a fortress as well? If the answer is no, then allow me to create some horror.
If someone could gain access to your Git repository, he could do one of the following if a pipeline has been granted
AdministratorAccess for CloudFormation:
- Change a DNS Name to re-route traffic (to a phishing site)
- Change your DeletionPolicy and throw away all your resources
- Change an instance profile or a security group
- I think this already showed you enough horror
If I now repeat my question: can I live with that? The answer is the complete opposite. It's a big deal, no way I can accept this!
So this is also a note to myself. Always think twice when cutting corners, especially if security might be at risk. Sometimes it's not easy to understand the blast radius of decisions at first. To make things even worse: in many cases, it will be very hard or even impossible for AWS Security tools to define the correct severity level. In this particular case, AWS will give you a warning but it cannot describe all the underlying risks.
I'm feeling lucky that I got this epiphany before this could turn into a real issue.
Although I have never been a fan of pipelines that maintain themselves, in this case, I need to give an extra warning. For clarity: by self-maintaining pipelines I mean pipelines that have a stage to update themselves. Having such a pipeline listening to Git means that the Pipeline policy itself can be changed by pushing a commit to Git. So if your self-maintaining pipeline doesn't have
AdministratorAccess, this is easily changed by having access to its Git repository.
For that reason, I would remove self-updating logic from a pipeline. We use makefiles that need a client to authenticate in order to make Pipelines changes.
In the end, the best way to close this security hole is to apply the least privilege rule to your Pipeline. The downside of this approach is the introduction of extra work. Whenever someone will make changes to the CloudFormation stack that incur security context changes he will need to update the Pipeline's policy first. Be prepared for people who get really upset by this.
If you are in a situation where your current pipeline policy is too relaxed, you could already consider starting with an intermediate solution. You could begin by denying all IAM and Route 53 access and/or deny delete access for all resources. Although this is already more secure, this approach has some downsides. So it's just to buy you a bit more time before applying the least privilege rule.
Sharing is caring ;-)
Until next time.
Note: the same security hole can be created on AWS CodeBuild.