⚠️ I am sorry if gists are sometimes rendered in the wrong order. It still seem to be an open issue on the forem platform: https://github.com/forem/forem/issues/14428
AWS access keys enables you to provide a programmatic access to your AWS cloud infrastructure. You can use those access keys to verify your identity and permissions in your AWS CLI, all AWS SDKs but also in your CI/CD pipeline to deploy your application which certain toolsets like SAM, CDK or the raw AWS CLI.
When using access keys, automated rotation is a very important aspect in terms of security and also embedded in the security pillar of the AWS Well-Architected Framework.
In this post, I will describe a solution to automatically rotate access keys that are used in your Bitbucket CI/CD pipeline using the Bitbucket API.
Best practices for managing AWS access keys
Besides the regular rotation of your access keys, AWS describes some additional best practices when managing your AWS access keys, including:
- protect or don't create your root user access key
- don't embed access keys into your source code
- rotate access keys periodically
- remove unused access keys
- configure MFA for your most sensitive operations
- use IAM roles instead of long-term access keys
But how to I rotate access keys that I configured in my CI/CD SaaS solution like Github, Bitbucket or Gitlab?
The good thing: all major providers provide APIs to enhance automation. Let's have a look how this can work for Bitbucket.
Rotate, deactivate, delete, repeat
What do we want to achieve? We want an automated process to
- Rotate our access keys every 90 days
- Deactivate unused access keys every 100 days
- Delete unused access keys every 110 days
- Update deployment variables in our Bitbucket pipeline configuration after every rotation.
Okay. Challenge accepted. Hold my beer and here we go:
In order to trigger our process we configure scheduled Cloudwatch events to invoke some lambda functions ever 90/100/110 days. Each lambda function has a dedicated responsibility to create a new, deactivate an or delete already deactivated access keys.
The rotation lambda is straight forward. It creates a new access key and writes the credentials in a secret provisioned in the AWS Secret Manager.
The secret will be the source of truth for the actice access key that is also used in our Bitbucket Pipeline configuration. In the next chapter, we take a deeper look how we now sync the secret with Bitbucket.
Sync credentials with Bitbucket
Bitbucket provides a REST API to access data or trigger operations on repositories, workspaces or pipeline configurations. In order to update environment variables in our deployment configurations in Bitbucket, we need to know:
- the name of the workspace,
- the name of the repository,
- the uuid of the deployment environment.
Having all those information in place, we can use the Bitbucket API to update the value of our deployment variables.
We can then use a Cloudwatch Event to capture all PutSecretValue events via CloudTrail and invoke the sync lambda function that is responsible to update your Bitbucket deployment variables. If you for example use the AWS CDK, configure the CloudWatch event like:
If you are more a fan of raw Cloudformation or SAM, the same will also be possible to setup in your favorite infrastructure-as-code tool.
What about IAM roles?
If you are more looking for a solution that does not rely on access keys but rather uses IAM roles for authorizing, I can highly recommend the solution to deploy on AWS using Bitbucket pipelines OIDC.
In order to use OpenID Connect on AWS, you will need to configure Pipelines as a Web Identity Provider, create an IAM role, and configure the build to assume the created role prior to running your build.
Web Identity Providers allow the system to receive an authentication token, and then use or exchange that token for temporary security credentials in AWS. These temporary security credentials map to an IAM role with permissions to use the resources in your AWS account. Learn more about Web Identity Providers from AWS
Alexa says, time is over...see you next time. Happy to get your feedback, experience and thoughts in the comments. 👋
Cover Image: https://unsplash.com/photos/DoWZMPZ-M9s
Top comments (2)
Very nice explanation to very old problem :) I'm surprised AWS still hasn't implemented IAM Access Keys rotation as part of Secrets Manager service (as for RDS). Maybe you should make it an CDK Construct and release it on Construct Hub (constructs.dev)?
That is an awesome idea! I think it would be also very valuable to also create a construct for Github and Gitlab...