Deploying AWS CDK apps using short-lived credentials and Github Actions

There are many ways to deploy an application to AWS. One might use AWS CloudFormation directly or a service like AWS CodePipeline. If AWS CloudFormation is not your cup of tea, you have services like AWS Elastic Beanstalk at your disposal.

There are a lot of choices.

This blog post will show you my preferred way of deploying AWS CDK applications - using GitHub Actions and utilizing short-lived IAM credentials for maximum security benefits.

Let us dive in.

You can find all the code presented in this blog post in this GitHub repository.

Typical AWS CDK deployment using GitHub Actions

Till recently, a typical AWS deployment using GitHub Actions might have looked similar to the following, simplified diagram.

To deploy the underlying application, one had to keep the long-lived IAM user credentials within the GitHub repository/organization secrets. While this setup certainly does the job, it leaves a relatively big security gap in our environment.

Let us explore why we should be moving away from IAM users in such scenarios next.

What is the issue with AWS IAM users

People who know much more about security than I do, like Ben Kehoe, have been advocating for abandoning the use of IAM users for most use-cases a long time. I completely agree with their arguments.

Ask any security professional if they would be more comfortable with long-lived or short-lived revokable credentials. I'm willing to bet a large sum of money that they would pick the latter option. The access keys are long-lived! Imagine an attacker getting a hold of those. They would be able to wreck your infrastructure. Not ideal.

Luckily an alternative appeared on the horizon recently. Let us explore that topic further.

An alternative

We were pretty much stuck with the IAM user setup when deploying AWS CDK applications for the longest time. All has changed when GitHub announced support for OpenID Connect for GitHub actions.

Now, instead of relying on the long-lived IAM User credentials, we can use the AssumeRoleWithWebIdentity AWS IAM call to get short-lived IAM role credentials to deploy AWS CDK applications. Neat!

Let us see, step by step, what goes into making this setup work.

In Action

Moving to the concrete now – the following is an example of how one might set up GitHub AWS CDK deployment pipeline to leverage the short-lived credentials utilizing GitHub OIDC provider.

Important: The following blog post assumes that you have the newStyleStackSynthesis AWS CDK feature flag turned on or are using AWS CDK v2. Read more about AWS CDK feature flags here.

Two separate stacks and AWS accounts

I'm a big believer in separating different concerns while writing application code or creating AWS infrastructures. In the spirit of keeping things separated, I will create two AWS CDK stacks on two separate AWS accounts.

The AWS account is a great way to reduce the potential blast radius when things go wrong. As a matter of fact, AWS recommends using multiple AWS accounts to achieve resource independence and isolation.

Please note that if you are deploying relatively simple architectures, you do not need separate accounts here – it might be an overkill. If you are not interested in the multi-account setup, read on. I will show you how to do all we discussed today using a single account.

• The first stack is called "infrastructure" and will hold the GitHub OIDC IAM Provider as well as the AWS IAM role used to deploy the "application". This stack will be deployed in account A.

• The second stack is called "application" and will hold the application itself. This stack will be deployed in account B.

The infrastructure stack

The OIDC provider

Creating a custom OIDC provider using AWS CDK is a breeze. The iam.OpenIdConnectProvider class is a great abstraction over the AWS::IAM::OIDCProvider AWS CloudFormation resource.

The following AWS CDK code will create a custom AWS IAM OIDC provider that developers can use in the context of GitHub Actions.

import { aws_iam } from "aws-cdk-lib";

// ...

const gitHubOIDCProvider = new aws_iam.OpenIdConnectProvider(
  this,
  "gitHubOIDCProvider",
  {
    url: "https://token.actions.githubusercontent.com",
    clientIds: ["sts.amazonaws.com"]
  }
);

Github has a great guide on how to integrate their OIDC provider with AWS. Give it a read!

The "deployer" role

As I eluded earlier, we will use the "deployer" role to deploy our main AWS CDK application. This role has to have a trust relationship with the custom OIDC provider we have created earlier – otherwise, we would be unable to assume it during GitHub Actions run.

import { aws_iam } from "aws-cdk-lib";

// ...

const gitHubOIDCProvider = ...

/**
 * Amend those to your needs.
 */
const yourGitHubUsername = "WojciechMatuszewski";
const yourGitHubRepoName = "github-oidc-aws-cdk-example";
const yourGitHubBranchName = "main";

const applicationDeployerRole = new aws_iam.Role(this, "applicationDeployerRole", {
  assumedBy: new iam.WebIdentityPrincipal(
    gitHubOIDCProvider.openIdConnectProviderArn,
    {
      StringLike: {
        "token.actions.githubusercontent.com:sub":
          // Notice the `ref:refs`. The `s` in the second `ref` is important!
          `repo:${yourGitHubUsername}/${yourGitHubRepoName}:ref:refs/heads/${yourGitHubBranchName}`
      }
    }
  ),
  inlinePolicies: {
    allowAssumeOnAccountB: new aws_iam.PolicyDocument({
      statements: [
        new aws_iam.PolicyStatement({
          effect: iam.Effect.ALLOW,
          actions: ["sts:AssumeRole"],
          resources: ["arn:aws:iam::ACCOUNT_B_ID:role/*"]
        })
      ]
    })
  }
});

There are two essential things to notice in this code snippet.

1. The conditions on the policy are very important. They make it so that untrusted repositories can't request the applicationDeployerRole access tokens.

2. The allowAssumeOnAccountB policy statement. Since we will use this role in a multi-account setup, we need to grant the role permissions to assume roles defined in account A. This policy statement is not needed when deploying both stacks in the single account.

---

With the stack deployed, copy the ARN of the applicationDeployerRole – you will need it later.

Let us move to the "application" stack.

The application stack

By default the AWS CDK bootstrapping process creates, amongst other resources, five IAM roles related to the deployment process.
You can learn more about them in the AWS Documentation – mainly the "Roles" section.

What is more, those roles have a trust relationship with the whole AWS account the stack was bootstrapped it (configurable). This makes it so that, in theory, an untrusted IAM entity could deploy the application at will.

We will change the AWS CDK bootstrapping template so that only the "deployer" role can assume the roles AWS CDK creates.

Let us get started.

Modifying the bootstrap template

Check out the AWS documentation for how to customize AWS CDK bootstrapping process further.

The first step is to get the bootstrapping template. Luckily AWS CDK bootstrap command exposes the --get-template flag.

npm run cdk bootstrap -- --get-template

The second step is to amend the trust relationship of the roles in the bootstrap template. Instead of having the whole AWS account as a principal, we will set it to the "deployer" role ARN.

For example, here is the unmodified expert from the bootstrapping template containing the FilePublishingRole.

FilePublishingRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Statement:
        - Action: sts:AssumeRole
          Effect: Allow
          Principal:
            AWS:
              Ref: AWS::AccountId

To amend the trust relationship so that only the "deployer" IAM role can assume it, change the value of the AWS key under the Principal.

FilePublishingRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Statement:
        - Action: sts:AssumeRole
          Effect: Allow
-          Principal:
-            AWS:
-              Ref: AWS::AccountId
+          Principal:
+            AWS: DEPLOYER_ROLE_ARN

Bootstrapping

With the bootstrap template modified, we can run the cdk bootstrap command with a special flag. This flag will tell the AWS CDK to use our modified template as the source of truth for AWS CloudFormation.

npm run cdk bootstrap -- --template YOUR_MODIFIED_TEMPLATE_NAME.yaml

When the bootstrapping process is successful, we are ready to move into creating GitHub Actions workflow and deploying our application.

Deployment

Okay, it's time we make the last push and deploy our application for the whole world to see. As per the title of this blog post, I will be using GitHub Actions to do that.

Our workflow file will be rather simplistic.

# .github/workflows/deployment.yaml
name: deployment

on:
  push:
    branches:
      - main

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - name: Pull repository
        uses: actions/checkout@v2
      - name: Install dependencies
        working-directory: ./application
        run: npm install
      - name: Assume deployer role
        uses: aws-actions/configure-aws-credentials@v1
        with:
          role-to-assume: YOUR_DEPLOYER_ROLE_ARN
          aws-region: eu-west-1 # Do not forget about adjusting the region!
      - name: Deploy the application
        working-directory: ./application
        run: npm run deploy:cicd

I want to focus on the Assume deployer role step. The aws-actions/configure-aws-credentials@v1 will interact with the GitHub OIDC provider and perform the sts:AssumeRoleWithWebIdentity call with the token provided by the GitHub OIDC provider.

Remember specifying conditions on the "deployer" role?

import { aws_iam } from "aws-cdk-lib";

// ...

const gitHubOIDCProvider = ...

/**
 * Amend those to your needs.
 */
const yourGitHubUsername = "WojciechMatuszewski";
const yourGitHubRepoName = "github-oidc-aws-cdk-example";
const yourGitHubBranchName = "main";

const applicationDeployerRole = new aws_iam.Role(this, "applicationDeployerRole", {
  assumedBy: new iam.WebIdentityPrincipal(
    gitHubOIDCProvider.openIdConnectProviderArn,
    {
      StringLike: {
        "token.actions.githubusercontent.com:sub":
          // Notice the `ref:refs`. The `s` in the second `ref` is important!
          `repo:${yourGitHubUsername}/${yourGitHubRepoName}:ref:refs/heads/${yourGitHubBranchName}`
      }
    }
  ),
  inlinePolicies: {
    allowAssumeOnAccountB: new aws_iam.PolicyDocument({
      statements: [
        new aws_iam.PolicyStatement({
          effect: iam.Effect.ALLOW,
          actions: ["sts:AssumeRole"],
          resources: ["arn:aws:iam::ACCOUNT_B_ID:role/*"]
        })
      ]
    })
  }
});

This is where they come into play. If we specify a different repository or branch, the Assume deployer role step would fail – the aws-actions/configure-aws-credentials@v1 would not be able to get short-lived credentials. The sts:AssumeRoleWithWebIdentity call would fail due to conditions mismatch.

If we configure everything correctly, all of our jobs should run successfully.

I do not want to use multiple AWS accounts

The example GitHub repository contains the single-account deployment AWS CDK code as well!

You might not want to use a separate account for the "infrastructure" stack, which is entirely understandable. In this case, let us look at what kind of modifications we would have to do to make our deployment work.

1. The "deployer" role does not need the allowAssumeOnAccountB policy. The policy is no longer relevant as the sts:AssumeRoleWithWebIdentity call will happen in the context of a single account.

import { aws_iam } from "aws-cdk-lib";

// ...

const gitHubOIDCProvider = ...

const applicationDeployerRole = new iam.Role(this, "applicationDeployerRole", {
  assumedBy: new iam.WebIdentityPrincipal(
    gitHubOIDCProvider.openIdConnectProviderArn,
    {
      StringLike: {
        "token.actions.githubusercontent.com:sub":
          // Notice the `ref:refs`. The `s` in the second `ref` is important!
          `repo:${yourGitHubUsername}/${yourGitHubRepoName}:ref:refs/heads/${yourGitHubBranchName}`
      }
    }
  ),
- inlinePolicies: {
-   allowAssumeOnAccountB: new iam.PolicyDocument({
-      statements: [
-         new iam.PolicyStatement({
-          effect: iam.Effect.ALLOW,
-          actions: ["sts:AssumeRole"],
-          resources: ["arn:aws:iam::ACCOUNT_B_ID:role/*"]
-        })
-      ]
-    })
+  inlinePolicies: {}
  }
});

1. Specify the qualifier parameter when bootstrapping the "infrastructure" and "application" stacks.

By default, AWS CDK will re-use resources created by bootstrapping process if you deploy multiple AWS CDK stacks in the same account.
We do not want this to happen as we would not be able to change the trust policies on the roles bootstrapped by the "application" stack.

• To change the qualifier when bootstrapping, use the qualifier CLI parameter.

npm run cdk bootstrap -- --qualifier=application --template ./bootstrap-template.yaml

• Specify the synthesizer property at the CDK app level.

const app = new cdk.App();
new ApplicationStack(app, "ApplicationStack", {
+  synthesizer: new cdk.DefaultStackSynthesizer({
+    qualifier: "YOUR_QUALIFIER"
+  })
});

The rest of the process is the same as in the case of a multi-account setup.

Closing words

There are many ways for deploying AWS CDK applications. This blog post aims to show you one of them – using GitHub Actions. I hope you found the setup valuable and helpful.

For some serverless/AWS CDK content, follow me on Twitter – @wm_matuszewski

Thank you for your valuable time.

Comments

[Parthib]

I gave implementing this a shot, but unfortunately I ran into issues. From my understanding, the deployer role is in account A, and this is the role that Github assumes through OIDC to deploy the application infrastructure to account B.

By giving the account A deployer role permissions to assume account B roles and bootstrapping account B so that the CFN roles generated can be assumed by account A, I was hoping Github would magically assume the CFN roles in account B and perform the CDK deployment. It doesn't seem to work like that unfortunately.

This is the error I get on Github:

Need to perform AWS calls for account ACCOUNT_B_ID, but the current credentials are for ACCOUNT_A_ID

This error makes complete sense to me since the role I assume is based in account A, but I'm trying to deploy to account B.

I tried looking into what official AWS documentation says about this:

aws.amazon.com/blogs/devops/cross-...

Though this is unrelated to short-term credentials, what they do is assume a role in account B itself. This is why I feel the approach described in this tutorial may not work at least for the multiple AWS account approach.

If I missed something obvious, please let me know!

[Michael Fecher]

The important piece is missing in the article IMHO.

CDK has a mechanism to derive the account from the credentials and stores this in the CDK_DEFAULT_ACCOUNT and CDK_DEFAULT_REGION env vars.

Instead, the Account of B is being passed as an additional information to the application CDK code, which will be deployed.
After the synth step, CDK detects that the current credentials (Account A) are diverging from the target B and it does an "assumeRole" under the hood for the Account B cdk-* role.
Now the cross-account deployment works.
Verified this on my own. ;)

Does this make sense?

[Parthib]

While what you said makes sense, even though I specified the account in the CDK code, Github (or whatever is happening under the hood in AWS) failed to detect the difference in current credentials and then perform the assumeRole under the account B CDK role. I have a suspicion that there might be an issue with my bootstrap version or something, but in the end my solution was to put the "infra" stack in the same account as the "application" stack. The errors I was getting didn't have enough information in them to tell me what I was doing wrong unfortunately.

[Peter Reid]

A note based on my findings using this, if you're utilising "environments" in GitHub actions, the string passed to STS is of format

${USERNAME}/${REPO}:environments/${ENVIRONMENT_NAME}

where env name is the lower-snake-case form of your env name e.g. Non-Prod becomes non-prod.

The heads attribute is not accessible on the request.

You can debug this by using CloudTrail events, be warned they can take 5-10 mins to appear, make sure you are in the correct region.

[jonyjalfon94]

Great guide, thank you so much! However i had to attach a policy that allows for the deployer role to actually create the resources in the account in order to make it work