DEV Community: Liran Aknin

Farewell Static Access Keys

Liran Aknin — Tue, 03 Dec 2024 09:23:22 +0000

Introduction

CI/CD pipelines frequently require integration with external cloud services to support a wide range of use cases. This may include uploading artifacts to object storage solutions (e.g. Amazon S3, Azure Blob Storage), publishing container images to artifact repositories (e.g. AWS Elastic Container Registry, Google Artifact Registry), or deploying applications to container orchestration platforms (e.g. Amazon ECS, Kubernetes, Azure AKS).

For example, in AWS, the common practice involves creating an IAM user, assigning the necessary permissions, and generating AWS access keys to facilitate this interaction.

These keys are then typically stored manually in the CI/CD platform, such as GitHub Secrets, CircleCI Contexts, or similar tools. However, this approach presents several challenges.

Storing static secrets in third-party platforms poses a significant risk of unauthorized access and potential exposure.
High-profile breaches, such as the CircleCI breach and the Codecov supply chain attack, highlight the appeal of CI/CD services as prime targets for attackers.
With these platforms' critical role in modern development pipelines, the question isn't "if" but "when" the next breach will occur.

If not handled perfectly, rotating these keys frequently can result in unexpected downtimes or operational challenges. Even small errors during the rotation process can disrupt workflows and cause delays. Moreover, the process of key rotation demands some time investment, diverting valuable engineering resources from other critical tasks.
Given the sensitive nature of CI/CD pipelines, it is evident that relying on such fragile and resource-intensive practices is a considerable risk.

Luckily, there is a better approach. OpenID Connect (OIDC) offers a solution by replacing static credentials with short-lived tokens.
These tokens are dynamically issued during each workflow run, removing the need for long-lived secrets in CI/CD platforms and significantly enhancing security and operational efficiency.

In this blog, I’ll demonstrate how I set up GitHub Actions to securely interact with AWS resources using OIDC.
My specific use case involves uploading files to a website hosted on S3 and clearing the CloudFront cache to ensure the new content is served.

By the end of this blog, you’ll gain a clear understanding of how OIDC overcomes the challenges discussed earlier.

Before diving into the technical details of this setup, let’s take a step back to explore what OIDC is and how it works so that we can lay a solid foundation for the implementation.

Understanding OIDC

OpenID Connect (OIDC) is a protocol built on top of OAuth 2.0 that simplifies verifying the identity of users or services. Instead of relying on static credentials like passwords or API keys, OIDC uses tokens to provide secure access via short-lived credentials.

Using an OIDC Identity Provider allows you to manage external identities, such as users or services, to access your internal resources without the need to create and maintain separate internal user accounts.

To illustrate this, consider the analogy of a pass for a convention or conference. The event spans multiple days with various sessions, but access to specific sessions requires an appropriate pass. This pass contains your name (identity), your registration type (role or scope), and a unique QR code (short-lived credentials) to grant you access to the sessions. The pass is only valid for the event and time it’s issued.

If we extend this analogy to our use case, GitHub workflows represent the attendees needing access to the specific event sessions (AWS resources).
To gain access, the workflows request a pass from the GitHub OIDC provider, which issues an OIDC token in the form of a JSON Web Token (JWT). This token includes key details, called claims, about the specific request.

Lets focus on the main ones:

iss: The issuer of the token (GitHub OIDC provider). This claim, along with other header parameters such as alg (algorithm) and kid (key ID), helps AWS validate the token's origin and verify its authenticity.
sub: The subject of the token. Represents the unique workflow making the request. It includes metadata like the repository, branch, and triggering event.
aud: The audience of the token. Specifies who the token is intended for (default to the URL of the repository owner). In my case, because I use the official aws-actions/configure-aws-credentials action in my workflow, it must be set to AWS Security Token Service - sts.amazonaws.com. More on that later.

AWS then validates the OIDC token’s details against its predefined trust configuration. If the token’s claims match, AWS generates a temporary set of credentials. These credentials provide the workflow with secure, time-limited access to the specified resources.

Putting It All Together

Now with the basic understanding of what OIDC is, it’s time to piece everything together.

I will use Terraform, my preferred tool for provisioning infrastructure. However, the same setup can be achieved using the AWS Management Console, the AWS Command Line Interface (CLI), or Tools for Windows PowerShell.
Detailed instructions can be found in the official AWS documentation.

It is important to note that support for custom claims for GitHub OIDC is unavailable in AWS.

Building The Trust

The first step is to add the GitHub OIDC Provider as a trusted identity Provider (IdP) in AWS. This is done by creating an OIDC provider entity in AWS IAM.
By doing so, we set the foundations to establish trust between our AWS account and GitHub’s OIDC provider.

resource "aws_iam_openid_connect_provider" "github_actions_oidc_provider" {
        url   = "https://token.actions.githubusercontent.com"

        client_id_list = [
            "sts.amazonaws.com",
        ]

        thumbprint_list = ["33e4e80807204c2b6182a3a14b591acd25b5f0db"] # Thumbprint of the GitHub OIDC provider's Intermediate Certificate
}

The OIDC provider resource has three key components that AWS uses during the validation process:

url: The OIDC provider URL, which corresponds to the iss (Issuer) claim in the token.
client_id_list: These correspond to the aud (Audience) claim.
thumbprint_list: A list of certificate thumbprints for AWS to validate the OIDC provider’s SSL certificate. This ensures that the OIDC provider is trusted. You can follow this procedure to get the thumbprint.

Creating a Role and Trust Policy

Once the OIDC provider is configured in AWS, the next step is to create an IAM role.
Unlike a user, an IAM role does not have long-term credentials. Instead, it is a temporary identity assigned to a federated user, such as a GitHub workflow, after successful authentication by the Identity Provider (IdP).

The role defines the trust relationship, allowing the workflow to request temporary security credentials from AWS STS using a token issued by the IdP. These credentials grant access to AWS resources, with the scope of access determined by the policies attached to the role.

Restricting which entities can assume the role by defining conditions in the trust policy is a best practice.
For example, you can use the token.actions.githubusercontent.com:sub condition key with string operators to limit access to a specific GitHub organization, repository, or branch.
This ensures that only trusted repositories can request tokens for your resources. In fact, in our use case, IAM will enforce that the token.actions.githubusercontent.com:sub condition key is defined and prohibits the use of wildcard (* or ?) or null values. If not explicitly set, any workflow or GitHub organization would be able to request access using the role name. Therefore, to prevent unauthorised access we would want to set it at least to our github organization/account.

resource "aws_iam_role" "github_actions_oidc_role" {
        name = "github-actions-oidc-role"

        assume_role_policy = jsonencode({
            "Version": "2012-10-17",
            "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "Federated": "${aws_iam_openid_connect_provider.github_actions_oidc_provider.arn}"
                },
                "Action": "sts:AssumeRoleWithWebIdentity",
                "Condition": {
                    "StringEquals": {
                        "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
                    },
                    "StringLike": {
                        "token.actions.githubusercontent.com:sub": "repo:GitHubOrg/GitHubRepo:ref:refs/heads/GitHubBranch" # Restrict access to a specific GitHub repository and branch
                    }
                }
            }
            ]
        })
}

Some Tips

Using Locals

Here’s an approach to restrict access to workflows from specific GitHub repositories, regardless of the branch, by leveraging Terraform locals.

# Environment-specific variable for repository list
    variable "github_repos" {
        type = list(string)
        default = [
          "repo1",
          "repo2",
          "repo3"
        ]
    }

      locals {
        repo_sub_list = [for repo in var.github_repos : "repo:GitHubUser/${repo}:*"]
      }

      # Use the local in the trust policy condition
        "StringLike": {
                  "token.actions.githubusercontent.com:sub": local.repo_sub_list
         }

This approach becomes especially handy for managing resource access across multiple environments. It allows you to bind specific repositories to environments based on the required access.
It also makes it more readable and maintainable.

You can explore this page to see some examples of how you might use these conditions in the GitHub OIDC trust policy.

Using ABAC

In larger environments, Attribute-Based Access Control (ABAC) can greatly simplify access management by utilizing tags in IAM policies.
Tags allow you to logically group resources, eliminating the need to explicitly list individual resources in your policies.

In fact, in a previous blog, I demonstrated how to control Lambda access to ECR repositories based on their tags.

To learn more, check out the AWS documentation on ABAC and access control with tags.

Creating The Policies

Once the role and its trust policy are in place, the next step is to create your policies.

In my use case, as described earlier, I need the workflow to upload files to an S3 bucket and create a CloudFront invalidation to clear the cache.

Here is a sample policy that allows such actions:

{
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
              ],
              "Resource": [
                "arn:aws:s3:::BucketName/*",
                "arn:aws:s3:::BucketName"
              ]
            },
            {
              "Effect": "Allow",
              "Action": [
                "cloudfront:CreateInvalidation"
              ],
              "Resource": "arn:aws:cloudfront::AccountID:distribution/DistributionID"
            }
          ]
}

Configuring GitHub Workflows for OIDC Integration

Great! We’re making steady progress.

With the AWS setup complete, we’ve configured it to validate OIDC tokens and issue short-lived credentials. Now, it’s time to update our GitHub workflows to request these tokens and utilize the generated credentials within the workflow jobs.

Defining Permissions

The first step is to define the necessary permissions in the workflow’s YAML configuration file.

permissions:
  id-token: write
  contents: read

The id-token: write permission allows the workflow to request an OIDC token from the GitHub OIDC provider.
The contents: read is needed if your workflow uses the actions/checkout action to access repository contents.

Requesting Access

With the permissions configured, the next step is to utilize the OIDC token within the workflow job. The aws-actions/configure-aws-credentials action simplifies this process by retrieving the JWT from GitHub’s OIDC provider and exchanging it for temporary credentials from AWS STS.
You can find more details about this action in the official documentation.

Here is an example of how its implemented in the workflow file:

- name: Configure AWS Credentials
  uses: aws-actions/configure-aws-credentials@v3
  with:
    role-to-assume: arn:aws:iam::${{ env.AWS_ACCOUNT_ID }}:role/github-actions-oidc-role
    aws-region: ${{ env.AWS_REGION }}

- name: Sync to S3
  run: aws s3 sync . s3://${{ env.S3_BUCKET_NAME }} --delete


- name: CloudFront Cache Invalidation
  run: aws cloudfront create-invalidation --distribution-id ${{ env.CLOUDFRONT_DISTRIBUTION_ID }} --paths "/*"

Configure AWS Credentials

Assumes the specified IAM role (github-actions-oidc-role) using the OIDC token and retrieves short-lived credentials valid for the duration of the workflow run.

Sync to S3

Uses the temporary credentials to upload files to the S3 bucket.

CloudFront Cache Invalidation

Uses the temporary credentials to clear the CloudFront cache.

And That’s It!

By combining OpenID Connect (OIDC) with GitHub Actions and AWS, we’ve built a secure, seamless workflow that eliminates the need for static credentials. Instead, we’re using dynamically generated, short-lived tokens to ensure secure access to AWS resources.

Following these steps, both improves the security of your workflows and simplifies credential management. Whether your workflows are deploying applications, managing infrastructure, or updating content like in my use case, this setup ensures that sensitive credentials remain out of reach.

Oh, and don’t forget to delete those static credentials from your CI/CD platform, you won’t need them anymore! 😉

Thank you for reading. I hope you found it helpful and enjoyable!
I’d love to hear how you’ve implemented similar setups in your own use cases. Whatever it is... feel free to drop a comment below.

How I Set Up Cross-Account ECR Access for Lambda Functions

Liran Aknin — Sat, 16 Nov 2024 19:19:41 +0000

The Use Case

Recently, I took on a project to deploy a new Lambda function in each of my AWS accounts to serve as a filter for Guard Duty alerts. Based on internal logic, this function would determine whether it requires immediate handling or can be addressed later. Because this function needed to be deployed across multiple accounts and regions (basically in each of my accounts), I faced the challenge of ensuring it could access a container image stored in a central Elastic Container Registry (ECR) repository in one of my AWS accounts.
This blog will focus on setting up cross-account access to ECR with region replication, rather than the specifics of my GuardDuty setup, which I might cover in a future post.

Reading the AWS documentation gave me a good sense of what configurations are necessary. Nonetheless, in this blog I will show you how I did it - the terraform way, while providing some tips and tricks to achieve the solution for my use case.
Terraform, an Infrastructure as Code (IaC) tool, provides a consistent and automated way to manage resources across multiple AWS accounts and regions. While this blog won't dive into all the details of Terraform, you can learn more about it here.

It is worth mentioning, that this blog requires a basic understanding of Terraform.

Overview

Many organizations in today’s cloud environments centralize container images within a single AWS account to streamline management, versioning, and security. However, additional configurations are required when Lambda functions across different accounts and regions need access to these images.

Following AWS’s same-region access requirement, each Lambda function must be able to access the image in the same region as its deployment. For that reason, I used the ECR’s region replication feature. This allowed the container image to be replicated to the necessary region.

A High-Level Overview of the Solution

Now that we have a high-level idea of what must be done, let's jump into the setup.

Some Facts to Consider

I manage multiple repositories in Account ID 111111111111 region us-east-2. Some of my repositories will require Lambda access to pull their images, but not all of my repositories' images will require a replication. However, all repositories replicated to another region will also need lambda access.

In short:

Lambda access ≠ replicating an image
Replicating an image = Lambda access

The Solution

The Terraform module iterates over a list of repository names (var.repository_names) using for_each.

Each repository is tagged with: Lambda-Access = true/false

The tag value is determined dynamically using the lookup function on a map (var.ecr_lambda_access) that specifies which repositories should have Lambda access.

ecr_lambda_access = {
        "repo_name_1"  = "true"
        "repo_name_2"  = "true"
        "repo_name_3"  = "true"
}

Here’s the implementation:

resource "aws_ecr_repository" "example_registry" {
        for_each        = toset(var.repository_names)
        name            = each.value

        tags = {
          Lambda-Access = lookup(var.ecr_lambda_access, each.value, false)
        }
}

To grant cross-account and same-account access to ECR repositories, along with Lambda-specific access, we add three statements to the aws_ecr_repository_policy resource:

Cross-account access grants general access to all repositories for specific external accounts.
Cross-account Lambda access grants Lambda functions in external accounts access to repositories tagged with Lambda-Access = true.
Same-account Lambda access grants Lambda functions in the same account access to repositories tagged with Lambda-Access = true.

Here’s the complete policy:

{
    "Version": "2008-10-17",
    "Statement": [
        {
        "Sid": "CrossAccountAccess",
        "Effect": "Allow",
        "Principal": {
            "AWS": [
            "arn:aws:iam::123456789012:root",
            "arn:aws:iam::222222222222:root"
            ]
        },
        "Action": [
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchGetImage",
        "ecr:BatchCheckLayerAvailability",
        "ecr:ListImages"
        ]
    },
    {
        "Sid": "LambdaECRImageCrossAccountRetrievalPolicy",
        "Effect": "Allow",
        "Action": [
        "ecr:BatchCheckLayerAvailability",
        "ecr:BatchGetImage",
        "ecr:GetDownloadUrlForLayer",
        "ecr:ListImages"
        ],
        "Principal": {
        "Service": "lambda.amazonaws.com"
        },
        "Condition": {
        "StringLike": {
            "aws:sourceArn": "arn:aws:lambda:us-east-2:123456789012:function:*"
        },
        "StringEquals": {
            "aws:ResourceTag/Lambda-Access": "true"
        }
        }
    },
    {
        "Sid": "LambdaECRSameAccountImageRetrievalPolicy",
        "Effect": "Allow",
        "Principal": {
        "Service": "lambda.amazonaws.com"
        },
        "Action": [
        "ecr:BatchGetImage",
        "ecr:GetDownloadUrlForLayer"
        ],
        "Condition": {
        "StringEquals": {
            "aws:ResourceTag/Lambda-Access": "true"
        }
        }
    }
    ]
}

After solving cross-account and same-account access for the default region, we address cross-account, same-region access using the aws_ecr_replication_configuration resource.

We replicate only to specific regions where the Lambda function resides. In our case, it is eu-west-1 in account ID 222222222222. The replication configuration includes a rule specifying destination regions and repository filters.

The dynamic block is used to iterate over var.repository_to_replicate, a list of repository prefixes to include in replication. It avoids hardcoding multiple repository_filter blocks, making the configuration scalable and easier to manage as the list of repositories grows.

For each prefix, it dynamically generates a repository_filter block with:

filter: The prefix for repositories to match.
filter_type: Set to PREFIX_MATCH

This filter will control which repositories will be replicated.

Here’s the implementation:

resource "aws_ecr_replication_configuration" "example" {
        replication_configuration {
          rule {
            destination {
              region      = "eu-west-1"
              registry_id = data.aws_caller_identity.current.account_id
            }
            dynamic "repository_filter" {
              for_each = toset(var.repository_to_replicate)
              content {
                filter      = repository_filter.value
                filter_type = "PREFIX_MATCH"
              }
            }
          }
        }
}

Region-Specific Repository Policies

We also need to define region-specific repository policies for the replicated repositories. This ensures that Lambda, in those regions, can access the replicated repositories.

To achieve this, we leverage the aws provider's alias for the replicated region (e.g. eu-west-1) and create repository policies for the replicated repositories.

Here's how it looks:

provider "aws" {
        alias  = "ireland_region"
        region = "eu-west-1"
}

resource "aws_ecr_repository_policy" "elastic_container_registry_policy_ireland" {
        for_each   = toset(var.repository_to_replicate)
        provider   = aws.ireland_region
        repository = each.value
        depends_on = [module.elastic_container_registry]
        policy     = <<EOF
        {
        "Version": "2012-10-17",
            "Statement": [
            {
                "Sid": "repositoryPerRegionStatement",
                "Effect": "Allow",
                "Principal": {
                "Service": "lambda.amazonaws.com",
                "AWS": "arn:aws:iam::222222222222:root"
                },
                "Action": [
                "ecr:BatchCheckLayerAvailability",
                "ecr:BatchGetImage",
                "ecr:GetDownloadUrlForLayer",
                "ecr:ListImages"
                ]
            }
            ]
        }
        EOF
}

This approach keeps our configuration modular and ensures seamless access to replicated resources across regions.

Conclusion

That’s how I tackled the challenge of setting up cross-account and cross-region access for Lambda functions to pull container images from a centralized ECR repository, using a scalable and modular solution with Terraform.

I hope this blog provided some insights or inspiration for your own projects. Feel free to share your thoughts or questions—I’d be happy to hear from you!

Thank you for reading!