DEV Community

Cover image for How I Removed AWS Access Keys from GitLab CI/CD with OIDC
Amira Abidi
Amira Abidi

Posted on

How I Removed AWS Access Keys from GitLab CI/CD with OIDC

#aws #terraform #gitlab #devops

When I first connected my GitLab CI/CD pipelines to AWS, I used the simplest solution: an IAM user with an Access Key and Secret Access Key stored as GitLab CI/CD variables.

It worked.

But there was one problem: those credentials were permanent.

They had to be stored, protected and eventually rotated. If they were accidentally exposed in logs or compromised, they could remain valid until manually revoked.

I wanted a cleaner solution.

So I replaced permanent AWS credentials with OIDC federation between GitLab and AWS.

The result is simple: GitLab pipelines can access AWS without storing any permanent AWS credentials.

In this post, I'll explain how I implemented it, how the authentication flow works, and one important issue I faced when using it with EKS and Terraform.

The architecture

The authentication flow looks like this:

┌──────────────┐
│  GitLab CI   │
└──────┬───────┘
       │
       │ OIDC token
       ▼
┌──────────────┐
│   AWS STS    │
└──────┬───────┘
       │
       │ Temporary credentials
       ▼
┌──────────────┐
│   IAM Role   │
└──────┬───────┘
       │
       ├──────────► Terraform
       │
       ├──────────► ECR
       │
       └──────────► EKS
Enter fullscreen mode Exit fullscreen mode

Instead of GitLab storing an AWS Access Key, it proves its identity to AWS using a short-lived OIDC token.

AWS verifies the token and returns temporary credentials.

How does OIDC authentication work?

The process can be summarized in five steps:

  1. GitLab creates an OIDC token for the CI/CD job.
  2. The pipeline sends this token to AWS.
  3. AWS verifies that the token really comes from GitLab.
  4. AWS checks that the project is allowed to assume the requested IAM role.
  5. AWS STS returns temporary credentials.

These credentials expire automatically.

So there is nothing permanent to store or rotate inside GitLab.

Step 1 — Register GitLab as an OIDC provider

AWS first needs to trust GitLab as an identity provider.

I configured the OIDC provider using Terraform:

data "tls_certificate" "gitlab" {
  url = "${var.gitlab_url}/.well-known/openid-configuration"
}

resource "aws_iam_openid_connect_provider" "gitlab" {
  url             = var.gitlab_url
  client_id_list  = [var.gitlab_url]
  thumbprint_list = [
    data.tls_certificate.gitlab.certificates[0].sha1_fingerprint
  ]
}
Enter fullscreen mode Exit fullscreen mode

This tells AWS that tokens issued by GitLab can be used for authentication.

But trusting GitLab itself is not enough.

AWS also needs to know which GitLab projects are allowed to access which IAM roles.

Step 2 — Restrict access with the IAM Trust Policy

This is one of the most important parts of the configuration.

We don't want any GitLab project to be able to assume our AWS roles.

The IAM Trust Policy checks information contained in the GitLab token.

For example:

condition {
  test     = "StringEquals"
  variable = "gitlab.com:aud"
  values   = [var.gitlab_url]
}

condition {
  test     = "StringLike"
  variable = "gitlab.com:sub"
  values   = [
    "project_path:${each.value}:ref_type:branch:ref:*"
  ]
}

condition {
  test     = "StringEquals"
  variable = "gitlab.com:namespace_id"
  values   = [var.gitlab_namespace_id]
}
Enter fullscreen mode Exit fullscreen mode

The important values here are:

  • aud: checks the expected audience.
  • sub: identifies the GitLab project and branch.
  • namespace_id: identifies the GitLab namespace.

I especially like using namespace_id because it is a stable numeric identifier.

GitLab project and group names can change, but the namespace ID provides an additional way to make sure the token comes from the expected namespace.

Step 3 — One IAM role per repository

My project uses four GitLab repositories, so I created a separate IAM role for each one.

Repository Main AWS permissions
Backend Push Docker images to ECR
Frontend Push Docker images to ECR
Infrastructure Manage AWS infrastructure with Terraform
Deploy Check ECR images and deploy to EKS

Why separate them?

Because the backend pipeline doesn't need permission to modify the EKS cluster.

And the deployment pipeline doesn't need permission to push Docker images.

Each pipeline gets only the permissions required for its job.

For example, the deployment role only needs to check whether an image exists:

actions = [
  "ecr:DescribeImages",
  "ecr:DescribeRepositories"
]
Enter fullscreen mode Exit fullscreen mode

It cannot push a new image.

This gives me a much cleaner separation of responsibilities.

Step 4 — Request the OIDC token from GitLab

On the GitLab side, the configuration is surprisingly small.

GitLab can generate the OIDC token directly inside the CI/CD job:

id_tokens:
  GITLAB_OIDC_TOKEN:
    aud: https://gitlab.com
Enter fullscreen mode Exit fullscreen mode

Then I define the AWS role and the location of the token:

variables:
  AWS_REGION: "us-east-1"
  AWS_ROLE_ARN: "arn:aws:iam::123456789012:role/gitlab-ci-book-infra"
  AWS_WEB_IDENTITY_TOKEN_FILE: "${CI_PROJECT_DIR}/.gitlab-oidc-token"
Enter fullscreen mode Exit fullscreen mode

And write the token to the file:

before_script:
  - echo "$GITLAB_OIDC_TOKEN" > "$AWS_WEB_IDENTITY_TOKEN_FILE"
Enter fullscreen mode Exit fullscreen mode

That's basically it.

Terraform and AWS SDKs understand the web identity authentication flow automatically.

There is no need to run:

aws configure
Enter fullscreen mode Exit fullscreen mode

And more importantly, I don't need to store permanent values such as:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
Enter fullscreen mode Exit fullscreen mode

in GitLab CI/CD variables anymore.

Step 5 — EKS needs an extra configuration

This was an important thing I learned while implementing the solution.

Successfully authenticating with AWS does not automatically mean that the pipeline can access Kubernetes.

For example:

aws eks update-kubeconfig
Enter fullscreen mode Exit fullscreen mode

may work correctly while:

kubectl get pods
Enter fullscreen mode Exit fullscreen mode

returns:

Unauthorized
Enter fullscreen mode Exit fullscreen mode

Why?

Because there are two different authorization levels:

AWS IAM
   │
   │ Can this role access the AWS/EKS API?
   ▼
EKS Cluster
   │
   │ Is this IAM role allowed inside Kubernetes?
   ▼
Kubernetes resources
Enter fullscreen mode Exit fullscreen mode

The IAM role therefore also needs access to the EKS cluster.

I configured this with an EKS access entry:

resource "aws_eks_access_entry" "ci_deploy" {
  cluster_name  = module.eks.cluster_name
  principal_arn = var.ci_deploy_role_arn
  type          = "STANDARD"
}
Enter fullscreen mode Exit fullscreen mode

And associated the required EKS access policy with it.

Once this was configured, the deployment pipeline could authenticate correctly to the cluster.

The Terraform + EKS issue I ran into

This was probably the most interesting problem I encountered.

EKS authentication tokens are short-lived.

Initially, I used:

data "aws_eks_cluster_auth" "this" {
  name = module.eks.cluster_name
}
Enter fullscreen mode Exit fullscreen mode

This works when terraform plan and terraform apply happen immediately.

But my pipeline separates them.

The flow is:

terraform plan
      │
      ▼
Manual approval
      │
      ▼
terraform apply
Enter fullscreen mode Exit fullscreen mode

The problem is that the EKS token can be generated during terraform plan.

If I wait before approving the apply, the saved token may already be expired.

Terraform then fails when trying to communicate with Kubernetes.

The solution: generate the token when it is needed

Instead of storing the token during the plan, I use an exec block:

provider "kubernetes" {
  host = module.eks.cluster_endpoint

  cluster_ca_certificate = base64decode(
    module.eks.cluster_certificate_authority_data
  )

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws-iam-authenticator"
    args        = [
      "token",
      "-i",
      module.eks.cluster_name
    ]
  }
}
Enter fullscreen mode Exit fullscreen mode

Now the token is generated when Terraform actually needs to connect to Kubernetes.

So even if I wait before approving terraform apply, Terraform gets a fresh token.

Another small GitLab CI lesson

Manual GitLab jobs can allow failures by default depending on how they are configured.

For an infrastructure deployment, that's something I definitely don't want.

If terraform apply fails, the pipeline should fail too.

So I explicitly use:

apply:
  rules:
    - if: '$CI_COMMIT_BRANCH'
      when: manual
      allow_failure: false
Enter fullscreen mode Exit fullscreen mode

It's a small configuration detail, but an important one for infrastructure pipelines.

Debugging OIDC authentication

When authentication fails, I usually check three things first:

aud
sub
namespace_id
Enter fullscreen mode Exit fullscreen mode

Typical errors include:

Error What I check
AssumeRoleWithWebIdentity denied Project path / sub
Incorrect token audience aud configuration
InvalidIdentityToken Token expiration / OIDC provider
Unauthorized from kubectl EKS access entry

I also keep this command in the pipeline:

aws sts get-caller-identity
Enter fullscreen mode Exit fullscreen mode

It's very useful because it immediately shows which AWS identity the pipeline is actually using.

What I learned

The biggest lesson from this implementation is that OIDC is not as complicated as it first looks.

The basic idea is simply:

GitLab proves its identity
        ↓
AWS verifies the GitLab project
        ↓
AWS STS provides temporary credentials
        ↓
The pipeline accesses AWS
Enter fullscreen mode Exit fullscreen mode

The main advantages for me are:

  • No permanent AWS credentials in GitLab.
  • No Access Keys to rotate.
  • Temporary credentials expire automatically.
  • Each repository has its own IAM role.
  • Permissions can be limited per pipeline.
  • AWS activity can be traced through CloudTrail.
  • GitLab can securely deploy to EKS.

There are a few extra details when EKS is involved, especially around cluster access and short-lived Kubernetes tokens, but once those pieces are understood, the overall architecture is quite simple.

For CI/CD pipelines, I now prefer this approach over storing permanent AWS credentials whenever OIDC federation is available.

Top comments (0)