Passing Dynamic Environment Variables Between GitLab CI Matrix Jobs: AWS OIDC Example

#gitlab #aws #cicd

In the previous article, we explored how to share build artifacts between GitLab CI matrix jobs using a React application example. Now, we'll tackle an even more complex challenge: passing dynamically generated environment variables between parallel matrix jobs.

GitLab CI's parallel:matrix feature is powerful for concurrent deployments, but lacks a direct mechanism for passing variables generated during runtime between matrix-generated jobs. In this article, we'll use AWS OIDC authentication to demonstrate how to generate and pass dynamic environment variables across GitLab CI matrix jobs.

Understanding Dynamic Environment Variables Challenge

As we saw in the previous article, GitLab's parallel:matrix feature allows running multiple jobs in parallel with different environment variable values. However, when those variables are generated during runtime (like temporary credentials), passing them to subsequent jobs becomes challenging.

For example, when deploying to multiple AWS environments, each environment might need unique temporary credentials obtained through OpenID Connect (OIDC). These credentials are generated during pipeline execution and must be passed to corresponding downstream jobs.

The solution we'll explore uses artifact files to reliably pass environment-specific data between matrix jobs, ensuring each parallel deployment has access to its own unique credentials.

Practical Example: AWS Multi-Tenant Deployment

Initial Setup

Here's a GitLab CI configuration that demonstrates our approach:

default:
  image:
    name: amazon/aws-cli:2.25.2
    entrypoint: [""]

stages:
  - pre-deploy
  - deploy

.tenants:
  parallel:
    matrix:
      - TENANT: alpha
      - TENANT: beta

get-aws-credentials:
  stage: pre-deploy
  extends:
    - .tenants
  needs: []
  environment: $TENANT
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gitlab.com # or custom GitLab domain if self-hosted
  script:
    - aws --version
    - >
      STS=($(aws sts assume-role-with-web-identity
        --role-arn ${ROLE_ARN}
        --role-session-name "GitLabRunner-${CI_PROJECT_ID}-${CI_PIPELINE_ID}"
        --web-identity-token ${GITLAB_OIDC_TOKEN}
        --duration-seconds 3600
        --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]'
        --output text))
    - echo "AWS_ACCESS_KEY_ID=${STS[0]}" >> .aws.$TENANT.env
    - echo "AWS_SECRET_ACCESS_KEY=${STS[1]}" >> .aws.$TENANT.env
    - echo "AWS_SESSION_TOKEN=${STS[2]}" >> .aws.$TENANT.env
  artifacts:
    paths:
      - .aws.$TENANT.env
    expire_in: 1 hour

deploy:
  stage: deploy
  extends:
    - .tenants
  needs:
    - get-aws-credentials
  script:
    - export $(cat .aws.$TENANT.env | xargs)
    - aws sts get-caller-identity

How It Works

Parallel Matrix: The .tenants template defines a matrix with two tenants: alpha and beta.
Credential Generation: The get-aws-credentials job uses AWS OIDC to dynamically assume a role for each tenant.
Environment File Creation: Credentials are written to tenant-specific .env files (.aws.alpha.env and .aws.beta.env).
Artifact Sharing: These .env files are shared as artifacts between jobs.
Credential Consumption: The deploy job imports these credentials using export $(cat .aws.$TENANT.env | xargs) command.

This approach results in a pipeline flow like this:

The key insight is that we're using the same naming convention we explored in the previous article — naming our environment files with the tenant identifier (.aws.$TENANT.env). This ensures each deployment job retrieves the correct credentials file.

Working Example

A complete working example of this implementation is available in our public GitLab repository

You can clone this repository to see the complete implementation and adapt it to your own needs.

Setting Up AWS OIDC Integration

To implement this approach, you'll need to configure AWS IAM with the following steps:

1) Add OpenID Connect Identity Provider

Open AWS Management Console
Navigate to IAM -> Identity Providers
Add a new provider with these details:
- Type: OpenID Connect
- Provider: https://gitlab.com (or your custom GitLab domain if self-hosted)
- Audience: https://gitlab.com (should match the domain where GitLab is hosted and GITLAB_OIDC_TOKEN:aud in .gitlab-ci.yml)

2) Create IAM Role

Go to IAM -> Roles -> Create
Configure as a Web entity with:
- Provider: https://gitlab.com (or your custom GitLab domain if self-hosted)
- Audience: https://gitlab.com (should match the domain where GitLab is hosted and GITLAB_OIDC_TOKEN:aud in .gitlab-ci.yml)
- Policy: Appropriate access level
Create a trust policy with example values:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::{ACCOUNT_NUMBER}:oidc-provider/gitlab.com"
                // Example: "arn:aws:iam::77766655544333:oidc-provider/gitlab.com"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "gitlab.com:aud": "https://gitlab.com"
                    // Or your custom GitLab domain if self-hosted
                },
                "StringLike": {
                    "gitlab.com:sub": "project_path:{PROJECT_PATH}*"
                    // Example: "project_path:u11d-michal-miler/dynamic-envs-aws*"
                }
            }
        }
    ]
}

3) Configure GitLab CI/CD

Copy the Role ARN
Set it as a variable in GitLab CI/CD settings (Settings -> CI/CD -> Variables)

Security Considerations

When implementing this approach, several security aspects deserve special attention:

Artifact Expiration: Set short expiration times for artifacts containing credentials to limit how long sensitive information remains accessible in GitLab storage. In our example, we use expire_in: 1 hour to ensure credentials are automatically removed.
Least Privilege IAM Roles: Follow the principle of least privilege by creating environment-specific roles with only the necessary permissions for each deployment task. Regularly audit these roles to maintain tight security.
Trust Policy Scope: Configure the project_path condition carefully in your trust policy. Consider using specific path constraints or additional conditions like gitlab.com:ref_type and gitlab.com:ref to restrict which branches can assume roles, especially for production environments.
Environment Variables vs. Files: While our approach uses files to pass credentials, this isolates them from GitLab's variable system and log outputs, reducing the risk of accidental exposure.

Remember that your deployment security depends on properly configuring each component of this pipeline. Review all settings thoroughly before using in production environments.

Conclusion

The technique explained in this article builds upon our previous approach for sharing artifacts and extends it to handle dynamic environment variables — particularly sensitive ones like cloud credentials.

This method isn't limited to AWS credentials. You can use similar approaches for:

Generating credentials for multiple cloud providers
Passing dynamic configuration between jobs
Managing database connection strings for different environments
Sharing temporary access tokens across deployment stages
Creating environment-specific configuration dynamically

The key insight is that environment-specific files, combined with GitLab's artifact sharing mechanism and a consistent naming pattern, provide a robust solution for passing dynamic variables between matrix jobs.

By leveraging these techniques, you can create truly flexible CI/CD pipelines that work seamlessly across multiple environments and tenants while maintaining proper isolation and security.