Solved: How to reference secrets during deployment?

🚀 Executive Summary

TL;DR: Securely referencing secrets during application deployment is crucial to prevent vulnerabilities. This guide details robust solutions, including dedicated secret management services, secure CI/CD environment variables, and encrypted configuration files, to safely handle sensitive data throughout CI/CD pipelines.

🎯 Key Takeaways

• Hardcoding secrets or storing unencrypted secrets in version control are critical security flaws that expose sensitive data.
• Dedicated secret management services (e.g., AWS Secrets Manager, HashiCorp Vault) provide centralized, auditable, and identity-based access control with features like automated rotation and dynamic secrets.
• Secure CI/CD environment variables (e.g., GitHub Actions Secrets) offer a simpler, integrated solution for static secrets, automatically masked in logs within the pipeline context.
• Encrypted configuration files (e.g., SOPS, Sealed Secrets) enable GitOps by storing encrypted secrets alongside code, with decryption occurring securely at deployment time using a managed key.
• The choice of solution depends on project scale, security requirements, and GitOps alignment, balancing complexity with features like dynamic credential generation and fine-grained access control.

Navigating the complexities of secure secret management during application deployments is a critical challenge for IT professionals. This guide explores common pitfalls and presents robust, detailed solutions for safely referencing sensitive data throughout your CI/CD pipelines.

Symptoms: The Pain Points of Poor Secret Management

Improperly handling secrets during deployment can lead to significant security vulnerabilities, operational inefficiencies, and compliance issues. Recognizing these symptoms is the first step toward building a more secure and robust deployment pipeline.

• ### Hardcoding Secrets

Problem: Developers often hardcode API keys, database credentials, or access tokens directly into source code or configuration files. This is the most egregious security flaw.

Impact: Secrets become visible to anyone with access to the codebase (even in private repositories, access can be broader than intended), making them difficult to rotate and highly susceptible to exposure if the repository is compromised.

• ### Secrets in Version Control (Unencrypted)

Problem: Storing unencrypted sensitive data in Git or other version control systems, even in separate files from the main codebase.

Impact: Similar to hardcoding, but slightly more organized. Historical commits can retain secrets even if later removed, creating a persistent risk.

• ### Passing Secrets as Plaintext Environment Variables (Locally/Improperly)

Problem: Relying on plaintext environment variables in local development or insecure deployment environments where they might be logged or exposed in process lists.

Impact: While better than hardcoding, environment variables can still be viewed by other processes on the same machine, or inadvertently logged by the application or deployment system if not handled carefully.

• ### Manual Secret Management

Problem: Manually entering secrets into deployment tools, servers, or applications during each deployment cycle.

Impact: Error-prone, slow, doesn’t scale, lacks auditability, and creates a single point of failure where a human operator could mistakenly expose a secret.

• ### Lack of Secret Rotation

Problem: Secrets are rarely or never updated, even if their lifespan should be short.

Impact: Increases the window of opportunity for a compromised secret to be exploited. Best practice dictates regular rotation, often automated.

Solution 1: Dedicated Secret Management Services

Dedicated secret management services provide a centralized, secure, and auditable way to store, access, and manage secrets. These platforms are purpose-built for enterprise-grade security and offer features like encryption, access control (RBAC), auditing, and automated rotation.

How it Works

Applications or deployment pipelines authenticate with the secret manager (typically using identity-based authentication like IAM roles, service accounts, or trusted client certificates), retrieve the necessary secrets at runtime, and use them. The secrets are never exposed in plaintext in the codebase or logs.

Example: AWS Secrets Manager with an EC2 Instance

Let’s illustrate with an EC2 instance needing to access a database password stored in AWS Secrets Manager.

Step 1: Store the Secret in AWS Secrets Manager

Navigate to AWS Secrets Manager, choose to store a new secret. For this example, we’ll store database credentials.

# Example JSON for a database secret
{
  "username": "dbuser",
  "password": "mySecurePassword123!",
  "engine": "mysql",
  "host": "mydb.c1abcdefghij.us-east-1.rds.amazonaws.com",
  "port": 3306
}

Give it a name like my-app/db-credentials.

Step 2: Create an IAM Role and Policy

Create an IAM policy that grants permission to retrieve the specific secret. Attach this policy to an IAM role that your EC2 instance will assume.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret"
            ],
            "Resource": "arn:aws:secretsmanager:REGION:ACCOUNT_ID:secret:my-app/db-credentials-*"
        }
    ]
}

Attach this policy to an IAM Role, e.g., MyAppEC2Role.

Step 3: Assign the IAM Role to Your EC2 Instance

When launching your EC2 instance, or by modifying an existing one, assign the MyAppEC2Role to it.

Step 4: Application Code to Retrieve Secret

Your application (e.g., Python) uses the AWS SDK to retrieve the secret at runtime. Because the EC2 instance has the assigned IAM role, it automatically authenticates without needing explicit credentials in the code.

import boto3
import json

def get_secret():
    secret_name = "my-app/db-credentials"
    region_name = "us-east-1" # Replace with your AWS region

    # Create a Secrets Manager client
    session = boto3.session.Session()
    client = session.client(
        service_name='secretsmanager',
        region_name=region_name
    )

    try:
        get_secret_value_response = client.get_secret_value(
            SecretId=secret_name
        )
    except Exception as e:
        print(f"Error retrieving secret: {e}")
        raise

    if 'SecretString' in get_secret_value_response:
        secret = get_secret_value_response['SecretString']
        return json.loads(secret)
    else:
        # For binary secrets, use get_secret_value_response['SecretBinary']
        return None

if __name__ == "__main__":
    db_credentials = get_secret()
    if db_credentials:
        print(f"Database Username: {db_credentials['username']}")
        print(f"Database Host: {db_credentials['host']}")
        # Now use these credentials to connect to your database
    else:
        print("Failed to retrieve database credentials.")

Key Benefits of Dedicated Secret Management

• Centralized Management: Single source of truth for all secrets.
• Strong Access Control: Fine-grained permissions (who can access which secret, under what conditions).
• Encryption at Rest and in Transit: Secrets are always encrypted.
• Auditing: Comprehensive logs of secret access, creation, and modification.
• Automated Rotation: Many services can automatically rotate credentials for databases, API keys, etc.
• Dynamic Secrets: Generate short-lived credentials on demand (e.g., HashiCorp Vault).

Solution 2: Secure CI/CD Environment Variables

For many deployments, especially those not requiring the full feature set of a dedicated secret manager, leveraging the secure secret management capabilities built into modern CI/CD platforms is an effective approach. These platforms provide mechanisms to store secrets securely and inject them as environment variables into your build and deployment jobs.

How it Works

You store secrets (e.g., API keys, tokens) directly in your CI/CD platform’s settings. When a pipeline runs, these secrets are made available as environment variables to the specific job that needs them. Crucially, they are typically masked in logs and not committed to source control.

Example: GitHub Actions Secrets

Let’s say your application needs an API key to deploy to an external service.

Step 1: Store the Secret in GitHub Repository Settings

In your GitHub repository, go to Settings > Secrets and variables > Actions > New repository secret.

Add a new secret, for instance, named DEPLOY_API_KEY with its value.

(Image for illustrative purpose, actual image not included in output.)

Step 2: Use the Secret in Your GitHub Actions Workflow

In your .github/workflows/deploy.yml file, you can reference this secret:

name: Deploy Application

on:
  push:
    branches:
      - main

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '18'

      - name: Install dependencies
        run: npm install

      - name: Run tests (optional)
        run: npm test

      - name: Deploy to Service
        env:
          MY_API_KEY: ${{ secrets.DEPLOY_API_KEY }} # Inject secret as environment variable
        run: |
          echo "Using API key for deployment..."
          # Example: Your deployment command using the environment variable
          npm run deploy -- --api-key $MY_API_KEY
          # Or, if your app directly reads process.env.MY_API_KEY
          echo "Deployment initiated successfully!"

      - name: Post-deployment checks
        run: echo "Deployment complete."

In this example, ${{ secrets.DEPLOY_API_KEY }} fetches the secret from GitHub’s secure storage, and it’s exposed as the MY_API_KEY environment variable only for the duration of that specific step.

Key Benefits and Considerations for CI/CD Environment Variables

• Simplicity: Easy to set up for smaller projects or where a dedicated secret manager is overkill.
• Integration: Seamlessly integrated into the CI/CD workflow.
• Masking: Most CI/CD platforms automatically mask secret values in logs to prevent accidental exposure.
• Auditing: CI/CD platforms usually log who created/modified secrets. Pipeline runs are auditable.
• Scope: Secrets can often be scoped to specific projects, environments, or even branches.
• Limitation: While secure within the CI/CD context, these secrets are typically static and don’t offer dynamic credential generation or complex rotation policies like dedicated secret managers.

Solution 3: Encrypted Configuration Files (e.g., SOPS, Sealed Secrets)

This approach involves encrypting configuration files that contain secrets and storing these encrypted files directly in version control alongside your application code. Decryption happens only at the deployment stage using a secure key, allowing for GitOps principles while keeping secrets safe.

How it Works

You use a tool like Mozilla SOPS or Bitnami Sealed Secrets to encrypt structured data files (YAML, JSON, .env). The encryption key is managed separately (e.g., in a KMS, GPG key, or Kubernetes controller). During deployment, your CI/CD pipeline or an in-cluster controller uses the key to decrypt the files just before the application needs them.

Example: Mozilla SOPS with AWS KMS

Let’s encrypt a config.yaml file containing sensitive database connection details.

Step 1: Install SOPS

Download and install SOPS from the official GitHub releases page (https://github.com/getsops/sops/releases).

# Example for Linux/macOS
curl -LO https://github.com/getsops/sops/releases/download/v3.7.3/sops-v3.7.3.linux.amd64
sudo mv sops-v3.7.3.linux.amd64 /usr/local/bin/sops
sudo chmod +x /usr/local/bin/sops

Step 2: Create a KMS Key (AWS)

Create an AWS KMS customer master key (CMK) that SOPS will use for encryption/decryption.

# Using AWS CLI
aws kms create-key --description "SOPS encryption key"
# Output will include KeyId and Arn. Note the Arn.
# Example Arn: arn:aws:kms:us-east-1:ACCOUNT_ID:key/YOUR_KMS_KEY_ID

Ensure the IAM user/role used by your CI/CD pipeline has permissions to encrypt/decrypt with this KMS key.

Step 3: Create Your Secret Configuration File

Create a config.yaml file with your secrets.

# config.yaml (This file will NOT be committed directly)
database:
  host: my-db-host.com
  port: 5432
  username: admin
  password: mySuperSecretPassword123
api:
  key: abcdef123456
  secret: anotherSuperSecret

Step 4: Encrypt the File with SOPS

Use SOPS to encrypt config.yaml using your KMS key. Replace REGION, ACCOUNT_ID, and YOUR_KMS_KEY_ID with your actual values.

sops --encrypt --kms arn:aws:kms:REGION:ACCOUNT_ID:key/YOUR_KMS_KEY_ID \
     config.yaml > encrypted_config.yaml

The encrypted_config.yaml file is safe to commit to version control.

# encrypted_config.yaml (Contents after encryption)
database:
  host: ENC[AES256_GCM,data:...,iv:...,tag:...]
  port: ENC[AES256_GCM,data:...,iv:...,tag:...]
  username: ENC[AES256_GCM,data:...,iv:...,tag:...]
  password: ENC[AES256_GCM,data:...,iv:...,tag:...]
api:
  key: ENC[AES256_GCM,data:...,iv:...,tag:...]
  secret: ENC[AES256_GCM,data:...,iv:...,tag:...]
sops:
  kms:
    - arn: arn:aws:kms:REGION:ACCOUNT_ID:key/YOUR_KMS_KEY_ID
      created_at: "2023-10-27T10:00:00Z"
      enc: ENC[AES256_GCM,data:...,iv:...,tag:...]
  mac: ENC[AES256_GCM,data:...,iv:...,tag:...]
  version: 3.7.3

Step 5: Decrypt in Your CI/CD Pipeline

In your CI/CD pipeline (e.g., GitHub Actions, GitLab CI, Jenkins), during the deployment step, you’ll decrypt the file. Ensure your CI/CD runner has the necessary AWS credentials (e.g., via OIDC, environment variables, or IAM role) to access the KMS key.

# Example CI/CD step
- name: Decrypt secrets
  run: sops --decrypt encrypted_config.yaml > decrypted_config.yaml
  env:
    AWS_REGION: us-east-1 # Important for KMS
    AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} # Or use OIDC/IAM roles
    AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

- name: Deploy application
  run: |
    # Your deployment logic that uses decrypted_config.yaml
    # e.g., load into environment variables or pass as file to app
    DB_PASSWORD=$(yq '.database.password' decrypted_config.yaml)
    API_KEY=$(yq '.api.key' decrypted_config.yaml)
    echo "Deploying with database password: $DB_PASSWORD (masked)"
    ./my-app-deploy-script --db-password "$DB_PASSWORD" --api-key "$API_KEY"

The decrypted_config.yaml should *not* be committed to version control and should be cleaned up after deployment.

Key Benefits and Considerations for Encrypted Configuration Files

• GitOps Friendly: Secrets are versioned alongside code, making rollbacks and environment management consistent.
• Auditability (of changes): Git history tracks changes to encrypted secrets.
• Flexibility: Supports various key management systems (KMS, GPG, Vault).
• No External API Calls at Runtime (for application): Application consumes a local file, reducing runtime dependencies on secret managers.
• Key Management: The encryption key itself must be securely managed outside of version control.
• Decryption Point: The CI/CD pipeline or deployment environment needs access to the decryption key, making that a critical security point.
• Sealed Secrets for Kubernetes: A Kubernetes-native alternative that encrypts Kubernetes Secret objects into SealedSecret resources, which can be safely stored in Git and decrypted by an in-cluster controller.

Comparison of Secret Management Solutions

Choosing the right solution depends on your specific needs, infrastructure, and security posture. Here’s a comparison to help you decide.

Feature/Solution	Dedicated Secret Manager (e.g., AWS Secrets Manager, Vault)	Secure CI/CD Environment Variables (e.g., GitHub Actions Secrets)	Encrypted Configuration Files (e.g., SOPS, Sealed Secrets)
Primary Use Case	Enterprise-grade, highly secure, dynamic secrets, complex access control, multi-cloud/hybrid environments.	Simpler deployments, cloud-native CI/CD, moderate security needs, static secrets.	GitOps-centric, versioned secrets alongside code, infrastructure as code (IaC), Kubernetes-native.
Secret Storage Location	Centralized, dedicated service (cloud provider or self-hosted).	CI/CD platform’s secure vault.	Version control (Git) in encrypted form.
Runtime Retrieval	Application makes API call to secret manager.	CI/CD injects into deployment script/container as ENV var. Application reads ENV var.	CI/CD decrypts to file. Application reads file/ENV vars from file. (Sealed Secrets: in-cluster controller decrypts).
Security Model	Robust, identity-based access (IAM roles, service accounts), encryption, auditing, auto-rotation.	Platform-level encryption, access controlled by CI/CD permissions, variable masking.	Key-based encryption (KMS, GPG), access determined by key permissions, version control history.
Complexity	High setup and operational overhead, but high reward for large scale.	Low setup, integrated into CI/CD.	Moderate setup (tool installation, key management), low runtime complexity.
GitOps Alignment	Good, CI/CD references secrets by name/path. Configuration in Git references secrets, but secrets themselves are external.	Moderate, CI/CD pipeline configuration is in Git, but secrets are external.	Excellent, encrypted secrets are stored directly in Git, facilitating full declarative deployments.
Dynamic Secrets	Yes (e.g., HashiCorp Vault for on-demand credentials).	No (typically static values).	No (typically static values, though content can be frequently updated).
Auditing	Comprehensive logs of all secret access attempts and changes.	CI/CD pipeline logs show secret usage; audit logs for secret changes.	Git history for changes to encrypted files; KMS/GPG logs for key usage.

Conclusion

Securely referencing secrets during deployment is non-negotiable for modern IT operations. While hardcoding and plaintext storage are strictly forbidden, multiple robust solutions exist, each with its strengths. Dedicated secret management services offer the highest level of security and flexibility for complex environments. CI/CD-managed environment variables provide a straightforward and secure option for simpler deployments. Encrypted configuration files, particularly with tools like SOPS or Sealed Secrets, excel in GitOps-driven workflows by keeping secrets versioned alongside code. By understanding these options and implementing them diligently, you can significantly enhance the security posture of your applications and deployment pipelines.

---

👉 Read the original article on TechResolve.blog