DEV Community

Cover image for Never Commit Secrets Again: Generate .env Files from AWS Secrets Manager
Mateen Anjum
Mateen Anjum

Posted on

Never Commit Secrets Again: Generate .env Files from AWS Secrets Manager

#aws #security #secret #dotenv

TL;DR: Store secrets in AWS Secrets Manager. Generate .env files on demand with a Python script. Never commit credentials again.
a

The Problem

Every team commits secrets eventually. GitHub detected over 12 million exposed credentials last year through their secret scanning.

The usual approaches all have failure modes:

  • .gitignore fails when developers forget to add it, or clone fresh and ask for the file via Slack
  • SOPS encryption still puts files in git, adds key management overhead, and creates merge conflict nightmares
  • .env.example templates get stale and require manual copying

We needed something better: secrets that live outside the repository entirely, with a frictionless developer experience.

The Solution

Secrets live in AWS Secrets Manager. Developers run one command to generate their .env file:

make env
# .env is generated locally, ready to use
Enter fullscreen mode Exit fullscreen mode

The file is gitignored. It never touches version control. When secrets change in AWS, developers regenerate and get the latest values.

Implementation

1. Organize Secrets in AWS

Structure your secrets by application and environment:

/myapp/dev/database      → {"DB_HOST": "...", "DB_PASSWORD": "..."}
/myapp/dev/api-keys      → {"STRIPE_KEY": "...", "SENDGRID_KEY": "..."}
/myapp/prod/database     → {"DB_HOST": "...", "DB_PASSWORD": "..."}
/myapp/prod/api-keys     → {"STRIPE_KEY": "...", "SENDGRID_KEY": "..."}
Enter fullscreen mode Exit fullscreen mode

Create secrets using AWS CLI:

aws secretsmanager create-secret \
  --name /myapp/dev/database \
  --secret-string '{"DB_HOST":"localhost","DB_PASSWORD":"devpass123"}'
Enter fullscreen mode Exit fullscreen mode

2. The Python Script

Here's the full script that generates .env files:

#!/usr/bin/env python3
"""
Generate .env file from AWS Secrets Manager.

Usage:
    python generate_env.py dev
    python generate_env.py prod --force
"""

import argparse
import json
import os
import sys
from pathlib import Path

import boto3
from botocore.exceptions import ClientError, NoCredentialsError

# Configuration
APP_NAME = "myapp"
AWS_REGION = os.environ.get("AWS_REGION", "us-east-1")
ENV_FILE = ".env"

SECRET_KEYS = ["database", "api-keys", "third-party"]


def get_secret(secret_name: str, region: str) -> dict:
    """Fetch a secret from AWS Secrets Manager."""
    client = boto3.client("secretsmanager", region_name=region)

    try:
        response = client.get_secret_value(SecretId=secret_name)
        return json.loads(response.get("SecretString", "{}"))
    except ClientError as e:
        if e.response["Error"]["Code"] == "ResourceNotFoundException":
            print(f"  Warning: Secret '{secret_name}' not found")
            return {}
        raise


def validate_aws_credentials() -> bool:
    """Check if AWS credentials are configured."""
    try:
        sts = boto3.client("sts")
        identity = sts.get_caller_identity()
        print(f"Authenticated as: {identity['Arn']}")
        return True
    except NoCredentialsError:
        print("Error: AWS credentials not found.")
        print("\nFix with one of:")
        print("  1. aws configure")
        print("  2. Set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY")
        print("  3. Use IAM role (if on AWS)")
        return False


def fetch_all_secrets(environment: str, region: str) -> dict:
    """Fetch all secrets for the environment."""
    all_secrets = {}
    for key in SECRET_KEYS:
        secret_path = f"/{APP_NAME}/{environment}/{key}"
        print(f"  Fetching: {secret_path}")
        all_secrets.update(get_secret(secret_path, region))
    return all_secrets


def generate_env_content(secrets: dict) -> str:
    """Generate .env content from secrets."""
    lines = [
        "# Auto-generated from AWS Secrets Manager",
        "# DO NOT COMMIT THIS FILE",
        "",
    ]
    for key, value in sorted(secrets.items()):
        if isinstance(value, str) and " " in value:
            value = f'"{value}"'
        lines.append(f"{key}={value}")
    return "\n".join(lines) + "\n"


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("environment", choices=["dev", "staging", "prod"])
    parser.add_argument("-f", "--force", action="store_true")
    parser.add_argument("-o", "--output", default=ENV_FILE)
    args = parser.parse_args()

    print(f"Generating .env for '{args.environment}'\n")

    if not validate_aws_credentials():
        sys.exit(1)

    secrets = fetch_all_secrets(args.environment, AWS_REGION)

    if not secrets:
        print(f"\nError: No secrets found at /{APP_NAME}/{args.environment}/*")
        sys.exit(1)

    print(f"\nFound {len(secrets)} secret values")

    content = generate_env_content(secrets)
    path = Path(args.output)

    if path.exists() and not args.force:
        if input(f"{args.output} exists. Overwrite? [y/N]: ").lower() != "y":
            sys.exit(0)

    path.write_text(content)
    print(f"Generated: {args.output}")


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

3. Shell Wrapper and Makefile

Create a shell wrapper for convenience:

#!/bin/bash
# generate-env.sh
set -e
ENV=${1:-dev}

if ! python3 -c "import boto3" 2>/dev/null; then
    pip3 install boto3 --quiet
fi

python3 "$(dirname "$0")/generate_env.py" "$ENV" "${@:2}"
Enter fullscreen mode Exit fullscreen mode

Add Makefile targets:

.PHONY: env env-dev env-prod

env:
    @./scripts/generate-env.sh dev

env-dev:
    @./scripts/generate-env.sh dev

env-prod:
    @./scripts/generate-env.sh prod

env-dry:
    @./scripts/generate-env.sh dev --dry-run
Enter fullscreen mode Exit fullscreen mode

4. GitHub Actions with OIDC

No stored credentials needed. Use OIDC to assume an AWS role:

name: Deploy

on:
  push:
    branches: [main]

permissions:
  id-token: write
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions
          aws-region: us-east-1

      - name: Generate .env
        run: |
          pip install boto3
          python scripts/generate_env.py prod --force

      - name: Deploy
        run: |
          # Your deployment commands
          echo "Deploying..."

      - name: Cleanup
        if: always()
        run: rm -f .env
Enter fullscreen mode Exit fullscreen mode

5. GitLab CI

Same pattern with GitLab's OIDC:

deploy:
  stage: deploy
  image: python:3.11-slim
  script:
    - pip install boto3
    - |
      export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s"
      $(aws sts assume-role-with-web-identity
      --role-arn ${AWS_ROLE_ARN}
      --role-session-name "gitlab-${CI_PIPELINE_ID}"
      --web-identity-token ${CI_JOB_JWT_V2}
      --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]'
      --output text))
    - python scripts/generate_env.py prod --force
    - echo "Deploying..."
  after_script:
    - rm -f .env
Enter fullscreen mode Exit fullscreen mode

IAM Permissions

Developers need:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue"],
      "Resource": "arn:aws:secretsmanager:us-east-1:*:secret:/myapp/dev/*"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

CI/CD roles need access to prod secrets:

{
  "Effect": "Allow",
  "Action": ["secretsmanager:GetSecretValue"],
  "Resource": "arn:aws:secretsmanager:us-east-1:*:secret:/myapp/prod/*"
}
Enter fullscreen mode Exit fullscreen mode

OIDC Setup for GitHub Actions

  1. Create the OIDC provider in AWS: 
aws iam create-open-id-connect-provider \
  --url https://token.actions.githubusercontent.com \
  --client-id-list sts.amazonaws.com
Enter fullscreen mode Exit fullscreen mode
  1. Create the trust policy: 
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repo:your-org/your-repo:*"
        }
      }
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Results

After implementing this:

Metric Before After
Secrets in git 47 0
Rotation time 2 hours 5 minutes
Setup time 45 min 10 min
Slack secret sharing Weekly Never

Repository Structure 

myapp/
├── scripts/
│   ├── generate_env.py
│   └── generate-env.sh
├── .github/
│   └── workflows/
│       └── deploy.yml
├── .gitignore          # includes .env
├── .env.example        # dummy values for reference
├── Makefile
└── README.md
Enter fullscreen mode Exit fullscreen mode

Try It

The full code is available at github.com/mateenali66/secrets-env-generator.

Clone it, configure your AWS credentials, create some test secrets, and run make env.

Questions? Contact me on .

Top comments (0)