I Almost Leaked My Production Database Password to GitHub (And How I Fixed CI/CD Security)

#cicd #devops #github #security

I Almost Leaked My Production Database Password to GitHub (And How I Fixed CI/CD Security)

Why GitHub Actions secrets aren't enough, and what solo developers actually need to know

The Moment

I'm reviewing a GitHub Actions log.

Something catches my eye in the build output:

Step 5: Running tests
DATABASE_URL=postgresql://root:MySecretPassword123!@prod-db.aws.rds.amazonaws.com:5432/vehicle_metrics
Running pytest...

Wait. My production database password is logged in plain text in a GitHub Actions run.

Not in a secret file. Not hidden. In the public build log.

Anyone with access to the repository could scroll through the logs and find it.

I had 30 seconds of pure panic.

How It Happened

Here's what I did wrong:

# ❌ VULNERABLE
name: Test
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run tests
        run: pytest backend/tests/
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}

This looks safe, but if my test code does this:

# ❌ BAD TEST CODE
def test_database_connection():
    print(f"Connecting to: {os.getenv('DATABASE_URL')}")  # LOGS THE SECRET!

The secret gets printed. GitHub logs it. Everyone sees it.

The Fix

Step 1: Clean Error Messages

# ✅ GOOD
def sanitize_for_logging(connection_string):
    return connection_string.split('@')[1] if '@' in connection_string else "***"

try:
    db_url = os.getenv('DATABASE_URL')
    logger.info(f"Connecting to: {sanitize_for_logging(db_url)}")
    connection = psycopg2.connect(db_url)
except Exception as e:
    logger.error(f"Connection failed: {type(e).__name__}")
    raise

Step 2: Use Temporary Credentials

# ✅ BEST: Use AWS IAM
name: Test
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    permissions:
      id-token: write

    steps:
      - uses: actions/checkout@v3

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v2
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions
          aws-region: us-east-1

      - name: Run tests
        run: pytest backend/tests/
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}

Step 3: Scan for Secrets

# ✅ ADD: Secret scanning
jobs:
  secret-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: TruffleHog secret scan
        uses: trufflesecurity/trufflehog@main
        with:
          path: ./
          base: ${{ github.event.repository.default_branch }}
          head: HEAD

The Complete Secure Pipeline

name: Secure CI/CD
on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main, develop]

jobs:
  secret-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - uses: trufflesecurity/trufflehog@main
        with:
          path: ./
          base: ${{ github.event.repository.default_branch }}
          head: HEAD

  dependency-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: aquasecurity/trivy-action@master
        with:
          scan-type: 'fs'
          scan-ref: '.'
          format: 'sarif'
          output: 'trivy-results.sarif'
      - uses: github/codeql-action/upload-sarif@v2
        with:
          sarif_file: 'trivy-results.sarif'

  test:
    runs-on: ubuntu-latest
    permissions:
      id-token: write

    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - uses: aws-actions/configure-aws-credentials@v2
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions
          aws-region: us-east-1

      - run: pip install -r requirements.txt pytest pytest-cov
      - run: pytest backend/tests/ --cov=backend --cov-report=xml
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}

      - uses: codecov/codecov-action@v3
        with:
          files: ./coverage.xml

  deploy:
    runs-on: ubuntu-latest
    needs: [secret-scan, dependency-scan, test]
    if: github.ref == 'refs/heads/main'
    permissions:
      id-token: write

    steps:
      - uses: actions/checkout@v3
      - uses: aws-actions/configure-aws-credentials@v2
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions-deploy
          aws-region: us-east-1
      - run: |
          aws eks update-kubeconfig --name vehicle-metrics
          kubectl apply -f infrastructure/kubernetes/

Security Checklist

☐ Secrets in GitHub Secrets (not in code)
☐ Secrets masked in logs
☐ No debug logging with sensitive data
☐ Use IAM roles instead of long-lived credentials
☐ Rotate credentials regularly
☐ Scan for secrets on every push (TruffleHog)
☐ Scan dependencies (Trivy)
☐ SAST scanning (SonarQube)
☐ Container scanning (Grype, Snyk)
☐ Limited permissions per job (RBAC)
☐ Audit logs reviewed regularly
☐ Separate test/staging/prod credentials
☐ Encrypted secrets at rest
☐ MFA for manual deployment
☐ Secrets expire automatically

Real World Gotchas

Gotcha 1: Secrets in Environment Variables

# ❌ BAD
export DATABASE_URL="postgresql://root:password@localhost/db"

# ✅ GOOD
export DATABASE_URL=$(aws secretsmanager get-secret-value --secret-id db-password)

Gotcha 2: Secrets in Docker Build

# ❌ BAD: Secret baked into image
ARG DB_PASSWORD=prod_secret_here
RUN echo $DB_PASSWORD > /app/config

# ✅ GOOD: Only runtime secrets
ARG VERSION=1.0
ENV VERSION=$VERSION

Gotcha 3: Secrets in Error Messages

# ❌ BAD
try:
    connect(os.getenv('DATABASE_URL'))
except Exception:
    print(traceback.format_exc())  # Includes the URL!

# ✅ GOOD
except Exception as e:
    logger.error(f"Connection failed: {type(e).__name__}")