Part 6: CI/CD Pipeline

#devops #docker #kubernetes #githubactions

Part 6: CI/CD Pipeline — GitHub Actions, Trivy, Cosign, and ECR

Part of the series: Building a Production-Grade DevSecOps Pipeline on AWS

Introduction

The CI/CD pipeline is the gateway between a developer's git push and a running container in production. Every security control that can be automated should live here — not as an afterthought but as a first-class gate that blocks bad artifacts from ever reaching a cluster.

This pipeline enforces:

No HIGH or CRITICAL CVEs — Trivy blocks the build before the image is pushed
No static AWS credentials — GitHub OIDC exchanges JWT tokens for temporary STS creds
Cryptographic image provenance — Cosign signs every image with an AWS KMS key; Kyverno verifies the signature before admitting the pod
Immutable image tags — sha-<full-commit-sha>, never :latest
Audit trail — every push is logged to S3 with digest, timestamp, and caller identity

The Application (myapp)

A minimal Node.js/Express API that represents any real production service:

myapp/
├── src/
│   └── index.js        # Express app with /health and /metrics
├── Dockerfile
├── package.json
└── .github/
    └── workflows/
        └── ci.yaml

// src/index.js
const express = require('express');
const promClient = require('prom-client');

const app = express();
const register = new promClient.Registry();
promClient.collectDefaultMetrics({ register });

const httpRequestsTotal = new promClient.Counter({
  name: 'myapp_http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status_code'],
  registers: [register],
});

app.use((req, res, next) => {
  res.on('finish', () => {
    httpRequestsTotal.inc({
      method: req.method,
      route: req.path,
      status_code: res.statusCode,
    });
  });
  next();
});

app.get('/health', (req, res) => {
  res.json({ status: 'healthy', region: process.env.AWS_REGION || 'unknown' });
});

app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

app.listen(8080, () => console.log('Listening on :8080'));

Dockerfile — Distroless Nonroot

# Stage 1: Build dependencies
FROM node:18-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

# Stage 2: Runtime — distroless (no shell, no package manager)
FROM gcr.io/distroless/nodejs18-debian12:nonroot

WORKDIR /app

# Copy only what's needed — no node_modules dev dependencies, no source maps
COPY --from=builder /app/node_modules ./node_modules
COPY src/ ./src/
COPY package.json ./

# nonroot image runs as uid 65532 by default — no root, ever
EXPOSE 8080

CMD ["src/index.js"]

Why distroless?
The nonroot variant contains only the Node.js runtime. There is no:

/bin/sh or /bin/bash — an attacker with RCE cannot spawn an interactive shell
apt, apk, yum — cannot install additional tools
curl, wget — cannot exfiltrate data or download payloads
Any other utility that would help lateral movement

Falco still alerts on any unexpected syscalls, but the attack surface is dramatically reduced.

Pipeline Overview

The complete pipeline: no static AWS credentials anywhere. OIDC authenticates
GitHub Actions to AWS. Every image is signed with Cosign before Kyverno will
admit it to the cluster.

┌─────────────────────────────────────────────────────────────────────────┐
│  git push → main                                                        │
│                    │                                                    │
│                    ▼                                                    │
│         ┌──────────────────┐                                            │
│         │  Job 1: test     │  npm ci + npm test                         │
│         └────────┬─────────┘                                            │
│                  │ needs: test                                          │
│                  ▼                                                      │
│         ┌──────────────────┐                                            │
│         │  Job 2: scan     │  trivy image → fail on HIGH/CRITICAL       │
│         └────────┬─────────┘                                            │
│                  │ needs: scan                                          │
│                  ▼                                                      │
│         ┌──────────────────────────────────────────────┐                │
│         │  Job 3: build-push-sign                      │                │
│         │  ├─ OIDC → assume IAM role (no static keys)  │                │
│         │  ├─ docker build (distroless)                │                │
│         │  ├─ push → ECR us-east-1                     │                │
│         │  ├─ push → ECR us-west-2                     │                │
│         │  ├─ cosign sign (AWS KMS)                    │                │
│         │  └─ S3 audit log                             │                │
│         └────────┬─────────────────────────────────────┘                │
│                  │ needs: build-push-sign                               │
│                  ▼                                                      │
│         ┌──────────────────────────────────────────────┐                │
│         │  Job 4: update-gitops                        │                │
│         │  └─ patch image.tag in values-*.yaml → push  │                │
│         └──────────────────────────────────────────────┘                │
└─────────────────────────────────────────────────────────────────────────┘

Full GitHub Actions Workflow

# .github/workflows/ci.yaml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  ECR_REGISTRY_USE1: 206617159586.dkr.ecr.us-east-1.amazonaws.com
  ECR_REGISTRY_USW2: 206617159586.dkr.ecr.us-west-2.amazonaws.com
  IMAGE_NAME: myapp
  AWS_REGION_USE1: us-east-1
  AWS_REGION_USW2: us-west-2

permissions:
  id-token: write   # Required for OIDC
  contents: write   # Required for gitops update commit
  packages: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '18'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run tests
        run: npm test

  scan:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build image for scanning
        run: |
          docker build -t ${{ env.IMAGE_NAME }}:scan-${{ github.sha }} .

      - name: Trivy vulnerability scan
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ env.IMAGE_NAME }}:scan-${{ github.sha }}
          format: table
          exit-code: '1'              # Fail the pipeline on findings
          severity: 'HIGH,CRITICAL'   # Only fail on HIGH and CRITICAL
          ignore-unfixed: true        # Skip CVEs with no available fix

  build-push-sign:
    needs: scan
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'  # Only push on main branch, not PRs
    outputs:
      image-digest-use1: ${{ steps.push-use1.outputs.digest }}
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ vars.AWS_ROLE_ARN_USE1 }}
          aws-region: ${{ env.AWS_REGION_USE1 }}
          # No static access keys — OIDC exchanges GitHub JWT for STS temp creds

      - name: Login to ECR us-east-1
        uses: aws-actions/amazon-ecr-login@v2
        with:
          region: ${{ env.AWS_REGION_USE1 }}

      - name: Login to ECR us-west-2
        uses: aws-actions/amazon-ecr-login@v2
        with:
          region: ${{ env.AWS_REGION_USW2 }}

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Build and push to us-east-1
        id: push-use1
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: |
            ${{ env.ECR_REGISTRY_USE1 }}/${{ env.IMAGE_NAME }}:sha-${{ github.sha }}
            ${{ env.ECR_REGISTRY_USE1 }}/${{ env.IMAGE_NAME }}:latest
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: Copy image to us-west-2
        run: |
          DIGEST="${{ steps.push-use1.outputs.digest }}"
          docker buildx imagetools create \
            --tag ${{ env.ECR_REGISTRY_USW2 }}/${{ env.IMAGE_NAME }}:sha-${{ github.sha }} \
            ${{ env.ECR_REGISTRY_USE1 }}/${{ env.IMAGE_NAME }}@${DIGEST}

      - name: Install Cosign
        uses: sigstore/cosign-installer@v3

      - name: Sign image with AWS KMS
        env:
          COSIGN_KEY: ${{ vars.COSIGN_KMS_KEY_ARN }}
        run: |
          DIGEST="${{ steps.push-use1.outputs.digest }}"
          IMAGE="${{ env.ECR_REGISTRY_USE1 }}/${{ env.IMAGE_NAME }}@${DIGEST}"

          # Sign the image — creates an OCI attestation artifact in ECR
          cosign sign --key awskms:///${COSIGN_KEY} \
            --yes \
            ${IMAGE}

          echo "Signed: ${IMAGE}"

      - name: Write S3 audit log
        run: |
          DIGEST="${{ steps.push-use1.outputs.digest }}"
          TIMESTAMP=$(date -u +%Y-%m-%dT%H:%M:%SZ)
          CALLER=$(aws sts get-caller-identity --query Arn --output text)

          echo "{
            \"timestamp\": \"${TIMESTAMP}\",
            \"repo\": \"${{ github.repository }}\",
            \"sha\": \"${{ github.sha }}\",
            \"digest\": \"${DIGEST}\",
            \"pushed_by\": \"${CALLER}\",
            \"workflow\": \"${{ github.workflow }}\",
            \"run_id\": \"${{ github.run_id }}\"
          }" | aws s3 cp - \
            s3://${{ vars.AUDIT_BUCKET }}/ci-push-audit/${{ github.sha }}.json

  update-gitops:
    needs: build-push-sign
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - name: Checkout myapp-gitops
        uses: actions/checkout@v4
        with:
          repository: MatthewDipo/myapp-gitops
          token: ${{ secrets.GITOPS_PAT }}
          path: myapp-gitops

      - name: Update image tags
        run: |
          cd myapp-gitops
          NEW_TAG="sha-${{ github.sha }}"

          # Update all environment value files
          for FILE in apps/myapp/values-dev.yaml \
                      apps/myapp/values-staging.yaml \
                      apps/myapp/values-production.yaml; do
            sed -i "s|tag: sha-[a-f0-9]*|tag: ${NEW_TAG}|g" $FILE
            echo "Updated $FILE → ${NEW_TAG}"
          done

      - name: Commit and push
        run: |
          cd myapp-gitops
          git config user.email "ci@github.com"
          git config user.name "GitHub Actions"
          git add apps/myapp/values-*.yaml
          git commit -m "ci: update image tag to sha-${{ github.sha }}"
          git push origin main

GitHub Repository Variables (not secrets)

These are set in the GitHub UI under Settings → Secrets and variables → Actions → Variables:

Variable	Value	Why not a secret?
`AWS_ROLE_ARN_USE1`	`arn:aws:iam::206617159586:role/myapp-dev-use1-github-ci`	Not sensitive — it's a role ARN, useless without the OIDC token
`COSIGN_KMS_KEY_ARN`	`arn:aws:kms:us-east-1:206617159586:key/...`	Not sensitive — KMS key ID alone grants nothing
`AUDIT_BUCKET`	`myapp-ci-audit-206617159586`	Not sensitive

Only GITOPS_PAT (GitHub Personal Access Token to push to myapp-gitops) is a secret.

No AWS_ACCESS_KEY_ID. No AWS_SECRET_ACCESS_KEY. These should never appear in a modern CI pipeline.

Cosign Image Signing

Cosign attaches a cryptographic signature to the image in the same ECR repository. The signature is stored as a separate OCI artifact tagged with the image digest.

ECR Repository: myapp

sha-abc123def456...          ← Your application image
sha256-abc123def456...sig    ← Cosign signature (OCI artifact)
sha256-abc123def456...att    ← Cosign attestation (SBOM, etc.)

Verification (what Kyverno does at admission time):

cosign verify \
  --key awskms:///arn:aws:kms:us-east-1:206617159586:key/YOUR_KEY_ID \
  206617159586.dkr.ecr.us-east-1.amazonaws.com/myapp:sha-abc123

# Output if valid:
# Verification for 206617159586.dkr.ecr.us-east-1.amazonaws.com/myapp:sha-abc123
# The following checks were performed on each of these signatures:
#   - The cosign claims were validated
#   - The signatures were verified against the specified public key

Image Naming Convention

206617159586.dkr.ecr.us-east-1.amazonaws.com/myapp:sha-<full-40-char-git-sha>

Why sha-<full-sha> and not semantic versioning?

Traceability: Given any running pod, you can run git show <sha> to see the exact commit
Immutability: Two builds of the same SHA produce the same image (deterministic builds)
No tag collisions: v1.0.0 can be overwritten; a git SHA cannot
No :latest: :latest is the devil — it means different things at different times and breaks reproducibility

Pre-commit Hooks

Before code ever reaches GitHub, pre-commit hooks catch common issues locally:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml
      - id: check-merge-conflict

  - repo: https://github.com/hadolint/hadolint
    rev: v2.12.0
    hooks:
      - id: hadolint-docker
        args: ['--ignore', 'DL3006']   # DL3006: always tag FROM image

  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.5.0
    hooks:
      - id: detect-secrets
        args: ['--baseline', '.secrets.baseline']

# Install pre-commit
pip install pre-commit
pre-commit install

# Run manually against all files
pre-commit run --all-files

Pipeline Security Properties

Property	How Achieved
No secrets in Git	detect-secrets pre-commit hook
No secrets in CI	OIDC replaces static AWS keys
No HIGH/CRITICAL CVEs	Trivy scan blocks pipeline
No unverified images	Kyverno admission webhook
Immutable artifacts	ECR `IMMUTABLE` tag mutability
Cryptographic provenance	Cosign + AWS KMS signing
Full audit trail	S3 + CloudTrail
Reproducible builds	Pinned base image SHA, `npm ci`

Summary

By the end of Part 6 you have:

✅ A secure Dockerfile using distroless/nonroot with multi-stage build
✅ GitHub Actions pipeline with 4 jobs: test → scan → build/push/sign → gitops update
✅ OIDC-based AWS authentication (zero static credentials)
✅ Trivy CVE scanning blocking HIGH/CRITICAL vulnerabilities
✅ Cosign image signing with AWS KMS
✅ Automatic image tag update in myapp-gitops triggering ArgoCD sync
✅ S3 audit log for every image push
✅ Pre-commit hooks catching issues before they reach CI

Screenshot Placeholders

SCREENSHOT: GitHub Actions — workflow run showing all 4 jobs passing (green checkmarks)

SCREENSHOT: ECR repository showing images with sha- tags and scan results (no HIGH/CRITICAL)

SCREENSHOT: GitHub Actions — Job 3 logs showing "Signed: ..." cosign output