DEV Community: Josh Duffney

Continuously patch GHCR images with Copacetic

Josh Duffney — Fri, 28 Feb 2025 15:05:37 +0000

Keep your GHCR-hosted images secure and updated with an effortless GitHub workflow. Follow along to see how!

⚙️ How It Works:
Here’s the gist of what we’re building:

✅ Query: Fetch all your images from GHCR.
🔍 Scan: Use Trivy to spot vulnerabilities.
🔧 Patch: Apply fixes with Copacetic and update the images on GHCR.

Gathering Images with Glue Code

To patch all your images, we first need to know what’s in your GHCR. Both Trivy and Copacetic have GitHub Actions, making automation a breeze—we can loop through every image in a workflow.

🧐 The Challenge:

Dynamically generating the list of images and tags.

💡 The Solution:

A bit of Go "glue code" that hits GitHub’s REST API, grabs all your images and tags under your username, and outputs them to a matrix.json file. This file feeds into the workflow via the fromJson function, driving the patching process.

🔄 Continuous Patching Workflow

This is where the magic happens—a GitHub workflow that ties everything together. It starts by generating the image list, then loops through each one to scan, patch, and update it on GHCR.

The workflow splits into two jobs:

1️⃣ Setup: Prepares the list of images.
2️⃣ Patch: Runs the scanning, patching, and updating.

1️⃣ The Setup Job

The setup job kicks things off by creating the matrix of images to patch:

setup:
  runs-on: ubuntu-latest
  permissions:
    packages: read
  outputs:
    matrix: ${{ steps.set-matrix.outputs.matrix }}
  steps:
    - name: Checkout to repository
      uses: actions/checkout@v3
    - name: Generate image list
      run: ./listImages
    - name: Set matrix data
      id: set-matrix
      run: echo "matrix=$(jq -c . < ./matrix.json)" >> $GITHUB_OUTPUT

🔹 What's Happening Here?

✔️ Checks out your repo to access necessary files.
✔️ Runs listImages, a Go program that builds matrix.json.
✔️ Stores the matrix in a GITHUB_OUTPUT variable for the next job.

2️⃣ The Patch Job

The patch job takes the matrix and runs the full patching cycle for each image:

patch:
  runs-on: ubuntu-latest
  permissions:
    packages: read
    contents: write
  needs: setup
  strategy:
    fail-fast: false
    matrix:
      images: ${{ fromJson(needs.setup.outputs.matrix) }}

🔹 Key Features

✔️ Depends on setup (via needs: setup).
✔️ Prevents failure from stopping all patches (fail-fast: false).
✔️ Loads image list from matrix.json using fromJson().

🔍 Patch Step 1: Scan with Trivy

- name: Generate Trivy Report
  uses: aquasecurity/trivy-action@0.29.0
  with:
    scan-type: "image"
    format: "json"
    output: "report.json"
    ignore-unfixed: true
    vuln-type: "os"
    image-ref: ${{ env.REGISTRY}}/${{ matrix.images }}

✅ Scans the image for OS vulnerabilities and outputs a report.json file.

📊 Patch Step 2: Count Vulnerabilities

- name: Check vulnerability count
  id: vuln_count
  run: |
    report_file="report.json"
    vuln_count=$(jq 'if .Results then [.Results[] | select(.Class=="os-pkgs" and .Vulnerabilities!=null) | .Vulnerabilities[]] | length else 0 end' "$report_file")
    echo "vuln_count=$vuln_count" >> $GITHUB_OUTPUT

✅ Extracts the number of fixable vulnerabilities and saves the count as vuln_count.

🏷️ Patch Step 3: Create Patch Tag

- name: Create patch tag
  id: patch_tag
  run: |
    imageName=$(echo ${{ matrix.images }} | cut -d ':' -f1)
    current_tag=$(echo ${{ matrix.images }} | cut -d ':' -f2)
    if [[ $current_tag == *-[0-9] ]]; then
      numeric_tag=$(echo "$current_tag" | awk -F'-' '{print $NF}')
      non_numeric_tag=$(echo "$current_tag" | sed "s#-$numeric_tag##g")
      incremented_tag=$((numeric_tag+1))
      new_tag="$non_numeric_tag-$incremented_tag"
    else
      new_tag="$current_tag-1"
    fi
    echo "tag=$new_tag" >> $GITHUB_OUTPUT
    echo "imageName=$imageName" >> $GITHUB_OUTPUT

✅ Generates a new patch tag (e.g., app:0.1.0 → app:0.1.0-1 → app:0.1.0-2) using the static incremental tagging strategy.

🔧 Patch Step 4: Patch with Copacetic

- name: Run copa action
  if: steps.vuln_count.outputs.vuln_count != '0'
  id: copa
  uses: project-copacetic/copa-action@v1.2.1
  with:
    image: ${{ env.REGISTRY}}/${{ matrix.images }}
    image-report: "report.json"
    patched-tag: ${{ steps.patch_tag.outputs.tag }}

✅ Applies patches only if vulnerabilities exist.

📤 Patch Step 5: Push to GHCR

- name: Login to GHCR
  if: steps.copa.conclusion == 'success'
  id: login
  uses: docker/login-action@3.3.0
  with:
    registry: ghcr.io
    username: ${{ github.actor }}
    password: ${{ secrets.GHCR_TOKEN }}
- name: Push patched image
  if: steps.login.conclusion == 'success'
  run: |
    docker push ${{ steps.copa.outputs.patched-image }}

✅ Authenticates with GHCR and pushes the patched image if patching was successful.

Ready to try it?

Check the repo's README to set up your own continuous patching workflow! 🎉

Note: This solution only works for public registries, like GHCR. To use copa to patch locally, see Option 2 in the Copa docs. And for private see Option 1.

⚡ Secure your containers faster—without disrupting your workflow

Josh Duffney — Fri, 28 Feb 2025 15:04:55 +0000

Security flaws in container images can't always wait. Instead of rebuilding or relying on a new base image, you can patch vulnerabilities directly—without disrupting your workflow.

🚀 What you'll learn:

Scan a container image with Trivy.
Apply OS-level patches using Copacetic.
Push the patched image to a container registry.

By the end, you'll know how to quickly patch container images and keep your deployments secure with minimal effort.

Prerequisites

Before you begin, ensure you have the following installed on your system:

Install Copacetic

On macOS with Homebrew:

brew install copa

On Linux:

# Define variables
VERSION="0.9.0"
URL="https://github.com/project-copacetic/copacetic/releases/download/v${VERSION}/copa_${VERSION}_linux_amd64.tar.gz"

# Download, extract, cleanup, and move copa binary
curl -L -o "copa_${VERSION}_linux_amd64.tar.gz" "$URL" && \
tar -xzf "copa_${VERSION}_linux_amd64.tar.gz" copa && \
rm "copa_${VERSION}_linux_amd64.tar.gz" && \
mv copa /usr/bin/

Future releases can be found, here.

Enable containerd image store

Copacetic uses BuildKit to execute patch commands, think apt update and apt upgrade, which are essential for updating and securing container images.

TODO: is this true/ accurate?
https://docs.docker.com/desktop/features/containerd/

The containerd image store simplifies the image-building process by allowing images to be constructed directly within the containerd environment, eliminating the need to start a separate buildkit instance.

Follow these steps to enable the image store:

In Docker Desktop, go to Settings
Select General
Then check Use containerd for pulling and storing images and click Apply & restart

For cases where the image store isn’t available or suitable, see the Buildkit connection examples.

Patch the container image

Copacetic provides two patching choices: update all packages to their latest versions, or apply targeted patches to fix only vulnerable packages, keeping changes minimal and precise.

Option 1: Update all packages

When you're not concerned with updating specific packages, this is the option you want.

Patch the image:
```
copa patch -i jduffney/hello-go:0.1.0
```

Give it a try: Replace jduffney/hello-go:0.1.0 with any container image you'd like and use copa to patch it!

Option 2: Update only targeted packages

This process is straightforward: Trivy scans your container image to generate a vulnerability report, identifying specific issues. Copacetic then ingests this report, utilizing the data to apply targeted patches exclusively to the flagged packages, ensuring precise and efficient remediation.

Generate a vulnerability report:

trivy image --pkg-types os \
--ignore-unfixed -f json \
-o vulnReport.json \
golang:1.24rc3-alpine3.21

Apply targeted patches:

copa patch -i golang:1.24rc3-alpine3.21 \
-r vulnReport.json

Compare Scan Results

Wanna see the patching magic in action? Fire up Trivy to scan your unpatched and patched images—boom, no more vulnerability.

Scan the unpatched image:

trivy image --pkg-types os \
--ignore-unfixed  \
jduffney/hello-go:0.1.0

Scan the patched image:

trivy image --pkg-types os \
--ignore-unfixed  \
jduffney/hello-go:0.1.0-patched

At the time of writing, Copacetic successfully patched all 10 vulnerabilities in the jduffney/hello-go:0.1.0 image.

Push the container image

The only thing left to do is to push the freshly patched container image to your registry.

Use docker push to upload the patched image:

docker tag jduffney/hello-go:0.1.0-patched \
<yourRegistry>/hello-go:0.1.0-patched
docker push <yourRegistry>/hello-go:0.1.0-patched

Note: Replace <yourRegistry> with your DockerHub or ghcr information.

🎉 Congrats! Your image is patched and deployment-ready.

Awesome, right? But imagine automatically keeping every image in your registry patched, all the time.

Discover how in the next post. 👈

Does Patching Containers Feel Like Whack-a-Mole?

Josh Duffney — Fri, 28 Feb 2025 15:03:44 +0000

Years ago, I thought switching to containers meant no more patching things—a dream come true—but the work just switched jerseys.

Instead of remoting into VMs and running updates, I’m now hunting down Dockerfiles and tweaking base images. If anything, patching has become less fun and more scavenger hunt.

Then I stumbled across Copacetic, a new project that’s makes it possible to automate container patching.

Meet the CNCF project Copacetic

Copacetic, or "Copa" for short, is a powerful CLI tool built in Go and based on BuildKit, designed to streamline container image patching. By leveraging vulnerability scanning tools like Trivy, Copa can automatically patch existing container images, saving developers time and effort.

Copa works by parsing vulnerability reports to identify necessary update packages, which are then processed using appropriate package managers like apt or apk. It then applies these updates to the container image with BuildKit, ensuring the image is up to date without requiring a rebuild.

One of Copa’s key benefits is that it doesn't require you to modify their container build process or use specific tools to support patching. Additionally, it reduces the need for specialized knowledge, as it relies on the vulnerability remediation embedded in existing reports from popular scanning tools. This makes it much easier to patch OS-level vulnerabilities without waiting for external dependencies or expert intervention.

Here's a short 4min lightning talk of Copacetic in action:

3 Ways Copacetic Tames CVEs

I’ve seen Copa shine in three practical patterns:

Local Patching: Run the Copa CLI or Docker Desktop plugin. Patch an image on your machine, push it to your registry—done. Perfect for quick fixes.
CI/CD Integration: Add Trivy and Copa to GitHub Actions. It patches mid-build, keeps tags steady, and cuts vulnerabilities without breaking your flow. Security, streamlined.
Registry Automation: Use a little scripting & Copa to scan your registry, patch images in bulk, and push them back with fresh tags. A cron job can drive it—devs grab the updates with no hasle.

Why Copacetic Matters (and What’s Next)

This post unpacked the mess: containers didn’t kill patching—they're still a headache, and Copacetic is like aspirin for the pain.

It’s a new tool that automates the slog—locally, in CI/CD, or at scale—without demanding a full rework. Not every vulnerability vanishes, but it dials the chaos down to manageable.

Ready to simplify patching? The next post walks you step-by-step through using Copacetic—those CVEs are sitting there, waiting for you.

Automate Container Image Patching with Copacetic and GitHub Actions

Josh Duffney — Mon, 02 Oct 2023 13:57:00 +0000

In an era where software deployment speed and security are paramount, automating critical tasks in the development pipeline is essential. GitHub Actions offers a robust platform to achieve this automation seamlessly.

In this article, we'll walk you through the creation of a GitHub Actions workflow that focuses on automating the patching and signing of container images using a CNCF sandbox project Copacetic.

Use copa-action to automate patching

Copacetic is a command-line application that uses reports from vulnerability scanners, such as trivy, to directly patch containers images.

It's able to do this by parsing the vulnerability reports generated by the scanners and creating additional layers on top of the container image that includes patches to the CVEs identified by the scanner.

For my Microsoft Build talk, I had to setup a shell task in the workflow to run the copa CLI, but luckily for you an official action now exists. Here's how you set it up.



# .github/workflows/patch.yml

on:
    push:
      branches:
          - copaGitOps

jobs:
      patch:
          runs-on: ubuntu-latest

          strategy:
            fail-fast: false
            matrix:
                  # provide relevant list of images to scan on each run
                  images: ['s3cexampleacr.azurecr.io/azure-voting-app-rust:v0.1.0-alpha']

          steps:
          - name: Login to Azure
            id: login
            uses: azure/login@v1
            with:
              creds: ${{ secrets.AZURE_CREDENTIALS }}

          - name: Login to Azure Container Registry
            run: |
                az acr login --name ${{ vars.ACR_NAME }}

          - name: Set up Docker Buildx
            uses: docker/setup-buildx-action@v3.0.0

          - name: Generate Trivy Report
            uses: aquasecurity/trivy-action@69cbbc0cbbf6a2b0bab8dcf0e9f2d7ead08e87e4
            with:
              scan-type: 'image'
              format: 'json'
              output: 'report.json'
              ignore-unfixed: true
              vuln-type: 'os'
              image-ref: ${{ matrix.images }}

          - name: Check Vuln Count
            id: vuln_count
            run: |
              report_file="report.json"
              vuln_count=$(jq '.Results | length' "$report_file")
              echo "vuln_count=$vuln_count" >> $GITHUB_OUTPUT

          - name: Extract Patched Tag 
            id: extract_tag
            run: |
              imageName=$(echo ${{ matrix.images }} | cut -d ':' -f1)
              current_tag=$(echo ${{ matrix.images }} | cut -d ':' -f2)

              if [[ $current_tag == *-[0-9] ]]; then
                  numeric_tag=$(echo "$current_tag" | awk -F'-' '{print $NF}')
                  non_numeric_tag=$(echo "$current_tag" | sed "s#-$numeric_tag##g")
                  incremented_tag=$((numeric_tag+1))
                  new_tag="$non_numeric_tag-$incremented_tag"
              else
                  new_tag="$current_tag-1"
              fi

              echo "patched_tag=$new_tag" >> $GITHUB_OUTPUT
              echo "imageName=$imageName" >> $GITHUB_OUTPUT

          - name: Copa Action
            if: steps.vuln_count.outputs.vuln_count != '0'
            id: copa
            uses: project-copacetic/copa-action@v1.0.0
            with:
              image: ${{ matrix.images }}
              image-report: 'report.json'
              patched-tag: ${{ steps.extract_tag.outputs.patched_tag }}
              buildkit-version: 'v0.11.6'
              # optional, default is latest
              copa-version: '0.3.0'

          - name: Docker Push Patched Image
            id: push
            if: steps.login.conclusion == 'success'
            run: |
                # docker push ${{ steps.copa.outputs.patched-image }}
                echo "DIGEST=$(docker push ${{ steps.copa.outputs.patched-image }} | grep -oE 'sha256:[a-f0-9]{64}')" >> $GITHUB_OUTPUT

There's a lot going on here, so let's walk through the steps in the job.

The workflow uses a matrix strategy with a list of images to generate a new job per image in the array of images.

A private Azure Container Registry is used the workflow authenticates to azure and then runs the az acr login command to populate the docker login credentials. Which allows you to use docker commands to interact with the registry.

The Trivy action is used to generate a list of vulnerabilities that exist for the targeted container image and outputs that to a report.json file to be used for patching by Copacetic.

A few lines of Bash are used to populate a variable that contains the number of vulnerabilities found.

Extract Patch Tag is another shell task that uses for awksedfu to determine what the container image patched tag should be following the format tag-1, with -1 indicating the image has been patched.

Copa action installs and runs the Copa CLI to ingest the reports.json and patch the remote container image, which results in a local version of the image that's been patched.

Once copa is done and a newly patched image has been created locally, the docker push command is used to push the image to the container registry, but it also grabs the digest from the image.

Curious what the digest is used for? Keep reading to find out. :)

Signed patched container images with Notation

Notation is another CNCF project that just hit version v1.0.0 in Aug. 202 and it allows you digitally sign artifacts. Those signatures become stamps of approval for those digital artifacts letting you know you can trust them to run in your environment.

And if you're using admission controllers on your cluster with projects like Ratify, those signatures are even more important because without them your containers won't be allowed to deploy on the cluster.

At any rate, adding the notary-actions to your GitHub Action Workflow is fairly straight forward:



- name: Setup Notation
  if: steps.push.conclusion == 'success'
  uses: notaryproject/notation-action/setup@v1
  with:
    version: "1.0.0"

- name: Notation Sign
  if: steps.push.conclusion == 'success'
  uses: notaryproject/notation-action/sign@v1
  with:
    plugin_name: azure-kv
    plugin_url: https://github.com/Azure/notation-azure-kv/releases/download/v1.0.1/notation-azure-kv_1.0.1_linux_amd64.tar.gz
    plugin_checksum: f8a75d9234db90069d9eb5660e5374820edf36d710bd063f4ef81e7063d3810b
    key_id: ${{ vars.KEY_ID }}
    target_artifact_reference: ${{ steps.extract_tag.outputs.imageName }}@${{ steps.push.outputs.DIGEST }}
    signature_format: cose
    plugin_config: |-
        name=${{ vars.CERT_NAME }}
        self_signed=false

notation-action/setup installs the notation CLI on the GitHub action runner at the specified version.

notation-action/sign' optionally installs a plugin, in this case, the azure-kv plugin. It then takes the necessary arguments as parameters and then passes them to thenotation sign` command that digitally signs the target artifact (container image)

Test the Workflow

If you've followed along with each tutorial in this series, you have the Flux resources for the Azure Voting app deployed along with Flux resources for the image automation to detect when new container images are pushed to the container registry. And now, you have a workflow that patches the Azure Voting app container images.

So, now the fun begins. Once you commit and push the .github.com/workflows/patch.ymlfile to the repository it will patch the Azure voting container image and push it to the container registry. Once that image is pushed the Flux image automation will detect that a new version of the image exists. When that happens the image automation will update the kustomization.yaml with the newer tag, which will in turn trigger the Flux resources monitor your application's repo to redeploy the application to the Azure Kubernetes Cluster.

Run the following commands & watch the magic happen:

git add . && git commit -m 'Add copa patching workflow' && git push

Configure Image Automation with Flux

Josh Duffney — Fri, 29 Sep 2023 15:05:24 +0000

In this post you'll continue the work built in Part 1 by adding Image Automation to the Azure Voting App deployment using FluxCD.

FluxCD's Image Automation allows you to monitor container image registries, allowing it to detect new image versions and automatically triggering updates according to the policies you define.

You can accomplish this by deploying three additional FluxCD resources, an image repository, image policy, and update automation. In addition to these three resources, you'll also make a slight change to the Kustomization.yml by marking the images so Flux knows what container images to update in the deployment.

Let's get started!

Create an Image Repository

Before Flux can monitor a registry for new images, it must know what repository to monitor and that's the purpose of the Image Repository resource.

Now, because this tutorial uses a private Azure Container Registry instance you must do a little leg work first to set up a credential that Flux can use to authenticate to the registry.

Flux has several authentication options, but in this tutorial, you'll create a Kubernetes secret that stores an ACR token.

Run the following commands to create the ACR token, then create the Kubernetes secret:

registry=s3cexampleacr.azurecr.io
tokenName=flux

tokenPassword=$(az acr token create \
        --name $tokenName \
        --registry $registry \
        --scope-map _repositories_admin \
        --query 'credentials.passwords[0].value' \
        --only-show-errors \
        --output tsv)

kubectl create secret docker-registry regcred \
  --docker-server=$registry \
  --docker-username=$tokenName \
  --docker-password=$tokenPassword \
  --docker-email=fluxcdbot@users.noreply.github.com \
  --namespace=flux-system

Once the ACR token is stored in a Kubernetes secret, run the following command to create the image repository:

flux create image repository azure-voting \
  --image=s3cexampleacr.azurecr.io/azure-voting-app-rust \
  --interval=1m \
  --secret-ref regcred \
  --export > ./clusters/my-cluster/azure-voting-image.yaml

Create an Image Policy

Next, you'll create an image policy. An image policy is the rules Flux follows to determine what images are newer. Often these rules are written to follow semantic versioning rules.

For example, if my application's container images are begins with the tag v0.1.0-alpha a valid select pattern would be >=0.1.0-0. This would allow Flux to detect that any semver higher than v0.1.0-alpha should be used to update the deployment.

Run the following command to create the image polciy:

flux create image policy azure-voting \
  --image-ref=azure-voting \
  --select-semver='>=0.1.0-0' \
  --export > ./clusters/my-cluster/azure-voting-image-policy.yaml

Create the image update

At this stage, Flux has knowledge of the container registry and the specific container image you intend to track. It also possesses a policy for identifying newer images; however, it lacks information on which deployment files to alter when a newer version is identified. This is where image update resources come into play, as they serve the purpose of specifying which deployment files need modification when updates are detected.

The image update is important for several reasons, but the main reason is, it's the resource that's going to be writing to your repository.

What do I mean it will be writing to my repository? Well, when a newer version of the container image is detected the image update resource updates the image reference in the Kustomization.yaml will the latest image and image tag and that commit or change to the repository is what triggers the actual redeployment of the Kubernetes manifests by the other Flux resources that handle the application deployment.

Having Flux commit directly to your existing branches might make you nervous, and for good reason. If that's the case, you can choose to have these changes staged in a new branch and merge them in before a deployment is triggered. To learn how to do that check out the Flux docs here.

But, for this tutorial, we'll keep it simple and have it commit directly to our existing branch.

Run the following command to create the image update resource:

flux create image update azure-voting \
  --interval=1m \
  --git-repo-ref=azure-voting \
  --git-repo-path="./manifests" \
  --checkout-branch=copaGitOps \
  --author-name=fluxcdbot \
  --author-email=fluxcdbot@users.noreply.github.com \
  --commit-template="{{range .Updated.Images}}{{println .}}{{end}}" \
  --export > ./clusters/my-cluster/azure-voting-image-update.yaml

Mark the Kustomization manifest

The last step in setting up image automation is to mark the Kustomization.yaml manifest with comments that indicate what container image should be modified by image update when new images version become available.

To make these changes, open the kustomization.yaml under /manifests and add the following comments:

images:
  - name: azure-voting-app-rust
    newName: s3cexampleacr.azurecr.io/azure-voting-app-rust # {"$imagepolicy": "flux-system:azure-voting:name"}
    newTag: latest # {"$imagepolicy": "flux-system:azure-voting:tag"}

It's crazy how often automation comes back to string/replace, isn't it? Anyway, that's it! You're finished.

Now when you push a new version of the container image to your container registry Flux will pick it up, modify the kustomization.yaml file, and that change will in turn trigger a new deployment to the Kubernetes cluster! Pretty awesome right?

Well, it gets even better. In the next post, you'll learn how to use a new CNCF sandbox project called Copacetic to automatically patch your container images and use FluxCD to deploy the patched version. Keep your Kubernetes environment nice and secure!

Follow me @joshduffney to catch my next post.

Also, a quick shout out to my teammate Paul Yu who wrote a series of awesome articles on FluxCD. I referenced them a lot while writing this post.

Automating Kubernetes Deployments with Flux

Josh Duffney — Wed, 27 Sep 2023 14:06:36 +0000

Interested in making Kubernetes deployments more secure AND automated? If so, keep reading. :)

FluxCD, commonly just called Flux, is an open-source and cloud-native Continuous Delivery (CD) and GitOps tool designed to automate deployments to Kubernetes.

And in this post, you'll learn how to install and configure Flux to automate the deployment of patched and signed containers images.

Prerequisites

In the graphic above, you'll notice that the secure pipeline has two phases; build and deploy. This post focuses on automating the deploy phase, but that requires you already have the Azure infrastructure setup and a GitHub Action Workflow configured to handle the build or CI portion of the pipeline. So, you'll need to do some prerequisite work to follow along.

Once you've completed the setup, you should have a working Azure environment with two signed container images for the Azure Voting app hosted in ACR, which is the necessary starting point for this post.

Note: The setup for this is heavy, even though it's automated. So, if you'd like just continue reading and learn how Flux can be used to automate Kubernetes deployments. :)

Setup the Local Dev Env

Flux is a GitOps tool and every GitOps tool needs a Git repository. To get the repo setup follow these steps:

Fork secure-supply-chain-on-aks repo.
Generate a token for Flux to access the repo. Here's a great clip to walk you through those steps.
Clone the repo to your local machine with, git clone.
Switch to the GitOps branch with git checkout GitOps

Bootstrap FluxCD on the AKS Cluster

In order to get FluxCD working on the cluster, you're going to have to first deploy FluxCD operator to the cluster. Luckily, the FluxCLI makes this super easy.

Here's what you need to do:

Install the FluxCLI

Export the environment variables GITHUB_TOKEN and GITHUB_USER

 export GITHUB_TOKEN=<your-token>
 export GITHUB_USER=<your-username>

Run the flux boostrap command

flux bootstrap github \
--components-extra=image-reflector-controller, image-automation-controller \
--owner=$GITHUB_USER \
--repository=secure-supply-chain-on-aks \
--branch=copaGitOps \
--path=clusters/my-cluster \
--read-write-key \
--personal

Once this command completes, issue a git pull command and you'll notice a new directory call cluster. What the flux bootstrap command did was commit all the necessary manifests to deploy Flux to your cluster.

Create a Flux Source

Now that Flux has been deployed the next thing you need to do is create a source.

Flux supports several source options, but in this post, you'll use git as the source.

Run the following command to create the flux source manifest file:

flux create secret git azure-voting \
  --url=https://github.com/duffney/secure-supply-chain-on-aks \
  --username=$GITHUB_USER \
  --password=$GITHUB_TOKEN

flux create source git azure-voting \
  --url=https://github.com/duffney/secure-supply-chain-on-aks/ \
  --branch=GitOps \
  --interval=1m \
  --secret-ref azure-voting \
  --export > ./clusters/my-cluster/azure-voting-source.yaml

In the above, command the FluxCLI was used to generate a Kubernetes manifest file that configures the desired Git repository as a source within the Flux system. What this essentially does is makes Flux aware of the Git repository that you want to monitor for changes.

Run the following command to view all Flux Sources that use Git:

flux get sources git

Didn't see your repo listed⁉️ Well, that's the Flux system is looking at your GitHub repository for its changes! Flux itself uses GitOps to manage its deployments. 😅

Run the following commands to push the new Git source to the Flux system:

git add -A && git commit -m "Add azure-votiong-app GitRepository"
git push

Rerun flux get sources git again and you'll see the newly created source. Next, you'll configure which files within that repository Flux will care about.

Create a Flux Kustomization

To use Flux with your Kubernetes deployments they need to be packaged somehow and one of the ways to do that is to use the popular resource configuration management tool, Kustomize.

Kustomize gives you the ability to create a single file, called a Kustmization.yaml that bundles your deployment. It's that bundle, so to speak, that Flux uses to determine which files to monitor for changes in your GitRepo.

For this post, kustomize has already been used to create a Kustomization.yaml file that defines the Kubernetes manifests that are considered part of the deployment.

If you're following along, but using your own appliaction you can create a Kustomization.yaml file by running kustomize create --autodetect within the directory that contains the manifests.

But just having a Kustomization.yaml file within your repository isn't enough, Flux needs to be made aware of it and how you'd like it to run the deployment.

So, to take care of that create a Flux Kustomization resources with the following command:

flux create kustomization azure-voting \
  --source=azure-voting \
  --path="./manifests" \
  --prune=true \
  --wait=true \
  --interval=1m \
  --retry-interval=2m \
  --health-check-timeout=3m \
  --export > ./clusters/my-cluster/azure-voting-kustomization.yaml

Oh, don't forget to push these changes to your repo too. :)

git add -A && git commit -m "Add azure-voting-app Kustomization"
git push

Verify Azure-Voting-App deployed

Given the intervals set in the Kustomization and Source it shouldn't take long for Flux to deploy the Azure-Voting-App to the AKS cluster, but if you're eager to see what's going on you can run the following commands:

flux events

flux logs

Test the Azure-Voting-App

Get the public IP address of the webapp with kubectl get ingress, and put it into a browsers and you should see the image below.

Next Steps

This post picked up after the build phase of the secure supply chain that is responsible for scanning, patching, and signing container images. If you'd like to learn more about that check out the following resources:

To dive deeper into GitOps, I highly recommend the follow articles written by my teammates:

Follow me @joshduffney to catch my next post where I'll walk through using Copacetic and FluxCD's Automate image updates to deploy patched container images.

Level-up Container Security: 4 Open-Source Tools for Secure Software Supply Chain

Josh Duffney — Tue, 05 Sep 2023 20:21:24 +0000

In a thought-provoking presentation by Kelsey Hightower, he compares the act of plugging in a random USB key discovered at a coffee shop to the common practice of pulling code from GitHub.

What's funny is that while people might give a suspicious look to someone plugging in a random USB key, they often don't think twice about grabbing a container image from a public registry and tossing it into production. But they really should. Lately, there have been a bunch of security breaches, and guess what? Many of them worked because they went after the weak spots in the software supply chain that no one bothered to check.

Recent supply chain attacks like SolarWinds, Log4j, and Kaseya have underscored the vulnerability of software supply chains. In response, US Federal agencies, under the influence of Executive Order 14028 and similar directives, are taking proactive steps to enhance the security of software supply chains. This has led to the emergence of an entirely new category of tools aimed at securing the software supply chain.

In this article, you'll be introduced to four open-source tools that, when integrated into your CI/CD pipeline, can significantly enhance the security of your container deployments, helping you avoid potential disasters arising from deploying compromised container images.

Overview

Before diving into the specifics of these open-source tools, let's talk about the high-level picture of how they can seamlessly integrate into your pipelines.

During the build stage, Trivy, Copacetic, and Notation play pivotal roles in automating vulnerability scanning, patching, and image signing. Once the build phase concludes, you'll have a trustworthy container image hosted in a registry, ready for deployment.

On the deployment end, two more tools, Gatekeeper and Ratify, are introduced to enforce policies that allow only signed container images to run on Kubernetes clusters.

Trivy: Vulnerability Scanner for Container Images

Trivy is CLI scanner tool and is a lifesaver for developers and DevOps teams navigating the intricate world of container security.

With its user-friendly command-line interface, Trivy makes it a breeze to detect vulnerabilities lurking within container images.

It has several different flags and options you can use to filter the results of the scanner to ensure the reports it generates remain actionable.



IMAGE=azure-voting-app-rust:v0.1-alpha
trivy image $IMAGE
trivy image --severity CRITICAL $IMAGE
trivy image --vuln-type os --severity CRITICAL $IMAGE

You can also export the vulnerability report generated by Trivy to a file, which is especially useful for the next tool in the lineup.



trivy image --exit-code 0 --format json --output ./patch.json --scanners vuln --vuln-type os --ignore-unfixed $IMAGE

Copacetic: Patching Container Images

Copacetic, another open-source gem, works in tandem with Trivy to tackle vulnerabilities in container images.

It utilizes Trivy's vulnerability reports to identify weak spots and then introduces patched layers to rectify these vulnerabilities. It's worth noting that Copacetic currently only works with remote container registries.

Here's an example of how it works:



# Push unpatched image to a remote registry
IMAGE=azure-voting-app-rust:v0.1-alpha
ACR_IMAGE=$ACR_NAME.azurecr.io/azure-voting-app-rust:v0.1-alpha
docker tag $IMAGE $ACR_IMAGE
docker push $ACR_IMAGE



#Start buildkit & run copa patch command
sudo ./bin/buildkitd &> /dev/null & 
sudo copa patch -i ${ACR_IMAGE} -r ./patch.json -t v0.1-alpha-1



#Rescan the newly patched image to verify a reduction in vulnerabilities
trivy image --severity CRITICAL --scanners vuln ${ACR_IMAGE_PATCHED}



#Push the patched image to the registry
ACR_IMAGE_PATCHED=${ACR_NAME}.azurecr.io/azure-voting-app-rust:v0.1-alpha-1

docker push ${ACR_IMAGE_PATCHED}

Notation: Image Signing for Enhanced Security

Notation is another command-line too that lets you digitally sign artifacts. And those signatures essentially become the stamps of approval for the different things in your software supply chain. For example, container images.

Using Notation is simple and straight forward. Once an image has gone through the scanning and patching steps, you sign the image with the notation sign command.

Once signed the container registry has an image index artifact with the signature. That signature will also be listed in the referrers of the container image.



APP_DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' ${ACR_IMAGE_PATCHED})
notation sign $APP_DIGEST

NOTE: It's a best practice to use the container image digest for signing because tags are mutable, meaning they can be overwritten. Digests on the other hand are unique and change each time a layer is added to the container. By using digests for signing, an additional layer of security is added, ensuring the integrity and authenticity of container images.

Ratify: Ensuring Policy Compliance

Ratify is an admission controller. It's available both as a binary and as a Kubernetes tool installed via a Helm Chart. It ensures that only signed images are deployed. It's an invaluable tool for safeguarding your AKS cluster by preventing unsigned container images from being deployed.

To install it on an existing Kubernetes cluster, run the following commands:



curl -L https://raw.githubusercontent.com/deislabs/ratify/main/helmfile.yaml | helmfile sync -f -

Once the installation is complete, any pods that use an unsigned image will be prevented from deploying.

Conclusion

As software supply chain security takes center stage, the importance of integrating open-source tools into your CI/CD pipeline cannot be overstated. By adopting these tools, you'll shield your deployments against potential threats, contributing to a more secure software supply chain.

Learn how to add these tools to your existing pipelines, with this step-by-step workshop!

What is Base62 Conversion?

Josh Duffney — Thu, 27 Jul 2023 20:58:06 +0000

In today's digital age, we are constantly dealing with large amounts of data, and finding efficient ways to store and process it has become crucial.

One technique that stands out is base62 conversion - a powerful encoding scheme that allows for compact data storage and is widely used in short URL services.

In this article, we'll explore the concept of base62 conversion, understand how it works, and see how it can be implemented using Golang.

What is Base62?

At its core, base62 is an encoding system that derives its name from the characters it uses for encoding. The base62 system employs a set of sixty-two characters, consisting of digits 0-9, lowercase letters a-z, and uppercase letters A-Z. If we break down the character count, we have 10 digits + 26 lowercase letters + 26 uppercase letters, resulting in a total of 62 characters.

The Conversion Process

To convert a decimal number to base62, we follow a simple process using long division:

Divide the decimal number by 62.
Record the quotient and keep track of the remainder.
Map the remainder to the corresponding base62 character using its position within the base62 digits.

Let's walk through an example to make things clearer. Suppose we want to convert the decimal number 123 to base62:

Start by dividing 123 by 62:
- 123 ÷ 62 = 1 quotient 61 remainder
Next, divide 1 by 62:
- 1 ÷ 62 = 0 quotient 1 remainder
Now, we map the remainders to their respective base62 characters. Since the base62 system starts its index at zero, the last character "Z" is at position sixty-one, and "1" is at position one.

Therefore, the decimal number 123 converts to "1Z" in base62.

Reverse Conversion: Base62 to Decimal

So far, we've looked at converting a decimal number to base62. But what if we want to go the other way around? How do we convert a base62 number back to its decimal equivalent? Let's find out:

Identify the position value of each base62 character.
Calculate the powers of 62 for each character based on its position.
Multiply each character by its corresponding power of 62.
Sum up the results to obtain the decimal representation.

Let's demonstrate this process using the base62 number "1Z" we obtained earlier:

Determine the position value of each character:
- "1" is in the 1st position.
- "Z" is in the 61st position.
Calculate the powers of 62:
- The power of "Z" is determined by using the formula len(base62) - i - 1 (where i represents the position of the digit):
- For "Z" in position 0, the calculation is 2 - 0 - 1 = 1.
- For "1" in position 1, the calculation is 2 - 1 - 1 = 0.
Convert back to decimals:
- Multiply each digit by its power of 62:
  - 1 * 62^1 = 62
  - Z * 62^0 = 61
Sum up the results: 62 + 61 = 123

As we can see, we successfully obtained the original decimal number 123 from the base62 representation "1Z".

Implementing Base62 Conversion in Golang

Now that we understand how base62 conversion works, let's explore how we can implement it using the Go programming language.

package main

import (
    "fmt"
)

const base62Digits = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"

func convertToBase62(number int) string {
    base62 := ""
    for number > 0 {
        remainder := number % 62
        base62 = string(base62Digits[remainder]) + base62
        number /= 62
    }
    return base62
}

func main() {
    decimalNumber := 123
    base62Number := convertToBase62(decimalNumber)
    fmt.Println(base62Number)
}

In the above code snippet, we have a convertToBase62 function that takes a decimal number as input and returns its base62 equivalent. The function uses successive division to calculate the remainders and builds the base62 representation by mapping each remainder to its respective character from the base62Digits string. Finally, we divide the number by 62 to continue the division process until the quotient becomes zero. The main function demonstrates how to use the convertToBase62 function by converting the decimal number 123 to base62 and printing the result.

Feel free to run the code on the Go Playground to see it in action: Go Playground - Base62 Conversion

Base62 conversion provides a straightforward and efficient way to represent numbers using a limited character set. It's particularly useful for applications that require compact data storage, such as short URL services. By leveraging the power of base62, developers can create more concise and user-friendly representations of numbers without sacrificing accuracy.

So, the next time you come across a shortened URL or need to optimize your data storage, think about the magic of base62 conversion and how it enables efficient handling of large numbers with just sixty-two characters! 🚀

Try it now challenge:

Write a second function the reverses the base62 digits into their original decimal.

Understanding Command-line (CLI) syntax for Gophers

Josh Duffney — Sat, 01 Jul 2023 20:08:21 +0000

The 2022 Go Developer Survey reveals that building CLI applications ranks as the second most popular application of Go, with an impressive majority of over 60% of respondents expressing their preference for writing in Go.

What this data means is, if you want to become a productive Go developer, learning how to build excellent and user friendly CLI tools is necessary. But, before you begin diving into the depths of os.Args, the flags package, and the Cobra framework, it essential that you have a mental model for how command-line applications word and the terminology used to describe them.

In this article, you'll dig into the world of CLI syntax, unravel its complexities, and empower you to navigate the command line with confidence.

Natural language comparison

Natural language comparison is a term used to describe a user's ability to interact with a CLI application by expressing their intentions in familiar, human-like language, which allows for a more intuitive and user-friendly experience.

The Cobra Framework, the most popular CLI framework written in Go, suggests the following pattern in their concept's documentation.

In this example: APPNAME is the executable, NOUN is the argument, VERB is a command, and ADJECTIVE is an option, also called a flag.

If you used CLIs for any length of time you'll have noticed that not every CLI follows this exact pattern, another common implementation of the natural language comparison is verb noun, which PowerShell, a popular scripting language uses. With that said, noun verb is a much more common pattern among CLIs.

Now that you've begun to build your mental model for building CLIs with natural language. Let's dive deeper into what exactly arguments are.

Understanding Arguments

Arguments are the nouns or things that commands act upon.

They are the values that provide data or instruction to a command when it's invoked. Influencing the commands behavior or determining the task to that's performed.

For example, in the command touch file.txt, "touch" is the command, and "file.txt" is the argument or noun, indicating the file to be acted upon.

Note: "argument" and "parameter" are sometimes used interchangeably, but they have distinct meanings.

Arguments are values passed during command execution, while parameters are variables defined within functions or methods.

Exploring Commands

Commands are - verbs - the main actions in a CLI application.

You use them by typing the executable's name followed by arguments or options\flags. Which allows users to perform specific operations.

For example, the following commands each represent different actions that can be executed within the respective CLI applications.

ls
[Command==Verb]

mkdir
[Command==Verb]

git commit
[Command] [Verb]

For simple command-line tools like ls and mkdir the names of the commands themselves are the verbs.

But that's not the case for more complex CLIs like git and npm. With more complex CLIs the verb shifts to the subcommands.

Subcommands create a hierarchical structure within a CLI, grouping related actions together.

They allow users to navigate and access different sets of functionality based on the context or domain they're working with.

For example, within git, there are subcommands like "git commit," "git pull," or "git branch."

> git commit
> git pull
> git branch

Each subcommand focuses on a specific aspect of version control within the overall "git" command.

Understanding Flags

Flags, also known as options, modify the behavior of a command or subcommand.

They act as adjectives, enabling users to customize the command's operation by specifying specific settings, enabling, or disabling features, or providing additional information.

Flags are typically represented by prefixed characters (short-flag) or words (long-flag) that indicate their purpose.

go -v #short-flag
go --version #long-flag

In the above command, the -v and --version flag instruct the go command to return version information.

Note: Short flags are reserved for the most commonly used flags only. Not every option or flag gets a short flag assigned to it.

Conclusion

In this post, you explored the components of natural language comparison: arguments, commands, and flags.

Arguments represent the nouns or entities acted upon by commands, while commands themselves are the verbs or actions performed within the CLI. Flags, acting as adjectives, modify the behavior of commands or subcommands, allowing users to customize their operations.

Understanding these components is crucial for building effective and user-friendly CLI applications. By leveraging natural language comparison, CLI developers can create intuitive and accessible interfaces that streamline command execution and enhance the overall user experience.

Setup a GitHub Action for signing container images with Notary

Josh Duffney — Wed, 02 Nov 2022 15:22:46 +0000

In this article, you'll configure a GitHub Action to digitally sign container images hosted on Azure Container Registry with Notary.

"GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline". You'll use it to automate the process of building, signing, and pushing a Docker image to Azure Container Registry.

Create GitHub Action Secrets

A GitHub Action Secret allows you to securely pass a value to the GitHub Action runner, which prevents sensitive data from being displayed in the build logs. You'll

Create several GitHub Action Secrets to securely pass your Azure and Notation credentials to your workflow.

Click the Settings on the repository
Select Secrets, then Actions,
Then click New repository secret
Enter the secret name and value
Click Add secret

Repeat steps 3-5 for each secrets listed below.

Name	Value
AZURE_CREDENTIALS	JSON object of the Azure Service Principal output from the `az ad sp create-for-rbac` command
NOTATION_USERNAME	Name of the Azure Container Registry token
NOTATION_PASSWORD	Password of the Azure Container Registry token

TIP
In case you didn't save the JSON output, rerunning the az ad sp create-for-rbac command will reset the password of the service principal and generate a new JSON object.

Update the GitHub Workflow

With the Azure infrastructure deployed and your GitHub Actions secrets configured, the last thing you have to do is update the GitHub workflow file.

Open the GitHub workflow file located at .github/workflows/docker-image.yml.
Replace all values from the table below with the appropriate information.
Issue the git add, git commit, and git push commands to push your changes to GitHub.

Placeholder	Description	AzCli command
`<registry-name>`	Name of the Azure Container Registry	az acr list --query '[].name' -o tsv
`<key-name>`	Name of the signing certificate	az keyvault certificate list --vault-name $vaultName --query '[].name' -o tsv
`<certificate-key-id>`	Key Id of the Azure Key Vault certificate	az keyvault certificate show --name example --vault-name $vaultName --query kid -o tsv

Replace $vaultName with the name of your Azure Key Vault instance.

Confirm the container image was signed

Congratulations! 🎉 You've made it to the end of the tutorial.

Your final tasks are to confirm the workflow executed properly and that there is a digital signature attached to the container image hosted on Azure Container Registry.

View the GitHub workflow run

To confirm your workflow executed properly, open the repository on GitHub and click the Actions tab. You should see a workflow run that is green.
Click to expand the steps within the workflow and examine the actions taken to sign the container image.

Confirm the digital signature exists

Open the Azure portal by going to portal.azure.com
Navigate to your Azure Container Registry instance
Under Services, select Repositories
Select the web-app-sample repository
Select the most recent tag
Click the Artifact tab
Confirm cncf.notary.v2.signature exists on the artifact

Deploy Azure Key Vault and Azure Container Registry for Document Signing with Notary

Josh Duffney — Wed, 02 Nov 2022 15:22:29 +0000

In this article, you'll deploy an Azure Key Vault and Azure Container Registry instance with Terraform.

Terraform is an infrastructure as code tool that lets you define your infrastructure resources with readable configuration files. You'll use it to deploy the necessary Azure infrastructure that your GitHub workflow depends on for signing container images.

By the end of this article, you'll have deployed all the Azure resources needed to digitally sign container images with Notary.

Create a service principal

Your GitHub workflow and Terraform both need an service principal for authenticating with Azure.

Create a new service principal by running the following az command:

az ad sp create-for-rbac --name notary-gh-sp --role contributor \
--scopes /subscriptions/<subscriptionId> --sdk-auth

Replace subscriptionId with your Azure subscriptions Id.

TIP Store the JSON object in a secure place. You'll use it to create a credential to authenticate to Azure with the Azure Login GitHub Action.

Export Terraform environment variables

One of several ways to pass credentials to Terraform is through environment variables, without these variables Terraform will failed to authenticate to Azure.

Use the following export commands to set the necessary environment variables for the Azure Terraform provider.

export ARM_CLIENT_ID="00000000-0000-0000-0000-000000000000"
export ARM_CLIENT_SECRET="00000000-0000-0000-0000-000000000000"
export ARM_SUBSCRIPTION_ID="00000000-0000-0000-0000-000000000000"
export ARM_TENANT_ID="00000000-0000-0000-0000-000000000000"

Replace the 00000000 with the values provided in the JSON from the az ad sp create-for-rbac command.

Apply the Terraform configuration

With the service principal created and the environment variable set, you're now ready to apply the Terraform configuration.

Change directories to the terraform folder.
```
cd terraform
```
Initialize Terraform
```
terraform init
```
Apply the Terraform configuration
```
terraform apply
```
When prompted type yes into the terminal and hit enter.

Create an Azure Container Registry Token

Your last task in this tutorial is to create a token that Notation, the command-line tool for Notary, will use to authenticate to the registry when signing images.

Run the following command to create an ACR token:

az acr token create \
--name exampleToken \
--registry <registryName> \
--scope-map _repositories_admin \
--query 'credentials.passwords[0].value' \
--only-show-errors \
--output tsv

TIP Store the password value in a secure place. You'll need to store it as a GitHub secret later in the demo.

Signing Container with Notary and GitHub Actions on Azure

Josh Duffney — Wed, 02 Nov 2022 15:22:16 +0000

In this series, you'll learn how to digitally sign a container image hosted in Azure Container Registry using Notary with a GitHub workflow.

Notary is a CNCF project that provides a set of tools that help you sign, store, and verify OCI artifacts using OCI-conformant registries. Digitally signing artifacts is one of many steps you can take to secure your software supply chains and improve the security of your software.

By the end of this series, you'll have a GitHub workflow that builds a simple web app container image, pushes that image to ACR, and signs the container image with Notation.

Prerequisites

Create a new repository with the template

A sample repository is provided to give you all the scaffolding needed to setup a GitHub workflow for signing container images using Notary.

Complete the following steps to create a new repository using the template.

Go to the acr-notary-sign-images-sample.
Click Use this template.
Select an Owner and enter a Repository name, then click Create repository from template.
Wait for the template to create, then the click Code button Under the Clone section, copy the URL.
Next open a terminal window and use the git clone command to pull down the new repository.
```
git clone <yourRepoURLHere>
```

Next steps

Continue to the next article to deploy the required Azure resources.