DEV Community: Adrian Mouat

On the Supply Chain Security Trail with Docker Scout

Adrian Mouat — Tue, 18 Jun 2024 10:23:19 +0000

I believe Docker Scout is one of the best supply-chain security and scanning tools available for the container ecosystem. Before we get into details, I want to make clear that I'm a Docker Captain, so I'm not entirely unbiased here! And Scout itself has a free tier, but if you want to monitor multiple repositories, then you'll need to purchase a subscription. Now that that's clear, please read on to understand why I like Scout and what separates it from other solutions.

The first thing to be aware of is that Scout isn't just a vulnerability scanner, and Docker's marketing materials tend to avoid that term. Scout takes a holistic approach to supply chain security. It starts by analysing containers and extracting or creating a complete Software Bill of Materials (SBOM) for images. With the Hub integration, these can be monitored over time for compliance to policies. Policies include – predictably – flagging images with high or critical vulnerabilities, but there are also policies such as "avoiding copy-left licences" and verifying "supply chain attestations".

Other features that go beyond simple vulnerability scanning include the ability to provide recommendations for mitigations and improvements and an already impressive list of integrations (not just with the Docker Hub, but also other registries, code editors, CI/CD platforms and build tooling).

There are two main parts to Scout: a web application that you can see in the below image, and a CLI tool that I will use throughout the rest of this article.

To dig a bit deeper, let's try analysing a few images. We said already that the starting point for everything is the SBOM, so let's try generating one. Interestingly, Scout has its own sbom command, separate from docker sbom which is powered by syft. I assume the Scout version will eventually replace docker sbom. Here's the SBOM for the official Redis image:

docker scout sbom --format list redis
{"level":"info","msg":"SBOM of image already cached, 133 packages indexed\n","time":"2024-06-12T14:50:48+01:00"}

           Name                  Version          Type
────────────────────────────────────────────────────────────
  acl                     2.3.1-3                deb
  …
  gosu                    (devel)                golang
  …
  redis                   7.2.5                  generic
  stdlib                  go1.18.2               golang
  stdlib                  1.18.2                 golang
  sys                     0.13.0                 golang
  sys                     0.1.0                  golang
  …

The majority of the output is deb packages, most of which I've clipped to reduce the size of the output. Scout has found these packages by reading the package manager database. This is as far as some other tooling goes, but we can see it's also picked up a few more things. Scout has found the binaries for two more applications – redis and gosu – that were downloaded directly from a website rather than being installed via the package manager. It's also analysed the gosu binary and found the libraries it links against. The effectiveness of supply chain tooling is strongly tied to its performance in identifying contents – if Scout had failed to find these binaries, it would not be able to scan them for vulnerabilities or ensure they match any policies.

In the equivalent Chainguard Image we can see everything is installed via the apk package manager:

$ docker scout sbom --format list cgr.dev/chainguard/redis
{"level":"info","msg":"SBOM of image already cached, 18 packages indexed\n","time":"2024-06-12T15:05:03+01:00"}

           Name               Version       Type
────────────────────────────────────────────────────
  bash                    5.2.21-r4         apk
  busybox                 1.36.1-r10        apk
  ca-certificates         20240315-r3       apk
  ca-certificates-bundle  20240315-r3       apk
  glibc                   2.39-r6           apk
  glibc-locale-posix      2.39-r6           apk
  ld-linux                2.39-r6           apk
  libcrypt1               2.39-r6           apk
  libcrypto3              3.3.0-r9          apk
  libssl3                 3.3.0-r9          apk
  libxcrypt               4.4.36-r7         apk
  ncurses                 6.4_p20231125-r3  apk
  ncurses-terminfo-base   6.4_p20231125-r3  apk
  openssl                 3.3.0-r9          apk
  posix-libc-utils        2.39-r6           apk
  redis-7.2               7.2.5-r2          apk
  redis-cli-7.2           7.2.5-r2          apk
  wolfi-baselayout        20230201-r11      apk

This is the full output – clearly demonstrating how much leaner Chainguard Images are.

An interesting question is what happens to containers you build yourself with a multi-stage process – will it pick everything up? As a test, I used an example Go application with a multi-stage build:

$ docker scout sbom test --format=list
{"level":"info","msg":"SBOM of image already cached, 7 packages indexed\n","time":"2024-06-12T15:46:18+01:00"}

           Name             Version      Type
──────────────────────────────────────────────────
  ca-certificates         20240315-r3   apk
  ca-certificates-bundle  20240315-r3   apk
  learninglabsstatic      (devel)       golang
  stdlib                  1.22.4        golang
  stdlib                  go1.22.4      golang
  tzdata                  2024a-r2      apk
  wolfi-baselayout        20230201-r11  apk

This looks pretty good! It has found the binary, the libraries it links against and the handful of packages that are in the Chainguard static image. If you regularly use another scanner, I'd recommend running a test like this to make sure it's aware of everything in your image. I'd also recommend running against a few different builds using different ecosystems to check that it properly identifies your Rust binaries or Java jars.

One of the best features in Scout is vulnerability scanning output. I often find it a struggle to get accurate high-level info from scanners, but it's easy with Scout. The simple tweak of displaying summary output at the end, instead of before a huge list of issues that you have to scroll backwards through, is much appreciated. Here's the output from scanning the current nginx:alpine image:

$ docker scout cves nginx:alpine
    ✓ SBOM of image already cached, 82 packages indexed
    ✗ Detected 2 vulnerable packages with a total of 5 vulnerabilities


## Overview

                    │       Analyzed Image
────────────────────┼──────────────────────────────
  Target            │  nginx:alpine
    digest          │  4f49228258b6
    platform        │ linux/arm64
    vulnerabilities │    0C     0H     5M     0L
    size            │ 22 MB
    packages        │ 82


## Packages and Vulnerabilities

   0C     0H     4M     0L  busybox 1.36.1-r15
pkg:apk/alpine/busybox@1.36.1-r15?os_name=alpine&os_version=3.19

    ✗ MEDIUM CVE-2023-42366
      https://scout.docker.com/v/CVE-2023-42366
      Affected range : <1.36.1-r16
      Fixed version  : 1.36.1-r16

    ✗ MEDIUM CVE-2023-42365
      https://scout.docker.com/v/CVE-2023-42365
      Affected range : <1.36.1-r19
      Fixed version  : 1.36.1-r19

    ✗ MEDIUM CVE-2023-42364
      https://scout.docker.com/v/CVE-2023-42364
      Affected range : <1.36.1-r19
      Fixed version  : 1.36.1-r19

    ✗ MEDIUM CVE-2023-42363
      https://scout.docker.com/v/CVE-2023-42363
      Affected range : <1.36.1-r17
      Fixed version  : 1.36.1-r17


   0C     0H     1M     0L  curl 8.5.0-r0
pkg:apk/alpine/curl@8.5.0-r0?os_name=alpine&os_version=3.19

    ✗ MEDIUM CVE-2024-0853
      https://scout.docker.com/v/CVE-2024-0853
      Affected range : <8.6.0-r0
      Fixed version  : not fixed

5 vulnerabilities found in 2 packages
  LOW       0
  MEDIUM    5
  HIGH      0
  CRITICAL  0


What's next:
    View base image update recommendations → docker scout recommendations nginx:alpine

It's also possible to only get the initial summary output by calling the quickview command.

In my testing, I've noticed that – compared to some other scanner tooling – Scout will report less CVEs. Docker claims this is because they are less vulnerable to false positives (where a CVE is wrongly attributed to an image). The primary reason they give for this is that Scout uses Package URLs or PURLs to match software, rather than CPEs.

On the other hand, Scout will report more matches than some tooling (I realise this is confusing, but I don't want to name names without providing a full breakdown and analysis). This is because Scout uses a wide range of sources to build its vulnerability database – as well as the security data from Linux distributions, it includes NVD, GitHub and other advisory data. The more data sources that are used, the more likely you are to find matches.

The last sentence in the command output hints at another feature in Scout – the ability to recommend fixes. One of the biggest flaws with current scanning tools is that they often flag issues that you can do nothing about – vulnerabilities for which there is no newer package or fix available. Scout helps you both filter out issues that you can't deal with and fix the ones you can.

Here's the recommendations output for the Debian-based nginx:latest image, which suggests changing to a different tag to reduce CVEs, and gives details of the potential impact:

docker scout recommendations nginx
    ✓ SBOM of image already cached, 231 packages indexed

    i Base image was auto-detected. To get more accurate recommendations, build images with max-mode provenance attestations.
      Review docs.docker.com ↗ for more information.
      Alternatively, use  docker scout recommendations --tag <base image tag>  to pass a specific base image tag.

  Target   │  nginx:latest
    digest │  705b7f60fea5

## Recommended fixes

  Base image is  debian:12-slim

  Name            │  12-slim
  Digest          │  sha256:6dc38501802c1554f0fd858d1153a6f0e18c71006c6d0b31cf19fa778900e658
  Vulnerabilities │    0C     0H     0M    23L
  Pushed          │ 1 month ago
  Size            │ 29 MB
  Packages        │ 125
  Flavor          │ debian
  OS              │ 12
  Slim            │ ✓


  │ The base image is also available under the supported tag(s)  12.5-slim ,  bookworm-slim . If you want to display recommendations specifically
  │ for a different tag, please re-run the command using the  --tag  flag.



Refresh base image
  Rebuild the image using a newer base image version. Updating this may result in breaking changes.


            Tag            │                  Details                   │    Pushed    │       Vulnerabilities
───────────────────────────┼────────────────────────────────────────────┼──────────────┼──────────────────────────────
   12-slim                 │ Benefits:                                  │ 11 hours ago │    0C     0H     0M    23L
  Newer image for same tag │ • Same OS detected                         │              │
  Also known as:           │ • Newer image for same tag                 │              │
  • 12.5-slim              │ • Tag was pushed more recently             │              │
  • bookworm-slim          │ • Image has similar size                   │              │
  • bookworm-20240612-slim │ • Image has same number of vulnerabilities │              │
                           │ • Image contains equal number of packages  │              │
                           │ • Tag is using slim variant                │              │
                           │                                            │              │
                           │ Image details:                             │              │
                           │ • Size: 29 MB                              │              │
                           │ • Flavor: debian                           │              │
                           │ • OS: 12                                   │              │
                           │ • Slim: ✓                                  │              │
                           │                                            │              │
                           │                                            │              │
                           │                                            │              │


Change base image
  The list displays new recommended tags in descending order, where the top results are rated as most suitable.


           Tag           │                    Details                    │    Pushed    │       Vulnerabilities
─────────────────────────┼───────────────────────────────────────────────┼──────────────┼──────────────────────────────
   stable-slim           │ Benefits:                                     │ 11 hours ago │    0C     0H     0M    23L
  Tag is preferred tag   │ • Same OS detected                            │              │
  Also known as:         │ • Tag is preferred tag                        │              │
  • stable-20240612-slim │ • Tag was pushed more recently                │              │
                         │ • Image has similar size                      │              │
                         │ • Image has same number of vulnerabilities    │              │
                         │ • Image contains equal number of packages     │              │
                         │ • Tag is using slim variant                   │              │
                         │ • stable-slim was pulled 46K times last month │              │
                         │                                               │              │
                         │ Image details:                                │              │
                         │ • Size: 29 MB                                 │              │
                         │ • Flavor: debian                              │              │
                         │ • OS: 12                                      │              │
                         │ • Slim: ✓                                     │              │
                         │                                               │              │
                         │                                               │              │
                         │                                               │              │
   12                    │ Benefits:                                     │ 11 hours ago │    0C     0H     0M    23L
  Tag is latest          │ • Same OS detected                            │              │
  Also known as:         │ • Tag was pushed more recently                │              │
  • 12.5                 │ • Tag is latest                               │              │
  • bookworm             │ • Image has same number of vulnerabilities    │              │
  • bookworm-20240612    │ • Image contains equal number of packages     │              │
  • latest               │                                               │              │
                         │ Image details:                                │              │
                         │ • Size: 50 MB                                 │              │
                         │ • Flavor: debian                              │              │
                         │ • OS: 12                                      │              │
                         │                                               │              │
                         │                                               │              │
                         │                                               │              │

Finally, let's take a look at the Chainguard nginx image.

docker scout cves chainguard/nginx
    ✓ SBOM obtained from attestation, packages found
    ✓ No vulnerable package detected


## Overview

                    │       Analyzed Image
────────────────────┼──────────────────────────────
  Target            │  chainguard/nginx:latest
    digest          │  1ffdf2c788d4
    platform        │ linux/arm64
    vulnerabilities │    0C     0H     0M     0L
    size            │ 20 MB
    packages        │ 55


## Packages and Vulnerabilities

  No vulnerable packages detected

As expected, there are no vulnerabilities, but also note the text at the start – "SBOM obtained from attestation, packages found". Scout has used the SBOM that Chainguard provided for the image and used that to enumerate the software in the image. This level of integration and support for external tooling is essential for creating a holistic supply chain security solution – every organisation uses a different set of tools and software from various vendors and it's imperative that we figure out how to get supply chain information from everyone in a verifiable and trustable manner.

Hopefully I've shown you why I like Docker Scout. It does a good job of finding all the packages in an image but also does a great job of reporting vulnerabilities. Vulnerabilities are less likely to be false positives through the use of PURLs and the summary output is easy to grab (which is great for videos!). Docker will tell you the real strengths lie in continuously monitoring repositories and reporting on progress towards supply chain policies, and I plan to create some further content on this in the future.

Please give Docker Scout a whirl (compare some Chainguard Images!) and let me know what you think.

Cover photo by Maël BALLAND on Unsplash

Debugging Distroless Images with kubectl and cdebug

Adrian Mouat — Fri, 31 May 2024 14:37:01 +0000

Ivan Velichko recently made me aware of issues with debugging distroless containers in Kubernetes with kubectl debug. This blog will take a look at the problem and show you can get access to the filesystem of a distroless pod for debugging purposes.

The problem Ivan found was a lack of permissions to access the filesystem of the container being debugged. This is best explained with some examples. With a regular (non-distroless) container, you can do the following to start an ephemeral debug container that shares various namespaces with the target container:

$ kubectl run nginx-pod --image nginx
pod/nginx-pod created
$ kubectl debug -it nginx-pod --image alpine --target nginx-pod
Targeting container "nginx-pod". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
Defaulting debug container name to debugger-h4fzv.
If you don't see a command prompt, try pressing enter.
/ # ps
PID   USER     TIME  COMMAND
    1 root      0:00 nginx: master process nginx -g daemon off;
   32 101       0:00 nginx: worker process
   33 101       0:00 nginx: worker process
   34 101       0:00 nginx: worker process
   35 101       0:00 nginx: worker process
   36 101       0:00 nginx: worker process
   37 101       0:00 nginx: worker process
   38 101       0:00 nginx: worker process
   39 101       0:00 nginx: worker process
   40 101       0:00 nginx: worker process
   41 101       0:00 nginx: worker process
  308 root      0:00 /bin/sh
  317 root      0:00 ps

(You could also just use kubectl exec -it nginx-pod -- /bin/sh, but of course that's not possible in a distroless container)

Note that the filesystem is the filesystem of the Alpine debug container, not the nginx container:

/ # cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.19.1
PRETTY_NAME="Alpine Linux v3.19"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"

But we can get to the nginx container filesystem via the /proc/1/root filesystem. To break this down:

/proc is a virtual filesystem created by the kernel that contains various metadata
1 refers to the process id, in this case our running nginx master process; and
root is a link to the root of the filesystem the process is running in.

So we can access the index.html file inside the nginx container like this:

/ # cat /proc/1/root/usr/share/nginx/html/index.html
<!DOCTYPE html>
<html>
…
/ #

Now let's try that with the cgr.dev/chainguard/nginx image, which is one of Chainguard's distroless images:

$ kubectl run nginx-distroless --image cgr.dev/chainguard/nginx
pod/nginx-distroless created
$ kubectl debug -it nginx-distroless --image alpine --target nginx-distroless
Targeting container "nginx-distroless". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
Defaulting debug container name to debugger-bcr26.
If you don't see a command prompt, try pressing enter.
/ # cat /proc/1/root/usr/share/nginx/html/index.html
cat: can't open '/proc/1/root/usr/share/nginx/html/index.html': Permission denied
/ # whoami
root

We get Permission denied.

It turns out that the problem is that the nginx container is running as the user nonroot with UID 65532, which we don't have permission to access despite being root (using --profile to set a different security profile didn't help either, but I suspect it might in the future). To fix this, we need our debug container to run as the same user as the nginx container. Unfortunately there's no --user flag for kubectl, so we need to have an image that runs as this user by default. We could create one e.g with a Dockerfile such as:

FROM alpine
USER 65532

But in the case of Chainguard Images there's a much easier solution. Most of our images come with -dev variants that run as the same user but also include a shell and can be used for debugging, so we can do:

$ kubectl debug -it nginx-distroless --image cgr.dev/chainguard/nginx:latest-dev --target nginx-distroless -- /bin/sh
Targeting container "nginx-distroless". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
Defaulting debug container name to debugger-nbvjt.
If you don't see a command prompt, try pressing enter.
/ $
/ $ cat /proc/1/root/usr/share/nginx/html/index.html
<!DOCTYPE html>
<html>
…

And everything works as expected.

There's actually a second wrinkle that is also solved by setting the user – if your pod is running with the runAsNonRoot policy, you won't be able to start a debug container that runs as root with the default profile.

This does point to some ways in which kubectl debug could be improved:

Add a --user option to set the user in the debug container
Add a formal way to access the target container filesystem. Going via /proc/1/root seems a little hacky and non-intuitive
Add some more docs to explain all of this (which is somewhere I plan to help).

(I do see there are some proposed enhancements related to profiles that might help here)

I should also point out that Ivan addresses these problems directly with his cdebug tool. You can use cdebug to directly debug a pod:

$ cdebug exec -it --privileged pod/nginx-distroless/nginx-distroless
Debugger container name: cdebug-20ba5985
Starting debugger container...
Waiting for debugger container...
Attaching to debugger container...
If you don't see a command prompt, try pressing enter.
/ # cat /proc/1/root/usr/share/nginx/html/index.html
<!DOCTYPE html>
<html>
…

cdebug also supports a --user flag if you have the runAsNonRoot policy e.g:

cdebug exec -it --user 65532 pod/nginx-distroless/nginx-distroless
…

That's about it. Running production workloads in distroless containers is a big improvement in terms of security. With a little bit of knowledge these containers can still be straightforward to debug.

Learning Labs: How to build on Chainguard's Hardened and Minimal Images

Adrian Mouat — Mon, 29 Jan 2024 16:31:53 +0000

At Chainguard we've just launched the first in a series of hands-on Learning Labs. This series will help engineers get started on the path to low-CVE, minimal images using Chainguard base images. We're planning to run labs for various stacks, starting with images for compiled languages such as Go, Rust and C on Feb 13.

What you will learn:
💜 Why reducing the size and CVE count of images is important
💜 How to migrate a Dockerfile build using a compiled language to use Chainguard Images
💜 Advanced techniques for producing truly minimal images and debugging

📆 Tuesday, Feb 13, 2024 @ 10 am ET (click through for your TZ)
📝 Register: https://www.crowdcast.io/c/learninglabs-feb24