Security starts before the production deployment

As a developer, you build new features, fix bugs, and ship code. That's what the original job description included. Software developers bear additional responsibilities today, such as expertise in deploying or securing software. You could make a convincing case that security is a production-only concern. As long as you're working on your features, security is not essential (yet). But what if security issues arise in production? These kinds of tickets return to engineering and interrupt everything you're doing. While these situations can happen anytime, you can play an active role in preventing them in the first place—the result: Fewer high-priority tickets and interruptions.

In this blog post, let's explore tools you can use to be proactive about a class of security incidents.

What Security Issues are we talking about here?

Over the last few years, companies like GitHub invested heavily in making software more secure. Dependabot automatically scans your dependency lists (i.e., package.json) and submits pull requests if you're using a vulnerable library version. Since it's such a low-effort workflow, developers can quickly patch vulnerable packages as part of their regular workflow.

In this blog post, we want to shed some light on another aspect of software security: Production runtime. If you're building Docker Images as part of your CI/CD workflow, this blog post is for you. No matter what specific build strategy you use for your images, you likely install additional packages to ensure your code runs without issues.

Insecure Docker Images?

Have you ever considered the security of the Docker images you run in production? If we use the current Ubuntu LTS image, it has 106 packages installed:

docker run --rm ubuntu:jammy dpkg -l | wc -l
  106

Not using Ubuntu? Alpine only has 15 preinstalled packages:

docker run --rm alpine:latest apk list -i | wc -l
      15

If a Docker image has 15 or over 100 installed packages, any of these packages could potentially be vulnerable to an attack. If you think, "Ok I'll patch once I see a report in the news" (like Heartbleed or Log4Shell), you're "missing out" on most patch opportunities. Most (severe) security vulnerabilities don't get a front-page listing on Hacker News.

There's no need to go through the CVE database manually. Instead, we can work towards a workflow that resembles GitHub's Dependabot. We can scan Docker Images as we build them to find out if there are any vulnerable packages and patch them. While we will never achieve 100% security,
we're getting a step closer with a low overhead process.

How to scan Docker Images

Introducing KubeClarity. KubeClarity is an open-source project to help you ship more secure software. While KubeClarity covers many different use cases, let's focus on image scanning for now.

Installation

Check out the KubeClarity README to find installation instructions for your specific platform. In this
tutorial, we mainly use the CLI but feel free to install the Dashboard for visualization.

Scan your Image

To illustrate image scanning, we use this repository,
containing an example Rust workload with a vulnerable OpenSSL version. Run the following command:

kubeclarity-cli scan ghcr.io/schultyy/rust-workload:0.0.3 --input-type image -o table
NAME           INSTALLED               FIXED-IN          VULNERABILITY     SEVERITY    SCANNERS
perl-base      5.28.1-6+deb10u1                          CVE-2023-31484    HIGH        grype
libgcc1        1:8.3.0-6                                 CVE-2018-12886    HIGH        grype
libsystemd0    241-7~deb10u9                             CVE-2019-3844     HIGH        grype
openssl        1.1.1n-0+deb10u3        1.1.1n-0+deb10u4  CVE-2023-0215     HIGH        grype
ncurses-base   6.1+20181013-2+deb10u3                    CVE-2023-29491    HIGH        grype
libudev1       241-7~deb10u9                             CVE-2019-3844     HIGH        grype
openssl        1.1.1n-0+deb10u3        1.1.1n-0+deb10u5  CVE-2023-0464     HIGH        grype
libstdc++6     8.3.0-6                                   CVE-2019-15847    HIGH        grype
openssl        1.1.1n-0+deb10u3        1.1.1n-0+deb10u5  CVE-2023-2650     HIGH        grype
openssl        1.1.1n-0+deb10u3        1.1.1n-0+deb10u4  CVE-2023-0286     HIGH        grype
libc-bin       2.28-10+deb10u2                           CVE-2020-1751     HIGH        grype
openssl        1.1.1n-0+deb10u3        1.1.1n-0+deb10u4  CVE-2022-4450     HIGH        grype
libss2         1.44.5-1+deb10u3                          CVE-2022-1304     HIGH        grype
libsystemd0    241-7~deb10u9                             CVE-2021-3997     MEDIUM      grype
libsystemd0    241-7~deb10u9                             CVE-2022-3821     MEDIUM      grype
openssl        1.1.1n-0+deb10u3        1.1.1n-0+deb10u5  CVE-2023-0466     MEDIUM      grype
libudev1       241-7~deb10u9                             CVE-2022-3821     MEDIUM      grype
openssl        1.1.1n-0+deb10u3        1.1.1n-0+deb10u4  CVE-2022-2097     MEDIUM      grype
libudev1       241-7~deb10u9                             CVE-2021-3997     MEDIUM      grype
libsystemd0    241-7~deb10u9                             CVE-2022-4415     MEDIUM      grype
libudev1       241-7~deb10u9                             CVE-2022-4415     MEDIUM      grype
openssl        1.1.1n-0+deb10u3        1.1.1n-0+deb10u4  CVE-2022-4304     MEDIUM      grype
openssl        1.1.1n-0+deb10u3        1.1.1n-0+deb10u5  CVE-2023-0465     MEDIUM      grype
libpcre3       2:8.39-12                                 CVE-2020-14155    MEDIUM      grype
libgcrypt20    1.8.4-5+deb10u1                           CVE-2019-13627    MEDIUM      grype
login          1:4.5-1.1                                 CVE-2023-29383    LOW         grype
bsdutils       1:2.33.1-0.1                              CVE-2021-37600    LOW         grype

(Output truncated for brevity)

The kubeclarity scan command outputs a list of packages for which CVEs have been filed. openssl comes up several times, with a version that fixes the issue (See FIXED-IN column).

Fix Vulnerable Packages

Open the Dockerfile and move to line 19. Line 19 right now installs a specific openssl version:

RUN apt-get update && apt-get install -y openssl=1.1.1n-0+deb10u3 && rm -rf /var/lib/apt/lists/*

Update this line to install the latest version according to the kubeclarity-cli scan result:

RUN apt-get update && apt-get install -y openssl=1.1.1n-0+deb10u5 && rm -rf /var/lib/apt/lists/*

It indicated that issues would subsequently be fixed in
1.1.1n-0+deb10u4 and 1.1.1n-0+deb10u5. Therefore, we'll go with the latest version available.

Let's build a new image:

docker build -t ghcr.io/schultyy/rust-workload:0.0.4 .

To verify the issue has been fixed, we rerun KubeClarity, this time with the LOCAL_IMAGE_SCAN=true variable set. We want to scan our local image before pushing, verifying that OpenSSL does not have any outstanding vulnerabilities:

LOCAL_IMAGE_SCAN=true kubeclarity-cli scan ghcr.io/schultyy/rust-workload:0.0.4 --input-type image -o table | grep openssl
 openssl        1.1.1n-0+deb10u5                  CVE-2010-0928     NEGLIGIBLE  grype
 openssl        1.1.1n-0+deb10u5                  CVE-2007-6755     NEGLIGIBLE  grype..

The scan still lists two openssl entries, though only with severity of NEGLIGIBLE. Therefore, the issue has been fixed.

What's next?

Check out the KubeClarity GitHub repository to learn about additional use cases, and make sure to star the repository!