DEV Community: gasparev

How to test Bash scripts (2020 guide)

gasparev — Thu, 03 Dec 2020 11:07:54 +0000

How do you test your bash scripts? Do you test them at all?

Bash scripting is one of the handiest tools every developer has to automate tasks, ease the everyday activities, help to reduce the toil. But automation tools can actually increase the toil if are not treated the same way as code. Bash scripts can get out of date and stop working, became hard to maintain, and increase your technical debt. You need to check them in version control, write tests, and use CI.

In this article, we'll go through how you can improve the quality of your automation scripts writing tests for Bash.

Bach

Bach is a testing framework for Bash that provides the possibility to write unit tests for your Bash scripts. Bach makes any command in the PATH an external dependency of the tested Bash script so that no command will be actually executed but will run as "dry run". In this way, you will be able to test the logic of the script and not the commands themself. Bach mocks all commands by default and provides a set of APIs for executing real commands if necessary.

How to install Bach

Prerequisites

Bash v4.3+
Coreutils (GNU/Linux)
Diffutils (GNU/Linux)

To install Bach Testing Framework download bach.sh to your project, use the source command to import bach.sh.

For example:

source path/to/bach.sh

What can be done

In Bach, we can test what the Bash script will actually execute.

Every test case in Bach is made of two functions: one for running tests and the other for asserting.

When you run your tests Bach will execute the two functions separately and will compare the sequence of commands executed by both functions. Every testing function must start with the name test-, the asserting function must end with -assert.

How to write test cases

Let's see some practical examples of how to write test cases.

#!/usr/bin/env bash
set -euo pipefail
source bach.sh

To enable the Bach Testing Framework the first thing to do is to source the bash.sh.

test-rm-your-dot-git() {
    @mock find ~ -type d -name .git === @stdout ~/src/your-awesome-project/.git \
                                                ~/src/code/.git
    find ~ -type d -name .git | xargs -- rm -rf
}
test-rm-your-dot-git-assert() {
    rm -rf ~/src/your-awesome-project/.git ~/src/code/.git
}

The first script we test is a command to find all the .git folders and remove them. We can test it by mocking the find command with those parameters to output two directories.

test-mock-script-with-custom-complex-action() {
    @mock ./path/to/script <<\SCRIPT
if [[ "$1" == foo ]]; then
  @echo bar
else
  @echo anything
fi
SCRIPT
    ./path/to/script foo
    ./path/to/script something
}
test-mock-script-with-custom-complex-action-assert() {
    bar
    anything
}

In the second script, we use Bach to test a script that returns a different output based on the input. In this case, we mock the script using @mock ./path/to/script and then we define the script behavior.

test-bach-framework-mock-commands() {
    @mock find . -name fn === @stdout file1 file2

    ls $(find . -name fn)

    @mock ls file1 file2 === @stdout file2 file1

    ls $(find . -name fn) | xargs -n1 -- do-something

    @mock ls === @stdout foo bar foobar
    ls | xargs -n2 -- bash -c 'do-something ${@}' -s
}
test-bach-framework-mock-commands-assert() {
    ls file1 file2

    do-something file2
    do-something file1

    bash -c 'do-something ${@}' -s foo bar
    bash -c 'do-something ${@}' -s foobar
}

In this script, you can see how to perform some complex operations like mocking a command and execute it in a $(...) expression and using pipes.

test-bach-framework-set--e-should-work() {
    set -e

    do-this
    builtin false

    should-not-do-this

}
test-bach-framework-set--e-should-work-assert() {
    do-this
    @fail
}

Here we test the behavior of set -e so we make the script fail with builtin false and we test the failure using @fail.

test-no-double-quote-star() {
    @touch bar1 bar2 bar3 "bar*"

    function cleanup() {
        rm -rf $1
    }

    # We want to remove the file "bar*", not the others
    cleanup "bar*"
}
test-no-double-quote-star-assert() {
    # Without double quotes, all bar files are removed!
    rm -rf "bar*" bar1 bar2 bar3
}

test-double-quote-star() {
    @touch bar1 bar2 bar3 "bar*"

    function cleanup() {
        rm -rf "$1"
    }

    # We want to remove the file "bar*", not the others
    cleanup "bar*"
}
test-double-quote-star-assert() {
    # Yes, with double quotes, only the file "bar*" is removed
    rm -rf "bar*"
}

Here we define a function that takes an argument and removes it. In the first example, we don't use double quotes in the function, this will remove all the files starting with bar-. In the second example, the function is using double quotes so only the file called bar* is removed.

How to run Bach tests

You can write all your test cases in a single .sh file remembering to add the source bach.sh in it. At this point to run the tests you just need to execute the .sh file.

This is it!

In this post, we've seen how to write unit tests for your Bash scripts to improve the quality and reliability of your automation.

Reach me on Twitter @gasparevitta and let me know your thoughts!

This article was originally published on my blog. Head over there if you like this post and want to read others like it!

Speed up Docker build time with cache warming

gasparev — Wed, 25 Nov 2020 13:26:42 +0000

Every time you run a build, Docker will try to use the local cache if present. What if you have never built the image locally but the image is already on a remote docker registry?

In this post, we'll go through how you can use a remote docker registry to warm the docker cache and speed up the local build. This can be also very useful in CI.

How to warm the cache in docker

From Docker 1.13 you can use a small trick. You can use the --cache-from flag to instruct docker to use as a cache the layer of another image. The only prerequisite is that you have the image locally.

Let's say you have your Dockerfile locally that you build and push:

$ docker build -t my-docker .
$ docker push my-docker

Now, on another machine or in your CI, you want to build my-docker again. Here you can use the image you pushed on your docker registry to speed up the build.

$ docker pull my-docker || true
$ docker build --cache-from my-docker .

We use || true to avoid failures if the image has not been pushed and then we use the image as a cache for the build.

How to warm the cache with Buildkit

With the classical approach, you must pull the image before running the build. We can avoid it using Buildkit and BUILDKIT_INLINE_CACHE. You can check on a previous post on how to set up Buildkit on your machine.

What we really need, together with the cache, is the JSON with cache metadata. We can use Buildkit to run the Docker build and add the cache information to the Docker image.

$ export DOCKER_BUILDKIT=1
$ docker build -t my-docker --build-arg BUILDKIT_INLINE_CACHE=1 .
$ docker push my-docker

Here we enable Buildkit with export DOCKER_BUILDKIT=1 and then we use --build-arg BUILDKIT_INLINE_CACHE=1 during the build. At this point, the built image can be used as a cache for the other builds.

After that, we can run the build on another machine or in CI using the same syntax as before:

$ docker build --cache-from my-docker .

What will happen this time is that the builder will pull the JSON metadata from the registry to figure out if there's a cache that can be used from the remote image. If there's a cache hit, only the needed layers will be fetched locally and will be used by the builder.

This is it!

In this post we've seen how to warm up the Docker build cache using another image, this technique will help in speeding up the build time.

If you are looking for other ways to speed up the build time you can try to use a multistage parallel Docker build.

Reach me on Twitter @gasparevitta and let me know your thoughts!
This article was originally published on my blog. Head over there if you like this post and want to read others like it!

Docker best practices: lint your Dockerfile!

gasparev — Mon, 09 Nov 2020 16:06:36 +0000

In software development, best practices are the way to go.
You must do the same while developing the infrastructure code!
In this post, we’ll go through how a linter can increase your productivity, how to use it with a Dockerfile, and how to implement it in a CI pipeline.

What is a linter? Why we need it?

According to Wikipedia, a linter is a static code analysis tool used to flag programming errors, bugs, stylistic errors, and suspicious constructs.
As a static code analysis tool, linters can’t be used to detect compiling time errors but are very useful in finding typos and syntax errors. Using a linter will allow you to detect errors early, fixing them faster, and reduce bugs before execution.

Hadolint

The tool we will use is called Hadolint and as you can recall from the name is a linter. It’s built to help you follow the docker best practices, and it also uses ShellCheck to inspect your RUN instructions.

How to set it up

It very easy to use both in a local environment and CI, you can find the integration docs here.

If you are a VS Code user, there is the Hadolint extension. If you want to use it directly in Github, there is the Hadolint Github action.

Define custom rules

If you don’t want to follow all the rules defined by Hadolint, you can easily deactivate some of them. You only need to create a file called ~/.config/hadolint.yaml, a full list of rules here. An example of a custom rule file is:

ignored:
  - DL3000
  - SC1010

How to run it in CI

To enforce this best practice, you can add a test in your Docker deployment pipeline. We can implement it in the Ansible pipeline we used to execute unit tests for Docker.

Let’s add a new role called “Run hadolint on Dockerfile”:

- name: Run hadolint on Dockerfile
  shell: |
    docker run --rm -i \
      -v "{{ role_path }}/files/hadolint.yaml":/root/.config/hadolint.yaml hadolint/hadolint \
      < {{ dockerfile_name }}

In this example, we directly run the official hadolint docker image against the Dockerfile. I’m mounting the hadolint.yaml file to use my custom rules configuration.

This is it!

Now you should know all you need to use Hadolint for your Dockerfile.

Reach me on Twitter @gasparevitta and let me know your thoughts!

You can find the code snippets on Github.
This article was originally published on my blog. Head over there if you like this post and want to read others like it!

Advanced Docker: how to use secrets the right way

gasparev — Tue, 27 Oct 2020 15:11:02 +0000

Secrets are one of the sneakiest vulnerability issues you can have in a Docker image if you don't know how to handle them.

If you need to clone a private repository or to download a private package you must use sensitive data during your docker build, there's no easy way around that.

In this tutorial on the advanced usage of Docker series, I'll explain how to use a build secret in a safe way.

What is Buildkit

I explained last week what is the Buildkit build engine, how to set it up, and how you can use Buildkit to speed up docker build.

Never use `COPY` and `rm`

If you are dealing with secrets during your development, I'm sure the first thing you've tried is to COPY a file with credentials from your Dockerfile and then remove it with rm when you don't need it anymore...

This is so wrong

because you are just deleting the file from that layer but the credentials are still in the layer above!

Unsafe code example

Let's use this Dockerfile.

FROM ubuntu:bionic
COPY .netrc /
RUN rm .netrc

And let's build it.

$ docker build -t unsafe . -f Dockerfile.not-safe
Sending build context to Docker daemon  4.096kB
Step 1/3 : FROM ubuntu:bionic
 ---> c14bccfdea1c
Step 2/3 : COPY .netrc /
 ---> 18d1eb74c6da
Step 3/3 : RUN rm .netrc
 ---> Running in fafd31acf728
Removing intermediate container fafd31acf728
 ---> d7d4315738a6
Successfully built d7d4315738a6
Successfully tagged unsafe:latest

Now we want to use the command docker history to list all the layers of the image.

$ docker history d7d4315738a6
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
d7d4315738a6        10 seconds ago      /bin/sh -c rm .netrc                            0B
18d1eb74c6da        14 seconds ago      /bin/sh -c #(nop) COPY file:a0fa732884be950b…   19B
c14bccfdea1c        4 weeks ago         /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B
<missing>           4 weeks ago         /bin/sh -c mkdir -p /run/systemd && echo 'do…   7B
<missing>           4 weeks ago         /bin/sh -c [ -z "$(apt-get indextargets)" ]     0B
<missing>           4 weeks ago         /bin/sh -c set -xe   && echo '#!/bin/sh' > /…   745B
<missing>           4 weeks ago         /bin/sh -c #(nop) ADD file:84f8ddc4d76e1e867…   63.2MB

Here we can see the latest layer created, d7d4315738a6, but you don't care about it.
The best part is the previous layer, 18d1eb74c6da, which we can analyze deeper.

$ docker run -it 18d1eb74c6da
root@09fb719ec3dc:/# cat .netrc
my secret password

This is the deal: every layer of your image is available, including the ones with your secrets!

Think about it next time you do COPY and rm.

Buildkit to the rescue with `--secret`

Buildkit adds a new flag called --secret for the docker build command. You can use it to provide safely a secret to your Dockerfile at build time! Buildkit mounts the secret using tmpfs in a temporary file located in /run/secrets that we can use to access a secret in the Dockerfile.

Using this feature we are sure that no secrets will remain in the image!

Safe code example

Dockerfile

Let's use the following Dockerfile

# syntax = docker/dockerfile:1.0-experimental

FROM ubuntu:bionic

RUN --mount=type=secret,id=mynetrc,dst=/.netrc cat /.netrc
RUN cat /.netrc

The first thing to notice is # syntax = docker/dockerfile:1.0-experimental, we tell Docker to use the new syntax to exploit the new Buildkit functionality.
Then, with the first RUN command, the magic happens. We tell Docker to mount a secret with the id mynetrc to the destination /.netrc and in the same line, we execute the cat command just for the sake of the example.
Then we RUN again the cat command on the same file.

Build command

To build our Dockerfile this is the command:

$ DOCKER_BUILDKIT=1 docker build --secret id=mynetrc,src=.netrc --progress=plain --no-cache -f Dockerfile.safe -t safe .

You can notice here the flag --secret which tells Docker the secret name and location. We also need to set DOCKER_BUILDKIT=1 to use the Buildkit build engine.

OK, let's build it.

$ DOCKER_BUILDKIT=1 docker build --secret id=mynetrc,src=.netrc --progress=plain --no-cache -f Dockerfile.safe -t safe .

#...

#8 [1/3] FROM docker.io/library/ubuntu:bionic
#8 CACHED

#9 [2/3] RUN --mount=type=secret,id=mynetrc,dst=/.netrc cat /.netrc
#9 0.808 my secret password
#9 DONE 1.7s

#10 [3/3] RUN cat /.netrc
#10 DONE 2.0s

#11 exporting to image
#11 exporting layers
#11 exporting layers 0.7s done
#11 writing image sha256:b86ed6e0585c2f2e5cb14796b896dae0004f75004ccece0949a3de0ca600b113 0.0s done
#11 naming to docker.io/library/safe 0.0s done
#11 DONE 1.0s

As you can see, in the RUN --mount=type=secret,id=mynetrc,dst=/.netrc cat /.netrc we can access the content of the file, instead on the following RUN there's no output.

The .netrc file, in fact, is present in the final layer of the image but it's empty.

Let's use the command docker history to list all the layers of this new image.

$ docker history b86ed6e0585c
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
b86ed6e0585c        5 hours ago         RUN /bin/sh -c cat /.netrc # buildkit           0B                  buildkit.dockerfile.v0
<missing>           5 hours ago         RUN /bin/sh -c cat /.netrc # buildkit           0B                  buildkit.dockerfile.v0
<missing>           4 weeks ago         /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B                  
<missing>           4 weeks ago         /bin/sh -c mkdir -p /run/systemd && echo 'do…   7B                  
<missing>           4 weeks ago         /bin/sh -c [ -z "$(apt-get indextargets)" ]     0B                  
<missing>           4 weeks ago         /bin/sh -c set -xe   && echo '#!/bin/sh' > /…   745B                
<missing>           4 weeks ago         /bin/sh -c #(nop) ADD file:84f8ddc4d76e1e867…   63.2MB

As you can see, it's not possible now to load an older layer to read the secret.

This is it!

I hope this was useful for you, now go and refactor your old Dockerfile!

Reach me on Twitter @gasparevitta and let me know how you use manage secrets.

You can find the code snippets on Github.

This article was originally published on my blog. Head over there if you like this post and want to read others like it!

3 steps to drastically improve your docker build performances

gasparev — Mon, 19 Oct 2020 14:08:43 +0000

Docker is the tool we use every day in our development, but how much time do you waste waiting for Docker build to complete? And how do you deal with gigantic image size?

What if I tell you there's a better way to build your containers?

Your favorite next tool is called Buildkit!

In this tutorial, we'll dive into the advanced usage of Docker to optimize your development process either in build time and in the size of the image itself. We will do it using Buildkit parallel multistage builds.

Buildkit

Buildkit is a toolkit developed by the Moby project to enhance the build and the packaging of software using containers.

Main features

Among the different features, Buildkit offers automatic garbage collection to clean up unneeded resources, concurrent dependency resolution, and efficient instruction caching. Buildkit is part of docker build since Docker 18.06.

How to enable Buildkit

If you want to use the Buildkit powered build engine you can do it using the environment variable DOCKER_BUILDKIT=1 docker build.

It's also possible to enable Buildkit by default:

Edit the daemon configuration in /etc/docker/daemon.json and add

{ "features": { "buildkit": true } }

Restart the daemon with

sudo systemctl daemon-reload
sudo systemctl restart docker

Code example

For this tutorial, we are going to prepare an image to deploy an instance of Prometheus in production. We will start from a standard Dockerfile and we will refactor it to improve performances.

Legacy Dockerfile

We are going to build Prometheus from source code, to do that we need a Docker image with all its build dependencies: golang, nodejs, yarn, and make.

FROM ubuntu:bionic

ENV GOPATH=$HOME/go
ENV PATH=$PATH:/usr/local/go/bin:$GOPATH/bin

RUN apt-get update \
    && apt-get install -y curl git build-essential \
    && curl -sL https://deb.nodesource.com/setup_14.x | bash - \
    && apt-get install -y nodejs \
    && npm install -g yarn \
    && curl -O https://storage.googleapis.com/golang/go1.15.2.linux-amd64.tar.gz \
    && tar -xvf go1.15.2.linux-amd64.tar.gz \
    && mv go /usr/local \
    && git clone https://github.com/prometheus/prometheus.git prometheus/ \
    && cd prometheus/ \
    && make build 
# RUN ./prometheus --config.file=your_config.yml

and let's build it with:

$ time docker build --no-cache -t prometheus . -f Dockerfile.prometheus

...

Successfully built 54b5d99ef76a
Successfully tagged prometheus:latest

real    19m56,395s
user    0m0,506s
sys     0m0,334s

The image size is:

$ docker images
REPOSITORY                    TAG                 IMAGE ID            CREATED              SIZE
prometheus                    latest              54b5d99ef76a        25 minutes ago       2.38GB

Legacy build performance

Looking at the results we needed almost 20 minutes to create an instance of Prometheus that has a size of 2.38GB.
This will be our starting point.

Multistage build

Now we have an image ready for production, so we are happy, right?

No, we definitely are not

As you may have noticed, the image we've just created is huuuge, we can definitely do better using an advanced Docker feature called multistage build.
The multistage build is available in Docker since the 17.05 version and it is the go-to way to optimize image size. You can use the FROM ... AS ... instruction to define a build stage and the COPY --from instruction to share artifacts between stages.

Refactor legacy Dockerfile to use multistage build

Let's apply these concepts to the old Dockerfile.

FROM ubuntu:bionic as base-builder

ENV GOPATH=$HOME/go
ENV PATH=$PATH:/usr/local/go/bin:$GOPATH/bin

RUN apt-get update \
    && apt-get install -y curl git build-essential \
    && curl -sL https://deb.nodesource.com/setup_14.x | bash - \
    && apt-get install -y nodejs \
    && npm install -g yarn \
    && curl -O https://storage.googleapis.com/golang/go1.15.2.linux-amd64.tar.gz \
    && tar -xvf go1.15.2.linux-amd64.tar.gz \
    && mv go /usr/local \
    && git clone https://github.com/prometheus/prometheus.git prometheus/ \
    && cd prometheus/ \
    && make build
FROM ubuntu:bionic as final
COPY --from=base-builder prometheus/prometheus prometheus
# RUN ./prometheus --config.file=your_config.yml

What we need to do is to create a tiny final stage that contains only the Prometheus executable. We can do it with COPY --from the previous stage.

It's time to build the Docker image.

$ time docker build --no-cache -t prometheus-multistage . -f Dockerfile.prometheus-multistage
...

Successfully built ab2217626102
Successfully tagged prometheus-multistage:latest

real    19m19,570s
user    0m0,418s
sys     0m0,459s

The image size is.

$ docker images
REPOSITORY              TAG                 IMAGE ID            CREATED             SIZE
prometheus-multistage   latest              ab2217626102        31 seconds ago      151MB

Multistage build performance

Looking at the new results we spent 19 minutes to build the image but the improvement on the size is a significant 99.94% reduction!

Parallel multistage build

So we were able to reduce the image size but the build time is still too much. We can still optimize that by exploiting the Buildkit build engine.
The legacy Docker build engine executes the build of the stages sequentially, on the other hand, Buildkit computes the dependency graph of the stages and parallelize the builds. With this in mind, we can refactor the Dockerfile to speed up the build time.

Refactor Dockerfile to use parallel multistage build

Let's see how this can be done.

FROM ubuntu:bionic as base-builder

ENV GOPATH=$HOME/go
ENV PATH=$PATH:/usr/local/go/bin:$GOPATH/bin

RUN apt-get update \
    && apt-get install -y curl git build-essential

FROM base-builder as base-builder-extended
RUN curl -sL https://deb.nodesource.com/setup_14.x | bash - \
    && apt-get install -y nodejs \
    && npm install -g yarn

FROM base-builder as golang
RUN curl -O https://storage.googleapis.com/golang/go1.15.2.linux-amd64.tar.gz \
    && tar -xvf go1.15.2.linux-amd64.tar.gz

FROM base-builder as source-code
RUN git clone https://github.com/prometheus/prometheus.git prometheus/

FROM base-builder-extended as builder
COPY --from=golang go /usr/local
COPY --from=source-code prometheus/ prometheus/
RUN cd prometheus/ && make build

FROM ubuntu:bionic as final
COPY --from=builder prometheus/prometheus prometheus
# RUN ./prometheus --config.file=your_config.yml

We create a first stage called base-builder that contains the basic tools and will act as a base for the next layers.
Inheriting from base-builder we define:

golang, that contains go;
source-code, that we use to fetch Prometheus source code;
base-builder-extended that is an enhancement of base-builder that contains nodejs and yarn;

The 3 stages don't depend on each other so the build will be parallelized.

At this point we are ready to build the code, we use builder for that. In this stage, we COPY --from the previous stages the artifacts we need to run the build. Then again we create a tiny final stage that contains only the Prometheus executable.

We can run the build now.

$ DOCKER_BUILDKIT=1 docker build --no-cache -t prometheus-parallel-multistage . -f Dockerfile.prometheus-parallel-multistage
[+] Building 734.4s (13/13) FINISHED                                                                                                  
 => [internal] load build definition from Dockerfile.prometheus-parallel-multistage                                              1.1s
 => => transferring dockerfile: 963B                                                                                             0.1s
 => [internal] load .dockerignore                                                                                                0.8s
 => => transferring context: 2B                                                                                                  0.1s
 => [internal] load metadata for docker.io/library/ubuntu:bionic                                                                 0.0s
 => CACHED [final 1/2] FROM docker.io/library/ubuntu:bionic                                                                      0.0s
 => [base-builder 2/2] RUN apt-get update     && apt-get install -y curl git build-essential                                   195.6s
 => [source-code 1/1] RUN git clone https://github.com/prometheus/prometheus.git prometheus/                                    77.6s 
 => [base-builder-extended 1/1] RUN curl -sL https://deb.nodesource.com/setup_14.x | bash -     && apt-get install -y nodejs   102.1s 
 => [golang 1/1] RUN curl -O https://storage.googleapis.com/golang/go1.15.2.linux-amd64.tar.gz     && tar -xvf go1.15.2.linux  149.8s 
 => [builder 1/3] COPY --from=golang go /usr/local                                                                              13.6s 
 => [builder 2/3] COPY --from=source-code prometheus/ prometheus/                                                                9.5s 
 => [builder 3/3] RUN cd prometheus/ && make build                                                                             338.6s 
 => [final 2/2] COPY --from=builder prometheus/prometheus prometheus                                                             2.6s 
 => exporting to image                                                                                                           1.9s 
 => => exporting layers                                                                                                          1.6s 
 => => writing image sha256:c0e59c47a790cb2a6b1229a5fec0014aa2b4540fc79c51531185c9466c9d5584                                     0.1s 
 => => naming to docker.io/library/prometheus-parallel-multistage                                                                0.1s

And check the image size.

$ docker images
REPOSITORY                       TAG                 IMAGE ID            CREATED             SIZE
prometheus-parallel-multistage   latest              c0e59c47a790        About a minute ago  151MB
prometheus-multistage            latest              ab2217626102        9 minutes ago       151MB
prometheus                       latest              54b5d99ef76a        39 minutes ago      2.38GB

Parallel multistage build performance

Looking at the new results we spent almost 12.5 minutes to build the image, a 30% reduction, keeping the same image size.

Results recap

The table below summarizes the build time and the image size in the three different examples.

Dockerfile	Build time	Image size
prometheus-parallel-multistage	12.5 m	151MB
prometheus-multistage	19 m	151MB
prometheus	20 m	2.38GB

As you can see the improvement, both in build time and in image size, is really huge. Using the multistage parallel build approach can be useful in production where a smaller Docker image can make the difference. All you have to do is to keep in mind how Buildkit works, think of what can be parallelized in your Dockerfile and develop it accordingly. You can easily integrate Buildkit in your Docker build/test/tag/push pipeline).

This is it!

I hope this was useful for you, now go and refactor your old Dockerfile!

Reach me on Twitter @gasparevitta and let me know your performance improvements.

You can find the code snippets on Github.

This article was originally published on my blog. Head over there if you like this post and want to read others like it!

Code security: evaluation of the new Github code scanning function

gasparev — Tue, 13 Oct 2020 16:44:55 +0000

Last week Github released their first official version of Code Scanning. It allows users to quickly scan their source code to identify vulnerabilities. But I have a question.

How reliable is it?

In this post, I'll try to give an answer to that based on a sample C code.

I will use CodeQl to scan a deliberately bugged code and then we will analyze the results to check the performance of the tool.

CodeQl code scanning

Github code scanning uses CodeQl, a semantic code analysis engine. It runs queries on your code to identify potential threats and bad patterns. It supports a huge variety of languages (C/C++, Go, Java, JavaScript, Python, ecc.) and it has a cool action called autobuild that you can use to build your code.

How to set it up

To enable CodeQl in your GitHub repository, you can follow the official documentation.

If you are a VS code user there's also an extension for you.

Reference Environment

For this evaluation I start from the standard CodeQl configuration, then I specify the language of the source code for the analysis: language: ['cpp'] plus an extended set of queries: queries: +security-extended, security-and-quality.

name: "CodeQL"

on:
  push:
    branches: [main]
  pull_request:
    # The branches below must be a subset of the branches above
    branches: [main]
  schedule:
    - cron: '0 13 * * 3'

jobs:
  analyze:
    name: Analyze
    runs-on: ubuntu-latest

    strategy:
      fail-fast: false
      matrix:
        language: ['cpp']

    steps:
    - name: Checkout repository
      uses: actions/checkout@v2
      with:
        fetch-depth: 2

    - run: git checkout HEAD^2
      if: ${{ github.event_name == 'pull_request' }}

    # Initializes the CodeQL tools for scanning.
    - name: Initialize CodeQL
      uses: github/codeql-action/init@v1
      with:
        languages: ${{ matrix.language }}
        queries: +security-extended, security-and-quality

    - run: |
        make

    - name: Perform CodeQL Analysis
      uses: github/codeql-action/analyze@v1

Reference code

I'm using a C code that contains, on purpose, some traits, and errors. The reference code is the fuzzgoat repository from fuzzstation and it contains 4 main vulnerabilities:

Use of memory after a free:

/******************************************************************************
WARNING: Fuzzgoat Vulnerability

The line of code below frees the memory block referenced by *top if
the length of a JSON array is 0. The program attempts to use that memory
block later in the program.
Diff       - Added: free(*top);
Payload    - An empty JSON array: []
Input File - emptyArray
Triggers   - Use after free in json_value_free()
******************************************************************************/

               free(*top);
/****** END vulnerable code **************************************************/

2 invalid frees:

   switch (value->type)
      {
         case json_object:

            if (!value->u.object.length)
            {
               settings->mem_free (value->u.object.values, settings->user_data);
               break;
            }

/******************************************************************************
  WARNING: Fuzzgoat Vulnerability

  The line of code below incorrectly decrements the value of 
  value->u.object.length, causing an invalid read when attempting to free the 
  memory space in the if-statement above.
  Diff       - [--value->u.object.length] --> [value->u.object.length--]
  Payload    - Any valid JSON object : {"":0}
  Input File - validObject
  Triggers   - Invalid free in the above if-statement
******************************************************************************/

            value = value->u.object.values [value->u.object.length--].value;
/****** END vulnerable code **************************************************/

            continue;

         case json_string:

/******************************************************************************
  WARNING: Fuzzgoat Vulnerability

  The code below decrements the pointer to the JSON string if the string
  is empty. After decrementing, the program tries to call mem_free on the
  pointer, which no longer references the JSON string.
  Diff       - Added: if (!value->u.string.length) value->u.string.ptr--;
  Payload    - An empty JSON string : ""
  Input File - emptyString
  Triggers   - Invalid free on decremented value->u.string.ptr
******************************************************************************/

            if (!value->u.string.length){
              value->u.string.ptr--;
            }
/****** END vulnerable code **************************************************/

Null pointer dereference:

  /******************************************************************************
  WARNING: Fuzzgoat Vulnerability
  The code below creates and dereferences a NULL pointer if the string
  is of length one.
  Diff       - Check for one byte string - create and dereference a NULL pointer
  Payload    - A JSON string of length one : "A"
  Input File - oneByteString
  Triggers   - NULL pointer dereference
******************************************************************************/

            if (value->u.string.length == 1) {
              char *null_pointer = NULL;
              printf ("%d", *null_pointer);
            }
/****** END vulnerable code **************************************************/

Code analysis results

These are the results from CodeQl.

Known vulnerabilities

As you can see in the analysis above, none of the vulnerability that were introduced on purpose on fuzzgoat are discovered by CodeQl.

free errors are difficult to spot

but this is not true for a NULL pointer dereference! It can easily be caught by other tools like clang:

$ scan-build-8 make all
scan-build: Using '/usr/lib/llvm-8/bin/clang' for static analysis
/usr/share/clang/scan-build-8/bin/../libexec/ccc-analyzer -o fuzzgoat -I. main.c fuzzgoat.c -lm
fuzzgoat.c:298:29: warning: Dereference of null pointer (loaded from variable 'null_pointer')
              printf ("%d", *null_pointer);
                            ^~~~~~~~~~~~~

New findings

On the other side, let's analyze the actual results.

CodeQl was able to locate a potential buffer overflow which is a crucial vulnerability.
That one is probably a false positive since the size of the buffer is fixed, but in general, it's a good practice to be cautious when performing a buffer write operation.

Apart from that, the other findings are:

Missing enum case in the switch;
Poorly documented large function;
Time-of-check time-of-use filesystem race condition;
Uncontrolled data used in path expression;
Potentially uninitialized local variable.

This is it!

After this evaluation, I think that CodeQl is a good tool for the daily use to get insight on the code quality: it's easy to use and configure, it can be quickly integrated into your GitHub repository and it provides small actionable results.

The downside is that you can have a high number of false positives to deal with.

Reach me on Twitter @gasparevitta and let me know your thoughts!

You can find the code snippets on Github

This article was originally published on my blog. Head over there if you like this post and want to read others like it!

Docker unit test: how to test a Dockerfile (Guide 2020)

gasparev — Wed, 07 Oct 2020 13:50:38 +0000

You know you should test everything...

Don't you?

Well, writing unit test for Docker should be part of your daily routine while developing a new Dockerfile. It can save you a loooot of time spent running a Docker image trying to figure out why is not working and it will drastically reduce your fear of rebuilding and updating a container (If you still don't believe me on testing, read this article by James Shore).

In this guide you will learn: which tools can help you testing your Dockerfile, how to write a unit test for Docker and how to automate it in a continuous integration pipeline.

Docker container structure

The best tool I can raccomand to write a unit test for a Docker is the Container Structure Test framework.
This framework, developed by Google, makes super easy to test the structure of your container image.

How to install

If you are using Linux run:

curl -LO https://storage.googleapis.com/container-structure-test/latest/container-structure-test-linux-amd64 && chmod +x container-structure-test-linux-amd64 && sudo mv container-structure-test-linux-amd64 /usr/local/bin/container-structure-test

Test options

Container Structure Test offers 4 types of test:

Command Tests: execute a command in your image and check the output
File Existence Tests: check if a file is, or isn't, present in the image
File Content Tests: check the content of a file
Metadata Test: check if a container metadata is correct

How to write a docker unit test

All you need is a Dockerfile and a .yaml or .json file that contains your test cases.

Write your first Docker unit test

For this example we will use the following Dockerfile for an image that can be used in the CI to build the code using Bazel.

FROM ubuntu:bionic

RUN apt-get update \
  && apt-get install -y curl gnupg \
  && curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor > bazel.gpg \
  && mv bazel.gpg /etc/apt/trusted.gpg.d/ \
  && echo "deb [arch=amd64] https://storage.googleapis.com/bazel-apt stable jdk1.8" > /etc/apt/sources.list.d/bazel.list \
  && apt-get update \
  && apt-get install -y bazel \
  && rm -rf /var/lib/apt/lists/*

RUN groupadd -g 1000 user \
  && useradd -d /home/user -m -u 1000 -g 1000 user \
  && chown -R user:user /home/user \
  && mkdir -p /bazel/cache \
  && chown -R user:user /bazel

RUN echo "build --repository_cache=/bazel/cache">/home/user/.bazelrc

And can be built with:

docker build -t docker-unit-test .

Now we have a Docker image that we set up as root but on the CI we want to mimic the developer build environment as much as possible, to do this we will run the build as a non root user.

What could go wrong?

A lot of things actually!

Do the user own build configuration files? Or the cache folder? Well you can check all of that before deploying your Docker image anywhere.

Let's create unit-test.yaml to test it!

schemaVersion: '2.0.0'
fileExistenceTests:
  - name: 'Check bazel cache folder'
    path: '/bazel/cache'
    shouldExist: true
    uid: 1000
    gid: 1000
    isExecutableBy: 'group'
fileContentTests:
  - name: 'Cache folder config'
    path: '/home/user/.bazelrc'
    expectedContents: ['.*build --repository_cache=/bazel/cache.*']

The first test Check bazel cache folder will check that the cache folder exists and is owned by the non-root user. The second test Cache folder config will check that the Bazel build configuration file content is as expected.

Everything is set, we can run our test in this way:

$ container-structure-test test --image docker-unit-test --config unit-test.yaml

=======================================
====== Test file: unit-test.yaml ======
=======================================
=== RUN: File Content Test: cache folder config
--- PASS
duration: 0s
=== RUN: File Existence Test: Check bazel cache folder
--- PASS
duration: 0s

=======================================
=============== RESULTS ===============
=======================================
Passes:      2
Failures:    0
Duration:    0s
Total tests: 2

PASS

This framework can be super useful for testing your Docker image before shipping it, it's fast and easy to use.

Automate the testing of Docker containers

Ok now we have our Dockerfile and tests ready, it's time to automate the testing process!

In this example I'm assuming that you have an Ansible pipeline that you use in Continuos Integration to build, tag and push a docker image. We are going to create a new task for that pipeline to execute the Docker unit test.

- name: unit test Docker Image
  shell: |
    container-structure-test test --image {{ docker_image }} --config {{ test_file }}
    if $?
    then
      echo "Test Failed"
      exit 1
    else
      echo "Test Succeeded"
      exit 0
    fi

This is it!

Reach me on Twitter @gasparevitta and let me know your thoughts!

I hope you find it useful and that you will start testing your Dockerfile from now on.

You can find the code snippets on Github

This article was originally published on my blog. Head over there if you like this post and want to read others like it!

DEV Community: gasparev

How to test Bash scripts (2020 guide)

Bach

How to install Bach

Prerequisites

What can be done

How to write test cases

How to run Bach tests

This is it!

Speed up Docker build time with cache warming

How to warm the cache in docker

How to warm the cache with Buildkit

This is it!

Docker best practices: lint your Dockerfile!

What is a linter? Why we need it?

Hadolint

How to set it up

Define custom rules

How to run it in CI

This is it!

Advanced Docker: how to use secrets the right way

What is Buildkit

Never use COPY and rm

Unsafe code example

Buildkit to the rescue with --secret

Safe code example

Dockerfile

Build command

This is it!

3 steps to drastically improve your docker build performances

Buildkit

Main features

How to enable Buildkit

Code example

Legacy Dockerfile

Legacy build performance

Multistage build

Refactor legacy Dockerfile to use multistage build

Multistage build performance

Parallel multistage build

Refactor Dockerfile to use parallel multistage build

Parallel multistage build performance

Results recap

This is it!

Code security: evaluation of the new Github code scanning function

CodeQl code scanning

How to set it up

Reference Environment

Reference code

Code analysis results

Known vulnerabilities

New findings

This is it!

Docker unit test: how to test a Dockerfile (Guide 2020)

Docker container structure

How to install

Test options

How to write a docker unit test

Write your first Docker unit test

Automate the testing of Docker containers

This is it!

Never use `COPY` and `rm`

Buildkit to the rescue with `--secret`