So you’ve set up continuous integration for your project. Everything looks good and now all you need is a container. Just build and run it, right? Not so fast!
Whether using containers to support development or for packaging an application, it’s easy to take them for granted. But many things can go wrong with them: moved files, incorrect permissions, a user is missing, the Dockerfile is incomplete, the list goes on…
Containers are as crucial as the code they support. In this tutorial, we’ll introduce a different way of testing them before deployment.
One of the many ways Google tests containers
Container Structure Tests (CST) is a container-testing tool developed by Google and open-sourced with the Apache 2.0 license. CST comes with a predefined set of tests for looking at what’s actually inside a container image.
For instance, CST can check if a file exists, run a command and validate its output, or check if the container exposes the correct ports. Almost every declarative keyword in the Dockerfile has a corresponding test.
One important note is that the project is not officially supported by Google, so it doesn’t show a lot of activity. But it’s popular enough to keep it active and it’s still accepting contributions.
More tests, less uncertainty
Any old container should do in order to try out CST. Though it will be easier if we build one from a known Dockerfile. So, if you want to follow along with me as I explore this tool, fork and clone our Ruby Kubernetes demo project:
semaphoreci-demos / semaphore-demo-ruby-kubernetes
A Semaphore demo CI/CD pipeline for Kubernetes.
Semaphore CI/CD demo for Kubernetes
This is an example application and CI/CD pipeline showing how to build, test and deploy a microservice to Kubernetes using Semaphore 2.0.
Ingredients:
- Ruby Sinatra as web framework
- RSpec for tests
- Packaged in a Docker container
- Container pushed to Docker Hub registry
- Deployed to Kubernetes
CI/CD on Semaphore
If you're new to Semaphore, feel free to fork this repository and use it to create a project.
The CI/CD pipeline is defined in .semaphore
directory and looks like this:
Local application setup
To run the microservice:
bundle install --path vendor/bundle
bundle exec rackup
To run tests:
bundle exec rspec
To build and run Docker container:
docker build -t semaphore-demo-ruby-kubernetes
docker run -p 80:4567 semaphore-demo-ruby-kubernetes
curl localhost
> hello world :))
Additional documentation
License
Copyright (c) 2022 Rendered Text
Distributed under the MIT License…
It’s a “Hello, World” application written on Ruby. It comes with a Dockerfile and a complete CI/CD pipeline.
You’ll also need a:
- A Docker installation.
- A CST config file.
- The CST executable.
Once you’ve cloned the demo, build the container image with:
$ docker build -t test-image .
Install the CST tool using the installation instructions. For instance, on Linux:
$ curl -LO https://storage.googleapis.com/container-structure-test/latest/container-structure-test-linux-amd64 && chmod +x container-structure-test-linux-amd64 && sudo mv container-structure-test-linux-amd64 /usr/local/bin/container-structure-test
If you’re running on a non-Intel architecture, you can build the project by yourself or try the CST image instead.
The command to run the test follows works like this:
$ container-structure-test test –config config-cst.yaml –image test-image:latest
But of course, that won’t work until we configure the tests. We’ll do that next.
Setting up CST tests
CST supports three categories of tests:
- Commands: starts the container, runs the command, and validates its result.
- Filesystem: checks for file existence, owner, permissions, and contents.
- Metadata: this category contains things such as environment variables, exposed ports, labels, among other image metadata.
Create a new file called config-cst.yaml
(JSON also works) and add the following mandatory line:
schemaVersion: 2.0.0
We’ll start with the command tests.
Command tests
Let’s try some command tests. We can use something like this to check that Ruby is installed. Of course, I cheated and looked where it’s actually located before trying.
commandTests:
- name: "Ruby is installed"
command: "which"
args: ["ruby"]
expectedOutput: ["/usr/local/bin/ruby"]
Now we’ll check that ruby --version
outputs the correct number:
- name: "Ruby version is correct"
command: "/usr/local/bin/ruby"
args: ["--version"]
expectedOutput: ["ruby 2.7.*"]
Should we be in a really security-conscious mindset, we can checksum the Ruby binary for extra safety. We can run preparation commands before the test with setup
.
- name: "Ruby binary checksum"
setup: [["apt-get", "update"], ["apt-get","install","-y","shatag"]]
command: "sha512sum"
args: ["/usr/local/bin/ruby"]
expectedOutput: ["df2fb393261ab88e5f991b96d958363f5e5185b51f1af319375be0d8b9ed6c27097ac8bfab399798497909c3e9e2bcc6d715a1e514f13fe5b7365344af555c7e /usr/local/bin/rub"]
Now that we have some initial tests, we can actually run the tool for the first time:
$ container-structure-test test --config config.yaml --image test-image:latest
=== RUN: Command Test: Ruby is installed
--- PASS
duration: 391.76725ms
stdout: /usr/local/bin/ruby
=== RUN: Command Test: Ruby version is correct
--- PASS
duration: 319.6335ms
stdout: ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [aarch64-linux]
=== RUN: Command Test: Ruby binary checksum
--- PASS
duration: 302.199834ms
stdout: df2fb393261ab88e5f991b96d958363f5e5185b51f1af319375be0d8b9ed6c27097ac8bfab399798497909c3e9e2bcc6d715a1e514f13fe5b7365344af555c7e /usr/local/bin/ruby
===============================
=========== RESULTS ===========
===============================
Passes: 3
Failures: 0
Duration: 1.013600584s
Total tests: 3
All command tests require a working Docker installation, as the tool must start a temporary container to run them. However, filesystem and metadata tests can be run without Docker appending the --driver tar
option.
Filesystem tests
Filesystem tests inspect the contents of the image; they check if files exist, their permissions, their contents, owner, and group.
Here’s how we can test that the code was correctly copied in the image. If we look at our Dockerfile, it should live in the /app/
folder. So, we define a fileExistenceTests
like this:
fileExistenceTests:
- name: 'app.rb exists and has correct permissions'
path: '/app/app.rb'
shouldExist: true
permissions: '-rw-rw-r--'
uid: 0
gid: 0
We can also check the reverse: that a file is not present in the image. Setting shouldExist
to false
is a great way to avoid shipping sensitive files by mistake. For example, we don’t need the unit tests contained in spec
for the final build.
- name: 'spec/ directory should not exist'
path: '/app/spec'
shouldExist: false
This time, CST should fail because the Docker image has the spec
folder.
$ container-structure-test test --config config.yaml --image test-image:latest
=== RUN: Command Test: Ruby is installed
--- PASS
duration: 308.371875ms
stdout: /usr/local/bin/ruby
=== RUN: Command Test: Ruby version is correct
--- PASS
duration: 312.744792ms
stdout: ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [aarch64-linux]
=== RUN: Command Test: Ruby binary checksum
--- PASS
duration: 286.408167ms
stdout: df2fb393261ab88e5f991b96d958363f5e5185b51f1af319375be0d8b9ed6c27097ac8bfab399798497909c3e9e2bcc6d715a1e514f13fe5b7365344af555c7e /usr/local/bin/ruby
=== RUN: File Existence Test: app.rb exists and has correct permissions
--- PASS
duration: 0s
=== RUN: File Existence Test: spec/ directory should not exist
--- FAIL
duration: 0s
Error: File /app/spec should not exist but does
===============================
=========== RESULTS ===========
===============================
Passes: 4
Failures: 1
Duration: 907.524834ms
Total tests: 5
We can fix the spec
failure by adding the following line into the .dockerignore
(you may also experience a permission error in /app/app.rb
, which can be quickly fixed with chmod
).
spec/
After rebuilding the image, the test should pass.
We have tested that a file exists. But, what about its contents? For that, we should use fileContentTests
. The following example shows how to test if the Ruby Gems have been installed from a safe repository.
fileContentTests:
- name: 'Gemfile remote is rubygems.org'
path: '/app/Gemfile.lock'
expectedContents: ['remote: https://rubygems.org/']
Both types of tests support regular expressions in the expected*
fields for more flexibility.
Metadata tests
Unlike the others, you can only have one metadata test. But it may check several things simultaneously, as metadata tests include a whole range of standard Docker variables.
Let’s say we want to test environment variables. In that case, we use env
.
metadataTest:
env:
- key: APP_HOME
value: /app
- key: RUBY_VERSION
value: 2.7.4
You can also check Docker declarations like WORKDIR
, EXPOSE
, VOLUME
, or USER
.
exposedPorts: ["4567"]
workdir: "/app"
volumes: []
And the always important CMD
and ENTRYPOINT
.
cmd: ["bundle", "exec", "rackup", "--host", "0.0.0.0", "-p", "4567"]
entrypoint: []
Once satisfied with the tests, commit the config file into the repository. This is the final version I got after experimenting with some extra tests follows.
schemaVersion: 2.0.0
commandTests:
- name: "Ruby is installed"
command: "which"
args: ["ruby"]
expectedOutput: ["/usr/local/bin/ruby"]
- name: "Ruby version is correct"
command: "/usr/local/bin/ruby"
args: ["--version"]
expectedOutput: ["ruby 2.7.*"]
- name: "Ruby binary checksum"
setup: [["apt-get", "update"], ["apt-get","install","-y","shatag"]]
command: "sha512sum"
args: ["/usr/local/bin/ruby"]
expectedOutput: ["df2fb393261ab88e5f991b96d958363f5e5185b51f1af319375be0d8b9ed6c27097ac8bfab399798497909c3e9e2bcc6d715a1e514f13fe5b7365344af555c7e /usr/local/bin/rub"]
- name: "Bundle is installed"
command: "which"
args: ["bundle"]
expectedOutput: ["/usr/local/bin/bundle"]
- name: "Bundler version is correct"
command: "/usr/local/bin/bundle"
args: ["--version"]
expectedOutput: ["Bundler version 2.1.*"]
fileExistenceTests:
- name: 'app.rb exists and has correct permissions'
path: '/app/app.rb'
shouldExist: true
permissions: '-rw-rw-r--'
uid: 0
gid: 0
- name: 'spec/ directory should not exist'
path: '/app/spec'
shouldExist: false
fileContentTests:
- name: 'Gemfile remote is rubygems.org'
path: '/app/Gemfile.lock'
expectedContents: ['remote: https://rubygems.org/']
metadataTest:
env:
- key: APP_HOME
value: /app
- key: RUBY_VERSION
value: 2.7.4
exposedPorts: ["4567"]
cmd: ["bundle", "exec", "rackup", "--host", "0.0.0.0", "-p", "4567"]
workdir: "/app"
entrypoint: []
volumes: []
Testing container structure in CI/CD
Of course, CST wouldn’t be a lot of help to us unless it’s part of a CI/CD pipeline. The logical place for structure tests is between container build and deployment.
Adding CST in your CI/CD pipeline is straightforward. To keep things simple, we’ll extend the one already included in the demo, but the steps should work with any pipeline.
First, ensure Semaphore has access to your repository. Follow the getting started guide to learn how to do this.
The demo pipeline builds and tests the Ruby app, then dockerizes it and finally deploys it to Kubernetes. We’ll add a structure test right between Docker build and Kubernetes deploy.
First, ensure you have stored your Docker Hub credentials as a Semaphore secret.
Next, open the workflow editor using the Edit Workflow button.
Expand the continuous delivery pipeline and add a block immediately after the Docker build step.
The CST block will have one job with five commands:
- Sign in with Docker Hub, where the container image is stored.
- Pull the image into the CI environment.
- Install the CST Linux binary.
- Clone the repository so we can access the CST config file.
- Run the tests. If the test fails, the process stops with an error.
The complete command sequence is:
echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
docker pull "${DOCKER_USERNAME}"/semaphore-demo-ruby-kubernetes:$SEMAPHORE_WORKFLOW_ID
curl -LO https://storage.googleapis.com/container-structure-test/latest/container-structure-test-linux-amd64 && chmod +x container-structure-test-linux-amd64 && sudo mv container-structure-test-linux-amd64 /usr/local/bin/container-structure-test
checkout
container-structure-test test --config config-cst.yaml --image "${DOCKER_USERNAME}"/semaphore-demo-ruby-kubernetes:$SEMAPHORE_WORKFLOW_ID
To finalize, enable the dockerhub
secret on the block and give it a try by clicking on Run the workflow.
Once the CI pipeline is complete, an auto-promotion should kick off the continuous delivery pipeline.
The image is ready and tested for the next stage. Now you can deliver containers with more confidence.
Conclusion
The more you know about your container, the less surprises you’ll get. Container Structure Test may not be the most flexible tool, but it certainly is a quick and easy way of adding some confidence to the release process. So, it should be on the radar of anyone using containers for serious work.
Read next:
Top comments (0)