Here’s a common situation that plagues many development teams. You run an application through your CI/CD pipeline and all of the tests pass, which is great. But when you deploy it to a live target environment the application just does not function as expected. You can’t always predict what will happen when your application is pushed live. The solution? Smoke tests are designed to reveal these types of failures early by running test cases that cover the critical components and functionality of the application. They also ensure that the application will function as expected in a deployed scenario. When implemented, smoke tests are often executed on every application build to verify that basic but critical functionality passes before jumping into more extensive and time-consuming testing. Smoke tests help create the fast feedback loops that are vital to the software development life cycle.
In this post I’ll demonstrate how to add smoke testing to the deployment stage of a CI/CD pipeline. The smoke testing will test simple aspects of the application post deployment.
Technologies used for smoke testing
This post will reference the following technologies:
- GitHub
- CircleCI
- Docker
- Kubernetes
- Google Kubernetes Engine (GKE)
- Bash
- smoke.sh - open source smoke testing framework by asm89
- Pulumi
Prerequisites
This post relies on configurations and code that are featured in my previous post Automate releases from your pipelines using Infrastructure as Code. The full source code can be found in this repo.
Getting the most from smoke tests
Smoke tests are great for exposing unexpected build errors, connection errors, and for validating a server’s expected response after a new release is deployed to a target environment. For example, a quick, simple smoke test could validate that an application is accessible and is responding with expected response codes like OK 200
, 300
, 301
, 404
, etc. The examples in this post will test that the deployed app responds with an OK 200
server code and will also validate that the default page content renders the expected text.
Running CI/CD pipelines without smoke tests
Let’s take a look at an example pipeline config that is designed to run unit tests, build, and push a Docker image to Docker Hub. The pipeline also uses infrastructure as code (Pulumi) to provision a new Google Kubernetes Engine (GKE) cluster and to deploy this release to the cluster. This pipeline config example does not implement smoke tests. Please be aware that if you run this specific pipeline example, a new GKE cluster will be created and will live on until you manually run the pulumi destroy
command to terminate all the infrastructure it created.
Caution: Not terminating the infrastructure will result in unexpected costs.
version: 2.1
orbs:
pulumi: pulumi/pulumi@2.0.0
jobs:
build_test:
docker:
- image: cimg/python:3.8.1
environment:
PIPENV_VENV_IN_PROJECT: 'true'
steps:
- checkout
- run:
name: Install Python Dependencies
command: |
pipenv install --skip-lock
- run:
name: Run Tests
command: |
pipenv run pytest
build_push_image:
docker:
- image: cimg/python:3.8.1
steps:
- checkout
- setup_remote_docker:
docker_layer_caching: false
- run:
name: Build and push Docker image
command: |
pipenv install --skip-lock
pipenv run pip install --upgrade 'setuptools<45.0.0'
pipenv run pyinstaller -F hello_world.py
echo 'export TAG=${CIRCLE_SHA1}' >> $BASH_ENV
echo 'export IMAGE_NAME=orb-pulumi-gcp' >> $BASH_ENV
source $BASH_ENV
docker build -t $DOCKER_LOGIN/$IMAGE_NAME -t $DOCKER_LOGIN/$IMAGE_NAME:$TAG .
echo $DOCKER_PWD | docker login -u $DOCKER_LOGIN --password-stdin
docker push $DOCKER_LOGIN/$IMAGE_NAME
deploy_to_gcp:
docker:
- image: cimg/python:3.8.1
environment:
CLOUDSDK_PYTHON: '/usr/bin/python2.7'
GOOGLE_SDK_PATH: '~/google-cloud-sdk/'
steps:
- checkout
- pulumi/login:
version: "2.0.0"
access-token: ${PULUMI_ACCESS_TOKEN}
- run:
name: Install dependencies
command: |
cd ~/
pip install --user -r project/requirements.txt
curl -o gcp-cli.tar.gz https://dl.google.com/dl/cloudsdk/channels/rapid/google-cloud-sdk.tar.gz
tar -xzvf gcp-cli.tar.gz
echo ${GOOGLE_CLOUD_KEYS} | base64 --decode --ignore-garbage > ${HOME}/project/pulumi/gcp/gke/cicd_demo_gcp_creds.json
./google-cloud-sdk/install.sh --quiet
echo 'export PATH=$PATH:~/google-cloud-sdk/bin' >> $BASH_ENV
source $BASH_ENV
gcloud auth activate-service-account --key-file ${HOME}/project/pulumi/gcp/gke/cicd_demo_gcp_creds.json
- pulumi/update:
stack: k8s
working_directory: ${HOME}/project/pulumi/gcp/gke/
workflows:
build_test_deploy:
jobs:
- build_test
- build_push_image
- deploy_to_gcp:
requires:
- build_test
- build_push_image
This pipeline deploys the new app release to a new GKE cluster, but we do not know if the application is actually up and running after all of this automation completes. How do we find out whether the application has been deployed and is functioning properly in this new GKE cluster? Smoke tests are a great way to quickly and easily validate the application’s status after deployment.
How do I write a smoke test?
The first step is to develop test cases that define the steps required to validate an application’s functionality. Identify the functionality that you want to validate, and then create scenarios to test it. In this tutorial, I’m intentionally describing a very minimal scope for testing. For our sample project, my biggest concern is validating that the application is accessible after deployment and that the default page that is served renders the expected static text.
I prefer to outline and list the items I want to test because it suits my style of development. The outline shows the factors I considered when developing the smoke tests for this app. Here is an example of how I developed test cases for this smoke test:
- What language/test framework?
- Bash
- smoke.sh
- When should this test be executed?
- After the GKE cluster has been created
- What will be tested?
- Test: Is the application accessible after it is deployed?
- Expected Result: Server responds with code
200
- Test: Does the default page render the text “Welcome to CI/CD”
- Expected Result:
TRUE
- Test: Does the default page render the text “Version Number: “
- Expected Results:
TRUE
- Post test actions (must occur regardless of pass or fail)
- Write test results to standard output
- Destroy the GKE cluster and related infrastructure
- Run
pulumi destroy
My test case outline (also called a test script) is complete for this tutorial and clearly shows what I’m interested in testing. For this post, I will write smoke tests using a bash-based, open source smoke test framework called smoke.sh
by asm89. For your own projects, you can write smoke tests in what ever language or framework you prefer. I picked smoke.sh
because it’s an easy framework to implement and it’s open source. Now let’s explore how to express this test script using the smoke.sh
framework.
Create smoke test using smoke.sh
The smoke.sh
framework’s documentation describes how to use it. The next block of sample code shows how I used the smoke_test
file found in the test/
directory of the example code’s repo.
#!/bin/bash
. tests/smoke.sh
TIME_OUT=300
TIME_OUT_COUNT=0
PULUMI_STACK="k8s"
PULUMI_CWD="pulumi/gcp/gke/"
SMOKE_IP=$(pulumi stack --stack $PULUMI_STACK --cwd $PULUMI_CWD output app_endpoint_ip)
SMOKE_URL="http://$SMOKE_IP"
while true
do
STATUS=$(curl -s -o /dev/null -w '%{http_code}' $SMOKE_URL)
if [$STATUS -eq 200]; then
smoke_url_ok $SMOKE_URL
smoke_assert_body "Welcome to CI/CD"
smoke_assert_body "Version Number:"
smoke_report
echo "\n\n"
echo 'Smoke Tests Successfully Completed.'
echo 'Terminating the Kubernetes Cluster in 300 second...'
sleep 300
pulumi destroy --stack $PULUMI_STACK --cwd $PULUMI_CWD --yes
break
elif [[$TIME_OUT_COUNT -gt $TIME_OUT]]; then
echo "Process has Timed out! Elapsed Timeout Count.. $TIME_OUT_COUNT"
pulumi destroy --stack $PULUMI_STACK --cwd $PULUMI_CWD --yes
exit 1
else
echo "Checking Status on host $SMOKE... $TIME_OUT_COUNT seconds elapsed"
TIME_OUT_COUNT=$((TIME_OUT_COUNT+10))
fi
sleep 10
done
Next, I’ll explain what’s going on in this smoke_test file.
Line by line description of the smoke_test file
Let’s start at the top of the file.
#!/bin/bash
. tests/smoke.sh
This snippet specifies the Bash binary to use and also specifies the file path to the core smoke.sh
framework to import/include in the smoke_test
script.
TIME_OUT=300
TIME_OUT_COUNT=0
PULUMI_STACK="k8s"
PULUMI_CWD="pulumi/gcp/gke/"
SMOKE_IP=$(pulumi stack --stack $PULUMI_STACK --cwd $PULUMI_CWD output app_endpoint_ip)
SMOKE_URL="http://$SMOKE_IP"
This snippet defines environment variables that will be used throughout the smoke_test
script. Here is a list of each environment variable and its purpose:
-
PULUMI_STACK="k8s"
is used by Pulumi to specify the Pulumi app stack. -
PULUMI_CWD="pulumi/gcp/gke/"
is the path to the Pulumi infrastructure code. -
SMOKE_IP=$(pulumi stack --stack $PULUMI_STACK --cwd $PULUMI_CWD output app_endpoint_ip)
is the Pulumi command used to retrieve the public IP address of the application on the GKE cluster. This variable is referenced throughout the script. -
SMOKE_URL="http://$SMOKE_IP"
specifies the url endpoint of the application on the GKE cluster.
while true
do
STATUS=$(curl -s -o /dev/null -w '%{http_code}' $SMOKE_URL)
if [$STATUS -eq 200]; then
smoke_url_ok $SMOKE_URL
smoke_assert_body "Welcome to CI/CD"
smoke_assert_body "Version Number:"
smoke_report
echo "\n\n"
echo 'Smoke Tests Successfully Completed.'
echo 'Terminating the Kubernetes Cluster in 300 second...'
sleep 300
pulumi destroy --stack $PULUMI_STACK --cwd $PULUMI_CWD --yes
break
elif [[$TIME_OUT_COUNT -gt $TIME_OUT]]; then
echo "Process has Timed out! Elapsed Timeout Count.. $TIME_OUT_COUNT"
pulumi destroy --stack $PULUMI_STACK --cwd $PULUMI_CWD --yes
exit 1
else
echo "Checking Status on host $SMOKE... $TIME_OUT_COUNT seconds elapsed"
TIME_OUT_COUNT=$((TIME_OUT_COUNT+10))
fi
sleep 10
done
This snippet is where all the magic happens. It’s a while
loop that executes until a condition is true or the script exits. In this case, the loop uses a curl
command to test if the application returns an OK 200
response code. Because this pipeline is creating a brand new GKE cluster from scratch, there are transactions in the Google Cloud Platform that need to be complete before we begin smoke testing.
- The GKE cluster and application service must be up and running.
- The
$STATUS
variable is populated with the results of the curl requests then tested for the value of200
. Otherwise, the loop increments the$TIME_OUT_COUNT
variable by 10 seconds, then waits for 10 seconds to repeat thecurl
request until the application is responding. - Once the cluster and app are up, running, and responding, the
STATUS
variable will produce a200
response code and the remainder of the tests will proceed.
The smoke_assert_body "Welcome to CI/CD"
and smoke_assert_body "Version Number: "
statements are where I test that the welcome and version number texts are being rendered on the webpage being called. If the result is false, the test will fail, which will cause the pipeline to fail. If the result is true, then the application will return a 200
response code and our text tests will result in TRUE
. Our smoke test will pass and execute the pulumi destroy
command that terminates all of the infrastructure created for this test case. Since there is no further need for this cluster, it will terminate all the infrastructure created in this test.
This loop also has an elif
(else if) statement that checks to see if the application has exceeded the $TIME_OUT
value. The elif
statement is an example of exception handling which enables us to control what happens when unexpected results occur. If the $TIME_OUT_COUNT
value exceeds the TIME_OUT
value, then the pulumi destroy
command is executed and terminates the newly created infrastructure. The exit 1
command then fails your pipeline build process. Regardless of test results, the GKE cluster will be terminated because there really isn’t a need for this infrastructure to exist outside of testing.
Adding smoke tests to pipelines
I’ve explained the smoke test example and my process for developing the test case. Now it’s time to integrate it into the CI/CD pipeline configuration above. We’ll add a new run
step below the pulumi/update
step of the deploy_to_gcp
job:
...
- run:
name: Run Smoke Test against GKE
command: |
echo 'Initializing Smoke Tests on the GKE Cluster'
./tests/smoke_test
echo "GKE Cluster Tested & Destroyed"
...
This snippet demonstrates how to integrate and execute the smoke_test
script into an existing CI/CD pipeline. Adding this new run block ensures that every pipeline build will test the application on a live GKE cluster and provide a validation that the application passed all test cases. You can be confident that the specific release will perform nominally when deployed to the tested target environment which in this case, is a Google Kubernetes cluster.
Wrapping up
In summary, I’ve discussed and demonstrated the advantages of using smoke tests and Infrastructure as Code in CI/CD pipelines to test builds in their target deployment environments. Testing an application in its target environment provides valuable insight into how it will behave when it’s deployed. Integrating smoke testing into CI/CD pipelines adds another layer of confidence in application builds.
If you have any questions, comments, or feedback please feel free to ping me on Twitter @punkdata.
Thanks for reading!
Top comments (1)
Shisha at home delivery provides a seamless way to enjoy your favorite flavors without stepping outside. Customers can order fresh shisha setups, complete with a variety of flavor options, delivered directly to their door. This service is ideal for creating a cozy atmosphere for gatherings or unwinding after a long day, all from the comfort of home.