In this blog, I will be talking about setting up a GitLab remote template for running a Chaos Experiment in the GitLab CI environment with LitmusChaos. Before jumping in, let's do a quick recap on Litmus. Litmus is a framework for practising chaos engineering in cloud-native environments. Litmus provides a chaos operator, a large set of chaos experiments on its hub, detailed documentation, and a friendly community. Litmus also has Github Chaos Actions and GitLab remote templates for the members who are using litmus experiments as part of their CI pipelines.
- Kubernetes 1.11 or later in your CI workflow
GitLab is a DevOps lifecycle tool which is extremely famous for its continuous integration and deployment pipeline features in the DevOps world nowadays which are configured using a YAML file called .gitlab-ci.yml within each project. The
.gitlab-ci.yaml defines the structure and determines what to execute using GitLab Runner .It deals with what decisions to be taken for when specific conditions are encountered. Like when a process is passed or failed. In GitLab we can include an external yaml file using the
include keyword these external files are called templates.
This helps to break down the CI/CD configuration into multiple files and increases readability for long configuration files.
It’s also possible to have template files stored in a central repository(like litmus-templates) and projects include their configuration files. This helps avoid duplicated configuration, for example, global default variables for all projects.
The addition of a template in GitLab is a very simple job. We just need to add the
include keyword that requires the external YAML file to have the extensions
.yaml, otherwise the external file is not included. include supports the following inclusion methods:
- local: It Includes a file from the local project repository and referenced using full paths relative to the root directory. we can use it like this:
include: - local: '/templates/.gitlab-chaos-template.yml'
- file: We can include files from another private project under the same GitLab instance using file along with include. It can be added with:
include: - project: 'mayadata-io/gitlab-remote-template' file: '/templates/pod-delete-template.yml'
- template: It used to include the pre-built GitLab template. It can be used like:
# File sourced from GitLab's template collection include: - template: Auto-DevOps.gitlab-ci.yml
- remote: It can be used to include a file from a different location, using HTTP/HTTPS, referenced by using the full URL.This makes the remote template very popular among all and it is widely used in many projects. The remote file must be publicly accessible through a simple GET request as authentication schemas in the remote URL are not supported. This brings LitmusChaos Remote Templates into the picture which is used to induce chaos on an application. The yaml templates are stored in a github repository and can be used as an external template.
include: - remote: 'https://raw.githubusercontent.com/mayadata-io/gitlab-remote-templates/master/templates/pod-delete-template.yml'
If you are using/willing to use litmus experiments as part of your CI pipelines and using GitLab as your CI platform then you are at the right place. LitmusChaos introduced GitLab remote templates to integrate the chaos experiments in your CI workflow. It helps to manage your configuration file and provide a simple way to perform the chaos testing on your application. We need to follow some simple steps to setup GitLab Remote Template in your CI workflow:
1. Setup a cluster: Before we begin make sure that we have a kubernetes cluster setup in our CI workflow this is more of a pre-requisite step. If we don’t have any cloud cluster like GKE or AWS we can set up a light weight simple clusters like KinD cluster.
2. Deploy an application: Once we setup a cluster we need to have an application on which we want to perform the chaos testing. If we don’t have the application running in a pod don’t worry we can set that in these two simple steps.
i. Create an Image of the application: We need to Dockerize the application for this we need to have a Dockerfile which can be used to create the application image. Let us suppose that we have a Dockerfile at location
/build from root. So we will build the image with that.
ii. Run a pod with the application container: Now as we have our application image in the Docker registry. We can deploy an application pod with that image. You can refer to a sample pod manifest that can be used to create an application pod from here replace the nginx image with your image and other attributes. Now apply the pod manifest and wait for it to come in Running state.
kubectl apply -f app.yaml
3. Select a Chaos template: Now select a chaos template from the list of templates available in the chaos template repository. Based on the experiment you want to perform, take the raw link of the template. Make sure before running any experiment we need to install litmus which can be done using the install litmus template present in the same list. A chaos template looks like:
--- variables: APP_NS: default APP_KIND: deployment APP_LABEL: APPLICATION_LABEL TARGET_CONTAINER: TARGET_CONTAINER_NAME TOTAL_CHAOS_DURATION: "30" CHAOS_INTERVAL: "10" FORCE: "false" EXPERIMENT_IMAGE: litmuschaos/go-runner EXPERIMENT_IMAGE_TAG: 1.7.0 .pod_delete_template: image: name: mayadata-io/chaos-ci-lib:ci entrypoint: ["./pod-delete"] script: - echo "Running Pod Delete Experiment"
Get the raw link by clicking on the raw button.
And copy the raw link
4. Add template in .gitlab-ci-yml: Now we need to add the chaos template in the GitLab configuration file for which we need to add a remote template URL with the
include: - remote: 'https://raw.githubusercontent.com/mayadata-io/gitlab-remote-templates/master/templates/node-cpu-hog-template.yml' stages: - chaosInject Inject Node CPU Hog: stage: chaosInject extends: .node_cpu_hog_template before_script: variables: APP_NS: default APP_LABEL: "run=nginx" APP_KIND: deployment TARGET_CONTAINER: nginx
Use the link which we copied in step 3 in
5. Add the Chaos Experiment: Once we are done with setting up the chaos infra we can add the chaos experiment which we want to induce on our application in the similar way for example the addition of pod delete chaos has been added in the below code.
include: - remote: 'https://raw.githubusercontent.com/mayadata-io/gitlab-remote-templates/master/templates/install-litmus-template.yml' - remote: 'https://raw.githubusercontent.com/mayadata-io/gitlab-remote-templates/master/templates/node-cpu-hog-template.yml' - remote: 'https://raw.githubusercontent.com/mayadata-io/gitlab-remote-templates/master/templates/uninstall-litmus-template.yml' stages: - appDeploy - chaosInfraSetup - chaosInject - chaosInfraCleanup Deploy to Staging: stage: appDeploy script: - kubectl run nginx --image=nginx --restart=Always Install LitmusChaos: stage: chaosInfraSetup extends: .install_litmus_template before_script: variables: APP_NS: default EXPERIMENT_IMAGE: litmuschaos/go-runner EXPERIMENT_IMAGE_TAG: 1.7.0 Inject Node CPU Hog: stage: chaosInject extends: .node_cpu_hog_template before_script: variables: APP_NS: default APP_LABEL: "run=nginx" APP_KIND: deployment TARGET_CONTAINER: nginx UnInstall LitmusChaos: stage: chaosInfraCleanup extends: .uninstall_litmus_template before_script: variables: APP_NS: default EXPERIMENT_IMAGE: litmuschaos/go-runner EXPERIMENT_IMAGE_TAG: 1.7.0
6. Trigger the pipeline: Once we are done with the set up we can trigger the pipeline and get the results under respective stages of the pipeline. In the same way we can add and perform any chaos experiment in the GitLab CI.
By using GitLab Remote template we can easily integrate LitmusChaos experiment with GitLab which helps to perform the chaos testing on an application in the CI workflow. We just need to make use of the
include:remote feature of GitLab and select the desired templates from the litmuschaos remote templates. So, What are your opinions on running chaos in the CI workflow? If you’re in favor of this then does it seem to be helpful? DevOps experts are welcome to comment and suggest on this!
Are you an SRE or a Kubernetes enthusiast? Does Chaos Engineering excite you?
Join Our Community On Slack For Detailed Discussion, Feedback & Regular Updates On Chaos Engineering For Kubernetes: https://kubernetes.slack.com/messages/CNXNB0ZTN
(#litmus channel on the Kubernetes workspace)
Check out the Litmus Chaos GitHub repo and do share your feedback: https://github.com/litmuschaos/litmus
Submit a pull request if you identify any necessary changes.
Litmus helps Kubernetes SREs and developers practice chaos engineering in a Kubernetes native way. Chaos experiments are published at the ChaosHub (https://hub.litmuschaos.io). Community notes is at https://hackmd.io/a4Zu_sH4TZGeih-xCimi3Q
Cloud-Native Chaos Engineering
Litmus is a toolset to do cloud-native chaos engineering. Litmus provides tools to orchestrate chaos on Kubernetes to help SREs find weaknesses in their deployments. SREs use Litmus to run chaos experiments initially in the staging environment and eventually in production to find bugs, vulnerabilities. Fixing the weaknesses leads to increased resilience of the system.
Litmus takes a cloud-native approach to create, manage and monitor chaos. Chaos is orchestrated using the following Kubernetes Custom Resource Definitions (CRDs):
- ChaosEngine: A resource to link a Kubernetes application or Kubernetes node to a ChaosExperiment. ChaosEngine is watched by Litmus' Chaos-Operator which then invokes Chaos-Experiments
- ChaosExperiment: A resource to group the configuration parameters of a chaos experiment. ChaosExperiment CRs are created by the operator when experiments are invoked by ChaosEngine.
- ChaosResult: A resource to hold the results of a chaos-experiment. The Chaos-exporter reads the…