GitHub Chaos Actions in Your CI/CD workflow [Part-1]

``In this blog, I will be talking about setting GitHub Chaos Actions in your CI workflow. Before jumping in, let's do a quick recap on Litmus. Litmus is a framework for practicing chaos engineering in cloud-native environments. Litmus provides a chaos operator, a large set of chaos experiments on its hub, detailed documentation, and a friendly community. Litmus is very easy to use you can also set up a very quick demo environment to install and run Litmus experiments.

What is Github Chaos Action?

Github actions automate the chaos execution on an application in the same place where the code is stored. You can write individual tasks along with chaos actions and combine them to create a custom GitHub workflow. GitHub Workflows are custom automated processes that you can set up in your repository to build, test, package, or deploy any code project on GitHub. Including the Github chaos actions in your workflow YAML, you can test the performance/resiliency of your application in a much simpler and better way. To know more visit Github chaos actions repository.

What do we need Github Chaos Action?

GitHub Action makes the execution of chaos on an application in a much simpler and easier way. This action could be triggered on the basis of parameter listed in the GitHub workflow YAML. This action can be performed over an application that can be deployed in the cluster or already present in the cluster. Github chaos action helps to fix the weaknesses leads to increased resilience of the system which helps to gain confidence to developers and SRE. It takes a cloud-native approach to create, manage, and monitor the chaos.

Pre-requisites

• Kubernetes 1.11 or later.
• A workflow in your Github Repository
• Application pod in a healthy state
• Pre-requisites of the Chaos Experiment you are executing. You can find this under the experiment section in litmus docs

A Sample Github Chaos Action Workflow
The workflow file is stored under .github/workflows directory of your repository and it uses YAML syntax and must have either a .yml or .yaml file extension. We just need to follow the below-mentioned steps to create a sample GitHub workflow for performing chaos engineering on a Kubernetes application using Github chaos action.

1. Setup name and event to trigger the GitHub workflow
`yaml
name: Push
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
`
This will trigger the GitHub workflow on a push or pull request basis on the master branch. Similarly, it can also be on a scheduled basis.
`yaml
on:
schedule:

• cron: ‘/15 * * * *’ ` **2. Setup jobs in the workflow* `yaml jobs: build: runs-on: ubuntu-latest ` A GitHub workflow run is made up of one or more jobs. Each job runs in an environment specified by runs-on. So in the above case, each job will run on the ubuntu-latest environment. By default jobs run in parallel to run them sequentially, we need to add `yaml needs: <Job-Name>. jobs: build: runs-on: ubuntu-latest needs: job1 steps: — uses: actions/checkout@v2 ` 3. Setup different tasks in a job A job contains a sequence of tasks called Steps. Steps can be used to run commands, tasks, or actions itself. Not all steps run actions, but all actions run as a step. `yaml jobs: build: runs-on: ubuntu-latest steps:
  • uses: actions/checkout@master Using job steps we can run the chaos action over an application and for doing so we need to use our [Github-chaos-actions](https://github.com/marketplace/actions/kubernetes-chaos) in a below-mentioned way: A sample pod deletes experiment workflow: yaml .github/workflows/main.yml name: main on: push: branches: [ master ] jobs: build: runs-on: ubuntu-latest steps: — uses: actions/checkout@master

— name: Running Litmus pod delete chaos experiment
uses: mayadata-io/github-chaos-actions@master
env:
##Pass kubeconfig data from secret in base 64 encoded form
KUBE_CONFIG_DATA: ${{ secrets.KUBE_CONFIG_DATA }}
##If litmus is not installed
INSTALL_LITMUS: true
##Give application info under chaos
APP_NS: default
APP_LABEL: run=nginx
APP_KIND: deployment
EXPERIMENT_NAME: pod-delete
##Custom image can also been used
EXPERIMENT_IMAGE: litmuschaos/ansible-runner
EXPERIMENT_IMAGE_TAG: latest
IMAGE_PULL_POLICY: Always

TOTAL_CHAOS_DURATION: 30
CHAOS_INTERVAL: 10
FORCE: false
##Select true if you want to uninstall litmus after chaos
LITMUS_CLEANUP: true
`
This is a complete single job that is running a chaos action over an application for every push on branch master and the kubeconfig of the cluster and application details are provided in the form of secrets and environment variables respectively. Other Variables are chaos action variables that change on the basis of the chaos experiment we are running.

How to connect the cluster with chaos actions?

1. To connect a healthy cluster with the chaos action we need to pass the kubeconfig of the cluster in the base 64 encoded form as done in the above example. For doing this we need to follow these steps: Convert the kubeconfig of your cluster in base 64 encoded form using this command:

`bash
cat $HOME/.kube/config | base64
`

• Add a secret KUBE_CONFIG_DATA with the value containing the output of the above command.
• Pass this secret as ENV in Github chaos actions to connect your cluster with the action. `yaml
• name: Running Litmus pod delete chaos experiment uses: mayadata-io/github-chaos-actions@master env: KUBE_CONFIG_DATA: ${{ secrets.KUBE_CONFIG_DATA }} `
• Creating a Cluster in the workflow itself is also one of the ways of performing chaos actions. We can even create a cluster in one of the steps of the job and then use Github-chaos-actions in further steps. In this case we don’t need to pass the kubeconfig externally through action’s ENV instead you can setup kubeconfig as an environment variable in base 64 encoded form using: `yaml
• name: "Setup kubeconfig ENV for Github Chaos Action" run: echo ::set-env name=KUBE_CONFIG_DATA::$(base64 -w 0 ~/.kube/config) TheKUBE_CONFIG_DATA` will be accessable to the Chaos Action which is running on a different container. Please visit [Part-2] of the GitHub actions series, we have discussed it in more detailed way.

Monitor the GitHub workflow jobs:

Once the workflow triggers we need to monitor the workflow. For monitoring the different jobs in the workflow:

• Click on the Actions option present in the repository containing workflow.
• It will show all the workflow runs. Select the one which triggered recently
• Check the job logs of the workflow.

Details of chaos action run

• To get the details of what happened when the chaos action was running check the logs on the Github-chaos-actions from the job logs.
• The Github-chaos-action contains the logs of the process involved in running the chaos actions.
• It also contains the logs of the experiment pod which will give a verbose idea of what was happening at the time of chaos injection.
• At the end of the logs, there will be a check of the verdict whether the chaos result verdict is Pass or Fail (It will be a pass for the successful execution of the experiment).
• To know more about the terms like Chaos Experiment, Chaos Engine and Chaos Result visit litmus docs.

To know more about GitHub Chaos Actions and different ways to setup Chaos Actions in your GitHub workflow read Part-2 now.

Conclusion:

We can automate the chaos execution on a review application to check its resiliency in the same place where the codes are stored using Github Action.It provides an easier way to perform different chaos experiments which helps in gaining confidence.

Are you an SRE or a Kubernetes enthusiast? Does Chaos Engineering excite you?
Join Our Community On Slack For Detailed Discussion, Feedback & Regular Updates On Chaos Engineering For Kubernetes: https://kubernetes.slack.com/messages/CNXNB0ZTN
(#litmus channel on the Kubernetes workspace)
Check out the Litmus Chaos GitHub repo and do share your feedback: https://github.com/litmuschaos/litmus
Submit a pull request if you identify any necessary changes.

litmuschaos / litmus

Litmus helps SREs and developers practice chaos engineering in a Cloud-native way. Chaos experiments are published at the ChaosHub (https://hub.litmuschaos.io). Community notes is at https://hackmd.io/a4Zu_sH4TZGeih-xCimi3Q

LitmusChaos

Open Source Chaos Engineering Platform

Read this in other languages.

🇰🇷 🇨🇳 🇧🇷 🇮🇳

Overview

LitmusChaos is an open source Chaos Engineering platform that enables teams to identify weaknesses & potential outages in infrastructures by inducing chaos tests in a controlled way. Developers & SREs can practice Chaos Engineering with LitmusChaos as it is easy to use, based on modern Chaos Engineering principles & community collaborated. It is 100% open source & a CNCF project.

LitmusChaos takes a cloud-native approach to create, manage and monitor chaos. The platform itself runs as a set of microservices and uses Kubernetes custom resources (CRs) to define the chaos intent, as well as the steady state hypothesis.

At a high-level, Litmus comprises of:

• Chaos Control Plane: A centralized chaos management tool called chaos-center, which helps construct, schedule and visualize Litmus chaos workflows
• Chaos Execution Plane Services: Made up of a…

View on GitHub