In this blog, I will talk about the generation of custom/new chaos experiment with the help of Litmus SDK . Before jumping in, let's do a quick recap on Litmus. Litmus is a framework for practicing Chaos Engineering in cloud-native environments. Litmus provides a chaos-operator, a large set of chaos experiments in its HUB, detailed documentaion, quick demo, and a friendly community.
What is Litmus SDK?
The Litmus SDK provides a simple way to bootstrap your experiment and helps create the aforementioned artifacts in the appropriate directory (i.e., as per the chaos-category) based on an attributes file provided as input by the chart-developer. The scaffolded files consist of placeholders which can then be filled as desired.
It generates the custom chaos experiments with some default Pre & Post Chaos Checks (AUT & Auxiliary Applications status checks). It can use the existing chaoslib (present inside
/chaoslib directory), if available else It will create a new chaoslib inside the corresponding directory.
Life Cycle of a Chaos Experiment
Each Chaos Experiment is divided into six main sections:
- Prepare: Prepare section contains the initialization steps (get ENVs), Updation of chaos result to marked the beginning of the chaos experiment.
- PreChaosCheck: PreChaosCheck contains some default checks (AUT & Auxiliary Application status check) and custom checks, which may vary on the basis of experiment i.e, liveness check, data persistence check, etc.
- ChaosInject: ChaosInject invoke the actual chaos, It contain the main business logic.
- CleanUp: CleanUp contains the steps to remove the helper/external pod, if any
- PostChaosCheck: PostChaosCheck contains the similar steps as the PreCheckCheck section have. It used to ensure the resiliency after chaos injection.
- Summary Summary updates the verdict(Pass/Fail) inside the chaos result and FailStep if the experiment fails.
- go should be is available & the GOPATH env is configured appropriately
- python3 is available (sudo apt-get install python3.6)
- jinja2 & pyYaml python packages are available (sudo apt-get install python3-pip, pip install jinja2, pip install pyYaml)
Steps to Generate Experiment via Litmus SDK
- Clone the litmus-go repository for go experiments and litmus-ansible for the ansible experiments and navigate to the contribute/developer-guide folder.
## for litmus-go $ git clone https://github.com/litmuschaos/litmus-go.git $ cd litmus-go/contribute/developer-guide
## for litmus-ansible $ git clone https://github.com/litmuschaos/litmus-ansible.git $ cd litmus-ansible/contribute/developer_guide
- Populate the attributes.yaml with details of the chaos experiment (or chart). Use the attributes.yaml.sample as reference.
As an example, let us consider an experiment to kill one of the replicas of an Nginx deployment. The attributes.yaml can be constructed like this:
$ cat attributes.yaml --- name: "pod-delete" version: "0.1.0" category: "sample-category" repository: "https://github.com/litmuschaos/litmus-go/tree/master/sample-category/pod-delete" community: "https://kubernetes.slack.com/messages/CNXNB0ZTN" description: "kills nginx pods in a random manner" keywords: - "pods" - "kubernetes" - "sample-category" - "nginx" scope: "Namespaced" auxiliaryappcheck: false permissions: - apigroups: - "" - "batch" - "litmuschaos.io" resources: - "jobs" - "pods" - "chaosengines" - "chaosexperiments" - "chaosresults" verbs: - "create" - "list" - "get" - "update" - "patch" - "delete" maturity: "alpha" maintainers: - name: "ksatchit" email: "email@example.com" provider: name: "Mayadata" minkubernetesversion: "1.12.0" references: - name: Documentation url: "https://docs.litmuschaos.io/docs/getstarted/"
- Run the following command to generate the necessary artifacts for submitting the sample-category chaos chart with pod-delete experiment.
## litmus-go $ go run generate_experiment.go -attributes=attributes.yaml -generateType=experiment
## litmus-ansible $ python3 generate_chart.py --attributes_file=attributes.yaml --generate_type=experiment
Note: In the
-generateType(litmus-go) attribute, select the appropriate type of manifests to be generated, where,
- chart: Just the chaos-chart metadata, i.e., chartserviceversion yaml
- experiment: Chaos experiment artifacts belonging to an existing or new chart.
Verify the Generated Files
$ cd /experiments $ ls -ltr total 8 drwxr-xr-x 3 shubham shubham 4096 May 15 12:02 generic/ drwxr-xr-x 3 shubham shubham 4096 May 15 13:26 sample-category/ $ ls -ltr sample-category/ total 12 -rw-r--r-- 1 shubham shubham 41 May 15 13:26 sample-category.package.yaml -rw-r--r-- 1 shubham shubham 734 May 15 13:26 sample-category.chartserviceversion.yaml drwxr-xr-x 2 shubham shubham 4096 May 15 13:26 pod-delete/ $ ls -ltr sample-category/pod-delete total 28 -rw-r--r-- 1 shubham shubham 791 May 15 13:26 rbac.yaml -rw-r--r-- 1 shubham shubham 734 May 15 13:26 pod-delete.chartserviceversion.yaml -rw-r--r-- 1 shubham shubham 792 May 15 13:26 experiment.yaml -rw-r--r-- 1 shubham shubham 1777 May 15 13:26 pod-delete-k8s-job.yml ## this file will be created in case of litmus-go -rw-r--r-- 1 shubham shubham 4533 May 15 13:26 pod-delete.go ## this file will be created in case of litmus-ansible -rw-r--r-- 1 shubham shubham 4533 May 15 13:26 pod-delete-ansible-logic.yml -rw-r--r-- 1 shubham shubham 813 May 15 13:26 engine.yaml
Proceed with construction of business logic inside the pod-delete.go file in litmus-go or pod-delete-ansible-logic.yml in litmus-ansible, by making the appropriate modifications listed below to achieve the desired effect:
- entry & exit criteria checks for the experiment
- helper utils in either pkg or new base chaos libraries
Update the experiment.yaml with the right chaos params in the
spec.definition.envwith their default values.
Update the chaoslib/litmus/pod-delete/pod-delete.go chaoslib to achieve the desired effect or reuse the existing chaoslib.
Create an experiment README explaining, briefly, the what, why & how of the experiment to aid users of this experiment.
Steps to Test Experiment
- Run the pod-delete-k8s-job.yml with the desired values in the ENV and appropriate chaosServiceAccount using a custom dev image instead of litmuschaos/litmus-go (say, ksatchit/litmus-go) that packages the business logic.
(Optional) Once the experiment has been validated using the above step, it can also be tested against the standard chaos workflow using the experiment.yaml. This involves:
- Launching the Chaos-Operator
- Creating the ChaosExperiment CR on the cluster (use the same custom dev image used in the above step)
- Creating the ChaosEngine to execute the above ChaosExperiment
- Verifying the experiment status via ChaosResult. refer Litmus Docs for more details on this procedure.
Steps to Include the Chaos Charts/Experiments into the ChartHub
- Send a PR to the litmus-go/litmus-ansible repo with the modified experiment files i.e, pod-memory-hog
- Send a PR to the chaos-charts repo with the modified experiment CR, experiment chartserviceversion, chaos chart (category-level) chartserviceversion & package (if applicable) YAMLs i.e, kubelet service kill
Litmus SDK will help developers & SREs to create their custom chaos experiment on demand. It will generate the templates, pre/post chaos checks, chaoslib (which can be modified according to use-case).
Are you an SRE or a Kubernetes enthusiast? Does Chaos Engineering excite you? Join Our Community #litmus channel in Kubernetes Slack
Contribute to LitmusChaos and share your feedback on Github
If you like LitmusChaos, become one of the many stargazers here
Litmus helps Kubernetes SREs and developers practice chaos engineering in a Kubernetes native way. Chaos experiments are published at the ChaosHub (https://hub.litmuschaos.io). Community notes is at https://hackmd.io/a4Zu_sH4TZGeih-xCimi3Q
Cloud-Native Chaos Engineering
Litmus is a toolset to do cloud-native chaos engineering. Litmus provides tools to orchestrate chaos on Kubernetes to help SREs find weaknesses in their deployments. SREs use Litmus to run chaos experiments initially in the staging environment and eventually in production to find bugs, vulnerabilities. Fixing the weaknesses leads to increased resilience of the system.
Litmus takes a cloud-native approach to create, manage and monitor chaos. Chaos is orchestrated using the following Kubernetes Custom Resource Definitions (CRDs):
- ChaosEngine: A resource to link a Kubernetes application or Kubernetes node to a ChaosExperiment. ChaosEngine is watched by Litmus' Chaos-Operator which then invokes Chaos-Experiments
- ChaosExperiment: A resource to group the configuration parameters of a chaos experiment. ChaosExperiment CRs are created by the operator when experiments are invoked by ChaosEngine.
- ChaosResult: A resource to hold the results of a chaos-experiment. The Chaos-exporter reads the…