In this blog, I will talk about the generation of custom/new chaos experiment with the help of Litmus SDK . Before jumping in, let's do a quick recap on Litmus. Litmus is a framework for practicing Chaos Engineering in cloud-native environments. Litmus provides a chaos-operator, a large set of chaos experiments in its HUB, detailed documentaion, quick demo, and a friendly community.
What is Litmus SDK?
The Litmus SDK provides a simple way to bootstrap your experiment and helps create the aforementioned artifacts in the appropriate directory (i.e., as per the chaos-category) based on an attributes file provided as input by the chart-developer. The scaffolded files consist of placeholders which can then be filled as desired.
It generates the custom chaos experiments with some default Pre & Post Chaos Checks (AUT & Auxiliary Applications status checks). It can use the existing chaoslib (present inside
/chaoslib directory), if available else It will create a new chaoslib inside the corresponding directory.
Life Cycle of a Chaos Experiment
Each Chaos Experiment is divided into six main sections:
- Prepare: Prepare section contains the initialization steps (get ENVs), Updation of chaos result to marked the beginning of the chaos experiment.
- PreChaosCheck: PreChaosCheck contains some default checks (AUT & Auxiliary Application status check) and custom checks, which may vary on the basis of experiment i.e, liveness check, data persistence check, etc.
- ChaosInject: ChaosInject invoke the actual chaos, It contain the main business logic.
- CleanUp: CleanUp contains the steps to remove the helper/external pod, if any
- PostChaosCheck: PostChaosCheck contains the similar steps as the PreCheckCheck section have. It used to ensure the resiliency after chaos injection.
- Summary Summary updates the verdict(Pass/Fail) inside the chaos result and FailStep if the experiment fails.
- go should be is available & the GOPATH env is configured appropriately
- python3 is available (sudo apt-get install python3.6)
- jinja2 & pyYaml python packages are available (sudo apt-get install python3-pip, pip install jinja2, pip install pyYaml)
Steps to Generate Experiment via Litmus SDK
- Clone the litmus-go repository for go experiments and litmus-ansible for the ansible experiments and navigate to the contribute/developer-guide folder.
## for litmus-go $ git clone https://github.com/litmuschaos/litmus-go.git $ cd litmus-go/contribute/developer-guide
## for litmus-ansible $ git clone https://github.com/litmuschaos/litmus-ansible.git $ cd litmus-ansible/contribute/developer_guide
- Populate the attributes.yaml with details of the chaos experiment (or chart). Use the attributes.yaml.sample as reference.
As an example, let us consider an experiment to kill one of the replicas of an Nginx deployment. The
attributes.yaml can be constructed like this:
$ cat attributes.yaml --- name: "sample-pod-delete" version: "0.1.0" category: "sample-category" repository: "https://github.com/litmuschaos/litmus-go/tree/master/sample-category/pod-delete" community: "https://kubernetes.slack.com/messages/CNXNB0ZTN" description: "kills nginx pods in a random manner" keywords: - "pods" - "kubernetes" - "sample-category" - "nginx" scope: "Namespaced" auxiliaryappcheck: false permissions: - apigroups: - "" - "batch" - "litmuschaos.io" resources: - "jobs" - "pods" - "chaosengines" - "chaosexperiments" - "chaosresults" verbs: - "create" - "list" - "get" - "update" - "patch" - "delete" maturity: "alpha" maintainers: - name: "ksatchit" email: "email@example.com" provider: name: "Mayadata" minkubernetesversion: "1.12.0" references: - name: Documentation url: "https://docs.litmuschaos.io/docs/getstarted/"
- Run the following command to generate the necessary artifacts for submitting the sample-category chaos chart with pod-delete experiment.
## litmus-go $ ./litmus-sdk generate <generate-type> -f=attributes.yaml
Note: Replace the
<generate-type>placeholder with the appropriate value based on the usecase:
chart:Just the chaos-chart metadata, i.e., chartserviceversion yaml
experiment:Chaos experiment artifacts belonging to a an existing OR new chart.
- provide the path of the
attribute.yamlmanifest in the
## litmus-ansible $ python3 generate_chart.py --attributes_file=attributes.yaml --generate_type=experiment
Note: In the
--generate_typeflag, select the appropriate type of manifests to be generated, where,
chart:Just the chaos-chart metadata, i.e, chartserviceversion yaml
experiment:Chaos experiment artifacts belonging to an existing or new chart.
- provide the path of the
attribute.yamlmanifest in the
Verify the Generated Files
$ cd /experiments $ ls -ltr total 8 drwxr-xr-x 3 shubham shubham 4096 May 15 12:02 generic/ drwxr-xr-x 3 shubham shubham 4096 May 15 13:26 sample-category/ $ ls -ltr sample-category/ total 12 -rw-r--r-- 1 shubham shubham 41 May 15 13:26 sample-category.package.yaml -rw-r--r-- 1 shubham shubham 734 May 15 13:26 sample-category.chartserviceversion.yaml drwxr-xr-x 2 shubham shubham 4096 May 15 13:26 sample-pod-delete/ $ ls -ltr sample-category/sample-pod-delete total 28 -rw-r--r-- 1 shubham shubham 791 May 15 13:26 rbac.yaml -rw-r--r-- 1 shubham shubham 734 May 15 13:26 sample-pod-delete.chartserviceversion.yaml -rw-r--r-- 1 shubham shubham 792 May 15 13:26 experiment.yaml -rw-r--r-- 1 shubham shubham 813 May 15 13:26 engine.yaml ## this file will be created in case of litmus-go --rw-r--r-- 1 shubham shubham 4096 May 15 13:26 test -rw-r--r-- 1 shubham shubham 4533 May 15 13:26 sample-pod-delete.go ## this file will be created in case of litmus-ansible -rw-r--r-- 1 shubham shubham 1777 May 15 13:26 pod-delete-k8s-job.yml -rw-r--r-- 1 shubham shubham 4533 May 15 13:26 pod-delete-ansible-logic.yml $ ls -ltr sample-category/sample-pod-delete/test total 4 -rw-r--r-- 1 shubham shubham 1039 May 15 13:26 test.yaml
Proceed with construction of business logic inside the
pod-delete.gofile in litmus-go or pod-delete-ansible-logic.yml in litmus-ansible, by making the appropriate modifications listed below to achieve the desired effect:
- entry & exit criteria checks for the experiment
- helper utils in either pkg or new base chaos libraries
experiment.yamlwith the right chaos params in the
spec.definition.envwith their default values.
chaoslib/litmus/sample-pod-delete/lib/pod-delete.gochaoslib to achieve the desired effect or reuse the existing chaoslib.
Create an experiment README explaining, briefly, the what, why & how of the experiment to aid users of this experiment.
Steps to Test Experiment
Follow the Litmus SDK: DevTest your chaos experiments with Okteto blog to test the generated experiment.
Steps to Include the Chaos Charts/Experiments into the ChartHub
- Send a PR to the
litmus-go/litmus-gorepo with the modified experiment files i.e, pod-memory-hog
- Send a PR to the chaos-charts repo with the modified experiment CR, experiment chartserviceversion, chaos chart (category-level) chartserviceversion & package (if applicable) YAMLs i.e, kubelet service kill
Litmus SDK will help developers & SREs to create their custom chaos experiment on demand. It will generate the templates, pre/post chaos checks, chaoslib (which can be modified according to use-case).
Are you an SRE or a Kubernetes enthusiast? Does Chaos Engineering excite you? Join Our Community #litmus channel in Kubernetes Slack Litmus is a toolset to do cloud-native chaos engineering. Litmus provides tools to orchestrate chaos on Kubernetes to help SREs find weaknesses in their deployments. SREs use Litmus to run chaos experiments initially in the staging environment and eventually in production to find bugs, vulnerabilities. Fixing the weaknesses leads to increased resilience of the system. Litmus takes a cloud-native approach to create, manage and monitor chaos. Chaos is orchestrated using the following Kubernetes Custom Resource Definitions (CRDs):
Contribute to LitmusChaos and share your feedback on Github
If you like LitmusChaos, become one of the many stargazers here
Litmus helps Kubernetes SREs and developers practice chaos engineering in a Kubernetes native way. Chaos experiments are published at the ChaosHub (https://hub.litmuschaos.io). Community notes is at https://hackmd.io/a4Zu_sH4TZGeih-xCimi3Q
Cloud-Native Chaos Engineering
Read this in other languages.
Litmus is a toolset to do cloud-native chaos engineering. Litmus provides tools to orchestrate chaos on Kubernetes to help SREs find weaknesses in their deployments. SREs use Litmus to run chaos experiments initially in the staging environment and eventually in production to find bugs, vulnerabilities. Fixing the weaknesses leads to increased resilience of the system.
Litmus takes a cloud-native approach to create, manage and monitor chaos. Chaos is orchestrated using the following Kubernetes Custom Resource Definitions (CRDs):