In this article we will describe how to perform chaos testing using Litmus (a popular chaos testing tool).
There are 4 major steps for running any chaos test.
- The first step is defining a steady state, which means defining how an ideal system would look like. For a web application, the home page is returning a success response, for a web service this would mean that it is healthy or it is returning a success for the health endpoint.
- The second step is actually introducing chaos such as simulating a failure such as a network bottleneck / disk fill etc.
- The third step is to verify a steady state, i.e, to check if the system is still working as expected.
- The fourth step which is the most important step (more important if you are running in production) is that we roll back the chaos that we caused.
To learn more about chaos testing, first we need to have an application under test, for this demo, we are going to have a BookInfo application deployed on a single node Kubernetes cluster. Along with it, we have Prometheus, Grafana, Jaeger & Kiali setup, along with Istio service mesh.
0.1) Setup Kubernetes Cluster: Get your Kubernetes cluster up and running with Docker1 as container runtime. To keep it simple install Docker for Desktop and also start Kubernetes along with it. However, you can also use Minikube, k3d, kind to set up local k8s clusters.
0.2) Setup Monitoring: Next to setup Istio along with all monitoring tools such as Prometheus, Grafana, Jaeger & Kiali
istioctl install --set profile=demo -y
# set ISTIO_RELEASE_URL to specific istio release version export ISTIO_RELEASE_URL=https://raw.githubusercontent.com/istio/istio/release-1.11/ kubectl apply -f $ISTIO_RELEASE_URL/samples/addons/jaeger.yaml kubectl apply -f ISTIO_RELEASE_URL/samples/addons/prometheus.yaml kubectl apply -f ISTIO_RELEASE_URL/samples/addons/grafana.yaml kubectl apply -f $ISTIO_RELEASE_URL/samples/addons/kiali.yaml
0.3) Install Bookinfo application with Istio service mesh enabled and envoy sidecar installed.
Download BookInfo yaml from Istio website: https://raw.githubusercontent.com/istio/istio/release-1.11/samples/bookinfo/platform/kube/bookinfo.yaml
Install Bookinfo with envoy proxy injected as sidecar container into default namespace:
istioctl kube-inject -f book-info.yaml | kubectl apply -f -
0.4) Verify applications is running and deployed with envoy proxy as sidecar.
Do port forwarding for the
productpage service and check in your browser.
kubectl port-forward service/productpage 9080:9080
If you open
localhost:9080 in your web browser, you should see something like this
So now I have my application up and running inside the k8s cluster.
Steady state for the bookinfo application is that the product page should keep rendering without any issues. Means
http://localhost:9080/productpage?u=normal should return
200 http status code under continuous load.
To check my steady state condition let me first generate continuous load on my bookinfo application using a command line tool called hey and monitor it.
hey -c 2 -z 200s http://localhost:9080/productpage
Above command generates continuous load on the product page for 200 seconds with 2 concurrent workers.
Here is a quick view of the Kiali dashboard showing all pods healthy and review service responding in a 100-200 ms timeframe, which internally calling rating service responding in avg 50-60 ms.
All set, now time to introduce chaos in the system. Let's first understand Litmus core concepts before we jump into execution.
2.1) Install Litmus: First step is to install an operator (Litmus Operator) into the Kubernetes cluster where we like to introduce chaos. Limus operator adds 3 custom resource definitions related to Litmus chaos into k8s cluster. You can also use Helm charts to install Litmus operators and its web UI. However, for simplicity I am going with direct yaml and install only operator.
$kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-v2.2.0.yaml
2.2) Setup Experiment: After that we need to add a specific experiment in the namespace where we like to introduce chaos. List of all the available chaos are listed here. Lets add a network deploy chaos experiment into the default namespace where we have our bookinfo application installed.
$kubectl apply -f https://hub.litmuschaos.io/api/chaos/2.2.0?file=charts/generic/pod-network-latency/experiment.yaml
2.3) Apply Permissions: Now we need to give permission using RBAC to allow chaos experiments to run.
$kubectl apply -f https://hub.litmuschaos.io/api/chaos/2.2.0?file=charts/generic/pod-network-latency/rbac.yaml
2.4) Run Chaos: Using ChaosEngine custom resource definition, we inject network delay chaos. Please look at the following yaml
network-delay-engine.yaml of kind ChaosEngine for introducing network delay of 2 sec for ratings deployment for about 100 seconds affecting all pods under deployment. Delay in ratings service response is going to indirectly delay review services and which indirectly adds delay to product page.
apiVersion: litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: name: bookinfo-network-delay namespace: default spec: jobCleanUpPolicy: 'retain' # It can be delete/retain annotationCheck: 'false' engineState: 'active' monitoring: false appinfo: appns: 'default' applabel: 'app=ratings' # application label matching appkind: 'deployment' # k8s object type chaosServiceAccount: pod-network-latency-sa experiments: - name: pod-network-latency spec: components: env: - name: NETWORK_INTERFACE value: 'eth0' # default interface used by pod - name: NETWORK_LATENCY value: '2000' # delay in milliseconds - name: TOTAL_CHAOS_DURATION value: '100' # chaos duration in seconds - name: PODS_AFFECTED_PERC value: '100' # effect # of pods in percentage
Please check comments in above yaml to learn more about different configurations. Details about each configuration can be found in documentation provided by Litmus toolkit here.
$kubectl apply -f network-delay-engine.yaml
Here is pod watch of default namespace and notice
pod-network-latency-helper-hhpofr pods doing the job of introducing network delay for rating service.
2.5) Observe Result: Use Kubernetes describe command to see output of the chaos run we had in previous steps. Lets first notice increased time in review service response time on Kiali.
ChaosResult to see the result in Litmus custom objects description.
Observe events using describe on
chaosengine custom resource
Observe events using describe on
chaosresult custom resource
Repeat chaos testing by increasing delay to 6 seconds (6000 ms) and repeating steps 2.4 and 2.5. Change
During the chaos test time we continuously accessed the system and observed 200 responses for the product page.
Observed 2 sec delay in response time on review service on Kiali dashboard.
In our case of network delay, since the chaos duration was set to 100 sec. It stopped automatically after 100 sec. So nothing to be done. Just observe that our system is back to normal.
On Kiali dashboard we see returning it to normal with review response time less than 100 ms timeframe and rating response time in 50-60 ms timeframe.
Can I use the Litmus tool with any other container runtime like contrainerd?
Yes, Steps in this article are keeping Docker as container runtime, however, if you have other runtimes like containerd, please read configuration on Chaos website for different configurations needed to run chaos experiments.
Where can I find a list of all chaos experiments available?
Litmus has some predefined chaos experiments available which can be found here, but it does not limit us to define our own experiments and run them in our own environments.
How to debug issues if any?
While running chaos using
ChoseEngine CRD, use following flag
jobCleanUpPolicy: 'retain' to keep pods in complete state (and not to be deleted after chaos run) which provides ability to look at logs of the pods.
Above commands and code checked into the public repository on Github.
Watch all of above in action in XConf 2021 online conference talk
Learn more with hands-on tutorial on Litmus site
Online hands-on with Litmus tool on KataCode