DEV Community: Shubham Chaudhary

Declarative Approach to Chaos Hypothesis using Litmus Probes

Shubham Chaudhary — Tue, 15 Sep 2020 15:47:00 +0000

Introduction

One of the foremost principles of chaos engineering is the need to build a hypothesis around steady-state or desired system behavior and be able to automate the process of gauging, analyzing & reporting it over the course of an experiment. This is indispensable to the persona executing these experiments in order to gain valuable insights on the resilience of their applications & deployment infrastructure. In other words, it is imperative for chaos engineering frameworks/toolsets to provide the ability to define their hypotheses & validate them against the same.

Further, in the world of cloud-native chaos engineering, the hypothesis definition is expected to be declarative in nature, much in the same way the chaos intent is. Another very interesting factor here is the diversity in the way the steady-state is defined. Most of the time, they are systemic behavior patterns: metrics around error rates, latency percentiles, etc., and at other times liveness of a crucial downstream service, state (values within) of a database, or even Kubernetes resources. Typically, they can be mapped to operational SLOs (Service Level Objectives) that have been agreed upon. These are the parameters & conditions that verify that the system “works” (as a discerning reader you might have understood the subtle difference here b/w chaos experimentation over standard failure testing: verification that the system works over validating how it works. Anyway, I digress..).

So, to summarize our understanding (courtesy: standard chaos principles & learnings from the awesome chaos community):

Hypotheses are crucial to chaos experiments
Chaos frameworks need to burn this into experiments (remember automated chaos!)
Hypothesis definitions should be declarative in nature & thereby easily tuned & scaled
Accommodate the diverse nature of hypotheses definitions, with the right schema & architecture.

Hypothesis in LitmusChaos Experiments

As a community, we identified the need for “steady-state” validations pretty early & introduced the pre-chaos & post-chaos checks in the experiments, with appropriate Kubernetes events generated in the respective stages. In the “generic” category of experiments, this corresponds to a vanilla check that verifies if all pods bearing the specified application label in a given namespace are in the “Running” state (the same check also verifies that all the containers within this pod have the right status).

In application-specific chaos experiments (such as the Kafka-broker-failure or Cassandra-node-failure) health checks native to the applications have been included in the experiment business logic. There are also examples and documentation resources available for users to add custom checks to implement a hypothesis within a newly bootstrapped/scaffolded chaos experiment using the Litmus SDK. In some ways, this ticks the points (a) & (b) in our summarization.

However, the aforementioned approach also places the onus on the chaos developers/users to create and maintain new experiments or modify existing ones with additional logic every time the hypothesis changes (for example, new criteria might be added to indicate steady-state).

This ties in directly to ( c), i.e., we needed a declarative way to specify checks to be carried out, while reusing the standard (generic) chaos experiments from the ChaosHub, and also accommodating standard interfaces to describe these checks (d).

The aforementioned reasons prompted us to come up with the “Litmus Probes” feature in the 1.7.0 release!

In this blog, I will talk about how you can use Litmus Probes to build custom checks against your application/infra in a reasonably declarative fashion, i.e., by providing additional inputs in the ChaosEngine YAML manifest.

Litmus Probes: Types & Characteristics

Litmus probes can be defined as “pluggable” checks you can define within the ChaosEngine for any chaos experiment. The experiment pods execute these checks based on the mode they are defined in & factor their success as necessary conditions in determining the verdict of the experiment (along with the standard “in-built” checks).

Litmus currently supports four types of probes:

httpProbe: To query health/downstream URIs
cmdProbe: To execute any user-desired health-check function implemented as a shell command
k8sProbe: To perform CRUD operations against native & custom Kubernetes resources
promProbe: To validate the SLO by prometheus endpoint

These probes can be used in isolation or in several combinations to achieve the desired checks. As we will see in subsequent sections, while the httpProbe, promProbe & k8sProbe are fully declarative in the way they are conceived, the cmdProbe expects the user to provide a shell command to implement checks that are highly specific to the application use case. Does it sound similar to the “command” or “shell” module in the ansible world? The intent is similar too!

The probes can be set up to run in different modes:

SoT: Executed at the Start of Test as a pre-chaos check
EoT: Executed at the End of Test as a post-chaos check
Edge: Executed both, before and after the chaos
Continuous: The probe is executed continuously, with a specified polling interval during the chaos injection.
OnChaos: The probe is executed continuously, with a specified polling interval strictly for chaos duration of chaos.

All probes share some common attributes:

probeTimeout: Represents the time limit for the probe to execute the check specified and return the expected data.
retry: The number of times a check is re-run upon failure in the first attempt, before declaring the probe status as failed.
interval: The period between subsequent retries
probePollingInterval: The time interval for which continuous probe should be sleep after each iteration.
initialDelaySeconds: Represents the initial waiting time interval for the probes.

Let us take a look at the different probe categories in some more detail.

httpProbe

The httpProbe allows developers to specify a URL which the experiment uses to gauge health/service availability (or other custom conditions) as part of the entry/exit criteria. The received status code is mapped against an expected status. It supports http Get and Post methods.

In HTTP Get method, it sends an HTTP Get request to the provided URL and matches the response code based on the given criteria(==, !=, oneOf).

In the HTTP Post method, it sends an HTTP Post request to the provided URL. The HTTP body can be provided in the body field. In the case of a complex Post request in which the body spans multiple lines, the bodyPath attribute can be used to provide the path to a file consisting of the same. This file can be made available to the experiment pod via a ConfigMap resource, with the ConfigMap name being defined in the ChaosEngine OR the ChaosExperiment CR.
It can be defined at .spec.experiments[].spec.probe inside ChaosEngine.

NOTE: body and bodyPath are mutually exclusive.

probe:
- name: "check-frontend-access-url"
  type: "httpProbe"
  httpProbe/inputs:
    url: "<url>"
    insecureSkipVerify: false
    method:
      get: 
        criteria: == # supports == & != and oneof operations
        responseCode: "<response code>"
  mode: "Continuous"
  runProperties:
    probeTimeout: 5
    interval: 5
    retry: 1
    probePollingInterval: 2

The httpProbe is better used in the Continuous mode of operation as a parallel liveness indicator of a target or downstream application. It uses the probePollingInterval property to specify the polling interval for the access checks.

NOTE: insecureSkipVerify can be set to true to skip the certificate checks.

cmdProbe

The cmdProbe allows developers to run shell commands and match the resulting output as part of the entry/exit criteria. The intent behind this probe was to allow users to implement a non-standard & imperative way of expressing their hypothesis. For example, the cmdProbe enables you to check for specific data within a database, parse the value out of a JSON blob being dumped into a certain path, or check for the existence of a particular string in the service logs.

In order to enable this behavior, the probe supports an inline mode in which the command is run from within the experiment image as well as a source mode, where the command execution is carried out from within a new pod whose image can be specified. While inline is preferred for simple shell commands, source mode can be used when application-specific binaries are required. The cmdProbe can be defined at .spec.experiments[].spec.probe the path inside the ChaosEngine.

probe:
- name: "check-database-integrity"
  type: "cmdProbe"
  cmdProbe/inputs:
    command: "<command>"
    comparator:
      type: "string" # supports: string, int, float
      criteria: "contains" #supports >=,<=,>,<,==,!= for int and contains,equal,notEqual,matches,notMatches for string values
      value: "<value-for-criteria-match>"
    source:
      image: "<repo>/<tag>" # it can be any image
  mode: "Edge"
  runProperties:
    probeTimeout: 5
    interval: 5
    retry: 1
    initialDelaySeconds: 5

k8sProbe

With the proliferation of custom resources & operators, especially in the case of stateful applications, the steady-state is manifested as status parameters/flags within Kubernetes resources. k8sProbe addresses verification of the desired resource state by allowing users to define the Kubernetes GVR(group-version-resource) with appropriate filters (field selectors/label selectors). The experiment makes use of the Kubernetes Dynamic Client to achieve this. The k8sProbe can be defined at .spec.experiments[].spec.probe the path inside the ChaosEngine.

It supports the following CRUD operations which can be defined at probe.operation.

create: It creates kubernetes resource based on the data provided inside probe.data field.
delete: It deletes matching kubernetes resource via GVR and filters (field selectors/label selectors).
present: It checks for the presence of kubernetes resource based on GVR and filters (field selectors/labelselectors).
absent: It checks for the absence of kubernetes resource based on GVR and filters (field selectors/labelselectors).

probe:
- name: "check-app-cluster-cr-status"
  type: "k8sProbe"
  k8sProbe/inputs:
    command:
      group: "<appGroup>"
      version: "<appVersion>"
      resource: "<appResource>"
      namespace: "default"
      fieldSelector: "metadata.name=<appResourceName>,status.phase=Running"
      labelSelector: "<app-labels>"
  operation: "present" # it can be present, absent, create, delete
  mode: "EOT"
  runProperties:
    probeTimeout: 5
    interval: 5
    retry: 1

PromProbe

PromProbe allows users to run Prometheus queries and match the resulting output against specific conditions. The intent behind this probe is to allow users to define metrics-based SLOs in a declarative way and determine the experiment verdict based on its success. The probe runs the query on a Prometheus server defined by the endpoint, and checks whether the output satisfies the specified criteria.

The promql query can be provided in the query field. In the case of complex queries that span multiple lines, the queryPath attribute can be used to provide the link to a file consisting of the query. This file can be made available in the experiment pod via a ConfigMap resource, with the ConfigMap being passed in the ChaosEngine OR the ChaosExperiment CR.

NOTE: query and queryPath are mutually exclusive

probe:
  - name: 'check-probe-success'
    type: 'promProbe'
    promProbe/inputs:
      endpoint: '<prometheus-endpoint>'
      query: '<promql-query>'
      comparator:
        criteria: '==' #supports >=,<=,>,<,==,!= comparision
        value: '<value-for-criteria-match>'
    mode: 'Edge'
    runProperties:
      probeTimeout: 5
      interval: 5
      retry: 1

Probe Status & Deriving Inferences

The Litmus chaos experiments run the probes defined in the ChaosEngine and update their stage-wise success in the ChaosResult custom resource, with details including the overall probeSuccessPercentage (a ratio of successful checks v/s total probes) and failure step, where applicable. The success of a probe is dependent on whether the expected status/results are met and also on whether it is successful in all the experiment phases defined by the probe’s execution mode. For example, probes that are executed in “Edge” mode, need the checks to be successful both during the pre-chaos & post-chaos phases to be declared as successful.

The pass criteria for the experiment is a logical AND function of all the probes defined in the ChaosEngine as well as inbuilt entry/exit criteria. Failure of either indicates a failed hypothesis and is deemed experiment failure. And an opportunity to fix the underlying problem!

Provided below is a chaosresult snippet containing the probe status for a mixed-probe chaosengine.

Name:         app-pod-delete
Namespace:    test
Labels:       name=app-pod-delete
Annotations:  <none>
API Version:  litmuschaos.io/v1alpha1
Kind:         ChaosResult
Metadata:
  Creation Timestamp:  2020-08-29T08:28:26Z
  Generation:          36
  Resource Version:    50239
  Self Link:           /apis/litmuschaos.io/v1alpha1/namespaces/test/chaosresults/app-pod-delete
  UID:                 b9e3638a-b7a4-4b93-bfea-bd143d91a5e8
Spec:
  Engine:      probe
  Experiment:  pod-delete
Status:
  Experimentstatus:
    Fail Step:                 N/A
    Phase:                     Completed
    Probe Success Percentage:  100
    Verdict:                   Pass

  Probe Status:
    Name:  check-frontend-access-url
    Status:
      Continuous:  Passed 👍 
    Type:          HTTPProbe

    Name:          check-app-cluster-cr-status
    Status:
      Post Chaos:  Passed 👍 #EoT
    Type:          K8sProbe

    Name:          check-database-integrity
    Status:
      Post Chaos:  Passed 👍 #Edge
      Pre Chaos:   Passed 👍 
    Type:          CmdProbe
Events:
  Type    Reason   Age   From                     Message
  ----    ------   ----  ----                     -------
  Normal  Summary  7s    pod-delete-0s2jt6-s4rdx  pod-delete experiment has been Passed

Conclusion

The probes are an effective mechanism to burn-in hypotheses into chaos experiments and arm them with more meaning & context. In some ways, they address the “opacity” today with respect to what constitutes an experiment pass or failure. Probes are also useful analytics aids indicating resiliency over a period of time. For example, the probeSuccessPercentage for a given experiment (with a set of mandatory probes) against a specific application can be tracked over time to gauge the progress being made in the feature & deploy practices.

One of the questions we got from the community as we set out to build this feature is whether the probes are a replacement for application-specific chaos experiments. The answer is “Not really”. While the probes do enable reuse of the generic experiments and give it an app context, they are not intended to perform deep application-level verification, much less inject app-specific faults. Take the example of the Kafka chaos experiments on the hub - for example. The focus here is on identifying a certain type of app replica to inject chaos on & use test downstream apps (to simulate the producer/consumer) for validation purposes. The probes can be useful in enhancing these experiments and therefore work more as aids than replacements.

Hope this feature helps you practice Chaos Engineering in an even better way. Do try it & let us know what you think.

Are you an SRE or a Kubernetes enthusiast? Does Chaos Engineering excite you? Join Our Community #litmus channel in Kubernetes Slack
Contribute to LitmusChaos and share your feedback on Github
If you like LitmusChaos, become one of the many stargazers here

litmuschaos / litmus

Litmus helps SREs and developers practice chaos engineering in a Cloud-native way. Chaos experiments are published at the ChaosHub (https://hub.litmuschaos.io). Community notes is at https://hackmd.io/a4Zu_sH4TZGeih-xCimi3Q

LitmusChaos

Open Source Chaos Engineering Platform

Read this in other languages.

🇰🇷 🇨🇳 🇧🇷 🇮🇳

Overview

LitmusChaos is an open source Chaos Engineering platform that enables teams to identify weaknesses & potential outages in infrastructures by inducing chaos tests in a controlled way. Developers & SREs can practice Chaos Engineering with LitmusChaos as it is easy to use, based on modern Chaos Engineering principles & community collaborated. It is 100% open source & a CNCF project.

LitmusChaos takes a cloud-native approach to create, manage and monitor chaos. The platform itself runs as a set of microservices and uses Kubernetes custom resources (CRs) to define the chaos intent, as well as the steady state hypothesis.

At a high-level, Litmus comprises of:

Chaos Control Plane: A centralized chaos management tool called chaos-center, which helps construct, schedule and visualize Litmus chaos workflows
Chaos Execution Plane Services: Made up of a…

View on GitHub

Cassandra Chaos Engineering With Litmus

Shubham Chaudhary — Sun, 19 Jul 2020 09:22:00 +0000

In this blog, I will talk about chaos engineering on Cassandra with Litmus. Before jumping in, let's do a quick recap on Litmus. Litmus is a framework for practicing Chaos Engineering in cloud-native environments. Litmus provides a chaos-operator, a large set of chaos experiments in its hub, detailed documentation, quick Demo, and a friendly community.

What Is Cassandra?

Apache Cassandra is a free and open-source, distributed, wide column store, NoSQL database management system designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure. Cassandra offers robust support for clusters spanning multiple datacenters with asynchronous masterless replication allowing low latency operations for all clients. Cassandra's support for replicating across multiple datacenters is best-in-class, providing lower latency for your users and the peace of mind of knowing that you can survive regional outages.

Cassandra is using Consistent Hashing, which is a distributed hashing scheme that operates independently of the number of servers or objects in a distributed hash table by assigning them a position on an abstract circle, or hash ring. This allows servers and objects to scale without affecting the overall system. Cassandra uniformly distributes the load over the Cassandra ring and re-distribute the load when the Cassandra Statefulset scales up/down.

With Kubernetes popularity skyrocketing and the adoption of Apache Cassandra growing as a NoSQL database well-suited to matching the high availability and scalability needs of cloud-based applications, it should be no surprise that more developers are looking to run Cassandra databases on Kubernetes. However, many devs are finding that doing so is relatively simple to get going with, but considerably more challenging to execute at a high level. We will discuss the installation part later in this blog.

On the positive side, Kubernetes helpfully offers StatefulSets — workload API objects that can be used to manage stateful applications. StatefulSets provide the requisite components to establish stable and unique network identifiers, stable persistent storage, smooth and ordering deployment and scaling (as well as deletion and termination), and automated rolling updates.

Why do we need chaos engineering on Cassandra?

As Cassandra is using consistent hashing and maintaining the Cassandra ring to distribute the load uniformly. If the number of replicas will scale up/down, the load will be redistributed. But we always have a few questions in our mind:

Does load redistribution always happen with 0 probability of failure in scale-up/down?
How will it behave if the one/multiple replicas of the Cassandra Statefulset are killed?
Is it resilient even if the rate of replica deletion is very high?

In the age of data evolution, data is very important and everyone is looking for a setup where the probability of downtime is least as possible. Here the chaos engineering comes into play. It will help you to find out all the corner cases of failure before they actually happen and if you really believe in "Prevention is better than cure" then the best real-life example award goes to chaos engineering. It can be used to test the resiliency of the Cassandra stateful set. As of now, we have a Cassandra-pod-delete experiment to test the case where one of the replicas of Cassandra Statefulset is deleted. More use-cases/experiments will be added soon.

Chaos On Cassandra: Pod-Delete Experiment

Now that we know the basics of a Cassandra statefulset on Kubernetes, let us execute a chaos experiment to kill one of the replicas of Cassandra while the load is distributed on all the replicas over the Cassandra ring & verify whether the load is redistributed. This example intends to introduce the user to the steps involved in carrying out a chaos experiment using Litmus.

Pre-Requisites

A (preferably) multi-node Kubernetes cluster. Ensure you are in the Kubernetes-admin context to setup RBAC for the various components involved.

Chaos Experiment Approach

The following steps are performed automatically upon execution of the Chaos Experiment:

An external liveness pod will be created, which will ensure the liveness of Cassandra statefulset during chaos execution. It is continuously running the liveness cycles of cqlsh commands. Liveness cycle stands for a set of cqlsh operations (create keyspace, create the table, insert data in the table, delete the tables, delete the keyspaces). It is running a webserver container as a side cart, which is exposing the status of the liveness cycle (cycleInProgress or cycleCompleted). The experiment looks for the status of the webserver service and cleans up the liveness deployment at the end of the cycle. In case of a timeout, the liveness container terminates ungracefully with an exception.
It will ensure that the load is distributed across all the replicas on the Cassandra ring before and after the chaos injection. It ensures that the load is re-distributed across all available replicas after every kill. In case if the load is not distributed on any of the available replicas, this check will be failed.
In its default mode, the experiment derives the random single/multiple replicas of Cassandra statefulset, performs a pod kill (delete), and checks the liveness and load distribution on the Cassandra ring across all replicas, which implies that the Cassandra statefulset remains alive and load redistribution takes place after every kill. If this is true, the experiment verdict is set to “pass,” indicating the current statefulset is tolerant to pod-failures. A terminated statefulset sets the verdict to “fail” and implies that the statefulset is not resilient enough, demanding a closer inspection.

Hypothesis

Upon killing the replica of Cassandra statefulset, the load will be re-distributed over the Cassandra ring.
After the deletion of the replica, the new replica will be created by the replica controller to maintain the desired count of the available replicas.

Preparing the Testbed

Setup the Cassandra Cluster

We are going to discuss two approaches to set up the Cassandra cluster quickly.

This approach will guide you to set up a Cassandra cluster in AWS EKS with OpenEBS as a storage orchestration tool. Follow the following tutorial to setup the cluster Setup Cassandra Cluster on EKS.
This approach will guide you to set up the Cassandra cluster in Minikube. We can follow the underlying steps to set up the Cassandra cluster.

Step-1: Creating a headless Service for Cassandra

Creating the headless service for Cassandra which is used for DNS lookups between Cassandra Pods and clients within your cluster:

root@demo:~# cat <<EOF > cassandra-service.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: cassandra
  name: cassandra
spec:
  clusterIP: None
  ports:
  - port: 9042
  selector:
    app: cassandra
EOF

root@demo:~# kubectl create -f cassandra-service.yaml -n cassandra
 service/cassandra created

Validation of Cassandra service: 

root@demo:~# kubectl get svc cassandra -n cassandra 
The response is:
     NAME        TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
     cassandra   ClusterIP   None         <none>        9042/TCP   45s

If you don't see a Service named Cassandra, that means creation failed.

Step-2: Using a StatefulSet to create a Cassandra ring

The StatefulSet manifest included below, creates a Cassandra ring that consists of three Pods.

Note: This example uses the default provisioner for Minikube. Please update the following StatefulSet for the cloud you are working with.

root@demo:~# cat <<EOF > cassandra-sts.yaml 

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cassandra
  labels:
    app: cassandra
spec:
  serviceName: cassandra
  replicas: 3
  selector:
    matchLabels:
      app: cassandra
  template:
    metadata:
      labels:
        app: cassandra
    spec:
      terminationGracePeriodSeconds: 1800
      containers:
      - name: cassandra
        image: gcr.io/google-samples/cassandra:v13
        imagePullPolicy: Always
        ports:
        - containerPort: 7000
          name: intra-node
        - containerPort: 7001
          name: tls-intra-node
        - containerPort: 7199
          name: jmx
        - containerPort: 9042
          name: cql
        resources:
          limits:
            cpu: "500m"
            memory: 1Gi
          requests:
            cpu: "500m"
            memory: 1Gi
        securityContext:
          capabilities:
            add:
              - IPC_LOCK
        lifecycle:
          preStop:
            exec:
              command: 
              - /bin/sh
              - -c
              - nodetool drain
        env:
          - name: MAX_HEAP_SIZE
            value: 512M
          - name: HEAP_NEWSIZE
            value: 100M
          - name: CASSANDRA_SEEDS
            value: "cassandra-0.cassandra.cassandra.svc.cluster.local"
          - name: CASSANDRA_CLUSTER_NAME
            value: "K8Demo"
          - name: CASSANDRA_DC
            value: "DC1-K8Demo"
          - name: CASSANDRA_RACK
            value: "Rack1-K8Demo"
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
        readinessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - /ready-probe.sh
          initialDelaySeconds: 15
          timeoutSeconds: 5
        # These volume mounts are persistent. They are like inline claims,
        # but not exactly because the names need to match exactly one of
        # the stateful pod volumes.
        volumeMounts:
        - name: cassandra-data
          mountPath: /cassandra_data
  # These are converted to volume claims by the controller
  # and mounted at the paths mentioned above.
  # do not use these in production until ssd GCEPersistentDisk or other ssd pd
  volumeClaimTemplates:
  - metadata:
      name: cassandra-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: fast
      resources:
        requests:
          storage: 1Gi
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: fast
provisioner: k8s.io/minikube-hostpath
parameters:
  type: pd-ssd
EOF

root@demo:~# kubectl create -f cassandra-sts.yaml -n cassandra
statefulset.apps/cassandra created
storageclass.storage.k8s.io/fast created

Validating the Cassandra StatefulSet
root@demo:~# kubectl get statefulset cassandra -n cassandra

The response should be similar to:
NAME        DESIRED   CURRENT   AGE
cassandra   3         0         13s

root@demo:~# kubectl get pods -l="app=cassandra" -n cassandra

It can take several minutes for all three Pods to deploy. Once they are deployed, the same command returns output similar to:

NAME          READY     STATUS    RESTARTS   AGE
cassandra-0   1/1       Running   0          10m
cassandra-1   1/1       Running   0          9m
cassandra-2   1/1       Running   0          8m

Step-3: Run the Cassandra nodetool inside the first Pod, to display the status of the ring.

kubectl exec -it cassandra-0 -- nodetool status

The response should look something like:

Datacenter: DC1-K8Demo
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  172.17.0.5  83.57 KiB  32           74.0%             e2dd09e6-d9d3-477e-96c5-45094c08db0f  Rack1-K8Demo
UN  172.17.0.4  101.04 KiB  32           58.8%             f89d6835-3a42-4419-92b3-0e62cae1479c  Rack1-K8Demo
UN  172.17.0.6  84.74 KiB  32           67.1%             a6a1e8c2-3dc5-4417-b1a0-26507af2aaad  Rack1-K8Demo

Setup the Litmus Infrastructure

Step-1: Install Litmus Chaos CRDs, Operator & RBAC

root@demo:~# kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-v2.11.0.yaml
namespace/litmus created
serviceaccount/litmus created
clusterrole.rbac.authorization.k8s.io/litmus created
clusterrolebinding.rbac.authorization.k8s.io/litmus created
deployment.apps/chaos-operator-ce created customresourcedefinition.apiextensions.k8s.io/chaosengines.litmuschaos.io created
customresourcedefinition.apiextensions.k8s.io/chaosexperiments.litmuschaos.io created
customresourcedefinition.apiextensions.k8s.io/chaosresults.litmuschaos.io created

kubectl apply -f https://litmuschaos.github.io/litmus/litmus-admin-rbac.yaml

Step-2: Create the Cassandra-Pod-Delete ChaosExperiment CR

root@demo:~# kubectl apply -f https://hub.litmuschaos.io/api/chaos/2.11.0?file=charts/cassandra/cassandra-pod-delete/experiment.yaml -n litmus
chaosexperiment.litmuschaos.io/cassandra-pod-delete created

Step-3: Annotate the Kafka statefulset for Chaos

root@demo:~# kubectl annotate sts/cassandra litmuschaos.io/chaos="true" -n cassandra 
statefulset.apps/cassandra annotated

Run the Chaos Experiment

Step-1: Construct the ChaosEngine for the Cassandra Pod Delete experiment:

root@demo:~# cat <<EOF > cassandra-chaos.yaml

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: cassandra-chaos
  namespace: default
spec:
  appinfo:
    appns: 'default'
    applabel: 'app=cassandra'
    appkind: 'statefulset'
  # It can be active/stop
  engineState: 'active'
  #ex. values: ns1:name=percona,ns2:run=nginx
  auxiliaryAppInfo: ''
  chaosServiceAccount: cassandra-pod-delete-sa
  experiments:
    - name: cassandra-pod-delete
      spec:
        components:
          env:
            # set chaos duration (in sec) as desired
            - name: TOTAL_CHAOS_DURATION
              value: '15'

            # set chaos interval (in sec) as desired
            - name: CHAOS_INTERVAL
              value: '15'

            # pod failures without '--force' & default terminationGracePeriodSeconds
            - name: FORCE
              value: 'false'

            # provide cassandra service name
            # default service: cassandra
            - name: CASSANDRA_SVC_NAME
              value: 'cassandra'

            # provide the keyspace replication factor
            - name: KEYSPACE_REPLICATION_FACTOR
              value: '3'

            # provide cassandra port 
            # default port: 9042
            - name: CASSANDRA_PORT
              value: '9042'

            # SET THE CASSANDRA_LIVENESS_CHECK
            # IT CAN BE `enable` OR `disable`
            - name: CASSANDRA_LIVENESS_CHECK
              value: ''

             ## percentage of total pods to target
            - name: PODS_AFFECTED_PERC
              value: ''
EOF

Step-2: Apply the ChaosEngine to launch the experiment

root@demo:~# kubectl apply -f cassandra-chaos.yaml -n litmus
chaosengine.litmuschaos.io/cassandra-chaos created

Step-3: Observe experiment execution

Watch the pods on the app namespace (cassandra) to view the chaos actions in progress.

watch -n 1 kubectl get pods -n cassandra

Look out for the following events.

The experiment job, as part of the experiment execution, launches a liveness pod (cassandra-liveness) that runs few cqlsh queries (create keyspaces, create tables, data insertion, and cleanup of table & keyspaces) running as separate containers of the same pod.
The liveness pod is failed if it is unable to run cqlsh commands(Cassandra is unavailable). View the cassandra-liveness pod logs during the pod-delete to verify the Uninterrupted availability of Cassandra statefulset.

kubectl logs -f cassandra-liveness -n litmus

Step-4: Verify Result Of the Chaos Experiment

View the verdict (spec.experimentStatus.verdict)of the cassandra-pod-delete chaos experiment to check whether the Cassandra cluster is resilient to the pod-delete.

root@demo:~# kubectl describe chaosresult cassandra-chaos-cassandra-pod-delete -n litmus

Name:         cassandra-chaos-cassandra-pod-delete
Namespace:    litmus
Labels:       chaosUID=22d5ba06-1fe8-4b01-bdbb-6246e1cdb2c9
              type=ChaosResult
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"litmuschaos.io/v1alpha1","kind":"ChaosResult","metadata":{"annotations":{},"labels":{"chaosUID":"22d5ba06-1fe8-4b01-bdbb-62...
API Version:  litmuschaos.io/v1alpha1
Kind:         ChaosResult
Metadata:
  Creation Timestamp:  2020-07-16T12:28:00Z
  Generation:          10
  Resource Version:    20451
  Self Link:           /apis/litmuschaos.io/v1alpha1/namespaces/litmus/chaosresults/cassandra-chaos-cassandra-pod-delete
  UID:                 32ea5093-47ee-41d8-bc34-e513edb46660
Spec:
  Engine:      cassandra-chaos
  Experiment:  cassandra-pod-delete
Status:
  Experimentstatus:
    Fail Step:  N/A
    Phase:      Completed
    Verdict:    Pass
Events:         <none>

Conclusion

The Cassandra chaos experiments are a good way to determine a potential breach of SLAs in terms of data consistency, performance & timeouts due to unexpected replica-kill. It will boost the confidence of the developers for those use-cases for which their setup returns a positive/passed result. In the future even if that chaos happens naturally(finger crossed) then the developers don’t have to worry. They are already trained with chaos management skills as they are chaos engineers/warriors after the adoption of chaos engineering practices.
Do try this experiment & let me know your findings!

Are you an SRE or a Kubernetes enthusiast? Does Chaos Engineering excite you? Join Our Community #litmus channel in Kubernetes Slack
Contribute to LitmusChaos and share your feedback on Github
If you like LitmusChaos, become one of the many stargazers here

litmuschaos / litmus

Litmus helps SREs and developers practice chaos engineering in a Cloud-native way. Chaos experiments are published at the ChaosHub (https://hub.litmuschaos.io). Community notes is at https://hackmd.io/a4Zu_sH4TZGeih-xCimi3Q

LitmusChaos

Open Source Chaos Engineering Platform

Read this in other languages.

🇰🇷 🇨🇳 🇧🇷 🇮🇳

Overview

At a high-level, Litmus comprises of:

Chaos Control Plane: A centralized chaos management tool called chaos-center, which helps construct, schedule and visualize Litmus chaos workflows
Chaos Execution Plane Services: Made up of a…

View on GitHub

Getting Started with Litmus SDK

Shubham Chaudhary — Tue, 16 Jun 2020 10:38:00 +0000

In this blog, I will talk about the generation of custom/new chaos experiment with the help of Litmus SDK . Before jumping in, let's do a quick recap on Litmus. Litmus is a framework for practicing Chaos Engineering in cloud-native environments. Litmus provides a chaos-operator, a large set of chaos experiments in its HUB, detailed documentaion, quick demo, and a friendly community.

What is Litmus SDK?

The Litmus SDK provides a simple way to bootstrap your experiment and helps create the aforementioned artifacts in the appropriate directory (i.e., as per the chaos-category) based on an attributes file provided as input by the chart-developer. The scaffolded files consist of placeholders which can then be filled as desired.

It generates the custom chaos experiments with some default Pre & Post Chaos Checks (AUT & Auxiliary Applications status checks). It can use the existing chaoslib (present inside /chaoslib directory), if available else It will create a new chaoslib inside the corresponding directory.

Life Cycle of a Chaos Experiment

Each Chaos Experiment is divided into six main sections:

Prepare: Prepare section contains the initialization steps (get ENVs), Updation of chaos result to marked the beginning of the chaos experiment.
PreChaosCheck: PreChaosCheck contains some default checks (AUT & Auxiliary Application status check) and custom checks, which may vary on the basis of experiment i.e, liveness check, data persistence check, etc.
ChaosInject: ChaosInject invoke the actual chaos, It contain the main business logic.
CleanUp: CleanUp contains the steps to remove the helper/external pod, if any
PostChaosCheck: PostChaosCheck contains the similar steps as the PreCheckCheck section have. It used to ensure the resiliency after chaos injection.
Summary Summary updates the verdict(Pass/Fail) inside the chaos result and FailStep if the experiment fails.

Pre-requisites

Go Experiments

go should be is available & the GOPATH env is configured appropriately

Ansible Experiments

python3 is available (sudo apt-get install python3.6)
jinja2 & pyYaml python packages are available (sudo apt-get install python3-pip, pip install jinja2, pip install pyYaml)

Steps to Generate Experiment via Litmus SDK

Clone the litmus-go repository for go experiments and litmus-ansible for the ansible experiments and navigate to the contribute/developer-guide folder.

## for litmus-go
$ git clone https://github.com/litmuschaos/litmus-go.git
$ cd litmus-go/contribute/developer-guide

## for litmus-ansible
$ git clone https://github.com/litmuschaos/litmus-ansible.git
$ cd litmus-ansible/contribute/developer_guide

Populate the attributes.yaml with details of the chaos experiment (or chart). Use the attributes.yaml.sample as reference.

As an example, let us consider an experiment to kill one of the replicas of an Nginx deployment. The attributes.yaml can be constructed like this:

$ cat attributes.yaml 

---
name: "sample-pod-delete"
version: "0.1.0"
category: "sample-category"
repository: "https://github.com/litmuschaos/litmus-go/tree/master/sample-category/pod-delete"
community: "https://kubernetes.slack.com/messages/CNXNB0ZTN"
description: "kills nginx pods in a random manner"
keywords:
  - "pods"
  - "kubernetes"
  - "sample-category"
  - "nginx"
scope: "Namespaced"
auxiliaryappcheck: false
permissions:
  - apigroups:
      - ""
      - "batch"
      - "litmuschaos.io"
    resources:
      - "jobs"
      - "pods"
      - "chaosengines"
      - "chaosexperiments"
      - "chaosresults"
    verbs:
      - "create"
      - "list"
      - "get"
      - "update"
      - "patch"
      - "delete"
maturity: "alpha"
maintainers:
  - name: "ispeakc0de"
    email: "shubham.chaudhary@harness.io"
provider:
  name: "Harness"
minkubernetesversion: "1.12.0"
references:
  - name: Documentation
    url: "https://docs.litmuschaos.io/docs/getstarted/"

Run the following command to generate the necessary artifacts for submitting the sample-category chaos chart with pod-delete experiment.

## litmus-go
$ ./litmus-sdk generate <generate-type> -f=attributes.yaml

Note: Replace the <generate-type>placeholder with the appropriate value based on the usecase:
- experiment: Chaos experiment artifacts belonging to a an existing OR new chart.
  - Provide the type of chaoslib in the -t flag. It supports the following values
    - exec: It creates the exec based chaoslib(default type)
    - helper: It creates the helper based chaoslib
- chart: Just the chaos-chart metadata, i.e., chartserviceversion yaml
  - Provide the type of chart in the -t flag. It supports the following values
    - category: It creates the chart metadata for the category i.e chartserviceversion, package manifests
    - experiment: It creates the chart for the experiment i.e chartserviceversion, engine, rbac, experiment manifests
    - all: it creates both category and experiment charts (default type)
- provide the path of the attribute.yaml manifest in the -f flag.

## litmus-ansible
$ python3 generate_chart.py --attributes_file=attributes.yaml --generate_type=experiment

Note: In the --generate_type flag, select the appropriate type of manifests to be generated, where,
- chart: Just the chaos-chart metadata, i.e, chartserviceversion yaml
- experiment: Chaos experiment artifacts belonging to an existing or new chart.
- provide the path of the attribute.yaml manifest in the --attributes_file flag.

Verify the Generated Files

$ cd /experiments

$ ls -ltr

total 8
drwxr-xr-x 3 shubham shubham 4096 May 15 12:02 generic/
drwxr-xr-x 3 shubham shubham 4096 May 15 13:26 sample-category/


$ ls -ltr sample-category/

total 12
-rw-r--r-- 1 shubham shubham   41 May 15 13:26 sample-category.package.yaml
-rw-r--r-- 1 shubham shubham  734 May 15 13:26 sample-category.chartserviceversion.yaml
drwxr-xr-x 2 shubham shubham 4096 May 15 13:26 sample-pod-delete/

$ ls -ltr sample-category/sample-pod-delete

total 28
-rw-r--r-- 1 shubham shubham  791 May 15 13:26 rbac.yaml
-rw-r--r-- 1 shubham shubham  734 May 15 13:26 sample-pod-delete.chartserviceversion.yaml
-rw-r--r-- 1 shubham shubham  792 May 15 13:26 experiment.yaml
-rw-r--r-- 1 shubham shubham  813 May 15 13:26 engine.yaml
## this file will be created in case of litmus-go
--rw-r--r-- 1 shubham shubham 4096 May 15 13:26 test
-rw-r--r-- 1 shubham shubham 4533 May 15 13:26 sample-pod-delete.go
## this file will be created in case of litmus-ansible
-rw-r--r-- 1 shubham shubham 1777 May 15 13:26 pod-delete-k8s-job.yml
-rw-r--r-- 1 shubham shubham 4533 May 15 13:26 pod-delete-ansible-logic.yml

$ ls -ltr sample-category/sample-pod-delete/test

total 4
-rw-r--r-- 1 shubham shubham  1039 May 15 13:26 test.yaml

Proceed with construction of business logic inside the pod-delete.go file in litmus-go or pod-delete-ansible-logic.yml in litmus-ansible, by making the appropriate modifications listed below to achieve the desired effect:
- variables
- entry & exit criteria checks for the experiment
- helper utils in either pkg or new base chaos libraries
Update the experiment.yaml with the right chaos params in the spec.definition.env with their default values.
Update the chaoslib/litmus/sample-pod-delete/lib/pod-delete.go chaoslib to achieve the desired effect or reuse the existing chaoslib.
Create an experiment README explaining, briefly, the what, why & how of the experiment to aid users of this experiment.

Steps to Test Experiment

Follow the Litmus SDK: DevTest your chaos experiments with Okteto blog to test the generated experiment.

Steps to Include the Chaos Charts/Experiments into the ChartHub

Send a PR to the litmus-go/litmus-go repo with the modified experiment files i.e, pod-memory-hog
Send a PR to the chaos-charts repo with the modified experiment CR, experiment chartserviceversion, chaos chart (category-level) chartserviceversion & package (if applicable) YAMLs i.e, kubelet service kill

Conclusion

Litmus SDK will help developers & SREs to create their custom chaos experiment on demand. It will generate the templates, pre/post chaos checks, chaoslib (which can be modified according to use-case).

litmuschaos / litmus

Litmus helps SREs and developers practice chaos engineering in a Cloud-native way. Chaos experiments are published at the ChaosHub (https://hub.litmuschaos.io). Community notes is at https://hackmd.io/a4Zu_sH4TZGeih-xCimi3Q

LitmusChaos

Open Source Chaos Engineering Platform

Read this in other languages.

🇰🇷 🇨🇳 🇧🇷 🇮🇳

Overview

At a high-level, Litmus comprises of:

Chaos Control Plane: A centralized chaos management tool called chaos-center, which helps construct, schedule and visualize Litmus chaos workflows
Chaos Execution Plane Services: Made up of a…

View on GitHub