DEV Community: Akash Shrivastava

Introduction to HTTP Chaos in LitmusChaos

Akash Shrivastava — Tue, 13 Sep 2022 15:58:52 +0000

This article is a getting-started guide for HTTP Chaos in LitmusChaos. We will be talking about

Introduction to LitmusChaos
How does HTTP Chaos work — Architecture
Types of HTTP Chaos Experiments
HTTP Chaos Demo

What is LitmusChaos

LitmusChaos is a toolset to do cloud-native chaos engineering. It provides tools to orchestrate chaos on Kubernetes to help SREs find weaknesses in their deployments. SREs use Litmus to run chaos experiments initially in the staging environment and eventually in production to find bugs and vulnerabilities. Fixing the weaknesses leads to increased resilience of the system.

Litmus takes a cloud-native approach to creating, managing and monitoring chaos. Chaos is orchestrated using the following Kubernetes Custom Resource Definitions (CRDs):

ChaosEngine: A resource to link a Kubernetes application or Kubernetes node to a ChaosExperiment. ChaosEngine is watched by Litmus’ Chaos-Operator which then invokes Chaos-Experiments
ChaosExperiment: A resource to group the configuration parameters of a chaos experiment. ChaosExperiment CRs are created by the operator when experiments are invoked by ChaosEngine.
ChaosResult: A resource to hold the results of a chaos experiment. The Chaos-exporter reads the results and exports the metrics into a configured Prometheus server.

For more information, you can visit litmuschaos.io or github.com/litmuschaos/litmus

Architecture

The experiments internally use two things to inject HTTP chaos and redirect traffic properly. First, it runs a proxy server that acts as a middleman and modifies the request/response per the experiment type. Second, it creates a routing rule in the network routing table using the IPtables library to redirect all incoming traffic on the targeted service port to the proxy port.

This diagram shows a request without HTTP chaos injected. The request to access Service A comes at port 80 and is forwarded to Service A to be processed

Now, after we inject HTTP chaos, the request to access Service A comes to port 80 but is forwarded to port 8000, on which the proxy server listens for requests. This is done by adding a routing rule in the routing table using IPtables. After the proxy server has modified the request, if required, it will forward the request to Service A to be processed. Now the response will follow the same path, the proxy server will modify the response if required and then send it back to the client to complete the request loop.

The proxy server is running inside the service pod and the service pod routing table is updated by running commands inside the service pod using the nsenter tool.

To facilitate the creation of a proxy server and adding rules to the routing table, a helper pod is run which uses nsenter to enter inside the target pod to run commands to achieve this.

Experiments

Currently, there are 5 different types of HTTP experiments available. They are

Let’s know more about them

HTTP Latency

HTTP latency adds latency to the HTTP requests by adding a sleep timer before sending the request forward from the proxy server. It can be used to simulate delayed responses from the APIs. To tune the latency value, use the LATENCY experiment variable and provide the value in milliseconds.

HTTP Reset Peer

HTTP Reset Peer simulates TCP connection reset error by closing the connection after a specified timeout. It can be used to simulate connection failures. To tune the timeout value, use the RESET_TIMEOUT experiment variable and provide the value in milliseconds

HTTP Status Code

HTTP Status Code can modify the status code of the response from the service as well as change the body of the response for the defined status code with a predefined template. It can be used to simulate API failures. To specify the status code using the STATUS_CODE experiment variable to tune the value. Supported values are available in the docs. You can also provide a comma-separated list of values and the experiment will select a random value from the list to use. If no value is provided then any random value from the supported values will be chosen.

You can use the MODIFY_RESPONSE_BODY variable to tune whether the response is changed with a predefined template according to the status code or not.

HTTP Modify Header

HTTP Modify Header can modify, add or remove headers from a request or response based on provided values. To specify whether you want to modify the request or response, use the HEADER_MODE variable. You can set it to request or response.

The HEADERS_MAP needs a JSON-type input. Suppose you want to add a header litmus with a value 2.12.0 then you should provide it like this {“litmus”: “2.12.0”} similarly for multiple values as well. To remove a header, you can overwrite its value to an empty string, currently removing the header key is not possible.

HTTP Modify Body

HTTP Modify Body can modify the request/response body completely. This can be used to modify API responses. You can use the RESPONSE_BODY variable to provide the overwrite value, this can be an HTML, plain text or JSON object.

Important Tuneable

These are the tuneable specific to all the HTTP Chaos experiments

Toxicity

TOXICITY can be used to provide the probability of requests being affected. Suppose you want only 50% of the requests to be affected, by setting the value TOXICITY to 50, the probability of a request getting affected is 50%. This doesn’t mean every alternate request will be affected, but each request has a 50–50 chance of being affected. In large requests count, this comes out to be around 50% of requests being affected.

Target Service Port

TARGET_SERVICE_PORT is the port of the service you want to target. This should be the port where the application runs at the pod level, not at the service level. This means if the application pod is running the service at port 8080 and we create a service exposing that at port 80, then the target service port should be 8080 and not 80, which is the port at pod-level.

Proxy Port

PROXY_PORT is the port at which the proxy server will be running. You are not required to change the default value (which is 20000) if this port is being used explicitly by any of your other services. If the experiment fails due to a port bind issue for the proxy server, you can change this value to an empty port to make it work.

Network Interface

NETWORK_INTERFACE is the interface name for the network which your service is using. The default value is eth0. If the experiment injection is failing due to a network interface error, you can use this to change it to the correct value.

Demo

Let us run the HTTP Status Code experiment. For simplicity, we will be injecting chaos into an Nginx service.

The service is running on port 80, we will be targeting this.

If we access the service, we are getting a 200 OK response with the default Nginx webpage. I will be using Postman to verify the status code.

Now, that we have the application set up on which we will be injecting chaos, let’s start creating a chaos scenario with the HTTP Status Code experiment.

Login to Chaos Centre and get to the Chaos Scenario section. Click on the Schedule a Chaos Scenario button. Select your agent and then select the chaos hub (HTTP experiments are available from ChaosHub version 2.11.0). Add a name for your scenario and move ahead. Now we are at selecting the experiments page.

We will be selecting the generic/pod-http-status-code experiment from the list of experiments. Moving ahead, we will tune the experiment variables.

Click on the pencil icon next to the experiment name to edit the experiment. Now we have to select the app to inject chaos into. The NGINX application we are using is running in the default namespace, it is of the deployment kind and has the label app=nginx. We will skip adding probes to keep it simple. The next section is to tune the experiment variables. Change the STATUS_CODE to 500 and the TARGET_SERVICE_PORT to the port of the service, in this case, it is port 80. The MODIFY_RESPONSE_BODY is a boolean to specify whether the response body should be changed to a pre-defined HTTP template according to the status code. Now we are done with tuning the required variables in this experiment, let’s move ahead and run this.

Now LitmusChaos will set up the experiment and then run it, once it starts injecting chaos we will be seeing the status code changing for the service. The output will be something similar to this.

That’s it, we have injected HTTP chaos into our application. The experiment passed because we haven’t specified any criteria to verify, we can do this using probes.

Summary

Through this article, we could understand how the HTTP chaos experiment works internally and then talk about the current types of HTTP chaos experiments available. Then we injected the HTTP Status code experiment on a sample NGINX service and saw the experiment in live action. In further tutorial blogs, I will be talking about running the other HTTP experiments as well.

You can join the LitmusChaos community on GitHub and Slack. The community is very active and tries to solve queries quickly.

I hope you enjoyed this journey and found the blog interesting. You can leave your queries or suggestions (appreciation as well) in the comments below.

Show your ❤️ with a ⭐ on our Github. To learn more about Litmus, check out the Litmus documentation. Thank you! 🙏

Thank you for reading

Akash Shrivastava

Software Engineer at Harness

Linkedin | Github | Instagram | Twitter

Setting up LitmusChaos on Raspberry Pi Cluster

Akash Shrivastava — Tue, 13 Sep 2022 15:43:06 +0000

This blog is a guide on how to set up LitmusChaos on a Raspberry Pi cluster. This kind of setup can be used for development or testing purposes, as it is cheaper than cloud-based services, and it overcomes any limitations on your system.

You can use this setup to see LitmusChaos in action, as well as this, can be used for development-level testing of services using Litmus.

Setting up a Raspberry Pi Cluster

This section is a guide on how to set up an RPi cluster to run Kubernetes with a Master and multiple Worker Nodes.

Hardware Required

For setting up the RPi Cluster, we need the following hardware (minimum requirement)

Raspberry Pis (at least 2, the 4 GB variant will be good enough)
Power Hub for powering the Raspberry Pis
Ethernet Cable(s)
Router (Optional Wi-Fi)
32 GB SD Card(s) (One for each RPi)
MicroSD Card Reader (or MicroSD Slot on your Laptop)

Installing Operating System on SD Card

There are many Linux-based distros available for RPis, you can go with the RaspiOS Lite, the only drawback is that it is only available for 32bit systems. Considering that, you can choose Ubuntu 20.02 Server, which is also lightweight (not as much as RaspiOS) but it has been working fine. For this article, I will be using Ubuntu 20.02 Server.

Raspberry Pi provides an official image tool for installing the operating system on SD Card, but you can use any other tool as well. Download the Ubuntu image from here. Next, connect the SD card and open the image tool. Select the Choose OS option and then select the Custom Image option, select the ubuntu image you downloaded. Next, select the storage device and click on Write. This will take some time (from 5–20 minutes), and once done, continue the same process for all other SD Cards.

After this, insert the SD Cards into the Raspberry Pis and power them on.

Connecting RPis to Wifi (Optional)

If you want to use the RPis connected with Ethernet only then you can skip this step. Also if you have a mini-HDMI to HDMI converter, you don’t need an Ethernet cable to set up wifi, you can connect your RPis to a screen and follow the same process.

To connect your RPis to Wifi, you will have to first connect it with an Ethernet cable. Go to your router settings and get the IP address of the RPis. Then SSH into the RPis one by one and repeat the same step.

ssh ubuntu@<ip-addr-rpi>

Note: The default password is ubuntu

You need to find the network interface name first

iw dev | grep Interface

Now to connect to Wifi you have to edit the netplan configuration

sudo nano /etc/netplan/50-cloud-init.yaml

Then add the following inside network block

wifis:
    <interface-name>:
        dhcp4: true
        optional: true
        access-points:
            "<your-wifi-ssid>”:
                password: "<your-wifi-password>"

Save and exit the editor and then apply the new configuration

sudo netplan apply

Now your device should be connected to wifi, you can check by

ip a

Now, repeat the same process on all the Raspberry Pis and then you can disconnect the Ethernet cable.

Note: the IP address has changed.

Configuring Raspberry Pis for SSH

First, change the hostname of the Pis so they are easy to distinguish.

For master node

sudo hostname-ctl set-hostname kmaster

For worker nodes

sudo hostname-ctl set-hostname knode<node number>

Now, on your system create SSH keys and authorise them for the RPis by following these steps

Note: Following steps are to be followed on your system

Create a .ssh directory if it doesn’t exist and cd into it

mkdir .ssh && cd .ssh

2. Use ssh-keygen to create SSH keys for master and all worker nodes, name the keys according to the hostname of the nodes so it’s easy to find.

3. Add the SSH keys to the ssh-agent

ssh-add kmaster
ssh-add knode<node number>

4. Copy the ssh-keys to the RPis

_\# Master node_
ssh-copy-id -i ~/.ssh/kmaster.pub ubunut@<RPI-IP-ADDRESS>

_\# Worker node_
ssh-copy-id -i ~/.ssh/knode<number>.pub ubuntu@<RPI-IP-ADDRESS>

If you had defined a static IP address for the RPis, you can use a hostname rather than an IP address

echo -e "<master node ip address>\\tkmaster" | sudo tee -a /etc/hosts
echo -e "<worker node 1 ip address>\\tknode1" | sudo tee -a /etc/hosts

Now, try to login into the RPis to verify that everything is working fine

ssh ubuntu@kmaster
ssh ubuntu@knode1

Installing Kubernetes on Raspberry Pi Cluster

=============================================

This section is a guide on how to install Kubernetes on Raspberry Pi Cluster with a Master and multiple Worker Nodes. We will be installing k3s because it is lightweight, you can install any other distribution as well.

Since we will be using Docker, follow the official docs to install, you can find them here.

Installing K3s Master

SSH into the master node

ssh ubuntu@kmaster

Now install K3s

curl -sfL [https://get.k3s.io](https://get.k3s.io) | sh -s - --docker

Verify that the installation was successful

sudo kubectl get nodes

Note: You can check the k3s service to debug if the installation was not successful

sudo systemctl status k3s

Installing K3s Nodes

On your system, run the following command to get the node token from the k3s master

MASTER\_TOKEN=$(ssh ubuntu@kmaster "sudo cat /var/lib/rancher/k3s/server/node-token")

Now SSH into the node

ssh ubuntu@knode1

Install K3s agent

curl -sfL [http://get.k3s.io](http://get.k3s.io) | K3S\_URL=https://kmaster:6443 K3S\_TOKEN=$MASTER\_TOKEN sh -s - --docker

Verify that the K3s agent was installed successfully

sudo systemctl status k3s-agent

Kubectlt

Install kubectl, a command-line interface tool that allows you to run commands against a remote Kubernetes cluster.

Now, create a config file to access the RPis K3s Cluster

mkdir -p $HOME/.kube/k3s
touch $HOME/.kube/k3s/config
chmod 600 $HOME/.kube/k3s/config

Next, copy the k3s cluster configuration from the master node

ssh pi@kmaster "sudo cat /etc/rancher/k3s/k3s.yaml" **\>** $HOME/.kube/k3s/config

Edit the k3s config file on the client machine and change the remote IP address of the k3s master from localhost/127.0.0.1 to kmaster

_\# Edit master config_
nano $HOME/.kube/k3s/config

_\# Search for the 'server' attribute located in -_
_\# clusters:_
_\# - cluster:_
_\#   server:_ [_https://127.0.0.1:6443_](https://127.0.0.1:6443) _or_ [_https://localhost:6443_](https://localhost:6443)
_#_
_\# Change 'server' value to_ [_https://kmaster:6443_](https://kmaster:6443)_
\# Do not change the port value_

Now, export the _k3s_ config file path as the KUBECONFIG environment variable to use the config

export KUBECONFIG=$HOME/.kube/k3s/config

Verify the setup

kubectl get nodes

Installing LitmusChaos on Raspberry Pi Cluster

==============================================

This section is a guide on how to install LitmusChaos 2.0 on Raspberry Pi Cluster with K3s

For installation, we will be following their docs. There are two ways to install, one is by using helm, other is by applying the YAML spec file. We will be installing using the YAML spec file, you can follow the other one if you want by going through their docs.

kubectl apply -f [https://litmuschaos.github.io/litmus/2.0.0/litmus-2.0.0.yaml](https://litmuschaos.github.io/litmus/2.0.0/litmus-2.0.0.yaml)

Note: You can find the latest version of litmus at docs.litmuschaos.io

Let’s verify that all the services are running and that there have been no issues

kubectl get pods -n litmus
kubectl get svc -n litmus

You can now use the LitmusChaos dashboard by using this address

<master-node-ip>:<port>

Change the with the master node IP and the with what is showing to you for the litmusportal-frontend-service external port value, the one after 9091: and then visit that address in your browser.

Add-Ons

=======

The /etc/hosts file sets to default after a restart, so you will have to keep adding the RPis IP every time you restart or you can run a startup script that will automatically set the values on every restart.

You can edit the bash profile file on your system to use this Kubeconfig and also add the ssh keys to the ssh-agent. In my system it was the /home/username/.profile file, it might differ in your system. I added these lines to the profile

eval $(ssh-agent)
ssh-add ~/.ssh/kmaster
ssh-add ~/.ssh/knode1export KUBECONFIG=$HOME/.kube/k3s/config

Summary

=======

In this article, we first set up the Raspberry Pis cluster and then installed K3s on the cluster. After that, we installed LitmusChaos onto the K3s cluster. We can now proceed with injecting chaos using the portal. This kind of setup is beneficially for local development purposes, and you will be saving money on AWS servers.

You can join the LitmusChaos community on Github and Slack. The community is very active and tries to solve queries quickly.

I hope you enjoyed this journey and found the blog interesting. You can leave your queries or suggestions (appreciation as well) in the comments below.

Show your ❤️ with a ⭐ on our Github. To learn more about Litmus, check out the Litmus documentation. Thank you! 🙏

Thank you for reading

Akash Shrivastava

Software Engineer at Harness

Linkedin | Github | Instagram | Twitter

How to run Azure Disk Loss Experiment in LitmusChaos

Akash Shrivastava — Fri, 29 Oct 2021 04:24:11 +0000

This article is a guide for setting up and running the Azure Virtual Disk Loss experiment on LitmusChaos 2.0. The experiment causes detachment of one or more virtual disks from the instance for a certain chaos duration and then re-attached them. The broad objective of this experiment is to extend support of LitmusChaos to non-Kubernetes targets while ensuring resiliency for all kinds of targets, as a part of a single chaos workflow for the entirety of a business.

Currently, the experiment is available only as a technical preview in the chaos hub, so we will have to use the master branch of the chaos hub to access it.

If you are looking for the Azure Instance Stop experiment, you can find it here

Pre-Requisites

To run this experiment, we need a few things beforehand

An Azure account
Disk(s) attached to Virtual Machine Scale Set (or an Instance only)
A Kubernetes cluster with LitmusChaos 2.0 installed (you can follow this blog to set up LitmusChaos 2.0 on AKS — Getting Started with LitmusChaos 2.0 in Azure Kubernetes Service)

Setting up Azure Credentials as Kubernetes Secret

To let LitmusChaos access your Azure instances, you need to set up the azure credentials as a Kubernetes secret. It is a very simple process, first, you need to install Azure CLI (if you already haven’t) and log in to it. Now run this command to get the azure credentials saved in an azure.auth file.

az ad sp create-for-rbac — sdk-auth > azure.auth

Next, create a secret.yaml file with the following content. Change the content inside azure.auth with the contents inside your azure.auth file

apiVersion: v1
kind: Secret
metadata:
  name: cloud-secret
type: Opaque
stringData:
  azure.auth: |-
    {
      "clientId": "XXXXXXXXX",
      "clientSecret": "XXXXXXXXX",
      "subscriptionId": "XXXXXXXXX",
      "tenantId": "XXXXXXXXX",
      "activeDirectoryEndpointUrl": "XXXXXXXXX",
      "resourceManagerEndpointUrl": "XXXXXXXXX",
      "activeDirectoryGraphResourceId": "XXXXXXXXX",
      "sqlManagementEndpointUrl": "XXXXXXXXX",
      "galleryEndpointUrl": "XXXXXXXXX",
      "managementEndpointUrl": "XXXXXXXXX"
    }

Now run the following command. Remember to change the namespace if you have installed LitmusChaos in any other namespace

kubectl apply -f secret.yaml -n litmus

Updating ChaosHub

As the experiment is only available as a technical preview right now, we will have to update the ChaosHub to use the technical preview (master) branch.

Now change the branch to “master”.

Click on Submit Now and the ChaosHub will now show the Azure Disk Loss experiment.

Scheduling the Experiment Workflow

Now move to the Workflows section and click on Schedule a Workflow. Select the Self-Agent (or any other one if you have multiple agents installed) and click on Next.

Select the third option to create a workflow from experiments using ChaosHub. Click on Next.

Click Next again (or edit the workflow name if you want to) and now on the Experiments page, click on Add a new Experiment and select the Azure Virtual Disk Loss experiment.

Next click on Edit YAML, you will now have to add the Disk Name(s) and Resource Group name in the ChaosEngine environments. Scroll down to the ChaosEngine artefacts, where you will see the environment variables, set the values accordingly. If your disks are connected to an instance that is a part of Scale Set, set the SCALE_SET to “enable”. Save the changes and schedule your workflow

Note: For Scale set and node pools, the experiment works only for disk(s) attached to a specific instance in the scale set and not to the scale set

Observing the Experiment Run

Great, now your workflow is running and you can check it out, click on Go to Workflow and then select your workflow.

You can check the status of your disk(s) in the Azure Portal to verify that the experiment is working as expected.

You can also click on azure-disk-loss to view the experiment logs. After the given chaos duration, the experiment will automatically re-attach the disk(s), and it will give a pass/fail verdict. In case the experiment fails, verify through the logs and portal that the disk have been re-attached.

This was it, you have successfully run the Azure Disk Loss experiment using LitmusChaos 2.0 Chaos Center.

In this blog, we saw how we can perform the Azure Disk Loss experiment using LitmusChaos 2.0. You can learn more about this experiment from the docs. This experiment is one of the many experiments Non-Kubernetes experiments in LitmusChaos, including experiments for AWS, GKS, VMWare, which are targeted towards making Litmus an absolute Chaos Engineering toolset for every enterprise regardless of the technology stack used.

You can join the LitmusChaos community on Github and Slack. The community is very active and tries to solve queries quickly.

I hope you enjoyed this journey and found the blog interesting. You can leave your queries or suggestions (appreciation as well) in the comments below.

Show your ❤️ with a ⭐ on our Github. To learn more about Litmus, check out the Litmus documentation. Thank you! 🙏

Thank you for reading

Akash Shrivastava

Software Engineer at ChaosNative

Linkedin | Github | Instagram | Twitter

How to run Azure Instance Stop Experiment in LitmusChaos

Akash Shrivastava — Wed, 25 Aug 2021 13:37:00 +0000

This article is a guide for setting up and running the Azure Instance Stop experiment on LitmusChaos 2.0. The experiment causes the power off of one or more azure instance(s) for a certain chaos duration and then power them on. The broad objective of this experiment is to extend the principles of cloud-native chaos engineering to non-Kubernetes targets while ensuring resiliency for all kinds of targets, as a part of a single chaos workflow for the entirety of a business.

Currently, the experiment is available only as a technical preview in the chaos hub, so we will have to use the master branch of the chaos hub to access it.

Pre-Requisites

To run this experiment, we need a few things beforehand

An Azure account
A Virtual Machine Scale Set (or an Instance only)
A Kubernetes cluster with LitmusChaos 2.0 installed (you can follow this blog to set up LitmusChaos 2.0 on AKS — Getting Started with LitmusChaos 2.0 in Azure Kubernetes Service)

Setting up Azure Credentials as Kubernetes Secret

az ad sp create-for-rbac — sdk-auth > azure.auth

Next, create a secret.yaml file with the following content. Change the content inside azure.auth with the contents inside your azure.auth file

apiVersion: v1
kind: Secret
metadata:
  name: cloud-secret
type: Opaque
stringData:
  azure.auth: |-
    {
      "clientId": "XXXXXXXXX",
      "clientSecret": "XXXXXXXXX",
      "subscriptionId": "XXXXXXXXX",
      "tenantId": "XXXXXXXXX",
      "activeDirectoryEndpointUrl": "XXXXXXXXX",
      "resourceManagerEndpointUrl": "XXXXXXXXX",
      "activeDirectoryGraphResourceId": "XXXXXXXXX",
      "sqlManagementEndpointUrl": "XXXXXXXXX",
      "galleryEndpointUrl": "XXXXXXXXX",
      "managementEndpointUrl": "XXXXXXXXX"
    }

Now run the following command. Remember to change the namespace if you have installed LitmusChaos in any other namespace

kubectl apply -f secret.yaml -n litmus

Updating ChaosHub

As the experiment is only available as a technical preview right now, we will have to update the ChaosHub to use the technical preview (master) branch.

Now change the branch to “master”.

Click on Submit Now and the ChaosHub will now show the Azure Instance Stop experiment.

Scheduling the Experiment Workflow

Now move to the Workflows section and click on Schedule a Workflow. Select the Self-Agent (or any other one if you have multiple agents installed) and click on Next.

Select the third option to create a workflow from experiments using ChaosHub. Click on Next.

Click Next again (or edit the workflow name if you want to) and now on the Experiments page, click on Add a new Experiment and select the Azure Instance Stop experiment.

Next click on Edit YAML, you will now have to add the Instance Name(s) and Resource Group name in the ChaosEngine environments. Scroll down to the ChaosEngine artefacts, where you will see the environment variables, set the values accordingly. If you are injecting chaos on a Scale Set, set the SCALE_SET to “enable”. Save the changes and schedule your workflow

Note: You need to provide the instance name from the Virtual Machine Scale Set section for Azure AKS nodes, not from the AKS node pools section.

Azure Virtual Machine Scale Set Instances section

Observing the Experiment Run

Great, now your workflow is running and you can check it out, click on Go to Workflow and then select your workflow.

You can check the status of your instance in the Azure Portal to verify that the experiment is working as expected.

You can also click on azure-instance-stop to view the experiment logs. After the given chaos duration, the experiment will automatically power on the instance(s), and it will give a pass/fail verdict. In case the experiment fails, verify through the logs and portal that the instances have started.

This was it, you have successfully run the Azure Instance Stop experiment using LitmusChaos 2.0 Chaos Center.

In this blog, we saw how we can perform the Azure Instance Stop experiment using LitmusChaos 2.0. You can learn more about this experiment from the docs. This experiment is one of the many experiments Non-Kubernetes experiments in LitmusChaos, including experiments for AWS, GKS, VMWare, which are targeted towards making Litmus an absolute Chaos Engineering toolset for every enterprise regardless of the technology stack used.

You can join the LitmusChaos community on Github and Slack. The community is very active and tries to solve queries quickly.

I hope you enjoyed this journey and found the blog interesting. You can leave your queries or suggestions (appreciation as well) in the comments below.

Show your ❤️ with a ⭐ on our Github. To learn more about Litmus, check out the Litmus documentation. Thank you! 🙏

Thank you for reading

Akash Shrivastava

Software Engineer at Harness

Linkedin | Github | Instagram | Twitter

Getting Started with LitmusChaos 2.0 in Azure Kubernetes Service

Akash Shrivastava — Fri, 09 Jul 2021 15:45:00 +0000

This is a quick tutorial on how to get started with LitmusChaos 2.0 in Azure Kubernetes Services. We will first create an AKS Cluster, followed by Installing LitmusChaos 2.0 on the cluster and then executing a simple pre-defined chaos workflow using LitmusChaos.

What is LitmusChaos

LitmusChaos is a toolset to do cloud-native chaos engineering. It provides tools to orchestrate chaos on Kubernetes to help SREs find weaknesses in their deployments. SREs use Litmus to run chaos experiments initially in the staging environment and eventually in production to find bugs, vulnerabilities. Fixing the weaknesses leads to increased resilience of the system.

Litmus takes a cloud-native approach to create, manage and monitor chaos. Chaos is orchestrated using the following Kubernetes Custom Resource Definitions (CRDs):

ChaosEngine: A resource to link a Kubernetes application or Kubernetes node to a ChaosExperiment. ChaosEngine is watched by Litmus’ Chaos-Operator which then invokes Chaos-Experiments
ChaosExperiment: A resource to group the configuration parameters of a chaos experiment. ChaosExperiment CRs are created by the operator when experiments are invoked by ChaosEngine.
ChaosResult: A resource to hold the results of a chaos experiment. The Chaos-exporter reads the results and exports the metrics into a configured Prometheus server.

For more information you can visit litmuschaos.io or github.com/litmuschaos/litmus

Pre-Requisites

Azure CLI — How to install on Linux/Debian
kubectl — How to install on Linux

If you feel lazy to install them, you can always use the Azure Cloud Shell, it already has the tools installed.

Creating an AKS Cluster

The first step to installing LitmusChaos on an AKS Cluster is to have an AKS Cluster. So let’s do that. Open Azure Portal and then log in with your account. You will be presented with the home screen. Now search for Kubernetes services and open it.

To create a cluster, click on create Create option in the menu and then select Create a Kubernetes cluster

Creating an AKS Cluster

Now you have to fill in details about what kind of cluster you want to create. Since Azure doesn’t charge for cluster management, you will only have to pay for the Node Instance you will be running. Fill in the name of the cluster, it can be anything, also create a new Resource Group if you haven’t. You can keep other settings as it is, or if you know what they do, can change it according to your need. For the Node Pool, select a B2ms size that has 2 vCPUs and 8 GiB of RAM and set the Node Count to 1 as we only want to run LitmusChaos, this will suffice for it. Although you are free to choose your configuration, keeping a minimum of 2 vCPUs and 8 GiB of RAM will help in seamless running. Remember to check that the Scale Method is set to Manual to keep a check on the cost.

Configuring AKS cluster

You can skip the rest of the configurations for now and directly click on Review + Create which will start the creation of Cluster. It will take around 5–10 minutes, so you can sit back for some time, grab a glass of water, also read about ChaosEngineering and LitmusChaos 2.0

AKS Cluster Deployment in Progress

The cluster is ready and you can now install LitmusChaos on it. You can use the Azure Cloud Shell or your local system terminal to connect to the Cluster, the steps are the same for both. I personally prefer using my local system so I will use that for this tutorial.

Connecting to AKS Cluster

Open your cluster and click on the Connect button, this will show you two commands to run. Copy the two commands and run them one by one. The first command sets the account as per the subscription id provided, and the second command fetches the credentials for the specific resource.

Connecting to AKS Cluster

Installing LitmusChaos

Now you have the credentials to access the Cluster, you can go ahead and install LitmusChaos 2.0 and do some chaos. For installation, I will be following their docs. There are two ways to install, one is by using helm, other is by applying the manifest file. I will follow the helm repo procedure, you can follow the other one if you want by going through their docs.

Note: You will need to have Helm installed on your system. You can refer from here

First, you will add the LitmusChaos Helm repository and then confirm that litmuschaos is present in the helm repository

helm repo add litmuschaos [https://litmuschaos.github.io/litmus-helm/](https://litmuschaos.github.io/litmus-helm/)
helm repo list

Adding litmus to helm repo

Next, you will create the namespace, by default we use *litmus *as the namespace name, you are allowed to use any name of your choice, just remember to change it in the following commands.

kubectl create ns litmus

Now, let’s install LitmusChaos using the helm repository you just added.

helm install chaos litmuschaos/litmus-2–0–0-beta --namespace=litmus --devel --set portalScope=namespace

Creating litmus namespace and installing LitmusChaos

Note: If you are using helm2, you will have to run this command

helm install --name chaos litmuschaos/litmus-2–0–0-beta --namespace=litmus --devel --set portalScope=namespace

The final step is to install the LitmusChaos CRDs

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/master/litmus-portal/litmus-portal-crds.yml

Installing LitmusChaos CRDs

Let’s verify that all the services are running, and there has been no issue

LitmusChaos Services

The services are running properly but there is one more change that you need to do, since AKS doesn’t provide public-IP to nodes by default, we need to change the litmusportal-frontend-service to a LoadBalancer service. You can do that by editing the service.

kubectl edit svc litmusportal-frontend-service -n litmus

At the very end inside spec there is type: NodePort, you have to change it to type: LoadBalancer

spec:
 clusterIP: xxxxxxx
 externalTrafficPolicy: Cluster
 ports:
 — name: http
 nodePort: xxxxx
 port: 9091
 protocol: TCP
 targetPort: 8080
 selector:
 app.kubernetes.io/component: litmus-2–0–0-beta-frontend
 sessionAffinity: None
 # Change the type here from NodePort to LoadBalancer
 type: LoadBalancer

Then save it, and list the services again. The External-IP might show pending for a minute, run the command again after a minute to get the IP.

LitmusChaos Services

<external-ip>:9091

Change the with what is showing to you for the litmusportal-frontend-service and then visit the address in your browser.

LitmusChaos Portal sign-in page

Ta-da! We are done with the installation of LitmusChaos 2.0 and now you can run a workflow. Login to the portal, the default credentials are

username: admin
password: litmus

It will ask you to set a new password, and then log in to the dashboard.

Note: Other than changing the frontend service to LoadBalancer, there is another way to make it work with NodePort by enabling public IP for individual nodes. I have not covered it in this article, but feel free to check it out here.

Running Chaos Workflows

LitmusChaos Portal Dashboard Page

Let’s see some chaos happening now. LitmusChaos comes with few predefined workflows, which setups a service and then wreak havoc in them. We will be running a podtato-head workflow, which creates a simple deployment and then injects the pod-delete experiment into it.

On the dashboard select Schedule a Workflow. In the Workflows dashboard, select the Self-Agent and then click on Next. In the next screen, select Create a Workflow from Pre-defined Templates and then select podtato-head and then click on Next.

Scheduling a podtato-head template-based workflow

On the next screen, you can define the Experiment name, description, and namespace, leave the default values and click on Next.

On this screen, you can tune the workflow by editing the experiment manifest and adding/removing or arranging the experiments in the workflow. The podtato-head template comes with its own defined workflow so simply click on Next.

Tuning the Workflow

The next screen is to adjust the weights of the experiment on the reliability score since you are running only one experiment, you can keep any value, in the case where you have multiple experiments running, you can set the importance of each experiment according to your requirements to get a meaningful reliability score. For now, click on Next and select Schedule now, you can also create a recurring schedule if you want the experiment to keep running at certain intervals. The final screen is to confirm the workflow and schedule it. Click on Finish to run the workflow

Workflow created

Yay! The workflow is created and is running now. Click on Go to Workflow, which will take you to the workflow screen, here you can see all your scheduled workflows. Click on the workflow to see its status.

Workflow Dashboard Page

The workflow will take some minutes to run, you can take a break until then. Meanwhile, you can join the LitmusChaos community on slack to stay updated with new releases and get help from the community.

Workflow Dashboard for the podtato-head workflow

The workflow run is now complete, you can access the workflow details using the graph view or the table view.

Workflow Completed Graph View

Workflow Completed Table View

Click on View Logs & Results to check out the logs and chaos results for the experiment

Experiment Logs and Results

And we are done. You were able to create an AKS Cluster, install LitmusChaos 2.0 on it, log in to the LitmusChaos Portal and then finally schedule a Workflow.

You can join the LitmusChaos community on Github and Slack. The community is very active and tries to solve queries quickly.

I hope you enjoyed this journey and found the blog interesting. You can leave your queries or suggestions (maybe appreciation as well) in the comments below.

Thank you for reading

Akash Shrivastava

Software Engineer at ChaosNative and a final year CSE student

Linkedin | Github | Instagram | Twitter