DEV Community: Udit Gaurav

LitmusChaos: Node Memory Hog Experiment

Udit Gaurav — Fri, 09 Apr 2021 12:18:54 +0000

In this blog, I will be discussing the LitmusChaos experiment that is used to exhaust the available memory resource on a Kubernetes node called Node Memory Hog. The main focus will be on the memory consumption and monitoring techniques used for the experiment. Before moving forward for those who are new to litmus and wanted to explore more about litmus features or chaos engineering in general, I would recommend checking out the litmus blog first also Litmus provides a chaos operator, a large set of chaos experiments on its hub, detailed documentation, and a friendly community do checkout them.

What you’ll need

For running this experiment you should have:

Kubernetes 1.11 or later
Litmus Installed 1.12.2 or later in the Cluster

What it’ll break

The Node Memory Hog experiment causes Memory resource exhaustion on the Kubernetes node which may lead to the eviction or unusual behavior of the application running on the cluster depending upon the extent up to which we exhaust the memory resource of the application node. The experiment uses a stress-ng tool to inject memory chaos.

The extent up to which we want to consume the node memory resource can be provided in two modes (from litmus 1.12.2):

In the percentage of the total memory capacity of the Node.
In the Mebibytes unit to consume available memory.

I'll demonstrate the experiment in both modes in upcoming sections. In both cases, the experiment consumes a set amount of memory, or as much as is available (whichever is lower), and holds onto it for the duration of the chaos.

Get Ready To Induce Memory Chaos On Kubernetes Node

Infrastructure

I'll be using a three-node GKE cluster of type e2-standard-2 (having 2 vCPU and 8 GB Memory).

root@cloudshell:$ kubectl get nodes
NAME                                       STATUS   ROLES    AGE    VERSION
gke-cluster-1-default-pool-3020340d-3vns   Ready    <none>   111s   v1.17.14-gke.400
gke-cluster-1-default-pool-3020340d-jfg6   Ready    <none>   111s   v1.17.14-gke.400
gke-cluster-1-default-pool-3020340d-k0lv   Ready    <none>   111s   v1.17.14-gke.400

Monitor the Node Memory resource

We can monitor the node memory resource consumption:

By ssh into the target node and run htop(if not available then install and run) OR
By Creating a pod on the target node having htop container for monitoring.

htop is an interactive system-monitor and process viewer tool for Unix:
We will use the second way to monitor the host memory resource. We'll not get the process information responsible for eating the memory resource in this way but we can have a watch over the current(consumed/total) memory. Follow the steps to setup the monitoring pod:

Create the monitoring file:

vim myhtop.yaml

Add the following to the myhtop.yaml

---
apiVersion: v1
kind: Pod
metadata:
  name: myhtop
spec:
  containers:
  - name: myhtop
    image: litmuschaos/go-runner:ci
    imagePullPolicy: Always
    command: ['sh', '-c', 'sleep 3600']
  ## Replace this with the target node name...
  nodeName: gke-cluster-1-default-pool-3020340d-3vns

Create the myhtop pod:

root@cloudshell:$ kubectl apply -f myhtop.yaml
pod/myhtop created

NOTE: To monitor more than one node you can use DaemonSet or Deployment.

Now exec into the pod and run htop command:

kubectl exec -it myhtop -- /bin/sh
/litmus $ htop

The Output will look like this:

The memory row in the output is representing that the node size is 7.77G out of which 502M is in use.

Deploy an Nginx application as AUT

We'll now deploy the Nginx application as Application Under Test.

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/chaos-ci-lib/master/app/nginx.yml

This will deploy an Nginx application with three replicas. It is Optional to deploy an application for running a node memory experiment we do this to check the application behavior(its availability)under chaos. A general observation after deploying the application is a slight increase in memory usage from 502M(to 518M) as one replica of the application pod was scheduled on the target node. For heavy application, it will be definitely more.

Setup Litmus and run chaos

Install litmus to run the experiment if not already installed.

kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-latest.yaml

For running chaos follow the instructions from experiment docs and create the CRs using hub. This will include the creation of RBAC, experiment, and engine(with appinfo and memory consumption values). When all the resources are created and the experiment starts executing in a traditional litmus way a runner pod will get created that will create the experiment pod for chaos execution.

root@cloudshell:$ kubectl get pods
NAME                           READY   STATUS    RESTARTS   AGE
myhtop                         1/1     Running   1          61m
nginx-756f79d98d-47d5f         1/1     Running   0          47m
nginx-756f79d98d-q48dv         1/1     Running   0          47m
nginx-756f79d98d-sqrfz         1/1     Running   0          47m
nginx-chaos-runner             1/1     Running   0          31s
node-memory-hog-47m4k0-pz9tx   1/1     Running   0          27s
node-memory-hog-bdvnde         1/1     Running   0          12s

Run and Monitor Node Memory hog Chaos

As we discussed in previous sections that we can run the node memory hog experiment in two different modes.

In Percentage Mode

By using the Percentage of total memory resource capacity of the node. In this mode, we use MEMORY_CONSUMPTION_PERCENTAGE env in chaosengine to provide the input percentage example 50(without percentage symbol) for 50% of total node capacity which is 7.77G. After running the chaos we can monitor the memory consumption from htop terminal.

Expectation: The 50% of total memory(7.77G) that is 3.885G Memory should be consumed.

Observation: We observe that the memory consumption is in between 3.56G to 3.93G. It is more fluctuating when compared with Mebibytes mode of execution.

In Mebibytes Mode

By using the Mebibytes unit to consume available memory. In this mode, we use MEMORY_CONSUMPTION_MEBIBYTES env in chaosengine to provide the input in Mebibytes example 3500(without Mebibytes symbol) for consuming nearly 3.5G of available node memory. After running the chaos we can monitor the memory consumption from htop terminal.

Expectation: The 3.5G of available memory to be consumed apart of 518M that was already in use. So the total amount of memory to be consumed should be nearly 4.0G.

Observation: From the above output we can observe that the memory consumed is 3.94G which is near to 4.0G. This mode of execution is less fluctuating and more accurate than the percentage mode.

Analysing The Memory Spike On Node

Now let us try to analyze the Memory Spike over the target nodes as the experiment takes memory from the process, simulating a massive memory burst. This could cause extreme memory demands to see how the API would identify and rectify such an issue. For analyzing the experiment behavior let's plot a memory hog graph which tells us about the different parameters like active memory bytes, free and available memory bytes, cpu usage, and disk read time.

In the above graph, the highlighted area marks the area under chaos. We can note the moment we start the experiment the free and available memory bytes spike up which represents that more amount of available or free bytes are consumed during the chaos and a very limited amount of memory is left to use. Similarly, the active memory and average free along with cache memory drop down which indicates the limited space left on the nodes.

Now let us try to max-out the memory consumption by the Kubernetes nodes and analyze the real memory consumption using htop, free -m, and kubectl top(metrics-server) commands.

We’ll re-run the experiment to check the upper limit of the memory consumption by changing the MEMORY_CONSUMPTION_PERCENTAGE value to consume 100 % of memory (that is 7.77G) in the chaosengine. To rerun the experiment we just need to create the chaosengine again with new inputs this time. And check the output using kubectl top (metrics-server), htop, free -m.

kubectl top Vs htop Vs free -m

kubectl top (metrics-server):

The kubectl top command returns current CPU and memory usage for a cluster’s pods or nodes. kubectl top pod uses a memory working set, the output of the kubectl top will be similar to the value of the metric "container_memory_working_set_bytes" in Prometheus.
If we run this query in Prometheus:

container_memory_working_set_bytes{pod_name=~"<pod-name>", container_name=~"<container-name>", container_name!="POD"}

you will get value in bytes that almost matches the output of kubectl top nodes.

We observe from the latest experiment run along with 100% of memory consumption, the memory usage output from kubectl top reached a maximum up to 92% and fluctuated between 70-90%.

htop:

htop is similar or alternative to the top command that is much easier to use for normal tasks. It is a cross-platform interactive process viewer.

The amount of memory consumption will raise to 6G.

free -m:

free is another useful command in Linux to get a detailed report on the system’s memory usage. When used without any option, the free command will display information about the memory and swap in kibibyte. 1 kibibyte (KiB) is 1024 byte. To get the vales in megabytes we need to use free -m.

Observation:

The experiment will not consume/hog the memory greater than the total memory available on Node in other words there will always be an upper limit for the amount of memory to be consumed equal to the total available memory.
There is more fluctuation in the memory consumption when we provide the amount of memory to be consumed in percentage while providing the memory in mebibytes will be less fluctuating but we need to make sure that we should not exceed the available memory otherwise our experiment helpe pod will get evicted.

What happens when both modes are provided

There are cases when both ENVs are provided or non of them are provided. Then priority will be given according to the following table:

MEMORY_CONSUMPTION_PERCENTAGE	MEMORY_CONSUMPTION_MEBIBYTES	Priority
Defined	Not Defined	Percentage
Defined	Defined	Percentage
Not Defined	Defined	Memory in Mebibytes
Not Defined	Not Defined	Default with 30%

WARNING: When a large amount of available memory is consumed like 95% or 100% of available memory then some services may stop working for a certain duration and the node may get crashed and go into a NotReady state.

Conclusion

You have installed litmus on your Kubernetes cluster and learned to execute LitmusChaos node memory hog experiments in different modes and at different values. You also configure a monitoring pod on the node using htop unix command. You also came across the different cases of node memory hog execution and its validation. If you haven't tried the experiment yet this is the best time for you to start and don't forget to give your opinion and share your feedback/experience with litmus in the comments below.

Are you an SRE or a Kubernetes enthusiast? Does Chaos Engineering excite you?
Join Our Community On Slack For Detailed Discussion, Feedback & Regular Updates On Chaos Engineering For Kubernetes: https://kubernetes.slack.com/messages/CNXNB0ZTN
(#litmus channel on the Kubernetes workspace)
Check out the Litmus Chaos GitHub repo and do share your feedback: https://github.com/litmuschaos/litmus
Submit a pull request if you identify any necessary changes.

litmuschaos / litmus

Litmus helps SREs and developers practice chaos engineering in a Cloud-native way. Chaos experiments are published at the ChaosHub (https://hub.litmuschaos.io). Community notes is at https://hackmd.io/a4Zu_sH4TZGeih-xCimi3Q

LitmusChaos

Open Source Chaos Engineering Platform

Read this in other languages.

🇰🇷 🇨🇳 🇧🇷 🇮🇳

Overview

LitmusChaos is an open source Chaos Engineering platform that enables teams to identify weaknesses & potential outages in infrastructures by inducing chaos tests in a controlled way. Developers & SREs can practice Chaos Engineering with LitmusChaos as it is easy to use, based on modern Chaos Engineering principles & community collaborated. It is 100% open source & a CNCF project.

LitmusChaos takes a cloud-native approach to create, manage and monitor chaos. The platform itself runs as a set of microservices and uses Kubernetes custom resources (CRs) to define the chaos intent, as well as the steady state hypothesis.

At a high-level, Litmus comprises of:

Chaos Control Plane: A centralized chaos management tool called chaos-center, which helps construct, schedule and visualize Litmus chaos workflows
Chaos Execution Plane Services: Made up of a…

View on GitHub

Multiarch Support In LitmusChaos

Udit Gaurav — Mon, 07 Dec 2020 09:10:19 +0000

In this blog, I will be discussing how to build a multi-arch image for any desired docker architecture to run litmus chore components. For those who are new to litmus and wanted to explore more about chaos engineering, I would recommend to check out litmus blog. Now coming back to the topic we will be using docker buildx for building the images in the following sections.

Pre-requisites:

Docker(version 19.03)

Understanding Multiarch Builds

Docker introduced the principle of multi-arch builds to support the "Build once, deploy anywhere" concept which helps to use ARM targets and reduce your bills such as AWS A1 and Raspberry Pis instances. But how do we produce them? and How do they work?
A multi-arch Docker image supports a different architecture behind the same imagetag. Let's compare the manifest of the multi-arch image with the image having a single arch.

Enable docker CLI if not already enabled using the following steps:

export DOCKER_CLI_EXPERIMENTAL=enabled
Add "experimental": "enabled", to ~/.docker/config.json (default location) at the beginning of the file and not at the end.

Image with single arch manifest

$ sudo docker manifest inspect --verbose litmuschaos/go-runner:1.9.0
{
    "Ref": "docker.io/litmuschaos/go-runner:1.9.0",
    "Descriptor": {
        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
        "digest": "sha256:bc14ce592d7d600eb14c0b6dbfb0f6e8211f7795b5bc9def057cba6b3c994185",
        "size": 2002,
        "platform": {
            "architecture": "amd64",
            "os": "linux"
        }
    },
    "SchemaV2Manifest": {
        "schemaVersion": 2,
        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
        "config": {
            "mediaType": "application/vnd.docker.container.image.v1+json",
            "size": 5502,
            "digest": "sha256:79fc16e2bbab6885589bd491c5ebf68c693239b8dbf13d45223b84e1a3aaa25b"
        },
        "layers": [ ... ]
    }
}

Multi arch image manifest

$ sudo docker manifest inspect --verbose litmuschaos/go-runner:1.10.0
[
    {
        "Ref": "docker.io/litmuschaos/go-runner:1.10.0@sha256:89717f4c87c7fc93c63d2d74c54c5e4457dd99dd37b1bbda3bbbabfc0f2eacb4",
        "Descriptor": {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "digest": "sha256:89717f4c87c7fc93c63d2d74c54c5e4457dd99dd37b1bbda3bbbabfc0f2eacb4",
            "size": 2207,
            "platform": {
                "architecture": "arm64",
                "os": "linux"
            }
        },
        "SchemaV2Manifest": {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "schemaVersion": 2,
            "config": {
                "mediaType": "application/vnd.docker.container.image.v1+json",
                "digest": "sha256:959a3aa87f6ae3add03a5a568d218e99e9394c785d924e5d6398d7cb4be0f472",
                "size": 5123
            },
            "layers": [ ... ]
        }
    },
    {
        "Ref": "docker.io/litmuschaos/go-runner:1.10.0@sha256:04cec97644444d564e256e87184e0daa73f3f335d187c65a1723f0f8dc61ad22",
        "Descriptor": {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "digest": "sha256:04cec97644444d564e256e87184e0daa73f3f335d187c65a1723f0f8dc61ad22",
            "size": 2210,
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        "SchemaV2Manifest": {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "schemaVersion": 2,
            "config": {
                "mediaType": "application/vnd.docker.container.image.v1+json",
                "digest": "sha256:679ad1c72c9be392b47b3fc6c455c383e28b8fcdbcf923dd5fb9db2cfb5c3cc2",
                "size": 5125
            },
            "layers": [ ... ]
        }
    }
]

We can observe that the manifest for the multi-arch image is a simple list of manifests prepared for each platform while an image with a single arch will have only one platform in its manifest.

Build Multi-arch image for Litmus Components

For building a multi-arch image for litmus core components just follow the following simple steps:

Setup buildx
Build go binary for different architectures.
Prepare Dockerfile for Multiarch.
Build and push the multi-arch image.

Build using Buildx instance

Introduction:

Docker Buildx is a CLI plugin that extends the docker command with the full support of the features provided by Moby BuildKit builder toolkit. It has new features like creating scoped builder instances and building against multiple nodes concurrently.

Setup:

The Docker Buildx is included in Docker 19.03 to install it run the following commands:

$ export DOCKER_BUILDKIT=1
$ docker build --platform=local -o . git://github.com/docker/buildx
$ mkdir -p ~/.docker/cli-plugins
$ mv buildx ~/.docker/cli-plugins/docker-buildx

Confirm the installation:

$ sudo docker buildx version

Output

github.com/docker/buildx v0.4.2-16-g1595352 159535261de8416d2a83d600ccdd7dbfa6d71303

for any issue in installation refer installation guide.

Now the buildx is setup in your system. Buildx allows you to create new instances of isolated builders. You can use this to get a scoped environment for your CI builds that does not change the state of the shared daemon, or for isolating builds for different projects. You can create a new instance for a set of remote nodes, forming a build farm, and quickly switch between them.

Setup Builder instance

1. Using the QEMU emulation support in the kernel

sudo apt-get install qemu-user-static -y
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes i

2. Command to create new builder instances.

docker buildx create --name multibuilder

Output

multibuilder

3. To show the builder instance created

docker buildx ls

Output

NAME/NODE       DRIVER/ENDPOINT             STATUS  PLATFORMS
multibuilder *  docker-container                    
  multibuilder0 unix:///var/run/docker.sock running linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
default         docker                              
  default       default                     running linux/amd64, linux/386

4. Inspect the builder

docker buildx inspect multibuilder

Name:   multibuilder
Driver: docker-container

Nodes:
Name:      multibuilder0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

Here the platforms will show the platform available to build images.

5. If the Status is not running run the following command
docker buildx inspect multibuilder --bootstrap

This will activate the Status of the builder instance.
Output


[+] Building 6.5s (1/1) FINISHED                                                                                                                                                       
 => [internal] booting buildkit                                                                                                                                                   6.5s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                                                                5.0s
 => => creating container buildx_buildkit_multibuilder0                                                                                                                           1.5s
Name:   multibuilder
Driver: docker-container

Nodes:
Name:      multibuilder0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/386, linux/arm/v7, linux/arm/v6, linux/s390x

6. Use the multibuilder instance

docker buildx use multibuilder

Build Go binary

We know that all the litmus core components (chaos experiment, operator, and runner) are written in Golang and we are building the go binaries and using them in the image.

Note:
You can build the go binaries and copy them in Dockerfile or we can build the binary in the Dockerfile itself and use them.

In this example, I'll be following the first way that is building binary and copying them in the Dockerfile.
The main key point here is to build different binaries with Go ENVs set to desired values like GOARCH=amd64 &GOOS=linux for which we have to build the image.

git clone https://github.com/litmuschaos/chaos-operator.git
cd chaos-operator
vi build/go-multiarch-build.sh

The following script will help to do that:

#!/usr/bin/env bash

package=$1
if [[ -z "$package" ]]; then
  echo "usage: $0 <package-name>"
  exit 1
fi

package_split=(${package//\// })
package_name=${package_split[-1]}

# add the arch for which we want to build the image
platforms=("linux/amd64" "linux/arm64")

for platform in "${platforms[@]}"
do
    platform_split=(${platform//\// })
    GOOS=${platform_split[0]}
    GOARCH=${platform_split[1]}
    output_name=build/_output/bin/chaos-operator$GOARCH

    # The script executes for the argument passed (in package variable)
    # here the arg will be "github.com/litmuschaos/chaos-operator/cmd/manager" for creating binary
    env GOOS=$GOOS GOARCH=$GOARCH go build -o $output_name $package

    if [ $? -ne 0 ]; then
        echo 'An error has occurred! Aborting the script execution...'
        exit 1
    fi
done

Command

./build/go-multiarch-build.sh github.com/litmuschaos/chaos-operator/cmd/manage

In the above script, we are creating the binary of github.com/litmuschaos/chaos-operator/cmd/manage with GOARCH & GOOS set in the platforms. This will create binaries as follow:

$ ls build/_output/bin/
chaos-operatoramd64*  chaos-operatorarm64*

Prepare Dockerfile for Multiarch

Prepare a Dockerfile that should work for all the different architectures. Make sure that:
1. The base image should be multi-arch: Use the base image which is having multiarch support.

FROM registry.access.redhat.com/ubi8/ubi-minimal:8.3

Which supports both ARM64 and AMD64 arch.

2. Use TARGETARCH variable for using binary in Dockerfile: All binaries used should support multi-arch builds and are built accordingly. The Docker predefines a set of ARG variables according to the --platform we pass in buildx. This is only available when using the BuildKit backend. Here the TARGETARCH is the architecture component of TARGETPLATFORM. Check out more platform env here.

Example

FROM registry.access.redhat.com/ubi8/ubi-minimal:8.3

ARG TARGETARCH

ENV OPERATOR=/usr/local/bin/chaos-operator \
    USER_UID=1001 \
    USER_NAME=chaos-operator

# install operator binary
COPY build/_output/bin/chaos-operator${TARGETARCH} ${OPERATOR}

COPY build/bin /usr/local/bin
RUN  /usr/local/bin/user_setup

ENTRYPOINT ["/usr/local/bin/entrypoint"]

USER ${USER_UID}

This will copy the binaries created in the previous step while building the image.

Build and push the multi-arch image

BuildKit is designed to work well for building for multiple platforms and not only for the architecture and operating system that the user invoking the build happens to run. When you invoke a build, you can set the --platform flag to specify the target platform for the build output, (for example, linux/amd64, linux/arm64, darwin/amd64).

A sample command to build and push the multiarch image with the Dockerfile created in the previous stage.

sudo docker buildx build --file build/Dockerfile --progress plane --platform linux/arm64,linux/amd64 --tag $(ORG_NAME)/$(IMAGE_NAME):$(IMAGE_TAG) . --push

The above command will build a multiarch image and push it to Dockerhub. The flags used are:

--file: Path of the Dockerfile (Default is ‘PATH/Dockerfile’)
--progress: Use to control verbosity. Set type of progress output (auto, plain, tty). Use plain to show container output.
--platform: Set target platform for build.
--tag: Set tag of the image in the ‘name:tag’ format.
--push: Used to push the image to the registry.

for more flags checkout buildx man page.

Multiarch image in Dockerhub:

We have successfully built and pushed the multiarch images on Dockerhub. The images will look like:

Troubleshooting

1. standard_init_linux.go:211: exec user process caused "exec format error":

There could be multiple reasons for this. Try to check out the base image you're using is supporting all the platforms for which you're building the images.

Make sure while building and copying the binary in Dockerfile that the arch and os should be same. Like you're building the binary with linux/amd64 and you're trying to run it on linux/arm64.

2. rpc error: code = Unknown desc = failed to load LLB: runtime execution on platform linux/arm/v7 not supported:

This means the platform you're looking for is not available. You can verify the available platforms using docker buildx inspect <builder-name>.Now you can run the following commands to resolve the above error.


sudo apt-get install qemu-user-static -y
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes i
docker buildx rm multibuilder
docker buildx create --name multibuilder
docker buildx ls
docker buildx inspect multibuilder
docker buildx inspect multibuilder --bootstrap
docker buildx use multibuilder
docker ps -a
docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7 -t <ORG_NAME>/<IMAGE_NAME>:<IMAGE_TAG> --push .

Conclusion

In this blog, we learn how to build and push an image that can be used at different docker architecture with the same image tag on it. This can be done using a powerful docker buildx feature which helps to build image with a single command. We can create images that can run on any desired platform using buildx and can be used to reduce your bills using ARM targets Raspberry Pis instance. So have ever tried building multiarch images? How useful it could be? Please leave your thoughts in the comment box.

🚀 Special Thanks 🙌

Thanks to Michael Fornaro for helping in suggesting, reviewing & adopting the multiarch support for Litmus Chore Components.

litmuschaos / litmus

Litmus helps SREs and developers practice chaos engineering in a Cloud-native way. Chaos experiments are published at the ChaosHub (https://hub.litmuschaos.io). Community notes is at https://hackmd.io/a4Zu_sH4TZGeih-xCimi3Q

LitmusChaos

Open Source Chaos Engineering Platform

Read this in other languages.

🇰🇷 🇨🇳 🇧🇷 🇮🇳

Overview

At a high-level, Litmus comprises of:

Chaos Control Plane: A centralized chaos management tool called chaos-center, which helps construct, schedule and visualize Litmus chaos workflows
Chaos Execution Plane Services: Made up of a…

View on GitHub

LitmusChaos GitLab Remote Templates

Udit Gaurav — Sat, 12 Sep 2020 20:32:01 +0000

In this blog, I will be talking about setting up a GitLab remote template for running a Chaos Experiment in the GitLab CI environment with LitmusChaos. Before jumping in, let's do a quick recap on Litmus. Litmus is a framework for practising chaos engineering in cloud-native environments. Litmus provides a chaos operator, a large set of chaos experiments on its hub, detailed documentation, and a friendly community. Litmus also has Github Chaos Actions and GitLab remote templates for the members who are using litmus experiments as part of their CI pipelines.

Pre-requisites:

Kubernetes 1.11 or later in your CI workflow

Brief Introduction to GitLab Remote Template

GitLab is a DevOps lifecycle tool which is extremely famous for its continuous integration and deployment pipeline features in the DevOps world nowadays which are configured using a YAML file called .gitlab-ci.yml within each project. The .gitlab-ci.yaml defines the structure and determines what to execute using GitLab Runner .It deals with what decisions to be taken for when specific conditions are encountered. Like when a process is passed or failed. In GitLab we can include an external yaml file using the include keyword these external files are called templates.

Benefits of using templates:

This helps to break down the CI/CD configuration into multiple files and increases readability for long configuration files.
It’s also possible to have template files stored in a central repository(like litmus-templates) and projects include their configuration files. This helps avoid duplicated configuration, for example, global default variables for all projects.

Different Types of Templates

The addition of a template in GitLab is a very simple job. We just need to add the include keyword that requires the external YAML file to have the extensions .yml or .yaml, otherwise the external file is not included. include supports the following inclusion methods:

local: It Includes a file from the local project repository and referenced using full paths relative to the root directory. we can use it like this:

include:
  - local: '/templates/.gitlab-chaos-template.yml'

file: We can include files from another private project under the same GitLab instance using file along with include. It can be added with:

include:
  - project: 'mayadata-io/gitlab-remote-template'
    file: '/templates/pod-delete-template.yml'

template: It used to include the pre-built GitLab template. It can be used like:

# File sourced from GitLab's template collection
include:
  - template: Auto-DevOps.gitlab-ci.yml

remote: It can be used to include a file from a different location, using HTTP/HTTPS, referenced by using the full URL.This makes the remote template very popular among all and it is widely used in many projects. The remote file must be publicly accessible through a simple GET request as authentication schemas in the remote URL are not supported. This brings LitmusChaos Remote Templates into the picture which is used to induce chaos on an application. The yaml templates are stored in a github repository and can be used as an external template.

Example:

include:
  - remote: 'https://raw.githubusercontent.com/mayadata-io/gitlab-remote-templates/master/templates/pod-delete-template.yml'

Setup LitmusChaos Remote Template in your CI workflow

If you are using/willing to use litmus experiments as part of your CI pipelines and using GitLab as your CI platform then you are at the right place. LitmusChaos introduced GitLab remote templates to integrate the chaos experiments in your CI workflow. It helps to manage your configuration file and provide a simple way to perform the chaos testing on your application. We need to follow some simple steps to setup GitLab Remote Template in your CI workflow:

1. Setup a cluster: Before we begin make sure that we have a kubernetes cluster setup in our CI workflow this is more of a pre-requisite step. If we don’t have any cloud cluster like GKE or AWS we can set up a light weight simple clusters like KinD cluster.

2. Deploy an application: Once we setup a cluster we need to have an application on which we want to perform the chaos testing. If we don’t have the application running in a pod don’t worry we can set that in these two simple steps.

i. Create an Image of the application: We need to Dockerize the application for this we need to have a Dockerfile which can be used to create the application image. Let us suppose that we have a Dockerfile at location /build from root. So we will build the image with that.

ii. Run a pod with the application container: Now as we have our application image in the Docker registry. We can deploy an application pod with that image. You can refer to a sample pod manifest that can be used to create an application pod from here replace the nginx image with your image and other attributes. Now apply the pod manifest and wait for it to come in Running state.

kubectl apply -f app.yaml

3. Select a Chaos template: Now select a chaos template from the list of templates available in the chaos template repository. Based on the experiment you want to perform, take the raw link of the template. Make sure before running any experiment we need to install litmus which can be done using the install litmus template present in the same list. A chaos template looks like:

---
variables:
  APP_NS: default
  APP_KIND: deployment 
  APP_LABEL: APPLICATION_LABEL
  TARGET_CONTAINER: TARGET_CONTAINER_NAME
  TOTAL_CHAOS_DURATION: "30"
  CHAOS_INTERVAL: "10"
  FORCE: "false"
  EXPERIMENT_IMAGE: litmuschaos/go-runner
  EXPERIMENT_IMAGE_TAG: 1.7.0

.pod_delete_template:
  image: 
    name: mayadata-io/chaos-ci-lib:ci
    entrypoint: ["./pod-delete"]
  script: 
    - echo "Running Pod Delete Experiment"

Get the raw link by clicking on the raw button.

And copy the raw link

4. Add template in .gitlab-ci-yml: Now we need to add the chaos template in the GitLab configuration file for which we need to add a remote template URL with the include keyword.

include:
  - remote: 'https://raw.githubusercontent.com/mayadata-io/gitlab-remote-templates/master/templates/node-cpu-hog-template.yml'

stages:
  - chaosInject

Inject Node CPU Hog:
  stage: chaosInject
  extends: .node_cpu_hog_template
  before_script:
  variables:
    APP_NS: default
    APP_LABEL: "run=nginx"
    APP_KIND: deployment
    TARGET_CONTAINER: nginx

Use the link which we copied in step 3 in include:remote.

5. Add the Chaos Experiment: Once we are done with setting up the chaos infra we can add the chaos experiment which we want to induce on our application in the similar way for example the addition of pod delete chaos has been added in the below code.

.gitlab-ci.yml

include:
  - remote: 'https://raw.githubusercontent.com/mayadata-io/gitlab-remote-templates/master/templates/install-litmus-template.yml'
  - remote: 'https://raw.githubusercontent.com/mayadata-io/gitlab-remote-templates/master/templates/node-cpu-hog-template.yml'
  - remote: 'https://raw.githubusercontent.com/mayadata-io/gitlab-remote-templates/master/templates/uninstall-litmus-template.yml'

stages:
  - appDeploy
  - chaosInfraSetup
  - chaosInject
  - chaosInfraCleanup

Deploy to Staging:
  stage: appDeploy
  script:
    - kubectl run nginx --image=nginx --restart=Always

Install LitmusChaos:
  stage: chaosInfraSetup
  extends: .install_litmus_template
  before_script:
  variables:
    APP_NS: default
    EXPERIMENT_IMAGE: litmuschaos/go-runner
    EXPERIMENT_IMAGE_TAG: 1.7.0

Inject Node CPU Hog:
  stage: chaosInject
  extends: .node_cpu_hog_template
  before_script:
  variables:
    APP_NS: default
    APP_LABEL: "run=nginx"
    APP_KIND: deployment
    TARGET_CONTAINER: nginx

UnInstall LitmusChaos:
  stage: chaosInfraCleanup
  extends: .uninstall_litmus_template
  before_script:
  variables:
    APP_NS: default
    EXPERIMENT_IMAGE: litmuschaos/go-runner
    EXPERIMENT_IMAGE_TAG: 1.7.0

6. Trigger the pipeline: Once we are done with the set up we can trigger the pipeline and get the results under respective stages of the pipeline. In the same way we can add and perform any chaos experiment in the GitLab CI.

Conclusion

By using GitLab Remote template we can easily integrate LitmusChaos experiment with GitLab which helps to perform the chaos testing on an application in the CI workflow. We just need to make use of the include:remote feature of GitLab and select the desired templates from the litmuschaos remote templates. So, What are your opinions on running chaos in the CI workflow? If you’re in favor of this then does it seem to be helpful? DevOps experts are welcome to comment and suggest on this!

litmuschaos / litmus

Litmus helps SREs and developers practice chaos engineering in a Cloud-native way. Chaos experiments are published at the ChaosHub (https://hub.litmuschaos.io). Community notes is at https://hackmd.io/a4Zu_sH4TZGeih-xCimi3Q

LitmusChaos

Open Source Chaos Engineering Platform

Read this in other languages.

🇰🇷 🇨🇳 🇧🇷 🇮🇳

Overview

At a high-level, Litmus comprises of:

Chaos Control Plane: A centralized chaos management tool called chaos-center, which helps construct, schedule and visualize Litmus chaos workflows
Chaos Execution Plane Services: Made up of a…

View on GitHub

Setup GitHub Chaos Action using KinD Cluster [Part-2]

Udit Gaurav — Fri, 07 Aug 2020 13:27:21 +0000

In this blog I’ll be talking about setting up GitHub Chaos Action using KinD cluster in your GitHub workflow, to know more about GitHub Chaos Actions and how to get started with chaos actions please refer Part-1 of the action series. This blog deals with automating the workflow by creating a kind cluster inside the CI workflow only in place of connecting an external cluster as we did in Part-1. Before jumping in, let's do a quick recap on Litmus. Litmus is a framework for practicing chaos engineering in cloud-native environments. Litmus provides a chaos operator, a large set of chaos experiments on its hub, detailed documentation, and a friendly community. Litmus is very easy to use; you can also set up a very quick demo environment to install and run Litmus experiments.

Pre-requisites:

GO
Docker

Note: The above is only required when you're not using the default runner otherwise the ubuntu-latest image has all the dependencies installed in it.

Brief Introduction to GitHub Chaos Action

GitHub Chaos Action helps you to automate the chaos testing in a Cloud Native way using Github Actions. It contains a number of litmuschaos experiments which will help to find the weaknesses or improvements required for your application.

We can also say that the Github chaos action is used to create custom software development life cycle (SDLC) workflows directly in your GitHub repository.To know more about setting up a sample workflow using GitHub chaos action please visit the Part-1 of the blog.

Why KinD Cluster?

KinD (Kubernetes in Docker) was primarily designed for testing Kubernetes itself, but now it has become very famous to be used for local development or CI. But “Why KinD?” You might ask this question to yourself while setting up GitHub CI with chaos action to answer this let’s do a small analysis of what is required.

Easily Setup: We mostly use GitHub default runner as it has almost all the capabilities installed in it to run a simple workflow and we don’t need an external runner until or unless we have a larger requirement. The Github Ubuntu image of the default runner comes preinstalled with kind so we just need to spin up a cluster with a configuration file or with default values. In this blog I will be using default values to setup KinD which will create a single node cluster with named ‘kind’
Compatible: We need a cluster which is very lightweight and easy to use. For this purpose, KinD fits the best. KinD is a tool for running local Kubernetes clusters using Docker container “nodes”.

These are enough reasons to use kind in our CI workflows.

New trend in Continuous Integration/Deployment:

One of the new trends in Continuous Integration/Deployment is to:

Create an application image.
Run tests against the created image.
Push image to a remote registry.
Deploy to a server from the pushed image.

It’s also useful when your application already has the Dockerfile that can be used to create and test an image.

We will create a Github CI workflow covering the above stages on a particular PR for that we need to follow the following ten simple steps:

Step-1: Setup a fresh workflow in case you don’t have one

Let’s get started with writing a GitHub CI YAML if you don’t have it already. We are now familiar with the fields and attributes of CI YAML from Part-1, so we will directly move toward steps involved in setting up the actions with the following template:

name: Litmus-CI
on:
  # Trigger the workflow on push or pull request,
  # but only for the master branch
  push:
    branches:
      - master
jobs:
  # Job name
  choas-tests:
    runs-on: ubuntu-latest
    steps:

Step-2: Checkout the latest commit on the Pull Request

When you commit code to your repository, you can continuously build and test the code to make sure that the commit doesn't introduce errors. The error could be in the form of some security issue, functional issue or performance issue which can be tested using different custom tests, linters or by pulling actions. This brings the need of having Chaos Actions which will perform a chaos test on the application over a particular commit which in-turn helps to track the performance of the application on a commit level. The following lines of code will help you checkout on the latest commit on the PR:

#Using the last commit id of pull request
      - uses: octokit/request-action@v2.x
        id: get_PR_commits
        with:
          route: GET /repos/:repo/pulls/:pull_number/commits
          repo: ${{ github.repository }}
          pull_number: ${{ github.event.issue.number }}
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: set commit to output
        id: getcommit
        run: | 
           prsha=$(echo $response | jq '.[-1].sha'  | tr -d '"')
           echo "::set-output name=sha::$prsha" 
        env: 
          response:  ${{ steps.get_PR_commits.outputs.data }}

      - uses: actions/checkout@v2
        with:
          ref: ${{steps.getcommit.outputs.sha}}

Just add these lines under steps in list format and we are on the latest commit on the PR.

Step-3: Create an Image of the application

In this stage we will Dockerize the application from the repository. For this we need to have a Dockerfile in the repository which can be used to create the application image. Here, let us suppose that we have a Dockerfile at location
/build from root. So for creating a Docker image of this application we need to run following command in CI YAML:

     - name: Build docker image
        run: |
          sudo docker build -f build/Dockerfile -t <image-name>:<image-tag>

Step-4: Install and configure KinD Cluster

As we discussed for running Chaos Actions we need to have a cluster and we will be creating a KinD cluster in the CI workflow only. As we’re using Ubuntu-latest image which already contain the kind installation we need to spin a cluster. For spinning the latest version of the KinD cluster we just need to run a kind create cluster command or for a specific version we need to add an action for that. For this blog I’ll be using kind 0.7.0 version which can be installed from the following line of code:

      #Install and configure a kind cluster
      - name: Installing Prerequisites (KinD Cluster)
        uses: engineerd/setup-kind@v0.5.0
        with:
            version: "v0.7.0"

      - name: Configuring and testing the Installation
        run: |
          kubectl cluster-info --context kind-kind
          kind get kubeconfig --internal >$HOME/.kube/config
          kubectl get nodes

Step-5: Load the image inside the node of the cluster

After creating the cluster we need to load the application image which was built in step 3 in the node of the cluster to be used locally for testing.

    - name: Load image on the nodes of the cluster
        run: |
          kind load docker-image --name=kind <image-name>:<image-tag

Step-6: Run a pod with the application container

Now we will run the application container in a pod. Different chaos testing on the application level and on the node level will be performed and the health of the pod will be verified after inducing chaos. You can refer to a sample pod manifest which can be used to create an application pod from here replace the nginx image with your image and place it at some location (say /path/to/application) in your repository. In this step we will create the application pod via the following command and wait for a few seconds to get it ready.

     - name: Deploy a sample application for chaos injection
        run: |
          kubectl apply -f /path/to/application
          sleep 30

Step-7: Setup kubeconfig ENV for GitHub Actions

Now we need to export KUBE_CONFIG_DATA containing kubeconfig of the the KinD cluster encoded in base64 to be used by Chaos Action.

      - name: Setting up kubeconfig ENV for Github Chaos Action
        run: echo ::set-env name=KUBE_CONFIG_DATA::$(base64 -w 0 ~/.kube/config)

Step-8: Run GitHub Chaos Actions on the application

Here comes the step where we will use GitHub Chaos Actions for chaos testing. The experiments and parameters are controlled by actions ENV as explained in the Chaos Action page.

Example:

      - name: Running Litmus pod delete chaos experiment
        if: startsWith(github.event.comment.body, '/run-e2e-pod-delete') || startsWith(github.event.comment.body, '/run-e2e-all')
        uses: mayadata-io/github-chaos-actions@v0.1.1
        env:
          INSTALL_LITMUS: true
          EXPERIMENT_NAME: pod-delete
          EXPERIMENT_IMAGE: litmuschaos/ansible-runner
          EXPERIMENT_IMAGE_TAG: ci
          IMAGE_PULL_POLICY: IfNotPresent
          LITMUS_CLEANUP: true

Step-9: Push the application image to the registry

This step will execute only when all other steps run fine. This means our application has passed the chaos test and now we are good to push the image in the registry.

      - name: Publish to Docker Repository
        uses: elgohr/Publish-Docker-Github-Action@master
        with:
          Name: <image-name>
          username: ${{ secrets.DUSER }}
          password: ${{ secrets.DPASS }}
          dockerfile: build/Dockerfile
          tags: "<image-tag>"

Step-10: Delete Kind Cluster

This is basically a cleanup step where we remove the cluster so the application pod and chaos components also get removed. This should In case some experiment fails, even then the cleanup stage should execute.

      - name: Deleting KinD cluster
        if: ${{ always() }}
        run: kind delete cluster

So the complete workflow YAML looks like:

main.yml

name: Litmus-CI
on:
 push:
 branches: [ master ]

jobs:
  tests:
    runs-on: ubuntu-latest
    steps:

      #Using the last commit id of pull request
      - uses: octokit/request-action@v2.x
        id: get_PR_commits
        with:
          route: GET /repos/:repo/pulls/:pull_number/commits
          repo: ${{ github.repository }}
          pull_number: ${{ github.event.issue.number }}
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: set commit to output
        id: getcommit
        run: | 
           prsha=$(echo $response | jq '.[-1].sha'  | tr -d '"')
           echo "::set-output name=sha::$prsha" 
        env: 
          response:  ${{ steps.get_PR_commits.outputs.data }}

      - uses: actions/checkout@v2
        with:
          ref: ${{steps.getcommit.outputs.sha}}      

      - name: Build docker image
        run: |
          sudo docker build -f build/ansible-runner/Dockerfile -t litmuschaos/ansible-runner:ci .
      #Install and configure a kind cluster
      - name: Installing Prerequisites (KinD Cluster)
        uses: engineerd/setup-kind@v0.4.0
        with:
            version: "v0.7.0"

      - name: Configuring and testing the Installation
        run: |
          kubectl cluster-info --context kind-kind
          kind get kubeconfig --internal >$HOME/.kube/config
          kubectl get nodes   

      - name: Load image on the nodes of the cluster
        run: |
          kind load docker-image --name=kind litmuschaos/ansible-runner:ci
      - name: Deploy a sample application for chaos injection
        run: |
          kubectl apply -f https://raw.githubusercontent.com/mayadata-io/chaos-ci-lib/master/app/nginx.yml
          sleep 30

      - name: Setting up kubeconfig ENV for Github Chaos Action
        run: echo ::set-env name=KUBE_CONFIG_DATA::$(base64 -w 0 ~/.kube/config)

      - name: Running node-memory-hog chaos experiment
        uses: mayadata-io/github-chaos-actions@v0.1.1
        env:
          INSTALL_LITMUS: true
          EXPERIMENT_NAME: node-memory-hog
          EXPERIMENT_IMAGE: litmuschaos/ansible-runner
          EXPERIMENT_IMAGE_TAG: ci
          IMAGE_PULL_POLICY: IfNotPresent
          LITMUS_CLEANUP: true


      - name: Publish to Docker Repository
        uses: elgohr/Publish-Docker-Github-Action@master
        with:
          Name: <image-name>
          username: ${{ secrets.DUSER }}
          password: ${{ secrets.DPASS }}
          dockerfile: build/Dockerfile
          tags: "<image-tag>"

      - name: Deleting KinD cluster
        if: ${{ always() }}
        run: kind delete cluster

View the Result:

Now to view the result of the run we need to navigate to the main page of the repository. Under your repository name, click Actions.

List of all workflows will appear. Out of these workflows select the latest one to view the logs and get an idea what happen during the execution of the chaos test.

The complete log will look like:

If you want to create and use your own experiment/test in the GitHub Chaos Action then you create it using litmus with the help of cool SDK or for more help you can also raise an issue in litmus for that. The community is extremely active and surely they will get back to you with some good help.

NOTE: If you're new to GitHub Actions and wanted to create a simple GitHub workflow then please also read the Part-1 of the action series.

Conclusion:

By using GitHub Chaos Actions along with KinD in the CI workflow automate, customize, and execute your software development workflows right in your repository. You can use the Chaos Action in a completely customized way to check the performance of your application for every commit coming in the Pull Request. This will help both developers and testers to maintain the quality of the software. So, what do you think? What else can we achieve using Chaos Actions and what other tests can be performed apart from what we have now? Kubernetes experts are welcome to comment and suggest on this!

litmuschaos / litmus

Litmus helps SREs and developers practice chaos engineering in a Cloud-native way. Chaos experiments are published at the ChaosHub (https://hub.litmuschaos.io). Community notes is at https://hackmd.io/a4Zu_sH4TZGeih-xCimi3Q

LitmusChaos

Open Source Chaos Engineering Platform

Read this in other languages.

🇰🇷 🇨🇳 🇧🇷 🇮🇳

Overview

At a high-level, Litmus comprises of:

Chaos Control Plane: A centralized chaos management tool called chaos-center, which helps construct, schedule and visualize Litmus chaos workflows
Chaos Execution Plane Services: Made up of a…

View on GitHub

GitHub Chaos Actions in Your CI/CD workflow [Part-1]

Udit Gaurav — Tue, 14 Jul 2020 12:54:01 +0000

``In this blog, I will be talking about setting GitHub Chaos Actions in your CI workflow. Before jumping in, let's do a quick recap on Litmus. Litmus is a framework for practicing chaos engineering in cloud-native environments. Litmus provides a chaos operator, a large set of chaos experiments on its hub, detailed documentation, and a friendly community. Litmus is very easy to use you can also set up a very quick demo environment to install and run Litmus experiments.

What is Github Chaos Action?

Github actions automate the chaos execution on an application in the same place where the code is stored. You can write individual tasks along with chaos actions and combine them to create a custom GitHub workflow. GitHub Workflows are custom automated processes that you can set up in your repository to build, test, package, or deploy any code project on GitHub. Including the Github chaos actions in your workflow YAML, you can test the performance/resiliency of your application in a much simpler and better way. To know more visit Github chaos actions repository.

What do we need Github Chaos Action?

GitHub Action makes the execution of chaos on an application in a much simpler and easier way. This action could be triggered on the basis of parameter listed in the GitHub workflow YAML. This action can be performed over an application that can be deployed in the cluster or already present in the cluster. Github chaos action helps to fix the weaknesses leads to increased resilience of the system which helps to gain confidence to developers and SRE. It takes a cloud-native approach to create, manage, and monitor the chaos.

Pre-requisites

Kubernetes 1.11 or later.
A workflow in your Github Repository
Application pod in a healthy state
Pre-requisites of the Chaos Experiment you are executing. You can find this under the experiment section in litmus docs

A Sample Github Chaos Action Workflow
The workflow file is stored under .github/workflows directory of your repository and it uses YAML syntax and must have either a .yml or .yaml file extension. We just need to follow the below-mentioned steps to create a sample GitHub workflow for performing chaos engineering on a Kubernetes application using Github chaos action.

1. Setup name and event to trigger the GitHub workflow
`yaml name: Push on: push: branches: [ master ] pull_request: branches: [ master ] `
This will trigger the GitHub workflow on a push or pull request basis on the master branch. Similarly, it can also be on a scheduled basis.
`yaml
on:
schedule:

cron: ‘/15 * * * *’ ` **2. Setup jobs in the workflow* `yaml jobs: build: runs-on: ubuntu-latest ` A GitHub workflow run is made up of one or more jobs. Each job runs in an environment specified by runs-on. So in the above case, each job will run on the ubuntu-latest environment. By default jobs run in parallel to run them sequentially, we need to add `yaml needs: <Job-Name>. jobs: build: runs-on: ubuntu-latest needs: job1 steps: — uses: actions/checkout@v2 ` 3. Setup different tasks in a job A job contains a sequence of tasks called Steps. Steps can be used to run commands, tasks, or actions itself. Not all steps run actions, but all actions run as a step. `yaml jobs: build: runs-on: ubuntu-latest steps:
- uses: actions/checkout@master Using job steps we can run the chaos action over an application and for doing so we need to use our [Github-chaos-actions](https://github.com/marketplace/actions/kubernetes-chaos) in a below-mentioned way: A sample pod deletes experiment workflow:yaml .github/workflows/main.yml name: main on: push: branches: [ master ] jobs: build: runs-on: ubuntu-latest steps: — uses: actions/checkout@master

— name: Running Litmus pod delete chaos experiment
uses: mayadata-io/github-chaos-actions@master
env:
##Pass kubeconfig data from secret in base 64 encoded form
KUBE_CONFIG_DATA: ${{ secrets.KUBE_CONFIG_DATA }}
##If litmus is not installed
INSTALL_LITMUS: true
##Give application info under chaos
APP_NS: default
APP_LABEL: run=nginx
APP_KIND: deployment
EXPERIMENT_NAME: pod-delete
##Custom image can also been used
EXPERIMENT_IMAGE: litmuschaos/ansible-runner
EXPERIMENT_IMAGE_TAG: latest
IMAGE_PULL_POLICY: Always

TOTAL_CHAOS_DURATION: 30
CHAOS_INTERVAL: 10
FORCE: false
##Select true if you want to uninstall litmus after chaos
LITMUS_CLEANUP: true
`
This is a complete single job that is running a chaos action over an application for every push on branch master and the kubeconfig of the cluster and application details are provided in the form of secrets and environment variables respectively. Other Variables are chaos action variables that change on the basis of the chaos experiment we are running.

How to connect the cluster with chaos actions?

To connect a healthy cluster with the chaos action we need to pass the kubeconfig of the cluster in the base 64 encoded form as done in the above example. For doing this we need to follow these steps: Convert the kubeconfig of your cluster in base 64 encoded form using this command:

`bash cat $HOME/.kube/config | base64 `

Add a secret KUBE_CONFIG_DATA with the value containing the output of the above command.
Pass this secret as ENV in Github chaos actions to connect your cluster with the action. `yaml
name: Running Litmus pod delete chaos experiment uses: mayadata-io/github-chaos-actions@master env: KUBE_CONFIG_DATA: ${{ secrets.KUBE_CONFIG_DATA }} `
Creating a Cluster in the workflow itself is also one of the ways of performing chaos actions. We can even create a cluster in one of the steps of the job and then use Github-chaos-actions in further steps. In this case we don’t need to pass the kubeconfig externally through action’s ENV instead you can setup kubeconfig as an environment variable in base 64 encoded form using: `yaml
name: "Setup kubeconfig ENV for Github Chaos Action" run: echo ::set-env name=KUBE_CONFIG_DATA::$(base64 -w 0 ~/.kube/config) TheKUBE_CONFIG_DATA` will be accessable to the Chaos Action which is running on a different container. Please visit [Part-2] of the GitHub actions series, we have discussed it in more detailed way.

Monitor the GitHub workflow jobs:

Once the workflow triggers we need to monitor the workflow. For monitoring the different jobs in the workflow:

Click on the Actions option present in the repository containing workflow.
It will show all the workflow runs. Select the one which triggered recently
Check the job logs of the workflow.

Details of chaos action run

To get the details of what happened when the chaos action was running check the logs on the Github-chaos-actions from the job logs.
The Github-chaos-action contains the logs of the process involved in running the chaos actions.
It also contains the logs of the experiment pod which will give a verbose idea of what was happening at the time of chaos injection.
At the end of the logs, there will be a check of the verdict whether the chaos result verdict is Pass or Fail (It will be a pass for the successful execution of the experiment).
To know more about the terms like Chaos Experiment, Chaos Engine and Chaos Result visit litmus docs.

To know more about GitHub Chaos Actions and different ways to setup Chaos Actions in your GitHub workflow read Part-2 now.

Conclusion:

We can automate the chaos execution on a review application to check its resiliency in the same place where the codes are stored using Github Action.It provides an easier way to perform different chaos experiments which helps in gaining confidence.

litmuschaos / litmus

Litmus helps SREs and developers practice chaos engineering in a Cloud-native way. Chaos experiments are published at the ChaosHub (https://hub.litmuschaos.io). Community notes is at https://hackmd.io/a4Zu_sH4TZGeih-xCimi3Q

LitmusChaos

Open Source Chaos Engineering Platform

Read this in other languages.

🇰🇷 🇨🇳 🇧🇷 🇮🇳

Overview

At a high-level, Litmus comprises of:

Chaos Control Plane: A centralized chaos management tool called chaos-center, which helps construct, schedule and visualize Litmus chaos workflows
Chaos Execution Plane Services: Made up of a…

View on GitHub

Get Started with LitmusChaos in Minutes

Udit Gaurav — Sun, 21 Jun 2020 13:03:27 +0000

In this blog, I will be talking about setting up a quick demo environment for Litmus. Before jumping in, let's do a quick recap on Litmus. Litmus is a framework for practicing chaos engineering in cloud-native environments. Litmus provides a chaos operator, a large set of chaos experiments on its hub, detailed documentation, and a friendly community. Litmus is very easy to use but a quick demo environment to install Litmus, run experiments, and learn chaos engineering on Kubernetes will be of course cooler.

Litmus Demo Environment

The Litmus Demo is a quick way to introduce yourself to the world of Cloud-Native Chaos Engineering. It helps you to familiarize yourself with running LitmusChaos experiments in a realistic application environment running multiple services on a Kubernetes cluster. By following the instructions in this blog, you will be able to create a cluster, install a sample application that will be subjected to chaos, build the chaos infrastructure & run the chaos experiments, all in a matter of minutes.

When, as a community, we were pondering about a quick, lightweight environment to achieve this automated “Demo” setup, we just couldn’t look beyond KinD (cluster infra) & the immensely popular Sock-Shop (sample microservices application). However, the demo script also provides for a more “extensive” platform in GKE, in case you want to explore the larger suite of experiments.

The idea here, as you may have guessed, is to get down to experiencing chaos injection without brooding over documentation & copy-pasting multiple Kubernetes manifests. This demo is designed to help get your hands dirty on injecting failures using a chaos framework & finding out what happens and not so much about inculcating in you, in-depth knowledge of standard practices around chaos engineering. We are hoping this will get you there eventually!

LitmusChaos PreRequisites

Docker, Kubectl & Python3.7+ (with the PyYaml package) are all you need for running the KinD platform based chaos demo. If GKE is your platform choice, you may need to configure gcloud on your workstation (or test-harness machines, if you are old fashioned!).

Know Your Cluster

As described earlier, this demo environment supports different platforms (KinD, GKE).

KinD: A 3 node KinD cluster pre-installed with the sock-shop demo application, litmus chaos CRDs, operator, and a minimal set of prebuilt chaos experiment CRs is setup.
GKE: A 3 node GKE cluster pre-installed with the sock-shop demo application, litmus chaos CRDs, operator, and the full generic Kubernetes chaos experiment suite is setup.

NOTE:The support for other platforms like AWS will be added very soon

Getting started:

To get started with any of the above platforms, we will follow the following steps.

1. Clone Litmus demo repository in your system.

This command will clone the master branch of the litmus demo repository in your system.

git clone https://github.com/litmuschaos/litmus-demo.git

2. Check out the available options on the demo script

 cd litmus-demo
./manage.py -h

Output:

usage: manage.py [-h] {start,test,list,stop} ...

Spin up a Demo Environment on Kubernetes.

positional arguments:
  {start,test,list,stop}
    start               Start a Cluster with the demo environment deployed.
    test                Run Litmus ChaosEngine Experiments inside litmus demo
                        environment.
    list                List all available Litmus ChaosEngine Experiments
                        available to run.
    stop                Shutdown the Cluster with the demo environment
                        deployed.

optional arguments:
  -h, --help            show this help message and exit

So, we can see that the available arguments are start, test, list, and stop with their usage.

3. Check the available experiments on the desired platform

You can also view the experiments supported by the platform, as of today (The folks in the litmus community are busy extending the platform interoperability for the chaos experiment, so stay tuned!).

For KinD platform

./manage.py list --platform kind

NOTE: The default value of --platform is kind so ./manage.py list will also give the same output.

OUTPUT:


Available Litmus Chaos Experiments:


    1. pod-delete
    2. container-kill
    3. node-cpu-hog
    4. node-memory-hog

For GKE platform

./manage.py list --platform GKE

OUTPUT:

Available Litmus Chaos Experiments:


    1. container-kill
    2. disk-fill
    3. node-cpu-hog
    4. node-memory-hog
    5. pod-cpu-hog
    6. pod-delete
    7. pod-memory-hog
    8. pod-network-corruption
    9. pod-network-latency
    10. pod-network-loss

4. Installing Demo Environment

Install the demo environment using one of the platforms with start argument:

KinD Cluster:

./manage.py start --platform kind

Once done, wait for all the pods to get in a ready state. You can monitor this using.

watch kubectl get pods --all-namespaces

Now when all pods come into Running state, we can access the sock-shop application through web-ui, which will help us to visualize the impact of chaos on the application and whether the application persists after chaos injections.

Follow the given steps to access through web-ui.

Get the port of frontend deployment:

kubectl get deploy front-end -n sock-shop -o jsonpath='{.spec.template.spec.containers[?(@.name == "front-end")].ports[0].containerPort}'

OUTPUT:
8079

Perform port forwarding on the port obtained above

 kubectl port-forward deploy/front-end -n sock-shop 3000:8079

OUTPUT:

Forwarding from 127.0.0.1:3000 -> 8079
Forwarding from [::1]:3000 -> 8079

Copy the IP to a web browser and get the web-ui of sock-shop using a kind cluster.

GKE CLuster

./manage.py start --platform GKE

Once done, you will get an output containing Ingress Details:

Ingress Details:

** RUNNING: kubectl get ingress basic-ingress --namespace=sock-shop
NAME            HOSTS   ADDRESS        PORTS   AGE
basic-ingress   *       34.120.15.86   80      95s

You can access the web application in a few minutes at http://34.120.15.86

After waiting for a few minutes, you can check the given URL (here http://34.120.15.86) it will take you to the sock-shop catalog UI on the localhost browser. This makes sure that the sock-shop application is deployed in the cluster-infra. The sock-shop catalog UI will look like:

To check all pods are ready, run the following command and check the status of all pods which should be in Running state.

watch kubectl get pods --all-namespaces

5. Running Chaos in Demo Environment

We can run all the chaos experiments we get from ./manage.py list or we can also run selective experiments that only we want to perform using ./manage test as shown:

For running all experiments

./manage.py test --platform <platform-name>

For running selective experiments

./manage.py test  --platform <platform-name> --test <test-name>

Follow step 3 to get the available test-name.

Example: For running the pod-delete experiment.

**./manage.py test  --platform kind --test pod-delete **

Take this opportunity to view what happens as the experiment runs its course. Pods may be killed; containers may be restarted, the resource usage may spike up - all impacting the application in its own unique way. In a typical run of the chaos experiment, a hypothesis is built around each of these failures. While we discuss this in detail in upcoming blogs, the endeavor in our case is to find out if the sock-shop app continues to stay available across these failures!!

The experiment results (Pass/Fail) are derived based on the simple criteria of app availability post chaos & are summarized on the console once the execution completes.

Get more details about the flags used to configure and run the chaos tests please refer to the param tables in the test section.

6. Deleting Cluster / Cluster Clean up

To shut down and destroy the cluster when you're finished, run the following commands:

KinD cluster

./manage.py --platform kind stop

GKE cluster

./manage.py --platform GKE stop --project {GC_PROJECT}

Conclusion

Did you get a taste of chaos in a Kubernetes environment? Did you end up killing different microservices to gauge the impact? Are you inclined to understand what happened when the failures were injected & what options the chaos framework provides to control & assess these? Did you make a mental map of how your microservices might behave if the same failures were injected on them? And about how to mitigate the impact of these failures?

If you asked yourself these questions, then the purpose of this litmus-demo is met. It also means you are onwards and upwards in your chaos engineering journey!!

litmuschaos / litmus

Litmus helps SREs and developers practice chaos engineering in a Cloud-native way. Chaos experiments are published at the ChaosHub (https://hub.litmuschaos.io). Community notes is at https://hackmd.io/a4Zu_sH4TZGeih-xCimi3Q

LitmusChaos

Open Source Chaos Engineering Platform

Read this in other languages.

🇰🇷 🇨🇳 🇧🇷 🇮🇳

Overview

At a high-level, Litmus comprises of:

Chaos Control Plane: A centralized chaos management tool called chaos-center, which helps construct, schedule and visualize Litmus chaos workflows
Chaos Execution Plane Services: Made up of a…

View on GitHub