DEV Community: Neelanjan Manna

Extending kubectl Utility With Plugins

Neelanjan Manna — Thu, 15 Sep 2022 15:54:18 +0000

Introduction

We can all agree on the fact that Kubernetes is quite expansive. It has become the de facto tool for cloud-native application deployment because of its flexibility. One often uses the kubectl CLI to interact with their Kubernetes cluster, and while kubectl is quite extensive in itself with regard to all the operations you can perform with it, its usability can be further extended using plugins.

kubectl plugins can extend the usability of the CLI tool by adding functional capabilities specific to a Kubernetes application or in general providing more features that can be accessed as kubectl sub-commands. The best part is it does not require editing kubectl’s source code or recompiling it, thereby making the plugins truly modular in the sense that they can be simply plugged in and used.

Installing Plugins

Although there’s more than one way to distribute and install kubectl plugins, the simplest way is to use krew. It is a package manager for kubectl and makes the installation and management of kubectl plugins a cakewalk. To get started, you would have to first install krew on your machine. You can refer to this installation document to do so.

Once installed, you shall first index all the available krew plugins, a list of which can be accessed here. To do so, run the following command:

kubectl krew update

Did you notice how krew itself is a kubectl plugin as well? A plugin to manage all other plugins! If you’re willing to, you can check its source code here.

Now, you can view the list of available plugins with the following command:

kubectl krew search

To install any plugin, say the whoami plugin, run the following command:

kubectl krew install whoami

Once installed, you can simply follow the usage guide to run the plugin commands as such:

kubectl whoami

You can also update, uninstall, and add a custom index for the plugins using krew among other things, for which you can refer to the documentation.

Developing Plugins

Developing plugins for kubectl is a fairly simple process, if we abstract out the business logic of the plugin. The plugin itself is nothing but a standalone CLI application that uses appropriate commands to achieve its intended functionalities. So the question remains how do we add this standalone CLI as a kubectl plugin? Let’s find out.

I have made a demo plugin called kubectl-count, which we will be using for this small demo. The complete source code can be found here: https://github.com/neelanjan00/kubectl-count. In a nutshell, this plugin allows you to count the instances of Kubernetes resources present in your cluster. So for example, you can count the number of pods in a namespace, the number of nodes in your cluster, or the total number of deployments in all the namespaces of your cluster.

The source code of the plugin is pretty straightforward, we use the Cobra CLI to create a boilerplate CLI project.

Then, we add the commands for each of the supported Kubernetes resources for which we will count the resources. We do so using the Kubernetes client-go library. For example, this is the logic for the command for counting the number of pods.

We are using flags here for specifying the namespace, the label selector, and all-namespaces, just as you would do in kubectl. Also, the aliases for the pods command such as po and pod are also available to be used. Similarly, we have created definitions for other commands as well, corresponding to the resources that they represent.

Lastly, if you’re wondering how we’re obtaining the Kubernetes client here, we use the existing .kubeconfig file in the machine and use it to generate a client.

We can simply run this CLI program as it is like the following:

go run kubectl-client.go po

The above command should give the number of pods in your default namespace, given that you have a valid Kubernetes cluster config file on your machine.

Now that we have our plugin ready, how shall we use it along with kubectl? To do so, first, we need to build a binary for this go program and then we need to rename the binary file (or an executable shell file, in case you want to make your plugins in bash) file such that the root sub-command of the plugin id preceded by kubectl-. For example, since the root sub-command for this plugin is count, the name of the binary will be kubectl-count. After this, we just need to move this file to the /usr/local/bin directory.

The naming convention we used here will allow kubectl to search for this plugin and any other plugin that you will be installing, given that the directory /usr/local/bin is in your system path. Now you can simply use this plugin to run the command that you had previously run to count the pods like so:

kubectl count po

Conclusion

kubectl plugins are an excellent way to increase the usability of the CLI tool. While there’s an ever-soaring list of plugins that you can simply plug and play using krew, it’s also pretty feasible to develop your own plugins which can abridge the lack of any functionality in kubectl without complicating the developer experience or the end-user experience.

How to Install Drone CI Server in Kubernetes

Neelanjan Manna — Fri, 09 Sep 2022 10:59:07 +0000

In this blog, we’ll setup a Drone CI server in Kubernetes using Helm. If you’re a beginner and are lost in the labyrinths of the documentations and GitHub READMEs, this simple blog will bootstrap you in just a few minutes.

Introduction

Drone by Harness is a continuous integration service that enables you to conveniently set up projects to automatically build, test, and deploy as you make changes to your code. Drone integrates seamlessly with Github, Bitbucket and Google Code as well as third party services such as Heroku, Dotcloud, Google AppEngine and more.

While Drone also has a cloud SaaS platform, we’ll be focusing on setting up the hosted version on Kubernetes. By default, Drone CI is configured to run as a Docker container and is meant to be setup in a server or a VM. In fact, all the CI build steps are performed using individual Docker containers.

However it is also possible to setup the Drone CI server on a Kubernetes cluster. Drone also provides a Helm chart to do so, which we will be using here. However this mode of installation is not present in the primary user documentation, hence the requirement of this blog. The Helm chart we will be using is present in this GitHub repository.

Pre-Requisites

A Kubernetes Cluster: Any Kubernetes cluster of sufficient capacity can be used. I will be using a GKE cluster of three e2-micro VMs for the setup.
Helm: Ensure that you have Helm installed on your local machine.
GitHub Account: For this blog, we’ll be using GitHub for creating an OAuth application for the Drone server.
kubectl: We’ll be using kubectl to access some of the Kubernetes cluster info.

Step-1: Setting up a GitHub OAuth Application for Drone

Before deploying Drone in Kubernetes, we need to create an OAuth application so that the users can authorise access of their GitHub repositories to Drone. We are using GitHub for this purpose here, however this can be achieved using other major Git hosting service providers as well like BitBucket, GitLab among others. Checkout the docs for the full list.

Go to https://github.com/settings/developers and create a new OAuth application and choose New OAuth App.

Put application name as Drone.

For the Homepage URL, we need to provide the Drone server application endpoint. Depending on how we’re configuring the access to the application in Kubernetes i.e. either using a NodePort or a LoadBalancer type of service, we will obtain the URL.
We’ll be using a NodePort type of service for the purpose of this demo. Currently, we don’t have the Drone server deployed in Kubernetes and hence the NodePort service isn’t available to us just yet. However, we can choose an unused NodePort value right away which will be later used for the server deployment. I will be using the port 32000, as an example.

With the NodePort value decided upon, we can simply use it with the external IP of any of the Kubernetes nodes to obtain the Homepage URL. In other words, this will be the endpoint of the Drone server once we install it. To get the external IP of the nodes, we can use the following command:



kubectl get nodes -o wide | awk '{ print $7 }'

You’ll obtain a similar result:



EXTERNAL-IP
35.238.118.197
34.68.253.216
35.222.5.86

You may choose any one of the external IPs available to you. For this demo I’ll be using the 35.238.118.197 IP. Hence, our Homepage URL becomes http://35.238.118.197:32000 and for the authorisation callback URL, we will use http://35.238.118.197:32000/login. Optionally, feel free to add any description that seems fit to you.

Once done, choose Register application. This should register your Drone application with the GitHub.

Now all we need to do is obtain a client secret key so that it can be used by Drone to authorize the login requests via GitHub OAuth. To do so, choose Generate a new client secret. This should prompt GitHub to authenticate you so that client secret key creation can be validated. Once done, you’ll obtain the secret key. Copy and save the key somewhere else as you won’t be able to access it from GitHub the next time, when we’ll make use of it to configure the Drone server. Also take a note of the client ID.

Step-2: Configuring the Drone server Helm chart

Before we configure the deployment options, we will add the Helm repo. To do so, execute the following commands:



helm repo add drone https://charts.drone.io

helm repo update

Once done, we can configure the server deployment options. To do so, we’ll use a YAML file, which can be obtained here. Download and save the file by the name values.yaml and open it using any text editor. We’ll be doing a minimal configuration setup in this demo, though you can explore all the other configuration options as well.

First, we’ll modify the Kubernetes service to be used for the server deployment. As decided earlier, we’ll be using a NodePort type of service, therefore service.type field value will be NodePort. Also, for the service.nodePort field, we will use the value 32000.



...
service:
  type: NodePort
  port: 8080
  annotations: {}
  nodePort: 32000
...

Next, we’ll configure the Drone specific options, which will be provided as environment variables. The first env is DRONE_SERVER_HOST, for which we will provide the external IP that we had selected earlier. Ensure that the IP address value is wrapped in quotes, to specify it as a string.

For the DRONE_SERVER_PROTO env, provide the value http as we will be using the HTTP protocol for the Drone server requests and runner polling.

For the DRONE_RPC_SECRET env, we need to provide a secret value that the Drone server and runners will share and use for authenticating the requests. To generate this value, you may use the following command to get a 32 character length secret:



openssl rand -hex 16

You can then provide this value as a string for the env.

Lastly, we will provide the GitHub OAuth application client ID and client secret key which we had created in the first step for the DRONE_GITHUB_CLIENT_ID and DRONE_GITHUB_CLIENT_SECRET envs respectively.



...
env:
  DRONE_SERVER_HOST: "35.238.118.197"
  DRONE_SERVER_PROTO: http
  DRONE_RPC_SECRET: "727b7fe17e8de56689f46943c76d25f8"

  DRONE_GITHUB_CLIENT_ID: 620a9d86236b7470558a
  DRONE_GITHUB_CLIENT_SECRET: 35e31b8fddd16c2cb3b681e2a4b84a1ee9c8ce57
...

Save and close the file.

Step-3: Create the Drone server deployment in Kubernetes

Open a terminal in the directory where your values.yaml file is located.

Firstly we’ll create the namespace where the deployment will take place with the following command. I will be using the drone namespace here.



kubectl create namespace drone

Next, run the following command which will install the Drone server in Kubernetes:



helm install --namespace drone drone drone/drone -f values.yaml

It will take a while for the command to execute and setup all the Kubernetes resources. Upon its completion, you’ll notice a similar output:



NAME: drone
LAST DEPLOYED: Sun Sep  4 17:39:07 2022
NAMESPACE: drone
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace drone -o jsonpath="{.spec.ports[0].nodePort}" services drone)
  export NODE_IP=$(kubectl get nodes --namespace drone -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT

This indicates that the server is successfully deployed in Kubernetes. We can validate that all the Kubernetes resources have been successfully setup using the following command:



kubectl get all --namespace drone

This should give you a similar output:



NAME                         READY   STATUS    RESTARTS   AGE

pod/drone-77cd496d5d-jf7gc   1/1     Running   0          3m26s

NAME            TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE

service/drone   NodePort   10.28.177.200   <none>        8080:32000/TCP   3m31s

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE

deployment.apps/drone   1/1     1            1           3m32s

NAME                               DESIRED   CURRENT   READY   AGE

replicaset.apps/drone-77cd496d5d   1         1         1       3m34s

Step-4: Access Drone server UI

Before accessing the Drone server UI, make sure that if there’s a Firewall for the Kubernetes nodes then there’s an ingress rule allowing all the http traffic on port 32000.

This would allow us to reach the server endpoint using the NodePort and node external IP. For example, I will be accessing http://35.238.118.197:32000 URL in my browser.

We can successfully access the Drone server UI now. Choose CONTINUE to proceed with the user authentication. Upon doing so, you’ll be prompted for GitHub OAuth authorization. This is by virtue of the OAuth application that we had created earlier in GitHub. Once done, you’ll be prompted to fill up your user details. Once done, choose SUBMIT.

You’ll be then able to access the Drone server dashboard, where all your repositories can be accessed, by virtue of the GitHub OAuth which gives permission to the Drone application to access all your repos.

Summary

In summary, we observed how Drone CI can be setup in Kubernetes using Helm. We created a GitHub OAuth application to authorize the users and provide access to the repositories. Then, we configured the server deployment by providing the relevant details using a config file and finally we used that config file to create the server deployment in Kubernetes using Helm, which allowed us to access the Drone server UI.

GCP IAM Integration for LitmusChaos with Workload Identity

Neelanjan Manna — Fri, 09 Sep 2022 10:55:11 +0000

In this blog, we’ll take a look at how to do a GCP IAM integration for LitmusChaos using Workload Identity for executing the GCP experiments in a keyless manner when using the Google Kubernetes Engine (GKE) as the execution plane.

Introduction

To execute LitmusChaos GCP experiments, one needs to authenticate with GCP using a service account before trying to access the target resources. Usually, you have only one way of providing the service account credentials to the experiment, using a service account key, but if you’re using a GKE cluster you have a keyless medium of authentication as well.
Therefore you have two ways of providing the service account credentials to your GKE cluster:

Using Secrets: As you would normally do, you can create a secret containing the GCP service account in your GKE cluster, which gets utilized by the experiment for authentication to access your GCP resources.
IAM Integration: When you’re using a GKE cluster, you can bind a GCP service account to a Kubernetes service account as an IAM policy, which can be then used by the experiment for keyless authentication using GCP Workload Identity. We’ll discuss more on this method in the following sections.

Why use Workload Identity for GCP authentication?

A Google API request can be made using a GCP IAM service account, which is an identity that an application uses to make calls to Google APIs. You might create individual IAM service accounts for the users who execute GCP experiments to enforce role-based access control, then download and save the keys as a Kubernetes secret that you manually rotate. Not only is this time-consuming, but service account keys only last ten years (or until you manually rotate them). An unaccounted-for key could give an attacker extended access in the event of a breach or compromise. Using service account keys as secrets is not an optimal way of authenticating GKE workloads due to this potential blind spot and the management cost of key inventory and rotation.

Workload Identity allows you to restrict the possible “blast radius” of a breach or compromise while enforcing the principle of least privilege across your environment. It accomplishes this by automating workload authentication best practices, eliminating the need for workarounds, and making it simple to implement recommended security best practices.

Your tasks will only have the permissions they require to fulfill their role with the principle of least privilege. It minimizes the breadth of a potential compromise by not granting broad permissions.
Unlike the 10-year lifetime service account keys, credentials supplied to the Workload Identity are only valid for a short time, decreasing the blast radius in the case of a compromise.
The risk of unintentional disclosure of credentials due to a human mistake is greatly reduced because Google controls the namespace service account credentials for you. It also eliminates the need for you to manually rotate these credentials.

Enabling service accounts to access GCP resources

STEP 1: Enable Workload Identity

You can enable Workload Identity on clusters and node pools using the Google Cloud CLI or the Google Cloud Console. Workload Identity must be enabled at the cluster level before you can enable Workload Identity on node pools. Workload Identity can be enabled for an existing cluster as well as a new cluster.

To enable Workload Identity on a new cluster using the console, choose to create a GKE cluster, and aside from the usual configuration, simply enable the Workload Identity under security.

You can also use the gcloud tool to create the Kubernetes cluster with Workload Identity enabled using the following command:

gcloud container clusters create CLUSTER_NAME \
   --region=COMPUTE_REGION \
   --workload-pool=PROJECT_ID.svc.id.goog

Replace the following:

CLUSTER_NAME: the name of your new cluster.
COMPUTE_REGION: the Compute Engine region of your cluster. For zonal clusters, use --zone=COMPUTE_ZONE.
PROJECT_ID: your Google Cloud project ID.

You can enable Workload Identity on an existing Standard cluster by using the gcloud CLI or the Cloud Console. Existing node pools are unaffected, but any new node pools in the cluster use Workload Identity by the modification of the same setting as shown above in the Cloud Console. Alternatively, you can use the following command to achieve it:

gcloud container clusters update CLUSTER_NAME \
   --region=COMPUTE_REGION \
   --workload-pool=PROJECT_ID.svc.id.goog

Replace the following:

CLUSTER_NAME: the name of your new cluster.
COMPUTE_REGION: the Compute Engine region of your cluster. For zonal clusters, use --zone=COMPUTE_ZONE.
PROJECT_ID: your Google Cloud project ID.

STEP 2: Configure LitmusChaos to use Workload Identity

Assuming that you already have LitmusChaos installed in your GKE cluster as well as the Kubernetes service account you want to use for your GCP experiments, execute the following steps.

Get Credentials for your cluster.

gcloud container clusters get-credentials CLUSTER_NAME

Replace CLUSTER_NAME with the name of your cluster that has Workload Identity enabled.

Create an IAM service account for your application or use an existing IAM service account instead. You can use any IAM service account in any project in your organization. For Config Connector, apply the IAMServiceAccount object for your selected service account. To create a new IAM service account using the gcloud CLI, run the following command:

gcloud iam service-accounts create GSA_NAME \
--project=GSA_PROJECT

Replace the following:

GSA_NAME: the name of the new IAM service account.
GSA_PROJECT: the project ID of the Google Cloud project for your IAM service account.

Alternatively, you can also use the Cloud Console UI to create a new GCP IAM Service Account.

Ensure that this service account has all the roles required for interacting with the Compute Engine resources including VM Instances and Persistent Disks according to the GCP experiments that you’re willing to run. You can grant additional roles either using the Cloud Console or using the following command:

gcloud projects add-iam-policy-binding PROJECT_ID \    
--member "serviceAccount:GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com" \
--role "ROLE_NAME"

Replace the following:

PROJECT_ID: your Google Cloud project ID.
GSA_NAME: the name of your IAM service account.
GSA_PROJECT: the project ID of the Google Cloud project of your IAM service account.
ROLE_NAME: the IAM role to assign to your service account, like roles/spanner.viewer.

Allow the Kubernetes service account to be used for the GCP experiments to impersonate the GCP IAM service account by adding an IAM policy binding between the two service accounts. This binding allows the Kubernetes service account to act as the IAM service account.

gcloud iam service-accounts add-iam-policy-binding GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com \
   --role roles/iam.workloadIdentityUser \
   --member "serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]"

Replace the following:

GSA_NAME: the name of your IAM service account.
GSA_PROJECT: the project ID of the Google Cloud project of your IAM service account.
KSA_NAME: the name of the Kubernetes service account to be used for LitmusChaos GCP experiments. If you’re using ChaosCenter, then by default the litmus-admin service account is used for all the experiments.
NAMESPACE: the namespace in which the Kubernetes service account to be used for LitmusChaos GCP experiments is present. If you’re using ChaosCenter, then by default litmus is the namespace.

Annotate the Kubernetes service account to be used for LitmusChaos GCP experiments with the email address of the GCP IAM service account.

kubectl annotate serviceaccount KSA_NAME \
   --namespace NAMESPACE \
iam.gke.io/gcp-service-account=GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com

Replace the following:

KSA_NAME: the name of the Kubernetes service account to be used for LitmusChaos GCP experiments. If you’re using ChaosCenter, then by default the litmus-admin service account is used for all the experiments.
NAMESPACE: the namespace in which the Kubernetes service account to be used for LitmusChaos GCP experiments is present. If you’re using ChaosCenter, then by default litmus is the namespace.
GSA_NAME: the name of your IAM service account.
GSA_PROJECT: the project ID of the Google Cloud project of your IAM service account.

STEP 3: Update ChaosEngine Manifest

When creating a new workflow with GCP experiments, edit the manifest YAML and add the following value to the ChaosEngine manifest field .spec.experiments[].spec.components.nodeSelector to schedule the experiment pod on nodes that use Workload Identity:

iam.gke.io/gke-metadata-server-enabled: "true"

As an example, say you’re adding the GCP VM Instance Stop experiment, you’d make the following change:

STEP 4: Update ChaosExperiment Manifest

Remove cloud-secret at .spec.definition.secrets in the ChaosExperiment manifest as we are not using a secret to provide our GCP Service Account credentials.

As an example, say you’re adding the GCP VM Instance Stop experiment, you’d remove the following:

Now you can run your GCP experiments with a keyless authentication provided by GCP using Workload Identity.

Summary

To conclude, we were able to set up a keyless authentication medium for the GCP experiments executing in a GKE that belongs to the target GCP environment. We saw how we can leverage Workload Identity for the GKE node pools to assign an IAM identity to our Kubernetes workloads, which was then used for the authentication of LitmusChaos GCP experiments.

Why Did I Contribute to the LitmusChaos Project for Hacktoberfest 2021

Neelanjan Manna — Mon, 30 May 2022 15:26:51 +0000

For the eighth edition of Hacktoberfest, I chose to contribute to LitmusChaos, a CNCF sandboxed project for Cloud-Native Chaos Engineering. It was a month-long celebration of making Chaos Engineering simpler for all the SREs and Developers who aspire to make their services more resilient. So if you are a software developer like me, then why should you consider contributing to the LitmusChaos project?

1. Buzzing Community

The Litmus community boasts of some 1000+ community members who profoundly promote the Cloud-Native resiliency paradigm shift by redefining the Chaos Engineering experience for everyone. It is the community members who bring out the best qualities of Chaos Engineering for their distinct use cases by leveraging the diverse set of chaos experiments offered by LitmusChaos. The regular community meetups and sync-ups highlight these interesting use cases, which helps every community member to learn from each other.

2. Excellent Code Quality

LitmusChaos is an awesome project for beginners who want to kick-start their journey into the world of open-source as the LitmusChaos project has been developed and contributed by all kinds of developers belonging to a varied number of organizations. This has enabled a robust project codebase that adheres to the best coding practices and is easy to understand and contribute. Further, the LitmusChaos project makes use of a wide range of open-source tools such as Kubernetes, GraphQL, Argo Workflows, React JS, etc. which allows every developer to contribute to something of their own choice.

3. Awesome Documentation

The Litmus project features extensive documentation that encompasses every single aspect of the project. It aids the developers who want to gain a deeper insight into the project and want to contribute to it with its detailed user documentation, experiment documentation, and API reference. Contributing to the documentation and tutorials is also an excellent option, which helps LitmusChaos to be used by more community members and end-users.

4. Flexible Litmus SDK

Litmus SDK allows developers to define their chaos experiments for the LitmusChaos framework. It helps developers to easily bootstrap the experiment files where the SDK is responsible for generating all the requisite artifacts and the developer is only responsible for defining the experiment business logic. The best part about Litmus SDK is that it’s available in multiple programming languages such as Go, Python, and Ansible, allowing developers to develop chaos experiments in any programming language of their choice.

5. Mentorship Programs

Apart from Hacktoberfest, LitmusChaos also takes part in all the major open-source mentorship programs such as Google Summer of Code (GSoC), GitHub India Externship, The Linux Foundation Mentorship Program (LFX Mentorship), and Google Summer of Docs (GSoD) among the others. These mentorship programs provide a very lucrative opportunity to the mentees for not only contributing to the open-source but also to learn and gain recognition as a quality developer since you’ll be making very significant contributions to the Litmus project under the guidance of a LitmusChaos project maintainer.

Join the Litmus Community:

Want to get help with queries, learnings, & contributions? Join the Litmus community on slack. To join the slack community please follow the following steps:
Step 1: Join the Kubernetes slack using the following link: https://slack.k8s.io/
Step 2: Join the #litmus channel on the Kubernetes slack or use this link after joining the Kubernetes slack: https://slack.litmuschaos.io/

Looking forward to seeing you in the world of Open Source!

GCP VM Disk Loss Experiment for LitmusChaos

Neelanjan Manna — Sun, 25 Jul 2021 15:06:31 +0000

In this beginner-friendly blog, we’ll be going through the GCP VM Disk Loss experiment for LitmusChaos. GCP VM Disk Loss experiment causes detachment of a non-boot persistent storage disk from a GCP VM instance for a specified duration of time and later re-attaches the disk to its respective VM instance. The broad objective of this experiment is to extend the principles of cloud-native chaos engineering to non-Kubernetes targets while ensuring resiliency for all kinds of targets, be it Kubernetes or non-Kubernetes ones, as a part of a single chaos workflow for the entirety of a business.

At the time of writing this blog, the experiment is available only as a technical preview in the chaos hub, but in the upcoming releases, the experiment will surely become an integral part of the chaos hub. That being said, we can still access and execute the experiment without any problem, as I am about to show you in this blog.

Pre-Requisites

Before we begin with the steps of the experiment, let’s check the pre-requisites for performing this experiment:

A GCP project containing the target Persistent Storage Disks attached to their respective VM instances
A GCP Service Account having project level Editor or Owner permissions
A Kubernetes cluster with Litmus 2.0 installed

STEP 1: Updating The Chaos Hub

Browse and log in to your Litmus portal. You should be on the dashboard.

Select ChaosHubs. Here you’d be able to see the default Chaos Hub.

Choose to Edit the default Chaos Hub and instead of the v1.13.x branch, choose the master branch.

Click Submit Now. Now you’d be able to access all the experiments, even those under the technical preview. To confirm that the experiments have been added successfully, click on Chaos Hub.

You should see the GCP Experiments listed here. Now we are all set to begin the steps of the experiment.

STEP 2: Setting Up the Chaos Experiment

We’d be using the experiment docs to help us with a few steps.

In this demo, we will inject chaos into two non-boot persistent storage disks named test-disk and test-disk-1, attached to the VM instances named test-instance and test-instance-1 respectively, the disks are located in the zones us-central1-a and us-central1-b respectively, and belong to the GCP project “Litmus GCP Instance Delete” with the ID of litmus-gcp-instance-delete.

Please notice that both the disks are initially attached to their respective VM instances, before the injection of the chaos. Now that we have our disks ready, we can set up our experiment. Before scheduling the chaos experiment, we need to make the GCP Service Account credentials available to Litmus, so that the instances can be shut down and later started as part of the experiment. To do that, we’ll make a Kubernetes secret named secret.yaml as follows:

The format of this secret is also available in the experiment docs. Make sure the name of the secret is cloud-secret and replace the respective fields of the secret with your own service account credentials. Once done, apply the secret in the litmus namespace using the command:

kubectl apply -f secret.yaml -n litmus

Once the secret is applied, we’re all set to schedule our experiment from the Litmus portal. In Dashboard, click on the Schedule a Workflow button. In the workflow creation page, choose the self-agent and click Next.

In the Choose a Workflow page, select “Create a new workflow using the experiments from MyHub” and select Chaos Hub in the dropdown. Then click Next.

In the Workflow Settings page, fill in the workflow name and description of your choice. Click Next.

In the Tune Workflow page, click on “Add a new experiment” and choose gcp/gcp-vm-disk-loss.

Click Done. Notice that the experiment has been added to the experiment graph diagram. Now click on “Edit YAML”. Here we will edit the workflow manifest to specify the experiment resource details.

Scroll down to the manifest of the ChaosExperiment:

Notice that the name of the secret that we had previously created is being passed to the ChaosExperiment to be mounted at the path /tmp/.

Scroll further down and similarly fill in the relevant experiment details in the manifest of the ChaosEngine as follows:

Please take note that the zone for each target disk is to be mentioned in DISK_ZONES in the same order of the DISK_VOLUME_NAMES. Similarly, the device name for each target disk is to be mentioned in DEVICE_NAMES in the same order of the DISK_VOLUME_NAMES. If you like, feel free to modify the parameters of the experiment such as the RAMP_TIME, TOTAL_CHAOS_DURATION, etc. As you would have noticed, some of the experiment tunables are common for both the ChaosEngine and ChaosExperiment, and the values of ChaosExperiment get overridden by that of the values of the ChaosEngine if they differ in both the manifests. Once you have made the changes, click Save Changes. We’ve now specified all the experiment details and are ready to go to the next step. Click Next.

In the Reliability Score, we will use the default score of 10. Click Next.

In Schedule, click Schedule Now. Click Next. On the Verify and Commit page verify all the details and once satisfied click on Finish. We’ve successfully scheduled our chaos experiment.

STEP 3: Observing the Chaos

Click on Go to Workflow and choose the workflow that we just created. Here we can observe the different steps of the workflow execution including chaos experiment installation, chaos injection, and chaos revert.

You can also determine if the chaos injection has taken place and as a result, the disks have detached from their respective VM instances or not from the GCP Console.

We can also view the Table View for the experiment logs as the experiment proceeds through the various steps.

Once completed, the workflow graph should have executed all the steps successfully.

We have got a 100% Resiliency Score. We can also check the ChaosResult verdict which should say the experiment has passed. The Probe Success Percentage should be 100% as all our disks re-attached successfully post their detachment.

Again you can check in the GCP console if the disks have been re-attached or not.

We can also perform post chaos analysis for the experiment results in the Analytics section.

In conclusion of this blog, we saw how we can perform the GCP VM Disk Loss chaos experiment using Litmus Chaos 2.0. This experiment is only one of the many experiments for the Non-Kubernetes experiments in LitmusChaos, including experiments for AWS, Azure, VMWare, and many more, which are targeted towards making Litmus an absolute Chaos Engineering toolset for every enterprise regardless of the technology stack used by them.

Come join me at the Litmus community to contribute your bit in developing chaos engineering for everyone. To join the Litmus community:

Step 1: Join the Kubernetes slack using the following link: https://slack.k8s.io/

Step 2: Join the #litmus channel on the Kubernetes slack or use this link after joining the Kubernetes slack: [https://slack.litmuschaos.io/](https://slack.litmuschaos.io/)

Show your ❤️ with a ⭐ on our Github. To learn more about Litmus, check out the Litmus documentation. Thank you! 🙏

GCP VM Instance Stop Experiment for LitmusChaos

Neelanjan Manna — Sat, 24 Jul 2021 11:06:12 +0000

This blog is a beginner-friendly guide for the GCP VM Instance Stop chaos experiment for LitmusChaos. The experiment causes the shutdown of one or more GCP VM instances for a specified duration of time and later restarts them. The broad objective of this experiment is to extend the principles of cloud-native chaos engineering to non-Kubernetes targets while ensuring resiliency for all kinds of targets, be it Kubernetes or non-Kubernetes ones, as a part of a single chaos workflow for the entirety of a business.

Pre-Requisites

Before we begin with the steps of the experiment, let’s check the pre-requisites for performing this experiment:

A GCP project containing the target VM instances
A GCP Service Account having sufficient permissions to stop or start the VM Instances
A Kubernetes cluster with Litmus 2.0 installed

STEP 1: Updating The Chaos Hub

Browse and log in to your Litmus portal. You should be on the dashboard.

Select ChaosHubs. Here you’d be able to see the default ChaosHub.

Choose to Edit the default Chaos Hub and instead of the v1.13.x branch, choose the master branch.

Click Submit Now. Now you’d be able to access all the experiments, even those under the technical preview. To confirm that the experiments have been added successfully, click on Chaos Hub and view the Chaos Hub.

You should see the GCP Experiments listed here. Now we are all set to begin the steps of the experiment.

STEP 2: Setting Up the Chaos Experiment

We’d be using the experiment docs to help us with a few steps.

In this demo, we will inject chaos into two VM instances named test-instance and test-instance-1, belonging to the zones us-central1-a and us-central1-b respectively, belonging to the GCP project “Litmus GCP Instance Delete” with the ID of litmus-gcp-instance-delete.

Please notice that the instances are in a running state initially, before the injection of the chaos. Now that we have our instances ready, we can set up our experiment. Before scheduling the chaos experiment, we need to make the GCP Service Account credentials available to Litmus, so that the instances can be shut down and later started as part of the experiment. To do that, we’d make a Kubernetes secret named secret.yaml as follows:

kubectl apply -f secret.yaml -n litmus

In the Choose a Workflow page, select “Create a new workflow using the experiments from MyHub” and select Chaos Hub in the dropdown. Then click Next.

In the Workflow Settings page, fill in the workflow name and description of your choice. Click Next.

In the Tune Workflow page, click on “Add a new experiment” and choose gcp/gcp-vm-instance-stop.

Scroll down to the manifest of the ChaosExperiment:

Notice that the name of the secret that we had previously created is being passed to the ChaosExperiment to be mounted at the path /tmp/.

Scroll further down and similarly fill in the relevant experiment details in the manifest of the ChaosEngine as follows:

Please take note that the zone for each target instance is to be mentioned in INSTANCE_ZONES in the same order of the VM_INSTANCE_NAMES. If you like, feel free to modify the other parameters of the experiment such as the RAMP_TIME, TOTAL_CHAOS_DURATION, etc. As you would have noticed, some of the experiment tunables are common for both the ChaosEngine and ChaosExperiment, and the values of ChaosExperiment get overridden by that of the values of the ChaosEngine if they differ in both the manifests. Once done, click Save Changes. We’ve now specified all the experiment details and are ready to go to the next step. Click Next.

In the Reliability Score, we will use the default score of 10. Click Next.

In Schedule, click Schedule Now. Click Next. On the Verify and Commit page verify all the details and once satisfied click on Finish. We’ve successfully scheduled our chaos experiment.

STEP 3: Observing the Chaos

You can also determine if the chaos injection has taken place and as a result, the instances have shutdown or not from the GCP Console.

We can also view the Table View for the experiment logs as the experiment proceeds through the various steps.

Once completed, the workflow graph should have executed all the steps successfully.

We can also check the ChaosResult verdict which should say the experiment has passed. The Probe Success Percentage should be 100% as all our instances restarted successfully post their shutdown.

Again you can check in the GCP console if the instances have restarted or not.

We can also perform post chaos analysis for the experiment results in the Analytics section.

In conclusion of this blog, we saw how we can perform the GCP VM Instance Stop chaos experiment using Litmus Chaos 2.0. This experiment is only one of the many experiments for the Non-Kubernetes experiments in LitmusChaos, including experiments for AWS, Azure, VMWare, and many more, which are targeted towards making Litmus an absolute Chaos Engineering toolset for every enterprise regardless of the technology stack used by them.

Show your ❤️ with a ⭐ on our Github. To learn more about Litmus, check out the Litmus documentation. Thank you! 🙏

Getting Started with Litmus 2.0 in Google Kubernetes Engine

Neelanjan Manna — Sat, 03 Jul 2021 04:53:08 +0000

This is a quick tutorial on how to get started with Litmus 2.0 in Google Kubernetes Engine. We’ll first set up our basic GKE cluster throughout this blog, then install Litmus 2.0 in the cluster, and finally, execute a simple chaos workflow using Litmus.

But before we kick off the demonstration, let’s have a brief introduction to Litmus. Litmus is a toolset to perform cloud-native Chaos Engineering. It provides tools to orchestrate chaos on Kubernetes to help developers and SREs find weaknesses in their application deployments. Litmus can be used to run chaos experiments initially in the staging environment and eventually in production to find bugs, vulnerabilities. Fixing the weaknesses leads to increased resilience of the system. Litmus adopts a “Kubernetes-native” approach to define chaos intent in a declarative manner via custom resources, although Litmus is not limited to only Kubernetes targets while injecting chaos and is able to target a plethora of non-Kubernetes targets as well, such as bare-metal infrastructure, public cloud infrastructure, hybrid cloud infrastructure, containerized services, etc. to fulfill the chaos engineering needs of an entire business, and not just a specific application microservice.

Pre-Requisites

A GCP project with sufficient permission for accessing GKE

Step-1: Setup the GKE Cluster

To set up the GKE cluster, we first need to enable the Kubernetes Engine API in our GCP project. To do that, we can access the APIs & Services Dashboard from the GCP console and search for the Kubernetes Engine API.

We can then enable the Kubernetes Engine API. Now we are all set to launch our GKE cluster. To do that, we will first go to the Kubernetes Engine dashboard in the GCP console and then choose to Create the cluster.

Go for a standard cluster and choose to Configure it.

Here you’d get all the options to configure your Kubernetes cluster. You can either choose to configure your own cluster as per your preference or you can simply go for My First Cluster option under the Cluster set-up Guide, which will set up a basic three-node cluster for you.

It will take a while for your cluster resources to set up and undergo a health check, after which it would finally be ready.

We can now connect to our cluster and proceed to install litmus. To do that, we need to choose our cluster and then choose the Connect option.

Next, choose the Run In Cloud Shell option.

This will open up a cloud shell terminal in your console. A cloud SDK command would be already there in your terminal which you need to execute. This command will configure the Kubectl to configure itself for the cluster we just created. If prompted, choose accept.

Now we are all set to install Litmus.

STEP 2: Install Litmus

We’d follow the instructions given in the Litmus 2.0 documentation to install Litmus. Here we will be installing Litmus in namespace mode using Helm. To do that, the very first step is to add the litmus helm repository using the following command:

helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/

This will add litmuschaos repository to the list of Helm chart repositories. We can verify the repository using the command:

helm repo list

You should see litmuschaos repository listed here. Next, we will create the litmus namespace as all the litmus infra components will be placed in this namespace. We will do this using the command:

kubectl create ns litmus

Once this is done, we can proceed to install the Litmus control plane using the following command:

helm install chaos litmuschaos/litmus-2-0-0-beta --namespace=litmus --devel --set portalScope=namespace

You should get a similar message:

Next, we can proceed to install the Litmus CRDs. Cluster-Admin or an equivalent user with the right permissions are required to install them CRDs. We’d use the following command to apply the CRDs:

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/master/litmus-portal/litmus-portal-crds.yml

And we’re done with installing Litmus. Just one final step before we can access the Litmus portal. If we use the command kubectl get svc -n litmus, we’ll be able to list down all the services running in the litmus namespace. Here you should be able to see the litmusportal-frontend-service and the litmusportal-server-service, both of which should be NodePort services, and their corresponding TCP ports should be something like 9091:xxxxx/TCP where xxxxx is the node port of the respective service. To access the portal, we need to expose both the node ports of the frontend service and the server service by applying a firewall rule. For the litmusportal-server-service we can expose only the first NodePort as there are two of them. We can do that using the following two commands:

gcloud compute firewall-rules create frontend-service-rule --allow tcp:<NODE_PORT>

gcloud compute firewall-rules create server-service-rule --allow tcp:<NODE_PORT>

Replace the <NODE_PORT> in the first command with your frontend service’s node port and similarly replace the <NODE_PORT> in the first command with your server service’s node port. Once done, we’re all set to access the Litmus portal. To do that, use the command kubectl get nodes -o wide to list all the pods in your cluster. You should see something like this:

For any of the nodes, copy its external IP and paste it into your browser URL section, followed by a :xxxxx where xxxxx corresponds to your node port. You should then be directed to the Litmus home screen.

Log in using the username admin and password litmus. Then you’d be asked to set up a password, which you’d use for any subsequent login. Once done, you’d be able to access the Litmus portal dashboard.

Step 3: Run a Chaos Workflow

For this demo, we’d see how do chaos workflows execute in Litmus with the help of a predefined chaos workflow template called podtato-head. It is a simple application deployed as a part of the Litmus installation in which we will try to inject pod-delete chaos.

Choose Schedule a Workflow. In the workflows dashboard, choose the self agent and then choose Next. Choose Create a Workflow from Pre-defined Templates and choose podtato-head.

Choose Next. Here we can define the experiment name, description, and namespace. Leave the default values and choose Next. Here we can tune the workflow by editing the experiment manifest and adding/removing or arranging the experiments in the workflow. The podtato-head template comes with its own defined workflow so simply choose Next.

Now we can set a reliability score per chaos experiment in our workflow. Since this workflow has only pod-delete chaos experiment, we can set a score for that in between 1–10. Then choose Next.

Here we can schedule our workflows to be executed in a determined frequency or we can simply schedule it to execute right away. Choose Schedule now. Choose Next. Here we can review the workflow details and make changes if required before finally executing the workflow. Choose Finish. You have successfully started the Chaos Workflow.

Choose Go to Workflow and you’d see the running workflow:

Select the workflow and you’d be able to see the workflow execute in its various stages in the graphical chart:

Wait for the workflow execution to complete, once completed the graph would appear as follows:

We can analyze the Table View tab for every step as follows:

Choose View Logs & Results to access the chaos result of the pod-delete experiment:

You can also observe the analytics of the workflow from the workflow dashboard page which shows a graphical analysis of the workflow run:

In conclusion of this blog, we saw how to set up a Google Kubernetes Engine cluster, how to install Litmus 2.0, and finally how to execute a chaos workflow using the Litmus portal. Though this is only the tip of the iceberg, I really hope you learned something new today, which will be helpful for your further journey in chaos engineering.

Come join me at the Litmus community to contribute your bit in developing chaos engineering for everyone. To join the Litmus community:
Step 1: Join the Kubernetes slack using the following link: https://slack.k8s.io/
Step 2: Join the #litmus channel on the Kubernetes slack or use this link after joining the Kubernetes slack: https://slack.litmuschaos.io/

Show your ❤️ with a ⭐ on our Github. To learn more about Litmus, check out the Litmus documentation. Thank you! 🙏

Part-2: A Beginner’s Practical Guide to Containerisation and Chaos Engineering with LitmusChaos 2.0

Neelanjan Manna — Fri, 02 Jul 2021 15:16:41 +0000

Part 2: Chaos Engineering 101, Using LitmusChaos 2.0 for a custom workflow chaos experiment

This blog is part two of a two blog series that details how to get started with containerization using Docker and Kubernetes for deployment, and later how to perform chaos engineering using LitmusChaos 2.0. Find Part-1 of the blog here.

Say you have deployed your E-Commerce application using Kubernetes and you’re very satisfied with how flexible and stable your application deployment has come to be. During the testing of the application, it had checked all the boxes and you’re very confident that your application deployment is all set to face the peak hours of the sale next week, which will have customers all over the country trying to buy products using your application. But alas, right on the peak hours of the sale, your customers face a service outage. What went wrong? You don’t have any idea, since on a superficial level nothing seems to be out of order. Could this infelicitous situation have been avoided? Yes, using chaos engineering.

In this blog, we will fundamentally explore chaos engineering; starting from what is chaos engineering, why chaos engineering is a necessity, how is it different from testing, processes, and principles of chaos engineering, introduction to cloud-native chaos engineering, a bird’s eye view of LitmusChaos, and finally, we will perform a pod-delete chaos experiment with a custom chaos workflow on the application that we had deployed in the previous blog.

What is Chaos Engineering?

As Wikipedia defines it:

Chaos engineering is the discipline of experimenting on a software system in production in order to build confidence in the system’s capability to withstand turbulent and unexpected conditions.

In our previous example of the E-Commerce application which faced a service downtime on account of the sharp rise in the number of users trying to simultaneously access the application, that situation could have been avoided by identifying the factors that were contributing to the service downtime.

Chaos Engineering emphasizes experiments, otherwise called hypotheses, and then compares the results to a defined steady-state. It is also regarded as the science of “breaking things on purpose” to identify the unforeseen weaknesses of the system before they wreak havoc in production. More than just fault injection, chaos engineering is an attempt at understanding the factors that contribute to the instability of a system, and gaining insights from the behavior of the system as a whole when it is subjugated to an adverse situation, with the ultimate goal of making systems more resilient in the production environment.

As an example, a distributed system could be checked for resiliency by randomly disabling the services responsible for the functioning of the system and analyzing its impact on the system as a whole.

Why Chaos Engineering?

Recognize the dangers and consequences: By allowing you to create an experiment and quantify how it affects your business, chaos engineering allows you to understand the influence of turbulent conditions on important applications. Companies can make informed judgments and react proactively to avoid or prevent losses when they understand what’s at risk.
Reaction to an incident: Because distributed systems are so complicated, there are several ways for things to go wrong. The notion of disaster recovery and business continuity is critical for firms in highly regulated contexts, such as the financial industry, because even a single instant of outage can be costly. These industries may rehearse, prepare, and put mechanisms in place for real-life situations by conducting chaotic experiments. When an incident occurs, chaos engineering allows teams to have the correct level of awareness, plans, and visibility.
Application Security & Observability: Chaos experiments help you figure out where your systems’ monitoring and observability capabilities are lacking, as well as your team’s ability to respond to crises. Chaos engineering will help you identify areas for improvement and motivate you to make your systems more visible, resulting in better telemetry data.
System Reliability: Chaos engineering enables firms to create dependable and fault-tolerant software systems while also increasing your team’s trust in them. The more reliable your systems are, the more confident you can be in their ability to perform as expected.

Is it the same as testing?

In one word, no.

A failure test looks at a single situation and determines whether or not a property is true. A test like this breaks the system in a predetermined way. The outcomes are often binary and do not reveal any additional information about the program, which is essential to understand the root cause of the problem.

Chaos Engineering’s purpose is to generate fresh information about the system when it is subjugated to adversity. One can learn more about the system’s behaviors, attributes, and performance because the scope is broader and the outcomes are unpredictable. Therefore it allows us to better understand the limitations of our system and act on them.

Process of conducting Chaos Engineering

By and large, chaos engineering can be abstracted into the following set of processes:

Define the steady-state hypothesis: You should begin by imagining what could go wrong. Start with a failed injection and forecast what will happen when it’s live.
Confirm the steady-state and perform several realistic simulations: Test your system using real-world scenarios to observe how it reacts to different stressors and events.
Collect data and monitor dashboards: You must assess the system’s dependability and availability. It’s ideal to employ key performance indicators that are linked to consumer success or usage. We want to see how the failure compares to our hypothesis, therefore we’ll look at things like latency and requests per second.
Changes and issues should be addressed: After conducting an experiment, you should have a good notion of what is working and what needs to be changed. We can now predict what will cause an outage and precisely what will cause the system to fail.

Principles of Chaos Engineering

An elaborate description of the same is the Principles of Chaos manifesto, which describes the core concerns that chaos engineering should address:

Understand your system’s normal state: Define your system’s steady state. Any chaotic experiment uses a system’s regular behavior as a reference point. You will have a better understanding of the effects of faults and failures if you understand the system when it is healthy.
Use of realistic bugs and failures: All experiments should be based on plausible and realistic settings. When a real-life failure is injected, it becomes clear which processes and technologies need to be upgraded.
Production-level testing: Only by running the test in a production setting can you see how disruptions influence the system. Allow your team to experiment in a development environment if they have little or no experience with chaotic testing. Once the production environment is ready, test it.
Control the radius of the blast: A chaotic test’s blast radius should always be kept as small as possible. Because these tests are conducted in a live setting, there is a potential that they will have an impact on end-users.
Automating chaos: Chaos experiments may be automated to the same degree as your CI/CD pipeline. Continuous chaos allows your team to continuously improve current and future systems.

Introduction to Cloud-Native Chaos Engineering

Businesses love cloud: About a third of companies’ IT budget goes to cloud services and the global public cloud computing market is set to exceed $330 billion in 2021. The groundbreaking shift towards cloud-native software products needs to be supplemented with the right set of tools to ensure that they are resilient against all the possible adverse situations that may arise in production.

Enter cloud-native chaos engineering, the best of both worlds. In its core essence, it's all about performing chaos engineering in a cloud-native or Kubernetes-first way. There are four major principles that define how cloud-native a chaos engineering tool or framework is:

CRDs for Chaos Management: For coordinating chaos on Kubernetes, the framework should have explicitly defined CRDs. These CRDs provide standard APIs for provisioning and managing chaos in large-scale production systems. These are the elements that make up a chaotic workflow orchestration system.
Open Source: To enable larger community engagement and examination, the framework must be totally open-source under the Apache License 2.0. The number of applications that are migrating to the Kubernetes platform is uncountable. Only the Open Chaos model will thrive and gain the requisite adoption at such a wide scale.
Extensible and Pluggable: The framework should be integrable with the vast number of existing cloud-native applications, essentially built as a component that can be easily plugged in for chaos engineering within an application and can be easily plugged out as well.
Broad Community adoption: The chaos will be carried out against well-known infrastructures such as Kubernetes, applications such as databases, and infrastructure components like as storage and networking. These chaos experiments can be utilized again, and a large community can help identify and contribute to more high-value scenarios. As a result, a Chaos Engineering system should have a central hub or forge where open-source chaos experiments can be shared and code-based collaboration is possible.

LitmusChaos is a cloud-native Chaos Engineering framework for Kubernetes that fulfills all the four criteria listed above.

A Bird’s Eye View of LitmusChaos

What is Litmus

Litmus is a toolset to do cloud-native Chaos Engineering. It helps both Developers and SREs automate the chaos experiments at different stages within the DevOps pipeline like development, during CI/CD, & in production. Fixing the weaknesses leads to increased resilience of the system.

Litmus adopts a “Kubernetes-native” approach to define chaos intent in a declarative manner via custom resources. It broadly defines Kubernetes chaos experiments into two categories: application or pod-level chaos experiments and platform or infra-level chaos experiments. The former includes pod-delete, container-kill, pod-cpu-hog, pod-network-loss, etc., while the latter includes node-drain, disk-loss, node-cpu-hog, etc. It is largely developed under Apache License 2.0 license at the project level.

Before we understand the architecture of Litmus, let us understand a few terminologies:

Chaos Experiment: Chaos Experiments are the building blocks of the Litmus architecture. Users can develop the desired chaos workflow by choosing from freely available chaos experiments or by creating new ones.
Chaos Workflow: A chaos workflow is a lot more than a chaos experiment. It helps the user define the intended result, observe the result, analyze the overall system behavior, and decide whether the system needs to be changed to improve resilience. For a normal development or operations team, LitmusChaos provides the infrastructure required to design, use, and manage chaotic workflows. Litmus’ teaming and GitOps features considerably aid in the collaborative control of chaotic processes within teams or software organizations.

Litmus Architecture

Litmus components can be classified into two parts:

Portal
Agents

Portal is a set of Litmus components that act as Cross-Cloud Chaos Control plane (WebUI) which is being used to orchestrate and observe the chaos workflows on Agents.

Agent is the set of Litmus components that induces Chaos using the chaos workflows on the K8s cluster component.

Portal Components

Litmus WebUI: Litmus UI provides a web user interface, where users can construct and observe the chaos workflow at ease. Also this act as a cross-cloud chaos control plane that is
Litmus Server: Litmus Server act as middleware which is used to handle API request from the user interface, store the config and results into the DB. This also acts as an interface to communicate between the requests and scheduling the workflow to Agent.
Litmus DB: Litmus DB act as a config store for chaos workflows and their results.

Agent Components

Chaos Operator: Chaos-Operator watches for the ChaosEngine CR and executes the Chaos-Experiments mentioned in the CR. Chaos-Operator is namespace scoped. By default, it runs in litmus namespace.
CRDs: During installation, the following three CRDs are installed on the Kubernetes cluster: chaosexperiments.litmuschaos.io, chaosengines.litmuschaos.io, and chaosresults.litmuschaos.io.
Chaos Experiment: Chaos Experiment is a CR and is available as YAML files on Chaos Hub.
Chaos Engine: ChaosEngine CR connects experiments to applications. The user must construct ChaosEngine YAML by giving the app label and experiments, as well as the CR.
Chaos Results: The results of a ChaosExperiment with a namespace scope are stored in the ChaosResult resource. The experiment itself creates or updates it in runtime. It contains critical information such as the ChaosEngine reference, Experiment State, Experiment Verdict (on completion), and key application/result properties. It can also be used to collect metrics.
Chaos Probes: Litmus probes are pluggable tests that can be defined for any chaotic experiment within the ChaosEngine. These checks are carried out by the experiment pods based on the mode they are defined in, and their success is used to determine the experiment’s judgment (along with the standard “in-built” checks).
Chaos Exporter: Metrics can be exported to a Prometheus database if desired. The Prometheus metrics endpoint is implemented by Chaos-Exporter.
Subscriber: Subscriber is a component on the Agent side that communicates with the Litmus Server component to obtain Chaos process data and return the results.

Demo: Performing the pod-delete experiment with a custom chaos workflow on a Kubernetes deployment

Let’s get a hang of Litmus with a very simple chaos experiment: the pod-delete experiment. Essentially, we’d like to see that whether our Kubernetes deployment from the last blog is resilient against the event of accidental pod-deletion.

Here’s what we’d do; firstly we will install Litmus, then we’d define our custom workflow, and finally we’d analyze the results of our experiment. Simple, isn’t it?

Our application deployment can be viewed using the kubectl get deployments command:

and the associated pods can be viewed using kubectl get pods:

Before installing Litmus, let us check the pre-requisites:

Kubernetes 1.15 or later
A persistent volume of 20GB (Recommended)
Helm3 or Kubectl

Once we’re good with these, let’s follow the installation steps. Litmus can be installed either using Helm or using Kubectl. Let’s try to install it using Helm:

STEP 1: Create a Litmus namespace in Kubernetes

Litmus installs within the litmus namespace. So let us create the namespace with the command: kubectl create namespace litmus

STEP 2: Add the Litmus Helm Chart

Clone the litmus repo and move into the cloned directory using the command: git clone https://github.com/litmuschaos/litmus-helm && cd litmus-helm

STEP 3: Install Litmus

The helm chart will install all the CRDs, required service account configuration, and chaos-operator required both for the core services as well as the portal to run. Use the command helm install litmuschaos — namespace litmus ./charts/litmus-2–0–0-beta/

And that’s it, you’re all set up to use Litmus! You can verify the installation by viewing all the resources installed under the litmus namespace using kubectl get all -n litmus command:

It might take a while for all the resources to get ready and eventually you’d see something as depicted above. Here we can observe that we have three pods running: litmus-frontend, litmus-backend, and mongo. These comprise the Litmus WebUI, Litmus Server, and Litmus DB respectively, as discussed earlier.

We also have three services for our application, namely litmusportal-frontend-service, litmusportal-backend-service, and mongo-service. These services maintain the endpoints for the pods which we saw earlier.

These pods are created using the two deployments namely litmusportal-frontend and litmusportal-backend, which are also responsible for specifying the replicasets for the same.

Lastly, the Mongo DB database being a stateful resource uses a statefulset named mongo to persist the data contained by the DB even if the mongo pod dies and restarts.

Once all these resources are ready, we can proceed to the Litmus portal. For this, we will try to access the NodePort service litmusportal-frontend-service. The nodePort assigned to the litmusportal-frontend-service in my machine has a mapping of 9091:30628 where 9091 is the specified targetPort while 30628 is the assigned nodePort.

If you’re using Minikube, you can port-forward any unused port of your choice to the litmusportal-frontend-service in order to access it at that port. For example, if I wish to access the litmusportal-frontend-service at the 3000 port, I’d use the command kubectl port-forward svc/litmusportal-frontend-service 3000:9091 -n litmus. Once done, simply access the Litmus portal at http://127.0.0.1:3000.

If you’re using any other Kubernetes platform, you can directly access the Litmus portal at the nodePort, given you have a firewall rule allowing ingress at that port. For example, I have a nodePort of 30628, hence I can directly access the Litmus portal at http://127.0.0.1:30628.

The default username is admin and the default password is litmus. Once you log in, you’d be prompted to enter a new password. Once that’s done, you’d find yourself in the dashboard:

Let’s take a quick tour of the portal. The Dashboard is where you see the different workflows which were either currently executing or had been previously executing, the number of agents connected to the portal, and the number of projects and invitations for collaboration. Inside Workflows, you will be able to get an elaborate view of the running and past workflow, along with their respective analytics and logs. ChaosHubs is where you can access the chaos experiments, by default you can access all the experiments listed under the Litmus’ ChaosHub but you can also set it up with your own hub. Under Analytics you’d be able to get a vivid analysis of the different aspects of your chaos workflows, using the graphs. Finally, Settings allow you to modify your personal information, automate workflows using GitOps, etc.

Let’s proceed on with the workflow creation for our pod-delete experiment. In Dashboard, click on the Schedule a Workflow button and choose the self-agent, since the application we’re targetting is deployed within the same Kubernetes cluster.

Click Next. Select “Create a new workflow using the experiments from MyHub” and from the dropdown, choose “Chaos Hub”. This is because the experiment we’re trying to perform here i.e. the pod-delete experiment, is a part of the Chaos Hub and we can directly use it, without the need to define it ourselves.

Click Next. Under the Workflow Settings, rename your workflow to any name of your choice. Don’t alter the namespace since our Litmus installation is supposed to use the litmus namespace only. Add a description of your own choice.

Click Next. Inside Tune Workflow, currently, you’d be able to view only one step listed in the flow chart and that is “install-chaos-experiments”.

Click on Add a new experiment and choose generic/pod-delete.

Click Done.

As you can observe, in the flow chart a second step “pod-delete” has been listed now. Although for the purpose of this demo we’d keep the workflow simple with only one experiment, one can design their entire workflow by adding more experiments as per the desired sequence. Its as simple as that!

Now, we’d specify the target deployment on which the chaos workflow will be executed. Click on Edit YAML and scroll down to line number 134 in the editor.

Here, we need to specify the appLabel of our hello-world application by overriding the default app=nginx value. To check the label of our deployment, we can use the kubectl get deployments --show-labels command:

The label of our deployment is app=hello-world so we’d simply replace app=nginx with app=hello-world.

Click Save Changes. Back in the portal, keep Revert Schedule as TRUE. This ensures that all the changes made to the system during the execution of the workflow will get reverted post-completion of the workflow. Click Next.

In Reliability Score, we can add a numeric score between 1 to 10 for each of our experiments. Litmus will simply use this weightage while calculating the resiliency score at the end of the workflow.

Click Next. Inside Schedule, we can schedule our workflows to be executed in a determined frequency or we can simply schedule it to execute right away. Choose “Schedule now”.

Click Next. Inside Verify and Commit, you can review your workflow details and make changes if required before finally executing the workflow.

Once you’re satisfied with the configuration, click Finish. You have successfully started the Chaos Workflow now:

Click Go to Worklow and choose the currently running workflow from the Workflows dashboard. You can view the graphical view of the workflow as it is actively performing the pod-delete experiment on our deployed application.

In effect, this experiment will first install the chaos experiment resources in the cluster, causing one of the three pods in our application deployment to be randomly chosen and deleted, and finally perform a cleanup of the chaos experiment resources.

The application deployment is therefore expected to spin up a new pod in replacement of the forcefully deleted pod for the experiment to be successful. Otherwise, if a new pod fails to spin up, then our system is definitely not resilient.

After a few minutes, you’d be able to see that the experiment has successfully completed:

All the steps of the experiment were successful and our system was able to successfully cope up even in the scenario of forceful pod-delete. We can further analyze the details regarding the experiment inside the Table View:

As we can see, we have obtained a 100% resilience score, and the other pieces of information of our experiment are also visible here. Furthermore, we can analyze the workflow analytics to better understand the impact of the workflow upon our system. Click Back and in the Workflow dashboard, click on the options ellipsis icon:

Choose Show the analytics.

The analytics graph shows the sequence of chaos experiments, the resiliency score, and the state of other parameters over the timeline of the execution of the workflow.

In conclusion of this blog series, let us appreciate that how we went all the way from containers to chaos engineering, starting from learning how to Dockerize a Node.js application, deploying our Docker container using Kubernetes, and finally using LitmusChaos to perform a pod-delete chaos experiment on our application.

Once again, welcome to the world of containers and chaos engineering. Come join me at the Litmus community to contribute your bit in developing chaos engineering for everyone. Stay updated on the latest Litmus trends through the Kubernetes Slack channel (Look for #litmus channel).

Don’t forget to share these resources with someone who you think might benefit from them. Thank you. 🙏

Part-1: A Beginner's Practical Guide to Containerisation and Chaos Engineering with LitmusChaos 2.0

Neelanjan Manna — Fri, 02 Jul 2021 14:46:31 +0000

Part 1: Containers 101, Deploy a Node.js App Using Docker and Kubernetes

This blog is part one of a two blog series that details how to get started with containerization using Docker and Kubernetes for deployment, and later how to perform chaos engineering using LitmusChaos 2.0. Find Part-2 of the blog here.

So you’ve just come across this term called Containerisation and now you’re wondering that as an aspiring software product engineer, or a DevOps engineer, what role will it play in your day-to-day work. After all, applications can be deployed without containers, and in fact, that has been the norm for a long time until containerization technologies like Docker, Kubernetes came into the big picture. So what’s all the fuss about?

In this blog, I’d try to answer all your questions, starting from what are containers, how containers got themselves into the limelight, why one should use them, what is container orchestration, advantages of container orchestration, and finally we will deploy a Node.js application using Docker and Minikube Kubernetes. Please take note that this blog will try to put more emphasis on the practical aspect of using Containers, and won’t encompass the basic theoretical concepts of Docker and Kubernetes at a large, but only those concepts which are necessary to understand the demo.

What are Containers?

As Docker defines it:

A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another.

Simply said, containers allow applications to run in an isolated environment of their own, along with all its dependencies. This decoupling makes it simple and consistent to bundle and deploy container-based applications along with all their dependencies, regardless of whether the target environment is a private data center, the public cloud, or even a developer’s personal laptop.

Difference between Containers and Virtual Machines

A hypervisor virtualizes physical hardware in conventional virtualization. As a result, each virtual machine has a guest OS, a virtual copy of the hardware that the OS needs to run, and an application with all of its related libraries and dependencies. On the same physical server, multiple virtual machines running different operating systems will coexist. A VMware VM, for example, can coexist with a Linux VM, which in turn can coexist with a Microsoft VM, and so on.

Containers virtualize the operating system (typically Linux or Windows) rather than the underlying hardware, so each container only contains the portable and its libraries and dependencies. Containers are small, fast, and portable since, unlike virtual machines, they do not need a guest OS in every instance and can instead rely on the host OS’s features and resources.

Containers, like virtual machines, allow developers to make better use of physical machines’ CPU and memory. On the other hand, Containers go much further because they support microservice architectures, which allow for more granular deployment and scaling of application components. This is a more appealing option than scaling up an entire monolithic framework because a single component is experiencing load issues.

Why we should Use Containers

Faster time to market: Keeping the competitive advantage needs new software and services. Organizations may use growth and organizational agility to accelerate the implementation of new services.
Deployment velocity: Containerization allows a quicker move from production to implementation. It allows DevOps teams to reduce deployment times and frequency by breaking down barriers.
Reduction of IT infrastructure: Containerization increases the density of device workloads, improve the utilization of your server compute density, and cut software licensing costs to save money.
Performance in IT operations: Containerization allows developers to streamline and automate the management of multiple applications and resources into a single operating model to improve operational performance.
Obtain greater freedom of choice: Any public or private cloud can be used to package, ship, and run applications.

What is Container Orchestration

As VMware defines it:

Container orchestration is the automation of much of the operational effort required to run containerized workloads and services. This includes a wide range of things software teams need to manage a container’s lifecycle, including provisioning, deployment, scaling (up and down), networking, load balancing and more.

Running containers in production can easily become a huge effort due to their lightweight and ephemeral nature. When used in conjunction with microservices, which usually run in their own containers, a containerized application will result in hundreds or thousands of containers being used to construct and run any large-scale system.
If handled manually, this can add a lot of difficulties. Container orchestration, which offers a declarative way of automating much of the job, is what makes the organizational complexity manageable for creation and operations, or DevOps. This makes it a natural match for DevOps teams and cultures, which aim for much greater speed and agility than conventional software development teams.

Advantages of Container Orchestration

Improved Resilience: Container orchestration software can improve stability by automatically restarting or scaling a container or cluster.
Simplification of Operations: The most significant advantage of container orchestration, and the primary explanation for its popularity, is simplified operations. Containers add a lot of complexity, which can easily spiral out of control if you don’t use container orchestration to keep track of it.
Enhanced Security: Container orchestration’s automated approach contributes to the protection of containerized applications by reducing or removing the risk of human error.

Demo: Deploy a Node.js App Using Docker and Kubernetes

Let’s get our hands dirty by deploying a simple Node.js application using a Docker container, followed by deploying the container image in a Minikube Kubernetes cluster in our own development machine.
Before we move on to the actual demo, let's check a few pre-requisites off the list so that we will all be on the same page:

Node.js version 10.19.0
Docker version 20.10.6
Minikube version 1.20.0
Virtualbox 6.1.6
Kubectl

It's worth mentioning that I will be using a machine running on Ubuntu 20.04 for this demo, though you should be fine with a Windows machine too. For this demo, we’ll not cover the installation part of Docker and Minikube since they are pretty straightforward and require no special instruction, and focus on the deployment part only.

1. Node.js Application

Let’s start with a very basic Node.js “Hello World” application for this demo. The application has been developed as any other Node.js application, after initializing an empty repository using npm init. The package.json generated is as follows:

Once that’s done, we can set up our index.js as follows:

Here we have a pretty basic Node.js server and all it does is serve the string ‘Hello World’ when a GET request is sent to the loopback address, at port 3000. Upon executing the above code using node index.js we obtain the following output:

And we do get our Hello World at http://127.0.0.1:3000/

Simple, right? Once that’s done, let’s move on to the sweet part, creating a container image of our application using Docker.

2. Docker

Once we have our application up and running, we can proceed to dockerizing the application. But before we do that, let us initialize a startup script in our package.json so that our application can be readily executed by Docker. Therefore, we’d modify our package.json as follows:

Now we are all set to Dockerize our application! To do that we need to simply create a Dockerfile in the same directory as of our index.js. A Dockerfile is simply a set of instructions required for building the container image of the application. Our Dockerfile looks something like this:

Let us walk through each of these commands to better understand their purpose.

The FROM instruction initializes a new build stage and sets the Base Image for subsequent instructions. A Base Image is simply a Docker image that has no parent image, which is created using the FROM scratch directive. As such, a valid Dockerfile must start with a FROM instruction. Here we are using the node v12.0 slim Base Image.

The WORKDIR command is used to set the working directory for all the subsequent Dockerfile instructions. If the WORKDIR is not manually created, it gets created automatically during the processing of the instructions. It does not create new intermediate Image layers. Here we set our working directory as /app.

The COPY instruction copies new files or directories from <src> and adds them to the filesystem of the container at the path <dest>. Here we are copying the package.json file to /app directory. Interestingly, we don’t copy the rest of the files into /app just yet. Can you guess why? This is because we’d like Docker to cache the first 3 commands so that every time we run docker build we won’t need to execute those commands, and thus improve our build speed.

RUN command can be used in two ways; either through a Dockerfile as shown here, otherwise through the Docker CLI. The RUN instruction will execute any commands in a new layer on top of the current image and commit the results. The resulting committed image will be used for the next step in the Dockerfile. To install the Node.js app dependencies from the package.json file, we use the RUN command here.

Next, we use COPY to move all the remaining files to the /app directory.

Finally, we use the CMD command to execute the Node.js application. The main purpose of a CMD is to provide defaults for an executing container. These defaults can include an executable, or they can omit the executable, in which case we must specify an ENTRYPOINT instruction as well. There can only be one CMD instruction in a Dockerfile. If we list more than one CMD then only the last CMD will take effect.

And we’re done with the Dockerfile. Now we’re all set to build our container image, which can be done using the command docker build -t hello-world:1.0 .

The only important takeaway here is that we MUST tag our image via -t flag, as it’d be very much beneficial for us to deploy and manage the container image later on. Here we have specified the name of our container image as hello-world and tagged it with its version 1.0. We had seen a similar practice with our Base Image node:12-slim as well. Lastly, we specified the directory from where the Dockerfile is located using the . path.

Upon building the image, Docker tries to fetch the Base Image if it's not present in the local registry. Next, it executes the set of instructions given in the Dockerfile in a sequential order to complete the build process. Here’s the build output:

Upon a successful build, we can view our image using the command docker images:

Notice the node image is also present here since we have used it as our Base Image. Now we’re all set to run our application. Let’s run our docker container using the command docker run -it -p 3000:3000 hello-world:1.0

Here we’re specifying that we want to run the container image in an interactive mode using the flag -it. Further, we specify the port mapping of our container using the -p flag, where we direct that the 3000 port of the host machine is mapped to the 3000 port of the container. Finally, we specify the name and tag of the image to be run. Hence we obtain:

Thus, we have successfully deployed a container image of our application using Docker, which we can verify by visiting http://127.0.0.1:3000/:

Further, we can run the container in detached mode by specifying -d flag in place of -it in the previous command: docker run -d -p 3000:3000 hello-world:1.0

The long alphanumeric string output is nothing but the UUID long identifier of the running container. We can further inspect the properties of this container using the command docker container ls:

Amazing! We have just deployed our very first Docker container and now we’re all set for our next destination: Kubernetes.

3. Kubernetes

As the container image of our application is now ready, all that’s left is to deploy our container image to a Kubernetes deployment. We’d use a Minikube cluster for the deployment of our container image locally.

It's important to take note that a Minikube cluster would have only one worker node, which will be created using a virtual machine, and the control plane will reside in our own machine only.

To start Minikube, we can use the command minikube start. It's worth pointing out that a minimum of 2 CPUs, 2GB RAM, and 20GB Disk Space is required for starting a Minikube cluster using this command. One may check the number of processing units in their machine using the npoc command:

Once the Minikube cluster starts up, you’d get the following output in the terminal:

Now that our Minikube cluster is up and running, let’s devise a deployment strategy for our container image.

Currently, our container image is stored locally in our own Docker Local Registry which is present in our machine. The Registry is nothing but a stateless, highly scalable server-side application that stores and allows the distribution of Docker images. So, either we can use the image directly from the Local Registry to deploy it in the Minikube server, or we can first push our image to a Hosted Registry such as Docker Hub and later pull it for the image deployment. The latter is a more suitable approach when working in a team.

The former approach, however, has an unnoticed aspect. Minikube comes with its own Docker ecosystem when we install it on our machine. If we create Docker images on our computer and try to use them in a Kubernetes deployment, we will get the ErrImageNeverPull error since it always tries to get the image from its own Local Registry or Docker Hub, resulting in an error as the pod is started.
To verify this, let’s do a small experiment. We can still view our Local Registry images using the command docker images:

Now, let’s try to access the Docker Images of Minikube. To do that we need to first SSH into the Minikube’s VM. To do that, we can use the command minikube ssh:

Now that we are inside the Minikube’s VM, we can again use the docker images command and this time the hello-world image or the node image is nowhere to be found:

The SSH can be exited using the exit command.
To get around this issue, we have two options. Either we can push our image first to a Hoster Registry and then pull it into our Minikube VM’s Docker, or we can directly build our Docker image using Minikube’s Docker daemon. Let’s try to deploy our container image using the second approach.

Firstly, we need to set the environment variable using the eval command as eval $(minikube docker-env). This will allow us to use the Docker daemon of Minikube to be used for the subsequent command. You can confirm that now the Minikube’s Docker is being used by using the docker images command again and this time you’d find the images from your Minikube Docker:

Next, use the build command for docker image as you’d normally do using the command docker build -t hello-world:1.0 .:

Now that our Docker image is in the right Registry, we can proceed towards the actual deployment. We can deploy our image either using a deployment manifest file or by using kubectl create deployment command directly. The first approach is a better approach since it gives us much more flexibility to specify our exact Pod configuration.

Let’s define our deployment.yml manifest:

A few interesting observations are to be made over here. Notice that replicas has been set to 1 for the time being. This means that there will always be exactly one pod for our deployment under the present configuration. The imagePullPolicy is set to “Never” as the image is expected to be fetched from the Minikube Docker’s Local Registry. Finally under ports, the containerPort has been set to 3000 because our Node.js application listens on that port.

Now, let’s create the container deployment using the following command kubectl create -f deployment.yml, assuming that the terminal is open in the directory where deployment.yml file is present. We get the following output:

Woohoo! We just deployed our container image in the Kubernetes cluster! To verify our deployment, we can see all the deployments in our cluster using the command kubectl get deployments:

As we can see, our deployment is successfully created. We can also inspect the pods associated with this deployment using the command kubectl get pods:

As per our manifest file, only one pod is created. Though, we can still increase or decrease the number of Pod replicas in our deployment using the following command kubectl scale --replicas=3 deployment hello-world. This command will create 2 more Pods which will be exact replicas of the Pod that we have created:

We can verify the newly created Pods by again using the command kubectl get pods:

Though we have deployed our container image, still we can’t access the application just yet, for the lack of a Service. As we know, Service in Kubernetes is an abstraction that defines a logical set of Pods and a policy by which to access them. Services enable a loose coupling between dependent Pods. For our purpose, we’d create a NodePort Service to be able to access our deployment. A NodePort exposes the Service on the same port of each selected Node in the cluster using NAT.

Let’s define a service.yml manifest:

A few points to notice: Under ports, we have defined the port and targetPort in relation to the Service itself i.e. port refers to the port exposed by the service and targetPort refers to the port used by our deployment aka. the Node.JS application.

To create this Service we’d use the following command kubectl create -f service.yml, assuming that the terminal is open in the directory where service.yml file is present. We get the following output:

We just created our NodePort service, which we can verify using the command kubectl get services:

As evident, we have successfully created a NodePort Service. Noticeably, it doesn’t have an external IP, therefore we can use port-forwarding to access our deployment at a specified port. The command to do so is specified as kubectl port-forward svc/hello-world-svc 3000:3000:

And now if we go to http://127.0.0.1:3000:

Congratulations! You have just deployed a containerized application using Docker and Kubernetes by adhering to the best practices. Though we have touched only the tip of the iceberg, I hope this demo has made you a little bit more familiar with the containers, and how Docker and Kubernetes can be used for containerizing and deploying applications.

In the next part of this series, we’d explore the world of Chaos Engineering using Litmus Chaos! We’d understand the best principles of Chaos Engineering and witness how Litmus Chaos performs Kubernetes-Native Chaos Engineering for attaining unparalleled resiliency in our Kubernetes application.

With that, I’d like to welcome you to the world of containers and chaos engineering. Come join me at the Litmus community to contribute your bit in developing chaos engineering for everyone. Stay updated on the latest Litmus trends through the Kubernetes Slack channel (Look for #litmus channel).

Don’t forget to share these resources with someone who you think might benefit from them. Thank you. 🙏