Chaos Engineering with LitmusChaos on AWS - Cross Account Implementation

In our last post, we learned about LitmusChaos, how to get started with a simple experiment, IAM Role for Service Account and more. In this blog post, we are going to see how you can run chaos experiments using LitmusChaos on AWS, if you have two different AWS accounts. Let us get started.

The deployment to production happens at a fast pace in a carefully constructed value stream. Primarily, we (performance engineers) work in a non-production environment. Though we run experiments in production, lower environments such as dev, QA, and performance environments can be your playground for your chaos experiments. Often, most of the organization will have more than one account, just to isolate the engineering units, environments, and more, just to avoid stepping on each other's toe.

As a chaos engineer, you may need to run experiments on the applications which are hosted in different AWS accounts.

Consider a simple application hosted on an EC2 instance in the AWS Account marketing-team. LitmusChaos on AWS EKS runs on a performance-team AWS account. In this case, we need to enable cross - AWS account access.

Chaos Engineering with LitmusChaos on AWS - Cross Account Implementation

Prerequisites

The following are the prerequisites for implementing this setup:

1. Two AWS Accounts with IAM privileges
2. LitmusChaos on AWS Account A
3. Sample application on AWS Account B
4. eksctl
5. kubectl

Creating two AWS accounts might be tricky as you need two different credit cards to sign up. I have borrowed an account from my friend, just to demonstrate this implementation.

Also, make sure you have valid privileges to play around with the IAM, Policy, Trust Relationships and more.

Let us name our AWS accounts to understand better. You are a Chaos Engineer who provides Chaos-as-a-Service. You run LitmusChaos on your AWS account called Chaos-Account.

A customer has approached you to perform chaos experiments on their application. Let us call their AWS account a Customer-Account.

On Chaos-Account

On Chaos-Account make sure you have LitmusChaos is up and running successfully and able to run experiments either using Self-Agent or external agent.

Create a new IAM Role using the below policy and trust relationship.

Before creating IAM Role, make sure you have the valid OIDC provider URL.

To validate OIDC URL, enter the below command

aws iam list-open-id-connect-providers

The above command must return the OIDC entry e.g. oidc.eks.us-east-2.amazonaws.com/id/F25CB34E2ACAA3CE669BA4E0XXXXXXXX"

If not, create an OIDC using the below command. In this case, our cluster name is chaos-cluster.

eksctl utils associate-iam-oidc-provider --cluster chaos-account --approve

Create a policy Chaos-Account-Policy using the below JSON. Please follow the Zero Trust framework while creating a policy and trust relationship. I am just being playful as it is my demo account.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "ec2:*",
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "sts:*",
            "Resource": [
                "*"
            ]
        }
    ]
}

Create a new IAM role called Chaos-Account-Role and attach the above policy and add the below trust relationship. Make sure you replace the OIDC ID, account ID, region etc.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<account_id>:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/F25CB34E2ACAA3CE669BA4EXXXXXXX"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-east-1.amazonaws.com/id/F25CB34E2ACAA3CE669BA4EXXXXXXX:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

On Customer-Account

Login into your Customer-Account or ask them to create a new policy called Customer-Account-Policy using the below JSON. Again, please follow the Zero Trust policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:*",
                "sts:*"
            ],
            "Resource": "*"
        }
    ]
}

Before creating a new role, let us create Identity provider in this account. Go to IAM, under Access management, click Identity providers.

Then, click Add provider button.

Select OpenID Connect

In Provider URL text box, enter the Chaos-Account identity provider URL, and then click Get thumbprint button.

In Audience text box, enter sts.amazonaws.com.

Click on Add provider button.

Add Provider

Create a new role by selecting Web identity as shown below and select your Identity provider and Audience [i.e. Chaos-Account] and then click Next

Adding Web Identity Role

In the next page, select the Customer-Account-Policy then create a role Customer-Account-Role.

Customer Account Role

Copy the Customer-Account-Role ARN to the clipboard as we need it to annotate in the LitmusChaos service account.

Annotation

Use the below command to annotate your LitmusChaos service account in the default namespace litmus.

kubectl annotate serviceaccount -n litmus litmus-admin eks.amazonaws.com/role-arn=arn:aws:iam::<customer-account-id>:role/Customer-Account-Role --overwrite

To validate the annotation, use the below command:

kubectl describe sa -n litmus litmus-admin

Chaos Experiment

Now it is time to create a chaos experiment to stop the EC2 instance in Customer-Account.

Copy the instance ID and the region which we need to feed it in the experiment YAML.

Create a ec2-terminate-by-id experiment and configure the instance ID and the region. Make sure you remove the secrets from the YAML and schedule it.

If all is well, it should stop and start the EC2 instance in your customer account.

Graphical View of Chaos Result

Chaos Verdict

Pod Logs

Conclusion

As you learned, cross - account implementation is easy and straightforward. It looks complex, but once you understand the OIDC concept, how role, policy, and trust relationships works, it is quick to implement. The above steps can be automated by leveraging AWS CLI and eksctl.