DEV Community: Shardul Srivastava

Getting Started with Contributing to CNCF: A Beginner's Guide

Shardul Srivastava — Sat, 08 Feb 2025 21:46:28 +0000

Contributing to CNCF is not difficult, but let me tell you what's difficult..

Finding out where to start contributing.

So Let's start to peel the onion, the first thing you should do is to go through this amazing CNCF Landscape. This landscape covers not just the CNCF projects but also member projects.

You can find an exhaustive list of all the CNCF Graduated and Incubating projects here.

Did I say Gradudated and Incubating... What's that ??

CNCF has a lot of projects under its umbrella, they are categorized based on their maturity levels, here are three different categories:

Graduated - CNCF Graduated projects are mature, widely adopted, and have a mind-boggling community behind them. Kubernetes is one of the examples of Graduated CNCF projects.
Incubating - CNCF Incubating projects are on their way to graduation but are still maturing. They have proven technical merit and community growth but may need to enhance their processes.
Sandbox - CNCF Sandbox projects are early-stage projects that are in the process of establishing their community and technical merit

Here is a very handy link to all the CNCF projects across maturity levels.

This would help you to understand what are the projects and you will find that most of the cloud-related open-source projects are maintained by CNCF.

How to find the right issues

Use these Github queries to find the beginner-friendly issues from CNCF graduated, incubating and sandbox projects.

CNCF Graduated Projects

is:issue is:open  label:"good first issue"  repo:kubernetes/kubernetes repo:prometheus/prometheus repo:etcd-io/etcd repo:containerd/containerd repo:linkerd/linkerd2 repo:fluent/fluent-bit repo:helm/helm repo:opencontainers/runc repo:cni-org/cni repo:envoyproxy/envoy repo:jaegertracing/jaeger repo:grpc/grpc repo:vitessio/vitess repo:tuf/tuf repo:opentelemetry/opentelemetry-collector repo:tikv/tikv repo:dragonflyoss/Dragonfly2 repo:argoproj/argo-cd repo:falcosecurity/falco repo:openpolicyagent/opa repo:thanos-io/thanos repo:fluxcd/flux2 repo:longhorn/longhorn repo:cert-manager/cert-manager repo:contour-org/contour repo:emissary-ingress/emissary repo:harvester/harvester repo:k3s-io/k3s repo:kubeedge/kubeedge repo:kubeflow/kubeflow repo:kubewarden/kubewarden-controller repo:kyverno/kyverno repo:openservicemesh/osm repo:paralus/paralus repo:serverlessworkflow/specification repo:spire/spire repo:tremor-rs/tremor-runtime repo:wasmCloud/wasmcloud repo:keptn/keptn repo:backstage/backstage repo:openfeature/spec repo:cloudevents/spec repo:knative/serving repo:crossplane/crossplane repo:openebs/openebs repo:opencost/opencost repo:open-cluster-management-io/OCM repo:operator-framework/operator-sdk repo:porter-dev/porter repo:pravega/pravega repo:submariner-io/submariner repo:telepresenceio/telepresence repo:tricksterproxy/trickster repo:virtual-kubelet/virtual-kubelet repo:volcano-sh/volcano repo:weaveworks/weave repo:openyurtio/openyurt repo:meshery/meshery repo:chaos-mesh/chaos-mesh repo:openkruise/kruise repo:openelb/openelb repo:sealer/sealer repo:devfile/api repo:antrea-io/antrea repo:litmuschaos/litmus repo:karmada-io/karmada repo:openfunction/openfunction repo:open-gateway/gateway-api

CNCF Incubating Projects

is:issue is:open label:"good first issue"
repo:artifacthub/hub repo:backstage/backstage repo:buildpacks/pack repo:chaos-mesh/chaos-mesh repo:cloud-custodian/cloud-custodian repo:containernetworking/cni repo:projectcontour/contour repo:cortexproject/cortex repo:crossplane/crossplane repo:dragonflyoss/Dragonfly2 repo:emissary-ingress/emissary repo:flatcar/flatcar repo:grpc/grpc repo:karmada-io/karmada repo:keptn/keptn repo:keycloak/keycloak repo:knative/serving repo:kubeflow/kubeflow repo:kubescape/kubescape repo:kubevela/kubevela repo:kubevirt/kubevirt repo:kyverno/kyverno repo:litmuschaos/litmus repo:longhorn/longhorn repo:nats-io/nats-server repo:notaryproject/notary repo:opencost/opencost repo:open-feature/spec repo:openkruise/kruise repo:opentelemetry/opentelemetry-collector repo:openyurtio/openyurt repo:operator-framework/operator-sdk repo:strimzi/strimzi-kafka-operator repo:thanos-io/thanos repo:volcano-sh/volcano repo:wasmCloud/wasmcloud

CNCF Sandbox Projects

is:issue is:open label:"good first issue"
repo:aeraki-mesh/aeraki repo:akri-sh/akri repo:antrea-io/antrea repo:armadaproject/armada repo:yahoo/athenz repo:runatlantis/atlantis repo:banzaicloud/bank-vaults repo:bfenetworks/bfe repo:bpfman/bpfman repo:projectcapsule/capsule repo:carina-io/carina repo:cartography-cncf/cartography repo:carvel-dev/ytt repo:cdk8s-team/cdk8s repo:chaosblade-io/chaosblade repo:cloudnative-pg/cloudnative-pg repo:clusternet/clusternet repo:clusterpedia-io/clusterpedia repo:confidential-containers/confidential-containers repo:connectrpc/connect-go repo:containerssh/containerssh repo:project-copacetic/copacetic repo:cozystack/cozystack repo:devfile/api repo:devspace-sh/devspace repo:devstream-io/devstream repo:dexidp/dex repo:distribution/distribution repo:easegress-io/easegress repo:eraser-dev/eraser repo:external-secrets/external-secrets repo:fluid-cloudnative/fluid repo:Project-HAMi/HAMi repo:headlamp-k8s/headlamp repo:hexa-org/policy-orchestrator repo:hwameistor/hwameistor repo:hyperlight-dev/hyperlight repo:inclavare-containers/inclavare-containers repo:inspektor-gadget/inspektor-gadget repo:interTwin-eu/interLink repo:k0sproject/k0s repo:k3s-io/k3s repo:k8gb-io/k8gb repo:k8sgpt-ai/k8sgpt repo:k8up-io/k8up repo:kairos-io/kairos repo:kanisterio/kanister repo:kcl-lang/kcl repo:kcp-dev/kcp repo:sustainable-computing-io/kepler repo:keylime/keylime repo:kgateway-dev/kgateway repo:kitops-ml/kitops repo:kmesh-net/kmesh repo:ko-build/ko repo:konveyor/operator repo:koordinator-sh/koordinator repo:kptdev/kpt repo:krkn-chaos/krkn repo:kuadrant/kuadrant-operator repo:kuasar-io/kuasar repo:kube-burner/kube-burner repo:kubeovn/kube-ovn repo:kube-rs/kube repo:kube-vip/kube-vip repo:kubean-io/kubean repo:kubearmor/kubearmor repo:kubeclipper/kubeclipper repo:kuberhealthy/kuberhealthy repo:kubeslice/kubeslice repo:kubestellar/kubestellar repo:kubewarden/kubewarden-controller repo:kudobuilder/kudo repo:kumahq/kuma repo:kubereboot/kured repo:KusionStack/kusion repo:lima-vm/lima repo:kube-logging/logging-operator repo:loxilb-io/loxilb repo:merbridge/merbridge repo:meshery/meshery repo:metallb/metallb repo:metal3-io/baremetal-operator

How to start doing the setup.

Well every project has it's own guide on how to get started, reading through CONTRIBUTING.md would help you with setting up the project local.

How to check your contributions

CNCF has a score system that tells how much you have contributed to CNCF projects, you can check your score here.

https://devstats.cluster.fun/

If you came this far, let me share something that will help you to get up to date with what's going on

How to interact with the community behind the projects

If you are interested in contributing into any projects, always join their slack channel in CNCF, here is how you can get invited to the CNCF slack - https://communityinviter.com/apps/cloud-native/cncf.

Running Apache Kafka on Containers

Shardul Srivastava — Sun, 30 Oct 2022 18:12:00 +0000

Apache Kafka is one of the most famous data stores. It's a go-to tool to collect streaming data at scale and process them with either Kafka streams or Apache Spark.

Getting started with Kafka is challenging as it involves familiarising a lot of new concepts such as topics, replication, and offsets, and then you have to understand what a Zookeeper is.

Confluent is a company specialising in Kafka with their cloud-based offering called Confluent cloud. Confluent is one of the biggest contributors to the Kafka open-source project. they have created some great tools to help with Kafka such as KsqlDB that allows us to query streaming data (It's amazing, try it).

Apart from that Confluent has great articles on understanding Kafka internals.

Kafka with Docker

To get started with Kafka on Docker, we are going to use confluent Kafka images.

Create a docker-compose.yaml file with one zookeeper and one Kafka container:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:latest
    container_name: broker
    ports:
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://kafka:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Start both containers in detached mode:

docker-compose up -d

docker-compose starts zookeeper on port 2181 and Kafka on port 9092 along with some configurations:

Zookeeper
1. ZOOKEEPER_CLIENT_PORT - Port where Zookeeper would be available.
2. ZOOKEEPER_TICK_TIME- the length of a single tick.
Kafka
1. KAFKA_ZOOKEEPER_CONNECT - Instructs Kafka how to connect to ZooKeeper.
2. KAFKA_LISTENER_SECURITY_PROTOCOL_MAP - Defines key/value pairs for the security protocol to use, per listener name.
3. KAFKA_ADVERTISED_LISTENERS - A comma-separated list of listeners with their host/IP and port. Read more about kafka listeners here.
4. KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR - Equivalent of broker configuration offsets.topic.replication.factor which is the replication factor for the offsets topic. Since we are running with just one Kafka node, we need to set this to 1.

Read more about how the connection to kafka broker works in a docker container here

Create Topics in Kafka

Kafka topics are like database tables. Just like a database, we have to create a table to start storing the data, for Kafka we have to create a topic.

Unlike a database that has a command to create a database, Kafka comes with some utility scripts, one of which is to create a topic that requires mandatory input as the topic name and a few other optional arguments.

Create a topic by the name kafka-test

/usr/bin/kafka-topics \
             --bootstrap-server broker:9092 \
             --create \
             --topic kafka-test
Created topic kafka-test.

Try running this command again and you will get this error:

Error while executing the topic command: Topic 'kafka-test' already exists.

Not so much CI friendly right ?? --if-not-exists allows you to create a topic if it doesn't exist and retuns exit code 0.

/usr/bin/kafka-topics \
             --bootstrap-server broker:9092 \
             --create \
             --if-not-exists \
             --topic kafka-test

There are a couple of other arguments that are essential for a good understanding of Kafka:

--partitions - Kafka topics are partitioned i.e the data of topics are spread across multiple brokers for scalability.
--replication-factor - To make data in a topic fault-tolerant and highly-available, every topic can be replicated, so that there are always multiple brokers that have a copy of the data.

What's a Kafka Message

Once we have the topic created, we can start sending messages to the topic. A Message consists of headers, a key, and a value. Let's talk about each of these.

Headers - Headers are key-value pairs and give the ability to add some metadata about the kafka message. Read the original KIP(Kafka Improvement Proposals) proposing headers here.
Key - Key for the kafka message. The key value can be null. Randomly chosen keys (i.e. serial numbers and UUID) are the best example of message keys. Read more about when you should use a key here.
Value - Actual data to be stored in kafka. Could be a string, json, Protobuf, or AVRO data format.

Writing to a Kafka Topic

Kafka provides a Producer API to send a message to the Kafka topic. This API is available in java with kafka-clients library and python with kafka-python package.

Luckily for us, we don't have to use any of these. Kafka comes with an out of box script kafka-console-producer that allows us to write data to a kafka topic.

Run the command and as soon as the command returns > with a new line, enter the Json message:

/usr/bin/kafka-console-producer \
    --bootstrap-server kafka:9092 \
    --topic kafka-test
>{"tickers": [{"name": "AMZN", "price": 1902}, {"name": "MSFT", "price": 107}, {"name": "AAPL", "price": 215}]}

We have successfully sent a message to the topic.

Enter Control + C to stop the script.

however this message was sent without any key, To send a key we have to set the properties parse.key to allow sending the key.

Default key separator is \t(tab),we can change it by setting the property key.separator. Eg: --property "key.separator=:"

Let's try to send a message with a random key stocks-123 this time:

/usr/bin/kafka-console-producer \
    --bootstrap-server kafka:9092 \
    --topic kafka-test \
    --property "parse.key=true"
>stocks-123 {"tickers": [{"name": "AMZN", "price": 1902}, {"name": "MSFT", "price": 107}, {"name": "AAPL", "price": 215}]}

With the release of kafka version 3.2.0, it's possible to send headers using ConsoleProducer by setting the property parse.headers to true.

Headers are metadata about the kafka message, souce of these stock prices would be a good candidate for the headers. Let's add a header key as stock_source and value as nyse to the Kafka message :

/usr/bin/kafka-console-producer \
    --bootstrap-server kafka:9092 \
    --topic kafka-test \
    --property "parse.key=true" \
    --property "parse.headers=true"
>stock_source:nyse  stocks-123  {"tickers": [{"name": "AMZN", "price": 1902}, {"name": "MSFT", "price": 107}, {"name": "AAPL", "price": 215}]}

Now we have successfully sent a kafka message with a header, key and value.

Check out supported properties for kafka-consoler-producer here.

Reading from a Kafka Topic

Kafka provides a Consumer API to read messages from a Kafka topic. This API is available in java with kafka-clients library and python with kafka-python package.

Kafka comes with an out of box script kafka-console-consumer to read messages from the kafka topic:

/usr/bin/kafka-console-consumer \
    --bootstrap-server kafka:9092 \
    --topic kafka-test

However, this command only prints the values of the kafka message. To print the key and headers, we have to set the properties print.headers, print.key to true. We can also print the timestamp of the message with the property print.timestamp.

/usr/bin/kafka-console-consumer \
    --bootstrap-server kafka:9092 \
    --topic kafka-test \
    --property print.headers=true \
    --property print.key=true \
    --property print.timestamp=true

There is other information such as partition and offset, they can be printed by setting the properties --property print.offset=true and --property print.partition=true.

Everytime we read from the a kafka topic, Kafka keeps track of the last offset the consumer read from and allows you to read from that point next time, however we can always read from the beginning using the arguments --from-beginning.

To always read from a kafka topic from the beginning:

/usr/bin/kafka-console-consumer \
    --bootstrap-server kafka:9092 \
    --topic kafka-test \
    --from-beginning \
    --property print.headers=true \
    --property print.key=true \
    --property print.timestamp=true

Is this too much to remember ??
Don't worry we have an easier way of reading and writing from Kafka topics.

kcat Utility

kcat is an awesome tool to make our lives easier, it allows us to read and write from kafka topics without tons of scripts and in a more user-friendly way.

As Confluent puts it, "It is a swiss-army knife of tools for inspecting and creating data in Kafka"

kcat has two modes, it runs in producer mode by specifying the argument -P and consumer mode by specifying the argument -C.It also automatically selects its mode depending on the terminal or pipe type. If data is being piped to kcat it will automatically select producer (-P) mode. If data is being piped from kcat (e.g. standard terminal output) it will automatically select consumer (-C) mode.

To read data from kafka topics, simply run
```
kcat -b localhost:9092 -t kafka-test
```

To write data to a Kafka topic, run

kcat -P -b localhost:9092 -t kafka-test

Take a look at the examples here to find out more about the usage.

here are some tips and tricks of using Kafka.

Getting Started with ACK RDS Controller

Shardul Srivastava — Tue, 18 Oct 2022 20:31:17 +0000

ACK controller for RDS is a Kubernetes Controller for provisioning RDS instances in a kubernetes native way.

ACK controller for RDS supports creating these database engines:

Amazon Aurora (MySQL & PostgreSQL)
Amazon RDS for PostgreSQL
Amazon RDS for MySQL
Amazon RDS for MariaDB
Amazon RDS for Oracle
Amazon RDS for SQL Server

Install ACK Controller for RDS

Let's start by setting up an EKS cluster. ACK Controller for RDS requires AmazonRDSFullAccess to create and manage RDS instances.
We will also create a service account ack-rds-controller for ACK Controller for RDS and attach AmazonRDSFullAccess IAM permissions to this service account with the help of IRSA.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: ack-cluster
  region: eu-west-1
  version: "1.22"
iam:
  withOIDC: true
  serviceAccounts:
    - metadata:
        # Service account used by rds-controller https://github.com/aws-controllers-k8s/rds-controller/blob/main/helm/values.yaml#L81
        name: ack-rds-controller
        namespace: ack-system
      # https://github.com/aws-controllers-k8s/rds-controller/blob/main/config/iam/recommended-policy-arn
      attachPolicyARNs: 
      - "arn:aws:iam::aws:policy/AmazonRDSFullAccess"

managedNodeGroups:
  - name: managed-ng-1
    minSize: 1
    maxSize: 10
    desiredCapacity: 1
    instanceType: t3.large
    amiFamily: AmazonLinux2

eksctl create cluster -f ack-cluster.yaml

Once the cluster is created, install the ACK Controller for RDS using Helm chart. ACK Controller for RDS helm charts are stored in the public ECR repository.

Note: Check the latest version of the helm chart here

Authenticate to the ECR repo using your AWS credentials :

 aws ecr-public get-login-password --region us-east-1 | helm registry login --username AWS --password-stdin public.ecr.aws
 Login Succeeded

Once you're authenticated, install ACK Controller for RDS in the ack-system namespace :

 helm upgrade \
    --install \
    --create-namespace \
    ack-rds \
    -n ack-system \
    oci://public.ecr.aws/aws-controllers-k8s/rds-chart \
    --version=v0.1.1 \
    --set=aws.region=eu-west-1 \
    --set=serviceAccount.create=false \
    --set=log.level=debug

During installation, we are setting these values in the Helm chart:

aws.region - AWS Region for API Calls.
serviceAccount.create - Setting this to false, since serviceAccount is created by eksctl during cluster creation above.
log.level - Setting this to debug to see detailed logs.

ACK controller for RDS installs several CRDs that allow you to provision RDS instances and related components:

GlobalCluster - Custom resource to create Aurora Global cluster.
DBCluster - Custom resource to create Amazon Aurora DB Cluster.
DBInstance - Custom resource to create Amazon RDS DB Instances.
DBClusterParameterGroup - Custom resource to create DB cluster parameter groups.
DBParameterGroup - Custom resource to create DB parameter groups.
DBProxy - Custom resource to create Amazon RDS Proxy
DBSubnetGroup - Custom resource to create DB Subnet Group.

Note:. Refer here for the reference of these CRDs.

Check ACK controller for RDS logs in cluster:

kubectl logs -f -l app.kubernetes.io/instance=ack-rds

Create Autora PostgreSQL Cluster with RDS Controller

To create an Aurora PostgreSQL Cluster with ACK controller for RDS, we have to first create a DB Subnet Group using DBSubnetGroup, then create a Aurora Cluster using DBCluster and add two Aurora Database instances in the cluster using DBInstance.

Create a DB Subnet Group test-ack-subnetgroup with all the subnets that are part of the VPC created by eksctl or use your existing subnets:

apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBSubnetGroup
metadata:
  name: "test-ack-subnetgroup"
spec:
  name: "test-ack-subnetgroup"
  description: "Test ACK Subnet Group"
  subnetIDs:
    - "subnet-011e4c822231b65fa"
    - "subnet-001ce16b9b7c3578f"
    - "subnet-0a90579d9b066ecf7"
    - "subnet-00e6d2e90ea132c8b"
    - "subnet-0fbc22e542749c98c"
    - "subnet-080c0b1ed39b0aa91"
  tags:
    - key: "created-by"
      value: "ack-rds"

Note: To get all the subnets created by eksctl, run command:

eksctl get cluster ack-cluster -o json| jq -r '.[0].ResourcesVpcConfig.SubnetIds'

Check the status of test-ack-subnetgroup DB Subnet Group :
```
kubectl get dbsubnetgroup test-ack-subnetgroup -ojson| jq -r '.status.subnetGroupStatus'
Complete
```
if it returns Complete, then DB Subnet group is successfully created.

Create a secret to store the password of the master user :

kubectl create secret generic "test-ack-master-password" --from-literal=DATABASE_PASSWORD="mypassword"
secret/test-ack-master-password created

Create an Aurora DB Cluster test-ack-aurora-cluster using DBCluster :

apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBCluster
metadata:
  name: "test-ack-aurora-cluster"
spec:
  engine: aurora-postgresql
  engineVersion: "14.4"
  engineMode: provisioned
  dbClusterIdentifier: "test-ack-aurora-cluster"
  databaseName: test-ack
  dbSubnetGroupName: test-ack-subnetgroup
  deletionProtection: true
  masterUsername: "test"
  masterUserPassword:
    namespace: "default"
    name: "test-ack-master-password"
    key: "DATABASE_PASSWORD"
  storageEncrypted: true
  enableCloudwatchLogsExports:
    - postgresql
  backupRetentionPeriod: 14
  copyTagsToSnapshot: true
  preferredBackupWindow: 00:00-01:59
  preferredMaintenanceWindow: sat:02:00-sat:04:00

Once created, check the status of DBCluster resource test-ack-aurora-cluster:
```
kubectl describe DBCluster test-ack-aurora-cluster
```
if the status is available then Aurora DB Cluster is successfully created.

After DBCluster is successfully created, add two Aurora DB Instances test-ack-aurora-instance-1 and test-ack-aurora-instance-2 to the DB cluster using DBInstance :

apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBInstance
metadata:
  name: test-ack-aurora-instance-1
spec:
  dbInstanceClass: db.t4g.medium
  dbInstanceIdentifier: test-ack-aurora-instance-1
  dbClusterIdentifier: "test-ack"
  engine: aurora-postgresql
---
apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBInstance
metadata:
  name: test-ack-aurora-instance-2
spec:
  dbInstanceClass: db.t4g.medium
  dbInstanceIdentifier: test-ack-aurora-instance-2
  dbClusterIdentifier: "test-ack"
  engine: aurora-postgresql

When these instances are available, one of them will be a Writer Instance and other one will be a Reader Instance.

Check the status of DBInstance :

kubectl describe DBInstance test-ack-aurora-instance-1
kubectl describe DBInstance test-ack-aurora-instance-2

Status available means that DB Instances are ready to use now.

Get the Writer Endpoint of the Aurora RDS cluster:

kubectl get dbcluster test-ack-aurora-cluster -ojson| jq -r '.status.endpoint'

test-ack-aurora-cluster.cluster-cncphmukmdde.eu-west-1.rds.amazonaws.com

Get the Reader Endpoint of the Aurora RDS cluster:

kubectl get dbcluster test-ack-aurora-cluster -ojson| jq -r '.status.readerEndpoint'

test-ack-aurora-cluster.cluster-ro-cncphmukmdde.eu-west-1.rds.amazonaws.com

When you have the Aurora DB Cluster provisioned, you would want to extract values such as Reader Endpoint and Writer Endpoint and configure them in the application to connect to the cluster.

ACK controller for RDS provides a way to import these values directly from the DBCluster or DBInstance resource to a desired Configmap or Secret.

Export the Database Details in a Configmap

FieldExport CRD allows you to export any spec or status field from a DBCluster or DBInstance resource into a ConfigMap.

Create an empty ConfigMap :

kubectl create cm ack-cluster-config
configmap/ack-cluster-config created

Create a FieldExport to import the value of .status.endpoint from test-ack-aurora-cluster DBCluster resource to the ConfigMap ack-cluster-config :

apiVersion: services.k8s.aws/v1alpha1
kind: FieldExport
metadata:
  name: ack-cm-field-export
spec:
  from:
    resource:
      group: rds.services.k8s.aws
      kind: DBCluster
      name: test-ack-aurora-cluster
    path: ".status.endpoint"
  to:
    kind: configmap
    name: ack-cluster-config
    key: DATABASE_HOST

Inspect the values of ConfigMap ack-cluster-config

kubectl get cm ack-cluster-config -ojson|jq -r '.data'

Export the Database Details in a Secret

FieldExport CRD allows you to export any spec or status field from a DBCluster or DBInstance into a Secret.

Create an empty Secret :

kubectl create secret generic ack-cluster-secret
secret/ack-cluster-secret created

Create a FieldExport to import the value of .status.endpoint from test-ack-aurora-cluster DBCluster resource to the Secret ack-cluster-secret :

apiVersion: services.k8s.aws/v1alpha1
kind: FieldExport
metadata:
  name: ack-secret-field-export
spec:
  from:
    resource:
      group: rds.services.k8s.aws
      kind: DBCluster
      name: test-ack-aurora-cluster
    path: ".status.endpoint"
  to:
    kind: secret
    name: ack-cluster-secret
    key: DATABASE_HOST

Inspect the values of Secret ack-cluster-secret
```
kubectl view-secret ack-cluster-secret --all
```
Note: view-secret is a kubectl plugin, check out my blog on how to use this plugin.

📌 FieldExport and ACK resource should be in the same namespace.

Important - Clean up

To clean up the clusters, first disable the delete protection by setting .spec.deletionProtection to false for the test-ack-aurora-cluster DBCluster resource:

kubectl apply -f test-ack-aurora-cluster.yaml

dbcluster.rds.services.k8s.aws/test-ack-aurora-cluster configured

Now, you can proceed to delete the instances first and then cluster:

kubectl delete -f test-ack-aurora-instance.yaml
kubectl delete -f test-ack-aurora-cluster.yaml

Most Useful kubectl Plugins

Shardul Srivastava — Sat, 15 Oct 2022 01:12:22 +0000

Kubernetes provides a convenient utility kubectl to interact with the cluster. kubectl talks to kube-apiserver and allows you to create, update and delete objects/resources in the cluster.

How To Pronounce kubectl

When you start using kubectl, the first thing that comes to mind is how the heck do you pronounce this. There are different pronunciations used by different people, as long as everybody is referring to the same command line tool, it's all good.

Here are the three different pronunciations that I have heard people using:

kube-control
kube-cuttle
kube-cee-tee-ell ←

P.S: I use the last one 😄

What are kubectl Plugins

Kubernetes provides a way to extend the functionality of kubectl using plugins. Plugins allow us to add additional functionality to the kubectl command line tool.

kubectl plugins are executables whose names start with kubectl-. These executables should be part of the PATH so that kubectl can discover them. kubectl automatically detects them and runs them for you.

Eg: If we have a plugin called hello then you can invoke it by using the command:

kubectl hello

here kubectl would look for an executable with the name kubectl-hello in the PATH

Install kubectl Plugins with Krew

kubectl plugins can be installed in numerous ways, the easiest way would be to install the official plugin manager called krew.

Install krew by following the instructions for your operating system here.

For Mac, it can be installed with the brew package manager:

brew install krew

Installing official plugins

krew maintains an index of officially maintained plugins called krew plugin index. There are about 206 plugins maintained in the official krew index by the maintainers.

Let's take a look at some of the most useful plugins

neat Plugin

Install neat plugin with krew :

kubectl krew install neat

neat is my favorite plugin. While working with Kubernetes, you often would want to check the resource spec in the cluster, however, when you run the command, you get more fields than intended as part of the spec.

kubectl get pods nginx-7fd68f74d-ntpdc -oyaml

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2022-10-14T16:18:08Z"
  generateName: nginx-7fd68f74d-
  labels:
    app: nginx
    pod-template-hash: 7fd68f74d
  name: nginx-7fd68f74d-ntpdc
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: nginx-7fd68f74d
    uid: 4ff93f8e-a3c3-4c81-9556-e69eb47e9011
  resourceVersion: "85985"
  uid: 714bf0d2-2456-4efc-a527-71f29943662c
spec:
  containers:
  - image: quay.io/shardul/nginx:v1
    imagePullPolicy: Always
    name: nginx
    resources:
      requests:
        cpu: "1"
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-qskm4
      readOnly: true
  dnsPolicy: ClusterFirst

the output is too verbose for troubleshooting. This is where neat plugin comes to our rescue.

Let's get the pod details, this time add | kubectl neat at the end of the command :

kubectl get pods nginx-7fd68f74d-ntpdc -oyaml | kubectl neat

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: nginx
    pod-template-hash: 7fd68f74d
  name: nginx-7fd68f74d-ntpdc
  namespace: default
spec:
  containers:
  - image: quay.io/shardul/nginx:v1
    imagePullPolicy: Always
    name: nginx
    resources:
      requests:
        cpu: "1"
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-qskm4
      readOnly: true

kubectl-neat omits managedFields, default values and status fields and some metadata fields such as creationTimestamp, resourceVersion,selfLink and uid.

view-secret Plugin

Install view-secret plugin with krew:

kubectl krew install view-secret

view-secret plugin saves a lot of time when you want to view a secret in the cluster, especially if it's a secret with multiple keys and values. Normally to view a secret you would do:

kubectl get secret my-secret -o yaml

apiVersion: v1
data:
  key1: c3VwZXJzZWNyZXQ=
  key2: dG9wc2VjcmV0
kind: Secret
metadata:
  creationTimestamp: "2022-10-14T19:58:25Z"
  name: my-secret
  namespace: default
  resourceVersion: "88915"
  uid: 4b6fbf40-f27f-4744-ab61-2d7457a41ce6
type: Opaque

then copy the values such as c3VwZXJzZWNyZXQ= and decode it with base64:

echo "c3VwZXJzZWNyZXQ=" | base64 -d
supersecret

With the view-secret plugin, you can just do :

kubectl view-secret my-secret --all
key1=supersecret
key2=topsecret

access-matrix Plugin

Install access-matrix plugin with krew :

kubectl krew install access-matrix

The access-matrix plugin is very useful to visualize your access in the cluster or to find out who can access a particular resource in the cluster.

blame Plugin

Install blame plugin with krew :

kubectl krew install blame

The blame plugin helps you to figure out who changed several fields of an object in the cluster - kubectl, kube-controller-manager or helm. It internally uses the .metadata.manageFields field of the object to get this information.

Read more about metadata.managedFields here.

If we edit a deployment nginx manually and update the replias to 2. We can see those details using the blame plugin that changes were done using kubectl edit

kubectl blame deploy nginx

                                                  spec:
kubectl-client-side-apply (Update    8 hours ago)   progressDeadlineSeconds: 600
kubectl-edit              (Update 26 minutes ago)   replicas: 2
kubectl-client-side-apply (Update    8 hours ago)   revisionHistoryLimit: 10

df-pv Plugin

Install df-pv plugin with krew :

kubectl krew install df-pv

If you are familiar with the df command in Linux and Mac, then you would love the df-pv plugin. It provides the same functionality as df provides, except that it provides details for Persistent volumes in a human-readable format.

The df-pv plugin comes in handy if you want to get an overall view of PVs in the cluster. It shows you details like Size, Used, Available, and %Used.

Installing plugins directly from repositories

Apart from the krew index, plugins can be installed from private repositories via manual steps or using a custom plugin index.

In the end plugins are just executables.

clean plugin

Install clean plugin manually :

git clone git@github.com:shardulsrivastava/kubectl-plugin.git
cd kubectl-plugin/plugins/clean
mv kubectl-clean /usr/local/bin
kubectl clean --help

The clean plugin comes handy if you're using EKS or GKE where you have orphaned pods lying around cluttering the cluster. It cleans them all up in one go.

To delete all the orphaned pods in your cluster

kubectl clean all

or you can clean up a particular namespace:

kubectl clean my-namespace

gke-outdated plugin

Install gke-outdated plugin manually :

git clone git@github.com:shardulsrivastava/kubectl-plugin.git
cd kubectl-plugin/plugins/gke-outdated
mv kubectl-gke_outdated print-table /usr/local/bin
kubectl gke-outdated --help

The gke-outdated plugin finds all the outdated GKE clusters in your GCP organization's folder.

To check all the GKE clusters running outdated kubernetes versions inside folder Id 907623304376 :

kubectl gke-outdated 907623304376 1.22

this will list all the GKE clusters inside of folder 907623304376 that are running a version less than 1.22.

Scaling in EKS with Karpenter - Part 1

Shardul Srivastava — Mon, 10 Oct 2022 22:06:12 +0000

Kubernetes has become a de-facto standard because it takes care of a lot of complexities internally. One of those complexities is cluster autoscaling i.e provisioning of nodes based on the increased number of workloads.

Cluster Autoscaler is a project maintained by a community called sig-autoscaling, one of the communities under Kubernetes. Check out more about Kubernetes Communities here.

Cluster autoscaler supports a number of cloud providers including EKS. here is the guide to setup cluster autoscaler on EKS and various configuration examples.

Cluster autoscaler runs in the cluster as an addon and adds or removes the nodes in the cluster to allow the scheduling of workloads. It kicks in when any of the pods are not able to schedule due to insufficient resources. Node groups in EKS are backed by EC2 Auto Scaling Groups and CA updates the number of nodes in the ASG dynamically to ensure pods are scheduled. It also supports Mixed Instance policy for Spot instances that allows users to save costs with Spot instances with the added risk of workload interruption.

While CA takes care of the scaling efficiently, there are still issues that EKS users face such as :

When there are no node in the Node group that matches the requirements of the pod and the pod remains unscheduled, which could cause an outage.

While debugging these types of issues a common error message in autoscaler logs is pod didn't trigger scale-up (it wouldn't fit if a new node is added)
Using too big instances in node groups, which leads to low resource utilization and increased cost.
Using too low instances in Node groups, which leads to node groups maxing out and resulting in unscheduled pods.
No way to identify the optimal choice of instance types based on workloads.

All of these issues and many more can be solved with...

Karpenter

Karpenter is an open-source project by AWS that improves the efficiency and cost of running workloads on Kubernetes clusters (not just EKS, it's designed to have support for multiple cloud providers).

Karpenter is group less i.e it works by provisioning nodes based on the workload and the provisioned nodes are not part of any ASG.

This allows Karpenter to scale nodes more efficiently, retry in a few milliseconds when node capacity is not available instead of waiting for minutes in case of an ASG and gives the flexibility to provision instances from a variety of instance types without creating hundreds of node groups.

Karpenter Installation

Karpenter controller runs on the cluster as a deployment and relies on a CRD called Provisioner to configure it's behaviour to handle workloads such as consolidate them to save costs, move them to another node with a different instance type and scale down nodes when they are empty.

Setup an EKS Cluster with Karpenter enabled

Karpenter can be installed by following the steps in the Karpenter - Getting Started Guide which involves creating several Cloudformation stacks, IAM role cluster bindings. IMO, the simplest way to get started is to use eksctl that has support for karpenter installation out-of-box.

Let's create a cluster with eksctl with a managed node group and karpenter installed:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: karpenter-cluster
  region: eu-west-1
  version: "1.22"
  tags:
    karpenter.sh/discovery: karpenter-cluster # Special tag for Karpenter that it uses to discover subnets and security groups
iam:
  withOIDC: true # required

karpenter:
  version: '0.15.0' # Current version of eksctl only supports 0.15.0 of karpenter.
  createServiceAccount: true 

managedNodeGroups:
  - name: managed-ng-1 # Managed node group for karpenter controller itself, could use fargate too that would be much simpler.
    minSize: 1
    maxSize: 10
    desiredCapacity: 1
    instanceType: t3.large
    amiFamily: AmazonLinux2

this would create an EKS cluster karpenter-cluster in the eu-west-1 region with a managed Node group managed-ng-1, install few of components for karpenter to work :

eksctl-KarpenterControllerPolicy-karpenter-cluster - An IAM policy with permissions to provision nodes and spot instances.
eksctl-karpenter-cluster-iamservice-role - An IAM role with IAM policy eksctl-KarpenterControllerPolicy-karpenter-cluster and attached to the service account used by karpenter to allow it to provision instances.
eksctl-KarpenterNodeRole-karpenter-cluster - An IAM role for provisioned nodes to work and get registered with the cluster.
eksctl-KarpenterNodeInstanceProfile-karpenter-cluster - An Instance profile for instance provisioned by Karpenter.
karpenter Helm Release - Installation of karpenter using Helm chart.

Setup Karpenter Provisioner

To configure Karpenter, you need to create a Provisioner resource that defines how Karpenter provisions nodes and removes them. It has several configuration options but we will start with the basic Provisioner :

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  provider: 
    subnetSelector:
      karpenter.sh/discovery: karpenter-cluster
    securityGroupSelector:
      karpenter.sh/discovery: karpenter-cluster

For the sake of simplicity, we are using provider definiton directly, however recommendaion is to create another CRD of type AWSNodeTemplate and refer to it using providerRef configuration.

Here subnetSelector and securityGroupSelector values allow karpenter to discover subnets and security groups i.e karpenter would discover the subnets and security groups using these tags and create the node in one of these subnets and attach the discovered security groups to the node.

Let's create a sample deployment for Nginx with CPU request as 1

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: quay.io/shardul/nginx:v1
        name: nginx
        imagePullPolicy: Always
        resources:
          requests:
            cpu: 1

Check if pods are running :

kubectl get pods

Initially Nginx pod is pending, karpenter detects a pending pod and kicks in to create a node to accommodate the pending pod, Check the karpenter controller logs to see the details :

kubectl -n karpenter logs -f -l app.kubernetes.io/instance=karpenter -c controller

2022-10-09T15:35:31.066Z  INFO  controller.provisioning Computed 1 new node(s) will fit 1 pod(s)  {"commit": "3d87474"}
2022-10-09T15:35:31.253Z  DEBUG controller.provisioning.cloudprovider Discovered subnets: [subnet-0bb805312e46db35f (eu-west-1b) subnet-05bd621d69320799a (eu-west-1c) subnet-02828f6cf56a0eabc (eu-west-1a) subnet-0d94022cc49b76aeb (eu-west-1a) subnet-09ebfc5011dbf69fb (eu-west-1b) subnet-084a73ba681d60241 (eu-west-1c)] {"commit": "3d87474", "provisioner": "default"}
2022-10-09T15:35:31.387Z  DEBUG controller.provisioning.cloudprovider Discovered security groups: [sg-03a9dd659d9a0b8fd sg-00a2bf520128db45b] {"commit": "3d87474", "provisioner": "default"}
2022-10-09T15:35:31.390Z  DEBUG controller.provisioning.cloudprovider Discovered kubernetes version 1.22  {"commit": "3d87474", "provisioner": "default"}
2022-10-09T15:35:31.425Z  DEBUG controller.provisioning.cloudprovider Discovered ami-04f335c3e4d6dcfad for query "/aws/service/eks/optimized-ami/1.22/amazon-linux-2-arm64/recommended/image_id"  {"commit": "3d87474", "provisioner": "default"}
2022-10-09T15:35:31.446Z  DEBUG controller.provisioning.cloudprovider Discovered ami-0ed22cc46dcbf16ed for query "/aws/service/eks/optimized-ami/1.22/amazon-linux-2/recommended/image_id"  {"commit": "3d87474", "provisioner": "default"}
2022-10-09T15:35:31.483Z  DEBUG controller.provisioning.cloudprovider Discovered launch template Karpenter-karpenter-cluster-5093968976540638239  {"commit": "3d87474", "provisioner": "default"}
2022-10-09T15:35:31.523Z  DEBUG controller.provisioning.cloudprovider Discovered launch template Karpenter-karpenter-cluster-9263159034123731516  {"commit": "3d87474", "provisioner": "default"}
2022-10-09T15:35:33.829Z  INFO  controller.provisioning.cloudprovider Launched instance: i-0aa11b1f2eb52b92d, hostname: ip-192-168-32-230.eu-west-1.compute.internal, type: t3.medium, zone: eu-west-1c, capacityType: spot {"commit": "3d87474", "provisioner": "default"}
2022-10-09T15:35:33.837Z  INFO  controller.provisioning Created node with 1 pods requesting {"cpu":"1125m","pods":"3"} from types t4g.micro, t3a.micro, t3.micro, t4g.small, t3a.small and 477 other(s) {"commit": "3d87474", "provisioner": "default"}

Immediately we can see that there is an additional node ip-192-168-32-230.eu-west-1.compute.internal launched by karpenter

and the pod is running now on this node:

Let's inspect the node created by the karpenter :

kubectl get node ip-192-168-32-230.eu-west-1.compute.internal -ojson | jq -r '.metadata.labels'

We can see that it's a spot instance of type t3.medium.

By default, karpenter provisions spot instance. However, we can change it to on-demand instance too based on our requirements. Let's try to delete the deployment to see if karpenter removes the instances.

kubectl delete deployment nginx

and the pod is gone, but the node is still there :

ttlSecondsAfterEmpty

With ttlSecondsAfterEmpty, Karpenter deletes empty instances after the TTL is elapsed.

Let's update the Provisioner to this value :

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  ttlSecondsAfterEmpty: 60
  provider:
    subnetSelector:
      karpenter.sh/discovery: karpenter-cluster
    securityGroupSelector:
      karpenter.sh/discovery: karpenter-cluster

Now karpenter will wait for 60 seconds before it cordons, drains, and deletes the empty node. We can check the details in the karpenter logs:

kubectl -n karpenter logs -f -l app.kubernetes.io/instance=karpenter -c controller

2022-10-09T16:17:08.000Z  INFO  controller.node Triggering termination after 1m0s for empty node  {"commit": "3d87474", "node": "ip-192-168-32-230.eu-west-1.compute.internal"}
2022-10-09T16:17:08.024Z  INFO  controller.termination  Cordoned node {"commit": "3d87474", "node": "ip-192-168-32-230.eu-west-1.compute.internal"}
2022-10-09T16:17:08.184Z  INFO  controller.termination  Deleted node  {"commit": "3d87474", "node": "ip-192-168-32-230.eu-west-1.compute.internal"}

Note:
For the nodes provisioned by Karpenter, it adds a finalizer for graceful node termination, we can check this by inspecting the node:

kubectl get nodes ip-192-168-32-230.eu-west-1.compute.internal -ojson | jq -r  '.metadata.finalizers'
[
  "karpenter.sh/termination"
]

Understanding Istio Access Logs

Shardul Srivastava — Sun, 06 Mar 2022 10:30:58 +0000

Istio access logs are very helpful to understand the incoming traffic pattern. These logs are produced by the Envoy proxy and can be viewed overall at the Istio Ingress gateway or at the individual pod that is injected with the envoy proxy sidecar.

Enable Istio Access Logs

Istio access logs are not enabled by default, it can be enabled by setting the meshConfig.accessLogFile of the IstioOperator resource.

A basic istio installation with access logs enabled would look like this:



apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    accessLogFile: /dev/stdout

Here the value /dev/stdout outputs the access logs to standard output. By default these access logs are in TEXT format i.e it looks like this :



[2022-10-23T20:38:15.413Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 37 0 0 "-" "curl/7.35.0" "3ce3e159-9dba-4617-9e85-feb1106a682c" "nginx" "192.168.42.152:80" inbound|80|| 127.0.0.6:52375 192.168.42.152:80 192.168.59.90:34854 outbound_.80_._.nginx.default.svc.cluster.local default

it can be changed to json by setting .spec.meshConfig.accessLogEncoding to JSON and logs would change to:



{
    "response_code_details": "via_upstream",
    "downstream_remote_address": "192.168.59.90:58478",
    "downstream_local_address": "192.168.42.152:80",
    "request_id": "e2873f51-0a42-4da4-a8e8-f711b46dea5c",
    "authority": "nginx",
    "requested_server_name": "outbound_.80_._.nginx.default.svc.cluster.local",
    "upstream_cluster": "inbound|80||",
    "path": "/",
    "bytes_sent": 37,
    "user_agent": "curl/7.35.0",
    "method": "GET",
    "upstream_local_address": "127.0.0.6:41549",
    "x_forwarded_for": null,
    "start_time": "2022-10-23T22:15:39.663Z",
    "response_code": 200,
    "upstream_transport_failure_reason": null,
    "duration": 0,
    "upstream_service_time": "0",
    "response_flags": "-",
    "upstream_host": "192.168.42.152:80",
    "route_name": "default",
    "bytes_received": 0,
    "protocol": "HTTP/1.1",
    "connection_termination_details": null
}

Istio Access Logs Format

When Istio access logs are enabled, they are by default in this format



[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS%
\"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\"
\"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n

here % is used to define a field to be printed in the access logs. To print the response code we can use %RESPONSE_CODE%. Let's look at some of the important fields here:

%START_TIME% - Request start time.
%RESPONSE_CODE% - Response code of the request.
%RESPONSE_FLAGS% - Additional details about response. More details here
%RESPONSE_CODE_DETAILS% - Response code details. More details here.
%DURATION% - Total duration of the request.
%UPSTREAM_HOST% - Upstream where the request is routed to such as pod. It would print the pod's IP and port where the request went. Eg - 192.168.42.152:80
%REQ()% - This is used to print request headers. Eg - %REQ(-FORWARDED-FOR)% would print the X-FORWARDED-FOR request header.
%RESP()% - This is used to print response headers. Eg - %RESP(CONTENT-TYPE)% would print the CONTENT-TYPE header in the response.

An exhausting list of available fields can be found here.

Customize Access Logs Format

If you want to parse your logs with a tool like Loki, json format logs would be well suited. Access logs format can be changed by setting the meshConfig.accessLogFormat.

Eg: To print json logs in a custom format



spec:
  meshConfig:
    accessLogFile: /dev/stdout
    accessLogEncoding: JSON
    accessLogFormat: |
      {
        "protocol": "%PROTOCOL%",
        "upstream_service_time": "%REQ(X-ENVOY-UPSTREAM_SERVICE_TIME)%",
        "upstream_local_address": "%UPSTREAM_LOCAL_ADDRESS%",
        "duration": "%DURATION%",
        "upstream_transport_failure_reason": "%UPSTREAM_TRANSPORT_FAILURE_REASON%",
        "route_name": "%ROUTE_NAME%",
        "downstream_local_address": "%DOWNSTREAM_LOCAL_ADDRESS%",
        "user_agent": "%REQ(USER-AGENT)%",
        "response_code": "%RESPONSE_CODE%",
        "response_flags": "%RESPONSE_FLAGS%",
        "start_time": "%START_TIME%",
        "method": "%REQ(:METHOD)%",
        "request_id": "%REQ(X-REQUEST-ID)%",
        "upstream_host": "%UPSTREAM_HOST%",
        "x_forwarded_for": "%REQ(X-FORWARDED-FOR)%",
        "client_ip": "%REQ(TRUE-Client-IP)%",
        "requested_server_name": "%REQUESTED_SERVER_NAME%",
        "bytes_received": "%BYTES_RECEIVED%",
        "bytes_sent": "%BYTES_SENT%",
        "upstream_cluster": "%UPSTREAM_CLUSTER%",
        "downstream_remote_address": "%DOWNSTREAM_REMOTE_ADDRESS%",
        "authority": "%REQ(:AUTHORITY)%",
        "path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%",
        "response_code_details": "%RESPONSE_CODE_DETAILS%"
      }

Printing Known Request and Response Headers

EnvoyProxy allows printing request headers and response headers with %REQ(X?Y):Z% and %RESP(X?Y):Z% respectively. This can be used to print any request header or response header and if the header is not present an alternative header can be given optionally.

%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% would print the X-ENVOY-ORIGINAL-PATH header, if it doesn't exist PATH header would be printed and if even the PATH header is also not present then - would be printed instead.

So we can print all the headers that we know the name of, but how about any dynamic headers whose name changes with every request.

Printing All the Request Headers and Request Body

Istio uses EnvoyProxy which is extensible by design, EnvoyFilter provides a mechanism to customize the Envoy configuration.

The HTTP Lua filter allows Lua scripts to be run during both the request and response flows. This Envoy HTTP Lua filter prints all the request headers :



apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: request-filter
  namespace: istio-system
spec:
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: envoy.filters.network.http_connection_manager
              subFilter:
                name: envoy.filters.http.router
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.lua
          typed_config:
            '@type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
            inlineCode: |
              -- Called on the request path.
              function envoy_on_request(request_handle)
                  request_handle:logCritical("Printing Request Headers => ")
                  for key, value in pairs(request_handle:headers()) do
                    request_handle:logCritical(key .. " => " .. value)
                  end
                  for chunk in request_handle:bodyChunks() do
                    request_handle:logCritical("Printing Request body => " .. chunk:getBytes(0, chunk:length()))
                  end
              end
  workloadSelector:
    labels:
      app: istio-ingressgateway



kubectl apply -f istio-filter.yaml

once applied, all the headers will be printed in the logs:



2022-10-23T23:48:39.750646Z critical  envoy lua script log: Printing Request Headers => 
2022-10-23T23:48:39.750688Z critical  envoy lua script log: :authority => 54.77.207.3
2022-10-23T23:48:39.750691Z critical  envoy lua script log: :path => /
2022-10-23T23:48:39.750693Z critical  envoy lua script log: :method => GET
2022-10-23T23:48:39.750695Z critical  envoy lua script log: :scheme => http
2022-10-23T23:48:39.750697Z critical  envoy lua script log: user-agent => Mozilla/5.0 (Windows NT 6.2;en-US) AppleWebKit/537.32.36 (KHTML, live Gecko) Chrome/56.0.3006.85 Safari/537.32
2022-10-23T23:48:39.750699Z critical  envoy lua script log: accept-encoding => gzip, deflate
2022-10-23T23:48:39.750701Z critical  envoy lua script log: accept => */*
2022-10-23T23:48:39.750714Z critical  envoy lua script log: x-forwarded-for => 192.168.59.163
2022-10-23T23:48:39.750716Z critical  envoy lua script log: x-forwarded-proto => http
2022-10-23T23:48:39.750717Z critical  envoy lua script log: x-envoy-internal => true
2022-10-23T23:48:39.750719Z critical  envoy lua script log: x-request-id => 0d1def76-2d84-4ddb-9aa3-62b2445cc1f8
2022-10-23T23:48:39.750747Z critical  envoy lua script log: x-envoy-peer-metadata-id => router~192.168.45.105~istio-ingressgateway-94b448d6d-pbsqj.istio-system~istio-system.svc.cluster.local

EKS Auth Deep Dive

Shardul Srivastava — Fri, 17 Sep 2021 20:58:54 +0000

If you use EKS then you have found yourself in a situation where a user can't access the cluster despite having all the IAM permissions and gets an Unauthorized message like Eddie here.

AWS EKS uses IAM credentials for authentication and Kubernetes RBAC for authorization. As per EKS docs:

EKS uses IAM permissions for authentication of valid entities such IAM users or roles. All the permissions for interacting with the EKS cluster is managed through Kubernetes RBAC

or simply put, EKS doesn't work the same way as other services such as S3 where if you have AmazonS3FullAccess, you can access any S3 bucket and create or delete files/folders. In EKS, IAM permissions are only used to check if the user has valid IAM credentials and permissions to run any command using kubectl such as kubectl get pods is managed by Kubernetes API that uses RBAC to control the access.

By default, the IAM Role or IAM User that was used to create the cluster, is added to the system:masters group and gets cluster-wide admin permission with cluster-admin ClusterRole.

As per Kubernetes documentation system:masters group is one of the default ClusterRoleBindings available in the Kubernetes cluster, it's attached to the cluster-admin ClusterRole that gives the user admin permissions in the cluster.

Default ClusterRole	Default ClusterRoleBinding	Description
cluster-admin	system:masters group	Allows super-user access to perform any action on any resource.

Note: This mapping of creator IAM User or Role to system:masters group is not visible in any configuration such as aws-auth configmap.

EKS allows giving access to other users by adding them in a configmap aws-auth in kube-system namespace. By default, this configmap is empty. However, If you are using eksctl to create the cluster, this config map will have the role created by eksctl for the node group and this role is attached to the system:bootstrappers and system:nodes groups.

aws-auth configmap is based on aws-iam-authenticator and has several configuration options:

mapRoles
mapUsers
mapAccounts

Using mapRoles to Map an IAM Role to the Cluster

mapRoles allows mapping an IAM role in the cluster to allow any entity or user assuming that role to access the cluster. After mapping an IAM role with mapRoles, any user or entity assuming this role is allowed to access the cluster, However, the level of access is defined by the groups attribute.

mapRoles has three attributes:

rolearn - IAM Role ARN to map to EKS cluster.
username - Username for the IAM Role to map in Kubernetes, this could be a static value like eks-developer or ci-account or a templated variable like {{AccountID}}/{{SessionName}}/{{EC2PrivateDNSName}} or both. This value would be printed in the aws-authenticator Cloudwatch logs if logging is enabled.
groups - List of Kubernetes groups that are defined in ClusterRoleBinding/RoleBinding. Example

  subjects:
  - kind: Group
    name: "my-group"
    apiGroup: ""

Let's create two IAM roles eks-admin and eks-dev and assume the eks-admin role to create a cluster with one node group:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: iam-auth-cluster
  region: us-east-1
  version: "1.21"

availabilityZones: 
  - us-east-1a
  - us-east-1b
  - us-east-1c

cloudWatch:
  clusterLogging:
    enableTypes: ["authenticator"]

managedNodeGroups:
  - name: managed-ng-1
    instanceType: t2.micro
    minSize: 1
    maxSize: 4
    desiredCapacity: 4

Once created, this cluster would have one NodeGroup and the IAM role associated with this node group would be added to the aws-auth configmap.

Check the contents of aws-auth configMap :

kubectl get configmap aws-auth -n kube-system -oyaml

apiVersion: v1
data:
  mapRoles: |
    - groups:
      - system:bootstrappers
      - system:nodes
      rolearn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/eksctl-iam-auth-cluster-nodegroup-NodeInstanceRole-1RNKIEA50ZD0B
      username: system:node:{{EC2PrivateDNSName}}
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system

By default, only IAM Role that created the cluster would have access to the cluster, any other IAM Role has to be added separately added in aws-auth. Let's try to assume the eks-developer IAM role and try to access the cluster with that role.

kubectl get pods

error: You must be logged in to the server (Unauthorized)

As expected, eks-developer IAM role would not be allowed access. To allow eks-developer IAM role access to the cluster, add the mapping in the aws-auth configMap to map this role to eks-developer Kubernetes user. We can either directly edit the configMap or use eksctl to add this mapping:

eksctl create iamidentitymapping \
  --cluster iam-auth-cluster \
  --region us-east-1 \
  --arn "arn:aws:iam::<AWS_ACCOUNT_ID>:role/eks-developer" \
  --username "eks-developer"

this would create an entry under the mapRoles section in aws-auth configMap :

apiVersion: v1
data:
  mapRoles: |
    - groups:
      - system:bootstrappers
      - system:nodes
      rolearn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/eksctl-iam-auth-cluster-nodegroup-NodeInstanceRole-1RNKIEA50ZD0B
      username: system:node:{{EC2PrivateDNSName}}

    - rolearn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/eks-developer
      username: eks-developer
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system

Let's try again accessing cluster by assuming the eks-developer IAM role:

kubectl get pods

Error from server (Forbidden): pods is forbidden: User "eks-developer" cannot list resource "pods" in API group "" at the cluster scope

This time, we can access the cluster, however not allowed to list pods in the cluster due to not having enough RBAC permissions. RBAC permissions can be assigned to this IAM role in two ways :

1. RBAC permissions with Kubernetes User
2. RBAC permissions with Kubernetes Groups

RBAC permissions with Kubernetes User

We can assign RBAC permissions to an IAM role by binding mapped Kubernetes User in aws-auth i.e eks-developer to a ClusterRole/Role.

Create a ClusterRole eks-developer-cluster-role with permissions to get, list or watch the pods resources :

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: eks-developer-cluster-role
rules:
  - apiGroups: [""] # Pod is part of Core API Group and "" indicates the core API group
    resources: ["pods"] # pods resource
    verbs: ["get", "list", "watch"] # Allow user to get, list of watch the pods.

We have mapped IAM role arn:aws:iam::<AWS_ACCOUNT_ID>:role/eks-developer to Kubernetes user eks-developer in aws-auth, now create a ClusterRoleBinding to bind developer-cluster-role to the Kubernetes user eks-developer.

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: eks-developer-user-cluster-role-binding
subjects:
  - kind: User
    name: eks-developer # Kubernetes User mapped to the IAM role in aws-auth configmap.
    apiGroup: ""
roleRef:
  kind: ClusterRole
  name: eks-developer-cluster-role
  apiGroup: ""

Access the cluster again by assuming the IAM role eks-developer

kubectl get pods -A

NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE
kube-system   aws-node-f584p             1/1     Running   0          79m
kube-system   coredns-66cb55d4f4-8hjj2   1/1     Running   0          91m
kube-system   coredns-66cb55d4f4-vtf6j   1/1     Running   0          91m
kube-system   kube-proxy-psjk5           1/1     Running   0          79m

and this time it works!!!

We can also verify the access granted to the users by checking the authenticator logs in Cloudwatch:

time="2021-09-13T16:29:14Z" level=info msg="access granted" arn="arn:aws:iam::01755xxxxx:role/eks-developer" client="127.0.0.1:50410" groups="[]" method=POST path=/authenticate sts=sts.us-east-1.amazonaws.com uid="heptio-authenticator-aws:01755xxxxx:AROAQIFUWO66PDOXKSLMQ" username=eks-developer

RBAC permissions with Kubernetes Groups

While assigning permissions directly to the Kubernetes User works just fine for most of the use-cases however this approach is not so great if you want to audit who is assuming the IAM role and accessing the cluster and would like additional information captured in the audit logs.

One such use-case is AWS SSO, where many users are assigned to a permission set and whenever these users log in using their credentials, they assume the same IAM role.

We can assign IAM permissions to an IAM role by creating a Kubernetes Group and add it to the mapRoles.groups field of IAM Role mapping in aws-auth.

Let's take the earlier example of the eks-developer IAM role and create a ClusterRoleBinding with Kubernetes Group developer bound to eks-developer-cluster-role and add Kubernetes Group developer in the mapping of this IAM role in aws-auth.

First delete the earlier ClusterRoleBinding eks-developer-user-cluster-role-binding :

kubectl delete clusterrolebindings eks-developer-user-cluster-role-binding
clusterrolebinding.rbac.authorization.k8s.io "eks-developer-user-cluster-role-binding" deleted

As soon as we delete the ClusterRolebinding, eks-developer IAM role won't be able to list the pods, let's check the access by assuming the eks-developer IAM role :

kubectl get pods -A
Error from server (Forbidden): pods is forbidden: User "eks-developer" cannot list resource "pods" in API group "" in the namespace "default"

Delete IAM Mapping from aws-auth :

eksctl delete iamidentitymapping \
      --cluster iam-auth-cluster \
      --region us-east-1 \
      --arn "arn:aws:iam::<AWS_ACCOUNT_ID>:role/eks-developer"

Create a ClusterRoleBinding to bind Kubernetes Group developer to cluster role eks-developer-cluster-role:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: eks-developer-group-cluster-role-binding
subjects:
  - kind: Group
    name: developer
    apiGroup: ""
roleRef:
  kind: ClusterRole
  name: eks-developer-cluster-role
  apiGroup: ""

Add Kubernetes Group developer to IAM role mapping of eks-developer in aws-auth and include the session name in username using templated variable {{SessionName}}:

eksctl create iamidentitymapping \
  --cluster iam-auth-cluster \
  --region us-east-1 \
  --arn "arn:aws:iam::<AWS_ACCOUNT_ID>:role/eks-developer" \
  --username "eks-developer:{{SessionName}}" \
  --group "developer"

this would create an entry under mapRoles section in aws-auth configmap as:

mapRoles: |
  - groups:
    - developer
    rolearn: arn:aws:iam::017558828988:role/eks-developer
    username: eks-developer:{{SessionName}}

Check the Cloudwatch authenticator logs for the authenticated user assuming eks-developer IAM role and we can see that this time session name is appended to the username in logs:

time="2021-09-13T17:57:46Z" level=info msg="access granted" arn="arn:aws:iam::0175XXXXXXXX:role/eks-developer" client="127.0.0.1:48520" groups="[developer]" method=POST path=/authenticate sts=sts.us-east-1.amazonaws.com uid="heptio-authenticator-aws:0175XXXXXXXX:AROAQIFUWO66PDOXKSLMQ" username="eks-developer:eks-developer-session"

If the session name consists of @, it would be replaced with -. Let's assume the IAM role eks-developer with session name containing @ :

aws sts assume-role \
    --role-arn "<IAM_ROLE_ARN>" \
    --role-session-name "my-develper-session@123456789" \
    --duration-seconds 3600

Now Cloudwatch logs would have session name printed as eks-developer:my-developer-session-123456789.

time="2021-09-14T17:50:25Z" level=info msg="access granted" arn="arn:aws:iam::017558828988:role/eks-developer" client="127.0.0.1:57794" groups="[developer]" method=POST path=/authenticate sts=sts.us-east-1.amazonaws.com uid="heptio-authenticator-aws:017558828988:AROAQIFUWO66PDOXKSLMQ" username="eks-developer:my-develper-session-123456789"

There is one problem here, if your EKS cluster is being accessed from multiple AWS accounts, it would not be possible to track the AWS account of the user who accessed the EKS cluster just by session name.

{{AccountID}} comes to the rescue, we can use this templated variable to get the AWS account ID of the user who is assuming the role, so we can set the username to :

aws:{{AccountID}}:eks-developer:{{SessionName}}

Please note that iamidentitymapping can't be overridden with eksctl, so you have to delete it and create it again.


eksctl delete iamidentitymapping \
      --cluster "iam-auth-cluster" \
      --region "us-east-1" \
      --arn "arn:aws:iam::<AWS_ACCOUNT_ID>:role/eks-developer"

eksctl create iamidentitymapping \
      --cluster "iam-auth-cluster" \
      --region "us-east-1" \
      --arn "arn:aws:iam::<AWS_ACCOUNT_ID>:role/eks-developer" \
      --username "aws:{{AccountID}}:eks-developer:{{SessionName}}" \
      --group "developer"

Now we would get the AWS Account ID along with Session Name in cloudwatch logs :

time="2021-09-14T18:26:33Z" level=info msg="access granted" arn="arn:aws:iam::017558828988:role/eks-developer" client="127.0.0.1:39752" groups="[developer]" method=POST path=/authenticate sts=sts.us-east-1.amazonaws.com uid="heptio-authenticator-aws:0175XXXXXXXX:AROAQIFUWO66PDOXKSLMQ" username="aws:0175XXXXXXXX:eks-developer:my-develper-session-123456789"

Note: If you want session name in raw format, you can use templated variable {{SessionNameRaw}} instead. However as of EKS 1.21, these two variables {{AccessKeyID}} and {{SessionNameRaw}} don't work.

Using mapUser to Map an IAM User to the Cluster

mapUsers allows mapping an IAM User to the cluster and add the user to one or more Kubernetes Groups. It has 3 attributes :

userarn - ARN of IAM User to map to EKS cluster. This could be an IAM user from the same AWS account or another account.
username - Static username to map this IAM User to, in Kubernetes.
groups - List of Kubernetes groups that are defined in ClusterRoleBinding/RoleBinding.

Note: Templated variables are not supported in username field with mapUser.

To add an IAM user with ARN arn:aws:iam::<AWS_ACCOUNT_ID>:user/dev-user in aws-auth configmap, we can run the below command:

eksctl create iamidentitymapping \
  --cluster iam-auth-cluster \
  --region us-east-1 \
  --arn "arn:aws:iam::<AWS_ACCOUNT_ID>:user/dev-user" \
  --username "dev-user"

this command would add these lines in aws-auth configMap:

  mapUsers: |
    - userarn: arn:aws:iam::<AWS_ACCOUNT_ID>:user/dev-user
      username: dev-user

Since we didn't specify any group, dev-user would be able to authenticate to the cluster, however wouldn't be able to list or get any resources.

For users mapped using mapUsers, RBAC permission can be given in two ways :

1. RBAC permissions with Kubernetes User
2. RBAC permissions with Kubernetes Groups

RBAC permissions with Kubernetes User

We can assign RBAC permissions to an IAM user by binding mapped Kubernetes User in aws-auth i.e dev-user to a ClusterRole/Role.

Create a ClusterRoleBinding to bind Kubernetes user dev-user to the developer-cluster-role:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: dev-user-cluster-role-binding
subjects:
  - kind: User
    name: dev-user # Kubernetes User mapped to the IAM user in aws-auth configmap.
    apiGroup: ""
roleRef:
  kind: ClusterRole
  name: eks-developer-cluster-role
  apiGroup: ""

Map IAM user arn:aws:iam::<AWS_ACCOUNT_ID>:user/dev-user to Kubernetes user dev-user in aws-auth configMap:

  eksctl create iamidentitymapping \
    --cluster iam-auth-cluster \
    --region us-east-1 \
    --arn "arn:aws:iam::<AWS_ACCOUNT_ID>:user/dev-user" \
    --username "dev-user"

Once this ClusterRoleBinding is created and the IAM user is mapped in aws-auth, IAM user dev-user would be able to get, list, or watch pods in any namespace.

RBAC permissions with Kubernetes Groups

If we need to give the same set of permissions to multiple users, then instead of creating multiple ClusterRoleBindings, we can use Kubernetes Groups and attach that group to the users for whom those permissions are required.

Create a ClusterRoleBinding to bind Kubernetes Group developer to cluster role eks-developer-cluster-role:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: dev-user-group-cluster-role-binding
subjects:
  - kind: Group
    name: dev
    apiGroup: ""
roleRef:
  kind: ClusterRole
  name: eks-developer-cluster-role
  apiGroup: ""

Map IAM user arn:aws:iam::<AWS_ACCOUNT_ID>:user/dev-user to Kubernetes user dev-user with dev group in aws-auth configMap:

eksctl create iamidentitymapping \
  --cluster iam-auth-cluster \
  --region us-east-1 \
  --arn "arn:aws:iam::<AWS_ACCOUNT_ID>:role/dev-user" \
  --username "dev-user" \
  --group "dev"

this would create an entry under mapUsers section in aws-auth configmap as:

mapUsers: |
  - groups:
    - dev
    userarn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/dev-user
    username: dev-user

Using mapAccounts to Map IAM ARN in an AWS Account to the Cluster

mapAccounts allows mapping all the IAM Users or IAM Roles of an AWS account to the cluster. It accepts the list of AWS Account IDs:

mapAccounts: |
  - "<AWS_ACCOUNT_ID_1>"
  - "<AWS_ACCOUNT_ID_2>"

After mapping the AWS accounts to the cluster, we can use Kubernetes User and Kubernetes Group to assign permissions to those IAM entities.

EKS IAM Deep Dive

Shardul Srivastava — Thu, 19 Aug 2021 18:56:24 +0000

Amazon EKS cluster consists of Managed NodeGroups, Self Managed NodeGroups and Fargate profiles. NodeGroups are autoscaling groups behind the scene, while Fargate is serverless and creates one fargate node per pod.

IAM permissions in EKS can be defined in two ways:

IAM Role for NodeGroups
IAM role for Service Accounts(IRSA)

IAM Role for NodeGroups

Whenever we create an EKS cluster using eksctl with node groups like below :



apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: monitoring-cluster
  region: us-east-1
  version: "1.21"

availabilityZones: 
  - us-east-1a
  - us-east-1b
  - us-east-1c

managedNodeGroups:
  - name: managed-ng-1
    instanceType: t3a.medium

eksctl automatically creates an IAM role with minimum IAM permissions required for the cluster to work and attaches it to the nodes part of the node group. All the pods running on these nodes inherit these permissions.

This role has 3 IAM policies attached that give basic access to the node :

AmazonEKSWorkerNodePolicy - This policy allows EKS worker nodes to connect to EKS clusters.
AmazonEC2ContainerRegistryReadOnly - This policy gives read-only access to ECR.
AmazonEKS_CNI_Policy - This policy is required for amazon-vpc-cni plugin to function properly on nodes which is deployed as part of aws-node daemonset in kube-system namespace.

These permissions may not be enough if you are running workloads that require various other IAM permissions. eksctl provides several ways to define additional permissions for the Node.

Attach IAM Policies using ARN to the Node Group.

eksctl allows you to attach IAM policies to a node group using attachPolicyARNs.



managedNodeGroups:
  - name: managed-ng-1
    iam:
      attachPolicyARNs:
      # Mandatory IAM Policies for NodeGroup.
      - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
      - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
      - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly

      # AWS Managed or Customer Managed IAM Policy ARN.
      - arn:aws:iam::aws:policy/AmazonS3FullAccess

Please note that while specifying IAM policy ARNs using attachPolicyARNs, it's mandatory to include the above 3 IAM policies as they are required for the node to function properly.

If you missed specifying these 3 IAM policies, you will get an error like this :



AWS::EKS::Nodegroup/ManagedNodeGroup: CREATE_FAILED – "The provided role doesn't have the Amazon EKS Managed Policies associated with it. Please ensure the following policies [arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy, arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly] are attached "

Attach IAM Role and Instance Profile to the Node Group

If you have an existing IAM Role and Instance Profile, then you can define that for a node group :



managedNodeGroups:
  - name: managed-ng-1
    iam:
      instanceProfileARN: "<Instance Profile ARN>"
      instanceRoleARN: "<IAM Role ARN>"

Attach Addons IAM Policy

eksctl provides out-of-box IAM policies for the cluster add-ons such as cluster autoscaler, external DNS, cert-manager.



managedNodeGroups:
  - name: managed-ng-1
    iam:
      withAddonPolicies:
        imageBuilder: true
        autoScaler: true
        externalDNS: true
        certManager: true
        appMesh: true
        ebs: true
        fsx: true
        efs: true
        albIngress: true
        xRay: true
        cloudWatch: true

imageBuilder - Full ECR access with AmazonEC2ContainerRegistryPowerUser.
autoScaler - IAM policy for Cluster Autoscaler.
externalDNS - IAM policy for External DNS.
certManager - IAM Policy for Cert Manager.
appMesh - IAM policy for AWS App Mesh.
ebs - IAM Policy for AWS EBS CSI Driver.
fsx - IAM Policy for AWS FSX Lustre.
efs - IAM Policy for AWS EFS CSI Driver.
albIngress - IAM Policy for ALB Ingress Controller.
xRay - IAM Policy AWSXRayDaemonWriteAccess for AWS X-ray Daemon
cloudWatch - IAM Policy CloudWatchAgentServerPolicy for AWS Cloudwatch Agent.

While all the above options allow you to define IAM permissions for the node group, there is a problem with defining IAM permissions at the node level.

All the pods running on nodes part of the node group will have these permissions thus not adhering to the principle of least privilege. For example, if we attach EC2 Admin and Cloudformation permissions to the node group to run CI server, any other pods running on this node group will effectively inherit these permissions.

One way to overcome this problem is to create a separate node group for the CI server. ( Use taints and affinity to ensure only CI pods are deployed on this node group)

However AWS access is not just required for CI servers, many applications use AWS services such as S3, SQS, and KMS and would require fine-grained IAM permissions. Creating one node group for every such application would not be an ideal solution and can lead to maintenance issues, higher cost, and low resource consumption.

IAM Roles for Service Accounts

Amazon EKS supports IAM Roles for Service Accounts (IRSA) that allows us to map AWS IAM Roles to Kubernetes Service Accounts. With IRSA, instead of defining IAM permissions on the node, we can attach an IAM role to a Kubernetes Service Account and attach the service account to the pod/deployment.

IAM role is added to the Kubernetes service account by adding annotation eks.amazonaws.com/role-arn with value as the IAM Role ARN.



apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-serviceaccount
  annotations:
    eks.amazonaws.com/role-arn: "<IAM Role ARN>"

EKS create a Mutating Admission Webhook pod-identity-webhook and uses it to intercept the pod creation request and update the pod spec to include IAM credentials.

Check the Mutating Admission Webhooks in the cluster:



kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io

NAME                                             WEBHOOKS   AGE
0500-amazon-eks-fargate-mutation.amazonaws.com   2          21m
pod-identity-webhook                             1          28m
vpc-resource-mutating-webhook                    1          28m

pod-identity-webhook supports several configuration options:

eks.amazonaws.com/role-arn - IAM Role ARN to attach to the service account.
eks.amazonaws.com/audience - Intended autience of the token, defaults to "sts.amazonaws.com" if not set.
eks.amazonaws.com/sts-regional-endpoints - AWS STS is a global service, however, we can use regional STS endpoints to reduce latency. Set true to use regional STS endpoints.
eks.amazonaws.com/token-expiration - AWS STS Token expiration duration, default is 86400 seconds or 24 hours. We can override this value by defining it at the pod level.
eks.amazonaws.com/skip-containers - A comma-separated list of containers to skip adding volume and environment variables. This annotation is only supported at the pod level.



apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-serviceaccount
  annotations:
    eks.amazonaws.com/role-arn: "<IAM Role ARN>"
    eks.amazonaws.com/audience: "sts.amazonaws.com"
    eks.amazonaws.com/sts-regional-endpoints: "true"
    eks.amazonaws.com/token-expiration: "86400"

How IRSA Works

When you define the IAM role for a service account using eks.amazonaws.com/role-arn annotation and add this service account to the pod, pod-identity-webhook mutates the pod spec to add environment variables and projected mount volumes.

These are the environment variables added by pod-identity-webhook in the pod spec:



  env:
    - name: AWS_DEFAULT_REGION
      value: us-east-1
    - name: AWS_REGION
      value: us-east-1
    - name: AWS_ROLE_ARN
      value: "<IAM ROLE ARN>"
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"

AWS_DEFAULT_REGION and AWS_REGION - Cluster Region
AWS_ROLE_ARN - IAM Role ARN.
AWS_WEB_IDENTITY_TOKEN_FILE - Path to the tokens file

pod-identity-webhook also adds a projected volume for service account token:



    volumeMounts:
    - mountPath: "/var/run/secrets/eks.amazonaws.com/serviceaccount"
      name: aws-iam-token
  volumes:
  - name: aws-iam-token
    projected:
      sources:
      - serviceAccountToken:
          audience: "sts.amazonaws.com"
          expirationSeconds: 86400
          path: token

a projected volume is created with the name aws-iam-token and mounted to the container at /var/run/secrets/eks.amazonaws.com/serviceaccount.

Let's test it out.

Create an EKS cluster with a Managed NodeGroup and an IAM service account s3-reader in the default namespace with AmazonS3ReadOnlyAccess IAM permissions :



apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: iam-cluster
  region: us-east-1
  version: "1.21"
availabilityZones: 
  - us-east-1a
  - us-east-1b
  - us-east-1c
iam:
  withOIDC: true
  serviceAccounts:
  - metadata:
      name: s3-reader
    attachPolicyARNs:
    - "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"
managedNodeGroups:
  - name: managed-ng-1
    instanceType: t3a.medium
    minSize: 1
    maxSize: 4
    desiredCapacity: 4

Note: We can also attach IAM Role directly using attachRoleARN. However, the role should have below trust policy to allow the pods to use this role :



{
 "Version": "2012-10-17",
 "Statement": [
  {
   "Effect": "Allow",
   "Principal": {
    "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/oidc.<REGION>.eks.amazonaws.com/<CLUSTER_ID>"
   },
   "Action": "sts:AssumeRoleWithWebIdentity",
   "Condition": {
    "StringEquals": {
     "oidc.<REGION>.eks.amazonaws.com/<CLUSTER_ID>:sub": "system:serviceaccount:default:my-serviceaccount"
    },
    "StringLike": {
     "oidc.<REGION>.eks.amazonaws.com/CLUSTER_ID:sub": "system:serviceaccount:default:*"
    }
   }
  }
 ]
}

Check the role created by eksctl for service account s3-reader



eksctl get iamserviceaccount s3-reader --cluster=iam-cluster --region=us-east-1
2021-08-25 02:51:14 [ℹ]  eksctl version 0.61.0
2021-08-25 02:51:14 [ℹ]  using region us-east-1
NAMESPACE NAME    ROLE ARN
default   s3-reader <IAM ROLE ARN>

Check the service account to verify eks.amazonaws.com/role-arn annotation:



kubectl get sa s3-reader -oyaml



apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: <IAM ROLE ARN>
  labels:
    app.kubernetes.io/managed-by: eksctl
  name: s3-reader
  namespace: default
secrets:
- name: s3-reader-token-5hnn7

we can see that eksctl has created the IAM role, service account and automatically added the annotation to the service account.

Since eksctl doesn't support all the annotations, let's add them manually:



kubectl annotate \
  sa s3-reader \
    "eks.amazonaws.com/audience=test" \
    "eks.amazonaws.com/token-expiration=43200"

Note: As of EKS v1.21, eks.amazonaws.com/sts-regional-endpoints annotation doesn't work due to this issue.

eks.amazonaws.com/sts-regional-endpoints has no effect #110

arunvelsriram posted on Mar 19, 2021

What happened:

eks.amazonaws.com/sts-regional-endpoints: "true" annotation is not injecting AWS_STS_REGIONAL_ENDPOINTS=regional to the Pods.

What you expected to happen:

AWS_STS_REGIONAL_ENDPOINTS=regional is injected in to the Pods.

How to reproduce it (as minimally and precisely as possible):

Create a serviceaccount as mentioned in the README with annotation eks.amazonaws.com/sts-regional-endpoints: "true"
Create a Pod, using that serviceaccount

Anything else we need to know?:

I noticed that in the README there are two versions of the annotation:

eks.amazonaws.com/sts-regional-endpoints: "true" - https://github.com/aws/amazon-eks-pod-identity-webhook/blob/master/README.md#eks-walkthrough
eks.amazonaws.com/sts-regional-endpoint: "true" - https://github.com/aws/amazon-eks-pod-identity-webhook/blob/master/README.md#aws_sts_regional_endpoints-injection

Which one is correct?

Environment:

AWS Region: ap-south-1
EKS Platform version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.platformVersion): eks.1
Kubernetes version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.version): 1.19
Webhook Version: Am using AWS EKS, so not sure how to find the version

View on GitHub

By default eks.amazonaws.com/audience is set to sts.amazonaws.com but we can set it to any value. To allow a different value for audience, we have to add this value in the Audiences section of the OIDC Provider and update the trust policy of the IAM Role to allow this audience.

Go to IAM Console, click on Identity Providers and add the audience to the OIDC Identity provider:

Modify the trust policy of the IAM Role used with the service account to add this audience :



{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/oidc.eks.<REGION>.amazonaws.com/id/<CLUSTER_ID>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.<REGION>.amazonaws.com/id/<CLUSTER_ID>:sub": "system:serviceaccount:default:s3-reader"
        },
        "oidc.eks.us-east-1.amazonaws.com/id/<CLUSTER_ID>:aud": "test"
      }
    }
  ]
}

Create a pod to test the IAM permissions:



apiVersion: v1
kind: Pod
metadata:
  name: iam-test
  annotations:
    # Skip injecting credentials to the sidecar container.
    eks.amazonaws.com/skip-containers: "sidecar-busybox-container"
spec:
  serviceAccountName: s3-reader
  containers:
  - name: iam-test
    image: amazon/aws-cli
    args: [ "sts", "get-caller-identity" ]

  - name: sidecar-busybox-container
    image: radial/busyboxplus:curl



kubectl apply -f iam-test.yaml

Once the pod is ready, check the environment variables in the pod:



kubectl get pods iam-test -ojson|jq -r '.spec.containers[0].env'



[
  {
    "name": "AWS_DEFAULT_REGION",
    "value": "us-east-1"
  },
  {
    "name": "AWS_REGION",
    "value": "us-east-1"
  },
  {
    "name": "AWS_ROLE_ARN",
    "value": "<IAM ROLE ARN>"
  },
  {
    "name": "AWS_WEB_IDENTITY_TOKEN_FILE",
    "value": "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
  }
]

Check the aws-iam-token volume in the pod:



kubectl get pods iam-test -ojson|jq -r '.spec.volumes[]| select(.name=="aws-iam-token")'



{
  "name": "aws-iam-token",
  "projected": {
    "defaultMode": 420,
    "sources": [
      {
        "serviceAccountToken": {
          "audience": "test",
          "expirationSeconds": 43200,
          "path": "token"
        }
      }
    ]
  }
}

we can see that expirationSeconds is 43200 as specified in the annotation eks.amazonaws.com/token-expiration.

Check the volumeMounts in the pod:



kubectl get pods iam-test -ojson|jq -r '.spec.containers[0].volumeMounts'



[
  {
    "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
    "name": "kube-api-access-xjjqv",
    "readOnly": true
  },
  {
    "mountPath": "/var/run/secrets/eks.amazonaws.com/serviceaccount",
    "name": "aws-iam-token",
    "readOnly": true
  }
]

Check the logs of the pod iam-test to see the role assumed by the pod:



kubectl logs iam-test -c iam-test

Running Apache Spark on EKS Fargate

Shardul Srivastava — Sat, 14 Aug 2021 20:18:18 +0000

Apache Spark is one of the most famous Big Data frameworks that allows you to process data at any scale.

Spark jobs can run on the Kubernetes cluster and have native support for the Kubernetes scheduler in GA from release 3.1.1 onwards.

Spark comes with a spark-submit script that allows submitting spark applications on a cluster using a single interface without the need to customize the script for different cluster managers.

spark-submit on Kubernetes cluster works as follows:

Spark creates a Spark driver running within a Kubernetes pod.
The driver creates executors which are also running within Kubernetes pods and connects to them and executes application code.
When the application completes, the executor pods terminate and are cleaned up, but the driver pod persists logs and remains in “completed” state in the Kubernetes API until it’s eventually garbage collected or manually cleaned up.

To submit a spark job on a kubernetes cluster using spark-submit :

./bin/spark-submit \
    --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \
    --deploy-mode cluster \
    --name spark-pi \
    --class org.apache.spark.examples.SparkPi \
    --conf spark.executor.instances=5 \
    --conf spark.kubernetes.container.image=<spark-image> \
    local:///path/to/examples.jar

While spark-submit provides support for several Kubernetes features such as secrets, persistentVolumes, rbac via configuration parameters, it still lacks a lot of features thus it's not suitable to use in production effectively.

Spark on K8s Operator

Spark on K8s Operator is a project from Google that allows submitting spark applications on Kubernetes cluster using CustomResource Definition SparkApplication.
It uses mutating admission webhook to modify the pod spec and add the features not officially supported by spark-submit.

The Kubernetes Operator for Apache Spark consists of:

A SparkApplication controller that watches events of creation, updates, and deletion of SparkApplication objects and acts on the watch events, a submission runner that runs spark-submit for submissions received from the controller,
A Spark pod monitor that watches for Spark pods and sends pod status updates to the controller,
A Mutating Admission Webhook that handles customizations for Spark driver and executor pods based on the annotations on the pods added by the controller,
A command-line tool named sparkctl for working with the operator.

The following diagram shows how different components interact and work together.

Setup Spark on K8s Operator on EKS Fargate

Setup EKS cluster using eksctl with fargate profile for default, kube-system, and spark namespaces.

    eksctl apply -f - <<EOF
    apiVersion: eksctl.io/v1alpha5
    kind: ClusterConfig
    metadata:
      name: spark-cluster
      region: us-east-1
      version: "1.21"
    availabilityZones: 
      - us-east-1a
      - us-east-1b
      - us-east-1c
    fargateProfiles:
      - name: fp-all
        selectors:
          - namespace: kube-system
          - namespace: default
          - namespace: spark
    EOF

Install Spark on K8s Operator using helm3 in the spark namespace.

    helm repo add spark-operator https://googlecloudplatform.github.io/spark-on-k8s-operator
    helm upgrade \
        --install \
        spark-operator \
        spark-operator/spark-operator \
        --namespace spark \
        --create-namespace \
        --set webhook.enable=true \
        --set sparkJobNamespace=spark \
        --set serviceAccounts.spark.name=spark \
        --set logLevel=10 \
        --version 1.1.6 \
        --wait

Verify Operator installation

    kubectl get pods -n spark

Submit SparkPi on EKS Cluster

Submit the SparkPi application to the EKS cluster

    kubectl apply -f - <<EOF
    apiVersion: "sparkoperator.k8s.io/v1beta2"
    kind: SparkApplication
    metadata:
      name: spark-pi
      namespace: spark
    spec:
      type: Scala
      mode: cluster
      image: "gcr.io/spark-operator/spark:v3.1.1"
      imagePullPolicy: Always
      mainClass: org.apache.spark.examples.SparkPi
      mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.1.1.jar"
      sparkVersion: "3.1.1"
      restartPolicy:
        type: Never
      driver:
        cores: 1
        coreLimit: "1200m"
        memory: "512m"
        labels:
          version: 3.1.1
        serviceAccount: spark
      executor:
        cores: 1
        instances: 1
        memory: "512m"
        labels:
          version: 3.1.1
    EOF

Note: hostPath volume mounts are not supported in Fargate.

Check the status of SparkApplication

    kubectl -n spark describe sparkapplications.sparkoperator.k8s.io spark-pi

Access Spark UI by port-forwarding to the spark-pi-ui-svc

   kubectl -n spark port-forward svc/spark-pi-ui-svc 4040:4040

Managing SparkApplication with sparkctl

sparkctl is CLI tool for creating, listing, checking status of, getting logs of, and deleting SparkApplications running on the Kubernetes cluster.

Build sparkctl from source:

   git clone git@github.com:GoogleCloudPlatform/spark-on-k8s-operator.git
   cd spark-on-k8s-operator/sparkctl
   go build -o /usr/local/bin/sparkctl

List SparkApplication objects in spark namespace:

   sparkctl list -n spark
   +----------+-----------+----------------+-----------------+
   |   NAME   |   STATE   | SUBMISSION AGE | TERMINATION AGE |
   +----------+-----------+----------------+-----------------+
   | spark-pi | COMPLETED | 1h             | 1h              |
   +----------+-----------+----------------+-----------------+

Check the status of SparkApplication spark-pi :

   sparkctl status spark-pi -n spark

    application state:
    +-----------+----------------+----------------+-----------------+--------------------+--------------------+-------------------+
    |   STATE   | SUBMISSION AGE | COMPLETION AGE |   DRIVER POD    |     DRIVER UI      | SUBMISSIONATTEMPTS | EXECUTIONATTEMPTS |
    +-----------+----------------+----------------+-----------------+--------------------+--------------------+-------------------+
    | COMPLETED | 1h             | 1h             | spark-pi-driver | 10.100.97.206:4040 |                  1 |                 1 |
    +-----------+----------------+----------------+-----------------+--------------------+--------------------+-------------------+
    executor state:
    +----------------------------------+-----------+
    |           EXECUTOR POD           |   STATE   |
    +----------------------------------+-----------+
    | spark-pi-418ac87b48d177c9-exec-1 | COMPLETED |
    +----------------------------------+-----------+

Check SparkApplication spark-pi logs:

   sparkctl log spark-pi -n spark

Port-forward to Spark UI:

   sparkctl forward spark-pi -n spark

you can access the Spark UI at http://localhost:4040

Canary Deployment with Istio on EKS

Shardul Srivastava — Sat, 14 Aug 2021 12:27:03 +0000

Canary deployment is a way of deploying the application in a phased manner. In this pattern, we deploy a new version of the application alongside the production version, then rollout the change to a small subset of servers.
Once new version of the application is tested by the real users, then rollout the change out to the rest of the servers.

Canary deployments can be complex and involve testing in production and manual verification.

To demonstrate Canary deployments, we will setup an EKS cluster, install Istio, deploy sample application and setup canary release of new version of application.

you can follow the steps below or use the script here.

Setup EKS Cluster

Setup an EKS cluster of version 1.21 in us-east-1 region with a managed node group default-pool of machine type t3a-medium.

Download and install the latest version of eksctl

   curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
   sudo mv /tmp/eksctl /usr/local/bin

for Mac and other operating systems, follow the steps here.

Create EKS cluster

   eksctl create cluster --name canary-cluster \
          --version 1.21 \
          --region us-east-1 \
          --nodegroup-name default-pool \
          --node-type t3a.medium \
          --nodes 3 \
          --nodes-min 0 \
          --nodes-max 4 \
          --managed

Once the Control plane is ready, you should see the output like this

   2021-08-13 23:59:52 [ℹ]  waiting for the control plane availability...
   2021-08-13 23:59:52 [✔]  saved kubeconfig as "/Users/shardulsrivastava/.kube/config"
   2021-08-13 23:59:52 [ℹ]  no tasks
   2021-08-13 23:59:52 [✔]  all EKS cluster resources for "canary-cluster" have been created
   2021-08-14 00:01:57 [ℹ]  kubectl command should work with "/Users/shardulsrivastava/.kube/config", try 'kubectl get nodes'
   2021-08-14 00:01:57 [✔]  EKS cluster "canary-cluster" in "us-east-1" region is ready

Note: You would need a minimum of these permissions to run the eksctl commands above.

Download and install kubectl

   curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
   sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

Check the pods running in cluster:

   kubectl get pods -A

At this point, your cluster would have only the core-dns, kube-proxy, and amazon-vpc-cni plugins.

Setup Istio

Istio provides a convenient binary istioctl to set up and interact with Istio components.

Install istioctl and if you're running these commands from an ec2 instance, add the installation directory to system path:

   curl -L https://istio.io/downloadIstio | sh -
   export PATH="$PATH:/home/ec2-user/istio-1.11.0/bin"

Istio has multiple configuration profiles, these profiles provide customization of the Istio control plane and of the sidecars for the Istio data plane. default profile is recommended for production deployments.

   istioctl install --set profile=default -y

alternatively, you can also take the dump of the manifests and apply using kubectl:

    istioctl manifest generate --set profile=default > generated-manifest.yaml
    kubectl apply -f generated-manifest.yaml

Once installed, you should see the output like this for a successful installation.

Note: If you get the below error, you should check the instance type you are using as there is a limit to the number of pods that can be scheduled on the node based on the node's instance type. See here for the list of instance types and the max pods.

Istio injects a envoy proxy side-car container to the pods to intercept traffic from pods, this behavior is enabled using label istio-injection=enabled on the namespace level and sidecar.istio.io/inject=true on the pod level.

Add this label to the default namespace to instruct Istio to automatically inject Envoy sidecar proxies.

   kubectl label namespace default istio-injection=enabled

Read more about Istio side-car injection here.

Install Kiali, Prometheus and Grafana to monitor and visualize mesh

   kubectl apply -f https://raw.githubusercontent.com/istio/istio/1.10.3/samples/addons/prometheus.yaml
   kubectl apply -f https://raw.githubusercontent.com/istio/istio/1.10.3/samples/addons/grafana.yaml
   kubectl apply -f https://raw.githubusercontent.com/istio/istio/1.10.3/samples/addons/kiali.yaml

Note: If you get an error like this unable to recognize "https://raw.githubusercontent.com/istio/istio/1.10.3/samples/addons/kiali.yaml": no matches for kind "MonitoringDashboard" in version "monitoring.kiali.io/v1alpha1", then re-run the command:

   kubectl apply -f https://raw.githubusercontent.com/istio/istio/1.10.3/samples/addons/kiali.yaml

Setup Sample Application

To demonstrate how canary deployment works, let's set up version v1 of an Nginx-based sample application and set up a service to expose it at port 80.

Deploy v1 vesion of sample application :

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: nginx
     name: nginx-v1
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: nginx
         version: v1
     template:
       metadata:
         labels:
           app: nginx
           version: v1
       spec:
         containers:
         - image: quay.io/shardul/nginx:v1
           name: nginx-v1
           imagePullPolicy: Always
           ports:
           - name: http
             containerPort: 80
           livenessProbe:
             httpGet:
               path: /health
               port: 80
           readinessProbe:
             httpGet:
               path: /health
               port: 80

Expose the application as a service:

   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: nginx
     name: nginx
   spec:
     ports:
     - name: http
       port: 80
     selector:
       app: nginx

Setup a test pod to test the connectivity to the nginx service :

   kubectl run -it test-connection --image=radial/busyboxplus:curl -- sh
   [ root@test-connection:/ ]$ while true; do curl nginx && echo ; done
   You're at the root of nginx server v1
   You're at the root of nginx server v1
   You're at the root of nginx server v1
   You're at the root of nginx server v1

Canary Deployment with Istio

With Istio, traffic routing and replica deployment are totally independent of each other. Istio routing rules provide fine-grained control over how to route traffic based on host, port, headers, uri, method, source labels and control the distribution of traffic.

Deploy another version v2 of the same application that uses image quay.io/shardul/nginx:v2.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: nginx
     name: nginx-v2
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: nginx
         version: v2
     template:
       metadata:
         labels:
           app: nginx
           version: v2
       spec:
         containers:
         - image: quay.io/shardul/nginx:v2
           name: nginx-v2
           imagePullPolicy: Always
           ports:
           - name: http
             containerPort: 80
           livenessProbe:
             httpGet:
               path: /health
               port: 80
           readinessProbe:
             httpGet:
               path: /health
               port: 80

Once the v2 version is deployed, when we hit the service again, we would get the below output :

   kubectl exec -it test-connection -- sh
   [ root@test-connection:/ ]$ while true; do curl nginx && echo ; done
   You're at the root of nginx server v2
   You're at the root of nginx server v2
   You're at the root of nginx server v1
   You're at the root of nginx server v1
   You're at the root of nginx server v2
   You're at the root of nginx server v1
   You're at the root of nginx server v1
   You're at the root of nginx server v2

Since there are two deployments with different versions exposed from the same service, whenever you hit the service, it will hit the different versions of the application in a round-robin manner and hence this output.

Istio Gateway acts as a load balancer receiving incoming and outgoing HTTP/TCP connections and it's bound to the istio-ingressgateway resource created during installation as a LoadBalancer service.

Setup a default-gateway for all the incoming traffic on port 80 :

   apiVersion: networking.istio.io/v1alpha3
   kind: Gateway
   metadata:
     name: default-gateway
     namespace: istio-system
   spec:
     selector:
       istio: ingressgateway
     servers:
     - hosts:
       - '*'
       port:
         name: http
         number: 80
         protocol: HTTP

Destination Rule allows you to define subsets of an application based on a set of labels. For example, we have deployed two subsets v1 and v2 of an application, and they are identified by the label version.

Setup a DestinationRule nginx-dest-rule to define two subsets v1 and v2:

   apiVersion: networking.istio.io/v1alpha3
   kind: DestinationRule
   metadata:
     name: nginx-dest-rule
   spec:
     host: nginx
     subsets:
     - name: v1
       labels:
         version: v1
     - name: v2
       labels:
         version: v2

Virtual Service acts just like an Ingress resource and matches traffic and directs it to a service based on the HTTP routing rules.

It can operate on internal as well as external service and can match traffic based on HTTP host, path (with full regular expression support), method, headers, ports, query parameters.

Setup a VirtualService nginx-virtual-svc for host nginx that receives the incoming traffic on default-gateway and routes 90% of the traffic to the subset v1 and 10% to subset v2 defined in nginx-dest-rule.

   apiVersion: networking.istio.io/v1alpha3
   kind: VirtualService
   metadata:
     name: nginx-virtual-svc
   spec:
     hosts:
     - nginx
     gateways:
     - istio-system/default-gateway
     http:
     - route:
       - destination:
           host: nginx
           subset: v1
         weight: 90
       - destination:
           host: nginx
           subset: v2
         weight: 10

After applying, 90% of the traffic will be routed to version v1 and 10% to v2 of the nginx service. Access the nginx service via istio-ingressgateway.

kubectl exec -it test-connection -- sh
while true; do curl -H "Host: nginx" istio-ingressgateway.istio-system && echo ; done

We can visualize the traffic flow using the Kiali dashboard:

istioctl dashboard kiali

In production, after testing the canary version to ensure that it's working fine, we can update the VirtualService to route 100% of the traffic to this version and rollout the newer version for all the users.

Cloud Native Chaos Engineering with Chaos Mesh

Shardul Srivastava — Mon, 09 Aug 2021 19:51:35 +0000

With Cloud, distributed architectures have grown even more complex and with complexity comes the uncertainty in how the system could fail.

Chaos Engineering aims to test system resiliency by injecting faults to identify weaknesses before they cause massive outages such as improper fallback settings for a service, cascading failures due to a single point of failure, or retry storms due to misconfigured timeouts.

History

Chaos Engineering started at Netflix back in 2010 when Netflix moved from on-prem servers to AWS infrastructure to test the resiliency of their infrastructure.

In 2012, Netflix open-sourced ChaosMonkey under Apache 2.0 license that randomly terminates instances to ensure that services are resilient to instance failures.

Cloud Native Chaos Engineering in CNCF Landscape

CNCF focuses on Cloud Native Chaos Engineering defined as engineering practices focused on (and built on) Kubernetes environments, applications, microservices, and infrastructure.

Cloud Native Chaos Engineering has 4 core principles:

Open source
CRDs for Chaos Management
Extensible and pluggable
Broad Community adoption

CNCF has two sandbox projects for Cloud Native Chaos Engineering

Chaos Mesh

Chaos Mesh is a cloud-native Chaos Engineering platform that orchestrates chaos on Kubernetes environments. It is based on Kubernetes Operator pattern and provides a Chaos Operator to inject into the applications and Kubernetes infrastructure in a manageable way.

Chaos Operator uses Custom Resource Defition(CRD) to define chaos objects. It provides a variety of these CRDs for fault injection such as :

Chaos Mesh Installation

Chaos Mesh can be installed quickly using installation script. However, it's recommended to use Helm 3 chart in production environments.

To install Chaos Mesh using Helm :

Add the Chaos Mesh repository to the Helm repository.

   helm repo add chaos-mesh https://charts.chaos-mesh.org

It's recommended to install ChaosMesh in a separate namespace, so you can either create a namespace chaos-testing manually or let Helm create it automatically, if it doesn't exist :

   helm upgrade \
        --install \
        chaos-mesh \
        chaos-mesh/chaos-mesh \
        -n chaos-testing \
        --create-namespace \
        --version v2.0.0 \
        --wait

Note: If you're using GKE or EKS with containerd, then use

   helm upgrade \
        --install \
        chaos-mesh \
        chaos-mesh/chaos-mesh \
        -n chaos-testing \
        --create-namespace \
        --set chaosDaemon.runtime=containerd \
        --set chaosDaemon.socketPath=/run/containerd/containerd.sock \
        --version v2.0.0 \
        --wait

Verify if pods are running :

   kubectl get pods -n chaos-testing

Run First Chaos Mesh Experiment

Chaos Experiment describes what type of fault is injected and how.

Setup an Nginx pod and expose it on port 80.

  kubectl run nginx --image=nginx --labels="app=nginx" --port=80

Get the IP of the nginx pod

  kubectl get pods nginx -ojsonpath="{.status.podIP}"

Open another terminal and setup a test pod to test the connectivity to nginx pod :

  kubectl run -it test-connection --image=radial/busyboxplus:curl -- sh
  ping <IP of the Nginx Pod> -c 2

this should show you the time it takes to ping the IP :

Create your first Chaos Experiment by running :

  kubectl apply -f - <<EOF
  apiVersion: chaos-mesh.org/v1alpha1
  kind: NetworkChaos
  metadata:
    name: nginx-network-delay
  spec:
    action: delay
    mode: one
    selector:
      namespaces:
        - default
      labelSelectors:
        'app': 'nginx'
    delay:
      latency: '1s'
    duration: '60s'
  EOF

this will create a CRD of type NetworkChaos that will introduce a latency of 1 second in the network of pods with labels app:nginx i.e nginx pod for the next 60 seconds.

Test the response of ping to the nginx pod now to see the delay of 1 second.

Run HTTPChaos Experiment

HTTPChaos allows you to inject faults in the request and response of an HTTP server. It supports abort,delay,replace,patch fault types.

Note: Before proceeding, delete the NetworkChaos experiment created earlier.

Check the response time of nginx pod :

   kubectl exec -it test-connection -- sh
   time curl <IP of the Nginx Pod>

Create HTTPSChaos experiment by running:

   kubectl apply -f - <<EOF
   apiVersion: chaos-mesh.org/v1alpha1
   kind: HTTPChaos
   metadata:
     name: nginx-http-delay
   spec:
     mode: all
     selector:
       labelSelectors:
         app: nginx
     target: Request
     port: 80
     delay: 1s
     method: GET
     path: /
     duration: 5m
   EOF

this will create a CRD of type HTTPChaos that will introduce a latency of 1 seconds to the requests sent to the pods with labels app:nginx i.e nginx pod on port 80 for the next 5 mins.

Note: If you get an error like admission webhook "vauth.kb.io" denied the request, as of version 2.0 there is an open issue 2187 and a temporary fix is to delete the validating webhook.

kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io validate-auth

Test the response time of nginx pod :

   time curl <IP of the Nginx Pod>

you will see the additional 1 second latency in the response.