DEV Community: Ben Sooraj

Ossa: A node.js server-side module (powered by Redis) for sending scheduled messages

Ben Sooraj — Tue, 12 May 2020 17:37:39 +0000

Disclaimer:

This is a side project! It is by no way complete, but it does get the job done.
This module is not production ready!
And is good for small projects/apps

At my work, I had to solve a problem where we send reminders to our users nudging them to prioritise a task pending with them (can't go beyond this with the details). These reminders were required to be configurable, that is, users get to choose when they want to be reminded: in 2 days, in 1 hour on a specific date and time etc.

After quite a bit of digging around the internet, I stumbled upon Redis Keyspace Notifications. There are many other ways you could implement this (cron jobs for example), but I decided to go with this approach. And since Redis was already one of our tech stacks, it was a viable option worth trying out.

I went ahead and wrapped up this feature into a module called Ossa.

Pretty easy to get started.

Create an ossa instance:

const Ossa = require('ossa');
const ossa = new Ossa({
    namespace: "ossa", // Default
    redis: {
        host: 'localhost', // Default
        port: 6379 // Default
    },
    debug: false, // Default
    mode: 0  // 0 => Send and receive (Default) | 1 => Send only
});

To schedule a notification/message:

try {
    const notificationID = await ossa.sendNotification({
        in: '10s',
        // on: moment().utc().add(30, 'seconds'),
        // on: '2020-05-02 03:23:00',
        // on: '2020-05-01T21:59:16Z',
        message: JSON.stringify({
            name: "Ben",
            age: 1000,
        })
    });
    console.log("notificationID: ", notificationID)
} catch (error) {
    throw new Error(error);
}

// Output:
// notificationID:  ossa::f1799e87-6740-4394-bf5e-d6e55eae3914

To receive the scheduled message:

ossa.on('notification-received', async (notificationID, notificationPayload) => {
    // Process the payload received
    console.log("notificationPayload: ", notificationPayload)
    console.table([
        { notificationID, message: notificationPayload.message }
    ]);
});

// Output:
// notificationPayload:  { in: '10s', message: '{"name":"Ben","age":1000}' }
// ┌─────────┬──────────────────────────────────────────────┬─────────────────────────────┐
// │ (index) │                notificationID                │           message           │
// ├─────────┼──────────────────────────────────────────────┼─────────────────────────────┤
// │    0    │ 'ossa::f1799e87-6740-4394-bf5e-d6e55eae3914' │ '{"name":"Ben","age":1000}' │
// └─────────┴──────────────────────────────────────────────┴─────────────────────────────┘

Do checkout the README page to learn more.

I have also created a sample ready-to-run example using docker so you can get started even faster.

So, go ahead and check it out and let me know your thoughts. Would love some community feedback.

Thanks!

Accessing Amazon RDS From AWS EKS

Ben Sooraj — Wed, 09 Oct 2019 07:08:11 +0000

Premise
Setup the MySQL Database - Amazon RDS
1. Create the VPC
2. Create the subnets
3. Create the DB subnet group
4. Create the VPC security group
5. Create a DB instance in the VPC
6. Amazon RDS setup diagram
Setup the EKS cluster
Let's build the bridge!
1. Create and Accept a VPC Peering Connection
2. Update the EKS cluster VPC's route table
3. Update the RDS VPC's route table
4. Update the RDS instance's security group
Test the connection

1. Premise

When moving your services to the Kubernetes ecosystem for the first time, it is best practice to port only the stateless parts to begin with.

Here's the problem I had to solve: Our service uses [Amazon RDS for MySQL][1]. Both the RDS instance(s) and EKS reside within their own dedicated [VPC][2]. How do resources running within AWS EKS communicate with the database?

Let's dive right in!

2. Setup the MySQL Database (Amazon RDS)

We will be using the AWS CLI for setting up MySQL database.

2.1 Create the VPC

We will first create a VPC with the CIDR block 10.0.0.0/24 which accommodate 254 hosts in all. This is more than enough to host our RDS instance.



$ aws ec2 create-vpc --cidr-block 10.0.0.0/24 | jq '{VpcId:.Vpc.VpcId,CidrBlock:.Vpc.CidrBlock}'
{
    "VpcId": "vpc-0cf40a5f6db5eb3cd",
    "CidrBlock": "10.0.0.0/24"
}

# Export the RDS VPC ID for easy reference in the subsequent commands
$ export RDS_VPC_ID=vpc-0cf40a5f6db5eb3cd

2.2 Create the subnets

RDS instances launched in a VPC must have a [DB subnet group][3]. DB subnet groups are a collection of subnets within a VPC. Each DB subnet group should have subnets in at least two Availability Zones in a given AWS Region.

We will divide the RDS VPC (RDS_VPC_ID) into two equal subnets: 10.0.0.0/25 and 10.0.0.128/25.

So, let's create the first subnet in the availability zone ap-south-1b:



$ aws ec2 create-subnet --availability-zone "ap-south-1b" --vpc-id ${RDS_VPC_ID} --cidr-block 10.0.0.0/25 | jq '{SubnetId:.Subnet.SubnetId,AvailabilityZone:.Subnet.AvailabilityZone,CidrBlock:.Subnet.CidrBlock,VpcId:.Subnet.VpcId}'
# Response:
{
  "SubnetId": "subnet-042a4bee8e92287e8",
  "AvailabilityZone": "ap-south-1b",
  "CidrBlock": "10.0.0.0/25",
  "VpcId": "vpc-0cf40a5f6db5eb3cd"
}

and the second one in the availability zone ap-south-1a



$ aws ec2 create-subnet --availability-zone "ap-south-1a" --vpc-id ${RDS_VPC_ID} --cidr-block 10.0.0.128/25 | jq '{SubnetId:.Subnet.SubnetId,AvailabilityZone:.Subnet.AvailabilityZone,CidrBlock:.Subnet.CidrBlock,VpcId:.Subnet.VpcId}'
# Response:
{
  "SubnetId": "subnet-0c01a5ba480b930f4",
  "AvailabilityZone": "ap-south-1a",
  "CidrBlock": "10.0.0.128/25",
  "VpcId": "vpc-0cf40a5f6db5eb3cd"
}

Each VPC has an implicit router which controls where network traffic is directed. Each subnet in a VPC must be explicitly associated with a route table, which controls the routing for the subnet.

Let's go ahead and associate these two subnet that we created, to the VPC's route table:



# Fetch the route table information
$ aws ec2 describe-route-tables --filters Name=vpc-id,Values=${RDS_VPC_ID} | jq '.RouteTables[0].RouteTableId'
"rtb-0e680357de97595b1"

# For easy reference
$ export RDS_ROUTE_TABLE_ID=rtb-0e680357de97595b1

# Associate the first subnet with the route table
$ aws ec2 associate-route-table --route-table-id rtb-0e680357de97595b1 --subnet-id subnet-042a4bee8e92287e8
{
    "AssociationId": "rtbassoc-02198db22b2d36c97"
}

# Associate the second subnet with the route table
$ aws ec2 associate-route-table --route-table-id rtb-0e680357de97595b1 --subnet-id subnet-0c01a5ba480b930f4
{
    "AssociationId": "rtbassoc-0e5c3959d360c92ab"
}

2.3 Create DB Subnet Group

Now that we have two subnets spanning two availability zones, we can go ahead and create the DB subnet group.



$ aws rds create-db-subnet-group --db-subnet-group-name  "DemoDBSubnetGroup" --db-subnet-group-description "Demo DB Subnet Group" --subnet-ids "subnet-042a4bee8e92287e8" "subnet-0c01a5ba480b930f4" | jq '{DBSubnetGroupName:.DBSubnetGroup.DBSubnetGroupName,VpcId:.DBSubnetGroup.VpcId,Subnets:.DBSubnetGroup.Subnets[].SubnetIdentifier}'
# Response:
{
  "DBSubnetGroupName": "demodbsubnetgroup",
  "VpcId": "vpc-0cf40a5f6db5eb3cd",
  "Subnets": "subnet-0c01a5ba480b930f4"
}
{
  "DBSubnetGroupName": "demodbsubnetgroup",
  "VpcId": "vpc-0cf40a5f6db5eb3cd",
  "Subnets": "subnet-042a4bee8e92287e8"
}

2.4 Create a VPC Security Group

The penultimate step to creating the DB instance is creating a VPC security group, an instance level virtual firewall with rules to control inbound and outbound traffic.



$ aws ec2 create-security-group --group-name DemoRDSSecurityGroup --description "Demo RDS security group" --vpc-id ${RDS_VPC_ID}
{
    "GroupId": "sg-06800acf8d6279971"
}

# Export the RDS VPC Security Group ID for easy reference in the subsequent commands
$ export RDS_VPC_SECURITY_GROUP_ID=sg-06800acf8d6279971

We will use this security group at a later point, to set an inbound rule to allow all traffic from the EKS cluster to the RDS instance.

2.5 Create a DB Instance in the VPC



$ aws rds create-db-instance \
  --db-name demordsmyqldb \
  --db-instance-identifier demordsmyqldbinstance \
  --allocated-storage 10 \
  --db-instance-class db.t2.micro \
  --engine mysql \
  --engine-version "5.7.26" \
  --master-username demoappuser \
  --master-user-password demoappuserpassword \
  --no-publicly-accessible \
  --vpc-security-group-ids ${RDS_VPC_SECURITY_GROUP_ID} \
  --db-subnet-group-name "demodbsubnetgroup" \
  --availability-zone ap-south-1b \
  --port 3306 | jq '{DBInstanceIdentifier:.DBInstance.DBInstanceIdentifier,Engine:.DBInstance.Engine,DBName:.DBInstance.DBName,VpcSecurityGroups:.DBInstance.VpcSecurityGroups,EngineVersion:.DBInstance.EngineVersion,PubliclyAccessible:.DBInstance.PubliclyAccessible}'

# Respone:
{
  "DBInstanceIdentifier": "demordsmyqldbinstance",
  "Engine": "mysql",
  "DBName": "demordsmyqldb",
  "VpcSecurityGroups": [
    {
      "VpcSecurityGroupId": "sg-06800acf8d6279971",
      "Status": "active"
    }
  ],
  "EngineVersion": "5.7.26",
  "PubliclyAccessible": false
}

We can verify that the DB instance has been created in the UI as well:

2.6 Amazon RDS setup diagram

3. Setup the EKS cluster

Spinning up an EKS cluster on AWS is as simple as:



$ eksctl create cluster --name=demo-eks-cluster --nodes=2 --region=ap-south-1
[ℹ]  using region ap-south-1
[ℹ]  setting availability zones to [ap-south-1a ap-south-1c ap-south-1b]
[ℹ]  subnets for ap-south-1a - public:192.168.0.0/19 private:192.168.96.0/19
[ℹ]  subnets for ap-south-1c - public:192.168.32.0/19 private:192.168.128.0/19
[ℹ]  subnets for ap-south-1b - public:192.168.64.0/19 private:192.168.160.0/19
[ℹ]  nodegroup "ng-ae09882f" will use "ami-09c3eb35bb3be46a4" [AmazonLinux2/1.12]
[ℹ]  creating EKS cluster "demo-eks-cluster" in "ap-south-1" region
[ℹ]  will create 2 separate CloudFormation stacks for cluster itself and the initial nodegroup
[ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=ap-south-1 --name=demo-eks-cluster'
[ℹ]  2 sequential tasks: { create cluster control plane "demo-eks-cluster", create nodegroup "ng-ae09882f" }
[ℹ]  building cluster stack "eksctl-demo-eks-cluster-cluster"
[ℹ]  deploying stack "eksctl-demo-eks-cluster-cluster"
[ℹ]  building nodegroup stack "eksctl-demo-eks-cluster-nodegroup-ng-ae09882f"
[ℹ]  --nodes-min=2 was set automatically for nodegroup ng-ae09882f
[ℹ]  --nodes-max=2 was set automatically for nodegroup ng-ae09882f
[ℹ]  deploying stack "eksctl-demo-eks-cluster-nodegroup-ng-ae09882f"
[✔]  all EKS cluster resource for "demo-eks-cluster" had been created
[✔]  saved kubeconfig as "/Users/Bensooraj/.kube/config"
[ℹ]  adding role "arn:aws:iam::account_number:role/eksctl-demo-eks-cluster-nodegroup-NodeInstanceRole-1631FNZJZTDSK" to auth ConfigMap
[ℹ]  nodegroup "ng-ae09882f" has 0 node(s)
[ℹ]  waiting for at least 2 node(s) to become ready in "ng-ae09882f"
[ℹ]  nodegroup "ng-ae09882f" has 2 node(s)
[ℹ]  node "ip-192-168-30-190.ap-south-1.compute.internal" is ready
[ℹ]  node "ip-192-168-92-207.ap-south-1.compute.internal" is ready
[ℹ]  kubectl command should work with "/Users/Bensooraj/.kube/config", try 'kubectl get nodes'
[✔]  EKS cluster "demo-eks-cluster" in "ap-south-1" region is ready

We will create a kubernetes Service named mysql-service of type ExternalName aliasing the RDS endpoint demordsmyqldbinstance.cimllxgykuy3.ap-south-1.rds.amazonaws.com.

Run kubectl apply -f mysql-service.yaml to create the service.



# mysql-service.yaml 
apiVersion: v1
kind: Service
metadata:
  labels:
    app: mysql-service
  name: mysql-service
spec:
  externalName: demordsmyqldbinstance.cimllxgykuy3.ap-south-1.rds.amazonaws.com
  selector:
    app: mysql-service
  type: ExternalName
status:
  loadBalancer: {}

Now, clients running inside the pods within the cluster can connect to the RDS instance using mysql-service.

Let's test the connect using a throwaway busybox pod:



$ kubectl run -i --tty --rm debug --image=busybox --restart=Never -- sh
If you don't see a command prompt, try pressing enter.
/ # nc mysql-service 3306
^Cpunt!

It is evident that the pod is unable to get through! Let's solve the problem now.

4. Let's build the bridge!

We are going to create a [VPC Peering Connection][5] to facilitate communication between the resources in the two VPCs. According to the documentation:

A VPC peering connection is a networking connection between two VPCs that enables you to route traffic between them using private IPv4 addresses or IPv6 addresses. Instances in either VPC can communicate with each other as if they are within the same network. You can create a VPC peering connection between your own VPCs, or with a VPC in another AWS account. The VPCs can be in different regions (also known as an inter-region VPC peering connection).

4.1 Create and Accept a VPC Peering Connection

To create a VPC peering connection, navigate to:

VPC console: https://console.aws.amazon.com/vpc/
Select Peering Connections and click on Create Peering Connection.
Configure the details as follows (select the EKS VPC as the Requester and the RDS VPC as the Accepter):
Click on Create Peering Connection
Select the Peering Connection that we just created. Click on Actions => Accept. Again, in the confirmation dialog box, click on Yes, Accept.

Don't forget to export the VPC Peering Connection ID:



$ export VPC_PEERING_CONNECTION_ID=pcx-0cc408e65493fe197

4.2 Update the EKS cluster VPC's route table



# Fetch the route table associated with the 3 public subnets of the VPC created by `eksctl`:
$ aws ec2 describe-route-tables --filters Name="tag:aws:cloudformation:logical-id",Values="PublicRouteTable" | jq '.RouteTables[0].RouteTableId'
"rtb-06103bd0704b3a9ee"

# For easy reference
export EKS_ROUTE_TABLE_ID=rtb-06103bd0704b3a9ee

# Add route: All traffic to (destination) the RDS VPC CIDR block is via the VPC Peering Connection (target)
$ aws ec2 create-route --route-table-id ${EKS_ROUTE_TABLE_ID} --destination-cidr-block 10.0.0.0/24 --vpc-peering-connection-id ${VPC_PEERING_CONNECTION_ID}
{
    "Return": true
}

4.3 Update the RDS VPC's route table



# Add route: All traffic to (destination) the EKS cluster CIDR block is via the VPC Peering Connection (target)
$ aws ec2 create-route --route-table-id ${RDS_ROUTE_TABLE_ID} --destination-cidr-block 192.168.0.0/16 --vpc-peering-connection-id ${VPC_PEERING_CONNECTION_ID}
{
    "Return": true
}

4.4 Update the RDS instance's security group

Allow all ingress traffic from the EKS cluster to the RDS instance on port 3306:



$ aws ec2 authorize-security-group-ingress --group-id ${RDS_VPC_SECURITY_GROUP_ID} --protocol tcp --port 3306 --cidr 192.168.0.0/16

5. Test the connection



$ kubectl run -i --tty --rm debug --image=busybox --restart=Never -- sh
If you don't see a command prompt, try pressing enter.
/ # nc mysql-service 3306
N
5.7.26-logR&=lk`xTH???mj   _5#K)>mysql_native_password
```

We can see that `busybox` can now successfully talk to the RDS instance using the service `mysql-service`.

That said, this is what our final setup looks like (lot of hard work guys):
![Final setup](https://thepracticaldev.s3.amazonaws.com/i/1ba38e5zu8i36egibtvc.jpeg)


**Note**:
This setup allows all pods in the EKS cluster to access the RDS instance. Depending on your use case, this may or may not be ideal to your architecture. To implement more fine-grained access control, considering setting up a [`NetworkPolicy`][6] resource.

Useful resources:
1. [Visual Subnet Calculator][4]
2. [jq - Command-line JSON processor][7]
3. [AWS CLI Command Reference][8]
4. [AWS VPC Peering][5]


[1]: https://aws.amazon.com/rds/mysql/
[2]: https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html
[3]: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.WorkingWithRDSInstanceinaVPC.html#USER_VPC.Subnets
[4]: http://www.davidc.net/sites/default/subnets/subnets.html
[5]: https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html
[6]: https://kubernetes.io/docs/concepts/services-networking/network-policies/
[7]: https://github.com/stedolan/jq
[8]: https://docs.aws.amazon.com/cli/latest/index.html

Up And Running With Kafka On AWS EKS Using Strimzi

Ben Sooraj — Thu, 03 Oct 2019 10:20:27 +0000

Disclaimer: This is not a tutorial per se, instead, this is me recording my observations as I setup a Kafka cluster for the first time on a Kubernetes platform using Strimzi.

Configure the AWS CLI
Create the EKS cluster
Enter Kubernetes
Install and configure Helm
Install the Strimzi Kafka Operator
Deploying the Kafka cluster
Analysis
Test the Kafka cluster with Node.js clients
Clean up!

Let's get right into it, then!

We will be using eksctl, the official CLI for Amazon EKS, to spin up our K8s cluster.

Configure the AWS CLI

Ensure that the AWS CLI is configured. To view your configuration:



$ aws configure list
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key     ****************7ONG shared-credentials-file    
secret_key     ****************lbQg shared-credentials-file    
    region               ap-south-1      config-file    ~/.aws/config

Note: The aws CLI config and credentials details are usually stored at ~/.aws/config and ~/.aws/credentials respectively.

Create the EKS cluster




$ eksctl create cluster --name=kafka-eks-cluster --nodes=4 --region=ap-south-1

[ℹ]  using region ap-south-1
[ℹ]  setting availability zones to [ap-south-1b ap-south-1a ap-south-1c]
[ℹ]  subnets for ap-south-1b - public:192.168.0.0/19 private:192.168.96.0/19
[ℹ]  subnets for ap-south-1a - public:192.168.32.0/19 private:192.168.128.0/19
[ℹ]  subnets for ap-south-1c - public:192.168.64.0/19 private:192.168.160.0/19
[ℹ]  nodegroup "ng-9f3cbfc7" will use "ami-09c3eb35bb3be46a4" [AmazonLinux2/1.12]
[ℹ]  creating EKS cluster "kafka-eks-cluster" in "ap-south-1" region
[ℹ]  will create 2 separate CloudFormation stacks for cluster itself and the initial nodegroup
[ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=ap-south-1 --name=kafka-eks-cluster'
[ℹ]  2 sequential tasks: { create cluster control plane "kafka-eks-cluster", create nodegroup "ng-9f3cbfc7" }
[ℹ]  building cluster stack "eksctl-kafka-eks-cluster-cluster"
[ℹ]  deploying stack "eksctl-kafka-eks-cluster-cluster"
[ℹ]  building nodegroup stack "eksctl-kafka-eks-cluster-nodegroup-ng-9f3cbfc7"
[ℹ]  --nodes-min=4 was set automatically for nodegroup ng-9f3cbfc7
[ℹ]  --nodes-max=4 was set automatically for nodegroup ng-9f3cbfc7
[ℹ]  deploying stack "eksctl-kafka-eks-cluster-nodegroup-ng-9f3cbfc7"
[✔]  all EKS cluster resource for "kafka-eks-cluster" had been created
[✔]  saved kubeconfig as "/Users/Bensooraj/.kube/config"
[ℹ]  adding role "arn:aws:iam::account_numer:role/eksctl-kafka-eks-cluster-nodegrou-NodeInstanceRole-IG63RKPE03YQ" to auth ConfigMap
[ℹ]  nodegroup "ng-9f3cbfc7" has 0 node(s)
[ℹ]  waiting for at least 4 node(s) to become ready in "ng-9f3cbfc7"
[ℹ]  nodegroup "ng-9f3cbfc7" has 4 node(s)
[ℹ]  node "ip-192-168-25-34.ap-south-1.compute.internal" is ready
[ℹ]  node "ip-192-168-50-249.ap-south-1.compute.internal" is ready
[ℹ]  node "ip-192-168-62-231.ap-south-1.compute.internal" is ready
[ℹ]  node "ip-192-168-69-95.ap-south-1.compute.internal" is ready
[ℹ]  kubectl command should work with "/Users/Bensooraj/.kube/config", try 'kubectl get nodes'
[✔]  EKS cluster "kafka-eks-cluster" in "ap-south-1" region is ready

A k8s cluster by the name kafka-eks-cluster will be created with 4 nodes (instance type: m5.large) in the Mumbai region (ap-south-1). You can view these in the AWS Console UI as well,

EKS:

CloudFormation UI:

Also, after the cluster is created, the appropriate kubernetes configuration will be added to your kubeconfig file (defaults to ~/.kube/config). The path to the kubeconfig file can be overridden using the --kubeconfig flag.

Enter Kubernetes

Fetching all k8s controllers lists the default kubernetes service. This confirms that kubectl is properly configured to point to the cluster that we just created.



$ kubectl get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.100.0.1   <none>        443/TCP   19m

Install and configure Helm

Helm is a package manager and application management tool for Kubernetes that packages multiple Kubernetes resources into a single logical deployment unit called Chart.

I use Homebrew, so the installation was pretty straightforward: brew install kubernetes-helm.

Alternatively, to install helm, run the following:



$ cd ~/eks-kafka-strimzi

$ curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > get_helm.sh

$ chmod +x get_helm.sh

$ ./get_helm.sh

Read through their installation guide, if you are looking for more options.

Do not run helm init yet.

Helm relies on a service called tiller that requires special permission on the kubernetes cluster, so we need to build a Service Account (RBAC access) for tiller to use.

The rbac.yaml file would look like the following:



---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system

Apply this to the kafka-eks-cluster cluster:



$ kubectl apply -f rbac.yaml
serviceaccount/tiller created
clusterrolebinding.rbac.authorization.k8s.io/tiller created

# Verify (listing only the relevant ones)
$ kubectl get sa,clusterrolebindings --namespace=kube-system
NAME                        SECRETS   AGE
.
serviceaccount/tiller       1         5m22s
.

NAME                                                                                                AGE
.
clusterrolebinding.rbac.authorization.k8s.io/tiller                                                 5m23s
.

Now, run helm init using the service account we setup. This will install tiller into the cluster which gives it access to manage resources in your cluster.



$ helm init --service-account=tiller

$HELM_HOME has been configured at /Users/Bensooraj/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.

To prevent this, run `helm init` with the --tiller-tls-verify flag.

For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation

Install the Strimzi Kafka Operator

Add the Strimzi repository and install the Strimzi Helm Chart:



# Add the repo
$ helm repo add strimzi http://strimzi.io/charts/
"strimzi" has been added to your repositories

# Search for all Strimzi  charts
$ helm search strim
NAME                            CHART VERSION   APP VERSION DESCRIPTION                
strimzi/strimzi-kafka-operator  0.14.0          0.14.0      Strimzi: Kafka as a Service

# Install the kafka operator
$ helm install strimzi/strimzi-kafka-operator
NAME:   bulging-gnat
LAST DEPLOYED: Wed Oct  2 15:23:45 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/ClusterRole
NAME                                 AGE
strimzi-cluster-operator-global      0s
strimzi-cluster-operator-namespaced  0s
strimzi-entity-operator              0s
strimzi-kafka-broker                 0s
strimzi-topic-operator               0s

==> v1/ClusterRoleBinding
NAME                                              AGE
strimzi-cluster-operator                          0s
strimzi-cluster-operator-kafka-broker-delegation  0s

==> v1/Deployment
NAME                      READY  UP-TO-DATE  AVAILABLE  AGE
strimzi-cluster-operator  0/1    1           0          0s

==> v1/Pod(related)
NAME                                       READY  STATUS             RESTARTS  AGE
strimzi-cluster-operator-6667fbc5f8-cqvdv  0/1    ContainerCreating  0         0s

==> v1/RoleBinding
NAME                                                 AGE
strimzi-cluster-operator                             0s
strimzi-cluster-operator-entity-operator-delegation  0s
strimzi-cluster-operator-topic-operator-delegation   0s

==> v1/ServiceAccount
NAME                      SECRETS  AGE
strimzi-cluster-operator  1        0s

==> v1beta1/CustomResourceDefinition
NAME                                AGE
kafkabridges.kafka.strimzi.io       0s
kafkaconnects.kafka.strimzi.io      0s
kafkaconnects2is.kafka.strimzi.io   0s
kafkamirrormakers.kafka.strimzi.io  0s
kafkas.kafka.strimzi.io             1s
kafkatopics.kafka.strimzi.io        1s
kafkausers.kafka.strimzi.io         1s

NOTES:
Thank you for installing strimzi-kafka-operator-0.14.0

To create a Kafka cluster refer to the following documentation.

https://strimzi.io/docs/0.14.0/#kafka-cluster-str

List all the kubernetes objects created again:



$ kubectl get all
NAME                                            READY   STATUS    RESTARTS   AGE
pod/strimzi-cluster-operator-6667fbc5f8-cqvdv   1/1     Running   0          9m25s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.100.0.1   <none>        443/TCP   90m

NAME                                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/strimzi-cluster-operator   1         1         1            1           9m25s

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/strimzi-cluster-operator-6667fbc5f8   1         1         1       9m26s

Deploying the Kafka cluster

We will now create a Kafka cluster with 3 brokers. The YAML file (kafka-cluster.Kafka.yaml) for creating the Kafka cluster would like the following:



apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: kafka-cluster
spec:
  kafka:
    version: 2.3.0 # Kafka version
    replicas: 3 # Replicas specifies the number of broker nodes.
    listeners: # Listeners configure how clients connect to the Kafka cluster
      plain: {} # 9092
      tls: {} # 9093
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: "2.3"
      delete.topic.enable: "true"
    storage:
      type: persistent-claim
      size: 10Gi
      deleteClaim: false
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim # Persistent storage backed by AWS EBS
      size: 10Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {} # Operator for topic administration
    userOperator: {}

Apply the above YAML file:



$ kubectl apply -f kafka-cluster.Kafka.yaml

Analysis

This is where things get interesting. We will now analyse some of the k8s resources which the strimzi kafka operator has created for us under the hood.



$ kubectl get statefulsets.apps,pod,deployments,svc
NAME                                       DESIRED   CURRENT   AGE
statefulset.apps/kafka-cluster-kafka       3         3         78m
statefulset.apps/kafka-cluster-zookeeper   3         3         79m

NAME                                                 READY   STATUS    RESTARTS   AGE
pod/kafka-cluster-entity-operator-54cb77fd9d-9zbcx   3/3     Running   0          77m
pod/kafka-cluster-kafka-0                            2/2     Running   0          78m
pod/kafka-cluster-kafka-1                            2/2     Running   0          78m
pod/kafka-cluster-kafka-2                            2/2     Running   0          78m
pod/kafka-cluster-zookeeper-0                        2/2     Running   0          79m
pod/kafka-cluster-zookeeper-1                        2/2     Running   0          79m
pod/kafka-cluster-zookeeper-2                        2/2     Running   0          79m
pod/strimzi-cluster-operator-6667fbc5f8-cqvdv        1/1     Running   0          172m

NAME                                                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.extensions/kafka-cluster-entity-operator   1         1         1            1           77m
deployment.extensions/strimzi-cluster-operator        1         1         1            1           172m

NAME                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/kafka-cluster-kafka-bootstrap    ClusterIP   10.100.177.177   <none>        9091/TCP,9092/TCP,9093/TCP   78m
service/kafka-cluster-kafka-brokers      ClusterIP   None             <none>        9091/TCP,9092/TCP,9093/TCP   78m
service/kafka-cluster-zookeeper-client   ClusterIP   10.100.199.128   <none>        2181/TCP                     79m
service/kafka-cluster-zookeeper-nodes    ClusterIP   None             <none>        2181/TCP,2888/TCP,3888/TCP   79m
service/kubernetes                       ClusterIP   10.100.0.1       <none>        443/TCP                      4h13m

Points to note:

The StatefulSet kafka-cluster-zookeeper has created 3 pods - kafka-cluster-zookeeper-0, kafka-cluster-zookeeper-1 and kafka-cluster-zookeeper-2. The headless service kafka-cluster-zookeeper-nodes facilitates network identity of these 3 pods (the 3 Zookeeper nodes).
The StatefulSet kafka-cluster-kafka has created 3 pods - kafka-cluster-kafka-0, kafka-cluster-kafka-1 and kafka-cluster-kafka-2. The headless service kafka-cluster-kafka-brokers facilitates network identity of these 3 pods (the 3 Kafka brokers).

Persistent volumes are dynamically provisioned:



$ kubectl get pv,pvc
NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                    STORAGECLASS   REASON   AGE
persistentvolume/pvc-7ff2909f-e507-11e9-91df-0a1e73fdd786   10Gi       RWO            Delete           Bound    default/data-kafka-cluster-zookeeper-1   gp2                     11h
persistentvolume/pvc-7ff290c4-e507-11e9-91df-0a1e73fdd786   10Gi       RWO            Delete           Bound    default/data-kafka-cluster-zookeeper-2   gp2                     11h
persistentvolume/pvc-7ffd1d22-e507-11e9-a775-029ce0835b96   10Gi       RWO            Delete           Bound    default/data-kafka-cluster-zookeeper-0   gp2                     11h
persistentvolume/pvc-a5997b77-e507-11e9-91df-0a1e73fdd786   10Gi       RWO            Delete           Bound    default/data-kafka-cluster-kafka-0       gp2                     11h
persistentvolume/pvc-a599e52b-e507-11e9-91df-0a1e73fdd786   10Gi       RWO            Delete           Bound    default/data-kafka-cluster-kafka-1       gp2                     11h
persistentvolume/pvc-a59c6cd2-e507-11e9-91df-0a1e73fdd786   10Gi       RWO            Delete           Bound    default/data-kafka-cluster-kafka-2       gp2                     11h

NAME                                                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/data-kafka-cluster-kafka-0       Bound    pvc-a5997b77-e507-11e9-91df-0a1e73fdd786   10Gi       RWO            gp2            11h
persistentvolumeclaim/data-kafka-cluster-kafka-1       Bound    pvc-a599e52b-e507-11e9-91df-0a1e73fdd786   10Gi       RWO            gp2            11h
persistentvolumeclaim/data-kafka-cluster-kafka-2       Bound    pvc-a59c6cd2-e507-11e9-91df-0a1e73fdd786   10Gi       RWO            gp2            11h
persistentvolumeclaim/data-kafka-cluster-zookeeper-0   Bound    pvc-7ffd1d22-e507-11e9-a775-029ce0835b96   10Gi       RWO            gp2            11h
persistentvolumeclaim/data-kafka-cluster-zookeeper-1   Bound    pvc-7ff2909f-e507-11e9-91df-0a1e73fdd786   10Gi       RWO            gp2            11h
persistentvolumeclaim/data-kafka-cluster-zookeeper-2   Bound    pvc-7ff290c4-e507-11e9-91df-0a1e73fdd786   10Gi       RWO            gp2            11h

You can view the provisioned AWS EBS volumes in the UI as well:

Create topics

Before we get started with clients we need to create a topic (with 3 partitions and a replication factor of 3), over which our producer and the consumer and produce messages and consume messages on respectively.



apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaTopic
metadata:
  name: test-topic
  labels:
    strimzi.io/cluster: kafka-cluster
spec:
  partitions: 3
  replicas: 3

Apply the YAML to the k8s cluster:



$ kubectl apply -f create-topics.yaml
kafkatopic.kafka.strimzi.io/test-topic created

Test the Kafka cluster with Node.js clients

The multi-broker Kafka cluster that we deployed is backed by statefulsets and their corresponding headless services.

Since each Pod (Kafka broker) now has a network identity, clients can connect to the Kafka brokers via a combination of the pod name and service name: $(podname).$(governing service domain). In our case, these would be the following URLs:

kafka-cluster-kafka-0.kafka-cluster-kafka-brokers
kafka-cluster-kafka-1.kafka-cluster-kafka-brokers
kafka-cluster-kafka-2.kafka-cluster-kafka-brokers

Note:

If the Kafka cluster is deployed in a different namespace, you will have to expand it a little further: $(podname).$(service name).$(namespace).svc.cluster.local.
Alternatively, the clients can connect to the Kafka cluster using the Service kafka-cluster-kafka-bootstrap:9092 as well. It distributes the connection over the three broker specific endpoints I have listed above. As I no longer keep track of the individual broker endpoints, this method plays out well when I have to scale up or down the number of brokers in the Kafka cluster.

First, clone this repo:

bensooraj / strimzi-kafka-aws-eks



# Create the configmap, which contains details such as the broker DNS names, topic name and consumer group ID
$ kubectl apply -f test/k8s/config.yaml
configmap/kafka-client-config created

# Create the producer deployment
$ kubectl apply -f test/k8s/producer.Deployment.yaml
deployment.apps/node-test-producer created

# Expose the producer deployment via a service of type LoadBalancer (backed by the AWS Elastic Load Balancer). This just makes it easy for me to curl from postman
$ kubectl apply -f test/k8s/producer.Service.yaml
service/node-test-producer created

# Finally, create the consumer deployment
$ kubectl apply -f test/k8s/consumer.Deployment.yaml
deployment.apps/node-test-consumer created

If you list the producer service that we created, you would notice a URL under EXTERNAL-IP:



$ kubectl get svc
NAME                             TYPE           CLUSTER-IP       EXTERNAL-IP                                                                PORT(S)                      AGE
.
.
node-test-producer               LoadBalancer   10.100.145.203   ac5f3d0d1e55a11e9a775029ce0835b9-2040242746.ap-south-1.elb.amazonaws.com   80:31231/TCP                 55m

The URL ac5f3d0d1e55a11e9a775029ce0835b9-2040242746.ap-south-1.elb.amazonaws.com is an AWS ELB backed public endpoint which we will be querying for producing messages to the Kafka cluster.

Also, you can see that there is 1 producer and 3 consumers (one for each partition of the topic test-topic):



$ kubectl get pod
NAME                                             READY   STATUS    RESTARTS   AGE
node-test-consumer-96b44cbcb-gs2km               1/1     Running   0          125m
node-test-consumer-96b44cbcb-ptvjd               1/1     Running   0          125m
node-test-consumer-96b44cbcb-xk75j               1/1     Running   0          125m
node-test-producer-846d9c5986-vcsf2              1/1     Running   0          125m

The producer app basically exposes 3 URLs:

/kafka-test/green/:message
/kafka-test/blue/:message
/kafka-test/cyan/:message

Where :message can be any valid string. Each of these URLs produce a message along with the colour information to the topic test-topic.

The consumer group (the 3 consumer pods that we spin-up) listening for any incoming messages from the topic test-topic, then receives these messages and prints them on to the console according to the colour instruction.

I curl each URL 3 times. From the following GIF you can see how message consumption is distributed across the 3 consumers in a round-robin manner:

Clean Up!




# Delete the test producer and consumer apps:
$ kubectl delete -f test/k8s/
configmap "kafka-client-config" deleted
deployment.apps "node-test-consumer" deleted
deployment.apps "node-test-producer" deleted
service "node-test-producer" deleted

# Delete the Kafka cluster
$ kubectl delete kafka kafka-cluster
kafka.kafka.strimzi.io "kafka-cluster" deleted

# Delete the Strimzi cluster operator
$ kubectl delete deployments. strimzi-cluster-operator
deployment.extensions "strimzi-cluster-operator" deleted

# Manually delete the persistent volumes
# Kafka
$ kubectl delete pvc data-kafka-cluster-kafka-0
$ kubectl delete pvc data-kafka-cluster-kafka-1
$ kubectl delete pvc data-kafka-cluster-kafka-2
# Zookeeper
$ kubectl delete pvc data-kafka-cluster-zookeeper-0
$ kubectl delete pvc data-kafka-cluster-zookeeper-1
$ kubectl delete pvc data-kafka-cluster-zookeeper-2

Finally, delete the EKS cluster:



$ eksctl delete cluster kafka-eks-cluster
[ℹ]  using region ap-south-1
[ℹ]  deleting EKS cluster "kafka-eks-cluster"
[✔]  kubeconfig has been updated
[ℹ]  2 sequential tasks: { delete nodegroup "ng-9f3cbfc7", delete cluster control plane "kafka-eks-cluster" [async] }
[ℹ]  will delete stack "eksctl-kafka-eks-cluster-nodegroup-ng-9f3cbfc7"
[ℹ]  waiting for stack "eksctl-kafka-eks-cluster-nodegroup-ng-9f3cbfc7" to get deleted
[ℹ]  will delete stack "eksctl-kafka-eks-cluster-cluster"
[✔]  all cluster resources were deleted

Hope this helped!