Use CDK to deploy a complete solution with Kafka, App Runner, EKS and DynamoDB

A previous blog post covered how to deploy a Go Lambda function and trigger it in response to events sent to a topic in a MSK Serverless cluster.

This blog will take it a notch further.

• The solution consists of a MSK Serverless cluster, a producer application on AWS App Runner and a consumer application in Kubernetes (EKS) persisting data to DynamoDB.
• The core components (MSK cluster, EKS and DynamoDB) and the producer application will be provisioned using Infrastructure-as-code with AWS CDK.
• Since the consumer application on EKS will interact with both MSK and DynamoDB, you will also need to configure appropriate IAM roles.

All the components in this solution have been written in Go.

• The MSK producer and consumer app use the franz-go library (it also supports MSK IAM authentication).
• The CDK stacks have been written using CDK Go library.

Prerequisites

You will need the following:

• An AWS account
• Install AWS CDK, AWS CLI, Docker, eksctl and curl.

Use CDK to provision MSK, EKS and DynamoDB

All the code and config is present in this GitHub repo. Clone the GitHub repo and change to the right directory:

git clone https://github.com/abhirockzz/msk-cdk-apprunner-eks-dynamodb
cd msk-cdk-apprunner-eks-dynamodb/cdk

Deploy the first CDK stack:

cdk deploy MSKDynamoDBEKSInfraStack

Wait for the all the components to get provisioned, including MSK Serverless cluster, EKS cluster and DynamoDB.
You can check its progress in the AWS CloudFormation console.

You can take a look at the CDK stack code here.

Deploy MSK producer application to App Runner using CDK

Deploy the second CDK stack.

Note that in addition to deploying the producer application to App Runner, it also builds and uploads the consumer application Docker image to an ECR repository.

Make sure to enter the MSK Serverless broker endpoint URL.

export MSK_BROKER=<enter endpoint>
export MSK_TOPIC=test-topic

cdk deploy AppRunnerServiceStack

Wait for the the producer application to get deployed to App Runner. You can check its progress in the AWS CloudFormation console.

You can take a look at the CDK stack code here and the producer application here.

Once complete, make a note of the App Runner application public endpoint as well as the ECR repository for the consumer application. You should see these in the stack output as such:

Outputs:
AppRunnerServiceStack.AppURL = <app URL>
AppRunnerServiceStack.ConsumerAppDockerImage = <ecr docker image>
....

Now, you can verify if the application is functioning properly. Get the publicly accessible URL for the App Runner application and invoke it using curl. This will create the MSK topic and send data specified in the HTTP POST body.

export APP_RUNNER_URL=<enter app runner URL>

curl -i -X POST -d '{"email":"[e-mail]","name":"user1"}' $APP_RUNNER_URL

Now you can deploy the consumer application to the EKS cluster. Before that, execute the steps to configure appropriate permissions for the application to interact with MSK and DynamoDB.

Configure IRSA for consumer application

Exit the cdk directory and change to the root of the project:

cd ..

Create an IAM OIDC identity provider for your cluster with eksctl

export EKS_CLUSTER_NAME=<EKS cluster name>

oidc_id=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)

aws iam list-open-id-connect-providers | grep $oidc_id

eksctl utils associate-iam-oidc-provider --cluster $EKS_CLUSTER_NAME --approve

Define IAM roles for the application

Configure IAM Roles for Service Accounts (also known as IRSA).

Refer to the documentation for details

Start by creating a Kubernetes Service Account:

kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eks-app-sa
EOF

To verify - kubectl get serviceaccount/eks-app-sa -o yaml

Set your AWS Account ID and OIDC Identity provider as environment variables:

ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)

export AWS_REGION=<enter region e.g. us-east-1>

OIDC_PROVIDER=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")

Create a JSON file with Trusted Entities for the role:

read -r -d '' TRUST_RELATIONSHIP <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:aud": "sts.amazonaws.com",
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:default:eks-app-sa"
        }
      }
    }
  ]
}
EOF
echo "${TRUST_RELATIONSHIP}" > trust.json

To verify - cat trust.json

Now, create the IAM role:

export ROLE_NAME=msk-consumer-app-irsa

aws iam create-role --role-name $ROLE_NAME --assume-role-policy-document file://trust.json --description "IRSA for MSK consumer app on EKS"

You will need to create and attach policy to role since we only want the consumer application to consume data from MSK cluster and put data to DynamoDB table - this needs to be fine-grained.

In the policy.json file, replace values for MSK cluster and DynamoDB. Create and attach the policy to the role you just created:

aws iam create-policy --policy-name msk-consumer-app-policy --policy-document file://policy.json

aws iam attach-role-policy --role-name $ROLE_NAME --policy-arn=arn:aws:iam::$ACCOUNT_ID:policy/msk-consumer-app-policy

Finally, associate the IAM role with the Kubernetes Service Account that you created earlier:

kubectl annotate serviceaccount -n default eks-app-sa eks.amazonaws.com/role-arn=arn:aws:iam::$ACCOUNT_ID:role/$ROLE_NAME

#confirm
kubectl get serviceaccount/eks-app-sa -o yaml

Deploy MSK consumer application to EKS

You can refer to the consumer application code here.

Make sure to update consumer application manifest (app-iam.yaml) with the MSK cluster endpoint and ECR image (obtained from the stack output).

kubectl apply -f msk-consumer/app-iam.yaml

# verify Pods
kubectl get pods -l=app=msk-iam-consumer-app

Verify end to end solution

Continue to send records using App Runner producer application:

export APP_RUNNER_URL=<enter app runner URL>

curl -i -X POST -d '{"email":"[e-mail]","name":"user2"}' $APP_RUNNER_URL
curl -i -X POST -d '{"email":"[e-mail]","name":"user3"}' $APP_RUNNER_URL
curl -i -X POST -d '{"email":"[e-mail]","name":"user4"}' $APP_RUNNER_URL

Check consumer app logs on EKS to verify:

kubectl logs -f $(kubectl get pods -l=app=msk-iam-consumer-app -o jsonpath='{.items[0].metadata.name}')

Scale out consumer app

The MSK topic created by the producer application has three topic partitions, so we can have maximum of three consumer instances.
Scale out to three replicas:

kubectl scale deployment/msk-iam-consumer-app --replicas=3

Verify the number of Pods and check logs for each of them. Notice how the data consumption is being balanced across the three instances.

kubectl get pods -l=app=msk-iam-consumer-app

Conclusion

You were able to deploy the end to end application using CDK. This comprised of a producer on App Runner sending data to MSK Serverless cluster and a consumer on EKS persisting data to DynamoDB. All the components were written using the Go programming language!