Getting started with MSK Serverless and AWS Lambda using Go

In this blog post you will learn how to deploy a Go Lambda function and trigger it in response to events sent to a topic in a MSK Serverless cluster.

• Prerequisites
• Infrastructure setup
• Send data to MSK Serverless using producer application
• Configure and deploy the Lambda function
• Verify the integration
• Conclusion

The following topics have been covered:

• How to use the franz-go Go Kafka client to connect to MSK Serverless using IAM authentication.
• Write a Go Lambda function to process data in MSK topic.
• Create the infrastructure - VPC, subnets, MSK cluster, Cloud9 etc.
• Configure Lambda and Cloud9 to access MSK using IAM roles and fine-grained permissions.

Prerequisites

You will need an AWS account, install AWS CLI as well a recent version of Go (1.18 or above).

Clone this GitHub repository and change to the right directory:

git clone https://github.com/abhirockzz/lambda-msk-serverless-trigger-golang
cd lambda-msk-serverless-trigger-golang

Infrastructure setup

Create VPC and other resources

Use a CloudFormation template for this.

aws cloudformation create-stack --stack-name msk-vpc-stack --template-body file://template.yaml

Wait for the stack creation to complete before proceeding to other steps.

Create MSK Serverless cluster

Use AWS Console to create the cluster.

Configure the VPC and private subnets created in the previous step.

Create AWS Cloud9 instance

Make sure its in the same VPC as the MSK Serverless cluster and choose the public subnet that you created earlier.

Configure MSK cluster security group

After the Cloud9 instance is created, edit the MSK cluster security group to allow access from the Cloud9 instance.

Configure Cloud9 to send data to MSK Serverless cluster

The code that we run from Cloud9 is going to produce data to the MSK Serverless cluster. So we need to ensure that it has the right privileges. For this, we need to create an IAM role and attach required permissions policy.

aws iam create-role --role-name Cloud9MSKRole --assume-role-policy-document file://ec2-trust-policy.json

Before creating the policy, update the msk-producer-policy.json file to reflect the required details including MSK cluster ARN etc.

aws iam put-role-policy --role-name Cloud9MSKRole --policy-name MSKProducerPolicy --policy-document file://msk-producer-policy.json

Attach the IAM role to the Cloud9 EC2 instance:

Send data to MSK Serverless using producer application

Log into the Cloud9 instance and run the producer application (its a Docker image) from a terminal.

export MSK_BROKER=<enter the MSK Serverless endpoint>
export MSK_TOPIC=test-topic

docker run -p 8080:8080 -e MSK_BROKER=$MSK_BROKER -e MSK_TOPIC=$MSK_TOPIC public.ecr.aws/l0r2y6t0/msk-producer-app

The application exposes a REST API endpoint using which you can send data to MSK.

curl -i -X POST -d 'test event 1' http://localhost:8080

This will create the specified topic (since it was missing to begin with) and also send the data to MSK.

Now that the cluster and producer application are ready, we can move on to the consumer. Instead of creating a traditional consumer, we will deploy a Lambda function that will be automatically invoked in response to data being sent to the topic in MSK.

Configure and deploy the Lambda function

Create Lambda execution IAM role and attach the policy

aws iam create-role --role-name LambdaMSKRole --assume-role-policy-document file://lambda-trust-policy.json
aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaMSKExecutionRole --role-name LambdaMSKRole

Before creating the policy, update the msk-consumer-policy.json file to reflect the required details including MSK cluster ARN etc.

aws iam put-role-policy --role-name LambdaMSKRole --policy-name MSKConsumerPolicy --policy-document file://msk-consumer-policy.json

Build and deploy the Go function and create a zip file

Build and zip the function code:

GOOS=linux go build -o app
zip func.zip app

Deploy to Lambda:

export LAMBDA_ROLE_ARN=<enter the ARN of the LambdaMSKRole created above e.g. arn:aws:iam::<your AWS account ID>:role/LambdaMSKRole>

aws lambda create-function \
    --function-name msk-consumer-function \
    --runtime go1.x \
    --zip-file fileb://func.zip \
    --handler app \
    --role $LAMBDA_ROLE_ARN

Lambda VPC configuration

Make sure you choose the same VPC and private subnets as the MSK cluster. Also, select the same security group ID as MSK (for convenience) - if you select a different one, make sure to update MSK security group to add an inbound rule (for port 9098), just like you did for the Cloud9 instance in an earlier step.

Configure the MSK trigger for the function

Make sure to choose the right MSK Serverless cluster and enter the correct topic name.

Verify the integration

Go back to the Cloud9 terminal and send more data using the producer application

I used a handy json utility called jo (sudo yum install jo)

APP_URL=http://localhost:8080

for i in {1..5};
    do jo email=user${i}@foo.com name=user${i} | curl -i -X POST -d @- $APP_URL; 
    done

In the Lambda function logs, you should see the messages that you sent.

Conclusion

You were able to setup, configure and deploy a Go Lambda function and trigger it in response to events sent to a topic in a MSK Serverless cluster!