DEV Community

Cover image for Automate Your AWS MSK Kafka Cluster with Terraform: A Complete Guide
AWS Community Builders profile image Yeshwanth L M
Yeshwanth L M for AWS Community Builders

Posted on

Automate Your AWS MSK Kafka Cluster with Terraform: A Complete Guide

#aws #terraform #kafka #iac

In a previous post, we walked through setting up an AWS MSK cluster manually using the AWS Console. While that's great for learning, it's not repeatable, scalable, or easy to manage. Today, we're taking it to the next level with Infrastructure as Code (IaC).

We'll use Terraform to define our entire MSK environment—VPC, subnets, security groups, IAM roles, the MSK cluster, and even a client EC2 instance—in a single set of configuration files. With a few simple commands, you can create, update, or destroy the whole setup.

What We'll Build:

  • A new VPC with public subnets across three Availability Zones.
  • All necessary networking (Internet Gateway, Route Tables).
  • A secure MSK Kafka cluster.
  • An EC2 instance pre-configured with Kafka tools and the correct authentication settings.
  • IAM roles and security groups that allow the EC2 instance to securely communicate with the MSK cluster.

Let's get started!

🛠️ Prerequisites

Before you begin, make sure you have the following:

  1. An AWS Account: You'll need an AWS account with programmatic access. If you haven't already, configure your credentials locally using the AWS CLI:

    aws configure

  2. Terraform Installed: You'll need the Terraform CLI installed on your machine.

How to Install Terraform

Terraform is easy to install. Here are instructions for common operating systems.

On macOS (using Homebrew):

brew tap hashicorp/tap
brew install hashicorp/tap/terraform
Enter fullscreen mode Exit fullscreen mode

On Windows (using Chocolatey):

choco install terraform
Enter fullscreen mode Exit fullscreen mode

On Linux (Debian/Ubuntu):

wget -O- [https://apt.releases.hashicorp.com/gpg](https://apt.releases.hashicorp.com/gpg) | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] [https://apt.releases.hashicorp.com](https://apt.releases.hashicorp.com) $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform
Enter fullscreen mode Exit fullscreen mode

After installation, verify it's working by running:

terraform --version
Enter fullscreen mode Exit fullscreen mode

📜 Understanding the Terraform Code

Save the code from the prompt into a file named main.tf. Let's break down what each section of our Terraform configuration does.

1. Networking (VPC, Subnets, IGW)

This section builds the foundational network for our resources. We create a new VPC and then provision three public subnets, one in each available AWS Availability Zone for high availability. The Internet Gateway and Route Tables ensure our EC2 instance can reach the internet.

2. Security (Security Groups & SSH Key)

  • SSH Key: Terraform dynamically generates an RSA key pair. The public key is uploaded to AWS (aws_key_pair), and the private key is saved locally as msk-client-key.pem so you can SSH into the EC2 instance.
  • Security Groups: We create two security groups: one for the MSK cluster and one for the EC2 client. The rules are configured to allow the EC2 instance and the MSK cluster to communicate freely with each other on any port, while the EC2 instance only accepts incoming SSH traffic from the internet.

3. IAM Role for EC2

Instead of hardcoding AWS keys, we use an IAM Role. This block creates an aws_iam_role that the EC2 instance can "assume." The attached aws_iam_policy grants the instance specific permissions to connect, describe, read from, and write to the MSK cluster topics. This is the most secure way to grant AWS permissions to services.

4. The MSK Cluster

This is the core resource. We define an aws_msk_cluster with three small broker nodes (kafka.t3.small). Critically, the client_authentication block is configured to use IAM (iam = true), which allows our EC2 instance to authenticate using its assigned IAM role.

5. The EC2 Client Instance

This is where the magic happens! We launch a t2.micro EC2 instance. The user_data script is a powerful feature that runs automatically on the first boot. This script:

  • Installs Java.
  • Downloads and extracts the correct version of Kafka.
  • Downloads the AWS MSK IAM Auth library, which is required for IAM authentication.
  • Creates the client.properties file with the exact configuration needed for our Kafka tools to authenticate with MSK via IAM.

This means that as soon as the instance is ready, it's already fully configured to act as a Kafka client!

🚀 Deploying the Infrastructure

Note: You can find the complete terraform script here: https://github.com/yeshwanthlm/AWS-MSK-Crash-Course/blob/main/terraform/msk-cluster-with-vpc-ec2-client.tf

With the msk-cluster-with-vpc-ec2-client.tf file saved, running the deployment is as simple as three commands.

Initialize Terraform

This command downloads the necessary AWS provider plugin.

terraform init
Enter fullscreen mode Exit fullscreen mode

Plan the Deployment

This is a dry run. Terraform shows you exactly what resources it will create, change, or destroy.

terraform plan
Enter fullscreen mode Exit fullscreen mode

Apply the Configuration

This command executes the plan and builds everything in your AWS account. Type yes when prompted.

terraform apply
Enter fullscreen mode Exit fullscreen mode

Grab a coffee! ☕ The MSK cluster creation is the longest step and will take around 20-30 minutes. Once finished, Terraform will display the outputs, including the public IP of your EC2 instance.

✅ Connecting and Testing Your Cluster

Once terraform apply is complete, let's verify everything works.

Get the EC2 Public IP

Find the public IP from the Terraform output. You can also run terraform output ec2_public_ip.

SSH into the EC2 Instance

The private key msk-client-key.pem was saved in your project directory.

# Make sure to set correct permissions if needed
chmod 400 msk-client-key.pem

ssh -i "msk-client-key.pem" ec2-user@<YOUR_EC2_PUBLIC_IP>
Enter fullscreen mode Exit fullscreen mode

Get Your Bootstrap Brokers String

In your local terminal (not the SSH session), get the connection string from the Terraform outputs.

terraform output bootstrap_brokers
Enter fullscreen mode Exit fullscreen mode

Copy this long string. You'll need it for the next steps.

Create a Kafka Topic

Inside your EC2 SSH session, navigate to the Kafka bin directory and create a topic.

bin/kafka-topics.sh --create \
  --bootstrap-server <bootstrapServerString> \
  --command-config /home/ec2-user/kafka_2.13-3.6.0/bin/client.properties \
  --replication-factor 3 \
  --partitions 1 \
  --topic my-first-topic
Enter fullscreen mode Exit fullscreen mode

Start a Producer

In the same terminal, start the console producer. This will give you a > prompt.

bin/kafka-console-producer.sh \
  --broker-list <bootstrapServerString> \
  --producer.config /home/ec2-user/kafka_2.13-3.6.0/bin/client.properties \
  --topic my-first-topic
Enter fullscreen mode Exit fullscreen mode

Type a message like Hello from Terraform! and press Enter.

Start a Consumer (in a new terminal)

Open a second terminal window and SSH into your EC2 instance again. Navigate to the same bin directory and run the consumer:

bin/kafka-console-consumer.sh \
  --bootstrap-server <bootstrapServerString> \
  --consumer.config /home/ec2-user/kafka_2.13-3.6.0/bin/client.properties \
  --topic my-first-topic \
  --from-beginning
Enter fullscreen mode Exit fullscreen mode

You should instantly see Hello from Terraform! appear in your consumer window. Success! 🎉

🧹 Cleaning Up

Don't forget to tear down your infrastructure to avoid ongoing AWS charges! The beauty of Terraform is that this is a single, simple command.

terraform destroy
Enter fullscreen mode Exit fullscreen mode

Type yes when prompted, and Terraform will neatly remove all the resources it created.

Conclusion

You've successfully automated a complete AWS MSK environment using Terraform. With Infrastructure as Code, you now have a repeatable, version-controlled, and reliable way to manage your Kafka clusters on AWS. This is the foundation for building powerful, event-driven applications at scale.

Top comments (0)