DEV Community

Cover image for A Beginner's Guide to AWS MSK: From Cluster Setup to Your First Message
AWS Community Builders profile image Yeshwanth
Yeshwanth for AWS Community Builders

Posted on

A Beginner's Guide to AWS MSK: From Cluster Setup to Your First Message

#aws #kafka #msk #devops

🚀 Getting Started with AWS MSK: Your First Kafka Cluster

Ever wondered how massive, data-driven apps handle real-time event streams for things like live analytics, log aggregation, or IoT data? A key technology behind this is Apache Kafka, a powerful open-source distributed event streaming platform.

However, setting up and managing Kafka on your own can be a complex and time-consuming task. This is where AWS Managed Streaming for Apache Kafka (MSK) comes in.

In this guide, we'll cover:

  • What AWS MSK is and why it's useful.
  • The different cluster types available.
  • A full, hands-on tutorial to create your own MSK cluster and send your first messages from an EC2 instance.

Let's dive in!

## 🤔 What is AWS MSK?

Think of Apache Kafka as a high-speed, central post office for your application's data. 📮 Applications can send messages (produce) to different mailboxes (topics), and other applications can pick them up (consume) when they're ready.

AWS MSK is a fully managed service that runs this post office for you. It handles the heavy lifting so you don't have to:

  • Provisioning Servers: No need to pick, set up, or configure EC2 instances.
  • Kafka Software Management: AWS handles the installation, patching, and upgrades of Kafka.
  • High Availability: MSK automatically distributes your cluster across multiple data centers (Availability Zones) to ensure it's resilient to failure.

In short, you get the full power of Apache Kafka without the operational overhead.

## ✨ Why Do You Need AWS MSK?

So, why choose MSK over managing Kafka yourself?

  • ✅ Simplified Operations: Spend your time building applications, not managing infrastructure.
  • 🌐 Highly Available & Scalable: MSK is built for resilience. You can easily scale your cluster's compute and storage with a few clicks and no downtime.
  • 🔒 Secure by Default: Integrates seamlessly with AWS services like IAM for authentication, VPC for network isolation, and KMS for encrypting your data at rest and in transit.
  • 💯 Fully Compatible: It's 100% compatible with open-source Apache Kafka. You can migrate existing applications, tools, and plugins without changing your code.

## Cluster Types: Provisioned vs. Serverless

When creating a cluster, MSK gives you two options:

### Provisioned Clusters

This is the traditional model. Think of it like leasing a fleet of trucks. 🚚 You choose the size and number of trucks (broker types and count), and you have full control over the configuration.

  • Best for: Predictable, high-volume workloads where you want fine-grained control.
  • You pay for: The resources you provision, 24/7.

### Serverless Clusters

This is a newer, more flexible option. Think of it like a pay-per-package delivery service. 📦 You don't manage any trucks; you just send your data, and the service automatically scales to handle the load.

  • Best for: New apps, or workloads with variable or unpredictable traffic.
  • You pay for: The data you stream and retain (throughput and storage).

For this tutorial, we'll use a Provisioned cluster to see all the underlying configurations.

## 🛠️ Hands-On Demo: Creating Your Cluster and Sending Messages

Time to build! We'll create an MSK cluster and an EC2 instance to communicate with it.

### Part 1: Create the MSK Cluster

  1. In the AWS Console, navigate to MSK.
  2. Click Create cluster and choose the Custom create method.
  3. Cluster settings:
    • Cluster name: my-demo-msk-cluster
    • Cluster type: Provisioned
    • Apache Kafka version: Use the recommended default.
  4. Networking:
    • Select your desired VPC.
    • Choose at least two Availability Zones and select a subnet in each. For a simple demo, public subnets are fine, but use private subnets for production.
  5. Security:
    • Under Access control methods, check the box for IAM role-based authentication. This is the most secure and straightforward way to connect from other AWS services.
  6. Review and Create. The cluster will take 20-30 minutes to become Active.

### Part 2: Set Up the EC2 Client & Security Groups

While the cluster is creating, let's set up our client machine.

  1. Launch an EC2 Instance:

    • Go to the EC2 service and click Launch instance.
    • Name: MSK-Client-EC2
    • AMI: Amazon Linux 2
    • Instance Type: t2.micro (Free Tier eligible)
    • Network: Crucially, select the same VPC and one of the subnets you used for your MSK cluster.

    • IAM Role: Attach an IAM role to the instance with a policy that allows it to connect to MSK. A simple policy for this demo would be:

      {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "kafka-cluster:*",
            "Resource": "<Your-MSK-Cluster-ARN>"
        }
    ]
}

  2. Configure Security Groups (The Important Part!):

    • MSK Cluster Security Group: Find the security group attached to your MSK cluster. Add an inbound rule to allow All traffic from the security group of your MSK-Client-EC2 instance.
    • EC2 Instance Security Group: Find the security group for your EC2 instance. Add an inbound rule to allow All traffic from the MSK cluster's security group. Also, make sure you have a rule to allow SSH from your IP.

This two-way rule allows the EC2 instance and the MSK brokers to communicate freely within the VPC.

### Part 3: Connect and Send Messages

Once your MSK cluster is Active and your EC2 instance is running, SSH into the instance.

  1. Install Tools:

    # Update and install Java
sudo yum update -y
sudo yum install java-1.8.0-openjdk -y

# Download and extract Apache Kafka tools
wget [https://archive.apache.org/dist/kafka/2.8.1/kafka_2.13-2.8.1.tgz](https://archive.apache.org/dist/kafka/2.8.1/kafka_2.13-2.8.1.tgz)
tar -xzf kafka_2.13-2.8.1.tgz
cd kafka_2.13-2.8.1/bin

  2. Create Client Properties File:
    We need to tell the Kafka tools to use IAM for authentication.

    # Create the config file
cat << EOF > client.properties
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
EOF

  3. Get Bootstrap Servers:
    In the MSK console, click your cluster, then View client information. Copy the Bootstrap servers endpoint for IAM.

  4. Create a Topic:
    Let's create a "mailbox" called my-first-topic. Replace <YOUR_BOOTSTRAP_SERVERS> with the endpoint you just copied.

    ./kafka-topics.sh --create --topic my-first-topic \
--bootstrap-server <YOUR_BOOTSTRAP_SERVERS> \
--command-config client.properties

  5. Start a Producer:
    This command gives you a prompt where you can type messages to send.

    ./kafka-console-producer.sh --topic my-first-topic \
--broker-list <YOUR_BOOTSTRAP_SERVERS> \
--producer.config client.properties

    Type Hello MSK! and hit Enter. Type a few more messages.

  6. Start a Consumer (in a new terminal):
    Open a second SSH session to your EC2 instance, navigate to the same bin directory, and run:

    ./kafka-console-consumer.sh --topic my-first-topic \
--bootstrap-server <YOUR_BOOTSTRAP_SERVERS> \
--consumer.config client.properties \
--from-beginning

You should see the messages you typed in the producer terminal appear instantly! 🎉

## Conclusion

Congratulations! You've successfully deployed a highly available Apache Kafka cluster using AWS MSK, configured secure access from an EC2 instance, and sent your first real-time messages.

By using MSK, you get to leverage the power of Kafka for your event-driven applications without the headache of managing the underlying infrastructure.

Thanks for reading! Let me know in the comments if you have any questions.

aws #kafka #cloud #devops #tutorial

Top comments (0)